Repository navigation
test(spec): hold the examples and every inferred type to the format - #188
Open
colinhacks wants to merge 4 commits into
Open
colinhacks wants to merge 4 commits into
colinhacks wants to merge 4 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
ℹ️ No critical issues — one small coverage gap inline.
Reviewed changes
Reviewed the whole PR at 814cf10: the spec's examples now run on each engine, the oracle checks every type it infers, and core's tests read one shared copy of the examples.
- Examples carry their SQL: each export of
spec/examples.tsis now anExamplewith itssetupandstatement.tests/spec-examples.test.tsruns the Postgres, MySQL, SQLite and DuckDB ones throughreylo/rtd'sanalyzeand compares every field each example shows. Several examples were corrected to what the engine actually prints. spec/examples.jsongenerated:spec/generate.tswrites it from the module's exports, and--check(intests/typedefs.test.ts) keeps it current. Core'stypes.rstests round-trip it and runcheckon every type in it.- Oracle validity check: the new
oracle/src/validity.rs(built onboon) holds each parameter's and column's type to$defs/Type, a serde round trip andType::check. A violation fails the case, and in a replay it counts as a contradiction. - CI: the test job's Postgres is now
pgvector/pgvector:pg18. A change tospec/typedefs.schema.jsonorspec/examples.jsonnow runs every engine check.
claude-opus-5-5 ｜ 𝕏
Contributor
There was a problem hiding this comment.
✅ No new issues found. The earlier coverage gap is closed.
Reviewed changes
Reviewed ca0015e, the one commit added since the review at 814cf10.
- Restored literal round trips for kinds no example shows: core's
types.rstests now round-tripuuidagain, and alsodatetime(with its requiredtick),multirangeandany. No export ofspec/examples.tsholds these four kinds, so now an example or a literal line reads every kind in the schema.
claude-opus-5-5 ｜ 𝕏
Each example now carries its schema and statement. A test runs them on the database and compares every field the example shows with what the engine writes, so an example can no longer promise what the engine does not write.
spec/examples.json, which the spec's generator writes, replaces the subset core's tests copied by hand.
Each parameter's and column's type must be a Type of the JSON Schema, read and write back as the same JSON, and pass rule 7. Nested JSON shapes, unions and structured parameters are built per statement, so no table of built-in types covers them.
colinhacks
force-pushed
the
spec-held-to-engine
branch
from
November 5, 2026 17:27
ca0015e to
b0f7969
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This makes the tests fail whenever the type descriptor spec promises something the engine does not print.
For example, the spec's MySQL example promised that a
varchar(100)column printssize: { max: 100 }. Nothing checked that, and the engine printed no size at all. Now a test runs each example's DDL and statement through the engine and fails if the result differs from the example. Its first run found 12 such differences, listed below.Two smaller changes go with it: the soundness oracle checks every type it infers against the format's own rules, and the Rust tests read the same examples instead of keeping a hand-copied subset.
What is now checked
tests/spec-examples.test.ts). Each example inspec/examples.tsnow carries its schema and statement beside its descriptor. For Postgres, MySQL, SQLite and DuckDB, the test runs them on an empty database of the example's own through the root module'scompileand compares every field the example shows, all but the server's version, and holds the whole descriptor to the JSON Schema with no diagnostic. SQL Server's and Oracle's stay hand-written, and the file says so. Compared rather than generated, so the file keeps its shared constants and its prose.oracle/src/validity.rs). Every parameter's and column's type of every analysis the oracle makes, in its cases and its replays, must be aTypeofspec/typedefs.schema.json, read and write back as the same JSON, and pass rule 7 (Type::check). A violation fails the case, or counts as a contradiction in a replay.spec/examples.json).nub run typedefswrites it fromspec/examples.tsand--checkholds it current. Core's tests read it, round-trip andcheckevery type in it, and keep literal JSON only for what no example shows.Other changes this brings
pgvector/pgvector:pg18, Postgres 18 with the extension installed, since CI fails on a skip and thevectorexample needs it. It is its own example, so everything else runs on any Postgres, and it skips where the extension is missing.boon, a JSON Schema 2020-12 validator, and eight crates under it. The oracle is not published and thereylobinary does not link them.spec/typedefs.schema.jsonorspec/examples.jsonnow runs every engine check, since the engine's tests and the oracle read them.Examples changed, where the engine is right
first_example,sqlselect id, email, bio from users where id = $1-- One user, by id.line: the field is the statement as the database runs itmysql_columns,next(:n + 1)bigint,doubleanddecimal: a type'snameis the database's name for itsqlite_columns,price(price numeric)name: "NUMERIC"name: "numeric": SQLite reports a declared type as written, but for its own words (INT,INTEGER,REAL,TEXT,BLOB)structured_param, its statementrows(kind varchar(32), at timestamptz?), which is a syntax error and leaveskindnullablerows(kind varchar(32) not null, at timestamptz); the descriptor it shows is unchangedExamples changed, where the engine is less precise
Each example now says what the engine writes today, with a note where the gap is. None is unsound: each type holds every value the column can.
eemail, a domain:varchar(255) check (value like '%@%')size: {max: 255},pattern: {dialect: "like", source: "%@%"},name: "public.email"size: {max: 255},name: "varchar": the server describes a domain's column by its base type, and no engine writespatternattimestamp(3)tick: "1/1000",max: "2038-01-19T03:14:07.999Z"tick: "1/1000000",max: "2038-01-19T03:14:07.999999Z"statusenum('b','a')enum: ["b", "a"]enumnamevarchar(100)size: {max: 100}sizebodytextsize: {unit: "byte", encoding: "UTF-8", max: 65535}sizetagsset('a','b')size: {max: 2}, elementenum: ["a", "b"]size, element plain textx{kind: "unknown"}nintegerin a STRICT tableintegeraloneINTEGER#186 makes
nintegeralone, and whichever of the two lands second updates the example.What the oracle found
Every type passed in the 170 cases. The replays found one limitation:
--replay postgresreports 8 contradictions, because the engine writes a Postgres range over an array, a composite or avarbitas it is, and the schema'sOrderedTyperefused those subtypes. It is settled: #177 letssubtypeonrangeandmultirangebe anyType, so the 8 go away when it lands. No case inoracle/caseshas such a type, so CI's oracle passes without it.What ran
cargo build --workspace --all-targets, no warning, andcargo testof every engine crate and the oracle.nub run typecheck;nub run teston Postgres 18.6 and MySQL 8.4.11: 365 passed, none skipped.--strict-coverage, on Postgres 18.6 with pgvector, PostGIS and the five validators, MySQL 8.4.11, SQLite and DuckDB (--features duckdb): 170 cases, none failed and no violation.Rust cannot catch foreign exceptions, before its report, and printed no violation up to there.--analysesfails its case under all three rules.