Repository navigation
feat(mysql): read each column's type modifier - #192
colinhacks wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
ℹ️ No blocking issues. One gap in the soundness argument, inline.
Reviewed changes
I reviewed the whole diff: the MySQL engine now gives a table's own result column the type its definition declares, and the oracle checks values against those bounds.
mysql_types::declared: narrowschar/varchar, theTEXTs,binary/varbinary/theBLOBs,DECIMAL, and the ticks ofDATETIME/TIMESTAMP/TIME, using the prepare's length and decimals. I checked themax_bytescollation table against every collation on a MySQL 8.0.46 server and found no mismatch and no extra entry.table_column/defined_type: these apply the narrowing only where the column names its table and that table is not a view. The views are now fetched whenever any column names a table. That also stopssettlefrom treating a view's column as the table's own.- UTF-8
TEXT: aTEXTis counted in UTF-8 bytes only whenTYPES_QUERY's newcharacter_set_namesays the column is UTF-8. Otherwise it is counted in code points, including through a derived table and inside a JSON document. - Describe protocol and fingerprint: the facade sends
decimals, and the MySQL fingerprint now includes each column's character set. - Oracle:
value_contradictionandscalar_contradictioncheck each value's length, digits and tick, in a column and inside a document. The seed adds strings at each column's longest length and times with six fractional digits.
I ran a throwaway MySQL 8.0.46 server to test the claim that an expression never names a table. It holds for a materialized derived table, a WITH query, a recursive WITH query and a union inside a derived table: each one's expression columns come back with an empty org_table. So a group_concat read through a derived table is not narrowed. The one exception I found is inline.
claude-opus-5-5 ｜ 𝕏
9e4ab13 to
8801751
Compare
There was a problem hiding this comment.
✅ No new issues found. The
information_schemagap is closed.
Reviewed changes
I reviewed what changed since 9e4ab13. Most of the range-diff comes from the rebase: main moved the short notation from either(a, b) to a | b, which rewrote many expected.txt lines. The real changes are:
- Added
stored_column: a column now takes its definition only where the prepare names a schema, and that schema is notinformation_schema. This covers both kinds ofinformation_schemacolumn: the expression columns, whose schema is empty though they name their view, and the columns read from the data dictionary. - Added
schemato the describe protocol: the facade sendsc.schema_str(), andMysqlColumn.schemaandMysqlDescribed.schema?carry it. A host that sends no schema gets no narrowing, which is the sound default, and the unit test holds that. - Stopped narrowing JSON_TABLE's columns: they name no schema, so
json_table.sqlnow typesvarcharanddecimalwith no bound. An omitted bound is sound. - Added
information_schema.sqltomysql_type_modifiers, which holdsTABLE_COMMENTandCREATE_OPTIONSto no bound.
I also checked that the UTF-8 TEXT lookup cannot be fooled by a table of another schema with the same name. TYPES_QUERY reads only database(), but traced_from traces nothing once a table name is schema-qualified. Such a column therefore falls back to counting code points.
claude-opus-5-5 ｜ 𝕏
A column stored under a definition the prepare reports now carries it: a CHAR or VARCHAR's length in code points, a TEXT's capacity (in UTF-8 bytes where the catalog says it stores UTF-8), a BINARY's, VARBINARY's or BLOB's bytes, a DECIMAL's precision and scale, and the fractional seconds of DATETIME, TIMESTAMP and TIME. An expression's, a view's or a union's reported length can be a display width, and JSON_TABLE's columns and information_schema's own views name no schema or a computed one, so each is left out. The oracle holds every MySQL value to its length, digits and tick.
7809293 to
b5b3bab
Compare
The MySQL engine dropped what a column declares. A column stored under its definition now carries it, read from the prepare's length, decimals and character set, as the Postgres engine has since #180.
Before and after, by type
varchar(20){ kind: "text", name: "varchar" }{ kind: "text", size: { max: 20 }, name: "varchar" }char(5){ kind: "text", name: "char" }{ kind: "text", size: { max: 5 }, name: "char" }textinutf8mb4orutf8mb3, read straight from its table{ kind: "text", name: "text" }{ kind: "text", size: { unit: "byte", encoding: "UTF-8", max: 65535 }, name: "text" }textin another character set, or read through a derived table or inside a JSON document{ kind: "text", name: "text" }{ kind: "text", size: { max: 65535 }, name: "text" }binary(3){ kind: "bytes", name: "binary" }{ kind: "bytes", size: { exact: 3 }, name: "binary" }varbinary(8){ kind: "bytes", name: "varbinary" }{ kind: "bytes", size: { max: 8 }, name: "varbinary" }blob{ kind: "bytes", name: "blob" }{ kind: "bytes", size: { max: 65535 }, name: "blob" }decimal(10,2){ kind: "decimal", name: "decimal" }{ kind: "decimal", precision: 10, min_scale: 2, max_scale: 2, name: "decimal" }datetimetick: "1/1000000"tick: "1"datetime(3)tick: "1/1000000"tick: "1/1000"timestamp(2)tick: "1/1000000",max: "2038-01-19T03:14:07.999999Z"tick: "1/100",max: "2038-01-19T03:14:07.99Z"time(4){ kind: "duration", max: "PT838H59M59S", tick: "1/1000000", name: "time" }{ kind: "duration", max: "PT838H59M59S", tick: "1/10000", name: "time" }information_schemaThe
tinytext,mediumtextandlongtexttypes carry 255, 16777215 and 4294967295, and so do the matching BLOBs.Where a modifier holds
Rule 2 of the spec says a bound appears only when it is true of every value.
group_concatfollows the session'sgroup_concat_max_len, and a placeholder's text has no length at the prepare. A CAST's bound does hold, and is left out here because the engine would have to read it from the tree.information_schematakes none. The views ofinformation_schemaare in no list of views, and name their view as the column's table. An expression among them names no schema (TABLES.TABLE_COMMENT, whose reported length is 8192), and the rest nameinformation_schemaand are read from the data dictionary, not stored under that definition. JSON_TABLE's columns name no schema either, so they take none, though their declared types do hold: an omitted bound is sound.CHARis at most n, not exactly n, since MySQL strips its padding as it reads one. ABINARY(n)is exactly n, since MySQL pads it with zero bytes and keeps them.TEXTcounts UTF-8 bytes only where the catalog says it stores UTF-8. Alatin1TEXTof 65535 bytes can reach 131070 bytes as UTF-8, and the prepare reports the same length for both. Every otherTEXTis bounded in code points, which holds for any character set, since each character takes at least one byte. Inside a JSON document or through a derived table the column's origin may be a derived name, so no catalog lookup is made and the bound is in code points.MySQL counts a string's length in bytes of the character set it is sent in, so
varchar(20)sent asutf8mb4is 80. The engine divides by that set's widest character, and a collation number MySQL 8.4 does not name leaves the bound out.Other changes
decimalsandschemabesidelength(engine/facade/,src/engine-core.ts).TEXT's size now turns on it. A.reylo/entry from before this is analyzed again.column_type, which holds every modifier but the character set. ATEXTconverted to another character set after the build is not caught there.Oracle checks
The oracle now holds every MySQL value to its type, in a column and inside a JSON document: its length in the size's unit, its digits against precision and scale, and its fraction against its tick. The seeding fills each column to its longest value in characters of one, two and four bytes, the longest string of bytes, and each time with six fractional digits, which a column of fewer rounds to its own.
The new case is
oracle/cases/mysql_type_modifiers. To check the checks, each bound in its analyses was tightened by one step and fed back through--analyses, and the oracle reported 104 contradictions, covering every column with a value at its bound. That includesTIME's tick in a column and in a document.In the 29 existing cases that change, 247 lines of
expected.txtdiffer. With the modifiers stripped each one equals main's line, so each change only adds a modifier:varchar(≤N)text(≤N)text(≤N bytes UTF-8)decimal(p,s)varbinary(≤N)blob(≤N)char(≤N)binary(=N)Ticks do not print in that notation, so the engine's tests and the case's dump hold them.
Checks run
Run locally, on MySQL 8.4.11 and Postgres 18:
--strict-coverage, none failed.cargo test -p reylo-engine-mysql -p reylo-oracle.cargo build --release -p reylo, thennub run typecheckandnub run teston it: 388 passed,skipped 0.Left to CI on this head: the Postgres, SQLite and DuckDB oracle shards, the other crates' tests,
cargo build --workspace --all-targets, the playground's engine and the package build.Left out
CAST's bound and JSON_TABLE's, as above.VECTOR(n)'s size, since MySQL 8.4 has no such type and the oracle cannot hold it.UNSIGNEDDECIMAL'sminof 0, which is no type modifier.spec/examples.ts, unchanged. It agrees with the engine but fortags, which writessize: { max: 2 }, andnext, whose members the engine names (bigint,double,decimal). Both differed before this.