Repository navigation
feat(sqlite): type the full-text search functions - #32
Conversation
bm25 is a real, highlight and snippet are text that may be null, and FTS4's offsets and matchinfo are text and a blob; each was unknown.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Important
bm25 with weight arguments can return NULL, but it is typed as never null.
Reviewed changes
I reviewed the whole PR: the FTS function typing in the SQLite analyzer, its parameter contexts, the native and oracle tests, and the docs rule.
- Result types:
bm25is a non-nullreal.highlightandsnippetare nullabletext.offsetsis a non-nulltextandmatchinfoa non-nullblob. All five are inPLAIN, so they carry no JSON subtype. - Parameter contexts:
highlight's column parameter is typedintegerand its marker parameterstext. Abm25weight is typedreal. - Tests: a native test
the_full_text_functions_are_typed, three newsqlite_ftsoracle statements and an FTS4notestable insetup.sql. - Docs: a "Full-text search" rule on the SQLite reference page.
I tested the FTS4 and no-MATCH claims on SQLite 3.45.1 and they hold. offsets, matchinfo and FTS4 snippet return '' or an empty blob, never NULL. FTS5 highlight and snippet are NULL exactly where their column is.
claude-opus-5-5 ｜ 𝕏
An infinite or overflowing weight makes the score NaN, which SQLite returns as NULL.
There was a problem hiding this comment.
✅ No new issues found. The weighted
bm25feedback is addressed cleanly.
Reviewed changes
I reviewed 0384c7d, the single commit since the prior review.
- Weighted
bm25nullable:bm25(t)stays a non-nullreal. With weight arguments it is now a nullablereal, chosen by the argument count the same wayjson_typeandjson_removealready do it. - Tests and docs follow: the native test asserts
weightednullable, thesqlite_ftsexpectation readsweighted: real?, and the docs rule gains abm25(docs, 1e999, 1.0)→nullpair.packages/sqlite/engine.wasmis rebuilt.
I checked the new docs pair on SQLite 3.45.1. bm25(docs, 1e999, 1.0) is NULL on a row that matches once in the weighted column, and bm25(docs) is a real on every row, with or without MATCH.
claude-opus-5-5 ｜ 𝕏
FTS5's
bm25is now typednumber,highlightandsnippetstring | null, and FTS4'soffsetsstringandmatchinfoUint8Array. Each wasunknown:A parameter passed as
highlight's column (integer) or markers (text), or as abm25weight (real), is typed too. Inside built JSON each nests as the value it is.Why these types. Measured on SQLite 3.53.4 over rows with a NULL in each column:
highlightand FTS5'ssnippetreturn NULL exactly where the column they read is NULL;bm25is a real on every row, 0 where no MATCH ranks it;offsetsandmatchinfoanswer every row. FTS4'ssnippetshares the name, sosnippetstays nullable.Checks
the_full_text_functions_are_typedinengine/sqlite.sqlite_ftsoracle case gains three statements (ranked and highlighted FTS5 matches, the functions with no MATCH, an FTS4 table), run over seeded edge values with none left unchecked.packages/sqlite/engine.wasmrebuilt.