Repository navigation
perf(engine): columnar fast path for exists-in-filter conjunction (#1183 finalize verify) - #1234
Merged
Merged
Conversation
#1183's finalize-verify step issues one count per changed group of the shape bool:[term: ax_file, exists: <field>]. In scoring context (must) the bool planner had NO leaf for exists, so the whole plan bailed to the brute path and source-scanned every row per call. Measured on the standing xerj-search node (:9200, index xc-xerj-search-...-lucene-docs, 91,143 docs): bool[term+exists] took=9638ms for 0 hits, while the bare term alone took=0ms with 7 survivors. The autoindex finalize loop pays that once per changed group per finalize cycle; the audit log showed 11.1s queries of the same shape (#1183). Fix (engine): lower filter-context exists into the scored columnar executor (filter_leaf, three arms): - keyword fields -> KeywordPrefix{prefix:""} (empty-prefix dictionary range; null rows excluded by FilterEval's null check) — same lowering the standalone root exists already uses. - numeric/boolean -> NumericWindow{±inf, inclusive} — likewise. - text/semantic_text (no doc-values column) -> new SourceExists leaf: per-segment cached stored slices (stored_slices_for), predicate = get_field_value + value_present — the SAME predicate the brute scan applies, so the hit set cannot drift. The row walk evaluates fev in order with short-circuit, so term+exists pays one stored-source parse per TERM survivor (7 on the measured segment), not per row. - meta fields (_id/_index/_seq_no/_version/_primary_term/_routing) keep the brute path: their exists semantics are envelope-derived, not source-derived. constant_plan gate now excludes plans carrying a SourceExists leaf (no closed-form count; FilterEval::count has an unreachable arm with the invariant documented). Client (xerj-autoindex): exact_semantic_count switches must -> filter context with the term first, to ride the new columnar order. Numbers: before measured 9638ms/0 hits on the 91k-doc segment vs 0ms for the bare term; after, the walk is a term-column scan + <=survivors source parses. Live after-number to be posted on #1183 when the standing rebuild node is swapped to this binary (tracked there). Tests: tests/exists_filter_conjunction.rs — 90-doc fixture covering present/absent/null/empty-string/array edges for text exists, keyword + numeric exists with nulls, must_not exists inversion, filter-context zero-score paging, and memtable-resident (brute) agreement; the scoring-context must variant must report the same total (cross-path drift guard). 5/5 pass; lib suite 746/0; clippy -D warnings clean. Reference-coding note: retrieval attempted on the xerj-search corpus twice; top hits were ES ESQL parquet readers (AGPL, approach-only) and the index is flood-damaged — fell back to this repo's own code per the mandate's own rule. Files: - crates/xerj-engine/src/index.rs: filter_leaf exists arms, SourceExists leaf, FilterEval::Source eval + count arm, constant_plan gate - crates/xerj-autoindex/src/sync_executor.rs: exact_semantic_count filter context - crates/xerj-engine/tests/exists_filter_conjunction.rs: new
Merged
This was referenced Oct 8, 2026
Merged
thomas-villani
pushed a commit
to thomas-villani/xerj
that referenced
this pull request
Oct 9, 2026
Window: xerj-org#1234 (exists-in-filter columnar fast path, xerj-org#1183), xerj-org#1240 (the exploit-hub A/B eval post + xerj-blogposts at 8 records), xerj-org#1241 (xerj-org#1238 per-source-index cap + text^0.5 recall leg). - engine/Cargo.toml 1.0.0-rc.90 -> 1.0.0-rc.91, Cargo.lock refreshed offline (58 members re-pinned). - CHANGELOG: rc.91 section. The xerj-org#1234 entry carries the stays-open marker for xerj-org#1183: the fix covers the scored lane; the count lane (finalize-verify) still brute-scans, measured 2026-10-08 on the 572,992-doc elasticsearch-docs index (fresh ax_file value: bare term 1.2 s, term+exists 13.9-16.2 s, exact repeat 0.02 s off the query cache; present and absent values cost the same). - ROADMAP: review-line roll to rc.91; same ten open; xerj-org#1238 recorded closed-inside-window; xerj-org#1183 gains the measured-mechanism paragraph. - Landing stamps RC.90 -> RC.91 across the 54 stamped pages (both patterns: data-latest-tag span + footer version); sitemap regenerated, no drift (185 URLs). - xerj-blogposts hub manifest on main mirrors the registry bump of xerj-org#1242: sha a9ad4fc, files 8, bytes 118,568 (read off the build's own corpus.json, not recomputed).
thomas-villani
pushed a commit
to thomas-villani/xerj
that referenced
this pull request
Oct 9, 2026
…digest windows (xerj-org#1183 count lane) The generation-wide verify barrier read changed groups back one group at a time: a term count per (group x index), a term+exists count per (group x dataset-with-semantic-field), and a catalog count per group - three serial round trips per file. Measured on the live crawl nodes (2026-10-08): cve-records finalize-verify ran at 2.2 groups/s across 402,695 changed groups (~50 h projected, :19285, 236 indices), vuln-fix-commits at 0.06/s across 29,258 groups (~120 h projected, :19295, 5,953 indices) - both against index phases that take hours, i.e. the barrier, not the writing, was the tail of every large corpus build (xerj-org#1183's count lane; the scored lane was fixed by xerj-org#1234 in rc.91). Every question has the same shape - 'how many rows in this index carry one of these content digests' - so they now batch. sync_executor.rs gains VERIFY_WINDOW=1024 and windowed_value_counts: one terms filter plus one exact terms aggregation per (index x 1,024 digests); the engine's terms doc_count is precise (no sketch in the metric path), so each bucket is the number the per-digest term search returned, verified live: term == bucket on sampled digests, and 524 absent digests in a 1,024 window produce no bucket and keep their seeded zero. The catalog leg cannot aggregate: the catalog is one global index (402,814 docs on the cve node) whose version map carries delete events - dataset documents are rewritten by every finalize-catalog - and the engine's columnar agg fast path refuses exactly that, so the brute agg corpus deep-clones the whole index (measured 786.7 MB estimate vs the 512 MB breaker -> 429, the xerj-org#1183 shape). windowed_catalog_counts therefore fetch-counts the matched set instead: _source-restricted paged fetch, 31 ms cold / 14 ms warm per window on that same catalog, delete-aware, with a mid-window total change and a per-window count/total mismatch both failing loud. Measured per-window costs on the live cve node: record windows 6-63 ms (columnar fast path, append-only indices), semantic windows ~16 s (the fast path does not yet columnarize exists; queued as the engine follow-up that takes this to single-digit ms). Net for cve-records: ~1.2 M serial searches collapse to ~800 windows, and the finalize tail drops from a projected ~50 h to low hours. The phase keeps its name and stays honest about progress: the denominator is now the number of read-back windows, ticked as each resolves (one pr.file per index/catalog window), so the line never sits at 0/N for the length of the walk (xerj-org#931's rule). The four failure messages are unchanged verbatim, and the per-group check order is unchanged, so the first disagreement names the same group it always did. xerj-org#971 group-equality-skip keeps unchanged groups out of the windows entirely. Tests: the HTTP fakes now evaluate a terms aggregation over the matched set (both the reconcile and the label fakes) and honour exists inside bool.filter, mirroring engine semantics; the xerj-org#1183 agg-free guard is narrowed to what it actually protects - no corpus-materialising agg (min/max on a time field) anywhere, and any terms agg must ride its digest-window filter - with the dataset read-back itself still agg-free. 1299 pass, 0 fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
existsinside a filter-contextboolhad no arm in the columnar filter executor, so the whole plan bailed to the brute path and source-scanned every row. This lowers it and adds a source-backed leaf for column-less fields.Why (#1183)
The autoindex finalize-verify step issues one count per changed group of the shape
bool:[term: ax_file, exists: <field>]. Measured on the standing xerj-search node (91,143-doc segment):boolmust[term: ax_file, exists: body]term: ax_fileOne such call per changed group per finalize cycle; the audit log showed 11.1 s queries of the same shape.
How
exists→ empty-prefix dictionary range (KeywordPrefix{prefix:""}); numeric/boolean → unboundedNumericWindow— the same lowerings the standalone rootexistsalready usessemantic_text(no doc-values column) → newSourceExistsleaf: per-segment cached stored slices, predicate =get_field_value+value_present— the same predicate the brute scan applies, so the hit set cannot drift. The row walk evaluates leaves in order with short-circuit, soterm + existspays one stored-source parse per term survivor (7 on that segment), not per row_id/_index/_seq_no/_version/_primary_term/_routing) stay on the brute path (envelope-derived semantics)constant_plangate excludes plans carrying aSourceExistsleaf (no closed-form count);FilterEval::countdocuments the invariantexact_semantic_countmovesmust→filterwith the term first (columnar order)Numbers
Before is measured (9638 ms / 0 hits vs 0 ms bare term, above). The after-number on the live 91k-doc segment will be posted on #1183 when the standing rebuild node is swapped to this binary — that swap is already tracked there. Locally: 5/5 new tests, lib suite 746 passed / 0 failed, clippy
-D warningsclean,bool_filter_is_non_scoring9/9.Tests
New
tests/exists_filter_conjunction.rs: 90-doc fixture covering present / absent / explicit-null / empty-string (PRESENT) / array-with-non-null (PRESENT) edges; keyword + numericexistswith nulls;must_not existsinversion; filter-context paging at score 0.0; memtable-resident (brute path) agreement. The scoring-contextmustvariant must report the same total — cross-path drift guard between the columnar leaf and the bruteexistsarm.Reference-coding note: retrieval attempted on the xerj-search corpus (twice); top hits were ES ESQL parquet readers (AGPL, approach-only) and the index is flood-damaged — fell back to this repo's own code per the mandate's own rule.
Refs #1183 (stays open until the standing-rebuild after-number verifies).