Repository navigation
perf(autoindex): batch the finalize-verify read-backs into per-index digest windows (#1183 count lane) - #1259
Merged
Merged
Conversation
…digest windows (#1183 count lane) The generation-wide verify barrier read changed groups back one group at a time: a term count per (group x index), a term+exists count per (group x dataset-with-semantic-field), and a catalog count per group - three serial round trips per file. Measured on the live crawl nodes (2026-10-08): cve-records finalize-verify ran at 2.2 groups/s across 402,695 changed groups (~50 h projected, :19285, 236 indices), vuln-fix-commits at 0.06/s across 29,258 groups (~120 h projected, :19295, 5,953 indices) - both against index phases that take hours, i.e. the barrier, not the writing, was the tail of every large corpus build (#1183's count lane; the scored lane was fixed by #1234 in rc.91). Every question has the same shape - 'how many rows in this index carry one of these content digests' - so they now batch. sync_executor.rs gains VERIFY_WINDOW=1024 and windowed_value_counts: one terms filter plus one exact terms aggregation per (index x 1,024 digests); the engine's terms doc_count is precise (no sketch in the metric path), so each bucket is the number the per-digest term search returned, verified live: term == bucket on sampled digests, and 524 absent digests in a 1,024 window produce no bucket and keep their seeded zero. The catalog leg cannot aggregate: the catalog is one global index (402,814 docs on the cve node) whose version map carries delete events - dataset documents are rewritten by every finalize-catalog - and the engine's columnar agg fast path refuses exactly that, so the brute agg corpus deep-clones the whole index (measured 786.7 MB estimate vs the 512 MB breaker -> 429, the #1183 shape). windowed_catalog_counts therefore fetch-counts the matched set instead: _source-restricted paged fetch, 31 ms cold / 14 ms warm per window on that same catalog, delete-aware, with a mid-window total change and a per-window count/total mismatch both failing loud. Measured per-window costs on the live cve node: record windows 6-63 ms (columnar fast path, append-only indices), semantic windows ~16 s (the fast path does not yet columnarize exists; queued as the engine follow-up that takes this to single-digit ms). Net for cve-records: ~1.2 M serial searches collapse to ~800 windows, and the finalize tail drops from a projected ~50 h to low hours. The phase keeps its name and stays honest about progress: the denominator is now the number of read-back windows, ticked as each resolves (one pr.file per index/catalog window), so the line never sits at 0/N for the length of the walk (#931's rule). The four failure messages are unchanged verbatim, and the per-group check order is unchanged, so the first disagreement names the same group it always did. #971 group-equality-skip keeps unchanged groups out of the windows entirely. Tests: the HTTP fakes now evaluate a terms aggregation over the matched set (both the reconcile and the label fakes) and honour exists inside bool.filter, mirroring engine semantics; the #1183 agg-free guard is narrowed to what it actually protects - no corpus-materialising agg (min/max on a time field) anywhere, and any terms agg must ride its digest-window filter - with the dataset read-back itself still agg-free. 1299 pass, 0 fail.
… verify comments
This was referenced Oct 8, 2026
thomas-villani
pushed a commit
to thomas-villani/xerj
that referenced
this pull request
Oct 9, 2026
Window since v1.0.0-rc.92: xerj-org#1259 (batched finalize-verify, the xerj-org#1183 count lane), xerj-org#1262 (closes xerj-org#1260: columnar exists + delete-aware fast-agg admission, 3.4s -> 2.7ms warm on the live crawl node, plus the test-race serialization caught by CI run 37816544794), xerj-org#1263 (chunxiaoxx CLA signature), xerj-org#1265 (agent-session-trajectories pack wiring on main), xerj-org#1268 (pack publish tag-burn fix, cited ahead of its merge). Corpus-hub side in prose: xerj-org#1255/xerj-org#1264 (recipe, stats, key, workflow, backlog flip) and xerj-org#1266 (vuln-fix-commits live, G7 blind-graded 4/5, 328,891 records). ROADMAP roll, shortlist live-verified at cut: twelve open — the ten carried from rc.92 (xerj-org#1094 xerj-org#1100 xerj-org#1122 xerj-org#1170 xerj-org#1173 xerj-org#1108 xerj-org#1110 xerj-org#1183 xerj-org#1212 xerj-org#1226) plus xerj-org#1250 and xerj-org#1267 (filed this window: the finalize read-back walk + doc-lane verify residual). Two corrections recorded: xerj-org#1254 was auto-closed by the rc.92 window's own xerj-org#1258 merge at 655ea93, 42 minutes BEFORE the rc.92 tag — the rc.92 roll's 'stays open' note predates the close, so it is de-linked from the shortlist and recorded in the closed list; xerj-org#1183's count lane is no longer 'the remaining half' (xerj-org#1259 + xerj-org#1262 landed it, first live proof the vuln-fix-commits rebuild completing 97 batched windows). Version 1.0.0-rc.92 -> rc.93 (workspace + lockfile via cargo update --workspace --offline), landing data-latest-tag stamped, sitemap regenerated (byte-identical, no new pages). release-notes-gate.sh PASSes locally: 13 shortlist issues open-checked, xerj-org#1260 closed-cited, window PRs all cited.
leninworld
pushed a commit
to leninworld/xerj
that referenced
this pull request
Oct 16, 2026
…rg#1267 The operational half of xerj-org#1267. The resume protocol could not tell that the server predated the client's fast-agg fix generation (xerj-org#1260/xerj-org#1270): on the cve-records resume a xerj-org#1259 client drove an older measure server, every finalize-verify `term ax_file` + `exists` window brute-scanned its index at ~3.4s, and it took four hours before anyone noticed. Nothing in the handshake differs between the two generations. What changed (client-only, sync_executor.rs): - `probe_server_generation` times ONE representative verify window right after `finalize-verify` opens. It prefers the semantic leg (the exact term+exists shape xerj-org#1260 fixed) and falls back to a record window when the generation has no semantic field. One digest is enough: the brute path's cost is index size, not window width. - A slow first answer (> 1s) is re-asked once before warning, because the first touch after a boot pages the index in cold (8.8s on the FIXED binary in xerj-org#1267). Warm numbers from xerj-org#1267: 3.4s old server vs 2.7ms fixed, three orders of magnitude either side of the 1s line. - If the warm answer is still slow it emits one `Progress::warn` line: the measured window time, a projection (windows x time), that the server is likely older than the client and lacks the xerj-org#1260/xerj-org#1270 lane, that the run continues and will be correct, and to upgrade the server and re-run (the journal resumes). It never aborts. `warn` reaches the plain/non-TTY surface, is one `{"event":"warning"}` object on `--progress json`, and survives `--quiet`. - Version: `GET /`'s `version.number` is the ES-compat version (8.13.0 or the --compat-version override), so it cannot be compared. `/v1/health` `data.version` carries the XERJ workspace version; the warning quotes it when it differs from the client's, and names a same-version slow server a branch/development build — the actual xerj-org#1267 shape, which is why latency, not version, is the signal. - A probe that errors returns silently: the verify window that follows asks the same thing and fails loud there. Tests: the decision is pinned through injected timings and version (the `rewalk_while_short` / `walk_run_id_pages` closure pattern): fast probe asks once and is silent; cold-then-warm is re-asked and silent; slow-warm warns on Plain with time, projection, versions and the upgrade advice and returns (no abort); same-version names a development build; Json surface emits exactly one parseable warning event; Silent surface still shows it; a failing probe neither warns nor propagates. cargo test -p xerj-autoindex: 1377 lib tests pass; fmt and clippy -D warnings clean. No engine/wire change, so the ES-YAML gate is untouched. Still open on xerj-org#1267: the engine residual (the run_id catalog read-back is O(corpus) per full-rewrite generation). Reference-coding mandate skipped: change is confined to this repository's own autoindex client code; no peer project has this protocol.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The generation-wide finalize-verify barrier asked its read-backs one changed group at a time — three serial searches per file. This batches them: one
termsfilter + one exacttermsaggregation per (index × 1,024 content digests) for the record and semantic legs, and a_source-restricted paged fetch-count for the catalog leg. The four failure messages and the per-group check order are unchanged, so the first disagreement names the same group it always did.Measured before (the defect, #1183's count lane)
Both against index phases measured in hours — the barrier, not the writing, was the tail of every large corpus build.
Measured after (live probes on the cve crawl node, 2026-10-08)
termcount == bucketdoc_counton sampled digests; 524 absent digests seeded in a 1,024-digest window produce no bucket and keep their seeded zero.existson the semantic field): ~16 s per window on a 53k-doc index — the fast path does not yet columnarizeexists, so this leg pays the brute path today. ~800 windows for cve-records ≈ low hours, against the 50 h serial baseline. The engine follow-up is fast-agg path: columnarize exists filters, and honor version-map deletes instead of bailing (measured: 16 s/window and 786 MB breaker trip) #1260 (columnarexists, delete-aware fast aggs) and takes this leg to single-digit ms._source:["file_key"]paged fetch, 31 ms cold / 14 ms warm per window on that same catalog, delete-aware, with a mid-window total change and a per-window count/total mismatch both failing loud.Net for cve-records: ~1.2 M serial searches collapse to ~800 windows.
Contract notes
finalize-verifyunchanged; the denominator is now the number of read-back windows, ticked as each resolves (one per index/catalog window), so the progress line never sits at0/Nfor the walk's duration (autoindex --no-graph: progress reads scan 100% / eta stalled for the whole indexing phase — indistinguishable from a hang #931's rule). Help text anddocs/recipes/zero-config-indexing.mdupdated.termsagg must ride its digest-window filter. The dataset read-back itself stays agg-free (PR fix(autoindex): agg-free catalog read-back + windowed finalize refresh #1207's shape).Tests
termsaggregation over the matched set and honourexistsinsidebool.filter, mirroring engine semantics.cargo test -p xerj-autoindex --lib: 1299 passed, 0 failed; fmt + clippy clean.Closes nothing yet — #1183 closes on a full xerj-search rebuild passing finalize end-to-end with the fixed client; this is the count-lane half of that. #1244's G7 suites depend on the same two crawler resumes this unblocks.