Repository navigation
fix(engine): let finalize-verify capture between catalog writes - #2237
Merged
Merged
Conversation
Motivation: `corpus index` finalize-verify sat at 0/N for seconds to minutes, or until the client gave up, when another corpus build was publishing to the shared autoindex-catalog index. The same corpora committed when the node was quiet. Root cause: explicit-id catalog bulks publish through index_document. Each call holds CollectionPublication across the WAL append and the memtable insert, then the next call opens another bracket immediately. Index::search will not admit a reader while any writer is in flight, and it retries a capture that straddles a bracket. The gap between those brackets is a few instructions — shorter than a capture that cloned the other run's buffered rows. The verify filter is bool(terms file_key, term run_id); `terms` was not a memtable bool predicate, so the capture took DocsForScan and cloned every buffered catalog document. The search then hit its deadline and returned timed_out with an empty page, which the verify walker either counted as zero or retried until the phase looked stuck. On a flushed catalog the keyword terms prefilter also scanned each segment's ordinal column once per terms value (1,024 full passes per verify window). Change: after a document or batch publication drops its bracket, the writer waits until a search that is blocked on this index validates its capture, or for 250ms. `terms` lowers into the fused columnar memtable walk, so that filter no longer clones the other run's rows. The keyword terms prefilter is one pass over the ordinal column. finalize-verify rejects a timed_out page instead of treating it as a live count of zero, and ticks one item per window so the phase does not stay at 0/N for the whole walk. The #2225 window tests call windowed_value_counts with tick=false; those calls are not phase windows. Evidence: with the handoff removed, catalog_verify_search_completes_while_another_run_writes failed in 2.000s with timed_out, total=0, hits=0. With the handoff restored it passed (0.15s). After rebasing onto origin/main 3a0fd82, `cargo +1.97.1 test -j 4 -p xerj-engine -p xerj-autoindex --lib -- catalog_verify_search_completes_while_another_run_writes columnar_terms_conjunction_matches_brute collection_publication_fail_closed_tests finalize_verify_stops_when_its_target_index_is_deleted finalize_verify_window_counts_a_live_index` passed: xerj-autoindex 2 tests in 1.02s, xerj-engine 26 tests in 0.79s. ES-YAML not run. A successful search response is unchanged; a missed deadline was already timed_out. An AI coding agent wrote this commit.
Keep #2231's resumed catalog read-back (credit_verify_items, watch_file, and the semantic exists preflight) together with #1761's rejection of a timed_out finalize-verify page. engine/crates/xerj-engine/src/index.rs auto-merged; the publication handoff is unchanged. Conflicted file: engine/crates/xerj-autoindex/src/sync_executor.rs. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Owner
Author
|
Xerj Merger made the fix: finalize-verify no longer sits at 0/N while another corpus build publishes to the shared autoindex-catalog index (#1761). After each per-document publication bracket the writer now hands off to a parked search (up to 250 ms) so the reader can validate its capture, and the verify walker treats a timed_out page as partial (bails with a resumable error) instead of counting it as zero. Adds a regression test with a concurrent catalog writer. Build + Test and all other checks green on c63ebad. |
cursor Bot
pushed a commit
that referenced
this pull request
Oct 23, 2026
Main moved ahead through ffc4e97: cluster metadata replication (#2257), the AVX2-free x86_64 baseline (#2161), _validate/query matching _search (#2263), finalize-verify between catalog writes (#2237), the unreadable-corpus refusal (#2258), and the ctutils bump (#2268). The only conflict was CHANGELOG.md. Both sides added a leading Fixed entry. Kept main's AVX2 note, then this branch's shutdown-flush note and the stale node.lock reclaim (#2233). engine.rs and main.rs merged without conflicts. Startup still takes the data-dir lock through node_lock::acquire before any index open, and HTTP listeners still stay bound until flush_all_force returns. Main's cluster proposer (attach_cluster, create_index_exact, delete_index_local) and the deferred cluster runner in async_main are intact. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An AI coding agent wrote this change.
Fixes #1761.
What this changes, and why
corpus indexfinalize-verify sat at 0/N for seconds to minutes, or until the client gave up, when another corpus build was publishing to the sharedautoindex-catalogindex. The same corpora committed when the node was quiet. This is the stall in #1761. The other items in that report are already fixed. #2129 (refresh admit timeout) and #2130 (bounded verify memory) do not cover it: finalize-refresh had already finished, and the hang is admission plus atermsscan, not the text-hydration path. #2225 (stop verify when its target index is deleted) is merged on main and is a different hang; this PR does not replace it.Explicit-id catalog bulks publish through
index_document. Each call holdsCollectionPublicationacross the WAL append and the memtable insert, then the next call opens another bracket immediately.Index::searchwill not admit a reader while any writer is in flight, and it retries a capture that straddles a bracket. The gap between those brackets is a few instructions — shorter than a capture that cloned the other run's buffered rows. The verify filter isboolofterms file_keyandterm run_id.termswas not a memtable bool predicate, so the capture tookDocsForScanand cloned every buffered catalog document. The search then hit its deadline and returnedtimed_outwith an empty page. The verify walker treated that page as a live count, or retried it, and the phase never ticked, so it stayed at 0/N. On a flushed catalog the keywordtermsprefilter also scanned each segment's ordinal column once per terms value (1,024 full passes per verify window).After a document or batch publication drops its bracket, the writer waits until a search blocked on that index validates its capture, or for 250ms, then opens the next bracket. The capture check still happens at the snapshot boundary, so the writer proceeds while the scan of that snapshot continues.
termslowers into the fused column 8000 ar memtable walk. The keywordtermsprefilter is one pass over the ordinal column. finalize-verify rejects atimed_outpage instead of counting it as zero, and ticks one item per window.Splitting publication into a structural bracket and an insert bracket was rejected: the WAL updates the version map to in-memory before the memtable insert, and the search verdict drops the old segment copy when it sees that state. A reader that captured during the gap would miss the new document.
Evidence
Without the handoff in
index_document,catalog_verify_search_completes_while_another_run_writesfailed in 2.000s:timed_out,total=0,hits=0. With the handoff restored, that test passed in 0.15s.After rebasing onto
origin/main3a0fd82ab(includes #2225 and #2226):xerj-autoindex: 2 passed, 0 failed, 1.02s. xerj-engine: 26 passed, 0 failed, 0.79s. That set includes the concurrent-writer reproduction, the columnar
termsconjunction, the publication-contract tests (search_waits_for_active_publication_then_revalidates,search_waiting_on_a_publication_stops_at_its_deadline,search_completes_under_publication_churn), and the #2225 deleted-index tests.cargo +1.97.1 fmt -p xerj-engine -p xerj-autoindexwas run before the commit. Clippy-D warningswas clean on these two crates before the rebase; it was not re-run after the twotick=falsecall-site edits.Checks
cargo fmtfor the crates touched (cargo +1.97.1 fmt -p xerj-engine -p xerj-autoindex). Notcargo fmt --all.cargo test -p <crate>for the whole crate (filtered tests above only)timed_out.timed_outverify page now fails the phase with a resume message, and the phase ticks once per window.after_flush_publish.Not run:
cargo fmt --all;cargo build --release -p xerj-engine -p xerj-autoindex; fullcargo test -p xerj-engineand fullcargo test -p xerj-autoindex; ES-YAML; clippy after the rebase. The 2.000s failure was measured on the pre-rebase tree by removing theindex_documentsuccess-path handoff; that negative control was not repeated after the rebase. The handoff inindex.rsapplied cleanly.Provenance
timed_out,total=0,hits=0); the passing run with the handoff (0.15s); the post-rebase filtered test command above (2 + 26 passed).xerj corpus indexon one node would falsify that if verify still sits at 0/N until the deadline. Also assumed: yielding 250ms per document while a capture is in flight does not poison publication or change committed documents. A publication-contract test failure, or a search that returns a torn overwrite under concurrent ingest, would falsify that.