…erj-org#440)
Two repair attempts on PR xerj-org#431 (built on the original xerj-org#428 scroll fix)
were refuted by live reproduction: resolving _seq_no/_version at render
time, then once at scroll-open time, both still read the version map in
a SECOND pass separate from the _source read -- a write landing in
between pairs a live seq_no with a stale/snapshot body. Since _seq_no is
fed back as if_seq_no for optimistic-concurrency writes, that is a
silent lost update, reproduced directly on xerj-org#440 on the reindex/
migration/CDC path scroll exists to serve.
xerj-org#431 was never merged, so main has none of its ScrollContext snapshot
work -- scroll continuation pages hardcoded seq_no/version/primary_term
on every hit unconditionally, ignoring the request's flags entirely,
and scroll_page_response's hand-built JSON never inserted these fields
into the wire response at all regardless of what EsHit carried. This
fixes that original xerj-org#428 gap and xerj-org#440 in one pass, on top of main.
Root fix per xerj-org#440's own diagnosis: xerj_query::executor::Hit now
carries seq_no/version, populated inside Index::search at every point a
hit's _source is resolved -- reusing the version-map entry already
fetched for that hit's own deleted/staleness check wherever one exists,
a single fresh lookup where none does. The two spots that deliberately
defer _source resolution for cheap candidate collection now resolve
seq_no/version at that SAME later moment, not before, so the window
never reopens regardless of which point in the pipeline produces the
final body.
es_compat.rs's four render-time re-lookups (PIT filter, fields._version,
top-level _seq_no/_version) now read h.seq_no/h.version off the hit
instead of re-querying the engine. ScrollContext gains
seq_no_primary_term/version flags captured once at scroll-open (real ES
semantics: the flag governs the whole scroll, not each page). The
disable_sequence_numbers -2/0 sentinel, previously search_impl-only, is
extracted into a shared helper and applied to scroll pages too --
xerj-org#440's own "adjacent, smaller, separate" gap.
Left out of scope: collapse inner_hits' _seq_no/_version -- members
there are already-rendered Values, not Hits, and _seq_no is currently a
fabricated array index, not even a real lookup. Pre-existing, separate
bug.
New engine/crates/xerj-api/tests/scroll_seq_no_torn_read.rs:
- multi_index_scroll_seq_no_matches_plain_search_per_index: 2 indices,
colliding ids, differing per-index write counts, asserts every
scrolled hit against a direct GET on its own index -- the shape
that caught xerj-org#431's second repair attempt (4/8 hits carrying the
wrong index's version-map entry).
- scroll_continuation_reports_the_snapshot_pair_not_a_torn_one: the
direct xerj-org#440 repro -- concurrent update between scroll page 1 and
page 2, asserts the scrolled page reports the snapshot pair, not
the live post-update one.
- scroll_continuation_honours_disable_sequence_numbers: the YAML
conformance gap xerj-org#440 names.
All three verified fail-before against the pre-fix tree.
cargo fmt: clean. cargo clippy -p xerj-api -p xerj-engine -p xerj-query
--lib --tests -- -D warnings: clean. cargo test -p xerj-engine --lib:
602 passed (2 pre-existing, unrelated snapshot-path-security failures,
confirmed reproducing identically on main -- an environment-specific
/tmp symlink-resolution issue). cargo test -p xerj-api: 223 lib tests
plus all integration suites passing, including the existing xerj-org#414 scroll
coverage (scroll_multi_index_identity, scroll_snapshot_cap_is_documented)
with no regressions.
Fixes xerj-org#440. Supersedes xerj-org#431 (never merged -- to be closed, not merged).
Claude-Session: https://claude.ai/code/session_01SYkHSEV3ofzGpF72efeW9w
Fixes #440. Fixes #428 (the original scroll-continuation hardcoding this PR also closes, folded into the same fix). Supersedes #431 (never merged — closing that PR, not merging it).
The bug (#440)
Two repair attempts on PR #431 (itself built on top of #428's original scroll fix) were refuted by the maintainer's own live reproduction:
_seq_no/_versionat render time — a liveidx.lookup_seq_no(&h.id)call ines_compat.rs, aftermerged_hitswas already built. Window: the whole scroll lifetime._source.Both let a write land between the
_sourceread and the_seq_noread, so a hit can report a live sequence number next to a stale/snapshot body. Since_seq_no< 8000 /code>/_primary_termare designed to be fed back asif_seq_no/if_primary_termfor optimistic-concurrency writes, that pairing is a silent lost update — reproduced directly on #440 on the reindex/migration/CDC path scroll exists to serve.#431 was never merged.
mainhas none of itsScrollContextsnapshot work — scroll continuation pages hardcodeseq_no: Some(0), version: Some(1), primary_term: Some(1)on every hit unconditionally, ignoring the request'sseq_no_primary_term/versionflags entirely, in bothscroll_page_responseandsearch_with_scroll's own first-page builder. Worse,scroll_page_response's hand-built JSON never inserted_seq_no/_version/_primary_terminto the wire response at all, regardless of whatEsHitcarried. This PR fixes the original #428 gap and #440 in one pass, on top ofmaindirectly.The fix
Per #440's own diagnosis: the value has to come from the same read that produced
_source, not a second pass. Lucene gets this structurally (a searcher pins anIndexReader;_seq_nodoc-values are immutable per segment, read alongside the body). xerj has no such structure, so this makes the same guarantee explicit:xerj_query::executor::Hitgainsseq_no: Option<u64>/version: Option<u64>, populated insideIndex::search, at every place a hit's_sourceis resolved — reusing the version-map entry already fetched for that hit's own deleted/staleness check wherever one exists, doing a single fresh lookup where none does. Two spots that deliberately defer_sourceresolution for cheap candidate collection (fill_memtable_sources, and the sort_topk inline fill) now resolveseq_no/versionat that same later moment, not before — so the source-vs-seq_no window never reopens, whichever point in the pipeline actually produces the final body.es_compat.rs's four render-time re-lookups (PIT filter,fields._version, top-level_seq_no/_version) now just readh.seq_no/h.versionoff the hit.ScrollContextgainsseq_no_primary_term/versionbool flags, captured once at scroll-open (matches real ES: the flag governs the whole scroll, not each page). Both scroll routes now read real, snapshotted per-hit values instead of hardcoding, and omit the fields entirely when not requested.scroll_page_response's hand-built JSON now actually inserts_seq_no/_version/_primary_termwhen present — the second half of the original scroll continuation pages (_search/scroll) never emit _seq_no/_version, even with seq_no_primary_term:true #428 bug, still present onmain.disable_sequence_numbers-2/0sentinel (previouslysearch_impl-only) is extracted into a shared helper and applied to scroll pages too — scroll's point-in-time view is torn: _seq_no is read in a second pass, so a hit can carry one moment's _source beside another's sequence number (silent lost update) #440's own "adjacent, smaller, separate" gap.Left deliberately out of scope: collapse
inner_hits'_seq_no/_version(es_compat.rs~13160-13175) —membersthere are already-renderedValues, notHits, and_seq_nois currently a fabricated array index (json!(i as u64)), not even a real lookup. Pre-existing, separate bug; worth its own issue.Testing
engine/crates/xerj-api/tests/scroll_seq_no_torn_read.rs:multi_index_scroll_seq_no_matches_plain_search_per_index— 2 indices, colliding ids, differing per-index write counts (so_seq_noisn't just a page ordinal), asserts every scrolled hit's_seq_no/_versionagainst a direct GET on its own index. This is the exact shape that caught DO NOT MERGE (refuted — snapshot not atomic) — fix(scroll): continuation pages resolve real _seq_no/_version (#428) #431's repair attempt feat(ai): prototype bounded ONNX embeddings #2 (4/8 hits carrying the wrong index's version-map entry).scroll_continuation_reports_the_snapshot_pair_not_a_torn_one— the direct scroll's point-in-time view is torn: _seq_no is read in a second pass, so a hit can carry one moment's _source beside another's sequence number (silent lost update) #440 repro: concurrently updates a snapshotted doc between scroll page 1 and page 2, asserts the scrolled page reports the SNAPSHOT_seq_no/_sourcepair, not the live post-update one.scroll_continuation_honours_disable_sequence_numbers— asserts the-2/0sentinel on a scroll continuation page, the YAML-conformance gap scroll's point-in-time view is torn: _seq_no is read in a second pass, so a hit can carry one moment's _source beside another's sequence number (silent lost update) #440 names (310_sequence_numbers_disabled.ymlhas no scroll case).git stashof every source change, test file only) — all 3 fail with the expected symptom, confirming they exercise the real bug and don't pass vacuously.cargo fmt: clean.cargo clippy -p xerj-api -p xerj-engine -p xerj-query --lib --tests -- -D warnings: clean.cargo test -p xerj-engine --lib: 602 passed, 0 failed (2 pre-existing failures insnapshot_path_security_tests, confirmed unrelated — same failures reproduce onmainbefore this change, an environment-specific/tmpsymlink-resolution issue).cargo test -p xerj-api --lib: 223 passed, 0 failed.cargo test -p xerj-api --test scroll_multi_index_identity --test scroll_snapshot_cap_is_documented: all passing, no regressions on multi-index scroll returns the raw index spec as per-hit _index, so (_index,_id) is not distinct — 6,100 unique keys for 12,000 docs #414's existing coverage.cargo check --workspace --lib --tests: clean.Claude-Session: https://claude.ai/code/session_01SYkHSEV3ofzGpF72efeW9w