8000 fix(scroll): carry real _seq_no/_version on the same read as _source (#440) by Vinz2168 · Pull Request #499 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

fix(scroll): carry real _seq_no/_version on the same read as _source (#440) - #499

Merged
xerj-org merged 1 commit into
xerj-org:mainfrom
Vinz2168:fix/issue-440-hit-carries-seq-no
Aug 19, 2026
Merged

xerj-org merged 1 commit into
xerj-org:mainfrom
Vinz2168:fix/issue-440-hit-carries-seq-no

Conversation

@Vinz2168
@Vinz2168 Vinz2168 commented Aug 19, 2026 •
Copy link
Copy Markdown
Collaborator

Fixes #440. Fixes #428 (the original scroll-continuation hardcoding this PR also closes, folded into the same fix). Supersedes #431 (never merged — closing that PR, not merging it).

The bug (#440)

Two repair attempts on PR #431 (itself built on top of #428's original scroll fix) were refuted by the maintainer's own live reproduction:

  1. Resolve _seq_no/_version at render time — a live idx.lookup_seq_no(&h.id) call in es_compat.rs, after merged_hits was already built. Window: the whole scroll lifetime.
  2. Resolve them once into the hit at scroll-open time instead. Narrower, but a multi-index scroll still spans every subsequent index's search, and — the real problem — the value is still read after, not from, the same read that produced _source.

Both let a write land between the _source read and the _seq_no read, so a hit can report a live sequence number next to a stale/snapshot body. Since _seq_no< 8000 /code>/_primary_term are designed to be fed back as if_seq_no/if_primary_term for optimistic-concurrency writes, that pairing is a silent lost update — reproduced directly on #440 on the reindex/migration/CDC path scroll exists to serve.

#431 was never merged. main has none of its ScrollContext snapshot work — scroll continuation pages hardcode seq_no: Some(0), version: Some(1), primary_term: Some(1) on every hit unconditionally, ignoring the request's seq_no_primary_term/version flags entirely, in both scroll_page_response and search_with_scroll's own first-page builder. Worse, scroll_page_response's hand-built JSON never inserted _seq_no/_version/_primary_term into the wire response at all, regardless of what EsHit carried. This PR fixes the original #428 gap and #440 in one pass, on top of main directly.

The fix

Per #440's own diagnosis: the value has to come from the same read that produced _source, not a second pass. Lucene gets this structurally (a searcher pins an IndexReader; _seq_no doc-values are immutable per segment, read alongside the body). xerj has no such structure, so this makes the same guarantee explicit:

  • xerj_query::executor::Hit gains seq_no: Option<u64> / version: Option<u64>, populated inside Index::search, at every place a hit's _source is resolved — reusing the version-map entry already fetched for that hit's own deleted/staleness check wherever one exists, doing a single fresh lookup where none does. Two spots that deliberately defer _source resolution for cheap candidate collection (fill_memtable_sources, and the sort_topk inline fill) now resolve seq_no/version at that same later moment, not before — so the source-vs-seq_no window never reopens, whichever point in the pipeline actually produces the final body.
  • es_compat.rs's four render-time re-lookups (PIT filter, fields._version, top-level _seq_no/_version) now just read h.seq_no/h.version off the hit.
  • ScrollContext gains seq_no_primary_term/version bool flags, captured once at scroll-open (matches real ES: the flag governs the whole scroll, not each page). Both scroll routes now read real, snapshotted per-hit values instead of hardcoding, and omit the fields entirely when not requested.
  • scroll_page_response's hand-built JSON now actually inserts _seq_no/_version/_primary_term when present — the second half of the original scroll continuation pages (_search/scroll) never emit _seq_no/_version, even with seq_no_primary_term:true #428 bug, still present on main.
  • The disable_sequence_numbers -2/0 sentinel (previously search_impl-only) is extracted into a shared helper and applied to scroll pages too — scroll's point-in-time view is torn: _seq_no is read in a second pass, so a hit can carry one moment's _source beside another's sequence number (silent lost update) #440's own "adjacent, smaller, separate" gap.

Left deliberately out of scope: collapse inner_hits' _seq_no/_version (es_compat.rs ~13160-13175) — members there are already-rendered Values, not Hits, and _seq_no is currently a fabricated array index (json!(i as u64)), not even a real lookup. Pre-existing, separate bug; worth its own issue.

Testing

Claude-Session: https://claude.ai/code/session_01SYkHSEV3ofzGpF72efeW9w

…erj-org#440)

Two repair attempts on PR xerj-org#431 (built on the original xerj-org#428 scroll fix)
were refuted by live reproduction: resolving _seq_no/_version at render
time, then once at scroll-open time, both still read the version map in
a SECOND pass separate from the _source read -- a write landing in
between pairs a live seq_no with a stale/snapshot body. Since _seq_no is
fed back as if_seq_no for optimistic-concurrency writes, that is a
silent lost update, reproduced directly on xerj-org#440 on the reindex/
migration/CDC path scroll exists to serve.

xerj-org#431 was never merged, so main has none of its ScrollContext snapshot
work -- scroll continuation pages hardcoded seq_no/version/primary_term
on every hit unconditionally, ignoring the request's flags entirely,
and scroll_page_response's hand-built JSON never inserted these fields
into the wire response at all regardless of what EsHit carried. This
fixes that original xerj-org#428 gap and xerj-org#440 in one pass, on top of main.

Root fix per xerj-org#440's own diagnosis: xerj_query::executor::Hit now
carries seq_no/version, populated inside Index::search at every point a
hit's _source is resolved -- reusing the version-map entry already
fetched for that hit's own deleted/staleness check wherever one exists,
a single fresh lookup where none does. The two spots that deliberately
defer _source resolution for cheap candidate collection now resolve
seq_no/version at that SAME later moment, not before, so the window
never reopens regardless of which point in the pipeline produces the
final body.

es_compat.rs's four render-time re-lookups (PIT filter, fields._version,
top-level _seq_no/_version) now read h.seq_no/h.version off the hit
instead of re-querying the engine. ScrollContext gains
seq_no_primary_term/version flags captured once at scroll-open (real ES
semantics: the flag governs the whole scroll, not each page). The
disable_sequence_numbers -2/0 sentinel, previously search_impl-only, is
extracted into a shared helper and applied to scroll pages too --
xerj-org#440's own "adjacent, smaller, separate" gap.

Left out of scope: collapse inner_hits' _seq_no/_version -- members
there are already-rendered Values, not Hits, and _seq_no is currently a
fabricated array index, not even a real lookup. Pre-existing, separate
bug.

New engine/crates/xerj-api/tests/scroll_seq_no_torn_read.rs:
  - multi_index_scroll_seq_no_matches_plain_search_per_index: 2 indices,
    colliding ids, differing per-index write counts, asserts every
    scrolled hit against a direct GET on its own index -- the shape
    that caught xerj-org#431's second repair attempt (4/8 hits carrying the
    wrong index's version-map entry).
  - scroll_continuation_reports_the_snapshot_pair_not_a_torn_one: the
    direct xerj-org#440 repro -- concurrent update between scroll page 1 and
    page 2, asserts the scrolled page reports the snapshot pair, not
    the live post-update one.
  - scroll_continuation_honours_disable_sequence_numbers: the YAML
    conformance gap xerj-org#440 names.
  All three verified fail-before against the pre-fix tree.

cargo fmt: clean. cargo clippy -p xerj-api -p xerj-engine -p xerj-query
--lib --tests -- -D warnings: clean. cargo test -p xerj-engine --lib:
602 passed (2 pre-existing, unrelated snapshot-path-security failures,
confirmed reproducing identically on main -- an environment-specific
/tmp symlink-resolution issue). cargo test -p xerj-api: 223 lib tests
plus all integration suites passing, including the existing xerj-org#414 scroll
coverage (scroll_multi_index_identity, scroll_snapshot_cap_is_documented)
with no regressions.

Fixes xerj-org#440. Supersedes xerj-org#431 (never merged -- to be closed, not merged).

Claude-Session: https://claude.ai/code/session_01SYkHSEV3ofzGpF72efeW9w
@xerj-org
Copy link
Copy Markdown
Owner

Independent verification — head 7a891349

Three skeptics, isolated worktrees, rustc 1.97.1, each told to REFUTE the torn-read fix. All three could not refute: 0 blocking. Merging on this.

  • Torn read closed: _seq_no/_version are now fields on executor::Hit, populated inside Index::search at the same point _source is resolved, and snapshotted into ScrollContext.hits at scroll-open. Continuation pages read h.seq_no/h.version off the snapshot — the four old render-time re-lookups are gone. hit_seq_version is arithmetically identical to the old lookup_seq_no/lookup_version.

  • Test is not vacuous: reverting only the 6 source hunks (keeping the test) fails all 3 new tests, including the torn-read assertion (scrolled hit must carry _seq_no). The torn-read test updates a doc between page 1 and 2 and asserts the continuation reports the SNAPSHOT pair (seq + body v=1), not the live post-update value — the exact lost-update the maintainer's repro caught on DO NOT MERGE (refuted — snapshot not atomic) — fix(scroll): continuation pages resolve real _seq_no/_versio 8000 n (#428) #431.

  • No regression: first page (both routes), plain _search, search_after/PIT, multi-index, deletion-mid-scroll all verified; Rust's exhaustive struct-init makes a forgotten field a compile error, not a silent 0. scroll continuation pages (_search/scroll) never emit _seq_no/_version, even with seq_no_primary_term:true #428 confirmed fixed (origin/main hardcoded seq_no:0/version:1 at 3 sites).

Two nits, filed as follow-ups, NOT blockers: collapse inner_hits still does a live lookup_version at render (es_compat.rs:13163 — pre-existing on main, explicitly scoped out); and a sub-millisecond intra-search-call window where source and version_map.get are separate reads (inherent to storage having no atomic source+version read; the same class as plain _search metadata staleness, and vastly narrower than the whole-scroll window it replaces).


Lens 1 — refuted: False

Verdict (verbatim):

REFUTATION FAILED — I cannot demonstrate a blocking torn read. The PR correctly closes the scroll-continuation window that refuted #431. HEAD confirmed at 7a89134.

Trace of the captured triple (_source, _seq_no, _version):

  • Capture point: seq_no/version are populated INSIDE Index::search at the same code point _source is resolved. Where a VersionEntry is already fetched for the doc's own deleted/superseded check (index.rs:16399, 23704, 23824, 24362), hit_seq_version reuses that SAME entry — genuinely one read producing all three. Where no entry is in hand, version_map.get is co-located with the source resolution; deferred-source paths (fill_memtable_sources, sort_topk inline, knn_result_from_scored) resolve seq_no at the same deferred fill moment, with a catch-all at ~19235 that fills any still-unresolved hit at the final bounded-page fill. hit_seq_version matches lookup_seq_no/lookup_version exactly (seq_no = ver.seq_no-1 saturating; version = external_versions wins else ver.version; deleted => (None,None)).
  • Snapshot point: search_impl clones merged_hits (all hits, resolved, size=SCROLL_SNAPSHOT_MAX_HITS) into scroll_snapshot -> ctx.hits at open (es_compat.rs:11219/14225/14251). search_with_scroll clones all_hits -> ctx.hits (19793/19865). ScrollContext also snapshots the emit flags once at open.
  • Render point: scroll_page_response reads h.seq_no/h.version straight off ctx.hits (es_compat.rs:20125-20160). No lookup_seq_no/lookup_version call remains in the scroll continuation top-level-hit path — the four old render-time re-lookups (PIT filter 10712, fields._version 12068, top-level _seq_no/_version 13250/13279) all now read the hit.

Attack results:

All 3 new tests + 2 pre-existing identity tests pass on the CI toolchain (rustc 1.97.1).

Findings (verbatim):

  • nit engine/crates/xerj-api/src/es_compat.rs:13163 (introduced_by_this_pr=False): Collapse inner_hits still does a live lookup_version at render time and fabricates _seq_no as the array index (json!(i as u64)); on a collapse+scroll this is a torn/synthetic pair for inner hits.

    emit_version_ih branch calls idx.lookup_version(id_str) live while _seq_no is a positional index, so an inner hit can report a live version beside a snapshot body. This is NOT introduced by this PR (identical on origin/main) and the PR explicitly scopes it out as a separate pre-existing bug; it is the collapse inner_hits path, not the top-level scroll hit this PR fixes. Not blocking.

  • nit engine/crates/xerj-engine/src/index.rs:12238 (introduced_by_this_pr=True): In a few construction sites the snapshot's seq_no is read via a fresh version_map.get separate from the already-materialized source (from scored), so the open-time triple is atomic only to within one Index::search call, not a truly single atomic read.

    A write landing in the sub-millisecond gap between source materialization and the version_map.get within the SAME search call could bake a (new seq_no, old source) pair into the snapshot. This window is inherent to the storage lacking an atomic source+version read (the PR acknowledges 'xerj has no such structure'), is the same class as a plain _search's source-vs-metadata staleness, is vastly narrower than the whole-scroll-lifetime window it replaces, and is not reproducible deterministically through the HTTP API. Not a continuation-window torn read; not blocking.


Lens 2 — refuted: False

Verdict (verbatim):

Could not refute PR #499. The central #440 claim holds: _seq_no/_version are captured at the same read as _source and snapshotted into ScrollContext.hits at scroll-open, so continuation pages report the snapshot pair, never a live (post-update/deleted) value. The new test is real and exercises the actual torn-read interleaving. HEAD verified at 7a89134. Nothing blocking; one nit.

Vacuity check: reverting ONLY the source hunks (git checkout origin/main on the 6 source files, keeping tests/scroll_seq_no_torn_read.rs) makes all 3 new tests FAIL under rustc 1.97.1 — including the torn-read assertion (panic at scroll_seq_no_torn_read.rs:253 'scrolled hit must carry _seq_no', because pre-fix continuation JSON never emitted the field), the disable_sequence_numbers test (got None, expected -2), and the multi-index test. Restoring source: all 3 pass.

Torn read is genuinely tested: scroll_continuation_reports_the_snapshot_pair_not_a_torn_one opens a scroll (size=1, sort v asc, seq_no_primary_term+version), does a concurrent PUT /torn/_doc/1 {v:999} AFTER the snapshot, pages to doc '1', and asserts scrolled _seq_no != live_seq_after_update AND _source.v == 1 (snapshot body). Runs through the real HTTP router via oneshot.

Mechanism traced: scroll_snapshot = merged_hits.clone() (es_compat.rs:11216); body.size bumped to SCROLL_SNAPSHOT_MAX_HITS 10k (es_compat.rs:8467) so the engine's final catch-all (index.rs:~19235) resolves seq_no for the WHOLE snapshot, not just page 1. hit_seq_version reads seq_no/version from the same version_map entry that gates the deleted/stale check and _source. Continuation pages (scroll_page_response) read h.seq_no/h.version off the snapshotted hit, gated by ctx.seq_no_primary_term/version captured once at open (real ES flag-governs-whole-scroll semantics). Deletion mid-scroll: doc was live at snapshot, stays in snapshot with its snapshot seq_no — matches ES point-in-time view.

#428 confirmed fixed: origin/main hardcoded version:Some(1)/seq_no:Some(0)/primary_term:Some(1) at 3 sites and the hand-built continuation JSON never inserted _seq_no/_version at all; PR replaces with flag-gated real values from the hit and adds the fields to the wire map. #431's live-lookup-at-render approach is correctly avoided (no lookup_seq_no at render time — the 4 es_compat re-lookups now read h.seq_no/h.version).

No regression: xerj-api 223 lib + all integration suites pass (incl. scroll_multi_index_identity, scroll_window_tests, search_after validation, PIT), xerj-query 183 lib pass, clippy -D warnings clean on 1.97.1. Memory: +~24 bytes/hit into snapshots capped at 10k hits and bounded by max_open_scrolls — snapshot cap counts hits not bytes, so #405/#501 unaffected.

Findings (verbatim):

  • nit engine/crates/xerj-api/src/es_compat.rs:20155 (and first-page 19819) (introduced_by_this_pr=True): Continuation/first-page render uses h.seq_no.unwrap_or(hit_idx as u64) as fallback; if any snapshot hit ever carried seq_no=None while emit_seq_no is true, it would silently emit a page-local ordinal (0 for each page's first hit) instead of a real or absent seq_no.

    Not demonstrable on any current path: for scroll, body.size is forced to the 10k cap (es_compat.rs:8467) so the engine's final catch-all (index.rs:~19235) resolves seq_no for every snapshotted hit before it reaches the context; the multi-index test confirms all 8 hits across continuation pages match a direct GET. The fallback is latent fragility only — a future query path leaving seq_no unresolved for hits beyond page 1 would surface it. Nit, not a defect.


Lens 3 — refuted: False

Verdict (verbatim):

Could not refute PR #499. HEAD confirmed at 7a89134. The fix is structurally sound across the whole scroll/search surface and I found no demonstrable blocking regression (no torn read, no pagination regression, no data loss, no crash).

WHY THE TORN READ IS ACTUALLY FIXED (not like #431): _seq_no/_version are now fields on xerj_query::executor::Hit, populated INSIDE Index::search at the same construction point _source is resolved, and snapshotted into ScrollContext.hits at scroll-open. Continuation pages read h.seq_no/h.version off the snapshot vector — never a live re-lookup. es_compat.rs's four former render-time re-lookups (PIT filter, fields._version, top-level _seq_no/_version) all now read the hit. The engine-side transform hit_seq_version() is arithmetically identical to the old lookup_seq_no/lookup_version (seq_no = ver.seq_no.saturating_sub(1); version = external_versions wins else ver.version; deleted -> (None,None)). Test scroll_continuation_reports_the_snapshot_pair_not_a_torn_one PASSES: a doc updated between page 1 and page 2 returns the SNAPSHOT (seq, source v=1) pair, not the live post-update seq — exactly the lost-update the maintainer's repro caught on #431.

REGRESSION LENS (my assignment):

  1. First page: correct. Both scroll routes fixed. search_impl first page renders via the same h.seq_no path (es_compat ~13250); search_with_scroll's first-page EsHit builder now reads h.version/h.seq_no instead of the hardcoded Some(0)/Some(1) still present on origin/main (confirmed: origin/main es_compat.rs lines 19845-47, 19935-37, 20111-13 hardcode seq_no:0/version:1). multi_index test asserts EVERY hit including page 1 against a direct GET — passes.
  2. Plain (non-scroll) _search: every hit returned from Index::search funnels through fill_memtable_sources (index.rs:17126), whose catch-all (19245) resolves seq_no/version for any hit left None. Deferred-None construction sites (14592, 15087) are on that path. kNN path resolves inline via knn_result_from_scored (now takes &Index). Sorted/keyed/scan collectors (22487, 23704, 23824, 24362) resolve inline. Rust's exhaustive struct init makes a forgotten field a compile error, not a silent 0.
  3. search_after/PIT: unaffected; published_search_after_recipe_walks_the_whole_corpus and scroll_multi_index_identity pass.
  4. Deletion mid-scroll: continuation returns the SNAPSHOTTED seq_no (doc was live at snapshot), matching ES point-in-time scroll semantics — this is the intended behavior, not a bug.
  5. Snapshot bloat: two Option (~16-24 bytes/hit) added to Hit; the scroll cap (es-compat: the scroll snapshot cap is applied per index on /{index}/_search_scroll, but to the summed total on /{index}/_search?scroll= (split from #370) #405/fix(scroll): enforce the summed scroll snapshot cap on the alias route too (#405) #501) is count-based, not byte-based, so no interaction — scroll_snapshot_cap_is_documented tests pass.

TEST RESULTS (rustc 1.97.1, CI toolchain):

  • cargo test -p xerj-api: 0 failures (all suites)
  • cargo test -p xerj-api --profile ci-test: 0 failures
  • cargo test -p xerj-engine --lib: 606 passed; 0 failed (the 2 snapshot_path_security_tests failures the PR body flagged as env-specific did NOT reproduce here)
  • cargo test -p xerj-api --test scroll_seq_no_torn_read: 3 passed
  • cargo clippy --workspace --all-targets -- -D warnings: exit 0, clean
  • cargo fmt --check: exit 0, clean

Nit only (not demonstrable, not blocking): the continuation-page fallback h.seq_no.unwrap_or(hit_idx as u64) uses a page-local ordinal that resets to 0 each page, so IF a live snapshotted hit ever carried seq_no=None (only reachable if it were absent from version_map), two docs on different pages could report the same fabricated seq_no. Live snapshot hits are always in version_map, so this branch is effectively dead, and it mirrors search_impl's pre-existing fallback style — I could not construct an input that reaches it. Refuted=false.

Findings (verbatim):

  • nit engine/crates/xerj-api/src/es_compat.rs (scroll_page_response continuation + search_with_scroll first page) (introduced_by_this_pr=True): The h.seq_no.unwrap_or(hit_idx as u64) fallback uses a per-page ordinal that resets to 0 each continuation page; if a live snapshotted hit were ever missing from the version_map (seq_no=None), two docs on different pages could report the same fabricated _seq_no.

    Code inspection only. I could not construct an input that reaches this branch: hit_seq_version returns Some for any live, non-deleted version_map entry, and a hit present in the scroll snapshot was live at snapshot time, so it is in the version_map. Not demonstrable; mirrors search_impl's pre-existing fallback. Non-blocking.


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants

0