Repository navigation
feat(storage): wire the index segment path to the object-storage backend (#965) - #1008
Merged
Merged
Conversation
…end (#965) storage.backend = "s3" starts now. Before this, the only constructor of StorageMode::ObjectStore outside tests hardcoded Local, and the startup guard refused the setting outright — honestly, because it was true: a flush uploaded one file per segment while a 25-field segment family is 104 files (.seg, .sidx, .ids, .dv, per-field .fst/.meta/.norms/.post), snapshot.json never left local disk, and a fresh node pointed at the bucket saw zero segments. The regression test object_store_mode_does_not_yet_make_an_index_stateless pinned that. Design — one immutable object per segment: - ZBM1 bundle format (xerj-storage/src/bundle.rs, new): the whole segment family packs into ONE object. Footer = the existing ZCM1 flush-completion manifest extended with absolute offsets, so family naming, CRC scheme and checksum validation stay on one code path. The trailer shape (footer_start u64 + version u32 + magic, 16 bytes) mirrors quickwit's split-footer trailer (quickwit-storage/src/ bundle_storage.rs:156-195, Apache-2.0 — approach adapted, no code taken): a cold open can later become a two-range-GET without a format change. v1 fetches the whole object. Unknown format versions are a typed error, never a best-effort parse. - Request economics, the reason for one object: at a 30 s flush interval, 104 PUTs per flush is ~899% of Cloudflare R2's free Class-A tier; one bundle + the catalogue PUT is ~18% (docs/OBJECT_STORAGE.md). - snapshot.json per index is the publication point: flush uploads the bundle then publishes the catalogue generation; merges publish their output bundle before retiring inputs; retirement deletes retired segments' bundles; boot fetches the catalogue and adopts backfill; reads hydrate families on demand (open_segment_arc); per-index meta (settings/schema/mapping) rides the same bucket. Config: 127 -> 128 settings (storage.s3_bucket). The guard that refused the backend is replaced by its one honest remainder: backend = "s3" with an empty bucket name is a half-written config and stays a hard error (XERJ never calls CreateBucket — an accidentally created bucket outlives the process and accrues cost). Fail-before, verified live on the base commit in this worktree (stash tracked changes, keep the new test): an_s3_index_round_trips_to_a_fresh _node panics "the flush must publish the snapshot catalogue". After: passes. Its predecessor test flips to the positive property (object_store_mode_makes_a_single_writer_index_stateless). Tests: bundle.rs unit tests (pack/unpack round-trip, footer CRCs, unknown-version refusal); index_store.rs object-store suite against the in-process filesystem simulation (XERJ_TEST_OBJECT_STORE_DIR — never a network endpoint, no credentials); engine journey object_storage_e2e.rs (flush uploads through real Index wiring, meta objects ride the bucket, a FRESH node on an empty data dir boots from the bucket and serves full-text over hydrated families); product_experience.rs count pin 128. The endpoint-backed s3_object_store suite still skips with a printed reason when no endpoint is set (eight explicit skips, verified). Honest limits, kept in docs/OBJECT_STORAGE.md "What is not wired": SINGLE-WRITER v1 (two nodes on one index clobber each other's catalogue — enforced by this paragraph, not by a lock); the WAL stays local (unflushed acked writes do not follow the bucket); bundles pack in memory and there is no multipart upload (single-PUT ceiling, RAM runs out first); crash-consistency is pinned only against the simulation, no killed-mid-upload test against a real endpoint; no GC command for orphaned bundles (deletes ride retirement; a crash between bundle and catalogue leaves an unlisted, harmless, billable object). Gates: fmt clean; clippy 0 (storage+engine, all targets); unit tests exit 0 (storage 199 + s3 suite + engine suites); ES-YAML conformance 1376 passed / 0 failed / 3 skipped (ci-test profile, private port, throwaway dir).
…ue-965-segment-object-storage
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Sep 22, 2026
…d-off PR xerj-org#1008's CI failed the browser security suite's policy tour: #/dashboards/ai-overview: a sample-backed view must be labelled (LIVE · Xerj · UPDATED JUST NOW · 74.3MS) That is the CORPUS view's pill — the tour starts at #/corpus — read on the first telemetry route. Root cause, in the test: after each setHash the tour waits for `aria-busy === 'false' && .h-scene, h1`. But app.js only ever sets aria-busy to 'false', at the END of a render — nothing flips it 'true' while the next route's query() is pending — so the condition is already satisfied by the PREVIOUS route's still-mounted DOM. The fixed 150ms sleep that follows then reads the old view's pill before the new render swaps nav and view in one piece. On a 2-4 vCPU runner the pending fetch outlasted the sleep; on a 32-core box it does not (6 runs under 24 spinners + 8 under 128: no reproduction — the window scales with runner latency, not scheduler pressure). Fail-before, deterministic: 400ms latency injected into the fake node's console-API 404s (uncommitted demonstration scaffold) fails the old test in one run — #/data: Xerj: MOCK FALLBACK · UPDATED JUST NOW · 0.8MS — the same race from the mirrored side: a LIVE-claiming route (data) read while the previous telemetry view's pill was still up. Fix: wait for the pill to reach the route's TERMINAL label instead of sleeping — MOCK FALLBACK|SAMPLE DATA on telemetry routes, ^LIVE on corpus/discover/data — bounded by the harness waitFor's 15s deadline. The wait IS the assertion: a view that never labels itself fails on timeout, showing the text it stalled on. The product is unchanged: while the new route's query() is pending the pill honestly describes the previous view — the one still on screen — and nav and view swap in the same innerHTML write, so no view ever shows another view's pill once rendered. Verified: old + latency → fails (above); fixed + latency → 11/11; fixed without latency → browser suite 24/24, unit suite 0 fail.
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Sep 22, 2026
The stateless-index and reader-fairness release. Two headline changes, each with a fail-before reproduction; everything else is a fix: - storage.backend = "s3" starts for real (xerj-org#965, PR xerj-org#1008): the whole segment family packs into ONE immutable ZBM1 bundle object (footer = the ZCM1 manifest + absolute offsets; trailer adapted from quickwit's split-footer bundle, Apache-2.0, approach only). One object + the catalogue PUT is ~18% of R2's free Class-A tier at a 30 s flush interval where per-file PUTs would be ~899%. snapshot.json is the publication point; merges publish before retiring; a FRESH node on an empty data dir boots from the bucket and serves full-text over hydrated families (journey test). Fail-before: the round-trip test panicked "the flush must publish the snapshot catalogue". Honest limits stay in docs/OBJECT_STORAGE.md: single-writer v1, WAL local, in-memory packs, simulation-pinned crash consistency. Config 127 -> 128 - readers no longer starve under sustained ingest (xerj-org#1013, PR xerj-org#1014): the seqlock token was validated at the END of a search, demanding a writer-free interval spanning the whole scan — under flush pressure those stop existing and search/delete_by_query 500'd at their deadline (reproduced twice in the xerj-org#950 profiling run). Validation now happens at the CAPTURE boundary with the evenness check; readers need a writer-free instant, not a writer-free search. Fail-before: churn test failed at its 2 s deadline with the production error; now passes in 0.37 s with exact totals (723/723 lib tests, ES-YAML 1376/0) - /v1/systemone votes on the payload, never the instruction prose: 0.9867 empty vs 0.6700 criteria-rich instructions was a 32-point swing from wording alone (xerj-org#1000); {"message": ...} voted at the spam base rate, 0.1733 vs 0.9667 as a string (xerj-org#1001). All eight wording/shape arms now score 0.9767 identically (PR xerj-org#1007) - the console tour's pill assertions raced the route hand-off (xerj-org#1011) — aria-busy never goes 'true', so the wait matched the PREVIOUS route's DOM; failed only on slow runners (bit PR xerj-org#1008's CI) - CI honesty: the reference-coding toolkit's tests never ran (one filter per invocation; xc_ matched nothing — green on zero tests), and its MCP schema gate looked for a release binary the job never builds; the roadmap was re-reviewed against rc.76 (release-drift guard) - follow-up filed: xerj-org#1015 (flush guard spans the whole finalize — a latency consideration now, not a starvation hazard) Version stamps: engine/Cargo.toml + Cargo.lock at 1.0.0-rc.77; landing footers V1.0.0-RC.77 (51 files). Two historical rc.76 mentions kept on purpose: the CHANGELOG section and the "since rc.76" feature-landing reference in landing/answers/rebuild-reference-corpus-index.
xerj-org
added a commit
that referenced
this pull request
Sep 26, 2026
Two CI failures on the sweep commit, both legitimate catches: 1. xerj-rerank egress_inventory::the_status_statement_names_every_node_ path_and_the_docs_quote_it requires docs/RERANK.md to quote xerj_rerank::DATA_EGRESS verbatim, and the sweep had reworded the doc without touching the const. The const was the stale side anyway: it still said "the `S3Backend` client, which nothing on the segment path constructs today" - false since rc.77 (#1008 closing #965 wired the segment path to the backend, and index bundles carry stored document text). Fixed at the source of truth: DATA_EGRESS now carves the exception explicitly ("One exception, since v1.0.0-rc.77: ... as the index's home rather than a third-party egress"), the doc quotes the new value verbatim, the published egress table's object-storage row and the air-gapped recipe row are rolled to the rc.77 reality (the recipe keeps its air-gapped guidance: keep `local`, or point the backend at an on-prem endpoint), and "nothing on the segment path constructs" joins FALSE_CLAIMS so it cannot come back on any published surface. The KNOWN-table comment that still said the backend "refuses to start" is corrected too. 2. gen_sitemap.py derives lastmod from each page file's last commit date, so the sweep commit bumped 94 URLs' lastmod after the gate had checked the pre-commit state; landing/sitemap.xml is regenerated (178 URLs, 20141 bytes). Verified locally: the doc contains the new 1,201-char const value byte-for-byte; the FALSE_CLAIMS grep over every surface in the test's list returns zero hits; gen_sitemap --check and build_articles --check pass post-commit.
xerj-org
added a commit
that referenced
this pull request
Sep 26, 2026
The Technical SEO gate failed on 880318b with 4 ERRORs, all on content/answers/does-xerj-support-s3-alerting-plugins.md — the page the sweep rewrote to say the index CAN live in a bucket since v1.0.0-rc.77 (#1008 closing #965). The rules still encoded the pre-rc.77 fact, so the TRUE statements tripped the false-claim gate: FC-S3-INDEX-IN-BUCKET x2 — \"an index can live in an S3-compatible bucket (storage.backend = \"s3\")\" and \"Storing the index in a bucket works, since v1.0.0-rc.77\"; the rule reason still said storage.backend = \"s3\" is \"rejected at startup\". FC-ALERTING — \"Alerting and user-supplied plugins are still not implemented\": _neg_near required is/are + \"not implemented\" to be adjacent, and \"are still not\" fell between them. FC-S3-BACKUP — \"snapshot.json catalogue\": the word \"snapshot\" put a real rc.77 artifact name into backup context. Same repair pattern as the DATA_EGRESS fix in d3fa17d: change the enforcement and the fact together, never weaken the enforcement to let a stale doc through. - FC-S3-INDEX-IN-BUCKET: premise comment, reason, evidence and rewrite now state the split truth (autoindex never puts an index in a bucket; storage.backend = \"s3\" is the opt-in bucket home since rc.77; the WAL stays local in every configuration; default is local). New paragraph-scoped exemptions for the qualified phrasings: storage.backend / storage.s3_bucket, since v1.0.0-rc.77 / rc.77, [#1008], ZBM1 / snapshot.json. An article that names the knob in one paragraph still cannot claim an unqualified index-in-bucket in another. - _neg_near: \"is/are (still|yet) not\" and \"is/are (still|yet) not implemented/...\" — the honest \"are still not implemented\" phrasing must never trip, in any rule using the helper. - FC-S3-BACKUP: snapshot.json named verbatim is the rc.77 publication catalogue, not the _snapshot repository feature the rule guards. - S3 THING-matrix gate: \"never imply the index lives in the bucket\" now carries the rc.77 opt-in exception with the knob named. Verified locally, exactly as CI runs them: factcheck.py --self-test -> 56 rules OK factcheck.py --fixture-check -> 58 TP / 0 FN / 0 FP (26 fixtures; bad_s3_index_in_bucket still trips, good_s3_source_honest still clean) factcheck.py --fail-on error -> 92 files, 0 ERROR (was 4), 93 WARN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
storage.backend = "s3"starts now (#965). Before, the only constructor ofStorageMode::ObjectStoreoutside tests hardcodedLocal, and the startup guard refused the setting outright — honestly, because it was true: flush uploaded one file per segment while a 25-field segment family is 104 files,snapshot.jsonnever left local disk, and a fresh node pointed at the bucket saw zero segments.Design — one immutable object per segment
engine/crates/xerj-storage/src/bundle.rs, new): the whole segment family packs into ONE object. The footer is the existing ZCM1 flush-completion manifest extended with absolute offsets — family naming, CRC scheme and checksum validation stay on one code path. The 16-byte trailer shape (footer_start u64+ versionu32+ magic) mirrors quickwit's split-footer trailer (quickwit-storage/src/bundle_storage.rs:156-195, Apache-2.0 — approach adapted, no code taken). Unknown format versions are a typed error, never a best-effort parse.snapshot.jsonper index is the publication point: flush uploads bundle → publishes catalogue; merges publish output before retiring inputs; retirement deletes retired bundles; boot fetches the catalogue with adoption backfill; reads hydrate families on demand; per-index meta (settings/schema/mapping) rides the same bucket.Config: 127 → 128 settings (
storage.s3_bucket). The refusal guard reduces to its one honest remainder:backend = "s3"with an empty bucket name stays a hard error (XERJ never calls CreateBucket).Fail-before (verified live in the worktree)
Stash tracked changes, keep the new test, run on the #965 base:
After the wiring: passes. Its storage-crate predecessor flips to the positive property (
object_store_mode_makes_a_single_writer_index_stateless).Tests
bundle.rsunit tests: pack/unpack round-trip, footer CRCs, unknown-version refusal.index_store.rsobject-store suite against the in-process filesystem simulation (XERJ_TEST_OBJECT_STORE_DIR) — never a network endpoint, no credentials.object_storage_e2e.rs: flush uploads through real Index wiring, meta objects ride the bucket, a FRESH node on an empty data dir boots from the bucket and serves full-text over hydrated families.product_experience.rssettings-count pin: 128.s3_object_storesuite still skips with a printed reason when no endpoint is set (eight explicit skips, verified).Gates
fmt clean · clippy 0 (storage+engine+common, all targets) · unit tests 1,544 passed / 0 failed · ES-YAML 1376 / 0 / 3 (ci-test profile, private port, throwaway dir) · docs guard 15/15 after merging main (rc.76 + #1007).
Honest limits (docs/OBJECT_STORAGE.md "What is not wired")
SINGLE-WRITER v1 (two nodes on one index clobber each other's catalogue); the WAL stays local; bundles pack in memory and there is no multipart upload; crash-consistency is pinned only against the simulation, no killed-mid-upload test against a real endpoint; no GC command for orphaned bundles (deletes ride retirement).
Fixes #965