fix(engine): #948 — prune published memtable entries, charge what they retain, bound merge batches, drain on breaker by xerj-org · Pull Request #1002 · xerj-org/xerj

xerj-org · 2026-09-21T03:36:42Z

Fixes #948

Problem: a 1 GB mailbox under the default 16 GiB cap aborted (25.4 GiB VmHWM, 510/2,530 needles missing, 0 edges); uncapped it peaked at 68.5 GiB.

Root cause (found with the ingest-memory ledger):

IndexStore::memtable_shards retained one parsed Arc<Value> per explicit-id write forever — flush drained the FTS memtable, never the storage shards (~2.4-3.1× source bytes invisible to every ledger category; ~7 KB/doc in the 100k-doc variant)
raw-bytes entries charged a flat 800 B regardless of payload
merge batches were cut by segment count while the executor materialises 5-9× input bytes
the RSS breaker only 429'd, never drained

Changes: prune_published_memtable() at flush finalise; structural entry charges (estimate_value_bytes) + WAL replay charging; raw_bytes_charge = max(800, len+id+160) with the 800 floor keeping the M5.17 cadence; merge batches cut by input bytes clamp(cap/64, 32, 512) MiB (XERJ_MERGE_BATCH_MAX_INPUT_MB), forcemerge too (approach after quickwit streaming_writer.rs:378 / merge streaming.rs:148, tantivy log_merge_policy.rs:22,94 — no code copied); breaker-engaged drain (flush + release caches + jemalloc arena purge hook, 2 s rate limit, coalescing channel), publish-warming skipped while engaged; ledger now charges hydration cache + merge buffers.

Measured (same harness both sides, sequential):

at-rest settled 972.8 → 481-488 MiB (variant A, 3 runs) and 883.0 → 280.8 MiB (variant C 100k×100B, retention ~677 → ~108 MB)
300M at 8 GiB cap: 14,766.6 → 7,963.6 MiB — under the cap; breaker engagements 178 → 2; verify identical figure-for-figure
1G mailbox at 16 GiB cap: abort → complete in 407.7 s with every needle and edge green (2,530/2,530, 21,544/21,544, 39,619/39,619)

NOT fixed (measured, kept visible):

the fixed 1G run's peak is still 21,405.3 MiB (1.3× its own 16 GiB cap — swap/OOM risk on a real 16 GiB machine)
verify-time reads under the cap thrash (p95 5.54 s vs base 7.5 ms — the drain drops the hydration cache reads rely on)
at-rest VmHWM unchanged (~1.94 GiB)
verify.py swallows request errors (the fixed 1G run's sender 0/40 was that; re-verified 40/40 on a rebooted copy)

Fail-before: behavioral revert fails drain_releases_memory.rs:87,:177, the memtable raw_bytes_charge floor test, index_store flush_finalize_prunes test; on pure base the drain API does not compile.

Gates: fmt + clippy -D warnings clean; engine/storage/server tests green except two pre-existing/environmental items (ROADMAP "on main now" doc test fails at base too; brain_e2e failed only while box disk ≥95% put nodes in read_only_allow_delete — passes in 569.4 s with the documented flood-stage watermark raise); ES-YAML 1,371 passed / 0 failed (port 9330, throwaway dir).

Run evidence: /root/948-runs (binaries xerj-948-base, xerj-948-after, xerj-948-commit).

…y retain, bound merge batches, drain on breaker Problem (issue #948): ingesting a 1 GB mailbox under the default 16 GiB auto cap aborted (exit 1) after the client's 600 s re-offer budget with server VmHWM 25,356.9 MiB, 85,722 of ~107k docs on the node, 510 of 2,530 planted needles missing and 0 edges written (run fix-mixed-1G-capped16g-base, 2026-09-20; base = 9c64a7c, harness benchmarks/mbox-ingest/run.sh, seed-42 mixed tree, 40,799-entry mbox, every needle checked). Uncapped, the same load peaked at 68.5 GiB (after-mixed-1G-runC-uncapped, mbox branch). Root cause, found with the ingest-memory ledger: 1. Retention (the big one): IndexStore::memtable_shards kept one parsed Arc<serde_json::Value> per explicit-id write for the life of the process. The engine's flush drains the FTS memtable and finalises from that, so the storage-side shards were never emptied: ~2.4-3.1x the source bytes of parsed JSON sat in RSS permanently, invisible to every ledger category (~533 MB unattributed at rest for a 210 MB load; ~7 KB retained per 100-byte doc in the 100k-doc variant). 2. Accounting: a raw-bytes (_bulk turbo) memtable entry charged a flat 800 bytes regardless of payload — a ~10 KB mailbox doc counted as 800, 12x under — and the explicit-id path charged a full JSON re-serialisation rather than the tree it retains. 3. Merge batching: batches were cut by segment COUNT, but the executor materialises decoded + re-encoded copies of every input doc (measured 5-9x input bytes), so a count-sized batch of large segments was an unbounded spike. 4. The parent RSS breaker only rejected (429); nothing ever drained, so engaged meant stuck until RSS fell on its own. What changed: - xerj-storage/index_store.rs: prune_published_memtable() at the end of every successful flush finalise drops entries the version map has repointed at a real segment (IN_MEMORY_SEGMENT_ID check); entries carry a structural charge (estimate_value_bytes) so shards and memtable_bytes agree; WAL replay charges too. - xerj-engine/memtable.rs: raw_bytes_charge() = max(800, len + id-len + 160) on the raw-bytes insert paths. The 800 floor keeps the compressed log-doc cadence byte-for-byte (the M5.17 regression stays fixed). - xerj-storage/merge.rs + xerj-engine/index.rs: merge batches are cut by input bytes, clamp(cap/64, 32, 512) MiB, XERJ_MERGE_BATCH_MAX_INPUT_MB overrides; forcemerge obeys the same cap. Approach after quickwit (quickwit-parquet-engine/src/storage/streaming_writer.rs:378 and merge/streaming.rs:148) and tantivy (src/indexer/log_merge_policy.rs:22,94); no code copied. - xerj-engine/engine.rs + governor.rs + xerj-server/main.rs: when the parent RSS breaker engages, the sampler requests a drain of every loaded index (flush + release rebuildable caches + a jemalloc arena.4096.purge hook the server installs), rate-limited to one per 2 s while engaged, through a capacity-1 coalescing channel and a worker thread; publish-warming is skipped while engaged; drain_rebuildable_memory() is public for tests. - xerj-engine/ingest_memory.rs: the ledger charges the segment-hydration cache at admission and the merge executor's working buffers (survivor/json_buffer/parsed/decoded/encoded), so the ledger's view and RSS agree where they used to diverge by gigabytes. Measured — same harness both sides, sequentially, on the same box (2026-09-20/21; binaries recorded in the result files): - At rest, variant A (20k x 10 KB = 210 MB via _bulk, cap lifted): settled RSS 972.8 -> 481.4 MiB (reproduced 481, 488.2), unattributed remainder ~533 MB -> ~3 MB; VmHWM unchanged ~1.94 GiB (fix-atrest-bisect). - At rest, variant C (100k x 100 B = 10 MB source): settled 883.0 -> 280.8 MiB, retention ~677 -> ~108 MB, VmHWM 1,973.8 -> 1,570.9 MiB; both sides landed exactly 100,000 docs (fix-atrest-bisect). - 300 MB mailbox at the 8 GiB cap a 16 GiB laptop gets: both complete; server VmHWM 14,766.6 -> 7,963.6 MiB (under the cap for the first time), breaker engagements 178 -> 2; verify identical figure for figure (735/735 needles, replies 5,992 = truth, attachment_of 11,169, sender 40/40, subjects 40/40) (fix-mixed-300M-cap8g-{base,fixed}.json). - 1 GB mailbox at the default 16 GiB cap: base aborts with VmHWM 25,356.9 MiB and 510 needles missing; fixed COMPLETES — exit 3, 407.7 s, 107,698 docs, 2,530/2,530 needles (0 missing, 0 duplicated, 0 wrong), replies_to 21,544 = truth, attachment_of 39,619 = attachment docs (fix-mixed-1G-capped16g-{base,fixed}.json). What this does NOT fix (measured, not guessed): - The 1 GB run's peak is still 21,405.3 MiB — 1.3x its own 16 GiB cap. The drain makes the run complete; it does not make it fit in 16 GiB. - Verify-time reads under the cap thrash: p95 5.54 s, 509 of 2,619 requests over 1 s (base p95 7.5 ms) — the drain drops the segment-hydration cache that reads rely on, and the next read re-hydrates. Median stays 9.2 ms. - At-rest VmHWM for the 210 MB load is unchanged (~1.94 GiB): the fix recovers the settled floor, not the transient peak. - The fixed 1 GB run's verify reports sender sampled 0/40 where base had 40/40: a verify.py gap (post() swallows request errors, buckets() reads []), triggered by that same thrash — not data loss. Re-verified on a rebooted byte copy of the same data dir with the cap lifted: the email_from_address aggregation returns 40 buckets and all 40 sender checks pass. Fail-before evidence: with the fix reverted in the working tree (prune call commented, drain body commented, flat 800 charge restored) the new tests fail for real — drain_releases_memory.rs:87 (drain_releases_caches_and_keeps_indexes_in_service), drain_releases_memory.rs:177 (flush_prunes_published_storage_memtable_ entries), memtable.rs raw_bytes_charge_tracks_payload_with_an_800_byte_ floor, index_store.rs flush_finalize_prunes_published_storage_memtable ("storage memtable entry must be pruned"). On pure base src the drain test does not compile (drain_rebuildable_memory does not exist there), which is exactly why the behavioral revert is the evidence that counts. Environment notes for the record: the shared box's root filesystem sat at 97-98% during the 2026-09-21 re-runs (other tenants), which crosses the node's 95% disk.watermark.flood_stage at boot; throwaway gate and measurement nodes were unblocked with the documented transient cluster setting (flood_stage 99%), which only changes which writes are admitted. The xerj-server brain_end_to_end_five_note_fixture failure seen under that disk condition is that block (edge bulks rejected, CLI retrying), not a regression: with the watermark raised mid-run the same binary completed and the test passed (569.4 s). The xerj-engine no_shipped_doc_promises_a_future_release failure is pre-existing at base (ROADMAP.md "on `main` now", present in git show HEAD:ROADMAP.md).

…heap-mailbox

…979 PR CI builds the merge-ref (head merged into main), not the head alone. Main gained Category::CacheVectorColumn in #979 (segment-cache-held decoded vector column, mapped in segment_cache_budget.rs), and measurement()'s exhaustive match has no catch-all arm -- so the branch compiled alone and failed E0004 in CI on the merged tree. Classification: Estimated, alongside CacheDecodedStored / CacheFtsReader -- the column is held by the segment cache, its size is derived from live-set dimensions, and it is released with the cache. Branch merged with origin/main (8d2fed4, clean) to reproduce the CI tree locally before the fix. Gates (this tree, release build): - cargo check / fmt / clippy: clean, 0 warnings - xerj-engine ingest_memory unit tests: 12/12 - ES-YAML conformance vs :9348, throwaway data dir: 1376 passed . 0 failed . 3 skipped . 1379 total

… Windows build Root cause ---------- #948 installed an allocator purge hook at boot whose closure calls tikv_jemalloc_ctl (epoch::advance + arena purge mallctl). The tikv crates are target-gated out on MSVC in xerj-server/Cargo.toml ([target.'cfg(not(target_env = "msvc"))'.dependencies]), but the closure was not cfg-gated, so xerj-server stopped compiling on windows-msvc: error[E0433]: cannot find module or crate `tikv_jemalloc_ctl` error: could not compile `xerj-server` (bin "xerj") due to 2 previous errors Every other jemalloc reference was already gated (the #[global_allocator] in main.rs, allocator_snapshot() in ingest_memory_trace.rs with its msvc default fallback) — this one site was missed. Blast radius ---------- The break is on main itself since #948's landing: CI's "Autoindex FD smoke (windows-latest)" has been failing on main's own merge commits (4f12618 = #1002's merge, failure) — earlier merges slid through because the job sits minutes-to-hours in the serialised runner queue and completed after the merge decision. Every PR merge-ref built off current main fails the same job (#999's run 35563877473, head 834958c, is the run that surfaced it). Fix -- The set_allocator_purge_hook statement (not the whole block) gains #[cfg(not(target_env = "msvc"))]: the engine's default no-op purge covers MSVC, and spawn_resource_sampler() keeps running on every target because allocator_snapshot() already falls back to default there. Statement-level cfg follows the existing pattern at main.rs:2011. Verification ---------- - cargo check -p xerj-server: ok (linux dev) - cargo fmt -p xerj-server -- --check: clean - the windows-msvc compile is proven by CI's FD smoke job on this head; locally only the absence of unguarded tikv references could be checked (grep: main.rs hook gated; global_allocator gated; trace snapshot gated with msvc fallback)

…-09-26) rc.75-rc.77 shipped rerank, share links, mail ingest, S3-as-index-home and the #1002/#975/#1023 measurements, and the 2026-09-26 post-merge audit (58 confirmed findings) found nearly every public surface still denying or misstating them. llms.txt had been fixed by #1033; this sweep is the rest. - llms-full.txt: NOT-IMPLEMENTED block rolled to the rc.77 reality (object storage removed from it; remaining rows re-stamped 2026-09-26 after re-reading the code); the #1017/#1009 dates corrected to each commit's own git timestamp; the ellipsis results-URL expanded to a real link. - ZERO_TOKEN_DIRECTION.md: rerank run pointers now name the actual benchmark dirs (RERANK.md carries only the pilot); the autoindex resilience row rolled to "fixed in rc.75 (#934)"; object-storage row already said rc.77 - kept. - XERJ_VS_LUCENE.md: _update_by_query "hard-capped at 10,000 hits" replaced with the post-#1023 _id-keyset search_after paging. - docs/RERANK.md: the "min_score means the same thing on every query" claim and the min_score example replaced with the measured calibration (ECE 0.10-0.31; rank by the probability, never threshold it); the egress inventory now records that storage.backend = "s3" writes index bundles since rc.77. - docs/recipes/mail-takeout-mbox.md and the takeout answer: pre-#1002 memory numbers reframed as dated before-state with the #1002 fix and the #1032 residual named. - answers/compare pages: "no email handler / no mbox handler" blanket claims replaced everywhere (mbox/.eml shipped rc.75; PST/Maildir still unreadable); capabilities tables and FAQs updated to match. - does-xerj-support-s3-alerting-plugins: the page llms.txt:50 cites as proof of the S3 status still said the opposite - rolled to the rc.77 reality (both directions work, one remaining startup refusal: an unnamed s3_bucket), with the correction stated in the open. - docs/SCRIPTING.md: every code anchor re-pinned after #1023 and the index.rs refactors (script query 26502->43775, script_score 33162-> 51918, script_fields es_compat 11051->14069, fault capture 977->997). - Landing twins, both compare/answers hubs and the sitemap regenerated via scripts/seo/build_articles.py --write; landing/llms.txt article index regenerated (title change propagation, 88 files). Verification: per-group adversarial review (12 groups), then a final gate - landing-constants-guard all checks passed; build_articles.py --check ok (189 outputs, 92 articles); gen_sitemap.py --check ok (178 URLs); stale-phrase greps ("no email handler", "local-directory simulation" as present tense, "memory is the limit today", ...) return zero hits; the diff touches only docs/, content/ and landing/ - no CHANGELOG, ROADMAP or engine changes. CHANGELOG entry follows after rebase on #1035 (single-writer rule on that file).

…/share/mbox ARE on main, xerj-org#948 is closed, residual retention re-anchored to xerj-org#1032 The hand-written status lines in landing/llms.txt drifted from the tree because nothing checks them against releases (the generated article tree is checked; these lines are not). Four defects, all filed by xerj-org#1028 and re-verified against main 2c31f96 before this commit: 1. "Not implemented" bullet listed rerank, share links and mbox/Takeout ingest as "in flight on branches, not on main". All three are on main and shipped since rc.75: rerank_stage.rs + docs/RERANK.md (xerj-org#946), `xerj share` + docs/SHARING.md (xerj-org#947), extract/mbox.rs (xerj-org#949) — and all three are already documented as usable elsewhere in the same file (rerank caveat L45, share L79/L203, mbox L3/L26/L38). Sentence removed from llms.txt L50 and llms-fu 8000 ll.txt L289. 2. The mail-export limit quoted the PRE-xerj-org#1002 numbers (66.9 GiB, did not finish under the 16 GiB cap) and linked xerj-org#948 as if open. xerj-org#948 was closed by xerj-org#1002 on 2026-09-21 (rc.76). Now states the measured post-fix truth, figures taken verbatim from the xerj-org#1002 PR body: completes in 407.7 s under the cap, peak 21,405.3 MiB (1.3x cap), verify p95 5.54 s while the breaker is engaged, 300 MB mailbox fits an 8 GiB cap (7,963.6 MiB). llms.txt L38, llms-full.txt L482. 3. Ingest-scale caveat called the RSS-runaway "an open, tracked defect". The runaway IS fixed (xerj-org#1002: at-rest retention ~677 -> ~108 MB on the 100k-doc variant), but xerj-org#1002's root cause 1 (memtable_shards) is a different structure from the 2026-07-09 scale ticket's remaining suspects — stored_value_cache / dv_cache / id_pos_cache are still "Left unbounded for now" as of index.rs:8569. Filed xerj-org#1032 so the caveat links a live tracked issue again instead of a closed one. The 20.2 GB sizing data point is now dated "before xerj-org#1002". 4. "First run" had two items numbered 4 (study M6, live since the research). Field-report step renumbered to 5. Scope deliberately narrow: obligation-language rework and the full llms.txt rewrite stay with the planned rewrite PR (research rules: one PR per llms.txt edit, docs/research/llms-txt-2026-09/README.md rule 10); xerj-org#1028's release-gate-check proposal is a separate change.

…, rc.78 queue The 2026-09-21 review of this file was stale within hours: xerj-org#950 closed 19:33, xerj-org#941 19:38, xerj-org#1015 20:49 (all 2026-09-21), xerj-org#874 2026-09-22, and the stage-1 gating trio xerj-org#937/xerj-org#938/xerj-org#939 closed the same evening — every one recorded here as open or "under way". This pass re-checks every status claim against live tracker state and rolls the file forward: - Next release: the two "in flight" items (xerj-org#1015, xerj-org#950) are closed with fixes on main riding to rc.78 — the section now records what actually landed since the rc.77 tag (xerj-org#1009, xerj-org#1017, xerj-org#1018, xerj-org#1020, xerj-org#1021, xerj-org#1023, xerj-org#1025, xerj-org#1026, xerj-org#1033, xerj-org#1034) with PR links, including xerj-org#874's budget met at ~3x margin (206 -> 64 kB per idle index) and discussion xerj-org#1012's email-labelling measurement (1.000 templated tier / 0.625 at 0.902 confidence on the hard tier — wrong-and-confident). - Open defects: now xerj-org#1031 and xerj-org#1032 only; xerj-org#1015/xerj-org#950 removed (closed); trackers sentence corrected (xerj-org#941, xerj-org#874 closed — only xerj-org#298 remains by design). - GA gate: "Close the CHANGELOG gap" marked closed 2026-09-26 (the rc.19-rc.70 backfill, PR xerj-org#1035); the record-complete note replaces the gap warning under "Shipping today". - Zero-token: stage-1 gating trio recorded as fixed (xerj-org#991, xerj-org#995, xerj-org#979) with the old measured costs kept as the before-state; the xerj-org#940 hash-seed spread annotated as a pre-fix measurement; stage-2 object storage marked done (xerj-org#965 wired in rc.77 — the old bullet contradicted this file's own "Shipping today" section); mail ingest "unreleased" corrected to "shipped in rc.75"; three stale "In flight:" labels on landed stage-1 items relabelled "History:". - Mail-ingest memory line: xerj-org#948's runaway is fixed (xerj-org#1002) — the line now carries the fixed numbers and points at xerj-org#1032 for the residual. Review line updated to 2026-09-26 (machine-checked format), scoped honestly as a desk review. The milestones line is true again: rc.78 milestone created, all four open issues triaged (xerj-org#1030/xerj-org#1031/xerj-org#1032 -> rc.78, xerj-org#298 -> GA).

cla-bot Bot added the cla-signed label Sep 21, 2026

xerj-org added 2 commits September 21, 2026 04:33

Merge remote-tracking branch 'origin/main' into fix/issue-948-ingest-…

3488068

…heap-mailbox

xerj-org merged commit 4f12618 into main Sep 21, 2026
2 of 3 checks passed

Vinz2168 mentioned this pull request Sep 24, 2026

llms.txt: "Not implemented" lists rerank, share links and mbox as not on main; #948 still presented as open after #1002 #1028

Closed

xerj-org mentioned this pull request Sep 26, 2026

Correct stale status claims across landing/, docs/ and content/ (2026-09-26 sweep) #1037

Merged

xerj-org mentioned this pull request Oct 3, 2026

engine: node dies SIGABRT under memory exhaustion — std thread-spawn sigaltstack panic hits panic="abort" #1122

Open

xerj-org mentioned this pull request Oct 20, 2026

autoindex of one small-record JSONL: ~240 KB node RSS per record (50k rows → 9.1 GiB, 300k OOM-killed); finalize-verify hydrates every doc #1940

Closed

Navigation Menu

Repository navigation

fix(engine): #948 — prune published memtable entries, charge what they retain, bound merge batches, drain on breaker - #1002

fix(engine): #948 — prune published memtable entries, charge what they retain, bound merge batches, drain on breaker#1002
xerj-org merged 3 commits into
mainfrom
fix/issue-948-ingest-heap-mailbox

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant