Repository navigation
fix(autoindex,engine): refuse or stream one oversized JSON document (#1122) - #2256
Merged
Merged
Conversation
Motivation: issue #1122, field report on xerj v1.0.0-rc.97. A BM25-only autoindex died mid-run on single JSON files over 50 MB. Two other parts of that issue are already covered (#2226 helper-thread spawn, #2236 Windows commit-space cap) and are not touched here. Root cause: a JSON value at or under the 64 MiB whole-file cap is parsed into one serde_json::Value and then tokenized. xerj-fts Token owns a String per word (analyzer.rs). extract_text_value clones every string and joins array text (memtable.rs). Release panic=abort dies on the allocation before the RSS governor samples. There was no per-document byte cap. Bulk already rejects action-count bombs; it did not reject one huge body. Autoindex's JSON extractor DOM-parsed any file up to 64 MiB (extract/json.rs). A standalone reproduction of that Token ownership, not a live analyze_doc (rustc -O /tmp/token_rss.rs, 2026-10-23): 8,388,605 bytes of "word " (1,677,721 tokens) moved VmRSS from 10,160 KB after the buffer to 101,912 KB after Vec<String>, a delta of 91,752 KB (~56 bytes/token). Not repeated at 50 MB. Change: MAX_INDEXED_DOCUMENT_BYTES is 8 MiB, a constant in xerj-common, not a new setting (limits stays 15, the census stays 137). _bulk refuses that item with HTTP 413 before IgnoredAny/from_slice; neighbours still index. index_document refuses an already-parsed value before semantic embedding and analyze. validate_raw_batch / parse_raw_batch reject an oversized raw body before the DOM. POST /{index}/_doc checks the raw body length first. Autoindex streams a file between 8 MiB and 64 MiB: top-level arrays and top-level record arrays are emitted one element at a time with the same locators as the DOM path; a value over the ceiling is skipped and the file is named. A file over 64 MiB is still not read; the skip now says so. JSONL lines in the 8–16 MiB gap are not indexed. ExtractStats.truncated is not set, so the note is not the section cap. Embedding thread counts were not changed. Proxy defaults are max_concurrent 4 and batch_size 64 (embed.rs), gated by a semaphore. neural_window_concurrency defaults to 1 (range 1..=64). ONNX intra-op threads are the CPU count, the session pool defaults to 1, inflight calls default to 8, and intra-op spinning is off (onnx.rs). No per-batch thread spawn was found. The Windows report was not reproduced. Evidence (rustc 1.97.1, from engine/): - cargo test -p xerj-common --lib: 290 passed, 0 failed - cargo test -p xerj-autoindex --lib: 1639 passed, 0 failed, 2 ignored - cargo test -p xerj-engine --test es_compat_tests oversized_json_document: 1 passed - cargo clippy -p xerj-common -p xerj-autoindex -p xerj-storage -p xerj-engine -p xerj-api --all-targets -- -D warnings: exit 0 - cargo fmt --all: exit 0 Not run: a release build, the workspace test suite, and the ES-YAML gate. The new 413 applies only to a document body over 8 MiB; documents under that ceiling take the previous path. Refs #1122. Does not close it: the Windows commit-space cap is #2236. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Motivation: use-case harness job 120333509428 (run 39916368511) failed in phase 5, `xerj brain --fresh`, after the refusal path. The catalog read-back returned HTTP 500 `Segment 412cfee1-… not found` and the run exited 1 with `write catalog: reading the catalog back failed`. Main at 5ad8572 (run 39909215501, job 120281112324) passed the same phase. The oversized-JSON checks in this branch do not create or delete segments; they are not the writer of the missing file. Root cause: the harness kills the brain server and `rm -rf`s its data directory while that process can still be creating files. `rm` then prints `Directory not empty` and leaves an index directory behind. Main's passing log did this for `ax-journal` only. This run also left `autoindex-catalog`. `snapshot.json` still named a segment whose `.seg` had been unlinked, and the `.ids` sidecar was still there. `rebuild_version_map_from_segments` loads that sidecar without opening the `.seg`. The following `_search` opens every snapshot segment and treats `SegmentNotFound` as a query error so a missing file cannot silently undercount. Retries cannot help: the id stays in the snapshot. `--fresh` had already written the new catalog rows; the read-back of those rows failed on the dangling id. Change: on local open, before the version map is rebuilt, drop snapshot entries whose segment file is not there and persist the repaired snapshot. `max_seq_no` is not lowered, so those sequence numbers are not reused. A sibling segment whose file is present stays. Object-store mode is unchanged: a catalogue whose bundle is missing still refuses to open rather than shrinking the view. Evidence (rustc 1.97.1, from engine/): - cargo test -p xerj-storage --lib: 229 passed, 0 failed, including reopen_drops_a_snapshot_segment_whose_file_is_gone - cargo clippy -p xerj-storage --lib -- -D warnings: exit 0 - rustfmt 1.97.1 --check on index_store.rs: exit 0 - Live node (target/quick/xerj): two flushed segments, stop, unlink one `.seg`, restart. match_all and an ids query both returned HTTP 200 with the surviving document. The log line was `dropped snapshot segments whose files are gone … removed=1`. The on-disk snapshot no longer named the missing id. - .github/scripts/usecase-smoke.sh with that binary: PASS. Phase 5 printed `Directory not empty` for `.xerj_dashboards` and `ax-docs`, the refusal passed, and `--fresh` exited 0. server.log recorded `removed=2` twice during that reopen. Not run: the ES-YAML gate and a fat-LTO release build. The repair runs only when a local segment file named by the snapshot is already absent at open. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Owner
Author
|
Xerj Merger made the fix: a single JSON document over 8 MiB (MAX_INDEXED_DOCUMENT_BYTES) is now refused or streamed instead of being parsed whole, so one huge file can't abort the node during autoindex or _bulk (_bulk returns 413 for that item; neighbours still index). Tests added. Refs #1122, which stays open for its remaining parts. |
cursor Bot
pushed a commit
that referenced
this pull request
Oct 23, 2026
The use-case harness on c763939 failed in phase 5 (brain --fresh after a wiped data dir) with `rm: Directory not empty` on ax-journal and autoindex-catalog, then HTTP 500 `Segment … not found` while reading the catalog back. That job merged this branch into c16fad5, which does not contain the open-time repair. This is the same torn-snapshot failure #2244 hit and #2256 fixed on main (`drop_local_segments_missing_files`, `reopen_drops_a_snapshot_segment_whose_file_is_gone`). The Windows commit-budget change does not run on that Linux job: Linux and macOS still take the previous cap. No conflict. CHANGELOG keeps both Unreleased notes. No .github file was edited. An agent wrote this merge. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
5 of 8 tasks
cursor Bot
pushed a commit
that referenced
this pull request
Oct 23, 2026
Main moved ahead through 8920c06: FTS case-aliased sidecar publish (#2261), MCP token-budget identity (#2255), oversized JSON refusal (#2256), memtable-pressure drain (#2244), AND-omitted match (#2278), _count min_score (#2277), cluster primary routing (#2271), and xerj_search match_line (#2269). The only conflict was CHANGELOG.md. Both sides inserted a leading Fixed entry. Kept main's _count min_score note, the shared AVX2 note, and main's MCP token-budget note, then this branch's shutdown-flush note and the stale node.lock reclaim (#2233). Main's oversized-JSON and missing-segment notes stayed below them. engine.rs and main.rs changed on both sides and merged without conflicts. Startup still takes the data-dir lock through node_lock::acquire before any index open, and HTTP listeners still stay bound until flush_all_force returns. Main's publish_staged rollback, FTS layout v3, and the cluster shard handler installed before HTTP accept are unchanged. No files under .github/ were modified. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
cursor Bot
pushed a commit
that referenced
this pull request
Oct 23, 2026
Conflicts were CHANGELOG.md only. engine.rs and landing/llms-full.txt auto-merged; governor.rs was unchanged on main since the merge base. #1981 (this branch): a flat RSS at or above the watermark releases the breaker after 2 seconds and does not re-arm inside the 4 MiB band. While that latch stays engaged, drain requests double from 2 seconds up to 15 minutes unless RSS falls by more than 4 MiB. corpus index reports eta_quality=stalled at 0/N and after 10 minutes of silence on a phase with no denominator. main, kept alongside that: - #2283 / #2281: a matching index template's semantic_text properties and index.default_pipeline are applied on index create. - #2240 packed f32 semantic embeddings (engine ingest/storage; no overlap with the drain schedule). - #2267 hub no-coverage routing, including the llms-full.txt exit text. The 0/N stall sentence stays. - #2261 case-aliased field sidecars, #2255 JSON token budgets, and #2256 oversized JSON documents. Their changelog bullets stay. The two drain clocks are unchanged: RSS uses MemoryDrainSchedule, the summed-memtable latch stays on the fixed 2 second window. No .github file was edited. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
cursor Bot
pushed a commit
that referenced
this pull request
Oct 23, 2026
CHANGELOG.md was the only conflict. Keep both Unreleased Fixed entries: this branch's #2234 refresh-during-merge note, and main's #2281 index-template semantic_text note. No engine source conflicted. Main's changes are taken as landed: - packed f32 semantic embeddings (#2240) - hub no-coverage routing (#2267) - template semantic_text and default_pipeline on index create (#2283) - case-aliased FTS field sidecars (#2261) - MCP JSON token-budget identity (#2255) - refusal or streaming of one oversized JSON document (#2256) index.rs, es_compat.rs, and the autoindex client auto-merged. _refresh still publishes an unflushed memtable without waiting on a background merge, and the merge encode still runs on the merge pool (#2234). That encode path now publishes the case-distinguished FTS filename layout from main. _flush still checkpoints the WAL. No .github files were touched. Verified with Rust 1.97.1, CARGO_BUILD_JOBS=3: - cargo test --profile ci-test -p xerj-api -p xerj-engine -p xerj-autoindex (205 targets ok; 4022 passed, 0 failed, 31 ignored) - cargo build -p xerj-api -p xerj-engine -p xerj-autoindex (Finished dev profile in 3m 12s) cargo +1.97.1 fmt -p xerj-api -p xerj-engine -p xerj-autoindex -- --check made no changes. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
This was referenced Oct 23, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
A single JSON document over 8 MiB is no longer parsed and tokenized inside one request, so one 50 MB file cannot abort the node during autoindex or
_bulk. Refs #1122. Other parts of that issue remain (#2226 is merged; the Windows commit-space cap is #2236).The field report (xerj v1.0.0-rc.97, BM25-only autoindex, ~191k docs,
--workers 8) says single JSON files over 50 MB killed the node mid-run. The mechanism is in this tree: a file at or under the 64 MiB whole-file cap is loaded andserde_json::from_slice'd into oneValue(extract/json.rs).Tokenowns aStringper word (xerj-ftsanalyzer.rs).extract_text_valueclones every string and joins array text (memtable.rs). The release profile ispanic = "abort", so the allocation dies before the RSS governor samples. Bulk already rejects an action-count bomb. It did not reject one huge body.The ceiling is a fixed constant,
MAX_INDEXED_DOCUMENT_BYTES= 8 MiB, inxerj-common. It is not a new setting (limitsstays 15; the census stays 137)._bulkreturns HTTP 413 for that item beforeIgnoredAny/from_slice. Neighbours still index. 413 is settled, so autoindex does not retry it and does not treat a lone item as the whole-request action-count refusal.index_documentrefuses an already-parsed value before semantic embedding andanalyze_doc.validate_raw_batch/parse_raw_batchreject an oversized raw body before the DOM. Bulk filters first, so one bad item does not fail the batch.POST /{index}/_docchecks the raw body length beforefrom_slice..jsonfile between 8 MiB and 64 MiB. A top-level array, and a top-level array of records inside an object, is emitted one element at a time with the same locators as the DOM path (e{i},{key}:e{i}, groupNoneon the longest array). A value over the ceiling is skipped. The skip names the file and the limit. Files at or under 8 MiB keep the DOM path. Files over 64 MiB are still not read; the skip now says so. JSONL lines in the former 8–16 MiB gap are not indexed.ExtractStats.truncatedis not set, so this is not reported as the 4096-section cap.Opening a local index whose snapshot names a segment file that is already gone now drops that segment instead of failing every later
_searchwithSegment … not found. A sibling segment that is still on disk stays readable, and sequence numbers from the missing segment are not reused. Object-store mode is unchanged: a catalogue whose bundle is missing still refuses to open. This is what letxerj brain --freshfinish its catalog read-back after a data directory was only partly removed (use-case harness job 120333509428).Evidence
Token ownership, not a live
analyze_docrun (rustc -O /tmp/token_rss.rs, 2026-10-23): 8,388,605 bytes ofword(1,677,721 tokens) moved VmRSS from 10,160 KB after the buffer to 101,912 KB afterVec<String>. Delta 91,752 KB, about 56 bytes per token. Not repeated at 50 MB.The engine test sends one small explicit id, one document of 8 MiB + 1, and one small auto-id in the same bulk. The oversized item is 413 and names the limit; the other two are 201 and searchable. A following
index_documentof the same string is refused and the small document is still there.Streaming vs DOM: files over a test ceiling and under 8 MiB in production emit the same locators, groups, and fields for an array, a wrapper object, a two-array split (#1856/#1978), and a scalar array. An oversized element keeps its index. A single object over the ceiling produces no records and a refusal that contains
t.jsonand the limit.Use-case harness (
Segment … not foundon--fresh)Main at
5ad85724epassed the same harness (run 39909215501, job 120281112324). This branch's run 39916368511, job 120333509428, failed only in phase 5. The oversized-JSON checks do not create or delete segments.The harness kills the brain server and
rm -rfs its data directory while that process can still be creating files.rmthen printsDirectory not emptyand leaves an index directory behind. Main's passing log did this forax-journalonly. This run also leftautoindex-catalog.snapshot.jsonstill named a segment whose.seghad been unlinked, and the.idssidecar was still there.rebuild_version_map_from_segmentsloads that sidecar without opening the.seg. The following_searchopens every snapshot segment and treatsSegmentNotFoundas a query error, so retries cannot clear it.--freshhad already written the new catalog rows; the read-back of those rows failed on the dangling id.Live node (
target/quick/xerj): two flushed segments, stop, unlink one.seg, restart.match_alland an ids query both returned HTTP 200 with the surviving document. The log line wasdropped snapshot segments whose files are gone … removed=1. The on-disk snapshot no longer named the missing id..github/scripts/usecase-smoke.shwith that binary: PASS. Phase 5 printedDirectory not emptyfor.xerj_dashboardsandax-docs, the refusal passed, and--freshexited 0.server.logrecordedremoved=2twice during that reopen.Checks
cargo fmt --all(CI runsrustfmt --checkandclippy -D warnings)cargo build --release -j 32 -p <crate>cargo test -p <crate>passes (cargo checkis not sufficient — test code is interleaved with production code)cargo run --release -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml), or: not applicable because this change is docs/landing-onlyengine/tests/es-compat-yaml/yaml/docs/CONTRIBUTION_REVIEW.mddocs/CONTRIBUTION_REVIEW.md. Record a negative control that exits non-zero with no artifact extracted, installed, published, or served. Keep verification and its action in one block; rerun the control with the failing exit discarded and with the next documented block appended, and require that neither reaches an artifact or unsafe/privileged path. List tested variants and unsupported variants separately, and verify every publication copy. Otherwise mark this itemN/Awith a reason.Not run: release build; workspace
cargo test; ES-YAML gate. No new YAML case: the refusal is an internal bulk/index limit for a body over 8 MiB, and documents under that ceiling take the previous path. The segment repair is local-open recovery of a file the snapshot already cannot open; object-store mode still refuses a missing bundle. Operational-recipe audit is N/A: this does not change a shell recipe, systemd unit, or install doc. CHANGELOG Unreleased is updated..github/workflowsis not touched.Contribution review: branch is independent of #2236, based on current
main(5ad85724e). Fail-closed only for the oversized value; arrays of small records still index; locators for those records match the DOM path. The dangling-segment repair drops only a local snapshot entry whose.segisNotFound; any other open error leaves the entry in place,max_seq_nois not lowered, and a present sibling stays. No new durable field. A per-item 413 does not journal the file as a server failure (settled item error). Linux and Windows share the byte check; there is no Windows-specific code.PUT /{index}/_doc/{id}and create-with-id still build aValuein axum'sJsonextractor before the value walk, so that path can still allocate the DOM. Autoindex uses_bulk, which checks the raw bytes first. A Keep note larger than 8 MiB is not recognised as a note.Embedding threads
Not changed. Inspected, not reproduced:
max_concurrentdefaults to 4 andbatch_sizeto 64 (engine/crates/xerj-ai/src/embed.rs). In-flight requests sit behind a semaphore ofmax_concurrent, and a batch is split into chunks ofbatch_size.embedding.neural_window_concurrencydefaults to 1, allowed range 1..=64.threads_for(Latency)(CPU count),onnx_session_pool_sizedefaults to 1,onnx_max_inflight_callsdefaults to 8, and intra-op spinning is disabled (onnx.rs). No path was found that spawns a thread per batch item.Provenance
cargo test -p xerj-common --lib(290 passed);cargo test -p xerj-autoindex --lib(1639 passed, 0 failed, 2 ignored); the onees_compat_testscase named above (passed in 0.05s); clippy-D warningsonxerj-common,xerj-autoindex,xerj-storage,xerj-engine, andxerj-apiwith--all-targets(exit 0);cargo fmt --all(exit 0). The embedding defaults cited above were read from those files in this session. For the harness failure:cargo test -p xerj-storage --lib(229 passed, 0 failed); clippy-D warningsonxerj-storage --lib(exit 0); a livetarget/quick/xerjreopen after unlinking one.seg(HTTP 200, surviving document only,removed=1);.github/scripts/usecase-smoke.shwith that binary (exit 0,--freshexit 0,removed=2inserver.log). Main's passing harness is run 39909215501.analyze_docof short tokens inside a process that previously died on a 50 MB file. Falsified if a document under 8 MiB still aborts the node, or if streaming a 50 MB array of small objects still builds a DOM of the whole file. Not measured: a live Windows run, a 50 MB node death, or the embedding-server thread count under batch-64 load. A thread-per-item spawn that this reading missed would falsify the decision to leave those counts alone. The ES-YAML gate and a fat-LTO release build were not run for the segment repair; it runs only when a local segment file named by the snapshot is already absent at open.