8000 fix(autoindex,engine): refuse or stream one oversized JSON document (#1122) by xerj-org · Pull Request #2256 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

fix(autoindex,engine): refuse or stream one oversized JSON document (#1122) - #2256

Merged
xerj-org merged 3 commits into
mainfrom
cursor/large-json-ingest-bound-a613
Oct 23, 2026
Merged

xerj-org merged 3 commits into
mainfrom
cursor/large-json-ingest-bound-a613

Conversation

@xerj-org
@xerj-org xerj-org commented Oct 23, 2026 •
Copy link
Copy Markdown
Owner

What this changes, and why

A single JSON document over 8 MiB is no longer parsed and tokenized inside one request, so one 50 MB file cannot abort the node during autoindex or _bulk. Refs #1122. Other parts of that issue remain (#2226 is merged; the Windows commit-space cap is #2236).

The field report (xerj v1.0.0-rc.97, BM25-only autoindex, ~191k docs, --workers 8) says single JSON files over 50 MB killed the node mid-run. The mechanism is in this tree: a file at or under the 64 MiB whole-file cap is loaded and serde_json::from_slice'd into one Value (extract/json.rs). Token owns a String per word (xerj-fts analyzer.rs). extract_text_value clones every string and joins array text (memtable.rs). The release profile is panic = "abort", so the allocation dies before the RSS governor samples. Bulk already rejects an action-count bomb. It did not reject one huge body.

The ceiling is a fixed constant, MAX_INDEXED_DOCUMENT_BYTES = 8 MiB, in xerj-common. It is not a new setting (limits stays 15; the census stays 137).

  • _bulk returns HTTP 413 for that item before IgnoredAny / from_slice. Neighbours still index. 413 is settled, so autoindex does not retry it and does not treat a lone item as the whole-request action-count refusal.
  • index_document refuses an already-parsed value before semantic embedding and analyze_doc.
  • validate_raw_batch / parse_raw_batch reject an oversized raw body before the DOM. Bulk filters first, so one bad item does not fail the batch.
  • POST /{index}/_doc checks the raw body length before from_slice.
  • Autoindex streams a .json file between 8 MiB and 64 MiB. A top-level array, and a top-level array of records inside an object, is emitted one element at a time with the same locators as the DOM path (e{i}, {key}:e{i}, group None on the longest array). A value over the ceiling is skipped. The skip names the file and the limit. Files at or under 8 MiB keep the DOM path. Files over 64 MiB are still not read; the skip now says so. JSONL lines in the former 8–16 MiB gap are not indexed. ExtractStats.truncated is not set, so this is not reported as the 4096-section cap.

Opening a local index whose snapshot names a segment file that is already gone now drops that segment instead of failing every later _search with Segment … not found. A sibling segment that is still on disk stays readable, and sequence numbers from the missing segment are not reused. Object-store mode is unchanged: a catalogue whose bundle is missing still refuses to open. This is what let xerj brain --fresh finish its catalog read-back after a data directory was only partly removed (use-case harness job 120333509428).

Evidence

Token ownership, not a live analyze_doc run (rustc -O /tmp/token_rss.rs, 2026-10-23): 8,388,605 bytes of word (1,677,721 tokens) moved VmRSS from 10,160 KB after the buffer to 101,912 KB after Vec<String>. Delta 91,752 KB, about 56 bytes per token. Not repeated at 50 MB.

cargo +1.97.1 test -j 2 -p xerj-common --lib
# 290 passed; 0 failed

cargo +1.97.1 test -j 2 -p xerj-autoindex --lib -- --test-threads=8
# 1639 passed; 0 failed; 2 ignored

cargo +1.97.1 test -j 2 -p xerj-engine --test es_compat_tests oversized_json_document -- --test-threads=1
# oversized_json_document_is_refused_and_its_neighbour_is_indexed ... ok

cargo +1.97.1 clippy -j 2 -p xerj-common -p xerj-autoindex -p xerj-storage -p xerj-engine -p xerj-api --all-targets -- -D warnings
# exit 0

cargo +1.97.1 fmt --all
# exit 0

The engine test sends one small explicit id, one document of 8 MiB + 1, and one small auto-id in the same bulk. The oversized item is 413 and names the limit; the other two are 201 and searchable. A following index_document of the same string is refused and the small document is still there.

Streaming vs DOM: files over a test ceiling and under 8 MiB in production emit the same locators, groups, and fields for an array, a wrapper object, a two-array split (#1856/#1978), and a scalar array. An oversized element keeps its index. A single object over the ceiling produces no records and a refusal that contains t.json and the limit.

Use-case harness (Segment … not found on --fresh)

Main at 5ad85724e passed the same harness (run 39909215501, job 120281112324). This branch's run 39916368511, job 120333509428, failed only in phase 5. The oversized-JSON checks do not create or delete segments.

The harness kills the brain server and rm -rfs its data directory while that process can still be creating files. rm then prints Directory not empty and leaves an index directory behind. Main's passing log did this for ax-journal only. This run also left autoindex-catalog. snapshot.json still named a segment whose .seg had been unlinked, and the .ids sidecar was still there. rebuild_version_map_from_segments loads that sidecar without opening the .seg. The following _search opens every snapshot segment and treats SegmentNotFound as a query error, so retries cannot clear it. --fresh had already written the new catalog rows; the read-back of those rows failed on the dangling id.

cargo +1.97.1 test -p xerj-storage --lib
# 229 passed, 0 failed, including
# reopen_drops_a_snapshot_segment_whose_file_is_gone
# and catalog_referencing_a_missing_bundle_refuses_to_open

cargo +1.97.1 clippy -p xerj-storage --lib -- -D warnings
# exit 0

Live node (target/quick/xerj): two flushed segments, stop, unlink one .seg, restart. match_all and an ids query both returned HTTP 200 with the surviving document. The log line was dropped snapshot segments whose files are gone … removed=1. The on-disk snapshot no longer named the missing id.

.github/scripts/usecase-smoke.sh with that binary: PASS. Phase 5 printed Directory not empty for .xerj_dashboards and ax-docs, the refusal passed, and --fresh exited 0. server.log recorded removed=2 twice during that reopen.

Checks

  • cargo fmt --all (CI runs rustfmt --check and clippy -D warnings)
  • Scoped release build of the crates touched: cargo build --release -j 32 -p <crate>
  • cargo test -p <crate> passes (cargo check is not sufficient — test code is interleaved with production code)
  • ES-YAML conformance suite at 0 failed (cargo run --release -p es-yaml-runner -- --dir tests/es-compat-yaml/yaml), or: not applicable because this change is docs/landing-only
  • New ES-compatible behaviour has a matching YAML case under engine/tests/es-compat-yaml/yaml/
  • Docs updated if user-visible behaviour changed
  • For non-trivial changes: the applicable audit in docs/CONTRIBUTION_REVIEW.md
  • For non-trivial changes: if this PR changes an operational document or runnable recipe, complete the operational audit in docs/CONTRIBUTION_REVIEW.md. Record a negative control that exits non-zero with no artifact extracted, installed, published, or served. Keep verification and its action in one block; rerun the control with the failing exit discarded and with the next documented block appended, and require that neither reaches an artifact or unsafe/privileged path. List tested variants and unsupported variants separately, and verify every publication copy. Otherwise mark this item N/A with a reason.

Not run: release build; workspace cargo test; ES-YAML gate. No new YAML case: the refusal is an internal bulk/index limit for a body over 8 MiB, and documents under that ceiling take the previous path. The segment repair is local-open recovery of a file the snapshot already cannot open; object-store mode still refuses a missing bundle. Operational-recipe audit is N/A: this does not change a shell recipe, systemd unit, or install doc. CHANGELOG Unreleased is updated. .github/workflows is not touched.

Contribution review: branch is independent of #2236, based on current main (5ad85724e). Fail-closed only for the oversized value; arrays of small records still index; locators for those records match the DOM path. The dangling-segment repair drops only a local snapshot entry whose .seg is NotFound; any other open error leaves the entry in place, max_seq_no is not lowered, and a present sibling stays. No new durable field. A per-item 413 does not journal the file as a server failure (settled item error). Linux and Windows share the byte check; there is no Windows-specific code. PUT /{index}/_doc/{id} and create-with-id still build a Value in axum's Json extractor before the value walk, so that path can still allocate the DOM. Autoindex uses _bulk, which checks the raw bytes first. A Keep note larger than 8 MiB is not recognised as a note.

Embedding threads

Not changed. Inspected, not reproduced:

  • Proxy max_concurrent defaults to 4 and batch_size to 64 (engine/crates/xerj-ai/src/embed.rs). In-flight requests sit behind a semaphore of max_concurrent, and a batch is split into chunks of batch_size.
  • embedding.neural_window_concurrency defaults to 1, allowed range 1..=64.
  • ONNX intra-op threads are threads_for(Latency) (CPU count), onnx_session_pool_size defaults to 1, onnx_max_inflight_calls defaults to 8, and intra-op spinning is disabled (onnx.rs). No path was found that spawns a thread per batch item.

Provenance

  • Written by: AI agent (Cursor Grok 4.7), run in a cloud agent on this branch.
  • Verified (commands run, output observed): the RSS probe numbers above; cargo test -p xerj-common --lib (290 passed); cargo test -p xerj-autoindex --lib (1639 passed, 0 failed, 2 ignored); the one es_compat_tests case named above (passed in 0.05s); clippy -D warnings on xerj-common, xerj-autoindex, xerj-storage, xerj-engine, and xerj-api with --all-targets (exit 0); cargo fmt --all (exit 0). The embedding defaults cited above were read from those files in this session. For the harness failure: cargo test -p xerj-storage --lib (229 passed, 0 failed); clippy -D warnings on xerj-storage --lib (exit 0); a live target/quick/xerj reopen after unlinking one .seg (HTTP 200, surviving document only, removed=1); .github/scripts/usecase-smoke.sh with that binary (exit 0, --fresh exit 0, removed=2 in server.log). Main's passing harness is run 39909215501.
  • Assumed (not tested, and what would falsify it): that an 8 MiB ceiling keeps a real analyze_doc of short tokens inside a process that previously died on a 50 MB file. Falsified if a document under 8 MiB still aborts the node, or if streaming a 50 MB array of small objects still builds a DOM of the whole file. Not measured: a live Windows run, a 50 MB node death, or the embedding-server thread count under batch-64 load. A thread-per-item spawn that this reading missed would falsify the decision to leave those counts alone. The ES-YAML gate and a fat-LTO release build were not run for the segment repair; it runs only when a local segment file named by the snapshot is already absent at open.
Open in Web Open in Cursor 

Motivation: issue #1122, field report on xerj v1.0.0-rc.97. A BM25-only
autoindex died mid-run on single JSON files over 50 MB. Two other parts
of that issue are already covered (#2226 helper-thread spawn, #2236
Windows commit-space cap) and are not touched here.

Root cause: a JSON value at or under the 64 MiB whole-file cap is parsed
into one serde_json::Value and then tokenized. xerj-fts Token owns a
String per word (analyzer.rs). extract_text_value clones every string
and joins array text (memtable.rs). Release panic=abort dies on the
allocation before the RSS governor samples. There was no per-document
byte cap. Bulk already rejects action-count bombs; it did not reject
one huge body. Autoindex's JSON extractor DOM-parsed any file up to
64 MiB (extract/json.rs).

A standalone reproduction of that Token ownership, not a live
analyze_doc (rustc -O /tmp/token_rss.rs, 2026-10-23): 8,388,605 bytes
of "word " (1,677,721 tokens) moved VmRSS from 10,160 KB after the
buffer to 101,912 KB after Vec<String>, a delta of 91,752 KB
(~56 bytes/token). Not repeated at 50 MB.

Change: MAX_INDEXED_DOCUMENT_BYTES is 8 MiB, a constant in xerj-common,
not a new setting (limits stays 15, the census stays 137). _bulk refuses
that item with HTTP 413 before IgnoredAny/from_slice; neighbours still
index. index_document refuses an already-parsed value before semantic
embedding and analyze. validate_raw_batch / parse_raw_batch reject an
oversized raw body before the DOM. POST /{index}/_doc checks the raw
body length first. Autoindex streams a file between 8 MiB and 64 MiB:
top-level arrays and top-level record arrays are emitted one element at
a time with the same locators as the DOM path; a value over the ceiling
is skipped and the file is named. A file over 64 MiB is still not read;
the skip now says so. JSONL lines in the 8–16 MiB gap are not indexed.
ExtractStats.truncated is not set, so the note is not the section cap.

Embedding thread counts were not changed. Proxy defaults are
max_concurrent 4 and batch_size 64 (embed.rs), gated by a semaphore.
neural_window_concurrency defaults to 1 (range 1..=64). ONNX intra-op
threads are the CPU count, the session pool defaults to 1, inflight
calls default to 8, and intra-op spinning is off (onnx.rs). No per-batch
thread spawn was found. The Windows report was not reproduced.

Evidence (rustc 1.97.1, from engine/):
- cargo test -p xerj-common --lib: 290 passed, 0 failed
- cargo test -p xerj-autoindex --lib: 1639 passed, 0 failed, 2 ignored
- cargo test -p xerj-engine --test es_compat_tests oversized_json_document: 1 passed
- cargo clippy -p xerj-common -p xerj-autoindex -p xerj-storage -p xerj-engine -p xerj-api --all-targets -- -D warnings: exit 0
- cargo fmt --all: exit 0

Not run: a release build, the workspace test suite, and the ES-YAML
gate. The new 413 applies only to a document body over 8 MiB; documents
under that ceiling take the previous path.

Refs #1122. Does not close it: the Windows commit-space cap is #2236.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
@cla-bot cla-bot Bot added the cla-signed label Oct 23, 2026
cursoragent and others added 2 commits October 23, 2026 11:01
Motivation: use-case harness job 120333509428 (run 39916368511) failed
in phase 5, `xerj brain --fresh`, after the refusal path. The catalog
read-back returned HTTP 500 `Segment 412cfee1-… not found` and the run
exited 1 with `write catalog: reading the catalog back failed`. Main at
5ad8572 (run 39909215501, job 120281112324) passed the same phase.
The oversized-JSON checks in this branch do not create or delete
segments; they are not the writer of the missing file.

Root cause: the harness kills the brain server and `rm -rf`s its data
directory while that process can still be creating files. `rm` then
prints `Directory not empty` and leaves an index directory behind.
Main's passing log did this for `ax-journal` only. This run also left
`autoindex-catalog`. `snapshot.json` still named a segment whose `.seg`
had been unlinked, and the `.ids` sidecar was still there.
`rebuild_version_map_from_segments` loads that sidecar without opening
the `.seg`. The following `_search` opens every snapshot segment and
treats `SegmentNotFound` as a query error so a missing file cannot
silently undercount. Retries cannot help: the id stays in the snapshot.
`--fresh` had already written the new catalog rows; the read-back of
those rows failed on the dangling id.

Change: on local open, before the version map is rebuilt, drop snapshot
entries whose segment file is not there and persist the repaired
snapshot. `max_seq_no` is not lowered, so those sequence numbers are
not reused. A sibling segment whose file is present stays. Object-store
mode is unchanged: a catalogue whose bundle is missing still refuses
to open rather than shrinking the view.

Evidence (rustc 1.97.1, from engine/):
- cargo test -p xerj-storage --lib: 229 passed, 0 failed, including
  reopen_drops_a_snapshot_segment_whose_file_is_gone
- cargo clippy -p xerj-storage --lib -- -D warnings: exit 0
- rustfmt 1.97.1 --check on index_store.rs: exit 0
- Live node (target/quick/xerj): two flushed segments, stop, unlink one
  `.seg`, restart. match_all and an ids query both returned HTTP 200
  with the surviving document. The log line was
  `dropped snapshot segments whose files are gone … removed=1`.
  The on-disk snapshot no longer named the missing id.
- .github/scripts/usecase-smoke.sh with that binary: PASS. Phase 5
  printed `Directory not empty` for `.xerj_dashboards` and `ax-docs`,
  the refusal passed, and `--fresh` exited 0. server.log recorded
  `removed=2` twice during that reopen.

Not run: the ES-YAML gate and a fat-LTO release build. The repair runs
only when a local segment file named by the snapshot is already absent
at open.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
@xerj-org
Copy link
Copy Markdown
Owner Author

Xerj Merger made the fix: a single JSON document over 8 MiB (MAX_INDEXED_DOCUMENT_BYTES) is now refused or streamed instead of being parsed whole, so one huge file can't abort the node during autoindex or _bulk (_bulk returns 413 for that item; neighbours still index). Tests added. Refs #1122, which stays open for its remaining parts.

@xerj-org
xerj-org merged commit 61c9c70 into main Oct 23, 2026
24 checks passed
cursor Bot pushed a commit that referenced this pull request Oct 23, 2026
The use-case harness on c763939 failed in phase 5 (brain --fresh
after a wiped data dir) with `rm: Directory not empty` on
ax-journal and autoindex-catalog, then HTTP 500
`Segment … not found` while reading the catalog back. That job
merged this branch into c16fad5, which does not contain the
open-time repair.

This is the same torn-snapshot failure #2244 hit and #2256 fixed
on main (`drop_local_segments_missing_files`,
`reopen_drops_a_snapshot_segment_whose_file_is_gone`). The Windows
commit-budget change does not run on that Linux job: Linux and
macOS still take the previous cap. No conflict. CHANGELOG keeps
both Unreleased notes. No .github file was edited.

An agent wrote this merge.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
cursor Bot pushed a commit that referenced this pull request Oct 23, 2026
Main moved ahead through 8920c06: FTS case-aliased sidecar publish
(#2261), MCP token-budget identity (#2255), oversized JSON refusal
(#2256), memtable-pressure drain (#2244), AND-omitted match (#2278),
_count min_score (#2277), cluster primary routing (#2271), and
xerj_search match_line (#2269).

The only conflict was CHANGELOG.md. Both sides inserted a leading
Fixed entry. Kept main's _count min_score note, the shared AVX2 note,
and main's MCP token-budget note, then this branch's shutdown-flush
note and the stale node.lock reclaim (#2233). Main's oversized-JSON
and missing-segment notes stayed below them.

engine.rs and main.rs changed on both sides and merged without
conflicts. Startup still takes the data-dir lock through
node_lock::acquire before any index open, and HTTP listeners still
stay bound until flush_all_force returns. Main's publish_staged
rollback, FTS layout v3, and the cluster shard handler installed
before HTTP accept are unchanged. No files under .github/ were
modified.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
cursor Bot pushed a commit that referenced this pull request Oct 23, 2026
Conflicts were CHANGELOG.md only. engine.rs and landing/llms-full.txt
auto-merged; governor.rs was unchanged on main since the merge base.

#1981 (this branch): a flat RSS at or above the watermark releases
the breaker after 2 seconds and does not re-arm inside the 4 MiB
band. While that latch stays engaged, drain requests double from
2 seconds up to 15 minutes unless RSS falls by more than 4 MiB.
corpus index reports eta_quality=stalled at 0/N and after 10 minutes
of silence on a phase with no denominator.

main, kept alongside that:
- #2283 / #2281: a matching index template's semantic_text properties
  and index.default_pipeline are applied on index create.
- #2240 packed f32 semantic embeddings (engine ingest/storage; no
  overlap with the drain schedule).
- #2267 hub no-coverage routing, including the llms-full.txt exit
  text. The 0/N stall sentence stays.
- #2261 case-aliased field sidecars, #2255 JSON token budgets, and
  #2256 oversized JSON documents. Their changelog bullets stay.

The two drain clocks are unchanged: RSS uses MemoryDrainSchedule,
the summed-memtable latch stays on the fixed 2 second window.

No .github file was edited.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
cursor Bot pushed a commit that referenced this pull request Oct 23, 2026
CHANGELOG.md was the only conflict. Keep both Unreleased Fixed
entries: this branch's #2234 refresh-during-merge note, and main's
#2281 index-template semantic_text note.

No engine source conflicted. Main's changes are taken as landed:

- packed f32 semantic embeddings (#2240)
- hub no-coverage routing (#2267)
- template semantic_text and default_pipeline on index create (#2283)
- case-aliased FTS field sidecars (#2261)
- MCP JSON token-budget identity (#2255)
- refusal or streaming of one oversized JSON document (#2256)

index.rs, es_compat.rs, and the autoindex client auto-merged.
_refresh still publishes an unflushed memtable without waiting on a
background merge, and the merge encode still runs on the merge pool
(#2234). That encode path now publishes the case-distinguished FTS
filename layout from main. _flush still checkpoints the WAL.

No .github files were touched.

Verified with Rust 1.97.1, CARGO_BUILD_JOBS=3:
- cargo test --profile ci-test -p xerj-api -p xerj-engine -p xerj-autoindex
  (205 targets ok; 4022 passed, 0 failed, 31 ignored)
- cargo build -p xerj-api -p xerj-engine -p xerj-autoindex
  (Finished dev profile in 3m 12s)

cargo +1.97.1 fmt -p xerj-api -p xerj-engine -p xerj-autoindex -- --check
made no changes.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0