Repository navigation
fix(xerj-autoindex): JSON records with no prose gain a text passage (#1158 build-time half) - #1200
Merged
Merged
Conversation
The rc.83 "finalize-catalog deadlock" was not a deadlock. Diagnosis, from the incident's own evidence re-read against the client code: - The client's retry envelope was always bounded: every request carries a 300 s total timeout (esclient.rs:856, present since the #977 port), and retry_loop gives a request six attempts before handing the error back — ~31 min worst case, then the caller's own downgrade path takes over (for the alias sweep that is the #345 non-fatal warning, by design). - "All 297 threads futex_do_wait, two in ep_poll, zero CPU" is exactly the signature of reqwest::blocking: the caller parks on a oneshot (futex) while the client's internal tokio runtime polls (ep_poll). wchan cannot distinguish parked-awaiting-HTTP from parked-on-a-condvar at that granularity, so a bounded silent wait read as a lock cycle. - since_progress_s=430 > 300 confirms the loop was cycling: attempt 1 timed out at 300 s, the backoff ran, attempt 2 was in flight when the diagnostics were captured. - The server-side half stays open: the reference node runs at/over its parent memory breaker during xerj-search rebuilds (measured 2026-10-06: rss 22.5 GB against a 16 GB limit, writes 429'd for 600 s+), and one read stalling past the 300 s window under that pressure is consistent with everything captured. Tracked with the #1122 memory-ceiling class. The defect that made it LOOK like a deadlock was silence: the retry loop announced 429s (throttle_notice) but nothing at all for transport errors or 5xx. This adds failure_notice on the Attempt::Failed arm — first failure prints immediately ("autoindex: search failed: <reason>; retrying (attempt 2 of 6, 310 s in this request so far)"), then at most once per NOTICE_EVERY, sharing throttle_notice's clock so alternating 429/timeout cannot double-announce. Same announce gate: loading runs speak; probes, MCP clients and --quiet stay silent — zero output-shape change for any non-loading caller. The incident's 1479-item finalize-catalog total (implying ~1.5 M sweep paths) is not reconcilable with the corpus's actual duplicate set (58 008 files, ≤ ~3.6 k duplicates -> ≤ 4 chunks) and the journal that produced it is gone; the standing --fresh rebuild (b1791330708, fresh journal, empty plan) will pin the real number when it reaches finalize. Noted in the tracker comment rather than guessed at here. Tests: failure_notices_announce_immediately_then_hold_the_cadence pins the gate (quiet never announces nor marks the clock; a loading run announces the first failure and holds the cadence on the next); a_dropped_connection_is_retried_after_announcing pins the wire shape (read-then-drop -> transport failure -> retry -> 200). Crate suite 1267 passed / 0 failed, clippy clean, fmt applied. Reference-coding: skipped — the change is confined to this repository's own client (a stderr notice inside our own retry loop); the transport precedent for the retry policy itself is already cited in-file (quickwit/meilisearch, esclient.rs BulkAdmission header).
…1158 build-time half) The ghsa-db class, measured live: 36,263 raw advisory JSONs indexed to 193k typed records (ids, enums, numbers, dates), and `xerj code ghsa-db "lodash command injection"` answered "No passage in 'ghsa-db' matches" while the same index returned 10,000+ hits over its typed fields — passage search reads body/defs/title/text and a raw-JSON record maps none of them. Every JSON-origin corpus on the hub worked around this by pre-flattening to .txt mirrors; the workaround was undocumented and nothing warned. Two changes: 1. extract::ensure_text_passage (extract/mod.rs): a record whose flattened fields carry no prose-like string (>= 24 chars AND >= 4 whitespace tokens — below both, a string is an id/enum/date that maps keyword and leaves the record invisible) gains a `text` passage rendered from its own fields: "key: value" lines, values capped at 256 chars, passage capped at 4,096 (chars throughout — a byte cap would stop a multibyte passage at a third of its budget), scalar arrays joined so identifier aliases stay matchable, cut on char boundaries. Never runs when the record already carries prose (the query side already searches text-typed fields since rc.86, and a key/value soup would outrank the real prose in the field order) and never clobbers a record-owned `text` field. Wired at the four JSON family Data sites: json.rs emit + wrapper-array + jsonl_fallback, jsonl.rs per-line. Scope is deliberately the JSON family only — the measured class is raw-JSON mirrors; csv/yaml/xml data sites can ride the same helper when a real corpus needs it, and until then they are covered by (2). 2. textless_dataset_note (lib.rs): any dataset whose sampled mapping has no text-searchable field (text | semantic_text) warns at plan time through the progress surface ("dataset 'x' maps no text-searchable field — `xerj code` passage search will find nothing in it (sampled fields: ...)"), so the class is caught at the source for every other shape too. --quiet stays silent. Identity note: the new `text` field changes the records, so a dataset's specs change and a new generation is provisioned — the normal extractor-change path, not a #755 identity-computation break; existing state dirs re-provision their JSON datasets once to gain passage search, and every already-built corpus keeps answering as before. Tests: crate suite 1272 passed / 0 failed (5 new extractor tests — synthesis fires, prose untouched, `text` never clobbered, per-record in a mixed dataset, hard cap on char boundaries incl. multibyte; 1 jsonl test; 1 plan-time warning test covering text/semantic_text both counting as searchable). Clippy clean, fmt applied. ES-YAML conformance runs in CI on this PR; the change is autoindex-side and touches no wire behaviour. Reference-coding: skipped — the change is confined to this repository's own extractors (a rendering helper plus a plan-time note); there is no peer-engine precedent to retrieve for synthesizing a passage field. Files: - engine/crates/xerj-autoindex/src/extract/mod.rs (ensure_text_passage, carries_prose, caps) - engine/crates/xerj-autoindex/src/extract/json.rs, jsonl.rs (call sites + tests) - engine/crates/xerj-autoindex/src/lib.rs (textless_dataset_note + call in build_phase_a + test) - CHANGELOG.md ([Unreleased] Fixed)
…lope # Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
This was referenced Oct 7, 2026
Merged
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 10, 2026
The corpus-index trust window. Four PRs, all against the corpus ingest path the rc.84-86 honesty work exposed: - xerj-org#1195 (merged just past the rc.86 tag): phase-B bulk bodies coalesce across files — the per-file round-trip ceiling of xerj-org#1147 (xerj-org#10^5 records/node-day on the ghsa-scale mirror class). A/B'd, no regression: rustsec 10.1s -> 9.7s, pysec 17.4s -> 16.9s, phase B at parity on both shapes; the structural win shows at scale, which the standing corpus rebuild re-measures. - xerj-org#1197: `xerj corpus index` proves the store answers a search before printing "searchable" — the xerj-org#1183 resume that certified 642 records over a store answering every _search with store_exception. An unsearchable build is retired by exact name, not switched in. - xerj-org#1199: a retrying client request announces itself (xerj-org#1183 item 1's client half) — the retry envelope was bounded (6 x 300s) but silent, which is exactly how it read as a deadlock. The server-side stall under memory pressure stays open in xerj-org#1183 with the xerj-org#1122 class. - xerj-org#1200: JSON records with no prose gain a synthesized text passage, plus a plan-time warning for any dataset mapping no text-searchable field — the build-time half of xerj-org#1158, whose query-time half shipped in rc.86. The issue closes with both halves shipped. ROADMAP rolled to the rc.87 cut: nine open at the cut (xerj-org#1147 and xerj-org#1158 closed inside the window), the rc.86 and earlier records stand as written. Release-notes gate: PASS locally. Files: engine/Cargo.toml (version stamp), CHANGELOG.md ([Unreleased] cut into [1.0.0-rc.87]; the xerj-org#1147 entry written at this roll — the PR merged past the tag, so its entry lands here), ROADMAP.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The build-time half of #1158 (the query-time half —
resolve_fieldsfalling back to an index's own text-typed fields — shipped in rc.86):textpassage rendered from its own fields —key: valuelines, values capped at 256 chars, the passage at 4,096, scalar arrays joined so identifier aliases stay matchable. A record that already carries prose is left untouched (the query side already searches it; a key/value soup would only outrank the real prose), and a record-ownedtextfield is never clobbered.dataset 'x' maps no text-searchable field — 'xerj code' passage search will find nothing in it (sampled fields: …). Through the progress surface, so--quietstays silent.Why
The measured case:
ghsa-db, 36,263 raw advisory JSONs → 193k typed records →No passage in 'ghsa-db' matcheson every query while the same index returned 10,000+ hits over its typed fields. Every JSON-origin corpus on the hub worked around this with an undocumented pre-flatten to.txt; osv-ecosystems (251,912 raw OSV JSONs) was caught before its build burned on the same class. With this, raw-JSON mirrors are passage-searchable by default and the class is named loudly at plan time for every other shape.The prose predicate
≥ 24 chars and ≥ 4 whitespace tokens — below both, a string is an id, an enum or a date: the values that map
keywordand leave a record invisible. A record with a real prose field already mapstextand is reachable through the rc.86 query-side fallback, so synthesis there would be a regression, not a fix.Scope and identity
prefixpre-exists as text (post-#737; e.g. after upgrade) #755 identity break. Existing state dirs re-provision their JSON datasets once to gain passage search; already-built corpora keep answering exactly as before; the 100 hub corpora are unaffected (their graded snapshots map standard fields).Tests
Crate suite 1272 passed / 0 failed (was 1267): synthesis fires / prose untouched /
textnever clobbered / per-record in a mixed dataset / hard cap on char boundaries including multibyte; a JSONL variant; and the plan-time warning (text and semantic_text both count as searchable). Clippy clean, fmt applied. ES-YAML conformance runs in CI; no wire behaviour touched.Reference-coding: skipped — confined to this repository's own extractors; there is no peer-engine precedent for synthesizing a passage field.