Repository navigation
fix(xerj-common): xerj code unions own text fields as recall legs - #1246
Merged
Merged
Conversation
Refs #1244 (stays open until the re-indexed corpus passes the pre-registered g7-cve-records-2026-10-08 suite). Motivation: the pre-registered g7-cve-records-2026-10-08 suite scored 0/7 against the cve-records corpus (402k CVE records from CVEProject/cvelistV5). Every needle's prose lives in schema-named fields (`containers_cna_descriptions`, `containers_cna_title`); the needle records are rank 1 (18.76) queried on those fields directly. But `resolve_fields` sent only `text^0.5`: the synthesized `text` field is mapped by a handful of shards and EMPTY in every record that carries schema-named prose (the #1158 synthesis correctly skips them), and its mere presence made `out` non-empty, so the #1158 own-field fallback never fired. The whole corpus answered noise. Fix (two rules in resolve_fields): 1. Own text fields join the multi_match as ^0.5 recall legs even when standard fields ARE mapped. Safe against score ballooning because the multi_match default best_fields executes as dis_max in this engine (max, not sum): a ^0.5 leg can only win by genuinely outranking the content fields at half weight — the same posture text^0.5 got from #1238. 2. Union legs must clear a quarter-coverage gate: an own field joins only when at least div_ceil(indices/4) indices in the mapping union map it. Measured on the two live corpora, coverage has an empty middle: cve-records schema prose at 37-96% of 118 indices (descriptions 95.8%), exploit-pocs-2026 frontmatter junk at <=1% of 5,644 (2,143 own text fields, all noise — the old sorted 24-cap kept a literal `$comment` and cut `Summary`). Verified by hand on exploit-pocs-2026: gated list = old list + code^0.5 only (code 46.2%), top-5 identical, totals 39 -> 43 (pure recall). The #1158 floor (no standard field mapped anywhere) keeps ANY own field, gate off: its only alternative is ["body"], a field no index maps — the MULTIMATCH_DEFECT silent zero. Layer 2 from the issue (synthesizing passages from descriptions[]) is UNNECESSARY: the prose already sits in searchable text-typed fields; carries_prose correctly skips synthesis. Verified in xerj-autoindex/src/extract/mod.rs. Verification: with the fix, both needles present in the salvaged half-index (CVE-2021-44228, CVE-2014-0160) jump to rank 1 with real scores (16.84 / 21.68). The other five needles were ABSENT from the index — the corpus was the salvaged 53% of a crashed autoindex run (213,851 of 402,695 files); a full re-index is running and the suite re-runs against it before #1244 closes. Tests: reversed the deliberate pin a_corpus_with_body_mapped_never_grows_own_fields -> own_text_fields_join_as_recall_legs_beside_body; added the_cve_records_shape_joins_prose_fields_beside_synthesized_text and union_legs_need_quarter_coverage_but_the_floor_takes_any. xerj-common --lib 199/0, xerj-autoindex --lib 1299/0. Files: - crates/xerj-common/src/xccode/fields.rs (resolve_fields, own_text_fields coverage counting)
This was referenced Oct 8, 2026
thomas-villani
pushed a commit
to thomas-villani/xerj
that referenced
this pull request
Oct 9, 2026
Round 2 of xerj-org#1244, found by the pre-registered g7-vuln-fix-commits suite (committed to corpus-hub as PR xerj-org#1249 BEFORE retrieval): with the quarter-union gate from xerj-org#1246, every query against vuln-fix-commits answered from the project-kb tooling source and ZERO payload records were reachable. Measured on the live corpus (:19295, 26 indices): - payload indices map message/patch/text and NO standard field; their prose lives in 'message' (needle CVE-2026-9856 is rank 1 at 36.35 queried on message directly) - 'text' is the xerj-org#1158 synthesized slot, EMPTY in every payload record (carries_prose skips records that already have a text-typed field) - the exact cve-records shape, one level deeper - 'message' sits in 4 of 26 union indices; the quarter gate demands 7, so the minority family could never pass it The union route assumed a corpus is ONE family. A multi-family corpus (repo source beside payload JSON beside jsonl inventories) breaks that assumption: a minority family's shared schema is invisible to union coverage forever. Fix: a second admission route in resolve_fields. Indices mapping none of the standard content fields are grouped by their exact text-field signature; every field of a signature carried by >= 2 indices joins the ^0.5 recall union (signature membership is 100% coverage of the family, so no second fraction is needed). The family that carries the field is the family that votes on it. Two guard rails keep the xerj-org#1246 junk hole closed: an index mapping ANY standard field (body and kin) is never in a family - README frontmatter junk lives in exactly those indices - and a signature needs a witness (>= 2), so a lone odd raw-JSON file inside a code corpus stays noise. MAX_OWN_FIELDS (24) hoisted to module scope and applied once at the merged union+family leg list; own_text_fields is now uncapped and both callers (floor, union merge) apply it explicitly. Reference-code mandate: skipped, confined to this repository's own measured lineage in fields.rs (xerj-org#1139 -> xerj-org#1158 -> xerj-org#1238 -> xerj-org#1246 -> here); no external corpus code informs the change. Tests: xerj-common --lib 201/0 (two new: a_minority_family_joins_by_signature_not_union_share, a_family_needs_a_witness_and_body_indices_stay_out), xerj-autoindex --lib 1299/0. Files: - crates/xerj-common/src/xccode/fields.rs (resolve_fields, own_family_fields new, MAX_OWN_FIELDS hoisted)
thomas-villani
pushed a commit
to thomas-villani/xerj
that referenced
this pull request
Oct 9, 2026
changelog: rc.92 window entries for <
8000
a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="5758121701" data-permission-text="Title is private" data-url="https://github.com/xerj-org/xerj/issues/1246" data-hovercard-type="pull_request" data-hovercard-url="/xerj-org/xerj/pull/1246/hovercard" href="https://github.com/xerj-org/xerj/pull/1246">xerj-org#1246 and xerj-org#1251 (x…
…erj-org#1244 own-field recall legs + family route) Both PRs landed in the rc.92 window without their CHANGELOG entries (xerj-org#1246 merged first, xerj-org#1251 after CI). The release-notes gate requires every window PR cited, and the two changes are one story told in two rounds: the quarter-union gate assumed a single-family corpus, and vuln-fix-commits measured the assumption false. Files: - CHANGELOG.md [Unreleased] ### Fixed (two entries)
thomas-villani
pushed a commit
to thomas-villani/xerj
that referenced
this pull request
Oct 9, 2026
Window: xerj-org#1246 (xerj code own-field recall legs + quarter-union gate, xerj-org#1244), xerj-org#1251 (multi-family family-route coverage + coverage-ranked cap, xerj-org#1244). - engine/Cargo.toml 1.0.0-rc.91 -> 1.0.0-rc.92, Cargo.lock refreshed offline (58 members re-pinned). - CHANGELOG: rc.92 section. xerj-org#1244 stays open — completion is the pre-registered G7 suites passing on rebuilt corpora; both rebuilds (cve-records on :19285, vuln-fix-commits on :19295) were in flight at cut time. - ROADMAP: review-line roll to rc.92; twelve open (ten carried + xerj-org#1250 breaker-cap + xerj-org#1254 unassigned-file drop, both filed inside the window). xerj-org#1250 carries the same-day attribution correction (parent RSS breaker, boot-time tier cap, not max_query_memory_mb) and the measured idle flush loop (644/858 GB writes on two parked A/B nodes). xerj-org#1183 gained the 2026-10-08 standing-rebuild outcome: abort on a store-lost segment, read-back half now working, success footer still prints after ok=false (348-doc overstatement, comment on the issue). - Landing stamps RC.91 -> RC.92 across the 54 stamped pages (both patterns: data-latest-tag span + footer version); sitemap regenerated, 185 URLs, no drift (lastmod rolls only).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
xerj codeover JSON corpora whose prose lives in schema-named fields (cve-records:containers_cna_descriptions) sent onlytext^0.5, because one shard's synthesized — and empty-in-real-records —textfield made the resolved list non-empty and the #1158 own-field fallback never fired. The pre-registered g7-cve-records-2026-10-08 suite scored 0/7 while the needle records ranked #1 (18.76) queried on their prose field directly.Fix
^0.5recall legs even when standard fields are mapped. Ranking-safe:best_fieldsexecutes as dis_max in this engine, so a half-weight leg can only win by genuinely outranking the content fields (xerj code multi_match: one index mapping a text-family field silences every index that does not (heterogeneous corpora) #1238 posture).div_ceil(indices/4)mapping indices). Measured spectrum has an empty middle: cve-records schema prose 37-96% of 118 indices; exploit-pocs-2026 frontmatter junk ≤1% of 5,644 (2,143 own text fields — the old sorted 24-cap kept a literal$commentand cutSummary). Hand-verified on exploit-pocs-2026: gated list = old +code^0.5only, top-5 identical, totals 39 → 43 (pure recall).textfield — passage search silently returns nothing #1158 floor (no standard field mapped anywhere) keeps any own field, gate off — its only alternative is["body"], a field no index maps (MULTIMATCH_DEFECT silent zero).The issue's layer 2 (passage synthesis from
descriptions[]) proved unnecessary: prose already sits in searchable text-typed fields;carries_prosecorrectly skips synthesis.Verification
xerj-common --lib199/0,xerj-autoindex --lib1299/0.a_corpus_with_body_mapped_never_grows_own_fields→own_text_fields_join_as_recall_legs_beside_body); two added, including the coverage-gate/floor-exemption pair.Refs #1244.