Repository navigation
benchmarks/map-gate: xerj_map unknown-field-400 gate #1055 — 20/30 -> 0/30 (PASS) - #1086
Merged
Merged
Conversation
…) — 20/30 -> 0/30, PASS The rc.79 gate for xerj_map reads 'unknown-field 400s from MCP-issued DSL down >=80% on the reference-coding harness'. PR #1074 shipped the tool and explicitly left this unmeasured ('needs a live multi-corpus node'); this is the deterministic measurement, on the release binary (xerj v1.0.0-rc.78, main @ a179e1e), port 9660, throwaway data dir. Discovery first: on this engine a wrong field in term/match/range/exists is a SILENT 0-hit 200 (match adds an unknown_field hint, term does not); the 400 class is exactly three MCP-reachable forms — sort without unmapped_type, knn on a non-vector field, semantic on a non-semantic_text field — plus a hybrid request killed whole by one bad leg. The gate counts only these. Corpus: the three small Apache-2.0 repos of pinned hub corpus xerj-vector (usearch cc23bbaf21ef, instant-distance 13ea89ac1ca0, hnswlib d9b3608c83d8), 'xerj corpus add --from' + 'xerj corpus index' — 9 datasets, 3472 records, spanning prose/code docs indexes, PascalCase config indexes (Version/Include case traps) and tiny structured datasets (a field NAMED vector that is keyword). Manifest committed verbatim in the results dir. Two arms, same 30 intents (benchmarks/map-gate/queries.json), guessed field names written down before any mapping check (5 coincidentally correct, kept as controls) vs field choices derived mechanically from the real 'xerj mcp' tools/call xerj_map response over stdio. Every request+response committed per intent per arm. Numbers: arm A 20/30 unknown-field 400s (9 sort, 5 semantic, 6 knn — every error body names the guessed field), arm B 0/30 -> 100% reduction, threshold >=80% -> PASS. Secondary, non-gate: arm A also silent-0-hit on 5 filter intents (lang/repo/line_number/lowercase-include/license-vs-licence) where arm B hit 174/3/325/0/3. The live model-driven agent loop (does an agent consult the map unprompted) is designed but not run — cost not approved for this wave, ~$120 at the 3-model study's per-trial figure; documented in the README's measured-vs-designed section.
Adds the [Unreleased] entry for the map-gate measurement landed in this PR's first commit: 20/30 -> 0/30 unknown-field 400s (100% reduction, threshold >=80%, PASS), with the silent-miss class reported separately and the live-agent loop recorded as designed-not-run.
This was referenced Oct 1, 2026
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
The packaging half of the ten-item program's rule (discussion xerj-org#1054): a benchmarks/<name> directory with raw results plus a blog post with the losses left in. The gate directories landed through xerj-org#1079, xerj-org#1085, xerj-org#1086, xerj-org#1087, xerj-org#1088 and xerj-org#1090; this is the post. Structure follows the Jev posts (verdict first, wins then losses with readings, then what the next window does about each loss). Every number is verbatim from the committed gate artifacts: - PASS: /_ask macro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run determinism, wall-clock p50 1.129 ms vs the 300 ms bar (xerj-org#1076, xerj-org#1079); xerj_map unknown-field 400s 20/30 guessed -> 0/30 map-informed, 100% vs the >=80% bar (xerj-org#1074, xerj-org#1086); calibration FiQA pair-level ECE 0.3109 raw -> 0.0088 held-out vs the <=0.10 bar, temperature 0.3163 published as the failing comparison arm, $0.2125 of provider calls over 648 queries (xerj-org#1080, xerj-org#1087). - FAIL, published as such: the lexical judge loses to its own first stage everywhere measured (0.7045->0.5967, 0.3419->0.2924, 0.2382->0.1650 vs the 0.30 FiQA bar) and stays opt-in with no quality claim per its own rule (xerj-org#1077, xerj-org#1090); the flywheel freezes as shipped (12 of 2,000 cached, xerj-org#1094) and forces 19.72% coverage at 0.9953 accuracy vs the 80%/0.97 bars (xerj-org#1075, xerj-org#1088); the tier-2 decide head misses SMS 0.9193/0.287 vs 0.95/0.05 and AG News 0.2599 vs 0.85, latency 8.4 ms PASS, ships behind the flag with the numbers published (xerj-org#1072/xerj-org#1073, xerj-org#1088). - SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 PASS, zero-hit 25->15 vs <=10 FAIL — the 15 have no stem anywhere in the corpus (xerj-org#1070, xerj-org#1085). - The gate that caught the stock release binary built without the decide-local cargo feature (flag a no-op, stock-binary-noop.txt) is told with the fix queued ahead of the cut, not claimed as done. - Honest gaps stated, not smoothed: the /_ask token arm designed not run, the _watcher detection gate not yet measured, no HF URL for xerj-decide-v1. Four issues filed from the measurements: xerj-org#1091-xerj-org#1094. Mechanics: hand-written page at landing/blog/the-rc80-gates.html, registered in scripts/seo/pagedata.py (published/updated 2026-09-30), card prepended to the blog hub (hub dateModified bumped), head and footer owned by fix_heads/fix_links, og falls back to the site card.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the measurement half of #1055 (plan item 1 of 10, rc.79). PR #1074 shipped
xerj_mapand stated plainly the gate was unmeasured; this addsbenchmarks/map-gate/with the raw results, per the packaging rule.Gate line, verbatim from the issue
Measured number: 20/30 -> 0/30 unknown-field 400s = 100% reduction (threshold >=80%) — PASS
Method
xerj v1.0.0-rc.78(main @ a179e1e), one node--insecure --embed-mode lexical --port 9660(private port band 9660-9679), throwawaymktemp -ddata dir, killed after the run. Counts, not timings — deterministic; a re-run reproduces the counts. 32-core sandbox, irrelevant to a count.xerj-vectorat their pinned commits (usearch cc23bbaf21ef, instant-distance 13ea89ac1ca0, hnswlib d9b3608c83d8), viaxerj corpus add --from+xerj corpus index— 9 datasets, 3472 records, 10.1s wall. Three index shapes on purpose: prose/code docs (semantic body/text), PascalCase config indexes (Version/Include/Platformcase traps), tiny structured datasets (a field namedvectorthat iskeyword; a json index whose only date iscloned_at). Manifest committed verbatim.term/match/range/existsis a silent 0-hit HTTP 200 (match carries anunknown_fieldhint; term does not) — not a 400. The 400 class is exactly three MCP-reachable forms:sortwithoutunmapped_type,knnon a non-vector field,semanticon a non-semantic_textfield; ahybridrequest dies whole on one bad leg. Only these count in the gate.queries.json, guessed field names written down BEFORE any mapping check, as the most plausible ES-fluent first guess (real convention splits:lang/language,license/licence,line_number/line, lowercaseversionvsVersion). 5 guesses coincided with real fields and were kept as controls.xerj mcpstdiotools/call xerj_mapresponse (saved raw, 16,828 bytes, 9 indexes, nothing trimmed): first compatible-es_typecandidate in the map's exact spelling; highest-coveragesemantic_textfor semantic needs; rewrites (not guesses) where the map lists no vector field; drops the clause where the map shows the concept absent (3 drops, each recorded asmap_action).results/2026-09-30-rc78gates/arm{A,B}/.Result
Arm A's 20 = 9 sort + 5 semantic + 6 knn (4 knn + 2 hybrid killed by their vector leg); all 20 error bodies name the guessed field (
error_names_used_fieldchecked, none counted on other grounds). Controls behave identically in both arms.BLOCKED (stated plainly)
The live model-driven agent loop — whether a live agent consults
xerj_mapbefore writing DSL, measured across its real tool-call stream with retries — is not run: live-model cost is not approved for this wave. Design: same 30 intents to an MCP-connected agent twice (map available vs withheld), count 400s in the actualtools/callstream; estimated$120 at the per-trial figures of$2 x 60 trials). The deterministic arms bound what it can add: the map eliminates the failure class whenever consulted; the live run measures consultation rate, a property of the agent.demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md(Caveats
Include, absent valuePACKAGE).benchmarks/only; no engine 8000 change. CHANGELOG entry follows as a second commit.🤖 Generated with Claude Code