8000 benchmarks/map-gate: xerj_map unknown-field-400 gate #1055 — 20/30 -> 0/30 (PASS) by xerj-org · Pull Request #1086 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

benchmarks/map-gate: xerj_map unknown-field-400 gate #1055 — 20/30 -> 0/30 (PASS) - #1086

Merged
xerj-org merged 2 commits into
mainfrom
measure/map-400s-1055
Oct 1, 2026
Merged

xerj-org merged 2 commits into
mainfrom
measure/map-400s-1055

Conversation

@xerj-org
Copy link
Copy Markdown
Owner

Closes the measurement half of #1055 (plan item 1 of 10, rc.79). PR #1074 shipped xerj_map and stated plainly the gate was unmeasured; this adds benchmarks/map-gate/ with the raw results, per the packaging rule.

Gate line, verbatim from the issue

unknown-field 400s from MCP-issued DSL down >=80% on the reference-coding harness.

Measured number: 20/30 -> 0/30 unknown-field 400s = 100% reduction (threshold >=80%) — PASS

Method

  • Binary/node: the release binary xerj v1.0.0-rc.78 (main @ a179e1e), one node --insecure --embed-mode lexical --port 9660 (private port band 9660-9679), throwaway mktemp -d data dir, killed after the run. Counts, not timings — deterministic; a re-run reproduces the counts. 32-core sandbox, irrelevant to a count.
  • Corpus (reference-coding shape): the three small Apache-2.0 repos of pinned hub corpus xerj-vector at their pinned commits (usearch cc23bbaf21ef, instant-distance 13ea89ac1ca0, hnswlib d9b3608c83d8), via xerj corpus add --from + xerj corpus index — 9 datasets, 3472 records, 10.1s wall. Three index shapes on purpose: prose/code docs (semantic body/text), PascalCase config indexes (Version/Include/Platform case traps), tiny structured datasets (a field named vector that is keyword; a json index whose only date is cloned_at). Manifest committed verbatim.
  • Discovery before design: on this engine, a wrong field in term/match/range/exists is a silent 0-hit HTTP 200 (match carries an unknown_field hint; term does not) — not a 400. The 400 class is exactly three MCP-reachable forms: sort without unmapped_type, knn on a non-vector field, semantic on a non-semantic_text field; a hybrid request dies whole on one bad leg. Only these count in the gate.
  • Arm A (no-map): 30 intents in queries.json, guessed field names written down BEFORE any mapping check, as the most plausible ES-fluent first guess (real convention splits: lang/language, license/licence, line_number/line, lowercase version vs Version). 5 guesses coincided with real fields and were kept as controls.
  • Arm B (with-map): field choices derived mechanically from the REAL xerj mcp stdio tools/call xerj_map response (saved raw, 16,828 bytes, 9 indexes, nothing trimmed): first compatible-es_type candidate in the map's exact spelling; highest-coverage semantic_text for semantic needs; rewrites (not guesses) where the map lists no vector field; drops the clause where the map shows the concept absent (3 drops, each recorded as map_action).
  • Every request+response committed per intent per arm under results/2026-09-30-rc78gates/arm{A,B}/.

Result

arm issued unknown-field 400s other 400s silent 0-hit ok, hits
A — no map 30 20 0 5 5
B — with xerj_map 30 0 0 1 29

Arm A's 20 = 9 sort + 5 semantic + 6 knn (4 knn + 2 hybrid killed by their vector leg); all 20 error bodies name the guessed field (error_names_used_field checked, none counted on other grounds). Controls behave identically in both arms.

BLOCKED (stated plainly)

The live model-driven agent loop — whether a live agent consults xerj_map before writing DSL, measured across its real tool-call stream with retries — is not run: live-model cost is not approved for this wave. Design: same 30 intents to an MCP-connected agent twice (map available vs withheld), count 400s in the actual tools/call stream; estimated $120 at the per-trial figures of demo/playbooks/REFCODING_BENCHMARK_3MODEL_2026-08-18.md ($2 x 60 trials). The deterministic arms bound what it can add: the map eliminates the failure class whenever consulted; the live run measures consultation rate, a property of the agent.

Caveats

  • 100% is the deterministic two-arm number on this corpus/query set, not a claim that no MCP query can ever 400: a map-informed agent can still write a wrong value (arm B's single 0-hit is exactly that — correct field Include, absent value PACKAGE).
  • The silent 0-hit class (5 vs 1) is the bigger usability win of the map but is NOT in the gate number; reported separately in the README and summary.
  • benchmarks/ only; no engine 8000 change. CHANGELOG entry follows as a second commit.

🤖 Generated with Claude Code

…) — 20/30 -> 0/30, PASS

The rc.79 gate for xerj_map reads 'unknown-field 400s from MCP-issued DSL
down >=80% on the reference-coding harness'. PR #1074 shipped the tool and
explicitly left this unmeasured ('needs a live multi-corpus node'); this is
the deterministic measurement, on the release binary (xerj v1.0.0-rc.78,
main @ a179e1e), port 9660, throwaway data dir.

Discovery first: on this engine a wrong field in term/match/range/exists is
a SILENT 0-hit 200 (match adds an unknown_field hint, term does not); the
400 class is exactly three MCP-reachable forms — sort without unmapped_type,
knn on a non-vector field, semantic on a non-semantic_text field — plus a
hybrid request killed whole by one bad leg. The gate counts only these.

Corpus: the three small Apache-2.0 repos of pinned hub corpus xerj-vector
(usearch cc23bbaf21ef, instant-distance 13ea89ac1ca0, hnswlib d9b3608c83d8),
'xerj corpus add --from' + 'xerj corpus index' — 9 datasets, 3472 records,
spanning prose/code docs indexes, PascalCase config indexes (Version/Include
case traps) and tiny structured datasets (a field NAMED vector that is
keyword). Manifest committed verbatim in the results dir.

Two arms, same 30 intents (benchmarks/map-gate/queries.json), guessed field
names written down before any mapping check (5 coincidentally correct,
kept as controls) vs field choices derived mechanically from the real
'xerj mcp' tools/call xerj_map response over stdio. Every request+response
committed per intent per arm.

Numbers: arm A 20/30 unknown-field 400s (9 sort, 5 semantic, 6 knn —
every error body names the guessed field), arm B 0/30 -> 100% reduction,
threshold >=80% -> PASS. Secondary, non-gate: arm A also silent-0-hit on
5 filter intents (lang/repo/line_number/lowercase-include/license-vs-licence)
where arm B hit 174/3/325/0/3. The live model-driven agent loop (does an
agent consult the map unprompted) is designed but not run — cost not
approved for this wave, ~$120 at the 3-model study's per-trial figure;
documented in the README's measured-vs-designed section.
Adds the [Unreleased] entry for the map-gate measurement landed in this
PR's first commit: 20/30 -> 0/30 unknown-field 400s (100% reduction,
threshold >=80%, PASS), with the silent-miss class reported separately
and the live-agent loop recorded as designed-not-run.
@cla-bot cla-bot Bot added the cla-signed label Sep 30, 2026
@xerj-org
xerj-org merged commit d7a0663 into main Oct 1, 2026
22 checks passed
neoneye pushed a commit to agent-memory-atlas-archive/xerj-org--xerj that referenced this pull request Oct 1, 2026
The packaging half of the ten-item program's rule (discussion xerj-org#1054): a
benchmarks/<name> directory with raw results plus a blog post with the
losses left in. The gate directories landed through xerj-org#1079, xerj-org#1085, xerj-org#1086,
xerj-org#1087, xerj-org#1088 and xerj-org#1090; this is the post.

Structure follows the Jev posts (verdict first, wins then losses with
readings, then what the next window does about each loss). Every number is
verbatim from the committed gate artifacts:

- PASS: /_ask macro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run
  determinism, wall-clock p50 1.129 ms vs the 300 ms bar (xerj-org#1076, xerj-org#1079);
  xerj_map unknown-field 400s 20/30 guessed -> 0/30 map-informed, 100%
  vs the >=80% bar (xerj-org#1074, xerj-org#1086); calibration FiQA pair-level ECE
  0.3109 raw -> 0.0088 held-out vs the <=0.10 bar, temperature 0.3163
  published as the failing comparison arm, $0.2125 of provider calls
  over 648 queries (xerj-org#1080, xerj-org#1087).
- FAIL, published as such: the lexical judge loses to its own first
  stage everywhere measured (0.7045->0.5967, 0.3419->0.2924,
  0.2382->0.1650 vs the 0.30 FiQA bar) and stays opt-in with no quality
  claim per its own rule (xerj-org#1077, xerj-org#1090); the flywheel freezes as
  shipped (12 of 2,000 cached, xerj-org#1094) and forces 19.72% coverage at
  0.9953 accuracy vs the 80%/0.97 bars (xerj-org#1075, xerj-org#1088); the tier-2
  decide head misses SMS 0.9193/0.287 vs 0.95/0.05 and AG News 0.2599
  vs 0.85, latency 8.4 ms PASS, ships behind the flag with the numbers
  published (xerj-org#1072/xerj-org#1073, xerj-org#1088).
- SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 PASS, zero-hit 25->15
  vs <=10 FAIL — the 15 have no stem anywhere in the corpus (xerj-org#1070,
  xerj-org#1085).
- The gate that caught the stock release binary built without the
  decide-local cargo feature (flag a no-op, stock-binary-noop.txt) is
  told with the fix queued ahead of the cut, not claimed as done.
- Honest gaps stated, not smoothed: the /_ask token arm designed not
  run, the _watcher detection gate not yet measured, no HF URL for
  xerj-decide-v1. Four issues filed from the measurements: xerj-org#1091-xerj-org#1094.

Mechanics: hand-written page at landing/blog/the-rc80-gates.html,
registered in scripts/seo/pagedata.py (published/updated 2026-09-30),
card prepended to the blog hub (hub dateModified bumped), head and
footer owned by fix_heads/fix_links, og falls back to the site card.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0