Repository navigation
feat(api): POST /_ask + xerj_plan MCP tool — prompt in, validated DSL out (#1056) - #1076
Merged
Merged
Conversation
… out (#1056) An agent filtering structured data hand-writes ES DSL today, and a wrong guess in a hand-written clause fails silently. POST /_ask (ES-compat) and POST /v1/ask (native) take {index, prompt} and return {query, plan[], confidence, indices[]} where query is an ES query clause that has ALREADY passed xerj_query::parse_request on this node — zero invalid DSL can leave the node, and a phrase the index cannot ground is a 422 that names it, never a silent guess. The planner is deterministic and rule-based (NOT an LLM): date literals and year phrases, comparator vocabulary ("or greater", "deeper than", "between ... and ..."), units (km, K, days, Earth radii), scale words ("100 million" -> 10^8) and a concept lexicon (magnitude->mag, "life expectancy"->lifeExp, "GDP per capita"->gdpPercap) resolve field phrases by scored matching against the index's real schema; equality values resolve against ONE batched terms agg (<=1,000 buckets per field, <=24 fields; fields whose agg reports sum_other_doc_count>0 or whose catalog cardinality_est exceeds the cap verify via <=4 BM25 match probes instead — the match arm). Numeric and date bounds are ONLY prompt literals; nothing is generated. Same request three times is byte-identical (ordered collections only, no clock in the body). Index patterns (ax-*) route by the autoindex-catalog dataset description read through the engine's own search API — xerj-api gains no xerj-autoindex dependency. Measured (benchmarks/ask-plan harness, PR #1071, unmodified; private port + throwaway data dir, 230 pairs x 3 runs): - round 1 (pre-fixes): macro result-set F1 0.9309, invalid_dsl_out 0, non_deterministic 0 - final: macro result-set F1 0.9975, invalid_dsl_out 0, non_deterministic 0 (gate >= 0.9 over >= 200 pairs) - the only 3 sub-1.0 pairs (usgs 057/058/059) are the "from A through B" date pairs, where English "through" is inclusive by design and the fixture gold treats the end day as exclusive — documented divergence - ES-YAML conformance on this tree: 1380 passed / 0 failed / 3 skipped Round-2 root cause worth remembering: an usize underflow in the separated-op scan ("stronger than MAGNITUDE 3.5") indexed out of bounds and, under panic=abort, took the whole node down mid-harness; fixed with a bound guard and pinned by two route tests (gapminder-052 shape). Also: xerj_plan, the 12th MCP tool — a plain engine proxy with index defaulting to ax-* exactly like xerj_search (#962). The published schema landing/docs/agents/schemas/mcp-tools.json regenerated via scripts/mcp-schema-check.sh --write (12 tools). Known follow-ups, deliberately NOT touched here: landing/llms.txt still describes the MCP tool set as "ten tools" (one llms.txt-editing PR at a time; fix queued post-release), landing/answers/* embed the same stale count, and anthropic/openai tool JSONs intentionally stay the six HTTP-core ops. Files: - engine/crates/xerj-api/src/ask_api.rs (new; planner + contract) - engine/crates/xerj-api/tests/ask_endpoint.rs (new; 16 route tests: canonical casing, comparators/units, gold ordering, date bounds, determinism 3x, adversarial battery re-parsed, catalog routing, BM25 arm with a real overflowing agg, native /v1/ask) - engine/crates/xerj-api/src/router.rs (/v1/ask + /_ask routes; same authz posture as /_decide — cluster verb, POST, fail-closed) - engine/crates/xerj-mcp/src/lib.rs (xerj_plan spec + proxy + tests) - landing/docs/agents/schemas/mcp-tools.json (regenerated)
CHANGELOG-only change: the Added entry for issue #1056, citing the measured harness numbers (macro F1 0.9975 final vs 0.9309 round 1, invalid_dsl_out 0, non_deterministic 0 over 230 pairs x 3 runs) and stating plainly that the p50 latency gate is a release-time measurement, not claimed here.
This was referenced Sep 30, 2026
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
The MCP registry grew twice without llms.txt or llms-full.txt noticing: xerj_code_search + xerj_plan landed via xerj-org#1005/xerj-org#1076 and xerj_map via xerj-org#1074 (merge f0b96d6). Both files still said the mcp-tools.json registry holds 'ten tools (the six HTTP operations plus the four brain tools)' and llms-full's own verify example printed the wrong count (jq '.tools | length' -> 10). The 'seventh canonical operation' arithmetic in the anthropic-tools.json line leaned on a seven-op numbering that no longer describes the surface (/_ask is HTTP since xerj-org#1076; the catalog read always was). What changed, all outside the generated article tree (hand-written regions per scripts/seo/README.md): - llms.txt: mcp-tools.json line says thirteen and names all seven additions; anthropic line states the six-op files are the HTTP-core surface BY DESIGN (not 'everything else is CLI-only') and says 'xerj mcp' serves all thirteen - llms.txt: endpoint map gains POST /_ask (validated DSL out, 422 naming ungrounded phrases, deterministic not-LLM) — it was entirely undocumented - llms.txt: llms-full section pointer no longer claims 'all seven operations' - llms-full.txt: section 3 retitled 'THE CORE AGENT OPERATIONS' (TOC + header; 'NOT an eighth operation' ordinals de-numbered), the schema paragraph carries thirteen + the full composition, and the jq verify example now reads -> 13 Checked against what is released, not asserted: the 13 names in landing/docs/agents/schemas/mcp-tools.json (pinned to tool_specs() by tests/published_schema_drift.rs) are exactly the 13 both files now name; all 13 appear by name across the two files; seo_lint 0 gate violations (30 pre-existing warnings, same as main). Scope deliberately excluded (the queued llms rewrite, one job per PR): the obligation-language block in llms-full section 9, capability branching, last-updated stamp. The first-time-agent harness rerun binds that rewrite, not an inventory correction. Follow-up from plan xerj-org#1054 items xerj-org#1055/xerj-org#1056.
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
….129 ms The one xerj-org#1056 gate line still unmeasured after xerj-org#1076 merged: 'p50 <= 300 ms CPU budget' per POST /_ask. The ask arm times every call but scores F1 only — the milliseconds were never recorded. New scripts/latency_arm.py (python3 stdlib, same conventions as ask_arm): one warmup pass + N measured passes over the same 230 pairs.jsonl prompts on the same booted node, per-request wall-clock ms to latency-raw.jsonl, nearest-rank p50/p90/p95/p99/max + a gate verdict to latency-summary.json. Honest scope stated in the file and the summary: wall-clock over loopback on an idle node is an UPPER BOUND on CPU time, not CPU time; prompts carry their gold index so catalog routing is out of the path (it has its own acceptance line); warmup reported separately so the steady-state choice is visible, not flattering. Measured run (release build of main f0b96d6, private :9681, throwaway data dir, fixtures loaded by the unmodified ask_arm.py loader, selfcheck green, run committed verbatim in results/2026-09-30-latency-gate/): measured n=690 (230 x 3 passes, all HTTP 200) p50 1.129 ms · p90 1.883 · p95 2.033 · p99 2.852 · max 155.719 warmup p50 1.102 ms · max 468.927 (first-touch) gate p50 <= 300 ms: PASS — 265x under the line Caveats kept in the README, not buried: the fixture indices are small (655 / 3,000 / 1,704 docs) — the p50 is a fact about THIS fixture, not a promise about larger corpora; the gate row claims wall-clock, nothing about CPU. README: the DESIGN banner is two runs stale — gate 2 (F1 0.9975, 0 invalid DSL, 0 nondeterminism, recorded in PR xerj-org#1076's body from an unmodified scratch copy whose raw files were never committed — said plainly in the row) and now gate 4 each carry MEASURED rows; the NOT-measured list shrinks to what is actually unmeasured (token arms, catalog routing). run.sh gains step 5 so a full run captures latency alongside F1 from here on.
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
The packaging half of the ten-item program's rule (discussion xerj-org#1054): a benchmarks/<name> directory with raw results plus a blog post with the losses left in. The gate directories landed through xerj-org#1079, xerj-org#1085, xerj-org#1086, xerj-org#1087, xerj-org#1088 and xerj-org#1090; this is the post. Structure follows the Jev posts (verdict first, wins then losses with readings, then what the next window does about each loss). Every number is verbatim from the committed gate artifacts: - PASS: /_ask macro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run determinism, wall-clock p50 1.129 ms vs the 300 ms bar (xerj-org#1076, xerj-org#1079); xerj_map unknown-field 400s 20/30 guessed -> 0/30 map-informed, 100% vs the >=80% bar (xerj-org#1074, xerj-org#1086); calibration FiQA pair-level ECE 0.3109 raw -> 0.0088 held-out vs the <=0.10 bar, temperature 0.3163 published as the failing comparison arm, $0.2125 of provider calls over 648 queries (xerj-org#1080, xerj-org#1087). - FAIL, published as such: the lexical judge loses to its own first stage everywhere measured (0.7045->0.5967, 0.3419->0.2924, 0.2382->0.1650 vs the 0.30 FiQA bar) and stays opt-in with no quality claim per its own rule (xerj-org#1077, xerj-org#1090); the flywheel freezes as shipped (12 of 2,000 cached, xerj-org#1094) and forces 19.72% coverage at 0.9953 accuracy vs the 80%/0.97 bars (xerj-org#1075, xerj-org#1088); the tier-2 decide head misses SMS 0.9193/0.287 vs 0.95/0.05 and AG News 0.2599 vs 0.85, latency 8.4 ms PASS, ships behind the flag with the numbers published (xerj-org#1072/xerj-org#1073, xerj-org#1088). - SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 PASS, zero-hit 25->15 vs <=10 FAIL — the 15 have no stem anywhere in the corpus (xerj-org#1070, xerj-org#1085). - The gate that caught the stock release binary built without the decide-local cargo feature (flag a no-op, stock-binary-noop.txt) is told with the fix queued ahead of the cut, not claimed as done. - Honest gaps stated, not smoothed: the /_ask token arm designed not run, the _watcher detection gate not yet measured, no HF URL for xerj-decide-v1. Four issues filed from the measurements: xerj-org#1091-xerj-org#1094. Mechanics: hand-written page at landing/blog/the-rc80-gates.html, registered in scripts/seo/pagedata.py (published/updated 2026-09-30), card prepended to the blog hub (hub dateModified bumped), head and footer owned by fix_heads/fix_links, og falls back to the site card.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes #1056 (engine side).
POST /_ask(ES-compat, next to/_decide) andPOST /v1/ask(native) take{index or pattern, prompt}and return{query, plan[], confidence, indices[]}wherequeryis an ES query clause the caller can send straight to/{index}/_searchas{"query": …}. Plusxerj_plan, the 12th MCP tool, a plain proxy for it withindexdefaulting toax-*exactly likexerj_search(#962). The published schemalanding/docs/agents/schemas/mcp-tools.jsonis regenerated (12 tools).The four hard lines, and where each is enforced
xerj_query::parse_requestbefore it is returned (ask_api.rs, step 5 of the handler). Pinned by the route test's adversarial battery: 18 prompts, every 200'squeryre-parsed through the same parser; injection prose,1e999,NaN,9999-99-99, CJK/needle runs all yield 4xx or valid DSL.error.phrasescarries the phrase. No silent guess anywhere; prefix value completion is restricted to digit-free names precisely sov9999can never "resolve" tov999.Vec,BTreeMap), ties reduced under explicit total orders, and no clock/timings/ids in the response body (which is why there is notook_ms).sum_other_doc_count > 0, or whose catalogcardinality_estexceeds the cap, is verified with a BM25matchprobe instead (≤4 per request) and the clause honestly comes back asmatch. The BM25 arm is route-tested with a real 1,006-distinct-value field whose agg provably overflows.Index routing over
ax-*is byautoindex-catalogdataset description (slug, field names, examples, notes), read through the engine's own index APIs — one search of the catalog index; xerj-api gains no xerj-autoindex dependency (the constant mirrorsxerj_autoindex::catalog::CATALOG_INDEX). Field gating is one batched typed call (Index::schema()), merged with catalog FieldSpecs. Numeric/date thresholds are only ever prompt literals — nothing generated; a threshold that would have to be invented is a 422.What the planner is (honesty)
A deterministic rule-based parser — not an LLM, not fuzzy retrieval. Date literals and year phrases, comparator vocabulary (
or greater,deeper than,between … and …,exactly), units (km,K,days,Earth radii), scale words (100 million→ 10⁸) and a concept lexicon (magnitude→mag,life expectancy→lifeExp,GDP per capita→gdpPercap) resolve field phrases by scored matching against the index's real fields; equality values resolve against the live value map. Anything it cannot ground in the index is refused by name.confidencereports how much soft matching the plan needed (0.9 fully exact, lower per fallback, 0.5 barematch_all).Measured (all numbers from this run, nothing asserted in code)
benchmarks/ask-planharness (PR #1071, unmodified — extracted to a scratch dir, nothing in this PR touches it), private port + throwaway data dir, 230 pairs × 3 runs:Gate (macro F1 ≥ 0.9 over ≥ 200 pairs) passes. Per dataset (final): gapminder 80/80, nasa 78/78, usgs 69/72. The only three sub-1.0 pairs (
usgs-earthquakes-057/058/059) are thefrom A through Bdate pairs: English "through" is inclusive, so this PR emits an inclusive next-day bound, while the fixture's gold treats the end day as exclusive — a documented semantics divergence, not a planner miss (~0.003 macro; the gate passes either way).Round-2 root cause, worth the history: an
usizeunderflow in the separated-op scan (stronger than MAGNITUDE 3.5with the number early in the prompt, gapminder-052 shape) indexed out of bounds and — underpanic = "abort"— took the whole node down mid-harness. Fixed with a bound guard and pinned by two route tests. This is also why the adversarial battery exists.ES-YAML conformance on this exact tree: 1380 passed / 0 failed / 3 skipped (private port, throwaway dir). Scoped tests: xerj-api 712 passed / 0 failed (16 of them the new route tests), xerj-mcp 57 lib + 4 schema-drift passed, clippy and
cargo fmt --checkclean.Latency (p50 ≤ 300 ms CPU) is NOT claimed here — that gate is a release-time measurement on the harness; this PR's unit is correctness.
Reference-code note
The peer reference corpora (
xerj code) are not available in this sandbox, so retrieval-before-writing could not be run against them. In-repo precedent followed instead, all read before writing:systemone_api.rs(/_decide— the body-carried-index + fail-closed authz posture this endpoint copies),es_compat.rs(search/agg execution throughxerj_query::parse_request), the ES-compat router's_-prefixed cluster verbs, andxerj-mcp'stool_specs()/#962ax-*defaulting. The catalog wire shape (doc_kind/slug/fields_json/notes/time_field) is read as untyped JSON so the no-dependency rule holds.Deliberately out of scope (follow-ups)
landing/llms.txtstill calls the MCP tool set "ten tools" — it was stale before this PR (eleven since [feature]: xc.py is not useful with xerj having the same capabilties inside the binary. Update LLMS and skills to have it nice #977, now twelve). One llms.txt-editing PR at a time (queued post-release), so it is not touched here; same forlanding/answers/*pages embedding the oldxerj mcphelp text.anthropic-tools.json/openai-tools.jsonintentionally remain the six HTTP-core tools — that smaller set is a published policy, not an oversight.