Repository navigation
feat(#977): reference-coding in the binary — xerj code, xerj corpus, xerj_code_search MCP - #1005
Merged
Merged
Conversation
…e, xerj corpus, xerj_code_search MCP Motivation ---------- The reference-coding method (#977) lived in three wrapper scripts — tools/xerj-code/scripts/xc.py, xc-index.sh, xc-corpus.sh — which meant a python3 dependency on every agent machine, a second implementation of the retrieval prose for the MCP surface that never existed, and contracts gated by pytest/sh test files instead of the scoped cargo gates every other subsystem uses. What landed ----------- - engine/crates/xerj-common/src/xccode/ — the ONE pure core: state ledger (30-day staleness refusal, salvaged/incomplete warnings, atomic write), licence map + restricted-licence warning, corpus.json + hub-manifest parsing with the untrusted-input gate, the path gate every manifest-derived path passes, mapping-resolved fields + semantic-capable discovery, symbol-passage/best-window selection, and the single renderer. HTTP is a 3-method trait (XcHttp) so the core stays client-free. - engine/crates/xerj-autoindex/src/xc.rs — the CLI shell: `xerj code` (bm25/semantic/hybrid, --json, --meatl, exit triangle 0/1/2/3), `xerj corpus add` (--from hub manifests, --as, sha pinning, review-block carry), `xerj corpus index` (the #930 build-verify-swap, unretired semantics: unknown counts never read as zero, deletes by exact name, sibling corpora excluded), `xerj corpus list`. Loopback admin-key fallback per #961/#962. - xerj-mcp: the 11th tool, xerj_code_search — deliberately NOT a thin proxy: it runs the same pipeline and returns rendered text, with licence_policy:"strict" stripping passage text from restricted-licence hits. This serves shell-less agents the same prose byte-for-byte. - Hybrid retrieval now fuses SERVER-SIDE: one native top-level hybrid query (RRF k=60, #943 shape) aimed at the semantic-capable indices, instead of the script's client-side fusion. No client-side fusion left to drift. - CI: the reference-coding job gates the Rust contracts (xccode + xc + MCP schema drift) instead of shell syntax + pytest. - Scripts and their python/shell suites deleted; SKILL.md, README, hub docs, AGENTS.md, CONTRIBUTING, case studies re-pointed at the subcommands. Review fixes applied before first push (audit of the port) - CRITICAL: get_mapping was called with "{pattern}/_mapping" (no leading slash); Es::get_json passes the path verbatim after base, so every mapping lookup was a malformed URL. Fixed at both call sites + a shape assertion in the test fake so a regression fails tests. - CRITICAL: `--from` was unusable — read_hub_manifest dropped the manifest's own `corpus` field, and no <name> was a usage error. HubManifest now carries corpus; resolve_corpus_name() is explicit > manifest > error naming both remedies; `--as` added and documented; two new tests pin resolution order and that resolution output still passes the pathgate. - CRITICAL: emit()'s early-return on empty text also swallowed --json output; the guard now covers the text arm only. - MAJOR: standalone semantic with an empty capable-set no longer POSTs — an empty comma-join is /_search = every index on the node. - MAJOR: hybrid_body now wraps BOTH legs with --lang (bool must), matching xc.py; the old port constrained BM25 only. - MAJOR: passage end-boundary snap used the FIRST newline in the window; xc.py's rfind takes the LAST — switching to rposition keeps the tail that earned the score. Honest gap (tracked) -------------------- The 81-check --fresh swap contract (fake node + PATH-shimmed fake binary) is NOT yet re-pinned: run_corpus_index constructs a real Es client and calls the real run_with_options, so it cannot run against fakes without injected seams. The state-ledger half IS re-pinned (xccode/state.rs, 8 tests). Follow-up: #1004. CHANGELOG states this plainly. Gates (local, this tree) ------------------------ - cargo test -p xerj-common --lib: 163 passed, 0 failed (xccode 49/49, state.rs included) - cargo test -p xerj-autoindex --lib: 1054 passed, 0 failed (xc:: 6/6) - cargo test -p xerj-mcp --lib: 36 passed, 0 failed - cargo check -p xerj-server: ok - cargo fmt --check: clean; cargo clippy: 0 warnings - mcp schema check + ES-YAML carried by CI (changes are CLI/MCP-facing; the engine query path is untouched except reading mappings it already served)
…erj code The published answer `rebuild-reference-corpus-index` walked agents through `xc-index.sh --fresh` / `xc.py` step by step — the exact commands the #977 port deletes. Every live instruction is re-pointed at the binary subcommands (`xerj corpus index [--fresh]`, `xerj code`, `xerj corpus list`), and the quoted stderr lines match the port's actual prose (`xerj corpus index:` / `xerj code` prefixes, the re-worded coverage warning from xccode/state.rs). The evidence and "how this page was checked" sections are re-aimed at what pins the behaviour NOW: the state-ledger tests in engine/crates/xerj-common/src/xccode/state.rs (salvaged warning, exit-3 not incomplete, index_prefix preferred over the bare glob, staleness refusal). The 81-check --fresh swap suite was deleted with the scripts and is not yet re-pinned in Rust — the page says so in plain words and links #1004 rather than citing tests that no longer exist. scripts/seo/article_data.py: XERJ_COMMANDS gains `code` and `corpus` (verified against the binary's dispatch, engine/crates/xerj-server/src/ main.rs) — without them the schema gate rejected the page's commands[].cmd lines, which is how the stale allowlist was caught. landing/: regenerated via build_articles.py --write (96 pages + sitemap re-check). The churn beyond the one answer page is the cross-links sidebar: every page lists every article's h1, and the h1 changed. gen_sitemap.py --check passes (176 URLs, unchanged — the slug did not change). Gates: test_article_schema.py 5/5 OK; build_articles.py --check ok; gen_sitemap.py --check ok.
…d a CorpusNode seam (#1004) Motivation ---------- PR #1005 ported the reference-coding scripts into the binary and deleted tools/xerj-code/tests/test_xc_index_fresh.py — 81 offline checks that pinned the DESTRUCTIVE side of `xerj corpus index --fresh`: which indexes may be deleted, in what order, and when the state file may switch. Issue #1004 was filed the same day because the port's logic carried those contracts but no test executed them; the answer page and CHANGELOG said so plainly ("carried by review, not by a test suite"). This commit closes that gap. Seams ----- The build/verify/swap flow moves verbatim into `corpus_index_flow(node, run_autoindex, now_secs, root, url, name, fresh)`. Three injections, each exactly the surface the old suite faked externally: - `CorpusNode` — the four node operations the flow performs: list indices under a glob, the tri-state `_count`, DELETE one index by exact name, and the scoped catalog delete. `impl CorpusNode for Es` is thin; the fake in the tests is the fifth implementation and cannot drift from what a real node is asked to do. - `run_autoindex(prefix, state_dir)` — the old suite PATH-shimmed a fake binary; the tests inject a fake runner that lands shards on rc 0/3, or on any rc in the `failing_but_writing` shape (#367: abort in finalisation after every document was written). - `now_secs` — the one-second build-id stamp; the old suite slept a real second to collide two rebuilds, the tests advance a fake clock. `run_corpus_index` keeps argv parsing, pathgates, `build_es`, and binds the seams to production values. Tests ----- Ten contract tests in xc.rs's test module, asserting on a FakeNode audit trail (every count read, delete, catalog scope) and the REAL state file — never on stdout prose: 1. a verified replacement switches state THEN retires, by exact name, count-before-delete ordering proven from the ops log 2. a build that never verifies (rc 0, zero records) removes only what it created; old index and old state untouched 3. a count the node does not answer (Unknown) authorises no delete and no switch; the unverified build is kept for the next verified run to retire 4. an unknown count for the OLD index presumes a working index: a failed build is not salvaged over it 5. a failed FIRST build that wrote records is kept and marked salvaged 6. a plain re-run resumes the recorded build: same prefix, same state dir, no deletes 7. sibling corpus indices (battle vs battle-terse) are never touched 8. a delete the node refuses leaves a duplicate, never a gap — the state file already names the new build 9. two rebuilds inside one second get different build ids (fake clock; the old suite's real 1s sleep) 10. a legacy corpus whose indices predate builds indexes under the bare namespace, no state dir Fail-before proof (both mutations caught, then reverted): - M1 `has_working_index = false` on unknown old count -> test 4 fails - M2 sibling-exclusion filter removed from corpus_indices -> test 7 fails The count-retry env knobs are pinned under a shared mutex: the new tests set XC_COUNT_TRIES=1, the existing defaults test asserts their absence, and run concurrently they would flake. Gates ----- - cargo test -p xerj-autoindex --lib: 1066 passed, 0 failed (was 1054; +12 incl. the two pre-existing tests now under the lock) - cargo clippy -p xerj-autoindex --all-targets: 0 warnings - cargo fmt -p xerj-autoindex -- --check: clean - cargo check -p xerj-server: ok - scripts/seo/build_articles.py --check + gen_sitemap.py --check + test_article_schema.py: 5/5 OK (answer page + CHANGELOG flipped from "not yet re-pinned" to naming the ten tests; landing regenerated) Closes #1004
One unreproduced lib-suite failure was observed once on the merged #977 branch (three other runs green). Process env is global, so a test that remove_var's XERJ_URL can race a concurrent reader; the #1004 suite already serialises its env knobs under ENV_LOCK — this test joins the same lock. No production change.
test(#1004): re-pin the --fresh swap contract as ten Rust tests behind a CorpusNode seam
…-in-binary # Conflicts: # CHANGELOG.md
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
The MCP registry grew twice without llms.txt or llms-full.txt noticing: xerj_code_search + xerj_plan landed via xerj-org#1005/xerj-org#1076 and xerj_map via xerj-org#1074 (merge f0b96d6). Both files still said the mcp-tools.json registry holds 'ten tools (the six HTTP operations plus the four brain tools)' and llms-full's own verify example printed the wrong count (jq '.tools | length' -> 10). The 'seventh canonical operation' arithmetic in the anthropic-tools.json line leaned on a seven-op numbering that no longer describes the surface (/_ask is HTTP since xerj-org#1076; the catalog read always was). What changed, all outside the generated article tree (hand-written regions per scripts/seo/README.md): - llms.txt: mcp-tools.json line says thirteen and names all seven additions; anthropic line states the six-op files are the HTTP-core surface BY DESIGN (not 'everything else is CLI-only') and says 'xerj mcp' serves all thirteen - llms.txt: endpoint map gains POST /_ask (validated DSL out, 422 naming ungrounded phrases, deterministic not-LLM) — it was entirely undocumented - llms.txt: llms-full section pointer no longer claims 'all seven operations' - llms-full.txt: section 3 retitled 'THE CORE AGENT OPERATIONS' (TOC + header; 'NOT an eighth operation' ordinals de-numbered), the schema paragraph carries thirteen + the full composition, and the jq verify example now reads -> 13 Checked against what is released, not asserted: the 13 names in landing/docs/agents/schemas/mcp-tools.json (pinned to tool_specs() by tests/published_schema_drift.rs) are exactly the 13 both files now name; all 13 appear by name across the two files; seo_lint 0 gate violations (30 pre-existing warnings, same as main). Scope deliberately excluded (the queued llms rewrite, one job per PR): the obligation-language block in llms-full section 9, capability branching, last-updated stamp. The first-time-agent harness rerun binds that rewrite, not an inventory correction. Follow-up from plan xerj-org#1054 items xerj-org#1055/xerj-org#1056.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #977. Closes #1004 (the swap-contract suite rides on this branch via PR #1006's merge into it).
What this is
The reference-coding loop (
tools/xerj-code/scripts/xc*.py|sh) becomes binary commands:xerj code,xerj corpus add|index|list, plus thexerj_code_searchMCP tool (the 11th) for shell-less agents. Nopython3/curldependency remains;gitstill is one (forxerj corpus add).xerj code <corpus> "<query>"— same retrieval, same exit triangle (0 hits / 1 no-match / 2 usage-transport-stale / 3 corpus-not-loaded-here), same flags (-k --lang --mode bm25|semantic|hybrid --full --no-symbol --json --meatl --stale-ok --url --api-key).xerj corpus add— clone or rebuild from a pinned manifest (--from,--as), licence detection with restrictive-first heuristics,review{}blocks preserved across rebuilds.xerj corpus index [--fresh]— the xc-index.sh --fresh fails on every previously indexed corpus (legacy state dir AND committed generation) #930 build-verify-swap, now with its destructive contracts re-pinned as ten Rust tests behind aCorpusNodeseam (Port the xc-index --fresh swap-contract suite (81 checks, fake node) to xerj corpus index #1004): switch-readers-before-retiring, deletes by exact name only, failed/empty builds leave the old index untouched, unknown counts never read as zero, siblings never touched.xerj corpus list— what is actually loaded on the node, honestly (404 → 0, other failures named).hybridquery, RRFk=60.~/.xerj-codecorpora, state files and indexes work unchanged: same layout, samecorpus.jsonschema, samexc-<corpus>namespaces.Semantics live in the new
xerj-common::xccodepure core, shared byte-for-byte by the CLI prose and the MCP tool text.Numbers
xerj-common::xccodetests: 163/163 ·xerj-autoindexlib: 1066 passed, 0 failed · MCP schema: 11 tools, drift-checkedMerge history on this branch
This branch carries merge commits resolving against main twice (after #999's windows MSVC fix landed, and again after main advanced past it); conflict resolution was CHANGELOG-only plus landing regeneration. The 81-check
--freshswap contract deleted with the scripts is re-pinned as ten Rust tests (PR #1006, merged into this branch).