Repository navigation
search: opt-in local judge stage on _search — _p_relevant per hit (#1060) - #1077
Merged
Merged
Conversation
…1060) The zero-token rerank. A _search body may now carry a "judge" block ({"local": true, "min_p": 0.5}); the node re-scores the hits it was about to return against the question, in process, and each judged hit carries _p_relevant, a 0..=1 relevance probability. min_p drops hits below the threshold and counts them in the response's "judged" block ({applied, scorer, query, judged, kept, dropped, took_ms}). No provider, no key, no tokens, no egress — unlike "rerank", the operator-pays hosted stage. Why a second stage instead of extending rerank: the hosted stage REPLACES _score with its probability and widens the page; the judge leaves _score alone (a client sorting by _score keeps the engine's number beside the new absolute scale) and judges exactly the emitted page. Different budget, different contract, different failure domain (503 judge_exception for a local model that cannot load, never a paid-call failure; no degrade path — a caller who asked for judged hits and silently got unjudged ones has been misled). Which judge judged is on the wire: judged.scorer is "lexical" or "model". The lexical arm is the always-compiled deterministic scorer (window-local BM25 saturation with page-local IDF, normalized by the max attainable share; K1/B from xerj-fts/src/bm25.rs) — NO semantic signal, no quality claim. The model arm (feature decide-local, the same xerj-api -> xerj-ai chain as the decide ladder) reuses the decide loader landed in #1072: same candle NLI head, same model directory, same process-wide registry — one loaded copy, two questions asked of it. The model arm is UNMEASURED in this sandbox (no local model loads here); the BEIR gate (0.699 SciFact / 0.345 NFCorpus / FiQA >= 0.30 beyond run spread) and the <= 40 ms p50 top-30 gate bind at release, not in this PR. Strictness mirrors the rerank stage: sort / search_after / collapse / scroll / size:0 / _source:false-without-fields are refused by name before the search runs, and every surface that does not run the stage (scroll continuation, _msearch, _search/template, _msearch/template, _async_search, _rank_eval, the native /v1 search API, the gRPC Search RPC) refuses a body carrying "judge" by name instead of silently dropping the block — the native API grew an explicit judge field for this, since its parser ignores unknown keys. "judge": null is absent everywhere; "local": false is refused (the local judge is the only judge). The judge reads the hit exactly as the response carries it: the candidate builder is rerank_stage::unrestricted_candidate, extracted from RerankPlan::candidate so the hosted stage's exhaustive-fields path and the judge share one implementation and cannot drift on what text a hit carries. MCP: xerj_search and xerj_hybrid_search take an optional "judge" argument (query required on hybrid, carried from the plain string on xerj_search, validated with the node's own JudgeConfig parser before the round trip). Tool descriptions state the honest ceiling and the zero-egress difference from rerank in those words. Published schema regenerated (11 tools, judge property added; drift test green). Scope note: xerj-server/src/grpc.rs is a one-hunk extension of the existing second-stage refusal there; judge-specific Prometheus metrics are deliberately skipped (would need xerj-common changes outside this PR's crate scope). Tests: xerj-rerank 49 (51 with judge-local), xerj-api lib 292 (14 new judge_stage unit tests), new tests/judge_stage_http.rs — 14 contract tests over HTTP: opt-in inertness, response shape unchanged without the block, _p_relevant present and ordered, _score untouched per id, min_p drop counting against a self-adjusting cut, the refusal table, per-surface named refusals, composition with the hosted provider, and zero egress asserted against the provider stub's own request counter. xerj-mcp 61 (7 new). Full suites green for -p xerj-rerank -p xerj-api -p xerj-mcp; clippy clean on touched crates with and without decide-local.
This was referenced Sep 30, 2026
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
The packaging half of the ten-item program's rule (discussion xerj-org#1054): a benchmarks/<name> directory with raw results plus a blog post with the losses left in. The gate directories landed through xerj-org#1079, xerj-org#1085, xerj-org#1086, xerj-org#1087, xerj-org#1088 and xerj-org#1090; this is the post. Structure follows the Jev posts (verdict first, wins then losses with readings, then what the next window does about each loss). Every number is verbatim from the committed gate artifacts: - PASS: /_ask macro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run determinism, wall-clock p50 1.129 ms vs the 300 ms bar (xerj-org#1076, xerj-org#1079); xerj_map unknown-field 400s 20/30 guessed -> 0/30 map-informed, 100% vs the >=80% bar (xerj-org#1074, xerj-org#1086); calibration FiQA pair-level ECE 0.3109 raw -> 0.0088 held-out vs the <=0.10 bar, temperature 0.3163 published as the failing comparison arm, $0.2125 of provider calls over 648 queries (xerj-org#1080, xerj-org#1087). - FAIL, published as such: the lexical judge loses to its own first stage everywhere measured (0.7045->0.5967, 0.3419->0.2924, 0.2382->0.1650 vs the 0.30 FiQA bar) and stays opt-in with no quality claim per its own rule (xerj-org#1077, xerj-org#1090); the flywheel freezes as shipped (12 of 2,000 cached, xerj-org#1094) and forces 19.72% coverage at 0.9953 accuracy vs the 80%/0.97 bars (xerj-org#1075, xerj-org#1088); the tier-2 decide head misses SMS 0.9193/0.287 vs 0.95/0.05 and AG News 0.2599 vs 0.85, latency 8.4 ms PASS, ships behind the flag with the numbers published (xerj-org#1072/xerj-org#1073, xerj-org#1088). - SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 PASS, zero-hit 25->15 vs <=10 FAIL — the 15 have no stem anywhere in the corpus (xerj-org#1070, xerj-org#1085). - The gate that caught the stock release binary built without the decide-local cargo feature (flag a no-op, stock-binary-noop.txt) is told with the fix queued ahead of the cut, not claimed as done. - Honest gaps stated, not smoothed: the /_ask token arm designed not run, the _watcher detection gate not yet measured, no HF URL for xerj-decide-v1. Four issues filed from the measurements: xerj-org#1091-xerj-org#1094. Mechanics: hand-written page at landing/blog/the-rc80-gates.html, registered in scripts/seo/pagedata.py (published/updated 2026-09-30), card prepended to the blog hub (hub dateModified bumped), head and footer owned by fix_heads/fix_links, og falls back to the site card.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The zero-token rerank (plan item 4 of 10, discussion #1054). A
_searchbody may now carry ajudgeblock:{ "query": {"match": {"body": "vitamin d supplementation"}}, "judge": {"local": true, "min_p": 0.5} }The node re-scores the hits it was about to return against the question in process — no provider, no key, no tokens, no egress. Each judged hit carries
_p_relevant(0..=1).min_pdrops hits below the threshold and counts them in the response'sjudgedblock:MCP:
xerj_searchandxerj_hybrid_searchtake an optionaljudgeargument (query required on hybrid, carried from the plain string onxerj_search, validated with the node's ownJudgeConfigparser before the round trip). Published schema regenerated.Closes #1060.
Why a second stage instead of extending
rerankThe hosted stage is operator-pays, sends document text off the node, REPLACES
_scorewith its probability, and widens the page. The judge is free, sends nothing anywhere, leaves_scoreuntouched beside the new absolute scale, and judges exactly the emitted page. Different budget, different contract, different failure domain (503 judge_exceptionfor a local model that cannot load — never a paid-call failure, and no degrade path: a caller who asked for judged hits and silently got unjudged ones has been misled).The judge reads the hit exactly as the response carries it: the candidate builder is
rerank_stage::unrestricted_candidate, extracted fromRerankPlan::candidateso the hosted and local stages share one implementation and cannot drift on what text a hit carries.Honest status of the scorer — read this before merging
ln(1+(N−df+0.5)/(df+0.5))), normalized by the max attainable share, K1=1.2/B=0.75 fromxerj-fts/src/bm25.rs. It has NO semantic signal. Scale semantics: share of maximum attainable saturation (single-term tf=1 on an avg-length doc ≈ 0.45, tf=2 short doc ≈ 0.63, no query word = exactly 0.0). The response names the scorer (judged.scorer:lexical|model) so a caller can see which judge judged;judged.modelappears only on the model arm.decide-local, samexerj-api → xerj-aichain and same shared model registry as the decide ladder) is interface + plumbing shipped, verified to compile and to fall back naming itself, nothing more. I did not fake a model.benchmarks/beir-hybrid; hosted Jev measured 0.3638) and ≤ 40 ms p50 for top-30. Loses → does not ship stays the release-time rule.Strictness (mirrors the rerank stage)
sort(incl.?sort=),search_after,collapse,scroll,size: 0,_source:falsewithout a fields channel,"local": false(the local judge is the only judge), unknown keys in the block, non-inferable queries withoutjudge.query._search/scrollcontinuation,_msearch,_search/template,_msearch/template,_async_search,_rank_eval, the native/v1search API (its request struct gained an explicitjudgefield for this — its parser ignores unknown keys), and the gRPC Search RPC."judge": nullis the same as nojudgekey everywhere. Opt-in inert by default: without the block the response is byte-identical in shape (pinned by test).Scope notes
xerj-server/src/grpc.rsis a one-hunk extension beyond the named crate set (task said xerj-rerank / xerj-api / xerj-mcp): it swaps the existing rerank-only refusal forcarries_second_stage, existing pub API only. Flagging it rather than silently widening scope.xerj-commonchanges outside this PR's crate scope. The stage reuses the rerank stage's refusal counter where a refusal happens.xerj coderetrieval was possible. In-repo precedent cited instead:xerj-fts/src/bm25.rs(K1/B and the saturation shape), systemone: local zero-shot decision head (tier 2) via candle (#1057) #1072'sdecide.rsloader (the model arm),rerank_stage.rs(the stage architecture this mirrors).Tests
Scoped runs, all green (default features and
--features decide-local):cargo test -p xerj-rerankjudge-local: 51)cargo test -p xerj-apijudge_stageunit tests)cargo test -p xerj-api --test judge_stage_httpcargo test -p xerj-mcpcargo build --release -p xerj-serverThe HTTP contract tests cover: opt-in inertness and unchanged shape without the block;
_p_relevantpresent and non-increasing with the all-terms doc first;_scoreidentical per id to the unjudged response;judgedblock accounting;min_pdrops counted against a self-adjusting cut (plus the 1.0-drops-everything and 0.0-keeps-everything edges); the 13-case refusal table; per-surface named refusals (_msearchper-item, scroll continuation, native/v1); composition with the hosted provider (both blocks,_rerankprecedesjudgedon the wire, two scales on every hit); and zero egress asserted against the provider stub's own request counter — not inferred from the search succeeding.fmt + clippy clean on xerj-rerank, xerj-api, xerj-mcp, xerj-server, with and without
decide-local.