8000 search: opt-in local judge stage on _search — _p_relevant per hit (#1060) by xerj-org · Pull Request #1077 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

search: opt-in local judge stage on _search — _p_relevant per hit (#1060) - #1077

Merged
xerj-org merged 2 commits into
mainfrom
judge-hits
Sep 30, 2026
Merged

xerj-org merged 2 commits into
mainfrom
judge-hits

Conversation

@xerj-org
Copy link
Copy Markdown
Owner

What

The zero-token rerank (plan item 4 of 10, discussion #1054). A _search body may now carry a judge block:

{ "query": {"match": {"body": "vitamin d supplementation"}},
  "judge": {"local": true, "min_p": 0.5} }

The node re-scores the hits it was about to return against the question in process — no provider, no key, no tokens, no egress. Each judged hit carries _p_relevant (0..=1). min_p drops hits below the threshold and counts them in the response's judged block:

"judged": { "applied": true, "scorer": "lexical", "query": "…",
             "judged": 4, "kept": 3, "dropped": 1, "min_p": 0.5, "took_ms": 0 }

MCP: xerj_search and xerj_hybrid_search take an optional judge argument (query required on hybrid, carried from the plain string on xerj_search, validated with the node's own JudgeConfig parser before the round trip). Published schema regenerated.

Closes #1060.

Why a second stage instead of extending rerank

The hosted stage is operator-pays, sends document text off the node, REPLACES _score with its probability, and widens the page. The judge is free, sends nothing anywhere, leaves _score untouched beside the new absolute scale, and judges exactly the emitted page. Different budget, different contract, different failure domain (503 judge_exception for a local model that cannot load — never a paid-call failure, and no degrade path: a caller who asked for judged hits and silently got unjudged ones has been misled).

The judge reads the hit exactly as the response carries it: the candidate builder is rerank_stage::unrestricted_candidate, extracted from RerankPlan::candidate so the hosted and local stages share one implementation and cannot drift on what text a hit carries.

Honest status of the scorer — read this before merging

  • The always-compiled arm is a deterministic lexical scorer, not a model. Window-local BM25 saturation with page-local IDF (ln(1+(N−df+0.5)/(df+0.5))), normalized by the max attainable share, K1=1.2/B=0.75 from xerj-fts/src/bm25.rs. It has NO semantic signal. Scale semantics: share of maximum attainable saturation (single-term tf=1 on an avg-length doc ≈ 0.45, tf=2 short doc ≈ 0.63, no query word = exactly 0.0). The response names the scorer (judged.scorer: lexical | model) so a caller can see which judge judged; judged.model appears only on the model arm.
  • The model arm reuses the decide loader from systemone: local zero-shot decision head (tier 2) via candle (#1057) #1072 and is UNMEASURED in this sandbox — no local cross-encoder model loads here, so the candle path (feature decide-local, same xerj-api → xerj-ai chain and same shared model registry as the decide ladder) is interface + plumbing shipped, verified to compile and to fall back naming itself, nothing more. I did not fake a model.
  • The issue's gates bind at release, not in this PR: BEIR bar (0.699 SciFact / 0.345 NFCorpus beyond run spread, FiQA ≥ 0.30; harness benchmarks/beir-hybrid; hosted Jev measured 0.3638) and ≤ 40 ms p50 for top-30. Loses → does not ship stays the release-time rule.

Strictness (mirrors the rerank stage)

  • Refused by name, before the search runs: sort (incl. ?sort=), search_after, collapse, scroll, size: 0, _source:false without a fields channel, "local": false (the local judge is the only judge), unknown keys in the block, non-inferable queries without judge.query.
  • Every surface that does not run the stage refuses the block by name instead of silently dropping it: _search/scroll continuation, _msearch, _search/template, _msearch/template, _async_search, _rank_eval, the native /v1 search API (its request struct gained an explicit judge field for this — its parser ignores unknown keys), and the gRPC Search RPC.
  • "judge": null is the same as no judge key everywhere. Opt-in inert by default: without the block the response is byte-identical in shape (pinned by test).

Scope notes

  • xerj-server/src/grpc.rs is a one-hunk extension beyond the named crate set (task said xerj-rerank / xerj-api / xerj-mcp): it swaps the existing rerank-only refusal for carries_second_stage, existing pub API only. Flagging it rather than silently widening scope.
  • Judge-specific Prometheus metrics deliberately skipped — they would need xerj-common changes outside this PR's crate scope. The stage reuses the rerank stage's refusal counter where a refusal happens.
  • Reference-code mandate: peer corpora are unavailable in this sandbox, so no xerj code retrieval was possible. In-repo precedent cited instead: xerj-fts/src/bm25.rs (K1/B and the saturation shape), systemone: local zero-shot decision head (tier 2) via candle (#1057) #1072's decide.rs loader (the model arm), rerank_stage.rs (the stage architecture this mirrors).

Tests

Scoped runs, all green (default features and --features decide-local):

suite result
cargo test -p xerj-rerank 49 passed (+2 with judge-local: 51)
cargo test -p xerj-api full suite green; lib 292 (14 new judge_stage unit tests)
cargo test -p xerj-api --test judge_stage_http 14 new contract tests, green
cargo test -p xerj-mcp 61 passed (7 new) + published-schema drift 4
cargo build --release -p xerj-server green post-rebase

The HTTP contract tests cover: opt-in inertness and unchanged shape without the block; _p_relevant present and non-increasing with the all-terms doc first; _score identical per id to the unjudged response; judged block accounting; min_p drops counted against a self-adjusting cut (plus the 1.0-drops-everything and 0.0-keeps-everything edges); the 13-case refusal table; per-surface named refusals (_msearch per-item, scroll continuation, native /v1); composition with the hosted provider (both blocks, _rerank precedes judged on the wire, two scales on every hit); and zero egress asserted against the provider stub's own request counter — not inferred from the search succeeding.

fmt + clippy clean on xerj-rerank, xerj-api, xerj-mcp, xerj-server, with and without decide-local.

…1060)

The zero-token rerank. A _search body may now carry a "judge" block
({"local": true, "min_p": 0.5}); the node re-scores the hits it was
about to return against the question, in process, and each judged hit
carries _p_relevant, a 0..=1 relevance probability. min_p drops hits
below the threshold and counts them in the response's "judged" block
({applied, scorer, query, judged, kept, dropped, took_ms}). No
provider, no key, no tokens, no egress — unlike "rerank", the
operator-pays hosted stage.

Why a second stage instead of extending rerank: the hosted stage
REPLACES _score with its probability and widens the page; the judge
leaves _score alone (a client sorting by _score keeps the engine's
number beside the new absolute scale) and judges exactly the emitted
page. Different budget, different contract, different failure domain
(503 judge_exception for a local model that cannot load, never a
paid-call failure; no degrade path — a caller who asked for judged
hits and silently got unjudged ones has been misled).

Which judge judged is on the wire: judged.scorer is "lexical" or
"model". The lexical arm is the always-compiled deterministic scorer
(window-local BM25 saturation with page-local IDF, normalized by the
max attainable share; K1/B from xerj-fts/src/bm25.rs) — NO semantic
signal, no quality claim. The model arm (feature decide-local, the
same xerj-api -> xerj-ai chain as the decide ladder) reuses the decide
loader landed in #1072: same candle NLI head, same model directory,
same process-wide registry — one loaded copy, two questions asked of
it. The model arm is UNMEASURED in this sandbox (no local model loads
here); the BEIR gate (0.699 SciFact / 0.345 NFCorpus / FiQA >= 0.30
beyond run spread) and the <= 40 ms p50 top-30 gate bind at release,
not in this PR.

Strictness mirrors the rerank stage: sort / search_after / collapse /
scroll / size:0 / _source:false-without-fields are refused by name
before the search runs, and every surface that does not run the stage
(scroll continuation, _msearch, _search/template, _msearch/template,
_async_search, _rank_eval, the native /v1 search API, the gRPC Search
RPC) refuses a body carrying "judge" by name instead of silently
dropping the block — the native API grew an explicit judge field for
this, since its parser ignores unknown keys. "judge": null is absent
everywhere; "local": false is refused (the local judge is the only
judge).

The judge reads the hit exactly as the response carries it: the
candidate builder is rerank_stage::unrestricted_candidate, extracted
from RerankPlan::candidate so the hosted stage's exhaustive-fields
path and the judge share one implementation and cannot drift on what
text a hit carries.

MCP: xerj_search and xerj_hybrid_search take an optional "judge"
argument (query required on hybrid, carried from the plain string on
xerj_search, validated with the node's own JudgeConfig parser before
the round trip). Tool descriptions state the honest ceiling and the
zero-egress difference from rerank in those words. Published schema
regenerated (11 tools, judge property added; drift test green).

Scope note: xerj-server/src/grpc.rs is a one-hunk extension of the
existing second-stage refusal there; judge-specific Prometheus metrics
are deliberately skipped (would need xerj-common changes outside this
PR's crate scope).

Tests: xerj-rerank 49 (51 with judge-local), xerj-api lib 292 (14 new
judge_stage unit tests), new tests/judge_stage_http.rs — 14 contract
tests over HTTP: opt-in inertness, response shape unchanged without
the block, _p_relevant present and ordered, _score untouched per id,
min_p drop counting against a self-adjusting cut, the refusal table,
per-surface named refusals, composition with the hosted provider, and
zero egress asserted against the provider stub's own request counter.
xerj-mcp 61 (7 new). Full suites green for -p xerj-rerank -p xerj-api
-p xerj-mcp; clippy clean on touched crates with and without
decide-local.
@cla-bot cla-bot Bot added the cla-signed label Sep 30, 2026
@xerj-org
xerj-org merged commit 618cb91 into main Sep 30, 2026
23 checks passed
neoneye pushed a commit to agent-memory-atlas-archive/xerj-org--xerj that referenced this pull request Oct 1, 2026
The packaging half of the ten-item program's rule (discussion xerj-org#1054): a
benchmarks/<name> directory with raw results plus a blog post with the
losses left in. The gate directories landed through xerj-org#1079, xerj-org#1085, xerj-org#1086,
xerj-org#1087, xerj-org#1088 and xerj-org#1090; this is the post.

Structure follows the Jev posts (verdict first, wins then losses with
readings, then what the next window does about each loss). Every number is
verbatim from the committed gate artifacts:

- PASS: /_ask macro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run
  determinism, wall-clock p50 1.129 ms vs the 300 ms bar (xerj-org#1076, xerj-org#1079);
  xerj_map unknown-field 400s 20/30 guessed -> 0/30 map-informed, 100%
  vs the >=80% bar (xerj-org#1074, xerj-org#1086); calibration FiQA pair-level ECE
  0.3109 raw -> 0.0088 held-out vs the <=0.10 bar, temperature 0.3163
  published as the failing comparison arm, $0.2125 of provider calls
  over 648 queries (xerj-org#1080, xerj-org#1087).
- FAIL, published as such: the lexical judge loses to its own first
  stage everywhere measured (0.7045->0.5967, 0.3419->0.2924,
  0.2382->0.1650 vs the 0.30 FiQA bar) and stays opt-in with no quality
  claim per its own rule (xerj-org#1077, xerj-org#1090); the flywheel freezes as
  shipped (12 of 2,000 cached, xerj-org#1094) and forces 19.72% coverage at
  0.9953 accuracy vs the 80%/0.97 bars (xerj-org#1075, xerj-org#1088); the tier-2
  decide head misses SMS 0.9193/0.287 vs 0.95/0.05 and AG News 0.2599
  vs 0.85, latency 8.4 ms PASS, ships behind the flag with the numbers
  published (xerj-org#1072/xerj-org#1073, xerj-org#1088).
- SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 PASS, zero-hit 25->15
  vs <=10 FAIL — the 15 have no stem anywhere in the corpus (xerj-org#1070,
  xerj-org#1085).
- The gate that caught the stock release binary built without the
  decide-local cargo feature (flag a no-op, stock-binary-noop.txt) is
  told with the fix queued ahead of the cut, not claimed as done.
- Honest gaps stated, not smoothed: the /_ask token arm designed not
  run, the _watcher detection gate not yet measured, no HF URL for
  xerj-decide-v1. Four issues filed from the measurements: xerj-org#1091-xerj-org#1094.

Mechanics: hand-written page at landing/blog/the-rc80-gates.html,
registered in scripts/seo/pagedata.py (published/updated 2026-09-30),
card prepended to the blog hub (hub dateModified bumped), head and
footer owned by fix_heads/fix_links, og falls back to the site card.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

search: self-judged hits — opt-in local judge rerank with _p_relevant per hit

1 participant

0