Repository navigation
fix(engine): hybrid total reports Eq at the exact per_query_topk boundary (#594) - #617
Conversation
Verification — 3 skeptics, verdicts verbatim (HEAD 62303a3). 3/3 NON_BLOCKING — but reworking on the evidence lens's finding.correctness — NON_BLOCKING
regression — NON_BLOCKING
evidence — NON_BLOCKING (with a finding)
Action: the evidence lens is right. Rather than ship redundant code + a misleading claim, I'm simplifying to the minimal load-bearing change (fix the boundary over-trigger; rely on the leg's own |
…dary (#594) The #569 capped→`gte` fix tripped `fused_count_capped` on `sub_result.hits.len() >= per_query_topk`. A hybrid sub-leg with EXACTLY `per_query_topk` real matches (and none beyond) fills a full page, so that arm reported `relation: gte` even though the fused total is exactly known — where ES reports `eq`. The `value` was always correct and "≥ value" held (safe direction), so it was a mild precision regression, deferred from #593 as NON_BLOCKING. The leg's own `total.value` / `total.relation` is the load-bearing capping signal, and it is already consulted here (`total.value > per_query_topk || relation == Gte`, from #569): a genuinely-capped leg has `total.value > per_query_topk` (or `Gte`), while a leg with exactly `per_query_topk` matches reports `total.value == per_query_topk` with `Eq`. So the only defect was the `hits.len() >= per_query_topk` arm over-triggering at the boundary. Change it to `> per_query_topk`: a full page no longer implies capped, and the exact-boundary leg stays `Eq`, restoring ES parity. (This is the "consult each leg's exact total.value" alternative the issue itself offered — no over-fetch needed.) The `hits.len() > per_query_topk` arm is now a cheap defensive guard: it never fires for `size == per_query_topk` (a leg cannot return more than requested), but it catches any future leg that over-returns without a matching total. Since `total.value >= hits.len()` always, it is subsumed by the `total.value` arm for every real leg — an earlier revision's extra `per_query_topk + 1` probe fetch (+ trim) was therefore redundant and has been dropped. Fail-before: reverting the capped check `>` → `>=` flips the new `hybrid_reports_eq_at_the_exact_per_query_topk_boundary` test (50 docs, kNN k=256) to `TotalHits { value: 50, relation: Gte }`. The #569 tests are unchanged (256-doc → Gte via `total.value`, 6-doc → Eq). Full `cargo test -p xerj-engine` green. Closes #594.
Round-2 verification (SIMPLIFIED revision e2bf5d7) — 3 skeptics, 8000 verdicts verbatim. 3/3 NON_BLOCKING.correctness — NON_BLOCKING
evidence / fail-before — NON_BLOCKING
regression — NON_BLOCKING
Gate: 3/3 NON_BLOCKING. Merging on all-checks-green + CLEAN. |
What
The #569 capped→
gtefix trippedfused_count_cappedonsub_result.hits.len() >= per_query_topk. A hybrid sub-leg with exactlyper_query_topkreal matches (and none beyond) fills a full page, so the fused total was reportedrelation: gteeven though it is exactly known — where ES reportseq. Thevaluewas always correct and "≥ value" held (safe direction), so this was a mild precision regression, deferred from #593 as NON_BLOCKING.Fix (index.rs, 3 hunks)
per_query_topk + 1per leg as a probe.> per_query_topkhits (the probe found a(per_query_topk+1)th match), or its own exact total exceeds the cap, or its own count is already a lower bound.per_query_topkbefore fusion, so the fused ranking and the returned page are byte-identical — onlytotal.relationat the exact boundary changes.A leg with exactly
per_query_topkmatches returnsper_query_topkhits → not capped →Eq, restoring ES parity at the boundary.Verification (rustc 1.97.1)
>check would falsely reportEqfor a genuinely-capped leg (the 256-doc test); without>the boundary reportsGte(the new 50-doc test).>→>=flips the newhybrid_reports_eq_at_the_exact_per_query_topk_boundarytest toTotalHits { value: 50, relation: Gte }.fmt --all --check✓ (exit-checked),clippy --workspace --all-targets -- -D warnings✓, fullcargo test -p xerj-engine→ 1044 passed / 0 failed.Closes #594.