Repository navigation
fix(engine): hybrid/fusion reports a capped total as relation:gte, not a false eq (#569) - #593
Conversation
…t a false eq (#569) `run_hybrid_with_deadline` fetches each sub-list at `per_query_topk = max(50, size+from)`, RRF/linear-fuses them, and reported `hits.total = { value: fused.len(), relation: Eq }` unless the request timed out. When any sub-list was truncated at the fetch cap the fused union under-counts the true match set, so a hybrid query over thousands of lexical matches with `size:10` claimed `{"total":{"value":<=100,"relation":"eq"}}` — a false EXACT count that `track_total_hits` cannot correct, while a plain lexical query reports the real count. Affects every hybrid query and, once #458 landed, the ES 8.x `knn`-beside-`query` shape that routes to hybrid. Track `fused_count_capped` across the sub-query loop: a sub-list is truncated if it filled the `per_query_topk` page, or its own total exceeds the page, or its own count is already a lower bound (`relation == Gte`). Report `Gte` when `any_timed_out || fused_count_capped`, and `Eq` only when every sub-list was fully retrieved — the honest, ES-style wire shape. The reported `value` (`fused.len()`) is a valid lower bound in the capped case. Tests: `hybrid_reports_gte_when_a_sub_list_is_capped` (256 vectors, a knn sub-list capped at 50, no timeout → Gte) and `hybrid_reports_eq_when_no_sub_list _is_capped` (6 docs, fully fetched → Eq, value 6). Fail-before (reverting only `|| fused_count_capped`): the capped hybrid reports a false Eq. Full xerj-engine suite pending; fmt clean; clippy -D warnings 0 under 1.98.0.
Independent 3-skeptic verification — all 3 NON_BLOCKINGVerdicts verbatim. Gate (b) satisfied. The single finding across all three lenses is the safe-direction exact-boundary over-report, filed as follow-up #594. 🟢 CORRECTNESS — NON_BLOCKING
🟢 REGRESSION — NON_BLOCKING
🟢 EVIDENCE — NON_BLOCKING
Tally: 3 NON_BLOCKING, 0 BLOCKING. Merge gate (b) ✅. Holding on (a) all-green + (c) CLEAN. Boundary nit → #594. |
Tracks the v1.0.0-rc.32 tag (88f9237), landing xerj-org#536 (deterministic HNSW rebuild await, xerj-org#592) and xerj-org#569 (hybrid/fusion total relation:gte when capped, xerj-org#593). Clears the tag-ahead-of-stamps landing-constants-guard transient. Guard validated.
Closes #569.
The bug (wrong results on a headline BM25+kNN feature)
run_hybrid_with_deadlinefetches each sub-list atper_query_topk = max(50, size+from), RRF/linear-fuses them, and reportedhits.total = { value: fused.len(), relation: Eq }unless the request timed out. When a sub-list is truncated at the fetch cap the fused union under-counts the true match set, so a hybrid query over thousands of lexical matches withsize:10claimed{"total":{"value":<=100,"relation":"eq"}}— a false exact count thattrack_total_hitscannot correct, while a plain lexical query reports the real count. Affects everyhybridquery and (since #458) the ES 8.xknn-beside-queryshape that routes to hybrid.The fix (ES-style, minimal — issue's option (a))
Track
fused_count_cappedacross the sub-query loop — a sub-list is truncated if it filled theper_query_topkpage, or its own total exceeds the page, or its own count is already a lower bound (relation == Gte). ReportGtewhenany_timed_out || fused_count_capped, andEqonly when every sub-list was fully retrieved. The reportedvalue(fused.len()) is a valid lower bound in the capped case — honest, not the exact tally (option (b), a union-aware exact count, is deliberately not attempted here).Verification
hybrid_reports_gte_when_a_sub_list_is_capped(256 vectors, a knn sub-list capped at 50, no timeout →Gte) andhybrid_reports_eq_when_no_sub_list_is_capped(6 docs, fully fetched →Eq, value 6).|| fused_count_capped): the capped hybrid reports a falseEq.cargo test -p xerj-engine: 0 failed (full suite, exit 0).cargo fmt --check: clean.cargo clippy -p xerj-engine --all-targets -- -D warnings(1.98.0): exit 0.The sibling
min_score-vs-RRF-micro-scores note in #569 is separate and left tracked there. An independent 3-skeptic verification follows in a comment.