Bench: document semantic permutations and natural-plan control limits #842
Description
Follow-up of #776: X07 in the remaining exclusions and follow-ups table, finalized in #834.
Disposition: EXCLUDED exhaustive semantics.
Milestone: v3.next — Exhaustive semantic and forced-plan permutations are deliberate standing exclusions, not missing nightly release routes.
Remaining scope recorded by the audit:
All numeric, bind, cross-kind join and forced-planner permutations remain correctness tests. Nightly controls select representative natural plans; broader hash controls include broader scans and keyword dynamic controls include binding/clone work.
C15/C16/C20/C34/C35 cover representative engine behavior while correctness suites own exhaustive semantics:
- Numeric cases: ordinary avg; exact and mixed sums, sum-distinct, extrema and tuple fallbacks; signed-zero/NaN and mixed value behavior, including set elements.
- Bind cases: value/count/time/entity slots, entity versus ref forms, keyword/ref equality, set binds and their applicable combinations. Native prepared cases and four-host set round trips are representatives, not exhaustive bind permutations.
- Cross-kind joins: entity/value/hash joins, pending/committed data and per-fact pseudo metadata; anti/union/negation/disjunction and correlated rule-body cases.
- Planner strategy permutations: representative natural plans use preflight operator checks; exhaustive forced strategies remain correctness tests.
The six query/operator_paths/{entity,value,hash}_{pending,committed}/10k controls each return 10k rows from paired 20k-fact fixtures, but hash controls deliberately scan a broader attribute domain. The expr/keyword_equality/{literal_eq,dynamic_eq,literal_neq,dynamic_neq,nested_literal_eq,prepared_keyword_eq}/10k controls share a 10k keyword fixture, yet dynamic cases include extra binding/clone work. Neither family is a clean single-optimization off switch.
Why outside the nightly shortlist: a Cartesian product of semantics, binds and planner choices duplicates correctness coverage and expands routine cost; natural-plan controls represent real engine paths with acknowledged confounders.
Acceptance criteria:
- Map each excluded family (numeric, bind, cross-kind join and forced strategy) to concrete correctness test targets at a pinned revision and identify any missing contract coverage.
- Retain representative nightly IDs/operator preflight evidence and explicitly document which permutations they do and do not establish.
- Record broader-scan and binding/clone confounders next to relevant comparisons; do not infer isolated hash or cache speedups from these controls.
- Publish the standing exclusion decision; any justified new representative must have a distinct cost hypothesis and bounded nightly budget.
Run heavy builds, enumeration, measurements and comparisons on idle hardware; retain reproducible evidence and clean temporary scratch afterward. This is benchmark/verification work aligned with PHILOSOPHY.md's reliability, portability and simplicity principles; any production change needs its own review.
Evidence: docs/bench-coverage-v3.md; validation run 41642356471. The validation run establishes only its recorded representative latency coverage; it does not complete this follow-up.
Activity
Merge pull request #864 from project-minigraf/docs/842-semantic-permu…
adityamukho commentedon Nov 6, 2026
Done in #864 (merged into v3 as a95e51f). Report pinned to v3 5d85c36:
test-hooksjob. No missing answer contract was found.IndexNestedLoopJoincounter name, and only the six join controls check operators before timing.