Repository navigation
docs(bench): map semantic permutations to correctness tests (Refs #842) - #864
Merged
Merged
Conversation
Adds the X07 report for core revision 5d85c36 on v3. It maps the numeric, bind, cross-kind join and forced-planner families excluded from nightly benchmarks to 128 curated correctness tests (130 targets in 29 cells), verified against the pinned plain and test-hooks inventories (17 tests are hooks-only; CI runs both builds). A standard-library script reproduces a lexical index of all 2,079 Rust tests and the target check byte for byte. No missing answer contract was found. Three attribution limits are recorded: no join-kind switch, entity and value probes share an operator counter name, and only the six join controls check operators before timing. The hash-scan and keyword binding/clone confounders sit next to the comparisons. X07 stays excluded; no nightly ID changes. No production code, public API, test or nightly-route change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H2a7QoYX6tc5G3sKVzvLjD
4 tasks done
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related #842 (X07 of the #776 close-out).
What this adds
docs/benchmarks/842-semantic-permutations/README.md: the X07 report pinned to v35d85c36f9a4ec3056d59a14982ff5bd3490a2263.targets.csv: 128 curated correctness tests (130 targets in 29 cells) for the numeric, bind, cross-kind join and forced-planner families.map.py(standard library): checks every target against the pinned plain andtest-hooksinventories, and writes a lexical index of all 2,079 Rust tests. Its output reproduces byte for byte from the committed lists inevidence/.bench-coverage-v3.md, plus CLAUDE, README, ROADMAP, CHANGELOG, TEST_COVERAGE and BENCHMARKS.No production code, public API, test or nightly-route change. Nothing was timed: the inventory is a build plus
--list.Findings
test-hooksbuild, which CI runs (rust.yml).prepared.rsbind), so a forced-plan × bind matrix would repeat the literal-query matrices.IndexNestedLoopJoin, so theoperator_pathsentity/value labels rest on query shape, not a counter.hash_*controls scan every attribute (20k facts) and then filter.dynamic_*keyword controls add a binding lookup and a clone per row.Verification
map.pyreruns on the committed evidence and reproducestests.csv,summary.jsonandtargets.jsonbyte for byte.🤖 Generated with Claude Code
https://claude.ai/code/session_01H2a7QoYX6tc5G3sKVzvLjD