8000 docs(bench): map semantic permutations to correctness tests (Refs #842) by adityamukho · Pull Request #864 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Documentation

- **Semantic permutation map** (#842): [X07's correctness-target map](docs/benchmarks/842-semantic-permutations/README.md) ties numeric, bind, cross-kind join and forced-planner permutations to 128 named tests at a pinned v3 revision, states what the representative nightly controls do and do not establish, and records the hash-scan and keyword binding/clone confounders. X07 stays excluded. No production, test, public API or nightly-route change.

- **Manual shape sweep** (#841): [native shape and retention experiments](docs/benchmarks/841-shape-sweeps/README.md) cover sort/window, set and UUID cases, matched recursive controls and sustained pins/FactLogs. The report distinguishes query timings from process RSS and query-meter observations, and retains input rejections, capped attempts and attribution limits. No production, public API or nightly-route change.

- **Regression history and reporting verification** (#840): [durable API/run evidence](docs/benchmarks/840-regression/completion.md) establishes numeric comparisons for all 427 native and 26 binding IDs with at least three independent v3 runs each. Separate main/v3 fixes #853/#854 preserve v3 history across measured main advances. Actual scheduled routes, 16 isolated live notification scenarios, four delivered messages and complete synthetic-resource cleanup are verified. This adds no database runtime behavior or Rust tests; further runtime lanes are tracked in #858.
Expand Down
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ checksummed evidence outlives Actions retention. This changes no database runtim
behavior or Rust test counts. Additional JVM/WASI and qualified mobile nightly
lanes are tracked separately in #858.

The [#842 semantic permutation map](docs/benchmarks/842-semantic-permutations/README.md)
ties the excluded numeric, bind, join and forced-plan permutations to 128
correctness tests and documents the nightly controls' attribution limits. It
changes no code, tests or nightly routes.

The [#841 manual shape sweep](docs/benchmarks/841-shape-sweeps/README.md)
records native sort/window, set, UUID, recursive-control and sustained-reader
experiments, including rejected inputs, capped attempts and attribution limits.
Expand Down
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ history, scheduled main/v3 routes and isolated live notification delivery with
cleanup. Raw evidence and offline verifiers are retained in Git. JVM/WASI and
qualified mobile nightly expansion are tracked separately in #858.

The [semantic permutation map](docs/benchmarks/842-semantic-permutations/README.md)
(#842) ties excluded numeric, bind, join and forced-plan permutations to
correctness tests and states what the natural-plan nightly controls do not
isolate. It adds no nightly benchmark.

The [manual shape sweep](docs/benchmarks/841-shape-sweeps/README.md) (#841)
examines native sort/window, set and UUID shapes, recursive controls, and
sustained reader retention. The report separates query latency, process RSS,
Expand Down
1 change: 1 addition & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ Scope — release-gate testing and documentation:
- Regression-history/reporting evidence (#840) is complete: [all 453 numeric comparisons, preserved independent v3 history, scheduled routes and live notification/cleanup evidence](docs/benchmarks/840-regression/completion.md). Separate main/v3 fixes #853/#854 are merged; raw evidence is retained beyond artifact expiry.
- Nightly runtime expansion (#858): add Java/JVM and WASI; qualify Swift/iOS simulator and Android emulator timings against physical-device workloads before enabling representative mobile lanes. The platform restriction from #839 applies only to that manual evidence task.
- Manual migration evidence (#838): native v7-to-v8 through 200k facts; Chromium/Firefox/WebKit through 20k facts, with separate read-only/reopen controls and bounded post-migration cursor churn. [Results, frozen fixtures and exclusions](docs/benchmarks/838-migration/README.md); no new nightly route or sustained-pin bound.
- Semantic permutations (#842): [correctness-target map and control limits](docs/benchmarks/842-semantic-permutations/README.md) for X07. The exclusion stands; no nightly route or acceptance gate is added.
- Manual shape evidence (#841): [native shape sweeps, recursive controls and sustained-reader experiments](docs/benchmarks/841-shape-sweeps/README.md), with explicit input, timeout and attribution limits. The small-nightly recommendation is separate from execution; this audit adds no nightly route or release-acceptance gate.
- Expression keyword/ref equality (#774; on `v3`): resolve literal keywords once per operator, with memory bounded by the expression. [Measurements and regression coverage](docs/benchmarks/774-expression-keywords.md); performance-only, with no v2.x backport.
- Higher storage/WAL coverage gates and branch-coverage gates (#396; on `v3`: line and branch floors per file in `coverage-thresholds.toml`, checked on every PR)
Expand Down
8 changes: 8 additions & 0 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

This document records reproducible local benchmark snapshots. Completed release-performance history belongs in the [CHANGELOG](../CHANGELOG.md); continuous CI history is available on [Bencher](https://bencher.dev/perf/minigraf/plots).

## Semantic permutations and control limits (#842, unreleased)

The [X07 map](benchmarks/842-semantic-permutations/README.md) assigns
numeric, bind, cross-kind join and forced-planner permutations to correctness
tests rather than nightly IDs. Hash-control ratios include a broader scan, and
literal/dynamic keyword ratios include binding and clone work: neither is an
isolated optimization measurement.

## Manual shape sweeps (#841, unreleased)

The [shape and retention report](benchmarks/841-shape-sweeps/README.md) records
Expand Down
6 changes: 6 additions & 0 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@ cleanup of all 46 synthetic resources. Fresh extraction reproduces the evidence
reports, and removing delivery/comparison evidence is rejected. No Rust tests
are added; the inventory above is unchanged. Additional runtime lanes are #858.

**#842 semantic permutation map**: [128 curated correctness tests](benchmarks/842-semantic-permutations/README.md)
(130 targets in 29 cells; 17 tests exist only in the `test-hooks` build) cover the numeric, bind,
cross-kind join and forced-planner families excluded from nightly benchmarks.
A reproducible lexical index tags all 2,079 Rust tests at the pinned revision.
No test is added; the inventory above is unchanged.

**#841 manual shape verification**: the [standalone audit harness](benchmarks/841-shape-sweeps/README.md)
checks full query answers before accepting timings and records sustained-reader
snapshot, quarantine-generation, release/reuse, integrity and reopen checks.
Expand Down
6 changes: 3 additions & 3 deletions docs/bench-coverage-v3.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,8 +213,8 @@ selected IDs in Appendix A, not every registered case in a family.
| C12 | Cursor full draining and early limit | `query/ordered_groups/…`, `query/partition_windows/…`, `query/cursor_lifecycle/drain/{10k,100k}` | F4 proof groups use batch 1000/full drain or limit 10; dedicated drain uses batch 256 | query; V+B | COVERED |
| C13 | Cursor lifecycle, pins and staged snapshot reads | `query/cursor_lifecycle/…`; `cursor_lifecycle.rs` | F 10k open/first batch/iterator/close; 10k/100k drain; old 10k snapshot across two setup checkpoints; staged 10k+100 pending+100 uncommitted | query; V+B | COVERED eight IDs; checkpoint setup is untimed, sustained quarantine growth MANUAL (X06) |
| C14 | Entity probes, pending filters and selective recursion | `query/join_3pattern/…`, `query/predicate_pushdown/…`, `query/operator_paths/entity_{pending,committed}/10k`, `recursion/ancestors/w10_d4` | M4 joins, M3 selective filters; paired 20k-fact joins; 11,110-edge ancestor tree | query; recursion `^recursion/`; V+B | COVERED; optimization-specific isolation remains MANUAL (X06) |
| C15 | Value/hash joins and per-fact metadata source | `query/operator_paths/{entity,value,hash}_{pending,committed}/10k`; `operator_paths.rs` | 10k output rows/20k facts; pending memory vs committed file; hash deliberately scans broader attribute domain; preflight operator checks | query; V+B | COVERED six natural-plan controls; exhaustive cross-kind/forced variants EXCLUDED (X07) |
| C16 | Anti/union operators and rule bodies | `negation/…`, `disjunction/…`, `query/operator_paths/{anti_committed,union_committed}/10k` | M 1k/10k selectivity/rule cases; committed correlated not plus hashed not-join, union with nested outer-variable expressions | negation `^negation/`; disjunction `^disjunction/`; query; V+B | COVERED representatives; forced strategy permutations EXCLUDED (X07) |
| C15 | Value/hash joins and per-fact metadata source | `query/operator_paths/{entity,value,hash}_{pending,committed}/10k`; `operator_paths.rs` | 10k output rows/20k facts; pending memory vs committed file; hash deliberately scans broader attribute domain; preflight operator checks (probe vs `HashJoin`; both probes report `IndexNestedLoopJoin`, [#842](benchmarks/842-semantic-permutations/README.md#missing-contract-coverage)) | query; V+B | COVERED six natural-plan controls; exhaustive cross-kind/forced variants EXCLUDED (X07) |
| C16 | Anti/union operators and rule bodies | `negation/…`, `disjunction/…`, `query/operator_paths/{anti_committed,union_committed}/10k` | M 1k/10k selectivity/rule cases; committed correlated not plus hashed not-join, union with nested outer-variable expressions | negation `^negation/`; disjunction `^disjunction/`; query; V+B | COVERED representatives (anti/union preflight checks row counts, not body strategy); forced strategy permutations EXCLUDED (X07) |
| C17 | Pseudo metadata and transaction/valid-time bounds | `temporal_metadata/…`, `time_travel/{as_of_counter,valid_at}/…`, `time_travel/operator_paths/timestamp_{monotone,irregular}/10k` | M3 metadata (tx_time case reads tx-count), M4 broad temporal points; committed 10k records, irregular timestamps require non-prefix cutoff | temporal_metadata `^temporal_metadata/`; time_travel `^time_travel/`; V+B | COVERED broad and selective committed timestamp representatives |
| C18 | General aggregates, exact arithmetic and :with | `aggregation/{count_scale,grouped_count_scale,sum_scale,with_grouped_sum,count_distinct_scale}/…` | M2 integer/count/:with; 10 departments, 50% count-distinct duplication | aggregation `^aggregation/`; V+B | COVERED; mixed/exact controls in C20 |
| C19 | Recursive engine/demand and rule limit checking | `recursion/{chain,fanout,ancestors}/…`, `recursion/operator_paths/ancestors_committed/10k`; rule-body negation/disjunction | Pending chain/tree; committed 10-edge ancestor chain amid 10k filler facts | recursion; negation; disjunction; V+B | COVERED positive recursion; nonlinear/multistratum/limit scaling ENVELOPE/acceptance (X02) |
Expand Down Expand Up @@ -393,7 +393,7 @@ shortlist. No manual timing or release sign-off is implied by this close-out.
| X04 | MANUAL binding checks | [#839's runtime evidence](benchmarks/839-bindings/README.md) records three series each for JVM, Android, Swift/iOS and WASI (1,350 samples); binding/release acceptance remains pending. New #839 timings use only Device Farm/the dedicated bench; macOS is excluded, existing GitHub nightlies unchanged. [The remaining API/lifecycle dispositions](benchmarks/839-bindings/dispositions.md) identify measured representatives, absent APIs and proposed exclusions, including reader churn beyond [#838's bounded persistence case](benchmarks/838-migration/README.md). Python shared UniFFI coverage does not measure other host runtimes; no binding RSS bound is asserted |
| X05 | Unverified operational follow-ups | Accumulate numeric regression baselines for 244 native and 26 foreign IDs (all have stored v3 observations/model settings, but baseline null in V). Verify live failed-upload and repeated-alert notification delivery; successful uploads and parser/audit unit tests do not exercise it. Cron execution of the final routes is unverified here: V was a dispatch. Preserve/refresh evidence beyond the 30-day artifact retention |
| X06 | MANUAL / ENVELOPE scaling and isolation | Large 10m sorting/window/envelope sweeps, wider set cardinalities, sustained pin/quarantine/log-reader churn and sparse/repeated-ref UUID projection are outside V's fixed shortlists. [#841's report](benchmarks/841-shape-sweeps/README.md) records the finite public shape matrix, a capped-window retry, matched recursive controls with baseline path counters, 544-generation pin/FactLog churn and charged query memory, with every rejected, capped, limit-error and deferred slot; it establishes no universal latency or memory bound. The [protocol](benchmarks/841-shape-sweeps/PROTOCOL.md) distinguishes the finite selection from broader extensions. [#838](benchmarks/838-migration/README.md) measures only eight commits with one old cursor plus a short-lived cursor per commit. Recursive controls are end-to-end comparisons: counters show which baseline paths each case uses, not each path's share of time, and the ancestor alternative's 128-row cutoff remains unmeasured |
| X07 | EXCLUDED exhaustive semantics | All numeric, bind, cross-kind join and forced-planner permutations remain correctness tests. Nightly controls select representative natural plans; broader hash controls include broader scans and keyword dynamic controls include binding/clone work |
| X07 | EXCLUDED exhaustive semantics | All numeric, bind, cross-kind join and forced-planner permutations remain correctness tests. Nightly controls select representative natural plans; broader hash controls include broader scans and keyword dynamic controls include binding/clone work. [#842's map](benchmarks/842-semantic-permutations/README.md) pins 128 correctness tests (130 targets in 29 cells) (v3 5d85c36f, all run in CI's plain or `test-hooks` job), finds no missing answer contract, and records three attribution limits: no join-kind switch, probes indistinguishable by counter, and answer-only preflight outside the six join controls. The exclusion stands; no nightly ID changes |
| X08 | EXCLUDED individual failure latency | Invalid bounds/windows/binds, deliberate callback errors, corrupt-page rejection, v1–v6 refusal, heavy single-entry WAL replay and individual PRS/API/QRY codes belong to correctness testing. C08 measures a successful repair of known damage and C30 measures cross-chunk rollback; neither claims exhaustive failure timing |
| X09 | EXCLUDED unshipped/non-runtime scope | Public profiler #185, OPFS #441 and rule persistence #241 are open/unshipped at the audited revision. Proof/test/docs infrastructure and #797's study add no shipped latency path; production uuid/dist size uses separate size gates. Re-audit new production features through the release cut |

Expand Down
Loading
Loading
0