Repository navigation
Audit nightly benchmark coverage for all v3 features added over v2 #776
Description
Audit nightly benchmark coverage for every feature added in v3 over v2 before v3.0.0 ships. A feature having unit tests, a manual performance report, an envelope shape, or a registered Criterion benchmark does not by itself establish nightly regression coverage.
Part of production readiness: #383. Related: #393 (regression alerting), #394 (performance envelope), and #775/#532 (post-streaming performance verification).
Scope and acceptance
- Build a complete feature inventory from the latest supported v2 release versus the
v3branch, recording both revisions and checking CHANGELOG, ROADMAP, public APIs, feature flags and binding additions. Distinguish v3-only additions from fixes also backported to v2. Refresh the inventory for features that land before the v3.0.0 cut. - Include, where present in the release:
:distinct;:limit/:offset/:order-byand top-k; set values, membership predicates andcollect-set;lag/leadand bounded row frames; streaming cursors and operator paths; query-memory limits; large/streaming transactions and snapshot reads; read-only open, fact-log iteration andLogWriter; v8 storage/index/checkpoint/migration and verify/rebuild APIs; and newly exposed prepared-query, UDF and options APIs in bindings where they introduce distinct costs. This is a starting list, not a substitute for the inventory. - Publish a coverage matrix: feature/subpath → benchmark IDs → fixture sizes/storage mode/features → nightly job/filter → evidence from an actual v3 run and regression reporting. Mark missing, manual/envelope-only and deliberately excluded coverage explicitly, with reasons.
- Audit the workflow actually used by scheduled runs (the default-branch workflow, its v3 checkout and matrix), Criterion registration and naming, and Bencher uploads/thresholds/alerts. Detect empty filters and registered groups that no scheduled job selects. Confirm that each intended benchmark executes and has a usable v3 regression history; a green workflow alone is insufficient.
- Add or route representative benchmarks for uncovered release features through small reviewable follow-ups. Exercise materially different execution paths and scaling, including committed/pending data, selective/scan queries, cursor draining and applicable binding overhead. Track memory/scaling where bounded-memory behavior is part of the feature; document any metric the nightly infrastructure cannot yet enforce.
- Validate workflow changes with a v3 dispatch and inspect the executed benchmark names and reporting. Keep nightly runtime practical. Run heavy enumeration/builds, measurements and comparisons on the idle bench machine, clean its scratch afterward, and record deliberate exclusions or remaining follow-ups before closing.
Concrete starting point
At v3 revision d0ad89c1d6df734afd5d4669ca9139c8f55d7702, benches/minigraf_bench.rs registers bench_window_rows under the window_rows/... group. The nightly window job uses ^window/, which does not select that group. Check this and all other new groups against the complete scheduled matrix; compiling the benchmark target does not mean a case is run nightly.
Done when every release feature has an explicit, verified nightly-coverage disposition and any required coverage additions have landed, with the matrix and run evidence linked here. This is development/CI work: preserve public API, file format, existing correctness checks and Minigraf's embedded, self-contained design.
Activity
Merge pull request #808 from project-minigraf/ci/776-bench-coverage-a…
ci(bench): share enumeration and preserve compiled cache keys
ci(bench): share enumeration and preserve compiled cache keys
ci(bench): gate history suite on checkout availability (Refs #776)
bench: retain lazy history fixtures across samples (Refs #776)
bench: clarify first-write set ingest costs
bench: distinguish join metadata sources (Refs #776)
ci(bench): verify benchmark selection and reporting provenance (P17 m…
ci(bench): route bounded row-frame shortlist
ci(bench): route row-frame shortlist in scheduled workflow
ci(bench): route bounded row-frame shortlist
58 remaining items
ci(bench): preserve verification and isolate binding experiments (Refs …
Merge pull request #832 from project-minigraf/bench/776-p16-bindings
ci: schedule P16 binding benchmarks from main (#833)
Merge pull request #834 from project-minigraf/docs/776-final-matrix
adityamukho commentedon Nov 4, 2026
All P01–P17 are merged; final matrix #834; 427 native IDs nightly versus 270 before, plus 26 foreign-runtime IDs; validation run 41642356471 passed all 31 jobs.
Every X01–X09 row now has a standalone follow-up with expanded scope, disposition, acceptance criteria and milestone rationale. Release sign-off, memory/scaling acceptance, and regression/reporting verification remain tracked in the v3.0.0 follow-ups; broader sweeps and standing exclusions are in v3.next. Closing this completed coverage audit does not certify those remaining checks.