Repository navigation
Bench: establish regression baselines and verify scheduled reporting #840
Description
Completed verification (2026-11-05)
PR #859 merged as 116d42c54312279dc0459db397cad318e50158f9 after all 19 checks passed. The final report and offline reproduction instructions complete all five acceptance criteria below. Production history fixes were separately reviewed and merged in #853 (main) and #854 (v3).
- All 427 native + 26 binding IDs now have finite numeric baselines and upper limits on the correct v3 testbeds. Every ID has at least three independently traced v3 runs; inherited main observations are excluded from those counts. The latency t-test remains upper probability 0.999, maximum sample size 30, not a percentage budget or a claim of thirty runs. Initial limits can be wide.
- Two full v3 dispatches and a C-only probe supply the fresh history. Real main and v3 UDF probes verify that the v3 head stays unchanged after main's measured Bencher version advances 84 → 85. The collection includes the original report that started one second before its head was created; the initial truncated query is retained as diagnostic evidence.
- The frozen original archive verifies actual main/v3 scheduled events and exact nonempty/disjoint selected, executed and uploaded sets. The history fix leaves those routes unchanged. The five fresh runs are explicitly dispatches.
- 16 isolated live job scenarios verify four actual GitHub issue/comment deliveries, branch/suite/ID scoping, immediate failed-upload notification with audit failure, and warning/empty-history behavior on a real previous-log 404. All 46 synthetic Bencher resources were removed and independently returned 404. Test issues Benchmark regression: binding_audit_840 (audit-840-20261105) #856/Benchmark regression: audit_840_native (audit-840-20261105) #857 are closed; both temporary workflow refs are deleted.
- Three checksummed archives preserve the complete evidence beyond Actions retention. Fresh extraction verifies 1,500 internal checksums and reproduces eight derived reports byte-for-byte. Negative controls reject missing comparisons, inherited-main substitutions and missing delivery evidence; optimized Python is refused.
The isolated delivery driver redirects the physical workflow lookup to its test branch while retaining the production script bodies. This proves real GitHub delivery in that scope, not human-mailbox receipt or unmodified default-branch delivery.
The second full v3 run also produced three real concurrent_file/serialized_writers/100k_{2,4,16}t alerts. Their values and reporting log are retained. None occurred in the preceding run, so the two-run policy suppressed notification. The measured Rust code, benchmark code and Cargo dependencies are unchanged between those two source revisions; the performance cause remains unconfirmed.
Java/JVM, WASI and qualified iOS-simulator/Android-emulator nightly expansion are tracked separately in #858, with the user's clarification that #839's platform restriction applies only to #839. Local measurement/notification scratch is cleaned up; the final evidence worktree is clean.
Original scope and acceptance criteria (retained)
Follow-up of #776: X05 in the remaining exclusions and follow-ups table, finalized in #834.
Disposition: Unverified operational follow-ups.
Milestone: v3.0.0 — Usable regression history and end-to-end reporting/schedule verification are still missing release evidence.
Remaining scope recorded by the audit:
Accumulate numeric regression baselines for 244 native and 26 foreign IDs (all have stored v3 observations/model settings, but baseline null in V). Verify live failed-upload and repeated-alert notification delivery; successful uploads and parser/audit unit tests do not exercise it. Cron execution of the final routes is unverified here: V was a dispatch. Preserve/refresh evidence beyond the 30-day artifact retention.
The validation run's public Bencher metrics prove all 427 native and 26 binding IDs have stored observations at the audited core SHA, branch v3, upper t-test probability 0.999 and maximum sample size 30. Numeric comparison baselines existed for only 183 native IDs: 244 native IDs and all 26 binding IDs still had null baselines. Stored models alone cannot enforce those comparisons. Older inherited main metrics are not independent v3 runs. Native data uses the native latency testbed; the four bindings each have a separate runtime testbed and no native-history inheritance.
The default branch owns .github/workflows/bench.yml: main cron 02:00 UTC, v3 cron 14:00 UTC. Dispatch uses its requested ref; compile resolves the measured SHA once for all jobs, while reporting retains the workflow SHA. Run 41642356471 was a dispatch, so execution of the final scheduled routes was unverified by the audit.
Uploads use continue-on-error, followed by an always-run completeness audit that must fail missing/partial/duplicate/unexpected/API-error uploads. Notification policy reports a latency alert only when the same benchmark alerted in the previous same-branch/suite run; upload failure is immediate. Previous-log lookup failure warns and supplies an empty prior-alert set. All validation uploads succeeded, all alert counts were zero and Report regression steps were skipped. Parser/audit unit tests therefore do not prove live failed-upload or repeated-alert delivery.
Artifacts retain exact filters/IDs, raw logs/reports, measured/workflow/core/binding hashes and upload outcome for 30 days. Why outside the shortlist: accumulation requires independent history, cron observation requires actual scheduled runs, and notification delivery requires controlled end-to-end operational checks rather than more latency IDs.
Acceptance criteria:
- Inventory the 244 native and 26 foreign IDs with null baselines, accumulate sufficient independent v3 observations on the correct branch/testbeds, and publish API evidence of usable numeric comparisons for every intended ID; explain any remaining exceptions.
- Confirm the shared 0.999/30 model settings without treating 0.999 as a latency-percentage budget or inherited main history as independent v3 runs.
- Observe actual final main/v3 cron routes and record event type, measured SHA, workflow SHA, nonempty/disjoint selections and exact executed/uploaded sets, including all four binding jobs on v3.
- Perform controlled end-to-end failed-upload and same-ID repeated-alert checks in isolated test history, verifying audit failure and actual notification delivery plus branch/suite scoping; record previous-log lookup failure behavior and clean test artifacts.
- Archive or refresh complete run/ID/report/raw-log/API evidence beyond the 30-day artifact retention and link a reproducible durable audit record.
Run heavy builds, enumeration, measurements and comparisons on idle hardware; retain reproducible evidence and clean temporary scratch afterward. This is benchmark/verification work aligned with PHILOSOPHY.md's reliability, portability and simplicity principles; any production change needs its own review.
Evidence: docs/bench-coverage-v3.md; validation run 41642356471. The validation run establishes only its recorded representative latency coverage; it does not complete this follow-up.
Activity
ci: preserve independent v3 benchmark history (#840) (#853)
ci: preserve independent v3 benchmark history (#840) (#854)
docs: archive scheduled benchmark and baseline evidence (#840) (#855)
test: constrain cleanup to archived #840 test resources
docs: verify live notification cleanup and preserved production models (
docs: complete #840 verification and track nightly runtime expansion
docs: complete regression-history and live reporting verification (#840…