Repository navigation
ci(bench): report regressions per suite, show muted alerts, cut insert_file time (#393) - #417
Merged
Merged
Conversation
…t_file time (#393) - Build each populate_file fixture once and copy it per call. Criterion runs the setup closure once per sample plus warm-up, so the 1m fixture was rebuilt ~11 times per benchmark and insert_file took 3h+. - One regression issue per suite ("Benchmark regression: <suite>"), commented on while open, instead of one repo-wide issue. - Write every outcome, including unreported single-run alerts, to the job summary and as a warning annotation. - Threshold 0.999 for insert_file, open and checkpoint (file I/O noise). - actions/checkout@v7, actions/cache@v6, actions/github-script@v9 (Node 24). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #393.
Changes
Alerting (
.github/workflows/bench.yml)Benchmark regression: <suite>. While it is open, new repeated alerts for that suite are added as comments. Before this, one open issue muted every suite.continue-on-error, so without this the run looks clean.0.999forinsert_file,openandcheckpoint, the same asconcurrent_file/*. These do file I/O and fsync, which is noisy on shared runners.actions/checkout@v7,actions/cache@v6(+cache/restore@v6),actions/github-script@v9, all Node 24. The script onlyrequiresfs, so the v9 ESM change does not affect it.insert_filerun time (benches/helpers/mod.rs)populate_file(1_000_000)was rebuilt about 11 times for each of the threeinsert_file/*/1mbenchmarks. That is most of the 3h20m in run 36106225599.populate_filenow builds each size once in a shared temp dir and copies it to the caller's path. Each caller still gets a fresh, identical checkpointed file, so the measurements do not change and Bencher history stays comparable.open/checkpointedandconcurrent_file/*get the same speedup.criterion_main!is written out by hand so the fixture dir is deleted at exit.Already done / not in this PR
checkpointis still slow (about 70 min). Its per-iteration setup has to be fresh by design (iter_batched,LargeInput), so it cannot use a shared fixture.Testing
cargo bench --no-runbuilds. A quick run ofinsert_file/single_fact/{1k,10k,100k}andopen/checkpointed/100kworks, and the fixture dir is removed afterwards.node --check.--branch main. It will run at the next nightly after merge.🤖 Generated with Claude Code
https://claude.ai/code/session_01BedF9ZmqVZLkvDwn91fuKy