Repository navigation
Improve nightly benchmark regression detection with repeat filtering - #401
Merged
Merged
Conversation
Shared runners make single-night Bencher alerts mostly noise. From Jul 1 to Sep 25, 31 of 87 nightly runs raised alerts, and none were real regressions (see #301). The report step now: - saves the Bencher output and reads the alerted benchmark names - reads the same suite's alerts from the previous completed run - reports only benchmarks that alerted on both runs - comments on an open regression issue instead of skipping it, so an old open issue no longer hides new alerts - still reports at once when Bencher fails without listing alerts Adds the actions: read permission to read the previous run's job logs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019dotPmboGEcQoQWSngbceY
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enhance the nightly benchmark regression detection workflow to reduce false positives from noisy shared CI runners. The workflow now:
set -o pipefailto catch Bencher command failures and captures output for alert parsingType of change
Changes
.github/workflows/bench.yml:actions: readpermission to allow reading previous workflow run logsteeand addset -o pipefailNotes for reviewer
The key insight is that shared CI runners are inherently noisy—a single alert often indicates a slow runner, not a real regression. By requiring alerts to repeat across two consecutive nightly runs, we filter out transient noise while still catching real regressions. The regex pattern
- (\S+) \(Latency\): https://bencher\.dev/console/projects/[^/\s]+/alerts/specifically targets Bencher's alert format in the output.The change to add comments instead of silently skipping when an issue exists ensures that if a regression persists across multiple nights, all alerts remain visible in the issue thread for investigation.
https://claude.ai/code/session_019dotPmboGEcQoQWSngbceY