8000 ci(bench): regression alerts have been muted since #301; fix alerting and noisy thresholds · Issue #393 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content

ci(bench): regression alerts have been muted since #301; fix alerting and noisy thresholds #393

Closed

Description

@adityamukho

Problem

  • .github/workflows/bench.yml skips filing a regression issue when any issue titled "Benchmark regression" is open. Benchmark regression - 2026-06-30 #301 has been open since 2026-06-30, so no regression reported after it has filed an issue.
  • Proof: in run 36106225599 (2026-09-25), Bencher flagged 6 alerts on insert_file/*. The job log says Skipping: open regression issue already exists (#301).
  • The Bencher upload step is continue-on-error: true, so the run still shows green.
  • insert_file results are very noisy on shared runners (95% CI such as 594 µs to 1.33 ms), so these alerts may be false. Nobody is triaging them either way.
  • The suite takes 75 minutes to 6 hours each night. insert_file alone took 3h20m, and 6 runs since August were cancelled.

Ask

  • Triage and close Benchmark regression - 2026-06-30 #301.
  • Remove duplicates per benchmark group, not per repo: add to an open issue for the same group instead of skipping. Or post one summary issue per run that lists every alert.
  • Move file-backed and fsync-heavy groups (insert_file, concurrent_file/*, checkpoint, open) to looser thresholds or a separate testbed. Better: use a dedicated or bare-metal runner for them.
  • Cut insert_file run time (fewer sizes or samples) so the suite finishes within one runner session.
  • Make a suppressed alert visible in the job summary.
  • Update pinned actions that GitHub warns run on deprecated Node 20 (actions/checkout@v4, actions/cache@v4, actions/github-script@v6).

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceBenchmark performance regressionsstabilityStability/panic issue

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0