Repository navigation
feat(ci): alert repository owners via GitHub issue on nightly test failure - #884
Merged
Merged
Conversation
…ilure Add automated alerting step in nightly test suite workflow that: - Reads OWNERS and ABSENCE.md to resolve active owners - Comments on existing open 'nightly-failure' issue or creates a new issue - Mentions active owners with failure metadata and logs link TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
There was a problem hiding this comment.
Code Review
This pull request introduces a new GitHub Actions script nightly-failure-alert.js and its corresponding test suite to automate alerting on nightly test failures. The script parses owners and absences, formats markdown alerts, and manages GitHub issues. Feedback on the implementation includes ensuring that pull requests are filtered out when querying for existing issues, safely handling cases where NEEDS_CONTEXT parses to null, and wrapping test executions in try...finally blocks to guarantee cleanup of temporary directories and environment variables.
TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
…tests - Safely access needsContext properties with optional chaining - Filter out pull requests when querying existing open nightly-failure issues - Wrap unit test orchestrator logic in try...finally for deterministic cleanup TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
…ctions Enforce single responsibility principle and 20-30 line function limit: - Extract resolveActiveOwners, lookupAssociatedPullRequest, findExistingNightlyFailureIssue, postOrUpdateAlertIssue, and parseNeedsContext - Keep main run orchestrator concise and linear TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
Add dedicated test suites for all decomposed functions in nightly-failure-alert.js: - resolveActiveOwners: file reading & missing file handling - lookupAssociatedPullRequest: PR resolution, empty results, and rate limit error safety - findExistingNightlyFailureIssue: PR exclusion filtering and network error handling - postOrUpdateAlertIssue: comment creation vs new issue creation - parseNeedsContext: valid JSON parsing and invalid JSON fallback TAG=agy CONV=acf882de-1561-404c-ba2e-ae0940451165
janzak-dev
marked this pull request as ready for review
September 15, 2026 10:08
janzak-dev
enabled auto-merge
September 15, 2026 10:10
andrzejfudro
previously approved these changes
Sep 15, 2026
andrzejfudro
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor th
7F22
at reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an automated failure reporting and alerting mechanism to the nightly test suite workflow (
nightly-test-suite.yml). When any nightly workflow job fails, GitHub Actions automatically resolves active repository owners fromOWNERS(excluding any absent members listed inABSENCE.md), searches for opennightly-failureissues to prevent notification spam, and either creates a new issue or posts a recurring failure comment pinging all active owners.Motivation & Context
Nightly test suite runs (such as the demo workload verification against the
xmc-nightlydeployment) run asynchronously on a cron schedule. When infrastructure or reconciliation failures occur (e.g. WorkUnits stuck inSTARTINGor cluster timeouts), there was previously no proactive notification mechanism alerting repository maintainers. Maintainers had to manually inspect GitHub Actions run history to discover breakages.Key Changes
.github/scripts/nightly-failure-alert.js):parseUsernamesandfilterActiveOwnersto extract candidate usernames fromOWNERSand filter out members fromABSENCE.md(case-insensitive).formatFailureIssueBodyandformatRecurringCommentBodyproducing structured tables with commit SHA, workflow run link, Experiment XID, and@ownermentions.nightly-failureto append a recurrence comment instead of creating duplicate issues..github/scripts/nightly-failure-alert.test.js):.github/workflows/nightly-test-suite.yml):report-nightly-failure(3. Report Failure & Alert OWNERS) configured withif: always() && (needs.trigger-demo-workload.result == 'failure' || needs.validate-workload-results.result == 'failure')andpermissions: issues: write.Example Alert Issue
Assumptions
None — requirements were fully specified.
Verification & Testing
node --test .github/scripts/nightly-failure-alert.test.js(11/11 tests passing).node --test .github/scripts/*.test.js(141/141 total repository script tests passing)../.githooks/pre-commit(Buildifier, gofmt, golangci-lint, determinism check, and addlicense all passed cleanly)../tools/analyze_diff_size.py --base-ref origin/main...(within thresholds, no split required).Related Issues / Bugs
N/A
TAG=agy
CONV=acf882de-1561-404c-ba2e-ae0940451165