Repository navigation
refactor(ci): consolidate demo workload launch and verification into unified nightly workflow - #895
Conversation
… script Update demo_workload_verify.py to verify that all WorkUnits of the demo experiment reach terminal COMPLETED state after execution, and add configurable CLI flags for timeout, queue, work unit count, title pattern, and polling interval. Add comprehensive unit tests in demo_workload_verify_test.py. TAG=agy CONV=df3c1c34-924a-42d3-81dc-9755d79b93f5
There was a problem hiding this comment.
Code Review
This pull request refactors the nightly demo workload verification script to support command-line flags for configuration (such as timeout, expected work units, and queue) and introduces a comprehensive unit test suite. The code reviewer identified several opportunities to simplify the implementation and tests. Specifically, they pointed out a redundant gRPC round-trip to re-fetch the experiment metadata and a redundant manual check for work unit completion, as the validation helper already guarantees this state. Consequently, they recommended updating the corresponding unit tests to reflect these simplifications and to test realistic failure scenarios, such as timeout exceptions.
Remove tools/nightly_tests/demo_workload_verify_test.py and its Bazel build target. TAG=agy CONV=df3c1c34-924a-42d3-81dc-9755d79b93f5
…deps in BUILD.bazel
…completion change
…ator TAG=agy CONV=df3c1c34-924a-42d3-81dc-9755d79b93f5
…unified nightly workflow TAG=agy CONV=df3c1c34-924a-42d3-81dc-9755d79b93f5
There was a problem hiding this comment.
watch out - python print("stuff") is not 100% reliable in containerized workloads.
please consider using print("stuff", flush=True) or logging package
I've updated all |
There was a problem hiding this comment.
side question - will nightly pipeline add a tag to repo at start?
No - I haven't considered it yet. We can discuss this. |
Summary
A standalone "deploy demo job" (
trigger-demo-workload.yml) provides limited value on its own and caused duplicated ~75s runner setup overhead before validation could even begin. This change consolidatestrigger-demo-workload.ymlandnightly-test-suite.ymlinto a single, unified workflow in.github/workflows/nightly-test-suite.yml.Executing launch and verification within a single job eliminates redundant runner boot and environment setup time, starts verification <1s after workload launch (enabling active polling across
STARTING->RUNNING->COMPLETEDstates), and simplifies variable passing via step outputs (steps.launch-demo-workload.outputs.xid). Additionally, configurableworkflow_dispatchinputs with boolean flags (verify_workload,report_failure) defaulting totruegive developers full control to optionally skip verification or failure alerts on manual runs.Motivation & Context
STARTINGstate..github/scripts/nightly-failure-alert.js) to track the unifiedrun-and-verify-demo-workloadjob while retaining backwards compatibility.Key Changes
.github/workflows/nightly-test-suite.ymlrun-and-verify-demo-workloadjob. Addedverify_workloadandreport_failureinputs (defaulting totrue)..github/workflows/trigger-demo-workload.yml.github/scripts/nightly-failure-alert.jstriggerOutcomeandverifyOutcometo inspect the unified job IDrun-and-verify-demo-workloadwith fallback support for legacy workflow step keys.tools/nightly_tests/demo_workload_verify.pywait_for_completion=Trueinvalidate_work_unitsto ensure complete execution verification before exit.tools/nightly_tests/xmc_entities_validator.pyExperiment does not yet match the expected state. Sleeping for {poll_interval_sec}s...) during status polling loops.Assumptions
None — requirements were fully specified.
Verification & Testing
Successful End-to-End Workflow Run
Unit Tests & Code Quality
node --test .github/scripts/nightly-failure-alert.test.js(27/27 tests passing)bazel test //tools/nightly_tests:all(all tests passing)./.githooks/pre-commit(all linters, buildifier, ruff, addlicense passing cleanly)TAG=agy
CONV=df3c1c34-924a-42d3-81dc-9755d79b93f5