Repository navigation
feat(tools): add common verification framework and demo workload verify CLI - #882
Merged
Merged
Conversation
This was referenced Sep 15, 2026
There was a problem hiding this comment.
Code Review
This pull request refactors the XMC nightly tests by introducing a shared common.py module containing standard flags, gRPC channel lifecycle managers, and logging helpers. The demo_workload_verify.py script is updated to leverage these common utilities, and the Bazel build configuration is adjusted accordingly. The review feedback highlights a potential parsing bug with IPv6 addresses in resolve_dashboard_endpoint, suggests restoring the required status of the xid flag to prevent tracebacks, and recommends using the new common.get_flag helper consistently instead of getattr.
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 08:00
d50bc58 to
5c3803d
Compare
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 08:06
5c3803d to
0ce79f0
Compare
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 08:12
0ce79f0 to
e9bfb06
Compare
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 08:18
e9bfb06 to
ffb9c5a
Compare
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 08:26
ffb9c5a to
5c9a4c1
Compare
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 08:41
5c9a4c1 to
e511742
Compare
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 09:28
e511742 to
d2a6ea7
Compare
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 09:36
d2a6ea7 to
65da709
Compare
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 12:59
25184e0 to
aa4d8db
Compare
janzak-dev
temporarily deployed
to
xmc-nightly
September 15, 2026 14:19 — with 
GitHub Actions
Inactive
janzak-dev
had a problem deploying
to
xmc-nightly
September 15, 2026 14:20 — with
GitHub Actions
Failure
janzak-dev
temporarily deployed
to
xmc-nightly
September 15, 2026 14:28 — with
GitHub Actions
Inactive
janzak-dev
had a problem deploying
to
xmc-nightly
September 15, 2026 14:30 — with
GitHub Actions
Failure
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 14:52
120d09b to
99f2abf
Compare
janzak-dev
temporarily deployed
to
xmc-nightly
September 15, 2026 14:55 — with
GitHub Actions
Inactive
janzak-dev
temporarily deployed
to
xmc-nightly
September 15, 2026 14:57 — with
GitHub Actions
Inactive
…fy CLI Extract common CLI flags, gRPC channel context managers, error handlers, and logging helpers to common.py and update demo_workload_verify.py to use the shared verification abstractions. TAG=agy CONV=2889d5f4-f25d-4694-93f6-70447e012e6b
- Use urllib.parse.urlparse in get_dashboard_url for robust host and port parsing - Mark xid flag as required if common.flags is present - Use common.get_flag to safely access FLAGS in demo_workload_verify TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…s and add common_test.py TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…oad_verify - Add repository root sys.path bootstrap to demo_workload_verify.py - Ensure PYTHONPATH is exported in nightly-test-suite.yml for defense-in-depth TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…rify TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…te workflow TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…kload_verify TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
force-pushed
the
feat/nightly-common-validation
branch
from
September 15, 2026 16:01
99f2abf to
59b3a47
Compare
TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
temporarily deployed
to
xmc-nightly
September 15, 2026 16:08 — with
GitHub Actions
Inactive
janzak-dev
temporarily deployed
to
xmc-nightly
September 15, 2026 16:09 — with
GitHub Actions
Inactive
janzak-dev
marked this pull request as ready for review
September 15, 2026 16:13
pachopepe
approved these changes
Sep 16, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🥞 Stack Navigation
feat(examples): add requirements.txt to all example workloads(Merged ✅)feat(ci): add parallel trigger-all-examples runner and fix launch scripts(Merged ✅)feat(tools): add common verification framework and demo workload verify CLI👈feat(ci,tools): add dedicated verification CLIs and e2e nightly test suite🏛️ Role in PR Stack Architecture
This PR serves as Layer 3 (Shared Verification Framework & Baseline Test) within the 4-part multi-example execution and validation stack.
xmc-nightly.tools/nightly_tests/common.py(centralized flag management, authenticated gRPC stub/channel context managers, REST client helpers, and error handling) and refactorsdemo_workload_verify.pyas the baseline verifier forsimple_job.Summary
Introduces the shared verification module
common.pyalongside a hermetic test suitecommon_test.py, refactorsdemo_workload_verify.pyto use clean top-level module imports, and configures.github/workflows/nightly-test-suite.ymlfor standardpython3 -m <module>execution.Successful run
https://github.com/google/xmc/actions/runs/34993099183
Motivation & Context
Extracting common verification logic reduces duplication across workload verifier scripts and standardizes banner logging, command-line arguments, channel lifecycle management, and error handling for the entire suite of verifiers built in Layer 4.
Key Changes
tools/nightly_tests/common.py: Shared verification helpers providing flag definitions, endpoint resolvers, channel contexts, REST client helpers, and banner formatters.tools/nightly_tests/common_test.py: Added 13 hermetic unit tests covering flag parsing, URL resolution, banner generation, and error handling.tools/nightly_tests/demo_workload_verify.py: Refactored to usecommon.pyas the baseline test verifier with clean top-level imports (removingsys.path.insertand defensivetry/exceptworkarounds).tools/nightly_tests/BUILD.bazel: Definedcommonpy_library,common_testpy_test, and updateddemo_workload_verifytarget dependencies..github/workflows/nightly-test-suite.yml: Updated validation step to execute via"${GITHUB_WORKSPACE}/.venv/bin/python3" -m tools.nightly_tests.demo_workload_verify.Assumptions
None — requirements were fully specified.
Verification & Testing
bazel test //tools/nightly_tests:all(all tests passing 100%)../.githooks/pre-commit check(100% compliance).Demo Workload(experiments/12642)cpu-queueWorkUnits verified asCOMPLETED.Related Issues / Bugs
N/A
TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339