Repository navigation
feat(ci): add parallel trigger-all-examples runner and fix launch scripts - #881
Conversation
There was a problem hiding this comment.
Code Review
This pull request improves endpoint parsing, environment variable fallback configuration, and work unit ID extraction across several example scripts. Feedback on these changes highlights two main issues: first, the work unit ID parsing in metrics_dashboard_job/launch.py incorrectly extracts the operation ID instead of the work unit ID, and should adopt the robust parsing logic used elsewhere; second, returning a success exit code of 0 when catching an XDashError in create_dashboard.py masks failures and should be reverted to a non-zero exit code to prevent issues in automated pipelines.
bf49503 to
1a0285c
Compare
There was a problem hiding this comment.
in the screenshot you're showing the example "ClickHouse Metrics Ingestion Experiment" as successfully done. but the Performance Dashboards are not enabled (not visible on tabs list on the left).
How did it work/run exactly? was the dashboard service deployed? if so - why the "performance dashboard" tab is not shown in UI?
@AntonTyb This is the first action that runs the launch scripts and for now does a naive check of whether the corresponding experiment/workunit was created successfully (has a More specific validation of whether all necessary resources have been created properly would be done later #878 |
…ipts Introduce parallel multi-example workload runner workflow alongside launch script fixes for workunit ID parsing, display name defaults, and dashboard endpoint resolution. TAG=agy CONV=2889d5f4-f25d-4694-93f6-70447e012e6b
- Robustly parse work unit ID from Operation name in metrics_dashboard_job - Return non-zero exit code on XDashError in create_dashboard TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…achable in create_dashboard TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…ults.js with unit tests - Move inline python aggregation logic into dedicated Node.js script .github/scripts/aggregate-example-results.js. - Add comprehensive unit test suite in .github/scripts/aggregate-example-results.test.js. - Update trigger-all-examples.yml to checkout repo and execute node .github/scripts/aggregate-example-results.js. TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…_dashboard.py TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
…ow and remove env lookup in create_dashboard.py TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
781b3a4 to
70a5e8d
Compare
this would mean that a partial fail of
|
|
Discussed offline. |
🥞 Stack Navigation
feat(examples): add requirements.txt to all example workloadsfeat(ci): add parallel trigger-all-examples runner and fix launch scripts👈feat(tools): add common verification framework and demo workload verify CLIfeat(ci,tools): add dedicated verification CLIs and e2e nightly test suite🏛️ Role in PR Stack Architecture
This PR serves as Layer 2 (Parallel Multi-Workload CI Execution) within the 4-part multi-example execution and validation stack.
xmc-nightly..github/workflows/trigger-all-examples.ymlto launch all 12 examples concurrently againstxmc-nightlyon GKE, and applies critical compatibility fixes to launcher scripts (managed_xprof_artifact_job,metrics_dashboard_job,xdash_dashboard) so all workloads execute successfully.Summary
Introduces the parallel
trigger-all-examplesGitHub Actions workflow runner (.github/workflows/trigger-all-examples.yml), matrix results aggregation script (.github/scripts/aggregate-example-results.js), unit tests, and launch script compatibility fixes across example workloads.🔗 Verified Workflow Run: Trigger All Example Workloads — Run #34948317955
Screenshot of the created experiments
Motivation & Context
Previously, PR #847 (and integrated into CI in PR #852) introduced
.github/workflows/trigger-demo-workload.ymlto trigger and launch a single demonstration workload (examples/simple_job).This PR replicates and generalizes that architecture for all 12 example workloads in parallel via GitHub Actions matrix execution:
.github/workflows/trigger-demo-workload.yml.fail-fast: false), capturing individual experiment XIDs and generating aggregated summary dashboards via a dedicated Node.js script (.github/scripts/aggregate-example-results.js).xmc-nightly.Key Changes
.github/workflows/trigger-all-examples.yml):fail-fast: false).workflow_dispatch(withexamples_filter,queue_name,timeout_minutes) andworkflow_call.$GITHUB_STEP_SUMMARY..github/scripts/aggregate-example-results.jsto parse artifact JSONs, assemble Markdown summary tables, and format outputs..github/scripts/aggregate-example-results.test.js(6 unit tests).examples/metrics_dashboard_job/launch.py: Robust WorkUnit ID extraction from operation path / resource names.examples/managed_xprof_artifact_job/launch.py: WorkUnit ID extraction from operation path and non-emptydisplay_namedefaults for artifact registration.examples/xdash_dashboard/create_dashboard.py:XMC_DASHBOARD_ENDPOINTenvironment variable resolution, protocol prefix stripping, and graceful fallback when standalone dashboard gRPC service is not exposed in CI.🛠️ Detailed Rationale for Launch Script Modifications
The following script updates were required to ensure seamless execution when triggered autonomously in CI against
xmc-nightly:examples/metrics_dashboard_job/launch.py:create_work_unitAPI returns a Long-Running Operation (experiments/<xid>/workUnits/<wid>/operations/<op_id>). Previously, the script extracted the ID usingcreated_wu.name.split('/')[-1], which inadvertently grabbed the trailing Operation ID instead of the WorkUnit ID. The logic was refactored to parseparts[3](the segment immediately followingworkUnits), ensuring the accurate WorkUnit ID is forwarded to downstream xDash dashboard builders.examples/managed_xprof_artifact_job/launch.py:parts[3]as the WorkUnit ID from the ESS Operation resource path.CreateArtifactenforces non-emptydisplay_namevalidation. A default fallback (display_name = "Managed XProf Profile") was added so artifact creation succeeds reliably during automated runs without requiring explicit CLI parameter overrides.examples/xdash_dashboard/create_dashboard.py:$XMC_DASHBOARD_ENDPOINT/$DASHBOARD_ENDPOINTwith automatic stripping of URI schemes (http://,https://) and path suffixes to construct well-formed host:port gRPC targets.xmc-nightlydeployed withenable_metrics: false), catchingXDashErrorwith a clear warning notice while still emitting the dashboard identifier prevents optional backend unavailability from failing the entire CI matrix job.Verification & Testing
./.githooks/pre-commit check(100% clean across Buildifier, gofmt, addlicense, ruff, determinism).node --test .github/scripts/aggregate-example-results.test.js(6/6 tests passing).Related Workflows & PRs
.github/workflows/trigger-demo-workload.yml(Single workload launcher introduced in PR #847 / PR #852)TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339