8000 Releases · google/xmc · GitHub
[go: up one dir, main page]
Skip to content

Releases

Release 0.18.0

Release 0.18.0 Latest
Latest

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 06 Nov 15:07
0.18.0
b103091
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.18.0

Release Version Determination: Minor bump (0.17.0 -> 0.18.0).
Rationale: Marks XMC's launch as a public open-source repository (pre-1.0 milestone), opens the 0.18.x one-time transition window for migrating legacy umbrella-managed ingestion-gateway metrics workloads to the stable-2026a Component profile (enable_components: true), promotes Components (enable_components / componentsDeployment.enabled) to Section 2 with an end-to-end ComponentStability contract across stable-2026a (redpanda, clickhouse, and ingestion-gateway as STABLE), introduces experimental Vizier (3P) and HPO (xmc-study-controller, 1P) components (EXPERIMENTAL with acknowledgeExperimental), adds ESS database schema migration 017 (Xid queue ordering tie-breaker), and delivers Resource Observability Dashboard (/resources) enhancements alongside BYO PostgreSQL support in Cluster Toolkit.

Important

Running ingestion-gateway without Components (enable_components: false)? 0.18.x is the only release for migrating to Components. Complete the cutover in docs/migrations/components.md before upgrading to 0.19.0. See the first [Action Required] item below.

🌍 Public Open-Source Repository Launch (Pre-1.0 Milestone)

XMC is now available as a public open-source repository! While XMC remains pre-1.0 as we continue stabilizing core APIs, database backends, and component profiles toward a future 1.0 release, 0.18.0 marks our transition to open-source development with public onboarding documentation, contribution workflows, and security reporting guidelines in place.

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

  • [Action Required] One-Time Components Transition Window (0.18.x Only) & Post-Migration xmc-profile Upgrades:
    • Supported Migration Window (0.18.x Only): 0.18.x is the only supported window for migrating an existing installation from legacy umbrella-managed metrics workloads (enable_components: false, umbrella ingestion-gateway) to the stable-2026a Component profile (enable_components: true). Starting with 0.19.0, the legacy non-component metrics templates and the one-time migration tool (tools/migrations/components/migrate.py) are scheduled for removal. Operators running ingestion-gateway from the umbrella chart must complete the cutover on 0.18.x before upgrading to 0.19.0 (clusters without a live umbrella ingestion-gateway do not need to run the handoff, as handoff is a no-op for them).
    • Cutover Runbook: Follow docs/migrations/components.md (prepare -> handoff -> upgrade with components -> install xmc-profile -> finalize) (#1461, #1606). The cutover keeps the live ingestion-gateway serving throughout: Helm ownership is handed over without deleting pods, and xmc-bridge then adopts the release with a rolling update.
    • Upgrading XMC After Migration: Because xmc-profile is a separate Helm release that gcluster deploy and umbrella helm upgrade do not upgrade, on every subsequent XMC upgrade operators must bump the pinned gateway image tag in profile-values.yaml (components.ingestion-gateway.imageTag, and components.ingestion-gateway.parameters.image if set) and re-run the xmc-profile upgrade (docs/migrations/components.md section "After Migration: Upgrading XMC", #1606).
    • New Installations: Fresh installations do not need the migration tool—start directly by explicitly setting enable_components: true (componentsDeployment.enabled: true; the default is still false) and install the stable-2026a profile (deploy/profiles/stable-2026a/README.md).
  • [Action Required] Cluster Toolkit Upgrades Must Re-Compile the Blueprint: INSTALL.md upgrade Step 4 (Option A) now runs ./cluster-toolkit/gcluster create deploy/ctk_v2/xmanager-cloud.yaml -d <deployment_config_path> -o ctk-out -l WARNING -w --force before ./cluster-toolkit/gcluster deploy ctk-out/<deployment_name> --only xmc_app --auto-approve. Running gcluster deploy alone re-applies a stale ctk-out/<deployment_name> without your updated image_tag, enable_components setting, or chart templates. This applies to every Cluster Toolkit upgrade, including the Components cutover (#1606).
  • [Action Required] ESS Database Schema Migration 017 (017_work_units_queue_index_xid_tiebreaker): Added migration 017_work_units_queue_index_xid_tiebreaker for both Spanner and PostgreSQL to append Xid as a deterministic tie-breaker to the active work unit queue ordering index (#1491). Ensure schema migrations are applied before upgrading workloads: they are not run automatically by gcluster deploy. Follow INSTALL.md → Step 3: Run Database Schema Migrations (or pass --run-migrations to dev_env.py gke-upgrade in development environments).

✨ New Features & Improvements

  • Resource Observability Dashboard (/resources) Enhancements:
    • Experiment Title & Live Running Duration in Queue Jobs Table: Added the parent experiment title (#1522) and live running duration for active jobs (#1588) to the Queue Jobs side panel table.
    • Draggable Pane Resize Divider & Persisted Full-Row Expandable Jobs Toggle: Added a draggable resize divider between the queues list and side panel (#1562) and a full-row expandable jobs layout toggle backed by localStorage persistence (#1570).
    • Generic Accelerator Parent Categories in Resource Type Picker: Updated the resource type filter picker to offer generic accelerator parent categories (e.g. grouping accelerator variants) (#1526).
  • Configurable Web UI Documentation Link (XMC_DOCS_URL): Made the documentation link in the Web UI navigation header configurable via the XMC_DOCS_URL environment variable and Helm chart values (#1584).

🐛 User-Facing Bug Fixes & Stability

  • Executing-Cluster Resolution for Work Unit Logging & Dashboard Links: Updated the Reconciler and Web UI URL processor so work unit Cloud Logging and GKE dashboard links point to the actual executing cluster (properly supporting MultiKueue delegated workloads on remote clusters) (#1551).
  • Deterministic Active Work Unit Queue Ordering: Added Xid as a secondary sort tie-breaker when fetching active work units by queue across Spanner and PostgreSQL (#1491).
  • Opaque Search Syntax Help Popover: Fixed the experiment list search syntax help popover background so it renders opaquely over table content (#1513).
  • Repo-Checkout SDK Verification in install_xmanager.py: Updated deploy/cli/install_xmanager.py to assert that local Python SDKs (xmc and xdash) resolve from the repository checkout after installation (#1485).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_components] Components

  • How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml), then install the stable-2026a Component profile (deploy/profiles/stable-2026a/README.md) or migrate an existing installation using docs/migrations/components.md.
  • stable-2026a Profile Catalog & Declared Stability:
    • redpanda: STABLE
    • clickhouse: STABLE
    • ingestion-gateway: STABLE
    • vizier: EXPERIMENTAL (requires acknowledgeExperimental: true — see Section 3)
    • hpo: EXPERIMENTAL (requires acknowledgeExperimental: true — see Section 3)
  • What's New:
    • ComponentStability Contract & CRD Admission Enforcement: Added the ComponentStability contract (STABLE, EXPERIMENTAL) to components.proto and the Component CRD admission rules, declaring explicit stability tiers across the stable-2026a catalog (#1538). Note: spec.stability is now a required Component field. If you author or commit Component manifests yourself (e.g. GitOps), re-render them from the 0.18.0 stable-2026a profile so they are not rejected on apply (#1564).
    • Stability Gating & Dependency Inversion Rules in CMS, xmc-bridge, Profile Chart & Linter: Enforced acknowledgeExperimental gating and dependency stability inversion rules (preventing STABLE components from depending on EXPERIMENTAL components) at runtime in CMS and xmc-bridge (#1539) as well as statically in the stable-2026a profile chart and cmd/component-linter (#1540).
    • End-to-End Component Stability Visibility (ESS, Health Service, Admin Panel UI & xmc-diag): Surfaced component stability tiers across ESS drift detection, Health Service responses, the Admin Panel UI (/admin), and tools/xmc-diag output (#1541).
    • **One-Time Legacy Metric...
Read more
Assets 2
Loading

Release 0.17.0

Release 0.17.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 30 Oct 14:33
0.17.0
681ee75
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.17.0

Release Version Determination: Minor bump (0.16.0 -> 0.17.0).
Rationale: Promotes Log Monitor (log_monitor_deployment_enabled: true / logMonitorDeployment.enabled: true) to enabled out-of-the-box by default across Cluster Toolkit and Helm deployments, adds database schema version health checks and db-config visibility in the Admin Panel, hardens XID ACL principal normalization and fail-closed production authentication guards, raises the minimum Python version for SDKs to Python 3.11+ (xmanager >= 0.8.5.post1), delivers the Pub/Sub Metrics Ingester and BigQuery DataEngine support for Dashboard Service, and completes the experimental PostgreSQL repository layer and database-agnostic Components dbstore.

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

  • [Promoted to Default] Log Monitor GKE Deployment (log_monitor_deployment_enabled: true): The Log Monitor service is now enabled out-of-the-box by default in deploy/ctk_v2/xmanager-cloud.yaml (deploy/ctk_v2/xmc-deployment-template.yaml) and deploy/helm/values.yaml (logMonitorDeployment.enabled: true), automatically scanning Cloud Logging for workload failures and surfacing root-cause status messages on experiments and work units (#1494).
  • [Action Required / Breaking] Python 3.11+ Minimum Requirement for SDKs & xmanager >= 0.8.5.post1 Default: Raised requires-python across the xmc and xdash Python SDKs to >=3.11 (#1452), bumped the default xmanager client version requirement in deploy/cli/install_xmanager.py and all example workloads to >=0.8.5.post1 (#1400), and updated install_xmanager.py to purge shadowed wheel xdash packages so local sdks/python/xdash installs cleanly (#1482).

✨ New Features & Improvements

  • Database Configuration (db-config) & Schema Version Health Checks in Admin Panel: Added Spanner and PostgreSQL schema version verification to the health checker (#1394) and exposed the shared db-config ConfigMap in the Health Service and Admin Panel UI (/admin) (#1293, #1380).
  • Declarative Kueue Queue & MultiKueue Dispatch Configuration in Helm: Extended deploy/helm/values.yaml and kueue-dev chart templates so deployments can declaratively configure CPU/GPU queues and their MultiKueue dispatch targets (#1359).

🐛 User-Facing Bug Fixes & Stability

  • Case-Insensitive XID ACL Matching, Principal Normalization & Group Query Injection Guard: Consolidated and normalized email principals on ACL write and OAuth2 sign-in (#1447, #1453), matched XID ACL principals case-insensitively during authorization checks (#1446), and escaped group membership filter queries to prevent query injection (#1448).
  • Build-Tag Isolation & Fail-Closed Production Guards for DISABLE_AUTHENTICATION and MOCK_AUTH: Gated DISABLE_AUTHENTICATION and MOCK_AUTH behind the dev build tag, built production container images exclusively from untagged binaries, and enforced fail-closed startup in production binaries (#1443, #1444, #1451).
  • Spanner bit_reversed_xid Filter Rewrite & Thread-Safe QueryGenerator: Stopped leaking Spanner-specific bit_reversed_xid filter predicates from ESS ListExperiments and ListArtifacts by rewriting xid equality filters inside the Spanner XID repository (#1481), and removed racy mutable lastFromTable state from QueryGenerator (#1480).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_metrics & dashboard_service_deployment_enabled] Metrics Pipeline & Dashboard Service

  • How to enable:
    • Metrics Ingestion: Set enable_metrics: true in deploy/ctk_v2/xmc-deployment-template.yaml (or metrics.enabled: true in deploy/helm/values.yaml).
    • Dashboard Service: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Note: Dashboard Service requires BYOD storage devices—see the deployment README—and also requires enable_metrics / metrics.enabled to be enabled for workload metrics data to be generated).
  • What's New:
    • Pub/Sub Metrics Ingester (cmd/metrics_ingester) with ClickHouse & BigQuery Writers: Added the Pub/Sub metrics ingester service with ClickHouse batch ingestion, subscription validation, health/readiness probes, and graceful shutdown batch flushing (#1002, #1003, #1004), alongside a BigQuery schema, row normalizer, and streaming writer (#1405, #1406).
    • Dashboard Service BigQuery DataEngine, Parallel Multi-Plot Queries & Live Component/bigquery Hot-Reload: Decoupled the DataEngine provider and parallelized multi-plot queries in Dashboard Service (#1407), added BigQuery client and static DataEngine query support (#1408), supported live Component/bigquery configuration hot-reloading (#1409), and added a BigQuery golden E2E parity harness and seeder (#1410).
    • xdash Auto-Discovery of xmanager login Credentials & Resilient Metrics Addons: Updated the xdash Python SDK and nightly verifiers to automatically discover cached xmanager login credentials (#1462), and hardened ClickHouse and Redpanda metrics addon manifests to survive pod rescheduling (#1459).

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud.yaml) and Helm values.yaml.

🔹 [Flag: enable_components] Components

  • How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml).
  • What's New:
    • Database-Agnostic dbstore Component Store & Serializable Catalog Transactions: Promoted the component store to a database-agnostic pkg/components/store/dbstore package backed by storage.Storage and RunInSerializableTransaction (#1393, #1398, #1402, #1403, #1404, #1415, #1416, #1471, #1495).
    • Gateway Parameter Parity, Config Checksum Rollouts & Multi-Broker Preservation: Added gateway parameter parity, custom namespace support, and image resolution across stable-2026a component charts (#1428), rotated config checksum annotations on Helm deployments while preserving multi-broker Redpanda endpoints in ingestion-gateway (#1427), and added --profile-values support with Helm >= 3.14 enforcement in dev_env.py (#1429).

🔹 [Flag: database.experimentalPostgres] Experimental PostgreSQL Control-Plane Backend

  • How to enable: Set database.backend: postgres and database.experimentalPostgres: true in deploy/helm/values.yaml.
  • What's New:
    • Shared dbconfig Loader & Backend-Agnostic Storage Factory Across Services: Added the shared dbconfig loader, postgres-secret delivery, explicit database.experimentalPostgres Helm opt-in, and backend-agnostic storage factory across all control-plane and auxiliary services (ess, xid_service, actuator, reconciler, dashboard_service, log_monitor, rod_data_provider, health_service) (#1293, #1302, #1387, #1388, #1396, #1401, #1419, #1493).
    • Complete PostgreSQL Domain Repositories & Spanner Parity: Implemented PostgreSQL repositories and row converters for Experiments (Create, Get, Update, ListExperiments, FetchDriftedComponents), WorkUnits (read...
Read more
Loading

Release 0.16.0

Release 0.16.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 23 Oct 13:46
0.16.0
1e34438
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.16.0

Release Version Determination: Minor bump (0.15.0 -> 0.16.0).
Rationale: Introduces interactive and headless per-user CLI OAuth2 login (xmanager login) backed by ESS schema migration 016_cli_login_codes, adds /v1/ Nginx proxying in the Web UI, hardens gRPC deployment admission and OAuth2 session verification, delivers dashboard ordering and xdash IAM fallback improvements, and advances behind-the-flag capabilities for Components (cluster ownership fencing via migration 015_component_ownership_fence, cold-standby takeover, per-CR UID isolation, and least-privilege RBAC) and PostgreSQL storage parity.

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Database Migrations

  • [Action Required / Database Migrations] Spanner & PostgreSQL ESS Schema Migrations (015 & 016): This release adds ESS schema migrations 015_component_ownership_fence (ComponentOwnershipFence table for cluster epoch fencing, #1234) and 016_cli_login_codes (CLILoginCodes table for one-time CLI OAuth2 login code exchange, #1314). When upgrading an existing deployment, ensure Spanner migrations are applied (executed automatically by default during dev_env.py gke-deploy and gke-upgrade).

✨ New Features & Improvements

  • Interactive & Headless CLI OAuth2 Login (xmanager login): Added end-to-end per-user CLI OAuth2 authentication flows in the Experiment State Server (pkg/auth/oauth2), including sealed cli_login ticket cookies (#1348), loopback callback completion on /api/callback persisted via CLILoginCodes (016_cli_login_codes, #1314), one-time authorization code exchange for per-user XMC session tokens (#1315), and a headless manual confirmation page for remote/SSH sessions (#1329). Migrated all example workloads and CLI guides to xmanager login (#1324) and standardized xmanager CLI installation on pip across examples, docs, and nightly E2E (#1377).
    • Note: Requires xmanager >= 0.8.5.post1 on the client (pip install -U "xmanager>=0.8.5.post1" or re-run python3 deploy/cli/install_xmanager.py).
  • Web UI /v1/ Reverse Proxy to Experiment State Server: Configured the Web UI Nginx server (cmd/ui/nginx.conf) to proxy /v1/ API requests directly to the Experiment State Server (#1369).

🐛 User-Facing Bug Fixes & Stability

  • gRPC Deployment Admission, Session Token Hardening & Production Auth Guards: Enforced deployment-level admission checks across gRPC servers in ESS, XID Service, Dashboard Service, and Health Service (#1312), required verified email claims and normalized user identities at OAuth2 sign-in and session token parsing (#1311, #1313), and blocked disableAuthentication and mockAuth in Helm templates outside development environments (#1321).
  • Pre-XID Creator ACL Check & Default Experiment Owner: Fixed CreateExperiment in ESS to validate the caller's creator ACL before provisioning an XID from the XID Service and to default the experiment owner to the authenticated creator when omitted (#1323).
  • Long Token Wrapping in Web UI Status Messages: Fixed long unbroken tokens in experiment and work unit status messages overflowing table cells in the Web UI (#1373).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_metrics & dashboard_service_deployment_enabled] Metrics Pipeline & Dashboard Service

  • How to enable:
    • Metrics Ingestion: Set enable_metrics: true in deploy/ctk_v2/xmc-deployment-template.yaml (or metrics.enabled: true in deploy/helm/values.yaml).
    • Dashboard Service: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Note: Dashboard Service requires BYOD storage devices—see the deployment README—and also requires enable_metrics / metrics.enabled to be enabled for workload metrics data to be generated).
  • What's New:
    • Newest-First Default Ordering in ListDashboards: Fixed ListDashboards in Spanner dashboard storage to order dashboards newest-first (create_time DESC) by default when no explicit order_by clause is provided (#1298).
    • Dashboards List Table Row Border Alignment & Screenshot Test: Fixed row border alignment in the Web UI Dashboards list page (/dashboards) and added golden Playwright screenshot verification (#1306).
    • xdash Python SDK IAM Token Fallback & Metrics Addon Setup Script: Added Google Cloud IAM identity token fallback to the xdash Python SDK client and added deploy/addons/metrics/setup_metrics_addons.sh to automate ClickHouse metrics addon setup (#1305).

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud.yaml) and Helm values.yaml.

🔹 [Flag: enable_components] Components

  • How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml).
  • What's New:
    • Cluster Ownership Fence, Epoch Declaration & Cold-Standby Takeover: Added the ComponentOwnershipFence schema (015_component_ownership_fence) and DeclareClusterEpoch RPC in CMS (#1234), declared xmc-bridge cluster identity and epoch to CMS on startup (#1273), enforced cluster ownership fencing and source-object ownership across sourced intents, observations, deactivations, and CR deletion cleanup (#1233, #1235, #1238, #1240), held fenced bridges in cold standby (#1236), and added store-backed CMS ownership Prometheus gauges and DR takeover documentation (#1243, #1275).
    • Per-CR UID Isolation, Same-Cluster Recreated CR Adoption & Status Tracking: Isolated per-CR Helm actuation state by Kubernetes UID (#1239), enabled seamless adoption of recreated Component CRs on the same cluster via expected_owner_uid (#1274), recorded status.componentId and status.componentGeneration on Component CRs (#1237, #1241), reduced steady-state bridge log noise (#1244), and added disjunctive or dependency rules to component-linter (#1341).
    • Least-Privilege xmc-bridge RBAC & Modular Bridge/Store Subpackages: Scoped xmc-bridge RBAC to least-privilege workload and wrapper resources (removing componentsDeployment.rbac.clusterScope, #1242), and decomposed pkg/components/bridge (helm, statusdetector, healthprobe) and pkg/components/store (memory, spanner, ownership, dependency, lifecycle) into dedicated subpackages (#1362, #1363, #1364, #1365).

🔹 [Flag: enable_managed_mldiagnostics] Managed ML Diagnostics

  • How to enable: Set enable_managed_mldiagnostics: true in deploy/ctk_v2/xmc-deployment-template.yaml.
  • What's New:
    • Configurable MLRun Project, Location & Service Account Impersonation: Added reconciler configuration flags to override the target GCP project, location, and service account impersonation for MLRun synchronization and managed XProf sessions (#1370).

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

  • PostgreSQL Storage Engine, Query Dialect & Schema Parity: Added the PostgreSQL connection pool, DBTX transaction runner, SQLSTATE error translator, and migration runner CLI (internal/storage/postgres, #1284, #1287, #1308), implemented the PostgreSQL pausestate repository with row locking (#1357), wired pgx pool and DBTX across all PostgreS...
Read more
Loading

Release 0.15.0

Release 0.15.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 16 Oct 14:01
0.15.0
5e94f2f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.15.0

Release Version Determination: Minor bump (0.14.0 -> 0.15.0).
Rationale: Promotes Pause Scheduling (pause_scheduling_enabled: true) to enabled out-of-the-box by default across Cluster Toolkit and Helm deployments (backed by Spanner schema migration 013_pause_scheduling), adds Work Units table sorting and status filtering in the Web UI, fixes JobSet reconciler STOPPED status persistence and ROD gateway routing, expands Nightly Playwright UI E2E coverage across Metrics Dashboards, ROD, and the Admin Panel, and advances behind-the-flag capabilities for Components (CMS authentication/authorization, Workload Identity, kstatus conditions, and 014_component_intent_source migration).

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

  • [Promoted to Default] Pause Scheduling (pause_scheduling_enabled: true): Cluster-wide workload scheduling pause and resume is now enabled out-of-the-box by default in deploy/ctk_v2/xmanager-cloud.yaml (deploy/ctk_v2/xmc-deployment-template.yaml) and deploy/helm/values.yaml (featureFlags.pause_scheduling_enabled: true) (#1283).
    • Operator & User Experience: Administrators can pause and resume workload scheduling directly from the Admin Panel (/admin) or Health Service API during cluster maintenance while researchers continue submitting experiments without disruption. While paused, the Experiments List page (/experiments) displays a global scheduling-paused banner (#1217) and the Actuator records a clear SCHEDULING_PAUSED system status message on buffered work units (#1258).
    • Spanner Persistence & Broadcast Resilience: Added the Spanner PauseState and PausedTasks schema (013_pause_scheduling) and storage repositories (#1046, #1085, #1116), hardened Health Service and the Actuator broadcast listener against Redis pub/sub outages and transient Spanner read errors (#1205, #1289), and documented the end-to-end architecture and operational flows (#1264).
  • [Action Required / Database Migrations] Spanner ESS Schema Migrations (013 & 014): This release adds Spanner ESS migrations 013_pause_scheduling (PauseState and PausedTasks tables, #1085) and 014_component_intent_source (request_source column on ComponentIntents, #1232). When upgrading an existing deployment, ensure Spanner migrations are applied (executed automatically by default during dev_env.py gke-deploy and gke-upgrade).

✨ New Features & Improvements

  • Work Units Table Sorting & Status Filtering in Web UI: Users can now sort the Work Units table on the Experiment Details page (/experiments/:id) by Work Unit ID, Status, Created time, and Completed time (#1251), and filter work units by status directly from the Work Units filters menu (#1252).

🐛 User-Facing Bug Fixes & Stability

  • Work Units Table Reload on Experiment Navigation: Fixed an issue in the Web UI where navigating directly between different experiment detail routes did not reload the Work Units table for the newly selected experiment ID (#1260).
  • ROD batchGetSecondaryData Gateway Route Matching: Fixed the Envoy Gateway HTTPRoute for the Resource Overview Dashboard (rod) to match the AIP-136 custom verb /v1/clusterqueues:batchGetSecondaryData, ensuring queue velocity and predictability metrics resolve reliably through the gateway (#1212).
  • JobSet Reconciler STOPPED State Ordering & Batcher Fault Isolation: Fixed the JobSet reconciler to flush and persist the STOPPED status in Spanner before releasing the JobSet protection finalizer (#1189), added bounded backoff retries for failed batcher flushes (#1200), isolated permanent per-work-unit batcher errors via typed WorkUnitNotFoundError so missing work units never block status updates for sibling work units (#1201, #1202), and mapped Spanner NotFound / AlreadyExists errors to canonical grpcerr sentinels (#1250).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_metrics & dashboard_service_deployment_enabled] Metrics Pipeline & Dashboard Service

  • How to enable:
    • Metrics Ingestion: Set enable_metrics: true in deploy/ctk_v2/xmc-deployment-template.yaml (or metrics.enabled: true in deploy/helm/values.yaml).
    • Dashboard Service: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Note: Dashboard Service requires BYOD storage devices—see the deployment README—and also requires enable_metrics / metrics.enabled to be enabled for workload metrics data to be generated).
  • What's New:
    • Nightly Playwright UI E2E Verification for Metrics Dashboards: Added automated Playwright browser verification (metrics_dashboard_ui_verify.py) to the Nightly E2E suite, asserting that dashboards created by metrics_dashboard_job and xdash_dashboard render cleanly on /dashboards and /dashboards/:id with hydrated chart cards and zero runtime or HTTP errors (#1256).
    • Backend-Neutral Dashboard Storage Logic & Sentinel Error Mapping: Extracted dashboard chart plot construction and validation into backend-neutral storage packages (internal/storage/data) (#1196) and mapped Spanner NotFound / AlreadyExists errors in GetChart and DeleteChart to canonical grpcerr status codes (#1250).

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud.yaml) and Helm values.yaml.

🔹 [Flag: enable_components] Components

  • How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml).
  • What's New:
    • CMS Authentication, Mutating RPC Authorization & Request-Source Ownership: Enforced gRPC authentication and mutating RPC authorization on the Component Management Service (CMS) in ESS (#1219), configured per-RPC bearer token credentials and GKE Workload Identity bindings for xmc-bridge (#1247), and recorded the authenticated request source as the owner of sourced component intents (014_component_intent_source, #1232).
    • controller-runtime Bridge Architecture, Leader Election & Configurable Concurrency: Migrated xmc-bridge to a controller-runtime controller running as a single leader-elected writer (#1224, #1225), made reconcile timeout and concurrency configurable via CLI flags and Helm values (#1272), and tracked CMS sync and deactivation rechecks in memory (#1229).
    • kstatus Conditions, Finalizer Ordering & CRD Lifecycle Retention: Emitted standard kstatus Reconciling and Stalled conditions alongside decoupled Synced and Ready conditions on Component Custom Resources (#1227, #1228), attached the CMS cleanup finalizer before executing any side effects (#1226), preserved the Component CRD across Helm uninstalls and feature-flag flips (#1223), and added KinD E2E lifecycle tests covering blocked deactivation and finalizer cleanup (#1151, #1276).
    • Unified Idempotent Intent Planning & Structured ErrorInfo Rejections: Consolidated component intent writes into a single shared plan with idempotent application and a Spanner no-op fast path (#1152, #1155, #1231), and attached structured google.rpc.ErrorInfo reasons to missing-dependency, cycle, and active-dependents rejections (#1230).

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

  • **Real Upstream JobSet & Kueue Controllers in...
Read more
Loading

Release 0.14.0

Release 0.14.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 09 Oct 13:58
0.14.0
94c5b8b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.14.0

Release Version Determination: Minor bump (0.13.0 -> 0.14.0).
Rationale: Adds out-of-the-box support for NVIDIA H100 Mega 80GB (a3-megagpu-8g) GPU accelerators on GKE and an XMC Lite batch job example for existing GKE clusters, resolves JobSet finalizer patching and reconciler error reporting bugs, brings dynamic ClickHouse Component CR resolution and hot-reloading DataEngine to the production-ready Metrics & Dashboard Service pipeline, and advances behind-the-flag capabilities across Components (stable-2026a meta-chart, drift self-healing), Pause Scheduling, and GKE-hosted Log Monitor deployments. (Note: No .sql or .ddl database schema migrations were introduced in this release; migrations/ changes were internal Go compatibility refactors).

🌟 1. Public Changes (Available Out-of-the-Box)

✨ New Features & Improvements

  • NVIDIA H100 Mega 80GB (a3-megagpu-8g) GPU Accelerator Support: Researchers and cluster users can now schedule workloads targeting GKE a3-megagpu-8g node pools using H100_MEGA (nvidia-h100-mega-80gb). The Actuator automatically injects cloud.google.com/gke-accelerator: nvidia-h100-mega-80gb into the JobSet PodSpec nodeSelector, and Kueue and Resource Overview Dashboard (ROD) resource classifiers recognize and track th 8000 e accelerator (#1115).
  • XMC Lite Batch Job Example on Existing GKE Clusters: Added examples/xmc_lite_job/, providing a ready-to-run workflow for launching batch experiments directly onto an existing GKE cluster via xm_local.create_experiment and xm_local.Kubernetes—complete with local SQLite experiment state inspection (inspect_sqlite_state.py)—without deploying the full XMC control plane (#1047).

🐛 User-Facing Bug Fixes & Stability

  • Atomic JobSet Finalizer Removal & Reconciler Error Visibility: Replaced full-list finalizer merge patches in the JobSet reconciler with an atomic RFC 6902 JSON Patch (test + remove) so concurrent finalizer updates on terminating workloads are never overwritten, enabled immediate reconciliation when malformed XMC labels are corrected on update, and surfaced non-retryable label/conversion failures as reconcile.TerminalError in controller-runtime metrics (#1025).
  • Corrected Kueue LocalQueue Targeting in simple_job Example: Updated examples/simple_job documentation and CLI usage to pass Kueue LocalQueue names (cpu-queue, gpu-queue) instead of ClusterQueue names—preventing example workloads from stalling as Inadmissible—and documented required client environment variables (XMANAGER_ENDPOINT, IAP_CLIENT_ID, XMC_CLIENT_SA) (#1020).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_metrics & dashboard_service_deployment_enabled] Metrics Pipeline & Dashboard Service

  • How to enable:
    • Metrics Ingestion: Set enable_metrics: true in deploy/ctk_v2/xmc-deployment-template.yaml (or metrics.enabled: true in deploy/helm/values.yaml).
    • Dashboard Service: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Note: Dashboard Service requires BYOD storage devices—see the deployment README—and also requires enable_metrics / metrics.enabled to be enabled for workload metrics data to be generated).
  • What's New:
    • Dynamic ClickHouse Connection Resolution & Hot-Reloading DataEngine: When running with components, dashboard-service dynamically resolves ClickHouse connection coordinates and credentials from the clickhouse Component CR (falling back to static CLICKHOUSE_* environment variables when absent), and automatically rebuilds and atomically swaps the cached ClickHouse DataEngine without requiring pod restarts whenever configuration or credentials change (#1037, #1038, #1039).
    • Expanded Nightly E2E Verification for Metrics & ClickHouse Workloads: Added dedicated automated E2E test verification for clickhouse_metrics_job, metrics_job, metrics_dashboard_job, and xdash_dashboard, continuously verifying that workloads successfully stream metrics through the Ingestion Gateway into ClickHouse and that charts hydrate correctly (#1057, #1064, #1099).
    • Hermetic KinD E2E 3P ClickHouse & Redpanda Integration Suite: Added a dedicated KinD E2E test suite running real OSS ClickHouse and Redpanda containers to validate metrics pipeline integration locally and in CI (#1105).

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in deploy/ctk_v2/xmc-deployment-template.yaml (or deploy/ctk_v2/xmanager-cloud-deployment.yaml) and Helm values.yaml.

🔹 [Flag: enable_components] Components

  • How to enable: Set enable_components: true in deploy/ctk_v2/xmc-deployment-template.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml).
  • What's New:
    • Single-Release stable-2026a Profile Meta-Chart: Packaged deploy/profiles/stable-2026a as a unified Helm meta-chart (xmc-profile-stable-2026a) that renders and validates every catalog Component Custom Resource (clickhouse, redpanda, ingestion-gateway) from a single values.yaml file with strict JSON Schema validation, extracted reusable profile chart contract tests, and updated component-linter to deduplicate nested chart Component definitions (#1016, #1034, #1059).
    • Self-Healing Drifted 1P Component Releases & Serialized Teardowns: xmc-bridge now verifies that all Kubernetes objects in a deployed 1P component's Helm manifest exist on cluster, reports missing resources as HelmReleaseDrifted, automatically repairs drifted or failed releases with rate-limited exponential backoff (MaxHistory=10), and serializes async Helm upgrades with uninstalls/deactivations so removed releases are never resurrected (#1030, #1061, #1072).
    • Dynamic ClickHouse Component CR Integration: Integrated pkg/components/crreader with dashboard-service so ClickHouse connection coordinates and credentials dynamically resolve from the clickhouse Component CR with hot-reloaded DataEngine updates (see Section 2 for details) (#1037, #1038, #1039).
    • Server-Side Apply (SSA) Status Applier, 3P Contract Breach Handling & Envtest Suite: Extracted ApplyComponentStatusSSA (fieldManager: "xmc-bridge") and conflict-retried finalizer synchronization, separated ContractBroken from InvalidWrapperSpec for 3P component status probes, modularized K8sHelmManager into focused sub-packages, and added a hermetic controller-runtime envtest integration suite covering CEL schema rules and ValidatingAdmissionPolicy deletion protection (#1067, #1068, #1069, #1070, #1071, #1077, #1104, #1118).
    • Dev Environment Deployment, CTK Verification & xmc-diag Component Diagnostics: Added --enable-components support to tools/dev_env (automatically installing the stable-2026a profile and verifying Component CRD, xmc-bridge, and Component CR readiness in verify_xmc.py) and extended tools/xmc-diag with --check-components to report Kubernetes Component CR health in text and JSON diagnostic reports (#1081, #1087).

🔹 [Flag: pause_scheduling_enabled] Pause Scheduling

  • How to enable: Set pause_scheduling_enabled: true in deploy/ctk_v2/xmc-deployment-template.yaml (or featureFlags.pause_scheduling_enabled: true in deploy/helm/values.yaml).
  • What's New:
    • Immediate STOPPED Transition for Paused Work Units: Stopping a work unit in the SCHEDULING_PAUSED state via StopWorkUnit or BatchStopWorkUnits now transitions its execution state directly to STOPPED in the Experiment State Server instead of waiting for a Kubernetes JobSet deletion watch event that never fires for unscheduled paused workloads (#1080).
    • Fast-Path Actuator Cache Advancement on Task Resumption: Replayed tasks now carry resumption_run_id and resumption_time metadata so an Actuator whose local pause cache has not yet refreshed immediately transitions to unpaused and dispatches replayed workloads to GKE in...
Read more
Loading

Release 0.13.0

Release 0.13.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 02 Oct 15:26
0.13.0
b017a89
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.13.0

Release Version Determination: Minor bump (0.12.0 -> 0.13.0).
Rationale: Introduces configurable login session lifetime for operators, resolves out-of-the-box pagination and JobSet pod diagnostic bugs, expands end-to-end verification for the production-ready Metrics & Dashboard Service pipeline, and advances experimental Components and Pause Scheduling capabilities.

🌟 1. Public Changes (Available Out-of-the-Box)

✨ New Features & Improvements

  • Configurable Login Session Lifetime: Cluster operators can now configure how long authenticated user login sessions remain active via auth.sessionTtl in deploy/helm/values.yaml (or the SESSION_TTL environment variable, defaulting to 1h). Both OAuth2 tokens and browser session cookies automatically align to the configured duration (#997).

🐛 User-Facing Bug Fixes & Stability

  • Reliable Cursor Pagination & Filtering Across List RPCs: Fixed server-side Spanner SQL errors and cursor deserialization issues when paginating ListExperiments, ListWorkUnits, ListXidEntries, ListArtifacts, and ListStatusMessages with page_size, page_token, and custom order_by parameters (#983).
  • Restored JobSet Startup Pod Diagnostics: Fixed a pod lookup bug in the reconciler that prevented JobSet startup pod checks from running in production, ensuring ImagePullBackOff, ErrImagePull, and Unschedulable pod warnings are surfaced to users when workloads fail to start (#1017).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled. Their default enablement is temporarily postponed solely to align with managed solution milestones or external BYOD prerequisites.

🔹 [Flag: enable_metrics & dashboard_service_deployment_enabled] Metrics Pipeline & Dashboard Service

  • How to enable:
    • Metrics Ingestion: Set enable_metrics: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or metrics.enabled: true in deploy/helm/values.yaml).
    • Dashboard Service: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or dashboardServiceDeployment.enabled: true in deploy/helm/values.yaml). (Note: Dashboard Service requires BYOD storage devices—see the deployment README—and also requires enable_metrics / metrics.enabled to be enabled for workload metrics data to be generated).
  • What's New:
    • End-to-End ClickHouse Metrics & Dashboard Hydration Verification: Enabled the metrics pipeline and ClickHouse/Redpanda addons in the continuous nightly environment, adding automated end-to-end verification of workload execution, direct ClickHouse SQL data ingestion, and Dashboard Service chart hydration (#910, #933, #1029, #1056).
    • Flexible Node-Pool Scheduling for Redpanda Addon: Removed the hardcoded main-pool node selector from deploy/addons/metrics/redpanda/values.yaml so Redpanda schedules cleanly across standard and custom GKE node pool topologies, and pinned the documented Redpanda chart version to 5.8.3 (compatible with ClickHouse 24.3) (#933).

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in your Cluster Toolkit deployment YAML or Helm values.yaml.

🔹 [Flag: enable_components] Components

  • How to enable: Set enable_components: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or componentsDeployment.enabled: true in deploy/helm/values.yaml).
  • What's New:
    • ⚠️ [Topology Change when enable_components: true] Ingestion Gateway Moves from Umbrella Chart to 1P Component: When enable_components: true (componentsDeployment.enabled: true), enabling enable_metrics: true (ingestionGatewayDeployment.enabled: true) no longer deploys the Ingestion Gateway from the umbrella Helm chart. Instead, ingestion-gateway is now packaged as an embedded 1P native component blueprint (deploy/profiles/stable-2026a/components/ingestion-gateway) and actuated dynamically by xmc-bridge once its Component resource (along with its required redpanda and clickhouse component dependencies) is active (#969, #974).
    • Dependency-Safe Component Actuation & Deactivation: Propagated Component.spec.dependencies into the CMS dependency graph and gated both component actuation and deactivation on dependency readiness and safety checks so dependent components are not orphaned or started prematurely (#989, #991, #992).
    • Live 1P Helm Release Health & Probe Diagnostics: xmc-bridge now reports live 1P Helm release health, verifies release status on cold-start hydration, and falls back to a clear diagnostic message when a 3P readiness probe fails without detail (#990, #1014).
    • Modular Bridge Reconciler Architecture: Decomposed pkg/components/bridge/reconciler.go into focused modules covering condition and spec parsing, CMS client and prober wiring, 3P wrapper reconciliation, and deletion/deactivation lifecycle management (#1009, #1010, #1011, #1012, #1015).

🔹 [Flag: pause_scheduling_enabled] Pause Scheduling

  • How to enable: Set pause_scheduling_enabled: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or featureFlags.pause_scheduling_enabled: true in deploy/helm/values.yaml).
  • What's New:
    • Paused Work Unit Status in the UI: Fixed the actuator pause handler update mask so paused tasks persist SCHEDULING_PAUSED without failing validation, and added a dedicated orange "Scheduling Paused" status chip and status bar segment in the Web UI (#984).
    • Backlog-Throttled Resumption Replay & Lease Extension: The Experiment State Server resumption worker now throttles paused-task replay when the Redis task stream (xmc-tasks-stream) backlog exceeds threshold (500 by default) to protect Redis memory, and automatically extends its Redis message claim lease (XCLAIM) during long replay runs (#986, #1007, #1042).
    • Transactional Pause Scheduling Storage Wiring: Centralized pause state and paused task repositories in the storage layer and bound actuator and ESS pause writes to transaction-scoped repositories (#1043, #1044).

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

  • Reconciler Performance, Observability & Lifecycle Modernization: Migrated the reconciler to upstream controller-runtime APIs, consolidated all reconciler and controller-runtime Prometheus metrics onto the :8082 metrics endpoint (removing the unexposed :8081 listener), scoped the Kubernetes informer cache strictly to XMC-managed JobSets and JobSet-owned Pods to lower memory usage, bounded reconcile loops with reconciliation_timeout (2m), and registered the update batcher and kueuesync loop as managed controller-runtime runnables (#1017, #1021, #1022, #1023, #1024).
  • Storage Transaction Runner Refactoring: Introduced a Spanner transaction runner and moved work unit restart orchestration out of the storage layer into Experiment State Server handlers (#950).
  • Release Orchestration & Developer Environment Tooling: Added the end-to-end xmc_release orchestration skill and workflow watcher (dispatch_and_watch.py), and updated the GKE dev environment upgrade script to restart xmc-bridge and ingestion-gateway during rollout (#980, #988).
  • CI, Test Scoping & Linting Standards: Hardened stacked-PR test target resolution and added concurrency controls to unit-tests.yml, propagated dynamic work unit counts to nightly verification scripts, enforced Python cyclomatic complexity checks in Ruff and unused-dependency linting across the stack, and updated engineering workflow rules (#987, #1000, #1001, #1008, #1041).
  • Dependency & Security Maintenance: Remediated Dependabot security alerts for qs in the UI and bumped npm/yarn development packages (#999, [#1...
Read more
Loading

Release 0.12.0

Release 0.12.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 28 Sep 12:54
0.12.0
5bc5692
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.12.0

Release Version Determination: Minor bump (0.11.0 -> 0.12.0).
Rationale: Promotes the Resource Observability Dashboard (ROD) and Health Service / Admin Panel to default-enabled (true) out-of-the-box across all Cluster Toolkit v2 blueprints and Helm deployments, introduces a comprehensive UI design system overhaul, hardens core task queue and workload scheduling reliability, and advances the preview Pause Scheduling and Custom Components frameworks.

🌟 1. Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

  • Resource Observability Dashboard (ROD) Enabled by Default: The Resource Observability Dashboard is now deployed and active out-of-the-box across all Cluster Toolkit v2 blueprints and Helm manifests (rod_deployment_enabled: true, rodDeployment.enabled: true), giving operators immediate, zero-configuration visibility into cluster queue velocity, resource utilization, and queue backlog predictability (#981).
  • Health Service & Admin Panel Enabled by Default: The Health Service and administrative interface are now enabled out-of-the-box (health_service_deployment_enabled: true, healthServiceDeployment.enabled: true), enabling real-time platform component health inspection, readiness probing, and centralized administrative controls without custom configuration (#981).
  • Comprehensive UI Design System & Visual Refresh: Overhauled the frontend visual styling, typography, and page structure across 8 synchronized updates to deliver a modern, cohesive developer experience:
    • Collapsible Sidenav Rail: Introduced a responsive navigation rail expandable from compact 72px icon view to 240px full-width label view, with restored documentation quick-links (#953, #973).
    • Global Theming & Typography: Aligned color tokens, typography hierarchies, and status chip styling across light and dark interfaces (#952).
    • Top Toolbar & Global Search: Refreshed the global application header with standardized height, borders, and an integrated search bar (#954).
    • Experiments & Work Units Layouts: Redesigned the Experiments List container, data table borders, metadata split cards, and Work Units table row headers and ID badges (#955, #956, #957, #958).
    • Artifact Filters & Cards: Polished artifact category toggle buttons and interactive artifact cards (#959).

✨ New Features & Runtime Improvements

  • Automated Default Resource Requests for Kueue Workloads: Workloads submitted without explicit resource limits are now automatically assigned safe default resource requests, preventing submitted jobs from becoming indefinitely stuck in Kueue scheduling queues (#916).
  • Strict Task Queue Error Propagation: The Experiment State Server (ESS) now rigorously checks and propagates Redis message broker errors during work unit task enqueueing, preventing silent task drops and ensuring immediate operational visibility into queue saturation (#937).

🐛 User-Facing Bug Fixes & Stability

  • Protobuf Compatibility & Python SDK Stability: Pinned googleapis-common-protos below version 1.75.4 across example workloads and deployments, eliminating runtime descriptor conflicts and import failures in Python SDK clients (#985).
  • Dev Environment Service Synchronization: Fixed the local development orchestrator (dev_env.py) to automatically rebuild, push, and restart the rod_data_provider container upon local code changes (#994).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully verified and safe to enable in production environments. Default enablement is optional to accommodate bring-your-own-device or cloud infrastructure prerequisites.

🔹 [Flag: dashboard_service_deployment_enabled] Dashboard Service (Custom Dashboards & Visualization)

  • How to enable: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmanager-cloud.yaml (or dashboardServiceDeployment.enabled: true in Helm values.yaml).
  • Why it's opt-in: Fully production-ready and safe to enable, but kept opt-in by default as it requires bring-your-own-devices (BYOD / external storage backends such as ClickHouse for metric ingestion and analytical query processing).
  • Prerequisite Flag: Note that for the dashboard service to fully function and ingest telemetry, the metrics pipeline must also be activated by enabling enable_metrics: true in deploy/ctk_v2/xmanager-cloud.yaml (or metrics.enabled: true in Helm values.yaml) for metric data to be generated and streamed.
  • What's New:
    • Powers custom metric charts, interactive dashboard creation, and high-throughput metric aggregation for experiment workloads.

🔹 [Flag: enable_managed_mldiagnostics] Google Cloud Managed ML Diagnostics on GKE

  • How to enable: Set enable_managed_mldiagnostics: true and managed_xprof_gcs_bucket: "gs://<bucket-name>" in deploy/ctk_v2/xmanager-cloud.yaml (or managedMLDiagnostics.enabled: true in Helm values.yaml).
  • What's New:
    • Automates performance profiling and diagnostic trace capture for JAX, PyTorch, and TensorFlow workloads running on GKE TPU/GPU clusters.
    • Injects XProf profiling sidecars, enables Google Cloud Hypercompute Cluster APIs, and integrates seamlessly with Diagon profiling consoles.

🛠️ 3. Internal Development / Upcoming Features (Under Construction)

Caution

The following capabilities are under active internal development and are not ready for use, preview, or testing. They are guarded by strict feature flags (defaulting to false). Enabling them manually is unsupported and will lead to environment instability.

🔹 Pause Scheduling & Task Resumption Subsystem

  • Status: 🚧 Under Construction (Do not enable)
  • What's New:
    • Real-Time Redis Broadcast Engine: Implemented production Redis Pub/Sub publisher and consumer channels for broadcasting cluster-wide pause and resume events between microservices (#864).
    • Actuator Task Interception & Buffering: Actuator now intercepts and safely buffers pending work unit tasks in Spanner storage when scheduling is paused, preventing premature execution on drained compute nodes (#927).
    • Resumption Worker Loop & Replay Execution: Added dedicated ESS resumption workers that progressively replay buffered tasks in configurable batches upon cluster resumption (#922).
    • Replay Claim Verification & Safety Checks: Replay workers verify cluster pause state and worker ownership claims on every batch to prevent split-brain processing during rapid pause/unpause toggles (#926).
    • Active Task State Filtering: Enhanced replay filtering to skip cancelled, expired, or superseded tasks during cluster unpause (#972).

🔹 Pluggable Custom Components Framework

  • Status: 🚧 Under Construction (Do not enable)
  • What's New:
    • ClickHouse 3P Component Blueprint: Added official 3rd-party component blueprint for deploying and managing ClickHouse analytical engines (#963).
    • Simplified Wrapper Component Architecture: Made wrapperRef optional for 3P components and migrated Redpanda to the unified 3P Wrapper Component blueprint (#939, #978).
    • Ingestion Gateway CR Resolution: Ingestion Gateway now automatically resolves Redpanda Custom Resources and configures RBAC policies when enable_components=true (#948).
    • Shared Component Dependency DAG: Introduced a unified directed acyclic graph (DAG) engine for validating and resolving complex inter-component dependencies (#930).
    • Dynamic Chart Resolution & Cold-Start Hydration: Bridge reconciler supports dynamic per-component chart resolution and cold-start state hydration on controller restarts (#931).
    • Integration Testing & Linter Hardening: Added Cloud Spanner emulator drift integration suites and extended the component linter to support Helm sub-charts and CRD enums (#826, #945).

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

  • Automated Release Tooling Suite:
    • Introduced the end-to-end AI release notes generator skill and taxonomy reference (xmc_generate_release_notes) (#836).
    • Added core release publisher CLI tool with headless GitHub REST API fallback (tools/release/publish_release.py) (#834, #835).
    • Built dynamic feature flag discovery, gated subsystem classification, and automated SemVer bump calculation engines (`tools/release/extrac...
Read more
Loading

XMC Release Notes 0.11.0

XMC Release Notes 0.11.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 18 Sep 16:22
0.11.0
ba64f3d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release Notes (Week of September 14 – September 18, 2026)

📌 Release Summary & SemVer Determination

  • Analysis Window: 2026-09-14T00:00:00Z to 2026-09-18T13:48:53Z
  • Total Merged Pull Requests: 40 PRs (100% accounted for & verified)
  • Recommended Version Bump: Minor Bump (0.Y.0 / X.Y.0)
    • Reasoning: Introduces backwards-compatible user and operator capabilities (out-of-the-box work unit artifacts navigation in the UI, General Availability of Queue Predictability throughput metrics, automatic Managed XProf metadata defaulting, Actuator JobSet adoption idempotency), standardizes deployment configuration switches in CTK v2 and modern Kueue v1beta2 manifests, and lays the foundational backend architecture for upcoming Pause Scheduling preview capabilities without breaking existing APIs or data contracts.

Section 1: 🌟 Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

  • Work-Unit Artifact Exploration Enabled by Default: Researchers and developers can now immediately inspect and navigate diagnostic artifacts generated by individual work units directly on the per-experiment page without requiring manual feature flag activation (#875).
  • Resource Observability Queue Predictability Graduated to GA: Removed the temporary "Work in Progress" banner from the Predictability section on Resource Observability queue cards, delivering production-ready queue velocity and throughput predictability metrics directly to cluster operators (#921).
  • Automatic Metadata Defaulting for Managed XProf Artifacts: The Experiment Storage Service (ESS) CreateArtifact API now automatically populates display_name and run_group with sensible defaults when omitted by client training workloads, streamlining artifact registration (#867).

✨ New Features & User-Facing Improvements

  • Deterministic MLRun Identifier Generation: The reconciler now generates MLRun IDs using deterministic truncated SHA-256 hashes of parent experiment and work unit identifiers, eliminating ID collisions and race conditions during reconciler retries or failovers (#865).
  • Contextual Work Unit Profiler Tooltips: Added explanatory tooltips to the "Profile" button within the work units table to provide operators with immediate clarity regarding profiling availability and active states (#872).

🐛 User-Facing Bug Fixes & Stability

  • Actuator JobSet Creation Idempotency: Actuator now idempotently reconciles and adopts existing Kubernetes JobSet resources when encountering AlreadyExists errors (such as after network timeout retries during submission), preventing unnecessary workload execution failures (#861).
  • OCI Container Startup & Security Hardening: Granted explicit read and execute permissions to the entrypoint script and Nginx configuration inside UI OCI images, ensuring reliable container startup in non-root and security-hardened environments (#886).

Section 2: 🚀 Production-Ready Features Behind Feature Flags (Safe to Enable)

1. Cluster Toolkit (CTK) v2 Blueprint Feature Switches

  • Configuration Toggle:
    • Cluster Toolkit Blueprint: Documented and exposed formal deployment configuration switches in the CTK v2 blueprint (deploy/ctk_v2/xmanager-cloud.yaml and README.md), allowing operators to configure optional platform modules such as the Resource Observability Dashboard (#896).
  • Customer Value & Capabilities:
    • Gives platform engineers clear, declarative controls to selectively activate or deactivate peripheral observability stacks and services during initial cluster blueprint provisioning.
  • Associated PRs: #896.

2. Kueue Resource Version Standardization (v1beta2)

  • Configuration Toggle:
    • Helm Chart: Standardized all Kueue queue manifests across CPU, GPU, and TPU queues in deploy/helm/templates/kueue-dev/ to serve kueue.x-k8s.io/v1beta2 (#915).
  • Customer Value & Capabilities:
    • Aligns local and remote deployment templates with modern Kueue Custom Resource Definition storage versions, eliminating runtime object conversion overhead and future-proofing queue definitions for cluster upgrades.
  • Associated PRs: #915.

Section 3: 🚧 In-Progress Development

Warning

The following changes lay the initial backend foundation and scaffolding for the upcoming Pause Scheduling capability. These components are strictly for internal development, are currently non-functional/incomplete, and are not yet available for customer or preview enablement.

Pause Scheduling Groundwork (Work-in-Progress)

  • Status: In-development backend scaffolding (internal dev flag pause_scheduling_enabled: false default in Helm) (#850, #879).

  • Work Merged This Cycle:

    • Admin UI Scaffolding: Added draft administrative card and confirmation modal components to the Admin Panel (#889, #891).
    • Health Service Endpoints & Messaging: Added preliminary :pause/:resume endpoints and Redis notification broadcast hooks (#877, #894, #920).
    • Actuator State Handling: Added initial stream listeners and local state cache interfaces (#912).
    • ESS Resumption Interfaces: Implemented base consumer worker interfaces and persistence repository structures (#857, #876, #890).

  • Associated PRs: #850, #857, #876, #877, #879, #889, #890, #891, #894, #912, #920.

Section 4: ⚙️ Operational, Infrastructure & Engineering Velocity

⚙️ Operational & CI/CD Infrastructure Updates

  • Automated Nightly Verification & Issue Dispatch: Added scheduled nightly test workflow automation with automated GitHub Issue alerts dispatched to repository maintainers upon failure (#852, #884, #888).
  • Parallel Workload Execution in Nightly CI: Added parallel example workload execution runner and unified nightly examples E2E pipeline (#881, #909).
  • Atomic Workload Launch & Validation: Consolidated workload launch and verification into an atomic CI runner pattern, eliminating legacy standalone trigger jobs (#895, #919).
  • Nightly Dashboard Deployment & Version Tagging: Enabled dashboard service deployment in nightly build and GKE deploy workflows with run-number version tagging upfront (nightly-YYYY-MM-DD.<run_number>) (#893, #918).

🛠️ Developer Velocity & Testing Infrastructure

  • E2E Test Orchestration Hardening: Hardened E2E test orchestration foundation, headless Dex OIDC authentication helpers, Playwright authenticator, dynamic KinD binary resolution, UI screenshot diagnostics, and dev NodePort forwarding manifests (#746).
  • Unified Verification Framework: Introduced common entity verification framework, queue/state resilience assertions, reusable in-process runner harness, and exhaustive Protobuf/priority validation CLI (tools/nightly_tests/) (#862, #868, #873, #882, #901, #908).
  • Examples & Architecture Documentation:
    • Standardized requirements.txt across all example workloads and simplified programmatic XProf profiler setup (#866, #880).
    • Documented architecture and injected environment variables for XMC managed XProf integration in docs/architecture/managed_xprof.md (#871).
  • Repository Hygiene: Updated developer availability schedule in .agents/absence.yaml (#870).
Loading

XMC Release Notes 0.10.0

XMC Release Notes 0.10.0

Choose a tag to compare

View all tags
@janzak-dev janzak-dev released this 11 Sep 12:35
0.10.0
ef8c353
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release Notes (Week of September 7 – September 11, 2026)

📌 Release Summary & SemVer Determination

  • Analysis Timeframe: 2026-09-07T00:00:00Z to 2026-09-11T12:18:09Z
  • Total Merged Pull Requests: 46 PRs (100% accounted for & verified)
  • Recommended Version Bump: Minor Bump (0.Y.0 / X.Y.0)
    • Reasoning: Introduces backwards-compatible public features (ChartArtifact full-width plot rendering and automated plot specs, permanent promotion of core platform status/error tracking, resilient task retry classification), new production-ready blueprint and Helm toggles (Managed ML Diagnostics on GKE, federated JWKS/OIDC authentication, hybrid remote Kubernetes cluster actuation), and major architectural additions across preview subsystems (Custom Components).

Section 1: 🌟 Public Changes (Available Out-of-the-Box)

🚀 Major Highlights & Promoted Capabilities

  • Core Platform Guarantees Promoted to Permanent Invariants: Status history tracking, inadmissible workload detection, and startup pod error inspection have been stabilized and promoted to permanent, out-of-the-box platform guarantees with the permanent removal of their legacy deployment flags (#801).
  • Full-Width Interactive Chart & Metric Visualization: Researchers can now view wide-aspect ChartArtifact time-series graphs spanning the entire Work Unit details page, dramatically improving metric readability and visual analysis across desktop displays (#811).
  • Automated Plot Specification Rendering for Artifacts: Users no longer need to write custom frontend charting code to visualize experiment results; declaring a ChartArtifact automatically provisions and renders complete interactive plot specifications in the dashboard (#810).

✨ New Features & User-Facing Improvements

  • Live Metric Streaming Example Workload: Developers and researchers can reference an updated example job demonstrating how to stream live multi-metric training series (such as loss, accuracy, learning rate, and step time) directly into XMC dashboards (#676).
  • Multi-Worker Job Template with Automated Queue Submission: Researchers can launch multi-workunit batch jobs across Kueue resource queues using a standardized example script that automatically manages dynamic project domains and worker replicas (#847).

🐛 User-Facing Bug Fixes & Stability

  • Resilient Workload State Handling on Network Glitches: Workloads no longer erroneously fail during transient Kubernetes API timeouts or network hiccups, as retryable network errors are automatically distinguished from terminal container failures in task workers (#814).
  • Intuitive UI Action Button Tooltip: Improved interface clarity by renaming the Work Units table action button tooltip from "Run Profile" to "Profile" (#851).

Section 2: 🚀 Production-Ready Features Behind Feature Flags (Safe to Enable)

1. Google Cloud Managed ML Diagnostics on GKE

  • Configuration Toggle:
    • Cluster Toolkit Blueprint: Set enable_managed_mldiagnostics: true and managed_xprof_gcs_bucket: "gs://<bucket-name>" in deploy/ctk_v2/xmanager-cloud.yaml (#762).
    • Helm Chart: Set managedMLDiagnostics.enabled: true in deploy/helm/values.yaml.
  • Functional Capability & Operator Enablement:
    • Delivers zero-configuration performance profiling and diagnostic trace capture for JAX, PyTorch, and TensorFlow workloads running on GKE TPU/GPU clusters.
    • Automatically injects profiling sidecars into training pods and provisions required Cloud IAM bindings and GKE cluster integrations (#762).
    • Single-Click Profiling Navigation: Reconciler automatically appends tab=profiles to MLRun Diagon console URLs, landing users directly on active profiling dashboards (#791).
    • Scheduling Latency Visibility: Exports JobSet creation timestamps to MLRun workload details, allowing operators to measure queuing and scheduling delay (#822).
    • Reliable Session Discovery: Eliminates profiling session discovery race conditions on active training runs (#839).
    • Safe Metadata Updates: Preserves workload ownership labels and orchestrator details during experiment updates (#845).
    • Standardized Label Formats: Migrated system labels to snake_case for consistent search and query filtering (#819).
    • Streamlined Configuration: Cleaned up redundant environment variables and CLI overrides in favor of per-work-unit bucket paths (#840, #842).
  • Associated PRs: #762, #791, #819, #822, #839, #840, #842, #845.

2. Hybrid Remote Cluster Management & Out-of-Cluster Kubernetes

  • Configuration Toggle:
    • Helm Chart: Set remoteK8s.enabled: true and mount cluster credentials via remoteK8s.secretName: "remote-k8s-config" in deploy/helm/values.yaml (#798).
    • Dev Env CLI: Run dev_env.py kind-up --remote-k8s --remote-k8s-context=<context> (#799).
  • Functional Capability & Operator Enablement:
    • Enables hybrid deployment topologies where a lightweight XMC control plane (running in local KinD or a central management cluster) schedules, monitors, and manages training workloads across remote compute clusters without deploying redundant control planes (#798, #799).
  • Associated PRs: #798, #799.

Section 3: 🧪 In-Development / Experimental Features Behind Flags

Warning

The following capabilities are in active preview and default to disabled (false). Schemas and interfaces are subject to change prior to general availability.

Pluggable Custom Components Framework

  • Configuration Flag:
    • Cluster Toolkit: enable_components: false in deploy/ctk_v2/xmanager-cloud.yaml
    • Helm Chart: componentsDeployment.enabled: false in deploy/helm/values.yaml
    • CLI / Env: --enable_components=false / ENABLE_COMPONENTS=false
  • Functional Capability & Operator Enablement (Preview):
    • Proactive Experiment Drift Detection: Experiment State Server checks Spanner consistency views to detect divergences between live component catalogs and immutable experiment snapshots, surfacing warning annotations in the UI (#805).
    • Automated Lifecycle Management for 3rd-Party Addons: Enables declarative installation, upgrading, and uninstallation of auxiliary Helm packages directly alongside experiment workloads via an in-process Go Helm SDK runner (#806).
    • Custom Component Parameterization: Allows users to customize addon behavior by declaring and passing configurable parameters to managed components (#827).
    • Live Addon Health & Status Probing: Automatically discovers Kubernetes Custom Resource Definitions (CRDs) via JSONPath evaluation (#581, #708), executes TCP socket connectivity checks (#582), and reports aggregated health status (#792).
    • Dependency-Aware Workload Placement: Workloads requiring specific cluster services can automatically wait for dependent components to become healthy before starting (#802).
    • Dynamic Value Persistence: Persists resolved component values inside cm-<release-name> ConfigMaps for traceable release state (#735).
    • Unified Component Schema & Test Doubles: Standardized component references and mock management tools to accelerate ecosystem extension development (#766).
  • Associated PRs: #766, #792, #802, #805, #806, #827, #735, #581, #582, #708.

Section 4: ⚙️ Operational, Infrastructure & Engineering Velocity

⚙️ Operational & Infrastructure Updates

  • Automated Nightly GKE Staging & Health Validation: Continuous nightly deployment pipelines now build container images, compile Cluster Toolkit blueprints, provision GKE environments, and execute end-to-end health verification automatically every day (#844).
  • **Cloud Quota & Capacity Res...
Read more
Loading

Release 0.9.0

Release 0.9.0

Choose a tag to compare

View all tags
@VLukiantsev VLukiantsev released this 04 Sep 13:43
0.9.0
519c6e6
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

XMC Release 0.9.0

Release Version Determination: minor bump (0.8.0 -> 0.9.0).
Rationale: Pre-1.0 zero-major versioning (0.Y.Z). Includes out-of-the-box user-facing UI enhancements, user-facing security remediations, approved production-ready preview features behind flags (enable_metrics, rod_deployment_enabled, health_service_deployment_enabled), and major database schema migration capabilities.

🌟 1. Public Changes (Available Out-of-the-Box)

✨ New Features & Improvements

  • Work Units Performance Profiling: Added a dedicated Run Profile action button directly within the Work Units table (#720), enabling operators and ML engineers to trigger runtime profiling directly from the UI without navigating through external submenus.
  • Frontend Strictness & Angular Guidelines: Expanded ESLint rules for TypeScript and modern Angular signals and control flow to prevent common UI memory leaks and non-reactive patterns (#723).

🐛 User-Facing Bug Fixes & Stability

  • Frontend Dependency Security Remediations: Updated vulnerable transitive npm dependencies (js-yaml and nanoid) to resolve Dependabot security alerts in the web UI console (#694).

🚀 2. Production-Ready Features Behind Feature Flags (Safe to Enable)

The following capabilities are fully developed, verified, and safe to be enabled in staging or production environments. Their default enablement is temporarily postponed solely to align with platform release milestones.

🔹 [Flag: enable_metrics] Metrics & ClickHouse Pipeline

  • How to enable: Set enable_metrics: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or metrics.enabled: true in Helm values.yaml).
  • What's New:
    • Secret Manager Integration: ClickHouse database credentials can now be securely sourced directly from Google Cloud Secret Manager via the Secret Manager CSI driver (SecretProviderClass), replacing static Secret references (#645).
    • Automated Schema Initialization: Automated ClickHouse database schema setup and worker permission assignment upon initial provisioning (#681).

🔹 [Flag: rod_deployment_enabled] Resource Observability Dashboard (ROD)

  • How to enable: Set rod_deployment_enabled: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or rod.enabled: true in Helm values.yaml).
  • What's New:
    • Production-ready telemetry collection and hardware resource utilization ingestion pipeline across distributed cluster nodes.

🔹 [Flag: health_service_deployment_enabled] Health Service

  • How to enable: Set health_service_deployment_enabled: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or healthServiceDeployment.enabled: true in Helm values.yaml).
  • What's New:
    • Centralized health aggregator microservice actively monitoring Spanner connectivity, schema migrations, and internal daemon endpoints.

🧪 3. In-Development / Experimental Features Behind Flags

Warning

The following capabilities are under active ongoing development, experimental, and may not yet be functional. They are available for early preview and evaluation by enabling their deployment flags in your Cluster Toolkit deployment YAML or Helm values.yaml.

🔹 [Flag: enable_components] Pluggable Components for XMC

  • How to enable: Set enable_components: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or components.enabled: true in Helm values.yaml).
  • What's New:
    • Production Cloud Spanner Store Client: Implemented the complete 11-method store.Store client interface backed by Google Cloud Spanner, featuring transactional soft-delete joins, mutations, and an emulator integration test suite (#682).
    • Dependency Resolution & SemVer Algebra: Added SemVer 2.0.0 constraint algebra engine, acyclic DAG validator, and component dependency validator for multi-component workflows (#514, #515, #777).
    • CRD Schema Conformance & Platform Rules: Introduced Component Custom Resource Definition (CRD) schema validation, platform constraint checks, and cluster CRD compatibility rule evaluations (#513, #593, #624).
    • 1P Helm Values Synthesis: Developed 1P Helm values synthesis engine and manager contract for automating sub-chart configuration generation (#578).
    • Platform Health Aggregator & AIP-158 Pagination: Added component platform health aggregation and standardized AIP-158 list pagination (#659).
    • Lifecycle & Deletion Protection: Built deletion protection finalizer lifecycle handling and safe teardown engine (#604).
    • UI Component Explorer: Enhanced UI with outlined input controls and tooltips (#684), dominant resource badges in smart resource finder (#675), clickable Xprof artifact links (#751), and shared error banners (#670, #677).

🔹 [Flag: dashboard_service_deployment_enabled] Dashboard Service & xdash SDK

  • How to enable: Set dashboard_service_deployment_enabled: true in deploy/ctk_v2/xmanager-cloud-deployment.yaml (or dashboardServiceDeployment.enabled: true in Helm values.yaml).
  • What's New:
    • Chart Data Engine On-Demand Rebuild: Rebuilt the chart data engine on demand with gated service readiness (#646).
    • Transactional Updates: Merged chart and dashboard updates inside write transactions to prevent race conditions (#663).
    • Structured Logging & Diagnostics: Migrated legacy logging to structured slog format (#721) and updated HTTPRoute regex matching (#649).
    • Examples & Job Harness: Added end-to-end metrics and dashboard job examples (#649, #664).

🔹 [Flag: reconciler.monitoring.enabled] Reconciler Prometheus Monitoring

  • How to enable: Set reconciler.monitoring.enabled: true in deploy/helm/values.yaml.
  • What's New:
    • Added new configuration toggle to export reconciler MLRun RED metrics into GCP Cloud Monitoring via Google Managed Prometheus (GMP) PodMonitoring (#391).

⚙️ 4. Operational, Infrastructure & Engineering Velocity

Changes relevant to cluster operators, CI/CD, and internal architecture that do not alter the end-user interface.

🛠️ Local Development Orchestration (tools/dev_env.py)

  • Unified Development Entrypoint: Decomposed local dev orchestration into focused domain packages:
    • LocalRunner orchestrator for single-command up / down cluster lifecycle (#559).
    • KindManager for automated local KinD cluster management (#573).
    • ImageBuilder for parallel OCI container compilation and multi-image KinD loading (#686).
    • HelmDeployer for local chart rendering, release deployment, and KinD sanitization (#683, #764).
    • SpannerManager for automated local Cloud Spanner emulator provisioning and schema migration execution (#572, #619).
    • Continuous watch mode, fast service live-reload, and rollout restarts (#561, #622, #696).
    • Deployment rollout diagnostics collector (#770).

🚀 CI/CD, Build Tooling & Test Reliability

  • Stacked PR Base Correlation: Correlated stacked PR parent bases via git patch-id to eliminate false merge conflicts (#763).
  • Modular PR Review & Approval Workflows: Extracted domain modules for PR approval timelines, approval policies, range-diff parsing, and severity badge tracking (#718, #753, #758, #759, #760, #768).
  • Test Concurrency & Determinism: Accelerated E2E execution via concurrent Spanner, KinD, and Playwright optimizations (#741) and eliminated time-dependent test non-determinism (#738).
  • In-Job Scope Resolution: Optimized unit test scheduling with fast in-job Bazel test target scoping (#650).
  • Frontend Packaging: Pre-compiled Angular UI directly on host runners and packaged via single-stage static containers (#740).

🔒 Linting & Repository Governance

  • Security Scanning: Added Helm chart linting and Kubernetes manifest security scanners (#772).
  • Expanded Linter Rules: Codified additional golangci-lint revive checks (#722) and Python Ruff rules (#719) based on PR review audits.
  • Workflow Guidelines: Enforced incremental follow-up commits, prohibited review force-pushing (#679), and added harness encapsulation rules (#743).
Loading
0