8000 feat(actuator): integrate component snapshot discovery and placement gating by VLukiantsev · Pull Request #802 · google/xmc · GitHub
[go: up one dir, main page]

Skip to content

feat(actuator): integrate component snapshot discovery and placement gating - #802

Merged
VLukiantsev merged 1 commit into
mainfrom
feat/components-snapshot-discovery-gate-b548436694
Sep 8, 2026
Merged

VLukiantsev merged 1 commit into
mainfrom
feat/components-snapshot-discovery-gate-b548436694

Conversation

@VLukiantsev
@VLukiantsev VLukiantsev commented Sep 7, 2026 •
Copy link
Copy Markdown
Collaborator

Summary

Integrates component snapshot discovery, feature enablement gating, and conditional placement gating into Actuator JobSet synthesis. Actuator reads required component snapshots from the xmc.dev/component-snapshots JobSet annotation, respects cluster feature enablement (enable_components), resolves Component CRs via controller-run 8000 time cache with cluster-wide fallback, gates workload placement by failing fast on missing/unhealthy/disabled components with retryable errors, and emits drift warnings on generation mismatches without blocking placement.

Motivation & Context

WorkUnits require access to platform-managed components (such as storage drivers, monitoring agents, or credentials). Per the AI Components Architecture, Actuator must discover and validate required component readiness during JobSet synthesis without querying Cloud Spanner directly. In addition, WorkUnits must not be scheduled if required components are degraded, disabled, or undergoing initial CR hydration lag. When components are disabled cluster-wide (enable_components=false), requests requiring components fail fast immediately rather than looping indefinitely in Redis retries.

This is Layer 1 of a 3-PR stack (gh-stack):

Key Changes

  • Feature Enablement Guard:
    • Propagated enable_components: {{ .Values.componentsDeployment.enabled | quote }} to Actuator ConfigMap (deploy/helm/templates/actuator/configmap.yaml).
    • Added conditional RBAC in deploy/helm/templates/actuator/clusterrole.yaml for xmc.dev/components (get, list, watch).
    • Registered --enable_components CLI flag and enable_components config env in cmd/actuator/config.go and k8sclient.ActuatorRuntimeConfig.
    • In evaluateComponentSnapshots, fails fast with a non-retryable error if component snapshots are requested on a cluster with enable_components=false.
  • Snapshot Discovery: Defined ComponentSnapshotsAnnotation (xmc.dev/component-snapshots) and ComponentDriftWarningAnnotation (xmc.dev/component-drift-warning) in internal/kubernetes/common/constants/constants.go.
  • Cached Lookup: In cmd/actuator/k8s_client/component_injection.go, implemented getComponent using c.ManagerClient (runtimeclient.Reader) falling back to dynamicClient and the cluster-wide "xmc" namespace.
  • Placement Gate: Validates component operational readiness before JobSet creation:
    • Missing component / hydration lag: Returns apierrors.NewServiceUnavailable to trigger task requeue with backoff.
    • Disabled state (spec.enabled == false or terminal status): Returns apierrors.NewServiceUnavailable.
    • Unhealthy conditions (Ready=False/Unknown/Pending or Degraded=True): Returns apierrors.NewServiceUnavailable.
  • Drift Warning: Detects generation mismatch between snapshot lock and observedGeneration, logging a warning and appending xmc.dev/component-drift-warning without blocking placement.
  • Preparatory PR fix(actuator): unwrap and classify retryable errors in workunit task handler #814: Extracted generic task handler retry error unwrapping (isRetryableError with errors.As) into preparatory PR fix(actuator): unwrap and classify retryable errors in workunit task handler #814 to enforce strict Single Responsibility Principle between worker loop and K8s JobSet construction.
  • Documentation: Added cmd/actuator/README.md and updated pkg/components/README.md with runtime discovery and gating architecture.

Assumptions

  • Component snapshot declarations are provided in the xmc.dev/component-snapshots annotation on the JobSet specification as a JSON map of component names to locked generations.
  • In-memory controller-runtime cache is initialized via c.ManagerClient on the actuator client; fallback to dynamic client is preserved for unit tests.
  • Non-restarting JobSets contract (MaxRestarts: 0, restartPolicy: Never/OnFailure) remains strictly enforced.

Verification & Testing

  • Bazel Test Matrix: bazel test //cmd/actuator/... executed and passed 100%:
    • //cmd/actuator:actuator_test (PASSED)
    • //cmd/actuator/k8s_client:k8s_client_test (PASSED, 10/10 shards)
    • //cmd/actuator/k8s_client/preconditions_validator:preconditions_validator_test (PASSED)
    • //cmd/actuator/workunittaskhandler:workunittaskhandler_test (PASSED)
  • Exhaustive Unit Test Coverage: Added comprehensive test cases in component_injection_test.go covering healthy admission, feature disabled with/without snapshots, missing CR hydration lag, disabled CR states, unhealthy condition states (Ready=False/Unknown/Pending, Degraded=True, uninitialized/empty conditions), generation drift detection (single and multiple), locked generation <= 0, malformed annotation JSON, canceled context during fetch, and cluster fallback.
  • Diff Sizing Check: ~330 production code lines (< 400), 540 test lines, ~930 total code lines (< 1,000), and 90 doc lines (< 1,000). Dedicated test suite created in component_injection_test.go to maintain Single Responsibility Principle.
  • Linters: ./.githooks/pre-commit passed cleanly (buildifier, gofmt, golangci-lint, determinism, addlicense).

TAG=agy
CONV=a22ce4d3-ff57-4515-bf2f-e810b9aec5ce

@VLukiantsev VLukiantsev changed the title feat/components snapshot discovery gate b548436694 feat(actuator): integrate component snapshot discovery, placement gate, and plain env injection Sep 7, 2026
@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8000

Code Review

This pull request introduces runtime component snapshot injection and placement gating into the XMC Actuator daemon. It enables the actuator to parse component snapshots, query component custom resources using a cached controller-runtime client (with dynamic client and cluster-wide fallbacks), evaluate placement readiness (enablement, health, and generation drift), and inject access configurations (environment variables) into JobSet containers. Additionally, error handling in the task handler has been updated to correctly detect wrapped retryable API errors. Feedback on the changes suggests explicitly validating that the status map is present and non-nil before accessing its keys in extractObservedGeneration and similar helper functions to improve robustness and readability.

@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements component snapshot injection and placement gating within the actuator daemon, allowing JobSets to resolve and inject configurations from Component custom resources while enforcing health and enablement constraints. The review feedback highlights several critical robustness improvements for the injection logic in component_injection.go: handling json.Number in extractObservedGeneration to avoid false-positive drift warnings, converting non-string scalar values to strings in extractStringMap to prevent environment variables from being silently dropped, and requiring a Ready condition in evaluateComponentHealth to avoid scheduling workloads on uninitialized components.

@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements component snapshot injection and placement gating in the actuator daemon, allowing JobSets to consume locked component configurations and inject their environment variables into primary and init containers. It also updates the task handler's retry logic to support wrapped API errors. The review feedback highlights two critical issues: first, the controller-runtime client is initialized as a direct client rather than a cached reader, which contradicts the documentation and could overload the API server under high load; second, the isRetryableError helper should explicitly handle context.DeadlineExceeded to ensure context timeouts are correctly retried.

Comment thread cmd/actuator/k8s_client/client.go Resolved
@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements component snapshot injection and placement gating within the actuator daemon. It introduces logic to fetch Component custom resources, validate their health and enablement status, detect generation drift, and inject their access environment variables into synthesized JobSets. Additionally, the task handler is updated to support retries on service-unavailable errors. The review feedback focuses on improving robustness when parsing unstructured Kubernetes objects by replacing manual map type assertions with safer helper functions from the unstructured package, such as NestedBool, NestedString, NestedSlice, NestedFieldNoCopy, and NestedMap.

@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements component snapshot injection and placement gating in the Kubernetes actuator daemon. It enables parsing component snapshots from annotations, retrieving the corresponding Component custom resources, evaluating their health and enablement states, and injecting their access environment variables into the synthesized JobSet containers. Additionally, error handling is updated to correctly identify wrapped retryable errors. The review feedback identifies a potential issue in the health evaluation logic, suggesting that any Ready condition status other than 'True' should be treated as unhealthy to prevent uninitialized states from being incorrectly considered healthy.

@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements runtime component snapshot injection and placement gating in the XMC Actuator Daemon. It introduces the capability to resolve Component custom resources via controller-runtime, validate their health and enablement, detect generation drift, and inject access configurations (environment variables) into JobSet containers. It also updates error handling in the task handler to support wrapped retryable errors. Feedback was provided to improve the detection of wrapped network timeouts and temporary errors in isRetryableError using errors.As with interface targets.

@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces runtime component snapshot injection and placement gating in the Actuator daemon. It adds the capability to fetch Component custom resources (using controller-runtime with a cluster-wide fallback), validate their health and enablement status, detect generation drift, and inject access configurations (environment variables) into JobSet containers. Additionally, error handling in the task handler was improved to correctly unwrap and identify retryable errors. The review feedback suggests enhancing error handling within the extraction of component access configurations to ensure that malformed statuses fail fast rather than being silently ignored.

@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces component snapshot injection into the XMC Actuator daemon, allowing it to resolve component configurations from Kubernetes Component custom resources rather than querying Cloud Spanner directly. Key changes include the implementation of component discovery, placement gating, drift detection, and environment variable injection into JobSets. Additionally, the isRetryableError logic in the task handler has been expanded to support more error types, and comprehensive tests have been added to verify these new capabilities. I have no feedback to provide.

@VLukiantsev VLukiantsev changed the title feat(actuator): integrate component snapshot discovery, placement gate, and plain env injection feat(actuator): integrate component snapshot discovery and placement gating Sep 7, 2026
@VLukiantsev
VLukiantsev requested a review from AntonTyb September 8, 2026 09:17
@VLukiantsev
VLukiantsev marked this pull request as ready for review September 8, 2026 09:17
Comment thread cmd/actuator/k8s_client/client.go Resolved
AntonTyb
AntonTyb previously approved these changes Sep 8, 2026
@AntonTyb AntonTyb left a comment
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one error message nitpick

@VLukiantsev
VLukiantsev force-pushed the feat/components-snapshot-discovery-gate-b548436694 branch from 619ef5a to c631228 Compare September 8, 2026 11:07
@VLukiantsev
VLukiantsev force-pushed the feat/components-snapshot-discovery-gate-b548436694 branch from c631228 to 54a62cc Compare September 8, 2026 11:20
@VLukiantsev
VLukiantsev requested a review from AntonTyb September 8, 2026 11:41
@VLukiantsev
VLukiantsev force-pushed the feat/components-snapshot-discovery-gate-b548436694 branch from 54a62cc to 4c5aa54 Compare September 8, 2026 12:50
…gating

Integrate Component snapshot discovery and placement gating into Actuator
JobSet synthesis. Read component snapshot annotations, evaluate placement
enablement and health, detect generation drift, and gate placement using
retryable errors for hydration lag and health failures.

TAG=agy
CONV=a22ce4d3-ff57-4515-bf2f-e810b9aec5ce
@VLukiantsev
VLukiantsev force-pushed the feat/components-snapshot-discovery-gate-b548436694 branch from 4c5aa54 to 082705d Compare September 8, 2026 14:49
@VLukiantsev
VLukiantsev added this pull request to the merge queue Sep 8, 2026
Merged via the queue into main with commit 2be8986 Sep 8, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0