8000 feat(actuator): integrate component snapshot discovery and placement gating by VLukiantsev · Pull Request #802 · google/xmc · GitHub
[go: up one dir, main page]

Skip to content
< 10000 /react-partial>
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions cmd/actuator/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# XMC Actuator Daemon (`cmd/actuator`)

`actuator` is the Kubernetes execution controller daemon responsible for consuming WorkUnit tasks from Redis, synthesizing Kubernetes `JobSet` resources with Kueue queueing, enforcing execution contracts, and managing the lifecycle of distributed training jobs in XMC.

---

## Architecture Overview

```mermaid
graph TD
subgraph ControlPlane["Control Plane & Queue"]
Redis[("Redis Task Queue")]
CMS["Component Manager Service (CMS)"]
end

subgraph ActuatorDaemon["cmd/actuator"]
TaskHandler["WorkUnit Task Handler<br>(HandleWorkUnitRedisTask)"]
K8sClient["K8s Client & Synthesizer<br>(constructJobSet)"]
PlacementGate["Component Placement Gate<br>(evaluateComponentSnapshots)"]
end

subgraph Cluster["Kubernetes Cluster"]
KubeAPI["Kubernetes API / Kueue"]
CompCRD["Component CRs (xmc.dev/v1alpha1)<br>(controller-runtime Reader)"]
JobSets["JobSets & WorkUnit Pods"]
end

Redis -->|"Dequeues Tasks"| TaskHandler
TaskHandler -->|"Synthesizes JobSet"| K8sClient
K8sClient --> PlacementGate
PlacementGate -->|"Reads Component Status"| CompCRD
PlacementGate -.->|"Fail-Fast Retryable Error"| TaskHandler
K8sClient -->|"Creates JobSet"| KubeAPI
KubeAPI --> JobSets
```

---

## WorkUnit Task Processing

1. **Redis Task Ingestion**: The actuator continuously polls and dequeues WorkUnit execution payloads (`actuatortaskpb.ActuatorPayload`) from Redis.
2. **Preconditions Validation**: Verifies that required quotas, namespaces, and node preconditions are satisfied before job scheduling.
3. **JobSet Synthesis**: Compiles WorkUnit specifications into Kubernetes `JobSet` resources (`sigs.k8s.io/jobset/api/jobset/v1alpha2`).
4. **Contract Invariant Enforcement**:
- **Non-Restarting JobSets Contract**: Primary container restart policy is strictly enforced (`Never` or `OnFailure` with `MaxRestarts: 0`) to guarantee deterministic execution tracking and prevent runaway restart loops.
5. **Execution Tracking**: Subscribes to JobSet status updates and reports state transitions back to the Experiment State Server (ESS).

---

## Component Snapshot Discovery & Placement Gating

When WorkUnits depend on reusable XMC components (such as storage drivers, monitoring agents, or sidecars), the Actuator inspects and resolves component configurations during JobSet construction:

### 1. Snapshot Discovery
Actuator inspects the `xmc.dev/component-snapshots` annotation on the JobSet specification. This annotation contains a JSON map of component names to locked generations:
```json
{
"metrics-collector": 2,
"storage-driver": 1
}
```
If the annotation is absent or empty, component evaluation is bypassed and standard JobSet construction proceeds.

### 2. Controller-Runtime Component Lookup
To avoid querying Cloud Spanner directly during JobSet construction, Actuator queries `Component` custom resources (`xmc.dev/v1alpha1`) using `controller-runtime`'s `runtimeclient.Reader` (initialized as a direct client by default, with support for an injected cached reader via `SetManagerClient`):
- **Namespace Resolution**: Actuator checks the WorkUnit's target queue namespace first.
- **Cluster Fallback**: If not found in the target namespace, it falls back to the default cluster-wide `"xmc"` namespace.
- **Dynamic Client Fallback**: In test environments or when `ManagerClient` is uninitialized, falls back to `dynamicClient`.

### 3. Conditional Placement Gate
For each required component, Actuator validates operational readiness:
- **Missing Component / Hydration Lag**: If a component CR is not found in the cache, returns `apierrors.NewServiceUnavailable`. The task handler catches this as a retryable error and requeues the task with exponential backoff.
- **Disabled State**: If `spec.enabled == false` or `status.state` is `COMPONENT_STATE_DISABLED`, `COMPONENT_STATE_DISABLING`, or `COMPONENT_STATE_DELETED`, returns `apierrors.NewServiceUnavailable`.
- **Unhealthy Conditions**: If conditions report `Ready != "True"` (including missing conditions or `Ready == "False"`), or `Degraded == "True"`, placement is blocked with `apierrors.NewServiceUnavailable`.

### 4. Generation Drift Detection
If the component is healthy and enabled, but `status.observedGeneration` differs from the `lockedGeneration` in the snapshot:
- Placement is **not blocked**.
- Actuator logs a warning (`"Component generation drift detected"`).
- Actuator attaches the `xmc.dev/component-drift-warning` annotation to the synthesized JobSet.
4 changes: 4 additions & 0 deletions cmd/actuator/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ var (
metricsEnabled bool
8000 allowPrivilegedContainers bool
allowHostNetwork bool
enableComponents bool
)

// Redis flags.
Expand All @@ -79,6 +80,7 @@ func init() {
flag.BoolVar(&metricsEnabled, "metrics_enabled", false, "Whether to enable XMC metrics in jobs.")
flag.BoolVar(&allowPrivilegedContainers, "allow_privileged_containers", false, "Whether to allow containers to run in privileged mode when requested.")
flag.BoolVar(&allowHostNetwork, "allow_host_network", false, "Whether to allow pods to run with host networking when requested.")
flag.BoolVar(&enableComponents, "enable_components", false, "Whether the components feature is enabled.")
registerConfigs()
}

Expand All @@ -94,6 +96,7 @@ func registerConfigs() {
configutil.DurationEnv(&redisProcessTimeout, "redis_process_timeout", constants.RedisProcessTimeout)
configutil.DurationEnv(&redisAbandonedMessageTimeout, "redis_abandoned_message_timeout", constants.RedisAbandonedMessageTimeout)
configutil.Int64Env(&redisReadBatchSize, "redis_read_batch_size", constants.RedisReadBatchSize)
configutil.BoolEnv(&enableComponents, "enable_components", false)
}

func validate() error {
Expand Down Expand Up @@ -132,6 +135,7 @@ func Load() Config {
MetricsEnabled: metricsEnabled,
AllowPrivilegedContainers: allowPrivilegedContainers,
AllowHostNetwork: allowHostNetwork,
EnableComponents: enableComponents,
},
Audience: aud,
XManagerServiceAccount: xmanagerServiceAccount,
Expand Down
4 changes: 4 additions & 0 deletions cmd/actuator/k8s_client/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ compat_go_library(
name = "k8s_client",
srcs = [
"client.go",
"component_injection.go",
"jobset.go",
"jobset_queries.go",
"k8sclient_test_utils.go",
Expand Down Expand Up @@ -71,6 +72,7 @@ compat_go_test(
name = "k8s_client_test",
srcs = [
"client_test.go",
"component_injection_test.go",
"jobset_queries_test.go",
"jobset_test.go",
"namespace_test.go",
Expand All @@ -91,9 +93,11 @@ compat_go_test(
"@io_k8s_api//batch/v1:batch",
"@io_k8s_api//core/v1:core",
"@io_k8s_apimachinery//pkg/api/resource",
"@io_k8s_apimachinery//pkg/apis/meta/v1/unstructured",
"@io_k8s_client_go//dynamic/fake",
"@io_k8s_client_go//kubernetes/fake",
"@io_k8s_client_go//testing:go_default_library",
"@io_k8s_sigs_controller_runtime//pkg/client/fake",
"@io_k8s_sigs_jobset//api/jobset/v1alpha2",
"@org_golang_google_protobuf//types/known/timestamppb",
],
Expand Down
28 changes: 25 additions & 3 deletions cmd/actuator/k8s_client/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,9 @@ type ActuatorRuntimeConfig struct {

// Decides whether pods can run with host networking when requested.
AllowHostNetwork bool

// Decides whether the components feature is enabled.
EnableComponents bool
}

// Priorities holds experiment and work unit priorities.
Expand All @@ -76,6 +79,14 @@ type Client struct {
preconditionsValidator preconditionsvalidator.Validator
}

var inClusterConfig = rest.InClusterConfig

// buildRESTConfig constructs a Kubernetes *rest.Config using the following resolution order:
// 1. Explicit KUBECONFIG environment variable (if set).
// 2. In-cluster pod service account environment (rest.InClusterConfig).
// If running inside a cluster (KUBERNETES_SERVICE_HOST is set) but in-cluster config fails,
// fail fast with an explicit error instead of falling back to host files.
// 3. Standard local kubeconfig loading rules (e.g. ~/.kube/config).
func buildRESTConfig() (*rest.Config, error) {
if kubeconfig := os.Getenv("KUBECONFIG&q 8BD0 uot;); kubeconfig != "" {
loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
Expand All @@ -86,18 +97,18 @@ func buildRESTConfig() (*rest.Config, error) {
return config, nil
}

config, err := rest.InClusterConfig()
config, err := inClusterConfig()
if err == nil {
return config, nil
}
if os.Getenv("KUBERNETES_SERVICE_HOST") != "" {
return nil, fmt.Errorf("failed to get in-cluster config: %w", err)
return nil, fmt.Errorf("running in-cluster (KUBERNETES_SERVICE_HOST is set), but failed to load in-cluster config: %w", err)
}

loadingRules := clientcmd.NewDefaultClientConfigLoadingRules()
config, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, &clientcmd.ConfigOverrides{}).ClientConfig()
if err != nil {
return nil, fmt.Errorf("failed to get kubernetes config (in-cluster or default): %w", err)
return nil, fmt.Errorf("failed to load default kubeconfig: %w", err)
}
return config, nil
}
Expand All @@ -119,16 +130,27 @@ func New(runtimeConfig ActuatorRuntimeConfig, storage storage.Storage, spannerCl
return nil, fmt.Errorf("failed to create dynamic kubernetes client: %w", err)
}

managerClient, err := runtimeclient.New(config, runtimeclient.Options{})
if err != nil {
return nil, fmt.Errorf("failed to create controller-runtime client: %w", err)
}
Comment thread
VLukiantsev marked this conversation as resolved.
Resolved

return &Client{
clientset: clientset,
dynamicClient: dynamicClient,
ManagerClient: managerClient,
storage: storage,
spannerClient: spannerClient,
RuntimeConfig: runtimeConfig,
preconditionsValidator: preconditionsvalidator.New(),
}, nil
}

// SetManagerClient sets the controller-runtime Reader client (e.g. cached reader or test double).
func (c *Client) SetManagerClient(managerClient runtimeclient.Reader) {
c.ManagerClient = managerClient
}

// NewWithClients creates a new Kubernetes client with the given clientset and dynamic client.
func NewWithClients(clientset kubernetes.Interface, dynamicClient dynamic.Interface, runtimeConfig ActuatorRuntimeConfig, preconditionsValidator preconditionsvalidator.Validator, storage storage.Storage, spannerClient *spanner.Client) *Client {
return &Client{
Expand Down
67 changes: 66 additions & 1 deletion cmd/actuator/k8s_client/client_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ package k8sclient

import (
"context"
"errors"
"fmt"
"os"
"strings"
Expand All @@ -24,6 +25,7 @@ import (
"golang/cmp/cmp"
"golang/k8s_io/apimachinery/v/v0_23/pkg/apis/meta/v1/v1"
"golang/k8s_io/apimachinery/v/v0_23/pkg/runtime/runtime"
"golang/k8s_io/client_go/v/v0_23/rest/rest"
clientgotesting "golang/k8s_io/client_go/v/v0_23/testing/testing"
"golang/protobuf/v2/proto/proto"
"golang/protobuf/v2/testing/protocmp/protocmp"
Expand Down Expand Up @@ -572,7 +574,6 @@ func TestNew_WithKubeconfig(t *testing.T) {
t.Fatalf("os.CreateTemp failed: %v", err)
}
defer os.Remove(tmpFile.Name())

kubeconfigContent := `apiVersion: v1
kind: Config
clusters:
Expand Down Expand Up @@ -613,3 +614,67 @@ users:
}
})
}

func TestBuildRESTConfig_InClusterDetected_FailsFast(t *testing.T) {
t.Setenv("KUBECONFIG", "")
t.Setenv("KUBERNETES_SERVICE_HOST", "10.0.0.1")
t.Setenv("KUBERNETES_SERVICE_PORT", "443")

orig := inClusterConfig
inClusterConfig = func() (*rest.Config, error) {
return nil, errors.New("open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory")
}
t.Cleanup(func() { inClusterConfig = orig })

_, err := buildRESTConfig()
if err == nil {
t.Fatal("buildRESTConfig() with in-cluster environment succeeded unexpectedly, want error")
}
expectedSub := "running in-cluster (KUBERNETES_SERVICE_HOST is set), but failed to load in-cluster config"
if !strings.Contains(err.Error(), expectedSub) {
t.Errorf("buildRESTConfig() error = %q, want substring %q", err.Error(), expectedSub)
}
}

func TestBuildRESTConfig_InClusterDetected_Succeeds(t *testing.T) {
t.Setenv("KUBECONFIG", "")
t.Setenv("KUBERNETES_SERVICE_HOST", "10.0.0.1")
t.Setenv("KUBERNETES_SERVICE_PORT", "443")

expectedCfg := &rest.Config{Host: "https://10.0.0.1:443"}
orig := inClusterConfig
inClusterConfig = func() (*rest.Config, error) {
return expectedCfg, nil
}
t.Cleanup(func() { inClusterConfig = orig })

cfg, err := buildRESTConfig()
if err != nil {
t.Fatalf("buildRESTConfig() with valid in-cluster environment failed: %v", err)
}
if cfg != expectedCfg {
t.Errorf("buildRESTConfig() = %v, want %v", cfg, expectedCfg)
}
}

func TestBuildRESTConfig_DefaultFallback_FailsWhenNoConfig(t *testing.T) {
t.Setenv("KUBECONFIG", "")
t.Setenv("KUBERNETES_SERVICE_HOST", "")
t.Setenv("KUBERNETES_SERVICE_PORT", "")
t.Setenv("HOME", t.TempDir())

orig := inClusterConfig
inClusterConfig = func() (*rest.Config, error) {
return nil, rest.ErrNotInCluster
}
t.Cleanup(func() { inClusterConfig = orig })

_, err := buildRESTConfig()
if err == nil {
t.Fatal("buildRESTConfig() without any config succeeded unexpectedly, want error")
}
expectedSub := "failed to load default kubeconfig"
if !strings.Contains(err.Error(), expectedSub) {
t.Errorf("buildRESTConfig() error = %q, want substring %q", err.Error(), expectedSub)
}
}
Loading
Loading
0