8000 fix(bridge): gate component deactivation on CMS dependency safety by VLukiantsev · Pull Request #992 · google/xmc · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions cmd/xmc_bridge/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@ func (m *mockCMS) CompleteDeactivation(ctx context.Context, in *componentspb.Com
return &componentspb.CompleteDeactivationResponse{}, nil
}

func (m *mockCMS) PrepareDeactivation(ctx context.Context, in *componentspb.PrepareDeactivationRequest, opts ...grpc.CallOption) (*componentspb.PrepareDeactivationResponse, error) {
return &componentspb.PrepareDeactivationResponse{}, nil
}

func TestParseConfig(t *testing.T) {
tests := []struct {
name string
Expand Down
1 change: 1 addition & 0 deletions pkg/components/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ Components cannot be deactivated or deleted arbitrarily if other active componen
#### GitOps Bridge Enforcement
The xmc-bridge enforces dependency ordering for Component CRs (see [bridge README](bridge/README.md#state-machine--lifecycle)):
- **Activation**: the bridge always registers an enabled Component's intent (including its dependency edges) with CMS first, then gates actuation. A not-yet-actuated Component whose `spec.dependencies` are missing, disabled, deleting, or not `Ready` at their current generation is held in `COMPONENT_STATE_PENDING_RECONCILIATION` (reason `DependenciesNotReady`). An already-running Component is never torn down during a dependency blip; it reports `DependenciesReady=False` and its Helm upgrades are held (a held upgrade to a new generation reports `Ready=False` reason `HelmUpgradePending`). Dependency cycles report `DependencyCycle`.
- **Deactivation**: setting `spec.enabled: false` calls `PrepareDeactivation` (blocking mode, `force = false`) before tearing down workloads. While active dependents (including gated but registered ones) exist the workload keeps running and the CR reports `DeactivationBlocked` (re-checked at most every 30s); otherwise the bridge deletes the workload and calls `CompleteDeactivation`, reporting `DeactivationIncomplete` and retrying if the handshake fails after teardown. The bridge does not wait for the recommended drain duration.

#### Draining Timeout Calculation
- `CalculateDrainTimeout(rec *ComponentRecord, cascaded []*ComponentRecord, reqTimeout *durationpb.Duration) time.Duration`
Expand Down
1 change: 1 addition & 0 deletions pkg/components/bridge/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ go_test(
embed = [":bridge"],
deps = [
"//api/protos/components/v1alpha1",
"//pkg/components/cms",
"//pkg/components/store",
"@com_github_google_go_cmp//cmp",
"@io_k8s_apimachinery//pkg/api/meta",
Expand Down
11 changes: 9 additions & 2 deletions pkg/components/bridge/README.md
8000
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,11 @@ The `Reconciler` evaluates Component CRs through a deterministic state machine:
- For 1P components, synthesizes Helm variable overrides and delegates release updates to `HelmManager`.
- For 3P wrapper components, queries `StatusDetector` to evaluate dynamic CR health and updates observed status.
- Transitions to `COMPONENT_STATE_ENABLED` upon success.
- **Disabled (`spec.enabled: false`)**: Deactivates 1P Helm variable overrides and transitions CMS state to `COMPONENT_STATE_DISABLED`.
- **Disabled (`spec.enabled: false`)**: Gated by CMS deactivation safety:
1. `PrepareDeactivation` (skipped once the CR already reports `Ready` reason `ComponentDisabled` at the current generation). If CMS returns `FailedPrecondition` because enabled components still depend on this one, the workload keeps running, `DeactivationBlocked=True` and `Ready=False`/`Degraded=False` reason `DeactivationBlocked` name the active dependents, and the pass returns without error. While blocked, CMS is re-checked at most every 30s (the `DeactivationBlocked` `lastTransitionTime` records the last check). `NotFound` (never registered) proceeds without the handshake.
2. Deletes the 1P Helm workload.
3. `CompleteDeactivation` finalizes the CMS handshake (`DISABLING` → `DISABLED`), then `UpdateObservedGeneration` reports `COMPONENT_STATE_DISABLED`. CMS RPC failures before teardown report reason `CMSDeactivationFailed` (`Degraded=True`).
4. If the handshake fails after the workload was deleted (or a later `PrepareDeactivation` is blocked or fails), `Ready=False`/`Degraded=True` reason `DeactivationIncomplete` is reported and the whole handshake is retried on the next pass (deletion is idempotent).

---

Expand All @@ -50,9 +54,10 @@ The `Reconciler` evaluates Component CRs through a deterministic state machine:
| `Ready` | `True` / `False` | Operational health and readiness of the underlying component. |
| `Degraded` | `True` / `False` | Presence of active probe failures, crash loops, or provisioning errors. |
| `DeletionBlocked` | `True` / `False` | Deletion request blocked due to active deletion protection policy. |
| `DeactivationBlocked` | `True` | `spec.enabled: false` blocked because enabled components depend on this one; the workload keeps running. |
| `DependenciesReady` | `True` / `False` | Informational, on actuated enabled components with `spec.dependencies`. `False` means Helm upgrades are held (see `HelmUpgradePending`). |

The bridge writes the full conditions list with server-side apply on every pass, so `DependenciesReady` is replaced wholesale and is absent whenever it does not apply (e.g. while a component is gated or disabled).
The bridge writes the full conditions list with server-side apply on every pass, so `DeactivationBlocked` and `DependenciesReady` are replaced wholesale and are absent whenever they do not apply (e.g. `DependenciesReady` is dropped while a component is gated or disabled).

Gating reasons (set on `Ready=False`, state `COMPONENT_STATE_PENDING_RECONCILIATION`). CMS receives the same coarse `Ready=False` status message as other unready paths; the details live on the CR conditions:

Expand All @@ -61,6 +66,8 @@ Gating reasons (set on `Ready=False`, state `COMPONENT_STATE_PENDING_RECONCILIAT
| `DependenciesNotReady` | `False` | A not-yet-actuated enabled component waits on missing, disabled, deleting, unready, stale, degraded, or CMS-unregistered `spec.dependencies`. |
| `DependencyCycle` | `True` | The component is on a `spec.dependencies` cycle; the message names the cycle path. |
| `HelmUpgradePending` | `False` | An actuated 1P component's Helm upgrade to the current generation is held until its dependencies are ready; the previous release keeps running. |
| `DeactivationBlocked` | `False` | CMS `PrepareDeactivation` rejected disabling the component because it has active dependents. |
| `DeactivationIncomplete` | `True` | The workload was deleted but the CMS deactivation handshake has not completed; retried every pass. |

---

Expand Down
124 changes: 113 additions & 11 deletions pkg/components/bridge/reconciler.go
Comment thread
VLukiantsev marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@
// Package bridge implements the GitOps Bridge Controller reconciliation loop.
package bridge

// TODO: Split this file by responsibility (conditions.go, spec_parser.go,
// wrapper_reconciler.go, lifecycle.go, reconciler.go); see google/xmc#992 review.

import (
"context"
"encoding/json"
Expand Down Expand Up @@ -45,10 +48,17 @@ const (
ConditionStatusFalse = "False"
// ReasonDependenciesNotReady marks an enabled component waiting on unready spec.dependencies.
ReasonDependenciesNotReady = "DependenciesNotReady"
// ReasonDeactivationBlocked marks a disabled component kept running because enabled components depend on it.
ReasonDeactivationBlocked = "DeactivationBlocked"
// ReasonDeactivationIncomplete marks a disabled component whose workload was removed before the CMS handshake completed.
ReasonDeactivationIncomplete = "DeactivationIncomplete"
// ReasonDependencyCycle marks an enabled component that is part of a spec.dependencies cycle.
ReasonDependencyCycle = "DependencyCycle"
// ConditionTypeDependenciesReady is informational on actuated components with dependencies; it does not affect Ready.
ConditionTypeDependenciesReady = "DependenciesReady"
reasonComponentDisabled = "ComponentDisabled"
// deactivationRecheckInterval throttles PrepareDeactivation while deactivation is blocked.
deactivationRecheckInterval = 30 * time.Second
// dependencyNotFoundMessagePrefix mirrors store.DependencyNotFoundMessagePrefix (pinned by a test).
dependencyNotFoundMessagePrefix = "dependency not found:"
)
Expand Down Expand Up @@ -85,6 +95,10 @@ type CMSClient interface {
UpdateObservedGeneration(ctx context.Context, in *componentspb.UpdateObservedGenerationRequest, opts ...grpc.CallOption) (*componentspb.UpdateObservedGenerationResponse, error)
// ExecuteDeletion executes component deletion in the CMS backend.
ExecuteDeletion(ctx context.Context, in *componentspb.ExecuteDeletionRequest, opts ...grpc.CallOption) (*componentspb.ExecuteDeletionResponse, error)
// PrepareDeactivation validates deactivation safety (FailedPrecondition on active dependents) and marks the component DISABLING.
PrepareDeactivation(ctx context.Context, in *componentspb.PrepareDeactivationRequest, opts ...grpc.CallOption) (*componentspb.PrepareDeactivationResponse, error)
// CompleteDeactivation transitions a DISABLING component to DISABLED.
CompleteDeactivation(ctx context.Context, in *componentspb.CompleteDeactivationRequest, opts ...grpc.CallOption) (*componentspb.CompleteDeactivationResponse, error)
}

// StatusDetectorHealthProber defines the status detection and active health checking interface for 3P custom resources.
Expand Down Expand Up @@ -229,6 +243,12 @@ func (a *serverClientAdapter) UpdateObservedGeneration(ctx context.Context, in *
func (a *serverClientAdapter) ExecuteDeletion(ctx context.Context, in *componentspb.ExecuteDeletionRequest, opts ...grpc.CallOption) (*componentspb.ExecuteDeletionResponse, error) {
return a.server.ExecuteDeletion(ctx, in)
}
func (a *serverClientAdapter) PrepareDeactivation(ctx context.Context, in *componentspb.PrepareDeactivationRequest, opts ...grpc.CallOption) (*componentspb.PrepareDeactivationResponse, error) {
return a.server.PrepareDeactivation(ctx, in)
}
func (a *serverClientAdapter) CompleteDeactivation(ctx context.Context, in *componentspb.CompleteDeactivationRequest, opts ...grpc.CallOption) (*componentspb.CompleteDeactivationResponse, error) {
return a.server.CompleteDeactivation(ctx, in)
}

// Condition represents a standardized Kubernetes custom resource condition.
type Condition struct {
Expand Down Expand Up @@ -318,7 +338,8 @@ type ReconcileComponentInput struct {
// 1. Validates input name and spec.
// 2. If IsDeleting, delegates to reconcileDeletion (gated by DeletionProtection).
// 3. Attaches deletion protection finalizer if not already present.
// 4. If !Enabled, synchronizes intent via applyIntent and handles deactivation via reconcileDisabled.
// 4. If !Enabled, synchronizes intent via applyIntent and handles deactivation via
// reconcileDisabled (gated by CMS deactivation safety).
// 5. Otherwise delegates to reconcileEnabled (dependency gating, intent, actuation, health).
func (r *Reconciler) ReconcileComponent(ctx context.Context, in ReconcileComponentInput) (*ReconcileResult, error) {
if in.Name == "" || in.Spec == nil {
Expand Down Expand Up @@ -514,25 +535,101 @@ func (r *Reconciler) applyIntent(ctx context.Context, in ReconcileComponentInput
return nil
}

// reconcileDisabled disables 1P components in Helm when deactivated (enabled: false),
// sets state to COMPONENT_STATE_DISABLED, sets conditions to Ready=False, Degraded=False,
// and updates the observed generation in CMS.
// reconcileDisabled deactivates a component (enabled: false). PrepareDeactivation blocks while
// active dependents exist (the workload keeps running; re-checked every deactivationRecheckInterval).
// Otherwise the 1P workload is deleted, CompleteDeactivation finalizes the handshake, and the
// component reports COMPONENT_STATE_DISABLED. Failures after teardown report DeactivationIncomplete.
func (r *Reconciler) reconcileDisabled(ctx context.Context, in ReconcileComponentInput, canonicalName string, res *ReconcileResult) (*ReconcileResult, error) {
if msg, ok := recentDeactivationBlock(in); ok {
return r.reportDeactivationBlocked(ctx, in, canonicalName, res, msg, false)
}
prepared, err := r.prepareDeactivation(ctx, in, canonicalName)
if status.Code(err) == codes.FailedPrecondition {
msg := fmt.Sprintf("Cannot disable %q: %s", in.Name, status.Convert(err).Message())
logging.Info(ctx, "Component deactivation blocked by active dependents", "name", canonicalName, "generation", in.Generation, "detail", msg)
return r.reportDeactivationBlocked(ctx, in, canonicalName, res, msg, true)
Comment thread
VLukiantsev marked this conversation as resolved.
Resolved
}
if err != nil {
err = fmt.Errorf("failed to prepare deactivation in CMS: %w", err)
markDeactivationFailed(in, res, "CMSDeactivationFailed", err.Error())
return res, err
}
if in.Spec.GetType() == componentspb.ComponentType_COMPONENT_TYPE_CUSTOM_1P_COMPONENT {
if err := r.deleteDisabledHelmComponent(ctx, in, res); err != nil {
return res, err
}
}
if prepared {
if _, err := r.cmsClient.CompleteDeactivation(ctx, &componentspb.CompleteDeactivationRequest{Name: canonicalName}); err != nil {
err = fmt.Errorf("failed to complete deactivation in CMS: %w", err)
markDeactivationFailed(in, res, ReasonDeactivationIncomplete, err.Error())
return res, err
}
}
res.State = componentspb.ComponentState_COMPONENT_STATE_DISABLED
setCondition(&res.Conditions, in.ExistingConditions, ConditionTypeReady, ConditionStatusFalse, in.Generation, "ComponentDisabled", "Component is disabled")
setCondition(&res.Conditions, in.ExistingConditions, ConditionTypeDegraded, ConditionStatusFalse, in.Generation, "ComponentDisabled", "Component is disabled")
setCondition(&res.Conditions, in.ExistingConditions, ConditionTypeReady, ConditionStatusFalse, in.Generation, reasonComponentDisabled, "Component is disabled")
setCondition(&res.Conditions, in.ExistingConditions, ConditionTypeDegraded, ConditionStatusFalse, in.Generation, reasonComponentDisabled, "Component is disabled")
if err := r.updateObservedCMS(ctx, canonicalName, in.Generation, res.State, "Ready=False"); err != nil {
return res, fmt.Errorf("failed to update observed generation in CMS: %w", err)
}
logging.Info(ctx, "Component disabled", "name", canonicalName, "generation", in.Generation)
return res, nil
}

// prepareDeactivation runs the CMS safety check unless already disabled at this generation. It returns
// true when CompleteDeactivation must follow; a CMS NotFound means there is nothing to guard.
func (r *Reconciler) prepareDeactivation(ctx context.Context, in ReconcileComponentInput, canonicalName string) (bool, error) {
if ready := findCondition(in.ExistingConditions, ConditionTypeReady); ready != nil && ready.Reason == reasonComponentDisabled && ready.ObservedGeneration == in.Generation {
return false, nil
}
_, err := r.cmsClient.PrepareDeactivation(ctx, &componentspb.PrepareDeactivationRequest{Name: canonicalName})
if status.Code(err) == codes.NotFound {
return false, nil
}
return err == nil, err
}

// recentDeactivationBlock returns the DeactivationBlocked message if CMS rejected deactivation at
// this generation less than deactivationRecheckInterval ago (its LastTransitionTime is the last check).
func recentDeactivationBlock(in ReconcileComponentInput) (string, bool) {
c := findCondition(in.ExistingConditions, ConditionTypeDeactivationBlocked)
if c == nil || c.Status != ConditionStatusTrue || c.ObservedGeneration != in.Generation || time.Since(c.LastTransitionTime) >= deactivationRecheckInterval {
return "", false
}
return c.Message, true
}

// reportDeactivationBlocked reports DeactivationBlocked=True with msg; checked marks a fresh CMS
// check, restarting the recheck interval. Ready keeps DeactivationIncomplete if the workload is gone.
func (r *Reconciler) reportDeactivationBlocked(ctx context.Context, in ReconcileComponentInput, canonicalName string, res *ReconcileResult, msg string, checked bool) (*ReconcileResult, error) {
setCondition(&res.Conditions, in.ExistingConditions, ConditionTypeDeactivationBlocked, ConditionStatusTrue, in.Generation, ReasonDeactivationBlocked, msg)
if cond := findCondition(res.Conditions, ConditionTypeDeactivationBlocked); checked && cond != nil {
cond.LastTransitionTime = time.Now().UTC()
}
if wasTornDown(in) {
markDeactivationFailed(in, res, ReasonDeactivationIncomplete, "workload already removed; "+msg)
return res, nil
}
return r.reportGated(ctx, in, canonicalName, res, ReasonDeactivationBlocked, msg, ConditionStatusFalse)
}

// wasTornDown reports whether a previous pass removed the workload without completing deactivation.
func wasTornDown(in ReconcileComponentInput) bool {
ready := findCondition(in.ExistingConditions, ConditionTypeReady)
return ready != nil && ready.Reason == ReasonDeactivationIncomplete
}

// markDeactivationFailed records Ready=False, Degraded=True with reason, or DeactivationIncomplete
// once an earlier pass removed the workload (the handshake is retried every pass).
func markDeactivationFailed(in ReconcileComponentInput, res *ReconcileResult, reason, msg string) {
if wasTornDown(in) {
reason = ReasonDeactivationIncomplete
}
res.State = componentspb.ComponentState_COMPONENT_STATE_PENDING_RECONCILIATION
setCondition(&res.Conditions, in.ExistingConditions, ConditionTypeReady, ConditionStatusFalse, in.Generation, reason, msg)
setCondition(&res.Conditions, in.ExistingConditions, ConditionTypeDegraded, ConditionStatusTrue, in.Generation, reason, msg)
}

// unreadyDependenciesMessage returns "waiting on dependencies: ..." naming every unready dependency,
// or "" when activation may proceed (disabled, no dependencies, or no resolver configured).
func unreadyDependenciesMessage(ctx context.Context, in ReconcileComponentInput) string {
Expand Down Expand Up @@ -654,13 +751,18 @@ func (r *Reconciler) ensure1PComponent(ctx context.Context, in ReconcileComponen
return false, nil
}

// wasComponentPreviouslyActive infers from the last Ready condition whether a workload may exist;
// gated (DependenciesNotReady, DependencyCycle) or disabled components never had one actuated.
func wasComponentPreviouslyActive(existing []Condition) bool {
for _, c := range existing {
if c.Type == ConditionTypeReady {
return c.Status == ConditionStatusTrue || (c.Reason != "" && c.Reason != "ComponentDisabled")
}
ready := findCondition(existing, ConditionTypeReady)
if ready == nil {
return false
}
return false
switch ready.Reason {
case "", reasonComponentDisabled, ReasonDependenciesNotReady, ReasonDependencyCycle:
return ready.Status == ConditionStatusTrue
}
return true
}

func isReadyAtGeneration(existing []Condition, generation int64) bool {
Expand Down
Loading
Loading
0