8000 fix(bridge): report real 1P helm release health and verify release on cold-start hydration by VLukiantsev · Pull Request #990 · google/xmc · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions pkg/components/bridge/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ go_test(
"@io_k8s_apimachinery//pkg/apis/meta/v1/unstructured",
"@io_k8s_apimachinery//pkg/runtime",
"@io_k8s_apimachinery//pkg/runtime/schema",
"@io_k8s_cli_runtime//pkg/genericclioptions",
"@io_k8s_client_go//dynamic/fake",
"@io_k8s_client_go//kubernetes/fake",
"@io_k8s_client_go//restmapper",
Expand All @@ -94,5 +95,6 @@ go_test(
"@org_golang_google_protobuf//types/known/durationpb",
"@sh_helm_helm_v3//pkg/chart",
"@sh_helm_helm_v3//pkg/release",
"@sh_helm_helm_v3//pkg/storage/driver",
],
)
26 changes: 24 additions & 2 deletions pkg/components/bridge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,8 @@ First-party (1P) components are managed natively through XMC Helm charts. Rather
- `RecordActuation(ref ComponentRef, generation int64)`
- `HydrateFromCluster(ctx context.Context) error`
- `HydrateComponentState(ctx context.Context, ref ComponentRef, observedGeneration int64, crReady bool) error`
- **`FakeHelmManager`**: In-memory test double supporting concurrent testing, state hydration assertions, and error injection.
- `ComponentHealth(ctx context.Context, ref ComponentRef) (healthy bool, reason, message string)`
- **`FakeHelmManager`**: In-memory test double supporting concurrent testing, state hydration assertions, and error injection (`EnsureErr`, `DeleteErr`, and `UnhealthyReason` for `ComponentHealth`, which defaults to healthy).

---

Expand All @@ -118,6 +119,27 @@ First-party (1P) components are managed natively through XMC Helm charts. Rather
- **Cold-Start State Hydration (`HydrateFromCluster` & `HydrateComponentState`)**:
- `HydrateFromCluster(ctx)` inspects the `xmc-component-values` ConfigMap in the target namespace on controller startup and restores `activeState` for all previously enabled components.
- `HydrateComponentState(ctx, ref, observedGeneration, crReady)` lazily reconciles in-memory `activeState` and `actuatedGenerations` from the `xmc-component-values` ConfigMap and the `Component` CR's `.status` (`observedGeneration` and `Ready=True` condition). This prevents redundant Helm release revision increments on cold start and ensures post-restart deactivation (`spec.enabled: false`) properly triggers `Delete1PComponent`.
- **Release Verification**: Because the values ConfigMap is persisted *before* the (asynchronous) Helm upgrade runs, neither the ConfigMap nor a `Ready=True` condition alone proves that the release exists. When hydration would record a new actuation (`crReady && observedGeneration > 0` with no actuation recorded yet), `HydrateComponentState` first calls `HelmActionRunner.Status` (outside the state mutex). The CR is distrusted only on positive evidence: `driver.ErrReleaseNotFound` or a terminal non-`deployed` status (e.g. `failed`, `superseded`, `uninstalled`). In that case the component is marked inactive and not actuated, so the next `Ensure1PCo 8000 mponent` re-installs it. Transient status errors trust the CR (and are logged) so an API blip during restart does not cause a revision bump. A `pending-install`/`pending-upgrade`/`pending-rollback` release left by a crashed bridge also trusts the CR, because a re-upgrade would only fail with "another operation is in progress"; `ComponentHealth` reports it as `HelmReleaseStuck` instead. The deactivation path (`observedGeneration == 0`) is unchanged and performs no release lookup.
- **1P Component Health (`ComponentHealth`)**:
- `evaluateHealth` in `reconciler.go` derives 1P health from `HelmManager.ComponentHealth` whenever a `HelmManager` is configured (the `BridgeProber` fallback, which reports non-3P components as healthy, is used only without a `HelmManager`).
- **`deployed` means applied, not running**: upgrades run without `--wait`, so a `deployed` release means Helm applied the manifests. It does not mean the pods are ready. Pod-level readiness is out of scope for this check.
- **In-flight upgrades**: `K8sHelmManager` tracks async upgrades per component from enqueue until the worker finishes. While one is queued or running, `Status` would still report the *previous* `deployed` revision, so `ComponentHealth` returns `HelmUpgradePending` instead. An upgrade error takes precedence over pending so `Degraded` stays stable across retries.
- **Throttling**: a healthy (`deployed`) verdict is cached per component for 1 minute (`healthCacheTTL`) to avoid a Helm `Status` call (which lists release Secrets) on every poll. Any upgrade enqueue or completion invalidates the cache. The upgrade-error and pending checks never use the cache.
- Checks are evaluated in order. Reason constants are exported from `reconciler.go`, and the reason below is set on both `Ready` and `Degraded`. Every non-healthy result sets `Ready=False` and state `COMPONENT_STATE_PENDING_RECONCILIATION`:

| Reason | Degraded | Meaning |
| :--- | :--- | :--- |
| `InvalidComponentRef` | `True` | The component reference is missing a name or namespace. |
| `HelmUpgradeFailed` | `True` | The last (sync or async) Helm upgrade for the component failed. |
| `HelmUpgradePending` | `False` | An async upgrade is queued or running; release status is not yet authoritative. |
| `HelmRunnerUninitialized` | `True` | No `HelmActionRunner` is configured, so release state cannot be verified. |
| `HelmReleaseNotFound` | `True` | The `xmc-cmp-<name>` release does not exist (`driver.ErrReleaseNotFound`). |
| `HelmStatusUnavailable` | *unchanged* | The Helm status query failed transiently (e.g. API server unreachable). The previous `Ready`/`Degraded` conditions are kept and a warning is logged. If there are no conditions at the current generation, it reports `Ready=False`, `Degraded=False`. |
| `HelmReleaseStuck` | `True` | The latest revision is `pending-install`, `pending-upgrade`, or `pending-rollback` with no in-flight upgrade in this bridge (e.g. the bridge crashed mid-operation). |
| `HelmReleaseNotReady` | `True` | The latest release revision is in another non-`deployed` state (e.g. `failed`). |
| `ComponentHealthy` | `False` | The release is `deployed` and no upgrade is pending or failed (`Ready=True`, state `COMPONENT_STATE_ENABLED`). |

- **Recovering a stuck release**: the bridge does not auto-recover `HelmReleaseStuck`. Helm records no lock owner, so an automatic rollback or uninstall could clobber an operation that another actor (an operator or another bridge replica) is actively running. To recover, run `helm rollback xmc-cmp-<name> -n <namespace>` (or `helm uninstall xmc-cmp-<name> -n <namespace>` for a `pending-install` with no prior revision). Then bump the `Component` spec or restart the bridge so it re-actuates.
- **Spec Generation Gating & Change Detection**:
- `reconcile1PWorkload` consults `IsComponentActuated(ref, in.Generation)`: steady-state reconciliation passes skip Helm upgrades when the current spec generation is already actuated and no errors are pending.
- `persistComponentValues`, `disableComponentValues`, and `removeComponentValues` return `(bool, error)` to detect whether ConfigMap contents actually changed, short-circuiting no-op upgrades.
Expand All @@ -129,7 +151,7 @@ First-party (1P) components are managed natively through XMC Helm charts. Rather
- `Purge1PComponent`: Permanently deletes the component JSON key from the ConfigMap upon component teardown, and uninstalls the release.
- `Orphan1PComponent`: Retains the ConfigMap values while detaching bridge ownership without triggering release upgrades.
- **Live Helm Go SDK Release Actuation (`SDKHelmActionRunner`)**:
- `HelmActionRunner` Interface: Declarative abstraction over Helm v3 SDK actions (`Upgrade`, `Uninstall`), allowing in-process Helm release upgrades and uninstalls without CLI child processes.
- `HelmActionRunner` Interface: Declarative abstraction over Helm v3 SDK actions (`Upgrade`, `Uninstall`, `Status`), allowing in-process Helm release upgrades, uninstalls, and read-only release status queries without CLI child processes. `Status` reports missing releases as errors matching `driver.ErrReleaseNotFound`.
- `SDKHelmActionRunner`: Uses `action.NewUpgrade` and dynamic Kubernetes REST client configurations to apply values directly to the component release. Resolves per-component chart paths via `WithRunnerChartPathResolver` and caches loaded `*chart.Chart` instances by path (`chartCache`) to avoid repeated disk I/O. Inspects release history with `action.NewHistory`; when `driver.ErrReleaseNotFound` occurs, falls back cleanly to `action.NewInstall` for initial release provisioning. Automatically falls back to an embedded in-memory default chart if no external chart directory is supplied.
- **Context-Aware Upgrade Lock**:
- Serializes concurrent component updates using a channel-based semaphore (`upgradeSem`) to prevent release state conflicts while immediately unblocking upon context cancellation.
Expand Down
17 changes: 17 additions & 0 deletions pkg/components/bridge/fake_helm_manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,12 @@ type FakeHelmManager struct {
ActuatedGenerations map[string]int64
EnsureErr error
DeleteErr error
// UnhealthyReason, when non-empty, makes ComponentHealth report unhealthy with this reason. Defaults to healthy.
UnhealthyReason string
}

var _ HelmManager = (*FakeHelmManager)(nil)

// NewFakeHelmManager creates an initialized FakeHelmManager.
func NewFakeHelmManager() *FakeHelmManager {
return &FakeHelmManager{
Expand Down Expand Up @@ -217,6 +221,19 @@ func (f *FakeHelmManager) HydrateComponentState(ctx context.Context, ref Compone
return nil
}

// ComponentHealth reports the component as healthy unless UnhealthyReason is set.
func (f *FakeHelmManager) ComponentHealth(ctx context.Context, ref ComponentRef) (bool, string, string) {
if f == nil {
return false, ReasonHelmRunnerUninitialized, "fake Helm manager is nil"
}
f.mu.Lock()
defer f.mu.Unlock()
if f.UnhealthyReason != "" {
return false, f.UnhealthyReason, "fake component unhealthy"
}
return true, ReasonComponentHealthy, "fake component healthy"
}

// GetAppliedValues returns a thread-safe deep copy of the applied Helm values for a component.
func (f *FakeHelmManager) GetAppliedValues(ref ComponentRef) map[string]any {
if f == nil || ref.Name == "" {
Expand Down
22 changes: 22 additions & 0 deletions pkg/components/bridge/fake_helm_manager_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -196,3 +196,25 @@ func TestFakeHelmManager_GetAppliedValues_CloneIsolation(t *testing.T) {
t.Errorf("expected internal timeout to remain 30, got %v", freshDepMap["config"])
}
}

func TestFakeHelmManager_ComponentHealth(t *testing.T) {
ref := ComponentRef{Namespace: "xmc", Name: "ingestion-gateway"}
tests := []struct {
name string
mgr *FakeHelmManager
wantHealthy bool
wantReason string
}{
{name: "defaults to healthy", mgr: NewFakeHelmManager(), wantHealthy: true, wantReason: ReasonComponentHealthy},
{name: "unhealthy reason injected", mgr: &FakeHelmManager{UnhealthyReason: ReasonHelmReleaseNotFound}, wantReason: ReasonHelmReleaseNotFound},
{name: "nil manager", mgr: nil, wantReason: ReasonHelmRunnerUninitialized},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
healthy, reason, _ := tt.mgr.ComponentHealth(context.Background(), ref)
if healthy != tt.wantHealthy || reason != tt.wantReason {
t.Errorf("ComponentHealth() = (%v, %q), want (%v, %q)", healthy, reason, tt.wantHealthy, tt.wantReason)
}
})
}
}
Loading
Loading
0