8000 fix(components): attach the CMS cleanup finalizer before any side effect by VLukiantsev · Pull Request #1226 · google/xmc · GitHub
[go: up one dir, main page]

Skip to content

fix(components): attach the CMS cleanup finalizer before any side effect - #1226

Merged
VLukiantsev merged 3 commits into
fix/components-bridge-single-writer-b548436300from
fix/components-finalizer-first-b548436300
Oct 16, 2026
Merged

VLukiantsev merged 3 commits into
fix/components-bridge-single-writer-b548436300from
fix/components-finalizer-first-b548436300

Conversation

@VLukiantsev
@VLukiantsev VLukiantsev commented Oct 13, 2026 •
Copy link
Copy Markdown
Collaborator

Summary

The bridge attaches its finalizer as the first step of every pass on a live Component, before any CMS or Helm side effect, so a Component created and deleted in quick succession can no longer leak a CMS record. The finalizer becomes cleanup-only and is renamed xmc.dev/cms-cleanup.

Motivation & Context

A fast create→delete could register intent in CMS before the finalizer existed, leaking the record. Using the finalizer as a deletion veto also hung namespace deletion for protected Components.

Key Changes

  • Finalizer attached with a JSON patch (test + add); the pass continues with the patched object. A rejected patch (for example a stale cache) returns an error and is retried with backoff.
  • Rename to xmc.dev/cms-cleanup, meaning "CMS cleanup pending" for every Component regardless of spec.deletionProtection.
  • The deletion ValidatingAdmissionPolicy is the sole deletion veto: DeletionBlocked is removed and CMS cleanup always runs on deletion (honoring Orphan).
  • One controller-runtime client replaces the separate dynamic client (AddCleanupFinalizer, ReleaseCleanupFinalizer, HasCleanupFinalizer); SyncComponentFinalizers and the DeletionProtection reconcile input are removed.
  • Shared bridge test fakes; envtests for the create/delete race and for deleting protected and unprotected Components.
  • Legacy finalizer: finalizers.xmc.dev/deletion-protection from earlier bridge versions counts as the cleanup finalizer (CMS cleanup runs on deletion, and releasing removes both names). Docs explain this; no migration is needed.

Stack Position

Assumptions

  • Existing Components that carry the legacy finalizers.xmc.dev/deletion-protection finalizer need no manual step: the bridge treats it as the cleanup finalizer and removes it on deletion. Such a CR keeps the legacy name instead of getting xmc.dev/cms-cleanup, unless it is stripped manually (optional).
  • Removing the DeletionBlocked condition is acceptable — the Component API is v1alpha1 and unlaunched — clients reading it must switch to the admission denial.
  • Clusters without ValidatingAdmissionPolicy support are out of scope — on such clusters protected Components would no longer be blocked from deletion.
  • The live GKE evidence still applies to this revision — it was collected before the last rebase onto main, and git range-diff shows no behavioral change — if wrong, the checkpoint must be re-run before merge.

Verification & Testing

  • Commands Executed: bazel test //pkg/components/bridge:bridge_test //cmd/xmc_bridge:xmc_bridge_test //test/e2e:components_lifecycle_test
  • Coverage & Test Status: all pass, including the create/delete race and the protected and unprotected deletion envtests.
  • KinD E2E: //test/e2e:components_lifecycle_test passed with the whole stack applied.
  • Diff size (tools/analyze_diff_size.py): 148 prod / 595 test / 16 doc lines.

Live Deployment & Manual Testing

Scenario Result Observation
Finalizer migration on existing profile CRs ✅ PASS After an optional strip of the legacy finalizer from clickhouse, redpanda and ingestion-gateway, the bridge attached and kept xmc.dev/cms-cleanup on the next pass.
Fast create→delete race (mt-cp1-race) ✅ PASS Back-to-back kubectl apply and kubectl delete attached the finalizer before any CMS write, executed CMS deletion and released the finalizer; the Spanner row ended State=5, ActiveName=NULL with no leak.
Admission veto and unprotected teardown (mt-cp1-fin) ✅ PASS kubectl delete with deletionProtection: true was denied by the deletion ValidatingAdmissionPolicy. With deletionProtection: false, delete soft-deleted the CMS row (State=5) and released the finalizer.

Related Issues / Bugs

Bug: 548436300

TAG=agy
CONV=d833b0e6-4821-48f6-8576-776d53f55a5c


Stack created with GitHub Stacks CLI • Give Feedback 💬

@VLukiantsev
VLukiantsev added this pull request to stack #1245 October 13, 2026 07:27
@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the deletion protection and finalizer logic in xmc-bridge. It replaces the legacy finalizers.xmc.dev/deletion-protection finalizer with xmc.dev/cms-cleanup, delegating deletion protection enforcement to an admission policy. The controller now ensures the cleanup finalizer is attached before executing any CMS or Helm side effects, and uses the controller-runtime client instead of a dynamic client for status and finalizer updates. Feedback suggests using /metadata/resourceVersion instead of /metadata/finalizers in the JSON patch test operation to provide a more robust and standard optimistic concurrency control mechanism, preventing potential failures when finalizers are empty or absent.

Comment thread pkg/components/bridge/status_applier.go Resolved
@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors finalizer management in xmc-bridge by replacing the legacy deletion-protection finalizer with a new cms-cleanup finalizer. It also transitions the controller and status applier to use the controller-runtime client instead of a dynamic client, replacing the optimistic concurrency update loop with JSON patches. The review feedback suggests optimizing finalizer operations by checking for their presence or absence before making API calls, and recommends testing /metadata/resourceVersion instead of /metadata/finalizers to avoid fragile JSON patch test operations when finalizers are empty.

Comment thread pkg/components/bridge/status_applier.go Resolved
Comment thread pkg/components/bridge/status_applier.go Resolved
8000
Comment thread pkg/components/bridge/status_applier.go Resolved
@VLukiantsev
Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the deletion and finalizer lifecycle in xmc-bridge. It replaces the legacy finalizers.xmc.dev/deletion-protection finalizer with xmc.dev/cms-cleanup, ensuring it is attached to a Component before any CMS or Helm side effects occur. Deletion protection enforcement is decoupled from the bridge reconciler and delegated to the component-deletion-protection admission policy. Additionally, finalizer updates are refactored to use safe JSON patches via AddCleanupFinalizer and ReleaseCleanupFinalizer instead of the previous optimistic concurrency loop, and ComponentController is updated to use a unified client.Client. There are no review comments to address, and I have no further feedback to provide.

@VLukiantsev
VLukiantsev force-pushed the fix/components-finalizer-first-b548436300 branch 2 times, most recently from d477d03 to f1f1f06 Compare October 14, 2026 09:37
@VLukiantsev
VLukiantsev force-pushed the fix/components-finalizer-first-b548436300 branch 2 times, most recently from 743063e to 75f9364 Compare October 14, 2026 11:58
@VLukiantsev
VLukiantsev removed this pull request from stack #1245 October 14, 2026 13:31
@VLukiantsev
VLukiantsev added this pull request to stack #1277 October 14, 2026 13:32
@VLukiantsev
VLukiantsev force-pushed the fix/components-finalizer-first-b548436300 branch from 75f9364 to f4fd439 Compare October 14, 2026 13:54
@VLukiantsev
VLukiantsev force-pushed the fix/components-finalizer-first-b548436300 branch 2 times, most recently from 2de933a to 3bcb131 Compare October 14, 2026 14:48
@VLukiantsev
VLukiantsev marked this pull request as ready for review October 15, 2026 07:41
@github-actions
github-actions Bot requested a review from andrzejfudro October 15, 2026 07:41
@VLukiantsev
VLukiantsev force-pushed the fix/components-finalizer-first-b548436300 branch 3 times, most recently from 179b0b6 to 80a8449 Compare October 15, 2026 10:02
andrzejfudro
andrzejfudro previously approved these changes Oct 15, 2026
@VLukiantsev
VLukiantsev force-pushed the fix/components-finalizer-first-b548436300 branch from 80a8449 to 70810e3 Compare October 15, 2026 12:50
@VLukiantsev VLukiantsev added the priority::2-high top of mind. should not wait. label Oct 15, 2026
The bridge now attaches its finalizer as the first step of every pass on a
live Component, before it touches CMS, so a Component that is created and
deleted in quick succession can no longer leak a CMS record. The finalizer is
attached with a JSON patch (test + add) and the pass continues with the
patched object.

- Rename the finalizer to xmc.dev/cms-cleanup. It now means "CMS cleanup
  pending" for every Component, regardless of spec.deletionProtection.
- Make the deletion ValidatingAdmissionPolicy the sole deletion veto: remove
  the DeletionBlocked condition, and always run CMS cleanup on deletion.
- Use a single controller-runtime client in the controller in place of the
  separate dynamic client. Add AddCleanupFinalizer, ReleaseCleanupFinalizer,
  and HasCleanupFinalizer, and remove SyncComponentFinalizers, AddFinalizer,
  HasFinalizer, and the DeletionProtection reconcile input.
- Share one set of test fakes across the bridge tests. Add envtests for the
  create/delete race and for deleting protected and unprotected Components.
- Document how to manually replace the legacy finalizer on existing
  Components. No migration code is added.

Bug: 548436300
TAG=agy
CONV=d833b0e6-4821-48f6-8576-776d53f55a5c
Treating every 422 from the finalizer patch as a stale cache and
requeueing after a fixed second hid persistent validation rejections in
an Info-logged loop. Return the error instead so the Component rate
limiter backs off and the failure keeps surfacing.

Bug: 548436300
TAG=agy
CONV=d833b0e6-4821-48f6-8576-776d53f55a5c
AddCleanupFinalizer returns the item unchanged when it already carries
the cleanup finalizer or its legacy name, and ReleaseCleanupFinalizer
returns early when neither is present, so neither issues a redundant
write.

Bug: 548436300
TAG=agy
CONV=d833b0e6-4821-48f6-8576-776d53f55a5c
@VLukiantsev
VLukiantsev force-pushed the fix/components-finalizer-first-b548436300 branch from 70810e3 to d513f91 Compare October 16, 2026 09:37
@github-actions github-actions Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Incremental PR Approval

Clean rebase detected with zero semantic patch lines changed.

Preserving prior approval by @andrzejfudro on commit 70810e3.

@VLukiantsev
VLukiantsev added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit 0fcb616 Oct 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority::2-high top of mind. should not wait.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0