Repository navigation
fix wait for replica inactivity and retry SYSTEM DROP REPLICA - #2047
Draft
bhardwajparth51 wants to merge 3 commits into
Draft
bhardwajparth51 wants to merge 3 commits into
bhardwajparth51 wants to merge 3 commits into
Conversation
…PLICA on Code 305 (Altinity#1943) Fixes issue Altinity#1943 where scaling down replicas causes SYSTEM DROP REPLICA to fail with Code: 305, Can't drop replica ... because it's active when executed before ZooKeeper session timeout expires. - Extracts ISchemer interface for dependency injection and testability. - Adds waitHostReplicaInactive polling until ZooKeeper session expires. - Adds retryDropReplica with linear backoff on Code 305 errors. - Adds getCoordinatorHost fallback helper for single-replica shard drops. - Adds comprehensive unit test suite in worker-deleter_test.go. Signed-off-by: Parth Bhardwaj <196071556+bhardwajparth51@users.noreply.github.com>
Conflict resolutions: - pkg/controller/chi/worker.go: 0.28.0 removed the shared worker.schemer (a data race under parallel shard reconcile); kept the field only as a test seam that is never set in production, read through the new clusterSchemer() - pkg/controller/chi/worker-migrator.go: ensureClusterSchemer keeps 0.28.0's per-call *ClusterSchemer (returns the concrete type); the injectable ISchemer lookup moved to clusterSchemer(), used by the PR's waitHostReplicaInactive/retryDropReplica - pkg/controller/chi/worker-deleter_test.go: add/add conflict, kept both the 0.28.0 deleteHost tests and the PR's replica-wait/retry tests in one file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…duler (Altinity#1943) During ClickHouse replica scale-down, SYSTEM DROP REPLICA can fail with DB::Exception: Replica is active. Code: 305 if executed before the terminated pod's ZooKeeper/Keeper session metadata expires. Previously, drop failures were ignored after promoting the ancestor, permanently orphaning Keeper metadata. This change brings replica cleanup to production readiness: - Reconciler Lifecycle: ZooKeeper replica cleanup now runs before finalizeReconcileAndMarkCompleted. If cleanup fails, reconciliation aborts with common.ErrCRUDAbort, preserving the previous ancestor in Status.NormalizedCRCompleted so removed hosts are rediscovered. - Async Retry Scheduler: Adds reconcileRetryScheduler using stoppable Go runtime timers to schedule delayed retries with exponential backoff (10s to 300s) without blocking worker threads or holding work queue items during delay. - Error Classification: Separates retryable errors (ErrZKReplicaDropRetryable: Code 305, Code 999 transient coordination errors, network timeouts, EOF) from permanent failures (ErrZKReplicaDropPermanent: Code 999 auth/ACL/argument errors, syntax errors, table not found). - Graceful Controller Shutdown: Closes and drains the retry scheduler before closing controller work queues, guaranteeing no callbacks enqueue onto closed queues. - Concurrency & Regression Tests: Adds unit tests for the retry scheduler, verifying callback linearization, preemption, shutdown draining, backoff progression, and comprehensive table-driven error classification tests. Fixes Altinity#1943 PR Altinity#2047
bhardwajparth51
90EF
marked this pull request as draft
November 4, 2026 20:13
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1943 where scaling down replicas causes
SYSTEM DROP REPLICAto fail withCode: 305, Can't drop replica ... because it's activewhen executed before ZooKeeper session timeout expires.ISchemerinterface for dependency injection and testability.waitHostReplicaInactivepolling until ZooKeeper session expires.retryDropReplicawith linear backoff on Code 305 errors.getCoordinatorHostfallback helper for single-replica shard drops.worker-deleter_test.go.Signed-off-by: Parth Bhardwaj 196071556+bhardwajparth51@users.noreply.github.com
Important items to consider before making a Pull Request
next-releasebranch, not intomasterbranch.