Repository navigation
Protect last healthy shard replica during interrupted rolls (#1704) - #2046
Merged
Merged
Conversation
Member
Author
Final review — #2046Commits: Verdict: approveAddresses #1704 without dropping the operator-IP reconcile needed for Behavior (correct)
Residual nits (non-blocking)
No further blockers from this pass. |
3 tasks done
sunsingerus
added a commit
that referenced
this pull request
Aug 3, 2026
Reworks PR #2046 (issue #1704) on three counts found by review: - A deferred host returned ErrCRUDAbort, which the shard[0] early-return turned into 'skip every sibling shard', and also skipped clean(), monitoring and the ancestor update. Deferral now has its own sentinel, is accumulated across shards and clusters, and is surfaced once at the end of the pass. It deliberately still reaches the top as an error: a pass that swallowed it would advance the ancestor past a host that was never rolled and let clean() purge that host. - hostMayRequireDisruption tested !Is(ObjectStatusSame), but at that point host statuses are only Requested/Modified/Found - Same is assigned later, inside ReconcileStatefulSet. Every converged host therefore looked disruptable, so a healthy replica in a degraded shard got no reconcile at all. Replaced by hostDisruptionWouldDegradeShard, evaluated in reconcileHostStatefulSet after PrepareHostStatefulSetWithStatus where the status is authoritative. - runConcurrently was last-writer-wins, so a benign deferral could mask a real shard failure. Deferrals are now tracked apart from errors. shouldExcludeHost keeps a shard-safety check, now for the opposite reason: since the restart is deferred later in the same pass, draining the host here would strip its ready label and drop it from the Service endpoints with nothing left to put it back - emptying the entrypoint Service of a shard whose only other replica is down. Deferral is surfaced via ReconcileAbortWithReason plus a dedicated event reason so 'Aborted' names its cause.
sunsingerus
added a commit
that referenced
this pull request
Aug 3, 2026
…tarving sibling shards Brings in PR #2046 (issue #1704 - an interrupted roll could take down a shard's last healthy replica) together with a rework of it. The deferral now has its own sentinel and is accumulated across shards and clusters, so one deferred host no longer skips every sibling shard. It still reaches the top as an error on purpose: swallowing it would advance the ancestor past a host that was never rolled and let clean() purge that host's StatefulSet. The disruption gate moved into reconcileHostStatefulSet, after the desired StatefulSet is known - the previous predicate tested a status that is only assigned later, so every converged host looked disruptable and a healthy replica in a degraded shard received no reconcile at all. shouldExcludeHost keeps its shard-safety check so a host that is about to be deferred is not drained out of the Service with nothing left in the pass to restore it. Tests: unit coverage for the shard-safety predicate and the sentinel, plus e2e test_010083_1, verified to fail against the unreworked PR on exactly the sibling convergence assertion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Si
66A2
gn in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reconcile unhealthy replicas in the shard prior to healthy ones. Solves #1704. Replaces #1967
Summary
test_010083for interrupted broken-image roll + operator restartTest plan
test_010083test_010008*,test_0100022,test_0100028