Repository navigation
[Draft] CHK Raft safety planner: observation, admission, and disruption - #2114
alex-zaitsev wants to merge 38 commits into
Conversation
[Draft] Enable authoritative CHK Raft planner for e2e
a5b78bd to
106f7d6
Compare
…knowledging abort
67eab8f to
c5a71d4
Compare
|
@alex-zaitsev question about scope of this planner vs #2041 #2041 is not planned to merge (#2041 (comment)), and #2087 + this PR cover 1→N / 2→3: staged admit, publish after Pod Started, then check live membership via but some parts of #2041 are not in 0.28.0 and not in this branch
do you plan to cover these in this PR, or is it better to extract them from #2041 into separate small PRs on top of the planner? |
|
@alex-zaitsev one more point, about what readiness actually checks Keeper auto is_leader = keeper_dispatcher->isLeader();
auto is_follower = keeper_dispatcher->isFollower() && keeper_dispatcher->hasLeader();
auto status = is_leader || is_follower || is_observer;it is local view of one node, it doesn't say the node is in committed membership of original ensemble or caught up with the log so gating next joiner on previous joiner Ready (#2087, and still in this PR) can't catch it, only |
# Conflicts: # pkg/controller/common/statefulset/statefulset-reconciler_test.go
Summary
Incrementally introduce an Observe → Decide → Act Raft safety planner for CHK.
raftplan.PlanAdmissionandraftplan.CheckDisruption, with unit and parity tests.raft_configurationbefore staged admissions.0.28.0).How to test
Build the operator from this PR's head branch
refactor/chk-raftplan-shadow-20261022and run the usual e2e suite; the planner is on by default. To run unit tests:go test -mod=vendor -race -count=1 ./pkg/controller/chk/...Prioritize staged 1→3 admission, intermediate unready joiner, interrupted Create/operator restart, 2→3, 3→5, degraded quorum during rollout, stop/resume, and scale-down.
The staged-join and readiness changes passed the owner's local suite, but CI for the later cleanup exposed a separate intermittent watcher test failure and a unit-test fixture panic. Commit
a5b78bdcorrects the fixture, tests the watcher using its recorded no-endpoint-change event and unchanged CHI ZooKeeper XML instead of an unrelated live session timestamp, and adds failure diagnostics for a distinct 0-2 scale-up stall. Re-run CI before marking the PR ready.Not in this change
Review notes
The authoritative implementation was developed as stacked PR #2115 and merged into this branch for a single end-to-end review. This PR still targets
0.28.0; it has not been merged into the release branch.Broken-image scale-up regression
The uncommitted
test_020003_4scenario found that a 1→3 scale-up with a broken image could remain indefinitelyInProgressrather than reportingAborted. Two startup paths were corrected:PodHasAllContainersStartednow requires reportedStarted=truestatus for every declared container; empty/partial container statuses are not successful startup.ObjectStatusSame), unready staged joiner on a later reconciliation, the StatefulSet reconciler performs the normal bounded Started, not Ready, wait and uses CHK's existingErrCRUDAbortfallback on timeout.The regression motivated a two-phase join flow: boot with host-local membership, wait for Pod Started, publish the new peer to the leader, then verify the leader's live Raft membership and the joiner's fresh StatefulSet/Pod readiness. A failed startup may abort and resume after a spec edit. No existing Keeper needs to restart for admission.
Cross-reconcile stalled-joiner timeout
After the initial broken-image startup fix, e2e still remained
InProgress: the next joiner was repeatedly returning a softErrCRUDDeferredfor a previously published, unready member. The admission executor now inspects the blocking member's live Pod: if all containers never started and its Pod age exceedsreconcile.statefulSet.update.timeout, it returns hardErrCRUDAbortinstead of repeating the deferral. This deadline is based on Kubernetes timestamps and survives reconcile retries/operator restarts; Pods that have started but are not yet Keeper Ready continue to defer safely. A missing Pod uses the blocking StatefulSet's age. Failed API observations remain errors, not absence.Regression tests cover pure deadline cases, abort propagation, resume, staged Raft publication after Pod Started, and the final committed-but-not-Ready join state. The owner previously confirmed the targeted join scenario passed. The broader e2e watch/scale-up scenario is not yet fully green on CI: two retries stalled creating the final Keeper Pod. The latest patch adds StatefulSet and Kubernetes event diagnostics to investigate any recurrence; no admission logic has been changed.
Cleanup after the working refactor
Removed the unused
rollbackStagedRaftAdmitpath and its obsolete tests, the emptyincludeAllHostsIntoClusterhook, the duplicate staged-join verification branch, and the unused boolean return frompublishStagedAdmit. Replaced pre-create publication test coverage with a test of the current Started → publish ordering. Updated stale inline comments. Legacy decision fallback, shadow diagnostics, quorum safety, and settle delays remain intentionally intact.