Repository navigation
fix: propagate preliminary aux ConfigMap errors during reconcile (fixes #2094) - #2110
Open
Tyagiquamar wants to merge 2 commits into
Open
Tyagiquamar wants to merge 2 commits into
Tyagiquamar wants to merge 2 commits into
Conversation
reconcileCRAuxObjectsPreliminary in both chi and chk workers previously swallowed errors from reconcileConfigMapCommon and reconcileConfigMapCommonUsers, logging the failure but returning nil. This allowed WalkTillError to proceed directly to reconciling clusters and creating StatefulSets/Pods while common ConfigMaps were absent, causing pods to hang in Init waiting on non-existent mounts and stalling for the 300s StatefulSet wait timeout. Return the error from reconcileCRAuxObjectsPreliminary immediately so WalkTillError halts promptly when preliminary ConfigMaps fail, and add regression tests. Fixes Altinity#2094
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reconcile stops when the preliminary common ConfigMaps fail, instead of continuing on to StatefulSets that mount those ConfigMaps.
Root cause
reconcileCRAuxObjectsPreliminaryin the ClickHouse and ClickHouse Keeper workers logged errors fromreconcileConfigMapCommonandreconcileConfigMapCommonUsers, then returned nil. The walk continued, pods entered Init waiting on mounts that were never created, and the reconcile sat in the StatefulSet wait (300s).Solution
Those ConfigMap errors are returned to the caller. A regression test drives a preliminary failure and expects reconcile to stop.
Test plan
pkg/controller/chi/worker-reconciler-chi_test.gocovers the preliminary-failure path.Fixes #2094