alex-zaitsev · Oct 22, 2026 · Sep 30, 2026 · Sep 30, 2026 · Sep 30, 2026 · Sep 30, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -16,6 +16,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * **EXPERIMENTAL.** **Backups and restores run per cluster.** Each backup, schedule and restore of an installation runs one Job (or `CronJob`) per backup-enabled cluster, all under one backup name, and reports every cluster in `status.clusters`: a failing cluster no longer fails or stops pruning the others (`PartiallyC
8000
ompleted`), operations and restore holds conflict per cluster only - but for a restore of access entities, which holds every cluster - and the metrics carry a `cluster` label. Storage paths default to `backup/{namespace}/{installation}/{cluster}/{shard}` and `object_disks/{namespace}/{installation}/{cluster}/{shard}`, so the installations of a Kubernetes cluster sharing one storage keep their backups apart, and every Job refuses, before changing anything, paths that two shards would share - and, in an installation with several backup-enabled clusters, paths it cannot check, such as those of a sidecar configuration file. Schemas are restored on every replica, so `Distributed` tables keep pointing at the clusters they named; an `overwrite` restore that drops replicated tables of a cluster with replicas first drops the restored tables `ON CLUSTER`, and every `overwrite` restore refuses, before changing anything, tables over `max_table_size_to_drop` that its drops could not get past. See [docs/backup.md](docs/backup.md#installations-and-clusters). by [@sunsingerus](https://github.com/sunsingerus)
 
 ### Changed
+* **A Keeper scale-up from one or two members admits one new replica at a time.** On 1→N, a member that does not exist yet stays out of `raft_configuration` until the previous join is Ready, so the operator does not create every new StatefulSet in one pass. A fresh ensemble is unchanged, and 3→N still adds replicas without that wait so the existing members can roll first. by [@alex-zaitsev](https://github.com/alex-zaitsev) in [#2087](https://github.com/Altinity/clickhouse-operator/pull/2087). Partially addresses [#2041](https://github.com/Altinity/clickhouse-operator/pull/2041)
+
 * **BACKWARD INCOMPATIBLE.** **The operator and the metrics exporter refuse to start when `security.fips.enforced` is not a boolean.** A mistyped value (`"ture"`) used to read as off, so the binaries ran without FIPS enforcement unnoticed; the startup log now names the value. Set it to `true` or `false`. by [@sunsingerus](https://github.com/sunsingerus)
 
 * **BACKWARD INCOMPATIBLE.** **A `ClickHouseInstallation` now wins over the templates it uses.** Where a template used to beat it silently - same-named pod, host and volume-claim templates, `configuration.users`, `profiles`, `quotas`, `settings` and `files`, `suspend`, `restart`, `taskID`, `namespaceDomainPattern`, `reconcile.statefulSet` and `reconcile.host` among them - the installation now wins, templates fill what it leaves unset, and a later template beats an earlier one. The operator's configuration rules (`clickhouse.addons.rules`) now yield the same way to a template or the installation setting the same key, such as `display_secrets_in_show_and_select`, though a grant spelled with the operator user's name still loses to the rules', which use the `{clickhouseOperatorUser}` placeholder. Containers, volumes and env now pair by name, mounts by mount path and ports by number and protocol, so differently named containers at the same position are now two. [docs/operator_configuration.md](docs/operator_configuration.md#template-precedence) lists the fields and what an upgrade changes. by [@sunsingerus](https://github.com/sunsingerus)

diff --git a/pkg/apis/common/types/reconcile_attributes.go b/pkg/apis/common/types/reconcile_attributes.go
@@ -96,6 +96,9 @@ func (a *ReconcileAttributes) SetExclude() *ReconcileAttributes {
 	if a == nil {
 		return a
 	}
+	if a.tags == nil {
+		a.tags = NewTags()
+	}
 	a.tags.Set(TagExclude)
 	return a
 }
@@ -117,7 +1
8000
20,10 @@ func (a *ReconcileAttributes) IsExclude() bool {
 	return a.tags.Has(TagExclude)
 }
 
-// SetLowPriority sets 'LowPriority' attribute
+// SetLowPriority sets 'LowPriority' attribute.
+// Nil tags stay a no-op: Tags.Set on a nil map does not store, and CHI
+// descendHostInClickHouseCluster calls this on every restart. Allocating
+// tags here would emit <priority>1000</priority> in remote_servers.
 func (a *ReconcileAttributes) SetLowPriority() *ReconcileAttributes {
 	if a == nil {
 		return a

diff --git a/pkg/controller/chk/worker-raft-safety.go b/pkg/controller/chk/worker-raft-safety.go
@@ -16,6 +16,7 @@ package chk
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"time"
 
@@ -89,6 +90,166
8000
 @@ func (f *chkStatefulSetFallback) OnStatefulSetUpdateFailed(
 // sizes defers every roll forever instead of preserving availability the user still has.
 const raftFaultTolerantMinMembers = 3
 
+// crHasEstablishedCluster is true when status.normalizedCompleted already lists hosts.
+// GetAncestor resolves from that status, and it is stamped only after a fully
+// successful pass. A CHK created with one replica whose first reconcile never
+// completed, then scaled to 3, has no ancestor and is treated as a fresh
+// bootstrap, so it is not staged. That fail-open is deliberate: a partial live
+// set with no completed generation is indistinguishable from an interrupted
+// bootstrap, and staging a bootstrap would publish a partial initial quorum.
+func crHasEstablishedCluster(cr api.ICustomResource) bool {
+	return cr != nil && cr.GetAncestor() != nil && cr.GetAncestor().HostsCount() > 0
+}
+
+// stageUncommittedScaleUpHosts keeps not-yet-created members out of
+// raft_configuration on 1→N / 2→3, so the published XML lists at most one
+// extra server while the previous join is still coming up. Only hosts whose
+// StatefulSet is confirmed absent are tagged: an interrupted pass must not
+// re-exclude a member that is already running. A Get error other than NotFound
+// fails closed — nothing is staged and the caller must requeue rather than
+// publish an expanded membership. Fresh bootstrap and 3→N (ancestor already
+// has quorum headroom) are left alone.
+func (w *worker) stageUncommittedScaleUpHosts(ctx context.Context, cr *apiChk.ClickHouseKeeperInstallation) error {
+	if cr == nil || !crHasEstablishedCluster(cr) || w.c == nil {
+		return nil
+	}
+	if ensembleHasQuorumHeadroom(cr.GetAncestor().HostsCount()) {
+		return nil
+	}
+
+	var toStage []*api.Host
+	var getErr error
+	_ = cr.WalkHosts(func(host *api.Host) error {
+		if host == nil || !host.GetReconcileAttributes().GetStatus().Is(types.ObjectStatusRequested) {
+			return nil
+		}
+		// Same predicate as rollbackStagedRaftAdmit: absent means NotFound.
+		sts, err := w.stagedJoinerStatefulSet(ctx, host)
+		if err != nil {
+			getErr = err
+			return err
+		}
+		if sts == nil {
+			toStage = append(toStage, host)
+		}
+		return nil
+	})
+	if getErr != nil {
+		return getErr
+	}
+	for _, host := range toStage {
+		host.GetReconcileAttributes().SetExclude()
+		w.a.V(1).M(host).F().Info(
+			"Staging new host out of raft_configuration until it is admitted: %s",
+			host.GetName(),
+		)
+	}
+	return nil
+}
+
+// hostIsStagedScaleUpJoin is a new member of an already-running ensemble that is too
+// small to protect quorum (1→N, 2→3). Identity is topological — established
+// ancestor without headroom, and this host is absent from that ancestor — not
+// ObjectStatusRequested. PrepareHostStatefulSetWithStatus overwrites Requested
+// to Same/Modified once the joiner STS exists, so a retry must not treat that
+// host as still waiting to be admitted. deferUnreadyStagedAdmit uses this to
+// postpone the next not-yet-created joiner until this one's live ReadyReplicas
+// is positive. Ready is not committed Raft membership (see
+// verifyHostEnsembleMembership). 3→N is not staged.
+func hostIsStagedScaleUpJoin(host *api.Host) bool {
+	if host == nil || host.GetCR() == nil {
+		return false
+	}
+	if !crHasEstablishedCluster(host.GetCR()) {
+		return false
+	}
+	if host.HasAncestor() {
+		return false
+	}
+	return !ensembleHasQuorumHeadroom(host.GetCR().GetAncestor().HostsCount())
+}
+
+// deferUnreadyStagedAdmit postpones a not-yet-created 1→N / 2→3 joiner while
+// another staged joiner already has a StatefulSet that is not Ready.
+//
+// The check is live ReadyReplicas, not ObjectStatus and not WaitUntilReady.
+// After an interrupted pass the previous joiner's StatefulSet exists, so
+// PrepareHostStatefulSetWithStatus marks it Same and ReconcileStatefulSet
+// returns without waiting. ErrCRUDDeferred is soft: the shard loop still
+// reconciles the other hosts, including the existing members that have to
+// reload or roll onto the published raft_configuration before the joiner can
+// become Ready. A Get error other than NotFound fails closed — membership is
+// not expanded while StatefulSet existence is unknown.
+func (w *worker) deferUnreadyStagedAdmit(ctx context.Context, host *api.Host) error {
+	if !hostIsStagedScaleUpJoin(host) || w.c == nil {
+		return nil
+	}
+	own, err := w.stagedJoinerStatefulSet(ctx, host)
+	if err != nil {
+		return err
+	}
+	if own != nil {
+		return nil
+	}
+	// Stopped: do not defer (the pass must reach clean) and do not create the
+	// StatefulSet (resume would start every missing member at once).
+	if host.IsStopped() {
+		w.a.WithEvent(host.GetCR(), a.EventActionReconcile, a.EventReasonReconcileInProgress).
+			WithAction(host.GetCR()).
+			M(host).F().
+			Info("Skip admitting %s while ClickHouseKeeper is stopped", host.GetName())
+		return errSkipStagedAdmit
+	}
+
+	var blocker *api.Host
+	var getErr error
+	_ = host.GetCR().WalkHosts(func(peer *api.Host) error {
+		if getErr != nil || blocker != nil || peer == nil || peer == host || !hostIsStagedScaleUpJoin(peer) {
+			return nil
+		}
+		sts, peerErr := w.stagedJoinerStatefulSet(ctx, peer)
+		if peerErr != nil {
+			getErr = peerErr
+			return nil
+		}
+		if sts != nil && sts.Status.ReadyReplicas < 1 {
+			blocker = peer
+		}
+		return nil
+	})
+	if getErr != nil {
+		return getErr
+	}
+	if blocker == ni
BD0
l {
+		return nil
+	}
+	w.a.WithEvent(host.GetCR(), a.EventActionReconcile, a.EventReasonReconcileInProgress).
+		WithAction(host.GetCR()).
+		M(host).F().
+		Info("Defer admitting %s into raft_configuration until %s is Ready", host.GetName(), blocker.GetName())
+	return common.ErrCRUDDeferred
+}
+
+// errSkipStagedAdmit tells reconcileHost to leave this not-yet-created member
+// alone and continue the pass. It is not a deferral and not a failure.
+var errSkipStagedAdmit = errors.New("skip staged keeper admit while stopped")
+
+// stagedJoinerStatefulSet returns the live StatefulSet, or nil when it is absent.
+// NotFound is not an error. Any other Get error is returned unchanged.
+func (w *worker) stagedJoinerStatefulSet(ctx context.Context, host *api.Host) (*apps.StatefulSet, error) {
+	sts, err := w.c.kube.STS().Get(ctx, host)
+	if apiErrors.IsNotFound(err) {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	if sts != nil {
+		host.Runtime.CurStatefulSet = sts
+	}
+	return sts, nil
+}
+
 // ensembleHasQuorumHeadroom reports whether the gate can ever pass for this ensemble size.
 func ensembleHasQuorumHeadroom(members int) bool {
 	return members >= raftFaultTolerantMinMembers
@@ -354,11 +515,21 @@ func (w *worker) isHostHealthyForReconcile(ctx context.Context, host *api.Host)
 // protect, and this falls back to the desired set (see the last paragraph). So read the result as
 // "what to size quorum on", never as "what is currently running".
 //
-// Membership is static. The generator emits keeper_server/raft_configuration as a plain config
-// section, and enable_reconfiguration is shipped explicitly disabled
-// (config/chk/keeper_config.d/01-keeper-03-enable-reconfig.xml), so a running Keeper holds the
-// membership it started with. Publishing the ConfigMap for a scale-up therefore does NOT admit the new
-// servers to the running Raft - they join only as the existing pods roll onto the new config.
+// Membership is static at the Raft protocol level: enable_reconfiguration is shipped
+// disabled, so there is no runtime reconfig command. The generator emits
+// keeper_server/raft_configuration as a mounted ConfigMap file. Publishing that
+// file does not mean live Raft has committed the new server_id — existing members
+// apply it when they reload or roll. Do not size quorum on the desired set as if
+// those servers were already voting.
+//
+// /ready on a newly created joiner is not a committed-membership barrier, and it
+// is not how 1→N / 2→3 serializes creates. A new peer cannot reach /ready until
+// existing members reload or roll onto the published file, and recovery-first
+// reconciles that peer before they do, so SetWaitUntilReady there expires into
+// ErrCRUDAbort. deferUnreadyStagedAdmit postpones the next missing joiner until
+// the previous joiner's live ReadyReplicas is positive, and that postponement
+// still lets this pass roll the existing members. verifyHostEnsembleMembership
+// is the extension point for /keeper/config + mntr.
 //
 // Sizing growth on the desired set inflates the denominator against a membership that does not
 // exist yet, and the damage is in the unsafe direction: a 3->5 with one member already down
@@ -369,8 +540,8 @@ func (w *worker) isHostHealthyForReconcile(ctx context.Context, host *api.Host)
 // An ancestor too small to tolerate a loss is not a quorum worth protecting, so fall back to the
 // desired set there. That also keeps growth classified as bootstrap: a 1->3 sized at 1 would read
 // rolling (members<=1 is rolling unconditionally), putting a Ready wait on the one EXISTING host
-// while the ensemble it must reach quorum with is still being created. The new hosts are already
-// safe either way - joinedEnsemble below denies them the Ready wait.
+// while the ensemble it must reach quorum with is still being created. 3→N added hosts still
+// skip Ready (joinedEnsemble). 1→N / 2→3 skip it too; the host loop defers the next admit.
 func quorumSizingEnsemble(cr api.ICustomResource) api.ICustomResource {
 	if cr == nil {
 		return nil
@@ -480,6 +651,12 @@ func (w *worker) prepareStsReconcileOptsWaitSection(
 	// no StatefulSet at all and so fails both terms. CurStatefulSet was refreshed moments ago by
 	// snapshotHostEnsemble.
 	joinedEnsemble := host.HasAncestor() || hostContributesReady(host)
+	// 1→N / 2→3 does not force Ready here. The new peer is reconciled before
+	// existing members roll, and it cannot reach /ready until they reload or
+	// roll onto the published raft_configuration. Forcing the wait would burn
+	// the STS timeout and abort, including when readiness is "no". One-at-a-time
+	// admission is deferUnreadyStagedAdmit. 3→N still skips Ready so existing
+	// members can roll onto the config that admits the peer.
 
 	// A host outside the live ensemble still has to wait to START. The code before this gate spelled
 	// this `probes.GetStartup().IsTrue() || !host.HasAncestor()`; narrowing it to !rolling alone