Releases

@fhoekstra

Security

BACKWARD INCOMPATIBLE. Removed the k8s_secret_ and k8s_secret_env_ user-settings syntax. A CHI that still uses the syntax is now rejected with RemovedSecretRefSyntax. Migrate to valueFrom/secretKeyRef syntax unstead (see docs/security_hardening.md):
```
users:
  user1/password:
    valueFrom:
      secretKeyRef:
        name: clickhouse-secret
        key: pwduser1
```
by @sunsingerus
Bumped the Go toolchain 1.26.5 → 1.26.8 to address stdlib CVEs in the operator and metrics-exporter images (GO-2026-6090, GO-2026-6089, GO-2026-6218, GO-2026-5972, GO-2026-5026). No API or behavior changes. The FIPS module (GOFIPS140=v1.0.0) is unchanged. by @sunsingerus
Bumped the OpenTelemetry SDK v1.43.0 → v1.45.0 (otel, otel/metric, otel/sdk, otel/sdk/metric, otel/trace). Clears CVE-2026-41178, an uncapped baggage header parse in otel/baggage and otel/propagation, both of which the operator links. Also clears CVE-2026-81870; it is filed against otel/sdk, so module-keyed scanners flagged us, but it logs trace exporter endpoints and the operator links no trace SDK or trace exporter. by @sunsingerus

Behavior Changes

A pod that will not terminate is force-deleted so StatefulSet Recreate finishes in one pass. A wedged shutdown previously aborted Recreate with the host left at Replicas=0. The operator now deletes that pod after a full wait timeout, emits HostPodForceDeleted, and completes Delete+Create in the same pass. A terminating pod is no longer treated as a healthy shard peer. by @alex-zaitsev in #2080. Fixes #2078
ZooKeeper endpoint changes no longer restart ClickHouse. configurationRestartPolicy has advertised zookeeper/*: "no" since 0.25.6; the policy is now honored, so endpoint edits reload via conf.d with no restarts. This also removes the one-time CHI→CHK restart noted in 0.27.2. by @identw in #2074
Keeper rolling updates no longer take an ensemble below Raft quorum. A Ready Keeper replica is deferred ([RaftQuorumUnsafe]) when disrupting it would lose majority; not-Ready replicas are recovered first. Affects ClickHouseKeeperInstallation rolls. by @alex-zaitsev in #2070. Fixes #2069
Helm CRD-install hook uses registry.k8s.io/kubectl:v1.36.3 instead of bitnami/kubectl:latest. Air-gapped and mirrored installs must add that image before upgrade, or set crdHook.image / crdHook.enabled: false. Helm installs only. by @fhoekstra in #2065
A failed schema migration no longer reports the CHI as Completed. HostCreateTables errors now abort the pass instead of being recorded as success. A CHI that already had an un-creatable object (for example a Dictionary with an unreachable source) will turn Aborted on first reconcile after upgrade. by @Tyagiquamar in #2077. Fixes #2021
Operator informers and caches no longer hold every object in the Kubernetes cluster. They are narrowed at the API server to clickhouse.altinity.com/app=chop, and clickhouse-keeper.altinity.com/app=chop respectively, so memory scales with the number of managed installations rather than cluster size. by @sunsingerus. Approach proposed by @gregakinman in #2058
ActionPlan is rendered once on status update. A status update invoked String() many times per retry, and each call reflection-dumped every diffed object, so on a large CHI that dominated operator CPU. by @erikasb in #2067
A failed Keeper start no longer disables Keeper reconciliation until operator restart. Start-up now contacts the API server (required by the narrowed cache), so a transient error can fail the first attempt. The operator retries indefinitely without exiting; ClickHouse reconciliation is not blocked. by @sunsingerus
OTEL_GO_X_CARDINALITY_LIMIT no longer caps operator metrics. The SDK bump brings a default of 2000 data points per instrument per scrape, which silently folds everything past it into one otel.metric.overflow series; the operator now pins the limit to unlimited, so series keep scaling with the installations watched. Anyone who set the variable to protect Prometheus must limit at the scrape instead. by @sunsingerus

Fixed

Host deletion is no longer reported as completed when the operator cannot tell whether the host still exists. Only NotFound means the host is gone; other API errors now raise DeleteFailed. Present since 0.10.0. Residual PVC-orphan on finalizer removal is tracked in #2056. by @somanchi004-code in #2057
Operator no longer keeps probing a ZooKeeper ensemble it can no longer reach. A disconnected ZK session leaked the client and re-dialed stale addresses until operator restart. Every terminal state now closes the connection. Root-path ensure is cancellable, skipped on empty recovery passes, and raises CreateFailed after retries (still non-fatal). by @alex-zaitsev in #2073
Oversized CRD ConfigMap broke GitOps installs of the Helm chart. The CRD-install hook packed three CRDs into one ConfigMap past the 256Ki annotation ceiling, so ArgoCD client-side apply and helm template | kubectl apply failed. Each CRD now gets its own ConfigMap. helm install / helm upgrade were unaffected. by @sunsingerus
Stale ClickHouse HTTP connection pools are reset after connection failures. by @norrs in #2068
A failed read is no longer mistaken for a missing object. Only NotFound means absent; any other error — RBAC still propagating, an overloaded API server — escalated to recovery, deleting a StatefulSet and its pods once a later read succeeded, or reporting phantom data loss on a Keeper PersistentVolumeClaim. Such reads now raise a host reconcile error and retry on the next pass. Affects ClickHouse and Keeper alike. by @sunsingerus
Lost-volume recovery no longer ignores shutdown. The poll that waits up to an hour for a PersistentVolumeClaim to disappear did not observe the operator context, so a stop issued during it could be held up until the poll ran out. by @sunsingerus

New Contributors

@fhoekstra made their first contribution in #2065
@somanchi004-code made their first contribution in #2057
@norrs made their first contribution in #2068
@erikasb made their first contribution in #2067
@identw made their first contribution in #2074
@Tyagiquamar made their first contribution in #2077

Full Changelog: release-0.27.3...release-0.27.4

@Slach

Added

Helm: scrape native ClickHouse Keeper Prometheus metrics when serviceMonitor.enabled is set, by @Slach in #2048. Closes #2038. Opt out with serviceMonitor.keeperMetrics.enabled: false. Keeper must expose a metrics port; clusters without Keeper are unaffected.

serviceMonitor:
  enabled: true
  keeperMetrics:
    enabled: true
    port: metrics

Changed

Protect the last healthy replica in a shard during an interrupted roll. Closes #1704. Replaces #1967. Unhealthy replicas in a shard are reconciled before healthy ones; a disruptive restart or image roll is deferred when it would take down the last healthy replica. Sibling shards keep converging — a deferred host no longer aborts the rest of the pass. The CHI still reports Aborted with event reason HostReconcileDeferredShardSafety until the deferred host can be rolled.

Fixed

Image upgrades again exclude the host first (LB drain, remote_servers de-prioritisation, in-flight query drain). Closes #2055. That was a regression since 0.26.3.
Honor onLostVolume: no so the operator does not SYSTEM DROP REPLICA when a volume is lost. The setting existed but was ignored on the ForceDrop path.

reconcile:
  host:
    drop:
      replicas:
        onLostVolume: no

Retry transient Kubernetes API errors on CHI kube Get calls, by @aaron276h in #2025. Closes #2026. Bounded exponential backoff (5 attempts, ~7.5s) for connection refused/reset, timeouts, and apiserver 5xx — a brief control-plane blip no longer aborts a large reconcile. Terminal errors (NotFound, Forbidden, …) still fail immediately.
Fix informer delete handling for DeletedFinalStateUnknown tombstones, by @27rohan in #2039. Closes #1882. Missed-watch relist payloads no longer panic the operator; malformed delete events are logged instead.
Skip cluster/host reconcile hooks on unreachable hosts (e.g. a scale-up replica that has no pod yet, or a dying host that cannot serve SQL). Closes #2052. instead of failing the hook and blocking reconcile. Emits HookSkippedUnreachableHost. Reachability overrides failurePolicy: Fail. The hook still runs on hosts that can be reached.

Security

Bumped dependencies to address CVEs in the operator and metrics-exporter images. An image scan flagged four CVEs; all are addressed by patch/minor bumps with no API or behavior changes:
- CVE-2026-39822 (HIGH) and CVE-2026-42505 (MEDIUM) — Go standard library. Fixed by building on Go 1.26.5 (go.mod 1.26.4 → 1.26.5).
- CVE-2026-46600 — golang.org/x/net bumped v0.55.0 → v0.56.0.
- CVE-2026-56852 — golang.org/x/text bumped v0.37.0 → v0.39.0.
Reachability note: govulncheck confirms the stdlib crypto/tls and x/text issues are reachable from operator/exporter code; the x/net and stdlib os issues are present-in-image but not reachable. All four are bumped for image hygiene regardless. The FIPS module (GOFIPS140=v1.0.0) is pinned independently of the Go toolchain and is unaffected.

New Contributors

@aaron276h made their first contribution in #2025
@27rohan made their first contribution in #2039

Full Changelog: release-0.27.2...release-0.27.3

@miguel-signoz

Added

Add configurable MaxConcurrentReconciles for CHK controller by @miguel-signoz in #2033. Default is 1. Closes #2032

reconcile:
  runtime:
    reconcileCHKsThreadsNumber: 1

Auto-heal Completed CHIs with sustained-NotReady hosts by @dashashutosh80 in #1998. Closes #1994. Tuned off by default:

reconcile:
  recovery:
    onStatus:
      # Recovery for a CHI whose .status.status is Aborted (reconcile did not complete)
      # when one of its host pods transitions to Ready — auto-resumes the reconcile.
      aborted:
      completed:
        # Action when a Completed CHI's pod flips Ready=True -> Ready=False and
        # stays NotReady for at least onPodNotReadyThreshold:
        #   none (default) — do nothing
        #   retry          — re-enqueue the CHI so the stuck host is force-restarted
        # OFF by default: force-recreating a Completed CHI's pod is destructive — it can
        # interrupt a replica's in-progress recovery and means hard downtime for a
        # single-replica shard. Opt in with `retry` only where that trade-off is acceptable.
        onPodNotReady: none
        # Minimum duration a pod must stay Ready=False before recovery fires, once enabled
        # (Go duration string; default 5m). Raise it for slow-recovering replicas.
        onPodNotReadyThreshold: 5m

Configure operator rootCA with a secret. Closes #1925

 clickhouse:
    access:
      scheme: https
      port: 8443
      rootCASecretRef:
        name: root-ca-secret
        key: ca.crt

Helm: Add podSecurityContext and podAnnotations to crdHook Job by @qlevasseur-genetec in #2009

Changed

Split Keeper services to internal and external ones. Closes #1982
Apply filter earlier when scraping metrics in order reduce memory usage by @dentiny in #2028. Closes #2019
Backward-incompatible config rename in ClickHouseOperatorConfiguration: reconcile.recovery.from.{aborted,completed} → reconcile.recovery.onStatus.{aborted,completed} (the from grouping level is removed; per-status action keys are unchanged, and the obsolete from key is silently ignored on load). If you set reconcile.recovery.from.aborted.onPodReady: none on 0.27.0/0.27.1 to disable Aborted auto-recovery, re-apply it as reconcile.recovery.onStatus.aborted.onPodReady: none — otherwise the default (retry) silently re-enables it on upgrade.

Security

Operator metrics port no longer exposes /debug/pprof. The operator's metrics HTTP listener (default :9999) served Go's shared http.DefaultServeMux, onto which net/http/pprof is transitively registered through a controller-runtime dependency — unintentionally exposing the /debug/pprof/* endpoints (including the CPU profile and trace handlers) to anything able to reach the operator's metrics Service, enabling unauthenticated profiling/DoS and heap/goroutine dumps. The listener now uses a dedicated mux that serves only the metrics endpoint; /metrics is unchanged.
Bumped versions of dependent libraries to address CVEs

Fixed

Add replica restart fix for operator when remote_servers configmap is propagated by @oo007 in #2024. Closes #2013
Escape reserved characters in element text by @AruneshDwivedi in #2034
Helm: Wire watchNamespaces value into operator ConfigMap by @jtomaszon in #2007. Closes #1919
Fixed incorrect watched namespace override #2008

FIPS Harness

Align FIPS test plan, requirements, and e2e FIPS coverage, fixed failing operator tests by @Elmo33 in #2011
Wire security.kubernetes.tls.minVersion into the Kubernetes API client transport by @Elmo33 in #2020
Expand FIPS TLS e2e coverage with host-run cipher probes and broader listener rejection checks by @Elmo33 in #2031

New Contributors

@oo007 made their first contribution in #2024
@jtomaszon made their first contribution in #2007
@dentiny made their first contribution in #2028
@miguel-signoz made their first contribution in #2033
@AruneshDwivedi made their first contribution in #2034

Full Changelog: release-0.27.1...release-0.27.2

@pkieszcz

NOTE: This is a major update to 0.27.0. Starting from 0.27.1 clickhouse-operator is FIPS-enabled

Added

secure/insecure flags to CHK in order to control ports by a single switch
FIPS-140 security hardening controls in operator configuration: https://github.com/Altinity/clickhouse-operator/blob/0.27.1/docs/security_hardening_fips.md
FIPS-140 reference configuration: https://github.com/Altinity/clickhouse-operator/blob/0.27.1/docs/fips_setup.md

Fixed

Fix permanent ArgoCD diff on resourceFieldRef.divisor by @pkieszcz in #1989
Fix inconsistent secret rendering in autogenerated remote_servers by @realyota in #1992. Closes #1991
Fixed skipped Delete when scale-to-0 Update fails by @dashashutosh80 in #1993. Closes #1990
Fixed .status.usedTemplates that were not properly cleaned when templates were removed
Bump dependent libraries to address CVEs

New Contributors

@pkieszcz made their first contribution in #1989

Full Changelog: release-0.27.0...release-0.27.1

@janeklb

Added

Reference CHK from CHI by a name rather than a service:

spec:
  configuration:
    zookeeper:
      keeper:
        name: my-keeper

[EXPERIMENTAL] Pre/post hooks. Allows to inject different SQL commands on reconcile events.
- supported events: Any, HostCreate, HostUpdate, HostStart, HostStop, HostConfigRestart, HostRollout, HostShutdown, HostDelete
- target controls where to run SQL: FirstHost (default), AllHosts, AllShards
- failurePolicy specifies what to do if hook execution fails with an error: Fail (default) or Ignore.
  See CRD for detailed description. Example:

  reconcile:
    host:
      hooks:
        pre:
          - sql:
              queries:
                - "SYSTEM STOP REPLICATION QUEUES"
            events:
              - HostShutdown

Now operator automatically restarts aborted reconcile if failed pod goes online after abort
Restart operator on operator configuration change. #1960. Closes #1930
Allow to exclude certain metrics from exporting. Noisy OS/CPU metrics are now excluded by default. #1975. Closes #1876
Exclusion rules are applied by default and can be changed in operator configuration:

    excludeMetricsRegexp:
      - "^metric\\.(OS.*CPU[0-9]+|CPUFrequencyMHz_[0-9]+)$"

Helm: Allow readiness and liveness probes for operator containers by @janeklb in #1976
Helm: Add security context for crdHook to enable security policy compliance by @qlevasseur-genetec in #1950

Changed

Enabled async_replication and use_xid_64 in Keeper default configuration. Requires Keeper 25.3 or above.
Whitelist Keeper four letter commands by default
Switched Keeper probes to ruok
Kubernetes client library has been upgraded to 0.30.14 by @dcoppa in #1970

Fixed

Fix watch ClickHouseOperatorConfiguration in operator namespace. #1959.
Fix chk probe crashloop by @hananbs in #1961. Closes #1962
Fix secret-backed env upgrade rollout. #1966. Closes #1963
Helm: fix configmap too long by @madrisan in #1949. Closes #1911
build(deps): bump go.opentelemetry.io/otel/sdk from 1.42.0 to 1.43.0 by @dependabot[bot] in #1953
Fix: format string issues by @destinyoooo in #1955
Fix: Prevent CrashLoopBackOff during image upgrade with RollingUpdate (#1926) by @dashashutosh80 in #1956. Closes #1926
Fixed a bug when host has not been removed from status fields hostsWithReplicaCaughtUp and hostsWithTablesCreated when removing from a cluster.
Prevent nil pointer panic in poll when CR Get fails by @wucm667 in #1974. Closes #1972
Scope schema discovery to target host's cluster in multi-cluster CHI by @lukas-pfannschmidt-tr in #1965. Closes #1964
Fix: use plain errors for clickhouse model messages in #1980 by @immanuwell

New Contributors

@qlevasseur-genetec made their first contribution in #1950
@destinyoooo made their first contribution in #1955
@hananbs made their first contribution in #1961
@wucm667 made their first contribution in in #1974
@immanuwell made their first contribution in #1980

Full Changelog: release-0.26.3...release-0.27.0

@dashashutosh80

Fixed

Prevent CrashLoopBackOff during image upgrade with RollingUpdate #1956 by @dashashutosh80. Closes #1926
Address CVEs in dependent libraries

Full Changelog: release-0.26.2...release-0.26.3

Fixed

Fixed a race condition when updating ClickHouse configuration and version altogether with a configuration setting that did not exist in the old version. Closes #1926
Fixed a bug when some Keeper nodes could be left offline after configuration changes
Fixed a bug when operator did not respect watched namespaces for Keeper. Closes #1923
Fixed potential races in configuration hash calculation. May close #1907
Updated dependent libraries to address CVE-2026-24051

Changed

Deleting multi-node CHI and CHK is now much faster
Added asynchronous_metrics_keeper_metrics_only to default Keeper configuration

Full Changelog: release-0.26.1...release-0.26.2]

@Slach

Fixed

Fixed Keeper startup that was slow due to missing quorum. Closes #1931 and #1856
Fixed Keeper deletion logic that could previously leave PVCs undeleted
Fixed hostName generation in status that might result in excessive schema propagation cycles
Fixed FQDN normalization to prevent trailing-dot inconsistencies between internal hostname representations
Bump stdlib version to address CVE
Bump base image version by @Slach in #1941. Closes #1940
Document custom service template behaviour for CHK by @realyota in #1939

Full Changelog: release-0.26.0...release-0.26.1

@lesandie

IMPORTANT: Due to ClickHouse upstream regression ClickHouse/ClickHouse#89693 DDL queries may not work on newly created ClickHouse pods. It affects Kubernetes deployments only in some new ClickHouse versions (25.8.10+ and above). The workaround is to restart ClickHouse pods. The problem is fixed by ClickHouse/ClickHouse#92339, see backports for different release branches. The fix is backported to Altinity Stable 25.8.16.10001 as well.
Closes #1883 and #1913

Added

Added an option to abort reconcile if STS needs to be recreated. It can be configured in operator configuration or CHI.

# Reconcile StatefulSet scenario
reconcile:
  statefulSet:
    recreate:
      # What to do in case operator is in need to recreate StatefulSet?
      # Possible options:
      # 1. abort - abort the process, do nothing with the problematic StatefulSet, leave it as it is,
      #    do not try to fix or delete or update it, just abort reconcile cycle.
      #    Do not proceed to the next StatefulSet(s) and wait for an admin to assist.
      # 2. recreate - proceed and recreate StatefulSet.
      # Triggered when StatefulSet update fails or StatefulSet is not ready
      onUpdateFailure: recreate

Added an option to configure system tables for metrics scrapping. The default is system.metrics and system.custom_metrics tables, but those can be changed with a regular expression if needed:

    tablesRegexp: "^(metrics|custom_metrics)$"

Changed

The suspend flag now immediately aborts a running reconcile. Previously, it did not affect the one that was running
When suspend flag is set, any reconcile attempt automatically sets CHI/CHK status to aborted.
Add optional registry prefix for operator and metrics images in Helm chart by @lesandie in #1928
Improve ClickHouse Keeper Grafana Dashboard by @discostur in #1872
Add CRDHook annotations by @eyyu in #1914
Hotfix crdhook, add imagePullSecrets by @Slach in #1917
Fix installer to default template URL to OPERATOR_VERSION by @realyota in #1910
sort keys in Settings.Keys() method for consistent order (fix manifest reconcile issue) by @mastercactapus in #1900
Multiple documentation fixes

Fixed

Fixed Keeper rolling update logic. Closes #1796 #1915
Fixed a bug when replica was not added to monitoring until it catches up the replication lag
Fixed version parsing for FIPS compatible builds of ClickHouse. Closes #1850
Fixed stop and suspend attributes for CHK that were previously ignored
Fix distributed_ddl.replicas_path mismatch that could prevent sharing (Zoo)Keeper between multiple clusters @Elmo33 in #1922
Fixed a bug when defaults.storageManagement.reclaimPolicy was not respected
Fixed slow initial connectivity to newly created pods caused by DNS search list exhaustion (ndots:5). Added trailing dot to FQDN and increased connect timeout
Fixed a bug where reconcile settings specified at CHI level (e.g. spec.reconcile.statefulSet.recreate.onUpdateFailure) were not inherited by cluster-level reconcile configuration

Other

stdlib has been upgraded to 1.25.6 to address CVEs
Operator has been certified for 25.8.16.10001 Altinity.Stable.

New Contributors

@rajdudhare1 made their first contribution in #1890
@siggy made their first contribution in #1906
@eyyu made their first contribution in #1914
@Elmo33 made their first contribution in #1922
@mastercactapus made their first contribution in #1900
@lesandie made their first contribution in #1928

Full Changelog: release-0.25.6...release-0.26.0

Changed

Hosts are not excluded from remote_servers anymore if restart is not needed. Previously, replicas in replicated cluster might be removed for a short time even if restart was not needed.
configuration.zookeeper section changes do not require restart anymore
Last reconciliation error and list of errors are now stored in CHI status

Fixed

actionPlan is now optional in status. That fixes operator upgrade problems that might happen in some environments.
Fixed excessive reconciles triggered by endpoint slices. Closes #1873
Fixed crash in CHK that might happen sometimes. Closes #1863
Fixed an issue with handling fractional requests/limits that would result in excessive reconcile. Closes #1849 #1821
Fixed a bug with operator crash in Terminating namespace. Closes #1871
stdlib was upgraded in order to address CVEs in dependent libraries

NOTE: Due to regression in upstream ClickHouse ClickHouse/ClickHouse#89693 schema propagation and DDL statements do not work with ClickHouse versions 25.8.10+ and newer until it is resolved.

Full Changelog: release-0.25.5...release-0.25.6

release-0.27.4

Security

Behavior Changes

Fixed

New Contributors

Contributors

Uh oh!

release-0.27.3

Added

Changed

Fixed

Security

New Contributors

Contributors

Uh oh!

release-0.27.2

Added

Changed

Security

Fixed

FIPS Harness

New Contributors

Contributors

Uh oh!

release-0.27.1

Added

Fixed

New Contributors

Contributors

Uh oh!

release-0.27.0

Added

Changed

Fixed

New Contributors

Contributors

Uh oh!

release-0.26.3

Fixed

Contributors

Uh oh!

release-0.26.2

Fixed

Changed

Uh oh!

release-0.26.1

Fixed

Contributors

Uh oh!

release-0.26.0

Added

Changed

Fixed

Other

New Contributors

Contributors

Uh oh!

release-0.25.6

Changed

Fixed

Uh oh!