Releases

v2.8.3

NEW FEATURES

support tables with local and plain_rewritable disks in one storage policy: ClickHouse rejects FREEZE for them, so create hardlinks the local parts into shadow the way FREEZE does and copies the parts of the plain disks from the bucket, with merges and moves of the table stopped (needs SYSTEM MERGES and SYSTEM MOVES grants, moves are now stopped for every table on a storage policy with plain disks), plain disks in skip_disks are skipped, clickhouse.disk_mapping is applied to the part paths, sync_replicated_tables, freeze_by_part_where and ignore_not_exists_error_during_freeze apply to such tables as they apply to FREEZE, parts on object disks with local metadata (including cache and encrypted wrappers over them) next to plain disks are rejected, default restore of such tables requires ClickHouse 25.4+ (older versions reject ATTACH PART for a storage policy with plain disks, use restore_as_attach there)
support Antalya metadata_type=cas disks with use_embedded_backup_restore: true, create_remote no longer fails with disk "cas" ... is not exists locally and restore_remote no longer fails with unsupported <metadata_type>cas</metadata_type>, without embedded mode a table on a CAS disk fails with an explicit error, because FREEZE on CAS leaves no local files to hardlink, fix #1562

IMPROVEMENTS

rebase copies parts, archives and object disk blobs of one table in parallel, previously every CopyObject inside a table ran strictly one by one and general.rebase_concurrency limited only the number of tables, now it also limits parallel CopyObject requests per table, so up to rebase_concurrency^2 copies are in flight, rebase logs the required backups chain, rebase_concurrency and max_copy_object on start and rebase table start / rebase table finish with progress, duration, copied_size and object_disk_size for each table at info level, fix #1648
update go dependencies: aws-sdk-go-v2, aws-sdk-go-v2/feature/s3/transfermanager 0.5.1, cloud.google.com/go/storage 1.70.0, clickhouse-go/v2 2.50.0, klauspost/pgzip 1.3.0, pkg/sftp 1.13.12, cos-go-sdk-v5 0.7.76, google.golang.org/api 0.309.0 and others

BUG FIXES

restore and restore_remote no longer drop and recreate every user-defined SQL function from the backup, previously each function got DROP FUNCTION IF EXISTS + CREATE OR REPLACE FUNCTION (function was unavailable between them) even with --tables for a single table, and silently rolled back definitions changed after the backup; now existing functions are read from system.functions first, a missing function is created with CREATE FUNCTION IF NOT EXISTS, an identical one is skipped without any query (so CREATE FUNCTION / DROP FUNCTION grants are not required), a changed one is kept with a warning and overwritten by atomic CREATE OR REPLACE FUNCTION only with --rm, the same as tables and RBAC objects; functions are restored before tables, so tables and materialized views which use UDF can be created on a clean server; restore --dry-run and restore_remote --dry-run show functions with create, replace, keep_existing or skip_identical for each function, fix #1636
FIPS builds no longer overwrite an explicit AWS_USE_FIPS_ENDPOINT, clickhouse-backup-fips set it to true unconditionally at startup, so with a custom s3.endpoint (MinIO, Ceph RGW, other S3-compatible storage) every S3 call failed with S3 ResolveEndpoint: endpoint rule error, A custom endpoint cannot be combined with FIPS and AWS_USE_FIPS_ENDPOINT=false had no effect; it is still true by default when unset, the variable only selects AWS s3-fips.* hostnames and does not affect the FIPS 140-3 crypto module
s3.allow_multipart_download now uses byte-range GETs by default, new s3.multipart_download_type (env S3_MULTIPART_DOWNLOAD_TYPE, default range): range requests byte ranges of chunk_size with If-Match set to the multipart object ETag, part keeps the previous partNumber GETs, which send the first part ETag as If-Match and get HTTP 412 PreconditionFailed on Ceph RGW (Hetzner Object Storage) from part 2 on (each part has its own ETag); the setting applies only when multipart download actually runs (allow_multipart_download: true and download_max_bytes_per_second: 0), on other S3-compatible storage (NuObject) part-number downloads failed with unexpected EOF, fix #1632, #1655
rebase with remote_storage: azblob no longer fails with BlobNotFound for databases and tables whose names need encoding, such as test-db or the .inner_id. table of a materialized view, the server-side copy source URL is escaped now, fix #1637
add s3.timeout (env S3_TIMEOUT, default 30m, empty or 0 disables it) as HTTP response header timeout for S3 requests including object disk connections, previously S3 had no per-request timeout and create_remote could hang forever in CopyObject / UploadPartCopy when S3 accepted the request and never answered, a timed out request is retried on a new connection; timeout already present in configs was silently ignored before, it accepts a duration (30m) or a number of seconds (3000), fix #1645
download of a regular (not embedded) backup no longer fails with Chown backup disk: lstat .../<backup>: no such file or directory when running as root with use_embedded_backup_restore: true and clickhouse.embedded_backup_disk configured, fix #1653
fix --link-dest quoting in the rsync upload example (test/integration/rsync/upload.sh), rsync reported --link-dest arg does not exist and copied every file instead of hardlinking unchanged parts, rsync calls in upload / download examples use --protect-args

@minguyen9988

v2.8.2

NEW FEATURES

add clickhouse.skip_restore_non_sharded_replicated_tables (env CLICKHOUSE_SKIP_RESTORE_NON_SHARDED_REPLICATED_TABLES, default false), clickhouse.skip_check_non_sharded_replicated_tables_cluster (env CLICKHOUSE_SKIP_CHECK_NON_SHARDED_REPLICATED_TABLES_CLUSTER, default {cluster}) and clickhouse.always_sharded_macros (env CLICKHOUSE_ALWAYS_SHARDED_MACROS, default ["shard"]), when true restore, restore_remote and restore --dry-run restore the data of non-sharded replicated tables (ZooKeeper path in the backup DDL resolves to the same value on every shard, macro values are read with cluster(..., system.macros), {database}, {table}, {uuid} are the same for the table everywhere) only on the first replica of the first shard of the cluster from system.clusters, every other node restores only their schema and gets the data through replication, so the same config works on every node, a macro listed as sharded with the same value on all shards is reported with a warning, such tables are never rebound to default_replica_path, not supported with use_embedded_backup_restore and restore_cloud, fix #1181
use_embedded_backup_restore_cluster now orchestrates the whole cluster instead of only adding ON CLUSTER to the BACKUP / RESTORE SQL: create, create_remote, restore, restore_remote on the initiator node send the same command with the new --embedded-on-cluster-worker flag to every other node of the cluster through its system.backup_actions table (clickhouse-backup server with api.create_integration_tables: true is required on every node) and wait for them, a worker skips the BACKUP / RESTORE SQL and only describes, uploads, downloads and fixes its own shards/N/replicas/M/ part of the shared destination, per table metadata/<db>/<table>.json moved under shards/N/replicas/M/ next to the .sql of each node (such backups are tagged embedded,cluster=<name>, older backups without the tag keep the flat layout), .backup, metadata.json, rbac, configs and named collections stay on the initiator, delete local is propagated to the other nodes as delete local --embedded-on-cluster-worker, so delete accepts the flag too, a single node cluster behaves as before, fix #928
add azblob.chunk_size (env AZBLOB_CHUNK_SIZE, default 0), a fixed upload block size used when ceil(file size / chunk_size) <= max_parts_count, clamped to [2MiB, 4000MiB], 0 keeps the automatic file size / max_parts_count clamped to [2MiB, 10MiB], brings back the explicit size removed together with AZBLOB_BUFFER_SIZE in v2.6.6, fix #1142
back up and restore Kafka offsets of Kafka engine tables with kafka_keeper_path / kafka_replica_name (StorageKafka2, ClickHouse 24.8+), previously only the schema was backed up and the restored table re-read from the consumer group offset (lost rows) or from earliest (duplicates); create detaches such tables before FREEZE so the offsets match the frozen data of the MV target (DETACH PERMANENTLY in ENGINE=Replicated databases), attaches them back after FREEZE (also on error and cancel) and stores the Keeper subtree without ephemeral nodes in kafka_keeper field of metadata/<db>/<table>.json; restore writes the offsets to Keeper right before CREATE (after the --rm DROP), when the path already exists without active replicas the offsets are overwritten with a warning, when it has active replicas they are left untouched with a warning; --restore-database-mapping / --restore-table-mapping rewrite kafka_keeper_path, a mapped restore with the same kafka_group_name next to a live source on ClickHouse < 25.6 fails before any Keeper write, because consumer group rebalance would steal the partitions of the source; between restore --schema and restore --data (Altinity ACM restore sequence) consumption is held by persistent partition lock nodes, which the data phase removes, add --skip-kafka-keeper-hold to restore / restore_remote (REST skip_kafka_keeper_hold) to disable it, embedded backups only log a warning, fix #1120
list remote and restore_remote auto-detect native ClickHouse Cloud backups (BACKUP ... TO S3 / AzureBlobStorage with <name>/.backup and without metadata.json) under s3.path / azblob.path, such backups are listed with data_format: cloud and size instead of broken (can't stat metadata.json), clean_remote_broken and backups_to_keep_remote retention no longer delete them, restore_remote <name> (CLI and POST /backup/restore_remote/{name}) switches to restore_cloud with --tables, --partitions, --skip-empty-tables, --rm, --dry-run and restore_schema_on_cluster support, options which restore_cloud can't apply and download fail explicitly; the base backup chain of incremental cloud backups is resolved from <base_backup> of each .backup (verified by base_backup_uuid) and passed to RESTORE with the configured credentials instead of the ones embedded in the manifest, a base in the same bucket and path is shown as +<base> in list remote and protected by the delete remote guard, fix #1574
add general.rbac_restore_always, general.config_restore_always, general.named_collections_restore_always (env RBAC_RESTORE_ALWAYS, CONFIG_RESTORE_ALWAYS, NAMED_COLLECTIONS_RESTORE_ALWAYS, default false), restore-side counterparts of *_backup_always, act as --rbac / --configs / --named-collections for restore and restore_remote (including --streaming), disabled by default because these restores overwrite existing objects and run restart_command
add general.download_transfer_concurrency (env DOWNLOAD_TRANSFER_CONCURRENCY, default 0 = download_concurrency^2, negative keeps the legacy per-table limit), one transfer budget per download shared by every part, archive and required part of all tables, so a single big table can use all the connections at the tail of a download, for FTP the budget is capped by 2*ftp.concurrency; add general.download_table_order (env DOWNLOAD_TABLE_ORDER, largest_first (default) or metadata), only the download dispatch order changes, restore order is not affected, fix #1591
add general.file_transfer_concurrency (env FILE_TRANSFER_CONCURRENCY, default 1), with compression_format: none files of one data part are transferred by several streams which only take slots of the shared download_transfer_concurrency (download) / upload_concurrency^2 (upload) budget left idle by other parts, so the total number of transfers never grows above the budget, fix #1454
add gcs.ranged_download_concurrency (env GCS_RANGED_DOWNLOAD_CONCURRENCY, default 0 = disabled), gcs.ranged_download_min_size (default 64MiB), gcs.ranged_download_chunk_size (default 32MiB) and gcs.ranged_download_max_buffers (default 64), archives larger than ranged_download_min_size are read by concurrent byte range requests and fed to decompression in order, without a temporary file, memory is capped process-wide by ranged_download_max_buffers * ranged_download_chunk_size, works with download_max_bytes_per_second, thanks @minguyen9988, fix #1592

IMPROVEMENTS

GCS: all pooled clients share one cached OAuth2 token, previously each client fetched its own token, so a cold delete remote with gcs.delete_concurrency: 50 opened about 50 parallel TLS handshakes to oauth2.googleapis.com and could fail with auth: cannot fetch token: ... TLS handshake timeout, and every retry after a delete error repeated it
download with clickhouse.force_rebalance spreads parts round-robin across local JBOD disks whose free space differs by less than 1% of total space (typically several disks on the same filesystem), previously least_used picked the disk with the largest free space from statvfs noise and all parts of a download could land on a single disk
Docker image: base image changed from alpine:3.23 to alpine:latest and pre-installed packages are upgraded with apk upgrade, the alpine:3.23 base shipped libcrypto3 / libssl3 3.5.7-r0 with CVE-2026-63073 and CVE-2026-75803, thanks @dfdemar, fix #1588
update go dependencies: aws-sdk-go-v2, azure-sdk-for-go, cloud.google.com/go/storage 1.69.0, clickhouse-go/v2 2.49.0, urfave/cli/v3 3.14.0, golang.org/x/crypto 0.58.0, klauspost/compress 1.20.3 and others

BUG FIXES

the SQL query log no longer prints passwords embedded in URLs: pass= / password= query string parameters and user:password@ are written as [MASKED], with api.create_integration_tables: true and api.password set every server start logged the DDL of system.backup_actions, system.backup_list and system.backup_version with the API password in plain text inside the URL() engine argument, at info level with the default clickhouse.log_sql_queries: true, the query sent to ClickHouse is unchanged, fix #1630
s3.chunk_size is now also used when ceil(file size / chunk_size) is exactly s3.max_parts_count, previously it fell back to file size / max_parts_count, ReadMe.md now shows the real default 5242880 and the semantics of s3.chunk_size, raise it to upload big files as fewer bigger parts instead of max_parts_count=1 workaround, each multipart upload allocates about (s3.concurrency + 2) * chunk_size bytes, s3.chunk_size ...

v2.8.1

NEW FEATURES

add clickhouse.drop_replica_if_exists (env CLICKHOUSE_DROP_REPLICA_IF_EXISTS, CLI --drop-replica-if-exists for restore/restore_remote, also accepted as the drop_replica_if_exists query argument by POST /backup/restore and POST /backup/restore_remote) — when the replica entry <replica_path>/replicas/<replica_name> of a restored Replicated*MergeTree still exists in ZooKeeper but no local table uses that path, execute SYSTEM DROP REPLICA '<replica_name>' FROM ZKPATH '<replica_path>' and restore the table with the ORIGINAL replication path from the backup, instead of the default behavior which silently rebinds the table to clickhouse.default_replica_path / clickhouse.default_replica_name; requires ClickHouse 20.4+, the restore fails when the drop is impossible (older server, or a local table still occupies the path), the statement is executed without ON CLUSTER because the path and replica name are resolved with the macros of the current node; MUST stay false during a concurrent multi-replica restore, fix #1162
add --streaming flag to create_remote, restore_remote and watch, --watch-streaming flag to server and the streaming query argument to POST /backup/create_remote, POST /backup/restore_remote and POST /backup/watch — process the backup table by table: create_remote --streaming runs freeze -> upload -> remove local copy per table, restore_remote --streaming runs download -> attach -> remove local copy per table, so only the tables in flight occupy local disk space and backups bigger than the free local space can be uploaded or restored; the local backup is not kept afterwards; interrupted runs are continued with --resume via the dedicated create_upload_streaming.state2 / download_restore_streaming.state2 state files (add these command names to api.complete_resumable_after_restart_commands to resume them after an API server restart); not available with use_embedded_backup_restore: true or remote_storage: custom, fix #780
add restore_cloud command and POST /backup/restore_cloud REST API endpoint — restore a ClickHouse Cloud native S3 backup (created via external backup / Bring Your Own Backup) onto self-managed ClickHouse: reads the .backup manifest directly from S3 (including data_file_name_generator: Checksum blob layout), rewrites ENGINE = Shared databases to Atomic and Shared*MergeTree tables to the matching Replicated*MergeTree (adding --replicated-zk-path/--replicated-replica arguments when Cloud DDL has none), applies the DDL and runs RESTORE TABLE ... FROM S3(...) / AzureBlobStorage(...) with allow_different_database_def=1, allow_different_table_def=1; source storage is S3-compatible (AWS, GCS over s3->endpoint: https://storage.googleapis.com with HMAC keys, MinIO) or AzureBlobStorage (when --container / --azblob-restore-url is passed or general->remote_storage: azblob); supports --tables filter, --base-prefix for incremental backups, --s3-restore-url / --azblob-restore-url overrides, --skip-empty-tables, --continue-on-error, --drop, --parallel and --dry-run; note: backups containing Packed parts (data.packed) require ClickHouse 26.8+ on the target server, fix #1508
add --dry-run flag to create, create_remote, upload, download, restore, restore_remote, delete commands and the dry-run (dry_run) query argument to the matching REST API endpoints — instead of performing the operation, report the number of tables, parts and the data size (split into local-disk and object-disk bytes, human readable in the log) which would be processed, honoring --tables, --partitions, --diff-from/--diff-from-remote, --schema and the other filters; create --dry-run additionally forecasts the disk space the backup hardlinks will occupy over time (bytes held in parts younger than 1/7/30 days approximate what background merges will rewrite within that horizon, up to the full local data size in the worst case) and delete --dry-run reports the dependent incremental backups; API dry-run requests execute synchronously, are allowed even when another operation is in progress regardless of api.allow_parallel, and return the report as JSON in the response body; every dry-run also fills the new result field of its status row, so the report is visible via GET /backup/status, GET /backup/actions and the system.backup_actions table (which gained a result String column, manually created tables need it added or input_format_skip_unknown_fields=1); estimation limits: create doesn't subtract --diff-from-remote increments, create_remote/upload can't know the compressed size before compression, parts of backups made by versions older than 2.7.0 have no per-part sizes and are counted in unknown_size_parts, fix #1012
add general.callback_url (env CALLBACK_URL) and general.callback_timeout (env CALLBACK_TIMEOUT, default 5s) — HTTP POST completion notification for API, one-shot CLI commands, and each watch iteration; API ?callback= overrides the global URL when non-empty; the payload keeps the existing status/error/operation_id fields and adds command and duration; status is now also cancel when the operation was killed via /backup/kill; callbacks are sent asynchronously, failures are logged and never change the backup result; one-shot CLI commands and watch iterations are now registered in /backup/status like API operations, fix #1481
add general.status_history_size (env STATUS_HISTORY_SIZE, default 1000) — upper bound on how many finished operations are kept in the in-memory status list exposed by /backup/status and system.backup_actions; needed because watch now records one operation per iteration, so the history would otherwise grow for as long as the process lives; operations still running are never dropped, whatever their age
delete local|remote <backup_name> and POST /backup/delete/{where}/{name} now refuse to delete a backup which other backups require via required_backup and report the dependent backup names, instead of silently breaking the incremental backups chain (the breakage surfaced only later, when a descendant was downloaded or restored, and for object disks the descendant required parts blobs were deleted together with the parent); pass --force (force=1 for the API) to get the old behavior, or set general.rebase_during_delete: true (env REBASE_DURING_DELETE, default false) to rebase every dependent increment first (same as the rebase command) so the chain stays restorable — rebase copies the deleted backup parts into its dependents, so deletion time grows with the copied data size and a rebase failure aborts the delete. backups_to_keep_local/backups_to_keep_remote retention is not affected, fix #1493
restore --rbac now converts RBAC objects between local_directory and replicated user directories, when the target server has only one of the two shapes: a Keeper dump from the backup is written as <access_control_path>/<uuid>.sql files when the target has no replicated user directory, and backup *.sql files are written to Keeper (<zookeeper_path>/uuid/<uuid> plus the <zookeeper_path>/<type char>/<name> index znode) when the target has no local_directory user directory; previously the half which had no matching storage was silently lost while the restore still reported success; restoring a backup with RBAC objects onto a server which has neither user directory now fails explicitly, fix #881

IMPROVEMENTS

support tables which declare their storage inline in the DDL via SETTINGS disk = disk(...) instead of a storage_policy (ClickHouse 23.2+, tested from 24.8), including nested cache and encrypted wrappers over an object storage disk; such disks are registered by ClickHouse under a generated __tmp_internal_<hash> name (or under name = '...') and are absent from preprocessed_configs/config.xml, so create/upload/download/restore used to fail with __tmp_internal_... is not present in object_disk.DisksCredentials; object storage credentials are now taken from the table DDL and the target disk is resolved again after the schema is restored, so the generated name may differ between the source and the target server. Note that the DDL stored in the backup metadata contains the secret_access_key in plain text, same as for S3/MySQL engine tables (#640), and that clickhouse-backup needs to run on the ClickHouse host (to read /var/lib/clickhouse/metadata/<db>/<table>.sql) or the server needs display_secrets_in_show_and_select=1, because SHOW CREATE TABLE masks disk(...) arguments as '[HIDDEN]', fix #943
fail fast instead of burning the whole retries_on_failure x retries_duration backoff budget per file when a remote object is permanently missing (S3 NoSuchKey/404, GCS 404, Azure BlobNotFound, FTP/SFTP not-found) during download, restore_remote and other retried remote operations; add general.allow_missing_files_on_download (env ALLOW_MISSING_FILES_ON_DOWNLOAD, default false), --allow-missing-files CLI flag for download/restore_remote and the allow_missing_files query argument for POST /backup/download and POST /backup/restore_remote — salvage mode which skips data parts missing on remote storage with an error-level log and a ...

v2.8.0

NEW FEATURES

add rebase command and POST /backup/rebase/{name} API endpoint — copy required parts from the required_backup chain into a remote incremental backup via server-side CopyObject (with streaming fallback) and remove the required_backup dependency, so the incremental backup becomes a full one without re-uploading data from the ClickHouse host; per-table parallelism is controlled by general.rebase_concurrency (env REBASE_CONCURRENCY, default = download_concurrency); requires backups made with upload_by_part: true and the same compression_format across the chain, fix #1344, #1444
add general.rebase_before_remove_old_remote (env REBASE_BEFORE_REMOVE_OLD_REMOTE, default false) — makes backups_to_keep_remote a strict limit: when deletion of old remote backups is blocked by required_backup links from kept backups, the oldest kept increment is rebased first (same as the rebase command) so the whole out-of-window chain becomes deletable; rebase failure is not fatal and falls back to the legacy keep-required behavior
add --schedule to watch command, general.watch_schedules config section (env WATCH_SCHEDULES, ;-separated) and the schedule query argument to POST /backup/watch — named cron-driven backup chains in name=<name>,full=<cron>[,increment=<cron>][,full_type=create|rebase][,delete_previous_cycle=true|false] format (standard 5-field cron, optional leading seconds field, @every/@daily descriptors), can be specified multiple times, mutually exclusive with --watch-interval/--full-interval; name is added as a prefix to watch_backup_name_template to isolate backup chains, full_type=rebase creates the scheduled full backup as increment + rebase (server-side copy of the previous chain instead of a full re-upload), delete_previous_cycle=true deletes all older backups of the chain after a successful full backup, fix #1354
add --list-parts and --partitions (--list-partitions) flags to tables command — additionally list every physical part (name, partition_id, size) or the distinct partitions (partition_id, partition, parts count, size) for each table; against the live server the data comes from system.parts, against --local-backup/--remote-backup part names come from backup metadata (partition_id derived from the part name), fix #959, #1333
add --rbac, --configs, --named-collections flags to server --watch — the server command never passed them through, so watch mode backed up only data even though the standalone watch command and the REST API endpoints already supported them, fix #955
add rebalance command and POST /backup/rebalance/{name} API endpoint (with --tables and --dry-run support) — move data parts inside a local backup between disks: hardlink from the live part when system.parts shows it on another disk; parts absent from system.parts are copied to the storage-policy disk of the same type with the most free space (unreserved_space - keep_free_space from system.disks), parts on a disk which is absent or left the policy always move; parts on object disks are skipped, fix #1024
add gcs.allow_multipart_upload (env GCS_ALLOW_MULTIPART_UPLOAD, default false, experimental) with gcs.upload_concurrency and gcs.multipart_upload_min_size (default 1GB) — files bigger than multipart_upload_min_size upload to GCS as multiple parallel parts (part size is chunk_size) via gRPC client and compose into the final object; not compatible with endpoint, force_http, encryption_key, fix #1028
add gcs.allow_multipart_download (env GCS_ALLOW_MULTIPART_DOWNLOAD, default false) with gcs.download_concurrency — download each file as parallel range reads (part size is chunk_size) into a temporary file, mirrors s3.allow_multipart_download, fix #1028
add general.disable_environment_override (settable ONLY in the config file, no environment variable name on purpose, default false) — when true, config values come only from the config file: all environment variables and the --env CLI flag are ignored during config loading; protects against accidental overrides such as Kubernetes service-discovery variables, fix #1079

IMPROVEMENTS

write a gzip-compressed bbolt file manifest manifest.bolt.gz during upload and read it during download/restore to eliminate the per-part Walk (ListObjectsV2 and analogs) calls to remote storage that dominated restore time on backups with many parts; memory stays bounded even with millions of files (batched writes on upload, mmap-backed prefix seek on read), with graceful fallback to Walk when the manifest is missing or unreadable, and manifest failures are non-fatal for the backup, fix #1375, #1380
store per-part size in table metadata during create and check the required free disk space right after table metadata download, before downloading any data — the check respects --tables/--partitions, resolves sizes of required parts through the required-backup chain, and when --hardlink-exists-files is used doesn't count parts which can be hardlinked from existing local parts (matched by hash_of_all_files via system.parts or by CRC64 of checksums.txt), fix #1268
fail fast on canceled context in the retrier — SIGTERM / POST /backup/kill no longer trigger a useless extra retry attempt with a misleading Will wait and retry warning per goroutine; during API server shutdown log which in-flight command and cancel_operation_timeout the server is waiting for instead of blocking silently

BUG FIXES

fix silent truncation of compressed backup uploads to S3 when s3.check_sum_algorithm: CRC32 (Object Lock buckets): the multipart upload stopped at the declared size (sum of raw file sizes), dropping the tar headers/padding/trailer tail of the archive; the stream is now consumed until EOF, fix #1471
fix rbac_conflict_resolution: ignore during restore --rbac — RBAC objects from the backup that already exist on the server are now really ignored: their access/*.sql files are not copied over the live ones and the correlated replicated uuid znodes and name-to-uuid mappings are not restored into Keeper, fix #1013

v2.7.4

NEW FEATURES

add clickhouse.parts_columns_max_bytes_before_external_group_by (env CLICKHOUSE_PARTS_COLUMNS_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY, default 100000000) and clickhouse.parts_columns_max_memory_usage (env CLICKHOUSE_PARTS_COLUMNS_MAX_MEMORY_USAGE, default 200000000) — make the memory caps injected into the system.parts_columns check query configurable instead of hardcoded, 0 means don't inject the corresponding setting (pre-2.7.2 behavior), fix #1420

BUG FIXES

fix --restore-database-mapping/--restore-table-mapping losing unrelated comma-separated --tables pattern items, or applying the mapping to the wrong item when the same source database/table name was repeated across items — each comma-separated pattern item is now matched and substituted independently, fix #1421

v2.7.3

NEW FEATURES

add general.max_broken_part_ratio (env MAX_BROKEN_PART_RATIO, default 0) — when >0, create/create_remote skips broken data parts instead of aborting the whole backup as long as the broken/total part ratio stays at or below the configured fraction; skipped parts are logged as warnings and the backup is marked successful but partial; 0 preserves the legacy any-broken-part-fails behavior, fix #1418
add clickhouse.rebind_replica_path_if_exists (env CLICKHOUSE_REBIND_REPLICA_PATH_IF_EXISTS, CLI --rebind-replica-path-if-exists, also accepted by the REST API POST /backup/restore endpoint) — opt-in rebind of the replicated table ZK path to default_replica_path during restore when the resolved path already has leftover children but no local table uses it (async-stale state after DROP); MUST stay false during a concurrent multi-replica restore to avoid split-brain, fix #849
add gcs.disable_http2 (env GCS_DISABLE_HTTP2) — force HTTP/1.1 for the GCS client to avoid single-TCP-connection HTTP/2 multiplexing becoming the bottleneck for high-throughput parallel downloads, fix #1437

IMPROVEMENTS

switch the S3 client retry mode from aws.RetryModeStandard to aws.RetryModeAdaptive, so upload/download retries back off with a client-side rate limiter under throttling (SlowDown/503) instead of a fixed token bucket
support tables with ENGINE=Alias during restore — strip the explicit column list (Alias inherits columns from the target table) and add allow_experimental_alias_table_engine=1 for ClickHouse 25.11+, fix #1426
skip tables from databases with engine DataLakeCatalog (read-only external catalogs) during backup, in addition to MySQL/PostgreSQL/MaterializedPostgreSQL, fix #1417
speed up create on servers with many tables: batch the in-progress system.mutations lookup instead of issuing one query per table (O(N^2) -> O(N)), fix #1431
speed up restore/download table list resolution: eliminate the O(N^2) linear duplicate scan in getTableListByPatternLocal/getTableListByPatternRemote via a shared O(1) dedup keyed on metadata.TableTitle, fix #1430
extend TestFlows FIPS 140-3 test coverage: dedicated test steps, stronger --fips-info/version-output and TLS-handshake assertions, updated requirements documentation, fix #1443

BUG FIXES

restore the default general.compression_use_multi_thread to true: #1378 defaulted it to false, which silently dropped gzip/zstd compression from multi-threaded (pre-1378 gzip always used pgzip) to single-threaded and caused ~30% slower upload throughput for backups dominated by one large table (where upload_concurrency provides no per-stream parallelism); the option is now silently ignored instead of failing config validation for formats other than gzip/zstd
fix v2.7.1 regression where restore --schema on a second replica rebound the ZK replica path even while the first replica was active and healthy — a foreign/renamed local table occupying the resolved ZK path is still detected automatically via system.replicas and rebound to default_replica_path, but a non-empty ZK path without a local table now defaults to the HA-safe join and requires the new opt-in rebind_replica_path_if_exists, fix #1428
fix inconsistent --rbac behavior: restore --rbac no longer returns an error when the backup contains no RBAC access directory, matching the download --rbac behavior, fix #1432
inject ON CLUSTER for LIVE VIEW and WINDOW VIEW restored with explicit column lists when restore_schema_on_cluster is set — enrichQueryWithOnCluster missed these statements and the views were created only on the local replica, fix #1433
mask sensitive values (passwords, storage credentials, encryption keys) when logging --env config overrides, and in the API server action logs and /backup/status output, fix #1441

v2.7.2

IMPROVEMENTS

include the data part file path and source object key (fPath, srcKey) in object_disk CopyObject/CopyObjectStreaming error messages during create and restore, so broken object_disk data (keys missing on remote storage) points to the exact failing file instead of just the shadow/table name
document GCS Workload Identity authentication for the gcs remote storage in Examples.md

BUG FIXES

fix clickhouse.skip_table_engines (env CLICKHOUSE_SKIP_TABLE_ENGINES) silently keeping some matching tables: the in-place slice removal advanced the cursor past the next element, so adjacent tables sharing a skipped engine were not all skipped; iterate in reverse so every match is dropped, fix #1416
ensure /backup/kill (and context cancellation in general) promptly aborts an in-flight download/restore even when a read is stalled on a slow/half-open network or disk backpressure — the source reader is now force-closed on cancellation so blocked Read calls in DownloadCompressedStream/DownloadPath, the Azure CopyObject poll backoff, and object_disk.CopyObjectStreaming return instead of running to completion, and the resumable .pid file is removed, fix #1365

v2.7.1

NEW FEATURES

add --fips-info app-level flag — prints binary name, version, git commit, build date, Go version, and the FIPS module build/runtime state (GOFIPS140 build setting, GODEBUG fips140 default/runtime) then exits, without requiring a Go toolchain, fix #1402
add Azure AD Workload Identity support for azblob — when AZBLOB_USE_MANAGED_IDENTITY=true and the AZURE_TENANT_ID/AZURE_CLIENT_ID/AZURE_FEDERATED_TOKEN_FILE env vars are injected (e.g. by the AAD Workload Identity webhook), the federated token is used to authenticate; see Examples.md for deployment, fix #1124

IMPROVEMENTS

add general.compression_use_multi_thread (env COMPRESSION_USE_MULTI_THREAD, default false), general.compression_threads (env COMPRESSION_THREADS, default 0 = auto/GOMAXPROCS) and general.compression_buffer_size (env COMPRESSION_BUFFER_SIZE, default 0) config options to tune per-stream zstd/gzip threading and the compression buffer (zstd encoder window / gzip DEFLATE window / pgzip block size); per-stream compression is now single-threaded by default to avoid CPU over-subscription, since upload_concurrency/download_concurrency already parallelize across tables, fix #1378
migrate compression from the archived, frozen github.com/mholt/archiver/v4 v4.0.0-alpha.8 to its maintained successor github.com/mholt/archives v0.1.5, removing the replace directive pinned in archiver#428
migrate S3 storage from the deprecated github.com/aws/aws-sdk-go-v2/feature/s3/manager to github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager; the forced CRC32 aws-chunked trailer that broke non-AWS S3-compatible providers is now disabled via RequestChecksumCalculation=WhenRequired, and the obsolete s3.buffer_size / S3_BUFFER_SIZE option was removed, fix #1409
add buffer-size and HTTP-transport tuning for high-bandwidth (10Gbps+) S3/GCS transfers: general.pipe_buffer_size (env PIPE_BUFFER_SIZE, default 128KB), general.download_copy_buffer_size (env DOWNLOAD_COPY_BUFFER_SIZE), gcs.upload_buffer_size (env GCS_UPLOAD_BUFFER_SIZE E870), s3.http_write_buffer_size (env S3_HTTP_WRITE_BUFFER_SIZE) and s3.http_read_buffer_size (env S3_HTTP_READ_BUFFER_SIZE), fix #1376
replace post-hoc sleep-based bandwidth throttling with a token-bucket rate limiter wrapped around the storage Reader/Writer interfaces, so the configured limit is enforced continuously during transfer instead of after each chunk, fix #934, #1377
parallelize ALTER TABLE ... UNFREEZE after backup create instead of running it inline inside each table goroutine, so an UNFREEZE no longer holds an upload_concurrency slot and blocks the next table, fix #1381
add clickhouse.parts_columns_batch_size (env CLICKHOUSE_PARTS_COLUMNS_BATCH_SIZE, default 25) to batch the system.parts lookups when computing hash_of_all_files, avoiding Max query size exceeded failures on tables with very many parts, fix #1408
resolve required data parts during restore the same way as during download — hardlink from the required backup on local disk when present, otherwise download the part from remote storage, fix #1023
ResumeOperationsAfterRestart now ignores create.state2/restore.state2 files instead of failing the API server startup with unknown command; only upload and download are auto-resumed after a server restart, fix #1083
fail fast with a clear error instead of retrying for ~35s when a remote table metadata .json is missing during download (covers S3 NoSuchKey, GCS 404, Azure BlobNotFound, FTP/SFTP not-found) — a missing table file is a permanent broken-backup condition, not a transient one, fix #1379
emit a clear error on --resume download when the local backup exists but download.state2 is missing (so it is unknown which parts are complete), instead of crashing or silently resuming on top of partial data, fix #1383
harden FIPS 140-3 verification: native Go GOFIPS140=v1.0.0 checks, ACVP reproducibility tests, outbound S3 TLS rejection checks and container cleanup in CI/CD, fix #1399, #1401, #1404

BUG FIXES

don't kill clickhouse-backup server with Fatal/os.Exit when the resumable state DB can't be written or read (e.g. no space left on device); the error now propagates so the server stays alive and returns it to the API client, while the CLI exits with a non-zero code, fix #1172
fix backup create failing with part "<name>" not found in system.parts ... after FREEZE when a ClickHouse cache disk (e.g. s3_cache) wraps an underlying S3 object disk — prefer the underlying disk name over the cache wrapper in getDisksFromSystemDisks, fix #1396
fix --hardlink-exists-files to also match parts whose hash_of_all_files is identical but that now live under a renamed table, fix #1398
fix --table combined with --resume on incremental backups: recursively downloading a required backup closed the parent b.dst connection and wiped the resumable state; the connection is now saved/restored, fix #1384
improve detection when clickhouse-backup runs on a host whose disks differ from clickhouse-server, instead of silently warning doesn't contain tables for restore, fix #1037

v2.7.0

NEW FEATURES

add clean_broken_retention CLI command — walks top-level of remote path and object_disks_path and batch-deletes (with retry) every entry that is not present in the live backup list and not matched by any --exclude=<glob> (and optionally scoped by --include=<glob>). Dry-run by default; pass --commit to actually delete. Useful for cleaning up orphans left by failed retention runs, fix #1371
add info CLI command for per-table backup size breakdown — shows per-table size, part count, and disk breakdown for local and remote backups, supports --tables=<db>.<table> glob filter and --format=text|json|yaml|csv|tsv, accepts all|local|remote scope, fix #1388
add force_rebalance config option (clickhouse.force_rebalance, env CLICKHOUSE_FORCE_REBALANCE) — distribute restored data across multiple JBOD disks under the same storage policy even when the source disk name (e.g. default) exists on the target machine, fix #1350
switch FIPS variant from FIPS 140-2 boringssl to native Go 1.24+ FIPS 140-3 (GODEBUG=fips140=on); embed an ACVP wrapper into the shipped clickhouse-backup-fips binary with dual entry points (clickhouse-backup-acvp argv0 dispatch and clickhouse-backup acvp subcommand) and ship a tracked public-scope ACVP reproducibility flow, fix #1341, #1364, #1391, #1395
add safety check to restore/restore_remote: fail without --rm/--drop when target tables already exist and contain rows (checked via clusterAllReplicas('{cluster}') when restore_schema_on_cluster is set) to avoid dangerous accidental DROP TABLE, fix #1325
add checksum verification during upload --diff-from / --diff-from-remote when part name matches, to avoid uploading mismatched data and to detect silent corruption, fix #1307

IMPROVEMENTS

speed up restore_remote from S3 incremental chains: cache backup list and avoid redundant ListObjects calls per table (previously 8h on 280GB / 3500 tables shrinks to minutes), fix #1362, #1361
reduce backup memory footprint for databases with thousands of tables (regression introduced in v2.6.42), fix #1360
wrap S3 credentials with aws.NewCredentialsCache() to avoid resolving credentials on every API call (IMDS/STS), reducing latency and throttling in IRSA + AssumeRole flows, fix #1335
simplify hash_of_all_files computation via a single post-FREEZE SELECT from system.parts instead of per-file hashing — also enables --hardlinks-exists-files to consult system.parts checksums during download, fix #1338
isolate FREEZE shadow directory per backup as /var/lib/clickhouse/shadow/backup-{uuid} so concurrent backups and cleanup-after-failure don't clobber each other's shadow data, fix #1345
add option to skip persisting list calls into the API server actions state — prevents unbounded growth of actions state when /backup/list is used as a monitoring endpoint during long-running backups, fix #1359
improve kill command to ensure all in-flight operations really finish and to remove leftover .pid files, fix #1365
document missing/incorrect concurrency defaults in ReadMe.md (download_concurrency, s3.concurrency, cos.concurrency, sftp.concurrency, ftp.concurrency), fix #1346
migrate integration tests to testcontainers-go for better parallelism and isolation, fix #1336
fix the list_duration log field formatting in pkg/storage/general.go (was emitting raw nanoseconds), fix #1337

BUG FIXES

fix restore_remote for tables using sparse-column serialization: accept empty sparse metadata files instead of treating StorageObjectCount=0 as corruption, affects ClickHouse 23.8+, fix #1372
fix restore_remote aborting the entire restore when an incremental backup contains a table absent from the required full backup; the missing table is now skipped with a warning, fix #1373
fix object_disk backup on S3 sources with SSE-C: handle 404 from server-side CopyObject by falling back to streaming and stop issuing HeadObject (returns 400) on SSE-C source objects before GetObject, fix #1374
fix --rbac-only backup failing with "is empty backup" when the database contains RBAC objects but no tables and allow_empty_backups=false, fix #1355
fix nested ssh consuming stdin from the while read loop in the rsync helper (use ssh -n) so all backup metadata files are processed instead of only the first, fix #1368
fix backup retention logic in the rsync helper: correct line counting, numeric comparison and arithmetic handling so old backups are properly cleaned up, fix #1369

v2.6.44

BUG FIXES

fix 1356, retry batch deletion in cleanBackupObjectDisks so transient errors (e.g. GCS 503) during retention no longer leave orphaned objects in object_disks_path

2.8.3

v2.8.3

Uh oh!

2.8.2

v2.8.2

Contributors

Uh oh!

2.8.1

v2.8.1

Uh oh!

2.8.0

v2.8.0

Uh oh!

2.7.4

v2.7.4

Uh oh!

2.7.3

v2.7.3

Uh oh!

2.7.2

v2.7.2

Uh oh!

2.7.1

v2.7.1

Uh oh!

2.7.0

v2.7.0

Uh oh!

2.6.44

v2.6.44

Uh oh!