Repository navigation
Reuse parts moved between local disks in --diff-from-remote, fix #1663 - #1667
Merged
Merged
Conversation
A part moved by TTL TO VOLUME or MOVE PARTITION keeps its name and hash_of_all_files, but create/upload looked it up only on the same disk of the diff backup, so every moved part was uploaded again. Look the part up on all disks of the diff backup, but reuse it only when both disks are plain local disks: object disks keep object metadata and encrypted disks keep ciphertext in the backup while hash_of_all_files is the same. 22.9+ reports type=Local for encrypted disks, so read system.disks.is_encrypted. The same plain-local filter now guards the cross-disk lookup in download, and restore of a local increment does not hardlink a required part from another disk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
replaceStorageDiskNameForReBalance renames disk_s3 only inside the s3 config files, a policy in a separate file kept referencing disk_s3 and clickhouse-server failed to start with UNKNOWN_DISK in TestS3 and TestAlibabaOverS3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
drwetter/testssl.sh moved to testssl/testssl.sh and the old URL no longer redirects for git, clone fails with "Remote branch v3.2rc3 not found" in every CI job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coverage Report for CI Build 42190604160Coverage increased (+0.07%) to 70.67%Details
Uncovered Changes
Coverage Regressions26 previously-covered lines in 5 files lost coverage.
Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1663
A part moved by
TTL ... TO VOLUME/ALTER TABLE ... MOVE PARTITION ... TO DISKkeeps its name andhash_of_all_files, butcreate --diff-from-remote/upload --diff-from-remotelooked it up only on the same disk of the diff backup, so every moved part was uploaded again.Changes
markDuplicatedParts(upload) anddiffTableMetadataForDisk(create) look a part up on all disks of the diff backup; the add check checksums during upload --diff-from / --diff-from-remote if data part name the same #1307hash_of_all_files/ checksums check is unchanged.hash_of_all_filesis the same, so reusing across disk types would corrupt restored data.type=Localfor an encrypted disk over a local one,GetDisksnow readssystem.disks.is_encrypted(Disk.IsEncrypted).findDiffBackupFilesRemote(download) skips other-disk candidates when only one of the disks is plain local (isCrossDiskDownloadIncompatible); a required disk absent insystem.disksis described bydisk_typesof the required backup, unknown type is not filtered.findRequiredPartLocal(restore) does not hardlink a required part stored on another disk, it is downloaded from remote.--diff-from(local) keeps same-disk only matching, its inode comparison never matches across disks.Tests
TestMarkDuplicatedPartsMovedBetweenDisks(fails on master),TestIsCrossDiskDownloadIncompatibleTestDiffFromRemoteMovedPartswith newmixed_disk_typespolicy (default,hdd1,disk_s3,hdd3_encrypted), moves partitions to each disk, checksrequiredflags and data after localrestoreandrestore_remote; passed locally on 26.8 and 22.8.🤖 Generated with Claude Code