8000 Various improvements to checksums by aadant · Pull Request #1139 · Altinity/clickhouse-sink-connector · GitHub
[go: up one dir, main page]
Skip to content

Various improvements to checksums - #1139

Merged
subkanthi merged 3 commits into
developfrom
checksum_improvements
Oct 14, 2025
Merged

Various improvements to checksums#1139
subkanthi merged 3 commits into
developfrom
checksum_improvements

Conversation

@aadant
@aadant aadant commented Sep 16, 2025 •
edited
Loading
Copy link
Copy Markdown
Collaborator

This PR introduced a new features and many bug fixes :

top_level_table_checksum.py

  • support for multiple databases
source:
    mysql:
        host: mysql1
        databases:
          - test
        ignored_columns:
            # https://github.com/Altinity/clickhouse-sink-connector/issues/1136
            - test.bit_table.bit1_column
           # https://github.com/Altinity/clickhouse-sink-connector/issues/1137
            - test.json_table.json_column

replicas:
    - clickhouse:
        host: clickhouse1
    - clickhouse:
        host: clickhouse2
  • --lock_tables_on_source locks tables on the source (it should be a replica not to block writes but only replication)
  • --sleep_after_lock sleeps after lock to wait for data to replicate

mysql_table_checksum.py

  • escapes table names
  • ability to exclude columns
  • attempts to format JSON columns (compact format)

clickhouse_table_checksum.py

  • fix checksums with NULL values in the first column
  • trim the trailing zeroes in DateTimes similar to MySQL checksums
  • support for is_deleted in the source table (previously it was ignored, leading to different checksums

@aadant
aadant requested a review from subkanthi September 16, 2025 22:31
@aadant
aadant requested a review from minguyen9988 September 17, 2025 00:31
@aadant aadant changed the title Various improvements in checksums Various improvements to checksums Sep 17, 2025
minguyen9988
minguyen9988 approved these changes Sep 17, 2025
View reviewed changes
@minguyen9988 minguyen9988 left a comment
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@subkanthi
subkanthi merged commit b412254 into develop Oct 14, 2025
@subkanthi
subkanthi deleted the checksum_improvements branch October 14, 2025 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@minguyen9988 minguyen9988 minguyen9988 approved these changes
@subkanthi subkanthi Awaiting requested review from subkanthi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@aadant @minguyen9988 @subkanthi
0