Repository navigation
fix(python): compare the source table column set; detect replica-only columns and fail by default (Spec 11.02 sections 3.3, 3.9, FM-11.02-10) - #1557
Merged
Conversation
…mns are named, not hashed (Spec 11.02 sections 3.3, 3.9, FM-11.02-10) The replica side of db_compare hashed every column in system.columns, the source side every column in information_schema. A column added on ClickHouse alone (for example ADD COLUMN name Nullable(String) DEFAULT JSONExtractString(payload, 'Name')) was therefore hashed on one side only: the table reported DIFFERENT on every later run, with nothing pointing at the column, although every replicated value agreed. - driver: reads the source table's column names (db.mysql.mysql_column_names) and passes them to every replica side as --source_columns, a JSON list so a column name holding a comma or a space survives the trip; the source side is unchanged. - replica side: restrict_to_source_columns() keeps only the columns the source table also has (case-insensitive, as MySQL names columns), names replica-only columns in one coverage WARNING per table (relayed by the driver at INFO, like the float/JSON notes), and names source columns the replica lacks (those still report DIFFERENT: a real replication gap). - without --source_columns (standalone side runs) behaviour is unchanged. - spec 11.02: column set (3.3), coverage (3.9), new FM-11.02-10; FM-11.02-9 notes that history columns are now left out as replica-only. Verified on a production table whose replicas carry such a column: with the column set restricted, the replica-side aggregate equals the source checksum; without it, it reproduces the failing replica checksum. Tests: TestReplicaOnlyColumns, TestSourceColumnSetReachesTheReplicaSide, mutation-checked (keeping every column, or dropping the option from the replica command, turns them red). db_compare suite: 325 passed, 1 skipped. scripts/validate_specs.py: PASSED. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…; --allow_replica_only_columns accepts them (Spec 11.02 section 3.9, FM-11.02-10) Adding columns on the ClickHouse destination is not supported practice: the replica must conform to the source. The checksum therefore detects such columns instead of only tolerating them. - replica side: the report is a distinct, parseable line, "Replica-only columns in table db.t: [...]". - driver: record_replica_only() parses it per (replica host, table). Default: one "WARNING REPLICA-ONLY COLUMNS -- host db.t: [...]" naming the remedy (stop adding columns on the destination; drop them there or add them to the source table), a summary WARNING at the end of the run, and exit code 1. With --allow_replica_only_columns the columns are logged at INFO and the exit code is unaffected. - either way the columns stay out of the row string, so the table verdict is about replicated columns only (no false "Checksum difference"). - a column in the job ignored_columns is excluded before the check: the reviewed per-column acceptance. Tests: TestSourceColumnSetReachesTheReplicaSide (default fails with the finding and keeps "No difference"; the flag keeps it green; no carry-over between runs), TestScheduledJobLogContract replica-only cases through main() with the job flags. Mutation-checked: forcing the flag on fails the two default tests, ignoring it fails the two flag tests. db_compare suite: 330 passed, 1 skipped; validate_specs.py PASSED. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions can
2DDC
not be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The replica side of db_compare hashes every column in system.columns, the source side every column in information_schema.columns. A column added on ClickHouse only (for example ADD COLUMN name Nullable(String) DEFAULT JSONExtractString(payload, Name)) is hashed on one side only, so the table reports a checksum difference on every later run, with nothing pointing at the column, even though every replicated value agrees. Adding columns on the destination is also not good practice: the replica must conform to the source, so such columns should be detected and reported, not silently tolerated. We hit this on a production table whose replicas gained such a column.
Change (Spec 11.02 section 3.3 column set, section 3.9, new FM-11.02-10)
Verification
Whole-Change-Verification: ran=cd sink-connector/python and python -m pytest db_compare/tests -q (330 passed, 1 skipped) plus python3 scripts/validate_specs.py (PASSED), both on this branch head on top of 2.11.0 befda7b; result=pass
Opened on behalf of our database team; the maintainers own the review and the merge.