Repository navigation
fix(history): make the two auxiliary modes match their specification -- SCD2 write protocol, table shape, log-only parity, DDL routing, same-transaction version floor (Domain 12, Spec 02.02 s3.1.2) - #1511
Conversation
…g-only, with Lean model
Two of the three operating modes had no single specification: per-table SCD2
history (replication.history.enable=true) was scattered over 04.04 s3.2,
02.01 s3.5, 08.05, 06.05, 06.09 and 05.02, and replication-log-only
(replication.history.replication_log_only=true) had none at all. This adds
Domain 12 and a Lean module that follow the 2.11.0 implementation exactly and
record its defects as gaps with machine-checked witnesses, so they can be
debugged at the spec level before the code changes. No src/main change.
specs/12-replication-history/
01-operating-modes.md enable x log_only matrix, per-batch writes,
database routing (every table -> history db),
startup, the degenerate combination
02-scd2-table-shape.md columns, sentinel 2100-01-01, engine,
PARTITION BY toDate(_valid_to),
ORDER BY (pk, _valid_to), TTL; record-path
vs DDL-path differences D-1..D-3
03-scd2-write-protocol.md INSERT binding, UPDATE 3-SELECT and DELETE
2-SELECT statements, versions, key predicate,
inline UPDATE vs flushed DELETE; gaps
G-12.03-1..6
04-binlog-history-audit-table.md the 19-column audit table: DDL, per-column
values, DML and DDL rows, TTL
05-replication-log-only.md audit rows only, DDL translated not
executed, what the offset certifies,
single-threaded parity gap
formal_specs/lean/Replication/History.lean (new lakefile root)
SCD2 row sets over the FINAL view at (pk, _valid_to):
update_supersedes_open_row, delete_hides_open_row, closed_row_continuity,
at_most_one_live_row_per_sort_key; gap witnesses
staged_insert_writes_no_closed_row (UPDATE runs inline without the flush the
DELETE path performs), closed_history_row_depends_on_insert_order (close row
and before copy tie on sorting key and version),
update_only_closes_after_image_key (key predicate from the after image);
log-only gating and routing on both executors (writer_path_ignores_log_only,
routing_diverges_on_log_only_alone). lake build clean, no sorry.
Registration: specs/README.md, CONSTITUTION.md s4/s5, 11.03 module list,
04.04 s3.2 pointer, formal_specs/lean/README.md, validate_specs.py
EXPECTED_DOMAINS (+12-replication-history).
Verification: scripts/validate_specs.py --lake PASSED (65 specs, 622
citations, lake build exit 0); scripts/tests 39 passed.
…-- SCD2 write protocol, table shape, log-only parity, DDL routing, same-transaction version floor (Domain 12, Spec 02.02 s3.1.2) The end-to-end history suite (three connectors side by side on one source -- standard, SCD2 history, replication-log-only -- then a kill -9 restart, the single-threaded variants and the degenerate flag combination) and the specs of Domain 12 exposed the gaps this change closes. No standard-mode behaviour changes except the version-floor fix, which the suite proved to lose a DELETE in standard mode as well. SCD2 write protocol (Spec 12.03) - UPDATE now flushes the rows staged on the shared statement before its inline history statement runs, exactly as DELETE did: a row created and updated in one batch keeps its pre-update version in the history (G-12.03-1). - The deleted before-copy at (pk, ts) is gone: it tied with the close row on sorting key and version and could hide the closed version under FINAL (G-12.03-2). A key-changing UPDATE writes a delete marker at the OLD key sentinel, and the close predicate uses the BEFORE image key (G-12.03-3). - Every row of one event carries the record standard version (no SnowFlake of ts_ms/gtid, no V+1), refused loudly when underivable (G-12.03-4). - Unknown columns in history mode raise StaleSchemaCacheException again; only the connector history columns are deferred (G-12.03-5). - A source truncation or table drop becomes a bulk close of every open row plus a marker per row (T / D), on the op=t record path AND on the DDL path (Debezium skips t events by default, so this is how a truncation arrives); snapshot-phase statements close nothing (G-12.03-6). Database-level DDL is ignored in history mode: a mode-2 connector snapshot used to drop the database a standard connector on the same ClickHouse was replicating into. Table shape (Spec 12.02) - Both creation paths emit _valid_from and _valid_to as DateTime[(tz)] DEFAULT sentinel and TTL from replication.history.ttl (D-1..D-3); the DDL path appends _valid_to on every ORDER BY branch as a flat tuple (a,b,_valid_to) instead of ((a,b),_valid_to) (G-12.02-3, G-12.02-4). Modes and routing (Specs 12.01, 12.04, 12.05) - ClickHouseBatchWriter skips data tables in log-only mode and routes on enable alone, like ClickHouseBatchRunnable (G-12.05-1, G-12.01-1). - History database / audit-table creation failure and the DDL audit-row failure are loud (G-12.01-2, G-12.04-1); banner and config doc strings name the mode (G-12.01-3, G-12.05-2); ReplicationLogOnlyIT asserts fixed. Version floor (Spec 02.02 s3.1.2) - Debezium stamps every row event of a MySQL transaction with the transaction position, so the rows after the first compared EQUAL to the high-water mark and were never floored: a late-committing INSERT+UPDATE+DELETE versioned the INSERT above its own DELETE and the deleted row stayed live (standard and history mode alike). Rows at the mark position are now floored at the effective timestamp the transaction first row received. Specs and Lean: Domain 12 rewritten to the corrected design with the old behaviour kept as old_* counterexamples; Replication.History reworked (update_supersedes_open_row, key_change_retires_old_key, closed_row_visible_at_close_key, bulk_close_hides_every_open_row, both_engines_skip_data_in_log_only, database_ddl_ignored_in_history_mode, ...); 02.01 s3.5, 02.02 s3.1.2, 04.04 s3.2, CONSTITUTION s5 updated. Verification: scripts/validate_specs.py --lake PASSED (68 specs, 746 citations); sink-connector units 617 run (the 1 failure / 30 errors are the pre-existing Docker-dependent tests and the Metrics registry ordering NPE, identical on the baseline: 601 run / 1 / 28); lightweight units MySqlDDLParserListenerImplTest 162, CommitOrderVersionClampTest 15, PrimaryKeyRebuildTest 29, DebeziumChangeEventCaptureTest 27 -- 0 failures; E2E on the built jar: history suite 109/28 (baseline) -> 147/1, the one remaining check being a pre-existing transient (JDBC addBatch NPE after an ADD COLUMN, self-healed by the retry); main-mode suites unchanged (7/1 t_geo, 25/1 t_twice, both pre-existing). Destructive-Op-Check: sites=25; result=pass
…odes Upstream squash-merged the Domain 12 specification as 7396c3c (Altinity#1507) and then landed Altinity#1508-Altinity#1510; the ten add/add conflicts were the squash artefact (upstream copies byte-identical to the spec commit) and were resolved to the corrected specs and Lean model of this branch. The Java merges were automatic and orthogonal (ResumeReplayLogSummary hooks in DebeziumChangeEventCapture, writer-connection close in ClickHouseBatchRunnable). Both modules compile; the history/DDL/clamp unit suites and the resume-replay and close-connection suites are green; validate_specs.py --lake passes (68 specs, 748 citations). Destructive-Op-Check: sites=0; result=pass
|
Rebased on 2.11.0 after #1507 was squash-merged and #1508-#1510 landed (merge commit fc93aa1; the spec-file conflicts were the squash artefact and resolve to this branch's corrected specs and Lean model; the Java merges were automatic and orthogonal). The PR now carries only the fix: 35 files, +3042/-686. End-to-end results on the merged head (local containers, ClickHouse 24.8 / MySQL 8.0, lightweight jar built from fc93aa1):
Unit: lightweight history/DDL/clamp and resume-replay suites 304 run, 0 failures; sink-connector history suites green; validate_specs.py --lake passes (68 specs, 748 citations). |
|
Upgrade / downgrade audit of this PR: a regression was found, fixed on this branch, and is now covered by an end-to-end upgrade/downgrade suite. Finding. The earlier head (fc93aa1) bound the raw standard version as _version for every history row. Releases up to 2.11.0 wrote the UPDATE close row with SnowFlakeId(ts_ms, gtid) and the after row / delete marker with that value + 1 (about 2.1e18 in 2026) while binding the INSERT row with the standard version. On a source without GTIDs (and on any source with snowflake.id=false) the standard version is the lightweight sequence ts_ms * 10^6 + counter (about 1.8e18), and FINAL keeps the greatest _version at a sorting key, so every open row or delete marker a 2.11.0 connector had left at (k, S) outranked everything the new build could write until 2032: after the upgrade, an UPDATE of any previously-updated key entered the history but never became current, a key change left the old key live, and a re-insert after a 2.11.0 delete stayed hidden. Standard mode was never affected; GTID sources with snowflake.id=true were never affected; downgrade was always safe. Reproduced by a new end-to-end upgrade/downgrade suite: three connectors (standard, SCD2 history, replication-log-only) hop 2.11.0 -> candidate -> 2.11.0 -> candidate on the same tables, offsets and schema history, a workload after every hop touches the keys the other build last wrote, every check is value-level against the source, and the run ends with a kill -9 restart on the candidate. Baseline (OLD = 2.11.0 at a1d51fa, NEW = fc93aa1, GTID off): 88 PASS / 13 FAIL, with source != FINAL current state right after the first hop. Fix (55a3fd8 + 0079ce0, Spec 12.03 section 3.5.1, S10): one version domain per SCD2 table. Every row -- the INSERT binding, close/after rows, key-change and delete markers, and the bulk close on both the record and the DDL path -- carries the snowflake encoding of the event's ordering key, which is the domain 2.11.0 already used for its UPDATE/DELETE rows: GTID rows as (versionTs | ts_ms, gtid) (the standard version itself with snowflake.id=true), sequence rows as (effectiveTs - 1, counter - seed). The first iteration decoded a sequence row as (seq / 10^6, seq mod 10^6) and broke GTID sources on the snapshot -> CDC boundary (the ten-digit counter seeds carry 500 ms into the quotient, and the first CDC rows are floored to the snapshot's last millisecond); the history suite caught it (31 FAIL) and 0079ce0 fixes it. Lean: snowflake_encode_strict_mono, sequence_row_below_gtid_row_of_its_millisecond, legacy_open_row_superseded_by_later_event, fixed_open_row_superseded_by_later_legacy_event, and the old_* witness old_raw_sequence_version_loses_to_legacy_open_row. Unit: ReplicationHistoryVersionDomainTest (12). Results on 0079ce0 (local containers, ClickHouse 24.8 / MySQL 8.0):
Behaviour changes to be aware of after an upgrade (by design, not regressions): history mode refuses to start when the audit table cannot be created and no longer swallows a failing audit insert; an SCD2 table with a target-only column that is not a history/engine column now raises StaleSchemaCacheException like standard mode instead of NULL-filling it; a source TRUNCATE / DROP TABLE in history mode closes rows instead of truncating/dropping the SCD2 table, and database-level DDL is ignored; the record-path CREATE TABLE honours replication.history.ttl (default 30, previously hard-coded 30) and declares both validity columns with the sentinel default (existing tables untouched); the single-threaded writer with replication_log_only=true but enable=false now behaves as standard mode, matching the multithreaded engine. |
Stacked on #1507 (the Domain 12 specs); this PR adds one commit on top of it. Once #1507 merges the diff of this PR is the fix commit alone.
Domain 12 specified the two auxiliary modes as implemented in 2.11.0 and recorded their gaps. This change closes every recorded gap so that the specification, the Lean model and the code agree, then verifies the result end to end with three connectors running side by side on one MySQL source (standard, SCD2 history, replication-log-only), a kill -9 restart of the history connector, the single-threaded variants of both auxiliary modes and the degenerate flag combination. The auxiliary modes are independent of the standard mode: every change is gated on the history flags, except the version-floor fix, which the same suite proved to lose a DELETE in standard mode as well.
SCD2 write protocol (Spec 12.03)
Table shape (Spec 12.02)
Modes and routing (Specs 12.01, 12.04, 12.05)
Version floor (Spec 02.02 s3.1.2) -- affects the standard mode too
Debezium stamps every row event of a MySQL transaction with the transaction binlog position, so the rows after the first compared EQUAL to the high-water mark and were never floored: a late-committing transaction that inserted, updated and deleted one key versioned its INSERT above its own UPDATE and DELETE, and the deleted row stayed live on the replica (reproduced in standard mode: 3 rows on ClickHouse versus 2 on MySQL). Rows at the mark position are now floored at the effective timestamp the transaction first row received. CommitOrderVersionClampTest.rowsOfOneTransactionShareTheFirstRowsFloor pins it.
Specs and Lean
Domain 12 rewritten to the corrected design with the old behaviour kept as old_* counterexamples; Replication.History reworked (update_supersedes_open_row, update_closes_before_image_key, key_change_retires_old_key, key_change_opens_new_key, closed_row_visible_at_close_key, bulk_close_hides_every_open_row, bulk_close_preserves_history, both_engines_skip_data_in_log_only, both_engines_route_alike, database_ddl_ignored_in_history_mode, plus the old_ witnesses); 02.01 s3.5, 02.02 s3.1.2, 04.04 s3.2 and the CONSTITUTION s5 bullet updated. lake build clean, no sorry.
Verification
Known limitations now stated in Spec 12.03 s3.9