Repository navigation
fix(versioning): durable _version floor across restarts, GTID path floored by commit order, tagged GTIDs, basename change, SCD2 composite keys - #1478
Merged
minguyen9988 merged 8 commits intoSep 22, 2026
Conversation
…e high-water mark; floor the GTID version; keep heartbeats out of the sequence The lightweight engine's version floor (sequenceMaxSourceTs) was a process-local static. After a restart it started at 0, so the first rows of the new run were versioned in their own (lagging) source second and ranked BELOW the rows the previous run had clamped to a later second; ReplacingMergeTree kept the stale rows until a later update arrived, with row counts matching. The window was the replication lag plus the ~0.5 s counter carry, not the ~1 ms of the documented arithmetic example, because every heartbeat ran through nextSequenceNumber with its envelope ts_ms (the connector wall clock) and pinned the floor there. Three changes, all inside the 2.8.0 _version domain (formula, seeds, multiplier and snowflake encoding are untouched; only what feeds them changes): * Durable high-water mark (new VersionHighWaterMark, table replica_version_high_water in the offset database, keyed by the offset table name, no new config key). On the dispatch thread every version assigned to a row is covered by a persisted horizon BEFORE the row is handed to the writers; the horizon is moved 5 s of source time ahead when exceeded, so it is written about once per 5 s under load and never on an idle source. A horizon that cannot be persisted fails the batch loudly. Written ahead of handoff rather than on acknowledgement because a unit a worker has already written can be parked behind an older sequence, so an ack-time mark would not cover it. * seedVersionFloor at engine start: sequenceMaxSourceTs = floorDiv(Vmax, 1e6) + 1, so every first delivery of the new run exceeds every version of the old one. With no mark row (first start after an upgrade) the seed falls back to max(_version) over the ReplacingMergeTree targets of the include-list databases, decoded in both version domains with a plausibility bound. * Control records (heartbeat, transaction metadata) no longer enter the sequence; rows and DDL do. The GTID path now builds the snowflake from the clamped effectiveTs (ClickHouseStruct.versionTs, set next to the sequence number) instead of the raw statement time, so a late-committing long transaction wins under GTID too. Tests (fail on the pre-fix code): DebeziumChangeEventCaptureTest .newerEventAfterSeededRestartRanksAboveOlderPreRestartEvent (lagging source, heartbeat through handleChangeEventBatch, restart, seed), .newerEventOneMillisecondAfterSeededRestartRanksAboveOlderPreRestartEvent (the former knownDefect pin, deliberately flipped: the seeded floor clamps the newer event to T+1001), .heartbeatAndTransactionMetadataDoNotTouchTheSequenceState, .replayAfterSeededRestartIsClampedAboveTheOldRun, .seedVersionFloorRaisesButNeverLowersTheFloor; GtidLateCommitVersionTest (T2 ts110/gtid200/pos100 then T1 ts100/gtid201/pos200 -> version(T1) > version(T2)); VersionHighWaterMarkTest (table, write-ahead horizon, load, loud failure, domain decoding, target scan, include-list mapping); ClickHouseStructTest.versionTsFallsBackToTsMsWhenUnset. The narrative of CommitOrderVersionClampTest.unpositionedCounterResetRaisesTheFloor now names a positionless row (its assertions are unchanged); SequenceSeedOverflowTest keeps pinning the raw arithmetic, re-described as inherited. Specs: 02.02 (algorithm, control-record exclusion, restart seeding, corrected within-run bound), 02.04 (post-restart replays ARE clamped; contiguity argument; dedup is Kafka-only), 02.06 (upgrade/downgrade matrix, additive table, remaining first-start-without-seed window), 02.01 (GTID path uses versionTs; history-mode gaps recorded), 02.03, 01.04 (clock skew covered by seeding), 01.06, 09.03 (the table), 11.03, CONSTITUTION I11 and section 5. Lean: new Replication/VersionFloor.lean models the shipped statics and proves restart_boundary (seeding orders every first delivery of the new run above the old run), version_ge_floor, floor_mono, seed_floor_gt, dispatch_control_preserves_state with the pre-fix counterexample, and the regression scenario by decide; registered in the lakefile, README and CONSTITUTION. Zero sorry; axioms [propext, Quot.sound] only.
…flake.id=false with a data snapshot at ERROR ClickHouseStruct.setAdditionalMetaData parsed source.gtid only when it split into exactly two segments (uuid:n). A MySQL 8.3+ tagged GTID (uuid:tag:n) left gtid at -1, so every tagged transaction silently took the sequence-number version path (~1.8e18) while its untagged neighbours stayed in the snowflake domain (~2.1e18) and lost every merge against them. The transaction number is now the LAST colon-separated segment of any value with at least two segments; a colon-less value (MariaDB domain-server-seq) still leaves gtid unset. snowflake.id=false versions GTID rows with the raw transaction number (1e6-1e10) while snapshot rows take the sequence path (1.7e18), so after a data snapshot every streamed change of a snapshotted key is discarded. DebeziumChangeEventCapture.setup now evaluates rawGtidVersioningWithDataSnapshot(props, config) and logs the combination at ERROR (once per start, naming both keys, the consequence and the remediation) whenever snapshot.mode is not one of never/no_data/schema_only/recovery/ schema_only_recovery; an unset snapshot.mode is Debezium's default initial and counts as a data snapshot. It is not refused: under Invariant I11 an existing configuration must keep starting after an upgrade. Tests (fail on the pre-fix code): ClickHouseStructTest.taggedGtidIsParsed (uuid:tag:42 -> 42; classic form unchanged; colon-less stays -1), SnowflakeIdSnapshotWarningTest.rawGtidVersioningWithDataSnapshotIsLoud and .noDataSnapshotModesAreSilent (captured log events at ERROR). Specs: 02.01 section 3.1 (GTID text formats), 02.06 section 3.2.1 and its verification list.
…g basename changes
SourcePosition.compareTo orders by the file prefix first (a total order), so
after a log basename change -- log_bin reconfigured, a failover to a
differently named log, or a RESET MASTER followed by the engine's in-JVM
re-creation -- every position of the new log ranked entirely below the old
high-water mark by plain string order ("binlog" < "mysql-bin"). The version
sequence treated the whole new log as a redelivery: nothing was clamped to
the floor, and a late commit in the new log was versioned in its own older
second and lost to the earlier write of the same key.
SourcePosition.sameLog(other) exposes prefix equality; nextVersionAssignment
treats a positioned record from a differently named log as a first delivery,
logs the identity change once at WARN and replaces the mark with it. In-log
rewinds inside the new log are redeliveries again. A RESET MASTER that keeps
the basename restarts numbering at 000001 and remains indistinguishable from
an in-run rewind by position alone; recorded as a gap (a process restart
clears the mark).
Tests: SourcePositionTest.sameLogIsByFilePrefix;
CommitOrderVersionClampTest.binlogBasenameChangeResetsTheHighWaterMark (fails
on the pre-fix rule: the renamed log's first record compared below the mark
and was not clamped).
Specs: 01.02 section 3.1.1 (new), 02.02 section 3 / 3.1 / verification.
…d the row index as `sequence` on the Kafka Connect path Replication history mode (replication.history.enable) had two row-identity defects: (a) ReplicationHistoryHandler.buildUpdateQueryParams closed the previous history row with a predicate on the FIRST primary-key column only (getPrimaryKey().get(0)), so on a table whose primary key has several columns every history row sharing that column's value was closed together -- every line of an order when one line changed. UpdateQueryParams now carries the whole key (column -> value, in key order); QueryFormatter gains composite-key overloads of getInsertQueryForUpdate / getInsertQueryForDelete that build the conjunction `c1`=v1 AND `c2`=v2 (formatPrimaryKeyPredicate, each value formatted for its column type) in both table-reading SELECTs; the single-column signatures delegate to them. A record without a primary key cannot be closed and is refused with IllegalStateException naming the table instead of failing with an index error. (d) The history table's sorting key is (server_id, logfile, position, sequence, _time). The Kafka Connect path assigns no sequence number, so BinLogHistory bound the -1 sentinel for every row and the rows of one multi-row statement collapsed into a single row under ReplacingMergeTree. Without a sequence number the record's row index within its event is bound instead, which is unique per (logfile, position); rows that carry a sequence number are unchanged. Tests (fail on the pre-fix code): ReplicationHistoryHandlerTest.compositePrimaryKeyClosesOnlyTheMatchingRow, QueryFormatterTest.updateAndDeleteQueriesUseEveryPrimaryKeyColumn, BinLogHistoryTest.kafkaPathUsesRowIndexAsSequence. Specs: 02.01 section 3.5 (a) and (d) and its verification list; 04.04 verification. (b) history versions ignore versionTs and collide under gtid_mode=OFF, and (c) the _operation spelling mismatch, remain recorded gaps in 02.01 section 3.5.
…and that GTID same-millisecond ties rest on the FINAL tie rule Spec 10.05 gains section 3.4: DeDuplicator is constructed only in ClickHouseSinkTask.start() and consulted only in ClickHouseSinkTask.put(), no other class references it; the lightweight engine has no record-identity de-duplication of any kind, so lightweight replay safety rests entirely on version ordering (the in-run high-water gate of 02.04 section 3.2 and the seeded floor of 02.02 section 3.5), never on deduplication.policy. The Lean README's FINAL tie-rule note now records that the connector relies on the later-insert-wins rule beyond relocation tombstones: under GTID versioning every row event of one transaction in one millisecond carries the same _version (spec 02.01 section 3.1.1), so two writes to one key inside one transaction are ordered by insertion order alone, which per-table routing on one worker establishes and the model assumes but does not model. No code change.
….0 leaves untouched Spec 02.06 section 6.1 now opens its upgrade/downgrade matrix with the exact compatibility statement the governing rule demands: the formula (effectiveTs * 1_000_000 + counter), the 2.8.0 seeds and the snowflake bit layout are byte-identical to 2.8.0, 2.9.1 and 2.10.x (SnowFlakeId only gained two public constants for the startup seed), and the only changes are in what feeds the formula -- restart seeding of the floor, control-record exclusion, the log-basename reset of the high-water position, and the clamped effectiveTs on the GTID path -- each cross-referenced to its spec section. No code change.
…rsionAssignment parameter The version-floor change replaced the trailing `long sequenceNumber` parameter of DebeziumChangeEventCapture.processEveryChangeRecord with a VersionAssignment (sequence number plus clamped effectiveTs). Two existing tests reach that private method reflectively and still looked it up with `long.class`, so DdlFailureLoudTest (3 tests) and NullParsedRecordSkipTest (1 test) failed with NoSuchMethodException on the full lightweight unit run; the module-filtered runs of the earlier commits did not include them. Both seams now look the method up with DebeziumChangeEventCapture.VersionAssignment.class and pass new VersionAssignment(1000000001L, 1000L) -- the same version the tests always passed, paired with the effectiveTs it decodes to (1000 * 1e6 + 1). No assertion is changed; the tests' subjects (loud DDL failure, unparseable record skipped without an NPE) are untouched. No production code change.
…-commit-order Resolves eight conflicts of merging the current 2.11.0 head (now containing #1475/#1476/#1477) into the version-floor branch. Additive/kept-both in the Lean and spec files: both BatchOrder (#1476) and VersionFloor (this branch) listed in lakefile.lean, formal_specs/lean/README.md, CONSTITUTION.md, and spec 11.03 (numbered proof list renumbered so BatchOrder is 13 and VersionFloor 14-16); spec 04.04 keeps #1476's collapsing/grouping bullets and this branch's composite-PK bullet; spec 01.06 keeps #1477's terminal classifier plus this branch's heartbeat-sequence-state test. The one semantic conflict is the version-assignment hot loop of DebeziumChangeEventCapture. Both behaviours are kept: #1478's VersionAssignment machinery (nextVersionAssignment + coverAssignedVersion, control records excluded from the sequence, chStruct versioned from assignment.sequenceNumber/effectiveTs) AND #1477's loud-failure path (rejectUnrepresentableValue before the sequence; parse() wrapped so a parser throw becomes RecordReplicationException; a null-parsed ROW record is terminal, not a WARN skip; the RecordReplicationException/DDLReplicationException rethrows ahead of the catch-all). NullParsedRowRecordIsTerminalTest keeps its terminal assertion, adapted to the VersionAssignment reflective signature. Verification: lightweight compile clean; NullParsedRowRecordIsTerminalTest + ControlRecordLogLevelTest + DebeziumChangeEventCaptureTest + UnparseableRowRecordIsTerminalTest 38/38; spec validator PASSED (59 specs, 519 citations); lake build completed successfully (BatchOrder + VersionFloor both built, 0 sorry); full sink-connector unit suite 547 run, 0 failures, 20 errors all Docker-environment (identical to base). config.yml.j2 rendered (minimal + full variants) and YAML-linted with duplicate-key detection: no unrendered markers, no duplicate keys. All 42 destructive sites in this merge are #1475/#1476/#1477 code already carrying DESTRUCTIVE: markers or test-code literals (statement text parsed/classified/logged, no execution against any live database), unchanged by this merge. Jinja-Render-Check: rendered=2; formats=yaml; result=pass Destructive-Op-Check: sites=42; result=pass Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Collaborator
Author
|
Pushed cd0d264: merged the current 2.11.0 head (d1663de, which now contains #1475/#1476/#1477) into the branch to resolve the conflict. Eight conflicts. Seven were additive (kept both sides): the Lean lakefile/README, CONSTITUTION, and spec 11.03 list both BatchOrder (from #1476) and this branch's VersionFloor (the numbered proof list renumbered so BatchOrder is 13 and VersionFloor 14-16); spec 04.04 keeps #1476's collapsing/grouping bullets plus this branch's composite-PK bullet; spec 01.06 keeps #1477's terminal classifier plus this branch's heartbeat-sequence-state test. The one semantic conflict was the version-assignment hot loop of DebeziumChangeEventCapture, where both behaviours are preserved: this branch's VersionAssignment machinery (nextVersionAssignment + coverAssignedVersion, control records excluded from the sequence, rows versioned from the assignment) AND #1477's loud-failure path (rejectUnrepresentableValue before the sequence; a parser throw or a null-parsed row record becomes a terminal RecordReplicationException instead of a WARN-and-skip; the RecordReplicationException/DDLReplicationException rethrows ahead of the catch-all). NullParsedRowRecordIsTerminalTest keeps its terminal assertion, adapted to the VersionAssignment reflective signature. Verification: lightweight compile clean; the four affected unit tests 38/38; spec validator PASSED (59 specs, 519 citations); lake build completed successfully with both BatchOrder and VersionFloor and zero sorry; full sink-connector unit suite 547 run, 0 failures, 20 Docker-environment errors identical to base. CI is running on the merged head now. |
minguyen9988
deleted the
fix-version-floor-restart-and-gtid-commit-order
branch
September 22, 2026 09:35
minguyen9988
added a commit
that referenced
this pull request
Sep 22, 2026
…ypes; fix inherited DdlIgnoreRulesTest signature Resolves nine conflicts of merging the current 2.11.0 head (now containing #1475/#1476/#1477/#1478) into the ddl-translation branch. MySqlDDLParserListenerImpl.java: both sides added independent fields (this branch's PK-change/ddlEventTimestamp fields and #1477's dropOrTruncateStatement field + isDropOrTruncateStatement method) - kept both. Lean lakefile/README, CONSTITUTION and spec 11.03 list all of BatchOrder (#1476), VersionFloor (#1478) and this branch's CreateTable. spec 07.01 keeps this branch's DDL-path unsigned rule (§3.2) and #1478's record-path untyped-Int64 report (renumbered §3.3). spec 07.02 keeps this branch's IEEE section plus #1475's value-comparison caveat. spec 06.05 keeps both verification sets including #1478's RecordSchemaVsDdlTypeAgreementTest bullet. spec 07.06 (spatial) is a genuine two-path merge: the record-schema path keeps POINT/POLYGON native (#1475), the DDL path types every spatial column String/WKB-hex (this branch, new §3.4); they agree on every non-polygon type and the RecordSchemaVsDdlTypeAgreementTest does not cover spatial, so the code coexists. CI fix: 2.11.0 itself is internally inconsistent - #1478 changed processEveryChangeRecord's last parameter to VersionAssignment but DdlIgnoreRulesTest (added by #1477) still reflected long.class, so disableDropTruncateIsScopedAndLive and disableDropTruncateExemptsSnapshotDdl threw NoSuchMethodException. This is the java-tests-lightweight failure the branch inherits through the merge. Fixed the reflective lookup and call to use VersionAssignment (mirroring NullParsedRowRecordIsTerminalTest). The same one-line fix should be back-ported to 2.11.0 directly. Verification: spec validator PASSED (59 specs, 534 citations); lightweight compiles; DdlIgnoreRulesTest + MySqlDDLParserListenerImplTest + DataTypeConverterTest + CreateTableNoKeySortKeyTest + CreateTableUniqueKeySortKeyTest + NullParsedRowRecordIsTerminalTest 175/175 (1 Docker skip); full lightweight unit suite 556 run, 0 failures, 62 errors all Testcontainers/Docker (no daemon here). config.yml.j2 rendered (minimal + full) and YAML-linted, no duplicate keys. All destructive sites in this merge are #1475/#1476/#1477/#1478 code with DESTRUCTIVE: markers or test-code literals, unchanged by this merge. Jinja-Render-Check: rendered=2; formats=yaml; result=pass Destructive-Op-Check: sites=42; result=pass Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base: 2.11.0 at 5780b62 (after 1470-1474). Part of the replication-correctness audit; spec-first, failing-then-passing test and mutation check per finding; Lean theorem for the floor. Everything stays inside the 2.8.0-compatible _version domain: the formula, seeds and encoding are untouched, only what feeds them changes (spec 02.06 section 6.1 states the compatibility contract).
Verification: sink-connector unit suite 470 green (20 Docker-only errors, identical to base); lightweight unit suite green after the seam fix in the last commit; lake build clean; spec validator passes. No end-to-end upgrade test (2.10.x writes then 2.11.0 restart) could be run without Docker; it is recorded as a gap in spec 02.06.
Merge note: see fix-writer-value-fidelity for the suggested order of the six audit PRs; forward-merge 2.11.0 after each merge to clear the shared registration files.