Repository navigation
fix(binlog): replicate compressed transactions correctly and verify the payload decoder at start (Spec 01.08) - #1531
Merged
Conversation
…he payload decoder at start (Spec 01.08) MySQL binlog_transaction_compression=ON (8.0.20+) writes each transaction as ONE zstd-compressed Transaction_payload_event. The binlog client (mysql-binlog-connector-java 0.40.2, zstd-jni shaded into the lightweight jar) decodes it and Debezium 3.1.3 dispatches the inner events directly, so every row of every statement of the transaction reaches the connector with the payload event position and a row index that restarts at 0 per rows event. Defect fixed: the version sequence classified a record against its high-water mark with SourcePosition.compareTo, which includes the row index. The second and later statements of a compressed transaction (and the later rows-event chunks of one large statement) therefore ranked BELOW the mark and fell into the un-clamped redelivery branch while the first statement rows had been floored: an UPDATE or DELETE of a key the same transaction inserted was versioned below the INSERT -- the update lost, the deleted row live, row counts intact. Reproduced end to end on the 2.11.0 head in steady state and after a restart, in both GTID modes (the GTID path versions with the same clamped timestamp). - SourcePosition.sameLogPosition: same log, file and byte position, any row index. - nextVersionAssignment: the rest of the marked event is recognised by that predicate BEFORE the above/below comparison and floored at what the event first row received; a redelivery from an earlier event keeps the un-floored assignment (spec 02.02 3.1.2). - BinlogTransactionCompressionPreflight (new): reads the source binlog_transaction_compression / level / version with one read-only SELECT, self-tests the Transaction_payload decoder on a synthetic zstd payload (one XID event) through the real deserializer, logs the result and exports three gauges. New key binlog.transaction.compression.check = auto (default) | require | skip: auto refuses only when the source is ON and the decoder does not work; require refuses unless the source is ON; skip runs nothing. - Spec 01.08 (new) with 01.02 3.3, 02.02 3.1.2 and 01.05 revised; Lean VersionFloor models the same-event rule (same_event_rows_share_floor, compressed_transaction_example, compressed_transaction_inverts_without_same_event_rule; lake build clean). - tests/e2e/binlog_transaction_compression.sh: MySQL 8.0 with compression on, GTID on and off, value-level comparison; steady-state compressed transactions incl. a 300k-row single statement, session-level toggle, DDL then DML, keyless; kill -9 with the transaction written before the restart under a mark-seeded and a clock-seeded floor; kill -9 mid-payload of a 400k-row transaction; source-side Transaction_payload proof. Verification: CommitOrderVersionClampTest 18/18 (3 new; the old predicate fails 2 of them), SourcePositionTest 15/15, BinlogTransactionCompressionPreflightTest 15/15 (mutation-checked), validate_specs PASSED (73 specs, 841 citations), e2e: pre-fix build 8/26 (GTID off) and 12/26 (GTID on) passing; fixed build 26/26 GTID off, 26/26 GTID on with check=require, 26/26 compression-off control.
…on unreadable source plus failed decoder Review findings (panel round 1): the e2e harness recorded failures in its counter but always exited 0 after cleanup, so a gate using it could accept a broken run -- cleanup now runs from an EXIT trap and the exit status is the verdict (0 only when FAIL=0 and at least one check ran). The configuration row for binlog.transaction.compression.check described the auto-mode case "source unreadable AND decoder self-test failed" as a warning; the code and spec 01.08 section 3.4 refuse it, and the row now says so.
Collaborator
Author
|
Full module suites on head e385a4e: sink-connector 670 run / 0 failures / 0 errors / 3 skipped (BUILD SUCCESS). sink-connector-lightweight (unit plus testcontainers ITs) is still running; totals will be added here when it finishes. e385a4e also makes the e2e harness exit non-zero when any check failed (cleanup from an EXIT trap) and aligns the doc/configuration.md row for auto mode (an unreadable source together with a failed decoder self-test is a refusal, as in spec 01.08 section 3.4). |
Collaborator
Author
|
Full lightweight module suite (unit plus testcontainers ITs, 118 test classes) on head e385a4e: Tests run 707, Failures 0, Errors 0, Skipped 4 -- BUILD SUCCESS (1 h 46 min). The core module was 670 run / 0 failures / 0 errors / 3 skipped. Both suites green. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Support for MySQL binlog transaction compression (binlog_transaction_compression=ON, MySQL 8.0.20+): the capability verified end to end, a correctness defect fixed, start-up detection added, spec first.
What MySQL does
With the variable ON each transaction is written as ONE zstd-compressed Transaction_payload_event (the GTID event stays outside; BEGIN, TABLE_MAPs, rows events and XID are inside), and the stream can be mixed (dynamic GLOBAL and SESSION variable). Debezium 3.1.3 with mysql-binlog-connector-java 0.40.2 (zstd-jni shaded into the lightweight jar) already decodes it per event type, so nothing has to be switched on in the connector -- but handleTransactionPayload dispatches the inner events directly, so every row of every statement of the transaction carries the payload event position, with a row index that restarts at 0 per rows event.
Defect fixed (both GTID modes)
nextVersionAssignment classified a record against the high-water mark with SourcePosition.compareTo, which includes the row index. Under a payload the second and later statements of the transaction (and the later rows-event chunks of one large statement) rank BELOW the mark and fell into the un-clamped redelivery branch while the first statement rows had been floored: an UPDATE or DELETE of a key inserted earlier in the same transaction was versioned below its own INSERT -- the update lost, the deleted row live, row counts off by one per undeleted row. Reproduced on the 2.11.0 head in steady state, not only after restarts: pre-fix build 8 of 26 (GTID off) and 12 of 26 (GTID on) end-to-end checks passing.
Fix: SourcePosition.sameLogPosition (same log, file and byte position, any row index) and a rest-of-marked-event branch tested before the above/below comparison (spec 02.02 section 3.1.2, spec 01.02 section 3.3). Uncompressed multi-row events receive exactly the versions they received before.
Detection, flag, metrics
New BinlogTransactionCompressionPreflight, run after the row-image check: one read-only SELECT of binlog_transaction_compression, its zstd level and the server version, plus a decoder self-test that decodes a synthetic zstd Transaction_payload (one XID event) through the real binlog-client deserializer, proving the zstd-jni native library and the decoder on the running platform. New key binlog.transaction.compression.check = auto (default) | require | skip: auto reports and refuses only when the source is ON and the decoder does not work; require refuses unless the source is ON (for deployments that rely on compression, e.g. a WAN link); skip runs nothing. Gauges clickhouse_sink_source_binlog_transaction_compression, clickhouse_sink_source_binlog_transaction_compression_level_zstd and clickhouse_sink_binlog_payload_decoder_ok.
Spec first
Verification