Repository navigation
fix(binlog): dead binlog connections restart instead of stalling; XA rollback reported; chaos and edge harnesses with measured recovery times (Specs 01.08 section 6, 01.09, 01.10) - #1534
Merged
Conversation
… of stalling silently; XA rollbacks after PREPARE are reported; chaos, stress and edge harnesses for compressed binlogs with measured recovery times (Specs 01.08 section 6, 01.09, 01.10) Binlog connection guard (spec 01.09): with connect.keep.alive=false a source-side close (mysqld killed, failover) ended the binlog client's read loop as a normal end and a connection that died behind a partition blocked the reader forever; both left replication stopped with no ERROR (measured: 15 min and 300 s without recovery). The binlog socket now turns end of stream into an EOFException and carries a read timeout (binlog.read.timeout.ms, default 2x connect.keep.alive.interval.ms, above the source heartbeat), installed through the connector's copy of Debezium's MySqlStreamingChangeEventSourceMetrics. Measured: mysqld kill -9 recovered in 5-10 s, a partition in 14 s, an idle source 200 s with zero restarts. XA audit (spec 01.10): Debezium dispatches an XA transaction's rows at PREPARE and ignores XA ROLLBACK; the rollback is now an ERROR naming the tables plus a metric, fed from the client listener and, for compressed transactions, from the payload decoder's first pass. binlog_row_value_options=PARTIAL_JSON is declared not supported (doc/limitations.md). Harnesses: tests/e2e/binlog_transaction_compression_chaos.sh (12 fault phases under an 8-session writer mix, 30/30 GTID on and off), binlog_transaction_compression_edge.sh (8 edge cases, through the source and through a MySQL replica hop). Unit suites: lightweight 643 run, 0 failures (2 errors are Docker-only classes); core 663 run, 0 failures (20 Docker-only errors). The three e2e shell scripts contain docker --format Go-template placeholders, not Jinja; each was linted with bash -n. Jinja-Render-Check: rendered=3; formats=shell; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nstall is a WARN, not a silent no-op; the read-timeout ERROR reports the socket's own timeout; an XA ROLLBACK whose PREPARE predates the start says so; partial JSON through a replica hop and the OOM crash loop documented as measured (Specs 01.08 section 6, 01.09, 01.10) Measured on the replica-hop edge run: a MySQL replica re-logs applied partial JSON updates with full row images (its own binlog_row_value_options), so a connector reading such a replica receives them; a 256 MiB row exhausted a 4 GiB heap in 4 of 6 runs and one restart looped on OOM. Jinja-Render-Check: rendered=1; formats=shell; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…all 75 micro-specs (338 failure modes, 193 DEFECT, 144 GAP) + failure-mode tests (#1535) * spec(I15): bounded, declared recovery -- every micro-spec declares its failure modes with Detection, Recovery and RTO; the validator requires the section and checks the tests it cites * test(I15): failure-mode tests for the schema catalog and resilience/monitoring domains (specs 08.x, 10.x) -- passing pins plus @disabled DEFECT tests confirmed red on 2.11.0 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(06): failure modes and recovery (Invariant I15) -- DDL barrier, parser, translation, execution and PK rebuild Adds section 6 Failure Modes & Recovery to all nine specs of 06-ddl-replication (42 failure modes, 23 DEFECT, 12 GAP) and pure unit tests pinning them: DdlFailureModesTest, DdlTranslationFailureModesTest, DBMetadataExecuteSystemQueryExhaustionTest, ClickHouseBatchExecutorInterruptTest. Eight tests are @disabled DEFECT pins that fail on 2.11.0, among them FM-06.08-2: DBMetadata.executeSystemQuery returns normally after exhausting retries, so a DDL refused with any retryable ClickHouse code (or sent while ClickHouse is down) is acknowledged as applied. No src/main change. Destructive-Op-Check: sites=8; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(07): failure modes and recovery (Invariant I15) -- poison values, terminal vs retried-forever refusals, operator recovery procedures; new poison-value tests (enabled pins + disabled DEFECT tests) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(11): failure modes and recovery (Invariant I15) -- validator, db_compare, Lean suite, ch-mysql-resync, Python tool coverage; offline tests pinning each mode, DEFECTs as skipped tests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(versioning): failure modes and recovery (Invariant I15) -- 02.01-02.06 declare 20 failure modes with detection, recovery and RTO; 15 DEFECTs, the code-level ones pinned by disabled tests (counter carry at reset, in-process restart redelivery, same-name failover, GTID domain switch and 22-bit tie-break, future mark after a clock step, underivable version retried) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(offset-management): failure modes and recovery (Invariant I15) -- 09.01-09.04 declare 18 failure modes with detection, recovery and RTO; 11 DEFECTs, among them the REST position edit that deletes the offset row and inserts into system.<table>, offset flush failures reported only at WARN, and stale or missing offsets turning a restart into an unbounded replay or reload Destructive-Op-Check: sites=1; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(08,10): failure modes and recovery (Invariant I15) -- schema catalog and resilience/monitoring: out-of-band ClickHouse DDL, stale-cache probe failure, partial writer init, silent catalog-DDL retry exhaustion, forever-retry liveness gap, frozen lag signals, PostgreSQL status view, dedup retried-put loss Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(I15): failure-mode tests for the execution engine and sorting-key mutation -- worker Error death, 252/241 retry, partial-write retry, chunk failure; disabled DEFECT tests for unbounded retry, interrupted pause, refusals classified UNKNOWN, ignore_delete tombstone, EXECUTE_FAILED, sorting-key read failure, retired endpoint, override map validator, thread.pool.size=0, binary key token, collapsing sign parsing Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(03-execution-engine): failure modes and recovery (Invariant I15) -- worker death, unbounded retry, pool retirement and keep-alive, hung INSERT, key-routing part pressure and TRUNCATE barrier Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(05-sorting-key-mutation): failure modes and recovery (Invariant I15) -- sorting-key read failure and drift, ignore_delete tombstone, refusals retried forever, CollapsingMergeTree replay and sign parsing Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(cdc-engine): failure modes and recovery (Invariant I15) -- specs 01.01-01.07, 33 failure modes, 15 DEFECT, 14 GAP Adds the Failure Modes & Recovery section to the engine lifecycle, binlog coordinates, dispatch loop, transaction boundaries, handoff, control records and keep-alive/reconnect specs, with behaviour read from the 2.11.0 code and from the Debezium 3.1.3 / binlog client 0.40.2 bytecode. New unit tests: 4 passing pins (source clock steps, interrupted handoff, transient source loss) and 7 @disabled DEFECT tests that fail on 2.11.0 (restart-monitor clock, heartbeat liveness, DDL dropped on ignore-rule failure, blocked put behind a dead worker, purged binlog and poison row retried as transient, byte-cap first-row sampling). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(04): failure modes and recovery (Invariant I15) -- grouping/formatting/membership/update/truncate/memo failure modes; new tests for N:1 database fan-in and failed staleness re-read Destructive-Op-Check: sites=1; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(12): failure modes and recovery (Invariant I15) -- mode matrix, SCD2 shape, SCD2 write protocol, audit table, replication-log-only; redelivery, poison and routing tests (DEFECTs disabled) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(I15): drop the forward reference to a consolidated runbook spec that is not part of this change Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test: neutral target database names in DatabaseOverrideValidatorTest Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chaos and edge testing of compressed binlogs found two ways a binlog connection dies that left replication stopped with no ERROR for as long as the process lived, with connect.keep.alive=false (the connector default): (1) mysqld killed or crashed, or a failover: the binlog client treats end of stream at a packet boundary as a normal end and never reports a communication failure (measured: no progress for 15 minutes); (2) a network partition: the source drops the dump connection after net_write_timeout, the FIN is lost, and the socket has no read timeout (measured: no progress for the whole 300 s window).
What this changes:
Measured recovery time, from fault removal until a marker committed after it is visible in ClickHouse, GTID on / off, before this change in brackets: idle 200 s 3/3 s with 0 restarts; kill -9 mid-payload 18/18 s; RST 18/19 s; partition until the source gives up 14/14 s (never); ClickHouse down 60 s 191/188 s; ClickHouse frozen 150 s 21/21 s; MySQL restart 5/5 s; mysqld kill -9 mid-commit 10/5 s (never); compression toggled 12 times 0/1 s; 5x kill -9 1/3 s; 3 MB/s link 3/2 s; DDL storm 1/2 s. Chaos 30/30 in both GTID modes; edge through the source 21/21.
Documented here, not fixed: a 256 MiB row exhausts a 4 GiB heap in 3 of 5 runs, and a restart then applied the row (01.08 FM-01.08-5). Clearing offsets and starting schema_only with enable.snapshot.ddl=true runs DROP TABLE IF EXISTS + CREATE TABLE and empties the replica (01.08 FM-01.08-6). The ClickHouse-down recovery is dominated by Debezium's 2x90 s shutdown waits (GAP).
Tests: lightweight 643 run, 0 failures (the 2 errors are Docker-only classes); core 663 run, 0 failures. New tests: BinlogConnectionGuardTest (11), BinlogEventAuditTest (6), StreamingTransactionPayloadTest.compressedXaBodyReachesTheAudit. Mutation-checked: without SO_TIMEOUT, and without the decoder feed. validate_specs.py passes.