Repository navigation
spec(I15): bounded, declared recovery -- Failure Modes & Recovery in all 75 micro-specs (338 failure modes, 193 DEFECT, 144 GAP) + failure-mode tests - #1535
Merged
minguyen9988 merged 24 commits intoOct 1, 2026
Conversation
…s failure modes with Detection, Recovery and RTO; the validator requires the section and checks the tests it cites
…onitoring domains (specs 08.x, 10.x) -- passing pins plus @disabled DEFECT tests confirmed red on 2.11.0 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…parser, translation, execution and PK rebuild Adds section 6 Failure Modes & Recovery to all nine specs of 06-ddl-replication (42 failure modes, 23 DEFECT, 12 GAP) and pure unit tests pinning them: DdlFailureModesTest, DdlTranslationFailureModesTest, DBMetadataExecuteSystemQu 8000 eryExhaustionTest, ClickHouseBatchExecutorInterruptTest. Eight tests are @disabled DEFECT pins that fail on 2.11.0, among them FM-06.08-2: DBMetadata.executeSystemQuery returns normally after exhausting retries, so a DDL refused with any retryable ClickHouse code (or sent while ClickHouse is down) is acknowledged as applied. No src/main change. Destructive-Op-Check: sites=8; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…, terminal vs retried-forever refusals, operator recovery procedures; new poison-value tests (enabled pins + disabled DEFECT tests) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…_compare, Lean suite, ch-mysql-resync, Python tool coverage; offline tests pinning each mode, DEFECTs as skipped tests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-02.06 declare 20 failure modes with detection, recovery and RTO; 15 DEFECTs, the code-level ones pinned by disabled tests (counter carry at reset, in-process restart redelivery, same-name failover, GTID domain switch and 22-bit tie-break, future mark after a clock step, underivable version retried) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…- 09.01-09.04 declare 18 failure modes with detection, recovery and RTO; 11 DEFECTs, among them the REST position edit that deletes the offset row and inserts into system.<table>, offset flush failures reported only at WARN, and stale or missing offsets turning a restart into an unbounded replay or reload Destructive-Op-Check: sites=1; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…alog and resilience/monitoring: out-of-band ClickHouse DDL, stale-cache probe failure, partial writer init, silent catalog-DDL retry exhaustion, forever-retry liveness gap, frozen lag signals, PostgreSQL status view, dedup retried-put loss Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…y mutation -- worker Error death, 252/241 retry, partial-write retry, chunk failure; disabled DEFECT tests for unbounded retry, interrupted pause, refusals classified UNKNOWN, ignore_delete tombstone, EXECUTE_FAILED, sorting-key read failure, retired endpoint, override map validator, thread.pool.size=0, binary key token, collapsing sign parsing Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… -- worker death, unbounded retry, pool retirement and keep-alive, hung INSERT, key-routing part pressure and TRUNCATE barrier Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…I15) -- sorting-key read failure and drift, ignore_delete tombstone, refusals retried forever, CollapsingMergeTree replay and sign parsing Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… 01.01-01.07, 33 failure modes, 15 DEFECT, 14 GAP Adds the Failure Modes & Recovery section to the engine lifecycle, binlog coordinates, dispatch loop, transaction boundaries, handoff, control records and keep-alive/reconnect specs, with behaviour read from the 2.11.0 code and from the Debezium 3.1.3 / binlog client 0.40.2 bytecode. New unit tests: 4 passing pins (source clock steps, interrupted handoff, transient source loss) and 7 @disabled DEFECT tests that fail on 2.11.0 (restart-monitor clock, heartbeat liveness, DDL dropped on ignore-rule failure, blocked put behind a dead worker, purged binlog and poison row retried as transient, byte-cap first-row sampling). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…atting/membership/update/truncate/memo failure modes; new tests for N:1 database fan-in and failed staleness re-read Destructive-Op-Check: sites=1; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…SCD2 shape, SCD2 write protocol, audit table, replication-log-only; redelivery, poison and routing tests (DEFECTs disabled) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ExecutorInterruptTest: FM-03.01-4 and FM-06.02-1 pin one code path; both methods kept in one class) Destructive-Op-Check: sites=8; result=pass
…ure-modes-recovery
…that is not part of this change Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ded-declared-recovery
minguyen9988
merged commit Oct 1, 2026
b780d32
into
fix-binlog-connection-guard-chaos-tests
12 checks passed
minguyen9988
added a commit
that referenced
this pull request
Oct 1, 2026
…rollback reported; chaos and edge harnesses with measured recovery times (Specs 01.08 section 6, 01.09, 01.10) (#1534) * fix(binlog): every dead binlog connection restarts the engine instead of stalling silently; XA rollbacks after PREPARE are reported; chaos, stress and edge harnesses for compressed binlogs with measured recovery times (Specs 01.08 section 6, 01.09, 01.10) Binlog connection guard (spec 01.09): with connect.keep.alive=false a source-side close (mysqld killed, failover) ended the binlog client's read loop as a normal end and a connection that died behind a partition blocked the reader forever; both left replication stopped with no ERROR (measured: 15 min and 300 s without recovery). The binlog socket now turns end of stream into an EOFException and carries a read timeout (binlog.read.timeout.ms, default 2x connect.keep.alive.interval.ms, above the source heartbeat), installed through the connector's copy of Debezium's MySqlStreamingChangeEventSourceMetrics. Measured: mysqld kill -9 recovered in 5-10 s, a partition in 14 s, an idle source 200 s with zero restarts. XA audit (spec 01.10): Debezium dispatches an XA transaction's rows at PREPARE and ignores XA ROLLBACK; the rollback is now an ERROR naming the tables plus a metric, fed from the client listener and, for compressed transactions, from the payload decoder's first pass. binlog_row_value_options=PARTIAL_JSON is dec 94DE lared not supported (doc/limitations.md). Harnesses: tests/e2e/binlog_transaction_compression_chaos.sh (12 fault phases under an 8-session writer mix, 30/30 GTID on and off), binlog_transaction_compression_edge.sh (8 edge cases, through the source and through a MySQL replica hop). Unit suites: lightweight 643 run, 0 failures (2 errors are Docker-only classes); core 663 run, 0 failures (20 Docker-only errors). The three e2e shell scripts contain docker --format Go-template placeholders, not Jinja; each was linted with bash -n. Jinja-Render-Check: rendered=3; formats=shell; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(binlog): review round -- a missing binlog client at guard/audit install is a WARN, not a silent no-op; the read-timeout ERROR reports the socket's own timeout; an XA ROLLBACK whose PREPARE predates the start says so; partial JSON through a replica hop and the OOM crash loop documented as measured (Specs 01.08 section 6, 01.09, 01.10) Measured on the replica-hop edge run: a MySQL replica re-logs applied partial JSON updates with full row images (its own binlog_row_value_options), so a connector reading such a replica receives them; a 256 MiB row exhausted a 4 GiB heap in 4 of 6 runs and one restart looped on OOM. Jinja-Render-Check: rendered=1; formats=shell; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(I15): bounded, declared recovery -- Failure Modes & Recovery in all 75 micro-specs (338 failure modes, 193 DEFECT, 144 GAP) + failure-mode tests (#1535) * spec(I15): bounded, declared recovery -- every micro-spec declares its failure modes with Detection, Recovery and RTO; the validator requires the section and checks the tests it cites * test(I15): failure-mode tests for the schema catalog and resilience/monitoring domains (specs 08.x, 10.x) -- passing pins plus @disabled DEFECT tests confirmed red on 2.11.0 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(06): failure modes and recovery (Invariant I15) -- DDL barrier, parser, translation, execution and PK rebuild Adds section 6 Failure Modes & Recovery to all nine specs of 06-ddl-replication (42 failure modes, 23 DEFECT, 12 GAP) and pure unit tests pinning them: DdlFailureModesTest, DdlTranslationFailureModesTest, DBMetadataExecuteSystemQueryExhaustionTest, ClickHouseBatchExecutorInterruptTest. Eight tests are @disabled DEFECT pins that fail on 2.11.0, among them FM-06.08-2: DBMetadata.executeSystemQuery returns normally after exhausting retries, so a DDL refused with any retryable ClickHouse code (or sent while ClickHouse is down) is acknowledged as applied. No src/main change. Destructive-Op-Check: sites=8; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(07): failure modes and recovery (Invariant I15) -- poison values, terminal vs retried-forever refusals, operator recovery procedures; new poison-value tests (enabled pins + disabled DEFECT tests) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(11): failure modes and recovery (Invariant I15) -- validator, db_compare, Lean suite, ch-mysql-resync, Python tool coverage; offline tests pinning each mode, DEFECTs as skipped tests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(versioning): failure modes and recovery (Invariant I15) -- 02.01-02.06 declare 20 failure modes with detection, recovery and RTO; 15 DEFECTs, the code-level ones pinned by disabled tests (counter carry at reset, in-process restart redelivery, same-name failover, GTID domain switch and 22-bit tie-break, future mark after a clock step, underivable version retried) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(offset-management): failure modes and recovery (Invariant I15) -- 09.01-09.04 declare 18 failure modes with detection, recovery and RTO; 11 DEFECTs, among them the REST position edit that deletes the offset row and inserts into system.<table>, offset flush failures reported only at WARN, and stale or missing offsets turning a restart into an unbounded replay or reload Destructive-Op-Check: sites=1; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(08,10): failure modes and recovery (Invariant I15) -- schema catalog and resilience/monitoring: out-of-band ClickHouse DDL, stale-cache probe failure, partial writer init, silent catalog-DDL retry exhaustion, forever-retry liveness gap, frozen lag signals, PostgreSQL status view, dedup retried-put loss Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(I15): failure-mode tests for the execution engine and sorting-key mutation -- worker Error death, 252/241 retry, partial-write retry, chunk failure; disabled DEFECT tests for unbounded retry, interrupted pause, refusals classified UNKNOWN, ignore_delete tombstone, EXECUTE_FAILED, sorting-key read failure, retired endpoint, override map validator, thread.pool.size=0, binary key token, collapsing sign parsing Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(03-execution-engine): failure modes and recovery (Invariant I15) -- worker death, unbounded retry, pool retirement and keep-alive, hung INSERT, key-routing part pressure and TRUNCATE barrier Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(05-sorting-key-mutation): failure modes and recovery (Invariant I15) -- sorting-key read failure and drift, ignore_delete tombstone, refusals retried forever, CollapsingMergeTree replay and sign parsing Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(cdc-engine): failure modes and recovery (Invariant I15) -- specs 01.01-01.07, 33 failure modes, 15 DEFECT, 14 GAP Adds the Failure Modes & Recovery section to the engine lifecycle, binlog coordinates, dispatch loop, transaction boundaries, handoff, control records and keep-alive/reconnect specs, with behaviour read from the 2.11.0 code and from the Debezium 3.1.3 / binlog client 0.40.2 bytecode. New unit tests: 4 passing pins (source clock steps, interrupted handoff, transient source loss) and 7 @disabled DEFECT tests that fail on 2.11.0 (restart-monitor clock, heartbeat liveness, DDL dropped on ignore-rule failure, blocked put behind a dead worker, purged binlog and poison row retried as transient, byte-cap first-row sampling). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(04): failure modes and recovery (Invariant I15) -- grouping/formatting/membership/update/truncate/memo failure modes; new tests for N:1 database fan-in and failed staleness re-read Destructive-Op-Check: sites=1; result=pass Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(12): failure modes and recovery (Invariant I15) -- mode matrix, SCD2 shape, SCD2 write protocol, audit table, replication-log-only; redelivery, poison and routing tests (DEFECTs disabled) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * spec(I15): drop the forward reference to a consolidated runbook spec that is not part of this change Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test: neutral target database names in DatabaseOverrideValidatorTest Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1534; it retargets to 2.11.0 when #1534 merges.
The DEFECTs that matter most for bounded recovery:
Fixes follow in separate PRs.
Checks: validate_specs.py passes (75 specs, 1173 citations). Core 738 run, 0 failures, 41 disabled DEFECT tests skipped. Lightweight 695 run, 0 failures, 28 skipped. Errors appear only in Docker-only classes. Python tools: 154 passed, 5 skipped.