Repository navigation
fix(logging): the resume replay is summarized, never dumped -- operation types and counts instead of one full row image per skipped event (Spec 01.07 section 3.5) - #1509
Conversation
…um's per-event 'Skipping previously processed ... event' lines (each a full row image; 5,227 of them, seven rotated 6 MB files in fourteen seconds, at one start) are denied by a log4j filter installed at setup() and reported as one line per resume with the operation types and their counts (Spec 01.07 section 3.5)
…onflict was the setup() preflight block, where #1505 added DebeziumQueueBytesPreflight next to the ResumeReplayLogSummary install; both are kept, queue preflight first
|
Conflict resolved by merging the current 2.11.0 head ( 8000 474c821, which now carries #1505, #1506 and #1507) into this branch as a merge commit (8230bb0, fast-forward push, no rebase). The only conflict was the setup() preflight block, where #1505 added DebeziumQueueBytesPreflight.apply(props) at the same spot as this branch's ResumeReplayLogSummary.install(); both are kept, queue preflight first. stop() carries both step 4a (close worker connections, from #1506) and step 4c (flush a replay still being counted, from this branch). The diff against 2.11.0 is now exactly this change (4 files, +464). Re-checked on the merge result: validate_specs PASSED (63 specs); sink-connector-lightweight 28/28 -- ResumeReplayLogSummaryTest 5/5, BinlogKeepAlivePreflightTest 7/7, DebeziumQueueBytesPreflightTest 4/4, StoppedEngineRetiresHandoffsTest 3/3, DeadWorkerRetryIsTerminalTest 3/3, EngineRestartFifoResetTest 6/6; BUILD SUCCESS. |
Noise
Every engine start resumes from the durable offset, which Debezium records as the transaction's BEGIN plus the number of events already delivered. The binlog client re-reads the transaction from BEGIN and Debezium skips the events it has already delivered -- logging EACH one at INFO from io.debezium.connector.binlog.BinlogStreamingChangeEventSource as 'Skipping previously processed row event: Event{header=..., data=...{rows=[ ... ]}}', i.e. the complete row image, some seventy lines per event. On one deployment a single start logged 5,227 such events: seven rotated 6 MB files in fourteen seconds, all of it row data, none of it an error. The operator's rule: do not print the row data; print the operation type and the count of that operation.
Rule (spec 01.07 section 3.5)
ResumeReplayLogSummary is a log4j filter the connector installs programmatically at setup() on that Debezium logger (idempotent per process; no deployment log4j2.xml change):
Example of the replacement line: 'Resume replay done (the reader moved past the resume point): skipped 5227 previously processed binlog event(s) already applied before this start, 14 s, binlog positions 1457290340..1502834139 -- INSERT=5227. Only the operation types and counts are logged, never the row images.'
Checks on this branch