Repository navigation
fix(memory): bound the data path in BYTES, not only in rows -- handoff byte cap, Debezium queue byte bound, byte-bounded INSERT chunks, bounded legacy queue (Spec 01.05 section 3.4 items 7-8, Spec 03.06 section 3.1) - #1505
Conversation
…cillates one unit at a time, so one WARN/INFO per wait was one per batch (~300 lines a minute, 990 WARNs into the error log in seven minutes on the first deployment that met the cap); now one WARN per pacing period, pauses inside a 60 s re-arm window counted not logged, one summary INFO per minute while paced, one 'pacing ended' line after a quiet gap or on reset() (Spec 01.05 section 3.4 item 6)
…lands 52 s into the period -- at 61 s the per-minute summary fires by design, which the test had mistaken for a stray line
… cap on the handoff (sink.connector.handoff.max.outstanding.bytes, default a quarter of the heap, estimated per group by RecordSizeEstimator and stamped on every row), a connector-pinned default for Debezium's max.queue.size.in.bytes (a sixteenth of the heap; Debezium's own default is 0 = off), INSERT chunks closed on estimated bytes as well as rows (buffer.max.bytes, default 256 MiB), and the single-threaded queue bounded by the ConfigDef default when the property is absent (Spec 01.05 section 3.4 items 7-8, section 3.6; Spec 03.06 section 3.1)
…g body -- checkIfBatchCanBeCommitted declares InterruptedException
… one squashed commit with the same content as the two pacing commits this branch was stacked on; the three conflicts (DebeziumOffsetManagement.java, spec 01.05, spec 09.01) resolve to this branch's versions, which carry the pacing text plus the byte-bound additions
|
Conflict resolved: #1504 landed on 2.11.0 as one squashed commit (5120dc5) whose content is byte-identical to the two pacing commits this branch was stacked on (git diff between them is empty), so the three conflicts -- DebeziumOffsetManagement.java, spec 01.05, spec 09.01 -- resolve to this branch's versions, which carry the pacing text plus the byte-bound additions. Merged 5120dc5 into the branch as a merge commit (2212fdd, fast-forward push, no rebase); the diff against 2.11.0 is now exactly the byte-bound change (16 files). Re-ran on the merge result: validate_specs PASSED (63 specs, 650 citations); sink-connector 62/62 (RecordSizeEstimatorTest 4/4, HandoffHardCapBytesTest 5/5, BatchChunkerTest 5/5, HandoffHardCapBackpressureTest 7/7, HandoffHardCapLogPacingTest 4/4, HandoffBacklogAdvisoryTest 4/4, DebeziumOffsetManagementTest 4/4, RetiredHandoffNotAcknowledgedTest 4/4, OffsetNoLossParameterIndependenceTest 5/5, PreparedStatementExecutor*Test green); sink-connector-lightweight 23/23 (DebeziumQueueBytesPreflightTest 4/4, BinlogKeepAlivePreflightTest 7/7, StoppedEngineRetiresHandoffsTest 3/3, DeadWorkerRetryIsTerminalTest 3/3, EngineRestartFifoResetTest 6/6). BUILD SUCCESS in both modules. |
…onflict was the setup() preflight block, where #1505 added DebeziumQueueBytesPreflight next to the ResumeReplayLogSummary install; both are kept, queue preflight first
Stacked on #1504 (its two commits are included here; the diff shrinks to the last two commits once #1504 merges).
Defect
Every bound on the data path was expressed in rows or in batches, none in bytes, on a JVM with a fixed heap and a source that can burst far faster than the sink writes:
Rule (spec 01.05 section 3.4 items 7-8, section 3.6; spec 03.06 section 3.1)
Checks on this branch