Repository navigation
perf(writer): coalesce.max.wait.ms -- bound ClickHouse part creation per wait instead of per poll (Spec 03.03 section 3.1.1 step 6) - #1526
Conversation
…for more queued batches before writing a small one, so parts are bounded per wait instead of per poll (Spec 03.03 section 3.1.1 step 6) Coalescing (3.1.1 steps 1-4) only takes what is ALREADY queued, so a worker that keeps up writes every poll as it arrives. Each INSERT is one ClickHouse part per partition it touches, of a few thousand rows, at the source poll rate, and under key-aware routing (spec 03.07) ten workers on one hot table multiply that by ten. Measured on the part-pressure matrix (MySQL 8.0 -> connector -> ClickHouse 24.8, month-partitioned target, 300,000 rows spanning 24 months, ten workers): 5,210 new parts averaging 58 rows from 779 INSERTs with key routing on, against 194 parts of 1,546 rows from 32 INSERTs with the table pinned to one worker, and 890 parts of 404 rows on the pre-coalescing build. Nothing was delayed or rejected in the lab, but that shape is the write amplification that drives a loaded server toward parts_to_delay_insert. New knob coalesce.max.wait.ms (default 0 = unchanged): when the queue is empty after step 1 and the write is still under BOTH buffer.max.records and buffer.max.bytes, the worker waits -- a timed poll on its own queue -- up to that long for more batches, adding each one that fits (and whatever arrived with it) and stopping the moment a bound is reached or the deadline passes. A batch taken during the wait that does not fit is kept as carriedOver and opens the next write, ahead of anything still queued (it is older than all of them), so nothing is split or reordered; every such batch is already outstanding in the offset FIFO from its handoff, so the DDL barrier and the quiescence predicate see it, and the write that follows reports every group in order exactly as before. Cost: up to the wait in latency on a trickle. Gain: parts bounded to one write per wait per worker per partition -- parts of buffer.max.records rows under load and far fewer merges. Specs: 03.03 section 3.1.1 step 6 and section 5; 01.05 configuration table. Tests: CoalescedQueuedBatchesTest.waitCoalescesABatchThatArrivesDuringTheWindow (400 ms wait: a batch handed off 80 ms later joins the write; default 0: two writes) and waitStopsAtTheRowBoundAndCarriesTheRestOver (bound 3 rows, 4 s wait: the second batch is carried over and opens the next write, and only one window is spent). Neighbours ParkedBatchWrittenOnceTest, OffsetHandoffOrderTest, ClickHouseBatchRunnableTest, HandedOffBatchVisibilityTest, ClickHouseBatchExecutorQuiescentTest, PauseDrainRaceTest green. Mutation check (wait ignored): both new tests fail. scripts/validate_specs.py --changed-base b8f29e4: PASSED (70 specifications, 812 citations).
|
Part-pressure / write-amplification matrix (MySQL 8.0 -> connector -> ClickHouse 24.8 in one pod, 10 workers, ClickHouse thresholds at defaults, active parts sampled once a second, every phase checked value-level MySQL vs ClickHouse FINAL). Workload per run: 300k rows into an unpartitioned table, 20k single-row transactions, 300k rows spanning 24 months into a MONTH-PARTITIONED table (q_part), 60k updates on it, then kill -9 + restart + ALTER + 100k rows. Figures are for the whole run unless noted; q_part is the partitioned table.
Reading: key-aware routing alone multiplies part creation on a partitioned hot table by ~6.5x against the previous build (5,666 vs 867 parts) and drives merge work to 12x the inserted bytes; a 500 ms wait brings it back to the previous build's level, 2 s below it, and the single-worker configuration remains the floor. No run delayed or rejected an insert and no TOO_MANY_PARTS (Code 252) occurred anywhere (connector log, query_log, system.errors); all runs converged value-identical, including the kill -9 restart with an ALTER after it. The wait costs latency, not sustained throughput: it only engages when a worker's queue is empty. Recommended deployment value for hot partitioned tables: 1-2 s; default stays 0 in this change so nothing changes without an operator's decision. |
Why:
Coalescing (spec 03.03 section 3.1.1 steps 1-4) only takes what is ALREADY queued, so a worker that keeps up writes every poll as it arrives: one ClickHouse part per partition touched, a few thousand rows, at the source poll rate -- and under key-aware routing (spec 03.07) ten workers on one hot table multiply that by ten. Measured end to end (MySQL 8.0 -> connector -> ClickHouse 24.8, a month-partitioned target fed 300,000 rows spanning 24 months, ten workers): 5,210 new parts averaging 58 rows from 779 INSERTs with key routing on; 194 parts of 1,546 rows from 32 INSERTs with the table pinned to one worker; 890 parts of 404 rows on the build before coalescing. Nothing was delayed or rejected in that lab (peak 17 active parts in one partition against the default 1000/3000 thresholds), but that is the write-amplification shape that drives a loaded server toward parts_to_delay_insert and TOO_MANY_PARTS.
What:
Verification: