8000 fix(writer): spill large INSERT chunks to disk instead of rendering them in memory (Spec 03.06 section 3.4) by minguyen9988 · Pull Request #1536 · Altinity/clickhouse-sink-connector · GitHub
[go: up one dir, main page]
Skip to content

fix(writer): spill large INSERT chunks to disk instead of rendering them in memory (Spec 03.06 section 3.4) - #1536

Merged
minguyen9988 merged 1 commit into
2.11.0from
fix-insert-spill-to-disk
Oct 1, 2026
Merged

fix(writer): spill large INSERT chunks to disk instead of rendering them in memory (Spec 03.06 section 3.4)#1536
minguyen9988 merged 1 commit into
2.11.0from
fix-insert-spill-to-disk

Conversation

@minguyen9988
@minguyen9988 minguyen9988 commented Oct 1, 2026
Copy link
Copy Markdown
Collaborator

Problem

The V2 JDBC driver renders every INSERT chunk as SQL text in memory: each value is encoded into a String, each row substituted into a StringBuilder, the chunk concatenated, turned into a String and then a UTF-8 byte[] body. One value of V bytes costs 5-6 live copies of its text, and a binary value's hex text is 2V. JFR on 2.11.0 (one 64 MiB and one 256 MiB LONGBLOB row, each inserted and updated in one transaction): 17.5 of 22.4 GiB allocated were these copies. With a 4 GiB heap the 256 MiB row exited on OutOfMemoryError and crash-looped on every restart (edge E5 failed 2 of 3 runs on 32b43c2).

Change (spec first: 03.06 section 3.4, FM-03.06-5/6/7; FM-01.08-5, FM-04.02-3, FM-07.04-3, FM-07.05-1 updated)

  • A chunk whose rendered size bound reaches insert.spill.threshold.bytes (default 32 MiB; 0 = always, negative = never) is written row by row to a spill file under insert.spill.directory/ and sent as INSERT INTO t(cols) FORMAT Values through the connection's own client with the statement's own settings. Still one INSERT per chunk.
  • The driver stays the encoder: every setter is delegated, and the row is written as the driver's own row template with its own encoded values at its own parameter positions. Values of 64 KiB or more are streamed from the row object with the driver's exact encoding (escaped quoted string, unhex(UPPER HEX), lower-case hex text) and are never built as a String.
  • Spill files are removed after the send or on failure. Leftovers of dead processes, and files older than this process in its own pid directory (containers run as pid 1), are swept on first use. If the driver internals are missing after a driver upgrade, the path refuses itself loudly and the in-memory path is kept; a unit test pins this.
  • Progress line marks spilled chunks: (sent from a spill file).

Verification

  • SpillingInsertStatementTest (14): byte-for-byte parity with the driver's own addBatch output for small and large strings, setBytes and hex text; send, settings, failure, unbound-parameter, sweep and decision cases. 6/6 mutants killed. Core 752 / lightweight 695 unit tests, 0 failures (the only errors are the Testcontainers tests that need Docker).
  • New tests/e2e/large_row_spill.sh at a 4 GiB heap, bytes mode: 64/128/256/512 MiB rows value-exact, kill -9 during the 512 MiB row re-applied exactly, 1,000,000-row incompressible transaction value-exact, 0 OOM, no spill file left. Peak heap after GC is 2.6 GiB. In base64 mode at 4 GiB, 64/128/256 MiB pass; 512 MiB passes at 8 GiB. Control on 32b43c2: the 512 MiB row exits with OOM four times out of four.
  • Edge harness at 4 GiB: 21/21. E5 now passes; it failed before.
  • Parity: the whole e2e corpus was run on a test build that spills EVERY chunk, with the same results as the in-memory path. That corpus is compression basic x3 26/26, hard on/off 49/49, large 25/25, edge 21/21, chaos (12 fault phases) GTID on/off, DDLxDML 26/26, history 148/0, PK 26/0, base 8/0, upgrade/downgrade 101/101, DST 15/15. In the time-zone matrix every deterministic cell is identical to the in-memory run.

Residual (declared)

The decoded row images still have to fit the heap while Debezium holds them (FM-01.08-5). MySQL's applier is the same: one event in memory, bounded by max_allowed_packet.

🤖 Generated with Claude Code

@minguyen9988 @claude
fix(writer): spill large INSERT chunks to disk instead of rendering t…
58e7afe 
…hem in memory (Spec 03.06 section 3.4)

A chunk whose rendered size bound reaches insert.spill.threshold.bytes (default 32 MiB) is written row by row to a spill file and sent as INSERT ... FORMAT Values through the driver's own client; values of 64 KiB or more are streamed from the row itself. The driver still encodes every small value and its own row template is used, so the text sent is byte-identical to the in-memory path (pinned by SpillingInsertStatementTest against the driver's addBatch output). A 256 MiB row that exhausted a 4 GiB heap and crash-looped now replicates value-exact up to 512 MiB at 4 GiB (bytes mode); new e2e harness tests/e2e/large_row_spill.sh.

Jinja-Render-Check: rendered=2; formats=shell,yaml; result=pass

Destructive-Op-Check: sites=3; result=pass

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@minguyen9988
minguyen9988 merged commit 9fdc612 into 2.11.0 Oct 1, 2026
11 checks passed
@minguyen9988
minguyen9988 deleted the fix-insert-spill-to-disk branch October 1, 2026 05:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@minguyen9988
0