Repository navigation
perf(writer): derive INSERT templates, column types and formatters once per batch instead of once per row -- byte-identical output, 1.9x per-batch throughput (Spec 04.06) - #1524
Merged
Conversation
…ce per batch instead of once per row -- byte-identical output, 1.9x per-batch throughput (Spec 04.06) A worker turned every batch into a JDBC batch by re-deriving, per ROW, metadata that is constant for the whole batch: the INSERT template (column list, placeholders, parameter-index map) was rebuilt per record from the same Connect schema and the same column map; the schema-staleness check rebuilt a lower-cased key set of the column map per record; every column declared type was re-parsed (ClickHouseColumn.of) per column per row and its range-policy label re-concatenated; and the temporal converters compiled their fixed DateTimeFormatter patterns per value. Measured on a production connector log (10,000-row batches of a 17-column ReplacingMergeTree table on one worker): 89 ms grouping + 172 ms bind/execute + 42 ms acknowledge/pace per batch, about 33k rows/s per worker, while system.query_log shows the same INSERT costs the server 11 ms. The new WriterHotPathBenchmarkTest reproduces the shape against a no-op JDBC surface: 116 ms grouping + 254 ms binding = 370 ms per 10,000 rows before this change; a JFR profile attributed 49% of samples to QueryFormatter.createColumns, 15% to PreparedStatementFieldMapper.parseColumn, 12% to refreshIfRecordHasUnknownColumn and 4% to DateTimeFormatter.ofPattern. Now: GroupInsertQueryWithBatchRecords memoises the template per (membership field list identity, column map identity, database) and the staleness verdict per (schema identity, column map identity) for the life of the grouper (one batch); PreparedStatementFieldMapper resolves each column parsed type, declared zone and range policy once per column per statement and discards the memo when the column map, configuration or table changes; DebeziumConverter holds its five fixed patterns as constants. Same SQL text, same parameter-index map object, same bound values, same failure paths -- nothing about membership, NULL handling, ordering or versioning is touched. After: 10 ms grouping + 189 ms binding = 199 ms per 10,000 rows (grouping 11.6x, overall 1.86x); the remaining bind cost is the per-value conversion and the JDBC surface itself. Spec: specs/04-query-generation/06-per-batch-metadata-memoisation.md (new; registered in specs/README.md). Tests: GroupInsertQueryTemplateMemoTest (2), PreparedStatementFieldMapperColumnBindingMemoTest (2); full sink-connector unit suite 649 tests, 0 failures, 0 errors, 3 skipped (pre-existing). Mutation checks: template memo disabled -> templateIsBuiltOncePerSchemaPerBatch fails (expected 2 template builds, was 300); map identity dropped from the template key -> templateIsRebuiltForADifferentColumnMap fails; mapper memo never invalidated -> bindingIsRebuiltForADifferentColumnMap fails (UTC rendering survives the map change). scripts/validate_specs.py --changed-base 93fde48: PASSED (70 specifications, 782 citations).
… identity; DDL test matrix (Spec 04.06 section 3.5) The template memo and the staleness-verdict memo were keyed on the column map by identity, and the column-binding memo was invalidated on a map identity change. The connector never mutates a column map in place -- DbWriter replaces it after every refresh -- and both memos live for one batch, behind the DDL barrier that drains every queue and every in-flight unit before a schema changes; still, correctness around a DDL must not rest on those two facts alone. The template key and the verified-schema entry now also carry the map content fingerprint (Map.hashCode over every name and declared type), and a bound column re-reads its declared type on every use and re-parses when it differs, so a map refreshed under the same reference mid-batch rebuilds the template, re-runs the staleness check and re-parses the column instead of serving anything pre-DDL. Cost: one pass over the map entries per record; the benchmark moves from 10 to 20 ms of grouping per 10,000 rows (209 ms total against 370 before the memos). Spec 04.06 section 3.5 enumerates the DDL cases and section 3.4 item 2 states the identity-and-content rule. Tests: GroupInsertQueryDdlMemoTest (5) -- pre- and post-ALTER records interleaved in one batch get their own templates with no metadata read; a stale cache is re-read exactly once and the fresh map keys every later record; an ALIAS column is proven absent once for 100 records (one listing, one default_kind probe); a pre-ALTER record carrying a dropped column still fails the batch with MissingTargetColumnException after a clean schema was memoised; a map changed in place rebuilds the template. PreparedStatementExecutorDdlMemoTest (1) -- a mixed pre/post-ALTER batch through addToPreparedStatementBatch against the recording JDBC surface: two statements, pre rows without a placeholder for the new column, post rows binding it including an explicit NULL. PreparedStatementFieldMapperColumnBindingMemoTest gains bindingIsRebuiltWhenTheDeclaredTypeChangesInPlace. Full sink-connector unit suite 656 tests, 0 failures, 0 errors, 3 skipped (pre-existing). Mutation checks: fingerprint dropped from the template key -> templateIsRebuiltWhenTheColumnMapChangesInPlace fails; declared-type re-check dropped -> bindingIsRebuiltWhenTheDeclaredTypeChangesInPlace fails. scripts/validate_specs.py --changed-base 93fde48: PASSED (70 specifications, 789 citations). The one destructive-looking site is a test display name describing a source-side DROP COLUMN scenario; the test destroys nothing. Destructive-Op-Check: sites=1; result=pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why:
A worker turned every batch into a JDBC batch by re-deriving, per ROW, metadata that is constant for the whole batch: the INSERT template (column list, placeholders, parameter-index map) was rebuilt per record from the same Connect schema and the same column map; the schema-staleness check rebuilt a lower-cased key set of the column map per record; every column's declared type was re-parsed (ClickHouseColumn.of) per column per row and its range-policy label re-concatenated; and the temporal converters compiled their fixed DateTimeFormatter patterns per value.
Measured on a production connector log (10,000-row batches of a 17-column ReplacingMergeTree(_version, is_deleted) table on one worker): 89 ms grouping + 172 ms bind/execute + 42 ms acknowledge/pace per batch, about 33k rows/s per worker, while system.query_log shows the same INSERT costs the ClickHouse server 11 ms. The new WriterHotPathBenchmarkTest reproduces that shape against a no-op JDBC surface: 116 ms grouping + 254 ms binding = 370 ms per 10,000 rows before this change. A JFR profile of it attributed 49% of samples to QueryFormatter.createColumns, 15% to PreparedStatementFieldMapper.parseColumn, 12% to refreshIfRecordHasUnknownColumn and 4% to DateTimeFormatter.ofPattern.
What:
Verification: