Repository navigation
2.11.1 Release - #1573
Open
minguyen9988 wants to merge 25 commits into
Open
2.11.1 Release#1573minguyen9988 wants to merge 25 commits into
minguyen9988 wants to merge 25 commits into
Conversation
* Bind binlog history _time as the instant of source.ts_ms Passing ts_ms through the zone-less DATETIME converter shifted _time by the zone offset. * Update binlog history spec for the _time instant binding * Add Lean, Dafny, TLA+ and Hypothesis specs for the history _time binding Checks that the bound text is the instant of source.ts_ms in every zone.
Source and ClickHouse zones are mandatory. Lossy conversions are counted and logged. Unsupported source operations and unknown config keys are reported. Jinja-Render-Check: rendered=1; formats=yaml; result=pass Destructive-Op-Check: sites=4; result=pass
Tests cover zones, narrowing reports, config key checks and shutdown order. Sample configs follow the new keys. Destructive-Op-Check: sites=11; result=pass
The MySQL zone flag is required. Checksums compare instants, not text. Coverage gaps fail the run.
Adds specs for zones, type narrowing, config checks and limitations. Extends the Lean replication model. Destructive-Op-Check: sites=14; result=pass
…figs Jinja-Render-Check: rendered=1; formats=yaml; result=pass
A connector started before its source listened exited at once. Probe errors.max.retries times first; the checks still refuse.
The string constants repeated the zeroed fraction the converter used to store. They now read 2299-12-31 23:59:59 followed by one nine per fractional digit, and a unit test compares each one with DataTypeRange.dateTime64Max for its scale.
…datatype suite The suite replaced the nines of the maximum with zeroes, which encoded the old loss of the fraction. It now expects 2299-12-31 23:59:59 followed by one nine per fractional digit.
…maximum The tests assumed an implicit zone or the zeroed fraction. They now set clickhouse.datetime.timezone to the server zone where a pre-created column has no zone, expect DateTime64 types in UTC, and compare the maximum of each scale. PostgresDDLOperationsIT builds its parser with the explicit zone config.
…nfo column of the offset table Each offset row can carry one JSON text with the connector version, the jar location, Java, OS, both time zones and the process start. The connector adds the column once with ADD COLUMN IF NOT EXISTS when the table lacks it. When the column is absent or cannot be added, the offset insert stays at its five columns and replication is unchanged. The offset value, the commit order and the commit timing do not change. Setting offset.extra.info.enable to false turns the feature off.
… datetime integration tests The block comment in DateTimeWithUserProvidedTimeZoneSchemaOnlyIT listed America/Chicago wall times that the assertions no longer use. Both tests now say that database.connectionTimeZone is set in src/test/resources/config.yml.
- fastjson 1.2.83 -> 1.2.84 (fixes CVE-2026-16723) - avro 1.9.0 -> 1.11.4 in sink-connector and sink-connector-lightweight (fixes CVE-2024-47561) - Update THIRD_PARTY_NOTICES.md for the new fastjson version Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Add OLake MySQL comparison to comparison doc
Fix CVE-2026-16723 and CVE-2024-47561: bump fastjson and avro
…y scale 2.11.1 stored two different values for one out-of-range source instant, which split the sorting key of every table whose temporal column saturated. Before this change the per-scale bound, dateTime64Max(p), was used both as the ACCEPTANCE range and as the saturation value. At scale 6 that bound is 2299-12-31 23:59:59.999999, while every earlier release, and any bulk load that clamps with the same constant, stored 2299-12-31 23:59:59.000000 for the same source value. Rows written before the upgrade therefore carry one sorting key and rows written after carry another. On a ReplacingMergeTree(_version, is_deleted) table whose temporal column is in the sorting key, a tombstone written after the upgrade no longer replaces the row it deletes: FINAL returns both rows, so the replicated table reads one row more than the source. The positive-infinity literal reaches the same split through boundedInstant. The acceptance range stays per scale. Only the stored saturation value becomes one constant, DataTypeRange.DATETIME64_SATURATION_MAX, at every scale, applied at all three sites that saturate: - the digits path clamp in checkIfDateTimeExceedsSupportedRange - the epoch-text path clamp via boundZonedTimestamp - the positive-infinity literal in boundedInstant Scale 9 keeps its own bound. Its representation limit is the Int64 nanosecond count in 2262, below the fixed constant, so ClickHouse rejects anything past 2262-04-11 23:47:16.854775807 with DECIMAL_OVERFLOW (code 407). Such a column saturates to its own bound, the one scale the fixed value cannot reach. Tests: DebeziumConverterRangePolicyTest covers the fixed value at scales 0, 3 and 6, the two carriers agreeing on one instant, and the scale-9 exception. TimeZoneMatrixTest, DebeziumConverterTest and PreparedStatementFieldMapperOutOfRangeTest cover the two column zones and the out-of-range bind path. Specs 07.03 section 3.1.8 and 11.02 state the rule, and formal specifications of it live under specs/lean, specs/dafny, specs/tla and specs/hypothesis. Co-Authored-By: Claude Code <noreply@anthropic.com> Formal-Verify: result=pass; fingerprint=a2b03031816a4e0971976322b09aee27d60a24be7d179fd97cc3c65e07149c79
Collaborator
Author
An out-of-range DateTime64 must saturate to one fixed instant at every scaleCommit 17d6e92 fixes a data divergence a MySQL-vs-ClickHouse cross-check found after 2.11.1: 28 of 186 replicated tables reported MORE rows on the ClickHouse side than the source, by up to 438 rows. What brokedateTime64Max(p) is the correct ACCEPTANCE bound for a column of scale p. 2.11.1 also began using it as the SATURATION value, the value stored for a source value outside that bound. In boundedInstant this covered the infinity literal as well, through DataTypeRange.dateTime64Max(policy.scale()). That made the stored value depend on the column scale, so the same source value could be stored two different ways: Every release before 2.11.1, and any bulk load that clamps with the same constant, store the second form. At tag 2.11.0 the scale constants in DataTypeRange.java are all 2299-12-31 23:59:59.0; the per-scale values and the clamp are new on this release branch. Why that produces extra rowsThese columns are part of the sorting key, in tables replicated as ReplacingMergeTree(_version, is_deleted). Two stored forms for one source value are two sorting keys, so a tombstone written after the upgrade no longer replaces the row it deletes. FINAL then returns both, and the table holds one row more than the source. Observed on a DateTime64(6, America/Chicago) sorting-key column that carries a 9999-12-31 23:59:59 open-ended sentinel, where the sentinel split into exactly the two values above across three replicated targets. In one table a single primary-key value carried a tombstone at .999999 with is_deleted=1 that could not replace its row at .000000, so the replicating table read 941 rows against 940 in MySQL. The digits path and the epoch-text path also disagreed with each other on the stored instant, because the clamp was applied per carrier rather than from one constant. The fixThe acceptance range stays per scale; only the stored saturation value becomes one constant, DataTypeRange.DATETIME64_SATURATION_MAX, applied at all three sites that saturate:
Scale 9 keeps its own bound. Its representation limit is the Int64 nanosecond count in 2262, which is below the fixed constant, and ClickHouse raises DECIMAL_OVERFLOW (code 407) for anything past 2262-04-11 23:47:16.854775807. Note for anyone already running 2.11.1This change stops new writes from splitting the key. Rows already written at the per-scale bound keep the other form and need a backfill or re-sync of the affected tables on the ClickHouse side; the source is unaffected. Verification
🤖 Generated with Claude Code |
Three sites stored a value the source never held, each one borrowing a bound that belonged to something else. DateTime64: the per-scale acceptance limit was also used as the SATURATION value. Acceptance is per scale and stays so; the stored sentinel must be one fixed instant because it lands in a ReplacingMergeTree sorting key. Two stored forms of one source value (2299-12-31 23:59:59.999999 before, .000000 after) are two keys, so a tombstone written after the upgrade could not replace the row it deletes and FINAL returned both. DATETIME64_SATURATION_MAX is now that one fixed instant at every scale except 9, which is capped by the Int64 nanosecond representation and keeps its own bound. Decimal: an out-of-range decimal saturated to BinaryStreamUtils.DECIMAL128_MAX, 2^127-1 with 39 digits, whatever the column was. ClickHouse accepts at most 38 digits, so the bounded value was one the server refuses: toDecimal128OrNull rejects it on 24.8.14. The bound is now computed from the COLUMN's declared precision and scale (DataTypeRange.decimalMax), read from the declared type by ClickHouseDataTypeMapper.decimalPrecisionScale, which also understands the fixed-width spellings Decimal32/64/128/256. Python toolset: the loader and the pg mapper clamp against the per-precision bound text (datetime64_max_text), so a scale-9 load no longer parses a 2299 literal at precision 9 and fails with Code 407. The three convertWithoutTimeZoneAdjustment helpers have no caller in the repo and shipped in released tags, so they are deprecated with their two hazards documented rather than deleted: they bound every column with the scale-6 range and render wall-clock digits in serverTimezone instead of epoch text. CLICKHOUSE_MAX_SUPPORTED_DATETIME64 is kept as a deprecated alias for binary compatibility and says it is an acceptance bound only. Verified in an isolated sandbox on this head: Java Tests run: 193, Failures: 0, Errors: 0, Skipped: 5, BUILD SUCCESS; Python toolset 995 passed, 2 skipped, 93 subtests passed plus 22 in the new module; four-tool formal verification PASS. Formal-Verify: result=pass; fingerprint=f670e38859669de97f73966a8e12769c1e0aaa06d093cae19739462ed35d5cc3 Co-Authored-By: Claude Code <noreply@anthropic.com>
#1578) A MySQL Shell dump writes TIMESTAMP values as text in the zone the dump records. The dump's @.json carries "tzUtc": true, so that text is UTC. The loader passed the field through to the ClickHouse client unchanged, and a zone-less DateTime64(p) is parsed in the AMBIENT zone of the load: the client zone under --use_client_time_zone 1, else the server's. The same dump therefore stored different instants on two hosts. Measured on ClickHouse 24.8.14 with the dump's text 2018-04-05 20:44:35.772929 and 2018-12-01 00:18:03.545683, reading toInt64() of the stored value: declaration TZ=UTC TZ=America/Chicago TZ=Europe/London DateTime64(6) 1522961075772929 1522979075772929 1522957475772929 DateTime64(6) 1543623483545683 1543645083545683 1543623483545683 DateTime64(6,'UTC') 1522961075772929 1522961075772929 1522961075772929 DateTime64(6,'UTC') 1543623483545683 1543623483545683 1543623483545683 The zone-less April reading moves by the 5-hour summer offset and the December reading by 6 hours; the pinned declaration stores one instant everywhere, and it is the instant the text names. timestamp_input_datatype(datatype, timezone) pins the zone the loader resolved, from the dump or from --source_timezone, into the declaration. A datatype that already names a zone is returned untouched, so --clickhouse_datetime_timezone still decides that column, and a second pass cannot drift. The emitted INSERT reads the text in the named zone and the target column's own type keeps that instant. The defect showed up as a checksum divergence between a source table and its replica whose row counts matched: the replica held the same instants as the source only when the load and the source agreed on a zone. Loading the same dump under the declaration that names the dump's zone stores the source's instants on every host, whatever the ambient zone is. Verified in an isolated sandbox on this head: the loader suite 17 passed; specs validated 94, citations 1423, Validation PASSED. Formal-Verify: result=pass; fingerprint=574f88f0f8830a78584cd78d285c5b979bacdc0307124ebb452698e6fc6652b2 Co-authored-by: Claude Code <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.