8000 Releases · Altinity/clickhouse-sink-connector · GitHub
[go: up one dir, main page]
Skip to content

Releases

Release v2.11.0

Release v2.11.0 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 15 Oct 23:26
2.11.0
f647404
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Lightweight: Docker

ClickHouse Sink Connector Release v2.11.0 Notes

2.11.0 is the spec-driven release. Every replication invariant is now declared in specs/ and machine-checked in Lean 4 (formal_specs/), and the runtime was hardened to match: offsets commit in FIFO order, _version is monotonic across restarts, the data path is bounded in bytes, and PostgreSQL is a first-class source.

Highlights

  • PostgreSQL CDC support — PostgreSQL is now a first-class source alongside MySQL: WAL streaming ported onto the 2.11.0 line, a ch-pg-dump snapshot tool anchored to the replication slot, honest PG verification verdicts, and a Postgres type mapper. (#1435, #1540, #1541, #1530)
  • Spec-driven development with Lean 4 formal verification — the repository now declares every behavioural invariant in specs/ and proves offset ordering, version monotonicity and convergence in formal_specs/lean, validated in CI on every pull request. (#1460, #1463, #1473, #1466)
  • Primary-key rebuild at the DDL barrier — when the source changes a primary key, the connector rebuilds the replica (swap + backfill, ON CLUSTER in replicated mode, cancellable safely) instead of silently drifting. (#1483)
  • SCD2 history and replication-log-only modes — per-table slowly-changing-dimension history and a replication-log-only mode, specified and proven with a formal model. (#1507, #1511, #1512)
  • Key-aware row routing — one hot table can spread across writer threads while preserving per-key ordering. (#1523, #1465)
  • Writer throughput — INSERT templates, column types and formatters derived once per batch instead of once per row (1.9x per-batch throughput, byte-identical output); queued batches behind the polled one coalesce into a single write bounded by buffer.max.records / buffer.max.bytes; coalesce.max.wait.ms bounds ClickHouse part creation per wait instead of per poll. (#1524, #1525, #1526)
  • Compressed binlog transactions — MySQL compressed transaction payloads replicate correctly and stream in constant memory regardless of uncompressed size; dead binlog connections restart instead of stalling. (#1531, #1533, #1534)
  • ch-mysql-resync repair tool — re-synchronise replica tables after a source change that bypassed the binlog (sql_log_bin=0 patch): dump, count-reconciled scratch load, atomic REPLACE PARTITION, connector offset rewind. (#1521, #1542)
  • Consistent checksums — db_compare can lock and wait for the connector, or take per-slice consistent snapshots, and a checksum driver never reports an unverified table as equal. (#1548, #1549, #1550, #1551, #1539)
  • Bounded memory and backpressure — a hard cap on rows handed off but not yet acknowledged (the reader pauses instead of filling the heap), the data path bounded in bytes not only rows, large INSERT chunks spilled to disk, and the process-lifetime leaks closed. (#1503, #1505, #1536, #1506)
  • Offset-commit correctness — offsets commit in handoff-sequence FIFO order, a written batch is never re-executed, worker death stops the pipeline loudly, and the OffsetStorageWriter flush-semaphore leak is closed. (#1471, #1447, #1464, #1566)
  • Monotonic _version across restarts — a durable version floor seeded at start, GTID path floored by commit order; a commit that reaches the binlog late can no longer lose to an earlier write of its key. (#1478, #1485, #1452)
  • DDL translation overhaul — MySQL ALTER/CREATE translated to what ClickHouse actually accepts, generated columns become DEFAULT (never MATERIALIZED), every handoff path drains before a DDL, and an unapplied DDL halts the pipeline loudly instead of being swallowed. (#1472, #1479, #1442, #1445, #1461)
  • Initial snapshot reliably completes — heartbeats are enabled and the post-snapshot heartbeat offset is committed, so snapshot_completed is persisted and the engine transitions to streaming. (#1438, #1428, #1467)
  • Apache-2.0 license compliance — no GPL/LGPL code in the connector JARs; THIRD_PARTY_NOTICES.md and full license texts ship with the distribution. (#1560, #1561, #1564)

Behaviour changes to note on upgrade

  • clamp.out.of.range defaults to true — an out-of-range source value saturates to the column's bound and is reported, instead of stopping replication. (#1490)
  • connect.keep.alive defaults to false — a lost binlog connection restarts the engine at a transaction boundary instead of resuming mid-statement and silently dropping rows. (#1500)
  • FATAL failures are terminal at once, never retried; the engine retry budget refills on progress, not on a bare start. (#1487, #1489)
  • TOO_MANY_PARTS (252) is retriable backpressure, not fatal. (#1462)
  • Per-batch progress lines moved to DEBUG; the resume replay is summarized instead of dumping every skipped row image; hard-cap and high-water-mark warnings are paced. (#1488, #1509, #1504)
  • Debezium JDBC storage keys renamed — shipped configs use the current offset.storage.jdbc.offset.table.* spellings; the pre-2.7.1 names still work but log a deprecation error on every start. (#1518)

What's Changed

  • fix: post-snapshot heartbeat offset is never committed, so the snapshot never finishes (#1379) by @minguyen9988 in #1428
  • Port #1431 + #1398 onto the recreated 2.11.0 branch (PostgreSQL CDC + DDL/DML race fix) by @minguyen9988 in #1435
  • fix: MATERIALIZED column metadata storm, and report a shadowed source value by @minguyen9988 in #1436
  • fix: enable heartbeats, without which the initial snapshot never completes (#1379) by @minguyen9988 in #1438
  • ci: the java-lightweight job has never run any t...
Read more

Contributors

subkanthi, alex-zaitsev, and minguyen9988
Assets 8
Loading

Release v2.10.3

Release v2.10.3

Choose a tag to compare

View all tags
@github-actions github-actions released this 31 Aug 21:54
2.10.3
e7da4d6
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

  • urgent fix: a keyless source table must not prevent the connector from starting by @minguyen9988 in #1426

Full Changelog: 2.10.2...2.10.3

Contributors

minguyen9988
Loading

Release v2.10.2

Release v2.10.2

Choose a tag to compare

View all tags
@github-actions github-actions released this 31 Aug 02:45
2.10.2
689fd32
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

  • fix: /metrics served with no Content-Type, breaking Prometheus 3.x scrapes by @minguyen9988 in #1413
  • fix: INT8 (Byte) hard-cast to Integer fails every TINYINT batch by @minguyen9988 in #1414
  • fix: ARRAY binding hard-casts to ArrayList, breaking every other List type by @minguyen9988 in #1415
  • fix: initial snapshot never finishes -- NPE on every unparseable record by @minguyen9988 in #1416
  • fix: MySQL DOUBLE auto-created as Float32, silently truncating precision by @minguyen9988 in #1417
  • fix: support PostgreSQL infinity values in timestamptz by @minguyen9988 in #1418
  • fix: batch containing a missing ClickHouse table is silently dropped and the offset advances by @minguyen9988 in #1419
  • fix: _version stores UInt64 max when the source has no GTID, permanently winning RMT dedup by @minguyen9988 in #1420
  • Added release notes for 2.10.1 release by @subkanthi in #1412

Full Changelog: 2.10.1...2.10.2

Contributors

subkanthi and minguyen9988
Loading

Release v2.10.1

Release v2.10.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 27 Aug 05:52
2.10.1
d41888a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

  • fix: history mode loses ~95% of rows when a CREATE is replayed by @minguyen9988 in #1410
  • fix: history-mode DELETE lost -- _operation dropped from INSERT, and the delete races its own batch by @minguyen9988 in #1411

Full Changelog: 2.10.0...2.10.1

Contributors

minguyen9988
Loading

Release v2.10.0

Release v2.10.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 25 Aug 22:17
2.10.0
b65433a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

ClickHouse Sink Connector Release v2.10.0 Notes

2.10.0 is a large reliability release. It stops several classes of silent data loss (DDL ordering, TRUNCATE timing, keyless tables, BIT/DOUBLE mapping, history-mode row drops, offset commits), adds dump/load/compare tooling, and hardens how the connector fails.

Full Changelog: 2.9.1...2.10.0

What's New

  • ch_sink_tools Python toolkit — dump, load, and compare MySQL/PostgreSQL against ClickHouse (ch-checksum, ch-pg-dump, ch-mysql-dump, and related commands). Passwords are masked in logs, eval() is removed from checksum formatting, and table locks are held only for the duration of each table checksum. (#1272, #1356, #1357, #1318)
  • Ansible + systemd deployment for the lightweight connector (no Docker, Kubernetes, or Kafka required). See deploy/ansible-systemd/. (#1338)
  • Optional REST API Basic Auth — set rest.api.auth.enabled=true with rest.api.auth.username / rest.api.auth.password. Auth remains off unless enabled. (#1300)
  • Non-locking snapshot options — additional snapshot locking modes so initial snapshots can avoid holding source table locks. (#1380)
  • db_time column on binlog-history tables (DateTime MATERIALIZED now()) for when the row was written to ClickHouse. (#1283)
  • ClickHouse JDBC driver 2.10. (#1278)
  • Fatal ClickHouse error classification — authentication, schema mismatch, type mismatch, too-many-parts, and similar deterministic failures stop the connector instead of retrying the same batch forever and blocking every table. Transient errors still retry. (#1315)
  • Helm chart — connector.snapshotMode (default schema_only) and connector.disableDropTruncate (default true) are now configurable. (#1359)
  • Updated show_replica_status view. (#1328)
  • Sysbench automated tests and pytest checksum integration tests. (#1354, #1351)

Breaking Changes

Upgrade notes — read these before rolling 2.10.0 into production.

  1. Keyless MySQL tables can refuse startup (#1401, #1405, #1406, #1407)

    A table with no PRIMARY KEY and no non-null UNIQUE key has no row identity in the binlog. The connector now:

    • Refuses to start if it finds such tables on the source (and prints the exact ALTER TABLE ... ADD COLUMN my_row_id ... fix).
    • Refuses keyless tables on MySQL < 8.0.30 and MariaDB (Generated Invisible Primary Key is not available).
    • Does not enable GIPK on the source. The connector is read-only against MySQL.
    • Does not block startup for tables that are already excluded from replication.

    What to do: add a GIPK (or a real primary key) on the source, or set keyless.table.check.skip=true if you accept silent row collapse.

  2. Helm defaults no longer snapshot existing data or apply DROP/TRUNCATE (#1359)

    Setting Previous chart behavior 2.10.0 default
    connector.snapshotMode pinned to initial schema_only
    connector.disableDropTruncate not configurable true

    Fresh Helm deploys will not load existing source rows and will ignore DROP / TRUNCATE. Override in values.yaml if you need the old behavior:

    connector:
  snapshotMode: "initial"
  disableDropTruncate: "false"

  3. Fatal ClickHouse errors stop the connector (#1315)

    Auth failures, unknown table/database/column, type mismatch, TOO_MANY_PARTS, and similar errors used to retry forever and stall the whole pipeline. They now fail the task. Configure alerting/restarts around a stopped connector; do not expect an infinite retry loop to hide a bad schema or credential.

  4. Sample Docker config no longer enables replicated table engines (#1301)

    auto.create.tables.replicated is commented out / false in the sample config.yml. New tables are created as ReplacingMergeTree, not ReplicatedReplacingMergeTree. Set auto.create.tables.replicated: "true" if you need replicated engines.

  5. Auto-created column types changed for new tables

    Existing ClickHouse tables are not rewritten. Newly auto-created tables use:

    • BIT(n) for n > 1 → binary string, not Bool (#1386, #1387, #1390)
    • DOUBLE / DOUBLE PRECISION / FLOAT8 → Float64 (was Float32) (#1386)
    • REAL → does not claim Float64 precision it never receives (#1397)
    • Unsigned integer mapping workaround for a Debezium bug (#1306)

    Recreate (or MODIFY COLUMN) any table that was auto-created under the old mapping if you need the corrected types.

Bug Fixes

CDC / DDL ordering

  • Apply DDL at its position in the stream so pre-ALTER rows are not written after the schema change. (#1385)
  • Execute TRUNCATE at its binlog position, not after the batch inserts. (#1382)
  • ALGORITHM / LOCK clauses no longer discard later operations in the same ALTER. (#1381)
  • Fix double database prefix on TRUNCATE, DROP, and RENAME. (#1316)
  • Invalidate the cached writer for the source table of a RENAME. (#1384)
  • Wait for ALTER TABLE visibility before cache invalidation. (#1321)
  • Fix cache invalidation logic (with integration test). (#1333)
  • Generated ADD / DROP / RENAME DDL is replay-safe after a restart (IF EXISTS / IF NOT EXISTS). Replay safety uses version ordering, not equality. (#1402, #1403, #1404)
  • DDL waiter no longer parses IF NOT EXISTS as the column name. (#1408)
  • Default MODIFY / CHANGE COLUMN to Nullable when ClickHouse schema metadata is unavailable. (#1339, #1341)
  • A column rename no longer NULLs rows still in flight. (#1395)
  • Pre-ALTER rows NULL-fill newly added columns instead of failing. (#1389)

Data integrity

  • Delete operations are persisted. (#1281)
  • _version is anchored to the source commit timestamp so a re-delivered DELETE cannot out-rank a later re-INSERT. Formula stays the 2.8.0 ts_ms * 1_000_000 + counter domain (upgrade- and rollback-safe). (#1378)
  • History mode no longer drops all but the first row of a multi-row binlog event. (#1391)
  • Kafka Connect preCommit only advances offsets after a durable ClickHouse insert. (#1345)
  • Retry offset flushing on process error. (#1335)
  • A nullable UNIQUE key no longer stalls the entire stream. (#1396)
  • Keyless / no-PK tables no longer collapse to a single row (see Breaking Changes for the GIPK requirement). (#1389, #1394, #1401)

Types and schema

Read more
Loading

Release v2.9.1

Release v2.9.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 14 Apr 02:42
2.9.1
999b63d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Lightweight
Docker: altinity/clickhouse-sink-connector:2.9.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.9.1-lt/images/sha256-bc181406a7169418c0bfe623d820bc2ae239db532c8ba8b5a833cf762fe4341b

What's Changed

  • Changed _time column in binloghistory from DateTime64 to DateTime to … by @subkanthi in #1239
  • enable publishing lightweight jar and disable kafka jar and docker pu… by @subkanthi in #1240
  • Fix NullPointerException when config properties are missing by @subkanthi in #1273
  • Added logic to cache serialized results. by @subkanthi in #1247
  • Fix DST transition offset. by @subkanthi in #1271

Full Changelog: 2.9.0...2.9.1

Contributors

subkanthi
Loading

Release v2.9.0

Release v2.9.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 16 Feb 04:04
2.9.0
546b3e4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Lightweight
Docker: altinity/clickhouse-sink-connector:2.9.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.9.0-lt/images/sha256-75bfc3a63d5b57bd712bcf41958f3bce3bc612d5e94a51ff70bf697e2f0e2c49

What's New

What's Changed

Full Changelog: 2.8.0...2.9.0

Contributors

aadant, subkanthi, and MyroTk
Loading

Release v2.8.0

Release v2.8.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 06 Nov 02:07
2.8.0
75b7bc9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Lightweight
Docker: altinity/clickhouse-sink-connector:2.8.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.8.0-lt/images/sha256-3799c182de1e91a1dd1434d2f2035a6bdbb036d9a1f74bad64c8b537446db7f5

What's New

  • Multi-platform(ARM) builds
  • Oracle support(Experimental)
  • Version History(Temporal tables) (Experimental)
  • Schema Override
  • support for more MySQL DDL statements.
  • Debezium upgrade to 3.1.3.Final

What's Changed

New Contributors

Full Changelog: 2.7.1...2.8.0

Contributors

aadant, subkanthi, and 3 other contributors
Loading

Release v2.7.1

Release v2.7.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 09 Jul 22:31
2.7.1
057e3ab

Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.1-lt/images/sha256-6aeb1dcbcd7b2f0e0f43f3f31cdc198a594977b2c2e338552465ad50eebd02ef

What's Changed

New Contributors

Full Changelog: 2.7.0...2.7.1

Contributors

subkanthi and mohaidoss
Loading

Release v2.7.0

Release v2.7.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 30 Jun 16:35
2.7.0
8aa907e

Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.0-lt/images/sha256-2e0f1a0487a71a11b19200320707545ec5c5992b0abc37ad09ec954c36352f37

Breaking Changes

Configuration Changes
offset.storage.jdbc.offset.table.name to offset.storage.jdbc.table.name
offset.storage.jdbc.offset.table.ddl to offset.storage.jdbc.table.ddl
offset.storage.jdbc.offset.table.delete to offset.storage.jdbc.table.delete

What's new

What's Changed

New Contributors

Full Changelog: 2.6.0...2.7.0

Contributors

subkanthi, philippevezina, and chimeyrock999
Loading
0