Repository navigation
Upgrade embedded Debezium to 3.7.0.Final (fixes #1450 schema-history recovery) - #1558
minguyen9988 wants to merge 8 commits into
Conversation
…recovery) Debezium 3.1.3 JdbcSchemaHistory splits a schema-history record longer than 65000 characters into rows with distinct ids and parses each row alone on recovery, so the first restart after a ~200-column table is captured fails with JsonEOFException (#1450). DBZ-8979 (3.3.0) fixed recovery. This moves to 3.7.0.Final and handles what the upgrade changes: schema-history store layout and migration (spec 09.05), MySQL TRUNCATE routing (spec 01.11), removed snapshot.mode values, legacy MySQL grammar package, removed ConnectorDescriptor, 3.7 constructors of the shadowed classes, debezium-storage-common, Kafka 4.3.1 (spec 02.06 3.4). Destructive-Op-Check: sites=12; result=pass; note=TRUNCATE mentions are log text, a routed MySQL truncate translated for the existing DDL path, and test assertions; no new destructive operation Jinja-Render-Check: rendered=2; formats=yaml; result=pass
Reset Debezium 3.7's lsn_events_processed when a PostgreSQL offset is edited to a new LSN; a stale counter made the connector skip real events at the new position (test: OffsetEditValidationTest.lsnEditClearsStaleEventsProcessedCounter, spec 09.03). Spec 01.08: restarts inside a compressed transaction on Debezium 3.3+, and FM-01.08-8, the downgrade hazard (a 3.3+ offset inside a compressed transaction resumed by a 3.1.3 build loses that transaction and every later compressed one; reproduced, workaround verified). Spec 01.07: Debezium 3.7 bundles mysql-binlog-connector-java 0.41.5 with the dbz#2359 fix.
…SQL driver, MariaDB Schema-history preflight (spec 09.05): on a first start the database in the schema-history JDBC URL (altinity_sink_connector by convention) does not exist yet; setup() creates it only after the preflight (createDatabaseForDebeziumStorage). The preflight got UNKNOWN_DATABASE (Code 81), treated it as an unreachable ClickHouse, waited 5 minutes and refused to start, so every suite that starts the connector against an empty ClickHouse (docker-compose suites, checksum, python e2e, testflows, and the ITs whose URL names the database) never replicated. The preflight now creates that database once with CREATE DATABASE IF NOT EXISTS over the same URL without the path, then verifies the table as before; a failed creation is reported and retried like an unreachable server. Reproduced end to end with the CI docker/config.yml and no storage database: the previous build never converged, this one converges in seconds. PostgreSQL: the quarkus BOM pins org.postgresql:postgresql 42.5.0, overriding the 42.7.13 Debezium 3.7 needs; debezium-connector-postgres 3.7 calls ChainedLogicalStreamBuilder.withAutomaticFlush(boolean) and every stream died with NoSuchMethodError. Pinned 42.7.13 in the module's dependencyManagement. MariaDB: Debezium 3.7's MySqlConnector treats MariaDB 10.x as MySQL >= 8.4 and fails with "should support SHOW BINARY LOG STATUS". Ship debezium-connector-mariadb (with the 3.x MariaDB JDBC driver Debezium 3.7 builds against, 3.5.3; its SPDX license id LGPL-2.1-or-later is accepted explicitly, the same library was accepted as LGPL-2.1) and run MariaDBIT with io.debezium.connector.mariadb.MariaDbConnector. Spec 02.06 section 3.4 and FM-02.06-5 document the connector.class change for MariaDB deployments. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
CI fix pushed (3c8ed22). Three causes, all from the Debezium 3.7 upgrade:
Local verification on this head: spec validator 85/85; lightweight unit suite 811 tests with only the container-only classes failing (same set as on 2.11.0); end-to-end wide-table restart, DDL churn/restart regression, and the CI config with and without a pre-created storage database all converge value-equal. |
….7. jackson-annotations was left unmanaged and resolved transitively through jackson-dataformat-yaml (pinned to 2.12.6), winning Maven nearest-dependency mediation over the 2.20.1 jackson-databind depends on; the classpath skew made jackson-databind 2.20.1 call JsonProperty.isRequired(), added to jackson-annotations only in 2.14+, throwing NoSuchMethodError in MariaDbCharsetRegistry static init under Debezium 3.7 MariaDbConnector and crash-looping the embedded engine on every restart (CI: java-lightweight / MariaDBIT#133 UNKNOWN_TABLE employees.audience, Restarting the engine - retry 5 of 10 observed before the assertion). Pin jackson-annotations to 2.20.1 and bump jackson-dataformat-yaml to 2.20.1 so all jackson modules converge. PR 1558 CI run 37259832067 job java-lightweight 111606331085. Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
|
java-lightweight was failing MariaDBIT#testMultipleDatabases and DDLIgnoreRegExIT#testDDLIgnoreRegex because the embedded engine crash-looped on startup: jackson-databind 2.20.1 called JsonProperty.isRequired() on jackson-annotations, which only gained that OptBoolean-returning overload in 2.14+, while jackson-annotations itself resolved transitively through jackson-dataformat-yaml (pinned to 2.12.6) and won Maven nearest-dependency mediation over the 2.20.1 jackson-databind needs. That threw NoSuchMethodError in MariaDbCharsetRegistry static init under the Debezium 3.7 MariaDbConnector on every restart attempt (log shows Restarting the engine - retry 5 of 10 before the test assertions fire), so neither test's expected rows ever replicated. Fixed by pinning jackson-annotations explicitly to 2.20.1 and bumping jackson-dataformat-yaml to 2.20.1 so all jackson-* modules converge on one version: commit f9b34fa. |
…on-annotations does not follow jackson-core/jackson-databind's X.Y.Z scheme for this release cycle: Maven Central published only 2.20-rc1 and 2.20 for that l 8000 ine (no 2.20.1), so the build failed dependency resolution outright (Could not find artifact com.fasterxml.jackson.core:jackson-annotations:jar:2.20.1 in central) on PR 1558 CI run 37371653966, job build-kafka-lightweight / build (111970021096), before any test could run. Pin jackson-annotations to 2.20 instead, which exists and still converges the classpath on jackson-databind 2.20.1'''s required JsonProperty.isRequired() surface. Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
|
CI fix: build-kafka-lightweight/build failed on the previous head (run 37371653966, job 111970021096) with Could not find artifact com.fasterxml.jackson.core:jackson-annotations:jar:2.20.1 in central -- the prior commit pinned jackson-annotations to 2.20.1 to match jackson-core/jackson-databind, but jackson-annotations does not follow that X.Y.Z scheme this release: Maven Central only publishes 2.20-rc1 and 2.20 for that line. Pinned jackson-annotations to 2.20 instead (jackson-dataformat-yaml stays at 2.20.1, which does exist), verified against Maven Central's metadata. Pushed as 770040a. |
…. testDDLIgnoreRegex started the Debezium engine's setup() asynchronously and immediately executed the first DDL (the partitioned CREATE TABLE), relying only on a blind 15s sleep afterward for the engine to catch up. If the engine's connect/snapshot/binlog-attach sequence had not finished by the time MySQL ran that DDL, the event was never captured at all, not delayed, missed, and the later analyze partition assertion read null instead of the DDL text (DDLIgnoreRegExIT.java:106, reproduced on heads 3c8ed22, 61591e8, and this PR's 770040a, run 37384341533 job 112017353417). DebeziumChangeEventCapture#markEngineStarted() sets ReplicationStatusSingleton.isReplicationRunning() true as the first action of the engine's connectorStarted() callback, and onConnectorStopped()/the terminal failure path reset it false, so the flag reflects an attached and running engine for exactly its lifetime. Poll it (up to 60s) before issuing any DDL instead of guessing a fixed delay. Verified: traced setup()/connectorStarted()/markEngineStarted() and onConnectorStopped() in DebeziumChangeEventCapture.java to confirm the singleton's semantics. Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
|
Fixed the DDLIgnoreRegExIT flake (java-tests-lightweight run 37384341533, job 112017353417): the 1 failure was testDDLIgnoreRegex asserting expected [alter table sales analyze partition p2022] but was [null]. Root cause: the test starts the Debezium engine's setup() on a separate executor thread and immediately issues the first DDL (the partitioned CREATE TABLE), relying only on a blind 15s sleep afterward. If the engine's connect/snapshot/binlog-attach sequence had not finished by the time MySQL ran that DDL, the event was never captured at all, not delayed but missed, and the later analyze-partition assertion read null instead of the DDL text. This reproduced consistently across the last several CI runs on this branch. Fix: poll ReplicationStatusSingleton.isReplicationRunning(), set true by DebeziumChangeEventCapture's connectorStarted() callback, the first action once the engine actually attaches, for up to 60s before issuing the first DDL, instead of guessing a fixed delay. Pushed as 9bb1f78. This PR's other known CI issues, checksum-lightweight, python-toolset-e2e-mysql and the testflows suites all timing out waiting for replication parity under the upgraded Debezium MariaDB connector, are still being triaged separately and are not yet fixed. |
…ocker compose logs was redirected straight to a .txt file only downloadable as a workflow artifact once the whole run completes; piping through tee keeps that file (and the upload-on-failure step) unchanged while also streaming the same output to the job's own stdout log, so a failing checksum-lightweight / e2e-mysql run can be triaged from the live/completed job log immediately instead of waiting on the artifact. Applies to python-toolset-e2e-mysql.yml (Dump compose logs on failure) and sink-connector-lightweight-checksum-tests.yml (init_mysql and sysbench failure dumps). Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
|
Pushed an observability-only CI change: pipe the compose/connector log dumps through tee instead of a pure redirect in python-toolset-e2e-mysql.yml and sink-connector-lightweight-checksum-tests.yml, so those logs are visible in the job's own live log instead of waiting on the artifact upload. This does not change any test or connector behavior, so the existing 7 failing checks are expected to still fail on this head; the goal is to get enough log visibility on the next run to find the actual root cause of the checksum-lightweight / python-toolset-e2e-mysql / testflows connector-table-creation failures. |
…MySQL grammar package. 2.11.0 added this class importing io.debezium.ddl.parser.mysql.generated.MySqlParser; this branch's Debezium 3.7 upgrade moved the generated MySQL DDL ANTLR parser to io.debezium.ddl.parser.mysql.legacy, so the merged build fails with cannot find symbol on DefaultValueContext/StringLiteralContext. All sibling files in this package already import .legacy (MySqlDDLParserListenerImpl, MySQLDDLParserService, DataTypeConverter, GeneratedExpressionBitOperators), and MySqlDDLParserListenerImpl's own identical inline literalDefault/clickHouseString/isCurrentTimestampDefault/firstEnumMember methods use the same context classes under .legacy, confirming they exist there unchanged. Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
|
New head c78815a fixes a build-break on the previous head (3c710e1): DefaultLiteralTranslator.java (added by this PR's Debezium 3.7 upgrade work) imported io.debezium.ddl.parser.mysql.generated.MySqlParser, but this branch relocated the generated MySQL DDL ANTLR parser to io.debezium.ddl.parser.mysql.legacy, so build-kafka-lightweight / build failed with cannot find symbol on DefaultValueContext/StringLiteralContext and every downstream job was skipped. Fixed by switching the import to .legacy, matching every sibling file in the same package (MySqlDDLParserListenerImpl, MySQLDDLParserService, DataTypeConverter, GeneratedExpressionBitOperators) which already use it. This class is not yet called from anywhere else in the codebase, so the fix is import-only with no behavior change. CI is running on the new head now. |
|
Status update: GitHub Actions has not triggered any check runs for this branch head (c78815a) for 8+ hours since it was pushed, even though the prior head on this branch triggered checks (Spec Governance / Python toolset unit tests / Pull Request Pipeline) within seconds of its own push. Confirmed via git ls-remote that the push landed correctly - refs/heads/fix-debezium-3.7-schema-history and refs/pull/1558/head both resolve to c78815a. This looks like a CI-trigger/Actions-infra gap rather than anything in the diff. The branch also now shows as not mergeable against 2.11.0 (base has moved since the merge-base). A maintainer may want to push an empty commit, or close/reopen, to retrigger checks. |
|
Status update: GitHub Actions has still not triggered any check runs for this branch head (c78815a) more than 24 hours after it was pushed. The previous head on this branch did trigger checks normally, so this looks like either a CI dispatch issue on the fork side or something related to the branch currently showing as not mergeable against the base. Could a maintainer confirm whether the workflow needs a manual re-run/re-dispatch, or whether rebasing onto the current base branch would unstick it? Happy to rebase and re-push once we know which it is. |
|
Status update: GitHub Actions has now gone more than 48 hours without triggering any check run for this branch head (c78815a), and the branch still shows as not mergeable against the base. The previous head on this branch did trigger checks normally. Could a maintainer confirm whether this needs a manual workflow re-dispatch, or whether we should rebase onto the current base and re-push to get CI moving? Happy to do either once we know which is needed. |
|
Status update: GitHub Actions has now gone more than 72 hours without triggering any check run for this branch head (c78815a), and the branch still shows as not mergeable against the base. The previous head on this branch did trigger checks normally within seconds of being pushed. At this point a manual workflow re-dispatch or a rebase-and-repush onto the current base seems like the most likely way to get CI moving again - happy to do either once a maintainer can confirm which is preferred, or whether something else is blocking dispatch on this head. |
|
Status update, 96h+ since the last push (head c78815a): CI has still never triggered a single check run on this head. The PR remains in CONFLICTING mergeable state against the moving 2.11.0 base, which appears to be the reason the pull_request workflows are not dispatching. Every earlier head on this branch triggered Spec Governance / Python unit tests / Pull Request Pipeline within seconds of being pushed, so this is specific to the current head. We are happy to either (a) push a rebase of this commit onto the current 2.11.0 tip to resolve the conflict and re-trigger CI, or (b) wait for a maintainer to confirm a preferred approach -- please let us know which you would prefer, or feel free to rebase/retrigger directly. |
|
Status update, 120h+ since the last push (head c78815a): CI has still never triggered a single check run on this head. The PR remains in CONFLICTING mergeable state against the moving 2.11.0 base, which appears to be the reason the pull_request workflows are not dispatching. We are still happy to either (a) push a rebase of this commit onto the current 2.11.0 tip to resolve the conflict and re-trigger CI, or (b) wait for a maintainer to confirm a preferred approach -- please let us know which you would prefer, or feel free to rebase/retrigger directly. |
|
Status update, 144h+ since the last push (head c78815a): CI has still never triggered a single check run on this head. The PR remains in CONFLICTING mergeable state against the moving 2.11.0 base, which appears to be the reason the pull_request workflows are not dispatching. We are still happy to either (a) push a rebase of this commit onto the current 2.11.0 tip to resolve the conflict and re-trigger CI, or (b) wait for a maintainer to confirm a preferred approach -- please let us know which you would prefer, or feel free to rebase/retrigger directly. |
Summary
Fixes #1450. The 2.11.0 branch still pins Debezium 3.1.3.Final, so #1450 is not fixed there. This PR upgrades the embedded Debezium to 3.7.0.Final, the latest release, and handles every behaviour change the upgrade brings. A plain version bump breaks existing deployments in several ways; each one is listed below with its fix, its spec and its tests. Related: #1451 (a 3.3.0 bump) leaves the TRUNCATE and schema-history layout problems below open.
This PR was created on behalf of a team that runs this connector; the maintainers own the review and the merge.
Root cause of #1450
Debezium 3.1.3 JdbcSchemaHistory stores a schema-history record longer than 65000 characters (the full tableChanges of a table with about 200 columns) as several rows, each with its own id, and on recovery parses each row on its own. The first restart after such a table is captured fails permanently with JsonEOFException: Unexpected end-of-input. DBZ-8979 (Debezium 3.3.0) writes every part of a record under one id and joins the parts back together on recovery.
What changes from 3.1.3 to 3.7.0, and how it is handled
Debezium 3.7 regression sweep (second commit)
After the first commit, a sweep looked for further Debezium-related regressions, comparing the 2.11.0 jar (3.1.3) and this branch (3.7.0):
Testing (final head)
Upgrade notes