Repository navigation
fix: wait for ALTER TABLE visibility before cache invalidation (#1222) - #1321
Merged
Merged
Conversation
Improve locking
After executing an ALTER TABLE (ADD/DROP COLUMN), poll system.columns until the schema change is visible before allowing the batch insert thread to rebuild its column metadata cache. Without this wait, the new DbWriter may read stale column metadata and silently drop values for newly added columns. The original fix (PR #1223) added CacheInvalidationManager to signal between the DDL and batch threads, but did not ensure the ALTER TABLE had propagated before the signal. Under concurrent load, ~37%% of newly added columns had their values silently lost. Adds DDLSchemaChangeWaiter, a utility that: - Parses ADD/DROP COLUMN from the DDL statement - Polls system.columns until the expected columns appear/disappear - Times out after 30s with a warning (configurable) - Falls back to a 500ms delay for unparseable ALTER TABLE types Includes unit tests for DDL parsing and null-safety.
Collaborator
|
@subkanthi what do you think ? do you like the implementation ? |
subkanthi
changed the base branch from
develop
to
2.10.0
June 26, 2026 20:36
Collaborator
maybe we can make this as config variables. |
Per review feedback from @subkanthi: - Made DDL schema change timeout and poll interval configurable as connector config variables (ddl.schema.change.timeout.ms and ddl.schema.change.poll.interval.ms) instead of using hardcoded defaults - Added DDLSchemaChangeWaiterRaceConditionTest with 19 tests across 5 categories proving the race condition exists (without the waiter) and is fixed (with the waiter), including a 20-thread concurrent stress test See GitHub issue #1319
Collaborator
Author
|
@subkanthi Done — addressed in commit 7392086. Both values are now configurable as connector config variables:
Added to Also added
|
…aceConditionTest The sink-connector module does not include Mockito as a test dependency. Rewrite the entire test class using manual JDBC stub implementations (StubConnection, StubStatement, StubResultSet) that implement the full JDBC interfaces with default no-op methods. Test categories: - Column extraction from ALTER TABLE DDL (ADD/DROP, single/multi-column) - Null safety and edge cases - Configuration variable integration - Visibility wait with pluggable query behavior via stubs - Race condition simulation (bug reproduction, fix verification, stress test) Fixes CI compilation error: package org.mockito does not exist
Collaborator
Author
|
Fixed the CI compilation error ( The sink-connector module doesn't include Mockito as a test dependency, so I rewrote the test class to use manual JDBC stub implementations instead:
Test coverage is unchanged — 22 test methods covering column extraction, null safety, config integration, visibility wait behavior (timeout, immediate visibility, staggered propagation, JDBC exception resilience), and race condition simulation (bug reproduction, fix verification, 20-thread stress test). |
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1319 (and the root cause behind #1222 that PR #1223 did not fully address).
Problem
After executing ALTER TABLE (ADD/DROP COLUMN), the connector immediately signals cache invalidation via CacheInvalidationManager. The batch insert thread then rebuilds its column metadata cache by querying system.columns. However, the ALTER TABLE may not have propagated to system.columns yet, causing the new DbWriter to get stale column metadata and silently drop values for newly added columns.
Under concurrent load, approximately 37% of newly added columns had their values silently lost (22 out of 60 columns showed NULL instead of the inserted value).
Solution
Added DDLSchemaChangeWaiter, a utility that polls system.columns after each ALTER TABLE until the expected columns appear (for ADD COLUMN) or disappear (for DROP COLUMN) before returning from executeDDL(). The visibility wait occurs before cache invalidation proceeds, ensuring the batch thread always sees the updated schema.
Changes
New: DDLSchemaChangeWaiter.java -- Polls system.columns with configurable timeout (default 30s) and poll interval (default 100ms). Falls back to a 500ms delay for unparseable ALTER TABLE types.
Modified: DebeziumChangeEventCapture.java -- Added call to DDLSchemaChangeWaiter.waitForSchemaVisibility() after each DDL query in executeDDL().
New: DDLSchemaChangeWaiterTest.java -- Unit tests covering DDL parsing, null safety, case insensitivity, and non-ALTER-TABLE passthrough.
Test Plan