8000 fix(writer): unify database and schema resolution between writer implementations and fix drift cache invalidation by minguyen9988 · Pull Request #1458 · Altinity/clickhouse-sink-connector · GitHub
[go: up one dir, main page]
Skip to content

fix(writer): unify database and schema resolution between writer implementations and fix drift cache invalidation - #1458

Merged
minguyen9988 merged 1 commit into
2.11.0from
fix-writer-db-resolution-and-schema-reconcile
Sep 18, 2026
Merged

fix(writer): unify database and schema resolution between writer implementations and fix drift cache invalidation#1458
minguyen9988 merged 1 commit into
2.11.0from
fix-writer-db-resolution-and-schema-reconcile

Conversation

@minguyen9988
@minguyen9988 minguyen9988 commented Sep 18, 2026
Copy link
Copy Markdown
Collaborator

Aligns ClickHouseBatchWriter and ClickHouseBatchRunnable database and table resolution, and ensures PostgreSQL schema drift detection invalidates the shared DbWriter cache.

Summary of Changes

  1. ClickHouseBatchWriter Database and Table Resolution:
  • Extracted resolveDatabaseName helper matching ClickHouseBatchRunnable.
  • Applied CLICKHOUSE_COMMON_DATABASE_PREFIX when resolving target database name.
  • Applied CLICKHOUSE_DATABASE_SCHEMA_SUFFIX with CLICKHOUSE_COMMON_SCHEMA_TEMPLATE formatting.
  • Protected replication history database name from being overwritten by CLICKHOUSE_DATABASE_OVERRIDE_MAP when replication.history.enable=true.
  • Updated getTableFromTopic to respect CLICKHOUSE_TABLE_SCHEMA_PREFIX and schema template formatting.
  • Invoked addRecordsToHistoryTable in persistRecords when replication history mode is enabled.
  • Guarded against NullPointerException during statement executor construction and error logging when table metadata retrieval fails.
  1. ClickHouseBatchRunnable Database Resolution:
  • Unified resolveDatabaseName so database override mapping and prefix/suffix handling match ClickHouseBatchWriter.
  • Guarded against NullPointerException during statement executor initialization when DbWriter is null.
  • Cleaned up disabled hash-routing stub in run().
  1. DebeziumChangeEventCapture and Schema Drift Reconciliation:
  • Applied CLICKHOUSE_DATABASE_OVERRIDE_MAP in extractDatabaseNameFromRecord so PostgreSQL schema drift reconciliation checks the true ClickHouse target database instead of failing against the un-overridden source database name.
  • Applied database prefix and schema suffix to invalidationDatabaseName during DDL cache invalidation so table keys match worker thread cache lookups.
  • In PostgresSchemaChangeDetector.invalidateCache, called CacheInvalidationManager.getInstance().invalidateTable(tableKey) so worker threads refresh their cached DbWriter after columns are added to ClickHouse.
  1. Testing:
  • Added ClickHouseBatchWriterDatabaseResolutionTest verifying prefix, schema suffix, override map, history database protection, and schema-prefixed table names.
  • Updated PostgresSchemaChangeDetectorTest to verify that invalidateCache increments CacheInvalidationManager table versions.
  • All unit tests passed on JDK 17.

@minguyen9988
fix(writer): unify database and schema resolution between writer impl…
7770cb3 
…ementations and fix drift cache invalidation
@minguyen9988
minguyen9988 merged commit 1e06244 into 2.11.0 Sep 18, 2026
5 of 10 checks passed
@minguyen9988
minguyen9988 deleted the fix-writer-db-resolution-and-schema-reconcile branch September 18, 2026 08:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@minguyen9988
0