8000 fix: an excluded keyless table must not block startup by minguyen9988 · Pull Request #1405 · Altinity/clickhouse-sink-connector · GitHub
[go: up one dir, main page]
Skip to content

fix: an excluded keyless table must not block startup - #1405

Merged
minguyen9988 merged 2 commits into
2.10.0from
omniwatcher/preflight-honours-exclusions
Aug 25, 2026
Merged

fix: an excluded keyless table must not block startup#1405
minguyen9988 merged 2 commits into
2.10.0from
omniwatcher/preflight-honours-exclusions

Conversation

@minguyen9988
@minguyen9988 minguyen9988 commented Aug 25, 2026 •
edited
Loading
Copy link
Copy Markdown
Collaborator

The keyless-table preflight refused to start whenever the source held a table with no row identity — even when the operator had already excluded that table from replication. A table the connector never reads cannot be replicated incorrectly, so it has no business blocking startup.

Why it matters

Before this, the only ways past a keyless table were:

  • add a primary key to a table you may not own, or
  • set keyless.table.check.skip=true and lose the protection for every other table on the source.

Neither is right when the table is already excluded.

Concretely: txnrepo uat and staging carry alembic_version in 18 databases — Alembic schema bookkeeping, 0–1 rows, no key. Excluding it is the correct fix, but the connector would still have refused to start.

What changed

withoutExcluded() drops out-of-scope tables before the refusal decision, honouring both Debezium list properties with Debezium's own semantics:

  • entries are regular expressions, full-matched against the fully-qualified db.table — not substring, so an exclusion written for one table cannot silently swallow a neighbour;
  • table.include.list, when set, means anything not listed is out of scope.

Failing safe, in the right direction

An uncompilable pattern is ignored rather than treated as a match.

This list can only ever remove tables from the refusal set. A broken pattern treated as matching everything would silently drop a genuinely keyless table from the check — a false pass, which is strictly worse than a false refusal. Same direction as the existing databaseFilter().

Tests — 20 in this class, 5 new

  • an excluded keyless table no longer blocks, using the exact ".*.temp_.*,.*[.]alembic_version" pattern shipped for txnrepo uat/staging, so this test fails if that production exclusion ever stops covering the table it was written for;
  • exclusion is anchored, not substring (alembic_version_history and xalembic_version still block);
  • a table outside table.include.list is out of scope;
  • no lists configured excludes nothing;
  • a malformed pattern is ignored while the valid entry beside it still applies.

Failing-first verified — with withoutExcluded() short-circuited, 3 of the 5 fail, each showing the excluded table still blocking:

testExcludedKeylessTableDoesNotBlockStartup
  expected:<[aerion_uat.trade]>
  but was:<[aerion_uat.alembic_version, txnrepo_staging.alembic_version,
            txnrepo_uat.temp_override_target, aerion_uat.trade]>

Full suite green: 44 across KeylessTablePreflightTest, SequenceSeedOverflowTest, SourceTsVersionAnchorTest, DdlReplayIdempotencyTest, CreateTableNoKeySortKeyTest.

Where this came from

Rolling 2.10.0 onto two of our environments, the preflight refused to start: the source carries the Alembic bookkeeping table alembic_version (0–1 rows, no key) in 18 databases. Excluding it from replication is the right answer, and we did — but the connector still would not start, because the preflight ran before the exclusion list was consulted.

Base branch 2.10.0, on top of d5cdf4fc.

Jenkins Continuous Build server and others added 2 commits August 25, 2026 03:25
Jenkins Continuous Build server
fix: an excluded keyless table must not block startup
f08e4e2 
The keyless-table preflight refused to start whenever the SOURCE held a
table with no row identity, even when the operator had already excluded
that table from replication. A table the connector never reads cannot be
replicated incorrectly, so it has no business blocking startup.

The practical effect was that the only ways past a keyless table were to
add a primary key to a table you may not own, or to set
keyless.table.check.skip=true and lose the protection for every OTHER
table on the source. Neither is right when the table is already excluded.

Concretely: txnrepo uat and staging carry alembic_version in 18
databases (Alembic schema bookkeeping, 0-1 rows, no key). Excluding it is
the correct fix, but before this change the connector would still refuse
to start.

withoutExcluded() drops out-of-scope tables before the refusal decision,
honouring both Debezium list properties with Debezium semantics:
  * entries are REGULAR EXPRESSIONS, full-matched against the
    fully-qualified db.table -- not substring, so an exclusion written
    for one table cannot silently swallow a neighbour;
  * table.include.list, when set, means anything NOT listed is out of
    scope.

Fails safe in the same direction as databaseFilter(): an uncompilable
pattern is IGNORED rather than treated as a match. This list can only
ever REMOVE tables from the refusal set, so a broken pattern that matched
everything would silently drop a genuinely keyless table from the check -
a false pass, which is worse than a false refusal.

Tests: KeylessTablePreflightTest 20 (5 new) - an excluded keyless table
no longer blocks (using the exact ".*.temp_.*,.*[.]alembic_version"
pattern shipped for txnrepo uat/staging, so the test fails if that
production exclusion ever stops covering the table it was written for);
exclusion is anchored not substring; a table outside table.include.list
is out of scope; no lists configured excludes nothing; a malformed
pattern is ignored while the valid entry beside it still applies.
Failing-first verified: with withoutExcluded() short-circuited, 3 of the
5 fail, each showing the excluded table still blocking.
Full suite green: 44 across KeylessTablePreflightTest,
SequenceSeedOverflowTest, SourceTsVersionAnchorTest,
DdlReplayIdempotencyTest, CreateTableNoKeySortKeyTest.
svc_dba_github
chore: drop build-generated churn in dependency-reduced-pom.xml
ddcc086 
The previous commit's test run let the maven-shade-plugin regenerate
dependency-reduced-pom.xml, which rewrote all 441 lines with CRLF endings
and silently bumped the checked-in lombok pin from 1.18.22 to 1.18.34.

Neither belongs in this change. Restored to the 2.10.0 state so the PR
contains only KeylessTablePreflight.java and its test.

Verified: after CRLF normalisation the only content difference between the
two versions was the lombok <version> line, so nothing else is lost by
reverting the file wholesale.
@minguyen9988 minguyen9988 changed the title WIP: DO NOT MERGE - fix: an excluded keyless table must not block startup fix: an excluded keyless table must not block startup Aug 25, 2026
@minguyen9988
minguyen9988 merged commit 67221f7 into 2.10.0 Aug 25, 2026
4 of 9 checks passed
@minguyen9988
minguyen9988 deleted the omniwatcher/preflight-honours-exclusions branch August 25, 2026 09:21
@minguyen9988 minguyen9988 mentioned this pull request Aug 25, 2026
Merged
minguyen9988 pushed a commit that referenced this pull request Aug 25, 2026
svc_dba_github
merge: reconcile with the pushed branch, keeping the rebuilt tree
25bc96d 
#1405 and #1406 merged upstream while this PR was open, so this branch's
original base carried those same commits and the PR showed CONFLICTING.
Rebuilt on the new 2.10.0 head (aa00431) by applying only this change's two
files. Force-push is never used here, so the previous tip is reconciled with
a -s ours merge instead.

Verified byte-identical to the tested branch: the diff of both files against
the 159-test-green commit is empty.
@minguyen9988 minguyen9988 mentioned this pull request Aug 27, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@minguyen9988
0