8000 docs: end-to-end architecture and flow specification for 2.11.0 by minguyen9988 · Pull Request #1456 · Altinity/clickhouse-sink-connector · GitHub
[go: up one dir, main page]
Skip to content

docs: end-to-end architecture and flow specification for 2.11.0 - #1456

Merged
minguyen9988 merged 1 commit into
2.11.0from
docs-architecture-and-flow-spec
Sep 18, 2026
Merged

docs: end-to-end architecture and flow specification for 2.11.0#1456
minguyen9988 merged 1 commit into
2.11.0from
docs-architecture-and-flow-spec

Conversation

@minguyen9988
@minguyen9988 minguyen9988 commented Sep 18, 2026
Copy link
Copy Markdown
Collaborator

Summary

Adds a comprehensive, ground-truth architectural specification and method-by-method operational mapping of the clickhouse-sink-connector on branch 2.11.0 to the repository at doc/architecture_and_flow_specification.md, and updates doc/architecture.md with a direct reference.

What is covered

  1. Topology and Prime Invariants:

    • Explicit formalization of the source of truth (MySQL/PostgreSQL) and replica conformity invariants.
    • Operating comparison between Embedded Lightweight Mode (sink-connector-lightweight) and Kafka Connect Mode (sink-connector).

  2. End-to-End Execution Flow (Detail to the Tee):

    • Phase 1: Startup, configuration loading, keyless table preflight checks, and target schema initialization.
    • Phase 2: CDC event capture, version generation via nextSequenceNumber with first-delivery high-water mark floor clamping, DDL interception, pre-DDL buffered row flushing, and quiescence drain protocol (drainBeforeDDL).
    • Phase 3: Worker batch dequeue, database/table resolution, DbWriter caching, schema staleness checks (refreshIfRecordHasUnknownColumn), unwritable column enforcement (enforceSourceColumnIsWritable), JDBC parameter binding, sorting key relocation tombstone generation, and serialized offset acknowledgement under OFFSET_COMMIT_LOCK.

  3. Method-by-Method Operation Mapping:

    • Detailed specifications of contracts, inputs, outputs, preconditions, lock boundaries, and state mutations for all critical classes:
      • DebeziumChangeEventCapture
      • ClickHouseBatchExecutor
      • GroupInsertQueryWithBatchRecords
      • PreparedStatementFieldMapper
      • PreparedStatementExecutor
      • DebeziumOffsetManagement

  4. Architectural Critique and Problem Identification:

    • Detailed analysis of the 8 systemic failure modes observed in production:
      1. Relational in-place mutations vs append-only columnar storage impedance mismatch.
      2. Version arithmetic bleed (ts_ms * 1e6 + counter) and post-restart inversion window where newer events can be discarded.
      3. DDL/DML synchronization fragility and thread pool deadlocks during queue drains.
      4. Tri-partite schema evolution engines and unbounded metadata query loops.
      5. Global static mutable state anti-patterns across threads and tasks.
      6. Code duplication and divergence between ClickHouseBatchWriter and ClickHouseBatchRunnable.
      7. OffsetStorageWriter flushInProgress semaphore leakage.
      8. Keyless table single-row collapse under ReplacingMergeTree.

  5. Blueprint for Architectural Simplification:

    • Deterministic 64-bit log-based versioning (binlog coordinates / LSN) eliminating arithmetic bleed and timestamp drift.
    • Barrier-based DDL synchronization replacing thread-pause and timeout-drain loops.
    • Unified ingestion core eliminating duplicate batch writers and global singletons.
    • Unified catalog manager with explicit column contracts and mandatory key validation.

@minguyen9988
minguyen9988 merged commit 90b01a4 into 2.11.0 Sep 18, 2026
1 check passed
@minguyen9988
minguyen9988 deleted the docs-architecture-and-flow-spec branch September 18, 2026 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@minguyen9988
0