8000 docs(specs): domain 13 - deep as-built specification of the Python toolset (snapshot, load, verification, resync) by minguyen9988 · Pull Request #1537 · Altinity/clickhouse-sink-connector · GitHub
[go: up one dir, main page]

Skip to content

docs(specs): domain 13 - deep as-built specification of the Python toolset (snapshot, load, verification, resync) - #1537

Merged
minguyen9988 merged 14 commits into
2.11.0from
docs-python-toolset-spec
Oct 2, 2026
Merged

minguyen9988 merged 14 commits into
2.11.0from
docs-python-toolset-spec

Conversation

@minguyen9988
@minguyen9988 minguyen9988 commented Oct 1, 2026 •
Copy link
Copy Markdown
Collaborator

Specification-only change. Adds domain 13 (Python toolset): a behaviour-as-built specification of the non-streaming tools under sink-connector/python, covering both the legacy top-level tree and the packaged REDACTED-SYSTEM tree, function by function.

Micro-specs:

  • 13.01 Toolset architecture, packaging and the two source trees (entry points, dependencies, grammars, Dockerfiles, legacy-vs-packaged divergence matrix)
  • 13.02 Shared connection, credential and configuration layer
  • 13.03 MySQL snapshot dump (mysql_dumper)
  • 13.04 ClickHouse snapshot loader and MySQL-to-ClickHouse DDL translation (full type-mapping table)
  • 13.05 PostgreSQL snapshot dump and load
  • 13.06 MySQL-to-ClickHouse verification runners (checksum and count)
  • 13.07 PostgreSQL-to-ClickHouse verification and auto_diff
  • 13.08 Ad hoc repair and re-synchronisation (ch-mysql-resync), with its deviations from Spec 11.04

Each spec has a CLI and statement-template contract, invariants, verification criteria, failure modes (Detection / Recovery / RTO / Test) and a defect register. The registers hold about 236 entries: 45 S1 (silent wrong or lost data, or a false match verdict), 50 S2, the rest S3/S4. Every entry cites file:line on 2.11.0 and is labelled reproduced (offline repro) or code-read. Some root causes are recorded in more than one spec when several tools reach them; for example, the packaged loader picks the dump time zone at random (13.01, 13.04, 13.08), and the packaged checksum driver reports No difference when both sides fail (13.01, 13.06). The fixes come later as separate PRs that work from these registers.

Verification: scripts/validate_specs.py PASSED (83 specs, 1179 verification citations, including the spec-first gate against 2.11.0). The offline Python suite is unchanged at 227 passed, 5 skipped. No source file changed.

🤖 Generated with Claude Code

minguyen9988 and others added 2 commits October 1, 2026 03:53
… its micro-specs

Adds 13-python-toolset to the validator's expected domains, the Constitution's domain table and the specs README index. The micro-specs 13.01-13.08 follow in this branch.

[spec-exempt: specification-only change, no source change]

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Behaviour-as-built specification of the non-streaming Python tools under sink-connector/python (legacy tree and the packaged tree): packaging and the two source trees, the shared connection and credential layer, MySQL snapshot dump, ClickHouse snapshot load and MySQL DDL translation, PostgreSQL snapshot dump and load, MySQL and PostgreSQL verification runners with auto_diff, and resync / ad hoc repair. Each spec carries failure modes and a defect register (S1-S4, reproduced or code-read evidence with file:line on 2.11.0) that the follow-up fixes work from. No source change; offline suite unchanged (227 passed, 5 skipped).

[spec-exempt: specification-only change, no source change]

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@minguyen9988
Copy link
Copy Markdown
Collaborator Author

Spec content is complete (8 micro-specs, validate_specs PASSED) and this is ready for review; it only needs the draft flag cleared. The S1 fix PRs #1538 to #1543 are stacked on it.

…t 1 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder. No file differs from 2.11.0 after this commit.

[spec-exempt: verbatim copy from the base branch, no new change]

Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 2 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder.

[spec-exempt: verbatim copy from the base branch, no new change]

Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 3 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder.

[spec-exempt: verbatim copy from the base branch, no new change]

Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 4 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder. The two DROP DATABASE IF EXISTS sites are the end-to-end suite's own fixtures on a disposable test stack, already annotated DESTRUCTIVE upstream.

[spec-exempt: verbatim copy from the base branch, no new change]

Destructive-Op-Check: sites=2; result=pass
Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 5 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder.

[spec-exempt: verbatim copy from the base branch, no new change]

Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 6 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder.

[spec-exempt: verbatim copy from the base branch, no new change]

Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 7 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder. The three DROP DATABASE IF EXISTS sites are the end-to-end seed's own databases on a disposable test MySQL, already annotated DESTRUCTIVE upstream.

[spec-exempt: verbatim copy from the base branch, no new change]

Destructive-Op-Check: sites=3; result=pass
Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 8 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder. The one DROP COLUMN site is a planted divergence on the test's own clone database, already annotated DESTRUCTIVE upstream.

[spec-exempt: verbatim copy from the base branch, no new change]

Destructive-Op-Check: sites=1; result=pass
Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 9 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder. The nine DROP sites are end-to-end test fixtures on the suite's own scratch databases on a disposable stack, each already annotated DESTRUCTIVE upstream.

[spec-exempt: verbatim copy from the base branch, no new change]

Destructive-Op-Check: sites=9; result=pass
Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 10 of 11)

Copies files verbatim from 2.11.0 (736c6d0) so the merge that follows reconciles a small remainder. The two workflow files are GitHub Actions YAML (their braces are Actions expressions, not Jinja); both parse with a duplicate-key-detecting YAML loader. The six destructive sites are CI cleanup of a never-started helper container and DROP DATABASE IF EXISTS on the suite's own scratch databases, each already annotated DESTRUCTIVE upstream.

[spec-exempt: verbatim copy from the base branch, no new change]

Jinja-Render-Check: rendered=2; formats=yaml; result=pass
Destructive-Op-Check: sites=6; result=pass
Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
…t 11 of 11)

Copies files verbatim from 2.11.0 (736c6d0). After this commit the tree equals 2.11.0, so the merge that follows is clean. pull-request.yml is GitHub Actions YAML and parses with a duplicate-key-detecting loader; the two mysql_table_checksum.py files are Python (their braces are format strings, not Jinja) and parse cleanly.

[spec-exempt: verbatim copy from the base branch, no new change]

Jinja-Render-Check: rendered=3; formats=yaml,text; result=pass
Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
Resolves the add/add conflicts in specs/13-python-toolset/01-08. #1544 was stacked on this branch and its squash-merge already brought domain 13 into 2.11.0 in a later revision: the same files, updated to describe the fixed behaviour (the 'Fixed since 2.11.0' and 'Fixed by the end-to-end suite' notes). The 2.11.0 revision is kept for every file (taken in the preceding commits, split so each commit stays reviewable); the older revision on this branch describes defects that are already fixed and would contradict the code and tests on 2.11.0. The domain registration (scripts/validate_specs.py, specs/CONSTITUTION.md, specs/README.md) is identical on both sides.

[spec-exempt: specification-only merge, no source change]

Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
@minguyen9988
Copy link
Copy Markdown
Collaborator Author

Conflict with 2.11.0 resolved (merge commit f83be39, no force-push). #1544 was stacked on this branch and its squash-merge already put domain 13 into 2.11.0, in a later revision that describes the fixed behaviour. The resolution keeps the 2.11.0 revision of every spec file, so this PR now has no remaining diff against 2.11.0: the spec is already on the base branch. Merging it only records the merge; closing it as delivered by #1544 has the same effect.

@minguyen9988
minguyen9988 merged commit 5fb43dc into 2.11.0 Oct 2, 2026
13 checks passed
@minguyen9988
minguyen9988 deleted the docs-python-toolset-spec branch October 2, 2026 03:55
minguyen9988 added a commit that referenced this pull request Oct 2, 2026
#1547 was squash-merged into 2.11.0 with the same content as its commit on this branch (40162c9), and #1537-#1543 were merged with no content change, so the two conflicting files keep this branch's version: 2.11.0's version of them plus this PR's change.

Co-Authored-By: Claude Opus 5.5 (1M context) [noreply@anthropic.com]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0