Rejoin a decommissioned CAS server root on server start by filimonov · Pull Request #2574 · Altinity/ClickHouse

filimonov · 2026-10-15T11:13:45Z

A CAS server whose server_root_id was decommissioned with SYSTEM CAS DROP POOL MEMBER now rejoins the pool as an empty member on its next start, with the same or a new server uuid. Before, the id was refused for good and needed a hand-edited owner object or a new server_root_id.

Built on PR B; until it merges, this diff also shows PR B's commits. Base: antalya-26.6.

Related: #2567

Changes

Rejoin on a server start. Pool::open over a tombstoned owner takes the id over when epoch and mount exist, the retirement is verified, the slot is terminal and the old life's data is gone. The owner moves to a new life by If-Match, the epoch counter continues, and the member starts empty. A conflict re-reads the owner and decides again. The rejoin logs a warning with the old and new uuid and life, and each of its two writes emits a MountClaim event (rejoin_mount_rewrite, rejoin_owner) with both uuids and lives.
New server uuid. The mount body is rewritten for the new uuid before the owner is claimed. A same-uuid rejoin leaves the mount body alone.
Refused: a self-remount (never rejoins), a server uuid of 0, a missing epoch or mount (an id retired by 26.6.4, or deleted by hand; checked first, so such an id is never sent to a rerun), an unverified tombstone, a held slot, data of the old life, and an owner or slot that changes on every decision.
Foreign terminal slot. A starter may reclaim a foreign terminal slot body only after a fresh owner read shows its own uuid, the recorded life and no tombstone. A live foreign body stays ForeignOwner, and the decommission tool never takes a foreign body.
Old access-probe key. Before the emptiness check, the rejoin deletes, by ETag, the previous owner's late roots/<srid>/clickhouse_access_check_<uuid> key. Any other object there still blocks the rejoin.
Refusal texts state the cause and point at the docs. previous life remains names the part left (for catalog rows, the first namespaces and their state) and, for catalog rows and manifests, SYSTEM CAS GC RUN '<disk>' on the GC leader; a row held by GC points at CASGCStuckRemovals. The foreign-owner refusal also names a duplicate server_root_id between two running servers.
system.cas_mounts shows a decommissioned member as retired (one more GET per terminated or fenced row) and adds the last column retirement_verified: 1 once the retirement is verified, else 0. A rerun of SYSTEM CAS DROP POOL MEMBER is for retired with retirement_verified = 0 only.
Docs: rejoin, its refusals and the rerun rule (operations, troubleshooting, mounts-and-leases, system.cas_mounts, SYSTEM CAS DROP POOL MEMBER). Decommission is no longer called permanent: the data is gone for good, the id is not.

Risks / notes

Late tombstone. The admin's fenced plane bounds when the tombstone is sent, not when the store applies it. PR B withholds the admin's farewell after an unresolved tombstone, so the victim waits for a GC fence; a tombstone applied later than that is a timing assumption, like an in-flight drain request. The docs keep the repair: if a server of the id ran after the tombstone time in its start error, do not rerun; rewrite the owner's body line to the next life and restart that member.
State and column change. There is no retiring state: a tombstoned owner's terminal row is retired, and retirement_verified (new last column) tells verified from not. Queries that matched retiring from the earlier revision of this PR must read the column.
Fail-closed becomes rejoin. A restarted decommissioned server used to be refused; it now comes back empty under the same server_root_id. Its local table definitions attach with no rows.
Late-object corner. A rejoin that died between its two writes, followed by a late roots/ object, leaves a slot the next start refuses, and a decommission rerun cannot clear it (the admin never takes a foreign body). The refusal says what to delete by hand.
From PR B: a claim lost during the decommission drain keeps the tombstone, so a half-drained member cannot start as an ordinary member; it rejoins only after a rerun verifies the retirement. A still-running process of the old life stops in IdentityLost at its next remount, also after a rejoin under a new uuid.
clickhouse disks without a server uuid file does not rejoin: it keeps the refusal.
Mixed versions: a 26.6.4 server still refuses a retired id. An id retired by 26.6.4 cannot rejoin; configure a new cas_server_root_id. Run SYSTEM CAS DROP POOL MEMBER only from upgraded servers: a 26.6.4 admin concurrent with a same-uuid rejoin can delete epoch and mount under it.
Residuals, not coded (CAS-385):
- A rerun over a verified retirement racing a same-uuid rejoin can land its tombstone on the rejoined owner and downgrade it to unverified. The rerun rule forbids a rerun outside retired with retirement_verified = 0; PR B's fenced tombstone narrows the window.
- A new-uuid mount rewrite racing another starter's full rejoin, claim and farewell between our owner read and our mount write leaves our clean body under their owner until their next start reclaims it; a DROP POOL MEMBER of them in that window is refused.
Rerunning a decommission bound to the original life is not done here.

Testing

gtests: a rejoined member's part (same uuid and new uuid) survives three GC rounds run from another member; a killed rerun's lease blocks the rejoin until GC fences it; an id without epoch and mount is refused with "configure a new <cas_server_root_id>" and no rerun advice; retired with retirement_verified 0 and 1 in listMounts; one test over every ServerRootLeftover (control objects unchanged, probe key built from toString(UUID), the catalog-rows refusal names the row); the two rejoin events with both uuids and lives; the old process after a new-uuid rejoin stops in IdentityLost and writes nothing; a fence during the drain keeps the tombstone and the start is refused with decommission is not finished.
test_cas_drop_pool_member: after each post-rejoin insert, SYSTEM CAS GC RUN on every node, then the rows are rechecked; a verified retirement reads retired with retirement_verified = 1. Module: 3/3.
CAS* unit tests on the head: release 2797, ASan 2801, all passed. The merge of PR B alone: release 2796, ASan 2800. Every commit through e20348282ab0 builds and passes release CAS* on its own.
TLA+ (CaEpochAllocCore round 14): with the tombstone allowed to land until GC fences the admin's slot, a clean farewell after an unresolved tombstone puts it on a running member (NoTombOverLive); with the farewell withheld the late-landing, paused-run and 26.6.4 rows are green under the rerun rule. The default battery (187 rows) meets every expectation.
Soak S47 (decommission under load through the SQL verb, the executor killed after each of the six steps, rerun, rejoin with the same and a new uuid), 60 min, seed 47, all eight cycles: 139 of 140 checks pass. Every kill after the tombstone shows retired with retirement_verified = 0 and the rerun converges; a kill after the verified mark shows retirement_verified = 1 and needs no rerun; every rejoin comes back empty and its new rows survive three GC rounds; dangling = 0 after every cycle; soak oracle PHASE3 OK. The one red is the NoWait refusal text, which does not name the state to wait for; that text belongs to the base series, not this PR.

Commits (on top of PR B)

ba336d2c60a9 Let a CAS start reclaim another uuid's terminal body of its own owner
b4ede038732f Show retired CAS members in system.cas_mounts
85df7b155084 Rejoin a decommissioned CAS server root at server start
e20348282ab0 Document CAS member rejoin and the retired state
734d2bcd885 Merge cas-363-member-lifecycle (CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567) into cas-member-lifecycle-pr-c
57c7944383b Report CAS rejoins in the event stream and name leftover catalog rows

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

A CAS server whose server_root_id was decommissioned with SYSTEM CAS DROP POOL MEMBER now comes back on its next start as an empty member, with the same or a new server uuid, instead of being refused. system.cas_mounts shows a decommissioned member as retired, with a new retirement_verified column.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-10-15T11:14:44Z

Workflow [PR], commit [5b7869d]

After a rejoin under a new server_uuid the slot can hold the previous uuid's farewell or self-fence. claimMount now reclaims such a terminal body below its epoch once a fresh owner read shows the claimer's uuid and life, not retired. A start and a self-remount pass that expectation; the decommission admin does not. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A terminated or fenced slot whose owner is tombstoned reads as `retired`, and the new last column `retirement_verified` says whether a decommission verified it. A terminal row whose owner does not decode reads as `corrupt`. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@ 8000 anthropic.com>

A start over a verified retirement whose data is gone takes the owner at the next life under its own server_uuid, same or new, and continues the kept epoch counter. A new uuid rewrites the mount body before the owner. An unverified tombstone, a held slot, leftover data or missing control objects refuse with the action to take. Remounts never rejoin, and a start without a server uuid does not rejoin. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Brings in the base merge of PR B: #2566 (member lifecycle PR A, final review) and #2517 (renewal until a definitive answer). - The foreign-owner refusal takes the #2566 text, which names `SYSTEM CAS DROP POOL MEMBER` for a dead owner. - Docs keep this branch's rejoin rules and add the #2566 rows for the terminal-slot claim and ids with `/`. - The unverified-tombstone test expects this branch's `decommission is not finished` refusal. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

alsugiliazova · 2026-10-23T10:48:23Z

PR #2574 Audit Review

Verdict

No new High defect in the rejoin itself. The High finding from #2567 is still in this head, and it matters more now.

The rejoin protocol is careful. It checks the counter and slot first, requires a verified retirement, refuses a held slot, removes the old access probe and checks for leftovers. For a new uuid it writes the mount before the owner, and it bounds its re-decisions. Every race I traced either fails closed or leaves a state that the next start resolves.

The trouble is the other way back into the pool. A decommission that loses its claim after the drain has started rolls the tombstone back. The member then restarts as an ordinary start, over a partly deleted life, and skips every check this PR adds. On top of that there is one Medium (also inherited, and wider now) and two new Lows, both about diagnosability.

Severity	Count	Findings
High	1 (inherited from #2567, unchanged)	A rollback after a partial drain lets the member start over a hole, bypassing the rejoin gate
Medium	1 (inherited, broadened)	A still-running process of a retired or rejoined id never reaches `identity_lost`
Low	2 (new)	A rejoin is not in the CAS event stream. The `previous life remains` refusal can be permanent with no hint why

What the change does

Before this PR, a server whose server_root_id had been decommissioned refused to start forever. Now Pool::open → mountWritable → claimOwnerOrThrow(..., RejoinIfVerified) takes the id over when the owner is tombstoned:

Control objects present. mount and epoch must exist. An id retired by 26.6.4 has neither and is refused with "configure a new <cas_server_root_id>". This is checked before anything else, so such an id is never sent to a rerun.
Retirement verified. Otherwise the start is refused with "decommission is not finished".
Slot terminal. The slot must be gc_fenced or a farewell. A live admin body refuses with ABORTED (still running).
Old probe removed. The old owner's roots/<srid>/clickhouse_access_check_<uuid> is deleted by ETag.
Leftover check. serverRootLeftover must find no catalog rows, no cas/manifests/<srid>/ and no roots/<srid>/. The refusal names the leftover.
New uuid only. The mount body is rewritten to the new uuid as a farewell, If-Match on the slot read in step 3.
Owner write. The owner becomes {our_uuid, owner_life + 1}, If-Match on the owner read. A lost write decides again from a fresh read, at most 4 times.

The ordinary start then continues: allocateWriterEpoch (the kept counter), claimMount, recheckOwnerAfterClaim. claimMount now also reclaims another uuid's terminal body below our epoch, if a fresh owner read shows our uuid and life and no tombstone. The decommission admin passes no expectation, so it never takes a foreign body. listMounts reads the owner of every terminated or fenced row and shows retired plus the new last column retirement_verified.

Model


Actors	The rejoining start (same or new `server_uuid`), the old process of the id (possibly still alive with a fenced lease), a decommission run or rerun (admin), the GC leader, other starters configured with the same id, 26.6.4 binaries
Channels	Conditional `GET`/`PUT`/`DELETE` on `owner`, `epoch`, `mount`, `cas/ref_catalog`, `cas/manifests/<srid>/`, `roots/<srid>/`. There is no direct RPC between actors. Ordering comes from ETags, `owner_life` and the epoch counter.
Authoritative state	The owner (`server_uuid`, `owner_life`, `retired_at_ms`, `retirement_verified`), the mount slot `(server_uuid, writer_epoch, gc_fenced, min_active)`, the epoch counter, the catalog
Failure model assumed	Crash-recover processes, a store that honours `If-Match`, in-flight writes that may land late (bounded only by the sender's lease), no trusted wall clock
Safe failure criterion	A rejoined member starts with nothing of the old life. No writer epoch is issued twice. No member runs under an owner life it did not start under. A refusal is fail-closed and says what to do.

Findings

Recovery and rollback

High (inherited from #2567, unchanged at this head): a decommission that loses its claim mid-drain un-retires the member, and its next start skips the rejoin gate.

Impact. Data the drain already deleted is gone: dropped namespaces (GC keeps deleting their Removing rows) and deleted roots/ objects. The owner is at a new life, not retired. The next start is an ordinary start, so none of this PR's checks run: no verified retirement, no leftover check, no "previous life remains". The member serves the surviving part of its old life with holes in it. This PR's own design shows why that is wrong: the rejoin refuses a retired id while any of its data remains, yet a rollback hands the same half-drained state back as a normal member.
Anchor. decommissionPoolMember (Tools/CasDecommission.cpp, 547–549 and 552–558) calls confirmClaimOrRollBack after drainVictim returns warnings and on the exception path. rollBackTombstone (138–159) writes nextOwnerLife(tombstone.owner, std::nullopt, false). This code is unchanged since PR-2567-AUDIT.md.
Trigger. GC fences the admin's mount body after at least one namespace drop or object delete, and before the farewell. A slow store, or one missed renewal in a long drain, is enough.
Why it's a defect. The confirm before the drain already proved the tombstone was exclusive. A later fence only means the admin stopped holding the slot. The docs now cover this state (migration.md, rerun rule: "A run that warns the tombstone was rolled back lost its claim and left the id unretired: running the command again is a new decommission of the id"). They do not say that data is already gone, or that starting the member now bypasses the rejoin checks.
Fix direction. Once any drop or delete has succeeded, keep the tombstone, withhold the farewell and report "rerun". Roll back only when the claim was lost before the drain began.
Regression test direction. Fence the admin after the first dropNamespace. Assert that the owner stays retired, and that the next Pool::open refuses with "decommission is not finished" instead of opening.

Observability and diagnosability

Medium (inherited from #2567, broadened by this PR): a still-running process of a retired id stays not_live forever. After a rejoin under a new uuid, it also never reaches identity_lost.

Impact. The old process keeps its fenced lease and retries remount every 30 seconds. It increments CASRemountFailed and logs a warning. system.cas_mounts.lifecycle stays not_live. It never writes, because it throws before allocateWriterEpoch. Before the rejoin it throws in throwIfOwnerRetired; the verified-retirement text now says to restart, which is better than CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567's text. After a same-uuid rejoin, the life check moves it to IdentityLost, as the docs say. After a new-uuid rejoin, claimOwnerOrThrow throws the foreign-owner CORRUPTED_DATA instead. tryRemountOnce catches it, returns false and loops. It never reaches IdentityLost.
Anchor. Pool::tryRemountOnce (Pool/CasPool.cpp, 1506–1515): only a returned life mismatch calls enterIdentityLost, and the catch (...) at 1649 returns false. claimOwnerOrThrow (Pool/CasServerRoot.cpp, 638 onward) throws for both the retired and the foreign case.
Trigger. The member was fenced by a partition, not stopped. The operator decommissions it and brings up a replacement host with the same server_root_id and a fresh uuid file. The partition heals.
Why it's a defect. troubleshooting.md says a process shows identity_lost with the owner moved from life N to life M when "a rejoin or a decommission rollback rewrote the owner while this process was fenced". That holds only for a same-uuid rejoin. The new-uuid case, which the migration page documents for a lost uuid file, has no row. An operator sees a zombie that looks like a lease blip.
Fix direction. In tryRemountOnce, call enterIdentityLost when the owner is retired or names another uuid, like a life mismatch. Add the troubleshooting row.
Regression test direction. Fence a live Pool, decommission it, rejoin the id with a different uuid, run tryRemountOnce, and assert lifecycle() == IdentityLost.

Low (new): a rejoin writes the mount slot and the owner without a CAS event.

Impact. The event sink receives MemberDecommission begin and end, and every claimMount write as MountClaim/MountConflict. It gets nothing for the rejoin's new-uuid mount rewrite or for the owner takeover. The only record is a LOG_WARNING in the rejoining server's text log. Afterwards, the stream shows a reclaim of "terminal body of another server_uuid", or a slot whose uuid changed with no writer recorded. That is the pattern the mount-slot writer audit (the "foreign writer" instrument in mountWritable) exists to flag.
Anchor. rejoinRetiredOwner (Pool/CasServerRoot.cpp, 556–636). claimOwnerOrThrow has no CasEventSink parameter, and the mount replace at the new-uuid branch is not routed through emitMountEvent.
Trigger. Any rejoin. The new-uuid case also changes the slot's uuid.
Why it's a defect. The pool's audit trail cannot answer "when and by whom did this id come back", and it records one slot write with no writer.
Fix direction. Pass the sink into claimOwnerOrThrow. Emit one MountClaim (outcome = "rejoin_mount_rewrite") for the mount write and one event for the owner takeover, carrying the old and new uuid and life.
Regression test direction. Run a new-uuid rejoin with a capturing sink and assert both events with the expected uuids and lives.

Low (new): the previous life remains: catalog rows refusal can be permanent, and it does not say which rows.

Impact. The start keeps refusing with "Run SYSTEM CAS GC RUN … it can take several rounds". A Removing row whose fold carries a hold is never drained (CatalogLifecycleReconciler::selectEligible skips coverage.hold). So the advice never helps, and the operator cannot tell "wait one more round" from "stuck". The same happens when no server on the pool runs GC.
Anchor. rejoinRetiredOwner, the ServerRootLeftover::CatalogRows case (587–591). serverRootLeftover returns only the kind, not the namespaces.
Trigger. An old-life namespace whose ref log has an undecodable body (HoldReason::BodyUndecodable). The decommission reports it under namespaces_pending_gc and still sets member_retired = 1.
Why it's a defect. This fails safe, but the refusal tells the operator to do something that cannot work, and it names nothing they could look up.
Fix direction. Return the first few offending namespaces and their NsState, and point at CASGCStuckRemovals / the leader's stuck-removal warning when a row has been Removing for many rounds.
Regression test direction. Seed a held Removing row for the retired srid and assert that the refusal names the namespace.

Checked and clean

Namespace reuse is inert. A re-attached table recreates its namespace under the same name, and the catalog mints a fresh incarnation, so the old life's cas/ns/stream|state/<life_id>/ debris is never adopted. Dropping and recreating a row never reuses an incarnation (CatalogEntry in CasRefCatalogFormat.h).
Epoch continuity. The counter is kept and continues. The staging sweep (epoch < live) and the orphan-manifest eligibility compare against the slot epoch, and every new-life epoch is above the old one. DelayedLowerEpochClaimAfterRejoinIsRefused pins a late lower-epoch claim.
Same-uuid twins. No mount write happens. One owner If-Match wins; the loser re-reads an unretired owner of its own uuid and returns that life. Twin detection then proceeds as before, through the mount claim and the epoch nonce.
Different-uuid starters. The mount If-Match comes before the owner If-Match. The loser either refuses (foreign owner) or leaves a farewell body that the winner reclaims through the owner-checked foreign reclaim (ConcurrentStartersWithDifferentUuidsOneWins, LostOwnerWriteDecidesAgainAndRefuses).
Foreign reclaim. The owner is read before the write and checked again after it (ForeignReclaimReadsTheOwnerFirst). A body at or above our epoch, or a live foreign body, stays ForeignOwner. The admin path passes no expectation (AdminDoesNotReclaimAForeignTerminalBody).
Ambiguous writes. A late-landing rejoin mount write is If-Match on the slot read and leaves at worst a farewell under the old retired owner. Any later start recovers from that. A late owner write loses to any intervening owner write.
Rerun racing a same-uuid rejoin. This is the PR's documented residual. The rerun re-tombstones the new life before the rejoined process has claimed the slot. The rejoined start then hits the admin's live body or the recheck, so it never writes. No data is lost; the retirement drops to unverified.
Refusals. 26.6.4-retired ids (no epoch/mount), zero uuid, a remount (tryRemountOnce keeps Refuse) and an unverified tombstone are all refused before any write. The decision loop is bounded at 4.
listMounts. An owner GET that throws degrades the disk to its synthesized row, the same as a mount GET failure before. An undecodable owner gives corrupt. Rows other than terminated and fenced cost no extra request.
Prefix selection. validateServerRootId rejects /, so srid + "/" cannot select a nested member's catalog rows, manifests or roots.
CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567's Low (late roots delete with identical bytes) is narrower now. The probe key carries the uuid, the probe is transient, and the rejoin removes the old one itself. It still applies to a killed rerun's in-flight delete, which is the timing assumption the PR states.

Coverage summary

Reviewed. Network and transport: If-Match on every rejoin write, ambiguous outcomes. Partition and availability: a zombie old process, a held slot, no GC leader. Timing and ordering: owner_life, epoch continuity, late writes. Partial and gray failure: a crash between the two rejoin writes, a late tombstone. Idempotency: the re-decision loop, twin starters. Recovery and rollback: the inherited rollback, kill points of the rejoin. Version skew: 26.6.4-retired ids, a 26.6.4 admin, downgrade rows. Observability: system.cas_mounts, events, refusal texts.
N/A. Security and trust across nodes. There is no new RPC path. A rejoin acts only on the server's own configured server_root_id, which is the existing trust boundary.
Deferred. Blob resurrection when a rejoined replica re-fetches parts whose blobs GC is condemning. That is the pool-wide BlobReuseResurrect protocol, not specific to rejoin, and it was not re-audited. ReplicatedMergeTree behaviour of an empty rejoined replica (Keeper still lists its old parts) is outside CAS.
Limits. This is a static review of 46d9e99d. The gtests were read, not re-run locally. dropNamespace and the GC fold were taken as audited elsewhere.

Operational readiness

Scope: restart of a decommissioned server_root_id and system.cas_mounts around it, at 46d9e99d.
Persona: Operator / DBA.
Environment: Shared CAS pool, at least one other member running GC.

Flow 1: Restart the same host after `member_retired = 1`

Happy path. Run SYSTEM CAS GC RUN on the leader until the catalog rows are gone, then restart. The server logs rejoined as an empty member and system.cas_mounts shows live. test_retired_member_rejoins_empty passed in CI.
1.1. Restart before GC finishes. The start fails with previous life remains, which is clear and names the action. It can be permanent with no reason given: see the second Low finding. Severity Low.
1.2. Tables come back empty without an error. This is intended, but it is easy to miss. A non-replicated table reads 0 rows. A replicated table re-fetches from its peers, which was not verified here.

Flow 2: A replacement host with a new uuid file

Happy path. The new uuid rewrites the mount and takes the owner.
2.1. The old host is still up behind a healed partition. It logs remount failures forever and stays not_live, with no identity_lost. See the Medium finding. Recovery: stop or restart it. A restart is refused as owned by a different server, which is the correct end state.

Flow 3: A decommission that warned `the tombstone was rolled back`

3.1. The id is not retired. A restart opens the member normally over whatever the drain had not deleted yet. See the High finding. The signal is a warning row only. Severity High. Recovery: do not start the member. Rerun SYSTEM CAS DROP POOL MEMBER until member_retired = 1, then restart it to rejoin empty.

Readiness verdict

Flows 1 and 2: ready with caveats (1.1, 2.1).
Flow 3: not ready until the inherited High is fixed. Use the guardrail below meanwhile.
Overall: ready with caveats, provided CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567's rollback finding is fixed before or together with this PR.

Operator guidance

After any decommission warning that says rolled back, do not start that member. Rerun until member_retired = 1 (3.1).
Before restarting a retired member, check system.cas_mounts: retired with retirement_verified = 1, and GC rounds run on the leader.
A process that keeps logging remount failures after its id was dropped or taken by a new uuid is dead weight. Stop it (2.1).

Recommended fixes

Keep the tombstone after the first successful delete (inherited High).
Make a retired or foreign owner terminal in tryRemountOnce, and add the troubleshooting row (Medium).
Emit CAS events for the rejoin's mount and owner writes (Low).
Name the leftover namespaces in the previous life remains refusal (Low).

alsugiliazova · 2026-10-23T10:48:36Z

PR #2574 CI Verification

Verdict

CI blocks approval. One failure is a regression and one is unknown.

The Unit tests (asan_ubsan) check reports only the Iceberg UBSan line, which is a pre-existing base bug. The same log also holds a CAS unit test that fails on every run since PR B merged the base branch. The cause is a test expectation that a base change made stale; the shipped code is not broken. The OAuth regression job hung for the full 210 minutes on this head, and passed on all 16 other heads. Both Grype failures come from the old base image and are gone on the current base.

Category	Count	Failures
`regression`	1	`ContentAddressedTransaction.StagingEpochMatchesAdmissionGeneration` (inside `Unit tests (asan_ubsan)`)
`pre-existing-flaky`	3	`IcebergMetadataGenerator.ModifyColumnAfterItselfWithoutTypeIsNoop` (UBSan), `GrypeScanKeeper`, `GrypeScanServer (-alpine)`
`infrastructure`	0	—
`cascade`	1	The `PR` rollup check, red because of the jobs above
`unknown`	1	`RegressionTestsRelease / OAuth / oauth` (cancelled at the timeout)

Job counts: 56 succeeded, 3 failed, 1 cancelled, 107 skipped.

How the list was reduced

Cascade. PR is the rollup and repeats the four red jobs.
The unit job holds two failures. The check summary names only the UBSan abort. The log (unit_tests_asan_ubsan.log) also has [ FAILED ] ContentAddressedTransaction.StagingEpochMatchesAdmissionGeneration at line 41844. The binary aborted at the Iceberg test, which is the last line of the log.
Comparison across the same base. The Iceberg abort and both Grype results match the other PR heads on this base, and are gone on #2614, which produced the current base tip.
Investigated individually. The CAS unit test (deterministic, mechanism read in source) and the OAuth hang (no output, no artifacts).

Findings

`ContentAddressedTransaction.StagingEpochMatchesAdmissionGeneration`: `regression`

Job 118317784888, unit_tests_asan_ubsan.log:

src/Disks/tests/gtest_cas_s3_staging.cpp:1136: Failure
Expected equality of these values:
  staged.size()
    Which is: 0
  1u
only the first write's key; the refused write staged nothing under the old epoch

Mechanism (read in source). The test was added by PR B's commit 133ee5786ae3 ("Put the writer epoch into CAS S3 staging keys"). It is not on the base branch. It stages a blob in a first transaction, lets that transaction go out of scope without committing, and later expects that blob's key to still be under staging/<srid>/<epoch_before>/.

The merge dc3396dc531a (Oct 19, "Merge antalya-26.6-20261008 into cas-363-member-lifecycle") brought in #2566. That PR changed ContentAddressedTransaction::cleanupPendingTempFiles so that an aborted transaction deletes its S3 staging object. Before, only a commit did; the old else if (committed) became else. Its own test CASS3Staging.AbortDeletesS3Staging pins the new behaviour. So the first transaction's key is now gone by the time the assertion runs, and the count is 0 every time.

Timeline, from the S3 unit logs:

Commit	Where	Result
`3a2641dc70`	#2567, before the base merge	OK (30 ms)
`d45685f0ca`	#2567 head, after `dc3396dc`	FAILED
`46d9e99d64`	#2574 head	FAILED

The product code is consistent: the abort cleanup is intended, and the staging-epoch property the test protects still holds. The test's expectation is stale. Fix it in PR B: count only the refused write's keys, or keep the first transaction alive across the assertion, or use leakWriterStagedObject the way the CASDecommissionStaging tests do. Either way, this check stays red after the base merge that clears the Iceberg abort.

This failure is not new to #2574. PR-2567-CI-VERIFICATION.md lists only two failing tests on d45685f0 and does not mention it, so that report should be corrected too. The PR description's "CAS* unit tests … ASan 2751, all passed" predates the merge.

This PR's own tests. The diff from base adds 130 TEST/TEST_F cases across PR B and this PR. All 130 ran in this job: 129 passed and the one above failed. That includes all 19 CASRejoin.*, both CASListMounts.*, and the new CASMount foreign-reclaim tests. The log has 2,746 CAS* tests run and all 2,746 passed.

`IcebergMetadataGenerator.ModifyColumnAfterItselfWithoutTypeIsNoop`: `pre-existing-flaky`

/ClickHouse/src/DataTypes/IDataType.h:380:67: runtime error: member call on null pointer of type 'DB::IDataType'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /ClickHouse/src/DataTypes/IDataType.h:380:67

This is a deterministic base bug, not flakiness. It fails the same way on 16 of 17 PR heads against this base, including unrelated ones such as #2573 ("Create custom disks outside storage_policies_mutex"). The one head where it passes is #2614, which merged c72ea99e999 ("Added nullcheck for type before calling unknown type check") into the base on Oct 21. That was after this PR last merged the base. The diff touches no Iceberg or IDataType file. Merging the current base clears it.

It does block one more thing: UBSan aborts the gtest binary, so any test ordered after it did not run on this head. All of this PR's tests ran before it (see above).

`RegressionTestsRelease / OAuth / oauth`: `unknown`

Job 118343500828. oauth/regression.py started at 22:32:50 with --output new-fails and printed nothing more. At the job's timeout_minutes: 210 it was cancelled: ##[error]The operation was canceled. at 02:01:09. No suite artifacts were uploaded under REFs/2574/merge/46d9e99d…/regression/.../oauth/.

On this job, across the PR heads on this base:

Who	Runs	Result
16 other PR heads (Oct 8–20)	16	16 success, 46–62 min
#2567 head `d45685f0`, started 3 minutes after this one	1	success, 48 min
#2574	1	hung, cancelled at 210 min

There is no code path. The OAuth suite's configs contain no CAS disk, and this diff only touches the CAS disk, system.cas_mounts and docs. Still, 0 of 16 against 1 of 1 cannot be called flaky from one run, and the hang left no log that names a mechanism.

What would resolve it: rerun the OAuth job. A pass makes it infrastructure (a hung Keycloak or the runner). A second hang needs regression-test-database-investigation with the CI database password, to find which scenario stalled.

Grype (`GrypeScanKeeper`, `GrypeScanServer (-alpine)`): `pre-existing-flaky`

The keeper image reports 4 High/Critical findings and the Alpine server image reports 1, including CVE-2026-85091, CVE-2026-84782, CVE-2026-84784 and CVE-2026-72897. The same counts appear on #2594, #2577 and #2567, all built as 26.6.8.20001. Their #2614 images (26.6.8.20002, the current base) report 0. The diff changes no Dockerfile or package list. A rebuild after merging the base should clear both. Jobs: 118351150468 and 118353852535.

What passed that matters for this PR

test_cas_drop_pool_member: all four tests passed in Integration tests (amd_asan_ubsan, db disk, old analyzer, 2/8) (118356079923). That includes test_retired_member_rejoins_empty (same uuid twice, then a new uuid, three GC rounds each).
CAS stateless jobs: amd_binary, cas storage, amd_binary, cas s3 storage, and amd_asan_ubsan, cas s3 storage 1/2 and 2/2.
CAS stress jobs: amd_debug and amd_asan_ubsan, both with CAS S3 storage.
The regression suites the PR selected: Parquet, ParquetS3 (minio and aws_s3), Iceberg 1 and 2, S3Export part and partition, Swarms.

Coverage gaps

No CAS regression suite ran. ci_regression_cas is unchecked in the PR body, so CAS, CASSuite, CASAlter, CASLightweightDelete, TieredStorageCAS and the CASS3Cache* jobs were all skipped. For a change to the CAS mount and owner protocol, this is the largest gap.
TSan, MSan, Aarch64 and coverage are excluded by the PR's CI options. The tsan/msan CAS stateless and stress jobs and Unit tests (tsan) did not run.
The targeted integration job ran nothing ("No failed tests found from previous runs").

Recommendation

Do not approve on CI grounds yet.

Fix StagingEpochMatchesAdmissionGeneration in CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567: the test, not the product. This branch picks the fix up through its next merge of CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567.
Merge the current antalya-26.6-20261008 (97ac10160b or later). That clears the Iceberg UBSan abort and should clear both Grype scans.
Rerun OAuth. If it hangs again, investigate it with the CI database.
Tick ci_regression_cas for the next run.

…ation Since #2566 an abandoned transaction deletes its S3 staging object, so the first write's key was gone by the time the test counted the keys under the old epoch and the count was 0 on every run (`Unit tests (asan_ubsan)` of #2567 and #2574 after the base merge). The first transaction now lives until the final assertion; the property under test — a self-remount between the two reads refuses the write and stages nothing under the old epoch — is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

Brings the StagingEpochMatchesAdmissionGeneration fix and the `isCleanRelease` rename; the two `isFarewell` call sites this branch added are renamed with it. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # docs/en/antalya/cas/architecture/mounts-and-leases.md # docs/en/antalya/cas/operations/troubleshooting.md # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.cpp # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasServerRoot.h

A rejoin rewrote the mount slot and the owner with no CAS event, so the audit trail showed a slot whose uuid changed with no writer. Each rejoin write now emits a `MountClaim` event (`rejoin_mount_rewrite`, `rejoin_owner`) with the retired and the new uuid and life. The `previous life remains` refusal for catalog rows named nothing, and a row GC holds never goes away, so the advice to run GC could not help. It now names the first rows with their state and points at `CASGCStuckRemovals`. Tests also pin that a still-running process of the old life stops in `IdentityLost` after a rejoin under a new uuid. Review: #2574 (comment) Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

filimonov · 2026-11-04T00:25:41Z

Thanks for the audit. Status per finding, on 57c7944:

High (from CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567), a rollback after a partial drain. Fixed in CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567 (54feac1), merged here in 734d2bc: a claim lost after the drain started keeps the tombstone, so the next start is refused with decommission is not finished instead of opening over the hole. FenceDuringTheDrainKeepsTheTombstone and FenceAfterANamespaceDropKeepsTheTombstone pin it; the rerun rule names the new warning.
Medium, the old process stays not_live. Fixed in CAS: decommission keeps a member's control objects; owner life and epoch nonce #2567 (bffad26): the remount reads the owner before the claim, and a foreign, retired or rewritten owner ends it in IdentityLost. CASRejoin.OldProcessStopsForGoodAfterANewUuidRejoin covers your new-uuid case; RemountDoesNotRejoin now also checks the lifecycle.
Low, no event for the rejoin writes. Fixed in 57c7944: MountClaim events rejoin_mount_rewrite and rejoin_owner with the retired and new uuid and life (NewUuidReclaimsTheOldLifesFarewell).
Low, previous life remains names nothing. Fixed in 57c7944: the refusal names the first rows with their state and points at CASGCStuckRemovals for a held row (RefusesWhileDataOfTheOldLifeRemains).

CI: StagingEpochMatchesAdmissionGeneration was fixed in #2567 (14f23d8); the Iceberg UBSan and Grype reds went with the move to the antalya-26.6 base. ci_regression_cas is ticked for the next run.

The merge-base changed after approval.

…e-pr-c

filimonov added CAS antalya-26.6 labels Oct 15, 2026

filimonov and others added 4 commits October 17, 2026 10:02

Document CAS member rejoin and the retired state

e203482

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

filimonov force-pushed the fix/antalya-26.6/cas-member-lifecycle-pr-c branch from 47bb142 to e203482 Compare October 17, 2026 08:34

filimonov mentioned this pull request Oct 19, 2026

CAS: refuse new nested server_root_id, delete S3 staging on abort, actionable member refusals #2566

Merged

11 of 31 tasks

svb-alt added the antalya label Oct 22, 2026

alsugiliazova mentioned this pull request Oct 23, 2026

Create custom disks outside storage_policies_mutex #2573

Merged

9 of 31 tasks

alsugiliazova added the verified-with-issues Verified by QA and issues found. label Oct 23, 2026

filimonov changed the base branch from antalya-26.6-20261008 to antalya-26.6 October 24, 2026 11:30

Merge cas-363-member-lifecycle (#2567) into cas-member-lifecycle-pr-c

b9f6ab8

Brings the StagingEpochMatchesAdmissionGeneration fix and the `isCleanRelease` rename; the two `isFarewell` call sites this branch added are renamed with it. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>

filimonov assigned k-morozov Nov 3, 2026

k-morozov self-requested a review November 3, 2026 11:14

filimonov and others added 2 commits November 4, 2026 00:57

k-morozov previously approved these changes Nov 4, 2026

View reviewed changes

Merge branch 'antalya-26.6' into fix/antalya-26.6/cas-member-lifecycl…

5b7869d

…e-pr-c

k-morozov approved these changes Nov 4, 2026

View reviewed changes

filimonov merged commit 11ab9a4 into antalya-26.6 Nov 4, 2026
363 of 367 checks passed

filimonov mentioned this pull request Nov 6, 2026

Antalya 26.8: port the CAS PRs merged into antalya-26.6 after the PR 2540 cut #2805

Open

11 of 31 tasks

Navigation Menu

Repository navigation

Rejoin a decommissioned CAS server root on server start - #2574

Rejoin a decommissioned CAS server root on server start#2574
filimonov merged 9 commits into
antalya-26.6from
fix/antalya-26.6/cas-member-lifecycle-pr-c

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

Uh oh!

Uh oh!

PR #2574 Audit Review

Verdict

What the change does

Model

Findings

Recovery and rollback

Observability and diagnosability

Checked and clean

Coverage summary

Operational readiness

Flow 1: Restart the same host after member_retired = 1

Flow 2: A replacement host with a new uuid file

Flow 3: A decommission that warned the tombstone was rolled back

Readiness verdict

Operator guidance

Recommended fixes

Uh oh!

PR #2574 CI Verification

Verdict

How the list was reduced

Findings

ContentAddressedTransaction.StagingEpochMatchesAdmissionGeneration: regression

IcebergMetadataGenerator.ModifyColumnAfterItselfWithoutTypeIsNoop: pre-existing-flaky

RegressionTestsRelease / OAuth / oauth: unknown

Grype (GrypeScanKeeper, GrypeScanServer (-alpine)): pre-existing-flaky

What passed that matters for this PR

Coverage gaps

Recommendation

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Flow 1: Restart the same host after `member_retired = 1`

Flow 3: A decommission that warned `the tombstone was rolled back`

`ContentAddressedTransaction.StagingEpochMatchesAdmissionGeneration`: `regression`

`IcebergMetadataGenerator.ModifyColumnAfterItselfWithoutTypeIsNoop`: `pre-existing-flaky`

`RegressionTestsRelease / OAuth / oauth`: `unknown`

Grype (`GrypeScanKeeper`, `GrypeScanServer (-alpine)`): `pre-existing-flaky`