8000 CAS improvements for antalya-26.8 by filimonov · Pull Request #2831 · Altinity/ClickHouse · GitHub
[go: up one dir, main page]

Skip to content

CAS improvements for antalya-26.8 - #2831

Merged
filimonov merged 85 commits into
antalya-26.8from
fix/antalya-26.8/cas-improvements-bundle
Nov 17, 2026
Merged

filimonov merged 85 commits into
antalya-26.8from
fix/antalya-26.8/cas-improvements-bundle

Conversation

@filimonov
@filimonov filimonov commented Nov 12, 2026 •
Copy link
Copy Markdown
Member

CAS (content-addressed storage) fixes for antalya-26.8, in three parts. Review commit by commit.

Changes

  • Late requests. A request that the caller gave up on can still land later. It can no longer revive a removed namespace, fence a live mount, or make ref cleanup skip work.
  • New defaults. The mount lease is 40 s, the request window is 120 s, the blob upload pool has 64 threads. Five GC round budgets are unbounded. See the table.
  • Faster writes. A ref-log append is one object-store request: no _ckpt write and no catalog read. Publishing a part no longer reads its own manifest back.
  • A manifest is checked before it is stored. The encoder refuses a bad or duplicate entry path and a record line over 64 KiB; then the sealed bytes are decoded as a reader would decode them. The encoder checks are two commits by Konstantin Morozov from 26.8 CAS avoid manifest get on promote #2794, which this PR replaces.
  • A table being removed refuses writes. Once a server reads a catalog row as Removing, it closes the table's write lane for good, including the read that confirms a recovery. The error says to retry later and points to the troubleshooting page.
  • A busy writer cannot hold a GC round. The orphan-manifest sweep replays a table's ref log to learn which manifests are owned. The replay now spends the round's budget per record and stops when it runs out; the table's manifests are then kept for this round.
  • A staged blob is copied verbatim at most once. A hardlink of a blob that is not published yet shared its staging object but not the once-only flag, so the same bytes could be published twice under one key.
Statement Requests before Requests after Time before, ms Time after, ms
INSERT 25 14 22.2 11.6
OPTIMIZE FINAL 41 31 23.4 17.7
mutation 38 28 29.0 22.7
DROP TABLE SYNC 33 23 126.8 54.3
first INSERT after a restart 24 15 12.0

Local RustFS stand, medians of 25. CREATE TABLE and SELECT do not change. The DROP TABLE SYNC time is noisy (15 to 180 ms).

utils/cas-yardstick, CAS policy, per operation, before and after:

Card GET PUT Wall, s
insert, merge, drop cycle 53 to 27 34 to 21 3.31 to 2.24
wide insert 12.1 to 6.1 9.0 to 6.0 27.7 to 24.0
mutation, per part 5.7 to 2.0 5.7 to 4.4 4.7 to 3.3
Setting Old New
cas_mount_lease_ttl_ms 30 s 40 s
cas_request_window_ms (new) fixed 90 s 120 s
cas_gc_round_*_budget (five keys) 5000 0 (unbounded)
cas_blob_upload_pool_size (server setting) 16 64

Risks / notes

  • The on-store format changes and there is no way back. The first writable mount with this build raises the pool to format generation 3. clickhouse-disks cas-drop-member does the same. Older builds then refuse to mount the pool. Stop all older servers of a pool first. Steps: docs/en/antalya/cas/operations/migration.md.
  • Mixed lease values are unsafe. A server without the declared-lease rule (before v26.6.8.20003, or 26.8 before Antalya 26.8: port the CAS PRs merged into antalya-26.6 after the PR 2540 cut #2805) still waits by its own 30 s and can take a slot from a live 40 s holder. Keep cas_mount_lease_ttl_ms = 30000 while such a server shares the pool. Steps: docs/en/antalya/cas/configuration.md.
  • A slow connect timeout now needs a longer lease. With a 40 s lease one request attempt may take at most 14 s. A disk with connect_timeout_ms of 0, or of 4500 ms or more, opens writable only with a raised cas_mount_lease_ttl_ms.
  • Restart after a kill waits a little longer: 47 s plus the start time instead of about 43 s, because the server waits out its old lease. The hard-restart regression scenarios allow 60 s.
  • The first GC round on a pool with a backlog runs long: the budgets no longer cut it.
  • A removal request that lands after a remount can leave a table Removing while it still takes inserts. Without the catalog read, the inserts continue until a restart, a remount, a retried DROP or the GC leader's next round; then writes are refused. Nothing is deleted. What finishes the removal depends on the statement that started it: docs/en/antalya/cas/operations/troubleshooting.md.
  • GC reads a few keys past the end of each ref log: about 8 extra reads per namespace per sweep page. CAS GC: read-ahead only over listed ref logs; resolve mounts from listed server roots #2542 removes them and is not part of this PR.
  • New error codes when a part cannot be published. A record line over 64 KiB (a very long file name) now fails with LIMIT_EXCEEDED that names the line and its size. A bad or duplicate entry path is LOGICAL_ERROR: only a bug in the server can build one.
  • A quieter log. A 404 answer to a conditional marker write that lost a race with GC is no longer logged at Error. It failed 02841_not_ready_set_bug once in CI.
  • Review closely: 18be198c7ab (the one-request append), 4a06eabac6c (when an epoch gets its closing record), c821aece830 (the closed lane for a Removing row), d0e2d1a80bd (the append without the catalog read), d383d7d79c4 (which errors the manifest encoder raises) and 20b02254350 (the bounded replay of the sweep).

Testing

On 9fbdefb5711; the commit after it changes one integration test config:

  • CAS* unit tests: 3169 pass (release), 3173 pass (ASan + UBSan). The Fast test configuration compiles.
  • Stateless: all 60 *_cas_* tests pass.
  • Integration: all 19 test_cas_* suites pass. test_cas_ref_move_compat runs a real 26.6.4 server against a generation 3 pool.
  • Soak: utils/ca-soak, 2 hours, 29 injected faults (kills, restarts, pauses, freezes of both replicas and of the object store), 22 checkpoints: row counts equal the model, cas-fsck reports nothing. Run on 1810fb2fef3, before the last six commits; not repeated.
  • Not run locally: the full stateless suite, MSan, TSan.
  • CI on the previous head 2e2bc6721ac: https://github.com/Altinity/ClickHouse/actions/runs/43913835330. None of its 36 red jobs came from this PR. The hard-restart regression jobs were red because of the 120 s lease; with 40 s they should fit their 60 s limit. Skip the CAS mount lease wait in the suites' CAS config clickhouse-regression#184 removes the wait from those suites.
  • test_cas_replicated_relink failed 4 of 12 local runs on a host with less than 10 % free disk: the background mover relocated the fetched part. Its tiered policy now sets move_factor to 0.
  • Red on the base branch too: sanitizer cas s3 storage shards (job timeout), aarch64 aggregate-function suites, ARM integration shards (helper image platform), test_storage_iceberg_with_spark under MSan, test_database_glue.

Related: #2570
Related: #2805
Related: #2542
Related: #2800
Related: #2794
Related: Altinity/clickhouse-regression#184
Related: ClickHouse#131967

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixes several cases where a slow or retried request to a CAS (content-addressed storage) disk could bring back a dropped table's data, lock out a healthy server, or leave garbage behind. Writes to a CAS disk are faster: an INSERT now needs 14 object-store requests instead of 25. The mount lease cas_mount_lease_ttl_ms defaults to 40 s, the new cas_request_window_ms to 120 s and cas_blob_upload_pool_size to 64, and GC rounds are no longer cut short by default. Upgrade with care: once a server with this version writes to a pool, older versions can no longer open it, so stop all older servers of the pool first.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

filimonov and others added 9 commits November 12, 2026 12:21
A caller that must re-decide its bytes after an attempt of unknown fate can ask for the attempt back with `Retry::resend_unresolved = false`. Resending the same bytes is unsafe for such a caller: the first attempt may have landed and been superseded, so a resend could revive it. `Retry::attempts_already_sent` lets the caller continue the attempt sequence after it re-decides. Tests pin the policy at the request engine: an unresolved write and a connect failure are not resent, a read-proven commit and a credential reissue are unaffected, and a continued sequence has no first-attempt timeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The incarnation was minted once before the admission loop. A resend, or
a re-decide after a lost response, could insert an incarnation that a
concurrent remove had already erased. The catalog has no revision field:
cancelling an added row restores the old bytes and ETag, so a late copy
of a resent row could revive a cancelled life.

Step 1 now mints inside the mutate callback and passes
`resend_unresolved = false`, so an attempt of unknown fate is decided
again on a fresh read instead of being resent. It returns the row that
landed; a landed write without that row raises `LOGICAL_ERROR`.

Tests cover a conflict retry, a lost response that landed and one that
did not, a refused catalog read, the deadline, a fence lost after an
unresolved attempt and during the pause before the next decision, and a
sibling that erases the row between a landed write and the next decision.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The lane close for a deposed writer moves into a local lambda so a
second caller can share it without duplicating the exception text. No
behaviour change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Five callers compare a `_ckpt` publish outcome with `FencedOut` only to
learn whether the contribution became durable. A named predicate lets
later outcome values be added without touching them. No behaviour change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A deposed writer's create can land in a ref-log slot that cleanup already
reclaimed, and the merge accepted its frontier by numeric comparison.
`publishCkpt` now refuses a non-seal frontier that sits at the durable epoch
seal (`SuccessorSeal`) or below a later epoch's frontier (`Superseded`), so
the append is rejected or left unacknowledged. `CASRefCheckpointFrontierRefused`
counts the refusals.

Callers:
- A wedge retry that a successor seal rejected is below the checkpoint, so
  recovery never installs it. Its survivors get the "outcome is unknown"
  answer, and the `Rejected` text no longer says "nothing is durable" where
  the log object is durable.
- A recovery walk whose frontier publish is refused ends with the
  retry-later error and writes nothing. The next recovery grounds on the
  successor's snapshot.
- Both ref-log sites that act on a publication outcome end with an assertion
  that the remaining outcome is a recorded one, so a new enumerator cannot
  fall through as "acknowledged".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Ref cleanup may delete a covered log after advancing `_ckpt` past the sample
`fsck` took. An absent tail id now gets the same `_ckpt` re-read as the
checkpoint base: a changed, absent or unreadable `_ckpt` gives `Unchecked`,
an unchanged one stays chain-broken. One local helper holds the re-read for
both the base validation and the tail walk.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Ref cleanup refused to delete whenever the catalog object's etag moved, so
unrelated namespace churn starved it. The chunk licence now depends on the
namespace's own row and life and on the GC lease. A moved row or life skips
that namespace and the pass continues; a moved lease, a failed read or an
ambiguous catalog stops the pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Operators could not tell a skipped namespace from a stopped pass, or see how
many planned keys were deleted. Add five counters and six `ref_object_cleanup`
phase metrics reporting plans, deletes, skips, stop reasons and chunks deleted
under a changed catalog etag.

- An absent `gc/state` is a pool anomaly, not a leader takeover: the licence
  check reports it as `StopAuthorityUnverified`. `StopLeaseMoved` means
  `gc/state` was read and its lease differs from the adopted one.
- A scope-exit guard attaches the phase metrics before the phase timer emits
  the row, so a delete chunk that throws still reports what was deleted.
- The race tests assert the counters: a moved row or life counts as a skip
  and not a pass stop, and a trim-disabled round reports zero metrics. The
  `gc/state` fault tests observe that the fault fires at the licence read and
  not in the namespace janitor.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Phase 17 licenses each chunk by the namespace's own catalog row and life plus
the GC lease, and a refusal skips one namespace or stops the pass. The
architecture pages describe that, the new counters and phase metrics, and the
`_ckpt` frontier refusal results. The operator pages cover the refusal
messages and the `fsck` unchecked finding.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
@github-actions
github-actions Bot commented Nov 12, 2026 •
Copy link
Copy Markdown

Workflow [PR], commit [44fc400]

filimonov and others added 20 commits November 13, 2026 07:17
`CASRefWriterSnapshotPublish.C4InFlightGateAdmitsAtMostOne`: a setup publisher that met the second
commit's `Writing` lane armed the backoff, so the later `dropRef` triggered no `_snap/` PUT and
`awaitBlockEntered` waited forever. The test now disables the backoff and asserts a compacted
tail. `awaitBlockEntered` gives up after 20 s and returns the result of its wait, and each caller
asserts it.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A slot that holds a lower epoch than the one being asked about is an older incarnation, so its GC
fence or clean farewell says nothing about the higher epoch. Only a greater epoch proves
supersession, whether the slot's body is live, GC-fenced or a clean release.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The renewer records the ETag of every body it lands and the attempt id of the renewal in flight,
so a reader on another thread can tell whether a slot body is one this mount wrote. Nothing reads
the record yet.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The in-memory fence can say live after another server claimed the mount slot or GC fenced it, so
a confirm answered `Yes` for a ref this mount no longer protects. After a ledger `Yes` the pool
reads the mount slot once, and the answer stays `Yes` only if the slot still holds a body this
renewer wrote. Otherwise, and when the read fails or no renewer exists, the answer is `Unknown`.
The ownership is captured before the ledger snapshot, so a remount between them cannot validate
an old snapshot.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The relink confirm reads the mount slot once before a ledger `Yes` and answers `Unknown` when the
slot is not held. The architecture pages no longer call the confirm I/O-free.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A `Live -> Removing` catalog write that was sent and not answered can still land after the removal
call gave up. Reopening the lane on one read of the old `Live` row then lets a late write stop the
table at a later append, with nothing linking it to the failed `DROP`. The lane now stays closed
until a retried removal completes or the disk is remounted, and the table's runtime is not evicted
meanwhile. A drop that fails before it sends a removal write still reopens the lane.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A drop gave every read its own standard window, and the creator-fence read nested in the catalog
update opened a fresh one on every decision. The drop freezes one policy at entry and spends it
across the catalog reads, the creator-fence read, the `gc/state` read and both catalog updates.
The resolution read after a failure keeps its own window so it can still reopen the lane.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A writer that already lost admission can hold a stale contribution that the durable body rejects
in the merge. It got `CORRUPTED_DATA` for a body nothing had corrupted. It now gets `FencedOut`;
an admitted writer still gets `CORRUPTED_DATA`.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An object removed from the bucket between the read and the write of a pool metadata update is an
environment fault, not a program bug, so it is reported as `INVALID_STATE` instead of
`LOGICAL_ERROR`, which also aborts debug and sanitizer builds.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The `GetBucketVersioning` probe in `checkPoolPreconditions` goes through the single-attempt client
with the attempt timeout, so the SDK does not retry it and the adaptive first-attempt timeouts do
not cut a slow answer short. The attempt timeout bounds each send and receive, and the connect cap
bounds the connect. Only a writable mount runs the probe.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A cold reader or writer that read a `Live` catalog row can install a runtime for a life that was
removed and drained before the insertion. The ref-name slot now counts retirements and outlives
eviction while the count is non-zero. Admission throws retry
8000
-later when the count moved since the
catalog read, and when the resident life has already applied its terminal removal.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The GC drain recorded a retired life only after the erase, so a cold admission that inserted its
runtime in between saw an unchanged retirement count. `deleteCompletedRemovingAtSnapshot` now calls
a `before_erase` hook once before the first erase, and the drain passes the invalidation of the
retired life.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Another server's GC leader can drain a life without telling the owner, so the owner evicted the
idle runtime of that life with a zero retirement count and erased its slot. Eviction of a runtime
whose removal is closed or applied now bumps the count and keeps the slot.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Several paths leave a runtime recovered as terminal with no local removal in progress: a presence
probe on a `Removing` row, the early return of `dropNamespaceImpl`, and a local GC whose retirement
announcement preceded the runtime's publication. Once the row is erased, `namespaceLife` refuses
the name until eviction. The warm path now reads the catalog for such a life and, when the row is
gone, detaches the runtime and births a new life.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every retired namespace name kept a `RefNameSlot` until the next remount, so create and drop cycles
grew `ref_name_slots` without bound. A cold admission now holds a `ColdAdmission` guard that keeps
the slot of its name from capture to publish. A slot with neither a runtime nor an admission is
erased.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The erase loop of a completed removal paused after its resolution read. A catalog write landing in
the pause refused the next erase, because that erase was conditioned on a read taken before the
pause. The pause schedule is unchanged (`backoff(attempt + 1)`, up to 5 s). The read now follows
the pause, so the next erase is conditioned on a read taken after it.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The reconciler keeps a count of completed erases that stays readable after `reconcile` throws.
`Gc::runRegularRound` writes it to `deleted` of the `pre_fold_ref_drain` phase on every exit of the
phase scope, so a drain that throws still reports the erases it completed.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`CasRefCatalog::deleteCompletedRemovingAtSnapshot` reports an acknowledged erase through
`erase_acknowledged` before its resolution read. The drain counts and announces that retirement
even when the read throws or the call ends `FencedOut` or `EntryChanged`. A warm `namespaceLife`
detaches a resident life that fails recovery once the catalog no longer names it. The presence
probe answers present instead of throwing when the drain announces during its admission, and
`before_erase` is required on the production erase. The aborted-drain test runs on the virtual
retry clock.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A 30 s lease leaves a 4 s write window in an outage. The default is now 120 s with the renew
period unchanged at 10 s, so the window is 94 s. Integration tests that kill a server keep 30 s in
their own configs. The upgrade notes require pinning `cas_mount_lease_ttl_ms` to `30000` while a
pool still has a build without the declared-lease rule, and name what an older server's early
fence leaves unprotected. The docs give the GC fence wait as up to two GC intervals after the
lease threshold.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The request budget log line carries `write_reserve_ms`, and `renewal_slack_ms` when background
renewal is on, so an operator can read the lease headroom from the log.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
filimonov and others added 9 commits November 16, 2026 15:35
`wait_and_read_output` cancelled the timeout timer of every command it
finished waiting for. When the wait itself gives up after
`DEFAULT_QUERY_TIMEOUT` with the client process st
10000
ill running, the timer
is what kills that process, so it has to stay armed. Cancel it only once
the process has ended. This is the text of upstream `813bdc10c98`.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A writer that replaces a blob's `.meta` marker with `If-Match` gets 404
when GC deletes the marker between the writer's read and its write.
`finalizeConditionalWrite` maps that to a lost precondition and the
writer starts over, but the HTTP client logged the response at Error.
A stateless test that checks stderr then fails on the log line alone:
`02841_not_ready_set_bug` did, once in a shard of 6891 tests.

`nativeConditionalPut` now sends an `If-Match` write under
`Expect404ResponseScope`, as `readUnder` does for its `GET`. The request
runs on the calling thread, since conditional writes switch the parallel
part upload off. A write with `If-None-Match` stays loud: it names no
existing object.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`DecodeRejectsMalformedEntryPaths` built its bad bytes with
`encodePartManifest`, so it stops working as soon as the encoder refuses
a bad path. Encode a legal placeholder path, splice the bad path in, and
compute the digest over the spliced bytes the way `computePayloadDigest`
does, without calling the encoder on the bad path. Only the decoder's
path check can refuse these bytes, and the test asserts its message
`invalid entry path`, not only the code. A control case proves the forge
reproduces the encoder's bytes for a legal path.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review: https://gist.github.com/k-morozov/7b34b402f18b051ced88f457c95d5df4
(cherry picked from commit 657c521)
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
(cherry picked from commit ff3b531)
Check encoded text lines in `encodePartManifest` against the decoder's
line cap before any manifest body is written. Measure escaped bytes
without the line terminator, and leave raw inline payload outside this
limit.

Cover entry and descriptor boundaries, JSON escaping and refusal before
the body write. Build total-size cap fixtures from many valid records.

Validation: focused release build and all 105 CAS tests passed.
Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com>
(cherry picked from commit 90acf02)
Conflicts resolved: `gtest_cas_part_manifest_format.cpp`: the new encode tests are placed before the forged decode test; `DecodeRejectsARecordLineOverTheLineCap` is added, because no writer-made body can carry an over-long line to the decoder any more.
Conflicts resolved: `gtest_cas_part_write.cpp`: `manifestWithEncodedSize` replaces `fillEntriesToTheEncodedCap` and its helpers for both object-cap boundary tests; `StageManifestRejectsOversizedEntryRecordBeforeBodyWrite` replaces `StageManifestRefusesARecordLineOverTheLineCap`, which asserted the same refusal with the decoder's `CORRUPTED_DATA`.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Adjust the checks that "Validate CAS manifest entry paths before
encoding" and "Reject oversized CAS manifest lines before staging" added
to `encodePartManifest`.

- A bad or duplicate entry path is a bug of the writer, so the encoder
  throws `LOGICAL_ERROR`. The decoder keeps `CORRUPTED_DATA`: there the
  same path is bad bytes. `validateManifestEntryPath` takes the code, so
  the path rule stays in one place.
- A line over the line cap keeps `LIMIT_EXCEEDED`. It comes from the
  size of the input, such as a very long file path, not from a broken
  invariant of the writer, so it is a limit, not a bug.
- The header and trailer lines have a fixed shape and can never reach
  the line cap; their checks are removed.
- `stageManifest` takes the object cap from
  `traitsFor(FormatId::PartManifest).object_cap` through `fitsObjectCap`
  instead of its own copy of the number, so the writer and the readers
  cannot drift apart. The tests derive their sizes from the same traits.

Tests that now expect `LOGICAL_ERROR` are split: the release build
checks the exception, sanitizer and debug builds check the abort.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The "was not staged by this transaction" errors of `precommitAdd` and
`promote` now carry the build id and the target namespace next to the
manifest and the ref, so the log names the transaction that refused.
The death tests of `precommitAdd` match that text instead of any abort.

Fix the text that "Validate CAS manifest entry paths before encoding"
made false. The encoder refuses what no writer may build: a bad or
duplicate entry path with `LOGICAL_ERROR`, a line over the line cap
with `LIMIT_EXCEEDED`. `stageManifest` still decodes the sealed bytes
with the readers' decoder before the `PUT`, so nothing a reader would
reject is written (`CORRUPTED_DATA`). The comment on
`validateManifestEntryPath` again gives the reason for the path rule:
manifest bytes also arrive over the interserver relink channel.

Also drop "after fail-closed reval" from the `promote` event reason
and the unused `#include <set>` from `CasPartWriteTxn.h`.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`decodePartManifest` checks the line cap on the stored line, but it
accepts strings the encoder writes longer, such as raw control bytes
that the encoder escapes to `\u0001`. `computePayloadDigest` re-encodes
the decoded body, and the encoder's line cap then threw
`LIMIT_EXCEEDED`. Callers treat only `CORRUPTED_DATA` as bad bytes:
`prepareAdoptFromManifest` falls back to a byte fetch on it alone. The
decoder now maps `LIMIT_EXCEEDED` from the digest computation to
`CORRUPTED_DATA`. Other codes still propagate: the decoder refuses a bad
or duplicate path before the digest, so a `LOGICAL_ERROR` there is a
real bug.

The simpler-looking alternative, a decoder that refuses non-canonical
escaping, was not taken. The string reader is shared by every CAS text
format, so a stricter reader changes all of them. Checking the cap on
the canonical form means re-encoding, which the digest already does.

A successfully decoded manifest now always re-encodes, so the relink
offer, which re-encodes a decoded manifest, needs no change.

Also state in the headers what `computePayloadDigest` and
`decodePartManifest` throw, keep the line cap out of "what no writer
builds", and wrap long comment lines.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
120 s is not enough under sanitizers: in CI the insert of 1100 one-row
parts hit the limit in all three MSan runs (133 s, 120 s, 121 s before
the client gave up) and once of three under TSan (119.6 s). 600 s is
`DEFAULT_QUERY_TIMEOUT`, the most `wait_and_read_output` waits anyway.
The timer no longer delays the exit of the test process, so the larger
value costs nothing.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@k-morozov k-morozov left a comment •
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked _chpt changes. LGTM.
I described several points but it's not a blocker so we could resolve it in the next PR.

However, this PR combines substantial CAS changes with a longer default mount lease. The lease increase currently prevents hard-restart tests from completing successfully, leaving important recovery paths unverified. I would prefer to move the default lease increase into a separate PR, together with the necessary test adjustments, and validate the remaining changes with the existing lease duration.

RefReplayBuilder builder(std::move(base_snapshot), base_snapshot_bytes);
if (grounding.walk_from && grounding.committed_through)
bool replay_failed = false;
const RefChainVisitor apply = [&](RefLogTxn && txn, uint64_t stored_bytes)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sweep budget does not bound this owner replay: sweep_recovery_ops_used is incremented only after recoverRefTableDetailedFromAuthority returns. If a writer keeps appending ahead of the reader, recovery can keep following the
growing chain and prevent the entire GC round from completing. Previously, the captured committed_through provided a finite boundary.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, fixed in 20b0225. Generation 3 has no committed_through, so the replay ran to the first absent key and was charged once, after it returned. Now the sweep charges it per record. When the budget runs out the replay stops and the namespace keeps its candidates for this round. A stopped replay returns no table: a prefix read as the owner set could delete an owned manifest. Test: a writer appends on every ref-log GET.

CasOperation op = mount_requests.resume(admitted_generation);
const CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, layout);
refuseUnlessAdmitted(op, "ref recovery catalog confirmation");
if (catalogStillNamesLife(snapshot.catalog, rt.life))
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confirmRecoveredLifeOrDetach accepts a same-life Removing row but does not set removal_admission_closed. If the row changes from Live to Removing during recovery, the server observes the removal yet still admits subsequent writes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, fixed in fc82bf0. The confirmation now latches Removing like every other read of that row; the next write is refused. No data was at risk: the only way to get there is a late beginRemoving from before a remount, and GC still folded the appends. The design excluded this site for a reason that does not hold, so the spec is corrected too.

@filimonov filimonov mentioned this pull request Nov 16, 2026
10 of 30 tasks
filimonov and others added 6 commits November 17, 2026 00:03
A pending S3-staged blob may be published once by a verbatim copy of its
staging object. The privilege lived in each `BlobSource`. `createHardLink`
of a pending blob into another part copies the `PendingBlob` record, and
`uploadPendingBlobs` built a fresh source per part, so one transaction
could copy the same envelope to the blob key twice. On a store with
content-derived ETags the second copy carries the token of the first, and
a delete `If-Match` that token issued for the first body removes the second.

`PendingBlob` now owns the `publication_attempted` flag and hands it to the
source. Hardlink copies share it, so only the first publication across all
of them may copy verbatim; every other one streams the staged payload under
a fresh envelope.

The test fake `FakeGenerationObjectStorage` gains an ETag mode with tokens
derived from the bytes, and an in-memory `copyObject`. Two tests hardlink a
pending blob into a later and into an earlier part, delete the body after
the first part publishes, and check one verbatim copy, a new token, and a
replayed delete that misses.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`confirmRecoveredLifeOrDetach` accepted a same-life `Removing` row and
left the lane open, so a write admitted as `Live` could keep committing
after the row turned. Every other read of a `Removing` row closes the
lane through `latchRemovingLocked`.

The confirmation now latches the same way, under `ref_queue_mutex`,
with `state_mutex` released at both call sites. Recovery still installs.
`appendRefOpsOnRuntime` re-reads the flag under `ref_queue_mutex` after
recovery, so the next positive write is refused as "is Removing in the
catalog" and reads see no data. No request is added.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Under generation 3 `_ckpt` stores no end of the ref chain, so the owner
replay of `recoverRefTableDetailedFromAuthority` reads to the first
absent key and follows records appended while it runs. The sweep charged
it one unit after it returned, so a writer that stays ahead of the reader
could hold a GC round.

`recoverRefTableDetailedFromAuthorityBounded` takes a check called after
each replayed record and reports `Recovered`, `BaseMoved` or `Stopped`.
Only `Recovered` carries a table: a stopped replay is a prefix of the
chain, and a prefix read as the owner set could delete an owned
manifest. `activeManifestKeys` charges the round's recovery budget per
record through that check and, when the replay stops, marks the
namespace `recovery_incomplete`, so its candidates are retained this
round. The old function keeps its type and passes no check; fsck and the
`REBUILD` path are unchanged.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The hard-restart scenarios of the regression suite wait 60 s for a
server killed with `SIGKILL`. A restarted server watches its own old
lease for `lease + lease / 20 + renew period / 2`
(`mountObservationThresholdMs`): 131 s at a 120 s lease, 47 s at 40 s.
40 s keeps those scenarios passing and still widens the write window of
an outage: `TTL - margin - period - 2 x envelope` is 14 s with the
defaults, against 4 s at 30 s.

With the default envelope of 7 s both startup checks pass: 7 s + 2 s
< 40 s, and 10 s + 14 s + 2 s = 26 s < 40 s. The envelope ceiling is
now 14 s, so a disk whose `connect_timeout_ms` is `0` or 4500 ms or more
opens writable only with a raised lease; the docs say so. The upgrade
notes keep the pin to `30000` while a pool has a build without the
declared-lease rule: such a server reclaims its own slot after 36.5 s,
inside a 40 s lease, and fences inside it with a renewal period below
8.5 s.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The pool is server-wide and shared by every concurrent insert and merge
that commits a CAS part. On a wide insert of 62,000 blobs 16 threads
were saturated and held the insert for most of its wall time. The
generic IO pool (`max_io_thread_pool_size`) has 100 threads.

Only the default changes. Nothing else is sized from the pool: its queue
is its thread count, the caller only submits and joins, and
`clickhouse-disks` mirrors the default.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The `cas_tiered` policy had no `move_factor`, so the default 0.1 applied to its local `hot` volume. On a host with less than 10% free disk the volume counts as full, and the background mover relocated a freshly fetched part to the CAS disk within a second, racing the tests that read `system.parts.disk_name`.

Set `move_factor` to 0 so placement no longer depends on the host's free space. TTL moves are unaffected.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@filimonov
Copy link
Copy Markdown
Member Author

On the lease: the default is 40 s now (feb86da), not 120 s. A killed server waits lease + lease/20 + 5 s = 47 s for its old lease, plus the start time. The hard-restart scenarios allow 60 s, so they should pass without the regression change; the margin is small, about 6 s. A longer default can come later, after Altinity/clickhouse-regression#184.

Both inline comments are fixed in this push. Also new: cas_blob_upload_pool_size 16 -> 64 (9fbdefb).

filimonov added a commit to filimonov/ClickHouse that referenced this pull request Nov 17, 2026
Scope of the wave on top of the late-request audit bundle: the port of
PR 2570 with the cold-admission guard (CAS-428), the drain pause order
(CAS-463), the 120 s mount lease and the configurable request window
(CAS-405.3, CAS-405.6), unbounded GC round budgets (CAS-29.11), one
request per list page (CAS-29.13), four observability fixes and five CI
fixes. Each story names its tests in failing-first order.

PR: Altinity#2831

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
@k-morozov
k-morozov marked this pull request as ready for review November 17, 2026 14:05
@filimonov
Copy link
Copy Markdown
Member Author

CI on 44fc40023c1 (run 44084349150): 110 jobs green, 15 red. One red comes from this PR.

Caused by this PR

Job Test Cause
Stateless tests (amd_binary, cas s3 storage, parallel) 03770_min_columns_to_activate_adaptive_write_buffer MEMORY_LIMIT_EXCEEDED at the test's 50 MiB limit. Every blob upload holds two 1 MiB buffers charged to the query. With cas_blob_upload_pool_size raised from 16 to 64 the peak grows from 32 MiB to 128 MiB for a part with many blobs.

Follow-up PR: size the upload buffers to the blob (clamp(blob size, 1, 1 MiB)), so a part of small blobs holds kilobytes, not megabytes. The fix is written and passes 03770 locally with CAS as the default storage. The same PR will look at the per-stream write buffers of the part writer.

Not caused by this PR

Jobs What Why not this PR
Stateless tests (amd_msan, cas s3 storage) 1-3/3, (amd_tsan, cas s3 storage) 1-2/2 job time limit after ~3 h 38 min, empty report These shards have not fitted the limit before this PR either.
RegressionTestsAarch64: CASAggregateFunctions (1-3), CASS3CacheAggregateFunctions (1-3) snapshot differences in the last digits on aarch64 The Release twins are green; no aggregate function code is touched.
RegressionTestsAarch64: CASS3CacheLightweightDelete (4) one leaf: concurrent delete/VersionedCollapsingMergeTree/concurrent delete without overlap The Release twin and the non-cache twins are green. Looks like the concurrent mutation ordering bug of ClickHouse#122934, not confirmed from the logs.
Stateless tests (arm_binary, parallel) 05484_cas_mounts_gc_success_age_null: correct output, failed on a server warning about high CPU usage in stderr 14 of 14 local reruns pass.
Unit tests (msan) FileCacheTest.ReserveUndoneWhenKeyDirectoryCannotBeCreated File cache test from a recent backport to the base branch; this PR does not touch the file cache.

What this run shows

  • Hard restart regression jobs are green on both architectures with the 40 s lease (CAS (atomic_insert), CASS3Cache (atomic_insert), CASLightweightDelete (2), CASS3CacheLightweightDelete (2)). They were red with the 120 s lease.
  • Both Stateless tests (amd_asan_ubsan, cas s3 storage) shards are green, in 2 h 15 min and 2 h 43 min.
  • All integration shards are green, including test_cas_replicated_relink.

@filimonov
filimonov merged commit 0aa837e into antalya-26.8 Nov 17, 2026
421 of 452 checks passed
@filimonov filimonov added the verified Approved for release label Nov 17, 2026
@DimensionWieldr
DimensionWieldr commented Nov 18, 2026 •
Copy link
Copy Markdown
Collaborator

No AI audit defects.

No fails in CI are PR-caused.

CAS checks: https://github.com/Altinity/clickhouse-regression/actions/runs/44138864219

  • No fails are relevant to this PR

LGTM

filimonov added a commit to filimonov/ClickHouse that referenced this pull request Nov 21, 2026
Altinity#2831 merged as 0aa837e.

- Done (19): CAS-1.1, 69, 148, 176.3, 354, 364, 373, 400, 428, 434, 443,
  448, 449, 455, 466, 480, 481, 489, 494.
- Done as obsolete (5): CAS-176.2, 167, 366, 451, 459; drafts 26 and 31
  archived. The one-request append writes no `_ckpt` per append.
- Partly delivered (24): notes name the criteria met and the ones open.
- Related tasks: the default mount lease is 40 s, an append is one
  `_log` create; CAS-405.3 and CAS-475 retitled.
- Adds tasks CAS-468 to CAS-496, created earlier and not committed.

Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

0