8000 Keep the CAS mount lease renewal alive over the server memory limit by filimonov · Pull Request #2511 · Altinity/ClickHouse · GitHub
[go: up one dir, main page]

Skip to content

Keep the CAS mount lease renewal alive over the server memory limit - #2511

Merged
filimonov merged 1 commit into
antalya-26.6-20261008from
fix/antalya-26.6/cas-renewal-memory-guard
Oct 8, 2026
Merged

filimonov merged 1 commit into
antalya-26.6-20261008from
fix/antalya-26.6/cas-renewal-memory-guard

Conversation

@filimonov
@filimonov filimonov commented Oct 8, 2026 •
Copy link
Copy Markdown
Member

The renewal thread allocated under the global memory tracker, so a server near max_server_memory_usage could throw MEMORY_LIMIT_EXCEEDED inside a lease renewal. The loop's catch-all then fenced the mount and ended the renewal thread for good, and the mount went through a remount under a fresh epoch because of memory pressure that had nothing to do with the object store. The renewal now survives the server memory limit.

Why it matters for the lease

The lease thread is the one thread that must not die first: without a renewal the holder loses its slot after the observation window, its unfinished work loses GC protection and every writer waits for the remount. The next change of the series (#2431) makes the renewal retry through an object-store outage instead of giving up; that promise needs a renewal thread that cannot be killed by the state of the server itself.

Changes

  • LockMemoryExceptionInThread(VariableContext::Global) for the whole renewal iteration, result handling included: the tracker still counts this thread's allocations but never throws on them. Global, because a User-level lock leaves the server limit enforced. Precedents: KeeperServer, TransactionLog. std::bad_alloc stays terminal.
  • Conditional writes set s3_allow_parallel_part_upload = false, so the PUT runs on the calling thread and the guard covers it. This applies to every Native conditional write (lease, manifests, gc/state, ref ledger); a body above max_single_part_upload_size on an ETag store now uploads its parts one after another, on a call that already waits for the whole write.
  • The conditional PUT buffer opens at min(body size, 1 MiB) instead of 1 MiB: a 200-byte control-plane write no longer costs a megabyte per attempt.

Risks / notes

  • Until the renewal rework lands, a memory-limit exception inside the remount worker still costs one remount attempt, not the thread.
  • Rejected: MemoryTrackerBlockerInThread (hides accounting), a deterministic classification of the error, a longer budget.

Testing

  • Regression test: the global limit is exceeded on the lease thread before the request, inside it and while the result is handled; the thread stays alive, the mount stays Live, CASMountLeaseLost does not move. Red on the base; red again with the guard at User level or below the admission block.
  • Behaviour-preservation tests: std::bad_alloc inside a renewal stays terminal; a 3 MiB conditional object is one PUT with every byte.
  • Tests for the calling thread and the buffer size (red on the base).
  • CAS* gate: 2613 (release), 2617 (ASan, no error); S3SingleAttemptClient.* under ASan; test_cas_mount_renewal_retry: 4 passed.

Closes: #2403
Closes backlog task CAS-292.

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

A server memory limit no longer ends the CAS mount lease renewal thread; the conditional PUT of a CAS control-plane write runs on the calling thread with a buffer of the body size.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Updated docs/en/antalya/cas/architecture/mounts-and-leases.md.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

The renewal thread allocated under the global memory tracker, so a server
near max_server_memory_usage could throw MEMORY_LIMIT_EXCEEDED inside a
renewal; the loop's catch-all then fenced the mount and ended the thread
for good, and the mount went through a remount under a fresh epoch
because of memory pressure that had nothing to do with the object store.

Hold LockMemoryExceptionInThread(VariableContext::Global) for the whole
renewal iteration: the tracker still counts this thread's allocations but
never throws on them. Run the conditional PUT on the calling thread
(s3_allow_parallel_part_upload = false for conditional writes), so the
guard covers the upload; open the PUT buffer at the body size instead of
the default 1 MiB. std::bad_alloc stays terminal.

Closes: #2403
Closes backlog task CAS-292.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions
github-actions Bot commented Oct 8, 2026
Copy link
Copy Markdown

Workflow [PR], commit [949578b]

@filimonov
filimonov merged commit d76b59f into antalya-26.6-20261008 Oct 8, 2026
306 of 316 checks passed
@svb-alt svb-alt added the port-antalya PRs to be ported to all new Antalya releases label Oct 11, 2026
zvonand pushed a commit that referenced this pull request Oct 12, 2026
…next commit)

---
Original cherry-pick message follows:

Merge pull request #2511 from Altinity/fix/antalya-26.6/cas-renewal-memory-guard

Keep the CAS mount lease renewal alive over the server memory limit
# Conflicts:
#	src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp
zvonand added a commit that referenced this pull request Oct 12, 2026
Kept both includes: `Common/ProfileEvents.h` from antalya-26.8 and `Common/LockMemoryExceptionInThread.h` from the source PR.

Source-PR: #2511 (#2511)
@k-morozov k-morozov added the verified Approved for release label Oct 20, 2026
@zvonand zvonand removed the port-antalya PRs to be ported to all new Antalya releases label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

0