Repository navigation
Keep the CAS mount lease renewal alive over the server memory limit - #2511
Merged
filimonov merged 1 commit intoOct 8, 2026
Merged
Conversation
The renewal thread allocated under the global memory tracker, so a server near max_server_memory_usage could throw MEMORY_LIMIT_EXCEEDED inside a renewal; the loop's catch-all then fenced the mount and ended the thread for good, and the mount went through a remount under a fresh epoch because of memory pressure that had nothing to do with the object store. Hold LockMemoryExceptionInThread(VariableContext::Global) for the whole renewal iteration: the tracker still counts this thread's allocations but never throws on them. Run the conditional PUT on the calling thread (s3_allow_parallel_part_upload = false for conditional writes), so the guard covers the upload; open the PUT buffer at the body size instead of the default 1 MiB. std::bad_alloc stays terminal. Closes: #2403 Closes backlog task CAS-292. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
k-morozov
approved these changes
Oct 8, 2026
zvonand
pushed a commit
that referenced
this pull request
Oct 12, 2026
…next commit) --- Original cherry-pick message follows: Merge pull request #2511 from Altinity/fix/antalya-26.6/cas-renewal-memory-guard Keep the CAS mount lease renewal alive over the server memory limit # Conflicts: # src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/Pool/CasMountRuntime.cpp
This was referenced Oct 12, 2026
10 of 30 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The renewal thread allocated under the global memory tracker, so a server near
max_server_memory_usagecould throwMEMORY_LIMIT_EXCEEDEDinside a lease renewal. The loop's catch-all then fenced the mount and ended the renewal thread for good, and the mount went through a remount under a fresh epoch because of memory pressure that had nothing to do with the object store. The renewal now survives the server memory limit.Why it matters for the lease
The lease thread is the one thread that must not die first: without a renewal the holder loses its slot after the observation window, its unfinished work loses GC protection and every writer waits for the remount. The next change of the series (#2431) makes the renewal retry through an object-store outage instead of giving up; that promise needs a renewal thread that cannot be killed by the state of the server itself.
Changes
LockMemoryExceptionInThread(VariableContext::Global)for the whole renewal iteration, result handling included: the tracker still counts this thread's allocations but never throws on them.Global, because aUser-level lock leaves the server limit enforced. Precedents:KeeperServer,TransactionLog.std::bad_allocstays terminal.s3_allow_parallel_part_upload = false, so thePUTruns on the calling thread and the guard covers it. This applies to every Native conditional write (lease, manifests,gc/state, ref ledger); a body abovemax_single_part_upload_sizeon an ETag store now uploads its parts one after another, on a call that already waits for the whole write.PUTbuffer opens atmin(body size, 1 MiB)instead of 1 MiB: a 200-byte control-plane write no longer costs a megabyte per attempt.Risks / notes
MemoryTrackerBlockerInThread(hides accounting), a deterministic classification of the error, a longer budget.Testing
Live,CASMountLeaseLostdoes not move. Red on the base; red again with the guard atUserlevel or below the admission block.std::bad_allocinside a renewal stays terminal; a 3 MiB conditional object is onePUTwith every byte.CAS*gate: 2613 (release), 2617 (ASan, no error);S3SingleAttemptClient.*under ASan;test_cas_mount_renewal_retry: 4 passed.Closes: #2403
Closes backlog task CAS-292.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
A server memory limit no longer ends the CAS mount lease renewal thread; the conditional
PUTof a CAS control-plane write runs on the calling thread with a buffer of the body size.Documentation entry for user-facing changes
Updated
docs/en/antalya/cas/architecture/mounts-and-leases.md.CI/CD Options
Exclude tests:
Regression jobs to run: