Repository navigation
CAS improvements for antalya-26.8 - #2831
Conversation
The incarnation was minted once before the admission loop. A resend, or a re-decide after a lost response, could insert an incarnation that a concurrent remove had already erased. The catalog has no revision field: cancelling an added row restores the old bytes and ETag, so a late copy of a resent row could revive a cancelled life. Step 1 now mints inside the mutate callback and passes `resend_unresolved = false`, so an attempt of unknown fate is decided again on a fresh read instead of being resent. It returns the row that landed; a landed write without that row raises `LOGICAL_ERROR`. Tests cover a conflict retry, a lost response that landed and one that did not, a refused catalog read, the deadline, a fence lost after an unresolved attempt and during the pause before the next decision, and a sibling that erases the row between a landed write and the next decision. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The lane close for a deposed writer moves into a local lambda so a second caller can share it without duplicating the exception text. No behaviour change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Five callers compare a `_ckpt` publish outcome with `FencedOut` only to learn whether the contribution became durable. A named predicate lets later outcome values be added without touching them. No behaviour change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A deposed writer's create can land in a ref-log slot that cleanup already reclaimed, and the merge accepted its frontier by numeric comparison. `publishCkpt` now refuses a non-seal frontier that sits at the durable epoch seal (`SuccessorSeal`) or below a later epoch's frontier (`Superseded`), so the append is rejected or left unacknowledged. `CASRefCheckpointFrontierRefused` counts the refusals. Callers: - A wedge retry that a successor seal rejected is below the checkpoint, so recovery never installs it. Its survivors get the "outcome is unknown" answer, and the `Rejected` text no longer says "nothing is durable" where the log object is durable. - A recovery walk whose frontier publish is refused ends with the retry-later error and writes nothing. The next recovery grounds on the successor's snapshot. - Both ref-log sites that act on a publication outcome end with an assertion that the remaining outcome is a recorded one, so a new enumerator cannot fall through as "acknowledged". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Ref cleanup may delete a covered log after advancing `_ckpt` past the sample `fsck` took. An absent tail id now gets the same `_ckpt` re-read as the checkpoint base: a changed, absent or unreadable `_ckpt` gives `Unchecked`, an unchanged one stays chain-broken. One local helper holds the re-read for both the base validation and the tail walk. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Ref cleanup refused to delete whenever the catalog object's etag moved, so unrelated namespace churn starved it. The chunk licence now depends on the namespace's own row and life and on the GC lease. A moved row or life skips that namespace and the pass continues; a moved lease, a failed read or an ambiguous catalog stops the pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Operators could not tell a skipped namespace from a stopped pass, or see how many planned keys were deleted. Add five counters and six `ref_object_cleanup` phase metrics reporting plans, deletes, skips, stop reasons and chunks deleted under a changed catalog etag. - An absent `gc/state` is a pool anomaly, not a leader takeover: the licence check reports it as `StopAuthorityUnverified`. `StopLeaseMoved` means `gc/state` was read and its lease differs from the adopted one. - A scope-exit guard attaches the phase metrics before the phase timer emits the row, so a delete chunk that throws still reports what was deleted. - The race tests assert the counters: a moved row or life counts as a skip and not a pass stop, and a trim-disabled round reports zero metrics. The `gc/state` fault tests observe that the fault fires at the licence read and not in the namespace janitor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Phase 17 licenses each chunk by the namespace's own catalog row and life plus the GC lease, and a refusal skips one namespace or stops the pass. The architecture pages describe that, the new counters and phase metrics, and the `_ckpt` frontier refusal results. The operator pages cover the refusal messages and the `fsck` unchecked finding. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`CASRefWriterSnapshotPublish.C4InFlightGateAdmitsAtMostOne`: a setup publisher that met the second commit's `Writing` lane armed the backoff, so the later `dropRef` triggered no `_snap/` PUT and `awaitBlockEntered` waited forever. The test now disables the backoff and asserts a compacted tail. `awaitBlockEntered` gives up after 20 s and returns the result of its wait, and each caller asserts it. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A slot that holds a lower epoch than the one being asked about is an older incarnation, so its GC fence or clean farewell says nothing about the higher epoch. Only a greater epoch proves supersession, whether the slot's body is live, GC-fenced or a clean release. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The renewer records the ETag of every body it lands and the attempt id of the renewal in flight, so a reader on another thread can tell whether a slot body is one this mount wrote. Nothing reads the record yet. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The in-memory fence can say live after another server claimed the mount slot or GC fenced it, so a confirm answered `Yes` for a ref this mount no longer protects. After a ledger `Yes` the pool reads the mount slot once, and the answer stays `Yes` only if the slot still holds a body this renewer wrote. Otherwise, and when the read fails or no renewer exists, the answer is `Unknown`. The ownership is captured before the ledger snapshot, so a remount between them cannot validate an old snapshot. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The relink confirm reads the mount slot once before a ledger `Yes` and answers `Unknown` when the slot is not held. The architecture pages no longer call the confirm I/O-free. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A `Live -> Removing` catalog write that was sent and not answered can still land after the removal call gave up. Reopening the lane on one read of the old `Live` row then lets a late write stop the table at a later append, with nothing linking it to the failed `DROP`. The lane now stays closed until a retried removal completes or the disk is remounted, and the table's runtime is not evicted meanwhile. A drop that fails before it sends a removal write still reopens the lane. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A drop gave every read its own standard window, and the creator-fence read nested in the catalog update opened a fresh one on every decision. The drop freezes one policy at entry and spends it across the catalog reads, the creator-fence read, the `gc/state` read and both catalog updates. The resolution read after a failure keeps its own window so it can still reopen the lane. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A writer that already lost admission can hold a stale contribution that the durable body rejects in the merge. It got `CORRUPTED_DATA` for a body nothing had corrupted. It now gets `FencedOut`; an admitted writer still gets `CORRUPTED_DATA`. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An object removed from the bucket between the read and the write of a pool metadata update is an environment fault, not a program bug, so it is reported as `INVALID_STATE` instead of `LOGICAL_ERROR`, which also aborts debug and sanitizer builds. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The `GetBucketVersioning` probe in `checkPoolPreconditions` goes through the single-attempt client with the attempt timeout, so the SDK does not retry it and the adaptive first-attempt timeouts do not cut a slow answer short. The attempt timeout bounds each send and receive, and the connect cap bounds the connect. Only a writable mount runs the probe. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A cold reader or writer that read a `Live` catalog row can install a runtime for a life that was removed and drained before the insertion. The ref-name slot now counts retirements and outlives eviction while the count is non-zero. Admission throws retry 8000 -later when the count moved since the catalog read, and when the resident life has already applied its terminal removal. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The GC drain recorded a retired life only after the erase, so a cold admission that inserted its runtime in between saw an unchanged retirement count. `deleteCompletedRemovingAtSnapshot` now calls a `before_erase` hook once before the first erase, and the drain passes the invalidation of the retired life. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Another server's GC leader can drain a life without telling the owner, so the owner evicted the idle runtime of that life with a zero retirement count and erased its slot. Eviction of a runtime whose removal is closed or applied now bumps the count and keeps the slot. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Several paths leave a runtime recovered as terminal with no local removal in progress: a presence probe on a `Removing` row, the early return of `dropNamespaceImpl`, and a local GC whose retirement announcement preceded the runtime's publication. Once the row is erased, `namespaceLife` refuses the name until eviction. The warm path now reads the catalog for such a life and, when the row is gone, detaches the runtime and births a new life. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Every retired namespace name kept a `RefNameSlot` until the next remount, so create and drop cycles grew `ref_name_slots` without bound. A cold admission now holds a `ColdAdmission` guard that keeps the slot of its name from capture to publish. A slot with neither a runtime nor an admission is erased. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The erase loop of a completed removal paused after its resolution read. A catalog write landing in the pause refused the next erase, because that erase was conditioned on a read taken before the pause. The pause schedule is unchanged (`backoff(attempt + 1)`, up to 5 s). The read now follows the pause, so the next erase is conditioned on a read taken after it. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The reconciler keeps a count of completed erases that stays readable after `reconcile` throws. `Gc::runRegularRound` writes it to `deleted` of the `pre_fold_ref_drain` phase on every exit of the phase scope, so a drain that throws still reports the erases it completed. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`CasRefCatalog::deleteCompletedRemovingAtSnapshot` reports an acknowledged erase through `erase_acknowledged` before its resolution read. The drain counts and announces that retirement even when the read throws or the call ends `FencedOut` or `EntryChanged`. A warm `namespaceLife` detaches a resident life that fails recovery once the catalog no longer names it. The presence probe answers present instead of throwing when the drain announces during its admission, and `before_erase` is required on the production erase. The aborted-drain test runs on the virtual retry clock. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A 30 s lease leaves a 4 s write window in an outage. The default is now 120 s with the renew period unchanged at 10 s, so the window is 94 s. Integration tests that kill a server keep 30 s in their own configs. The upgrade notes require pinning `cas_mount_lease_ttl_ms` to `30000` while a pool still has a build without the declared-lease rule, and name what an older server's early fence leaves unprotected. The docs give the GC fence wait as up to two GC intervals after the lease threshold. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The request budget log line carries `write_reserve_ms`, and `renewal_slack_ms` when background renewal is on, so an operator can read the lease headroom from the log. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`wait_and_read_output` cancelled the timeout timer of every command it finished waiting for. When the wait itself gives up after `DEFAULT_QUERY_TIMEOUT` with the client process st 10000 ill running, the timer is what kills that process, so it has to stay armed. Cancel it only once the process has ended. This is the text of upstream `813bdc10c98`. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A writer that replaces a blob's `.meta` marker with `If-Match` gets 404 when GC deletes the marker between the writer's read and its write. `finalizeConditionalWrite` maps that to a lost precondition and the writer starts over, but the HTTP client logged the response at Error. A stateless test that checks stderr then fails on the log line alone: `02841_not_ready_set_bug` did, once in a shard of 6891 tests. `nativeConditionalPut` now sends an `If-Match` write under `Expect404ResponseScope`, as `readUnder` does for its `GET`. The request runs on the calling thread, since conditional writes switch the parallel part upload off. A write with `If-None-Match` stays loud: it names no existing object. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`DecodeRejectsMalformedEntryPaths` built its bad bytes with `encodePartManifest`, so it stops working as soon as the encoder refuses a bad path. Encode a legal placeholder path, splice the bad path in, and compute the digest over the spliced bytes the way `computePayloadDigest` does, without calling the encoder on the bad path. Only the decoder's path check can refuse these bytes, and the test asserts its message `invalid entry path`, not only the code. A control case proves the forge reproduces the encoder's bytes for a legal path. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review: https://gist.github.com/k-morozov/7b34b402f18b051ced88f457c95d5df4 (cherry picked from commit 657c521) Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com> (cherry picked from commit ff3b531)
Check encoded text lines in `encodePartManifest` against the decoder's line cap before any manifest body is written. Measure escaped bytes without the line terminator, and leave raw inline payload outside this limit. Cover entry and descriptor boundaries, JSON escaping and refusal before the body write. Build total-size cap fixtures from many valid records. Validation: focused release build and all 105 CAS tests passed. Signed-off-by: Konstantin Morozov <just.morozov.k@gmail.com> (cherry picked from commit 90acf02) Conflicts resolved: `gtest_cas_part_manifest_format.cpp`: the new encode tests are placed before the forged decode test; `DecodeRejectsARecordLineOverTheLineCap` is added, because no writer-made body can carry an over-long line to the decoder any more. Conflicts resolved: `gtest_cas_part_write.cpp`: `manifestWithEncodedSize` replaces `fillEntriesToTheEncodedCap` and its helpers for both object-cap boundary tests; `StageManifestRejectsOversizedEntryRecordBeforeBodyWrite` replaces `StageManifestRefusesARecordLineOverTheLineCap`, which asserted the same refusal with the decoder's `CORRUPTED_DATA`. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Adjust the checks that "Validate CAS manifest entry paths before encoding" and "Reject oversized CAS manifest lines before staging" added to `encodePartManifest`. - A bad or duplicate entry path is a bug of the writer, so the encoder throws `LOGICAL_ERROR`. The decoder keeps `CORRUPTED_DATA`: there the same path is bad bytes. `validateManifestEntryPath` takes the code, so the path rule stays in one place. - A line over the line cap keeps `LIMIT_EXCEEDED`. It comes from the size of the input, such as a very long file path, not from a broken invariant of the writer, so it is a limit, not a bug. - The header and trailer lines have a fixed shape and can never reach the line cap; their checks are removed. - `stageManifest` takes the object cap from `traitsFor(FormatId::PartManifest).object_cap` through `fitsObjectCap` instead of its own copy of the number, so the writer and the readers cannot drift apart. The tests derive their sizes from the same traits. Tests that now expect `LOGICAL_ERROR` are split: the release build checks the exception, sanitizer and debug builds check the abort. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The "was not staged by this transaction" errors of `precommitAdd` and `promote` now carry the build id and the target namespace next to the manifest and the ref, so the log names the transaction that refused. The death tests of `precommitAdd` match that text instead of any abort. Fix the text that "Validate CAS manifest entry paths before encoding" made false. The encoder refuses what no writer may build: a bad or duplicate entry path with `LOGICAL_ERROR`, a line over the line cap with `LIMIT_EXCEEDED`. `stageManifest` still decodes the sealed bytes with the readers' decoder before the `PUT`, so nothing a reader would reject is written (`CORRUPTED_DATA`). The comment on `validateManifestEntryPath` again gives the reason for the path rule: manifest bytes also arrive over the interserver relink channel. Also drop "after fail-closed reval" from the `promote` event reason and the unused `#include <set>` from `CasPartWriteTxn.h`. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`decodePartManifest` checks the line cap on the stored line, but it accepts strings the encoder writes longer, such as raw control bytes that the encoder escapes to `\u0001`. `computePayloadDigest` re-encodes the decoded body, and the encoder's line cap then threw `LIMIT_EXCEEDED`. Callers treat only `CORRUPTED_DATA` as bad bytes: `prepareAdoptFromManifest` falls back to a byte fetch on it alone. The decoder now maps `LIMIT_EXCEEDED` from the digest computation to `CORRUPTED_DATA`. Other codes still propagate: the decoder refuses a bad or duplicate path before the digest, so a `LOGICAL_ERROR` there is a real bug. The simpler-looking alternative, a decoder that refuses non-canonical escaping, was not taken. The string reader is shared by every CAS text format, so a stricter reader changes all of them. Checking the cap on the canonical form means re-encoding, which the digest already does. A successfully decoded manifest now always re-encodes, so the relink offer, which re-encodes a decoded manifest, needs no change. Also state in the headers what `computePayloadDigest` and `decodePartManifest` throw, keep the line cap out of "what no writer builds", and wrap long comment lines. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
120 s is not enough under sanitizers: in CI the insert of 1100 one-row parts hit the limit in all three MSan runs (133 s, 120 s, 121 s before the client gave up) and once of three under TSan (119.6 s). 600 s is `DEFAULT_QUERY_TIMEOUT`, the most `wait_and_read_output` waits anyway. The timer no longer delays the exit of the test process, so the larger value costs nothing. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
There was a problem hiding this comment.
I've checked _chpt changes. LGTM.
I described several points but it's not a blocker so we could resolve it in the next PR.
However, this PR combines substantial CAS changes with a longer default mount lease. The lease increase currently prevents hard-restart tests from completing successfully, leaving important recovery paths unverified. I would prefer to move the default lease increase into a separate PR, together with the necessary test adjustments, and validate the remaining changes with the existing lease duration.
| RefReplayBuilder builder(std::move(base_snapshot), base_snapshot_bytes); | ||
| if (grounding.walk_from && grounding.committed_through) | ||
| bool replay_failed = false; | ||
| const RefChainVisitor apply = [&](RefLogTxn && txn, uint64_t stored_bytes) |
There was a problem hiding this comment.
The sweep budget does not bound this owner replay: sweep_recovery_ops_used is incremented only after recoverRefTableDetailedFromAuthority returns. If a writer keeps appending ahead of the reader, recovery can keep following the
growing chain and prevent the entire GC round from completing. Previously, the captured committed_through provided a finite boundary.
There was a problem hiding this comment.
Confirmed, fixed in 20b0225. Generation 3 has no committed_through, so the replay ran to the first absent key and was charged once, after it returned. Now the sweep charges it per record. When the budget runs out the replay stops and the namespace keeps its candidates for this round. A stopped replay returns no table: a prefix read as the owner set could delete an owned manifest. Test: a writer appends on every ref-log GET.
| CasOperation op = mount_requests.resume(admitted_generation); | ||
| const CasRefCatalog::Snapshot snapshot = CasRefCatalog::read(op, layout); | ||
| refuseUnlessAdmitted(op, "ref recovery catalog confirmation"); | ||
| if (catalogStillNamesLife(snapshot.catalog, rt.life)) |
There was a problem hiding this comment.
confirmRecoveredLifeOrDetach accepts a same-life Removing row but does not set removal_admission_closed. If the row changes from Live to Removing during recovery, the server observes the removal yet still admits subsequent writes.
There was a problem hiding this comment.
Correct, fixed in fc82bf0. The confirmation now latches Removing like every other read of that row; the next write is refused. No data was at risk: the only way to get there is a late beginRemoving from before a remount, and GC still folded the appends. The design excluded this site for a reason that does not hold, so the spec is corrected too.
A pending S3-staged blob may be published once by a verbatim copy of its staging object. The privilege lived in each `BlobSource`. `createHardLink` of a pending blob into another part copies the `PendingBlob` record, and `uploadPendingBlobs` built a fresh source per part, so one transaction could copy the same envelope to the blob key twice. On a store with content-derived ETags the second copy carries the token of the first, and a delete `If-Match` that token issued for the first body removes the second. `PendingBlob` now owns the `publication_attempted` flag and hands it to the source. Hardlink copies share it, so only the first publication across all of them may copy verbatim; every other one streams the staged payload under a fresh envelope. The test fake `FakeGenerationObjectStorage` gains an ETag mode with tokens derived from the bytes, and an in-memory `copyObject`. Two tests hardlink a pending blob into a later and into an earlier part, delete the body after the first part publishes, and check one verbatim copy, a new token, and a replayed delete that misses. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`confirmRecoveredLifeOrDetach` accepted a same-life `Removing` row and left the lane open, so a write admitted as `Live` could keep committing after the row turned. Every other read of a `Removing` row closes the lane through `latchRemovingLocked`. The confirmation now latches the same way, under `ref_queue_mutex`, with `state_mutex` released at both call sites. Recovery still installs. `appendRefOpsOnRuntime` re-reads the flag under `ref_queue_mutex` after recovery, so the next positive write is refused as "is Removing in the catalog" and reads see no data. No request is added. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Under generation 3 `_ckpt` stores no end of the ref chain, so the owner replay of `recoverRefTableDetailedFromAuthority` reads to the first absent key and follows records appended while it runs. The sweep charged it one unit after it returned, so a writer that stays ahead of the reader could hold a GC round. `recoverRefTableDetailedFromAuthorityBounded` takes a check called after each replayed record and reports `Recovered`, `BaseMoved` or `Stopped`. Only `Recovered` carries a table: a stopped replay is a prefix of the chain, and a prefix read as the owner set could delete an owned manifest. `activeManifestKeys` charges the round's recovery budget per record through that check and, when the replay stops, marks the namespace `recovery_incomplete`, so its candidates are retained this round. The old function keeps its type and passes no check; fsck and the `REBUILD` path are unchanged. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The hard-restart scenarios of the regression suite wait 60 s for a server killed with `SIGKILL`. A restarted server watches its own old lease for `lease + lease / 20 + renew period / 2` (`mountObservationThresholdMs`): 131 s at a 120 s lease, 47 s at 40 s. 40 s keeps those scenarios passing and still widens the write window of an outage: `TTL - margin - period - 2 x envelope` is 14 s with the defaults, against 4 s at 30 s. With the default envelope of 7 s both startup checks pass: 7 s + 2 s < 40 s, and 10 s + 14 s + 2 s = 26 s < 40 s. The envelope ceiling is now 14 s, so a disk whose `connect_timeout_ms` is `0` or 4500 ms or more opens writable only with a raised lease; the docs say so. The upgrade notes keep the pin to `30000` while a pool has a build without the declared-lease rule: such a server reclaims its own slot after 36.5 s, inside a 40 s lease, and fences inside it with a renewal period below 8.5 s. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The pool is server-wide and shared by every concurrent insert and merge that commits a CAS part. On a wide insert of 62,000 blobs 16 threads were saturated and held the insert for most of its wall time. The generic IO pool (`max_io_thread_pool_size`) has 100 threads. Only the default changes. Nothing else is sized from the pool: its queue is its thread count, the caller only submits and joins, and `clickhouse-disks` mirrors the default. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The `cas_tiered` policy had no `move_factor`, so the default 0.1 applied to its local `hot` volume. On a host with less than 10% free disk the volume counts as full, and the background mover relocated a freshly fetched part to the CAS disk within a second, racing the tests that read `system.parts.disk_name`. Set `move_factor` to 0 so placement no longer depends on the host's free space. TTL moves are unaffected. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
On the lease: the default is 40 s now (feb86da), not 120 s. A killed server waits Both inline comments are fixed in this push. Also new: |
Scope of the wave on top of the late-request audit bundle: the port of PR 2570 with the cold-admission guard (CAS-428), the drain pause order (CAS-463), the 120 s mount lease and the configurable request window (CAS-405.3, CAS-405.6), unbounded GC round budgets (CAS-29.11), one request per list page (CAS-29.13), four observability fixes and five CI fixes. Each story names its tests in failing-first order. PR: Altinity#2831 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
|
CI on Caused by this PR
Follow-up PR: size the upload buffers to the blob ( Not caused by this PR
What this run shows
|
|
No AI audit defects. No fails in CI are PR-caused. CAS checks: https://github.com/Altinity/clickhouse-regression/actions/runs/44138864219
LGTM |
Altinity#2831 merged as 0aa837e. - Done (19): CAS-1.1, 69, 148, 176.3, 354, 364, 373, 400, 428, 434, 443, 448, 449, 455, 466, 480, 481, 489, 494. - Done as obsolete (5): CAS-176.2, 167, 366, 451, 459; drafts 26 and 31 archived. The one-request append writes no `_ckpt` per append. - Partly delivered (24): notes name the criteria met and the ones open. - Related tasks: the default mount lease is 40 s, an append is one `_log` create; CAS-405.3 and CAS-475 retitled. - Adds tasks CAS-468 to CAS-496, created earlier and not committed. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
CAS (content-addressed storage) fixes for
antalya-26.8, in three parts. Review commit by commit.Changes
_ckptwrite and no catalog read. Publishing a part no longer reads its own manifest back.Removing, it closes the table's write lane for good, including the read that confirms a recovery. The error says to retry later and points to the troubleshooting page.INSERTOPTIMIZE FINALDROP TABLE SYNCINSERTafter a restartLocal RustFS stand, medians of 25.
CREATE TABLEandSELECTdo not change. TheDROP TABLE SYNCtime is noisy (15 to 180 ms).utils/cas-yardstick, CAS policy, per operation, before and after:GETPUTcas_mount_lease_ttl_mscas_request_window_ms(new)cas_gc_round_*_budget(five keys)cas_blob_upload_pool_size(server setting)Risks / notes
clickhouse-disks cas-drop-memberdoes the same. Older builds then refuse to mount the pool. Stop all older servers of a pool first. Steps:docs/en/antalya/cas/operations/migration.md.v26.6.8.20003, or 26.8 before Antalya 26.8: port the CAS PRs merged into antalya-26.6 after the PR 2540 cut #2805) still waits by its own 30 s and can take a slot from a live 40 s holder. Keepcas_mount_lease_ttl_ms = 30000while such a server shares the pool. Steps:docs/en/antalya/cas/configuration.md.connect_timeout_msof0, or of 4500 ms or more, opens writable only with a raisedcas_mount_lease_ttl_ms.Removingwhile it still takes inserts. Without the catalog read, the inserts continue until a restart, a remount, a retriedDROPor the GC leader's next round; then writes are refused. Nothing is deleted. What finishes the removal depends on the statement that started it:docs/en/antalya/cas/operations/troubleshooting.md.LIMIT_EXCEEDEDthat names the line and its size. A bad or duplicate entry path isLOGICAL_ERROR: only a bug in the server can build one.Error. It failed02841_not_ready_set_bugonce in CI.18be198c7ab(the one-request append),4a06eabac6c(when an epoch gets its closing record),c821aece830(the closed lane for aRemovingrow),d0e2d1a80bd(the append without the catalog read),d383d7d79c4(which errors the manifest encoder raises) and20b02254350(the bounded replay of the sweep).Testing
On
9fbdefb5711; the commit after it changes one integration test config:CAS*unit tests: 3169 pass (release), 3173 pass (ASan + UBSan). The Fast test configuration compiles.*_cas_*tests pass.test_cas_*suites pass.test_cas_ref_move_compatruns a real26.6.4server against a generation 3 pool.utils/ca-soak, 2 hours, 29 injected faults (kills, restarts, pauses, freezes of both replicas and of the object store), 22 checkpoints: row counts equal the model,cas-fsckreports nothing. Run on1810fb2fef3, before the last six commits; not repeated.2e2bc6721ac: https://github.com/Altinity/ClickHouse/actions/runs/43913835330. None of its 36 red jobs came from this PR. The hard-restart regression jobs were red because of the 120 s lease; with 40 s they should fit their 60 s limit. Skip the CAS mount lease wait in the suites' CAS config clickhouse-regression#184 removes the wait from those suites.test_cas_replicated_relinkfailed 4 of 12 local runs on a host with less than 10 % free disk: the background mover relocated the fetched part. Its tiered policy now setsmove_factorto 0.cas s3 storageshards (job timeout), aarch64 aggregate-function suites, ARM integration shards (helper image platform),test_storage_iceberg_with_sparkunder MSan,test_database_glue.Related: #2570
Related: #2805
Related: #2542
Related: #2800
Related: #2794
Related: Altinity/clickhouse-regression#184
Related: ClickHouse#131967
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fixes several cases where a slow or retried request to a CAS (content-addressed storage) disk could bring back a dropped table's data, lock out a healthy server, or leave garbage behind. Writes to a CAS disk are faster: an
INSERTnow needs 14 object-store requests instead of 25. The mount leasecas_mount_lease_ttl_msdefaults to 40 s, the newcas_request_window_msto 120 s andcas_blob_upload_pool_sizeto 64, and GC rounds are no longer cut short by default. Upgrade with care: once a server with this version writes to a pool, older versions can no longer open it, so stop all older servers of the pool first.Documentation entry for user-facing changes
CI/CD Options
Exclude tests:
Regression jobs to run: