8000 CAS orphan sweep scheduling by k-morozov · Pull Request #2800 · Altinity/ClickHouse · GitHub
[go: up one dir, main page]

Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 41 additions & 7 deletions docs/en/antalya/cas/architecture/garbage-collection.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,18 +220,30 @@ phase 5 and builds a new in-degree snapshot; `defer` skips generation constructi
graduation check refuses to defer and the plan read surfaces the bad state
- **Fails the round if:** the plan-building seal read reports an invalid adopted seal
- **Observability:** phase row `defer_decision`; metrics `changed_shards` (despite the name, the
number of changed namespace-life rows), `namespaces_seen`, `ref_log_keys_listed`
number of changed namespace-life rows), `namespaces_seen`, `ref_log_keys_listed`,
`run_orphan_manifest_sweep`, `force_orphan_manifest_sweep`, `orphan_manifest_sweep_active`, and
`orphan_manifest_sweep_waiting_for_clock`

The plan has one row per admitted catalog life (`Live` or `Removing`; `Creating` excluded), joining
its last folded position, its greatest listed `_log` position, and the keys later phases need. A row
is *changed* when the listed `_log` is newer than the last folded position. Phase 4 chooses `fold`
if any of: changed rows ≥ `gc_fold_threshold` (default 1); an adopted shard has a published pending
delete; an adopted shard has a condemned blob due to graduate
(`oldest_nonpending_condemn_round < round + 1`); or `gc_fold_max_defer_rounds` (default 8)
(`oldest_nonpending_condemn_round < round + 1`); `force_orphan_manifest_sweep` is true; or
`gc_fold_max_defer_rounds` (default 8)
consecutive defers were reached. Both thresholds are internal `PoolConfig` fields, not disk
settings. On `defer`, one suppressed namespace-janitor page runs (phase 16's work) and the round
returns without a commit. On `fold`, phase 6 reuses this plan and the same `LIST`.

`run_orphan_manifest_sweep` requests a new or continued traversal according to
`cas_orphan_manifest_sweep_interval_sec`. Its first attempt forces a fold even on an idle pool.
After a suppressed attempt is committed, a leader-local marker names the newly adopted generation.
While that generation remains current, `force_orphan_manifest_sweep` is false and ordinary bounded
defer applies. A fold started by any other trigger still plans the requested sweep when its safety
gates permit it. A committed non-suppressed fold clears the marker, so successful active pages keep
forcing the next fold. A new generation, rebuild or fresh `Gc` instance does not inherit the old
generation's retry delay. No retry timestamp or marker is added to `gc/state`.

## Phase 5 — parent seal read {#phase-5-parent-seal-read}

Copies the *previously* adopted fold seal's run references into memory before the new fold can
Expand Down Expand Up @@ -582,7 +594,9 @@ from the catalog.
## Phase 18 — orphan sweep {#phase-18-orphan-sweep}

The last phase: executes the [orphan-manifest sweep](/antalya/cas/architecture/manifests-and-refs#orphan-sweep)
planned in phase 9 and adopted by phase 13's `CAS`.
planned in phase 9 and adopted by phase 13's `CAS`. Planning runs only when the schedule requests
it and the destructive gate permits it. The default minimum wait between completed traversals is
one hour; successful pages of an active traversal continue without that wait.

- **Runs on:** post-`CAS` (fold path)
- **Reads / writes:** one `HEAD` per nomination, then a `DELETE` conditional on the observed etag
Expand All @@ -594,7 +608,8 @@ planned in phase 9 and adopted by phase 13's `CAS`.
closing seal is consumed and no tail record above the cursor names it; any uncertainty retains.
- **Fails the round if:** a `TokenMismatch` — an immutable manifest identity must never change token
(illegal ABA); stricter than every other post-`CAS` delete (`CORRUPTED_DATA`)
- **Observability:** phase row `orphan_sweep`; metrics `listed`, `floor_lookups` / `floor_reads`
- **Observability:** phase row `orphan_sweep`; metrics `orphan_manifest_sweep_page_listed`,
`orphan_manifest_sweep_completed`, `listed`, `floor_lookups` / `floor_reads`
(mount-floor lookups per namespace and the reads they cost), `deleted`, `skipped`,
`undecodable`, `cursor_advanced`, `suppressed`, and the retained share of `skipped` by reason:
`retained_no_coverage`, `retained_hold`, `retained_unconsumed_seal`, `retained_tail_removal`
Expand All @@ -604,6 +619,10 @@ planned in phase 9 and adopted by phase 13's `CAS`.
Under `suppress_destructive` phase 9 planned nothing, so the nomination list is empty and the cursor
does not move.

`orphan_manifest_sweep_page_listed` reports a successful page `LIST`, including an empty page;
`orphan_manifest_sweep_completed` reports that the committed traversal reached the end of the key
space. Physical body deletion follows adoption and is counted separately in `deleted`.

## GC anomalies {#gc-anomalies}

A GC round records *anomalies* and per-namespace *holds* instead of failing, unless a fail-closed
Expand Down Expand Up @@ -645,12 +664,26 @@ checkpoint states are counted only in the suppression log line's frontier-defici
## The one-pass commit {#gc-state}

`<pool_prefix>/gc/state` is the durable safety and round-adoption state: `round`, `gc_shards`,
`snap_generation`, `snap_pruned_through`, `snap_attempt`, `manifest_sweep_cursor`, and the lease. A
`snap_generation`, `snap_pruned_through`, `snap_attempt`, `manifest_sweep_cursor`,
`orphan_manifest_sweep_active`, `orphan_manifest_sweep_started_at_ms`,
`orphan_manifest_sweep_completed_at_ms`, and the lease. A
folding round publishes it with exactly one commit `CAS` in phase 13, `round_commit`; the fold
itself performs no `CAS` of its own, and within a round execution phase 1's lease `CAS` over the
same object is the only other writer. Outside the round, `SYSTEM CAS GC REBUILD` replaces the
baseline with a `CAS` of its own.

The orphan-sweep fields control pacing, not deletion authority. The start timestamp records the
first page's accepted progress; the completion timestamp records the accepted end of traversal.
They are sampled near the round commit in UTC milliseconds, with `0` meaning unknown or never
recorded. Legacy state without the three schedule fields resumes a nonempty cursor or permits an
immediate first traversal for an empty cursor. A partial schedule group is corruption. Renewal and
takeover preserve these fields; rebuild resets them and the cursor.

If a completed traversal has a timestamp ahead of the current leader's clock, another traversal
waits and `orphan_manifest_sweep_waiting_for_clock` is set. This observation can come from clock
skew between leaders or a local clock change. It does not stop an active traversal; interval `0`
also permits a new traversal without waiting for that timestamp.

**The fold seal *is* the coverage record**: generation, parent generation, one `ref_lives` row per
catalog-admitted opaque life (coverage plus optional cleanup evidence), references to the
source-edge run segments, and a per-shard condemned summary. It is encoded deterministically, so a
Expand Down Expand Up @@ -798,8 +831,9 @@ the chunk size reject `0`:
| `cas_gc_round_prefix_wholesale_budget` | 20000 | listed objects the retention prune may process per round, gone ones included (phase 13) |
| `cas_gc_round_handoff_prefix_wholesale_budget` | 5000 | listed objects the hand-off reclaim may process per round, reserved separately (phase 14) |
| `cas_gc_round_ref_cleanup_budget` | 5000 | covered `_log` / `_snap` deletes per round (phase 17) |
| `cas_manifest_sweep_list_budget_keys` | 1000 | orphan-manifest sweep `LIST` budget in keys per round; `0` disables the sweep (phase 9) |
| `cas_manifest_sweep_delete_budget_keys` | 100 | orphan-manifest sweep `DELETE` budget per round (phases 9, 18) |
| `cas_orphan_manifest_sweep_interval_sec` | 3600 | minimum wait after a completed traversal before starting another; `0` removes the wait |
| `cas_manifest_sweep_list_budget_keys` | 1000 | orphan-manifest sweep `LIST` budget in keys per permitted round; `0` pauses progress (phase 9) |
| `cas_manifest_sweep_delete_budget_keys` | 100 | orphan-manifest sweep candidate budget per page; `0` lists without nominations (phases 9, 18) |
| `cas_gc_round_sweep_namespace_budget` | 20 | namespaces whose protection view the sweep may build per page (phase 9) |
| `cas_gc_round_sweep_recovery_op_budget` | 5000 | committed-tail ref-log reads the sweep's recovery walk may spend (phase 9) |
| `cas_gc_bulk_delete_chunk_keys` | 1000 | keys per batch `DELETE` request for write-once families (phases 15, 17); `1` to `1000` |
Expand Down
27 changes: 23 additions & 4 deletions docs/en/antalya/cas/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,13 +214,14 @@ removes that churn, with no measured cost.

## Advanced GC pacing settings {#advanced-gc-pacing-settings}

These settings bound individual phases of a `GC` round. The first two accept any `UInt64` value;
for the remaining caps, `0` means unbounded.
These settings control the schedule and work limits of a `GC` round. For per-round caps, `0` means
unbounded unless the table states otherwise.

| Setting | Default | Bounds | Description |
|---|---|---|---|
| `cas_manifest_sweep_list_budget_keys` | `1000` | `UInt64` | Orphan-manifest sweep `LIST` budget per round |
| `cas_manifest_sweep_delete_budget_keys` | `100` | `UInt64` | Orphan-manifest sweep `DELETE` budget per round |
| `cas_orphan_manifest_sweep_interval_sec` | `3600` | `UInt64` | Minimum wait after a completed orphan-manifest traversal before starting another; `0` means no wait |
| `cas_manifest_sweep_list_budget_keys` | `1000` | `UInt64` | Orphan-manifest sweep `LIST` budget per permitted round; `0` pauses the traversal without losing its cursor |
| `cas_manifest_sweep_delete_budget_keys` | `100` | `UInt64` | Orphan-manifest sweep candidate budget per page; `0` lists without nominating deletions |
| `cas_gc_bulk_delete_chunk_keys` | `1000` | `1`–`1000` | Most keys per batch delete request in GC's write-once families (owner-removed manifest bodies, covered ref logs and snapshots, the namespace janitor's dead ref logs and snapshots), capped by the storage's batch-delete limit |
| `cas_gc_round_graduation_budget` | `5000` | `0` = unbounded | Blob-graduation (`condemned` → `delete_pending`) cohort cap per round |
| `cas_gc_round_redelete_budget` | `5000` | `0` = unbounded | Exact-token re-delete cohort cap for prior `delete_pending` rows per round |
Expand All @@ -231,6 +232,24 @@ for the remaining caps, `0` means unbounded.
| `cas_gc_round_handoff_prefix_wholesale_budget` | `5000` | `0` = unbounded | Post-`CAS` hand-off generation-prefix reclaim cap per round, reserved separately so pruning cannot starve the one-shot hand-off |
| `cas_gc_round_outcome_entry_budget` | `5000` | `0` = unbounded | `GcOutcomes` entry cap across the re-delete/spared audit log per round |

### Orphan-manifest sweep schedule {#orphan-manifest-sweep-schedule}

The first eligible traversal starts without waiting. Once started, it continues in bounded pages
without waiting for `cas_orphan_manifest_sweep_interval_sec` between successful pages. The interval
starts after the traversal's completion is committed. Completion means reaching the end of the key
space, not that every later manifest-body deletion succeeded. Repeated `SYSTEM CAS GC RUN` commands
respect the same schedule. Setting the interval to `0` allows another traversal on the next eligible
round; setting the `LIST` budget to `0` pauses progress.

When destructive work is suppressed, the traversal's cursor and timestamps remain unchanged.
After a suppressed fold is committed, idle retries on that same generation use the existing bounded
fold cadence: up to eight deferred invocations by default, then another fold. Ref-log changes and
due blob graduation can force a fold sooner. A new leader may make an immediate attempt. These retry
controls are internal `PoolConfig` fields, not additional disk settings.

Normal blob reclamation and cleanup of manifests whose owners were removed keep their own GC rules;
the hourly interval applies to the orphan-manifest traversal.

## Migration from unprefixed keys {#migration-from-unprefixed-keys}

The unprefixed spelling of a `CAS` setting is accepted for now and reported at server startup. It
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ namespace ContentAddressedSetting
extern const ContentAddressedSettingsUInt64 gc_shards;
extern const ContentAddressedSettingsUInt64 manifest_sweep_list_budget_keys;
extern const ContentAddressedSettingsUInt64 manifest_sweep_delete_budget_keys;
extern const ContentAddressedSettingsUInt64 orphan_manifest_sweep_interval_sec;
extern const ContentAddressedSettingsUInt64 gc_round_graduation_budget;
extern const ContentAddressedSettingsUInt64 gc_round_redelete_budget;
extern const ContentAddressedSettingsUInt64 gc_round_sweep_namespace_budget;
Expand Down Expand Up @@ -294,6 +295,7 @@ ContentAddressedMetadataStorage::ContentAddressedMetadataStorage(
, gc_shards(settings_[ContentAddressedSetting::gc_shards].value)
, manifest_sweep_list_budget_keys(settings_[ContentAddressedSetting::manifest_sweep_list_budget_keys].value)
, manifest_sweep_delete_budget_keys(settings_[ContentAddressedSetting::manifest_sweep_delete_budget_keys].value)
, orphan_manifest_sweep_interval_sec(settings_[ContentAddressedSetting::orphan_manifest_sweep_interval_sec].value)
, gc_round_graduation_budget(settings_[ContentAddressedSetting::gc_round_graduation_budget].value)
, gc_round_redelete_budget(settings_[ContentAddressedSetting::gc_round_redelete_budget].value)
, gc_round_sweep_namespace_budget(settings_[ContentAddressedSetting::gc_round_sweep_namespace_budget].value)
Expand Down Expand Up @@ -794,6 +796,7 @@ ContentAddressedMetadataStorage::PoolView ContentAddressedMetadataStorage::openP
pool_config.gc_shards = gc_shards;
pool_config.manifest_sweep_list_budget_keys = manifest_sweep_list_budget_keys;
pool_config.manifest_sweep_delete_budget_keys = manifest_sweep_delete_budget_keys;
pool_config.orphan_manifest_sweep_interval_sec = orphan_manifest_sweep_interval_sec;
pool_config.gc_round_graduation_budget = gc_round_graduation_budget;
pool_config.gc_round_redelete_budget = gc_round_redelete_budget;
pool_config.gc_round_sweep_namespace_budget = gc_round_sweep_namespace_budget;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -640,6 +640,7 @@ class ContentAddressedMetadataStorage final : public IMetadataStorage, public IC
const uint64_t gc_shards; /// Blob-hash-prefix reducer shard count, fixed at pool creation.
const uint64_t manifest_sweep_list_budget_keys;
const uint64_t manifest_sweep_delete_budget_keys;
const uint64_t orphan_manifest_sweep_interval_sec;
const uint64_t gc_round_graduation_budget;
const uint64_t gc_round_redelete_budget;
const uint64_t gc_round_sweep_namespace_budget;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ constexpr std::string_view CAS_KEY_PREFIX = "cas_";
DECLARE(UInt64, gc_snapshot_generations_to_keep, 3, "GC snapshot generations retained", 0) \
DECLARE(UInt64, gc_shards, 1, "Blob-hash-prefix reducer shards (>= 1); creation-time only", 0) \
DECLARE(UInt64, manifest_sweep_list_budget_keys, 1000, "Orphan-manifest sweep LIST budget per round", 0) \
DECLARE(UInt64, orphan_manifest_sweep_interval_sec, 3600, "Seconds to wait after a completed orphan-manifest sweep before starting the next one (0 = no wait)", 0) \
DECLARE(UInt64, manifest_sweep_delete_budget_keys, 100, "Orphan-manifest sweep candidate budget per round: keys that pass every retain check, whose bodies are read and decided", 0) \
DECLARE(UInt64, gc_round_graduation_budget, 5000, "Blob graduation (condemned -> delete_pending) cohort cap per round (0 = unbounded)", 0) \
DECLARE(UInt64, gc_round_redelete_budget, 5000, "Blob redelete (exact-token delete of a prior delete_pending row) cohort cap per round (0 = unbounded)", 0) \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ namespace GcStateWire
constexpr WireKey snap_pruned_through{"snap_pruned_through"};
constexpr WireKey snap_attempt{"snap_attempt"};
constexpr WireKey manifest_sweep_cursor{"manifest_sweep_cursor"};
constexpr WireKey orphan_manifest_sweep_active{"orphan_manifest_sweep_active"};
constexpr WireKey orphan_manifest_sweep_started_at_ms{"orphan_manifest_sweep_started_at_ms"};
constexpr WireKey orphan_manifest_sweep_completed_at_ms{"orphan_manifest_sweep_completed_at_ms"};
constexpr WireKey lease_owner{"lease_owner"};
constexpr WireKey lease_seq{"lease_seq"};
}
Expand All @@ -38,6 +41,8 @@ String encodeGcState(const GcState & state)
{
if (state.gc_shards < 1)
throw Exception(ErrorCodes::LOGICAL_ERROR, "encodeGcState: gc_shards must be >= 1 -- refusing to persist an unreadable gc/state");
if (!state.orphan_manifest_sweep_active && !state.manifest_sweep_cursor.empty())
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: inactive manifest sweep has a nonempty cursor");
CasJsonWriter out(256);
writeHeaderLine(out, FormatId::GcState);
bool first = true;
Expand All @@ -47,6 +52,9 @@ String encodeGcState(const GcState & state)
writeU64StringField(out, GcStateWire::snap_pruned_through, state.snap_pruned_through, first);
writeU64StringField(out, GcStateWire::snap_attempt, state.snap_attempt, first);
writeStringField(out, GcStateWire::manifest_sweep_cursor, state.manifest_sweep_cursor, first);
writeBoolField(out, GcStateWire::orphan_manifest_sweep_active, state.orphan_manifest_sweep_active, first);
writeU64StringField(out, GcStateWire::orphan_manifest_sweep_started_at_ms, state.orphan_manifest_sweep_started_at_ms, first);
writeU64StringField(out, GcStateWire::orphan_manifest_sweep_completed_at_ms, state.orphan_manifest_sweep_completed_at_ms, first);
writeHex128Field(out, GcStateWire::lease_owner, state.lease.owner, first);
writeU64StringField(out, GcStateWire::lease_seq, state.lease.seq, first);
closeObject(out, first);
Expand All @@ -64,6 +72,7 @@ GcState decodeGcState(std::string_view data)

GcState state;
bool saw_gcs = false;
unsigned schedule_fields_mask = 0;
String key;
while (r.nextKey(key))
{
Expand All @@ -82,6 +91,21 @@ GcState decodeGcState(std::string_view data)
state.snap_attempt = r.readU64String();
else if (key == GcStateWire::manifest_sweep_cursor)
state.manifest_sweep_cursor = r.readString();
else if (key == GcStateWire::orphan_manifest_sweep_active)
{
state.orphan_manifest_sweep_active = r.readBool();
schedule_fields_mask |= 1;
}
else if (key == GcStateWire::orphan_manifest_sweep_started_at_ms)
{
state.orphan_manifest_sweep_started_at_ms = r.readU64String();
schedule_fields_mask |= 2;
}
else if (key == GcStateWire::orphan_manifest_sweep_completed_at_ms)
{
state.orphan_manifest_sweep_completed_at_ms = r.readU64String();
schedule_fields_mask |= 4;
}
else if (key == GcStateWire::lease_owner)
state.lease.owner = r.readHex128();
else if (key == GcStateWire::lease_seq)
Expand All @@ -95,6 +119,12 @@ GcState decodeGcState(std::string_view data)
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: missing gc_shards");
if (state.gc_shards == 0)
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: gc_shards must be >= 1");
if (schedule_fields_mask == 0)
state.orphan_manifest_sweep_active = !state.manifest_sweep_cursor.empty();
else if (schedule_fields_mask != 7)
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: incomplete manifest sweep schedule");
if (!state.orphan_manifest_sweep_active && !state.manifest_sweep_cursor.empty())
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: inactive manifest sweep has a nonempty cursor");
if (!body_in.eof() || !in.eof())
throw Exception(ErrorCodes::CORRUPTED_DATA, "CAS gc/state: trailing bytes");
return state;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ struct GcState
uint64_t snap_pruned_through = 0; /// highest snap generation fully pruned (retention cursor)
uint64_t snap_attempt = 0; /// adopted attempt id (folding leader's lease.seq) for snap_generation
String manifest_sweep_cursor; /// best-effort orphan part-manifest cleanup cursor; reachability ignores it
bool orphan_manifest_sweep_active = false;
uint64_t orphan_manifest_sweep_started_at_ms = 0;
uint64_t orphan_manifest_sweep_completed_at_ms = 0;
GcLease lease;
};

Expand Down
Loading
Loading
0