feat(inkless:consolidation): bound the WAL prune per request [KC-444] by viktorsomogyi · Pull Request #843 · aiven/inkless

viktorsomogyi · 2026-10-28T16:14:18Z

Summary

A consolidation prune of a deep already-tiered WAL backlog held every requested partition's logs row for an unbounded delete, in one transaction for the whole run. prune_batches_below_highest_tiered_offset_v2 caps each delete and takes the lock only when that partition has work. v1 stays, so a broker still on the old build keeps a function that resolves during a rolling upgrade.

ConsolidatedDisklessLogPruner
  read highestOffsetInRemoteStorage
  controlPlane.pruneDisklessLogs(requests, cap)
    PruneDisklessLogsJob
      one retried transaction per partition
        prune_batches_below_highest_tiered_offset_v2
          skip the logs lock when the prune is already past the tiered offset
          otherwise lock that partition
          keep-offset = first covering-index entry past the tiered offset
          delete at most the cap through delete_records_v1

 prune run
-  one transaction for every eligible partition
-  lock every logs row, then delete every tiered batch
+  one transaction per partition, retried on its own
+  lock only a partition that still has work
+  delete at most inkless.consolidation.cleanup.max.batches.per.request (default 2000)
+  interval default 1 minute

A successful run records ConsolidationPruneTotalTime, ConsolidationPruneTotalBatchesDeleted, and ConsolidationPruneTotalBytesDeleted. A thrown run records none of them. PruneDisklessLogsQueryTime and PruneDisklessLogsQueryRate record one sample per partition.

Evidence

Before: prune_batches_below_highest_tiered_offset_v1 holds the logs lock for one pass over the whole tiered backlog. The only bound is statement_timeout, which aborts the transaction, and the lock hold grows with the backlog depth.
After: v2 holds the lock for one capped pass at a time and releases it between passes, so the longest hold is bounded by the cap, not by backlog depth.

A benchmark clears the same 200,000-batch tiered backlog (plus a 200,000-batch untiered tail) two ways: once under v1 in a single unbounded pass, and once under v2 through 100 capped 2,000-batch passes. A background thread commits continuously through each scenario, so the commit-latency sample covers the whole run, not a couple of lucky data points.

Tiered batches deleted by each scenario: 200000; untiered tail: 200000; v2 cap: 2000 batches
over 100 passes; control-plane socket timeout: 5000 ms
                scenario |   total ms |   pass p50 ms |   pass max ms |  commits | commit p50 ms | commit max ms
           v1 (one pass) |    214.754 |       214.754 |       214.754 |        2 |         2.826 |       191.921
         v2 (100 passes) |    532.275 |         4.244 |        42.736 |      101 |         4.224 |        41.804

Both scenarios' worst-case commit latency sits inside the 5-second control-plane socket timeout on this fixture. Postgres clears an indexed range fast either way, so total prune time doesn't tell the real story here. The longest lock hold does: it sets the worst case a produce commit can stall behind. v2's 100 passes take longer in total than v1's one pass from per-transaction overhead, yet across four local runs v1's lo 8000 ck hold ran 4.3x-5.0x as long as v2's longest pass, for the same total deleted volume. A deeper backlog grows v1's lock hold further; v2's worst case stays capped regardless of backlog depth.

PruneBatchesBelowHighestTieredOffsetV1Test calls v1 directly. PruneBatchesBelowHighestTieredOffsetV2Test, PruneDisklessLogsJobTest, ConsolidatedDisklessLogPrunerTest, and ConsolidatedDisklessLogPrunerMetricsTest pass. BatchesPlanRegressionTest checks the keep-offset lookup is one index entry.

Merge Danger

Door: two-way

v1 stays in the schema, so a broker rolled back to the old build still calls a function that resolves. V31 only adds v2. The new interval and cap defaults apply to brokers on this build and can be set back in config.

Blast Radius: produce

The prune lock sits on the same logs row a produce commit needs. A wrong keep-offset can delete WAL batches that are not fully tiered, or leave a switched topic's log start behind the seal.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

…444] Set `consolidation.cleanup.max.batches.per.request` to 2000 and `consolidation.cleanup.interval.ms` to 1 minute. These defaults match retention's per-minute 2000-batch rate, so consolidation sustains about 33 batches per second while each locked delete stays within retention's batch count. A value of 0 remains uncapped, and the config reference records the roughly eight-hour drain time for a 1,000,000-batch backlog. Testing - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.config.InklessConfigTest` - `PATH="$HOME/Library/Python/3.13/bin:$PATH" make vale` Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7

Keep the existing v1 function available during rolling upgrades while v2 bounds each prune delete and reports deleted counts. The new function locks only partitions with work, preserves the cross-tier start bootstrap, and advances empty switched logs to the seal. The keep-offset lookup reads one covering-index entry, so the `logs` row lock covers only that read. The log start sitting at or before the tiered offset doesn't by itself mean a batch is deletable: the oldest remaining batch can still straddle the tiered offset when it's a multi-record or coalesced batch only partially tiered. v2 probes the keep-offset unlocked before falling back to the lock, mirroring retention enforcement's unlocked boundary scan, and probes it again right after acquiring the lock. Either probe finding nothing to delete returns early, so a prune with nothing to delete never takes the `logs` lock or calls `delete_records_v1` purely to re-set `log_start_offset` to its current value. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:compileTestJava --no-watch-fs --rerun-tasks - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV1Test - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.config.InklessConfigTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.BatchesPlanRegressionTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --max-workers=1 Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7

The control plane passes the per-request cap to the v2 prune function and returns deletion counts. The job isolates each partition in its own retried transaction so a retry doesn't repeat committed partitions. The in-memory control plane mirrors the cap, handoff, and no-op behavior. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --rerun-tasks --tests io.aiven.inkless.control_plane.InMemoryControlPlaneTest --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneTest --tests io.aiven.inkless.control_plane.InMemoryControlPlanePruneDisklessLogsTest --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV1Test --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test --tests io.aiven.inkless.control_plane.postgres.LogsEarliestBatchTimestampTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --rerun-tasks --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest - make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7

Bound each consolidation prune request with the broker cap and report successful-run deletion totals and duration. This keeps prune work bounded and makes partial runs invisible to completion metrics. The generated metrics reference now lists the per-partition `PruneDisklessLogs` query metrics that the docs describe. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.delete.ConsolidatedDisklessLogPrunerMetricsTest --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneMetricsTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:genInklessMetricsDoc - make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7

The bounded prune must skip a lock for completed partitions while retaining the lock for bootstrap and delete work. Postgres tests hold `logs` rows from a second connection, observe `pg_blocking_pids`, verify per-partition lock release, and exercise a transient `40P01` retry. They also check one duration sample per partition, a log soft-deleted while the prune waits for its lock, a straddling oldest batch that leaves the log start at or before the tiered offset with nothing actually deletable, and a later partition failure that leaves earlier partitions committed. The benchmark seeds an identical 200,000-tiered-batch backlog on two partitions, then compares the longest lock hold of the old unbounded `v1` call against the longest single hold across 100 capped `v2` passes that clear the same total batch count. A background thread commits continuously through each scenario's whole measurement window, so the reported commit-latency sample covers the run, not a couple of commits that happened to land inside one short pass. Testing: - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsJobTest --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test --tests io.aiven.inkless.control_plane.postgres.EnforceRetentionJobTest --no-daemon` passes. - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:benchmarkTest --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsCommitBlockingBenchmarkTest --no-daemon` passes. Across four local runs, v1's single lock hold ran 196ms, 164ms, 179ms, and 179ms; v2's longest single pass among 100 ran 41ms, 38ms, 38ms, and 40ms, a 4.3x-4.8x gap. - `make vale` passes. Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7

A completed no-op prune needs zero-valued observability, while a partial run must not look successful. Document and test the batch-boundary keep offset, per-attempt cap, empty switched-log handoff, and partition-scoped deadlock retry. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest --no-daemon --rerun-tasks - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsJobTest --no-daemon - PATH="$HOME/Library/Python/3.13/bin:$PATH" make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7

Vale's reStructuredText parser calls rst2html before it applies any style rule. The runner installs Vale and its packages, and it has no docutils, so a pull request that changes an owned rst file fails the lint job with E100 before any alert is reported. Assisted-by: Grok 4.7

jeqo

LGMT. Just added a comment on a missing error metric, but should be good after that.

jeqo · 2026-10-30T11:22:12Z

        }
      }
    }
+    metrics.recordConsolidationPruneFinishedSuccessfully(


Could we add a an error rate metric as well?

Ok, added it.

The consolidated diskless log pruner updates its prune totals only after a run completes. A run that throws left no trace in the metrics, so operators could not see that pruning had stopped working. A new error rate counts each run that fails with an exception. A tick that the control plane reports as unavailable does not count. The availability gate already skips that tick, so it is not a prune failure. Testing ConsolidatedDisklessLogPrunerMetricsTest covers the new counter and its removal on close. ConsolidatedDisklessLogPrunerTest checks that a failed run increments the counter and that a control-plane-unavailable run does not. Assisted-by: Auto

…#843) * feat(inkless:config): set a 2000-batch cap and 1-minute interval [KC-444] Set `consolidation.cleanup.max.batches.per.request` to 2000 and `consolidation.cleanup.interval.ms` to 1 minute. These defaults match retention's per-minute 2000-batch rate, so consolidation sustains about 33 batches per second while each locked delete stays within retention's batch count. A value of 0 remains uncapped, and the config reference records the roughly eight-hour drain time for a 1,000,000-batch backlog. Testing - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew : 8000 storage:inkless:test --tests io.aiven.inkless.config.InklessConfigTest` - `PATH="$HOME/Library/Python/3.13/bin:$PATH" make vale` Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * feat(inkless:storage): add bounded prune v2 function [KC-444] Keep the existing v1 function available during rolling upgrades while v2 bounds each prune delete and reports deleted counts. The new function locks only partitions with work, preserves the cross-tier start bootstrap, and advances empty switched logs to the seal. The keep-offset lookup reads one covering-index entry, so the `logs` row lock covers only that read. The log start sitting at or before the tiered offset doesn't by itself mean a batch is deletable: the oldest remaining batch can still straddle the tiered offset when it's a multi-record or coalesced batch only partially tiered. v2 probes the keep-offset unlocked before falling back to the lock, mirroring retention enforcement's unlocked boundary scan, and probes it again right after acquiring the lock. Either probe finding nothing to delete returns early, so a prune with nothing to delete never takes the `logs` lock or calls `delete_records_v1` purely to re-set `log_start_offset` to its current value. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:compileTestJava --no-watch-fs --rerun-tasks - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV1Test - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.config.InklessConfigTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.BatchesPlanRegressionTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --max-workers=1 Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * feat(inkless:storage): wire bounded prune through control plane [KC-444] The control plane passes the per-request cap to the v2 prune function and returns deletion counts. The job isolates each partition in its own retried transaction so a retry doesn't repeat committed partitions. The in-memory control plane mirrors the cap, handoff, and no-op behavior. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --rerun-tasks --tests io.aiven.inkless.control_plane.InMemoryControlPlaneTest --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneTest --tests io.aiven.inkless.control_plane.InMemoryControlPlanePruneDisklessLogsTest --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV1Test --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test --tests io.aiven.inkless.control_plane.postgres.LogsEarliestBatchTimestampTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --rerun-tasks --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest - make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * feat(inkless:storage): wire consolidation prune cap and metrics [KC-444] Bound each consolidation prune request with the broker cap and report successful-run deletion totals and duration. This keeps prune work bounded and makes partial runs invisible to completion metrics. The generated metrics reference now lists the per-partition `PruneDisklessLogs` query metrics that the docs describe. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.delete.ConsolidatedDisklessLogPrunerMetricsTest --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneMetricsTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:genInklessMetricsDoc - make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * test(inkless:storage): verify prune lock behavior [KC-444] The bounded prune must skip a lock for completed partitions while retaining the lock for bootstrap and delete work. Postgres tests hold `logs` rows from a second connection, observe `pg_blocking_pids`, verify per-partition lock release, and exercise a transient `40P01` retry. They also check one duration sample per partition, a log soft-deleted while the prune waits for its lock, a straddling oldest batch that leaves the log start at or before the tiered offset with nothing actually deletable, and a later partition failure that leaves earlier partitions committed. The benchmark seeds an identical 200,000-tiered-batch backlog on two partitions, then compares the longest lock hold of the old unbounded `v1` call against the longest single hold across 100 capped `v2` passes that clear the same total batch count. A background thread commits continuously through each scenario's whole measurement window, so the reported commit-latency sample covers the run, not a couple of commits that happened to land inside one short pass. Testing: - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsJobTest --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test --tests io.aiven.inkless.control_plane.postgres.EnforceRetentionJobTest --no-daemon` passes. - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:benchmarkTest --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsCommitBlockingBenchmarkTest --no-daemon` passes. Across four local runs, v1's single lock hold ran 196ms, 164ms, 179ms, and 179ms; v2's longest single pass among 100 ran 41ms, 38ms, 38ms, and 40ms, a 4.3x-4.8x gap. - `make vale` passes. Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * fix(inkless:consolidation): finalize prune bounds behavior [KC-444] A completed no-op prune needs zero-valued observability, while a partial run must not look successful. Document and test the batch-boundary keep offset, per-attempt cap, empty switched-log handoff, and partition-scoped deadlock retry. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest --no-daemon --rerun-tasks - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsJobTest --no-daemon - PATH="$HOME/Library/Python/3.13/bin:$PATH" make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * ci(inkless): install rst2html for the Vale runner Vale's reStructuredText parser calls rst2html before it applies any style rule. The runner installs Vale and its packages, and it has no docutils, so a pull request that changes an owned rst file fails the lint job with E100 before any alert is reported. Assisted-by: Grok 4.7 * feat(inkless:consolidation): add consolidation prune error rate [KC-444] The consolidated diskless log pruner updates its prune totals only after a run completes. A run that throws left no trace in the metrics, so operators could not see that pruning had stopped working. A new error rate counts each run that fails with an exception. A tick that the control plane reports as unavailable does not count. The availability gate already skips that tick, so it is not a prune failure. Testing ConsolidatedDisklessLogPrunerMetricsTest covers the new counter and its removal on close. ConsolidatedDisklessLogPrunerTest checks that a failed run increments the counter and that a control-plane-unavailable run does not. Assisted-by: Auto

…#843) * feat(inkless:config): set a 2000-batch cap and 1-minute interval [KC-444] Set `consolidation.cleanup.max.batches.per.request` to 2000 and `consolidation.cleanup.interval.ms` to 1 minute. These defaults match retention's per-minute 2000-batch rate, so consolidation sustains about 33 batches per second while each locked delete stays within retention's batch count. A value of 0 remains uncapped, and the config reference records the roughly eight-hour drain time for a 1,000,000-batch backlog. Testing - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.config.InklessConfigTest` - `PATH="$HOME/Library/Python/3.13/bin:$PATH" make vale` Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * feat(inkless:storage): add bounded prune v2 function [KC-444] Keep the existing v1 function available during rolling upgrades while v2 bounds each prune delete and reports deleted counts. The new function locks only partitions with work, preserves the cross-tier start bootstrap, and advances empty switched logs to the seal. The keep-offset lookup reads one covering-index entry, so the `logs` row lock covers only that read. The log start sitting at or before the tiered offset doesn't by itself mean a batch is deletable: the oldest remaining batch can still straddle the tiered offset when it's a multi-record or coalesced batch only partially tiered. v2 probes the keep-offset unlocked before falling back to the lock, mirroring retention enforcement's unlocked boundary scan, and probes it again right after acquiring the lock. Either probe finding nothing to delete returns early, so a prune with nothing to delete never takes the `logs` lock or calls `delete_records_v1` purely to re-set `log_start_offset` to its current value. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:compileTestJava --no-watch-fs --rerun-tasks - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV1Test - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.config.InklessConfigTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --tests io.aiven.inkless.control_plane.postgres.BatchesPlanRegressionTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --no-watch-fs --max-workers=1 Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * feat(inkless:storage): wire bounded prune through control plane [KC-444] The control plane passes the per-request cap to the v2 prune function and returns deletion counts. The job isolates each partition in its own retried transaction so a retry doesn't repeat committed partitions. The in-memory control plane mirrors the cap, handoff, and no-op behavior. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --rerun-tasks --tests io.aiven.inkless.control_plane.InMemoryControlPlaneTest --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneTest --tests io.aiven.inkless.control_plane.InMemoryControlPlanePruneDisklessLogsTest --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV1Test --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test --tests io.aiven.inkless.control_plane.postgres.LogsEarliestBatchTimestampTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --rerun-tasks --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest - make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * feat(inkless:storage): wire consolidation prune cap and metrics [KC-444] Bound each consolidation prune request with the broker cap and report successful-run deletion totals and duration. This keeps prune work bounded and makes partial runs invisible to completion metrics. The generated metrics reference now lists the per-partition `PruneDisklessLogs` query metrics that the docs describe. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.delete.ConsolidatedDisklessLogPrunerMetricsTest --tests io.aiven.inkless.control_plane.postgres.PostgresControlPlaneMetricsTest - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:genInklessMetricsDoc - make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * test(inkless:storage): verify prune lock behavior [KC-444] The bounded prune must skip a lock for completed partitions while retaining the lock for bootstrap and delete work. Postgres tests hold `logs` rows from a second connection, observe `pg_blocking_pids`, verify per-partition lock release, and exercise a transient `40P01` retry. They also check one duration sample per partition, a log soft-deleted while the prune waits for its lock, a straddling oldest batch that leaves the log start at or before the tiered offset with nothing actually deletable, and a later partition failure that leaves earlier partitions committed. The benchmark seeds an identical 200,000-tiered-batch backlog on two partitions, then compares the longest lock hold of the old unbounded `v1` call against the longest single hold across 100 capped `v2` passes that clear the same total batch count. A background thread commits continuously through each scenario's whole measurement window, so the reported commit-latency sample covers the run, not a couple of commits that happened to land inside one short pass. Testing: - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsJobTest --tests io.aiven.inkless.control_plane.postgres.PruneBatchesBelowHighestTieredOffsetV2Test --tests io.aiven.inkless.control_plane.postgres.EnforceRetentionJobTest --no-daemon` passes. - `JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:benchmarkTest --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsCommitBlockingBenchmarkTest --no-daemon` passes. Across four local runs, v1's single lock hold ran 196ms, 164ms, 179ms, and 179ms; v2's longest single pass among 100 ran 41ms, 38ms, 38ms, and 40ms, a 4.3x-4.8x gap. - `make vale` passes. Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * fix(inkless:consolidation): finalize prune bounds behavior [KC-444] A completed no-op prune needs zero-valued observability, while a partial run must not look successful. Document and test the batch-boundary keep offset, per-attempt cap, empty switched-log handoff, and partition-scoped deadlock retry. Testing - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :core:test --tests io.aiven.inkless.consolidation.ConsolidatedDisklessLogPrunerTest --no-daemon --rerun-tasks - JAVA_HOME=/opt/homebrew/opt/openjdk@21 ./gradlew :storage:inkless:test --tests io.aiven.inkless.control_plane.postgres.PruneDisklessLogsJobTest --no-daemon - PATH="$HOME/Library/Python/3.13/bin:$PATH" make vale Assisted-by: GPT-5.6 Luna Max Assisted-by: Grok 4.7 * ci(inkless): install rst2html for the Vale runner Vale's reStructuredText parser calls rst2html before it applies any style rule. The runner installs Vale and its packages, and it has no docutils, so a pull request that changes an owned rst file fails the lint job with E100 before any alert is reported. Assisted-by: Grok 4.7 * feat(inkless:consolidation): add consolidation prune error rate [KC-444] The consolidated diskless log pruner updates its prune totals only after a run completes. A run that throws left no trace in the metrics, so operators could not see that pruning had stopped working. A new error rate counts each run that fails with an exception. A tick that the control plane reports as unavailable does not count. The availability gate already skips that tick, so it is not a prune failure. Testing ConsolidatedDisklessLogPrunerMetricsTest covers the new counter and its removal on close. ConsolidatedDisklessLogPrunerTest checks that a failed run increments the counter and that a control-plane-unavailable run does not. Assisted-by: Auto

viktorsomogyi force-pushed the svv/async-prune branch from 50c9388 to 8df1ff3 Compare October 29, 2026 09:07

viktorsomogyi requested a balanced review from Copilot October 29, 2026 10:06

Copilot AI reviewed Oct 29, 2026

viktorsomogyi force-pushed the svv/async-prune branch 3 times, most recently from d1f0874 to 30af211 Compare October 29, 2026 11:49

viktorsomogyi added 7 commits October 29, 2026 12:51

viktorsomogyi force-pushed the svv/async-prune branch from 30af211 to c19eba0 Compare October 29, 2026 11:52

viktorsomogyi marked this pull request as ready for review October 29, 2026 11:56

viktorsomogyi requested review from giuseppelillo and jeqo October 29, 2026 11:56

jeqo reviewed Oct 30, 2026

View reviewed changes

jeqo approved these changes Nov 2, 2026

View reviewed changes

jeqo merged commit 1af55a0 into main Nov 2, 2026
7 checks passed

jeqo deleted the svv/async-prune branch November 2, 2026 08:26

Navigation Menu

Repository navigation

feat(inkless:consolidation): bound the WAL prune per request [KC-444] - #843

feat(inkless:consolidation): bound the WAL prune per request [KC-444]#843
jeqo merged 8 commits into
mainfrom
svv/async-prune

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Uh oh!

Summary

Evidence

Merge Danger

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants