Repository navigation
feat(inkless:retention): bound the file-cleaner deletion worklist [KC-404] - #743
Conversation
80fbfbc to
ad49a61
Compare
ad49a61 to
f30f274
Compare
f30f274 to
90f28d9
Compare
There was a problem hiding this comment.
Pull request overview
Bounds file-cleaner work per cycle and moves grace-period filtering into control-plane queries.
Changes:
- Adds configurable per-cycle deletion limits.
- Adds saturation observability.
- Updates PostgreSQL/in-memory implementations and tests.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
FileCleaner.java |
Applies grace-period and cycle bounds. |
FileCleanerMetrics.java |
Adds saturation metric. |
ControlPlane.java |
Defines bounded deletion query contract. |
InMemoryControlPlane.java |
Implements filtering and limits. |
PostgresControlPlane.java |
Passes query bounds to the job. |
FindFilesToDeleteJob.java |
Adds SQL filtering and limits. |
InklessConfig.java |
Adds the cycle-cap configuration. |
FileCleanerMockedTest.java |
Tests bounded cleaning and metrics. |
FileCleanerIntegrationTest.java |
Updates integration assertions. |
FindFilesToDeleteJobTest.java |
Tests eligibility and limits. |
InMemoryControlPlanePruneDisklessLogsTest.java |
Updates deletion queries. |
AbstractControlPlaneTest.java |
Adds an unbounded test helper. |
InklessConfigTest.java |
Verifies the default cap. |
docs/inkless/metrics.rst |
Documents saturation metrics. |
docs/inkless/configs.rst |
Documents the new configuration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
90f28d9 to
0078bfe
Compare
FileCleaner fetched every file in state 'deleting' and filtered the grace period client-side, so the control-plane query was proportional to the whole deletion backlog. After a bulk topic delete that is unbounded: FindFilesToDeleteJob has no LIMIT and no JDBC fetch size, so the full result set materializes in broker heap while holding a connection from the jobs pool (shared with the produce commit, delete-records and retention paths) and pinning a snapshot against vacuum. Every broker runs its own cleaner, so the cost is paid per broker per cycle. The same unbounded set was then handed to the backend, which issues 1000-key DeleteObjects calls back to back with no pacing. Push both bounds into the control plane, which is the only place they can be enforced: by the time a caller-side filter runs, the query cost is already paid. - getFilesToDelete(markedBefore, limit) replaces the no-arg form. markedBefore is the grace-period cutoff (now - file.cleaner.retention.period.ms), previously applied in FileCleaner; limit caps the rows returned, 0 meaning unbounded (the max.batches.per.request convention). Both control planes enforce both bounds. - file.cleaner.max.files.per.cycle (default 20000) caps the worklist. It doubles as the delete pacing knob: the drain rate is the cap over file.cleaner.interval.ms, and pacing (not slice size) is the lever that reduces S3 request-rate throttling. The default is sized against the rate at which files are created, since in steady state files are marked for deletion as fast as they are produced: a broker rotates a file every produce.commit.interval.ms (250 ms, so 4/s) or whenever produce.buffer.max.bytes (8 MiB) fills, whichever comes first, so above ~32 MiB/s of ingest per broker the size trigger dominates and the rate is ingest/8 MiB - tens of files/s on a busy broker, not 4. Pushing the grace period down is what makes the limit safe rather than an independent cleanup: with the age filter on the broker, a truncated result could consist entirely of not-yet-eligible files while eligible ones went unreturned, and the cleaner would spin without draining. Applying it server-side makes every returned row actionable, so any truncation still makes progress. The query deliberately has no ORDER BY. Ordering by file_id would let the planner walk files_pkey ascending and filter, making the scan cost depend on how many low-file_id non-deleting rows it skips rather than on the limit; it also orders by commit time, while a bulk topic delete marks a whole topic at once, so commit order and mark order are unrelated exactly when the backlog is large. Nor is ordering needed semantically: a marked file is unreferenced, so there is no fairness claim between candidates. What the limit does not bound yet is scan work: files_by_state_only_deleting_idx covers `state` only, so the grace-period predicate is a residual filter that the LIMIT cannot stop. A later commit indexes it. Bounding a cycle changes how a permanently-undeletable key behaves. Today any error throws the whole slice, so one bad key anywhere stalls every cycle for the entire backlog; with a cap it stalls only once the window reaches it, and the files ahead of it still drain. That is an improvement, but the stall is now silent, so the following commit adds a counter for saturated cycles. The default is generous enough that a small set of poison keys cannot fill a cycle; a per-key failure count (which would need a column) is the durable fix if that proves insufficient. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
adccc6a to
31f82e8
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 137 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/FindFilesToDeleteJob.java:66
- PostgreSQL does not guarantee index order without
ORDER BY; the planner may choose a sequential scan and return a heap-order prefix. That contradicts the oldest-first behavior deliberately implemented by the in-memory plane and described by the API/migration, so capped cycles can process a different prefix in production. Order bymarked_for_deletion_atbefore applying the limit.
// No ORDER BY: the returned rows are all actionable, so any prefix makes progress. The scan is
// bounded by files_by_marked_for_deletion_deleting_idx, which covers the grace-period predicate
// and lets Postgres stop once `limit` rows are found.
storage/inkless/src/main/resources/db/migration/V25__File_cleaner_marked_for_deletion_index.sql:41
- The lock awaited by the index build is
SHARE, notACCESS EXCLUSIVE. The queueing warning still applies because it conflicts with writers, but naming the wrong mode may lead operators to expect reads to be blocked as well.
storage/inkless/src/main/resources/db/migration/V25__File_cleaner_marked_for_deletion_index.sql:18 - A regular PostgreSQL
CREATE INDEXholds aSHARElock on the table during the scan, notACCESS EXCLUSIVE. It still blocks writes, so the operational warning remains important, but the documented lock mode should be accurate (the later non-concurrentDROP INDEXhas different locking behavior).
This issue also appears on line 40 of the same file.
A cycle that stops at file.cleaner.max.files.per.cycle is indistinguishable from one that found nothing more to do, so a deletion backlog that keeps growing cannot be attributed to the cap rather than to deletes that are failing. The distinction decides the response: raise the cap, or go find out why deletes fail. It matters more now that a cycle is bounded, because a permanently-undeletable key occupies a slot in the window every pass and holds up the files behind it. Backlog depth itself is observed out-of-band from the files table; this counter is what turns a growing backlog into a diagnosis. Add FileCleanerCycleSaturatedRate, incremented when the worklist comes back over the cap. The query asks for one row beyond the cap and the extra row is dropped before deleting, so a cycle that happens to find exactly cap eligible files is not reported as saturated - "at the cap" and "more remain" are not the same state, and only the second one is actionable. The add saturates at Integer.MAX_VALUE: overflowing would hand the control planes a negative limit, which they read as unbounded, so the one setting that asks for the largest possible cap would remove it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
31f82e8 to
6b406f1
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 137 changed files in this pull request and generated no new comments.
Suppressed comments (3)
storage/inkless/src/main/resources/db/migration/V25__File_cleaner_marked_for_deletion_index.sql:43
- The lock request that can wait for a long-running writer before this build is
SHARE, notACCESS EXCLUSIVE. It still queues later writers behind it, so the timeout rationale remains valid, but the documented lock mode should match PostgreSQL'sCREATE INDEXbehavior.
storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/FindFilesToDeleteJob.java:66 - Without
ORDER BY, PostgreSQL does not guarantee index order; a sequential or bitmap plan can return a different bounded prefix as statistics change. That breaks the oldest-first behavior and the stated parity withInMemoryControlPlane. Order explicitly bymarked_for_deletion_atso the new index defines the worklist.
// No ORDER BY: the returned rows are all actionable, so any prefix makes progress. The scan is
// bounded by files_by_marked_for_deletion_deleting_idx, which covers the grace-period predicate
// and lets Postgres stop once `limit` rows are found.
storage/inkless/src/main/resources/db/migration/V25__File_cleaner_marked_for_deletion_index.sql:18
- This overstates the lock held during the full index build. PostgreSQL's non-concurrent
CREATE INDEXholds aSHARElock on the table (which still blocks inserts, updates, and deletes);ACCESS EXCLUSIVEis only needed later by the non-concurrent drop. Correcting the lock mode is important operator guidance when diagnosing blocked sessions.
This issue also appears on line 40 of the same file.
c039fc4 to
692b53c
Compare
…icate Bounding the worklist bounded the rows returned, not the work done to find them. files_by_state_only_deleting_idx covers `state` alone, so marked_for_deletion_at was a residual filter: Postgres inspected every deleting row and discarded the ones still in the grace period. LIMIT cannot stop that, and the worst case is the one the cap was added for - right after a bulk delete, when the whole backlog is inside the grace period, each broker scans all of it to return zero rows. Replace it with a partial index on marked_for_deletion_at over deleting rows. The range predicate is now an index scan that ends at the first non-matching entry, and a limited cycle stops as soon as it has its rows. The old index is dropped rather than kept: the new one serves `state = 'deleting'` equally well, and the only other query with that predicate, delete_files_v1, is keyed by object_key. Building it scans all of `files` to evaluate the predicate and holds ACCESS EXCLUSIVE while it does, since Flyway wraps migrations in a transaction and CONCURRENTLY is not available inside one. The index itself is small - deleting rows only - but the lock is held for a scan of the whole table, and `files` is on the produce commit path. Both statements are therefore idempotent: an operator can run the CONCURRENTLY equivalents before upgrading and the migration turns into a no-op. The DDL to run is in the migration header, together with the INVALID-index caveat, since CREATE INDEX IF NOT EXISTS matches on name alone and would accept a failed concurrent build. The migration stays non-concurrent rather than switching to executeInTransaction=false. As written, a failure rolls back index and history row together and the next broker start retries from scratch; a failed concurrent build would instead leave an invalid index - unusable by queries, still maintained on every write - plus a failed history row that wedges every subsequent start until someone runs flyway repair. What it does add is lock_timeout: an ACCESS EXCLUSIVE request that queues behind a long transaction blocks every commit arriving after it, so the stall would be bounded by that transaction rather than by the build. Timing out instead rolls back into the clean retry path. The ordering is a behaviour change, not just a plan change. The worklist is now the oldest eligible files instead of an arbitrary prefix, so keys the object store never confirms - which stay marked, and stay oldest - are re-served first every cycle and would crowd out later files once there were as many of them as the cap. Nothing gets there: deleting a missing object succeeds, throttling is transient and fails a different subset each cycle, and the refusals that do persist (bucket policy, Object Lock) apply to the bucket, so no file is deletable and which ones are retried first stops mattering. Recorded on ControlPlane#getFilesToDelete so the next caller does not have to derive it, with FileCleanerFilesFailedRate as the check if the assumption ever looks wrong. InMemoryControlPlane sorts the same way. It is a test double, and leaving it on hash order would hide exactly the truncation prefix the Postgres plane now guarantees. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… generator The changelog draws config changes from configs.rst and metrics from metrics.rst, so a migration reached the changelog only if a commit subject happened to mention it. That is the one change class an operator cannot react to after the fact: a migration that takes a table-level lock blocks the produce path during the upgrade, and V25 is exactly that case, avoidable only by running the DDL concurrently beforehand. Diff the filenames under db/migration between the two tags. Flyway files are append-only, so a set difference is exact - no parsing of SQL semantics, which the generator has no business interpreting. Annotate each entry with the blocking DDL it contains (CREATE INDEX, ALTER TABLE, ...) and whether it runs CONCURRENTLY, so the reader knows to open the migration header. Comments are stripped before scanning: headers quote the concurrent form as operator guidance, which would otherwise report a lock-taking migration as concurrent. The section is titled "Postgres schema changes" rather than "schema": in a Kafka fork "schema" reads as record schemas or the metadata log, and the reader needs to know this is the control-plane database. It appears in the summary output too. Unlike metric deltas, which are noisy because metrics.rst tracks a hand-maintained registry list, this source is exact and needs no curation step beyond reading the migration header. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
692b53c to
6e8d2ed
Compare
There was a problem hiding this comment.
had 2 minor comments, otherwise LGTM
…od predicate perf(inkless:control_plane): index the file-cleaner grace-period predicate Bounding the worklist bounded the rows returned, not the work done to find them. files_by_state_only_deleting_idx covers `state` alone, so marked_for_deletion_at was a residual filter: Postgres inspected every deleting row and discarded the ones still in the grace period. LIMIT cannot stop that, and the worst case is the one the cap was added for - right after a bulk delete, when the whole backlog is inside the grace period, each broker scans all of it to return zero rows. Replace it with a partial index on marked_for_deletion_at over deleting rows. The range predicate is now an index scan that ends at the first non-matching entry, and a limited cycle stops as soon as it has its rows. The old index is dropped rather than kept: the new one serves `state = 'deleting'` equally well, and the only other query with that predicate, delete_files_v1, is keyed by object_key. Building it scans all of `files` to evaluate the predicate and holds ACCESS EXCLUSIVE while it does, since Flyway wraps migrations in a transaction and CONCURRENTLY is not available inside one. The index itself is small - deleting rows only - but the lock is held for a scan of the whole table, and `files` is on the produce commit path. Both statements are therefore idempotent: an operator can run the CONCURRENTLY equivalents before upgrading and the migration turns into a no-op. The DDL to run is in the migration header, together with the INVALID-index caveat, since CREATE INDEX IF NOT EXISTS matches on name alone and would accept a failed concurrent build. The migration stays non-concurrent rather than switching to executeInTransaction=false. As written, a failure rolls back index and history row together and the next broker start retries from scratch; a failed concurrent build would instead leave an invalid index - unusable by queries, still maintained on every write - plus a failed history row that wedges every subsequent start until someone runs flyway repair. What it does add is lock_timeout: an ACCESS EXCLUSIVE request that queues behind a long transaction blocks every commit arriving after it, so the stall would be bounded by that transaction rather than by the build. Timing out instead rolls back into the clean retry path. The ordering is a behaviour change, not just a plan change. The worklist is now the oldest eligible files instead of an arbitrary prefix, so keys the object store never confirms - which stay marked, and stay oldest - are re-served first every cycle and would crowd out later files once there were as many of them as the cap. Nothing gets there: deleting a missing object succeeds, throttling is transient and fails a different subset each cycle, and the refusals that do persist (bucket policy, Object Lock) apply to the bucket, so no file is deletable and which ones are retried first stops mattering. Recorded on ControlPlane#getFilesToDelete so the next caller does not have to derive it, with FileCleanerFilesFailedRate as the check if the assumption ever looks wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-contained setUp inserted one deleting file that every case then depended on implicitly. It showed up as assertions that do not match their own bodies: honoursLimit inserted two files and expected three, and the mixed-eligibility case named an expected key that appears nowhere in the test. Verifying either one required scrolling up to setUp and re-deriving the fixture. Move the insert into each case and have the helper return the FileToDelete it wrote, so the expected values come from the same call that created the row instead of being rebuilt by hand. setUp is left with database creation and migration, which every case needs and none asserts on. The fileId field goes with it: it was assigned from a returning() clause and never read, since FileToDelete carries no id. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-404] (#743) FileCleaner asked the control plane for every file in state deleting and filtered the grace period on the broker, so one cycle cost was proportional to the whole deletion backlog: no LIMIT, no fetch size, the full result set in broker heap while holding a jobs-pool connection and pinning a snapshot against vacuum, per broker, every cycle. Bounding it is only safe because the grace-period cutoff moves into the query too: with the age filter server-side every returned row is actionable, so a truncated worklist still drains.
…-404] (#743) FileCleaner asked the control plane for every file in state deleting and filtered the grace period on the broker, so one cycle cost was proportional to the whole deletion backlog: no LIMIT, no fetch size, the full result set in broker heap while holding a jobs-pool connection and pinning a snapshot against vacuum, per broker, every cycle. Bounding it is only safe because the grace-period cutoff moves into the query too: with the age filter server-side every returned row is actionable, so a truncated worklist still drains.
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
FileCleaner asked the control plane for every file in state
deletingand filtered the grace period on the broker, so one cycle cost was proportional to the whole deletion backlog: noLIMIT, no fetch size, the full result set in broker heap while holding a jobs-pool connection and pinning a snapshot against vacuum, per broker, every cycle. Bounding it is only safe because the grace-period cutoff moves into the query too: with the age filter server-side every returned row is actionable, so a truncated worklist still drains.Commits
bound the file-cleaner deletion worklist-getFilesToDelete(markedBefore, limit)replaces the no-arg form; both control planes enforce both bounds; addsfile.cleaner.max.files.per.cycle.count saturated file-cleaning cycles- addsFileCleanerCycleSaturatedRateso a backlog held up by the cap can be told apart from one held up by failing deletes.index the file-cleaner grace-period predicate- adds a partial index onmarked_for_deletion_atso the cap bounds scan work, not just rows returned.flag schema changes in the changelog generator- migrations added in a range now show up in the changelog with the blocking DDL they run, so an upgrade lock is not discovered by an operator after the fact.See the commit messages for the reasoning behind each step.
Config
file.cleaner.max.files.per.cycle(int, default20000,0= unbounded). Caps the worklist and doubles as the delete pacing knob: per-broker drain rate is the cap overfile.cleaner.interval.ms. Set it above the rate at which files are marked, otherwise the backlog grows.Metrics
FileCleanerCycleSaturatedRate(new) - cycles whose worklist came back over the cap. The query asks for one row beyond the cap, so an exactly-full cycle is not misreported as saturated.8000 Schema
V25__File_cleaner_marked_for_deletion_index.sqlcreatesfiles_by_marked_for_deletion_deleting_idx ON files (marked_for_deletion_at) WHERE state = 'deleting'and dropsfiles_by_state_only_deleting_idx, which coveredstateonly and left the grace-period predicate as a residual filter (soLIMITbounded rows returned but not scan work: a backlog entirely inside the grace period was walked in full to return zero rows).Operator action recommended. The index is small - it holds only
deletingrows - but building it scans all offilesto evaluate the predicate, and Flyway runs migrations in a transaction, soCONCURRENTLYis unavailable and the build holdsACCESS EXCLUSIVE. Commits tofilesblock for that scan, which scales with the whole table rather than with thedeletingsubset, andfilesis on the produce commit path. Both statements are idempotent (IF NOT EXISTS/IF EXISTS), so the lock can be avoided entirely by running the DDL concurrently before upgrading, which makes the migration a no-op:If the concurrent build fails it leaves an INVALID index behind; drop it and retry before upgrading, because
CREATE INDEX IF NOT EXISTSmatches on name only and would otherwise keep an unusable index. The migration itself stays non-concurrent on purpose: Flyway rolls it back as a unit, so a failure retries cleanly on the next broker start, whereas a failed concurrent build inside the migration would leave both an invalid index and a failed history row that blocks every broker start until someone runsflyway repair. It setslock_timeout = 5sso a pendingACCESS EXCLUSIVErequest cannot queue behind a long transaction and stall every commit arriving behind it.Behaviour change worth knowing
The worklist is now the oldest eligible files rather than an arbitrary prefix, because the plan walks the new index in
marked_for_deletion_atorder. Keys the object store never confirms stay marked and stay oldest, so in principlelimitor more permanently-failing keys could crowd out later files. In practice there is no such state to reach: deleting a missing object succeeds, throttling is transient and hits a different subset each cycle, and the only persistent failures (bucket policy, Object Lock) are bucket-wide - at which point nothing is deletable and rotating the worklist would not slow the backlog either way. Documented onControlPlane#getFilesToDelete, withFileCleanerFilesFailedRateas the signal.Testing
FileCleanerMockedTest: probe limit passed to the control plane, saturation at cap+1, no saturation at exactly the cap, and the delete set trimmed back to the cap in both cases.FindFilesToDeleteJobTest: eligible file returned, grace-period boundary exclusive,limithonoured and0unbounded - run against the migrated schema.InMemoryControlPlanenow returns oldest-first to match Postgres, so tests that exercise truncation see the same prefix in both planes.inkless-release-0.46..HEAD; V24 is reported asALTER TABLE, function-only migrations produce no note, and a header quoting theCONCURRENTLYform is not mistaken for one.