Repository navigation
feat(inkless:retention): shorten the file-cleaner interval to 2 minutes [KC-404] - #745
Merged
Merged
Conversation
jeqo
force-pushed
the
jeqo/file-cleaner-defaults
branch
from
August 12, 2026 13:25
263e4ad to
77a6b0f
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Shortens the Inkless file-cleaner cadence to improve deletion backlog drain rate.
Changes:
- Changes the default interval from five to two minutes.
- Updates configuration documentation and default-value test.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
InklessConfig.java |
Updates the default and configuration description. |
InklessConfigTest.java |
Verifies the new default. |
configs.rst |
Updates generated configuration documentation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
jeqo
force-pushed
the
jeqo/file-cleaner-defaults
branch
2 times, most recently
from
August 12, 2026 15:24
b46f302 to
a45f288
Compare
jeqo
marked this pull request as ready for review
August 12, 2026 15:34
jeqo
force-pushed
the
jeqo/file-cleaner-defaults
branch
from
August 18, 2026 16:07
a45f288 to
303f94d
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
storage/inkless/src/main/java/io/aiven/inkless/config/InklessConfig.java:110
- The PR rationale still treats this limit as half of
file.cleaner.interval.ms, calculates a 60-second allowance, and calls the rule documentation-only. This corrected text matchesSharedState.java:130-134: the broker enforces half offile.cleaner.retention.period.ms, so the unchanged one-minute default permits 30 seconds regardless of this PR's interval change. Update the PR body so the squashed commit doesn't preserve an incorrect safety argument.
"The time to live must be <= half of the value of file.cleaner.retention.period.ms.";
A cycle used to fetch the whole deletion backlog, so running it less often was the only brake on query cost, and 5 minutes bought that brake. `file.cleaner.max.files.per.cycle` now bounds the fetch, which caps per-cycle cost and frees the interval to serve drain rate instead. The old cadence has less headroom than it looks. Steady-state marking tracks file creation: a broker rotates a file every `produce.commit.interval.ms` (250 ms), or sooner when `produce.buffer.max.bytes` (8 MiB) fills. The timer sets a floor of 4 files/s per broker. Above roughly 32 MiB/s of ingest per broker the size trigger dominates at ingest / 8 MiB, about 12 files/s at 100 MiB/s. Cluster drain is harder to pin down than a single cycle. The bounded query is deterministic and there is no leasing, so cycles that overlap on different brokers select the same rows and delete the same objects. Cluster drain therefore lands between 1x and Nx the cap per interval, and only the 1x end is safe to assume. On that pessimistic assumption, 20000 files per 5 minutes drains 67 files/s. Six timer-bound brokers create 24 files/s, and six brokers at 100 MiB/s each create 75 files/s, so the old default goes underwater exactly when ingest is highest. 20000 files per 2 minutes drains 167 files/s. Shortening the interval beats raising the cap for the same throughput. It holds the per-cycle burst at 20 `DeleteObjects` calls instead of 50, and S3 request-rate throttling responds to the burst. It also keeps the transient worklist smaller and shortens the jobs-pool connection hold time. A cycle holds that worklist about four times over, across the fetch, the key sets, and the confirmed-deleted set. The `consume.batch.coordinate.cache.ttl.ms` documentation pointed the reader at the wrong config, which made the shorter interval look like it tightens the cache bound. `SharedState.initialize` enforces a ceiling of half of `file.cleaner.retention.period.ms` and rejects the broker config at startup otherwise. The retention period stays at 1 minute here, so the ceiling stays at 30 s and the interval change doesn't touch it. The doc string now names that config. `FileCleanerCycleSaturatedRate` is the feedback signal. If it stays flat in production, the cap is over-provisioned and the interval can go back up. Testing `InklessConfigTest` asserts the new default, and `docs/inkless/configs.rst` carries the new default and doc strings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jeqo
force-pushed
the
jeqo/file-cleaner-defaults
branch
from
August 18, 2026 16:36
303f94d to
ef77801
Compare
tvainika
approved these changes
Aug 19, 2026
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
…es (#745) A cycle used to fetch the whole deletion backlog, so running it less often was the only brake on query cost, and 5 minutes bought that brake. `file.cleaner.max.files.per.cycle` now bounds the fetch, which caps per-cycle cost and frees the interval to serve drain rate instead. The old cadence has less headroom than it looks. Steady-state marking tracks file creation: a broker rotates a file every `produce.commit.interval.ms` (250 ms), or sooner when `produce.buffer.max.bytes` (8 MiB) fills. The timer sets a floor of 4 files/s per broker. Above roughly 32 MiB/s of ingest per broker the size trigger dominates at ingest / 8 MiB, about 12 files/s at 100 MiB/s. Cluster drain is harder to pin down than a single cycle. The bounded query is deterministic and there is no leasing, so cycles that overlap on different brokers select the same rows and delete the same objects. Cluster drain therefore lands between 1x and Nx the cap per interval, and only the 1x end is safe to assume. On that pessimistic assumption, 20000 files per 5 minutes drains 67 files/s. Six timer-bound brokers create 24 files/s, and six brokers at 100 MiB/s each create 75 files/s, so the old default goes underwater exactly when ingest is highest. 20000 files per 2 minutes drains 167 files/s. Shortening the interval beats raising the cap for the same throughput. It holds the per-cycle burst at 20 `DeleteObjects` calls instead of 50, and S3 request-rate throttling responds to the burst. It also keeps the transient worklist smaller and shortens the jobs-pool connection hold time. A cycle holds that worklist about four times over, across the fetch, the key sets, and the confirmed-deleted set. The `consume.batch.coordinate.cache.ttl.ms` documentation pointed the reader at the wrong config, which made the shorter interval look like it tightens the cache bound. `SharedState.initialize` enforces a ceiling of half of `file.cleaner.retention.period.ms` and rejects the broker config at startup otherwise. The retention period stays at 1 minute here, so the ceiling stays at 30 s and the interval change doesn't touch it. The doc string now names that config. `FileCleanerCycleSaturatedRate` is the feedback signal. If it stays flat in production, the cap is over-provisioned and the interval can go back up. Testing `InklessConfigTest` asserts the new default, and `docs/inkless/configs.rst` carries the new default and doc strings. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
…es (#745) A cycle used to fetch the whole deletion backlog, so running it less often was the only brake on query cost, and 5 minutes bought that brake. `file.cleaner.max.files.per.cycle` now bounds the fetch, which caps per-cycle cost and frees the interval to serve drain rate instead. The old cadence has less headroom than it looks. Steady-state marking tracks file creation: a broker rotates a file every `produce.commit.interval.ms` (250 ms), or sooner when `produce.buffer.max.bytes` (8 MiB) fills. The timer sets a floor of 4 files/s per broker. Above roughly 32 MiB/s of ingest per broker the size trigger dominates at ingest / 8 MiB, about 12 files/s at 100 MiB/s. Cluster drain is harder to pin down than a single cycle. The bounded query is deterministic and there is no leasing, so cycles that overlap on different brokers select the same rows and delete the same objects. Cluster drain therefore lands between 1x and Nx the cap per interval, and only the 1x end is safe to assume. On that pessimistic assumption, 20000 files per 5 minutes drains 67 files/s. Six timer-bound brokers create 24 files/s, and six brokers at 100 MiB/s each create 75 files/s, so the old default goes underwater exactly when ingest is highest. 20000 files per 2 minutes drains 167 files/s. Shortening the interval beats raising the cap for the same throughput. It holds the per-cycle burst at 20 `DeleteObjects` calls instead of 50, and S3 request-rate throttling responds to the burst. It also keeps the transient worklist smaller and shortens the jobs-pool connection hold time. A cycle holds that worklist about four times over, across the fetch, the key sets, and the confirmed-deleted set. The `consume.batch.coordinate.cache.ttl.ms` documentation pointed the reader at the wrong config, which made the shorter interval look like it tightens the cache bound. `SharedState.initialize` enforces a ceiling of half of `file.cleaner.retention.period.ms` and rejects the broker config at startup otherwise. The retention period stays at 1 minute here, so the ceiling stays at 30 s and the interval change doesn't touch it. The doc string now names that config. `FileCleanerCycleSaturatedRate` is the feedback signal. If it stays flat in production, the cap is over-provisioned and the interval can go back up. Testing `InklessConfigTest` asserts the new default, and `docs/inkless/configs.rst` carries the new default and doc strings. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
jeqo
added a commit
that referenced
this pull request
Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
jeqo
added a commit
that referenced
this pull request
Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A cycle used to fetch the whole deletion backlog, so running it less often was the only brake on query cost, and 5 minutes bought that brake.
file.cleaner.max.files.per.cyclenow bounds the fetch, which caps per-cycle cost and frees the interval to serve drain rate instead.The old cadence has less headroom than it looks. Steady-state marking tracks file creation: a broker rotates a file every
produce.commit.interval.ms(250 ms), or sooner whenproduce.buffer.max.bytes(8 MiB) fills. The timer sets a floor of 4 files/s per broker. Above roughly 32 MiB/s of ingest per broker the size trigger dominates at ingest / 8 MiB, about 12 files/s at 100 MiB/s.Cluster drain is harder to pin down than a single cycle. The bounded query is deterministic and there is no leasing, so cycles that overlap on different brokers select the same rows and delete the same objects. Cluster drain therefore lands between 1x and Nx the cap per interval, and only the 1x end is safe to assume.
On that pessimistic assumption, 20000 files per 5 minutes drains 67 files/s. Six timer-bound brokers create 24 files/s, and six brokers at 100 MiB/s each create 75 files/s, so the old default goes underwater exactly when ingest is highest. 20000 files per 2 minutes drains 167 files/s.
Shortening the interval beats raising the cap for the same throughput. It holds the per-cycle burst at 20
DeleteObjectscalls instead of 50, and S3 request-rate throttling responds to the burst. It also keeps the transient worklist smaller and shortens the jobs-pool connection hold time. A cycle holds that worklist about four times over, across the fetch, the key sets, and the confirmed-deleted set.FileCleanerCycleSaturatedRateis the feedback signal. If it stays flat in production, the cap is over-provisioned and the interval can go back up.Config changes
inkless.file.cleaner.interval.msdefault: 300000 (5 minutes) -> 120000 (2 minutes). Clusters that pinned the old value keep it.inkless.consume.batch.coordinate.cache.ttl.msdoc string: pointed atfile.cleaner.interval.ms, which made the shorter interval look like it tightens the cache bound.SharedState.initializeenforces a ceiling of half offile.cleaner.retention.period.msand rejects the broker config at startup otherwise. The retention period stays at 1 minute here, so the ceiling stays at 30 s and the interval change doesn't touch it. The doc string now names that config, and drops a duplicated word.Operator impact
File-cleaner cycles run 2.5x more often, so per-cycle counters and rates re-baseline. Total deleted objects per unit of ingest doesn't change. Expect
FileCleanerCycleSaturatedRateto drop wherever the cap was binding at the old cadence.Testing
InklessConfigTestasserts the new default, anddocs/inkless/configs.rstcarries the new default and doc strings.