8000 feat(inkless:retention): bound the file-cleaner deletion worklist [KC-404] by jeqo · Pull Request #743 · aiven/inkless · GitHub
[go: up one dir, main page]

Skip to content

feat(inkless:retention): bound the file-cleaner deletion worklist [KC-404] - #743

Merged
viktorsomogyi merged 9 commits into
mainfrom
jeqo/file-cleaner-bound
Aug 18, 2026
Merged

viktorsomogyi merged 9 commits into
mainfrom
jeqo/file-cleaner-bound

Conversation

@jeqo
@jeqo jeqo commented Aug 11, 2026 •
Copy link
Copy Markdown
Contributor

FileCleaner asked the control plane for every file in state deleting and filtered the grace period on the broker, so one cycle cost was proportional to the whole deletion backlog: no LIMIT, no fetch size, the full result set in broker heap while holding a jobs-pool connection and pinning a snapshot against vacuum, per broker, every cycle. Bounding it is only safe because the grace-period cutoff moves into the query too: with the age filter server-side every returned row is actionable, so a truncated worklist still drains.

Commits

  1. bound the file-cleaner deletion worklist - getFilesToDelete(markedBefore, limit) replaces the no-arg form; both control planes enforce both bounds; adds file.cleaner.max.files.per.cycle.
  2. count saturated file-cleaning cycles - adds FileCleanerCycleSaturatedRate so a backlog held up by the cap can be told apart from one held up by failing deletes.
  3. index the file-cleaner grace-period predicate - adds a partial index on marked_for_deletion_at so the cap bounds scan work, not just rows returned.
  4. flag schema changes in the changelog generator - migrations added in a range now show up in the changelog with the blocking DDL they run, so an upgrade lock is not discovered by an operator after the fact.

See the commit messages for the reasoning behind each step.

Config

  • file.cleaner.max.files.per.cycle (int, default 20000, 0 = unbounded). Caps the worklist and doubles as the delete pacing knob: per-broker drain rate is the cap over file.cleaner.interval.ms. Set it above the rate at which files are marked, otherwise the backlog grows.

Metrics

  • FileCleanerCycleSaturatedRate (new) - cycles whose worklist came back over the cap. The query asks for one row beyond the cap, so an exactly-full cycle is not misreported as saturated.

8000 Schema

V25__File_cleaner_marked_for_deletion_index.sql creates files_by_marked_for_deletion_deleting_idx ON files (marked_for_deletion_at) WHERE state = 'deleting' and drops files_by_state_only_deleting_idx, which covered state only and left the grace-period predicate as a residual filter (so LIMIT bounded rows returned but not scan work: a backlog entirely inside the grace period was walked in full to return zero rows).

Operator action recommended. The index is small - it holds only deleting rows - but building it scans all of files to evaluate the predicate, and Flyway runs migrations in a transaction, so CONCURRENTLY is unavailable and the build holds ACCESS EXCLUSIVE. Commits to files block for that scan, which scales with the whole table rather than with the deleting subset, and files is on the produce commit path. Both statements are idempotent (IF NOT EXISTS / IF EXISTS), so the lock can be avoided entirely by running the DDL concurrently before upgrading, which makes the migration a no-op:

SET LOCAL lock_timeout = '5s';
CREATE INDEX CONCURRENTLY IF NOT EXISTS files_by_marked_for_deletion_deleting_idx
    ON files (marked_for_deletion_at) WHERE state = 'deleting';
DROP INDEX CONCURRENTLY IF EXISTS files_by_state_only_deleting_idx;

If the concurrent build fails it leaves an INVALID index behind; drop it and retry before upgrading, because CREATE INDEX IF NOT EXISTS matches on name only and would otherwise keep an unusable index. The migration itself stays non-concurrent on purpose: Flyway rolls it back as a unit, so a failure retries cleanly on the next broker start, whereas a failed concurrent build inside the migration would leave both an invalid index and a failed history row that blocks every broker start until someone runs flyway repair. It sets lock_timeout = 5s so a pending ACCESS EXCLUSIVE request cannot queue behind a long transaction and stall every commit arriving behind it.

Behaviour change worth knowing

The worklist is now the oldest eligible files rather than an arbitrary prefix, because the plan walks the new index in marked_for_deletion_at order. Keys the object store never confirms stay marked and stay oldest, so in principle limit or more permanently-failing keys could crowd out later files. In practice there is no such state to reach: deleting a missing object succeeds, throttling is transient and hits a different subset each cycle, and the only persistent failures (bucket policy, Object Lock) are bucket-wide - at which point nothing is deletable and rotating the worklist would not slow the backlog either way. Documented on ControlPlane#getFilesToDelete, with FileCleanerFilesFailedRate as the signal.

Testing

  • FileCleanerMockedTest: probe limit passed to the control plane, saturation at cap+1, no saturation at exactly the cap, and the delete set trimmed back to the cap in both cases.
  • FindFilesToDeleteJobTest: eligible file returned, grace-period boundary exclusive, limit honoured and 0 unbounded - run against the migrated schema.
  • InMemoryControlPlane now returns oldest-first to match Postgres, so tests that exercise truncation see the same prefix in both planes.
  • Changelog generator run over past release ranges and over inkless-release-0.46..HEAD; V24 is reported as ALTER TABLE, function-only migrations produce no note, and a header quoting the CONCURRENTLY form is not mistaken for one.

@jeqo
jeqo force-pushed the jeqo/file-cleaner-bound branch 2 times, most recently from 80fbfbc to ad49a61 Compare August 12, 2026 13:24
@jeqo
jeqo requested a balanced review from Copilot August 12, 2026 13:24
@jeqo
jeqo force-pushed the jeqo/file-cleaner-bound branch from ad49a61 to f30f274 Compare August 12, 2026 14:33
@jeqo
jeqo marked this pull request as ready for review August 12, 2026 14:40
@jeqo jeqo changed the title feat(inkless:retention): bound the file-cleaner deletion worklist feat(inkless:retention): bound the file-cleaner deletion worklist [KC-404] Aug 12, 2026
@jeqo
jeqo requested a review from viktorsomogyi August 13, 2026 09:05
@jeqo
jeqo force-pushed the jeqo/file-cleaner-bound branch from f30f274 to 90f28d9 Compare August 13, 2026 12:07
@jeqo
jeqo requested a balanced review from Copilot and removed request for Copilot August 13, 2026 12:08
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Bounds file-cleaner work per cycle and moves grace-period filtering into control-plane queries.

Changes:

  • Adds configurable per-cycle deletion limits.
  • Adds saturation observability.
  • Updates PostgreSQL/in-memory implementations and tests.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
FileCleaner.java Applies grace-period and cycle bounds.
FileCleanerMetrics.java Adds saturation metric.
ControlPlane.java Defines bounded deletion query contract.
InMemoryControlPlane.java Implements filtering and limits.
PostgresControlPlane.java Passes query bounds to the job.
FindFilesToDeleteJob.java Adds SQL filtering and limits.
InklessConfig.java Adds the cycle-cap configuration.
FileCleanerMockedTest.java Tests bounded cleaning and metrics.
FileCleanerIntegrationTest.java Updates integration assertions.
FindFilesToDeleteJobTest.java Tests eligibility and limits.
InMemoryControlPlanePruneDisklessLogsTest.java Updates deletion queries.
AbstractControlPlaneTest.java Adds an unbounded test helper.
InklessConfigTest.java Verifies the default cap.
docs/inkless/metrics.rst Documents saturation metrics.
docs/inkless/configs.rst Documents the new configuration.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@jeqo
jeqo force-pushed the jeqo/file-cleaner-bound branch from 90f28d9 to 0078bfe Compare August 13, 2026 12:56
@jeqo
jeqo marked this pull request as draft August 13, 2026 13:29
FileCleaner fetched every file in state 'deleting' and filtered the grace period
client-side, so the control-plane query was proportional to the whole deletion
backlog. After a bulk topic delete that is unbounded: FindFilesToDeleteJob has no
LIMIT and no JDBC fetch size, so the full result set materializes in broker heap
while holding a connection from the jobs pool (shared with the produce commit,
delete-records and retention paths) and pinning a snapshot against vacuum. Every
broker runs its own cleaner, so the cost is paid per broker per cycle. The same
unbounded set was then handed to the backend, which issues 1000-key DeleteObjects
calls back to back with no pacing.

Push both bounds into the control plane, which is the only place they can be
enforced: by the time a caller-side filter runs, the query cost is already paid.

- getFilesToDelete(markedBefore, limit) replaces the no-arg form. markedBefore is
  the grace-period cutoff (now - file.cleaner.retention.period.ms), previously
  applied in FileCleaner; limit caps the rows returned, 0 meaning unbounded (the
  max.batches.per.request convention). Both control planes enforce both bounds.
- file.cleaner.max.files.per.cycle (default 20000) caps the worklist. It doubles
  as the delete pacing knob: the drain rate is the cap over
  file.cleaner.interval.ms, and pacing (not slice size) is the lever that reduces
  S3 request-rate throttling. The default is sized against the rate at which files
  are created, since in steady state files are marked for deletion as fast as they
  are produced: a broker rotates a file every produce.commit.interval.ms (250 ms,
  so 4/s) or whenever produce.buffer.max.bytes (8 MiB) fills, whichever comes
  first, so above ~32 MiB/s of ingest per broker the size trigger dominates and
  the rate is ingest/8 MiB - tens of files/s on a busy broker, not 4.

Pushing the grace period down is what makes the limit safe rather than an
independent cleanup: with the age filter on the broker, a truncated result could
consist entirely of not-yet-eligible files while eligible ones went unreturned, and
the cleaner would spin without draining. Applying it server-side makes every
returned row actionable, so any truncation still makes progress.

The query deliberately has no ORDER BY. Ordering by file_id would let the planner
walk files_pkey ascending and filter, making the scan cost depend on how many
low-file_id non-deleting rows it skips rather than on the limit; it also orders by
commit time, while a bulk topic delete marks a whole topic at once, so commit order
and mark order are unrelated exactly when the backlog is large. Nor is ordering
needed semantically: a marked file is unreferenced, so there is no fairness claim
between candidates. What the limit does not bound yet is scan work:
files_by_state_only_deleting_idx covers `state` only, so the grace-period predicate
is a residual filter that the LIMIT cannot stop. A later commit indexes it.

Bounding a cycle changes how a permanently-undeletable key behaves. Today any error
throws the whole slice, so one bad key anywhere stalls every cycle for the entire
backlog; with a cap it stalls only once the window reaches it, and the files ahead of
it still drain. That is an improvement, but the stall is now silent, so the following
commit adds a counter for saturated cycles. The default is generous enough
that a small set of poison keys cannot fill a cycle; a per-key failure count (which
would need a column) is the durable fix if that proves insufficient.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jeqo
jeqo force-pushed the jeqo/file-cleaner-bound branch 2 times, most recently from adccc6a to 31f82e8 Compare August 14, 2026 11:59
@jeqo
jeqo requested a balanced review from Copilot August 14, 2026 12:01
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 137 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/FindFilesToDeleteJob.java:66

  • PostgreSQL does not guarantee index order without ORDER BY; the planner may choose a sequential scan and return a heap-order prefix. That contradicts the oldest-first behavior deliberately implemented by the in-memory plane and described by the API/migration, so capped cycles can process a different prefix in production. Order by marked_for_deletion_at before applying the limit.
        // No ORDER BY: the returned rows are all actionable, so any prefix makes progress. The scan is
        // bounded by files_by_marked_for_deletion_deleting_idx, which covers the grace-period predicate
        // and lets Postgres stop once `limit` rows are found.

storage/inkless/src/main/resources/db/migration/V25__File_cleaner_marked_for_deletion_index.sql:41

  • The lock awaited by the index build is SHARE, not ACCESS EXCLUSIVE. The queueing warning still applies because it conflicts with writers, but naming the wrong mode may lead operators to expect reads to be blocked as well.
    storage/inkless/src/main/resources/db/migration/V25__File_cleaner_marked_for_deletion_index.sql:18
  • A regular PostgreSQL CREATE INDEX holds a SHARE lock on the table during the scan, not ACCESS EXCLUSIVE. It still blocks writes, so the operational warning remains important, but the documented lock mode should be accurate (the later non-concurrent DROP INDEX has different locking behavior).

This issue also appears on line 40 of the same file.

A cycle that stops at file.cleaner.max.files.per.cycle is indistinguishable from one
that found nothing more to do, so a deletion backlog that keeps growing cannot be
attributed to the cap rather than to deletes that are failing. The distinction
decides the response: raise the cap, or go find out why deletes fail.

It matters more now that a cycle is bounded, because a permanently-undeletable key
occupies a slot in the window every pass and holds up the files behind it. Backlog
depth itself is observed out-of-band from the files table; this counter is what turns
a growing backlog into a diagnosis.

Add FileCleanerCycleSaturatedRate, incremented when the worklist comes back over the
cap. The query asks for one row beyond the cap and the extra row is dropped before
deleting, so a cycle that happens to find exactly cap eligible files is not reported
as saturated - "at the cap" and "more remain" are not the same state, and only the
second one is actionable. The add saturates at Integer.MAX_VALUE: overflowing would
hand the control planes a negative limit, which they read as unbounded, so the one
setting that asks for the largest possible cap would remove it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jeqo
jeqo force-pushed the jeqo/file-cleaner-bound branch from 31f82e8 to 6b406f1 Compare August 17, 2026 08:03
@jeqo
jeqo requested a balanced review from Copilot August 17, 2026 08:03
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 137 changed files in this pull request and generated no new comments.

Suppressed comments (3)

storage/inkless/src/main/resources/db/migration/V25__File_cleaner_marked_for_deletion_index.sql:43

  • The lock request that can wait for a long-running writer before this build is SHARE, not ACCESS EXCLUSIVE. It still queues later writers behind it, so the timeout rationale remains valid, but the documented lock mode should match PostgreSQL's CREATE INDEX behavior.
    storage/inkless/src/main/java/io/aiven/inkless/control_plane/postgres/FindFilesToDeleteJob.java:66
  • Without ORDER BY, PostgreSQL does not guarantee index order; a sequential or bitmap plan can return a different bounded prefix as statistics change. That breaks the oldest-first behavior and the stated parity with InMemoryControlPlane. Order explicitly by marked_for_deletion_at so the new index defines the worklist.
        // No ORDER BY: the returned rows are all actionable, so any prefix makes progress. The scan is
        // bounded by files_by_marked_for_deletion_deleting_idx, which covers the grace-period predicate
        // and lets Postgres stop once `limit` rows are found.

storage/inkless/src/main/resources/db/migration/V25__File_cleaner_marked_for_deletion_index.sql:18

  • This overstates the lock held during the full index build. PostgreSQL's non-concurrent CREATE INDEX holds a SHARE lock on the table (which still blocks inserts, updates, and deletes); ACCESS EXCLUSIVE is only needed later by the non-concurrent drop. Correcting the lock mode is important operator guidance when diagnosing blocked sessions.

This issue also appears on line 40 of the same file.

@jeqo
jeqo force-pushed the jeqo/file-cleaner-bound branch 2 times, most recently from c039fc4 to 692b53c Compare August 17, 2026 09:35
jeqo and others added 2 commits August 17, 2026 12:48
…icate

Bounding the worklist bounded the rows returned, not the work done to find them.
files_by_state_only_deleting_idx covers `state` alone, so marked_for_deletion_at was a
residual filter: Postgres inspected every deleting row and discarded the ones still in
the grace period. LIMIT cannot stop that, and the worst case is the one the cap was
added for - right after a bulk delete, when the whole backlog is inside the grace
period, each broker scans all of it to return zero rows.

Replace it with a partial index on marked_for_deletion_at over deleting rows. The
range predicate is now an index scan that ends at the first non-matching entry, and a
limited cycle stops as soon as it has its rows. The old index is dropped rather than
kept: the new one serves `state = 'deleting'` equally well, and the only other query
with that predicate, delete_files_v1, is keyed by object_key.

Building it scans all of `files` to evaluate the predicate and holds ACCESS EXCLUSIVE
while it does, since Flyway wraps migrations in a transaction and CONCURRENTLY is not
available inside one. The index itself is small - deleting rows only - but the lock is
held for a scan of the whole table, and `files` is on the produce commit path. Both
statements are therefore idempotent: an operator can run the CONCURRENTLY equivalents
before upgrading and the migration turns into a no-op. The DDL to run is in the
migration header, together with the INVALID-index caveat, since CREATE INDEX IF NOT
EXISTS matches on name alone and would accept a failed concurrent build.

The migration stays non-concurrent rather than switching to executeInTransaction=false.
As written, a failure rolls back index and history row together and the next broker
start retries from scratch; a failed concurrent build would instead leave an invalid
index - unusable by queries, still maintained on every write - plus a failed history row
that wedges every subsequent start until someone runs flyway repair. What it does add is
lock_timeout: an ACCESS EXCLUSIVE request that queues behind a long transaction blocks
every commit arriving after it, so the stall would be bounded by that transaction rather
than by the build. Timing out instead rolls back into the clean retry path.

The ordering is a behaviour change, not just a plan change. The worklist is now the
oldest eligible files instead of an arbitrary prefix, so keys the object store never
confirms - which stay marked, and stay oldest - are re-served first every cycle and
would crowd out later files once there were as many of them as the cap. Nothing gets
there: deleting a missing object succeeds, throttling is transient and fails a different
subset each cycle, and the refusals that do persist (bucket policy, Object Lock) apply
to the bucket, so no file is deletable and which ones are retried first stops mattering.
Recorded on ControlPlane#getFilesToDelete so the next caller does not have to derive it,
with FileCleanerFilesFailedRate as the check if the assumption ever looks wrong.

InMemoryControlPlane sorts the same way. It is a test double, and leaving it on hash
order would hide exactly the truncation prefix the Postgres plane now guarantees.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… generator

The changelog draws config changes from configs.rst and metrics from metrics.rst, so a
migration reached the changelog only if a commit subject happened to mention it. That
is the one change class an operator cannot react to after the fact: a migration that
takes a table-level lock blocks the produce path during the upgrade, and V25 is exactly
that case, avoidable only by running the DDL concurrently beforehand.

Diff the filenames under db/migration between the two tags. Flyway files are
append-only, so a set difference is exact - no parsing of SQL semantics, which the
generator has no business interpreting. Annotate each entry with the blocking DDL it
contains (CREATE INDEX, ALTER TABLE, ...) and whether it runs CONCURRENTLY, so the
reader knows to open the migration header. Comments are stripped before scanning:
headers quote the concurrent form as operator guidance, which would otherwise report a
lock-taking migration as concurrent.

The section is titled "Postgres schema changes" rather than "schema": in a Kafka fork
"schema" reads as record schemas or the metadata log, and the reader needs to know this
is the control-plane database. It appears in the summary output too. Unlike metric
deltas, which are noisy because metrics.rst tracks a hand-maintained registry list, this
source is exact and needs no curation step beyond reading the migration header.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jeqo
jeqo force-pushed the jeqo/file-cleaner-bound branch from 692b53c to 6e8d2ed Compare August 17, 2026 09:49
@jeqo
jeqo marked this pull request as ready for review August 17, 2026 10:04
@jeqo
jeqo requested a review from viktorsomogyi August 17, 2026 16:26
@viktorsomogyi viktorsomogyi left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

had 2 minor comments, otherwise LGTM

jeqo and others added 3 commits August 18, 2026 12:09
…od predicate

perf(inkless:control_plane): index the file-cleaner grace-period predicate

Bounding the worklist bounded the rows returned, not the work done to find them.
files_by_state_only_deleting_idx covers `state` alone, so marked_for_deletion_at was a
residual filter: Postgres inspected every deleting row and discarded the ones still in
the grace period. LIMIT cannot stop that, and the worst case is the one the cap was
added for - right after a bulk delete, when the whole backlog is inside the grace
period, each broker scans all of it to return zero rows.

Replace it with a partial index on marked_for_deletion_at over deleting rows. The
range predicate is now an index scan that ends at the first non-matching entry, and a
limited cycle stops as soon as it has its rows. The old index is dropped rather than
kept: the new one serves `state = 'deleting'` equally well, and the only other query
with that predicate, delete_files_v1, is keyed by object_key.

Building it scans all of `files` to evaluate the predicate and holds ACCESS EXCLUSIVE
while it does, since Flyway wraps migrations in a transaction and CONCURRENTLY is not
available inside one. The index itself is small - deleting rows only - but the lock is
held for a scan of the whole table, and `files` is on the produce commit path. Both
statements are therefore idempotent: an operator can run the CONCURRENTLY equivalents
before upgrading and the migration turns into a no-op. The DDL to run is in the
migration header, together with the INVALID-index caveat, since CREATE INDEX IF NOT
EXISTS matches on name alone and would accept a failed concurrent build.

The migration stays non-concurrent rather than switching to executeInTransaction=false.
As written, a failure rolls back index and history row together and the next broker
start retries from scratch; a failed concurrent build would instead leave an invalid
index - unusable by queries, still maintained on every write - plus a failed history row
that wedges every subsequent start until someone runs flyway repair. What it does add is
lock_timeout: an ACCESS EXCLUSIVE request that queues behind a long transaction blocks
every commit arriving after it, so the stall would be bounded by that transaction rather
than by the build. Timing out instead rolls back into the clean retry path.

The ordering is a behaviour change, not just a plan change. The worklist is now the
oldest eligible files instead of an arbitrary prefix, so keys the object store never
confirms - which stay marked, and stay oldest - are re-served first every cycle and
would crowd out later files once there were as many of them as the cap. Nothing gets
there: deleting a missing object succeeds, throttling is transient and fails a different
subset each cycle, and the refusals that do persist (bucket policy, Object Lock) apply
to the bucket, so no file is deletable and which ones are retried first stops mattering.
Recorded on ControlPlane#getFilesToDelete so the next caller does not have to derive it,
with FileCleanerFilesFailedRate as the check if the assumption ever looks wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…-contained

setUp inserted one deleting file that every case then depended on implicitly. It showed
up as assertions that do not match their own bodies: honoursLimit inserted two files and
expected three, and the mixed-eligibility case named an expected key that appears nowhere
in the test. Verifying either one required scrolling up to setUp and re-deriving the
fixture.

Move the insert into each case and have the helper return the FileToDelete it wrote, so
the expected values come from the same call that created the row instead of being rebuilt
by hand. setUp is left with database creation and migration, which every case needs and
none asserts on. The fileId field goes with it: it was assigned from a returning() clause
and never read, since FileToDelete carries no id.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jeqo
jeqo requested a review from viktorsomogyi August 18, 2026 10:02
@viktorsomogyi
viktorsomogyi merged commit 6e14f04 into main Aug 18, 2026
7 checks passed
@viktorsomogyi
viktorsomogyi deleted the jeqo/file-cleaner-bound branch August 18, 2026 12:14
jeqo pushed a commit that referenced this pull request Aug 19, 2026
…-404] (#743)

FileCleaner asked the control plane for every file in state deleting and filtered the grace period on the broker, so one cycle cost was proportional to the whole deletion backlog: no LIMIT, no fetch size, the full result set in broker heap while holding a jobs-pool connection and pinning a snapshot against vacuum, per broker, every cycle. Bounding it is only safe because the grace-period cutoff moves into the query too: with the age filter server-side every returned row is actionable, so a truncated worklist still drains.
jeqo pushed a commit that referenced this pull request Aug 19, 2026
…-404] (#743)

FileCleaner asked the control plane for every file in state deleting and filtered the grace period on the broker, so one cycle cost was proportional to the whole deletion backlog: no LIMIT, no fetch size, the full result set in broker heap while holding a jobs-pool connection and pinning a snapshot against vacuum, per broker, every cycle. Bounding it is only safe because the grace-period cutoff moves into the query too: with the age filter server-side every returned row is actionable, so a truncated worklist still drains.
jeqo added a commit that referenced this pull request Aug 19, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the
inkless-changelog skill and curated:

- Metric deltas cross-checked against source commits (#716, #735, #743,
  #746, #715); all trace to real, shipped metrics, no documentation
  catch-up noise this increment.
- Postgres migration headers (V23-V26) checked for lock impact; V25 takes
  ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26
  offers a pre-upgrade CONCURRENTLY workaround.
- Added two config default changes the generator's diff-only detection
  missed (existing keys, not new ones): file.cleaner.interval.ms
  (5min -> 2min, #745) and retention.enforcement.interval.ms /
  max.batches.per.request (#744).
jeqo added a commit that referenced this pull request Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the
inkless-changelog skill and curated:

- Metric deltas cross-checked against source commits (#716, #735, #743,
  #746, #715); all trace to real, shipped metrics, no documentation
  catch-up noise this increment.
- Postgres migration headers (V23-V26) checked for lock impact; V25 takes
  ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26
  offers a pre-upgrade CONCURRENTLY workaround.
- Added two config default changes the generator's diff-only detection
  missed (existing keys, not new ones): file.cleaner.interval.ms
  (5min -> 2min, #745) and retention.enforcement.interval.ms /
  max.batches.per.request (#744).
jeqo added a commit that referenced this pull request Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the
inkless-changelog skill and curated:

- Metric deltas cross-checked against source commits (#716, #735, #743,
  #746, #715); all trace to real, shipped metrics, no documentation
  catch-up noise this increment.
- Postgres migration headers (V23-V26) checked for lock impact; V25 takes
  ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26
  offers a pre-upgrade CONCURRENTLY workaround.
- Added two config default changes the generator's diff-only detection
  missed (existing keys, not new ones): file.cleaner.interval.ms
  (5min -> 2min, #745) and retention.enforcement.interval.ms /
  max.batches.per.request (#744).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

0