8000 fix(inkless:consolidation): cap the WAL prune at the end of remote storage by giuseppelillo · Pull Request #868 · aiven/inkless · GitHub
[go: up one dir, main page]

Skip to content

fix(inkless:consolidation): cap the WAL prune at the end of remote storage - #868

Draft
giuseppelillo wants to merge 1 commit into
mainfrom
giuseppelillo/cap-wal-prune-at-remote-end
Draft

giuseppelillo wants to merge 1 commit into
mainfrom
giuseppelillo/cap-wal-prune-at-remote-end

Conversation

@giuseppelillo
Copy link
Copy Markdown
Contributor

After an operator lowers a committed seal with kafka-topic-switch.sh seal, the WAL has no batches between the end of remote storage and the first diskless batch. The prune takes the base offset of the first retained batch as the new WAL start, so it moves the WAL start past that range. A replica that rebuilds from remote storage resumes at the end of remote storage, fetches below the WAL start, and can't make progress.

The keep-offset now never passes highestOffsetInRemoteStorage + 1. The WAL start stays at the end of remote storage, and a read in the empty range returns the first diskless batch.

  • V32__Cap_prune_at_remote_end.sql replaces prune_batches_below_highest_tiered_offset_v2 with the capped version.
  • InMemoryControlPlane applies the same cap.
  • The jOOQ classes change only in their schema version annotation.

This change is a prerequisite for the phase 1 recovery work in PR #855 (KC-717).

Test plan

handoffBelowLostRangeKeepsWalStartAtRemoteEnd in AbstractControlPlaneTest prunes a switched log whose remote storage ends below the seal. It checks that the WAL start stays at the end of remote storage, that a read in the empty range returns the first diskless batch, and that a later prune past the seal still deletes that batch. The test runs in InMemoryControlPlaneTest and PostgresControlPlaneTest, and fails without the cap.

🤖 Generated with Claude Code

…orage

After an operator lowers a committed seal with `kafka-topic-switch.sh seal`,
the WAL has no batches between the end of remote storage and the first
diskless batch. The prune takes the base offset of the first retained batch
as the new WAL start, so it moves the WAL start past that range. A replica
that rebuilds from remote storage resumes at the end of remote storage,
fetches below the WAL start, and can't make progress.

The keep-offset now never passes `highestOffsetInRemoteStorage + 1`. The WAL
start stays at the end of remote storage, and a read in the empty range
returns the first diskless batch. `V32` replaces
`prune_batches_below_highest_tiered_offset_v2` with the capped version, and
`InMemoryControlPlane` applies the same cap.

Assisted-by: Claude Fable 5.2
@giuseppelillo
giuseppelillo marked this pull request as draft November 13, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0