Repository navigation
feat(inkless:switch): add operator recovery for a phase 1 partition that lost its classic prefix [KC-717] - #855
Draft
giuseppelillo wants to merge 8 commits into
Conversation
…emote storage [KC-717] A switched partition that starts phase 2 without part of its classic prefix has no WAL batches between the end of remote storage and the first diskless batch. The prune moved the WAL start to that first batch, past the lost range. Until the leader uploaded a segment that spans the lost range, a leader that lost its local log couldn't rebuild: remote storage didn't cover the WAL start minus one, and the WAL rejected a fetch at the end of remote storage. The keep-offset is now at most `highestOffsetInRemoteStorage + 1` in `prune_batches_below_highest_tiered_offset_v2` (migration `V32`) and in the in-memory control plane. Batches are contiguous in every other case, so the cap doesn't change them. ## Testing `AbstractControlPlaneTest` covers the prune on both control planes. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
…out a restart [KC-717] In phase 1 of the classic-to-diskless switch, a leader whose local log ends below the seal goes offline. That fence left two problems for an operator who wants to recover the partition: - Going offline clears the partition's log, and the remote log manager tracks only partitions with a log. A broker that started with no local log never registered the partition, so it couldn't report what remote storage holds of the classic prefix. - An offline partition with the same topic ID ignores every later metadata change until the broker restarts. Electing another replica or starting phase 2 didn't bring it back. A phase 1 leader of a consolidating topic now creates an empty log when it has none, so the fence runs on it. Before the fence takes the partition offline, it registers the partition with the remote log manager as a follower. The broker records the partitions it fenced this way, and the next metadata change for one of them creates the partition again. A log directory failure clears the record, so a partition in a failed directory stays offline as before. ## Testing - `testFencedPhaseOneLeaderRegistersWithRemoteLogManagerAsFollower` checks the follower registration. - `testFencedPhaseOneLeaderComesBackOnClassicPrefixFlagCommit` and `testFencedPhaseOneLeaderComesBackAsFollowerAfterElection` check that the partition comes back as the phase 2 leader with consolidation armed, or as a follower. - `testFencedPhaseOneLeaderStaysOfflineAfterLogDirFailure` checks that a failed directory keeps the partition offline. - `testApplyDeltaCreatesEmptyLogForWipedPhaseOneLeaderAndFencesIt` checks the empty log, the fence, and the registration. - `ReplicaManagerInklessTest` and `ReplicaManagerTest` pass. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
…fix [KC-717] To recover a phase 1 partition that lost part of its classic prefix, an operator needs two facts the existing APIs don't give. The first is which replicas still hold the whole prefix. The second is how much of the prefix remote storage holds. `ListOffsets` reaches only the leader, and the leader of such a partition is offline. Remote coverage lives only in the remote log metadata. The new `DescribeDisklessSwitchReplica` broker RPC (API key 504) reports one broker's replica. It reports the local log start, log end offset, and high watermark, and whether the broker took the partition offline. It also reports the cross-tier log start and the run of copied remote segments without a gap from that start, capped at the seal. The broker reads the log from the log manager, so a partition that the phase 1 fence took offline still answers. The RPC requires `DESCRIBE` on the cluster. `Admin#describeDisklessSwitchReplica` sends it to a chosen broker. ## Testing - `ReplicaManagerInklessTest` covers a fenced leader, a gap in remote storage, segments that aren't copied, remote storage that starts above the cross-tier log start, the seal cap, no remote segments, a remote log manager that isn't ready, a classic topic, and a partition that the broker doesn't host. - `RequestResponseTest` and `RequestQuotaTest` pass. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
… is at the seal [KC-717] In phase 1 of the classic-to-diskless switch, a follower left the classic fetcher as soon as its log end offset was at the seal and it was in the ISR. The fetch response that returns the last records has the leader's high watermark from before them. The follower stops fetching, so its high watermark stays below the seal for all of phase 1. An operator who recovers a phase 1 partition needs that high watermark. A replica outside the ISR can have an uncommitted tail at or past the seal. Only a high watermark at the seal shows that the replica has the committed prefix, and `kafka-topic-switch.sh elect` rejects a replica with a lower one. The follower now also waits for its high watermark to be at the seal. Until then, it backs off like a follower that waits for the ISR, and the next fetch at the seal returns the new high watermark. A consolidating partition hands off without waiting, as before. ## Testing - `shouldNotEvictPartitionAtSealUntilHighWatermarkReachesSeal` checks that the follower waits and backs off. - `shouldEvictConsolidatingPartitionAtSealWithHighWatermarkBelowSeal` checks that a consolidating partition doesn't wait. - `verifyDisklessSwitchEviction` also checks which partitions wait at the seal. `ReplicaFetcherThreadTest` passes. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
…ic prefix [KC-717] In phase 1 of the classic-to-diskless switch, the part of the classic prefix that remote storage doesn't have yet exists only on the replicas' local disks. If the leader loses its local log while it's the only ISR member, the partition can't recover by itself. The broker fences it, and the controller elects only ISR members and blocks unclean leader election. Only a leader with the whole prefix starts phase 2. `AlterDisklessSwitch` gets two tagged fields for an operator recovery. An unset field doesn't change the request: - `LeaderId` elects that replica with an ISR of that replica alone, even if it's outside the ISR. The request must name the committed seal. The controller checks that the replica is active and that its log directory for the partition is online. `kafka-topic-switch.sh elect` checks that the replica has the whole prefix. - `ClassicPrefixInRemoteStorage` sets the flag and starts phase 2 at `SealOffset`, which can be below the committed seal. The partition loses the records between them. The record adds every active replica to the ISR and elects a leader, as the flag commit does. `PartitionChangeBuilder` writes the lower seal in the same record as the flag. Both fields need a committed seal and a partition in phase 1. The controller rejects them together or with `ClearProducerStates`. `AlterDisklessSwitchOptions` exposes them to `Admin#alterDisklessSwitch`. ## Testing - `RecoverLocalClassicPrefixTests` in `ReplicationControlManagerInklessTest` covers both actions and each rejection. - `testClassicPrefixFlagWritesLoweredSeal` checks the tag order and the merged seal. - `ReplicationControlManagerTest`, `PartitionChangeBuilderTest`, `RequestResponseTest`, and `MessageTest` pass. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
…itch [KC-717] The controller can't see the replicas' logs. Before an operator elects a replica outside the ISR or gives up part of the classic prefix, someone has to check what each replica and remote storage have. Three new `kafka-topic-switch.sh` subcommands run that check through `DescribeDisklessSwitchReplica`: - `describe` reports each replica's log and whether the replica, together with remote storage, has the whole classic prefix. It also reports the offsets that `start-phase-2` would lose. - `elect --replica` checks that the replica's high watermark is at the seal and that its log and remote storage cover the prefix without a gap. Then it sends `AlterDisklessSwitch` with `LeaderId`. - `start-phase-2` fails while a replica has the whole prefix. It sets the seal to the end of the prefix in remote storage, and it needs `--accept-data-loss` if that loses records. When the seal changes, it reminds the operator to run `repair`. The command uses the remote view of the replica with the highest remote end, because brokers read the remote log metadata at different speeds. `elect` and `start-phase-2` take `--dry-run`. ## Testing - `TopicSwitchCommandTest` covers parsing, each check, the dry runs, and the requests that the commands send. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
The recovery actions depend on broker behavior that unit tests mock: the fence of a leader without its prefix, the follower registration with the remote log manager, and the partition that comes back on the next metadata change. These tests run both actions end to end. Consolidating topics reject `remote.log.copy.disable`, so a test can't keep a partition in phase 1 through config. `GatedRemoteStorageManager` wraps the Aiven remote storage manager and fails the copy of a topic's segments from a chosen offset on. - `testElectReplicaHoldingClassicPrefixOfFencedPhaseOnePartition` wipes the only ISR member and elects the follower outside the ISR. The old leader rebuilds its prefix from the new leader without a restart, and consumers read every record. Once the gate opens, the partition starts phase 2. - `testStartPhaseTwoAtRemoteEndAfterEveryReplicaLostClassicPrefix` wipes both replicas while remote storage has only the first segment, then starts phase 2 at its end. The fenced leader comes back without a restart. Consumers read the remote prefix and then the diskless records at the old seal. ## Testing - `InklessConsolidatedDisklessTopicsTest` and `InklessTopicTypeSwitcherClusterTest` pass. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
…C-717] An operator who faces an offline phase 1 partition needs to know which recovery applies, what each command checks, and what consumers read after `start-phase-2` gives up part of the classic prefix. The new "Recover a phase 1 partition that lost its classic prefix" section walks through an example with `describe`, `elect`, and `start-phase-2`, and it lists the limitations of the checks. `DISKLESS_CONSOLIDATION.md` already links to it. The switch document also covers these changes: - A fenced phase 1 leader comes back on the next metadata change. - A phase 1 follower leaves the classic fetcher once its high watermark is at the seal. - `AlterDisklessSwitch` has the `LeaderId` and `ClassicPrefixInRemoteStorage` tagged fields. ## Testing - `python3 .vale/inkless.py --base ed97707` reports no alerts. Assisted-by: Claude Opus 5.5 Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
giuseppelillo
marked this pull request as draft
November 6, 2026 14:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In phase 1 of the classic-to-diskless switch, the part of the classic prefix that the remote log manager hasn't copied yet exists only on the replicas' local disks. KC-636 blocks unclean leader election in phase 1, so a replica with a partial prefix can't lead. The guard also leaves the partition stuck when the leader loses its local log while it's the only ISR member. The broker takes the partition offline, and the controller elects only ISR members. Only a leader with the whole prefix starts phase 2.
For example, partition 0 has replicas 1, 2, and 3, the committed seal is 1000, and remote storage has the prefix up to offset 600. Broker 2 replicated the whole prefix, then left the ISR when it shut down. Broker 1 is the leader and the only ISR member, and it loses its disk. Broker 2 has every record, and only an unclean election could make it the leader. If broker 2 has also lost its disk, the records from 600 to 1000 are gone, and the partition is offline instead of serving the rest.
This change adds two recovery actions to
kafka-topic-switch.sh. Each one runs only on an explicit operator command, and the tool checks the replicas through a new broker RPC before it asks the controller.Changes
DescribeDisklessSwitchReplicabroker RPC (API key 504) reports one replica's local log, its high watermark, and whether the broker fenced the partition. It also reports the cross-tier log start and the run of remote segments without a gap from it, capped at the seal. The RPC needsDESCRIBEon the cluster.Admin#describeDisklessSwitchReplicasends it to a chosen broker, andkafka-topic-switch.sh describeshows every replica.LeaderIdtagged field onAlterDisklessSwitchelects a replica with an ISR of that replica alone, even if it's outside the ISR. The controller checks that the broker is an active replica and that its log directory for the partition is online.kafka-topic-switch.sh electfirst checks that the replica's high watermark is at the seal and that its log and remote storage cover the prefix.ClassicPrefixInRemoteStoragetagged field sets the flag and starts phase 2 atSealOffset, which can be below the committed seal.kafka-topic-switch.sh start-phase-2sets the seal to the end of the prefix in remote storage. It fails while a replica has the whole prefix, and it needs--accept-data-losswhen it loses records. A consumer that reads in the lost range gets the diskless records from the old seal on.describecan report the remote copy. The next metadata change brings the partition back, so neither action needs a broker restart.electcan't tell its committed prefix from an uncommitted tail.highestOffsetInRemoteStorage + 1(migrationV32). Afterstart-phase-2lowers the seal, the WAL has no batches between the end of remote storage and the first diskless batch. Without the cap, the prune moves the WAL start past the lost range, and a leader that lost its local log can't rebuild.CLASSIC_TO_DISKLESS_SWITCH.mdgets the "Recover a phase 1 partition that lost its classic prefix" section with the example, andDISKLESS_CONSOLIDATION.mddescribes the prune cap.The recovery actions reuse
AlterDisklessSwitch, the operator RPC for the switch state, instead of a new election RPC. It already needsALTERon the cluster and accepts requests only before phase 2.Limitations
electrejects it even if it has every committed record.start-phase-2needs a replica that reports the remote copy of the prefix. A broker that fen 8000 ced the partition reports it only on a consolidating topic.start-phase-2lowers the seal, the control-planediskless_start_offsetkeeps the old seal until the operator runsrepair. The command prints that step.Testing
RecoverLocalClassicPrefixTestsinReplicationControlManagerInklessTestcovers both actions and each rejection.testClassicPrefixFlagWritesLoweredSealinPartitionChangeBuilderTestchecks the tag order and the merged seal.TopicSwitchCommandTestcovers the three subcommands: parsing, each check, the dry runs, and the requests they send.ReplicaManagerInklessTestcovers the describe RPC, the fence of a leader without a log, the follower registration, and the partition that comes back after an election or the flag commit.shouldNotEvictPartitionAtSealUntilHighWatermarkReachesSealandshouldEvictConsolidatingPartitionAtSealWithHighWatermarkBelowSealinReplicaFetcherThreadTestcheck the follower change.AbstractControlPlaneTestcovers the prune cap on both control planes.testElectReplicaHoldingClassicPrefixOfFencedPhaseOnePartitionandtestStartPhaseTwoAtRemoteEndAfterEveryReplicaLostClassicPrefixinInklessConsolidatedDisklessTopicsTestrun both actions on a cluster.GatedRemoteStorageManagerkeeps the partition in phase 1 by failing segment copies from a chosen offset on, because consolidating topics rejectremote.log.copy.disable.InklessConsolidatedDisklessTopicsTest,InklessTopicTypeSwitcherClusterTest,ReplicationControlManagerTest,RequestResponseTest,MessageTest, andRequestQuotaTestpass../gradlew check -x test -x generateJooqClassespasses.tests/kafkatest/tests/inkless/didn't run on this change.python3 .vale/inkless.py --base ed97707229doesn't report any alerts.Assisted-by: Claude Opus 5.5