Repository navigation
test(inkless:switch): run the stale high watermark promotion test in phase 2 [KC-721] - #854
Merged
viktorsomogyi merged 1 commit intoNov 5, 2026
Conversation
…phase 2 [KC-721] `test_switched_leader_promotion_restores_stale_hw` fails on main since #849 (KC-636). The test sets `unclean.leader.election.enable=true` right after the seal commit, and the controller now rejects that config until phase 2. The test cluster had no remote storage, so its switched partition never left phase 1. The old scenario can't happen in phase 1 anymore. There, the controller elects only ISR members, and the old test needed unclean election to promote a replica outside the ISR. In phase 2, remote storage holds the prefix and the controller can elect any live replica. A replica with a stale high watermark can still become leader there. This change reworks the test to cover that case instead of removing it. The test runs on a consolidation cluster with tiered storage on MinIO. It switches with `remote.storage.enable` in the same alter as `diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before it injects the stale high watermark checkpoint. The test stops the promoted replica last instead of isolating it with a network partition. It's then the last ISR member of the test topic and of the internal topics, so the controller elects it without unclean election when it restarts alone. It also leads `__remote_log_metadata`, which it needs to serve the prefix from remote storage. The consumer must read every record across the seal. On main, the follower consolidation fetcher repairs the stale high watermark before the promotion. The leader-side repair in `maybeReconcileSwitchedLeader` doesn't run in this test, and `testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers it. The change also removes `_run_unclean_leader_election`, which has no other callers. ## Testing - `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker on main. The logs show phase 2 at seal 5000, a broker reload with high watermark 2500, a local log start offset of 5000, and 6000 records consumed. - `make vale` passes. Assisted-by: Claude Opus 5.5
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the phase-2 stale high-watermark promotion test update.
No open issues 🎉
What changed in this PR
Updates the stale high-watermark promotion system test to run in switch phase 2, where remote storage holds the classic prefix. This adapts the test to the controller’s phase-1 unclean-election restrictions while preserving consumer coverage across the seal.
Changes:
- Adds consolidation-cluster support, combined switch configuration, and phase-2 state polling.
- Replaces network isolation and unclean election with ordered broker shutdowns and a single-replica restart.
- Removes the unused unclean-election helper.
| File | Description |
|---|---|
tests/kafkatest/tests/inkless/inkless_topic_switch_test.py |
Adds phase-2 test helpers and reworks stale-checkpoint recovery coverage to verify consumption across the classic prefix and diskless tail. |
Review effort: Balanced
viktorsomogyi
approved these changes
Nov 5, 2026
viktorsomogyi
deleted the
giuseppelillo/kc-721-phase-2-stale-hw-promotion
branch
November 5, 2026 13:18
gqmelo
pushed a commit
that referenced
this pull request
Nov 5, 2026
…phase 2 [KC-721] (#854) `test_switched_leader_promotion_restores_stale_hw` fails on main since #849 (KC-636). The test sets `unclean.leader.election.enable=true` right after the seal commit, and the controller now rejects that config until phase 2. The test cluster had no remote storage, so its switched partition never left phase 1. The old scenario can't happen in phase 1 anymore. There, the controller elects only ISR members, and the old test needed unclean election to promote a replica outside the ISR. In phase 2, remote storage holds the prefix and the controller can elect any live replica. A replica with a stale high watermark can still become leader there. This change reworks the test to cover that case instead of removing it. The test runs on a consolidation cluster with tiered storage on MinIO. It switches with `remote.storage.enable` in the same alter as `diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before it injects the stale high watermark checkpoint. The test stops the promoted replica last instead of isolating it with a network partition. It's then the last ISR member of the test topic and of the internal topics, so the controller elects it without unclean election when it restarts alone. It also leads `__remote_log_metadata`, which it needs to serve the prefix from remote storage. The consumer must read every record across the seal. On main, the follower consolidation fetcher repairs the stale high watermark before the promotion. The leader-side repair in `maybeReconcileSwitchedLeader` doesn't run in this test, and `testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers it. The change also removes `_run_unclean_leader_election`, which has no other callers. ## Testing - `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker on main. The logs show phase 2 at seal 5000, a broker reload with high watermark 2500, a local log start offset of 5000, and 6000 records consumed. - `make vale` passes. Assisted-by: Claude Opus 5.5
gqmelo
pushed a commit
that referenced
this pull request
Nov 5, 2026
…phase 2 [KC-721] (#854) `test_switched_leader_promotion_restores_stale_hw` fails on main since #849 (KC-636). The test sets `unclean.leader.election.enable=true` right after the seal commit, and the controller now rejects that config until phase 2. The test cluster had no remote storage, so its switched partition never left phase 1. The old scenario can't happen in phase 1 anymore. There, the controller elects only ISR members, and the old test needed unclean election to promote a replica outside the ISR. In phase 2, remote storage holds the prefix and the controller can elect any live replica. A replica with a stale high watermark can still become leader there. This change reworks the test to cover that case instead of removing it. The test runs on a consolidation cluster with tiered storage on MinIO. It switches with `remote.storage.enable` in the same alter as `diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before it injects the stale high watermark checkpoint. The test stops the promoted replica last instead of isolating it with a network partition. It's then the last ISR member of the test topic and of the internal topics, so the controller elects it without unclean election when it restarts alone. It also leads `__remote_log_metadata`, which it needs to serve the prefix from remote storage. The consumer must read every record across the seal. On main, the follower consolidation fetcher repairs the stale high watermark before the promotion. The leader-side repair in `maybeReconcileSwitchedLeader` doesn't run in this test, and `testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers it. The change also removes `_run_unclean_leader_election`, which has no other callers. ## Testing - `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker on main. The logs show phase 2 at seal 5000, a broker reload with high watermark 2500, a local log start offset of 5000, and 6000 records consumed. - `make vale` passes. Assisted-by: Claude Opus 5.5
gqmelo
pushed a commit
that referenced
this pull request
Nov 5, 2026
…phase 2 [KC-721] (#854) `test_switched_leader_promotion_restores_stale_hw` fails on main since #849 (KC-636). The test sets `unclean.leader.election.enable=true` right after the seal commit, and the controller now rejects that config until phase 2. The test cluster had no remote storage, so its switched partition never left phase 1. The old scenario can't happen in phase 1 anymore. There, the controller elects only ISR members, and the old test needed unclean election to promote a replica outside the ISR. In phase 2, remote storage holds the prefix and the controller can elect any live replica. A replica with a stale high watermark can still become leader there. This change reworks the test to cover that case instead of removing it. The test runs on a consolidation cluster with tiered storage on MinIO. It switches with `remote.storage.enable` in the same alter as `diskless.enable`, and it waits for `classicPrefixInRemoteStorage` before it injects the stale high watermark checkpoint. The test stops the promoted replica last instead of isolating it with a network partition. It's then the last ISR member of the test topic and of the internal topics, so the controller elects it without unclean election when it restarts alone. It also leads `__remote_log_metadata`, which it needs to serve the prefix from remote storage. The consumer must read every record across the seal. On main, the follower consolidation fetcher repairs the stale high watermark before the promotion. The leader-side repair in `maybeReconcileSwitchedLeader` doesn't run in this test, and `testApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeader` covers it. The change also removes `_run_unclean_leader_election`, which has no other callers. ## Testing - `test_switched_leader_promotion_restores_stale_hw` passes twice in ducker on main. The logs show phase 2 at seal 5000, a broker reload with high watermark 2500, a local log start offset of 5000, and 6000 records consumed. - `make vale` passes. Assisted-by: Claude Opus 5.5
jeqo
pushed a commit
that referenced
this pull request
Nov 6, 2026
) The 0.51 entry records the switch phase gates, the bounded consolidation prune, and their config, metric, and schema changes. The generator draft missed the new default of `inkless.consolidation.cleanup.interval.ms` and the broker and controller metrics that `metrics.rst` doesn't cover. Its `PostgresControlPlane` delta is documentation catch-up for metrics that 0.50 already registered. The session logs record the cherry-pick of #823 through #854 to `inkless-4.1`, `inkless-4.2`, and `inkless-4.3`, the conflict resolutions, and the release-branch build fixes. ## Testing - `make vale` reports no alerts. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_switched_leader_promotion_restores_stale_hwfails on main since#849 (KC-636). The test sets
unclean.leader.election.enable=truerightafter the seal commit, and the controller now rejects that config until
phase 2. The test cluster had no remote storage, so its switched partition
never left phase 1.
The old scenario can't happen in phase 1 anymore. There, the controller
elects only ISR members, and the old test needed unclean election to
promote a replica outside the ISR. In phase 2, remote storage holds the
prefix and the controller can elect any live replica. A replica with a
stale high watermark can still become leader there. This change reworks
the test to cover that case instead of removing it.
The test runs on a consolidation cluster with tiered storage on MinIO. It
switches with
remote.storage.enablein the same alter asdiskless.enable, and it waits forclassicPrefixInRemoteStoragebeforeit injects the stale high watermark checkpoint. The test stops the
promoted replica last instead of isolating it with a network partition.
It's then the last ISR member of the test topic and of the internal
topics, so the controller elects it without unclean election when it
restarts alone. It also leads
__remote_log_metadata, which it needs toserve the prefix from remote storage. The consumer must read every record
across the seal.
On main, the follower consolidation fetcher repairs the stale high
watermark before the promotion. The leader-side repair in
maybeReconcileSwitchedLeaderdoesn't run in this test, andtestApplyDeltaRestoresStaleHwmWhenSwitchedFollowerBecomesLeadercoversit.
The change also removes
_run_unclean_leader_election, which has no othercallers.
Testing
test_switched_leader_promotion_restores_stale_hwpasses twice in duckeron main. The logs show phase 2 at seal 5000, a broker reload with high
watermark 2500, a local log start offset of 5000, and 6000 records
consumed.
make valepasses.Assisted-by: Claude Opus 5.5