Repository navigation
test(inkless:control-plane): cover the availability gate across outages, restarts, and migration locks [KC-619] - #850
Merged
Conversation
8000
View session
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The tests retain a connection-order race that can cause spurious failures and untracked asynchronous restarts that can outlive teardown, risking controller shutdown and directory deletion during broker startup.
2 open issues
What changed in this PR
This PR expands cluster coverage for the control-plane availability gate introduced in #759, checking that outages and migration delays isolate diskless traffic without disrupting classic topics.
Changes:
- Tests classic traffic, offset commits, diskless request errors, broker restarts, and connection-string recovery during outages.
- Adds migration-lock tests for startup fencing and classic availability after migration timeout.
- Introduces a raw request helper that exposes diskless errors without client retries.
| File | Description |
|---|---|
core/src/test/java/kafka/server/InklessDisklessRequestClient.java |
Sends raw produce, fetch, and list-offsets requests to a selected broker. |
core/src/test/java/kafka/server/InklessControlPlaneOutageClusterTest.java |
Expands outage, restart, connection-attempt, and recovery coverage. |
core/src/test/java/kafka/server/InklessControlPlaneMigrationLockClusterTest.java |
Adds two-broker coverage for migration-lock waits and timeouts. |
Review effort: Balanced
…es, restarts, and migration locks [KC-619] The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5
gqmelo
force-pushed
the
gqmelo/kc-619-control-plane-gate-test-cases
branch
from
November 3, 2026 08:47
961c9d6 to
036b6df
Compare
gqmelo
marked this pull request as ready for review
November 3, 2026 12:38
viktorsomogyi
approved these changes
Nov 4, 2026
gqmelo
added a commit
that referenced
this pull request
Nov 5, 2026
…es, restarts, and migration locks [KC-619] (#850) The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5
gqmelo
added a commit
that referenced
this pull request
Nov 5, 2026
…es, restarts, and migration locks [KC-619] (#850) The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5
gqmelo
added a commit
that referenced
this pull request
Nov 5, 2026
On 4.2, `MetadataDelta` has no public constructor, only `MetadataDelta.Builder`. `HostedPartition.Online` and `HostedPartition.Offline` take no type parameter, `MemoryRecords` and `SimpleRecord` are in `org.apache.kafka.common.record`, and the Scala `BrokerReconfigurable` returns a Scala set. The tests added by #835, #840, #841, #850, and #852 used the main signatures. The Java `UnifiedLogTest` on 4.2 is a stub, and the log tests still live in the Scala `kafka.log.UnifiedLogTest`. The cherry-pick of #840 ports its four `forceDeleteLocalSegmentsBelow` tests to the Scala suite instead of pulling the upstream Java tests in through the conflict. Assisted-by: Claude Opus 5.5
gqmelo
added a commit
that referenced
this pull request
Nov 5, 2026
…es, restarts, and migration locks [KC-619] (#850) The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5
gqmelo
added a commit
that referenced
this pull request
Nov 5, 2026
sync(compile): adapt cherry-picked changes to the 4.1 APIs
<
8000
span class="hidden-text-expander inline">
On 4.1, `RemoteStorageFetchInfo` carries a `TopicPartition` instead of a `TopicIdPartition`, so the remote fetch callback in `ReplicaManager` looks up the topic ID with `topicIdPartition`. `LogReadResult` takes no `Errors` argument, `RetriableRemoteStorageException` doesn't exist and the reclaim floor throws `RemoteStorageException`, and `remote.log.manager.thread.pool.size` sizes the follower pool. `HostedPartition.Online` and `HostedPartition.Offline` take no type parameter, `MemoryRecords` and `SimpleRecord` are in `org.apache.kafka.common.record`, and the Scala `BrokerReconfigurable` returns a Scala set. The changes added by #831, #834, #835, #840, #841, #847, #850, and #852 used the main signatures. The Java `UnifiedLogTest` on 4.1 is a stub, and the log tests still live in the Scala `kafka.log.UnifiedLogTest`. The cherry-pick of #840 ports its four `forceDeleteLocalSegmentsBelow` tests to the Scala suite instead of pulling the upstream Java tests in through the conflict. After #847, `kafka.server.ReplicaManager` on 4.1 is too big for SpotBugs to analyze, and `:core:spotbugsMain` fails with `SKIPPED_CLASS_TOO_BIG`. The class is about 400 lines longer on 4.1 than on 4.2 and 4.3, and SpotBugs 4.9.8 fails the same way. The exclude filter now skips that pattern for `ReplicaManager` only. Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration.
The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable
KAFKA_STORAGE_ERROR. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again.The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts
migration.timeout.ms, the starting broker serves classic traffic and keeps diskless traffic gated.InklessDisklessRequestClientsends raw requests to one broker, because clients retryKAFKA_STORAGE_ERRORand would hide it.Test plan
./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTestmake vale