Repository navigation
feat(inkless:control_plane): make control plane connection config dynamic - #759
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new dynamic cluster-wide broker config (diskless.control.plane.availability) that allows an external management plane to signal control plane availability, and uses that signal to proactively gate diskless operations (control-plane calls, produce/fetch hot paths, and periodic background tasks) to reduce timeouts and make failures immediately retriable.
Changes:
- Introduces
ControlPlaneAvailability+AvailabilityGatedControlPlane, and wires them into broker/controller startup and dynamic reconfiguration. - Gates diskless produce/fetch and skips periodic diskless background jobs when the control plane is reported unavailable.
- Adds metrics and documentation for control plane availability/gated-call counts, plus unit/integration tests for gating and dynamic updates.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| storage/inkless/src/test/java/io/aiven/inkless/produce/AppendHandlerTest.java | Updates produce handler tests for availability gating and new constructor signature. |
| storage/inkless/src/test/java/io/aiven/inkless/delete/FileCleanerIntegrationTest.java | Passes ControlPlaneAvailability into SharedState initialization for integration coverage. |
| storage/inkless/src/test/java/io/aiven/inkless/control_plane/ControlPlaneAvailabilityTest.java | New unit tests for availability state handling, config parsing, and listener behavior. |
| storage/inkless/src/test/java/io/aiven/inkless/control_plane/AvailabilityGatedControlPlaneTest.java | New unit tests for gating semantics and delegate lifecycle behavior. |
| storage/inkless/src/test/java/io/aiven/inkless/consume/FetchHandlerTest.java | Updates fetch handler tests for availability gating and retriable error behavior. |
| storage/inkless/src/test/java/io/aiven/inkless/common/SharedStateTest.java | Updates SharedState.initialize calls to include control plane availability. |
| storage/inkless/src/main/java/io/aiven/inkless/produce/AppendHandler.java | Gates diskless produce when control plane is reported unavailable. |
| storage/inkless/src/main/java/io/aiven/inkless/doc/MetricsDocs.java | Adds generated docs output for control plane availability metrics. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/ControlPlaneAvailabilityMetrics.java | New Yammer/JMX metrics for reported availability and gated-call count. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/ControlPlaneAvailability.java | New availability state holder with dynamic updates, metrics, and listener hook. |
| storage/inkless/src/main/java/io/aiven/inkless/control_plane/AvailabilityGatedControlPlane.java | New wrapper that rejects calls while unavailable and lazily manages delegate lifecycle. |
| storage/inkless/src/main/java/io/aiven/inkless/consume/FetchHandler.java | Gates diskless fetch when control plane is reported unavailable. |
| storage/inkless/src/main/java/io/aiven/inkless/common/SharedState.java | Threads ControlPlaneAvailability through shared state so handlers can gate consistently. |
| server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java | Defines new config key, allowed values, default, and docs; marks it dynamically updatable. |
| docs/inkless/metrics.rst | Documents the new control plane availability metrics in Inkless docs. |
| core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala | Adds coverage for runIfControlPlaneAvailable gating helper. |
| core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala | Adds test ensuring dynamic updates propagate into ControlPlaneAvailability. |
| core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala | Adds tests for dynamic-config startup catch-up and config validation/update mode. |
| core/src/main/scala/kafka/server/SharedServer.scala | Creates and wires availability + availability-gated control plane during SharedServer start. |
| core/src/main/scala/kafka/server/ReplicaManager.scala | Skips periodic diskless tasks when gated; passes availability to fetch handler. |
| core/src/main/scala/kafka/server/KafkaConfig.scala | Exposes disklessControlPlaneAvailability on KafkaConfig. |
| core/src/main/scala/kafka/server/DynamicBrokerConfig.scala | Registers dynamic reconfigurable to update availability; adds startup catch-up logic. |
| core/src/main/scala/kafka/server/BrokerServer.scala | Passes availability into SharedState.initialize. |
Suppressed comments (1)
storage/inkless/src/test/java/io/aiven/inkless/produce/AppendHandlerTest.java:201
- This test creates an
AppendHandlerbut does not close it, and it doesn't wait for the returned future. Closing via try-with-resources and awaiting the future makes the test more robust and avoids leaking any resources managed byAppendHandler.
@Test
void availableAppendDelegatesToWriter() {
final Writer writer = Mockito.mock(Writer.class);
Mockito.when(writer.write(Mockito.any(), Mockito.any(), Mockito.any()))
.thenReturn(CompletableFuture.completedFuture(Map.of()));
final AppendHandler appendHandler = new AppendHandler(
writer, topic -> new LogConfig(Map.of()), AVAILABLE);
appendHandler.handle(Map.of(T0P0, MemoryRecords.EMPTY), RequestLocal.noCaching());
Mockito.verify(writer).write(Mockito.any(), Mockito.any(), Mockito.any());
}
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
c410814 to
be219d1
Compare
d045039 to
ac813dd
Compare
There was a problem hiding this comment.
Oops, clicked on the wrong review previously :)
ac813dd to
8c07f8e
Compare
f0f90bc to
71a9e77
Compare
71a9e77 to
80f0274
Compare
There was a problem hiding this comment.
Started reviewing it, will do a second pass soon but I wanted to share some comments iteratively so you can address them.
There was a problem hiding this comment.
Couple more comments I could think of.
e70e452 to
36fd8dd
Compare
…amic This config is made dynamic with the intent to allow a management plane to report the state of the control plane coordinator service. The config should be dynamicaly updated whenever the service becomes unavailable or available. Inkless can then use this state to avoid trying to connect to the control plane coordinator, reducing the noise in logs and metrics, and replying with retriable errors (i.e. fail fast) when appropriate. This needed changes in how the connection pools are instantiated. The new class AvailabilityGatedControlPlane instantiates a new ControlPlane on dynamically when it is first needed and closed when it is reported unavailable, so that the connection pool is closed and there are no attempts to reconnect. When marking control plane as available, a reason needs to be passed and it is included in the metrics. Currently there is only one reason (not configured), but in the future we intend to add a circuit breaker too, so being able differentiate this in metrics can be helpful. [KC-419] Assisted-by: Claude Sonnet 5 Assisted-by: Claude Opus 5.5
9c59310 to
f7498c4
Compare
…p deadline A fixed 60-second wait for the first control plane build lets a broker unfence while a schema migration is still running. Migrations such as V30 build an index on `batches`, so their duration grows with the data and can reach `migration.timeout.ms`. Until the build finishes, the broker fails diskless requests that another broker could serve. The broker and the dedicated controller now wait for the first build with `FutureUtils.waitWithLogging` and the existing `startupDeadline`, like their other startup waits. The broker waits before it asks to be unfenced. The delegate's own timeouts, including `migration.timeout.ms`, bound the build, so the wait has no timeout of its own. Assisted-by: Claude Opus 5.5
There was a problem hiding this comment.
A general AI comment that I think is relevant because it can prevent internal topics from working as expected:
Location: BrokerServer.startup, the wait immediately above lifecycleManager.setReadyToUnfence()
Issue: The wait sits before unfence, so classic partitions on that broker stay unavailable for the whole first build. Followers blocked on Flyway's advisory lock wait one second per retry, and lockRetryCount is set to migration.timeout.ms (default 300 seconds). A rolling restart leaves the other brokers up. A simultaneous restart leaves every broker fenced until its own attempt settles, including brokers that are only waiting for the lock.
Recommendation: If classic traffic must stay up, unfence on the startup deadline and keep failing diskless calls until availability() is AVAILABLE. If the full fence is the intended cost, state that in the commit message so operators know a control-plane migration stalls classic produce and fetch on that node.
But that is intended as per #759 (comment) |
|
@gqmelo looking at it again I don't think that unavailability is entirely intended, however I agree with the rest of your comments. If it existed before, then we don't have to fix it now. |
There was a problem hiding this comment.
I think it looks good, however before merging we should finish the discussion on slack. As a follow-up, we could implement further tests to ensure the feature's stability:
- While the control plane is out of service, classic produce, classic consume, and consumer-group offset commits succeed, and diskless produce, fetch, and list-offsets fail with a retriable error until the connection string is restored.
- A broker that restarts while the control plane is out of service comes up, serves classic traffic, keeps failing diskless traffic, and serves diskless traffic again after the connection string is restored.
- When the control plane is unreachable, diskless produce fails immediately without a new connection per request, and succeeds again once the connection string points at a reachable database.
- While a control-plane migration holds its advisory lock, brokers that are already running keep serving classic traffic, the starting broker stays fenced, and it accepts diskless produce after the lock is released.
- When a migration outlasts its timeout, the starting broker serves classic traffic and keeps diskless traffic gated.
- Emptying the dynamic connection string stops diskless produce, and removing that override resumes it through the static connection string.
There was a problem hiding this comment.
🟡 Changes recommended
Outage handling still disrupts classic-topic administration and contains availability and pool-lifecycle defects.
5 open findings
Preserve empty diskless topic creation no-op before delegate resolution · New Preserve empty diskless topic deletion no-op before delegate resolution · New Capture unavailable reason once to avoid null metrics failures · New Close retired delegates before replacement I/O begins · New Wait for initial build completion before returning attempt status · New
5 resolved since last review
This warning executes once per rejected produce request. During a control-plane outage, normal… A failed factory attempt can leak a partially initialized delegate on every retry.… This drops every gated call while the state isUNKNOWN, because that state has a null reason.… This starts the first build before the Raft manager exists and before either role installs its… A combined broker/controller node registers the same shared gate here and through the broker path…
🧠 Review effort: Balanced
| * {@link State#UNKNOWN}, where there is no reason to report yet. | ||
| */ | ||
| public void recordGatedCall() { | ||
| metrics.recordGatedCall(unavailableReason == null ? UnavailableReason.NOT_READY : unavailableReason); |
| final ControlPlane stale = current.delegate(); | ||
| final ControlPlane created; | ||
| try { | ||
| created = delegateFactory.apply(configSupplier.get()); |
|
|
||
| private void awaitFactoryInvocations(final int expected) { | ||
| await().atMost(AWAIT_SECONDS, TimeUnit.SECONDS) | ||
| .until(() -> factoryInvocations.get() == expected); |
|
@gqmelo I've merged this but please take a look at the copilot review points and raise follow-up tickets if they're relevant (and please add a reason to the comments if they aren't). |
…amic (#759) This config is made dynamic with the intent to allow a management plane to report the state of the control plane coordinator service. The config should be dynamicaly updated whenever the service becomes unavailable or available. Inkless can then use this state to avoid trying to connect to the control plane coordinator, reducing the noise in logs and metrics, and replying with retriable errors (i.e. fail fast) when appropriate. This needed changes in how the connection pools are instantiated. The new class AvailabilityGatedControlPlane instantiates a new ControlPlane on dynamically when it is first needed and closed when it is reported unavailable, so that the connection pool is closed and there are no attempts to reconnect. When marking control plane as available, a reason needs to be passed and it is included in the metrics. Currently there is only one reason (not configured), but in the future we intend to add a circuit breaker too, so being able differentiate this in metrics can be helpful. [KC-419] Assisted-by: Claude Sonnet 5 Assisted-by: Claude Opus 5.5 * fix(inkless:control_plane): wait for the first build until the startup deadline A fixed 60-second wait for the first control plane build lets a broker unfence while a schema migration is still running. Migrations such as V30 build an index on `batches`, so their duration grows with the data and can reach `migration.timeout.ms`. Until the build finishes, the broker fails diskless requests that another broker could serve. The broker and the dedicated controller now wait for the first build with `FutureUtils.waitWithLogging` and the existing `startupDeadline`, like their other startup waits. The broker waits before it asks to be unfenced. The delegate's own timeouts, including `migration.timeout.ms`, bound the build, so the wait has no timeout of its own. Assisted-by: Claude Opus 5.5
…amic (#759) This config is made dynamic with the intent to allow a management plane to report the state of the control plane coordinator service. The config should be dynamicaly updated whenever the service becomes unavailable or available. Inkless can then use this state to avoid trying to connect to the control plane coordinator, reducing the noise in logs and metrics, and replying with retriable errors (i.e. fail fast) when appropriate. This needed changes in how the connection pools are instantiated. The new class AvailabilityGatedControlPlane instantiates a new ControlPlane on dynamically when it is first needed and closed when it is reported unavailable, so that the connection pool is closed and there are no attempts to reconnect. When marking control plane as available, a reason needs to be passed and it is included in the metrics. Currently there is only one reason (not configured), but in the future we intend to add a circuit breaker too, so being able differentiate this in metrics can be helpful. [KC-419] Assisted-by: Claude Sonnet 5 Assisted-by: Claude Opus 5.5 * fix(inkless:control_plane): wait for the first build until the startup deadline A fixed 60-second wait for the first control plane build lets a broker unfence while a schema migration is still running. Migrations such as V30 build an index on `batches`, so their duration grows with the data and can reach `migration.timeout.ms`. Until the build finishes, the broker fails diskless requests that another broker could serve. The broker and the dedicated controller now wait for the first build with `FutureUtils.waitWithLogging` and the existing `startupDeadline`, like their other startup waits. The broker waits before it asks to be unfenced. The delegate's own timeouts, including `migration.timeout.ms`, bound the build, so the wait has no timeout of its own. Assisted-by: Claude Opus 5.5
…amic (#759) This config is made dynamic with the intent to allow a management plane to report the state of the control plane coordinator service. The config should be dynamicaly updated whenever the service becomes unavailable or available. Inkless can then use this state to avoid trying to connect to the control plane coordinator, reducing the noise in logs and metrics, and replying with retriable errors (i.e. fail fast) when appropriate. This needed changes in how the connection pools are instantiated. The new class AvailabilityGatedControlPlane instantiates a new ControlPlane on dynamically when it is first needed and closed when it is reported unavailable, so that the connection pool is closed and there are no attempts to reconnect. When marking control plane as available, a reason needs to be passed and it is included in the metrics. Currently there is only one reason (not configured), but in the future we intend to add a circuit breaker too, so being able differentiate this in metrics can be helpful. [KC-419] Assisted-by: Claude Sonnet 5 Assisted-by: Claude Opus 5.5 * fix(inkless:control_plane): wait for the first build until the startup deadline A fixed 60-second wait for the first control plane build lets a broker unfence while a schema migration is still running. Migrations such as V30 build an index on `batches`, so their duration grows with the data and can reach `migration.timeout.ms`. Until the build finishes, the broker fails diskless requests that another broker could serve. The broker and the dedicated controller now wait for the first build with `FutureUtils.waitWithLogging` and the existing `startupDeadline`, like their other startup waits. The broker waits before it asks to be unfenced. The delegate's own timeouts, including `migration.timeout.ms`, bound the build, so the wait has no timeout of its own. Assisted-by: Claude Opus 5.5
…pic requests [KC-621] `ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down to their diskless topics before calling the control plane. It still called the control plane when nothing was left. Since #759, the gated control plane rejects every call while it's unavailable, so classic-only requests failed during an outage: creation returned `UNKNOWN_SERVER_ERROR` after KRaft created the topic, and deletion failed before KRaft deleted anything. The guard belongs in `ControllerApis` because that's where the diskless filtering happens. With no diskless topics, there's nothing to send to the control plane. `createPartitions` already guards the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pic requests [KC-621] `ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down to their diskless topics before calling the control plane. It still called the control plane when nothing was left. Since #759, the gated control plane rejects every call while it's unavailable, so classic-only requests failed during an outage: creation returned `UNKNOWN_SERVER_ERROR` after KRaft created the topic, and deletion failed before KRaft deleted anything. The guard belongs in `ControllerApis` because that's where the diskless filtering happens. With no diskless topics, there's nothing to send to the control plane. `createPartitions` already guards the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pic requests [KC-621] `ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down to their diskless topics before it calls the control plane, but it made the call even when no diskless topics were left. Since #759, the gated control plane rejects every call while it's unavailable. During an outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after KRaft had already created the topic. A classic-only deletion failed before KRaft deleted anything. `ControllerApis` now calls the control plane only when the filtered set isn't empty. The guard belongs in `ControllerApis` because the diskless filtering happens there. If a request has no diskless topics, there's nothing to send to the control plane. `createPartitions` already guards its call the same way. Assisted-by: Claude Opus 5.5
…pic requests [KC-621] (#832) `ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down to their diskless topics before it calls the control plane, but it made the call even when no diskless topics were left. Since #759, the gated control plane rejects every call while it's unavailable. During an outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after KRaft had already created the topic. A classic-only deletion failed before KRaft deleted anything. `ControllerApis` now calls the control plane only when the filtered set isn't empty. The guard belongs in `ControllerApis` because the diskless filtering happens there. If a request has no diskless topics, there's nothing to send to the control plane. `createPartitions` already guards its call the same way. Assisted-by: Claude Opus 5.5
…es, restarts, and migration locks [KC-619] The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5
…es, restarts, and migration locks [KC-619] (#850) The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5
…pic requests [KC-621] (#832) `ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down to their diskless topics before it calls the control plane, but it made the call even when no diskless topics were left. Since #759, the gated control plane rejects every call while it's unavailable. During an outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after KRaft had already created the topic. A classic-only deletion failed before KRaft deleted anything. `ControllerApis` now calls the control plane only when the filtered set isn't empty. The guard belongs in `ControllerApis` because the diskless filtering happens there. If a request has no diskless topics, there's nothing to send to the control plane. `createPartitions` already guards its call the same way. Assisted-by: Claude Opus 5.5
…es, restarts, and migration locks [KC-619] (#850) The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5
…pic requests [KC-621] (#832) `ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down to their diskless topics before it calls the control plane, but it made the call even when no diskless topics were left. Since #759, the gated control plane rejects every call while it's unavailable. During an outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after KRaft had already created the topic. A classic-only deletion failed before KRaft deleted anything. `ControllerApis` now calls the control plane only when the filtered set isn't empty. The guard belongs in `ControllerApis` because the diskless filtering happens there. If a request has no diskless topics, there's nothing to send to the control plane. `createPartitions` already guards its call the same way. Assisted-by: Claude Opus 5.5
…es, restarts, and migration locks [KC-619] (#850) The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5
…pic requests [KC-621] (#832) `ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down to their diskless topics before it calls the control plane, but it made the call even when no diskless topics were left. Since #759, the gated control plane rejects every call while it's unavailable. During an outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after KRaft had already created the topic. A classic-only deletion failed before KRaft deleted anything. `ControllerApis` now calls the control plane only when the filtered set isn't empty. The guard belongs in `ControllerApis` because the diskless filtering happens there. If a request has no diskless topics, there's nothing to send to the control plane. `createPartitions` already guards its call the same way. Assisted-by: Claude Opus 5.5
…es, restarts, and migration locks [KC-619] (#850) The control-plane availability gate from #759 had cluster coverage only for diskless produce and topic management while the connection string is empty. These tests pin the rest of the behavior operators depend on during a control-plane outage or a slow schema migration. The outage tests check that classic produce, consume, and consumer-group offset commits keep working while diskless produce, fetch, and list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also cover a broker restart during the outage, an unreachable database that must not get a connection per produce request, and removing the emptied dynamic override so the static connection string applies again. The migration lock tests hold Flyway's advisory lock from a separate session and restart one of two brokers. While the lock is held, the running broker keeps serving classic traffic and the starting broker stays fenced until the lock is released. When the migration outlasts `migration.timeout.ms`, the starting broker serves classic traffic and keeps diskless traffic gated. `InklessDisklessRequestClient` sends raw requests to one broker, because clients retry `KAFKA_STORAGE_ERROR` and would hide it. ## Test plan - `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest` - `make vale` Assisted-by: Claude Opus 5.5

This config is made dynamic with the intent to allow a management plane to report the state of the control plane coordinator service.
The config should be dynamicaly updated whenever the service becomes unavailable or available. Inkless can then use this state to avoid trying to connect to the control plane coordinator, reducing the noise in logs and metrics, and replying with retriable errors (i.e. fail fast) when appropriate.
This needed changes in how the connection pools are instantiated.
The new class AvailabilityGatedControlPlane instantiates a new ControlPlane on dynamically when it is first needed and closed when it is reported unavailable, so that the connection pool is closed and there are no attempts to reconnect.
When marking control plane as available, a reason needs to be passed and it is included in the metrics. Currently there is only one reason (not configured), but in the future we intend to add a circuit breaker too, so being able differentiate this in metrics can be helpful.
KC-419