8000 feat(inkless:control_plane): make control plane connection config dynamic by gqmelo · Pull Request #759 · aiven/inkless · GitHub
[go: up one dir, main page]

Skip to content

feat(inkless:control_plane): make control plane connection config dynamic - #759

Merged
viktorsomogyi merged 2 commits into
mainfrom
gqmelo/control-plane-availability
Oct 16, 2026
Merged

viktorsomogyi merged 2 commits into
mainfrom
gqmelo/control-plane-availability

Conversation

@gqmelo
@gqmelo gqmelo commented Aug 18, 2026 •
Copy link
Copy Markdown
Collaborator

This config is made dynamic with the intent to allow a management plane to report the state of the control plane coordinator service.

The config should be dynamicaly updated whenever the service becomes unavailable or available. Inkless can then use this state to avoid trying to connect to the control plane coordinator, reducing the noise in logs and metrics, and replying with retriable errors (i.e. fail fast) when appropriate.

This needed changes in how the connection pools are instantiated.
The new class AvailabilityGatedControlPlane instantiates a new ControlPlane on dynamically when it is first needed and closed when it is reported unavailable, so that the connection pool is closed and there are no attempts to reconnect.

When marking control plane as available, a reason needs to be passed and it is included in the metrics. Currently there is only one reason (not configured), but in the future we intend to add a circuit breaker too, so being able differentiate this in metrics can be helpful.

KC-419

Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new dynamic cluster-wide broker config (diskless.control.plane.availability) that allows an external management plane to signal control plane availability, and uses that signal to proactively gate diskless operations (control-plane calls, produce/fetch hot paths, and periodic background tasks) to reduce timeouts and make failures immediately retriable.

Changes:

  • Introduces ControlPlaneAvailability + AvailabilityGatedControlPlane, and wires them into broker/controller startup and dynamic reconfiguration.
  • Gates diskless produce/fetch and skips periodic diskless background jobs when the control plane is reported unavailable.
  • Adds metrics and documentation for control plane availability/gated-call counts, plus unit/integration tests for gating and dynamic updates.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
storage/inkless/src/test/java/io/aiven/inkless/produce/AppendHandlerTest.java Updates produce handler tests for availability gating and new constructor signature.
storage/inkless/src/test/java/io/aiven/inkless/delete/FileCleanerIntegrationTest.java Passes ControlPlaneAvailability into SharedState initialization for integration coverage.
storage/inkless/src/test/java/io/aiven/inkless/control_plane/ControlPlaneAvailabilityTest.java New unit tests for availability state handling, config parsing, and listener behavior.
storage/inkless/src/test/java/io/aiven/inkless/control_plane/AvailabilityGatedControlPlaneTest.java New unit tests for gating semantics and delegate lifecycle behavior.
storage/inkless/src/test/java/io/aiven/inkless/consume/FetchHandlerTest.java Updates fetch handler tests for availability gating and retriable error behavior.
storage/inkless/src/test/java/io/aiven/inkless/common/SharedStateTest.java Updates SharedState.initialize calls to include control plane availability.
storage/inkless/src/main/java/io/aiven/inkless/produce/AppendHandler.java Gates diskless produce when control plane is reported unavailable.
storage/inkless/src/main/java/io/aiven/inkless/doc/MetricsDocs.java Adds generated docs output for control plane availability metrics.
storage/inkless/src/main/java/io/aiven/inkless/control_plane/ControlPlaneAvailabilityMetrics.java New Yammer/JMX metrics for reported availability and gated-call count.
storage/inkless/src/main/java/io/aiven/inkless/control_plane/ControlPlaneAvailability.java New availability state holder with dynamic updates, metrics, and listener hook.
storage/inkless/src/main/java/io/aiven/inkless/control_plane/AvailabilityGatedControlPlane.java New wrapper that rejects calls while unavailable and lazily manages delegate lifecycle.
storage/inkless/src/main/java/io/aiven/inkless/consume/FetchHandler.java Gates diskless fetch when control plane is reported unavailable.
storage/inkless/src/main/java/io/aiven/inkless/common/SharedState.java Threads ControlPlaneAvailability through shared state so handlers can gate consistently.
server-common/src/main/java/org/apache/kafka/server/config/ServerConfigs.java Defines new config key, allowed values, default, and docs; marks it dynamically updatable.
docs/inkless/metrics.rst Documents the new control plane availability metrics in Inkless docs.
core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala Adds coverage for runIfControlPlaneAvailable gating helper.
core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala Adds test ensuring dynamic updates propagate into ControlPlaneAvailability.
core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala Adds tests for dynamic-config startup catch-up and config validation/update mode.
core/src/main/scala/kafka/server/SharedServer.scala Creates and wires availability + availability-gated control plane during SharedServer start.
core/src/main/scala/kafka/server/ReplicaManager.scala Skips periodic diskless tasks when gated; passes availability to fetch handler.
core/src/main/scala/kafka/server/KafkaConfig.scala Exposes disklessControlPlaneAvailability on KafkaConfig.
core/src/main/scala/kafka/server/DynamicBrokerConfig.scala Registers dynamic reconfigurable to update availability; adds startup catch-up logic.
core/src/main/scala/kafka/server/BrokerServer.scala Passes availability into SharedState.initialize.
Suppressed comments (1)

storage/inkless/src/test/java/io/aiven/inkless/produce/AppendHandlerTest.java:201

  • This test creates an AppendHandler but does not close it, and it doesn't wait for the returned future. Closing via try-with-resources and awaiting the future makes the test more robust and avoids leaking any resources managed by AppendHandler.
    @Test
    void availableAppendDelegatesToWriter() {
        final Writer writer = Mockito.mock(Writer.class);
        Mockito.when(writer.write(Mockito.any(), Mockito.any(), Mockito.any()))
            .thenReturn(CompletableFuture.completedFuture(Map.of()));
        final AppendHandler appendHandler = new AppendHandler(
            writer, topic -> new LogConfig(Map.of()), AVAILABLE);

        appendHandler.handle(Map.of(T0P0, MemoryRecords.EMPTY), RequestLocal.noCaching());

        Mockito.verify(writer).write(Mockito.any(), Mockito.any(), Mockito.any());
    }

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@gqmelo
gqmelo force-pushed the gqmelo/control-plane-availability branch 4 times, most recently from c410814 to be219d1 Compare August 20, 2026 08:39
@gqmelo
gqmelo marked this pull request as ready for review August 20, 2026 09:55
@gqmelo
gqmelo marked this pull request as draft September 2, 2026 08:34
@gqmelo
gqmelo force-pushed the gqmelo/control-plane-availability branch 2 times, most recently from d045039 to ac813dd Compare September 3, 2026 15:42
@viktorsomogyi viktorsomogyi left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, clicked on the wrong review previously :)

@gqmelo
gqmelo force-pushed the gqmelo/control-plane-availability branch from ac813dd to 8c07f8e Compare September 7, 2026 13:55
@gqmelo
gqmelo force-pushed the gqmelo/control-plane-availability branch 2 times, most recently from f0f90bc to 71a9e77 Compare September 7, 2026 14:40
@gqmelo
gqmelo marked this pull request as ready for review September 8, 2026 06:39
@gqmelo
gqmelo marked this pull request as draft September 8, 2026 06:39
@gqmelo gqmelo changed the title feat(inkless:control_plane): add dynamic diskless.control.plane.availability config feat(inkless:control_plane): make control plane connection config dynamic Sep 8, 2026
@gqmelo
gqmelo requested a review from viktorsomogyi September 8, 2026 06:52
@gqmelo
gqmelo force-pushed the gqmelo/control-plane-availability branch from 71a9e77 to 80f0274 Compare September 9, 2026 10:25
@gqmelo
gqmelo marked this pull request as ready for review September 9, 2026 12:32
@viktorsomogyi viktorsomogyi left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Started reviewing it, will do a second pass soon but I wanted to share some comments iteratively so you can address them.

@viktorsomogyi viktorsomogyi left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple more comments I could think of.

@gqmelo
gqmelo force-pushed the gqmelo/control-plane-availability branch 4 times, most recently from e70e452 to 36fd8dd Compare September 14, 2026 15:00
…amic

This config is made dynamic with the intent to allow a management
plane to report the state of the control plane coordinator service.

The config should be dynamicaly updated whenever the service becomes
unavailable or available. Inkless can then use this state to avoid
trying to connect to the control plane coordinator, reducing the noise
in logs and metrics, and replying with retriable errors (i.e. fail fast)
when appropriate.

This needed changes in how the connection pools are instantiated.
The new class AvailabilityGatedControlPlane instantiates a new
ControlPlane on dynamically when it is first needed and closed
when it is reported unavailable, so that the connection pool is
closed and there are no attempts to reconnect.

When marking control plane as available, a reason needs to be
passed and it is included in the metrics. Currently there is only
one reason (not configured), but in the future we intend to add
a circuit breaker too, so being able differentiate this in metrics
can be helpful.

[KC-419]

Assisted-by: Claude Sonnet 5
Assisted-by: Claude Opus 5.5
@gqmelo
gqmelo force-pushed the gqmelo/control-plane-availability branch from 9c59310 to f7498c4 Compare October 13, 2026 10:22
…p deadline

A fixed 60-second wait for the first control plane build lets a broker
unfence while a schema migration is still running. Migrations such as
V30 build an index on `batches`, so their duration grows with the data
and can reach `migration.timeout.ms`. Until the build finishes, the
broker fails diskless requests that another broker could serve.

The broker and the dedicated controller now wait for the first build
with `FutureUtils.waitWithLogging` and the existing `startupDeadline`,
like their other startup waits. The broker waits before it asks to be
unfenced. The delegate's own timeouts, including `migration.timeout.ms`,
bound the build, so the wait has no timeout of its own.

Assisted-by: Claude Opus 5.5
@viktorsomogyi viktorsomogyi left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A general AI comment that I think is relevant because it can prevent internal topics from working as expected:

Location: BrokerServer.startup, the wait immediately above lifecycleManager.setReadyToUnfence()
Issue: The wait sits before unfence, so classic partitions on that broker stay unavailable for the whole first build. Followers blocked on Flyway's advisory lock wait one second per retry, and lockRetryCount is set to migration.timeout.ms (default 300 seconds). A rolling restart leaves the other brokers up. A simultaneous restart leaves every broker fenced until its own attempt settles, including brokers that are only waiting for the lock.
Recommendation: If classic traffic must stay up, unfence on the startup deadline and keep failing diskless calls until availability() is AVAILABLE. If the full fence is the intended cost, state that in the commit message so operators know a control-plane migration stalls classic produce and fetch on that node.

@gqmelo
gqmelo commented Oct 15, 2026
Copy link
Copy Markdown
Collaborator Author

Location: BrokerServer.startup, the wait immediately above lifecycleManager.setReadyToUnfence()
Issue: The wait sits before unfence, so classic partitions on that broker stay unavailable for the whole first build.

But that is intended as per #759 (comment)
Or is there any other issue I'm missing?
Also, that happens today (I think it is actually worse because failing to start the control plane makes the kafka process to exit).

@viktorsomogyi
viktorsomogyi commented Oct 15, 2026 •
Copy link
Copy Markdown
Contributor

@gqmelo looking at it again I don't think that unavailability is entirely intended, however I agree with the rest of your comments. If it existed before, then we don't have to fix it now.

@viktorsomogyi viktorsomogyi left a comment •
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it looks good, however before merging we should finish the discussion on slack. As a follow-up, we could implement further tests to ensure the feature's stability:

  • While the control plane is out of service, classic produce, classic consume, and consumer-group offset commits succeed, and diskless produce, fetch, and list-offsets fail with a retriable error until the connection string is restored.
  • A broker that restarts while the control plane is out of service comes up, serves classic traffic, keeps failing diskless traffic, and serves diskless traffic again after the connection string is restored.
  • When the control plane is unreachable, diskless produce fails immediately without a new connection per request, and succeeds again once the connection string points at a reachable database.
  • While a control-plane migration holds its advisory lock, brokers that are already running keep serving classic traffic, the starting broker stays fenced, and it accepts diskless produce after the lock is released.
  • When a migration outlasts its timeout, the starting broker serves classic traffic and keeps diskless traffic gated.
  • Emptying the dynamic connection string stops diskless produce, and removing that override resumes it through the static connection string.

Copilot AI left a comment
Copy link
Copy Markdown
Contributor

* {@link State#UNKNOWN}, where there is no reason to report yet.
*/
public void recordGatedCall() {
metrics.recordGatedCall(unavailableReason == null ? UnavailableReason.NOT_READY : unavailableReason);
final ControlPlane stale = current.delegate();
final ControlPlane created;
try {
created = delegateFactory.apply(configSupplier.get());

private void awaitFactoryInvocations(final int expected) {
await().atMost(AWAIT_SECONDS, TimeUnit.SECONDS)
.until(() -> factoryInvocations.get() == expected);
@viktorsomogyi
viktorsomogyi merged commit 3649de7 into main Oct 16, 2026
18 checks passed
@viktorsomogyi
viktorsomogyi deleted the gqmelo/control-plane-availability branch October 16, 2026 08:48
@viktorsomogyi
viktorsomogyi commented Oct 16, 2026 •
Copy link
Copy Markdown
Contributor

@gqmelo I've merged this but please take a look at the copilot review points and raise follow-up tickets if they're relevant (and please add a reason to the comments if they aren't).

jeqo pushed a commit that referenced this pull request Oct 19, 2026
…amic (#759)

This config is made dynamic with the intent to allow a management
plane to report the state of the control plane coordinator service.

The config should be dynamicaly updated whenever the service becomes
unavailable or available. Inkless can then use this state to avoid
trying to connect to the control plane coordinator, reducing the noise
in logs and metrics, and replying with retriable errors (i.e. fail fast)
when appropriate.

This needed changes in how the connection pools are instantiated.
The new class AvailabilityGatedControlPlane instantiates a new
ControlPlane on dynamically when it is first needed and closed
when it is reported unavailable, so that the connection pool is
closed and there are no attempts to reconnect.

When marking control plane as available, a reason needs to be
passed and it is included in the metrics. Currently there is only
one reason (not configured), but in the future we intend to add
a circuit breaker too, so being able differentiate this in metrics
can be helpful.

[KC-419]

Assisted-by: Claude Sonnet 5
Assisted-by: Claude Opus 5.5

* fix(inkless:control_plane): wait for the first build until the startup deadline

A fixed 60-second wait for the first control plane build lets a broker
unfence while a schema migration is still running. Migrations such as
V30 build an index on `batches`, so their duration grows with the data
and can reach `migration.timeout.ms`. Until the build finishes, the
broker fails diskless requests that another broker could serve.

The broker and the dedicated controller now wait for the first build
with `FutureUtils.waitWithLogging` and the existing `startupDeadline`,
like their other startup waits. The broker waits before it asks to be
unfenced. The delegate's own timeouts, including `migration.timeout.ms`,
bound the build, so the wait has no timeout of its own.

Assisted-by: Claude Opus 5.5
jeqo pushed a commit that referenced this pull request Oct 19, 2026
…amic (#759)

This config is made dynamic with the intent to allow a management
plane to report the state of the control plane coordinator service.

The config should be dynamicaly updated whenever the service becomes
unavailable or available. Inkless can then use this state to avoid
trying to connect to the control plane coordinator, reducing the noise
in logs and metrics, and replying with retriable errors (i.e. fail fast)
when appropriate.

This needed changes in how the connection pools are instantiated.
The new class AvailabilityGatedControlPlane instantiates a new
ControlPlane on dynamically when it is first needed and closed
when it is reported unavailable, so that the connection pool is
closed and there are no attempts to reconnect.

When marking control plane as available, a reason needs to be
passed and it is included in the metrics. Currently there is only
one reason (not configured), but in the future we intend to add
a circuit breaker too, so being able differentiate this in metrics
can be helpful.

[KC-419]

Assisted-by: Claude Sonnet 5
Assisted-by: Claude Opus 5.5

* fix(inkless:control_plane): wait for the first build until the startup deadline

A fixed 60-second wait for the first control plane build lets a broker
unfence while a schema migration is still running. Migrations such as
V30 build an index on `batches`, so their duration grows with the data
and can reach `migration.timeout.ms`. Until the build finishes, the
broker fails diskless requests that another broker could serve.

The broker and the dedicated controller now wait for the first build
with `FutureUtils.waitWithLogging` and the existing `startupDeadline`,
like their other startup waits. The broker waits before it asks to be
unfenced. The delegate's own timeouts, including `migration.timeout.ms`,
bound the build, so the wait has no timeout of its own.

Assisted-by: Claude Opus 5.5
jeqo pushed a commit that referenced this pull request Oct 19, 2026
…amic (#759)

This config is made dynamic with the intent to allow a management
plane to report the state of the control plane coordinator service.

The config should be dynamicaly updated whenever the service becomes
unavailable or available. Inkless can then use this state to avoid
trying to connect to the control plane coordinator, reducing the noise
in logs and metrics, and replying with retriable errors (i.e. fail fast)
when appropriate.

This needed changes in how the connection pools are instantiated.
The new class AvailabilityGatedControlPlane instantiates a new
ControlPlane on dynamically when it is first needed and closed
when it is reported unavailable, so that the connection pool is
closed and there are no attempts to reconnect.

When marking control plane as available, a reason needs to be
passed and it is included in the metrics. Currently there is only
one reason (not configured), but in the future we intend to add
a circuit breaker too, so being able differentiate this in metrics
can be helpful.

[KC-419]

Assisted-by: Claude Sonnet 5
Assisted-by: Claude Opus 5.5

* fix(inkless:control_plane): wait for the first build until the startup deadline

A fixed 60-second wait for the first control plane build lets a broker
unfence while a schema migration is still running. Migrations such as
V30 build an index on `batches`, so their duration grows with the data
and can reach `migration.timeout.ms`. Until the build finishes, the
broker fails diskless requests that another broker could serve.

The broker and the dedicated controller now wait for the first build
with `FutureUtils.waitWithLogging` and the existing `startupDeadline`,
like their other startup waits. The broker waits before it asks to be
unfenced. The delegate's own timeouts, including `migration.timeout.ms`,
bound the build, so the wait has no timeout of its own.

Assisted-by: Claude Opus 5.5
gqmelo added a commit that referenced this pull request Oct 20, 2026
…pic requests [KC-621]

`ControllerApis` filters `CreateTopics` and `DeleteTopics` requests
down to their diskless topics before calling the control plane.

It still called the control plane when nothing was left.
Since #759, the gated control
plane rejects every call while it's unavailable, so classic-only
requests failed during an outage: creation returned
`UNKNOWN_SERVER_ERROR` after KRaft created the topic, and deletion failed
before KRaft deleted anything.

The guard belongs in `ControllerApis` because that's where the diskless
filtering happens. With no diskless topics, there's nothing to send to
the control plane. `createPartitions` already guards the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gqmelo added a commit that referenced this pull request Oct 20, 2026
…pic requests [KC-621]

`ControllerApis` filters `CreateTopics` and `DeleteTopics` requests
down to their diskless topics before calling the control plane.

It still called the control plane when nothing was left.
Since #759, the gated control
plane rejects every call while it's unavailable, so classic-only
requests failed during an outage: creation returned
`UNKNOWN_SERVER_ERROR` after KRaft created the topic, and deletion failed
before KRaft deleted anything.

The guard belongs in `ControllerApis` because that's where the diskless
filtering happens. With no diskless topics, there's nothing to send to
the control plane. `createPartitions` already guards the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gqmelo added a commit that referenced this pull request Oct 20, 2026
…pic requests [KC-621]

`ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down
to their diskless topics before it calls the control plane, but it made
the call even when no diskless topics were left. Since #759, the gated
control plane rejects every call while it's unavailable. During an
outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after
KRaft had already created the topic. A classic-only deletion failed
before KRaft deleted anything.

`ControllerApis` now calls the control plane only when the filtered set
isn't empty. The guard belongs in `ControllerApis` because the diskless
filtering happens there. If a request has no diskless topics, there's
nothing to send to the control plane. `createPartitions` already guards
its call the same way.

Assisted-by: Claude Opus 5.5
jeqo pushed a commit that referenced this pull request Oct 21, 2026
…pic requests [KC-621] (#832)

`ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down
to their diskless topics before it calls the control plane, but it made
the call even when no diskless topics were left. Since #759, the gated
control plane rejects every call while it's unavailable. During an
outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after
KRaft had already created the topic. A classic-only deletion failed
before KRaft deleted anything.

`ControllerApis` now calls the control plane only when the filtered set
isn't empty. The guard belongs in `ControllerApis` because the diskless
filtering happens there. If a request has no diskless topics, there's
nothing to send to the control plane. `createPartitions` already guards
its call the same way.

Assisted-by: Claude Opus 5.5
gqmelo added a commit that referenced this pull request Nov 3, 2026
…es, restarts, and migration locks [KC-619]

The control-plane availability gate from #759 had cluster coverage only
for diskless produce and topic management while the connection string
is empty. These tests pin the rest of the behavior operators depend on
during a control-plane outage or a slow schema migration.

The outage tests check that classic produce, consume, and consumer-group
offset commits keep working while diskless produce, fetch, and
list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also
cover a broker restart during the outage, an unreachable database that
must not get a connection per produce request, and removing the emptied
dynamic override so the static connection string applies again.

The migration lock tests hold Flyway's advisory lock from a separate
session and restart one of two brokers. While the lock is held, the
running broker keeps serving classic traffic and the starting broker
stays fenced until the lock is released. When the migration outlasts
`migration.timeout.ms`, the starting broker serves classic traffic and
keeps diskless traffic gated.

`InklessDisklessRequestClient` sends raw requests to one broker, because
clients retry `KAFKA_STORAGE_ERROR` and would hide it.

## Test plan

- `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest`
- `make vale`

Assisted-by: Claude Opus 5.5
viktorsomogyi pushed a commit that referenced this pull request Nov 4, 2026
…es, restarts, and migration locks [KC-619] (#850)

The control-plane availability gate from #759 had cluster coverage only
for diskless produce and topic management while the connection string
is empty. These tests pin the rest of the behavior operators depend on
during a control-plane outage or a slow schema migration.

The outage tests check that classic produce, consume, and consumer-group
offset commits keep working while diskless produce, fetch, and
list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also
cover a broker restart during the outage, an unreachable database that
must not get a connection per produce request, and removing the emptied
dynamic override so the static connection string applies again.

The migration lock tests hold Flyway's advisory lock from a separate
session and restart one of two brokers. While the lock is held, the
running broker keeps serving classic traffic and the starting broker
stays fenced until the lock is released. When the migration outlasts
`migration.timeout.ms`, the starting broker serves classic traffic and
keeps diskless traffic gated.

`InklessDisklessRequestClient` sends raw requests to one broker, because
clients retry `KAFKA_STORAGE_ERROR` and would hide it.

## Test plan

- `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest`
- `make vale`

Assisted-by: Claude Opus 5.5
gqmelo added a commit that referenced this pull request Nov 5, 2026
…pic requests [KC-621] (#832)

`ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down
to their diskless topics before it calls the control plane, but it made
the call even when no diskless topics were left. Since #759, the gated
control plane rejects every call while it's unavailable. During an
outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after
KRaft had already created the topic. A classic-only deletion failed
before KRaft deleted anything.

`ControllerApis` now calls the control plane only when the filtered set
isn't empty. The guard belongs in `ControllerApis` because the diskless
filtering happens there. If a request has no diskless topics, there's
nothing to send to the control plane. `createPartitions` already guards
its call the same way.

Assisted-by: Claude Opus 5.5
gqmelo added a commit that referenced this pull request Nov 5, 2026
…es, restarts, and migration locks [KC-619] (#850)

The control-plane availability gate from #759 had cluster coverage only
for diskless produce and topic management while the connection string
is empty. These tests pin the rest of the behavior operators depend on
during a control-plane outage or a slow schema migration.

The outage tests check that classic produce, consume, and consumer-group
offset commits keep working while diskless produce, fetch, and
list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also
cover a broker restart during the outage, an unreachable database that
must not get a connection per produce request, and removing the emptied
dynamic override so the static connection string applies again.

The migration lock tests hold Flyway's advisory lock from a separate
session and restart one of two brokers. While the lock is held, the
running broker keeps serving classic traffic and the starting broker
stays fenced until the lock is released. When the migration outlasts
`migration.timeout.ms`, the starting broker serves classic traffic and
keeps diskless traffic gated.

`InklessDisklessRequestClient` sends raw requests to one broker, because
clients retry `KAFKA_STORAGE_ERROR` and would hide it.

## Test plan

- `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest`
- `make vale`

Assisted-by: Claude Opus 5.5
gqmelo added a commit that referenced this pull request Nov 5, 2026
…pic requests [KC-621] (#832)

`ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down
to their diskless topics before it calls the control plane, but it made
the call even when no diskless topics were left. Since #759, the gated
control plane rejects every call while it's unavailable. During an
outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after
KRaft had already created the topic. A classic-only deletion failed
before KRaft deleted anything.

`ControllerApis` now calls the control plane only when the filtered set
isn't empty. The guard belongs in `ControllerApis` because the diskless
filtering happens there. If a request has no diskless topics, there's
nothing to send to the control plane. `createPartitions` already guards
its call the same way.

Assisted-by: Claude Opus 5.5
gqmelo added a commit that referenced this pull request Nov 5, 2026
…es, restarts, and migration locks [KC-619] (#850)

The control-plane availability gate from #759 had cluster coverage only
for diskless produce and topic management while the connection string
is empty. These tests pin the rest of the behavior operators depend on
during a control-plane outage or a slow schema migration.

The outage tests check that classic produce, consume, and consumer-group
offset commits keep working while diskless produce, fetch, and
list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also
cover a broker restart during the outage, an unreachable database that
must not get a connection per produce request, and removing the emptied
dynamic override so the static connection string applies again.

The migration lock tests hold Flyway's advisory lock from a separate
session and restart one of two brokers. While the lock is held, the
running broker keeps serving classic traffic and the starting broker
stays fenced until the lock is released. When the migration outlasts
`migration.timeout.ms`, the starting broker serves classic traffic and
keeps diskless traffic gated.

`InklessDisklessRequestClient` sends raw requests to one broker, because
clients retry `KAFKA_STORAGE_ERROR` and would hide it.

## Test plan

- `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest`
- `make vale`

Assisted-by: Claude Opus 5.5
gqmelo added a commit that referenced this pull request Nov 5, 2026
…pic requests [KC-621] (#832)

`ControllerApis` filters `CreateTopics` and `DeleteTopics` requests down
to their diskless topics before it calls the control plane, but it made
the call even when no diskless topics were left. Since #759, the gated
control plane rejects every call while it's unavailable. During an
outage, a classic-only creation returned `UNKNOWN_SERVER_ERROR` after
KRaft had already created the topic. A classic-only deletion failed
before KRaft deleted anything.

`ControllerApis` now calls the control plane only when the filtered set
isn't empty. The guard belongs in `ControllerApis` because the diskless
filtering happens there. If a request has no diskless topics, there's
nothing to send to the control plane. `createPartitions` already guards
its call the same way.

Assisted-by: Claude Opus 5.5
gqmelo added a commit that referenced this pull request Nov 5, 2026
…es, restarts, and migration locks [KC-619] (#850)

The control-plane availability gate from #759 had cluster coverage only
for diskless produce and topic management while the connection string
is empty. These tests pin the rest of the behavior operators depend on
during a control-plane outage or a slow schema migration.

The outage tests check that classic produce, consume, and consumer-group
offset commits keep working while diskless produce, fetch, and
list-offsets fail with the retriable `KAFKA_STORAGE_ERROR`. They also
cover a broker restart during the outage, an unreachable database that
must not get a connection per produce request, and removing the emptied
dynamic override so the static connection string applies again.

The migration lock tests hold Flyway's advisory lock from a separate
session and restart one of two brokers. While the lock is held, the
running broker keeps serving classic traffic and the starting broker
stays fenced until the lock is released. When the migration outlasts
`migration.timeout.ms`, the starting broker serves classic traffic and
keeps diskless traffic gated.

`InklessDisklessRequestClient` sends raw requests to one broker, because
clients retry `KAFKA_STORAGE_ERROR` and would hide it.

## Test plan

- `./gradlew :core:test --tests kafka.server.InklessControlPlaneOutageClusterTest --tests kafka.server.InklessControlPlaneMigrationLockClusterTest`
- `make vale`

Assisted-by: Claude Opus 5.5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

0