test(inkless:switch): run switch system tests with consolidation and cover faults before phase 2 [KC-747] by giuseppelillo · Pull Request #869 · aiven/inkless · GitHub
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(inkless:switch): run switch system tests with consolidation and cover faults before phase 2 [KC-747] - #869
Only one of the 14 switch system tests started brokers with consolidation.
The rest never left phase 1, so forced deletion of the local prefix, the
divergent-tail check, the rebuild from remote storage, and consolidation
from the seal ran under faults only in unit tests. _wait_for_switch_complete
also returned at control-plane init, because InFlightPartitions counts only
steps 3 to 5.
The brokers now consolidate by default, and the switch alter adds remote.storage.enable. The completion wait then requires classicPrefixInRemoteStorage on every partition through kafka-topic-switch.sh state, and records the seal of each partition.
Each data check now compares every consumed record with the acked ledger:
partition, offset, key, and value. It fails on a missing, duplicated,
out-of-order, changed, or never-acked record, and labels each one below or at
or above the seal. Producers fail the test on any send error.
test_fault_between_seal_commit_and_phase_2 holds every partition in WaitingForUpload by dropping broker traffic to object storage, asserts the
hold through JMX and the switch state, then injects one fault: a leader
restart, SIGKILL, or SIGSTOP, a follower SIGKILL, a rolling restart, a leader
network partition, a reassignment, or a topic deletion. Three more faults
start as the hold ends, so they overlap the flag request and the phase 2
transition: a failover of the active controller of three, a Postgres outage,
and a rolling restart. The hold alone is the object-storage outage case.
…cover faults before phase 2 [KC-747]
Only one of the 14 switch system tests started brokers with consolidation.
The rest never left phase 1, so forced deletion of the local prefix, the
divergent-tail check, the rebuild from remote storage, and consolidation
from the seal ran under faults only in unit tests. `_wait_for_switch_complete`
also returned at control-plane init, because `InFlightPartitions` counts only
steps 3 to 5.
The brokers now consolidate by default, and the switch alter adds
`remote.storage.enable`. The completion wait then requires
`classicPrefixInRemoteStorage` on every partition through
`kafka-topic-switch.sh state`, and records the seal of each partition.
Each data check now compares every consumed record with the acked ledger:
partition, offset, key, and value. It fails on a missing, duplicated,
out-of-order, changed, or never-acked record, and labels each one below or at
or above the seal. Producers fail the test on any send error.
`test_fault_between_seal_commit_and_phase_2` holds every partition in
`WaitingForUpload` by dropping broker traffic to object storage, asserts the
hold through JMX and the switch state, then injects one fault: a leader
restart, SIGKILL, or SIGSTOP, a follower SIGKILL, a rolling restart, a leader
network partition, a reassignment, or a topic deletion. Three more faults
start as the hold ends, so they overlap the flag request and the phase 2
transition: a failover of the active controller of three, a Postgres outage,
and a rolling restart. The hold alone is the object-storage outage case.
Testing
The new diff and parsing helpers were checked offline against synthetic
records. `pyflakes` and `make vale` are clean. The Inkless System Tests
workflow runs the suite on this branch.
Assisted-by: Claude Opus 5.5
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The test can hide missing records by accepting repeated partition-offset acknowledgments with identical content and overwriting ledger entries; it must reject every repeated position.
This PR strengthens classic-to-diskless switch system tests by exercising consolidation and phase 2, checking record integrity, and injecting faults after seal commit.
Changes:
Enable consolidation by default and wait for every partition’s classicPrefixInRemoteStorage flag, recording its seal offset.
Compare consumed partition, offset, key, and value against acknowledged records, and fail on producer send errors.
Add a fault matrix covering broker failures, dependency outages, reassignment, topic deletion, and controller failover before or during phase 2.
…h test ledger
Each producer run restarts the same key and value sequence. If offsets
regressed, a new batch could be acked at existing positions with identical
content, and the ledger accepted it and hid the lost records. Any second
ack for a partition and offset now fails the test.
Assisted-by: Claude Opus 5.5
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The unconditional phase 1 wait can time out after a broker restart in consolidating clusters even when the switch has completed, so those clusters need to wait directly on phase 2 metadata.
If a partition reaches phase 2 before its broker restarts in rolling_restart_at_phase_2, the restarted consolidating replica no longer contributes to SealedPartitionsCount. The unconditional phase 1 wait can then time out even though the switch has completed. For consolidating clusters, wait directly on the phase 2 metadata; keep the sealed-partition check for non-consolidating clusters.
A leader that takes over a consolidating partition in phase 2, for
example after a restart, doesn't seal it, so the partition drops out of
`SealedPartitionsCount`. The phase 1 wait then times out after a rolling
restart even though the switch finished. The leader requests the phase 2
flag only after control-plane init, so a consolidating cluster now waits
on the topic config and the phase 2 flag only.
Assisted-by: Claude Opus 5.5
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Both rolling-restart variants time out waiting for an unassigned broker to join the ISR, while the URP and controller-failover tests can pass without exercising their intended regression scenarios.
No open issues 🎉
3 previously missed issues
In code that hasn't changed since last review
Rolling-restart tests wait for an unassigned broker to join the ISR
Both rolling-restart variants time out with four brokers and replication factor three: _rolling_restart() waits for every broker to enter partition 0's ISR, which the unassigned broker can never do. Use three brokers for these variants, or make the helper wait only for assigned replicas. Both calls at lines 2423 and 2463 are affected.
Phase 2 lets the internal topic mask the switched-partition URP regression
Changing this default also moves test_switched_topic_urp_clears_after_replica_recovery into phase 2 before it stops a follower. ReplicaManager.underReplicatedPartitionCount explicitly excludes partitions with active consolidation (ReplicaManager.scala:551–553), so the switched partition no longer contributes to the gauge that test checks. The newly enabled, replicated __remote_log_metadata topic can instead satisfy the positive URP assertion, letting the test pass without exercising its intended regression. Keep that test explicitly in phase 1, or update it to assert the intended phase 2 metric separately from internal-topic URP.
Controller failover test does not ensure a flag request overlaps failover
This variant can pass without a flag request encountering the controller failover. The brokers still have to upload the classic prefix after heal_dependency, and the remote log manager polls every 5 seconds; the remaining controllers can elect a replacement before that upload finishes. The test only checks eventual completion, so it doesn't distinguish that schedule from a request overlapping failover. Synchronize the controller fault with a pending flag request, or assert a failed/retried flag request during the failover, so this case actually covers the intended transition fault.
…n the flag request to the controller failover
The Category D cluster has four brokers and a replication factor of three.
`_rolling_restart()` waited for every broker to join the ISR of partition 0,
and the broker outside its replicas never does. The helper now waits only
for the replicas of partition 0.
The URP gauge leaves out partitions that consolidate. In phase 2, the
switched partition no longer counts, and `__remote_log_metadata` could raise
the gauge instead. `test_switched_topic_urp_clears_after_replica_recovery`
now runs without consolidation, so it stays in phase 1.
`controller_failover_at_phase_2` could pass without a flag request meeting
the failover, because the remaining controllers elect a leader within
seconds. The test now stops two of three controllers, waits until a flag
request is pending for 10 s, and then starts one of them.
Assisted-by: Claude Opus 5.5
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The test can pass without a broker restart overlapping the phase 2 transition, so it must explicitly enforce that overlap to verify the intended scenario.
…n in the rolling restart fault
`rolling_restart_at_phase_2` lifted the hold and then started a rolling
restart. Every broker could finish the upload and get the flag before the
first broker stopped, so the test could pass without a restart overlapping
phase 2. The test now stops the leader of partition 0 before it lifts the
hold and keeps it down until every partition is in phase 2. The leader then
rejoins in phase 2, and the other brokers restart one at a time.
Assisted-by: Claude Opus 5.5
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The distributed fault matrix requires system-test validation of controller-quorum loss, dependency blackholes, and phase-2 recovery, since static review cannot establish that these coordinated scenarios complete reliably.
…sions into later tests
The `postgres_outage_at_phase_2` fault failed, and so did the four Category D
tests that ran after it.
The fault waited for phase 2 with `kafka-topic-switch.sh state`. That command
lists offsets, which needs the control plane, so every call failed during the
outage. The broker logs show that the controller accepted every flag about
20 s into the outage. The test now waits on the `WaitingForUpload` and
`SendingFlagToController` gauges, which don't need Postgres, and confirms
phase 2 with `state` after it lifts the block.
The outage drops broker traffic to Postgres. Postgres never sees the brokers
close their connections, so it keeps those sessions for about two hours.
The four tests after the fault failed with `too many clients already`. After
a test blocks the control plane, the teardown now stops the brokers and ends
the Postgres sessions from their addresses.
Assisted-by: Claude Opus 5.5
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because transformed client metadata can misidentify partition leaders, allowing fault tests to pass without exercising the intended leader and follower roles.
…, not the client metadata
For a diskless topic, `InklessTopicMetadataTransformer` rewrites the leader
in Metadata and DescribeTopicPartitions responses to a replica it picks by
hash. `_get_leader_node` read `kafka-topics.sh --describe`, so after the
switch it could return a follower. A leader fault could then hit a follower,
and a follower fault could hit the leader, and the test would still pass.
The helper now reads the `ReplicasCount` gauge of the partition on each live
broker with a one-time `JmxTool` call. Only the leader reports a nonzero
value, in every phase of the switch. The helper waits until exactly one
broker reports it.
Assisted-by: Claude Opus 5.5
…n't stop in teardown
`KafkaService.stop_node` raises when a broker doesn't exit within 60 s.
The teardown then skipped `terminate_broker_sessions()`, and the stale
sessions filled `max_connections` for the tests that ran next. The teardown
now kills the brokers and ends the sessions in a `finally` block.
Assisted-by: Claude Opus 5.5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Only one of the 14 switch system tests started brokers with consolidation.
The rest never left phase 1, so forced deletion of the local prefix, the
divergent-tail check, the rebuild from remote storage, and consolidation
from the seal ran under faults only in unit tests.
_wait_for_switch_completealso returned at control-plane init, because
InFlightPartitionscounts onlysteps 3 to 5.
The brokers now consolidate by default, and the switch alter adds
remote.storage.enable. The completion wait then requiresclassicPrefixInRemoteStorageon every partition throughkafka-topic-switch.sh state, and records the seal of each partition.Each data check now compares every consumed record with the acked ledger:
partition, offset, key, and value. It fails on a missing, duplicated,
out-of-order, changed, or never-acked record, and labels each one below or at
or above the seal. Producers fail the test on any send error.
test_fault_between_seal_commit_and_phase_2holds every partition inWaitingForUploadby dropping broker traffic to object storage, asserts thehold through JMX and the switch state, then injects one fault: a leader
restart, SIGKILL, or SIGSTOP, a follower SIGKILL, a rolling restart, a leader
network partition, a reassignment, or a topic deletion. Three more faults
start as the hold ends, so they overlap the flag request and the phase 2
transition: a failover of the active controller of three, a Postgres outage,
and a rolling restart. The hold alone is the object-storage outage case.
🤖 Generated with Claude Code