fix(inkless:switch): defer timestamp ListOffsets below the diskless log start [KC-761] by giuseppelillo · Pull Request #870 · aiven/inkless

giuseppelillo · 2026-11-12T12:34:29Z

A timestamp ListOffsets lookup can return an offset past records that the partition still stores. The consumer then skips those records without an error.

On a hybrid partition, the broker searches its local log first and asks the control plane only if nothing matches. The control plane searches from the first retained WAL batch. If the local log ends before that batch, neither search reads the offsets in between.

The following replicas hit this:

A switched leader below the seal. The KC-702 divergent-tail check recreates it empty, or it lost its disk, and the broker keeps it online so that consolidation can rebuild it.
A consolidating replica that the WAL prune passed, such as a reassignment target, a replica that lost its data, or a follower that lags the prune. Any replica of a consolidating topic answers ListOffsets, and the pruner doesn't wait for replicas.

Example

A born-diskless consolidating topic holds offsets 0 to 1199, with timestamps 10000 + offset. The leader has tiered up to 799, and the prune moved the WAL start to 800. Replica B is a reassignment target with an empty log. A consumer in the zone of B calls offsetsForTimes(10100):

Replica	What happens	Consumer gets
Leader	The local search finds offset 100.	100
B, before this change	The local search finds nothing. The control plane returns its oldest batch, 800, because its timestamp 10800 is at or past 10100.	800, and skips 100 to 799
B, after this change	The local log ends before the WAL start.	`OFFSET_NOT_AVAILABLE`, and the consumer retries

Changes

The router returns the retriable OFFSET_NOT_AVAILABLE (LEADER_NOT_AVAILABLE before version 5) when the local log ends before the diskless log. The Fetch path already does this for the same window. A match in the local log is still returned.

Switched replica below the seal: the lookup fails without a control-plane call.
Consolidating partition outside phase 1 of the switch: FetchOffsetHandler.Job.addWalStart reads the WAL start in the same control-plane call as the timestamp lookup. The lookup fails if the local log ends before the WAL start, or before the seal if that's later. Job now matches responses to requests by position, because a partition can appear twice. A response for another partition fails the lookups.
Searchable end of the local log: this is the log end offset. While the leader epoch cache can't resolve the log start, the searchable end is the log start. A tier-state rebuild truncates the log before it restores that cache, and until then the remote search finds nothing.
Search range snapshot: the broker reads the truncation count and the log start before the searchable end. When the lookup decides, the range counts only if the log is the same instance with the same truncation count and log start. A rebuild, truncation, recreation, or log start move during the lookup fails the lookup.
ConsolidationDeferredListOffsetsRate: a new ReplicaManager meter counts the deferred lookups. Errors from the control plane don't count. DISKLESS_CONSOLIDATION.md lists it.
LeaderEpochFileCache.assign(List): it validates the entries before it changes the cache, and it holds the write lock across them. A reader sees none of the restored epochs or all of them. INKLESS_OWNERSHIP lists the file as interleaved.

The WAL start check applies only while the broker consolidates the partition. A switched partition that doesn't consolidate, or one in phase 1, keeps its log at the seal. There, diskless retention or a DeleteRecords past the seal would fail every local miss.

Trade-off

A consolidating replica that can't rebuild answers timestamp lookups with OFFSET_NOT_AVAILABLE until it rebuilds. This happens when the remote prefix is unknown or when the consolidation fetcher parks the partition. Before this change, the replica returned a wrong offset. ConsolidationDeferredListOffsetsRate shows a replica in this state.

Test 8000 ing

DisklessFetchOffsetRouterTest covers a switched replica below the seal, with sync and async local lookups, the version mapping, and a local match. It also covers a consolidating replica before and past the WAL start, a diskless lookup that doesn't match, a search range that stops being current, the seal bound before the first prune, forwarded errors, and the fresh-job fallback.
FetchOffsetHandlerTest covers request order, positional matching, the cross-tier earliest cache, failures, and a response-count mismatch.
ReplicaManagerInklessTest checks that the KC-702 recreated leader and an empty consolidating replica get OFFSET_NOT_AVAILABLE, and that a caught-up replica gets the diskless offset. It also covers the rebuild window with no epoch for the log start, a rebuild during the lookup, and a log start that moves during the lookup. The existing fallback tests that stub the local lookup now create a local log at the seal.
DisklessFetchOffsetRouterTest and ReplicaManagerInklessTest check that each deferred lookup marks ConsolidationDeferredListOffsetsRate once, and that other answers don't.
LeaderEpochFileCacheTest checks that a reader sees none or all of a restored list of epoch entries.
With each guard turned off, the tests that expect the error fail.
InklessDisklessListOffsetsTest passes against Postgres, and RemoteLogManagerTest, UnifiedLogTest, and TierStateMachineTest pass.

KC-761

🤖 Generated with Claude Code

Assisted-by: Claude Opus 5.5

…og start [KC-761] A `ListOffsets` lookup by timestamp on a hybrid partition searches the local log first and asks the control plane only on a "no match". The control plane answers from the first retained WAL batch. A replica whose local log ends before that batch leaves a range that neither search reads, so a consumer that seeks into it starts at the first diskless offset and skips records without an error. Two kinds of replica hit it: - A switched leader below the seal: the KC-702 divergent-tail check recreates it empty, or it lost its disk and stays online for consolidation to rebuild it. - A consolidating replica that the WAL prune passed: a reassignment target, a replica that lost its data, or a follower that lags the prune. The pruner doesn't wait for replicas, and any replica of a consolidating topic answers `ListOffsets`. The router now returns the retriable `OFFSET_NOT_AVAILABLE` (`LEADER_NOT_AVAILABLE` before version 5) instead, as the Fetch path already does for the same window. A switched replica below the seal gets the error without a control-plane call. For a consolidating partition, `FetchOffsetHandler.Job.addWalStart` reads the WAL start in the same control-plane call as the timestamp lookup, and the router fails the lookup if the local log ends before the WAL start or the seal. A match in the local log is still returned. The searchable end of the local log is its log end offset, or its log start offset while the leader epoch cache can't resolve the log start. A tier-state rebuild truncates the log past the WAL start before it restores that cache, and until then the remote search finds nothing. The router reads this end before the local search and again when the control plane answers, and uses the smaller value. The WAL start check applies only to consolidating partitions. A switched partition that doesn't consolidate keeps its log at the seal, so diskless retention past the seal would block it permanently. `FetchOffsetHandler.Job` now matches control-plane responses to requests by position, because a partition can appear twice. Every control plane answers in request order. Testing `DisklessFetchOffsetRouterTest` covers a switched replica below the seal (sync and async local lookups, version mapping, a local match), a consolidating replica below and past the WAL start, a diskless "no match" below the WAL start, a log that a rebuild extends or a recreation cuts back during the lookup, the seal bound before the first prune, forwarded errors, and the fresh-job fallback. With the guard disabled, the 9 tests that expect the error fail. `FetchOffsetHandlerTest` covers request order, positional matching, the cross-tier earliest cache, failures, and a response-count mismatch. `ReplicaManagerInklessTest` checks that the KC-702 recreated leader and an empty replica of a consolidating topic get `OFFSET_NOT_AVAILABLE`, and that a caught-up replica gets the diskless offset. Four existing fallback tests now create a local log at the seal. `InklessDisklessListOffsetsTest` passes against Postgres. Assisted-by: Claude Opus 5.5

Copilot

🟡 Changes recommended

The tests do not cover a remote prefix with an unresolved epoch, leaving regressions that skip remote data during tier-state rebuilds undetected.

1 open issue

Add coverage for unsearchable remote prefixes during timestamp lookup · New

What changed in this PR

This PR prevents timestamp-based ListOffsets lookups from silently skipping records when a switched or consolidating replica can't search the range before the diskless log.

Changes:

Return retriable errors when the searchable local range ends before the seal or WAL start.
Batch WAL-start and timestamp lookups, and match control-plane responses by position.
Add regression coverage and document the deferral behavior.

File	Description
`storage/inkless/src/test/java/io/aiven/inkless/consume/FetchOffsetHandlerTest.java`	Tests batching, response matching, cache isolation, and failures.
`storage/inkless/src/main/java/io/aiven/inkless/consume/FetchOffsetHandler.java`	Adds WAL-start lookups and positional response handling.
`docs/inkless/DISKLESS_CONSOLIDATION.md`	Documents timestamp lookup deferral during replica recovery.
`core/src/test/scala/unit/kafka/server/ReplicaManagerInklessTest.scala`	Tests recreated and consolidating replicas and updates fallback fixtures.
`core/src/test/scala/unit/kafka/server/DisklessFetchOffsetRouterTest.scala`	Tests range guards, asynchronous fallback, and error propagation.
`core/src/main/scala/kafka/server/ReplicaManager.scala`	Calculates the searchable range using log offsets and epoch-cache state.
`core/src/main/scala/kafka/server/DisklessFetchOffsetRouter.scala`	Guards timestamp fallback against gaps before the diskless log.

_{Review effort: Balanced}

…timestamp lookups [KC-761] No test exercised the branch of the searchable end that drops to the log start while the leader epoch cache can't resolve it. The router tests stub the searchable end, and the broker cases had no remote prefix, so returning the log end offset unconditionally still passed. The new `ReplicaManagerInklessTest` case leaves a remote-enabled log in the state that a tier-state rebuild leaves before it restores the cache: log start 0, local log start and log end 900, past the WAL start at 800, and no epoch for the log start. It checks that a timestamp lookup gets `OFFSET_NOT_AVAILABLE`, and that after the cache is restored the lookup gets the diskless offset. With the branch disabled, only this test fails. The consolidating replica tests now share their broker setup. Assisted-by: Claude Opus 5.5

Copilot

🟡 Changes recommended

Unsynchronized range reads can race with truncation and allow a lookup to skip the remote prefix, requiring a consistent snapshot with truncation invalidation and a regression test.

1 open issue

Prevent inconsistent log-range snapshots across truncation · New

1 resolved since last review

Add coverage for unsearchable remote prefixes during timestamp lookup

_{Review effort: Balanced}

…mestamp lookup decides [KC-761] The searchable end of the local log came from unsynchronized reads. A tier-state rebuild could truncate the log between them and leave a range that claims the remote prefix even though the local search can't read it. If the rebuild restores the leader epoch cache before the control plane answers, a second read agrees with the first, and the guard returns a diskless offset that skips the prefix. The broker now reads the log's truncation count before the range, as `updateHighestOffsetInRemoteStorage` callers do, and the range counts only while the log is the same instance with the same count. A truncation that the reads straddle, a rebuild or truncation during the lookup, and a recreation all make the range unknown, and the lookup gets `OFFSET_NOT_AVAILABLE`. This check replaces the second read. Testing `ReplicaManagerInklessTest` runs a rebuild inside the remote search: it truncates the log and restores the epoch cache before the control plane answers. The lookup gets `OFFSET_NOT_AVAILABLE`, and a lookup without a rebuild gets the diskless offset. Without the check, the test fails. `DisklessFetchOffsetRouterTest` covers a range that stops being current on a switched and on a consolidating partition, before routing and before the control plane answers, and that the router reads the range once before the classic lookup. Assisted-by: Claude Opus 5.5

Copilot

🟡 Changes recommended

Critical range-validation gaps during log-start advancement or partial epoch restoration can cause the diskless fallback to skip retained records.

1 open issue

Invalidate timestamp lookup range when log start advances · New

1 previously missed issue

In code that hasn't changed since last review

Publish rebuilt epoch checkpoints atomically

core/src/main/scala/kafka/server/ReplicaManager.scala:2230

During a multi-epoch rebuild, LeaderEpochFileCache.assign(List) publishes entries under separate locks, so resolving the log-start epoch doesn't mean the full checkpoint is available. A concurrent timestamp lookup can accept this range, reject remote segments whose later epochs aren't restored yet, and fall through to a diskless offset that skips retained records; neither the log instance nor the truncation count changes during restoration. Publish the checkpoint atomically or track rebuild readiness until all epochs are available, and add a multi-epoch restore interleaving test.

1 resolved since last review

Prevent inconsistent log-range snapshots across truncation

_{Review effort: Balanced}

… epochs restore in part [KC-761] Two more states let a remote timestamp search read nothing while the search range still counted as current, so the lookup returned a diskless offset that skips retained records: - Retention or `DeleteRecords` moves the log start while the remote search is queued. The search keeps the log start it read, and `truncateFromStartAsyncFlush` drops the epoch of that offset. The truncation count doesn't change. - A tier-state rebuild restores the epoch cache with `LeaderEpochFileCache.assign(List)`, which took the write lock once per entry. A reader could see the epoch of the log start before the later epochs, and `RemoteLogManager` rejects segments in epochs that aren't in the cache yet. The search range now also records the log start and counts only while it's unchanged. `assign(List)` holds the write lock across all entries, so a reader sees none of them or all of them. `INKLESS_OWNERSHIP` lists `LeaderEpochFileCache.java` as interleaved. Testing `ReplicaManagerInklessTest` moves the log start inside the remote search, checks that the epoch of the search's start offset is gone, and expects `OFFSET_NOT_AVAILABLE`. A lookup without the move gets the diskless offset. `LeaderEpochFileCacheTest` assigns 2000 entries 20 times while a reader thread records the cache sizes it sees, and checks that it only sees 0 or 2000. Each test fails without its fix. `LeaderEpochFileCacheTest`, `RemoteLogManagerTest`, `UnifiedLogTest`, and `TierStateMachineTest` pass. Assisted-by: Claude Opus 5.5

Copilot

🔵 Needs a closer look

The data-loss guard’s ordering guarantees across asynchronous timestamp searches, concurrent log rebuilds, epoch-cache restoration, and WAL pruning warrant human review and concurrent rebuild/prune validation.

No open issues 🎉

1 resolved since last review

Invalidate timestamp lookup range when log start advances

_{Review effort: Balanced}

viktorsomogyi · 2026-11-13T10:38:24Z

+        val logStartOffset = log.logStartOffset
+        val remotePrefixUnsearchable = log.remoteLogEnabled() && logStartOffset < log.localLogStartOffset &&
+          log.leaderEpochCache.epochForOffset(logStartOffset).isEmpty
+        val end = if (remotePrefixUnsearchable) logStartOffset else log.logEndOffset


A stale high watermark still lets a lookup skip records (Warning, pre-existing)

This predates the PR, so it doesn't need a fix here. A follow-up ticket is enough. Not sure if you had this already in what you showed this morning?

The issue: classicSearchRange treats the log end offset as the searchable end. A consumer lookup can't see past the high watermark (or the last stable offset for READ_COMMITTED), though. If the local search matches a record at or past that bound, classicFetchOffset hides the match, and the router falls back to diskless. The WAL-start check then compares the log end offset with the WAL start and passes, so the consumer gets an offset at or past the WAL start. It skips the records in [highWatermark, walStart) without an error.

This can happen after a restart with a stale HW checkpoint. testFetchConsolidatingTieredRangeAboveHighWatermarkReadsFromUnifiedLog sets up exactly that state for Fetch. It lasts until the first consolidation fetch, or for as long as the fetcher has the partition parked. The seal check has the same gap: a switched leader whose log ends at the seal but whose HW is below it skips [highWatermark, seal).

Suggested fix: for client requests, compute the searchable end from the same bound that Partition.fetchOffsetForTimestamp uses: the last stable offset for READ_COMMITTED, and the high watermark for READ_UNCOMMITTED. Keep the log end offset for other replica IDs. In steady state the HW is at or past the WAL start, so this change shouldn't add errors.

viktorsomogyi

Gemini 4 Argon + Opus 5.5 review came up with a couple of points. Overall I think the approach is good and these are mostly minor issues that they noticed.

…checks [KC-761] Review follow-ups for the timestamp lookup guard: - The WAL start check ran in phase 1 of the switch, where the broker doesn't consolidate and the local log stays at the seal. A `DeleteRecords` past the seal moves the WAL start over deleted offsets, and every local miss then failed until phase 2. The check now runs only while consolidation is active, as `consolidationActiveFor` decides. The seal check still applies. - A missing or negative WAL start now fails the check. Before, a negative offset passed it. - `FetchOffsetHandler.Job` checks that each control-plane response is for the partition of the request at the same position, and fails every lookup on a mismatch. - `LeaderEpochFileCache.assign(List)` validates every entry before it changes the cache, so an invalid entry leaves none of the list applied. Testing `DisklessFetchOffsetRouterTest` covers phase 1 with a WAL start past the seal, the seal check in phase 1, and a negative WAL start. `FetchOffsetHandlerTest` covers responses in another order. `LeaderEpochFileCacheTest` covers a list with an invalid entry. Each test fails without its fix. `ReplicaManagerInklessTest` and `TierStateMachineTest` pass. Assisted-by: Claude Opus 5.5

…-761] A replica that can't search the range before the diskless log answers timestamp `ListOffsets` lookups with `OFFSET_NOT_AVAILABLE` until it rebuilds. If it never rebuilds, only client errors show it. The Fetch path already counts its deferrals in `ConsolidationDeferredPrefixFetchRate`. `ReplicaManager` now registers `ConsolidationDeferredListOffsetsRate`, and the router marks it each time it defers a timestamp lookup: on a switched replica below the seal, and on a consolidating replica that ends before the WAL start. Errors that the router forwards from the control plane don't count. `DISKLESS_CONSOLIDATION.md` lists the metric. Testing `DisklessFetchOffsetRouterTest` checks that each deferred lookup counts once, and that a diskless result, forwarded errors, and phase 1 lookups don't count. `ReplicaManagerInklessTest` reads the meter for the KC-702 recreated leader and for an empty and a caught-up consolidating replica. Without the marks, those tests fail. Assisted-by: Claude Opus 5.5

giuseppelillo requested a balanced review from Copilot November 12, 2026 13:02

Copilot started reviewing on behalf of giuseppelillo November 12, 2026 13:03

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

Comment thread core/src/main/scala/kafka/server/ReplicaManager.scala Resolved

giuseppelillo requested a balanced review from Copilot November 12, 2026 13:23

Copilot started reviewing on behalf of giuseppelillo November 12, 2026 13:23

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

Comment thread core/src/main/scala/kafka/server/ReplicaManager.scala Resolved

giuseppelillo requested a balanced review from Copilot November 12, 2026 13:39

Copilot started reviewing on behalf of giuseppelillo November 12, 2026 13:40

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

Comment thread core/src/main/scala/kafka/server/ReplicaManager.scala Resolved

giuseppelillo requested a balanced review from Copilot November 12, 2026 13:59

Copilot started reviewing on behalf of giuseppelillo November 12, 2026 13:59

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

viktorsomogyi reviewed Nov 13, 2026

View reviewed changes

jeqo reviewed Nov 13, 2026

View reviewed changes

Comment thread core/src/main/scala/kafka/server/DisklessFetchOffsetRouter.scala Resolved

viktorsomogyi approved these changes Nov 13, 2026

View reviewed changes

jeqo approved these changes Nov 13, 2026

View reviewed changes

jeqo merged commit 12c5120 into main Nov 13, 2026
8 checks passed

jeqo deleted the claude/kc-761-investigation-36f1c8 branch November 13, 2026 12:25

jeqo mentioned this pull request Nov 13, 2026

fix(inkless:consolidation): defer timestamp lookups during a rebuild [KC-719] #848

Closed

Navigation Menu

Repository navigation

fix(inkless:switch): defer timestamp ListOffsets below the diskless log start [KC-761] - #870

fix(inkless:switch): defer timestamp ListOffsets below the diskless log start [KC-761]#870
jeqo merged 6 commits into
mainfrom
claude/kc-761-investigation-36f1c8

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Uh oh!

Example

Changes

Trade-off

Test 8000 ing

Uh oh!

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants