Repository navigation
feat(inkless:consume): measure the fetch data phase and its queueing [KC-446] - #784
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The in-flight memory gauges omit active downloads, and the documented pool-size bound is inaccurate.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds fetch-stage latency and memory metrics for the diskless consumer path.
Changes:
- Measures object-fetch queue and data-phase duration.
- Tracks cached, cold, and in-flight object bytes.
- Adds provenance and metric-focused tests and documentation.
File summaries
| File | Description |
|---|---|
ReaderTest.java |
Tests timing and byte accounting. |
FetchPlannerTest.java |
Tests queue timing and cache provenance. |
Reader.java |
Records phase timing and retained bytes. |
InklessFetchMetrics.java |
Defines and registers six metrics. |
FetchPlanner.java |
Records queue time and fetch provenance. |
metrics.rst |
Documents the new metrics. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
a7f72f7 to
ada3917
Compare
There was a problem hiding this comment.
🟡 Changes recommended
In-flight accounting misses active downloads during submission and charges rejected jobs that never allocate memory.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Balanced
ada3917 to
cd8c452
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The FetchDataTime documentation misstates its timing boundary and relationship to per-object metrics.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Balanced
cd8c452 to
8250c25
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The in-flight gauge incorrectly counts queued jobs that haven't allocated memory.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Balanced
8250c25 to
ac6f3ab
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Queue timing includes pre-submission cache work, and the memory gauge undercounts materialized backend buffers.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
docs/inkless/PERFORMANCE.md:357
- The repository guide requires sentence-case concept headings and recommends noun phrases rather than headings that start with an
-ingverb. Rename this new heading accordingly.
#### Reading the Fetch Memory Metrics
- Files reviewed: 8/8 changed files
- Comments generated: 6
- Review effort level: Balanced
ff30262 to
1c9ef8b
Compare
1c9ef8b to
83b938a
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The memory guidance incorrectly excludes Azure's duplicate buffer, and two headings violate repository style.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
docs/inkless/PERFORMANCE.md:357
- Use sentence case for this new heading, as required by
AGENTS.md:129-132.
#### Fetch Memory Metrics
- Files reviewed: 8/8 changed files
- Comments generated: 2
- Review effort level: Balanced
83b938a to
fc7a62a
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
The transfer-to-request accounting gap can underreport live bytes and the high-water mark.
Review details
Suppressed comments (1)
storage/inkless/src/main/java/io/aiven/inkless/consume/FetchPlanner.java:572
- Releasing the transfer charge here creates an accounting gap: the returned
FileExtentalready owns its payload, butReader.allOfFileExtentscannot charge that payload until theCompletableFuturecompletes afterward. Another worker can start during this gap, so bothInFlightLaggingObjectBytesand especially its high-water mark can undercount live bytes. Hand off the existing charge to the request before completing the future (and release it here only for failures and losing hedges), so transfer and payload ownership overlap without a decrement/reincrement window.
metrics.addInFlightLaggingObjectBytes(-bytes);
- Files reviewed: 8/8 changed files
- Comments generated: 0 new
- Review effort level: Balanced
fc7a62a to
20e1cbf
Compare
08e776a to
8338c90
Compare
There was a problem hidi 10BC8 ng this comment.
🟡 Changes recommended
The advertised recent-object byte metric is missing, and four existing metrics are removed despite the stated compatibility guarantee.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 2
- Review effort level: Balanced
8338c90 to
fc15d33
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The operator guide omits rate limiting from one diagnosis and overlooks rejected jobs as a no-sample queue case.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 2
- Review effort level: Balanced
fc15d33 to
a27c0a5
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Download accounting can overreport clipped ranges, and the operator guidance can misattribute limiter-induced queueing.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
storage/inkless/src/main/java/io/aiven/inkless/consume/FetchPlanner.java:566
chargeWhileFetchingcharges the requested range rather than the buffer that the backend allocates. A consolidation fetch can plan a fixed-size recent-data block and then bypass the cache; S3 clips that range at the object end (S3Storage.java:135-138), so the returnedSizedReadableByteChanneland download buffer can be much smaller thanrequest.byteRange().size(). The gauge and high-water mark then overreport memory, which also makes the planned byte budget reject work too early. Charge the channel's actual content length afterfetch()returns and beforereadToByteBuffer()allocates the destination.
final long bytes = request.byteRange().size();
- Files reviewed: 8/8 changed files
- Comments generated: 1
- Review effort level: Balanced
a27c0a5 to
7354f9b
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Concurrency-sensitive accounting across asynchronous completion, hedging, and multiple metrics groups warrants final human validation.
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 0 new
- Review effort level: Balanced
A consumer fetch has four stages: find batches, plan, wait for every object fetch, assemble the response. Three of them are timed - FindBatchesTime, FetchPlanTime, FetchCompletionTime - and the wait is not, even though it is nearly all of the latency. It cannot be recovered by subtracting the others from FetchTotalTime either, because percentiles do not subtract. Per-object FetchFileTime does not stand in for it: a request touching more objects than the fetch pools serve at once queues the remainder, so the stage is longer than any single object's fetch. One number for that stage would still not say why it is slow. Storage throttling and a fetch pool that cannot drain as fast as requests arrive both show up as a slow wait, and they call for opposite responses. So the stage is timed as two parts: FetchQueueTime for the wait before a fetch job starts, and FetchFileTime, which already exists, for the transfer. Rate-limit wait stays where it is, under LaggingConsumerRateLimitWaitTime, since it is spent on the worker thread after the queue. The bytes that stage holds are equally invisible. CacheEntrySize measures one fetched range and FetchResponseSize measures what reaches the caller, which on the hot path is a fraction of what is retained, since a fetched range covers a whole cache block. Nothing sums a request's ranges, so the largest memory term on the read path can only be estimated by multiplying two histogram means, and a limit on in-flight fetch memory needs the spread rather than an average. Add five meters. FetchQueueTime and FetchDataTime time the stage and its queueing half. FetchLaggingObjectBytes sums, per request, the bytes held from cache-bypassing fetches, and InFlightLaggingObjectBytes with its Max track the running total for the requests one metrics group serves - consumer fetches and consolidation fetches report under separate groups. That total is what a per-request distribution cannot give: p99 per request times the pool size repeats the same product-of-means error one level up. All three byte meters cover the cache-bypassing population only, because it is the one nothing caps. Cache-resident bytes are bounded by the cache and already sized by CacheSize, CacheEntrySize and CacheHitBytesPerSec, so metering them per request would add a number with no decision attached to it. Lagging matches the LaggingConsumerRequestRate counter over the same population. Each doc string stays one or two sentences so the generated table in docs/inkless/metrics.rst remains readable. How to read the metrics together, and what they undercount, lives in docs/inkless/PERFORMANCE.md. Nothing records these yet.
Wire the fetch data phase meters into the read path. FetchQueueTime is read as the first statement of the fetch job, so it ends where the queue ends: the transfer and any rate-limit wait, which is spent on the worker thread afterwards, stay in their own meters and a slow phase can be attributed to one of the three. The clock starts where the job is handed to the executor - immediately before supplyAsync on the cold path, and in the cache's registration callback on the hot path, since capturing before computeIfAbsent would fold the cache lookup into the wait. A cache hit records nothing, because no job was queued for it; so does a caller that joins another caller's in-flight load, whose wait the loader's own sample already carries, and a hedged retry, whose wait would muddy the primary's. Reader.allOfFileExtents is where the whole set of fetched ranges is in hand, so it times the phase from its own entry to the aggregate completion and records the byte sum there. A request that planned nothing to fetch records no sample at all, for the same reason the byte histogram skips a request that fetched only from the cache: a deployment of caught-up consumers would otherwise bury the distributions under zeros. Two further decisions worth stating. Which bytes count cannot be decided outside FetchPlanner. The classification is three-way and each arm sits in a different branch of submitSingleRequest: the hot path goes through computeIfAbsent, a consolidation peek returns a cached entry, and everything else fetches cold without writing back. So a consolidation request can hold a shared cache entry for one object and a cache-bypassing range for another, and a split by request path would report it wrong. FetchRequestWithFuture carries a cacheBypassing flag instead, set next to the request counter that already meters the same population. A hot-path miss is not bypassing: computeIfAbsent registers the entry before the load runs, so concurrent callers coalesce onto it and the cache owns the bytes either way. The in-flight gauge is charged in two halves, each where its bytes exist. The download buffer is charged by FetchPlanner at the requested range for as long as the transfer runs, which covers a hedge's second buffer as well, since the loser is never cancelled and both are real until each finishes. The payload that survives is charged by Reader as it arrives and returned when the request completes. A job still waiting in a pool queue, or rejected at submission, holds nothing and is charged nothing - charging at submission instead would make the gauge track queue depth, and a bounded lagging queue holds a hundred jobs per worker. The release needs no mutual exclusion, and the reason is worth recording because it is an ordering the chain provides rather than one the accounting enforces: every extent future is wrapped so that none of them fails, so allOfFileExtents completes only once each charge has run, and the fetch response cannot complete before that. A stage that settles the response early - a timeout, or a short circuit on the first failure - would break the ordering and leak the gauge upward, which is why the class says so, why the release happens exactly once from the response's own completion, and why it zeroes its total so a second call cannot subtract twice. Test plan FetchPlannerTest covers the queue clock on both submission paths with a clock the test advances, that the hot path excludes the cache lookup, and that a cache hit records nothing; the charge window through a transfer, asserted from inside the read so it pins the boundary rather than the end state; a queued job charging nothing; a hedge charging and releasing its own buffer alongside the primary's; and cacheBypassing against the branch taken, on a mixed hot/cold fetch and on both consolidation peek outcomes. ReaderTest pins that only cache-bypassing bytes are counted, including on a request that hits the cache for one object and misses for another, and that a cache-resident-only request records nothing; that a failed fetch counts nothing; that an empty plan records no phase sample; that the phase duration spans the slowest fetch rather than the first; and, end to end through Reader.fetch, that the gauge nets to zero on both the success and the failure path. InklessFetchMetricsTest covers the gauge arithmetic itself: the value follows charges and releases, and the high-water mark ratchets only upward.
7354f9b to
f3d91f1
Compare
…[KC-446] (#784) * feat(inkless:consume): add fetch data phase metrics A consumer fetch has four stages: find batches, plan, wait for every object fetch, assemble the response. Three of them are timed - FindBatchesTime, FetchPlanTime, FetchCompletionTime - and the wait is not, even though it is nearly all of the latency. It cannot be recovered by subtracting the others from FetchTotalTime either, because percentiles do not subtract. Per-object FetchFileTime does not stand in for it: a request touching more objects than the fetch pools serve at once queues the remainder, so the stage is longer than any single object's fetch. One number for that stage would still not say why it is slow. Storage throttling and a fetch pool that cannot drain as fast as requests arrive both show up as a slow wait, and they call for opposite responses. So the stage is timed as two parts: FetchQueueTime for the wait before a fetch job starts, and FetchFileTime, which already exists, for the transfer. Rate-limit wait stays where it is, under LaggingConsumerRateLimitWaitTime, since it is spent on the worker thread after the queue. The bytes that stage holds are equally invisible. CacheEntrySize measures one fetched range and FetchResponseSize measures what reaches the caller, which on the hot path is a fraction of what is retained, since a fetched range covers a whole cache block. Nothing sums a request's ranges, so the largest memory term on the read path can only be estimated by multiplying two histogram means, and a limit on in-flight fetch memory needs the spread rather than an average. Add five meters. FetchQueueTime and FetchDataTime time the stage and its queueing half. FetchLaggingObjectBytes sums, per request, the bytes held from cache-bypassing fetches, and InFlightLaggingObjectBytes with its Max track the running total for the requests one metrics group serves - consumer fetches and consolidation fetches report under separate groups. That total is what a per-request distribution cannot give: p99 per request times the pool size repeats the same product-of-means error one level up. All three byte meters cover the cache-bypassing population only, because it is the one nothing caps. Cache-resident bytes are bounded by the cache and already sized by CacheSize, CacheEntrySize and CacheHitBytesPerSec, so metering them per request would add a number with no decision attached to it. Lagging matches the LaggingConsumerRequestRate counter over the same population. Each doc string stays one or two sentences so the generated table in docs/inkless/metrics.rst remains readable. How to read the metrics together, and what they undercount, lives in docs/inkless/PERFORMANCE.md. Nothing records these yet. * feat(inkless:consume): record fetch queue, phase time and object bytes Wire the fetch data phase meters into the read path. FetchQueueTime is read as the first statement of the fetch job, so it ends where the queue ends: the transfer and any rate-limit wait, which is spent on the worker thread afterwards, stay in their own meters and a slow phase can be attributed to one of the three. The clock starts where the job is handed to the executor - immediately before supplyAsync on the cold path, and in the cache's registration callback on the hot path, since capturing before computeIfAbsent would fold the cache lookup into the wait. A cache hit records nothing, because no job was queued for it; so does a caller that joins another caller's in-flight load, whose wait the loader's own sample already carries, and a hedged retry, whose wait would muddy the primary's. Reader.allOfFileExtents is where the whole set of fetched ranges is in hand, so it times the phase from its own entry to the aggregate completion and records the byte sum there. A request that planned nothing to fetch records no sample at all, for the same reason the byte histogram skips a request that fetched only from the cache: a deployment of caught-up consumers would otherwise bury the distributions under zeros. Two further decisions worth stating. Which bytes count cannot be decided outside FetchPlanner. The classification is three-way and each arm sits in a different branch of submitSingleRequest: the hot path goes through computeIfAbsent, a consolidation peek returns a cached entry, and everything else fetches cold without writing back. So a consolidation request can hold a shared cache entry for one object and a cache-bypassing range for another, and a split by request path would report it wrong. FetchRequestWithFuture carries a cacheBypassing flag instead, set next to the request counter that already meters the same population. A hot-path miss is not bypassing: computeIfAbsent registers the entry before the load runs, so concurrent callers coalesce onto it and the cache owns the bytes either way. The in-flight gauge is charged in two halves, each where its bytes exist. The download buffer is charged by FetchPlanner at the requested range for as long as the transfer runs, which covers a hedge's second buffer as well, since the loser is never cancelled and both are real until each finishes. The payload that survives is charged by Reader as it arrives and returned when the request completes. A job still waiting in a pool queue, or rejected at submission, holds nothing and is charged nothing - charging at submission instead would make the gauge track queue depth, and a bounded lagging queue holds a hundred jobs per worker. The release needs no mutual exclusion, and the reason is worth recording because it is an ordering the chain provides rather than one the accounting enforces: every extent future is wrapped so that none of them fails, so allOfFileExtents completes only once each charge has run, and the fetch response cannot complete before that. A stage that settles the response early - a timeout, or a short circuit on the first failure - would break the ordering and leak the gauge upward, which is why the class says so, why the release happens exactly once from the response's own completion, and why it zeroes its total so a second call cannot subtract twice. Test plan FetchPlannerTest covers the queue clock on both submission paths with a clock the test advances, that the hot path excludes the cache lookup, and that a cache hit records nothing; the charge window through a transfer, asserted from inside the read so it pins the boundary rather than the end state; a queued job charging nothing; a hedge charging and releasing its own buffer alongside the primary's; and cacheBypassing against the branch taken, on a mixed hot/cold fetch and on both consolidation peek outcomes. ReaderTest pins that only cache-bypassing bytes are counted, including on a request that hits the cache for one object and misses for another, and that a cache-resident-only request records nothing; that a failed fetch counts nothing; that an empty plan records no phase sample; that the phase duration spans the slowest fetch rather than the first; and, end to end through Reader.fetch, that the gauge nets to zero on both the success and the failure path. InklessFetchMetricsTest covers the gauge arithmetic itself: the value follows charges and releases, and the high-water mark ratchets only upward.
…[KC-446] (#784) * feat(inkless:consume): add fetch data phase metrics A consumer fetch has four stages: find batches, plan, wait for every object fetch, assemble the response. Three of them are timed - FindBatchesTime, FetchPlanTime, FetchCompletionTime - and the wait is not, even though it is nearly all of the latency. It cannot be recovered by subtracting the others from FetchTotalTime either, because percentiles do not subtract. Per-object FetchFileTime does not stand in for it: a request touching more objects than the fetch pools serve at once queues the remainder, so the stage is longer than any single object's fetch. One number for that stage would still not say why it is slow. Storage throttling and a fetch pool that cannot drain as fast as requests arrive both show up as a slow wait, and they call for opposite responses. So the stage is timed as two parts: FetchQueueTime for the wait before a fetch job starts, and FetchFileTime, which already exists, for the transfer. Rate-limit wait stays where it is, under LaggingConsumerRateLimitWaitTime, since it is spent on the worker thread after the queue. The bytes that stage holds are equally invisible. CacheEntrySize measures one fetched range and FetchResponseSize measures what reaches the caller, which on the hot path is a fraction of what is retained, since a fetched range covers a whole cache block. Nothing sums a request's ranges, so the largest memory term on the read path can only be estimated by multiplying two histogram means, and a limit on in-flight fetch memory needs the spread rather than an average. Add five meters. FetchQueueTime and FetchDataTime time the stage and its queueing half. FetchLaggingObjectBytes sums, per request, the bytes held from cache-bypassing fetches, and InFlightLaggingObjectBytes with its Max track the running total for the requests one metrics group serves - consumer fetches and consolidation fetches report under separate groups. That total is what a per-request distribution cannot give: p99 per request times the pool size repeats the same product-of-means error one level up. All three byte meters cover the cache-bypassing population only, because it is the one nothing caps. Cache-resident bytes are bounded by the cache and already sized by CacheSize, CacheEntrySize and CacheHitBytesPerSec, so metering them per request would add a number with no decision attached to it. Lagging matches the LaggingConsumerRequestRate counter over the same population. Each doc string stays one or two sentences so the generated table in docs/inkless/metrics.rst remains readable. How to read the metrics together, and what they undercount, lives in docs/inkless/PERFORMANCE.md. Nothing records these yet. * feat(inkless:consume): record fetch queue, phase time and object bytes Wire the fetch data phase meters into the read path. FetchQueueTime is read as the first statement of the fetch job, so it ends where the queue ends: the transfer and any rate-limit wait, which is spent on the worker thread afterwards, stay in their own meters and a slow phase can be attributed to one of the three. The clock starts where the job is handed to the executor - immediately before supplyAsync on the cold path, and in the cache's registration callback on the hot path, since capturing before computeIfAbsent would fold the cache lookup into the wait. A cache hit records nothing, because no job was queued for it; so does a caller that joins another caller's in-flight load, whose wait the loader's own sample already carries, and a hedged retry, whose wait would muddy the primary's. Reader.allOfFileExtents is where the whole set of fetched ranges is in hand, so it times the phase from its own entry to the aggregate completion and records the byte sum there. A request that planned nothing to fetch records no sample at all, for the same reason the byte histogram skips a request that fetched only from the cache: a deployment of caught-up consumers would otherwise bury the distributions under zeros. Two further decisions worth stating. Which bytes count cannot be decided outside FetchPlanner. The classification is three-way and each arm sits in a different branch of submitSingleRequest: the hot path goes through computeIfAbsent, a consolidation peek returns a cached entry, and everything else fetches cold without writing back. So a consolidation request can hold a shared cache entry for one object and a cache-bypassing range for another, and a split by request path would report it wrong. FetchRequestWithFuture carries a cacheBypassing flag instead, set next to the request counter that already meters the same population. A hot-path miss is not bypassing: computeIfAbsent registers the entry before the load runs, so concurrent callers coalesce onto it and the cache owns the bytes either way. The in-flight gauge is charged in two halves, each where its bytes exist. The download buffer is charged by FetchPlanner at the requested range for as long as the transfer runs, which covers a hedge's second buffer as well, since the loser is never cancelled and both are real until each finishes. The payload that survives is charged by Reader as it arrives and returned when the request completes. A job still waiting in a pool queue, or rejected at submission, holds nothing and is charged nothing - charging at submission instead would make the gauge track queue depth, and a bounded lagging queue holds a hundred jobs per worker. The release needs no mutual exclusion, and the reason is worth recording because it is an ordering the chain provides rather than one the accounting enforces: every extent future is wrapped so that none of them fails, so allOfFileExtents completes only once each charge has run, and the fetch response cannot complete before that. A stage that settles the response early - a timeout, or a short circuit on the first failure - would break the ordering and leak the gauge upward, which is why the class says so, why the release happens exactly once from the response's own completion, and why it zeroes its total so a second call cannot subtract twice. Test plan FetchPlannerTest covers the queue clock on both submission paths with a clock the test advances, that the hot path excludes the cache lookup, and that a cache hit records nothing; the charge window through a transfer, asserted from inside the read so it pins the boundary rather than the end state; a queued job charging nothing; a hedge charging and releasing its own buffer alongside the primary's; and cacheBypassing against the branch taken, on a mixed hot/cold fetch and on both consolidation peek outcomes. ReaderTest pins that only cache-bypassing bytes are counted, including on a request that hits the cache for one object and misses for another, and that a cache-resident-only request records nothing; that a failed fetch counts nothing; that an empty plan records no phase sample; that the phase duration spans the slowest fetch rather than the first; and, end to end through Reader.fetch, that the gauge nets to zero on both the success and the failure path. InklessFetchMetricsTest covers the gauge arithmetic itself: the value follows charges and releases, and the high-water mark ratchets only upward.
…[KC-446] (#784) * feat(inkless:consume): add fetch data phase metrics A consumer fetch has four stages: find batches, plan, wait for every object fetch, assemble the response. Three of them are timed - FindBatchesTime, FetchPlanTime, FetchCompletionTime - and the wait is not, even though it is nearly all of the latency. It cannot be recovered by subtracting the others from FetchTotalTime either, because percentiles do not subtract. Per-object FetchFileTime does not stand in for it: a request touching more objects than the fetch pools serve at once queues the remainder, so the stage is longer than any single object's fetch. One number for that stage would still not say why it is slow. Storage throttling and a fetch pool that cannot drain as fast as requests arrive both show up as a slow wait, and they call for opposite responses. So the stage is timed as two parts: FetchQueueTime for the wait before a fetch job starts, and FetchFileTime, which already exists, for the transfer. Rate-limit wait stays where it is, under LaggingConsumerRateLimitWaitTime, since it is spent on the worker thread after the queue. The bytes that stage holds are equally invisible. CacheEntrySize measures one fetched range and FetchResponseSize measures what reaches the caller, which on the hot path is a fraction of what is retained, since a fetched range covers a whole cache block. Nothing sums a request's ranges, so the largest memory term on the read path can only be estimated by multiplying two histogram means, and a limit on in-flight fetch memory needs the spread rather than an average. Add five meters. FetchQueueTime and FetchDataTime time the stage and its queueing half. FetchLaggingObjectBytes sums, per request, the bytes held from cache-bypassing fetches, and InFlightLaggingObjectBytes with its Max track the running total for the requests one metrics group serves - consumer fetches and consolidation fetches report under separate groups. That total is what a per-request distribution cannot give: p99 per request times the pool size repeats the same product-of-means error one level up. All three byte meters cover the cache-bypassing population only, because it is the one nothing caps. Cache-resident bytes are bounded by the cache and already sized by CacheSize, CacheEntrySize and CacheHitBytesPerSec, so metering them per request would add a number with no decision attached to it. Lagging matches the LaggingConsumerRequestRate counter over the same population. Each doc string stays one or two sentences so the generated table in docs/inkless/metrics.rst remains readable. How to read the metrics together, and what they undercount, lives in docs/inkless/PERFORMANCE.md. Nothing records these yet. * feat(inkless:consume): record fetch queue, phase time and object bytes Wire the fetch data phase meters into the read path. FetchQueueTime is read as the first statement of the fetch job, so it ends where the queue ends: the transfer and any rate-limit wait, which is spent on the worker thread afterwards, stay in their own meters and a slow phase can be attributed to one of the three. The clock starts where the job is handed to the executor - immediately before supplyAsync on the cold path, and in the cache's registration callback on the hot path, since capturing before computeIfAbsent would fold the cache lookup into the wait. A cache hit records nothing, because no job was queued for it; so does a caller that joins another caller's in-flight load, whose wait the loader's own sample already carries, and a hedged retry, whose wait would muddy the primary's. Reader.allOfFileExtents is where the whole set of fetched ranges is in hand, so it times the phase from its own entry to the aggregate completion and records the byte sum there. A request that planned nothing to fetch records no sample at all, for the same reason the byte histogram skips a request that fetched only from the cache: a deployment of caught-up consumers would otherwise bury the distributions under zeros. Two further decisions worth stating. Which bytes count cannot be decided outside FetchPlanner. The classification is three-way and each arm sits in a different branch of submitSingleRequest: the hot path goes through computeIfAbsent, a consolidation peek returns a cached entry, and everything else fetches cold without writing back. So a consolidation request can hold a shared cache entry for one object and a cache-bypassing range for another, and a split by request path would report it wrong. FetchRequestWithFuture carries a cacheBypassing flag instead, set next to the request counter that already meters the same population. A hot-path miss is not bypassing: computeIfAbsent registers the entry before the load runs, so concurrent callers coalesce onto it and the cache owns the bytes either way. The in-flight gauge is charged in two halves, each where its bytes exist. The download buffer is charged by FetchPlanner at the requested range for as long as the transfer runs, which covers a hedge's second buffer as well, since the loser is never cancelled and both are real until each finishes. The payload that survives is charged by Reader as it arrives and returned when the request completes. A job still waiting in a pool queue, or rejected at submission, holds nothing and is charged nothing - charging at submission instead would make the gauge track queue depth, and a bounded lagging queue holds a hundred jobs per worker. The release needs no mutual exclusion, and the reason is worth recording because it is an ordering the chain provides rather than one the accounting enforces: every extent future is wrapped so that none of them fails, so allOfFileExtents completes only once each charge has run, and the fetch response cannot complete before that. A stage that settles the response early - a timeout, or a short circuit on the first failure - would break the ordering and leak the gauge upward, which is why the class says so, why the release happens exactly once from the response's own completion, and why it zeroes its total so a second call cannot subtract twice. Test plan FetchPlannerTest covers the queue clock on both submission paths with a clock the test advances, that the hot path excludes the cache lookup, and that a cache hit records nothing; the charge window through a transfer, asserted from inside the read so it pins the boundary rather than the end state; a queued job charging nothing; a hedge charging and releasing its own buffer alongside the primary's; and cacheBypassing against the branch taken, on a mixed hot/cold fetch and on both consolidation peek outcomes. ReaderTest pins that only cache-bypassing bytes are counted, including on a request that hits the cache for one object and misses for another, and that a cache-resident-only request records nothing; that a failed fetch counts nothing; that an empty plan records no phase sample; that the phase duration spans the slowest fetch rather than the first; and, end to end through Reader.fetch, that the gauge nets to zero on both the success and the failure path. InklessFetchMetricsTest covers the gauge arithmetic itself: the value follows charges and releases, and the high-water mark ratchets only upward.
* docs(inkless:sync): archive the 0.48 cherry-pick sessions for 4.1, 4.2, 4.3 Session logs for bringing the three active release branches up to main (5fa6669, #784) ahead of the 0.48 release. inkless-4.3 took its 8 commits clean. inkless-4.2 and inkless-4.1 took 19 each; the logs record the conflict resolutions and the sync(compile)/sync(test) adaptations needed for the 4.2.1 and 4.1.2 APIs (Record package, LogReadResult constructor, Seq-based DelayedFetch statuses). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> docs(inkless:sync): complete the 0.48 cherry-pick session logs The three branches took six more main commits (#794, #787, #792, #793, tables and the verification that covered them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(inkless:release): list inkless-4.3 as an active release branch inkless-4.3 is on Kafka 4.3.1 (#779) and gets its first Kafka-base tag with 0.48, so it moves from Planned to Active in RELEASES.md. The release workflow header names it among the branches the cherry-pick sync has to cover before a run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(inkless:release): add 0.48 changelog entry Generated from inkless-release-0.47..inkless-release-0.48 and curated: every metric delta traced to its commit, the four config default changes the configs.rst diff shows added next to the four new keys, and the V28 lock on logs described for operators. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* docs(inkless:sync): archive the 0.48 cherry-pick sessions for 4.1, 4.2, 4.3 Session logs for bringing the three active release branches up to main (5fa6669, #784) ahead of the 0.48 release. inkless-4.3 took its 8 commits clean. inkless-4.2 and inkless-4.1 took 19 each; the logs record the conflict resolutions and the sync(compile)/sync(test) adaptations needed for the 4.2.1 and 4.1.2 APIs (Record package, LogReadResult constructor, Seq-based DelayedFetch statuses). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> docs(inkless:sync): complete the 0.48 cherry-pick session logs The three branches took six more main commits (#794, #787, #792, #793, tables and the verification that covered them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(inkless:release): list inkless-4.3 as an active release branch inkless-4.3 is on Kafka 4.3.1 (#779) and gets its first Kafka-base tag with 0.48, so it moves from Planned to Active in RELEASES.md. The release workflow header names it among the branches the cherry-pick sync has to cover before a run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(inkless:release): add 0.48 changelog entry Generated from inkless-release-0.47..inkless-release-0.48 and curated: every metric delta traced to its commit, the four config default changes the configs.rst diff shows added next to the four new keys, and the V28 lock on logs described for operators. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* docs(inkless:sync): archive the 0.48 cherry-pick sessions for 4.1, 4.2, 4.3 Session logs for bringing the three active release branches up to main (5fa6669, #784) ahead of the 0.48 release. inkless-4.3 took its 8 commits clean. inkless-4.2 and inkless-4.1 took 19 each; the logs record the conflict resolutions and the sync(compile)/sync(test) adaptations needed for the 4.2.1 and 4.1.2 APIs (Record package, LogReadResult constructor, Seq-based DelayedFetch statuses). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> docs(inkless:sync): complete the 0.48 cherry-pick session logs The three branches took six more main commits (#794, #787, #792, #793, tables and the verification that covered them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(inkless:release): list inkless-4.3 as an active release branch inkless-4.3 is on Kafka 4.3.1 (#779) and gets its first Kafka-base tag with 0.48, so it moves from Planned to Active in RELEASES.md. The release workflow header names it among the branches the cherry-pick sync has to cover before a run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(inkless:release): add 0.48 changelog entry Generated from inkless-release-0.47..inkless-release-0.48 and curated: every metric delta traced to its commit, the four config default changes the configs.rst diff shows added next to the four new keys, and the V28 lock on logs described for operators. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* docs(inkless:sync): archive the 0.48 cherry-pick sessions for 4.1, 4.2, 4.3 Session logs for bringing the three active release branches up to main (5fa6669, #784) ahead of the 0.48 release. inkless-4.3 took its 8 commits clean. inkless-4.2 and inkless-4.1 took 19 each; the logs record the conflict resolutions and the sync(compile)/sync(test) adaptations needed for the 4.2.1 and 4.1.2 APIs (Record package, LogReadResult constructor, Seq-based DelayedFetch statuses). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> docs(inkless:sync): complete the 0.48 cherry-pick session logs The three branches took six more main commits (#794, #787, #792, #793, tables and the verification that covered them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(inkless:release): list inkless-4.3 as an active release branch inkless-4.3 is on Kafka 4.3.1 (#779) and gets its first Kafka-base tag with 0.48, so it moves from Planned to Active in RELEASES.md. The release workflow header names it among the branches the cherry-pick sync has to cover before a run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(inkless:release): add 0.48 changelog entry Generated from inkless-release-0.47..inkless-release-0.48 and curated: every metric delta traced to its commit, the four config default changes the configs.rst diff shows added next to the four new keys, and the V28 lock on logs described for operators. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A consumer fetch has four stages: find batches, plan, wait for every object fetch, assemble the response. Three are timed; the wait is not, even though it is nearly all of the latency. It cannot be derived either - percentiles do not subtract, and per-object
FetchFileTimemisses the queueing, which is most of the wait once a request touches more objects than the pools serve at once.One number for the stage would still not say why it is slow: storage throttling and a pool that cannot drain fast enough both look like a slow wait and call for opposite fixes. So the stage is timed in parts. The bytes it holds are invisible too, and a limit on in-flight fetch memory needs their spread rather than an average.
No behavior change and no new config. A byte budget over in-flight fetch memory is the follow-up; this is the measurement it needs.
New metrics
Under
InklessFetchMetrics, and separately underConsolidationFetchMetricsfor consolidation fetches.FetchQueueTime- per object: wait in the pool queue before its job started. Read as the job's first statement, so the transfer (FetchFileTime) and the rate-limit wait, both on the worker thread afterwards, stay in their own meters.FetchDataTime- per request: from the plan being handed back until the last of its fetches completes.FetchLaggingObjectBytes- per request: bytes held from cold, cache-bypassing fetches.InFlightLaggingObjectBytesand itsMax- the running total of those bytes, and its high-water mark. A per-request distribution cannot give this: p99 per request times pool size repeats the product-of-means error one level up.The three byte meters cover cache-bypassing bytes only, because those are the ones nothing caps - the pools bound concurrent downloads, not bytes held. Cache-resident bytes are bounded by the cache and already sized by
CacheSize/CacheEntrySize/CacheHitBytesPerSec.docs/inkless/PERFORMANCE.mdhas the reading guide - which combination points at storage versus at the pool - and what the gauge counts, does not count and undercounts. Doc strings stay one or two sentences so the generated table inmetrics.rstremains readable.Two decisions
Which bytes count is decided per object, not per request. A consolidation fetch peeks the cache read-only and falls through to a cold fetch on a miss, so one request can hold a shared cache entry for one object and a cache-bypassing range for another.
FetchPlanner.submitSingleRequestis the only place that knows which arm ran, soFetchRequestWithFuturecarries acacheBypassingflag. A hot-path miss is not bypassing:computeIfAbsentregisters the entry before the load runs.The gauge is charged where each buffer lives.
FetchPlannercharges the download buffer for the length of the transfer, hedges included since the loser is never cancelled;Readercharges the surviving payload from arrival until the request completes. A queued or rejected job holds nothing and is charged nothing - charging at submission would make the gauge track queue depth, and the lagging queue holds a hundred jobs per worker.Commits
add fetch data phase metrics- the five meters, no callers, so the metric contract can be reviewed on its own.record fetch queue, phase time and object bytes- the wiring and the tests.See the commit messages for the reasoning behind each.
Operator impact
Five new metric names, nothing renamed or removed, so no dashboard re-baselining.
metrics.rstis regenerated. Getting them onto the dashboards responders actually open is not part of this PR.Touches
InklessFetchMetricsandmetrics.rst, which #787 also touches, so whichever merges second needs a rebase. No dependency otherwise.Test plan
FetchPlannerTestcovers the queue clock on both submission paths and that the hot path excludes the cache lookup; the charge window asserted from inside the transfer, so it pins the boundary rather than the end state; a queued job charging nothing; a hedge charging and releasing its own buffer alongside the primary's; andcacheBypassingagainst the branch taken.ReaderTestcovers which bytes count, including a request that hits the cache for one object and misses for another; that an arrived payload is on the gauge while the request still fetches; that a failed fetch and an empty plan record nothing; and, end to end throughReader.fetch, that the gauge nets to zero on both the success and failure paths.InklessFetchMetricsTestcovers the gauge arithmetic and that the high-water mark ratchets only upward.Follow-ups
FetchCompleterchange: a missing extent is already a per-partition outcome.ObjectFetchercopies it into the extent, so the gauge counts one of the two copies. Removing that copy is worth its own change.