8000 feat(inkless:consume): measure the fetch data phase and its queueing [KC-446] by jeqo · Pull Request #784 · aiven/inkless · GitHub
[go: up one dir, main page]

Skip to content

feat(inkless:consume): measure the fetch data phase and its queueing [KC-446] - #784

Merged
viktorsomogyi merged 2 commits into
mainfrom
jeqo/fetch-file-extends-metrics
Sep 8, 2026
Merged

viktorsomogyi merged 2 commits into
mainfrom
jeqo/fetch-file-extends-metrics

Conversation

@jeqo
@jeqo jeqo commented Sep 3, 2026 •
Copy link
Copy Markdown
Contributor

A consumer fetch has four stages: find batches, plan, wait for every object fetch, assemble the response. Three are timed; the wait is not, even though it is nearly all of the latency. It cannot be derived either - percentiles do not subtract, and per-object FetchFileTime misses the queueing, which is most of the wait once a request touches more objects than the pools serve at once.

One number for the stage would still not say why it is slow: storage throttling and a pool that cannot drain fast enough both look like a slow wait and call for opposite fixes. So the stage is timed in parts. The bytes it holds are invisible too, and a limit on in-flight fetch memory needs their spread rather than an average.

No behavior change and no new config. A byte budget over in-flight fetch memory is the follow-up; this is the measurement it needs.

New metrics

Under InklessFetchMetrics, and separately under ConsolidationFetchMetrics for consolidation fetches.

  • FetchQueueTime - per object: wait in the pool queue before its job started. Read as the job's first statement, so the transfer (FetchFileTime) and the rate-limit wait, both on the worker thread afterwards, stay in their own meters.
  • FetchDataTime - per request: from the plan being handed back until the last of its fetches completes.
  • FetchLaggingObjectBytes - per request: bytes held from cold, cache-bypassing fetches.
  • InFlightLaggingObjectBytes and its Max - the running total of those bytes, and its high-water mark. A per-request distribution cannot give this: p99 per request times pool size repeats the product-of-means error one level up.

The three byte meters cover cache-bypassing bytes only, because those are the ones nothing caps - the pools bound concurrent downloads, not bytes held. Cache-resident bytes are bounded by the cache and already sized by CacheSize / CacheEntrySize / CacheHitBytesPerSec.

docs/inkless/PERFORMANCE.md has the reading guide - which combination points at storage versus at the pool - and what the gauge counts, does not count and undercounts. Doc strings stay one or two sentences so the generated table in metrics.rst remains readable.

Two decisions

Which bytes count is decided per object, not per request. A consolidation fetch peeks the cache read-only and falls through to a cold fetch on a miss, so one request can hold a shared cache entry for one object and a cache-bypassing range for another. FetchPlanner.submitSingleRequest is the only place that knows which arm ran, so FetchRequestWithFuture carries a cacheBypassing flag. A hot-path miss is not bypassing: computeIfAbsent registers the entry before the load runs.

The gauge is charged where each buffer lives. FetchPlanner charges the download buffer for the length of the transfer, hedges included since the loser is never cancelled; Reader charges the surviving payload from arrival until the request completes. A queued or rejected job holds nothing and is charged nothing - charging at submission would make the gauge track queue depth, and the lagging queue holds a hundred jobs per worker.

Commits

  • add fetch data phase metrics - the five meters, no callers, so the metric contract can be reviewed on its own.
  • record fetch queue, phase time and object bytes - the wiring and the tests.

See the commit messages for the reasoning behind each.

Operator impact

Five new metric names, nothing renamed or removed, so no dashboard re-baselining. metrics.rst is regenerated. Getting them onto the dashboards responders actually open is not part of this PR.

Touches InklessFetchMetrics and metrics.rst, which #787 also touches, so whichever merges second needs a rebase. No dependency otherwise.

Test plan

FetchPlannerTest covers the queue clock on both submission paths and that the hot path excludes the cache lookup; the charge window asserted from inside the transfer, so it pins the boundary rather than the end state; a queued job charging nothing; a hedge charging and releasing its own buffer alongside the primary's; and cacheBypassing against the branch taken. ReaderTest covers which bytes count, including a request that hits the cache for one object and misses for another; that an arrived payload is on the gauge while the request still fetches; that a failed fetch and an empty plan record nothing; and, end to end through Reader.fetch, that the gauge nets to zero on both the success and failure paths. InklessFetchMetricsTest covers the gauge arithmetic and that the high-water mark ratchets only upward.

Follow-ups

  • A byte budget over in-flight fetch memory, using this gauge as its accounting object. Shedding needs no FetchCompleter change: a missing extent is already a per-partition outcome.
  • The S3 and Azure clients materialize a response before ObjectFetcher copies it into the extent, so the gauge counts one of the two copies. Removing that copy is worth its own change.
  • A stalled fetch stays invisible: both timers record on completion, like every other fetch signal.

Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The in-flight memory gauges omit active downloads, and the documented pool-size bound is inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds fetch-stage latency and memory metrics for the diskless consumer path.

Changes:

  • Measures object-fetch queue and data-phase duration.
  • Tracks cached, cold, and in-flight object bytes.
  • Adds provenance and metric-focused tests and documentation.
File summaries
File Description
ReaderTest.java Tests timing and byte accounting.
FetchPlannerTest.java Tests queue timing and cache provenance.
Reader.java Records phase timing and retained bytes.
InklessFetchMetrics.java Defines and registers six metrics.
FetchPlanner.java Records queue time and fetch provenance.
metrics.rst Documents the new metrics.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from a7f72f7 to ada3917 Compare September 4, 2026 09:07
@jeqo
jeqo requested a balanced review from Copilot September 4, 2026 09:09
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

In-flight accounting misses active downloads during submission and charges rejected jobs that never allocate memory.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Balanced

@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from ada3917 to cd8c452 Compare September 4, 2026 12:57
@jeqo
jeqo requested a balanced review from Copilot September 4, 2026 12:57
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The FetchDataTime documentation misstates its timing boundary and relationship to per-object metrics.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread docs/inkless/metrics.rst Resolved
@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from cd8c452 to 8250c25 Compare September 4, 2026 13:15
@jeqo
jeqo requested a balanced review from Copilot September 4, 2026 13:20
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The in-flight gauge incorrectly counts queued jobs that haven't allocated memory.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Balanced

@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from 8250c25 to ac6f3ab Compare September 4, 2026 15:05
@jeqo
jeqo requested a balanced review from Copilot September 4, 2026 15:05
Copilot AI 8000 left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Queue timing includes pre-submission cache work, and the memory gauge undercounts materialized backend buffers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

docs/inkless/PERFORMANCE.md:357

  • The repository guide requires sentence-case concept headings and recommends noun phrases rather than headings that start with an -ing verb. Rename this new heading accordingly.
#### Reading the Fetch Memory Metrics
  • Files reviewed: 8/8 changed files
  • Comments generated: 6
  • Review effort level: Balanced

Comment thread docs/inkless/PERFORMANCE.md Resolved
Comment thread docs/inkless/PERFORMANCE.md Resolved
Comment thread docs/inkless/PERFORMANCE.md Resolved
@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch 2 times, most recently from ff30262 to 1c9ef8b Compare September 4, 2026 17:21
@jeqo
jeqo requested a balanced review from Copilot September 4, 2026 17:23
@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from 1c9ef8b to 83b938a Compare September 4, 2026 17:23
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The memory guidance incorrectly excludes Azure's duplicate buffer, and two headings violate repository style.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

docs/inkless/PERFORMANCE.md:357

  • Use sentence case for this new heading, as required by AGENTS.md:129-132.
#### Fetch Memory Metrics
  • Files reviewed: 8/8 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread docs/inkless/PERFORMANCE.md Resolved
Comment thread docs/inkless/PERFORMANCE.md Resolved
@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from 83b938a to fc7a62a Compare September 4, 2026 17:35
@jeqo
jeqo requested a balanced review from Copilot September 4, 2026 17:52
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The transfer-to-request accounting gap can underreport live bytes and the high-water mark.

Review details

Suppressed comments (1)

storage/inkless/src/main/java/io/aiven/inkless/consume/FetchPlanner.java:572

  • Releasing the transfer charge here creates an accounting gap: the returned FileExtent already owns its payload, but Reader.allOfFileExtents cannot charge that payload until the CompletableFuture completes afterward. Another worker can start during this gap, so both InFlightLaggingObjectBytes and especially its high-water mark can undercount live bytes. Hand off the existing charge to the request before completing the future (and release it here only for failures and losing hedges), so transfer and payload ownership overlap without a decrement/reincrement window.
            metrics.addInFlightLaggingObjectBytes(-bytes);
  • Files reviewed: 8/8 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from fc7a62a to 20e1cbf Compare September 6, 2026 08:18
@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from 08e776a to 8338c90 Compare September 7, 2026 10:28
@jeqo
jeqo requested a balanced review from Copilot September 7, 2026 10:29
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The advertised recent-object byte metric is missing, and four existing metrics are removed despite the stated compatibility guarantee.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 8/8 changed files
  • Comments generated: 2
  • Review effort level: Balanced

@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from 8338c90 to fc15d33 Compare September 7, 2026 10:43
@jeqo
jeqo requested a balanced review from Copilot September 7, 2026 10:47
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The operator guide omits rate limiting from one diagnosis and overlooks rejected jobs as a no-sample queue case.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 8/8 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread docs/inkless/PERFORMANCE.md Resolved
Comment thread docs/inkless/PERFORMANCE.md Resolved
@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from fc15d33 to a27c0a5 Compare September 7, 2026 10:57
@jeqo
jeqo requested a balanced review from Copilot September 7, 2026 10:59
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Download accounting can overreport clipped ranges, and the operator guidance can misattribute limiter-induced queueing.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

storage/inkless/src/main/java/io/aiven/inkless/consume/FetchPlanner.java:566

  • chargeWhileFetching charges the requested range rather than the buffer that the backend allocates. A consolidation fetch can plan a fixed-size recent-data block and then bypass the cache; S3 clips that range at the object end (S3Storage.java:135-138), so the returned SizedReadableByteChannel and download buffer can be much smaller than request.byteRange().size(). The gauge and high-water mark then overreport memory, which also makes the planned byte budget reject work too early. Charge the channel's actual content length after fetch() returns and before readToByteBuffer() allocates the destination.
        final long bytes = request.byteRange().size();
  • Files reviewed: 8/8 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread docs/inkless/PERFORMANCE.md Resolved
@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from a27c0a5 to 7354f9b Compare September 7, 2026 11:05
@jeqo
jeqo requested a balanced review from Copilot September 7, 2026 11:07
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Concurrency-sensitive accounting across asynchronous completion, hedging, and multiple metrics groups warrants final human validation.

Review details
  • Files reviewed: 8/8 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@jeqo
jeqo marked this pull request as ready for review September 7, 2026 11:14
jeqo added 2 commits September 7, 2026 13:19
A consumer fetch has four stages: find batches, plan, wait for every object
fetch, assemble the response. Three of them are timed - FindBatchesTime,
FetchPlanTime, FetchCompletionTime - and the wait is not, even though it is
nearly all of the latency. It cannot be recovered by subtracting the others from
FetchTotalTime either, because percentiles do not subtract. Per-object
FetchFileTime does not stand in for it: a request touching more objects than the
fetch pools serve at once queues the remainder, so the stage is longer than any
single object's fetch.

One number for that stage would still not say why it is slow. Storage throttling
and a fetch pool that cannot drain as fast as requests arrive both show up as a
slow wait, and they call for opposite responses. So the stage is timed as two
parts: FetchQueueTime for the wait before a fetch job starts, and FetchFileTime,
which already exists, for the transfer. Rate-limit wait stays where it is, under
LaggingConsumerRateLimitWaitTime, since it is spent on the worker thread after
the queue.

The bytes that stage holds are equally invisible. CacheEntrySize measures one
fetched range and FetchResponseSize measures what reaches the caller, which on
the hot path is a fraction of what is retained, since a fetched range covers a
whole cache block. Nothing sums a request's ranges, so the largest memory term
on the read path can only be estimated by multiplying two histogram means, and a
limit on in-flight fetch memory needs the spread rather than an average.

Add five meters. FetchQueueTime and FetchDataTime time the stage and its
queueing half. FetchLaggingObjectBytes sums, per request, the bytes held from
cache-bypassing fetches, and InFlightLaggingObjectBytes with its Max track the
running total for the requests one metrics group serves - consumer fetches and
consolidation fetches report under separate groups. That total is what a
per-request distribution cannot give: p99 per request times the pool size
repeats the same product-of-means error one level up.

All three byte meters cover the cache-bypassing population only, because it is
the one nothing caps. Cache-resident bytes are bounded by the cache and already
sized by CacheSize, CacheEntrySize and CacheHitBytesPerSec, so metering them per
request would add a number with no decision attached to it. Lagging matches the
LaggingConsumerRequestRate counter over the same population.

Each doc string stays one or two sentences so the generated table in
docs/inkless/metrics.rst remains readable. How to read the metrics together, and
what they undercount, lives in docs/inkless/PERFORMANCE.md.

Nothing records these yet.
Wire the fetch data phase meters into the read path.

FetchQueueTime is read as the first statement of the fetch job, so it ends where
the queue ends: the transfer and any rate-limit wait, which is spent on the
worker thread afterwards, stay in their own meters and a slow phase can be
attributed to one of the three. The clock starts where the job is handed to the
executor - immediately before supplyAsync on the cold path, and in the cache's
registration callback on the hot path, since capturing before computeIfAbsent
would fold the cache lookup into the wait. A cache hit records nothing, because
no job was queued for it; so does a caller that joins another caller's in-flight
load, whose wait the loader's own sample already carries, and a hedged retry,
whose wait would muddy the primary's.

Reader.allOfFileExtents is where the whole set of fetched ranges is in hand, so
it times the phase from its own entry to the aggregate completion and records the
byte sum there. A request that planned nothing to fetch records no sample at all,
for the same reason the byte histogram skips a request that fetched only from the
cache: a deployment of caught-up consumers would otherwise bury the distributions
under zeros.

Two further decisions worth stating.

Which bytes count cannot be decided outside FetchPlanner. The classification is
three-way and each arm sits in a different branch of submitSingleRequest: the hot
path goes through computeIfAbsent, a consolidation peek returns a cached entry,
and everything else fetches cold without writing back. So a consolidation request
can hold a shared cache entry for one object and a cache-bypassing range for
another, and a split by request path would report it wrong.
FetchRequestWithFuture carries a cacheBypassing flag instead, set next to the
request counter that already meters the same population. A hot-path miss is not
bypassing: computeIfAbsent registers the entry before the load runs, so
concurrent callers coalesce onto it and the cache owns the bytes either way.

The in-flight gauge is charged in two halves, each where its bytes exist. The
download buffer is charged by FetchPlanner at the requested range for as long as
the transfer runs, which covers a hedge's second buffer as well, since the loser
is never cancelled and both are real until each finishes. The payload that
survives is charged by Reader as it arrives and returned when the request
completes. A job still waiting in a pool queue, or rejected at submission, holds
nothing and is charged nothing - charging at submission instead would make the
gauge track queue depth, and a bounded lagging queue holds a hundred jobs per
worker.

The release needs no mutual exclusion, and the reason is worth recording because
it is an ordering the chain provides rather than one the accounting enforces:
every extent future is wrapped so that none of them fails, so allOfFileExtents
completes only once each charge has run, and the fetch response cannot complete
before that. A stage that settles the response early - a timeout, or a short
circuit on the first failure - would break the ordering and leak the gauge
upward, which is why the class says so, why the release happens exactly once
from the response's own completion, and why it zeroes its total so a second call
cannot subtract twice.

Test plan

FetchPlannerTest covers the queue clock on both submission paths with a clock the
test advances, that the hot path excludes the cache lookup, and that a cache hit
records nothing; the charge window through a transfer, asserted from inside the
read so it pins the boundary rather than the end state; a queued job charging
nothing; a hedge charging and releasing its own buffer alongside the primary's;
and cacheBypassing against the branch taken, on a mixed hot/cold fetch and on
both consolidation peek outcomes. ReaderTest pins that only cache-bypassing bytes
are counted, including on a request that hits the cache for one object and misses
for another, and that a cache-resident-only request records nothing; that a failed
fetch counts nothing; that an empty plan records no phase sample; that the phase
duration spans the slowest fetch rather than the first; and, end to end through
Reader.fetch, that the gauge nets to zero on both the success and the failure
path. InklessFetchMetricsTest covers the gauge arithmetic itself: the value
follows charges and releases, and the high-water mark ratchets only upward.
@jeqo
jeqo force-pushed the jeqo/fetch-file-extends-metrics branch from 7354f9b to f3d91f1 Compare September 7, 2026 11:19
@viktorsomogyi
viktorsomogyi merged commit 5fa6669 into main Sep 8, 2026
5 checks passed
@viktorsomogyi
viktorsomogyi deleted the jeqo/fetch-file-extends-metrics branch September 8, 2026 15:21
jeqo added a commit that referenced this pull request Sep 9, 2026
…[KC-446] (#784)

* feat(inkless:consume): add fetch data phase metrics

A consumer fetch has four stages: find batches, plan, wait for every object
fetch, assemble the response. Three of them are timed - FindBatchesTime,
FetchPlanTime, FetchCompletionTime - and the wait is not, even though it is
nearly all of the latency. It cannot be recovered by subtracting the others from
FetchTotalTime either, because percentiles do not subtract. Per-object
FetchFileTime does not stand in for it: a request touching more objects than the
fetch pools serve at once queues the remainder, so the stage is longer than any
single object's fetch.

One number for that stage would still not say why it is slow. Storage throttling
and a fetch pool that cannot drain as fast as requests arrive both show up as a
slow wait, and they call for opposite responses. So the stage is timed as two
parts: FetchQueueTime for the wait before a fetch job starts, and FetchFileTime,
which already exists, for the transfer. Rate-limit wait stays where it is, under
LaggingConsumerRateLimitWaitTime, since it is spent on the worker thread after
the queue.

The bytes that stage holds are equally invisible. CacheEntrySize measures one
fetched range and FetchResponseSize measures what reaches the caller, which on
the hot path is a fraction of what is retained, since a fetched range covers a
whole cache block. Nothing sums a request's ranges, so the largest memory term
on the read path can only be estimated by multiplying two histogram means, and a
limit on in-flight fetch memory needs the spread rather than an average.

Add five meters. FetchQueueTime and FetchDataTime time the stage and its
queueing half. FetchLaggingObjectBytes sums, per request, the bytes held from
cache-bypassing fetches, and InFlightLaggingObjectBytes with its Max track the
running total for the requests one metrics group serves - consumer fetches and
consolidation fetches report under separate groups. That total is what a
per-request distribution cannot give: p99 per request times the pool size
repeats the same product-of-means error one level up.

All three byte meters cover the cache-bypassing population only, because it is
the one nothing caps. Cache-resident bytes are bounded by the cache and already
sized by CacheSize, CacheEntrySize and CacheHitBytesPerSec, so metering them per
request would add a number with no decision attached to it. Lagging matches the
LaggingConsumerRequestRate counter over the same population.

Each doc string stays one or two sentences so the generated table in
docs/inkless/metrics.rst remains readable. How to read the metrics together, and
what they undercount, lives in docs/inkless/PERFORMANCE.md.

Nothing records these yet.

* feat(inkless:consume): record fetch queue, phase time and object bytes

Wire the fetch data phase meters into the read path.

FetchQueueTime is read as the first statement of the fetch job, so it ends where
the queue ends: the transfer and any rate-limit wait, which is spent on the
worker thread afterwards, stay in their own meters and a slow phase can be
attributed to one of the three. The clock starts where the job is handed to the
executor - immediately before supplyAsync on the cold path, and in the cache's
registration callback on the hot path, since capturing before computeIfAbsent
would fold the cache lookup into the wait. A cache hit records nothing, because
no job was queued for it; so does a caller that joins another caller's in-flight
load, whose wait the loader's own sample already carries, and a hedged retry,
whose wait would muddy the primary's.

Reader.allOfFileExtents is where the whole set of fetched ranges is in hand, so
it times the phase from its own entry to the aggregate completion and records the
byte sum there. A request that planned nothing to fetch records no sample at all,
for the same reason the byte histogram skips a request that fetched only from the
cache: a deployment of caught-up consumers would otherwise bury the distributions
under zeros.

Two further decisions worth stating.

Which bytes count cannot be decided outside FetchPlanner. The classification is
three-way and each arm sits in a different branch of submitSingleRequest: the hot
path goes through computeIfAbsent, a consolidation peek returns a cached entry,
and everything else fetches cold without writing back. So a consolidation request
can hold a shared cache entry for one object and a cache-bypassing range for
another, and a split by request path would report it wrong.
FetchRequestWithFuture carries a cacheBypassing flag instead, set next to the
request counter that already meters the same population. A hot-path miss is not
bypassing: computeIfAbsent registers the entry before the load runs, so
concurrent callers coalesce onto it and the cache owns the bytes either way.

The in-flight gauge is charged in two halves, each where its bytes exist. The
download buffer is charged by FetchPlanner at the requested range for as long as
the transfer runs, which covers a hedge's second buffer as well, since the loser
is never cancelled and both are real until each finishes. The payload that
survives is charged by Reader as it arrives and returned when the request
completes. A job still waiting in a pool queue, or rejected at submission, holds
nothing and is charged nothing - charging at submission instead would make the
gauge track queue depth, and a bounded lagging queue holds a hundred jobs per
worker.

The release needs no mutual exclusion, and the reason is worth recording because
it is an ordering the chain provides rather than one the accounting enforces:
every extent future is wrapped so that none of them fails, so allOfFileExtents
completes only once each charge has run, and the fetch response cannot complete
before that. A stage that settles the response early - a timeout, or a short
circuit on the first failure - would break the ordering and leak the gauge
upward, which is why the class says so, why the release happens exactly once
from the response's own completion, and why it zeroes its total so a second call
cannot subtract twice.

Test plan

FetchPlannerTest covers the queue clock on both submission paths with a clock the
test advances, that the hot path excludes the cache lookup, and that a cache hit
records nothing; the charge window through a transfer, asserted from inside the
read so it pins the boundary rather than the end state; a queued job charging
nothing; a hedge charging and releasing its own buffer alongside the primary's;
and cacheBypassing against the branch taken, on a mixed hot/cold fetch and on
both consolidation peek outcomes. ReaderTest pins that only cache-bypassing bytes
are counted, including on a request that hits the cache for one object and misses
for another, and that a cache-resident-only request records nothing; that a failed
fetch counts nothing; that an empty plan records no phase sample; that the phase
duration spans the slowest fetch rather than the first; and, end to end through
Reader.fetch, that the gauge nets to zero on both the success and the failure
path. InklessFetchMetricsTest covers the gauge arithmetic itself: the value
follows charges and releases, and the high-water mark ratchets only upward.
jeqo added a commit that referenced this pull request Sep 9, 2026
…[KC-446] (#784)

* feat(inkless:consume): add fetch data phase metrics

A consumer fetch has four stages: find batches, plan, wait for every object
fetch, assemble the response. Three of them are timed - FindBatchesTime,
FetchPlanTime, FetchCompletionTime - and the wait is not, even though it is
nearly all of the latency. It cannot be recovered by subtracting the others from
FetchTotalTime either, because percentiles do not subtract. Per-object
FetchFileTime does not stand in for it: a request touching more objects than the
fetch pools serve at once queues the remainder, so the stage is longer than any
single object's fetch.

One number for that stage would still not say why it is slow. Storage throttling
and a fetch pool that cannot drain as fast as requests arrive both show up as a
slow wait, and they call for opposite responses. So the stage is timed as two
parts: FetchQueueTime for the wait before a fetch job starts, and FetchFileTime,
which already exists, for the transfer. Rate-limit wait stays where it is, under
LaggingConsumerRateLimitWaitTime, since it is spent on the worker thread after
the queue.

The bytes that stage holds are equally invisible. CacheEntrySize measures one
fetched range and FetchResponseSize measures what reaches the caller, which on
the hot path is a fraction of what is retained, since a fetched range covers a
whole cache block. Nothing sums a request's ranges, so the largest memory term
on the read path can only be estimated by multiplying two histogram means, and a
limit on in-flight fetch memory needs the spread rather than an average.

Add five meters. FetchQueueTime and FetchDataTime time the stage and its
queueing half. FetchLaggingObjectBytes sums, per request, the bytes held from
cache-bypassing fetches, and InFlightLaggingObjectBytes with its Max track the
running total for the requests one metrics group serves - consumer fetches and
consolidation fetches report under separate groups. That total is what a
per-request distribution cannot give: p99 per request times the pool size
repeats the same product-of-means error one level up.

All three byte meters cover the cache-bypassing population only, because it is
the one nothing caps. Cache-resident bytes are bounded by the cache and already
sized by CacheSize, CacheEntrySize and CacheHitBytesPerSec, so metering them per
request would add a number with no decision attached to it. Lagging matches the
LaggingConsumerRequestRate counter over the same population.

Each doc string stays one or two sentences so the generated table in
docs/inkless/metrics.rst remains readable. How to read the metrics together, and
what they undercount, lives in docs/inkless/PERFORMANCE.md.

Nothing records these yet.

* feat(inkless:consume): record fetch queue, phase time and object bytes

Wire the fetch data phase meters into the read path.

FetchQueueTime is read as the first statement of the fetch job, so it ends where
the queue ends: the transfer and any rate-limit wait, which is spent on the
worker thread afterwards, stay in their own meters and a slow phase can be
attributed to one of the three. The clock starts where the job is handed to the
executor - immediately before supplyAsync on the cold path, and in the cache's
registration callback on the hot path, since capturing before computeIfAbsent
would fold the cache lookup into the wait. A cache hit records nothing, because
no job was queued for it; so does a caller that joins another caller's in-flight
load, whose wait the loader's own sample already carries, and a hedged retry,
whose wait would muddy the primary's.

Reader.allOfFileExtents is where the whole set of fetched ranges is in hand, so
it times the phase from its own entry to the aggregate completion and records the
byte sum there. A request that planned nothing to fetch records no sample at all,
for the same reason the byte histogram skips a request that fetched only from the
cache: a deployment of caught-up consumers would otherwise bury the distributions
under zeros.

Two further decisions worth stating.

Which bytes count cannot be decided outside FetchPlanner. The classification is
three-way and each arm sits in a different branch of submitSingleRequest: the hot
path goes through computeIfAbsent, a consolidation peek returns a cached entry,
and everything else fetches cold without writing back. So a consolidation request
can hold a shared cache entry for one object and a cache-bypassing range for
another, and a split by request path would report it wrong.
FetchRequestWithFuture carries a cacheBypassing flag instead, set next to the
request counter that already meters the same population. A hot-path miss is not
bypassing: computeIfAbsent registers the entry before the load runs, so
concurrent callers coalesce onto it and the cache owns the bytes either way.

The in-flight gauge is charged in two halves, each where its bytes exist. The
download buffer is charged by FetchPlanner at the requested range for as long as
the transfer runs, which covers a hedge's second buffer as well, since the loser
is never cancelled and both are real until each finishes. The payload that
survives is charged by Reader as it arrives and returned when the request
completes. A job still waiting in a pool queue, or rejected at submission, holds
nothing and is charged nothing - charging at submission instead would make the
gauge track queue depth, and a bounded lagging queue holds a hundred jobs per
worker.

The release needs no mutual exclusion, and the reason is worth recording because
it is an ordering the chain provides rather than one the accounting enforces:
every extent future is wrapped so that none of them fails, so allOfFileExtents
completes only once each charge has run, and the fetch response cannot complete
before that. A stage that settles the response early - a timeout, or a short
circuit on the first failure - would break the ordering and leak the gauge
upward, which is why the class says so, why the release happens exactly once
from the response's own completion, and why it zeroes its total so a second call
cannot subtract twice.

Test plan

FetchPlannerTest covers the queue clock on both submission paths with a clock the
test advances, that the hot path excludes the cache lookup, and that a cache hit
records nothing; the charge window through a transfer, asserted from inside the
read so it pins the boundary rather than the end state; a queued job charging
nothing; a hedge charging and releasing its own buffer alongside the primary's;
and cacheBypassing against the branch taken, on a mixed hot/cold fetch and on
both consolidation peek outcomes. ReaderTest pins that only cache-bypassing bytes
are counted, including on a request that hits the cache for one object and misses
for another, and that a cache-resident-only request records nothing; that a failed
fetch counts nothing; that an empty plan records no phase sample; that the phase
duration spans the slowest fetch rather than the first; and, end to end through
Reader.fetch, that the gauge nets to zero on both the success and the failure
path. InklessFetchMetricsTest covers the gauge arithmetic itself: the value
follows charges and releases, and the high-water mark ratchets only upward.
jeqo added a commit that referenced this pull request Sep 9, 2026
…[KC-446] (#784)

* feat(inkless:consume): add fetch data phase metrics

A consumer fetch has four stages: find batches, plan, wait for every object
fetch, assemble the response. Three of them are timed - FindBatchesTime,
FetchPlanTime, FetchCompletionTime - and the wait is not, even though it is
nearly all of the latency. It cannot be recovered by subtracting the others from
FetchTotalTime either, because percentiles do not subtract. Per-object
FetchFileTime does not stand in for it: a request touching more objects than the
fetch pools serve at once queues the remainder, so the stage is longer than any
single object's fetch.

One number for that stage would still not say why it is slow. Storage throttling
and a fetch pool that cannot drain as fast as requests arrive both show up as a
slow wait, and they call for opposite responses. So the stage is timed as two
parts: FetchQueueTime for the wait before a fetch job starts, and FetchFileTime,
which already exists, for the transfer. Rate-limit wait stays where it is, under
LaggingConsumerRateLimitWaitTime, since it is spent on the worker thread after
the queue.

The bytes that stage holds are equally invisible. CacheEntrySize measures one
fetched range and FetchResponseSize measures what reaches the caller, which on
the hot path is a fraction of what is retained, since a fetched range covers a
whole cache block. Nothing sums a request's ranges, so the largest memory term
on the read path can only be estimated by multiplying two histogram means, and a
limit on in-flight fetch memory needs the spread rather than an average.

Add five meters. FetchQueueTime and FetchDataTime time the stage and its
queueing half. FetchLaggingObjectBytes sums, per request, the bytes held from
cache-bypassing fetches, and InFlightLaggingObjectBytes with its Max track the
running total for the requests one metrics group serves - consumer fetches and
consolidation fetches report under separate groups. That total is what a
per-request distribution cannot give: p99 per request times the pool size
repeats the same product-of-means error one level up.

All three byte meters cover the cache-bypassing population only, because it is
the one nothing caps. Cache-resident bytes are bounded by the cache and already
sized by CacheSize, CacheEntrySize and CacheHitBytesPerSec, so metering them per
request would add a number with no decision attached to it. Lagging matches the
LaggingConsumerRequestRate counter over the same population.

Each doc string stays one or two sentences so the generated table in
docs/inkless/metrics.rst remains readable. How to read the metrics together, and
what they undercount, lives in docs/inkless/PERFORMANCE.md.

Nothing records these yet.

* feat(inkless:consume): record fetch queue, phase time and object bytes

Wire the fetch data phase meters into the read path.

FetchQueueTime is read as the first statement of the fetch job, so it ends where
the queue ends: the transfer and any rate-limit wait, which is spent on the
worker thread afterwards, stay in their own meters and a slow phase can be
attributed to one of the three. The clock starts where the job is handed to the
executor - immediately before supplyAsync on the cold path, and in the cache's
registration callback on the hot path, since capturing before computeIfAbsent
would fold the cache lookup into the wait. A cache hit records nothing, because
no job was queued for it; so does a caller that joins another caller's in-flight
load, whose wait the loader's own sample already carries, and a hedged retry,
whose wait would muddy the primary's.

Reader.allOfFileExtents is where the whole set of fetched ranges is in hand, so
it times the phase from its own entry to the aggregate completion and records the
byte sum there. A request that planned nothing to fetch records no sample at all,
for the same reason the byte histogram skips a request that fetched only from the
cache: a deployment of caught-up consumers would otherwise bury the distributions
under zeros.

Two further decisions worth stating.

Which bytes count cannot be decided outside FetchPlanner. The classification is
three-way and each arm sits in a different branch of submitSingleRequest: the hot
path goes through computeIfAbsent, a consolidation peek returns a cached entry,
and everything else fetches cold without writing back. So a consolidation request
can hold a shared cache entry for one object and a cache-bypassing range for
another, and a split by request path would report it wrong.
FetchRequestWithFuture carries a cacheBypassing flag instead, set next to the
request counter that already meters the same population. A hot-path miss is not
bypassing: computeIfAbsent registers the entry before the load runs, so
concurrent callers coalesce onto it and the cache owns the bytes either way.

The in-flight gauge is charged in two halves, each where its bytes exist. The
download buffer is charged by FetchPlanner at the requested range for as long as
the transfer runs, which covers a hedge's second buffer as well, since the loser
is never cancelled and both are real until each finishes. The payload that
survives is charged by Reader as it arrives and returned when the request
completes. A job still waiting in a pool queue, or rejected at submission, holds
nothing and is charged nothing - charging at submission instead would make the
gauge track queue depth, and a bounded lagging queue holds a hundred jobs per
worker.

The release needs no mutual exclusion, and the reason is worth recording because
it is an ordering the chain provides rather than one the accounting enforces:
every extent future is wrapped so that none of them fails, so allOfFileExtents
completes only once each charge has run, and the fetch response cannot complete
before that. A stage that settles the response early - a timeout, or a short
circuit on the first failure - would break the ordering and leak the gauge
upward, which is why the class says so, why the release happens exactly once
from the response's own completion, and why it zeroes its total so a second call
cannot subtract twice.

Test plan

FetchPlannerTest covers the queue clock on both submission paths with a clock the
test advances, that the hot path excludes the cache lookup, and that a cache hit
records nothing; the charge window through a transfer, asserted from inside the
read so it pins the boundary rather than the end state; a queued job charging
nothing; a hedge charging and releasing its own buffer alongside the primary's;
and cacheBypassing against the branch taken, on a mixed hot/cold fetch and on
both consolidation peek outcomes. ReaderTest pins that only cache-bypassing bytes
are counted, including on a request that hits the cache for one object and misses
for another, and that a cache-resident-only request records nothing; that a failed
fetch counts nothing; that an empty plan records no phase sample; that the phase
duration spans the slowest fetch rather than the first; and, end to end through
Reader.fetch, that the gauge nets to zero on both the success and the failure
path. InklessFetchMetricsTest covers the gauge arithmetic itself: the value
follows charges and releases, and the high-water mark ratchets only upward.
jeqo added a commit that referenced this pull request Sep 10, 2026
* docs(inkless:sync): archive the 0.48 cherry-pick sessions for 4.1, 4.2, 4.3

Session logs for bringing the three active release branches up to main
(5fa6669, #784) ahead of the 0.48 release. inkless-4.3 took its 8
commits clean. inkless-4.2 and inkless-4.1 took 19 each; the logs record
the conflict resolutions and the sync(compile)/sync(test) adaptations
needed for the 4.2.1 and 4.1.2 APIs (Record package, LogReadResult
constructor, Seq-based DelayedFetch statuses).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(inkless:sync): complete the 0.48 cherry-pick session logs

The three branches took six more main commits (#794, #787, #792, #793,
tables and the verification that covered them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(inkless:release): list inkless-4.3 as an active release branch

inkless-4.3 is on Kafka 4.3.1 (#779) and gets its first Kafka-base tag
with 0.48, so it moves from Planned to Active in RELEASES.md. The release
workflow header names it among the branches the cherry-pick sync has to
cover before a run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(inkless:release): add 0.48 changelog entry

Generated from inkless-release-0.47..inkless-release-0.48 and curated:
every metric delta traced to its commit, the four config default changes
the configs.rst diff shows added next to the four new keys, and the V28
lock on logs described for operators.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
giuseppelillo pushed a commit that referenced this pull request Sep 18, 2026
* docs(inkless:sync): archive the 0.48 cherry-pick sessions for 4.1, 4.2, 4.3

Session logs for bringing the three active release branches up to main
(5fa6669, #784) ahead of the 0.48 release. inkless-4.3 took its 8
commits clean. inkless-4.2 and inkless-4.1 took 19 each; the logs record
the conflict resolutions and the sync(compile)/sync(test) adaptations
needed for the 4.2.1 and 4.1.2 APIs (Record package, LogReadResult
constructor, Seq-based DelayedFetch statuses).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(inkless:sync): complete the 0.48 cherry-pick session logs

The three branches took six more main commits (#794, #787, #792, #793,
tables and the verification that covered them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(inkless:release): list inkless-4.3 as an active release branch

inkless-4.3 is on Kafka 4.3.1 (#779) and gets its first Kafka-base tag
with 0.48, so it moves from Planned to Active in RELEASES.md. The release
workflow header names it among the branches the cherry-pick sync has to
cover before a run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(inkless:release): add 0.48 changelog entry

Generated from inkless-release-0.47..inkless-release-0.48 and curated:
every metric delta traced to its commit, the four config default changes
the configs.rst diff shows added next to the four new keys, and the V28
lock on logs described for operators.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
giuseppelillo pushed a commit that referenced this pull request Sep 18, 2026
* docs(inkless:sync): archive the 0.48 cherry-pick sessions for 4.1, 4.2, 4.3

Session logs for bringing the three active release branches up to main
(5fa6669, #784) ahead of the 0.48 release. inkless-4.3 took its 8
commits clean. inkless-4.2 and inkless-4.1 took 19 each; the logs record
the conflict resolutions and the sync(compile)/sync(test) adaptations
needed for the 4.2.1 and 4.1.2 APIs (Record package, LogReadResult
constructor, Seq-based DelayedFetch statuses).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(inkless:sync): complete the 0.48 cherry-pick session logs

The three branches took six more main commits (#794, #787, #792, #793,
tables and the verification that covered them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(inkless:release): list inkless-4.3 as an active release branch

inkless-4.3 is on Kafka 4.3.1 (#779) and gets its first Kafka-base tag
with 0.48, so it moves from Planned to Active in RELEASES.md. The release
workflow header names it among the branches the cherry-pick sync has to
cover before a run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(inkless:release): add 0.48 changelog entry

Generated from inkless-release-0.47..inkless-release-0.48 and curated:
every metric delta traced to its commit, the four config default changes
the configs.rst diff shows added next to the four new keys, and the V28
lock on logs described for operators.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
giuseppelillo pushed a commit that referenced this pull request Sep 24, 2026
* docs(inkless:sync): archive the 0.48 cherry-pick sessions for 4.1, 4.2, 4.3

Session logs for bringing the three active release branches up to main
(5fa6669, #784) ahead of the 0.48 release. inkless-4.3 took its 8
commits clean. inkless-4.2 and inkless-4.1 took 19 each; the logs record
the conflict resolutions and the sync(compile)/sync(test) adaptations
needed for the 4.2.1 and 4.1.2 APIs (Record package, LogReadResult
constructor, Seq-based DelayedFetch statuses).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(inkless:sync): complete the 0.48 cherry-pick session logs

The three branches took six more main commits (#794, #787, #792, #793,
tables and the verification that covered them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(inkless:release): list inkless-4.3 as an active release branch

inkless-4.3 is on Kafka 4.3.1 (#779) and gets its first Kafka-base tag
with 0.48, so it moves from Planned to Active in RELEASES.md. The release
workflow header names it among the branches the cherry-pick sync has to
cover before a run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(inkless:release): add 0.48 changelog entry

Generated from inkless-release-0.47..inkless-release-0.48 and curated:
every metric delta traced to its commit, the four config default changes
the configs.rst diff shows added next to the four new keys, and the V28
lock on logs described for operators.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

0