Repository navigation
feat(inkless:storage): split S3 error metrics by operation - #716
Merged
Merged
Conversation
The S3 MetricCollector recorded every SDK error (throttling, server, timeout, IO, other) into a single global sensor per error type, so a throttle could not be attributed to a specific operation. The ops question "are our deletes being throttled?" was unanswerable from metrics. Record each error into an additional operation-tagged sensor (operation=<OperationName>, e.g. DeleteObjects) alongside the existing untagged global sensor. The global metrics are unchanged, so existing dashboards/alerts keep working; the tagged variants appear under a new MBean (type=s3-client-metrics,operation=...). Tagged sensors are created lazily and stored in a ConcurrentHashMap to stay safe under concurrent SDK publish callbacks. Per-operation attribution is skipped when the operation name is ambiguous. Adds MetricCollectorTest which builds SDK MetricCollections directly to exercise publish() without WireMock, and regenerates metrics.rst.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves observability of the Inkless S3 storage backend by attributing AWS SDK error metrics (throttling, server, timeout, IO, other) to specific S3 operations, while keeping the existing global (untagged) metrics intact for backwards-compatible dashboards and alerts.
Changes:
- Add per-operation (tagged) variants of S3 error metrics via new
MetricNameTemplates inMetricRegistry. - Extend
MetricCollector.publish()to record SDK errors into both the existing global sensors and lazily-created operation-tagged sensors (when operation name is unambiguous). - Add
MetricCollectorTestthat constructs SDKMetricCollections directly to validate global vs per-operation error recording behavior, and regeneratedocs/inkless/metrics.rstaccordingly.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| storage/inkless/src/test/java/io/aiven/inkless/storage_backend/s3/MetricCollectorTest.java | Adds unit coverage for per-operation vs global error metrics and ambiguous-operation behavior. |
| storage/inkless/src/main/java/io/aiven/inkless/storage_backend/s3/MetricRegistry.java | Introduces operation-tagged metric templates and includes them in the registry’s exported list for docs/JMX. |
| storage/inkless/src/main/java/io/aiven/inkless/storage_backend/s3/MetricCollector.java | Records SDK errors into new operation-tagged sensors (lazily, concurrency-safe) in addition to existing global sensors. |
| docs/inkless/metrics.rst | Documents the new type=s3-client-metrics,operation=... MBean and its attributes. |
jeqo
marked this pull request as ready for review
July 22, 2026 18:31
tvainika
approved these changes
Aug 12, 2026
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
The S3 MetricCollector recorded every SDK error (throttling, server, timeout, IO, other) into a single global sensor per error type, so a throttle could not be attributed to a specific operation. The ops question "are our deletes being throttled?" was unanswerable from metrics. Record each error into an additional operation-tagged sensor (operation=<OperationName>, e.g. DeleteObjects) alongside the existing untagged global sensor. The global metrics are unchanged, so existing dashboards/alerts keep working; the tagged variants appear under a new MBean (type=s3-client-metrics,operation=...). Tagged sensors are created lazily and stored in a ConcurrentHashMap to stay safe under concurrent SDK publish callbacks. Per-operation attribution is skipped when the operation name is ambiguous. Adds MetricCollectorTest which builds SDK MetricCollections directly to exercise publish() without WireMock, and regenerates metrics.rst.
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
The S3 MetricCollector recorded every SDK error (throttling, server, timeout, IO, other) into a single global sensor per error type, so a throttle could not be attributed to a specific operation. The ops question "are our deletes being throttled?" was unanswerable from metrics. Record each error into an additional operation-tagged sensor (operation=<OperationName>, e.g. DeleteObjects) alongside the existing untagged global sensor. The global metrics are unchanged, so existing dashboards/alerts keep working; the tagged variants appear under a new MBean (type=s3-client-metrics,operation=...). Tagged sensors are created lazily and stored in a ConcurrentHashMap to stay safe under concurrent SDK publish callbacks. Per-operation attribution is skipped when the operation name is ambiguous. Adds MetricCollectorTest which builds SDK MetricCollections directly to exercise publish() without WireMock, and regenerates metrics.rst.
jeqo
added a commit
that referenced
this pull request
Aug 19, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's 8000 diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
jeqo
added a commit
that referenced
this pull request
Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
jeqo
added a commit
that referenced
this pull request
Sep 9, 2026
Covers inkless-release-0.46..inkless-release-0.47. Generated via the inkless-changelog skill and curated: - Metric deltas cross-checked against source commits (#716, #735, #743, #746, #715); all trace to real, shipped metrics, no documentation catch-up noise this increment. - Postgres migration headers (V23-V26) checked for lock impact; V25 takes ACCESS EXCLUSIVE on files for a full-table scan at broker startup, V26 offers a pre-upgrade CONCURRENTLY workaround. - Added two config default changes the generator's diff-only detection missed (existing keys, not new ones): file.cleaner.interval.ms (5min -> 2min, #745) and retention.enforcement.interval.ms / max.batches.per.request (#744).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The S3 MetricCollector recorded every SDK error (throttling, server, timeout, IO, other) into a single global sensor per error type, so a throttle could not be attributed to a specific operation. The ops question "are our deletes being throttled?" was unanswerable from metrics.
Record each error into an additional operation-tagged sensor (operation=, e.g. DeleteObjects) alongside the existing untagged global sensor. The global metrics are unchanged, so existing dashboards/alerts keep working; the tagged variants appear under a new MBean (type=s3-client-metrics,operation=...).
Tagged sensors are created lazily and stored in a ConcurrentHashMap to stay safe under concurrent SDK publish callbacks. Per-operation attribution is skipped when the operation name is ambiguous.
Adds MetricCollectorTest which builds SDK MetricCollections directly to exercise publish() without WireMock, and regenerates metrics.rst.