Releases

Inkless version 0.51

Docker Images

docker pull ghcr.io/aiven/inkless:0.51
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.51
Kafka 4.2.2: inkless-4.2.2-0.51
Kafka 4.3.1: inkless-4.3.1-0.51

0.51 (Kafka 4.1.2, 4.2.2, 4.3.1)

Features

(inkless:switch) gate broker ISR shrink on the switch phase (#841)
(inkless:switch) gate consolidation on the switch phase (#840)
(inkless:consolidation) bound the WAL prune per request [KC-444] (#843)
(inkless:switch) detect classic-prefix upload and request the tiered flag [KC-588] (#835)
(inkless:controller) gate ISR, reassignment, and leader election on switch phase (#833)

Fixes

(inkless:controller) skip the config record when the connection string of the control plane doesn't change (#852)
(inkless:controller) block unclean leader election in switch phase 1 [KC-636] (#849)
(inkless:consolidation) defer prefix reads on a rebuilding replica [KC-673] (#847)
(inkless:controller) validate remote storage on consolidated diskless creates (#845)
(inkless:metadata) route consolidating partitions with one replica to the replica [KC-651] (#844)
(inkless:controller) parse diskless.enable case-insensitively on topic creation (#846)
(inkless:controller) return a retriable error when the control plane is unavailable [KC-649] (#839)
(inkless:control-plane) read unavailableReason once in recordGatedCall [KC-624] (#836)
(inkless:control-plane) close the retired delegate before building its replacement [KC-630] (#838)
(inkless:controller) skip control plane calls for classic-only topic requests [KC-621] (#832)

Configuration

Added inkless.consolidation.cleanup.max.batches.per.request (default 2000): caps the batches one consolidation cleanup request deletes per partition. 0 removes the cap (#843).
Changed the default of inkless.consolidation.cleanup.interval.ms from 5 minutes to 1 minute (#843).

Metrics

Added io.aiven.inkless.delete:type=ConsolidatedDisklessLogPruner: ConsolidationPruneTotalBatchesDeleted, ConsolidationPruneTotalBytesDeleted, ConsolidationPruneTotalTime, and ConsolidationPruneErrorRate (#843).
Added kafka.server:type=ReplicaManager,name=ConsolidationDeferredPrefixFetchRate and the per-partition gauge ConsolidationTierRebuildRequests (#847).
Added kafka.controller:type=KafkaController,name=DisklessSwitchedPartitionsPendingUploadCount (#835).

Postgres schema

Migration V31__Bound_prune_batches.sql adds a new prune function and type. It takes no table lock, and the v1 function stays for brokers on 0.50 during a rolling upgrade (#843).

Upgrade notes

The classic-to-diskless switch now has two phases. Until the controller commits classicPrefixInRemoteStorage, brokers keep the classic ISR rules, don't consolidate the partition, and the controller rejects unclean leader elections (#833, #835, #840, #841, #849).
Diskless CreateTopics, DeleteTopics, and CreatePartitions return the retriable KAFKA_STORAGE_ERROR when the control plane is unavailable (#839).
With consolidation on, a diskless topic whose retention is below the broker log.local.retention.ms now fails at create time (#845).

For the full list, see CHANGELOG.md.

Inkless version 0.50

Docker Images

docker pull ghcr.io/aiven/inkless:0.50
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.50
Kafka 4.2.2: inkless-4.2.2-0.50
Kafka 4.3.1: inkless-4.3.1-0.50

0.50 (Kafka 4.1.2, 4.2.2, 4.3.1)

Features

(inkless:controller) add the SetClassicPrefixInRemoteStorage RPC [KC-582] (#827)
(inkless:control_plane) make control plane connection config dynamic (#759)
(inkless:metrics) add inkless-tag and kafka-version to app-info (#821)
(inkless:metadata) add classicPrefixInRemoteStorage flag [KC-580] (#824)
(inkless:ai) support Cursor in the Vale gate hook (#817)

Fixes

(inkless:control-plane) drop the batches timestamp index before release (#828)
(inkless:config) require consolidation for the classic-to-diskless switch (#822)
(inkless:consume) stop a slow ListOffsets from hanging the broker [KC-537] (#798)
(inkless:consolidation) exclude consolidation from ConsumerLag [KC-546] (#816)
(inkless:test) raise S3 timeouts and avoid the isolated broker in switch systests (#813)
(inkless:build) enable Zip64 for the JMH shadow jar (#810)
(inkless:test) replace removed minio images with pgsty fork (#811)

Configuration

Added inkless.control.plane.migration.timeout.ms
Added inkless.fetch.offset.thread.pool.size

Postgres schema

Migration V30__List_offsets_min_offset_lookup.sql

Inkless version 0.49

Docker Images

docker pull ghcr.io/aiven/inkless:0.49
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.49
Kafka 4.2.1: inkless-4.2.1-0.49
Kafka 4.3.1: inkless-4.3.1-0.49

0.49 (Kafka 4.1.2, 4.2.1, 4.3.1)

Fixes

(inkless:config) allow routine alters on diskless topics with remote.storage.enable=false (#804)
(inkless:consolidation) freeze cross-tier start during prune (#806)
(inkless:retention) reject new copy-disabled consolidating topics [KC-552] (#809)
(inkless:retention) skip consolidating topics in WAL retention [KC-552] (#808)
(inkless:controller) count diskless topics with remote storage effectively off [KC-540] (#805)
(inkless:consolidation) start consolidation when remote storage is enabled later [KC-540] (#802)

Postgres schema

Migration V29__Bootstrap_cross_tier_start_on_prune.sql

Inkless version 0.48

Docker Images

docker pull ghcr.io/aiven/inkless:0.48
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.48
Kafka 4.2.1: inkless-4.2.1-0.48
Kafka 4.3.1: inkless-4.3.1-0.48

0.48 (Kafka 4.1.2, 4.2.1, 4.3.1)

Upstream sync: main development base moved to Kafka 4.3.0-SNAPSHOT (from 4.2.0-SNAPSHOT), Scala 2.13.17 -> 2.13.18.

Features

(inkless:storage) default S3 API call timeouts to 2s and 1s (#791)
(inkless:consume) measure the fetch data phase and its queueing [KC-446] (#784)
(inkless:storage) bound and shorten the GCS upload path (#783)
(inkless:delete) soft-delete diskless topics and purge in the background [KC-349] (#782)
(inkless:consume) default the lagging request-rate limit off, cap batches per partition (#769)
(inkless:control-plane) enable batch coalescing by default (#766)

Fixes

(inkless:storage) count retried GCS error responses (#789)
(inkless:ci) copy integrationTest JUnit XML into build/junit-xml (#793)
(inkless:delete) do not sleep inside the topic purger tick [KC-349] (#792)
(inkless:release) exclude sync-tooling commits from the branch-consistency gate (#794)
(inkless:ci) update Inkless CI workflow and JUnit catalog parsing (#788)
(inkless:gcs) narrow fetch length before opening the reader (#781)
(inkless:storage) report an oversized fetch as a storage failure [KC-481] (#778)
(inkless:consume) size fetch buffers to the payload [KC-481] (#772)
(inkless:consume) serve consumer reads from a lagging AZ replica [KC-461] (#774)
(inkless:gcs) count batch object deletions in metrics [KC-483] (#775)
(inkless-sync) use trunk merge base for --before-version (#770)
(inkless-sync) avoid arithmetic error in conflict count check (#768)
(inkless:control_plane) make the diskless fetch budget fair across partitions [KC-407] (#750)
(inkless:fetch) preserve legacy fetch request partition identity in response [KC-353] (#762)
(inkless:metadata) accept pipe as a client.id AZ separator (#767)
(inkless:build) stop reusing inklessTag as both the override key and the resolved value [KC-384] (#765)

Configuration

Added inkless.storage.gcs.connect.timeout
Added inkless.storage.gcs.read.timeout
Added inkless.topic.purger.interval.ms
Added inkless.topic.purger.max.batches.per.cycle

Postgres schema

Migration V27__Find_batches_request_level_min_one_message.sql
Migration V28__Soft_delete_logs.sql -- ALTER TABLE, CREATE INDEX: takes a table-level lock, check the migration header for operator impact

Inkless version 0.47

Docker Images

docker pull ghcr.io/aiven/inkless:0.47
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.47
Kafka 4.2.1: inkless-4.2.1-0.47

0.47 (Kafka 4.1.2, 4.2.1)

Features

(inkless:metrics) add classic-to-diskless replicas outside ISR gauge (#763)
(inkless:retention) shorten the file-cleaner interval to 2 minutes (#745)
(inkless:retention) bound the file-cleaner deletion worklist [KC-404] (#743)
(inkless:retention) add LastSuccessfulFileCleanupAgeMs (#746)
(inkless:control_plane) add pg table auto-vacuum tuning (#747)
(inkless:release) add changelog generator and backfill 0.33..0.46 (#713)
(inkless:storage) split S3 error metrics by operation (#716)
(inkless) enable AZ parsing for generated client IDs (#699)
(inkless:metrics) add last-successful-operation age gauges [KC-316] (#735)

Fixes

(inkless:retention) size retention enforcement to keep up with produce (#744)
(inkless:test) Adjust date in license (#761)
(inkless:storage) bound bulk-delete failure logging [KC-430] (#742)
(inkless) repair missing control-plane rows on CreatePartitions retry [KC-420] (#752)
(inkless:controller) skip switched partitions in unfence ISR expansion (#754)
(inkless:switch) clear under-replicated partitions after classic-to-diskless switch (#697)
(inkless) bound error-message size on fetch/commit failure paths (#741)
(inkless:storage) drain object-store deletes monotonically under throttling [KC-413] (#715)
(inkless:consume) stop counting a fast cache load as a cache hit (#738)
(inkless:switch) preserve classic prefix on partition increase during switch [KC-387] (#737)
(inkless) refresh cached topic config for diskless topics with a local log (#736)
(inkless:build) resolve the inkless release tag without git (#734)
(inkless:retention) Limit the size of AWS delete error messages (#733)
(inkless:release) run finalize-release on the workflow_call path (#729)

Configuration

Added inkless.file.cleaner.max.files.per.cycle

Postgres schema

Migration V23__Init_diskless_log_authoritative_seal.sql
Migration V24__Table_storage_tuning.sql -- ALTER TABLE: takes a table-level lock, check the migration header for operator impact
Migration V25__File_cleaner_marked_for_deletion_index.sql -- CREATE INDEX: takes a table-level lock, check the migration header for operator impact
Migration V26__Drop_files_by_state_only_deleting_index.sql -- DROP INDEX: takes a table-level lock, check the migration header for operator impact

Inkless version 0.46

Docker Images

docker pull ghcr.io/aiven/inkless:0.46
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.46
Kafka 4.2.1: inkless-4.2.1-0.46

0.46 (Kafka 4.1.2, 4.2.1)

Features

(inkless) surface inkless tag at startup and in --version output (#717)
(inkless:consume) add diskless reader throughput metrics (#727)
(inkless:consolidation) separate ConsolidationFetchBytesInPerSec from replication metric (#723)
(inkless:consume) meter control-plane fetch errors separately from data-plane (#726)
(inkless:consolidation) delayed fetcher to honor minBytes/maxWaitMs (#685)

Fixes

(inkless:ci) green the upstream test suite (diskless test fixes + flaky quarantine) (#700)
(inkless:release) unblock the release workflow (#714)
(inkless) allow changing RF for diskless managed topics (#724)
(inkless:consolidation) reject follower reads below the cross-tier earliest [KC-332] (#720)
(inkless:retention) bound retention enforcement boundary scan to O(cap) (#722)
(inkless:consolidation) stop over-reclaiming the classic prefix on rebuilt leaders [KC-332] (#711)
(inkless:consolidation) route DeleteRecords to the real leader and report cross-tier low watermark [KC-332] (#710)
(inkless:consolidation) route ListOffsets(EARLIEST) to the control plane f 8000 or consolidating topics [KC-332] (#709)
(inkless:switch) [KC-234] force-roll boundary segment at seal to enable tiering (#694)

Inkless version 0.45

Docker Images

docker pull ghcr.io/aiven/inkless:0.45
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.45
Kafka 4.2.1: inkless-4.2.1-0.45

0.45 (Kafka 4.1.2, 4.2.1)

Features

(inkless:consume) coalesce per-batch buffers and isolate corrupt-batch failures [KC-279] (#706)
(inkless:retention) track earliest batch timestamp per log (#704)
(inkless:switch) implement repair tool for diskless switch (#686)
(inkless) add INKLESS_OWNERSHIP manifest as single source of truth (#693)

Fixes

(inkless:retention) decouple enforce_retention_v2 from the log lock (#705)
(inkless:consume) bound find_batches_v2 scan to the fetch budget (O(result), not O(read-depth)) [KC-327] (#701)
(inkless:ci) guard JUnit parsing against killed-test reports (#696)
(inkless:ts-unification) allow diskless+remote.storage on creation when allow-switch flag is on (#691)
(inkless:ci) fix RequestQuotaTest and make CI test failures visible (#692)

@viktorsomogyi

Inkless version 0.44

Docker Images

docker pull ghcr.io/aiven/inkless:0.44
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.44
Kafka 4.2.1: inkless-4.2.1-0.44

What's Changed

fix(inkless:consolidation): recover a switched consolidated leader after local-log loss by @viktorsomogyi in #673
feat(inkless): add partition fan-in commit metrics by @jeqo in #675
test(inkless:consolidation): add born-consolidated pipeline + WAL-prune system test by @viktorsomogyi in #674
fix(inkless:controller): fix leader skew for managed diskless after rolling restart by @jeqo in #643
feat(inkless): coalesce contiguous batches into one row in commit_file_v2 by @jeqo in #671
fix(inkless:consolidation): don't fence a consolidating leader below the seal by @viktorsomogyi in #677
fix(inkless): find_batches rejects offset below log start offset by @jeqo in #666
test(inkless:consolidation): add read-from-remote system tests, harness support by @viktorsomogyi in #654
test(inkless:consolidation): add dependency-outage system test for consolidation pipeline by @viktorsomogyi in #669
docs(inkless): document the interceptors introduced for CREATE_TOPIC by @gqmelo in #680
feat(inkless:consolidation): route consolidation fetcher through coldpath to avoid cache pollution [KC-171] by @jeqo in #679
feat(inkless): add inkless.client.az.listener.map config by @gqmelo in #683
feat(inkless): resolve client AZ from listener map in metadata transformer by @gqmelo in #684
feat(inkless:switch): implement AlterDisklessSwitch and tooling [KC-97] by @EelisK in #665
feat(inkless): [KC-298] track and serve the cross-tier earliest offset for consolidated topics by @viktorsomogyi in #670
feat(inkless:consolidation): [KC-298] add JMX metrics for the cross-tier log start offset by @viktorsomogyi in #682
feat(inkless:switch): auto-enable remote.storage atomically on diskless switch by @jeqo in #678
fix(inkless:consolidation): serve cross-tier earliest offset promptly after startup by @jeqo in #688

Full Changelog: inkless-release-0.43...inkless-release-0.44

@giuseppelillo

Inkless version 0.43

Docker Images

docker pull ghcr.io/aiven/inkless:0.43
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.43
Kafka 4.2.1: inkless-4.2.1-0.43

What's Changed

fix(inkless): truncate above the seal only if those messages are not consolidated by @giuseppelillo in #667
fix(inkless:metrics): exclude consolidating partitions from URP metrics by @jeqo in #640
fix(inkless): Add PostgreSQL socket timeouts for Inkless control plane by @giuseppelillo in #668
fix(inkless:consolidation): complete the classic-to-consolidated switch by always sealing and registering by @viktorsomogyi in #651
fix(inkless:consolidation): start consolidation when remote storage is enabled on a diskless topic by @viktorsomogyi in #672

Full Changelog: inkless-release-0.42...inkless-release-0.43

@jeqo

Inkless version 0.42

Docker Images

docker pull ghcr.io/aiven/inkless:0.42
docker pull ghcr.io/aiven/inkless:latest

Kafka Versions

Kafka 4.1.2: inkless-4.1.2-0.42
Kafka 4.2.1: inkless-4.2.1-0.42

What's Changed

refactor(inkless): clarify fetch routing for consolidating partitions by @jeqo in #633
docs(inkless:switch): Document classic to diskless switch by @giuseppelillo in #632
fix(inkless): Remove consolidating fetchers unconditionally by @viktorsomogyi in #629
feat(inkless:consolidation): add separate wiring for consolidation fetcher [KC-160] by @jeqo in #635
feat(inkless): add dedicated rate limit for consolidation fetch [KC-145] by @jeqo in #636
feat(inkless:consolidation): make fetch quota config dynamic by @jeqo in #637
feat(inkless:test): implement system tests for classic to diskless switch by @giuseppelillo in #581
feat(inkless): add controller guards for classic-to-diskless switch [POD-2464] by @EelisK in #634
fix(inkless): reject topic creation with config conflicting with diskless regex by @gqmelo in #642
fix(inkless): create tiered topic when diskless.enable=false and system is disabled [KC-158] by @EelisK in #641
docs(inkless): fix rendering of release sync prompt by @gqmelo in #648
test(inkless): Integration test for diskless + consolidate separately by @viktorsomogyi in #639
feat(inkless): KC-156 diskless leader epoch for consolidation truncation by @viktorsomogyi in #631
feat(inkless): ensure unclean leader election is disabled on classic-to-diskless switch [KC-129] by @EelisK in #647
docs(inkless): Enrich documentation of classicToDisklessStartOffset by @giuseppelillo in #658
feat(inkless:consolidation): serve consolidated reads from remote via OFFSET_MOVED_TO_TIERED_STORAGE by @viktorsomogyi in #650
test(inkless:systest): run ducktape system tests against consolidated topics [KC-153] by @viktorsomogyi in #645
feat(inkless:consolidation): supplement local log with diskless data on fetch [KC-168] by @jeqo in #638
fix(inkless:systest): fix sigstop and slow consumer giving false negatives by @giuseppelillo in #659
feat(inkless:tools): add operator tooling for topic type switch [KC-97] by @EelisK in #661
fix(inkless:switch): Fix stale HW on switched leader promotion by @giuseppelillo in #660

Full Changelog: inkless-release-0.41...inkless-release-0.42

Inkless 0.51

Docker Images

Kafka Versions

0.51 (Kafka 4.1.2, 4.2.2, 4.3.1)

Features

Fixes

Configuration

Metrics

Postgres schema

Upgrade notes

Uh oh!

Inkless 0.50

Docker Images

Kafka Versions

0.50 (Kafka 4.1.2, 4.2.2, 4.3.1)

Features

Fixes

Configuration

Postgres schema

Uh oh!

Inkless 0.49

Docker Images

Kafka Versions

0.49 (Kafka 4.1.2, 4.2.1, 4.3.1)

Fixes

Postgres schema

Uh oh!

Inkless 0.48

Docker Images

Kafka Versions

0.48 (Kafka 4.1.2, 4.2.1, 4.3.1)

Features

Fixes

Configuration

Postgres schema

Uh oh!

Inkless 0.47

Docker Images

Kafka Versions

0.47 (Kafka 4.1.2, 4.2.1)

Features

Fixes

Configuration

Postgres schema

Uh oh!

Inkless 0.46

Docker Images

Kafka Versions

0.46 (Kafka 4.1.2, 4.2.1)

Features

Fixes

Uh oh!

Inkless 0.45

Docker Images

Kafka Versions

0.45 (Kafka 4.1.2, 4.2.1)

Features

Fixes

Uh oh!

Inkless 0.44

Docker Images

Kafka Versions

What's Changed

Contributors

Uh oh!

Inkless 0.43

Docker Images

Kafka Versions

What's Changed

Contributors

Uh oh!

Inkless 0.42

Docker Images

Kafka Versions

What's Changed

Contributors

Uh oh!