All notable changes to minikv will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- No unreleased entries yet.
A correction release: where 2.0.0 reported work that it did not do, minikv now fails explicitly, and the data losses found on the way are fixed.
Some earlier entries describe features that the code never provided. They stay as they were written. In short:
- There are no virtual shards and no rebalancing: placement ranks the live volumes by rendezvous hashing (HRW) on each key.
verify,repair,compact,rebalance,upgradeandstream, and the matching admin routes, never ran: they returned fixed reports.- Backups copied nothing, restores restored nothing, and TTLs were never applied.
- The Kubernetes operator, io_uring, the Kafka CDC sink, tiering compression, time-series downsampling and OpenTelemetry tracing do not exist, and nothing encrypts data at rest.
- BREAKING (HTTP):
POST /admin/verify,/admin/repair,/admin/compactand/admin/scaleanswer501 Not Implemented, with{"error":"not implemented","feature":…,"roadmap":…}, instead of200and an invented report. So do the backup and restore routes, which answered503or an empty list. A write with anX-Minikv-TTLheader answers501and stores nothing. - BREAKING (CLI):
verify,repair,compact,rebalance,upgradeandstreamexit with status 1 and name the release planned for them.get,putanddeleteexit with status 1 when the coordinator answers an error. - BREAKING (metrics):
/metricsno longer exports the series that nothing updated (minikv_requests_total,minikv_errors_total,minikv_bytes_read_total,minikv_bytes_written_total,minikv_active_connections,minikv_keys_with_ttl,minikv_rate_limited_requests,minikv_request_duration_msand the per-endpoint series), norminikv_volume_free_bytesandminikv_s3_objects_with_ttl, which were always 0.minikv_raft_rolebecomes one gauge per role:minikv_raft_role{role="leader"} 1. - BREAKING (library): functions that reported work they did not do now return errors:
MiniKVController::run,reconcileandhandle_delete;IoUring::newandUringFile::openwhenenabledis set;BackupManager::start_restore;TimeseriesEngine::run_downsampling; the tieringcompressanddecompressfor any algorithm butNone;CDCManager::create_sinks_from_configwith a Kafka sink;CDCManager::flushwhen a sink fails;BlobStore::compact. - The vector index and
audit.logare stored under--db; a vector index left by 2.0.0 incoord-data/is loaded. Audit entries name the actorunauthenticated. minikv getwrites the raw bytes, to standard output when--outputis absent.--coordinatoris accepted before or after the subcommand.- The remote datacenters of
ReplicationManagerstart unhealthy, never contacted. - Removed
scripts/admin.sh,scripts/test_raft_cluster.sh,fix_ci_complete.shandcobertura.xml, and the unused dependenciesraft,raft-proto,opentelemetryandtracing-opentelemetry.
- A restarted volume server rebuilds its index from its segments and its WAL: deleted blobs and old versions no longer come back, and the index holds the BLAKE3 hash of the values. LZ4-compressed values can be read back.
- Time series: a second write in the same hour no longer replaces the first, a query that starts inside an hour finds its points, and a range that ends before it starts no longer panics.
/metricsfollows the Prometheus text format, withHELPandTYPElines./admin/exportand/searchreport the values that they cannot read./searchmatches bytes, and answers500when the key listing fails./admin/statusreportsnb_s3_objectsasnull, instead of 0, when the key listing fails. The time-series stats list only what the engine supports, and the geo and CDC status answers drop their invented fields.- The admin dashboard reads the fields that
/admin/statusreturns, and shows the real version. minikv-coordlogs a warning for each[coordinator]setting of the configuration file that it does not apply, andconfig.toml.exampleloads.- The tests no longer rewrite
config.toml. - HRW placement breaks ties by volume id.
- The benchmarks count only
200as a success, andscripts/benchmark.shsends its requests to the leader of a cluster that can elect one. The coordinator image checks/health/live, and the alert rules on metrics that do not exist are gone. - Documentation: placement, what is implemented and what answers
501, configuration, and the module docs.
These items stay for compatibility and will be removed in 3.0.0:
ConsistentHashRing,blob_prefix,Error::ShardNotFound, andPlacementManager::get_shard,rebalanceandget_shard_volumes: not wired into placement.StoreStats::bloom_false_positivesandcompressed_blobs: always 0.volume::http::Locationandget_location, andVolumeGrpcService::new.maybe 8000 _encryptandmaybe_decrypt: they return their input when encryption or decryption fails.UringFile::async_read,async_writeandflush_async: they do nothing.- The
shardfield ofBlobLocationholds a segment number: read it withBlobLocation::segment().
- Reworked the distributed layer: coordinators replicate metadata through a persistent Raft log (log up-to-date checks on elections, majority commit, leader step-down when the quorum is lost)
- Writes use two-phase commit between the leader coordinator and the volume servers; each object version gets its own blob, and older versions are cleaned up after the switch
- Reads are linearizable on every coordinator (ReadIndex)
minikv-volume serveruns a gRPC storage service, a/healthendpoint and heartbeats to every coordinator- BREAKING:
--coordinatorsis required and should list every coordinator - BREAKING: batch
getreturns the stored value instead of the key metadata - BREAKING: public Rust API:
KeyMetadatahas a newblob_idfield,VolumeServer::open(VolumeServerConfig)replacesVolumeServer::new,VolumeClient::preparetakes the payload, and the unusedRaftNodesnapshot and election helpers were removed - BREAKING: coordinator data directories from 1.x are not compatible; start with fresh ones
DELETE /s3/:bucket/:keyandGET/PUT/POST/DELETE /:key, used by theminikvCLIPOST /internal/volumes/heartbeat/admin/statusreports the term, leader, commit index and log length- End-to-end test with 3 coordinators and 3 volumes, including leader failover
- The coordinator starts without a
config.toml - Peer ports and coordinator list in
docker-compose.yml
- Notebook-first Python SDK with analytics-oriented client helpers
- End-to-end quickstart example for vector search and time-series workflows
- Dataframe conversion helpers for pandas, polars, and pyarrow
- New vector endpoints:
POST /vector/upsertPOST /vector/queryGET /admin/vector/stats
- Cosine similarity top-k matching
- Persistent on-disk vector index snapshot (
coord-data/vector_index.json)
- Production handlers for
POST /ts/writeandPOST/GET /ts/query - Automatic initialization of the time-series engine
- Query support for tags, aggregation, resolution, and limits
- Admin time-series stats endpoint now backed by live engine stats
- Helm chart added with dev/staging/prod values profiles
- Observability bundle:
- Grafana dashboard provisioning and default minikv overview dashboard
- Prometheus alert rules for availability and latency/error SLOs
- Backup/restore runbook for release and disaster-recovery drills
- GA preflight release script and Makefile targets for release validation
- README restructured for GA usage and operations
- CONTRIBUTING guide updated for v1 release workflow
- Release engineering runbook added for reproducible v1.0.0 process
- Updated shell automation scripts to use Kubernetes-aligned health probes (
/health/live)
- Custom Resource Definition (CRD) for
MiniKVCluster - Automated deployment and scaling of coordinator and volume nodes
- RBAC configuration with proper permissions
- StatefulSet management for persistent storage
- ConfigMap generation from cluster spec
- Horizontal Pod Autoscaler (HPA) support
- ServiceMonitor for Prometheus integration
- Example manifests for basic and production clusters
- Dedicated
TimeseriesEnginefor time-series workloads - Multiple resolution levels (raw, 1min, 5min, 1hour, 1day)
- Automatic downsampling with configurable retention
- Aggregation functions: sum, avg, min, max, count, first, last, stddev
- Delta and gorilla compression for efficient storage
- Time-range queries with aggregation support
- Geographic data locality for compliance (GDPR, data residency)
- Multiple routing strategies:
- Nearest region (latency-based)
- Primary region (consistency-based)
- Round-robin (load distribution)
- Geo-fenced (compliance-based)
- Haversine distance calculation for region selection
- Per-key prefix region assignment
- Region health monitoring with automatic failover
- Geo-fencing rules for data sovereignty
- Automatic data movement between storage tiers
- Tier levels: Hot, Warm, Cold, Archive
- Policy-based tiering rules:
- Access frequency (access count in time window)
- Data age (time since creation)
- Value size (bytes)
- Last access time (idle duration)
- Access pattern tracking with history
- Compression per tier (none, lz4, zstd, snappy, gzip)
- S3-compatible archive tier support
- Statistics and reporting for tier distribution
- Zero-copy I/O operations using Linux io_uring
- Batched submissions for reduced syscalls
- Configurable submission/completion queue depths
- Optional kernel polling (SQPOLL) for ultra-low latency
- Registered buffers for zero-copy transfers
- Direct I/O bypass of page cache
- Graceful fallback to standard I/O on unsupported systems
- Write batching for small operations
k8s_operator.rs- Kubernetes controller and CRD typestimeseries.rs- Time-series storage enginegeo.rs- Geo-partitioning routertiering.rs- Data tiering managerio_uring.rs- io_uring I/O backend
k8s/crds/minikvcluster.yaml- CRD definitionk8s/rbac/operator-rbac.yaml- Operator permissionsk8s/operator/deployment.yaml- Operator deploymentk8s/examples/basic-cluster.yaml- Minimal cluster examplek8s/examples/production-cluster.yaml- Production-ready example
- Asynchronous replication to remote datacenters
- Multiple conflict resolution strategies:
- Last-Write-Wins (timestamp-based)
- Vector Clocks (causality tracking)
- Local-First (prefer local DC writes)
- Primary-First (prefer primary DC writes)
- DC-aware routing for read/write operations
- Configurable replication lag monitoring and alerts
- Per-datacenter replication status tracking
- Real-time capture of all data changes (INSERT, UPDATE, DELETE)
- Configurable sinks for event delivery:
- Webhook sink (HTTP POST to external endpoints)
- Kafka sink (for event streaming platforms)
- File sink (for local debugging/archival)
- Memory sink (for testing)
- Event filtering by operation type and key prefix
- Sequence numbers for guaranteed ordering
- Old value capture for UPDATE and DELETE operations
- Embedded web dashboard for cluster monitoring
- Real-time cluster status visualization
- API key management interface
- Backup/restore controls with progress tracking
- Plugin management UI
- Cross-DC replication status monitoring
- R 8000 esponsive dark theme design
- Full backup support (complete snapshot)
- Incremental backup support (changes since last backup)
- Backup compression (configurable)
- Backup encryption support
- Multiple backup destinations:
- Local filesystem
- S3-compatible storage
- Point-in-time recovery
- Checksum verification during restore
- Backup manifest with metadata
- Extensible plugin architecture
- Plugin types:
- Storage plugins (custom backends)
- Auth plugins (custom authentication)
- Hook plugins (event listeners)
- Middleware plugins (request interceptors)
- Plugin lifecycle management (load, enable, disable, unload)
- Plugin dependencies and version compatibility
- Built-in logging hook plugin example
GET /admin/ui- Admin web dashboardPOST /admin/backup- Create a new backupGET /admin/backups- List all backupsGET /admin/backups/:id- Get backup detailsDELETE /admin/backups/:id- Delete a backupPOST /admin/restore- Restore from backupGET /admin/replication/status- Replication statusGET /admin/plugins- List pluginsPOST /admin/plugins/:id/enable- Enable a pluginPOST /admin/plugins/:id/disable- Disable a pluginGET /admin/cdc/status- CDC status
- Added
async-traitfor async plugin traits - New modules:
replication,cdc,backup,plugin,admin_ui - Comprehensive unit tests for all new features
- Vector clock implementation for distributed causality
POST /admin/import- Batch import key-value pairs from JSON payloadGET /admin/export- Streaming export of all key-value pairs as NDJSON
POST /transaction- Execute multiple operations (put/delete) in a single request- Returns detailed results for each operation with success/error status
GET /search?value=<substring>- Search for keys whose values contain the specified substring
- S3-compatible API now supports pluggable persistent storage backends (RocksDB, Sled)
- Objects can be stored durably by configuring storage backend in config.toml
- API Key Authentication - Secure access control with API keys
- Generate and manage API keys via admin endpoints
- Keys are securely hashed using Argon2id
- Support for key expiration and revocation
- Headers:
Authorization: Bearer <api_key>orX-API-Key: <key>
- JWT Token Support - Stateless authentication tokens
- Generate JWT tokens from valid API keys
- Configurable token expiration (default: 24 hours)
- HMAC-SHA256 signature verification
- Role-Based Access Control (RBAC) - Fine-grained permissions
- Three role levels: Admin, ReadWrite, ReadOnly
- Middleware enforcement on all protected routes
- Role-based endpoint restrictions
- Multi-tenancy - Tenant isolation for data
- Tenant identifier attached to each API key
- S3 objects tagged with tenant ownership
- Tenant extraction from authenticated requests
- Encryption at Rest - AES-256-GCM data encryption
- HKDF-SHA256 key derivation from master key
- Per-object random nonces for security
- Separate keys for data and WAL encryption
- Transparent encryption/decryption with backward compatibility
- Tenant Quotas - Resource limits per tenant
- Storage limits (bytes)
- Object count limits
- Request rate limiting per tenant
- Prometheus metrics for quota usage
- Audit Logging - Structured audit logs for all admin and sensitive actions (file + stdout)
- Persistent Storage Backends - Pluggable storage: in-memory, RocksDB, Sled (configurable via config.toml)
- Watch/Subscribe System - Real-time key change notifications (WebSocket & SSE endpoints, production-ready)
- Subscribe to key changes via
/watch/sse(SSE) or/watch/ws(WebSocket) - Events: PUT, DELETE, REVOKE (with key, tenant, timestamp)
- Integrated with all S3/data and admin modification endpoints
- Subscribe to key changes via
POST /admin/keys- Create new API keyGET /admin/keys- List all API keysGET /admin/keys/:id- Get specific API key detailsPOST /admin/keys/:id/revoke- Revoke an API keyDELETE /admin/keys/:id- Delete an API keyGET /admin/audit- Download or stream audit logs (NEW)GET /admin/subscribe- Subscribe to key change events (NEW, preview)
- Constant-time password verification with Argon2
- Secure key generation using cryptographic RNG
- Authentication middleware for route protection
- Request validation and tenant context propagation
- Audit log hooks in all admin and data modification endpoints
- Pluggable backend: select in-memory, RocksDB, or Sled via config
- S3/data endpoints refactored to use trait abstraction
- Persistent storage for all S3/data paths when enabled
- Audit log file and stdout output
- Prometheus metrics for audit, quota, and storage backend
- Watch/subscribe system for real-time notifications (preview)
- S3 store entries now include tenant field
- Authorization required for protected endpoints (when auth enabled)
- Storage backend must be selected in config (default: in-memory)
- TTL Support - Keys can now expire automatically with millisecond precision
- Set TTL via
X-Minikv-TTLheader on PUT requests - Automatic cleanup of expired keys
- Set TTL via
- LZ4 Compression - Optional transparent compression for values > 1KB
- Configurable via
CompressionMode::Lz4 - Automatic decompression on read
- Configurable via
- Rate Limiting - Token bucket algorithm with per-IP tracking
- Configurable requests per second and burst size
- Returns
X-RateLimit-*headers
- Kubernetes Health Probes - Production-ready health endpoints
/health/ready- Readiness probe (checks volumes + Raft)/health/live- Liveness probe (always returns OK)
- Enhanced Metrics - Prometheus histograms and per-endpoint stats
- Request latency histograms with configurable buckets
- Per-endpoint request/error counters
- TTL and rate limiting metrics
- Request Tracing - Structured logging with request IDs
- Unique
X-Request-IDheader for each request - Correlation across distributed components
- Unique
- Updated blob format to support compression metadata
- Index snapshots now include TTL expiration data (KVINDEX3 format)
- Better WAL replay with v0.5.0 format support
- Blob storage format changed (existing data will be migrated on read)
- Index snapshot format updated to KVINDEX3
- Admin dashboard endpoint (
/admin/status) for live cluster state and monitoring (NEW)- Shows role, leader, volumes, S3 object count, and more, for monitoring and UI integration
- S3-compatible API (PUT/GET, in-memory demo) as a new object storage interface (NEW)
- Store and retrieve objects via
/s3/:bucket/:key
- Store and retrieve objects via
- Full documentation and automated tests for all new endpoints
- Better system observability: admin and metrics endpoints now cover all cluster state
- Clean separation of admin/user APIs
- Documentation expanded and migrated for new features
- Test coverage increased for dashboard and S3 features
- All v0.4.0 roadmap features implemented and tested
- Ready for integration with UIs, external metrics dashboards, and S3-demo clients
- Cluster state easily visible and integrable via admin endpoint
- Range queries (efficient scans across keys)
- Batch operations API (multi-put/get/delete)
- TLS encryption for HTTP and gRPC (production-ready security)
- Flexible configuration: file, environment variables, and CLI override
- All code, comments, and documentation now in English
- 100% green CI: build, test, lint, format
- CLI and API fully support new batch and range operations
- Example config and all templates now in English
- Refined error handling and configuration merging
- Documentation and README updated for v0.3.0
- All v0.3.0 roadmap features are implemented and production-ready
- No incomplete logic remains
- Ready for enterprise deployment and future advanced features
- Full multi-node Raft consensus: leader election, log replication, snapshots, commit index, recovery, partition detection
- Advanced Two-Phase Commit (2PC) streaming: chunked blob streaming, error propagation, retry, timeouts
- Automatic cluster rebalancing: detects overloaded/underloaded volumes, moves blobs and updates metadata
- Prometheus metrics endpoint: /metrics exposes cluster and volume stats, Raft role, replication lag, health
- Professional integration, stress, and recovery tests
- All scripts, test templates, and documentation translated/adapted to English
- All core features are implemented and production-ready
- No incomplete logic remains
- All documentation, comments, and scripts are in professional English
- Ready for enterprise deployment and further extension
- Raft consensus for coordinator high availability
- 2PC (Two-Phase Commit) for distributed writes
- Write-Ahead Log (WAL) for durability with configurable fsync
- Dynamic sharding across 256 virtual shards
- HRW (Highest Random Weight) placement for replica selection
- gRPC internal coordination protocol
- HTTP REST API for public access
- Segmented append-only log architecture
- In-memory HashMap index for O(1) lookups
- Bloom filters for fast negative lookups
- Index snapshots for 5ms restarts (vs 500ms rebuild)
- CRC32 checksums on every record
- Automatic background compaction
- RocksDB metadata store for key → replicas mapping
- Raft leader election (simplified single-node for v0.1)
- Volume health monitoring
- Placement manager with shard rebalancing
- RESTful HTTP API: PUT, GET, DELETE
- Blob storage with segmented logs
- WAL for durable writes
- gRPC service for 2PC operations
- HTTP API for direct blob access
- Automatic compaction based on threshold
- Index snapshot persistence
- Coordinator commands:
serve,compact,rebalance - Volume commands:
serve,compact - CLI: verify, repair, batch, range