You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Repository navigation
Sorted sets (800 members): 53% more memory than Redis, and BGSAVE peak grows ~0.46 GiB under writes (Mastodon home feeds) #825
On a Mastodon-shaped home-feed workload (20,000 sorted sets of 800 members each — Mastodon's FeedManager::MAX_ITEMS), KiviDB 1.2.2 is much faster than Redis 8.10 under concurrency (up to 2.0× ops/s, half the read latency), but it:
stores the sorted sets in 53% more memory — 1.86 GiB vs 1.22 GiB used_memory for the same 16M members, and
grows by ~0.46 GiB during BGSAVE under write load (RSS 1.97 → 2.43 GiB peak), while Redis's fork-based save barely moves (1.208 → 1.219 GiB) on the same data and load.
These sets are 800 members, above Redis's 128-member listpack threshold, so Redis is using its skiplist encoding here — this is not the compact-encoding gap in #297. For sorted-set-heavy data the fork-free snapshot is currently a loss on peak memory, the opposite of what we see with string data (#4619-style workloads: ~half Redis's peak).
Numbers (two runs per server, each started fresh, same host)
Redis 8.10.3
KiviDB 1.2.2
used_memory after load (20k × 800 zset + 20k small zsets)
1,216 MiB
1,860 MiB
RSS after load
1.208 GiB
1.967 GiB
Peak RSS during BGSAVE under 200-connection write/read load
1.219 GiB
2.43 GiB
BGSAVE time
2 s
1 s
ops/s during the save
151k
263k
For context, the same workload's throughput (memtier, 4 threads):
Connections
Redis ops/s
KiviDB ops/s
read p50 / p99 (Redis)
read p50 / p99 (KiviDB)
48
157k
182k
0.30 / 0.40 ms
0.30 / 0.54 ms
200
155k
269k
1.25 / 1.68 ms
0.59 / 2.02 ms
500
157k
316k
3.17 / 4.13 ms
1.40 / 5.54 ms
(The p99 at 200/500 connections is also worth a look, but memory is the reason for this issue.)
Repro
Load (redis-py): for f in 1..20000: ZADD feed:home:u{f} with 800 members str(n) scored n (18-digit integers, e.g. 113000000000001000…), plus ZADD feed:home:u{f}:reblogs with one member.
Load (memtier_benchmark 2.6.1, --threads 4 --clients 50 --test-time 25 --key-prefix=u --key-minimum=1 --key-maximum=20000 --data-size 18), commands — the per-follower mix of Mastodon's FeedManager#add_to_feed + #trim, plus the home-timeline read:
5 s into that load: BGSAVE, sample docker stats every 0.5 s for 15 s, take the peak.
Environment
kividb/kividb:v1.2.2 vs redis:8 (8.10.3, --save ""), AWS CodeBuild 8 vCPU linux x86_64. Context: kividbio outreach to Mastodon (mastodon/mastodon#41702), where Redis memory is mostly these feeds.
Reopened: #926 packs sorted sets of up to 128 members (19-30 B a member against 116-132 before, Redis 20-25), but this issue's 800-member feeds are above that limit and unchanged (121 B a member against Redis's 80). A commit message in #926 closed this by mistake. What is left here is the large representation (#875).
Activity
murtazayusuf commentedon Nov 18, 2026
Reopened: #926 packs sorted sets of up to 128 members (19-30 B a member against 116-132 before, Redis 20-25), but this issue's 800-member feeds are above that limit and unchanged (121 B a member against Redis's 80). A commit message in #926 closed this by mistake. What is left here is the large representation (#875).
Small sorted sets in one block (collection storage, step 4)