Repository navigation
Benchmark vector sets (VADD/VSIM) against Garnet and Redis 8 #627
Description
The vector suite in benchmarks/1.2.0/vector compares KiviDB, Redis 8.10 and Dragonfly through the FT.* search commands. Garnet doesn't implement FT.*, so it isn't in that suite.
Garnet does have Redis-style vector sets (VADD/VSIM/VREM/VEMB/...) as a preview feature: --enable-vector-set-preview, DiskANN-based, about 15 open vector sets per server. Docs: https://microsoft.github.io/garnet/docs/commands/vector-sets. KiviDB implements the same commands (src/commands/vset.rs), and so does Redis 8.
To do for 1.3.0:
- Check whether redis/vector-db-benchmark can drive vector sets. If it can't, write a small harness: load with
VADD(batched, N clients), query withVSIM ... COUNT 10 EF <ef>at 1/16/100 clients, and compute recall@10 against the dataset's ground truth. - Use the same datasets and VMs as the 1.2.0 vector suite: dbpedia-100K and 1M (1536-d), glove-25. h-and-m filtered too, if
VSIM ... FILTERis comparable across engines; Garnet's inline filtering is still in progress. - Engines: KiviDB, Redis 8.10 vector sets, Garnet (latest release, preview flag on). Dragonfly too, if it implements vector sets by then.
- Report q/s at each recall level, ingest time, memory per vector, and p99, with the harness committed under
benchmarks/<version>/vector-sets/so anyone can rerun it. - State clearly that Garnet's vector sets are a preview.
Activity