A DAOS storage backend for NIXL, the NVIDIA Inference Xfer Library. It moves data between local memory and DAOS objects through the DAOS object API (dkey/akey), not the DFS file layer, and is meant for KV cache offload from Dynamo/KVBM and the LMCache NIXL backend.
Out of tree, on its way upstream. NIXL has no DAOS backend today. The design goes to NIXL as an RFC issue first and the PR follows once it is agreed; the draft of both is in
upstream/. This repository holds the plugin until then, plus its container build and testbed scripts.
- 34.17 GB/s reading 4.69 GiB (120 objects x 40 layers x 1 MiB) from 2 DAOS ranks (2.9.100 development build) over 400G verbs. Without folding the 40 layers into one RPC: 14.40 GB/s.
- DFS costs 0.63 ms fixed per object, the object API 0.0137 ms, on the same 4800 reads of 1 MiB.
- With
daos_serverkilled mid-read, the event-queue deadline returns in 8 s with every thread reclaimed; a blocking call was still stuck at 120 s. - On NIXL main, against an upstream DAOS 2.8.0-6 server (1 rank,
ofi+tcp), thereg,xferandagenttests pass. - vLLM 0.30 + LMCache 0.5.5 storing the KV cache through LMCache's NIXL storage backend into DAOS (4 ranks, kdev): every repeated prompt was a full hit read back from DAOS, 1,837 of 1,837 retrievals; TTFT p50 0.166 s against 0.749 s recomputed (e2e/vllm/).
The design and the descriptor mapping are in
src/plugins/daos/README.md. The full reports, with
method, raw numbers and the conclusions later retracted, are in
doc/measurements/:
throughput,
DFS vs object API, folding,
failure modes and
GPU memory.
| path | contents |
|---|---|
src/plugins/daos/ |
the plugin: nixlDaosEngine, meson build, user documentation |
test/unit/plugins/daos/ |
programs that need a live DAOS pool: test_reg, test_xfer, test_agent, plus test_gpu and bench_nixl (not part of the upstream submission) |
upstream/ |
the upstream submission: integration.patch and apply-integration.sh wire the plugin into a NIXL source tree; 0001-*.patch is the commit as it will be submitted; issue and PR drafts; ci/ holds an Ubuntu DAOS client image and build check for NIXL CI |
ci/build.sh |
builds NIXL + the plugin inside a DAOS client image and produces the runtime image nixl-daos:dev (images/Dockerfile.runtime) |
ci/e2e-testbed.sh |
runs the tests from that image against an existing DAOS pool on a client host |
e2e/vllm/ |
vLLM + LMCache image whose NIXL storage backend writes to DAOS through this plugin, with the LMCache patch it needs and the measured result |
doc/measurements/ |
measurement reports behind the numbers above |
doc/adr/ |
design decisions specific to this repository |
# NIXL source checkout next to this repository; DAOS client image with daos-devel
ci/build.sh ../nixl registry.example/daos-client:2.8.0 4
# against a live pool, from a host running daos_agent
ci/e2e-testbed.sh root@client-host <pool> <daos-system-name> nixl-daos:devInside a container, reaching the host daos_agent socket needs
--security-opt label=disable on SELinux hosts.
Shared with the other Gluesys DAOS repositories (daos-operator, daos-csi, daos-images): fixes go upstream first (daos-stack, ai-dynamo/nixl, LMCache), and destructive DAOS operations (format, wipe) are never automated.
Apache-2.0. See LICENSE.