English | 简体中文
AMD RDNA4 backend: Radiance on Windows and Linux, with build and deployment commands, default settings, and full benchmark comparisons.
Radiance 1.17: observer replay, transfer and execution-overlap measurements, including opt-in controls and qualification limits.
Long-context conversations and AI agents on limited GPU memory.
KVMem keeps complete native KV history in system RAM, with optional SSD tiering. The GPU holds the history needed for the current request and space for generation. It supports NInfer, llama.cpp, PolyStrata and Radiance through a local Web UI and OpenAI-compatible APIs. See the Radiance integration guide for its AMD qualification and supported combinations.
These are the main model families supported with KVMem history management enabled. Choose the backend and model format together.
| Backend | Models | Model format |
|---|---|---|
| NInfer | Qwen3.5 Dense (0.8B / 2B / 4B / 9B / 27B), Qwen3.6 / Qwen3.8 27B, and compatible fine-tunes, including Ternary Bonsai 2 27B | .ninfer — use a compatible artifact or convert the weights first |
| llama.cpp | Qwen3.5 Dense, Qwen3.6 / Qwen3.8 27B, Ternary Bonsai 2 27B, and other compatible full-attention GGUF models | .gguf |
| PolyStrata | Qwen3.6-35B-A3B, Ornith 1.5, Tiel Coder / Cyber-Tiel 35B-A3B, compatible qwen35moe models, and Gemma 4 26B-A4B |
.gguf; image input also needs a matching mmproj |
NInfer's KVMem integration currently covers Dense models; PolyStrata's covers the qwen35moe MoE architecture. Other models supported by those upstream engines are outside this KVMem scope. llama.cpp's KVMem adapter skips sliding-window-attention (SWA) and purely recurrent models.
Architecture support does not mean every size, quantization or feature combination has been tested. See the Qwen3.5 Dense family coverage, Bonsai integration and PolyStrata integration. Availability, vision and MTP support depend on the model components and the release package.
The table covers the main GPU families for KVMem, with package and source-build differences noted below.
| Backend | GPU families / example cards | Scope |
|---|---|---|
| NInfer | NVIDIA RTX 20 / 30 / 40 / 50, TITAN RTX, Tesla V100 | CUDA only. V100 needs an sm_70 / CUDA 12 build or a matching older package. A100-class sm_80 is a source compatibility target without KVMem hardware qualification. |
| llama.cpp | NVIDIA RTX 20 / 30 / 40 / 50 (CUDA); AMD Radeon RX 6800 / 6900, RX 7700 XT / 7800 XT / 7900 XT / XTX, RX 9060 XT / 9070 / 9070 XT (HIP/ROCm) | AMD availability depends on the SDK, OS and compiled GPU targets; see the ROCm guide. RX 9060 XT has recorded KVMem long-context tests. |
| PolyStrata | NVIDIA RTX 20 / 30 / 40 / 50 | KVMem uses one CUDA GPU. The opt-in AMD HIP source integration is experimental, pending AMD inference qualification; see integration status. |
RTX examples include 2060 / 2080 Ti, 3060 / 3090, 4060 Ti / 4090, and 5050 Laptop / 5060 Ti / 5090. The series name alone does not guarantee enough VRAM for a chosen model and configuration.
Choose a release package matching your GPU, OS and backend. The unified Windows packages cover RTX 20 / 30 / 40 / 50 with NInfer and PolyStrata; llama.cpp and AMD use separate packages or source builds. A compiled target is not a test of every card: check the package's instructions and validation report for driver requirements and actual hardware coverage.
- Long context: Retrieve relevant KV history to manage long conversations and tool histories within a fixed GPU budget.
- Extra-long / xhigh reasoning: Reuse GPU generation space to sustain long reasoning and answers on limited VRAM, including the
xhighreasoning setting. - State reuse: Reuse prefixes and checkpoints across conversations and tool continuations according to each backend's rules.
- Memory tiering: Manage KV history across RAM and SSD, loading selected history onto the GPU as needed.
RTX 5060 Ti 16 GB · 27B IQ3 model · real local runs. The pelican cycling animations below are original HTML outputs from the release video, generated with medium and xhigh reasoning settings.
medium |
xhigh |
|---|---|
KVMem makes room as generation proceeds, so Gen reserve is reusable workspace, not the cumulative output limit. Context, request limits and model stop conditions still apply. These are individual examples; see the demo notes for provenance and scope.
Stable release · All releases, including previews and older versions
Choose a package for your operating system, GPU and backend on the release page. Each release contains its downloads, checksums, driver requirements, model support and qualification notes. Supply model weights separately. Releases marked Pre-release are for testing.
- Download and fully extract a package. Use English paths for the package and models; the full paths must not contain Chinese characters.
- For a Windows console package, double-click
START-HERE-EN.bat, then select the backend, model and GPU. UseSTART-HERE-CN.batfor Chinese. - Start the service. Once ready, open Chat from the console or connect your client to the API address shown there.
NInfer typically uses .ninfer models; llama.cpp uses GGUF. Model, quantization and vision support depend on the backend and release. For older packages without a one-click entry, follow the included instructions.
See Windows one-click startup, the Linux launcher and the console guide.
The service provides OpenAI-compatible /v1/chat/completions and /v1/responses endpoints with streaming. Tool and multimodal support vary by backend. The API base is http://127.0.0.1:<service-port>/v1; query /v1/models for the loaded model ID.
Configure context limits, GPU history budgets, generation reserve and RAM quotas through the console or launch options. Long contexts still need sufficient system RAM; model weights and inference workspaces also use GPU memory. See the runtime guide and backend differences for parameters, defaults and supported combinations.
git clone --recurse-submodules https://github.com/kvmem/kvmem.git
cd kvmem
python scripts/prepare-backends.py
python scripts/prepare-backends.py --checkEngines use pinned submodules and integration patches. GitHub-generated source ZIPs omit submodule contents. See source preparation and builds for requirements and commands, and the ROCm guide for AMD. Mainline source and published packages may have different capabilities.
| Topic | Guides |
|---|---|
| Startup, settings and support | Runtime guide · Backend differences |
| KV retrieval and memory | Graph recall · RAM / SSD tiering |
| Design and backend integration | Architecture · Memory backend contract |
| Builds and releases | Development workflow · Backend versions and licenses |
| Tests and performance | Validation records · Performance report · Benchmark data |
Reports retain their original versions, hardware and settings; their results do not apply to every package or device. Browse docs/ for more topics.
QQ group: 1040777853 · GitHub Issues
Include the version, GPU, driver, model, launch settings and error logs when reporting an issue. Remove private conversation content from logs.
Thanks to ggml-org/llama.cpp, Neroued/ninfer, iamwavecut/ninfer-all and community contributors, including Ryan-gsq and contributors to 3090/4090 adaptations, adaptive MTP, fast prefill and ngram. LRFU/GQA max references melis2023/kvmem-actkv. Thanks to testers including melis and redsnow23.
Each backend retains its upstream license and third-party notices; see the backend manifest and package licenses. cuBLAS follows the CUDA EULA. Models are distributed separately under their own terms.
KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU
BibTeX
@misc{chai2026kvmem,
title = {{KVMem}: Virtualizing Million-Token Agent Workspaces on a Consumer {GPU}},
author = {Di Chai and Leye Wang and Zeshen Su and Zhiguo Xia and Zhihang Yu},
year = {2026},
eprint = {2609.04852},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2609.04852}
}