Need help? Join us on Discord: https://discord.gg/8EAvYPV2rE
gr8 serves GLM-5.3, GLM-5.3-Flash and DeepSeek-V4.1-Flash on one to six DGX Spark (GB10) boxes (the count depends on
the model, see below). The model is split across all the boxes (tensor parallel, TP). gr8 is written from scratch in
CUDA and uses no vLLM, PyTorch or NCCL at serve time. Each box runs one engine process (g6m4) and is one rank. The
ranks talk over RoCE v2 through gr8's own RDMA code, and an OpenAI-compatible API runs on the first box. Beta
release. If you run it, please tell us what breaks: issues and PRs are welcome.
- Three models, one to six boxes. GLM-5.3 on three to six boxes, GLM-5.3-Flash on one to four, DeepSeek-V4.1-Flash on four (Models).
- Boots from its own cache. On GLM-5.3 each box fetches only its own part of the release, once. After that, GLM-5.3 boots in about 40 s and GLM-5.3-Flash on one box in 45 s.
- Speed. On one stream writing code, GLM-5.3 decodes 57.4 tok/s on six boxes (RigMark), GLM-5.3-Flash 54.3 tok/s on one box and DeepSeek-V4.1-Flash 104.6-105.9 tok/s on four (gr8's own bench). The Models table has prose, prefill and dates.
- Many chats at once, and long ones. GLM-5.3 decodes 8 requests at once from a KV pool of up to 614,400 tokens
(TP=6). GLM-5.3-Flash on one box decodes 4 at once from a 700,032-token pool. DeepSeek-V4.1-Flash runs 5 chats at
once by default and up to 16, and with
CAP=1048576one request can hold up to 1,048,566 tokens, prompt and reply (docs/FIRST-BOOT.md). - Prefix reuse and KV below the pool. A request that continues a conversation prefills only its new part: a continued 20K-token conversation takes 7.1 s instead of 17.3 s (GLM-5.3, TP=4). On GLM-5.3, evicted prefixes can go to an SSD or to a KV node with spare memory, over RDMA. From a KV node, the next turn of a 445,855-token prompt answered in 11.4 s (experimental, Remote KV). GLM-5.3-Flash can keep its session checkpoints in the same kinds of store (experimental). DeepSeek-V4.1-Flash has no store below the device.
- Image input. DeepSeek-V4.1-Flash takes images in chat requests (opt-in,
GR8_DS41_VISION=1, docs/DS41-VISION.md). - A Docker image.
ghcr.io/kindlingai/gr8has the engine for every TP width and all three models, so GLM-5.3 on three to six boxes needs no build (Docker Compose). The one-box and two-box recipes publish their own images.
gr8 is available in easy-to-use recipes preconfigured for the best performance for a given
model and cluster size.
- One DGX Spark: https://github.com/kindlingai/glm-5.3-flash-exlr8-1x-spark
- Two DGX Sparks: https://github.com/kindlingai/glm-5.3-flash-exlr8-2x-spark
| model | boxes | weights on Hugging Face | one stream: decode code / prose, prefill of an 8K prompt | page |
|---|---|---|---|---|
| GLM-5.3 | 3 to 6 | kindlingai/glm-5.3-exlr8-k2-k3-k4 | six boxes: 57.4 / 43.1 tok/s, 1,808 tok/s (RigMark, 2026-10-24; one box clock-capped) | docs/GLM-5.3.md |
| GLM-5.3-Flash | 1 to 4 | kindlingai/glm-5.3-flash-exlr8-k2-k3-k4 | one box: 54.3 / 29.1 tok/s, 1,214 tok/s (gr8 glm53f bench, 2026-11-06) |
docs/FLASH.md |
| DeepSeek-V4.1-Flash | 4 | kindlingai/deepseek-v4 8000 .1-flash-exlr8-native | four boxes: 104.6-105.9 / 61.7-62.0 tok/s, 3,197-3,700 tok/s (bench/bench.py, two runs, 2026-11-02) |
docs/DEEPSEEK.md |
Each page has the model's results, how to start it and where its known issues are.
| boxes | model | start here | download |
|---|---|---|---|
| 1 | GLM-5.3-Flash | glm-5.3-flash-exlr8-1x-spark: one docker run (below) |
87 GB |
| 2 | GLM-5.3-Flash | glm-5.3-flash-exlr8-2x-spark: the same image on both boxes | about 86 GB a box |
| 3 or 4 | GLM-5.3-Flash | docs/FLASH.md | see the page |
| 3 to 6 | GLM-5.3 | Docker Compose (below), or docs/DEPLOY.md for bare metal and Kubernetes | about 90 GB a box with 4 boxes (TP=4) |
| 4 | DeepSeek-V4.1-Flash | docs/DEEPSEEK-BOOT.md: fetch the whole release to every box, then write a token map once | 510.6 GB a box, fetched before the first boot |
Each recipe repo pins one image and one set of settings for its box count.
One box (GLM-5.3-Flash), with Docker and the NVIDIA container toolkit (both come with DGX OS). This is the 1x recipe's command; its README has the settings and an API example:
docker run -d --name glm53f-1x-spark --gpus all --network host --pid host \
--ulimit memlock=-1 --ulimit core=0 --cap-drop ALL --cap-add IPC_LOCK \
--security-opt no-new-privileges --read-only --tmpfs /tmp:size=1g,mode=1777 --init --oom-score-adj 1000 \
-v glm53f-cache:/var/lib/gr8/cache -v glm53f-prompts:/var/lib/gr8/prompts -v glm53f-home:/var/lib/gr8/home \
ghcr.io/kindlingai/glm-5.3-flash-exlr8-1x-spark:latest
docker logs -f glm53f-1x-spark # ready at "serving: http://..."Three to six boxes (GLM-5.3), with Docker Compose. On every box:
mkdir -p ~/gr8 && cd ~/gr8
curl -fsSLO https://raw.githubusercontent.com/kindlingai/gr8/main/docker/compose.yaml
curl -fsSL https://raw.githubusercontent.com/kindlingai/gr8/main/docker/.env.example -o .env
nano .env # GR8_MODEL, MODEL, PEERS and (glm53 at TP=4) DRAFTER: the same file on every box
docker compose up -dThen docker compose exec gr8 /opt/gr8/gr8 glm53 smoke on the head. docs/DEPLOY.md explains each
setting, and has bare metal and Kubernetes. An agent that starts gr8 for someone asks them the questions in
docs/FIRST-BOOT.md first.
DeepSeek-V4.1-Flash has no one-command path yet. docs/DEEPSEEK-BOOT.md is its boot in
order, from the four boxes to a passing ./gr8 ds41 smoke.
model-loader can fetch the weights instead of gr8: the boxes copy the pieces between them over the LAN, from one Hugging Face fetch or a NAS. Each surface takes it with one more file or value: docs/DEPLOY.md.
Each model's page gives its box count. docs/GLM-5.3.md and docs/DEEPSEEK-BOOT.md add memory and disk.
- DGX Spark or other GB10 boxes (sm_121a, 128 GB unified memory), one rank per box, with nothing else holding much memory.
- One box needs no fabric and no LAN: at TP=1 the launcher skips the RDMA checks (docs/FLASH.md).
- Two or more boxes need a RoCE v2 fabric with two rails: each box's ConnectX-7 has one address on each PCIe root. Two
boxes need no switch: one 200G cable joins their ConnectX ports, as in the
two-box recipe. For three or more we use one
200G switch. At TP=3 or 4,
--shape ringruns without one: each box's f0 port is cabled to the next box's f1 port, and at TP=4 the diagonal's writes are relayed by the box between, through a 32 MB ring per fabric (about 0.25 GB pinned per rank). It costs about 10% of single-stream decode (4-5% at 4 streams). memlock must be unlimited. With mentat (--mentat localhost --tp 3 --shape ring, mentatd 0.24.0 or later on every box), the cabling can be any ring: the boxes claim a ring from mentat and take their rank order and RDMA devices from its one answer, so a scheduler such as a Kubernetes LeaderWorkerSet can place them in any order. Tag both addresses of each ConnectX portrdmain mentat (MENTAT_ANNOUNCE_IFACES). - Two or more boxes also need a LAN between them with TCP ports 7340 and 7341 open. The boxes need no ssh between them: the head engine passes each request to the other ranks over its control links.
- Proactive memory compaction off on every box (docs/DEPLOY.md has the commands). A rank fills the box to about 2 GB under its limit, and at that level the kernel's background compaction moves pages under the engine in bursts that delay every collective.
- To build: CUDA 13 (
nvccwithsm_121a), g++ with C++20, CMake, clang and libclang-dev (the front end builds XGrammar), rdma-core / libibverbs-dev, Rust through rustup (the front end needs Rust 1.85 or later; Ubuntu 24.04'scargopackage is 1.75), and Python 3 with venv. The stock DGX OS (Ubuntu 24.04) has all of them except Rust. Or, in place of all of these, Docker with the NVIDIA container toolkit (DGX OS has both): see Docker Compose.
Every box runs the same ./gr8 <model> up, where the model is glm53, glm53f or ds41
(docs/DEPLOY.md). The first address in --peers is the
head: it runs the API on port 8100. Before any engine starts, the boxes compare their options, model and engine build,
and if one box differs, every box stops and prints the difference. Without --peers, the boxes find each other through
a local mentat daemon (--mentat localhost, the default).
| command | what it does |
|---|---|
up [OPTIONS] |
start this box's supervisor and follow it until the cluster serves |
down, restart |
stop every box, or stop and boot again (in Docker: down, then up -d on every box) |
status |
every box's state, engine and free memory, and the head's front end and /health |
health |
this box alone, in one line: exit 0 when it serves (on the head, also its API's /health), else 1. The compose healthcheck, and a Kubernetes exec readiness probe on any rank |
logs [rank|shim|front|supervisor] [-f] |
a log from any box |
prepare [OPTIONS] |
fill --model-cache with this box's part, without booting |
check, show-config |
what would stop this box's boot; every option and where it came from |
smoke, bench |
the smoke test, and prefill and decode benchmarks, against the API on this box (run them on the head) |
build |
the engine and the Rust front end, into bin/ |
Options come from the command line, then a config file (@FILE, else ./cluster.env; see
config/cluster.env.example), then the environment. With the options in cluster.env,
./gr8 glm53 up alone is enough. ./gr8 knobs lists every option, and KNOBS.md says how they combine.
When a rank's engine exits, the head stops every box: status names the rank and shows its last log line, and the
supervisor log names the line that says why. Logs are in out/ on each box (supervisor.log, g6m4.r<r>.err, and
shim.log and front.log on the head). Each boot moves the previous engine logs to out/prev1, and keeps the three
boots before it (prev1 is the newest), so a restart does not overwrite the boot you want to look at.
- "handshake refused": every box prints the field that differs and each box's value. It is an option, the model (a different commit or variant), the engine build, a box whose checks failed, or a box that does not answer on a rail.
- The boot stops with "GiB left after the weights and part of a KV pool": lower
--cap, or use a smaller--variant. - A box waits for the others: with mentat, check that every box's daemon sees the rest (
mentatd status); with--peers, the workers wait for the head. - Each rank checks its weights at boot: the GPU sums what it holds and compares with the bytes the loader read. A
difference stops the boot with exit 3 and names the rank and the chunk ("g6: weight check: ..."). Once serving,
curl localhost:8100/health?deepon the head checks every rank again (frontend/README.md).EXTRA_ENV=GR8_WEIGHT_CHECK=0turns both off. - GLM-5.3 troubleshooting is in docs/GLM-5.3.md. Open bugs for every model are in KNOWN-ISSUES.md.
The head serves the OpenAI API on port 8100. frontend/README.md has the full API.
| port | what | where |
|---|---|---|
| 8100 | the OpenAI-compatible API (/v1/chat/completions, /v1/completions, /v1/models, /health) |
head |
| 7340 | the supervisors: the handshake before boot, then status, logs and down | every box |
| 7341 | the multi-session control channel between the ranks | head |
| 18600-18659 | fabric bootstrap and the per-collective regions | head |
| path | what |
|---|---|
gr8 |
the entry point: ./gr8 <model> <command> |
tools/launch/ |
the per-box supervisor behind up: options, the model and its cache, the handshake, mentat |
engine/runtime/ |
g6m4.cu (one rank), m4sim.cu (one GPU simulating every rank), the step, prefill and the multi-session scheduler (s2/) |
engine/kernels/ |
experts and dense trellis matrices, GEMV/GEMM, router, norms, sampling |
engine/attn/ |
MLA decode and prefill attention and the sparse indexer |
engine/comms/ |
the RoCE v2 verbs fabric: collectives, bulk proxy, context-parallel layout |
engine/loader-rs/ |
the Rust loader: plans each rank's image from the release, fetches from Hugging Face, converts the drafter |
engine/kvman-rs/ |
the Rust S2 page allocator: every rank's KV pages, live and retained runs, prefix-reuse matches |
frontend/ |
the HTTP front end (g6front-rs) and the shim that drives the ranks |
scripts/ |
build, tests and engine_args.sh (the engine command line) |
tools/convert/ |
the readers, writers and test fixtures behind the exlr8 release |
smoketest/, bench/ |
the smoke test, benchmarks and the quality gate |
docs/ |
one page per model, deploy and boot guides, engine settings, the prefill attention design, the model-specific parts |
| KNOBS.md, KNOWN-ISSUES.md, PROTOCOL.md, AGENTS.md | how the options combine (./gr8 knobs lists them); open bugs; the front end to engine protocol; how to build, test and land a change |
gr8 is not a general engine: it is written for the shapes of the three models above. docs/MODEL-SPECIFIC.md lists every place GLM 5.3 is built in, and what would carry over to another model.
KindlingAI is:
- @adapt-ai-systems
- Prefill optimization, TP=6 S2 serving and context-budget improvements (#7, #9, #27, #29).
- Earlier engine, TP=3 conversion and gate foundations preserved in the Git history (range fetching and manifests, gate harness).
- DeepSeek-V4.1-Flash on four boxes: the
ds41tuning knobs and served defaults (#327-#333, #335-#338), and this boot guide.
- @coffee-the-dev
- @mmastrac
The following contributions are greatly appreciated. If your name is missing, please file a PR: we want to ensure your work is recognized!
Thanks to:
- @turboderp
- EXL3 ideas, research and trellis quantization through exllamav3.
- @OsakaTX
- @jakejharris
- @knapcio
- Performance experiments and proposals for speculative decoding and INT4 draft weights (#83, #85, #88, #89).
- Fabric MTU negotiation, distributed GEMVs, MoE overlap, attention kernels and decode profiling proposals (#82, #84, #86, #87, #90).
- Shared-scope steering in S2 batched decode (the MoE shared-expert finalize of shared steps, #209).
- @eqvse and @cl83
- TP=3/TP=4 ring fabric and relay support (#4).
- @chuck-ads
- TP=4/TP=6 KL reference baselines and prompt-set benchmarking (TP=4 baseline, TP=6 baseline, prompt sets).
- @spectra-the-bot
- @unsaltedbutter-ai
- API compatibility and structured-output proposals (#71).
- @msuiche
- Native GLP (GGUF) steering input: weightless GLP vectors load as steering profiles through the schema-v1 validation path (GLP spec).
- @alexellis
- RigMark, used for decode and concurrency benchmarking.
- @0xr3ngar
- Usage cache reporting for the glm dialect: the launcher now passes
--enable-prompt-tokens-detailsto the front end, sousage.prompt_tokens_details.cached_tokensreaches clients on glm clusters.
- Usage cache reporting for the glm dialect: the launcher now passes
- exllamav3 by @turboderp and its contributors: the EXL3 format and its trellis quantizer, a streamlined variant of Cornell RelaxML's QTIP. This engine would not exist without their quantization work. The trellis decoders here were written from the EXL3 format, and the exlr8 weights were encoded with exllamav3's quantizer.
- davidsyoung/GLM-5.3-EXL3-TR3-3.25bpw, whose
per-expert bit distribution
homekeeps expert for expert. - RigMark by @alexellis, for decode and concurrency benchmarking.
- weightless by @msuiche: the GLP vector format (spec) and its reader-conformance contract, which the native steering input implements.
- The tool-call cases in
frontend/g6front-rs/src/parser.rsare ported from vLLM's GLM tool-parser tests (Apache-2.0).
Credits are also maintained in CREDITS.md.
MIT, see LICENSE. The weights and steering profiles are not part of this repository and carry their own licenses (GLM 5.3's license, each release's model card and notice, and the DFlash2 drafter's CC BY-NC-ND 4.0).