
Built and tested on one home machine. If QEngine helps you, a coffee keeps it going.
QEngine is an open-source (MIT) inference engine for five models on NVIDIA RTX 3000, 4000 and 5000 series cards, with nothing extra to install or set for any of them (tuned on the RTX 3090):
- Qwen3.8-Flash-Next, run from my checkpoint bpx03/Qwen3.8-Flash-Next-QEngine (routed experts INT4 from Intel's AutoRound quantization, repacked bit-exact; the parts every token passes through in INT8 or calibrated INT4): a sparse mixture-of-experts model with 512 routed experts, 262K context (up to 1M opt-in) and image input. Runs on 2 or more cards (tuned for 3 and 2).
- Qwen3.8-27B (INT4, the RedHatAI W4A16 checkpoint RedHatAI/Qwen3.8-27B-INT4 plus its MTP head; INT8 opt-in: bpx03/Qwen3.8-27B-INT8-QEngine): a dense model with 262K context (up to 1M opt-in, with optional sparse attention for long prompts) and image input. Tuned for 3 cards, also runs on 2, and on 1 as an experiment.
- DeepSeek-V4.1-Flash (preview), run from the third-party Maya-S GGUF files as they are (peasantsmith/DeepSeek-V4.1-Flash-1017-Maya-GGUF): a large sparse mixture-of-experts model (384 routed experts per layer, Engram n-gram memory, compressed sparse attention) on three 24 GB cards plus about 170 GB of host RAM, with a DSpark draft model, 262K context. RTX 3000 / 4000 series only.
- GLM-5.3-Flash (preview), run from my INT4 quant bpx03/GLM-5.3-Flash-INT4-QEngine (routed experts and KDA layers): a sparse mixture-of-experts model (288 routed experts per layer, 8 active) on three 24 GB cards plus about 192 GB of host RAM, with the experts in host RAM and a CPU expert lane next to the cards, MTP drafts, 262K context.
- gpt-oss-120B (preview), run from OpenAI's official checkpoint as it is, openai/gpt-oss-120b (MXFP4 experts): a sparse mixture-of-experts model (128 experts per layer, 4 active) on three 24 GB cards, all in VRAM, 128K context, up to 8 conversations decoded together. RTX 3090-class cards (compute capability 8.6) only.
One server serves any of them through an OpenAI-compatible and an Anthropic-compatible API (it works with agent clients such as Claude Code). The C++/CUDA engines are built as shared libraries; the Python server loads them and handles the API, tool calls, structured output, scheduling and caching.
Measured on my machine (3× RTX 3090, 24 GB, PCIe 4.0 x16, GPU P2P on), QEngine 1.5.0 with its default settings and the club-3090 benchmark tools (November 19, 2026):
| output tok/s unless noted | Qwen3.8-27B | Qwen3.8-Flash-Next | DeepSeek-V4.1-Flash | GLM-5.3-Flash | gpt-oss-120B |
|---|---|---|---|---|---|
| single-stream decode, narrative / code | 275 / 524 | 310 / 452 | 88 / 120 | 110 / 92 | 301 / 301 |
| prefill, 10K-token prompt: tok/s, time to first token | 6,188, 1.60 s | 8,254, 1.23 s | 4,217, 2.35 s | 2,920, 3.42 s | 13,280, 0.74 s |
| prefill, 90K-token prompt: tok/s, time to first token | 4,994, 18.0 s | 8,390, 10.5 s | 4,895, 17.8 s | 3,572, 25.4 s | 5,055, 17.6 s |
| total output, 1 / 4 / 8 / 16 streams at once | 312 / 575 / 848 / 999 | 300 / 642 / 880 / 1,061 | 70 / 73 / 78 / 78 | 72 / 55 / 50 / 44 | 306 / 594 / 731 / 835 |
| model | weights | quality against a reference |
|---|---|---|
| Qwen3.8-27B | INT4 W4A16, group 128 (RedHatAI) plus its MTP head | next-token KL 0.0154 against BF16, top-1 agreement 95.3 % |
| Qwen3.8-Flash-Next | routed experts INT4 (Intel's AutoRound, repacked bit-exact); shared expert and dense parts INT8 or calibrated INT4; DeltaNet projections INT8 | next-token KL 0.114 against BF16, top-1 agreement 89.6 % |
| DeepSeek-V4.1-Flash | the Maya-S GGUF files as they are: experts IQ2_XXS / IQ2_S / IQ3_XXS / IQ3_S, dense Q8_0, Engram Q6_K | negative log-likelihood 1.635 on my test texts (no full-precision reference fits on this machine) |
| GLM-5.3-Flash | my INT4 quant: routed experts and KDA layers INT4 (details) | next-token KL 0.59 against the official FP8 model (three prompts: 0.45-0.77) |
| gpt-oss-120B | OpenAI's checkpoint as it is: MXFP4 experts, bf16 dense weights | next-token KL 0.118 against an fp32 reference of the model, top-1 agreement 86.9 % |
(Single-stream decode: club-3090 bench.sh with each model's own sampling; streams: its concurrency-probe.sh,
greedy, 256-token prompts and replies, total output of all streams; each number is the mean of 2 to 6 runs. The two
Qwen servers ran with thinking on at effort low and vision on. For 16 streams the servers that hold 8 conversations
by default were started with 16: GLM_CONVS=16, DeepSeek-V4.1-Flash --max-active 16, GPTOSS_SLOTS=16.
DeepSeek-V4.1-Flash and GLM-5.3-Flash decode one conversation at a time and give the streams turns, so their total
output stays at about one stream's. gpt-oss-120B at 90K: prompts of 86,786 and 89,998 tokens. Quality: the same
builds; the 8-pack results are below.)
Flash-Next quality, in more detail: next-token KL divergence against the official BF16 model on the public set
jyohukuchan/qwen38-flash-next-bf16-logits,
scored with its own compare.py: mean KL 0.114, top-1 agreement 89.6 %, WikiText-2 KL 0.125 (0.126 when
the context went through batched prefill). 1.5.0 build, November 19, 2026, the same as 1.3.6 to every digit; details
and the command under Quality.
What is new in 1.5.0:
-
Three new models, as previews, in the same server, API and Docker image as the two Qwen models (what each one supports: Feature matrix):
- DeepSeek-V4.1-Flash, run from the Maya-S GGUF files as they are, on three 24 GB cards plus about 170 GB of host RAM: 262K context, DSpark drafts, thinking, tool calls and structured output, a host-RAM archive and shared system prompts for agent teams. 88 / 120 tok/s single-stream (narrative / code), a 90K-token prompt in 17.8 s, 8-pack 122 / 133 of 150 (thinking off / on).
- GLM-5.3-Flash, run from my INT4 quant, on three 24 GB cards plus about 192 GB of host RAM, with the experts in host RAM and a CPU expert lane next to the cards: 262K context, MTP drafts, logprobs, up to 8 conversations at once by exact time slicing. 110 / 92 tok/s single-stream, a 90K-token prompt in 25.4 s, 8-pack 121 / 133 of 150.
- gpt-oss-120B, run from OpenAI's checkpoint as it is, all in VRAM on three 24 GB cards, with or without GPU P2P: 128K context, up to 8 conversations decoded together, lookup drafts, presence / frequency penalties. 301 tok/s single-stream, 835 tok/s total with 16 streams, a 10K-token prompt in 0.74 s and a 90K-token prompt in 17.6 s, 8-pack 110 / 116 of 150 (reasoning effort low / medium).
-
27B: the first token comes much sooner with many agents at once. The 27B used to keep half of the GPU time for running replies however many requests waited for their prompt. Now the time split follows the queue: short prompts are prefilled back to back with one decode step between passes, long ones share the time by head count, and running replies keep at least 10 % of it (
--prefill-share-min,--prefill-short;--prefill-share-min 0.5gives the old split). It counts a running reply at admission by itsmax_tokens, and no longer copies a whole conversation to the host archive when the next turn only replaces the last reply. On my 2 cards, in an agent replay with short tool results, the median time to the first token at 16 agents in flight went from 4.5 s to 0.86 s, with 7 % more total output; single-stream speed is unchanged (numbers). Beyond its 16 decode seats a request is prefilled on one of two seats kept for prompts, gets its first token, and its reply waits parked in host memory (KV and state, resumed exactly where it stopped) until a decode seat frees (--park,--park-reserve;--park 0waits for a seat first, as before). Archived conversations keep two recurrent-state checkpoints 10BC8 instead of four (QE_ARCH_SUBS), so the host archive holds about 1.5 times as many agents. At 32 and 64 agents in flight with the defaults on 2 cards (release build): median first token 1.0 / 0.48 / 1.0 s at 16 / 32 / 64 agents in flight, with 243 / 359 / 285 tok/s total output and no agent's prompt computed again.QE_MAX_SEQS=32(opt-in) seats 32 conversations on 2 and 3 cards: at 32 agents in flight on 2 cards the median first token went from 25 s to 0.48 s with 106 % more total output, at the cost of a smaller KV pool (2 cards: 417,792 tokens instead of 623,616, so two very long conversations no longer fit at once). -
Flash-Next: the first token comes much sooner with several agents at once. Every prompt now keeps the state at its own end, so an agent's next turn resumes there even when the client sends back another text for the reply or drops its reasoning; before, a turn shorter than 4,096 tokens was recomputed on the next turn (about a third of all prefill work in an agent replay). A conversation keeps three recurrent states instead of six (its system end, its latest prompt end and the newest other one;
FLASH_CKPT_KEEP=6as before), so the host archive holds 64 agents. On 3 cards the time split follows the queue as on the 27B: while short prompts are next, running replies get 10 % of the time after every prefill pass instead of half (--prefill-share-min; 2 cards and mode B keep the fixed split). In an agent replay with short tool results the median time to the first token went from 0.16 / 0.44 / 1.91 / 3.93 s to 0.13 / 0.25 / 0.37 / 2.83 s at 1 / 4 / 16 / 32 agents in flight, with 15 / 14 / 16 / 10 % more total output (numbers). Beyond the 16 decode seats Flash-Next now parks as the 27B does (--park, on 3 cards intpmode): a reply gets its first token from one of 4 seats kept for prompts, waits in host memory and resumes exactly where it stopped when a seat frees. At 32 / 64 agents the median time to the first token went from 2.83 / 8.0 s to 0.26 / 0.75 s, with 5 / 9 % more total output. -
Flash-Next: no more stops at a slot hand-off when the host is short of memory (#32, #25). The engine's log lines go through a writer thread, so a slow log reader can no longer hold up a GPU step (
QE_LOG_ASYNC); the archive's memory is pinned in the background only while the engine is idle, 1 GB at a time, and pinning pauses (and backs off) when a registration is slow, which is a sign of host memory pressure. Thanks to cristianflorett, dgaglioni and gbinka for the reports and measurements behind the Flash-Next hand-off fix (#32, #25). -
Both Qwen models give memory back when other programs squeeze the host while they serve. A background thread watches the kernel's memory-pressure and swap-in counters next to
MemAvailable; under pressure the engine stops growing, frees archive memory, shrinks the archive, drops pinned prompts and, last (Flash-Next), moves the n-gram table to the page cache, one logged stage at a time, and it returns slowly once the host stays calm (QE_MEMP=0turns it off; details). With 179 GB taken by another program on my machine, Flash-Next went down to the last stage and served every request, with the same greedy text. -
For every model: streamed replies send a keep-alive after 10 s without an event (an OpenAI
: keep-alivecomment, an Anthropicpingevent;QE_SSE_KEEPALIVE), so proxies and clients with a read timeout keep a stream whose request waits for its turn. At start the server logs one[settings]line for every flag or environment variable that is set but does nothing for the loaded model. A top-levelenable_thinkingin a request is read too (#31). -
27B and Flash-Next single-stream decode and prefill speed and their quality are the same as in 1.3.6 within the run-to-run spread (same benchmark, 1.3.6: 27B 281 / 507 tok/s, prefill 6,133 / 4,997 tok/s; Flash-Next 306 / 449 tok/s, prefill 8,207 / 8,394 tok/s).
1.5.0 went through the release checks on my machine and passed all of them (the 262K-token stress runs with images on 3 and on 2 cards, VRAM and host-RAM hogs taken mid-run, soaks, Docker smoke, per model). The 8-pack quality suite ran once per model (Quality), next to the KL / NLL measurement and the speed runs of all five models.
Older releases and what changed in each: releases page.
- Quick start
- Hardware and interconnect
- Models
- Feature matrix
- API
- Serving behaviour
- Settings
- Benchmarks
- Troubleshooting and FAQ
- Repository layout and tests
- Changelog
- Support
- License, Disclaimer
More settings, measurements and the troubleshooting table: docs/advanced.md.
You need Linux, two or three RTX 3000, 4000 or 5000 series cards (see Hardware), an NVIDIA driver with CUDA 13.1 support, and enough host RAM (Flash-Next: about 140 GB for the full defaults; with less, the start-up sizes its host memory to the machine, from 64 GB up; see Host memory).
The hf command comes with the Hugging Face hub client: pip install -U "huggingface_hub[cli]".
# Qwen3.8-Flash-Next: my checkpoint is the supported way to run it (~129 GB, ready to run: converted weights,
# INT8 n-gram table and three small files)
hf download bpx03/Qwen3.8-Flash-Next-QEngine --local-dir models/Qwen3.8-Flash-Next-QE-INT4
# (the previous, 1.3.4 weights: add --revision v1.3.4; see "Model files" under Models)
# Qwen3.8-27B (incl. its MTP head); two small tuned files are fetched automatically at the first start
hf download RedHatAI/Qwen3.8-27B-INT4 --local-dir models/Qwen3.8-27B-INT4Requirements on the host: the NVIDIA driver and the
NVIDIA Container Toolkit. One image serves all
models; it picks the engine from the model's config.json (DeepSeek-V4.1-Flash: from the GGUF files).
docker run -d --name qengine --restart unless-stopped --gpus all --ipc=host --ulimit memlock=-1 \
-p 127.0.0.1:8080:8080 -p 127.0.0.1:9393:9393 \
-v /path/to/models:/app/models:ro -v qengine-cache:/root/.cache/qengine \
-e QE_MODEL=/app/models/Qwen3.8-Flash-Next-QE-INT4 \
ghcr.io/bpx03/qengine:latest- Without
QE_MODELthe container serves/app/models/Qwen3.8-27B-INT4. - The API listens on 127.0.0.1 only, as the source start does. To reach it from other machines, publish it as
-p 8080:8080(all interfaces) and put a firewall or an authenticating proxy in front: the server has no API keys. --restart unless-stoppedmatters: after a stalled GPU the server exits with code 75 and relies on a restart (see Failure handling).- Images:
latest/1.5.0(text only) andvision/1.5.0-vision(with the image encoder). Earlier:1.3.6,1.3.6-vision,1.3.5.9,1.3.5.9-vision,1.3.5.8,1.3.5.8-vision,1.3.5.7,1.3.5.7-vision,1.3.5.6,1.3.5.6-vision,1.3.5and1.3.5-vision; experimental1.3.5.5,1.3.5.5-vision,1.3.5.4,1.3.5.4-vision,1.3.5.3,1.3.5.3-vision,1.3.5.2,1.3.5.2-vision,1.3.5.1,1.3.5.1-vision. Earlier releases stay available under their version tags. QE_GPUS=0,2picks the GPUs (host ids with--gpus all; default0,1,2;QE_DEVICES, if set, wins overQE_GPUS). Settings are environment variables (-e FLASH_PAR=pp); anything after the image name, orQE_EXTRA_ARGS, goes to the server (--think 0,--vision 1).- Keep the
qengine-cachevolume: it holds the GEMM tuning table, the downloaded 27B files and the stats. --ipc=hostand--ulimit memlock=-1are needed for the pinned host memory. On SELinux hosts (Fedora, RHEL) add--security-opt label=disable.- Port 8080: the API. Port 9393: monitoring dashboard,
/api/stats, Prometheus/metrics. - Compose:
QE_MODELS_DIR=/path/to/models docker compose up -d(seedocker-compose.yml). It builds the image from this checkout (it does not pullghcr.io/bpx03/qengine) and serves the 27B; for Flash-Next addQE_MODEL: /app/models/Qwen3.8-Flash-Next-QE-INT4underenvironment:. Unlike thedocker runline above, it publishes the API on all interfaces (QE_HOST_PORT, default 8080) and the monitoring on 127.0.0.1 (QE_METRICS_HOST_PORT, 9393). The server has no API keys, so anyone who can reach that port can use it: setQE_HOST_PORT=127.0.0.1:8080to keep it on this machine, or put it behind a firewall or an authenticating proxy.QE_VISION_BUILD=0builds the text-only image,QE_REVISIONsets the build hash. - DeepSeek-V4.1-Flash (preview): mount the folder that holds both
Maya-S/anddspark/and pointQE_MODELatMaya-S, e.g.-v /path/to/DeepSeek-V4.1-Flash-1017-GGUF:/app/models/dsv4:ro -e QE_MODEL=/app/models/dsv4/Maya-S. Symbolic links inside the mount must resolve inside the container (mount their targets at the same paths). The image carries the DSV4 engine for sm_86 / sm_89 only. Engine settings go in with-e(-e DSV4_EGMIRROR=...), server flags inQE_EXTRA_ARGS(--route-margin 0.05). One image serves all models; the DSV4, GLM and gpt-oss engines add cuBLAS to it. - GLM-5.3-Flash (preview): mount the
bpx03/GLM-5.3-Flash-INT4-QEnginedirectory, e.g.-v /path/to/GLM-5.3-Flash-INT4-QEngine:/app/models/glm:ro -e QE_MODEL=/app/models/glm. The experts are pinned in host RAM (about 157 GB), so--ipc=host --ulimit memlock=-1matter here. Engine settings go in with-e(-e GLM_CPU_ISA=avx2), server flags inQE_EXTRA_ARGS(--split 704,704,640). - gpt-oss-120B (preview): mount the
openai/gpt-oss-120bdirectory, e.g.-v /path/to/gpt-oss-120b:/app/models/gpt-oss-120b:ro -e QE_MODEL=/app/models/gpt-oss-120b. Engine settings go in with-e(-e GPTOSS_SLOTS=4). The image carries the gpt-oss engine for sm_86 only. - Build it yourself:
docker build -t qengine .(with vision),--build-arg VISION=0(text only),--build-arg CUDA_ARCHS=86(RTX 3090 code only, faster build),--build-arg REVISION=$(git rev-parse HEAD)(the build hash the monitoring headers show; without it they saybuild unknown).
Requirements: CUDA Toolkit 13.1, CMake ≥ 3.24 and Ninja (or Make), a host compiler nvcc 13.1 accepts (it
rejects GCC 16; CMakeLists.txt uses /usr/bin/g++-15 when present, otherwise pass e.g.
-DCMAKE_CXX_COMPILER=g++-14 -DCMAKE_CUDA_HOST_COMPILER=g++-14), Python 3.12.
cmake -B build -G Ninja
cmake --build build # -> build/libqengine.so (27B), build/libqflash.so + build/flash_run (Flash-Next)
python3 -m pip install -r requirements.txt # torch: install the CUDA build matching your driver (see the file)A source build contains code for the GPUs of the machine it is configured on (their compute capabilities from
nvidia-smi; the cmake output names them), always plus RTX 3090 code (sm_86). To build for other cards, name
them: cmake -B build -G Ninja -DCMAKE_CUDA_ARCHITECTURES="86;120" (or "86;89;120", or just 120).
server/run.sh 8080 --model models/Qwen3.8-Flash-Next-QE-INT4 # Flash-Next on GPUs 0,1,2, API on 127.0.0.1:8080
server/run.sh 8080 # Qwen3.8-27B (the default model directory)
python3 server/app.py --help # all command-line options| setup | command (source; Docker: the same variables with -e, GPUs with QE_GPUS) |
notes |
|---|---|---|
| 3 cards (tuned) | server/run.sh 8080 --model <dir> |
the default: GPUs 0,1,2 |
| 2 cards | QE_DEVICES=0,2 server/run.sh 8080 --model <dir> |
Flash-Next keeps about half of its experts in host RAM; the 27B serves 16 slots, as on 3 cards (QE_MAX_SEQS=32 for 32) |
| 1 card (27B only, experimental) | QE_DEVICES=0 QE_MAX_SEQS=1 server/run.sh 8080 |
one conversation, 206,848 tokens of context; QE_MAX_SEQS up to 4; MTP drafts unless you lower --max-ctx to what the start-up line says (--max-ctx 165K on my 24 GB card) or set QE_DRAFTER=dflash2 (DFlash2) |
| a card also drives a desktop | Flash-Next FLASH_EC_MARGIN_MB=256,2048,256; 27B QE_RESERVE_GB=1.5,3,1.5 |
more free VRAM on that card (values in GPU order) |
| 4 or more cards (Flash-Next) | QE_DEVICES=0,1,2,3 server/run.sh 8080 --model <dir> |
exactly 4 equal cards with verified P2P: tp, otherwise a pipeline (pp); see 4 or more cards |
| long documents, short answers (Flash-Next) | FLASH_PAR=pp |
pipeline mode: faster long prompts, slower single-stream decode |
| many long conversations (Flash-Next) | FLASH_KV=ram FLASH_KV_RAM_TOKENS=4M |
KV cache in host RAM, ~74K tokens per GB |
| more KV per GB (Flash-Next) | FLASH_KV_BITS=4 |
4-bit KV cache: 7,488 instead of 13,520 bytes per token (~134K tokens per GB with FLASH_KV=ram); see Host memory |
| small cards, more context (27B) | QE_KV_BITS=4 |
4-bit KV cache: about 1.5x the context on 8-12 GB cards; see Smaller cards |
The first start on a new card count tunes the GEMMs and caches the table in ~/.cache/qengine.
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen3.8-flash-next-qengine",
"messages": [{"role": "user", "content": "Write a haiku about PCIe."}],
"max_tokens": 1024
}'The served names are qwen3.8-flash-next-qengine and qwen3.8-27b-qengine (--served-name changes it).
For Claude Code, set ANTHROPIC_BASE_URL=http://127.0.0.1:8080 and CLAUDE_CODE_ATTRIBUTION_HEADER=0.
I run QEngine behind llama-swap myself. A source build:
healthCheckTimeout: 900 # the first start of a model takes longer (see below)
models:
qwen3.8-flash-next-qengine:
cmd: bash -c 'cd /path/to/QEngine && exec server/run.sh ${PORT} --model /path/to/models/Qwen3.8-Flash-Next-QE-INT4'
checkEndpoint: /health
qwen3.8-27b-qengine:
cmd: bash -c 'cd /path/to/QEngine && exec server/run.sh ${PORT} --model /path/to/models/Qwen3.8-27B-INT4'
checkEndpoint: /healthThe Docker form (the switcher stops the container by name):
qwen3.8-flash-next-qengine:
cmd: >
docker run --rm --name qengine-fn --gpus all --ipc=host --ulimit memlock=-1
-p 127.0.0.1:${PORT}:8080 -v /path/to/models:/app/models:ro -v qengine-cache:/root/.cache/qengine
-e QE_MODEL=/app/models/Qwen3.8-Flash-Next-QE-INT4 ghcr.io/bpx03/qengine:latest
cmdStop: docker stop -t 60 qengine-fn
checkEndpoint: /healthexecmatters: it makes the server the process the switcher stops, so a stop ends the engine and frees the cards.GET /healthanswers only once the model is loaded and the API is up, so it is the readiness check.- A start takes longer the first time on a machine: the GEMMs are tuned for the card count, the 27B downloads its
tuned MTP head and draft ranking and the DFlash2 drafter (about 1.28 GB), and with
--lsathe index map. All of it is cached in~/.cache/qengine(Docker: keep theqengine-cachevolume), so later starts skip it. - Only one engine can hold a card. A second one that starts while the first is still exiting waits up to
QE_GPU_LOCK_WAIT_S(60 s) and then stops with the first one's pid and command line. So the switcher must stop one QEngine model before it starts another on the same GPUs (llama-swap does that by default: one model at a time). Two models side by side need separate cards (QE_DEVICES, DockerQE_GPUS).
QEngine is tuned on 3× RTX 3090 with GPU P2P and plenty of host RAM, but it adapts by itself to machines that differ from that:
- no P2P (stock driver, x4 / x8 slots, dual-socket or NUMA): both engines exchange through pinned host memory by DMA, close to the P2P speed (details);
- long prompts on slow links:
FLASH_PAR=pp, a layer pipeline, picked automatically where it wins (Parallel mode); - small host RAM (e.g. 64-96 GB): the host-memory plan adapts, with no swapping at the defaults (Host memory);
- mixed or uneven cards: a per-card calibration and a layer split by speed;
- 4 or more cards: see 4 or more cards;
FLASH_PAR=pptpis experimental; - small cards (8-16 GB): see Smaller cards;
- RTX 3000, 4000 and 5000 series: built in, nothing to set (what has run where).
If a setup like this is still slow or misbehaves, please open an issue with the start-up log, started with
FLASH_BOX_REPORT=1 (it adds [box] lines on each card and link; on 4 or more cards they are on by default): those,
the [qflash] parallel mode line and the host RAM plan line are what I need to look into it.
QEngine runs on the RTX 3000, 4000 and 5000 series with no extra steps. The Docker images carry the code for all three,
and a source build targets the GPUs it finds (-DCMAKE_CUDA_ARCHITECTURES sets others).
| series | code | what has run on it |
|---|---|---|
| RTX 3000 | sm_86 | 3090: all tuning (82 SMs, tile shapes, split counts, memory budgets) targets it, and everything published here was measured on it. Community: 4× modded 3080 20 GB (#9) |
| RTX 4000 | sm_89 | built, but no run on record yet; reports are welcome |
| RTX 5000 | sm_120 | community: 2× 5090 (club-3090 #1826), 4× 5080 16 GB (#13) |
The 4000 and 5000 series use the settings tuned for the 3090, so they may have speed left unused. What limits a card is its memory; see below.
How many cards each model runs on:
| model | cards | how |
|---|---|---|
| Qwen3.8-Flash-Next | 3 (tuned), 2 | tensor parallel or pipeline, picked at start (FLASH_PAR=auto) |
| Qwen3.8-Flash-Next | 4 | tensor parallel when peer access between all 4 equal cards passes the start-up check, else pipeline (FLASH_PAR=auto); community runs only, see 4 or more cards |
| Qwen3.8-Flash-Next | 5 or more | pipeline (FLASH_PAR=pp); tested only as 4 pipeline stages on my 3 cards |
| Qwen3.8-Flash-Next | 1 | not supported |
| Qwen3.8-27B | 3 (tuned), 2 | tensor parallel |
| Qwen3.8-27B | 1 | experimental, QE_MAX_SEQS 1 to 4 |
Smaller cards (Flash-Next; measured on my 3090s with all but 12 GB or 8 GB of each card taken by another process before the start, November 3, 2026; defaults otherwise, 16 conversation slots; c=1 / c=4: greedy decode of 400-token replies, thinking off; prefill of an 11.5K-token prompt; a 53K-token needle; the 27B follows below):
| model | cards x free VRAM | context | c=1 / c=4 tok/s | prefill tok/s | needle |
|---|---|---|---|---|---|
| Flash-Next | 3 x 12 GB | 262,144 | 66.5 / 145.1 | 4,612 | pass |
| Flash-Next | 3 x 8 GB | 262,144 | 47.2 / 91.1 | 3,819 | pass |
| Flash-Next | 2 x 12 GB | 262,144 | 40.7 / 78.7 | 3,546 | pass |
| Flash-Next | 2 x 8 GB | 65,536 (fitted at start) | 34.7 / 55.1 | 2,263 | pass |
- Flash-Next keeps its dense weights, slot state and KV cache on the cards and puts the experts that do not fit in
host RAM (on 8 GB cards nearly all of them, so decode is host-link bound). The minimum per card is about 7.5 GB on
3 cards and about 8 GB on 2 cards at a 65K context (about 10 GB at 262K); when the requested context does not fit,
it starts with the largest that does and says so (
--max-ctxsets it,FLASH_MAX_SEQSfewer slots leave room for more). Below the minimum it stops with a message naming what to change. - 27B needs its weights on the cards: about 5 GB per card on 3 cards and 7.2 GB on 2, plus buffers, the conversation
slots' state (110 MB per slot and card on 3 cards, about 200 MB on 2) and the KV cache. On a small card it starts
with fewer slots and a smaller VRAM reserve (logged;
QE_MAX_SEQS,QE_RESERVE_GBset them) so the KV cache holds at least a 32K context: 8 GB cards work on 3 cards, 12 GB cards on 2. Below that it stops with a message (exit code 3). - 27B on one card needs about 15.8 GB for the weights alone, so a single 16 GB card does not start; 2 x 16 GB does.
27B on smaller cards, and with a 4-bit KV cache (opt-in, QE_KV_BITS=4) for more context. Each key and value
row is rotated (a fixed random Hadamard transform, which spreads the outlier channels) and stored in 4 bits with one
fp16 scale per 32 values: 144 bytes per row instead of 256 with FP8, so the pool holds 1.6-1.7x the tokens (the MTP
layer stays FP8). Measured like the table above (November 14, 2026, same day for both caches, before the INT4
verify head of 1.3.6, which leaves each card more room for the KV pool; the 3 x 8 GB FP8 row is from November 3;
needle at about 60 % of the served context; slots as fitted for FP8):
| cards x free VRAM | KV cache | context | c=1 / c=4 tok/s | prefill tok/s | needle |
|---|---|---|---|---|---|
| 3 x 12 GB | FP8 | 255,649 | 220.2 / 630.1 | 6,372 | pass |
| 3 x 12 GB | 4-bit | 262,144 | 189.8 / 653.4: the context now fits DFlash2, which is slower than MTP on this story prompt (QE_DRAFTER=mtp: 212.9 / 664.6) |
6,231 | pass |
| 3 x 8 GB | FP8 | 44,675 (9 slots) | 220.6 / 619.3 | 6,288 | pass |
| 3 x 8 GB | 4-bit | 67,584 (9 slots) | 215.2 / 630.2 | 6,364 | pass |
| 2 x 12 GB | FP8 | 30,720 (14 slots) | 171.5 / 338.5 | 4,568 | pass |
| 2 x 12 GB | 4-bit | 46,080 (14 slots) | 190.7 / 513.3 | 4,537 | pass |
1 x 24 GB, QE_MAX_SEQS=1 |
4-bit | 240,640 with DFlash2 (262,144 with QE_DRAFTER=mtp; FP8 on 1.3.5.8: about 191K) |
102.9 / - | 2,550 | pass |
| 2 x 8 GB | either | does not start (the weights do not fit) | - | - | - |
- Quality: next-token KL against a bf16 KV cache over 100K / 32K / 64K-token documents is 4.4e-3 with 4 bits and 1.5e-3 with FP8 (for scale: the INT4 weights against BF16 are 1.45e-2); top-1 agreement stays at FP8's level, a passphrase needle at 130K (3 depths) and 239K passes, MTP / DFlash2 acceptance within 2 %. It is opt-in because it is not lossless against FP8; on cards with room the FP8 default is the better choice.
- Speed: decode within 2 % of FP8 at 32K and faster at long context (128K: -9 % time per step); prefill 0.4 % slower
at 128K and 3 % at 256K (the prompt's keys are converted for the prefill kernels, about 0.5 KB per key and KV head
of the served context). On 3 x 12 GB the context now fits the DFlash2 drafter, which
QE_DRAFTER=autothen loads, as on 24 GB cards: warm c=1, it is 10 % faster than MTP alone on code and 7 % slower on prose, the same on both memory sizes (the c=1 numbers in the table are a first, cold request). - Also checked with 4 bits: conversations restored from the RAM archive continue bit-identically, the disk cache
resumes after a restart, the KV pool shrinks and grows back when another process takes and frees 16 GB of a card,
and
--max-ctx 512K(YaRN) serves 524,288 tokens on 3 cards with a 331K-token needle found. - Not with
QE_LSA(sparse attention reads FP8 keys; it is turned off with a start-up line) and not withQE_KV_FP8=0. Conversations archived or saved to disk with one cache format are not reused by the other.
GPU selection: QE_DEVICES=0,2 (physical ids; Docker QE_GPUS), or --devices (indices within
CUDA_VISIBLE_DEVICES). By default every visible GPU is used. Each engine locks the GPUs it uses: a second engine on
a card in use stops at start with the first one's pid and command line (QE_ALLOW_SHARED_GPU=1 skips the check).
GeForce cards have no GPU-to-GPU P2P with the stock NVIDIA driver. A community patch of the open kernel modules (aikitoria/open-gpu-kernel-modules, a fork of tinygrad's patch that tracks current driver versions, with Resizable BAR on) enables it. QEngine detects which one you have and needs no setting.
Since 1.3.4 both engines are fast without P2P: their tensor-parallel exchanges copy through pinned host memory by
DMA (the copy engines), and the GPU cores only wait on flags in their own memory. Measured on 3× RTX 3090 (PCIe 4.0
x16), with P2P forced off (QE_P2P=0) against P2P on (November 1, 2026):
| measured (s, lower is better) | without P2P | with P2P | 1.3.3 without P2P |
|---|---|---|---|
Flash-Next prefill, 10K / 32K / 90K-token prompt (flash_run) |
1.83 / 6.06 / 17.03 | 1.35 / 4.54 / 13.06 | 3.75 / 12.55 / 34.95 |
| Flash-Next decode step, one conversation | 9.9 ms | 9.5 ms | 12.7 ms |
| 27B prefill, 16K-token prompt | 2.68 | 2.59 | 3.94 |
The 27B's decode speed does not depend on P2P. Without P2P the 27B uses a smaller KV pool (its position-split KV
layout needs P2P). Inside Docker, P2P works exactly when it works on the host (check the P2P: / [comm] lines of the
start-up log).
Flash-Next can share the cards in several ways; the 27B always runs tensor parallel. FLASH_PAR selects the mode
(source: FLASH_PAR=pp server/run.sh ...; Docker: -e FLASH_PAR=pp).
| mode | FLASH_PAR |
what it does | recommended for | picked by auto? |
how to force / options |
|---|---|---|---|---|---|
| automatic (default) | auto |
picks tp or pp at start from a link probe, a short per-card speed calibration and a model of a typical agent request |
almost every machine | - | FLASH_PAR_AUTO_P=32768 weighs 32K-token prompts instead of 8K (favours pp) |
| tensor parallel | tp |
every layer split over the cards; all cards work on every token, exchanges in every layer (P2P, or host-staged without it) | 2-3 equal cards: fastest decode and short prompts, with or without P2P | yes: 2-3 equal cards; exactly 4 equal cards with verified peer access between all of them, if it fits | FLASH_PAR=tp; FLASH_COMM sets the exchange path |
| pipeline | pp |
each stage holds a contiguous range of whole layers; prompts flow through in micro-batches, several conversations decode as groups moving stage to stage; only a 40 KB row per token crosses between stages, so it needs no P2P | long uncached prompts with short replies (RAG, long documents); uneven cards; 4 or more cards | yes: 5+ cards; 4 cards when peer access is missing or unverified, the cards are uneven, or tp does not fit; and uneven cards (or 2-3 cards without P2P) when it wins the mix | FLASH_PAR=pp; FLASH_PP_LAYERS=17,17,14 sets layers per stage; a card may hold several stages (--devices 0,1,2,0; a test layout, it adds no compute) |
| pipeline over TP groups (experimental, untested on 4 cards) | pptp |
a pipeline whose stages are tensor-parallel groups of 2-3 cards; per-layer exchanges stay inside a group | 4+ cards where P2P exists only inside card pairs (two PCIe switches, two CPU sockets) | no, opt-in only | FLASH_PAR=pptp FLASH_PP_MAP=0,1|2,3 (, inside a stage, | between stages); without a map FLASH_PP_TP=2 pairs consecutive cards |
auto models an 8K-token uncached prompt and a 512-token reply, averaged over 1, 4 and 8 conversations at once
(capped at FLASH_MAX_SEQS); pp must beat tp by 10 % over that mix to be picked. With P2P on equal cards it is
always tp. Settings of the pick and the pipeline: docs/advanced.md.
Recipes
| your machine and workload | setting |
|---|---|
| 2-3 equal cards, with or without P2P, agents / chat | leave auto (= tp) |
| mostly long uncached prompts (RAG, long documents, short answers) | FLASH_PAR=pp, or FLASH_PAR_AUTO_P=32768 to let auto decide on that basis |
| uneven or mixed cards | leave auto: it calibrates each card and picks pp with a split by speed when that wins |
| exactly 4 cards | leave auto: tp with verified peer access between all of them, otherwise pp; mostly long uncached prompts: FLASH_PAR=pp (why) |
| 5 or more cards | leave auto (= pp); if your cards pair up on P2P, try FLASH_PAR=pptp FLASH_PP_MAP=0,1|2,3 |
steering vectors or FLASH_KV=ram needed |
FLASH_PAR=tp (2-4 cards) |
FLASH_PAR=pp server/run.sh 8080 --model models/Qwen3.8-Flash-Next-QE-INT4
docker run ... -e FLASH_PAR=pp -e QE_MODEL=/app/models/Qwen3.8-Flash-Next-QE-INT4 ghcr.io/bpx03/qengine:latest
docker run ... -e FLASH_PAR=pptp -e FLASH_PP_MAP='0,1|2,3' -e QE_GPUS=0,1,2,3 ... ghcr.io/bpx03/qengine:latestWhat was picked. The start-up log has one line with the mode and the reason, e.g.
[qflash] parallel mode: tp (auto: no peer access, but TP not beaten by the margin over the mix); probe: ..., and a
[qflash] cards: line with each card's measured speed and free memory (cards of the same model count as uneven only
when a longer re-calibration confirms the gap). /api/stats on the monitoring port shows the same under parallel.
The trade-off. pp prefills 32K-90K-token prompts up to 1.8× faster than tp (3 cards with P2P: a 90K prompt
in 9.9 s against 13.6 s; on 2 cards with P2P the two are about even), decodes 4-8 conversations at 0.83-1.03× of
tp, but one conversation at only 0.54-0.65×; 10K prompts with P2P are faster in tp. Measured on the server, same build, cold prompts end to end and decode of 1 / 4 / 8
conversations (November 1, 2026):
| cards, link | mode | 10K prompt | 32K prompt | 90K prompt | decode tok/s, 1 / 4 / 8 at once |
|---|---|---|---|---|---|
| 3, P2P | tp |
1.52 s | 5.20 s | 13.55 s | 244 / 541 / 773 |
| 3, P2P | pp |
1.72 s | 3.89 s | 9.88 s | 132 / 447 / 696 |
| 3, no P2P | tp |
2.05 s | 7.09 s | 18.03 s | 232 / 487 / 708 |
| 3, no P2P | pp |
1.77 s | 3.95 s | 10.21 s | 136 / 438 / 703 |
| 2, P2P | tp |
2.54 s | 5.67 s | 15.96 s | 91 / 200 / 246 |
| 2, P2P | pp |
3.08 s | 5.59 s | 14.75 s | 56 / 203 / 245 |
| 2, no P2P | tp |
3.78 s | 8.31 s | 23.45 s | 88 / 195 / 236 |
| 2, no P2P | pp |
2.97 s | 5.63 s | 14.85 s | 57 / 201 / 241 |
Each row is one fresh server; the decode runs come first with a cold expert cache, which costs 2-card single-stream decode most. The prompts and method differ from the headline table, so compare rows within this table only.
Uneven cards. pp splits the layers by each card's measured speed and free memory. With one 3090 slowed to a
third of its compute (a test setting), the split moved 4 layers off it: prefill about 30 % and decode 13 % faster than
an equal split.
What pp and pptp do not serve: image input in pptp (refused with a clear message; pp serves images since 1.3.5.1, experimental), steering vectors (not loaded),
FLASH_KV=ram (on 2 or 3 cards the server runs tp instead and logs it; on 4 or more cards the start is refused with
a message), activation capture; lookup drafts are off. Structured output and the thinking budget work as in tp
(a constrained step runs through the stages as a batch of one).
I have no 4-card machine: the 4-way split was
tested as 4 pipeline stages on my 3 cards (two stages on one card), where it gave the same output bit for bit as 3
stages. pptp was tested only in a 3-card layout (0,1|2), where its next-token probabilities match tp within the
run-to-run noise. A community run on four RTX 5080 cards (tp, 1.3.5.7 and 1.3.5.9, #24)
confirmed Flash-Next's long-context recall there: needles found up to 240,664 tokens at 262K, and at 1M up to 843,147
on 1.3.5.7 and 813,180 on 1.3.5.9 (one ladder rung lower), the same level as on 3 cards. More reports from 4-card machines are welcome.
Since 1.3.5.4 (experimental): on exactly 4 equal cards, auto picks tp when peer access between all of them passes the data check
at start and tp fits the free VRAM; otherwise it picks pp. With 5 or more cards it is always pp. If tp can't start
(out of device or host memory during the load, or a CUDA error or a stall in the check), the server restarts itself in
pp and says so in the log, at the cost of the partial load. On a card whose free VRAM is badly fragmented, a tp that did
load can still stop later on a request: use FLASH_PAR=pp there. This rule rests on one user's measurements on one rig
(4× RTX 5080 16 GB behind one PCIe switch, 1.3.5): one 1,000-token stream at 310.3 tok/s in tp against 145.4 in pp. On
1.3.4.3, prefill at 32K was higher in pp (7,367 against 5,466 tok/s), so for mostly long uncached prompts set
FLASH_PAR=pp. The rest was tested on 3 cards only. On 1.3.5 to 1.3.5.3, auto always picks pp on 4 cards and
prints peer access no even if your cards have P2P, so set FLASH_PAR=tp yourself.
At start both engines measure every GPU pair for a few milliseconds (P2P loads and stores, copy-engine copies, copies staged through host memory), how much each transfer slows a kernel running next to it (the co-run tax), and whether data sent over P2P really arrives intact:
[comm] topology: numa -1,-1,-1 (1 node), PCIe x16,x16,x16, P2P rank 0-1:0 0-2:0 1-2:0
[comm] P2P data check: SM loads, SM stores and copy engine ok on every pair (...)
[comm] prefill transfers: P2P SM loads / stores, ... (FLASH_COMM=auto, auto: ...: P2P kept: tax over the cap, under the 1 NUMA node cap)
[comm] decode collectives (QE_DECODE_COMM=auto, real decode steps, median of 4 interleaved rounds of 12): single sequence: P2P SM stores (P2P ..., host memory ... ms/step, host faster in 0/4); batched (4 sequences): ...; calibration ... ms
With auto (the default) Flash-Next keeps P2P on fast links, moves the prefill transfers to the copy engine on slow,
narrow (e.g. an x4 slot) or cross-socket links, and never uses a transport that failed the data check. The line ends
with the reason. FLASH_COMM (Flash-Next) and QE_COMM (27B) force a path: p2p, ce (copy engine), host.
The rule and its thresholds: docs/advanced.md.
Decode sends only small messages, but several per layer and step (all-reduces, the attention exchanges): either as
SM stores into the other cards' memory (P2P) or through the host path, with the same results. The host path is the
one the engines use without peer access (QE_P2P=0): each card writes its words once into pinned host memory and the
others read them; peer access stays on for prefill. Which one is faster depends on the board: peer stores can stall the
other kernels running on a card, and only the real model shows by how much. With auto both engines time a few real
decode steps on each path at the end of the start-up (one sequence, and four batched sequences), keep P2P or the host
path for each, and print both timings on the [comm] decode collectives line. QE_DECODE_COMM=p2p or host forces
the decode path (both engines; prefill keeps its own path). Before 1.3.5.6 the host candidate was a slower path
(about 45 % more time per step on 3 cards), so boards where P2P stores are expensive decoded slower than on 1.3.4.3. The measurements are cached in ~/.cache/qengine (the Docker volume) and reused while the cards, links, driver and
build stay the same; QE_COMM_CACHE=refresh measures again. A cached decode line says from the start-up cache, not measured this boot and carries the date of the measurement, so compare its ms/step across boots with care; a round
that was not measured (with fewer than 4 slots there is no batched round) prints n/a.
Set FLASH_COMM=ce (27B: QE_COMM=ce) yourself when the [comm] line says P2P but:
- long prompts fail with
non-finite values(HTTP 500) or the log showsGUARDlines; - a card sits behind something the link width does not show (a PCIe switch, a riser or extension cable, a chipset slot, a bifurcated slot);
- the cards differ (e.g. a 3090 next to a 3090 Ti) or sit in slots of different PCIe generations, and the line shows a P2P co-run tax of about 1.9 or more;
- prefill is clearly slower than the numbers here.
The copy engine gives the same output; on fast x16 links P2P is a few percent faster in prefill.
NUMA. On a machine with several NUMA nodes, Flash-Next places pinned host memory by GPU (each card's host-backed
experts and staging buffers on its node, the n-gram table interleaved over the GPUs' nodes). When the system does not
report a card's node (GPU NUMA ID: N/A), those buffers are interleaved over all nodes; QE_NUMA sets the nodes by hand.
Both engines plan their host-RAM parts from the machine's size, not from what happens to be free at start. If other
programs hold RAM at start, the plan warns and stays: the archive keeps below MemAvailable minus the margin while
memory is short and grows when it comes back. Only the Flash-Next n-gram table is decided at start: when it does not fit
next to the experts as it is read, it comes from the page cache until the next restart (decode about 30 % slower at
c=1). If the free RAM is short at start, the server first waits up to 20 s for it (a previous server still exiting;
QE_RAM_WAIT_S). If the model can load it runs, with warnings; the load stops with a clear message only when the
experts really do not fit. The start-up line host RAM plan: ... says what was reduced. While serving, when another
program takes RAM, the archive gives memory back (log lines archive: host memory below the margin ...) and the server
keeps running.
Measured minimu 8 m free h 10BC8 ost RAM at start (3 cards, November 3, 2026): Flash-Next about 28 GB (the host-backed experts must be pinned: about 14 GB on 3 cards, about 31 GB on 2, plus a margin), full speed (the n-gram table pinned) from about 85 GB; 27B started with 24 GB free (the lowest I tried). Below its minimum Flash-Next stops at start with a message naming the shortfall, before it pins anything.
| part | host RAM |
|---|---|
| host-resident experts (what VRAM cannot hold) | about 12 GB (3 cards), about 31 GB (2 cards) |
| n-gram embedding table (INT8, pinned) | 51.8 GB |
conversation archive (FLASH_KV_ARCHIVE_GB) |
48 GB, filled as conversations leave VRAM |
pinned system prompts (FLASH_PREFIX_GB) |
8 GB, as agent kinds appear |
| server process (Python, vision encoder, CUDA) | about 5 GB |
FLASH_KV=ram (opt-in) |
13.5 GB per 1M tokens of KV, pinned at start; smaller when the host is short (see below) |
The defaults fit the machine (since 1.3.4). At start-up the engine plans its host memory from MemTotal and
MemAvailable (and a cgroup / docker --memory limit, if one is set). It first sets aside a reserve for the system
and the desktop: max(8 GB, 8 % of RAM). The rest goes, in this order, to:
- the host-resident experts, which are not optional;
- the n-gram table. It is pinned when it fits next to an 8 GB minimum archive. Otherwise it is read through the page
cache (as with
FLASH_NGRAM_PIN=0, below), and part of the remaining RAM stays free for the page cache to hold its rows; - the archive and the pinned system prompts, which get what is left, up to 48 + 8 GB.
With FLASH_KV=ram the KV pool in host RAM (FLASH_KV_RAM_TOKENS, 13,520 bytes per token, 13.5 GB at the default 1M;
7,488 bytes and 7.5 GB with FLASH_KV_BITS=4) is in the plan as well. It is pinned at start, like the experts and the n-gram table, so these three together must fit
both the RAM the machine has and what is free at start (MemAvailable minus the margin). When they do not, the plan
gives way in this order: the archive and the pinned prompts first; then the KV pool, down to the seats times the
context (FLASH_MAX_SEQS x --max-ctx, so every seat still holds a full window); then down to one window; then the
n-gram table moves to the page cache (it costs decode speed for every request, so it goes last). The experts are never
reduced; if even the smallest plan does not fit, the log warns and the server starts anyway. The start-up line shows
the pool and what is pinned at start next to what was free.
Flash-Next with a 4-bit KV cache (opt-in, FLASH_KV_BITS=4). Each key and value row of the attention layers (the
MTP layer too) is rotated (a fixed random Hadamard transform, which spreads the outlier channels) and stored in 4 bits
with one fp16 scale per 32 values: 576 bytes per layer and token instead of 1,040 with FP8, 7,488 instead of 13,520
bytes per token in all. The VRAM pool keeps its 262,144 tokens and gives the freed memory to the expert cache; with
FLASH_KV=ram the same host RAM holds 1.8x the tokens (1.05M tokens: 7.9 GB pinned instead of 14.2). Measured on 3 cards
(November 14, 2026, same day for both caches): KL against the FP8 cache 4.95e-2 on 32K-100K documents (two FP8 runs
that differ only in summation order: 2.52e-2), top-1 agreement 92.2 %; needle at 130K and at 1M (FLASH_KV=ram,
YaRN) pass; prefill of a 90K prompt 1-2 % faster, decode steps unchanged, MTP drafts accepted 3 % less (3.00 vs 3.10
tokens per step at 90K). Other values of FLASH_KV_BITS and settings that select an attention kernel without a
4-bit reader (FLASH_ATTN_TC, FLASH_ATTN_DEC=0) keep the FP8 cache and log it. On 2 cards, prompts of about
115K-188K prefill in 4096-row batches with the 4-bit cache (the smaller K/V arrays have less room for the big
batches). Known limit, from one community report on 4 cards
(#25): on a 1M-token recall ladder the 4-bit cache filled 83 % of the
window (every miss a refusal) against 91 % with FP8, so keep FP8 when maximum reach matters.
Nothing is planned beyond the RAM the machine has, so the defaults do not push it into swap. With about 140 GB or
more (3 cards) or about 160 GB or more (2 cards), the plan is exactly the defaults above. The log has one line with
the plan, and /api/stats reports it as host_memory:
[qflash] host RAM plan: MemTotal 98.8 GB, MemAvailable 92.8 GB, reserve 8.0 GB (OS + desktop), server base 6.0 GB,
engine buffers 1.3 GB; experts in host memory 31.2 GB (estimate 31.3 GB), n-gram table 51.8 GB from the page cache
(12.0 GB kept for it); KV archive 34.5 GB + pinned prefixes 5.7 GB; reduced: n-gram table not pinned: ...
Planned sizes by machine (start-up plan, desktop using about 6 GB):
| RAM | cards | n-gram table | archive + pinned prompts |
|---|---|---|---|
| 64 GB | 2 / 3 | page cache | 9.6 + 1.5 / 25.2 + 4.1 GB |
| 92 GB | 2 | page cache | 34.5 + 5.7 GB |
| 92 GB | 3 | pinned | 16.8 + 2.8 GB |
| 128 GB | 2 / 3 | pinned | 30.9 + 5.1 / 47.4 + 7.9 GB |
| 160 GB and more | 2 / 3 | pinned | 48 + 8 GB (the defaults) |
Overrides. A value you set wins: FLASH_NGRAM_PIN=1 pins the table, =0 keeps it in the page cache;
FLASH_KV_ARCHIVE_GB and FLASH_PREFIX_GB set the archive and the pinned prompts (0 turns them off). If a value you
set does not fit, the log says so and the value is kept. On Flash-Next FLASH_KV_ARCHIVE_GB=0 also turns off the pinned
prompts and every reuse that goes through the archive: no checkpoints, no copy of a cached prefix from another slot, and
a conversation that leaves its slot is prefilled again on its next turn. This holds with FLASH_KV=ram too: the KV pool
in host RAM holds only the conversations in the slots, it does not replace the archive. QE_HOST_BASE_GB (6) changes the share counted for the
server process.
Keeping RAM free for other programs: two settings, used at different times.
| setting | when it acts | what it does | raise it when |
|---|---|---|---|
QE_HOST_RESERVE_GB |
once, at start | RAM the start-up plan leaves out when it sizes the n-gram table, the archive and the pinned prompts | other programs will need more RAM later, or the plan comes out too large |
QE_MEM_MARGIN_GB |
all the time, while serving | free RAM (MemAvailable) the archive keeps: below it, the archive stops growing and gives memory back |
programs that start later get squeezed, or swap appears |
On a small machine, lower them to leave more for the archive: with both at their 8 GB default, a 16 GB machine has almost no archive, so a conversation that leaves VRAM is recomputed when it returns (a planned archive under 2 GB gets a warning line at start; the start goes on). The lower you go, the closer the kernel's OOM killer gets, and it would end the server.
- On 2 cards about half of the expert groups stay in pinned host memory (about 15 GB per card) and are read over PCIe or copied into the expert cache when hot.
- While it fills, the archive keeps at least max(8 GB, 5 % of RAM) free (
QE_MEM_MARGIN_GB), and gives memory back when other programs need it. The engine also watches the kernel's memory-pressure and swap-in counters while it runs: when other programs squeeze the host, it stops growing, shrinks the archive, drops pinned prompts and, last, moves the n-gram table to the page cache, one logged stage at a time, and keeps serving (QE_MEMP=0turns this off; details). - n-gram table in the page cache (
FLASH_NGRAM_PIN=0, or chosen at start-up): the table stays in the file mapping and a host thread copies the rows each step needs. With the table fully in the page cache, agent throughput at 8 conversations was 793 tok/s (pinned: 805); with the whole server limited to 12 GB of RAM, 493 tok/s (3 cards, October 25, 2026). After the load the engine drops the page cache of the weight files (it never reads them again), so they don't push the table out of the page cache (FLASH_WEIGHTS_COLD=0keeps it). Details: docs/advanced.md. - The 27B needs much less: its archive defaults to 32 GB (
QE_KV_ARCHIVE_GB) plus 8 GB of pinned prompts (QE_PREFIX_GB), sized the same way. Its defaults fit from 64 GB of RAM.
Linux only (developed on Fedora 44). Windows, including Docker Desktop / WSL2, is untested: WSL2 has no GPU P2P and limits pinned host memory, which the archive and Flash-Next's host-resident parts rely on.
48 layers (36 Gated DeltaNet + 12 sparse-attention layers), 512 routed experts (top-10) plus a shared expert, 4-stream hyper-connections and a per-layer n-gram embedding table. What the engine does:
- experts split over the cards in 128-wide groups (3 cards: 176/160/176, so the middle card can drive a desktop;
--splitsets one weight per card); INT4 expert GEMV / GEMM, dense parts INT4 or INT8 (stored that way in the checkpoint, or converted to INT8 at load where the file carries BF16; see the precision table below); - an expert cache: the least-used expert groups live in pinned host memory and are copied into free VRAM while hot;
- attention KV in FP8, split by position over the cards, with the indexer's top-512 block selection beyond 2K tokens; 262,144 tokens of context per conversation;
- MTP drafts plus lookup drafts (one conversation decoding alone: 3 to 5 drafts per step, picked from the draft head's confidence; batched steps: up to 3), steps launched ahead, CUDA graphs;
- up to 16 conversations at once with batched decoding; prompts that arrive together share prefill batches (the expert weights are read once for all of them);
- prefix reuse, checkpoints and a host-RAM archive of conversations; image input (
--vision 1); structured output.
Model files. The Hugging Face download is ready to
run. Since 1.3.4.1 its default revision stores the DeltaNet projections (in_proj_qkv / z / out_proj) as INT8 (group
128) and has the shared expert and the n-gram key/value projections calibrated with GPTQ (INT4); the routed experts
are unchanged. That brings the model about twice as close to BF16 (see
Quality) and costs decode speed: about 5 % with 8 conversations, about 4 % single-stream on the club-3090 bench and up to
9 % at one request in the concurrency test (the DeltaNet projections are read on every step, 0.35 GB more per card);
prefill speed is unchanged. The download is about 1 GB larger. The engine reads the format from the files, so the previous weights
still work as they are (QEngine 1.3.4 and older stop at load on the new files with tensor not found: ...linear_attn.in_proj_qkv.weight_packed: update QEngine, or download --revision v1.3.4):
hf download bpx03/Qwen3.8-Flash-Next-QEngine --revision v1.3.4 --local-dir models/Qwen3.8-Flash-Next-QE-INT4-v1.3.4
server/run.sh 8080 --model models/Qwen3.8-Flash-Next-QE-INT4-v1.3.4 # Docker: -e QE_MODEL=/app/models/...-v1.3.4What runs in which precision (1.3.4.1 default weights; the files store exactly these formats except where noted):
| part | precision | note |
|---|---|---|
| routed experts (512 per layer) | INT4, group 128 | Intel's AutoRound quantization, repacked bit-exact |
| DeltaNet projections (in_proj_qkv / z / out_proj) | INT8, group 128 | 1.3.4 weights: INT4 |
| shared expert, n-gram key / value projections | INT4, group 128 | calibrated with GPTQ (1.3.4 weights: round-to-nearest) |
| attention q / k / v / o, lm_head, router, hyper-connection mix matrices | INT8, group 128 | BF16 in the files, converted at load |
| embeddings, n-gram table | BF16 / INT8 | the n-gram table ships as INT8 |
| norms, gates, DeltaNet a / b and conv, indexer | BF16 | |
| activations, accumulation | BF16 / FP16, FP32 accumulation | prefill GEMMs use INT8 activations |
| attention KV cache | FP8 (E4M3) | 4-bit opt-in: FLASH_KV_BITS=4 |
| MTP draft head | INT8, group 128 (experts and shared expert INT4) | attention and fc BF16 in the files, converted at load |
Context up to 1M tokens (opt-in): FLASH_KV=ram server/run.sh 8080 --model <dir> --max-ctx 1M turns on the
model card's YaRN RoPE scaling (Qwen3.8-27B: --max-ctx 1M on 3 cards, --max-ctx 512K on 2, no FLASH_KV). It applies to every request of that server, so use it only on a server meant for long
documents; see docs/advanced.md.
KV cache in host RAM (opt-in): by default the attention KV of all conversations shares one VRAM pool of 262,144
tokens. With FLASH_KV=ram it lives in pinned host RAM (FLASH_KV_RAM_TOKENS, default 1M, 13,520 bytes per token
for the whole machine) and VRAM keeps the indexer keys and a cache of recent blocks; output is bit-identical. Many
long conversations then stay resident: with 4 conversations of ~64K tokens sending follow-ups at once, output went
from 455 to 580 tok/s on 3 cards and from 233 to 332 tok/s on 2 cards (October 30, 2026). It is not the default
because short-request throughput is about 12 % lower (the indexer keys and block cache take VRAM from the experts).
Details: docs/advanced.md.
- INT4 weights repacked at load (a few seconds, no conversion) into tensor-core layouts.
- Verify head: an INT4 copy of
lm_headscreens the vocabulary, and each row's top 16 candidates (top 64 for sampled rows and logprobs) are rescored exactly from the BF16 rows in pinned host RAM, so greedy output takes the BF16 argmax. It needs 2.5 GB of pinned host RAM on one card (0.85 GB per card on 3) and 0.64 GB less VRAM per card than an INT8 head. Without that host RAM, or withQE_HEAD4R=0, every card uses the INT8 head. - Tensor parallel over PCIe; the four full-attention KV heads are split by position (3 cards with P2P), so no KV head is duplicated. All-reduces run inside the kernels and inside CUDA graphs.
- Speculative decoding, verified over the full vocabulary, so outputs are unchanged: the DFlash2 block drafter (7 drafts per step) for a conversation that decodes alone, when its memory fits; MTP with 1 to 7 drafts per step, chosen from the measured acceptance, for everything else. Lookup drafts on the GPU.
- Up to 32 conversations (
QE_MAX_SEQS, default 16; more open requests are parked, see Serving behaviour); KV cache in FP8 in one pool per card (1,232,896 tokens on 3 cards with P2P and 16 slots, 1,075,200 with 32;QE_KV_BITS=4stores it in 4 bits for small cards, see GPUs); 262,144 tokens per conversation. - Prompt-cache states (snapshots of the recurrent state, about 150 MB of pinned host RAM each, split over the cards):
up to 8 per sequence slot while they take at most 8 % of the host's RAM (or of a container's memory limit), at least
4 per slot and at least 32 (
QE_SNAPSHOTS); with 16 slots on a 256 GB host that is 128 states, about 19 GB. The states within the 8 % are pinned at start while the host has room, before the first request (QE_ARCHIVE_WAIT_S); the rest only as the store fills. With 32 for every slot count, 8 agent conversations on 32 slots dropped each other's states and re-prefilled from the system prompt; with 4 per slot, 16 conversations on 16 slots still did. The state at the end of each prompt is copied to the host beside the first decode step instead of before it (about 6 ms less to the first token and per reply on one card, 2 ms on three; 16 MB of VRAM per card for the staged convolution rings;QE_SNAP_ASYNC=0copies it before the first step, as before).
Release files. Two files are attached to every release:
model_mtp_ft.safetensors (849 MB, an MTP head fine-tuned on the engine's own answers) and token_rank.bin (the
draft-vocabulary ranking). The server downloads them on the first start into ~/.cache/qengine/assets, checks them
against the release's SHA256SUMS and reuses them offline (the model directory is never written). Without them the
engine runs, just slower. Offline machines: put them into the model directory by hand:
cd models/Qwen3.8-27B-INT4
for f in model_mtp_ft.safetensors token_rank.bin SHA256SUMS; do
curl -LO https://github.com/bpx03/QEngine/releases/latest/download/$f
done
sha256sum -c SHA256SUMSQE_FETCH_ASSETS=0 turns the download off; the start waits for it at most QE_FETCH_WAIT_S (120) seconds.
Since 1.3.5.8 the 27B can draft with DFlash2, a small block drafter (the W4A16 build syvai/Qwen3.8-27B-DFlash2-W4A16, Apache-2.0). It reads the hidden states of five layers of the model and proposes 7 tokens per step; the model verifies them like MTP drafts, so greedy output is identical to MTP, and sampled output follows the same distribution (only the random draws differ).
Where it helps. It is used for a conversation that decodes alone: one chat, one agent, one long answer. Measured on this hardware (1.3.5.8; server, one conversation, against the MTP default of the same build):
| setup | short code | short prose | short math | 131K context |
|---|---|---|---|---|
1 card (QE_DRAFTER=dflash2) |
+38 % | +17 % | +22 % | +10.5 % |
| 3 cards, P2P | +34 % (321 → 432 tok/s) | +6 % (230 → 244) | +15 % (297 → 341) | – |
The gain depends on the text: code and other predictable output gain most; long reasoning that MTP and lookup drafts already predict well gains little (a 32K prompt with a long thinking part: +2 %).
Second choices on one card: when the drafter is unsure about its first three drafts (greedy replies), the step
verifies four drafts plus the drafter's second choice for each of the first three positions in the same 8 rows; a
right second choice keeps one more token. Output is identical. Measured on one card (engine, same build on / off):
+3.5 % on a 30-prompt mix, +4.6..4.9 % on the three short benchmark prompts, +2.7 % / +1.6 % at 32K / 96K, about level
on file edits. QE_DF_SIB=0 turns it off.
Two conversations at once on one card: when exactly two conversations decode in the same step, each drafts with its
own DFlash2 context (8 rows each); output stays identical to each conversation decoding alone. Measured on one card
(server, two 512-token replies at once): +2.8 % for the pair as a whole; +8 % per step while both run (engine benchmark). With three or more
conversations in a step, and on 2 or 3 cards, batched steps draft with MTP as before (three at once on one card was
level to slightly slower, four 5 % slower). QE_DF_BATCH=0 turns it off; QE_DF_BATCH=n allows it up to n
conversations per step. Grammar-constrained replies (response_format, a forced tool call until its header, logprobs) always keep MTP; a tool_choice: none reply drafts as usual. Agent turns that edit a file already in the context come out about 3 % slower,
because the model copies text the drafter no longer sees (it keeps the last ~2K positions); turns that write new text
are faster.
What it costs. VRAM on every card, plus 0.25 GB of pinned host RAM:
| cards | VRAM per card | KV pool (all conversations together) | context per conversation |
|---|---|---|---|
| 1 | 1.20 GB (three context rings for the two-conversation steps; QE_DF_BATCH=0: 1.16 GB) |
about −38K tokens | QE_MAX_SEQS=1: 206,848 → about 165,000 (default QE_RESERVE_GB=1.5) |
| 2 | 0.82 GB | 675,840 → 623,616 tokens | 262,144, unchanged |
| 3 | 0.70 GB | 1,298,432 → 1,232,896 tokens | 262,144, unchanged |
On 2 and 3 cards the default context still fits. On one card it does not, which is why a single card stays on MTP by default.
When it is on (QE_DRAFTER):
auto(default): at start the server loads DFlash2 only when the requested context (--max-ctx) and slots still fit with it, and prints one line saying which drafter it loaded and why (memory needed against memory free). The context is never lowered to make room for it. On 2 and 3 cards it is on with the defaults. On one card the default--max-ctx 262144does not fit, so a single card stays on MTP, and the start-up line tells you what this machine needs, e.g.For DFlash2: --max-ctx <= 165K or QE_DRAFTER=dflash2(the number is computed from your free VRAM). Either ask for that context or force it:With the default 16 slots one card has about 43K tokens per conversation and stays on MTP; DFlash2 on one card is meant for# one card: room for DFlash2 by asking for less context (use the number from your start-up line) QE_DEVICES=0 QE_MAX_SEQS=1 server/run.sh 8080 --model <dir> --max-ctx 150K # or force it: the context becomes whatever still fits (logged) QE_DEVICES=0 QE_MAX_SEQS=1 QE_DRAFTER=dflash2 server/run.sh 8080 --model <dir>
QE_MAX_SEQS1 to 4.dflash2: always load it; the served context shrinks to what fits, with a start-up line.mtp: never load it (and nothing is downloaded); the previous behaviour, full context.
Files. On the first start the server downloads the drafter (1.28 GB, a pinned revision checked against its
sha256) into ~/.cache/qengine/dflash2 (Docker: the qengine-cache volume) and reuses it offline. Without network
access and without the file the server starts on MTP and says so in one line; it never stops for it.
QE_FETCH_ASSETS=0 turns the download off; QE_DFLASH_URL points it at a mirror. A local copy: QE_DFLASH=<dir or file>, or a dflash2/ directory inside the model directory.
INT4 stays the default. Pointing the server at an 8-bit checkpoint runs the INT8 layer weights: closer to the BF16 model and slower, because a decode step reads twice the weight bytes. Three cards are recommended; one card can't hold them. The weights are at bpx03/Qwen3.8-27B-INT8-QEngine.
hf download bpx03/Qwen3.8-27B-INT8-QEngine --local-dir models/Qwen3.8-27B-INT8
server/run.sh 8080 --model models/Qwen3.8-27B-INT8 # Docker: -e QE_MODEL=/app/models/Qwen3.8-27B-INT8To build one from the official BF16 weights (no GPU, no calibration data; Python packages in requirements-ref.txt):
hf download Qwen/Qwen3.8-27B --local-dir models/Qwen3.8-27B-BF16
python3 tools/quant_int8.py --src models/Qwen3.8-27B-BF16 --out models/Qwen3.8-27B-INT8 \
--kv-from models/Qwen3.8-27B-INT4 # copies the FP8 KV scales, which keep the full KV poolINT8 KL and top-1 are from 1.3.5.2. Everything else, the INT4 KL and top-1 included, is from November 1, on older code, and not rerun since; 3 cards:
| metric | INT8 | INT4 | INT8 / INT4 |
|---|---|---|---|
| KL vs BF16, mean (320 positions) | 6.66e-4 | 1.47e-2 | 22× closer |
| top-1 match with BF16 | 99.7 % | 95.3 % | |
| decode narrative / code, tok/s | 184.6 / 276.7 | 266.6 / 387.7 | 0.69 / 0.71 |
| prefill 10K / 90K, tok/s | 4,816 / 4,098 | 6,077 / 4,866 | 0.79 / 0.84 |
| agent c=8, output tok/s | 810 | 1,215 | 0.67 |
| KV pool, 32 slots, 262K context, tokens | 566,272 | 932,864 | 0.61 |
The 8-pack has not run on INT8, so whether the lower KL gives better answers in real tasks is not known yet.
Your own checkpoint goes in through --model. The engine reads one model.safetensors (not shards),
compressed-tensors pack-quantized, symmetric, group 128: weight_packed int32 [N][K/4] with bytes LSB-first and the
stored value w + 128, weight_scale bf16 [N][K/128], tensor names as in the 27B INT4 checkpoint. A format it would
misread stops the start with the reason. Without FP8 KV scales it runs a bf16 KV cache with half the pool and no position
split (QE_KV_SCALES=<a 27B INT4 model.safetensors> borrows the scales; this fallback has not run on cards). INT8
checkpoints use their own model_mtp.safetensors, not the fine-tuned head (QE_INT8_MTP_FT=1 uses it). Not tried:
lued/Qwen3.8-27B-INT8-W8A16-MTP; it is sharded, so the start refuses it as it is.
Above 262K tokens the 27B needs YaRN (--max-ctx 512K or 1M), and for prompts that long most of the time goes to
attention; --lsa makes that attention sparse (the index map is downloaded once and cached, see the docs). Both are off by default. YaRN is static: it applies to every
request of that server, short ones too, so run a separate server for long documents. What --lsa changes
(3× RTX 3090, measured November 3, 2026; details in docs/advanced.md):
| you gain | you give up |
|---|---|
| 512K prompt about 2× faster (225 → 103–114 s), 1M about 3× (736 → 239 s) | exactness: each token reads 2,048–4,096 of its keys, which carry about 60–80 % of the full attention weight in my measurement |
| decode at 1M about 2× (106 → 227 tok/s), at 512K 1.5× | tested only on 2 retrieval tasks at 128K and 512K plus one 1M needle (same answers as full attention): not RULER, not reasoning quality over long documents |
| nothing changes below a prompt position of ~64K or a context of ~192K (full attention runs there) | only one conversation at a time benefits (the last long prompt's); others, and requests on a pinned shared system prompt, use full attention |
| about 0.5 GB per card less KV pool; needs P2P and the FP8 KV cache; at most 1.87M tokens |
--max-ctx(default 262,144) is the context of one conversation; the KV pool of all conversations together is sized from free VRAM. The served context is never larger than the pool: when the pool is smaller than--max-ctx, the server still starts, serves at most the pool's length (max_model_lenin/v1/models, the length check) and says so in one start-up line (27B on 2 cards with--max-ctx 1M):context: requested 1048576 tokens, KV pool 659456 tokens, served 659456 tokens per conversation.- Conversations that leave VRAM go to a host-RAM archive (
FLASH_KV_ARCHIVE_GB48,QE_KV_ARCHIVE_GB32; smaller on machines with less RAM, see Host memory): returning to a 50K-token conversation is a copy, not a recompute. - Pinned system prompts: the KV and state at the end of each agent kind's system prompt and tools are kept apart
from the archive (
FLASH_PREFIX_GB/QE_PREFIX_GB, 8 GB), so a new agent of that kind starts after its system prompt even when the archive is full. - There is no disk cache by default. With
--disk-cache DIR(both models, 200 GB at most,--disk-cache-gb) conversations are also written to files and survive a restart: a conversation that returns is read back instead of recomputed. Flash-Next has it since 1.3.5.3, experimental; details in docs/advanced.md. A failing disk costs only the cache; the server keeps serving.
DeepSeek-V4.1-Flash-1017: 40 layers, 384 routed experts per layer plus a shared expert, Engram n-gram memory tables and compressed sparse attention. QEngine runs it on three 24 GB cards plus host RAM, with the DSpark draft model for speculative decoding. It is a preview in 1.5: it runs in the same server and API as the other models, with the limits listed below.
Model files. I do not ship weights for this model. QEngine runs from the Maya-S GGUF files on Hugging Face
(peasantsmith/DeepSeek-V4.1-Flash-1017-Maya-GGUF,
revision c33d87b8), read as they are: the routed experts (IQ2_XXS / IQ2_S / IQ3_XXS / IQ3_S), the dense weights
(Q8_0), the Engram tables (Q6_K), the tokenizer and the chat format from the GGUF metadata, and the DSpark drafter from
the dspark/ folder next to Maya-S/. Nothing is converted on disk and the model directory is never written. About
348 GB with the drafter:
hf download peasantsmith/DeepSeek-V4.1-Flash-1017-Maya-GGUF --revision c33d87b8 \
--include "Maya-S/*" "dspark/*" --local-dir models/DeepSeek-V4.1-Flash-1017-GGUF
server/run.sh 8080 --model models/DeepSeek-V4.1-Flash-1017-GGUF/Maya-S
The server recognises the model from the GGUF metadata (architecture deepseek41) and loads the DSV4 engine
(build_dsv4/libqdsv4.so, built by src/dsv4/build.sh; the Docker image has it). Other revisions are not tested: the
start checks the model's dimensions and stops with the reason if they differ. The first start writes a few derived
files (tokenizer, the drafter's experts re-packed for the GPU, Engram hash constants) into
~/.cache/qengine/dsv4/<model id> (--cache-dir moves it; Docker: the qengine-cache volume), so the first start takes longer. After that the server
is ready about 41 s after start with the files in the page cache.
What it needs
- Exactly three NVIDIA cards with 24 GB each, compute capability 8.6 or 8.9 (RTX 3090 / 3090 Ti, RTX 4090; tuned
on the 3090). The DSV4 engine is built for sm_86 only, which runs on RTX 4000 (sm_89) cards as is; RTX 5000 cards
(sm_120) are not supported in 1.5. The engine uses the first three visible cards (
QE_DEVICES); a fourth is ignored, two are not enough. The expert pool on each card is sized from its free VRAM, so a card that drives a desktop works, with fewer experts in VRAM. - GPU P2P between every pair of cards is recommended, not required. With it, the context reservation is lazy:
--max-ctxdefaults to 262,144 and short chats run as fast as at 32K, because the long-context memory is taken from the expert pool only while a long prompt needs it and given back after. Without it (a stock driver, orDSV4_NOCTXLAZY=1) the reservation is static:--max-ctxstill defaults to 262,144, but its memory is held for the whole run, so fewer experts fit in VRAM (on my cards without P2P, decode 2-3 % slower at short context than with--max-ctx 32768). When that context does not leave 8 GiB free on the fullest card, the start halves it (down to 32,768) with a WARNING line. Without P2P the cards also exchange through pinned host memory. A start-up line says which reservation runs. An explicit--max-ctxalways wins. - Host RAM: about 170 GB free for the full setup (on my machine
MemAvailabledrops by about 165 GB while it serves a 233K-token conversation). With less, the engine keeps fewer host copies of the experts and reads the rarest ones from the GGUF files; I tested that under a RAM hog taken while it serves, not on a machine with less RAM. It never allocates pastMemAvailableminus a safety margin: it serves slower instead of running into the out-of-memory killer, and gives memory back while serving when the host runs short. - An NVMe drive for the model files. The two Engram shards (5 and 6, about 161 GB) stay on disk and are read row by row while a prompt is processed. A hard disk or a network share is far too slow for this.
- A CPU with AVX-512 is recommended; AVX2 is the minimum (x86-64-v3: AVX2, FMA, F16C, BMI2). The experts that are
not in VRAM run on the CPU. The engine picks its CPU kernels at start: the AVX-512 kernels need the Zen 4 set
(AVX-512 with VNNI, VBMI, BITALG, BF16: AMD Zen 4 / Zen 5, Intel Sapphire Rapids and later); other CPUs get the AVX2
kernels, with the same output but slower decode. The start-up line
CPU lane: N threads, AVX-512 kernelsshows which. The CPU part uses 24 threads by default (--engine-args "--cpu-threads N"). - Linux and an NVIDIA driver with CUDA 13.1 support, as for the other models.
What works
- The same server and API as the other models: OpenAI
POST /v1/chat/completions(streaming or not), AnthropicPOST /v1/messagesandcount_tokens,GET /v1/models,GET /health. Responses carrytimings(prompt and decode tokens and seconds). - Thinking on by default (
--think, a request'senable_thinking).reasoning_effort:low,medium/high,max/xhigh, or a number 1-100 (default 75);noneturns thinking off. The reasoning comes back inreasoning_content(OpenAI) or thinking blocks (Anthropic). The thinking budget (--thinking-budget, a request'sthinking.budget_tokens/thinking_budget) closes the reasoning at its count. - Tool calls in the OpenAI and Anthropic formats (the model's own DSML call blocks are parsed into tool calls),
tool_choice(auto,none,required, a named tool) and structured output (response_formatjson_object/json_schema). - Several requests at once (agents): up to
--max-active 8are in service, more wait in the queue. They take turns on the GPU: a long prompt in chunks of 10,240 tokens, an answer in slices of 16 decode passes, so a short request does not wait behind a long prompt or a long answer. The engine decodes one conversation at a time, so the total speed stays that of one request; a switch between conversations is exact (each one's text is the same as served alone with the same expert state). - Shared system prompts (agent teams): when a second conversation starts with the same system message (tools
included, at least 1,024 tokens) as an earlier one, that part is kept once, and every later conversation with it
starts from the kept copy and computes only its own part.
--no-prefix-shareturns it off. - Multi-turn reuse and a host-RAM archive: a follow-up turn reuses the cached prompt, also with thinking on;
regenerating a reply reuses it too. Idle conversations move to host RAM (
--archive-gib, default 32 GiB, least recently used out first) and come back with a copy instead of a recompute.cache_prompt: falsein a request forces a full prefill.usage.prompt_tokens_details.cached_tokensshows what was reused. - Stop strings (
stop), matched in the answer only, never inside the reasoning. - Sampling:
temperature,top_p,top_k,min_p,seed; defaults from the GGUF metadata (temperature 1.0, top_p 1.0) unless set with--temperature/--top-p. Speculative decoding with DSpark keeps the sampled distribution exact. - Monitoring:
qetopand the dashboard on the metrics port (/api/statshas a DSV4 panel: held conversation, archive, K/V pool, prefix entries, passes). A conversation that has to be computed again because it left the archive shows up as a refill, counted with the prefill.
Known limits (preview)
- Text only: no images. No logprobs, no presence / frequency penalty (the DSV4 sampler has none), no activation steering (a request that asks for it gets "not supported").
- Constrained replies: structured output and
tool_choicerequired/ a named tool decode greedy replies (temperature 0) on DSpark passes, each pass checked against the grammar (the same text as one masked step per token, about 1.6x faster); sampled constrained replies decode one token per step without drafts, so they are slower.QE_DSV4_GRAM_PASS=0turns the passes off. - One conversation decodes at a time. Several requests share the GPU in turns, so the total output speed with 8 agents is about that of one (on my cards 66 tok/s for 4 agents with unrelated prompts, 98 tok/s for 8 agents that share a system prompt), and each agent gets its share.
- Greedy text depends on more than the prompt. Which experts sit in VRAM changes as the model runs, and the GPU and
CPU paths round differently, so the same greedy request can give a different text after unrelated requests, with
another amount of free VRAM (a desktop on a card), with another
--cpu-threads, with another--max-ctxand when a prompt is fed in chunks (beside other requests, or after a shared system prompt). The last three come from the prefill's CPU expert split:DSV4_NOCPUPF=1removes them at a prefill cost.--tier-resetmakes repeats of a request give the same text (see Settings). - Editing an earlier message re-processes the prompt from that point (reuse works when the new prompt extends an old one or a kept system prompt).
- Quality is that of the Maya-S quantization (2- to 3-bit experts, Q8_0 dense weights, Q6_K Engram), not of the original checkpoint.
- Decode speed at long context (256-token replies, greedy, on my cards): 103 tok/s with 117K tokens of context and 91 tok/s with 233K, against 76-106 tok/s for three prompts at short context.
Z.ai's GLM-5.3-Flash: 45 layers, 288 routed experts per layer (8 active per token) plus a shared expert, attention
layers with a sparse indexer for long prompts next to KDA linear-attention layers, and one MTP layer. QEngine runs it
on three 24 GB cards plus host RAM, with its own engine (src/glm): the experts used most often sit in VRAM, the full
set in pinned host RAM, and the rest of a step's experts run on the CPU next to the cards. It is a preview in 1.5: it
runs in the same server and API as the other models, with the limits listed below.
Model files. My INT4 quant of GLM-5.3-Flash, bpx03/GLM-5.3-Flash-INT4-QEngine (routed experts and KDA layers in INT4; lossy, see the limits). The GLM engine runs only from this checkpoint: it holds my GPTQ INT4 experts and KDA layers in QEngine's own layout, so other GLM-5.3-Flash quants, GGUF files and the original weights do not load. About 186 GB:
hf download bpx03/GLM-5.3-Flash-INT4-QEngine --local-dir models/GLM-5.3-Flash-INT4-QEngine
server/run.sh 8080 --model models/GLM-5.3-Flash-INT4-QEngine
The server recognises the model from config.json (model_type glm5_next) and loads the GLM engine
(build/libqglm.so, built by the normal CMake build when src/glm is in the tree; the Docker image has it). The MTP
draft vocabulary and the expert prior (tools/glm/draft_vocab_49k.i32, tools/glm/expert_prior_gen.f32) are read
from next to the library (<root>/build/libqglm.so -> <root>/tools/glm/); without them the engine still runs, with
the full drafter head and an empty expert cache at start (a start-up line says so). The chat template is the model's
own chat_template.jinja. Loading takes about 60 s with the files in the page cache.
What it needs
- Three NVIDIA cards with 24 GB each (tuned and tested on RTX 3090s). Two cards also work, with fewer expert slots per card (slower decode). With four or more cards the engine uses the first three.
- GPU P2P is optional. Without it (the stock GeForce driver) the all-reduces go through host memory. A start-up
line says which path runs;
GLM_P2P=0forces the host path. - About 192 GB of host RAM: the experts (about 157 GB) stay pinned in host RAM, plus the conversation archive (below, up to 32 GB by default). Less RAM means a smaller archive; the experts themselves have to fit.
- An x86-64 CPU with AVX2. The CPU expert lane is built twice and picks its kernels at start: AVX-512 when the
CPU has the full Zen 4 set (with VNNI and BF16: Zen 4, Sapphire Rapids and later), else AVX2 (
GLM_CPU_ISAforces one). The lane's thread placement is derived from the CPU topology at start (one log line). - Linux and an NVIDIA driver with CUDA 13.1 support, as for the other models.
What works
- The same server and API as the other models: OpenAI
POST /v1/chat/completions(streaming or not), AnthropicPOST /v1/messagesandcount_tokens,GET /v1/models,GET /health. - Tool calls in the GLM format (
<tool_call>with<arg_key>/<arg_value>), returned in the OpenAI and Anthropic formats;tool_choiceauto,none,requiredand a named tool; structured output (response_formatjson_schema/json_object). - Reasoning on or off (
enable_thinking, Anthropic thinking),reasoning_effortas the model's template takes it (low,high,max); the thinking budget (--thinking-budget, a request's budget) closes the reasoning at its count. - logprobs /
top_logprobs, stop strings (stop) with an exact rollback of the tokens past the match. - Prompt cache: a follow-up turn reuses the prefix it shares with earlier turns; idle conversations go to a conversation archive in pinned host RAM and come back with a copy instead of a new prefill.
- MTP speculative decoding with the model's own MTP layer.
- Several conversations at once (8 by default,
GLM_CONVS): replies take turns on the GPUs. A switch parks the running reply between two decode steps (its state, and its expert cache contents, saved to host RAM) and resumes the other one exactly where it stopped, so each reply is bit-identical to the same reply served alone. A new prompt starts once the running reply has had its turn (GLM_PREEMPT_MS, by default the slice length), so short replies end without a switch; long replies are sliced (GLM_SLICE_MS2000, at least 8x the measured cost of a switch: a park, ~0.06 s, plus the later resume, ~0.3-0.6 s, so about 4-5 s per turn). With 8 agents (agent replay: 2-16K-token prompts, 64-token replies) the first token comes in 13.2 s p50 instead of 28.3 s one at a time, and the total output is 22.4 instead of 12.6 tok/s; 4 clients with 256-token replies get the same total output as one at a time (70.5 vs 70.7 tok/s); one conversation alone runs as before. - Long prompts return between layers: the prefill runs in host slices of about one second, so the server keeps reporting progress; a cancel (or a client that disconnects) stops the prompt at the next layer, within about half a second, and the cache goes back exactly to the start of the interrupted prefill window.
Known limits (preview)
- One reply decodes at a time: several conversations share the GPUs in turns (see above), not in one batch, so the total output speed does not grow with the number of conversations; a switch between two long replies costs ~0.3 s (the expert cache contents of the reply that resumes are copied back).
- Parked replies and the snapshots of up to 3 x
GLM_CONVSconversations use pinned host RAM (about 150 MB each, plus the archived attention rows); under host-RAM pressure a parked reply whose rows were given back is prefilled again (logged; that reply is then no longer bit-identical to a run alone). - Greedy output is deterministic for the same request on the same machine and settings, but it depends on the number of expert slots (free VRAM, the context setting, the number of cards) and on how the prompt was split across prefill windows: a prompt resumed from the cache can differ in the last bits from one prefilled at once.
- Constrained replies (structured output, forced tool calls) and logprobs are slower: each step reads the full logits on the host.
- The default checkpoint is a lossy INT4 quant (routed experts and KDA layers).
- Text only (no image input), no presence / frequency penalties (accepted and ignored), no activation steering, no disk cache, no smaller K/V formats.
OpenAI's gpt-oss-120B: 36 layers, 128 experts per layer (4 active per token), MXFP4 expert weights, grouped attention
with every second layer on a 128-token window, and the harmony chat format. QEngine runs it on three 24 GB cards, all of
it in VRAM, with its own engine (src/gptoss). It is a preview in 1.5: it runs in the same server and API as the other
models, with the limits listed below.
Model files. The official checkpoint as OpenAI publishes it, read as it is: the MXFP4 experts and the bf16 dense
weights, nothing converted, nothing of mine to download. I tested revision b5c939de; about 65 GB without the
original/ and metal/ folders:
hf download openai/gpt-oss-120b --revision b5c939de --exclude "original/*" "metal/*" \
--local-dir models/gpt-oss-120b
server/run.sh 8080 --model models/gpt-oss-120b
The server recognises the model from config.json (model_type gpt_oss) and loads the gpt-oss engine
(build_gptoss/libqgptoss.so, built by src/gptoss/build.sh; the Docker image has it). Without that library the
start stops and says how to build it. The chat template is the model's own chat_template.jinja. The server is ready
about 11 s after start with the files in the page cache.
What it needs
- Three NVIDIA cards with 24 GB each, compute capability 8.6 (RTX 3090 / 3090 Ti). The engine is built for sm_86 only. RTX 4000 cards (8.9) run the same code untested, with the output head computed in full: its fast two-stage form has a rounding margin I measured on 3090s only, so a start-up line says the full head is used (slower decode: about 6 % for one conversation on my 3090s). Other cards (A100 8.0, RTX 5000 12.0, ...) are refused at start in 1.5. Two or four cards are accepted but not tested (two 24 GB cards cannot hold the 65 GB of weights: the start stops with what each card would need); one card is refused.
- GPU P2P between the cards is not needed. With it the cards write into each other's memory. Without it (the
stock GeForce driver) they exchange through pinned host memory, with the same output bit for bit: on my cards the
host path reaches about 94 % of the P2P speed (2026-11-19, engine tool: decode 314 against 332 tok/s for one
conversation, 951 against 1,009 tok/s for eight, a 10K-token prompt at 13.1K against 13.8K tok/s). A start-up line
says which path runs;
GPTOSS_COMM=hostforces the host path. The host path takes the same VRAM as the P2P path and about 0.5 GB of pinned host RAM;GPTOSS_PF_HOSTLANES=0withGPTOSS_LM=fullis the older, slower host path with about 0.5 GB more K/V room per card. When the host cannot pin that memory at start, the engine falls back by itself (the one-lane host form, a smaller prefill chunk, pageable row copies; one WARNING line each, slower prefill) and keeps serving. - Host RAM only for the conversation archive (below); the model itself stays in VRAM.
- Linux and an NVIDIA driver with CUDA 13.1 support, as for the other models. 10BC8
What works
- The same server and API as the other models: OpenAI
POST /v1/chat/completions(streaming or not), AnthropicPOST /v1/messagesandcount_tokens,GET /v1/models,GET /health. - Reasoning always runs (the model has no off switch).
reasoning_effortlow/medium(default) /high; other APIs' names are mapped:maxandxhightohigh,minimalandnonetolow, thinking off (enable_thinking: false) tolow. The reasoning comes back inreasoning_content(OpenAI) or thinking blocks (Anthropic); the thinking budget (--thinking-budget, a request's budget) closes it at its count. - Tool calls (the harmony
functionsnamespace) in the OpenAI and Anthropic formats,tool_choice(auto,none,required, a named tool) and structured output (response_formatjson_object/json_schema). A history with several tool calls in one assistant turn (from another model) is rendered as one call and its result after the other, as harmony expects; several system / developer messages at the start are joined into one. - Several conversations at once: 8 slots when the context per conversation is at least 32,768 tokens (the default
131,072), else 4 (
GPTOSS_SLOTS); they share one K/V pool (3x24 GB: 144,611 rows in bf16). Greedy and plain-temperature replies decode together in one step, and a reply's tokens are the same as when it runs alone. A reply that cannot grow because the pool is full ends withfinish_reasonlength; a new prompt waits until the running replies have room to grow. - Prompt cache: a follow-up turn reuses its conversation's K/V; a shared system prompt (tools included) is kept
once for all conversations that start with it; idle conversations move to host RAM (
QE_KV_ARCHIVE_GB, default 128 GiB, never closer thanQE_KV_ARCHIVE_MARGIN_GB= 8 GiB toMemAvailable; least recently used out first) and come back with a copy.cache_prompt: falseforces a full prefill. - Lookup drafts for greedy replies (
GPTOSS_LOOKUP): tokens that repeat earlier text (edits, code) are checked in one multi-row step; the text is exactly the plain greedy text. - Sampling:
temperature(default 1.0, OpenAI's recommendation),top_p(default 1.0),top_k,min_p,seed. Plain temperature sampling decodes at greedy speed; requests withtop_p< 1,top_kormin_prun the full output head one row per step (slower: about 6 % for one conversation on my cards, more with several at once). - Presence / frequency penalties (
presence_penalty,frequency_penalty, OpenAI's definition: a token's logit drops byfrequency_penaltytimes its count in the reply so far, pluspresence_penaltyonce it has appeared; before the temperature), per conversation, on the GPU and at the plain decode speed, also for several conversations at once. A greedy request with a penalty takes the highest penalised logit. A reply with a penalty runs without lookup drafts. - Stop strings (
stop), matched in the final answer only, never in the reasoning. - A device fault ends the process (exit code 75) after the open requests get an error, so llama-swap or Docker
restarts it instead of a server that answers
/healthand fails every request.
Known limits (preview)
- Text only: an image part is answered with
[image omitted: this server runs without vision]in its place. No logprobs (HTTP 400). No activation steering, no disk cache (--disk-cacheis ignored with a start-up line). - One tool call per reply (the model's format); the built-in
browserandpythontools are not run by the server. - Constrained replies (structured output,
tool_choicerequired/ named /none) decode one row per step. - The DFlash drafter is not used by the server in 1.5 (
GPTOSS_DFLASHis ignored with a start-up line). - A greedy or seeded reply in a reused conversation can differ from the same prompt sent fresh where two tokens are nearly tied: rows computed by decode and by prefill differ in rounding. The same holds for a conversation that shares its beginning with another one the server still holds (the same system prompt, or just the harmony header): the shared rows are reused, so the rest of the prompt is computed in a call of another size, which rounds differently. The text is equally valid either way; requests that start from the same server state give the same reply.
- Context per conversation: 131,072 tokens (the model's own limit,
--max-ctx); the pool is shared by all slots.
The five engines don't all do the same things. This table is what each one supports today (✅ yes, 🟡 partly, ❌ no).
| feature | Qwen3.8-27B | Qwen3.8-Flash-Next | DeepSeek-V4.1-Flash | GLM-5.3-Flash | gpt-oss-120B |
|---|---|---|---|---|---|
| Tool calls, streaming, OpenAI + Anthropic APIs | ✅ | ✅ | ✅ | ✅ | ✅ |
Structured output (response_format) |
✅ | ✅ | ✅ | ✅ | ✅ |
tool_choice required / named / none |
✅ | ✅ | ✅ | ✅ | ✅ |
logprobs |
✅ | ✅ | ❌ HTTP 400 | ✅ | ❌ HTTP 400 |
| Presence / frequency penalty | ✅ | ✅ | ❌ ignored | ❌ ignored | ✅ |
| Thinking budget | ✅ | ✅ | ✅ | ✅ | ✅ |
| Thinking off | ✅ | ✅ | ✅ | ✅ | 🟡 always reasons (effort low) |
Images (--vision 1) |
✅ | ✅ | ❌ | ❌ | ❌ |
| Activation steering | ✅ | ✅ | ❌ | ❌ | ❌ |
| Prefix cache, shared system prompt | ✅ | ✅ | ✅ | ✅ | ✅ |
| Idle conversations in host RAM | ✅ | ✅ | ✅ | ✅ (8 at once, one decodes at a time) | ✅ |
Disk cache (--disk-cache) |
✅ | ✅ | ❌ | ❌ | ❌ |
KV cache in host RAM (FLASH_KV=ram) |
❌ | ✅ | ❌ | ❌ | ❌ |
| Smaller K/V formats (opt-in) | ✅ 4-bit | ✅ 4-bit | ❌ | ❌ | ✅ FP8, 12-bit K |
| Max context (default) | 256K, 1M with YaRN | 256K, 1M with YaRN + KV in RAM | 256K | 256K | 128K |
Parked replies beyond the seats (--park) |
✅ | ✅ (3+ cards) | ❌ | ❌ | ❌ |
| Drafts | MTP, DFlash2, lookup | MTP, n-gram, lookup | DSpark | MTP | prompt lookup |
| Sparse attention for long prompts | ✅ LSA (opt-in) | built in | built in | built in | ❌ |
| Cards | 1-4 | 2-4 TP, more in PP | 3 | 3 (2 with fewer expert slots) | 3 (2-4 accepted, untested) |
| Without GPU P2P | ✅ | ✅ | ✅ | ✅ | ✅ |
| Docker image | ✅ | ✅ | ✅ | ✅ | ✅ |
n > 1, logit_bias and parallel_tool_calls: false are ignored on every model.
At start the server logs one [settings] line for every flag or environment variable that is set but does nothing for
the loaded model, e.g. FLASH_KV=ram is set but has no effect for gpt-oss-120B (KV stays in VRAM ...). A request
parameter the model doesn't support gets one [api] line the first time it shows up; the request is then served as
the table says (ignored, or HTTP 400).
- OpenAI:
POST /v1/chat/completions(streaming or not),GET /v1/models. - Anthropic:
POST /v1/messages,POST /v1/messages/count_tokens. GET /health,GET /props(llama.cpp style),GET /v1/steering.
/v1/models and /props report what a client needs to size its requests: max_model_len (prompt + reply tokens of
one conversation), max_num_seqs (conversations served at once), max_prompt_tokens, max_output_tokens (null:
no cap beyond the context), kv_pool_tokens, kv_cache (vram / ram) and input_modalities; /props adds
total_slots and default_generation_settings.n_ctx.
Too-long prompts get HTTP 400 before any streaming starts. The Anthropic path says
prompt is too long: N tokens > M maximum (Claude Code compacts the conversation on it). The OpenAI path uses
OpenAI's wording, This model's maximum context length is M tokens. However, your messages resulted in N tokens ...
with code context_length_exceeded, which harnesses such as verifiers / AutomationBench and LiteLLM recognise as a
context overflow.
Tool calls: OpenAI tools and Anthropic tool_use / tool_result. Tools are rendered sorted by name with
sorted JSON keys, so a client that reorders the same tools keeps the prompt cache (--sort-tools 0: the client's
order). The chat template is server/templates/qwen38_syssafe.jinja (the model's template, accepting several leading
system messages). --template replaces it. A reply starts as reasoning only if the template's generation prompt opens
<think>: with a custom template that does not, a model reasoning without writing the tag shows that text in content,
followed by a literal </think>. Since 1.3.6 a <tool_call> that does not parse as a call (prose inside the tag,
broken markup), or a call that tool_choice filters out, reaches the client as plain text in content, streamed or
not, on both APIs, instead of an empty reply.
tool_choice on both APIs: auto (default), none (Anthropic {"type": "none"}: the tools stay in the prompt, the
tool-call token is masked, so the model answers in text), required (Anthropic any) and a named function (OpenAI
{"type": "function", "function": {"name": ...}}, Anthropic {"type": "tool", "name": ...}): the answer starts with
that call after any thinking. Enforced in the engine: none bans the tool-call token in the normal decode path, so
such a reply keeps its batching and speculative drafts and runs as fast as auto (with logprobs it is masked on the
constrained path instead). A forced call runs through its thinking and the call header on the same masks as structured
output (on Flash-Next such a reply steps outside the batch until then). Calls to other tools than a named one are
dropped. Client stop sequences do not cut a forced reply before its call is complete; they
apply after it. Under none a call the model writes out as plain text is text and arrives in content. An
unknown name, or a forcing choice without tools or together with response_format, is a 400.
Structured output: response_format with json_object or json_schema, enforced in the engine (llguidance
grammars) together with speculative decoding. Stop sequences are supported.
Sampling: values a request doesn't send come from the Qwen3.8 model cards, per reply mode. Thinking: temperature 1.0,
top_p 0.95, top_k 20, min_p 0, presence penalty 0. Non-thinking: temperature 0.7, top_p 0.80, top_k 20, min_p 0, presence
penalty 1.5. --temperature, --top-p, --top-k, --min-p and --presence-penalty set one value for both modes, and a
value in the request wins. Greedy requests (temperature 0) get no penalty. The cards warn that a higher presence penalty may
occasionally mix languages and slightly lower quality: --presence-penalty 0, or presence_penalty: 0 in a request, turns
it off. The speed cost of presence 1.5 was measured on Flash-Next (3 cards, 8 agents at once, November 15, 2026):
with a step's verify rows sampled one after another it took 17-20 % of the throughput; since 1.3.6 they are sampled
in parallel (QE_SAMPLE_PAR), which gave +16 % on that test (697 → 808 tok/s). Its effect on answer quality and
tool calls has not been measured here.
presence_penalty and frequency_penalty are applied on the GPU.
logprobs / top_logprobs on /v1/chat/completions (top_logprobs 0-20, streamed too; since 1.3.5.4, experimental). The numbers
are the target model's own next-token distribution: raw logits through log-softmax over the full vocabulary, before
temperature, top-p, top-k, min-p and penalties, never a draft head's. Steering, when used, is included. content covers
only the tokens of content, not reasoning or tool-call tokens; with logprobs the non-streamed content is not stripped,
so the entries join to it. Such a request decodes one token per step without draft speedup, about 3x slower, and while it
runs a normal reply beside it drops to about 0.4-0.45 of its speed alone. In pp the numbers carry the mode's own
rounding, so use tp for KL-grade measurements. On the 27B the top 64 values of each row are exact BF16 logits; the
normaliser also sums the INT4 screen's values for the rest of the vocabulary (log-prob error about 0.01 at most).
Prompt logprobs and /v1/completions are not available.
Thinking and reasoning effort: thinking is on by default (--think 0 turns it off by default;
chat_template_kwargs.enable_thinking per request; a top-level enable_thinking is accepted too when
chat_template_kwargs does not set it, and a null counts as not sent). reasoning_effort (xhigh = the template default, high is the
same, medium, low) per request or as a server default with --reasoning-effort / QE_REASONING_EFFORT; a
request's value wins. max is taken as xhigh and minimal as low (the names other APIs use); other unknown
values render as medium. On /v1/messages the Anthropic output_config.effort is used when the request sends no
reasoning_effort (so max gets the most reasoning). The OpenAI Responses / OpenRouter form reasoning: {"effort": ...}
is read too, on both paths, after chat_template_kwargs.reasoning_effort and the top-level reasoning_effort.
none (any case, in any of these fields) turns thinking off,
the same as enable_thinking: false, also when the same request sets enable_thinking: true (an "off" always wins).
QE_REASONING_EFFORT=none makes thinking off the default; a request that names another effort, sets
enable_thinking: true or the Anthropic thinking.type enabled / adaptive still thinks.
The OpenRouter switches are read last: reasoning: {"enabled": false} is thinking off when the request sends no other
thinking setting or effort; "enabled": true turns thinking on over the server default (--think 0,
--reasoning-effort none) at the effort the request names, unless the same request turns thinking off.
reasoning: {"exclude": true} on /v1/chat/completions: the model reasons as usual (the reasoning tokens are in
usage), but the reply carries no reasoning_content, streamed or not.
Effort only asks the model to think less. A model can still reason until max_tokens runs out and return a turn
with reasoning only (no answer, no tool call), which an agent takes as the end of its task and a benchmark scores as
zero. A budget stops that:
- per request: Anthropic
thinking: {"type": "enabled", "budget_tokens": N}; OpenAI paththinking_budget(top level or inchat_template_kwargs;thinking_token_budget,max_thinking_tokens,custom_params.thinking_budgetandreasoning.max_tokensare accepted too); - server default for requests that send none:
--thinking-budget N/QE_THINKING_BUDGET=N(0 = off), one budget for every request. A request's value wins; a negative one turns the budget off for that request; 0 turns thinking off. - or a default per reasoning effort:
QE_THINKING_BUDGET=low:4096,medium:16384,xhigh:32768(the same value works for--thinking-budget), optionally withdefault:N(or*:N) for efforts the map does not list. The effort is the one the prompt is rendered with (the request's, else--reasoning-effort, else the template'sxhigh), named the way the template renders it:highandmaxuse thexhighentry,minimalthelowentry, other unknown values themediumentry. An entry of 0 or below means no budget for that effort (as a plain 0 or negative number). Order: the request's budget, then its effort's entry, thendefault:, then none. With a custom--templatewhose efforts the server does not recognise, a warning is printed at start-up, efforts are matched as sent, and requests without one use thexhighentry. - since 1.3.4.3 a malformed value (e.g.
8k,1.5,low:4k, an unknown effort) stops the server at start-up with an error naming the entry; 1.3.4.2 and earlier warned and ran without a budget. - the start-up line prints the map, and
/api/stats(thinking.budget_from) and the per-request log line (think 66/4096 (effort:low)) show where each budget came from:request,effort:<level>,default,off(the request turned the budget off) ornone.
At the budget the server writes a one-sentence wrap-up and </think> (27 tokens; QE_THINK_WRAP changes the
sentence), and the model goes on with its answer or tool call. The reasoning holds exactly the budget plus the
wrap-up: near the budget (and near the answer tail) the server caps how many drafts a step may verify, so the close
lands at the same token whatever the drafts (before 1.3.6 it could come up to one decode step late, and the
text then depended on the drafter).
max_tokens counts thinking and answer together. Answer tail: when max_tokens would run out inside <think>,
the same close comes early enough to leave the answer the last eighth of max_tokens (64 to 2048 tokens;
QE_THINK_TAIL, a token count, 0 = off), with or without a budget (not with ignore_eos). finish_reason is length only when the answer
itself is cut off; usage.completion_tokens_details.reasoning_tokens reports the thinking tokens.
usage.completion_tokens (Anthropic output_tokens) counts the end-of-turn token the model generated, as vLLM
does: a reply of 7 text tokens that ends on its own reports 8. A reply cut by max_tokens or a stop string has no
end token, and neither has structured output that ends when its grammar is complete (the server ends that turn
itself). Before 1.3.6 the end token was never counted.
Cost of the forced tokens. They are verified like drafts, up to 4 per step on Flash-Next and up to 8 on the 27B, together with MTP and lookup drafts. On the 27B a constrained request stays in the batched step with the others. On Flash-Next a constrained step (a forced close, and structured output in general) runs as a step of its own, so with several conversations decoding at once it adds steps the others wait for. A request without a budget that never reaches the tail decodes exactly as before.
Now and then a thinking model ends its turn with reasoning only (end-of-turn sampled inside or right after
<think>) or with a tool call that does not parse. Agent harnesses (AutomationBench / verifiers and many OpenAI-API
agent loops) end the task on a turn without text and without a tool call. Such a non-streamed, sampled reply
(finish stop) is sampled again up to 2 times (--empty-retry N / QE_EMPTY_RETRY, 0 = off). The prompt is
cached, so a retry costs only its decode. Never retried: streams (the bytes are already sent), greedy (temperature
0) and seeded requests, structured output, ignore_eos, requests with stop sequences. usage.completion_tokens
counts every try; finish_reason is the last try's. A tool call that does not parse is retried the same way; when
the last try still does not parse, its text is returned as content (streams: at once), so the client never gets an
empty turn from it.
--vision 1 (both models; Docker: the vision image) accepts images in OpenAI (image_url, data URLs or http(s))
and Anthropic (image blocks) messages. The vision encoder runs in PyTorch on the last GPU (Flash-Next: ~0.6 GB of
VRAM there, weights streamed from host RAM; 27B: ~1.3 GB). Image tokens take part in prefix caching, keyed on the image
content. A 1600×900 screenshot is encoded in about 0.14 s on Flash-Next. Unreadable images and video / audio parts get
HTTP 400. No vision quality benchmark yet. Details: docs/advanced.md.
Both engines can steer the model with control vectors in the llama.cpp control-vector GGUF format (plus a JSON
sidecar), in add or project mode, chosen per request with a steering field. QEngine ships one vector for
Flash-Next, off by default: refusal_projection, which projects a refusal direction out of the residual stream so
the model declines far fewer requests. Removing refusals removes a safety behaviour of the model; you are responsible
for what it outputs. Turn it on in Docker with -e QE_STEERING=1, from source with
QE_STEER=vectors/refusal_projection.gguf QE_STEER_DEFAULT=refusal_projection. Vector format, loading your own
vectors, per-request selection and credits: docs/advanced.md.
Scheduling. Shortest-prefill-first queue with aging; long prompts are prefilled in chunks between decode steps
of the running conversations; short requests can cut in between the chunks of a long prefill (Flash-Next: requests
up to 8K tokens, e.g. tool results, join the next prefill batch at once while the long prompt keeps half of it, on
servers whose prefill is bound by host-backed expert copies; --long-share). 27B: the time split follows the queue:
while the prompt to prefill next is short (up to --prefill-short, 4,096 tokens), prefill passes run back to back
with one decode step between them; a longer one leaves the running conversations a share by head count (running /
(running + waiting), between --prefill-share-min 0.10 and --prefill-share 0.5), and over any 2 seconds they keep
at least 10 % of the time. With many agents in flight the first token comes several times sooner (numbers under
Benchmarks); --prefill-share-min 0.5 restores the fixed half split. 27B with all decode seats
taken and more requests open than seats: two seats (Flash-Next on 3 cards: four, whose prompts share prefill passes)
stay for prompts; a reply prefilled there sends its first token and is parked in host memory until a seat frees
(oldest first; --park 0 off). A request that does not
fit the KV pool lets smaller ones start first, until it has waited --backfill-wait seconds (30).
Agents. Requests are grouped by the client's session id (Claude Code metadata.user_id) and by message history
(a client without a session id: by system prompt);
the main agent thread (the conversation that receives the others' results) is recognised for the logs and
/api/stats only: the prefill queue, the shared prefill batches and memory treat every conversation alike (least
left to compute, aged by waiting time). When the pool is full, a request is admitted by pausing the most
recently admitted decoding conversations (the pool's own order) once it has waited --pause-wait seconds (3), and not
while another one is paused; they resume where they stopped. --main-agent-re
forces the main-thread label by a regex on the system prompt or tool names.
KV pool under pressure (both models). Idle conversations go to the host-RAM archive first. The archive frees memory
in one order: least recently used whole conversation first, every conversation alike (the main agent thread too), each
loss with a log line. The same order frees VRAM slots on both models; a generating conversation never leaves VRAM, and an
idle one never makes a request wait. When the running conversations still do not fit, the most recently admitted one
is paused: its KV goes to the archive and its stream pauses without an end token; it resumes exactly where it stopped
once the others keep their headroom. A reply ends early only when a single conversation is larger than the pool.
On Flash-Next every slot release, archive drop and restore from the archive prints one line (conversation id, tokens,
seconds idle, where it went and why), and /api/stats (engine.conversations, also in qetop) lists the known
conversations least recently used first, in a slot or in the archive, so the next one to leave is the first.
Speculative decoding. MTP drafts plus lookup drafts (when recent tokens repeat an earlier part of the conversation, as in file edits); on the 27B the DFlash2 block drafter for a conversation that decodes alone. Greedy and sampled outputs keep the model's distribution.
Loops. A reply whose last 600 tokens are one short fragment repeated is ended (--loop-guard-tokens). When an
agent keeps repeating earlier turns, whether one call or a short cycle of a few calls (same tool, exactly the same
arguments apart from the description of a shell call), each getting exactly the same result as before, for 3 turns
(--loop-guard-turns; calls that differ only in an id, such as a batch of deletes, and polls whose output changes are
not a loop), the next turn is generated with reasoning on even when thinking is off (at most 2048 reasoning tokens,
--loop-guard-think; the cached prompt is kept), with a short note on the tool result, a presence penalty on the
repeated reply and no lookup drafts. The reasoning comes on top of the request's max_tokens, and stop sequences apply
only to the answer. A guarded turn thinks even when the client disabled thinking (enable_thinking: false, thinking: {"type": "disabled"},
thinking_budget: 0); --loop-guard-think 0 turns this off. A long stretch of unchanged results can be legitimate waiting, so the guard backs off: it applies at 3,
6 and 12 turns, then every 24 turns (also for a client that sends only its last turns). Right after a user message it only turns reasoning on (harness text such as
<system-reminder> or <task-notification> blocks is not a user message). The note is left out when it would push the prompt over the context.
Conversations without tool results are never flagged. "loop_guard": false in a request turns it off.
Turn end. A generated <|im_start|> ends the reply like the end-of-turn token, so a reply never continues into a
turn the model made up. The log line (stopping at a generated <|im_start|> ...) shows the first and last 16 tokens of
the reply.
Template markers in content. Marker strings that a conversation only quotes (<|im_start|>, <|im_end|>,
<think>, <tool_call>, image placeholders, any added token of the tokenizer) are tokenized as plain text, wherever
they appear: user, system, assistant and tool messages, reasoning, tool-call names and arguments and tool
definitions, also a partial marker at the end of a name that the template would complete. Control tokens appear only
where the chat template puts them, so a conversation that pastes a chat template or server source does not split into
made-up turns, and the model does not copy a real end-of-turn token out of it. One exception follows the template's own
format: a user message that consists only of <tool_response>...</tool_response> blocks (tool results sent as a user
message) keeps the tags between and around the blocks. On the output side only the real <think>, </think> and
<tool_call> tokens switch between reasoning, answer and tool call; the same strings written as text stream as part of
the reply, and a tool-call argument that quotes </parameter> or </function> is not cut there. Conversations
without such strings in their content are tokenized exactly as before.
Failure handling.
- A stalled GPU or driver: an engine step that has not finished after
FLASH_RUN_TIMEOUT_S(120 s, Flash-Next), or no scheduler progress forQE_HANG_S(180 s, both models), fails the open requests with an error at once, refuses new ones, writes bounded diagnostics (stacks,nvidia-smi;QE_HANG_DIAG_S, 30 s) to the log and exits with code 75 so Docker (--restart) or systemd restarts it. The step monitor is off underncu,nsys,compute-sanitizerand a debugger.kill -USR1 <pid>writes the Python stacks at any time. - A prompt the engine cannot start fails at once with the reason: an invalid or too-long prompt gets HTTP 400
(
invalid_request_error), so clients do not retry it unchanged. Only "no room yet" is retried, while other conversations run, for up toQE_START_RETRY_S(60 s). - A request that produces no tokens fails with an error instead of waiting forever.
- Flash-Next's output guard fails a request whose forward produced non-finite values, instead of emitting garbage; the server goes on.
- Flash-Next keeps VRAM for its CUDA graphs (
FLASH_GRAPH_MB, 256 MB per card, next to the expert cache) and frees its least recently used graphs to stay within it, prompt-processing graphs first (/api/stats->graphsshows the use per card). A prompt-shaped graph is kept only when its shape came back within the last 64 prompt steps, and it never pushes out the decode graphs (it runs without a graph instead), so many different prompt lengths do not slow decoding down. If VRAM still runs short for a new graph, it gives up a small reserve, then drops cached graphs, then runs the step without a graph; it does not stop. - Both engines keep serving when another process (a browser, the desktop, another application) takes GPU memory
while they run, and take the memory back once it is free again (log lines
VRAM pressure/VRAM free again): Flash-Next moves its least-used resident expert groups to host memory (they are read from there, as the other host-resident experts; with the pipeline ofFLASH_PAR=pp_prefill, between pipelined prompts); 27B shrinks its KV cache (idle conversations go to the host-RAM archive; a reply that needs more room pauses with its KV in the archive, or waits, until the memory is free again; afterQE_PAUSE_TIMEOUT_S(900 s) of waiting for memory another process holds, it ends with an error that says so; a queued request that fits nowhere with nothing running ends afterQE_NO_ROOM_S(10 s) with such an error, and the server keeps serving).FLASH_VRAM_LOW_MB,FLASH_VRAM_TARGET_MB,QE_VRAM_LOW_MB,QE_VRAM_TARGET_MBset the marks,FLASH_VRAM_ADAPT=0/QE_VRAM_ADAPT=0turn it off. Measured: an application taking 1, 2 and 4 GB of one card during four streams and a 120K-token prompt, then freeing it (Flash-Next on 2 cards, also in the pipeline mode; 27B on 3 cards): every request served, speed back after the memory was freed. - A start that finds the GPUs still held by a previous engine waits up to
QE_GPU_LOCK_WAIT_S(60 s), then names the processes holding them.
Logs.
- The whole output (engine messages and tracebacks too) is appended to
~/.cache/qengine/logs/<served-name>.logwith timestamps, survives crashes and restarts, and ends with the exit status and the kernel's lines about it (OOM kill, GPU Xid). Rotated at 64 MB (--log-mb);--log-filemoves it,--log-file ''turns it off. - After every request: one line in vLLM's
SpecDecoding metricsformat and llama.cpp'sprompt eval time/eval timeformat (QE_STATS_LOG=0turns them off). WARNING empty reply: a generated turn with no text and no tool call, with the slot, conversation, finish reason, thinking tokens, themax_tokens/ budget that applied, whether</think>came, the last thinking tokens, and whether a retry follows; one more line reports the retry's outcome.history: conversation N: the client's history has N assistant turn(s) with reasoning only; left out of the prompt(Flash-Next): once per conversation and again only when the count grows. Such turns make the model end later turns empty, so they are left out (--drop-empty-turns 0keeps them). This is not an error./api/statscounts thinking budgets, forced closes, empty replies and retries, history drops, and shows the parallel mode and the CUDA-graph memory per card.
Monitoring. Live dashboard and Prometheus metrics on http://127.0.0.1:9393/ (--metrics-port), terminal
monitor server/qetop.py. Both show the running version and build in the header (e.g. QEngine 1.3.6 (build abc1234)).
The defaults are the best configuration I measured; most settings exist to fit a machine or to turn a feature off. Environment variables are read at start; an empty value counts as unset. A value per card is a comma list in GPU order (a short list repeats its last value). The long tail is in docs/advanced.md.
| setting | default | what it does |
|---|---|---|
server/run.sh <port> |
- | starts the server on 127.0.0.1:<port>; extra arguments go to server/app.py |
--model <dir> (Docker QE_MODEL) |
models/Qwen3.8-27B-INT4 |
the model directory; the engine is picked from its config.json |
QE_DEVICES (Docker: QE_DEVICES, else QE_GPUS) |
all visible GPUs (Docker 0,1,2) |
physical GPU ids, e.g. 0,2 |
--max-ctx |
262144 |
context per conversation (256K, 512K, 1M accepted); above 262,144 turns on YaRN RoPE scaling (factor context / 262,144; Flash-Next also needs FLASH_KV=ram; 27B: up to what the VRAM pool holds, 1,232,896 tokens on 3 cards and 623,616 on 2 with the defaults (16 slots; with QE_MAX_SEQS=32 1,075,200 and 417,792); QE_KV_BITS=4 raises it) |
--lsa / QE_LSA=auto |
off | 27B: sparse attention for long prompts (opt-in, lossy): 512K prompts about 2x faster, 1M about 3x, decode at 1M 2x. The released index map (Prefrontal AI, Apache-2.0) is downloaded on the first start into ~/.cache/qengine/lsa (Docker: the qengine-cache volume); offline without it the server runs with full attention. --lsa MAP / QE_LSA=MAP takes a map file (docs) |
--think |
1 |
thinking on by default (a request's chat_template_kwargs.enable_thinking, or a top-level enable_thinking, wins) |
--reasoning-effort / QE_REASONING_EFFORT |
the template's (xhigh) |
default effort: xhigh, high / max (= xhigh), medium, low, minimal (= low), none (thinking off) |
--thinking-budget / QE_THINKING_BUDGET |
0 (off) |
default thinking budget in tokens, or one per effort (low:4096,medium:16384,xhigh:32768[,default:N]) |
QE_THINK_TAIL |
auto |
room kept for the answer: 1/8 of max_tokens, 64 to 2048 tokens; a number; 0 = off |
--empty-retry / QE_EMPTY_RETRY |
2 |
resample a non-streamed empty reply up to N times; 0 = off |
--vision |
0 |
1: image input (Docker: the vision image) |
--temperature, --top-p, --top-k, --min-p |
model card, per mode: 1.0, 0.95, 20, 0 thinking; 0.7, 0.80, 20, 0 non-thinking |
sampling defaults for requests that send none; a set value applies to both modes |
--presence-penalty / QE_PRESENCE_PENALTY |
model card, per mode: 0 thinking, 1.5 non-thinking |
presence penalty for sampled requests that send none; a set value applies to both modes |
--frequency-penalty / QE_FREQUENCY_PENALTY |
0 |
frequency penalty for sampled requests that send none |
--served-name |
per model | the model id in the API |
--host, --port, --metrics-port |
127.0.0.1, 8080, 9393 |
where the API and the monitoring listen (--metrics-port 0 = off) |
QE_HANG_S |
180 |
server watchdog in seconds; 0 = off |
QE_SSE_KEEPALIVE |
10 |
streamed replies: a keep-alive after this many seconds without an event (OpenAI : keep-alive comment, Anthropic ping event), e.g. while a reply is queued or waits for its turn; 0 = off |
--log-file, --log-mb |
~/.cache/qengine/logs/<name>.log, 64 |
persistent log |
QE_STEERING (Docker) |
0 |
1: the included steering vector is the default; <name>: that vector |
| setting | default | what it does |
|---|---|---|
FLASH_PAR |
auto |
auto, tp, pp, pptp (experimental); see Parallel mode |
FLASH_PAR_AUTO_P |
8192 |
prompt length auto weighs; 32768 favours long-prompt work |
FLASH_COMM |
auto |
prefill transfer path: auto, p2p, ce, host (also pull / push) |
QE_DECODE_COMM |
auto |
decode collectives: auto (timed on real decode steps at start-up, cached; keeps P2P or the host path per mode), remeasure, p2p (SM stores into the peers), host (the no-peer host path, the QE_P2P=0 kernels, with peer access kept for prefill); also for the 27B |
QE_COMM_CACHE |
1 |
start-up cache of the link probe and decode calibration: 0 off, refresh measure again |
FLASH_MAX_SEQS |
16 |
conversations served at once (1 to 32; pipeline mode pp at most 16); fewer leave more VRAM for the expert cache. More than 16 step in groups of 16 and page more conversations through the host archive: on 3 cards 32 gave less total output than 16 with parking (--park) |
FLASH_EC_MARGIN_MB |
256 |
VRAM per card left free next to the expert cache; e.g. 256,2048,256 when GPU1 drives a desktop |
FLASH_GRAPH_MB |
256 |
VRAM per card for cached CUDA graphs: 128 MB of it (at most half of FLASH_EC_MARGIN_MB) comes out of that margin, the rest is kept on top; 0 = graphs share the margin (1.3.4) |
FLASH_PFGRAPH_WINDOW |
64 |
a prompt-shaped graph is captured only when its shape was used within the last N prompt steps, and never evicts a decode graph; 0 = off |
FLASH_ATTN_DMIN |
16 |
decode attention of batched steps (more than one wave of blocks): a row uses one key split per N key tiles (at most 8), so short contexts do not pay for 8 splits; 0 = always 8 splits |
FLASH_KV |
vram |
ram: KV cache in pinned host RAM |
FLASH_KV_RAM_TOKENS |
1M |
with FLASH_KV=ram: KV tokens of all conversations together (K / M / G); the host-RAM plan lowers it when the host is short (at least one window) |
FLASH_KV_BITS |
8 |
4: 4-bit KV cache (opt-in, not lossless against FP8): 7,488 instead of 13,520 bytes per token, about 1.8x the tokens with FLASH_KV=ram; see Host memory |
FLASH_KV_ARCHIVE_GB |
48 (less on smaller machines) |
host-RAM archive of conversations; 0 = off, which on Flash-Next also turns off the pinned prompts and prefix reuse across slots, with FLASH_KV=ram too; a set value wins over the host-RAM plan. The archive's memory is pinned in the background, 1 GB at a time, only while the engine is idle and the host has at least 8 GB to spare above QE_MEM_MARGIN_GB (never next to a running step: #32); a registration slower than 1 s, which is a sign of host memory pressure, pauses pinning for 30 s, and the pause doubles each time it repeats, up to 8 min. Until the pool has grown, new archive copies use plain memory (slower copies, same result) |
FLASH_PREFIX_GB |
8 (less on smaller machines) |
host RAM for pinned system prompts; 0 = off |
FLASH_NGRAM_PIN |
set by the host-RAM plan (1 when it fits) |
1: pin the n-gram table; 0: read it through the page cache |
FLASH_WEIGHTS_COLD |
1 |
after the load, drop the page cache of the model files the engine no longer reads (all but the n-gram table in page-cache mode); 0 = keep it, and also turns off the load-time drop (QE_LOAD_DROP) |
FLASH_TP_SPIN_US |
5000 |
µs a tensor-parallel helper thread spins after its last job before it sleeps (an idle server then uses no CPU); -1 = always spin |
FLASH_VRAM_ADAPT |
1 |
Flash-Next: when another process takes GPU memory while serving, move resident expert groups to host memory and back later; 0 off |
FLASH_VRAM_LOW_MB / FLASH_VRAM_TARGET_MB |
128 / 256 |
free memory per card below which groups move to host memory / up to which they move; they come back as soon as the other processes give the memory back (keeping 16 MB above the low mark free) |
FLASH_CTX_FIT |
1 |
Flash-Next: lower the context at start to the largest that fits the smallest card (logged); 0 keeps --max-ctx as given |
FLASH_PRELOAD |
1 |
Flash-Next: when CUDA was started before the engine (the vision encoder) with lazy module loading, load the engine's kernels at start instead of at their first use; 0 off |
QE_RAM_WAIT_S |
20 |
seconds the start waits for free host RAM (MemAvailable) when it is short of the full plan (as long as it rises, or while the previous QEngine server on these cards is still exiting with 1 GB or more on them); 0 plans at once |
QE_NO_ROOM_S |
10 |
27B: seconds a queued request that fits nowhere waits with nothing running, then it ends with an error that says why; 0 = no deadline |
QE_PAUSE_TIMEOUT_S |
900 |
27B: seconds a reply may wait for GPU memory another process holds before it ends with an error, counted from when that wait began: one that cannot continue, or a paused one that cannot resume because the KV cache gave memory back; a reply paused only to make room for another start has no deadline |
FLASH_NGRAM_SPIN_MS |
50 |
page-cache mode of the n-gram table: ms the row-copy thread spins after the last request, then it polls every 50 µs (1 ms after a second); -1 = always spin |
QE_HOST_RESERVE_GB |
max(8, 8 % of RAM) | at start only: RAM the plan leaves to the system and the desktop (both models). QE_MEM_MARGIN_GB is the one that acts while serving |
FLASH_RUN_TIMEOUT_S |
120 |
engine step monitor in seconds; 0 = off |
| setting | default | what it does |
|---|---|---|
QE_MAX_SEQS |
16 |
conversations kept in the KV pool, up to 32; 1 to 4 on one card. More open requests than slots are parked beyond the seats (--park). 32 seats 32 at once, but leaves a smaller KV pool (2 cards: 417,792 against 623,616 tokens; one 262,144-token conversation still fits, two very long ones at once do not) |
QE_ADMIT_BY_LIMIT |
1 |
a running conversation is counted at admission with at most its reply's max_tokens + 1,024 tokens of growth (else 8,192); 0 = always 8,192 |
QE_ARCH_REPLY |
1 |
a conversation's next turn that leaves its cached tokens only inside its last reply (a client that sends another assistant text or drops the reasoning) no longer copies the whole conversation to the host archive first; 0 = copy it, as before |
QE_ARCH_SUBS |
2 |
27B: recurrent-state checkpoints an archived conversation keeps besides its end state (1 to 4; each is about 151 MB of host RAM). With many agents at once, 1 lets the host archive hold about a third more conversations; the second one is what an edited or retried last turn resumes from |
QE_ARCH_DEMOTE |
1 |
27B: an archived conversation whose next turn replaced its last reply (a client that sends another assistant text) goes first when the archive needs room, instead of being kept as recently used; 0 = keep it as before |
QE_SNAP_DEMOTE |
1 |
27B: when a prompt-end state is saved, the same slot's older prompt-end states of that conversation (its earlier turns) go first when the state store needs room, instead of aging like the live ones; with many conversations per slot count a waiting turn no longer loses its own state. The resume point of a repeated prompt and of a fork is kept. 0 = plain least-recently-used order, as before |
QE_DF_BATCH |
2 on one card, 0 on more |
batched steps of up to this many conversations draft with DFlash2 (one context ring each, 43 MB per card); 0: batched steps draft with MTP |
QE_DRAFTER |
auto |
drafter of a conversation that decodes alone: auto loads DFlash2 when the requested context and slots fit with it (logged, with the --max-ctx that would fit; one card with the default --max-ctx stays on MTP), dflash2 always (context shrinks to fit), mtp never (no download) |
QE_DFLASH |
– | path to a local DFlash2 checkpoint (directory or file) instead of the download |
QE_KV_BITS |
8 |
4: 4-bit KV cache (opt-in, not lossless against FP8): about 1.5x the context on small cards; --lsa is then off (logged); see Smaller cards |
QE_HEAD4R |
16 |
27B verify head: INT4 screen + exact BF16 rescore of the top N per row (pinned host RAM); 0 = INT8 head |
QE_RESERVE_GB |
1.5 |
VRAM per card left outside the KV pool; e.g. 1.5,3,1.5 for a desktop on GPU1 |
QE_KV_ARCHIVE_GB |
32 (less below 64 GB of RAM) |
host-RAM archive of conversations; 0 = off; a set value wins |
QE_PREFIX_GB |
8 (less below 64 GB of RAM) |
host RAM for pinned system prompts |
QE_COMM |
auto |
prefill transfer path: auto, p2p, ce, host |
QE_FETCH_ASSETS |
1 |
0: no downloads at start: not the tuned MTP head and draft ranking, not the DFlash2 drafter (MTP drafts only), not the --lsa index map (full attention) |
--disk-cache DIR |
off | persistent prompt cache that survives restarts (Flash-Next: experimental, since 1.3.5.3) |
The common settings above apply (--max-ctx, --think, --reasoning-effort, --thinking-budget, --temperature,
--top-p, --served-name, --host, --port, --metrics-port); presence / frequency penalties, --vision and
steering do not. The full list is in docs/advanced.md.
| setting | default | what it does |
|---|---|---|
--model <dir> (Docker QE_MODEL) |
– | the Maya-S directory of the GGUF files; the DSpark drafter is read from <dir>/../dspark |
--max-ctx |
262144 |
context per conversation; with P2P the long-context memory is taken from the expert pool only while needed (short chats as at 32K); without it, it is reserved at start (halved, down to 32,768, when it would leave less than 8 GiB free on a card; logged); a start-up line says which; an exp 10BC8 licit value wins |
--archive-gib |
32 |
host RAM for idle conversations (least recently used out first; it shrinks when the host runs short); 0 = off |
--max-active |
8 |
requests in service at once (they take turns); more wait in the queue |
--route-margin |
off | opt-in, approximates the routing: when the router picks an expert that is not in VRAM and a VRAM-resident expert scored within this margin, the resident one runs; 0.05 is the tested value (decode 7-20 % faster on three prompts, negative log-likelihood 1.641 against 1.623 on my gate texts); logged at start |
--tier-reset |
off | opt-in: after each request the expert placement goes back to its state after loading, so repeating a greedy request gives the same text; serves one request at a time and turns prefix sharing off; costs decode speed |
--no-prefix-share |
off | off-switch: no shared copies of system prompts |
--no-dspark |
off | off-switch: decode without the draft model (much slower) |
--engine-args "<flags>" |
– | engine flags, e.g. --cpu-threads N (default 24), --reserve-gb X (VRAM left free per card, default 0.6) |
DSV4_EGMIRROR |
off | faster long prompts on btrfs / XFS through reflink copies of the two Engram shards (docs) |
The common settings above apply (--max-ctx, --devices / QE_DEVICES, --reasoning-effort, --thinking-budget,
--temperature, --top-p, --served-name, --host, --port, --metrics-port); presence / frequency penalties,
--vision, --disk-cache and steering do not. The engine's own switches are environment variables; a few more are in
docs/advanced.md.
| setting | default | what it does |
|---|---|---|
--model <dir> (Docker QE_MODEL) |
– | the bpx03/GLM-5.3-Flash-INT4-QEngine directory |
--max-ctx / GLM_MAX_CTX |
262144 |
context; the engine lowers it with a warning when its attention cache does not fit (GLM_MAX_CTX caps it). The cache of a 262K context leaves room for 32K-row prefill windows instead of 64K: if your prompts stay under 64K, --max-ctx 65536 prefills prompts above 32K about 6 % faster |
--devices / QE_DEVICES |
every visible card | the cards: two or three (of more, the first three) |
--split |
704,640,704 |
routed expert columns per card, multiples of 64 summing to 2048; by default the 640 goes to the card with the least free VRAM |
GLM_PF_SLICE_MS |
1000 |
host slice of a prefill in ms (progress and cancel between slices); 0 = the whole prompt in one call |
GLM_CONVS |
8 |
conversations served at once (time-sliced, exact); 1 = one at a time, the others wait in the queue |
GLM_SLICE_MS / GLM_SLICE_X |
2000 / 8 |
a reply's turn while others wait: at least this many ms and this many times the measured switch cost (park + resume) |
GLM_PREEMPT_MS |
the slice | how long the running reply decodes before a new prompt's prefill parks it |
GLM_PF_CANCEL_MID |
1 |
0 = a cancel waits for the next prefill window start (the older rule) |
GLM_CPU_ISA |
auto |
CPU expert lane kernels: auto (AVX-512 when present), avx2, avx512 |
GLM_LANE_CPUS / GLM_CPU_THREADS |
from the CPU topology | the CPU lane's cores (a list of ranges) / its thread count, instead of the derived placement |
GLM_P2P |
1 |
0 = off-switch: the all-reduces go through host memory even when P2P is there |
The common settings above apply (--max-ctx, --reasoning-effort, --thinking-budget, --temperature, --top-p,
--served-name, --host, --port, --metrics-port, --presence-penalty, --frequency-penalty); --vision,
--disk-cache and steering do not. The engine's own switches are environment variables; the full list is in
docs/advanced.md.
| setting | default | what it does |
|---|---|---|
--model <dir> (Docker QE_MODEL) |
– | the openai/gpt-oss-120b directory |
--max-ctx / GPTOSS_MAX_CTX |
131072 |
context per conversation (the model's limit; GPTOSS_MAX_CTX caps it) |
GPTOSS_SLOTS |
8 (context per conversation >= 32,768), else 4 |
conversations decoded at once (1-16) |
GPTOSS_POOL_ROWS |
every row that fits | K/V rows shared by all slots; a number pins it (never below one conversation) |
GPTOSS_POOL_RESERVE_MB |
320 |
VRAM per card left free beside the pool (decode graphs, other processes) |
GPTOSS_KV |
bf16 |
opt-in k12 (12-bit K, 1.56x the rows) or fp8 (8-bit K and V, 1.94x the rows): lossy, for many agents with long contexts (KL to the bf16 store at 32K / 64K context: k12 1.2e-3 / 2.2e-3, fp8 2.9e-3 / 2.3e-3) |
GPTOSS_ADMIT_REPLY_ROWS |
2048 |
rows a new prompt leaves free for the running replies to grow (at most 1/16 of the pool) |
GPTOSS_LOOKUP |
9 |
rows per step of the lookup drafts on greedy replies (a token and up to 8 drafts); 0 = off-switch |
GPTOSS_COMM |
auto | host forces the all-reduce through host memory, p2p requires P2P |
GPTOSS_CHUNK |
10240 |
prefill chunk in tokens |
GPTOSS_ARCHIVE_SYNC_ROWS |
all | an evicted conversation goes to host RAM whatever its size; a number bounds the copy |
QE_KV_ARCHIVE_GB / QE_KV_ARCHIVE_MARGIN_GB |
128 / 8 |
host RAM for idle conversations / what it leaves free below MemAvailable |
GPTOSS_BROKEN_EXIT |
1 |
after a device fault the process exits with code 75 (for a restart); 0 = off-switch (keeps the failed process for debugging) |
All numbers were measured by me on one machine: 3× RTX 3090 (24 GB, PCIe 4.0 x16, P2P through the patched driver unless noted), one NUMA node; "2 cards" are two of its cards. QEngine runs with its defaults. They describe that machine; your numbers depend on the cards, links, driver and host.
Since 1.3.6 the server counts the end token a reply generates in usage (see
Thinking budget and the answer tail), and benchmark clients take their token
counts from it, so 1.3.6 numbers include one more token per reply that ends on its own than the older numbers below.
1.5.0 (November 19, 2026; 3 cards, P2P, the club-3090 rebench-full.sh setup: 8 slots, thinking on, effort low,
vision on): single-stream decode narrative / code 310 / 452 tok/s, prefill 10K / 90K 8,254 / 8,390 tok/s (first token
1.23 / 10.5 s), decode at 10K / 90K depth 350 / 306 tok/s, 1 / 2 / 4 requests at once 302 / 421 / 647 tok/s total, soak
median decode 402.7 tok/s, verify-stress 8/8, soak PASS. Concurrency probe (16 slots, greedy, 256-token prompts and
replies): 300 / 642 / 880 / 1,061 tok/s total at 1 / 4 / 8 / 16 streams.
1.3.6 (November 15, 2026; 3 cards, P2P, the club-3090 rebench-full.sh setup: 8 slots, thinking on, effort low,
vision on): single-stream decode narrative / code 305.6 / 449.3 tok/s, prefill 10K / 90K 8,207 / 8,394 tok/s, decode
at 10K / 90K depth 315 / 296 tok/s, 1 / 2 / 4 requests at once 305 / 422 / 653 tok/s total, soak median decode 411.4
tok/s, verify-stress 8/8, soak PASS. The club bench sends no presence penalty, so the server's non-thinking default
1.5 applies; with QE_PRESENCE_PENALTY=0 (the default before 1.3.5.3) the same run gives 325.0 / 444.4 tok/s
single-stream. Plain defaults (server/run.sh, 16 slots): 311 / 454 tok/s single-stream, prefill 8,057 / 8,372, 8
agents at once 760 tok/s (mean of 7 runs, 709-828; real-text prompts 976-993). 2 cards (0, 2), plain defaults: 201 /
226 tok/s single-stream, prefill 4,265 / 7,134, 8 agents at once 198 tok/s (224-256 with presence 0); a 262K-token
prompt with image traffic passes on both.
Many agents in flight (1.5.0 code, November 19, 2026; 3 cards, P2P, defaults; the 27B's agent replay below
with 64-token replies: bench/q27_ttft_conc.py --mix light --gen 64, warm-up 20 s (64 agents: 60 s), 60 s window,
one run per point). Median / 95th percentile time to first token, total output:
| agents in flight | 1 | 4 | 16 | 32 | 64 |
|---|---|---|---|---|---|
| before (1.3.6 scheduling, same day) | 0.16 / 0.46 s, 131 tok/s | 0.44 / 1.42 s, 154 | 1.91 / 7.2 s, 165 | 3.93 / 9.4 s, 154 | 7.1 / 33 s, 144 |
| prompt-end state + queue-aware split | 0.13 / 0.38 s, 150 tok/s | 0.25 / 0.67 s, 175 | 0.37 / 1.46 s, 191 | 2.83 / 3.54 s, 170 | 8.0 / 9.0 s, 157 |
| 1.5.0 (+ parking beyond the seats) | 0.13 / 0.38 s, 149 tok/s | 0.25 / 0.65 s, 169 | 0.36 / 1.23 s, 197 | 0.26 / 0.79 s, 179 | 0.75 / 2.0 s, 171 |
At 32 and 64 agents the 16 decode seats are full. Without parking a prompt got a seat only when a reply ended; now
it is prefilled on one of four seats kept for prompts, sends its first token and waits parked in host memory (greedy
replies resume bit-identical). At 64 parked replies may hold up to 3/4 of the host archive (FLASH_KV_ARCHIVE_GB,
48 GB): past that a reply waits for a seat as before, so a larger archive parks more. Up to 32 seats can be set
(FLASH_MAX_SEQS); on 3 cards 32 seats gave less total output than 16 with parking.
1.3.5 (November 6, 2026; 3 cards, P2P, the club-3090 rebench-full.sh setup: 8 slots, thinking on, effort low, vision on): single-stream decode narrative / code 313.6 / 404.8 tok/s, prefill 10K / 90K 6,128 / 7,515 tok/s, decode at 10K / 90K depth 350 / 290 tok/s (one reply each), 1 / 2 / 4 requests at once 293 / 430 / 651 tok/s total, soak median decode 372.4 tok/s, verify-stress 8/8, soak PASS. At the plain defaults, thinking off: 3 cards 312.9 / 410.7 tok/s, prefill 6,186 / 7,249, 8 agents at once (vllm bench serve, 500 tokens in, 2,000 out) 867 tok/s, a noisy point, other runs 935 and 956. 2 cards (0, 2): 208.7 / 220.7 tok/s, prefill 3,797 / 6,357, 8 agents at once 285 tok/s, a 262K-token prompt and image traffic pass.
1.3.4.2 (November 2, 2026; 3 cards, P2P, the club-3090 rebench-full.sh setup: 8 slots, thinking on, effort low,
vision on; in brackets 1.3.4.1 on the same day with the same desktop load on GPU1): single-stream decode narrative /
code 314.3 / 401.4 tok/s (292.4 / 368.4), prefill 10K / 90K 6,137 / 6,877 tok/s (6,061 / 6,675), decode at 10K / 90K
depth 328 / 307 tok/s (294 / 275), 1 / 2 / 4 requests at once 288 / 414 / 581 tok/s total (273 / 382 / 555), soak
median decode 359.9 tok/s (333.9), 8 agents at once (vllm bench serve, ~500-token real-text prompts, 2,000 output
tokens) 1,022 tok/s (897), verify-stress 8/8, soak PASS. 2 cards (0, 2): 8 agents at once 359 tok/s, a 262K-token
prompt and image traffic pass.
1.3.4.1 (November 2, 2026; 3 cards, P2P, the club-3090 rebench-full.sh setup: 8 slots, thinking on, effort low,
vision on; in brackets the 1.3.4 weights on the same day with the same desktop load on GPU1): single-stream decode
narrative / code 292.4 / 368.4 tok/s (CV 2.7 / 2.2 %; 300.0 / 389.9), prefill 10K / 90K 6,061 / 6,675 tok/s (5,940 /
6,749), decode at 10K / 90K depth 294 / 275 tok/s (318 / 282), 1 / 2 / 4 requests at once 273 / 382 / 555 tok/s total
(293 / 407 / 566), 8 agents at once (vllm bench serve, ~500-token real-text prompts, 2,000 output tokens) 897 tok/s
(944 with the 1.3.4 weights, same day), verify-stress 8/8, soak PASS. flash_run: 10K / 90K-token prefill 1.561 /
14.995 s (1.554 / 14.96 s). 2 cards (0, 2): 8 agents at once 370 tok/s, a 262K-token prompt and image traffic pass.
Measured during the 1.3.1 release validation (October 30, 2026). Single-stream decode and prefill: club-3090
scripts/bench.sh (thinking off); agents: vllm bench serve as the client, ~500-token real-text prompts, 2,000
output tokens; long conversations: 4 conversations of ~64K tokens, then rounds where all 4 send a follow-up at once.
| tok/s, higher is better | 3 cards | 2 cards |
|---|---|---|
| single-stream decode, narrative | 307 | 210 |
| single-stream decode, code | 388 | 248 |
| prefill, 10K-token prompt | 5,960 | 3,760 |
| prefill, 90K-token prompt | 6,410 | 5,179 |
| agents, 8 at once (total output) | 941 | 390 |
| agents, 16 at once (total output) | 1,104 | 468 |
| 4 × 64K conversations, follow-up rounds (output) | 455 (FLASH_KV=ram: 580) |
233 (FLASH_KV=ram: 332) |
1.3.4 speed check against 1.3.3 (3 cards, P2P, flash_run, November 1, 2026): 90K-token prefill 13.01 s (1.3.3:
13.10 s); decode on a code prompt 343.8 tok/s (deterministic mode 367.6 tok/s; 1.3.3: 365.5).
The parallel modes and the no-P2P path (1.3.4, November 1, 2026) are measured in Parallel mode and With and without GPU P2P. Burst prefill, slow links and the KV-in-RAM runs: docs/advanced.md.
1.5.0 (November 19, 2026; 3 cards, P2P, defaults, DFlash2 loaded, the club-3090 bench.sh settings: thinking on,
effort low, vision on): single-stream decode narrative / code 275 / 524 tok/s, prefill 10K / 90K 6,188 / 4,994 tok/s
(first token 1.60 / 18.0 s); concurrency probe (greedy, 256-token prompts and replies): 312 / 575 / 848 / 999 tok/s
total at 1 / 4 / 8 / 16 streams.
1.3.6 (November 15, 2026; 3 cards, P2P, defaults, DFlash2 loaded): single-stream decode narrative / code 278.7 /
515.5 tok/s, prefill 10K / 90K 6,309 / 5,113 tok/s (5,769 / 4,784 right after other runs), 8 agents at
once 1,129 tok/s (1,210 with QE_PRESENCE_PENALTY=0). With the club-3090 bench.sh settings: 281.2 / 507.0 tok/s,
prefill 6,133 / 4,997. Short single replies (385 tokens, greedy: a Python function, a short story, a proof): 474 / 264
/ 401 tok/s on 3 cards; 212 / 150 / 196 tok/s on one card (QE_MAX_SEQS=1 --max-ctx 150K, DFlash2).
Many agents in flight (1.5.0 code, November 18, 2026; closed loop, no think time: each agent sends its next
turn when the reply ends; a shared 2K-token system prompt, a first message of 1-3K tokens, then tool results of
30-2,000 tokens (about 650 new tokens per turn) up to 16K tokens of context; 128-token replies, greedy, thinking off;
bench/q27_ttft_conc.py --mix light --gen 128, one run per point). Median time to first token / total output:
| agents in flight | 1 | 4 | 16 | 32 | 64 |
|---|---|---|---|---|---|
| 2 cards, 1.3.6 (16 slots, fixed half split) | 0.18 s / 140 tok/s | 0.64 s / 157 | 4.5 s / 231 | 25.0 s / 159 | 73.0 s / 111 |
| 2 cards, 1.5.0 defaults (16 seats, parking beyond them; release build, November 20) | – | – | 1.04 s / 243 tok/s | 0.48 s / 359 | 1.0 s / 285 (after a 90 s warm-up) |
2 cards, 1.5.0, QE_MAX_SEQS=32 |
0.19 s / 142 tok/s | 0.33 s / 173 | 0.86 s / 248 | 0.32 s / 361 | 5.0 s / 155 |
3 cards, 1.5.0, QE_MAX_SEQS=32 |
0.13 s / 193 tok/s | – | 1.31 s / 258 | 0.66 s / 403 | 3.3 s / 242 |
At 64 agents the measured window (after 15 s) still holds the 64 agents' first turns (about 140K new tokens, the
queue is prefill-bound). After a 90 s warm-up, at 64 agents: 2 cards 30.3 s / 147 tok/s without parking and with four checkpoints per archive,
0.27 s / 318 tok/s with 1.5.0 (QE_MAX_SEQS=32); 3 cards 1.45 s / 305 tok/s, where past ~10K tokens per agent the default 32 GB host
archive no longer holds all 64 conversations (QE_ARCH_SUBS=1: 1.06 s / 317; a larger QE_KV_ARCHIVE_GB removes
the limit).
1.3.5 (November 6, 2026; 3 cards, P2P): single-stream decode narrative / code 274.1 / 422.5 tok/s, prefill 10K / 90K 6,437 / 5,159 tok/s, 8 agents at once 1,205 tok/s. With the club-3090 bench.sh settings: 276.1 / 418.9 tok/s. Prefill measured on its own.
Serving benchmark (October 2026, QEngine 1.0), 3 cards, vllm bench serve as the client; output tokens/s:
| scenario (input / output tokens) | 1 at once | 8 | 16 | 32 |
|---|---|---|---|---|
| short (1024 / 512) | 208 | 794 | 949 | 1,001 |
| agent (500 / 2000) | 287 | 1,080 | 1,302 | 1,423 |
| ShareGPT | 243 | 819 | 935 | 1,110 |
Later changes (W4A8 GEMM for batched passes of 33 rows or more, less admission headroom) raised agent at 32 to 1,703 and short at 32 to 1,039 tok/s in a same-day A/B. Prefill of one new prompt (median time to first token): 32K tokens 5.35 s, 200K tokens 50.7 s.
1.5.0 (November 19, 2026; 3 cards, P2P, defaults; club-3090 tools): single-stream decode narrative / code 88 /
120 tok/s, prefill 10K / 90K 4,217 / 4,895 tok/s (first token 2.35 / 17.8 s), 70 / 73 / 78 / 78 tok/s total at 1 / 4
/ 8 / 16 streams (requests take turns; --max-active 16 for 16). Quality: negative log-likelihood 1.635 on my gate
texts (no full-precision reference fits on this machine); 8-pack under Quality.
More points (November 20, 2026, release build; 3 cards, defaults unless noted, greedy, DSpark on; one run each):
| DeepSeek-V4.1-Flash, 3x RTX 3090 | P2P | no P2P |
|---|---|---|
| single-stream decode, engine, three prompts: git log / code / narrative (tok/s) | 104.0 / 105.5 / 76.3 | 99.8 / 97.9 / 69.7 |
same, --route-margin 0.05 |
124.7 / 119.3 / 81.9 | – |
| prefill, 2.5K-token prompt first / second time, 10K-token prompt (s) | 1.44 / 1.28, 2.87 | – |
| prefill, 117K / 233K-token prompt, server (s) | 27.0 / 56.2 | – |
| decode with 117K / 233K tokens of context, server (tok/s) | 103.0 / 91.2 | – |
| 4 agents at once, unrelated prompts, server (total output tok/s; slowest first token) | 65.5; 5.2 s | – |
| 4 / 8 agents at once, shared 7K system prompt, server (total output tok/s; slowest first token) | 99.6 / 97.6; 2.3 / 10.2 s | – |
| ready after start, files in the page cache (s) | 40.9 | – |
The no-P2P column ran with the default 262,144 context, so its reservation is static (DSV4_NOP2P=1 on the same
machine).
1.5.0 (November 19, 2026; 3 cards, P2P, defaults; club-3090 tools): single-stream decode narrative / code 110 /
92 tok/s, prefill 10K / 90K 2,920 / 3,572 tok/s (first token 3.42 / 25.4 s), 72 / 55 / 50 / 44 tok/s total at 1 / 4 /
8 / 16 streams (replies take turns; GLM_CONVS=16 for 16). Server, one conversation: prefill of an 8K / 32K /
143K-token prompt 2,996 / 3,443 / 3,917 tok/s, decode at 100K context 94.9 tok/s. Quality: next-token KL against the
official FP8 model 0.448 / 0.541 / 0.772 on three prompts (a README, a git log, an agent turn), argmax agreement 79.4
/ 74.6 / 69.0 %; 8-pack under Quality. Ready after start: about 60 s with the files in the page cache.
1.5.0 (November 19, 2026; 3 cards, P2P, defaults; club-3090 tools): single-stream decode narrative / code 301 /
301 tok/s, prefill 10K / 90K 13,280 / 5,055 tok/s (first token 0.74 / 17.6 s), 306 / 594 / 731 / 835 tok/s total at 1
/ 4 / 8 / 16 streams (GPTOSS_SLOTS=16 for 16).
More points (November 20, 2026, release build; 3 cards, defaults, greedy, engine tool, mean of two runs; the no-P2P
column with GPTOSS_COMM=host):
| gpt-oss-120B, 3x RTX 3090 | P2P | no P2P |
|---|---|---|
| single-stream decode, short context (tok/s) | 337 | 321 |
| 8 conversations at once (total output tok/s) | 1,013 | 952 |
| prefill, 10K-token prompt (tok/s) | 13,120 | 12,310 |
| ready after start, server, files in the page cache (s) | 10.6 | – |
In the release stress run a 129K-token prompt with all 8 slots busy found its needle.
Quality (1.5.0 build): every layer's output agrees with an fp32 reference of the model within the bar (worst median row error 3.96e-4 through the decode path, 5.24e-4 through the prefill path); next-token KL against fp32 0.118 and top-1 agreement 86.9 % on 11 items through the prefill path (0.112 through the decode path); needles at 90K: 3 of 3.
Closeness to BF16. Next-token KL divergence against the official BF16 model on a public reference set (21 cases,
32,359 positions: prose, code, Japanese, Chinese, SQL, agent turns and WikiText-2), scored with the set's own
compare.py; every position through QEngine's decode path (forwards of up to 16 tokens) on 3 cards (1.3.6 release
build, November 15, 2026, identical to 1.3.5 to every digit; the 1.3.4 row November 2). The last column scores the last 512 positions of the long cases after the context before them went through
the server's batched prefill (my own harness, not part of the command below). Lower is closer to BF16:
| Flash-Next weights | mean KL | top-1 agreement | WikiText-2 KL | KL, context by batched prefill (last 512 positions) |
|---|---|---|---|---|
| default | 0.114 | 89.6 % | 0.125 | 0.126 |
1.3.4 (--revision v1.3.4) |
0.267 | 83.3 % | 0.329 | 0.358 |
To measure it yourself (3 cards; ~32 GB download, ~65 GB of scratch space):
hf download jyohukuchan/qwen38-flash-next-bf16-logits --repo-type dataset \
--revision 5294aa9ab94b0956742fc4c6528c21c1c3c08e42 --local-dir data/kl-ref \
--include "reference/*" "corpus/*" compare.py vocab.json manifest.json
python3 tools/flash_kl_ref.py --model models/Qwen3.8-Flash-Next-QE-INT4 --ref data/kl-ref --work /path/to/scratchThe same from the Docker image (no source build; the downloaded set at /ref, the scratch space at /work):
docker run --rm --gpus all -e QE_GPUS=0,1,2 -e QE_MODEL=/app/models/Qwen3.8-Flash-Next-QE-INT4 \
-v /path/to/models:/app/models:ro -v $PWD/data/kl-ref:/ref:ro -v /path/to/scratch:/work \
ghcr.io/bpx03/qengine:latest kl --ref /ref --work /work--cases en_prose_129,python_code_513 scores only those cases (a quick check, not the numbers above); the result
is printed and written to /work/summary.json.
club-3090 8-pack (benchlocal, 150 scenarios, 4,096-token answer budget per scenario), QEngine 1.5.0 (November 19-20, 2026), one full run per model on 3 cards, each on a fresh server with its defaults (the two Qwen models with vision on and the club's launch settings, reasoning effort low). Scenarios passed, thinking off / on (gpt-oss-120B always reasons: reasoning effort low / medium):
| pack | N | Qwen3.8-27B | Qwen3.8-Flash-Next | DeepSeek-V4.1-Flash | GLM-5.3-Flash | gpt-oss-120B |
|---|---|---|---|---|---|---|
| toolcall | 15 | 12 / 13 | 13 / 14 | 12 / 13 | 13 / 13 | 11 / 13 |
| instructfollow | 15 | 11 / 15 | 13 / 14 | 12 / 15 | 9 / 15 | 14 / 14 |
| structoutput | 15 | 12 / 12 | 14 / 14 | 12 / 14 | 12 / 14 | 13 / 14 |
| dataextract | 15 | 10 / 13 | 10 / 12 | 12 / 13 | 13 / 14 | 11 / 9 |
| reasonmath | 15 | 13 / 15 | 13 / 14 | 12 / 14 | 14 / 15 | 12 / 13 |
| bugfind | 15 | 14 / 15 | 15 / 14 | 15 / 15 | 15 / 15 | 11 / 12 |
| hermesagent | 20 | 15 / 14 | 12 / 12 | 15 / 14 | 15 / 15 | 12 / 13 |
| cli | 40 | 27 / 31 | 30 / 31 | 32 / 35 | 30 / 32 | 26 / 28 |
| total | 150 | 114 / 128 | 120 / 125 | 122 / 133 | 121 / 133 | 110 / 116 |
No engine errors and no graph fallbacks in any of the runs. Earlier releases published the mean of three runs; from 1.5 on it is a single run, because running it for all models takes a long time.
The earlier result, Flash-Next on QEngine 1.3.6 (November 15, 2026), three full runs, each on a fresh server. Scenarios passed: run 1 / 2 / 3, mean pass@1, mean pass@3 (up to 3 attempts per scenario):
| pack | N | thinking off: runs | mean | pass@3 | thinking on: runs | mean | pass@3 |
|---|---|---|---|---|---|---|---|
| toolcall | 15 | 13 / 13 / 13 | 13.0 | 13.0 | 14 / 14 / 13 | 13.7 | 14.0 |
| instructfollow | 15 | 15 / 13 / 13 | 13.7 | 13.7 | 14 / 15 / 15 | 14.7 | 15.0 |
| structoutput | 15 | 13 / 14 / 14 | 13.7 | 14.0 | 13 / 13 / 13 | 13.0 | 13.7 |
| dataextract | 15 | 11 / 9 / 11 | 10.3 | 11.0 | 11 / 13 / 12 | 12.0 | 13.3 |
| reasonmath | 15 | 13 / 15 / 15 | 14.3 | 14.3 | 15 / 15 / 14 | 14.7 | 14.7 |
| bugfind | 15 | 15 / 14 / 15 | 14.7 | 15.0 | 15 / 15 / 14 | 14.7 | 15.0 |
| hermesagent | 20 | 14 / 14 / 14 | 14.0 | 16.7 | 14 / 15 / 14 | 14.3 | 17.0 |
| cli | 40 | 31 / 29 / 30 | 30.0 | 35.0 | 29 / 30 / 29 | 29.3 | 32.3 |
| total | 150 | 125 / 121 / 125 | 123.7 | 132.7 | 125 / 130 / 124 | 126.3 | 135.0 |
No engine errors, no empty replies and no graph fallbacks in the three runs. On 1.3.5 (November 6) the same suite gave 120.7 / 126.7 (thinking off / on).
AutomationBench sales (Zapier, 100 tasks, a 1.3.4 pre-release build with default settings, Flash-Next on 3 cards, 16 at once, November 1, 2026): 47 % of tasks passed, mean partial credit 0.78; the 5 zero-score tasks were wrong decisions of the model, none came from the server. The image encoder was checked against Hugging Face transformers and the INT8 n-gram table against the BF16 one (next-token KL); see docs/advanced.md.
python3 -m pip install -r requirements-dev.txt
python3 bench/serving_suite.py --engine-name qengine --base-url http://127.0.0.1:8080 \
--model qwen3.8-flash-next-qengine --tokenizer models/Qwen3.8-Flash-Next-QE-INT4 --quick--quick runs short requests at 1 at once plus two live-traffic scenarios. For the 27B use
--model qwen3.8-27b-qengine (no --tokenizer). The ShareGPT scenario needs
ShareGPT_V3_unfiltered_cleaned_split.json from
anon8231489123/ShareGPT_Vicuna_unfiltered
in bench/data/.
An agent stops early, or a benchmark scores tasks as zero.
- Look for
WARNING empty replyin the log. A reasoning-only turn ends many agent loops. Non-streamed requests are resampled (--empty-retry); streamed ones cannot be. If the line saysfinish lengthand</think> no, the reply ran out ofmax_tokenswhile thinking: set a thinking budget (--thinking-budget, or per request) or raisemax_tokens. - Check the context. Agent prompts grow large (AutomationBench sales reaches 94K tokens); a harness ends a task when a
prompt no longer fits. In a replay of AutomationBench sales (November 1, 2026), a 32K-token context cut 48 of 100
tasks and left 40 at zero, against 6 with the default 262K. Check
max_model_lenin/v1/models:--max-ctx, a smallQE_MAX_SEQSor a single card lower it, and so can a proxy in between. history: ... reasoning only; left out of the promptis informational, not an error.
No P2P (stock driver, dual-socket server, cloud box). Nothing to set: both engines detect it and use host-staged
exchanges (see With and without GPU P2P). For long-prompt work on Flash-Next try
FLASH_PAR=pp, which needs no fast link.
Long prompts fail with non-finite values, or the log shows GUARD lines. Set FLASH_COMM=ce (27B:
QE_COMM=ce) and report the [comm] start-up lines.
Slow decode, or poor scaling with concurrent requests, on a board with a high P2P co-run tax. Look at the
[comm] decode collectives start-up line: it shows the measured ms per decode step on both paths and the pick for
one and for several sequences. If the two timings are close, or your own runs disagree with the pick, compare
QE_DECODE_COMM=host and QE_DECODE_COMM=p2p at 1 and 4 concurrent requests and report the [comm] lines with
both numbers. Before 1.3.5, decode always used P2P stores when peer access worked; QE_P2P=0 was the only way off
them, and it also moves prefill to the slower host-staged path. On 1.3.5 to 1.3.5.5 the host candidate was slower than
QE_P2P=0 itself; on such a board use 1.3.5.6 or later, or QE_P2P=0 on the older versions.
not enough GPU memory for a KV cache (27B). Too many slots for the free VRAM, e.g. on one card: lower
QE_MAX_SEQS, leave --vision off.
A desktop on one of the cards stutters. Flash-Next FLASH_EC_MARGIN_MB=256,2048,256, 27B
QE_RESERVE_GB=1.5,3,1.5 (the larger value at that card's position).
The server says another engine holds the GPU. Each card serves one engine process. Stop the other one, or wait if
it is still exiting after a crash; check dmesg for Xid lines.
The server restarted with exit code 75. The engine stalled (collective timeout) or the watchdog saw no progress
([hang]). The open requests got an error first. Please report the log, which holds the diagnostics.
Does it run on Windows / WSL2? Untested. WSL2 has no GPU P2P and limits pinned host memory.
Is greedy output reproducible? Flash-Next: not by default. At temperature 0, two runs can differ where two tokens are nearly tied, depending on what shares a step and on what ran on the cards before. FLASH_ATTN_CHUNK=1 makes the same request give the same text whatever shares a step and whatever state the machine is in; use it to compare runs, reproduce a report or run tests. It does not make answers closer to BF16, and it costs speed (see docs/advanced.md). It holds per setup: a different number of cards can give different text, and I didn't measure it across cache hits and misses. I found the class of the cause (float sums in the decode path), but I can't explain why earlier load on the cards changes the result. For one conversation run through long prefills, add FLASH_DETERMINISTIC=1 FLASH_LOOKUP=0. 27B: not bit for bit across cache hits and
misses, batch sizes or shared prefill batches (different kernels split near-ties differently). On every engine this
includes conversations that share their beginning with one the server still holds (the same system prompt, for
example): the shared part is reused, the rest is computed in a call of another size, and where two tokens are nearly
tied the reply can differ from the same request sent to a freshly started server. Greedy output is reproducible within
one installation: the 27B times its GEMM kernels once and keeps the choice in ~/.cache/qengine, so a fresh tune (a
new version, a cleared cache, or a container without a persistent ~/.cache/qengine volume) can change tokens where
two are nearly tied.
More symptoms and log lines: docs/advanced.md.
src/ 27B engine: slots, prompt cache, MTP, scheduling, per-GPU worker, C API; kernels in src/kernels/
src/flash/ Qwen3.8-Flash-Next engine: libqflash.so, flash_run, kernels, pipeline modes
src/dsv4/ DeepSeek-V4.1-Flash engine (libqdsv4.so, build.sh): GGUF reader, CPU expert lane, DSpark drafts
src/glm/ GLM-5.3-Flash engine (libqglm.so, from the CMake build): host-RAM experts, CPU lane, MTP
src/gptoss/ gpt-oss-120B engine (libqgptoss.so, build.sh): MXFP4 experts, harmony format (server/harmony.py)
server/ app.py (OpenAI + Anthropic API, tool calls, grammars, scheduling), engine.py (ctypes),
model_profile.py (per-model defaults), think_budget.py, vision.py, steering.py,
dashboard + qetop.py (monitoring), templates/ (chat template), run.sh
tests/ regression tests (engines, server, kernels)
tools/ model conversion (convert_flash_next.py, convert_ngram_int8.py, convert_ngram_fp8.py), reference dumps, draft vocabulary,
GLM-5.3-Flash data files (tools/glm/)
bench/ serving_suite.py and the benchmark harness
docs/ advanced.md: more settings, measurements, steering, troubleshooting
The tests in tests/ check invariants of the engines (cached, batched, speculative and archived paths reproduce a
fresh computation; sampling keeps its distribution; kernels match torch). CPU-only examples:
python3 -m pip install -r requirements-dev.txt
python3 tests/test_server_mock.py && python3 tests/test_think_budget.py # server end to end on a mock engine
python3 tests/test_grammar.py && python3 tests/test_toolcall_parse.pyMost other tests need the GPUs and a model (QE_TEST_MODEL, QE_LIB=build/libqflash.so for Flash-Next).
Every release has notes on the releases page, with the fixes, new settings and how to upgrade.
QEngine is built and tested on one machine at home, and every release goes through long benchmark, quality and stress runs before it ships. If it saves you money or lets you run something you couldn't before, you can buy me a coffee. It pays for power, disks and memory for new models, and the time it takes to fix the bugs you report. Bug reports and benchmark results from your own hardware help just as much.
QEngine is licensed under the MIT License, see LICENSE. Versions up to 1.3.5.7 were released under the PolyForm Noncommercial License 1.0.0.
The model weights are not part of this repository; they are distributed by their authors under their own licenses.
server/templates/qwen38_syssafe.jinja is a modified version of the model's chat template. The included steering
vector falls under the model's license (see docs/advanced.md).
QEngine is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and non-infringement. As far as the law allows, the author is not liable for any claim, damages or other liability arising from the use of the software, including hardware damage, data loss, incorrect model output or downtime. It is experimental software, tested on a single machine (3× RTX 3090); use it at your own risk. Benchmark results describe that machine and may not reproduce on other hardware, drivers or software versions. See LICENSE for the binding terms.