A small transformer inference engine written in Rust. It loads decoder-only checkpoints from the HuggingFace Hub and runs autoregressive text generation on CPU or CUDA, with a KV cache that can store keys and values at full precision, per-token int8, or per-token packed int4.
Three architectures, one interface:
| GPT-2 | Qwen3 | Qwen3.5 | |
|---|---|---|---|
| checkpoints | openai-community/gpt2{,-medium,-large,-xl} |
Qwen/Qwen3-{0.6B,1.7B,4B,8B,14B} |
Qwen/Qwen3.5-{0.8B,2B,4B,9B,27B}, Qwen3.6-27B, Qwen3.8-27B |
| token mixer | multi-head attention | grouped-query attention | hybrid: Gated DeltaNet ×3, gated GQA ×1 |
| decode state | KV cache | KV cache | KV cache in 1 layer of 4, fixed-size recurrent state in the rest |
| positions | learned wpe table |
RoPE (rope_theta 1e6) |
RoPE on 25% of each head (rope_theta 1e7) |
| normalization | LayerNorm | RMSNorm | RMSNorm, zero-centred gain x̂·(1+w) |
| Q/K scaling | — | per-head RMSNorm (QK-norm) before RoPE | QK-norm; L2-norm in DeltaNet |
| MLP | 4×, tanh-GELU | SwiGLU | SwiGLU |
| weight layout | Conv1D [in, out] |
nn.Linear [out, in] |
nn.Linear, under model.language_model. |
| context | 1024 | 40960 | 262144 |
| published dtype | fp32 | bf16 | bf16 |
All three are verified against HuggingFace Transformers to floating-point noise, on the forward path and the KV-cache path (see Benchmarks).
| Branch | What it is |
|---|---|
main (this branch) |
Built on candle for tensor ops and the nn primitives. GPU support, less low-level code to maintain. This is the base for ongoing work. |
scratch |
The original from-scratch version: a hand-written tensor library (src/tensor.rs, ~1D–4D, matmul / softmax / layernorm) with no ML dependencies, plus the full GPT-2 forward pass on top of it. CPU only. Kept as a reference for the ground-up implementation. |
If you want to see the transformer built without a tensor framework, read the
scratch branch. If you want to run models, use main.
- Works: GPT-2, Qwen3 and Qwen3.5 (dense) weight loading (single-file and sharded safetensors), forward pass, greedy / temperature / top-k / top-p sampling, CPU and CUDA execution.
- No weight quantization. Weights load at the checkpoint's own
torch_dtype— fp32 for GPT-2, bf16 for Qwen3 — or at an explicitPrecision. candle has no CPU bf16 gemm, so a bf16 checkpoint on CPU falls back to fp32 with a note; on CUDA it stays bf16. - KV cache.
forward_with_cachedecodes one token per step against a per-layer key/value cache — O(n) instead of the O(n²) full-recomputeforward. ~11× faster greedy decode on CPU at seq 64→192 (up to ~31× by seq 512→544, see the sweep in benchmarks), and the cached path still matches HuggingFace to fp32 noise. The plainforwardis kept as the reference and pre-cache baseline; both it and the cached path skip the LM head on positions the caller won't use (forward_last/forward_with_cache_last). - Grouped-query attention is respected in the cache and in the math. Only
the narrow
n_kv_headtensors are stored, and attention folds each KV head's group of query heads into one matmul instead of materialising an expanded copy of K/V every step. On Qwen3-0.6B (16 query heads over 8 KV heads) that is a 2× smaller cache before any quantization. - Preallocated cache buffer. At full precision K/V are written in place into a buffer that doubles when full, and attention reads a strided view of it, so a decode step no longer copies the whole cache.
- Hardware-aware execution plan. At startup the engine probes the device
(which half-precision dtypes actually run there), the CPU (physical cores,
SIMD features available vs. compiled in) and picks thread count, weight
precision and where greedy argmax runs. Every binary prints the plan and
sweeprecords it in each CSV row. See Hardware. - Quantized KV cache.
KvQuant::Int8(per-token symmetric) is 3.76× smaller andKvQuant::Int4(per-token asymmetric, packed) 6.4× smaller. On GPT-2 int8 costs effectively nothing (Δppl ≈ +0.008 on WikiText-2) — but that result does not transfer to Qwen3, which is ~7× more sensitive to int8 and ~15× more to int4, to the point where int8 changes the top-1 token on prompts GPT-2 handles unaffected. Qwen3.5, which rotates only a quarter of each head and caches K/V in only 6 of its 24 layers, sits far closer to GPT-2 (int8 drift 0.054σ against Qwen3's 0.266σ, on one prompt). See Benchmarks. - Hybrid linear attention (Qwen3.5). Three of every four layers are Gated
DeltaNet, whose decode state is a fixed-size matrix per head, so the cache
holds per-layer state, either a growing KV cache or a constant recurrent
one. On Qwen3.5-0.8B the KV cache is 9.3× smaller per token than
Qwen3-0.6B's, and cached decode loses 8% of its speed from 64 to 960 tokens of
context against Qwen3's 27%. The vision tower and MTP head ship in the
checkpoint but are not loaded; the MoE variants (
35B-A3B) are rejected. - Larger sizes (
gpt2-medium/large/xl,Qwen3-1.7Band up,Qwen3.5-2Band up) share their family's architecture and load through the same path, but onlygpt2124M,Qwen3-0.6BandQwen3.5-0.8Bare regularly exercised. - Inference only — no training.
Architecture-specific code lives under models/, behind the CausalLM trait.
Everything above it — the decode loop, the cache, the benchmark harnesses — is
generic and never names a concrete model type.
src/
├── lib.rs library root
├── main.rs CLI demo: [MODEL] [PROMPT], KV-cache decode
├── loader.rs config parsing, architecture dispatch, sharded safetensors, precision choice
├── dtype.rs Precision (f32/f16/bf16) + what each device can actually run
├── device.rs pick CUDA if built with --features cuda, else CPU
├── hardware.rs probe: cores, CPU SIMD (available vs. compiled in), half-precision support
├── plan.rs ExecutionPlan: the hardware-dependent choices for a run
├── generation.rs Generator (decode loop) + SamplingConfig / sample(): greedy / temp / top-k / top-p
├── kv_cache.rs KvCache / LayerKvCache: per-layer K/V, model precision or per-token int8 / int4
│
├── layers/ primitives shared across architectures
│ ├── activation.rs tanh-approx GELU (GPT-2's gelu_new) + SiLU (Qwen3's SwiGLU gate)
│ ├── mask.rs additive causal + sliding-window masks, dtype-aware
│ ├── rope.rs rotary embeddings, offset-aware for cached decode
│ └── mod.rs grouped_attention (GQA without expanding K/V), repeat_kv reference
│
└── models/
├── mod.rs CausalLM trait + ModelMeta (layers, q/kv heads, head_dim, ctx, params)
├── gpt2/ config, attention (fused QKV, MHA), block (pre-LN + GELU MLP), model
├── qwen3/ config, attention (GQA + QK-norm + RoPE), block (RMSNorm + SwiGLU), model
└── qwen3_5/ config, delta_net (Gated DeltaNet), attention (gated GQA, partial RoPE),
norm (zero-centred + gated RMSNorm), block, model
src/bin/
├── bench.rs throughput + size benchmark, no-cache vs KV-cache decode
├── sweep.rs seq-len sweep (cache off/on, quant, --repeats N, --precision) → CSV
├── ppl.rs sliding-window perplexity on a text file (--kv-quant off|int8|int4)
├── memprobe.rs peak-RSS probe for one decode path (KV-cache memory cost)
└── parity_dump.rs dump logits for the parity check (--cache exercises the cache path)
examples/
└── generate.rs minimal library-usage example (KV-cache decode loop)
tests/
└── cache_parity.rs cache == full-recompute, incremental == bulk prefill, GQA cache
shape, quantization drift — per architecture (ignored; needs weights)
benchmarks/ prompt set, parity scripts, methodology + results, GPU runbook
src/models/<name>/withconfig.rs(deserializeconfig.json, expose aModelMeta),attention.rs,block.rs,model.rs.impl CausalLM for <Name>Model.- One arm in
loader::Architectureand itsconfig.jsonprobe.
Reuse layers/ for anything that isn't specific to the architecture. Nothing
else in the crate needs to change — the binaries, the cache and the decode loop
are already generic.
CPU build (default, no CUDA toolkit needed):
cargo build --releaseGPU build (requires a CUDA toolkit and a compatible driver):
cargo build --release --features cudaRust 1.91+ required. Some dependencies compile native code (ring,
aws-lc-rs, onig_sys), so a C toolchain is required: on Linux/macOS the
system compiler is enough; on Windows use the MSVC build tools, or MSYS2
mingw-w64-gcc with the x86_64-pc-windows-gnu Rust target.
On x86_64-pc-windows-gnu, aws-lc-sys (pulled in via hf-hub) compiles
against the MSYS2 winpthreads headers but rustup's bundled self-contained
mingw libs are older and lack nanosleep64, so the link fails. .cargo/config.toml
sets -C link-self-contained=no for that target, which hands CRT + winpthreads
resolution to the MSYS2 gcc driver (whose libwinpthread has the symbol).
Keep MSYS2 mingw-w64-x86_64-toolchain on PATH. See the comments in that file
for the MSVC link.exe-shadowing case.
.cargo/config.toml builds with -C target-cpu=native, so a local build uses
every SIMD extension the build machine has. This matters more than it sounds:
candle compiles its AVX2/F16C kernels (f16 and bf16 dot products, the quantized
matmuls) only when the target enables them, and a default x86-64 build leaves
them out. fp32 matmul is unaffected either way, because the gemm crate picks
its SIMD at run time.
The flip side is that the binary only runs on CPUs with the same features. To
build one for distribution, set RUSTFLAGS (which replaces the config file's
flags), e.g. RUSTFLAGS="-C target-cpu=x86-64-v2" cargo build --release.
Everything else is decided at run time by ExecutionPlan, printed at startup:
plan : device=cpu threads=6 (6C/12T) cpu_simd=built:avx2+fma+f16c available:avx2+fma+f16c half=f16 argmax=host
| choice | CPU | CUDA |
|---|---|---|
| threads | physical cores (decode is bandwidth-bound; SMT siblings add little). Override with RAYON_NUM_THREADS or sweep --threads |
— |
| weight precision | checkpoint dtype if the device runs it (probed), else fp32 | bf16 on Ampere+, fp32 fallback |
| greedy argmax | on the host | on the device; only the token id is copied back |
If the CPU supports more than the binary was built for, the plan prints a warning saying so.
cargo run --release # gpt2, CPU
cargo run --release --features cuda # gpt2, GPU
cargo run --release -- Qwen/Qwen3-0.6B "Once upon a time" # qwen3Arguments are [MODEL] [PROMPT], where MODEL is a Hub id or a local
directory. Downloads on first run and prints a 50-token completion, followed by
a tok/s line on stderr.
loader::load returns a Box<dyn CausalLM> — the architecture is picked from
the checkpoint's config.json, so the calling code is identical either way.
Generator owns the KV cache and drives the decode loop: prefill the prompt
once, then take one token per step.
use minillm::generation::{Generator, SamplingConfig};
use minillm::{device, loader};
fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let dev = device::best();
// Swap for "Qwen/Qwen3-0.6B" and nothing below changes.
let (model, tokenizer) = loader::load("openai-community/gpt2", &dev)?;
let ids = tokenizer.encode("The future of AI is", true)?.get_ids().to_vec();
let cfg = SamplingConfig { temperature: 0.8, top_k: Some(40), top_p: Some(0.95) };
let mut generator = Generator::new(model.as_ref());
// Holds back a character split across tokens until it is whole.
let mut text = tokenizer.decode_stream(false);
generator.prefill(&ids)?;
for _ in 0..40 {
if let Some(piece) = text.step(generator.next_token(&cfg)?)? {
print!("{piece}");
}
}
Ok(())
}model.meta() exposes the shape facts a caller might need — layer count, query
and KV head counts, head_dim, context window, parameter count, and
kv_cache_bytes_per_token_with(quant, precision) for cache accounting.
use minillm::dtype::Precision;
let (model, tok) = loader::load_with("Qwen/Qwen3-0.6B", &dev, Some(Precision::BF16))?;Omit the override and the checkpoint's own torch_dtype is used. candle has no
CPU bf16 gemm, so on CPU a bf16 checkpoint falls back to fp32 with a note, while
an explicit Precision::BF16 on CPU is refused outright rather than failing
later inside a matmul.
Run the same thing as an example:
cargo run --release --example generate -- "Once upon a time"Other entry points:
Generator::with_quant(model, KvQuant::Int8) |
quantized KV cache |
generation::greedy(model, &ids, n, quant) |
one-shot greedy decode |
generation::greedy_no_cache(model, &ids, n) |
the O(n²) reference path |
generation::sample_with(&logits, &cfg, &mut rng) |
seeded, reproducible sampling |
Generator::next_token_with(&cfg, &mut rng) |
same, inside the decode loop |
CausalLM::forward_last / forward_with_cache_last |
only the final position's logits, skipping the LM head on the rest — what Generator and greedy_no_cache use internally |
loader::load_with(spec, &dev, Some(Precision::BF16)) |
pin the weight dtype |
model.meta() |
layers, q/kv heads, head_dim, context, params, KV bytes/token |
Greedy decoding (temperature <= 1e-6; top_k/top_p cannot change the
argmax, so they are ignored) consumes no randomness, so every benchmark in this repo is deterministic. For reproducible
sampled output, pass a seeded RNG:
use rand::SeedableRng;
let mut rng = rand::rngs::StdRng::seed_from_u64(42);
let next = generator.next_token_with(&cfg, &mut rng)?;SamplingConfig controls sample(&logits, &cfg):
| field | effect |
|---|---|
temperature: f64 |
divides logits before softmax; <= 1e-6 selects greedy (argmax) |
top_k: Option<usize> |
keep only the k highest-scoring tokens |
top_p: Option<f64> |
nucleus: keep the shortest ranked prefix whose probability mass reaches p |
Sampling runs on the CPU over a Vec<f32> — one token per step is cheap and the
scalar code is easy to verify.
Weights, config.json and tokenizer.json are fetched via hf-hub and cached
under ~/.cache/huggingface. For gated or private repos, provide a token:
echo "HF_TOKEN=hf_your_token_here" > .env # loaded via dotenvy
# or: export HF_TOKEN=hf_your_token_hereSee benchmarks/ for the harness, methodology and full results.
# one-time: fetch GPT-2 weights into a local dir (see benchmarks/README.md)
cargo run --release --bin bench -- benchmarks/gpt2 64 128 # no-cache vs KV-cache decode
cargo run --release --bin parity_dump -- --cache # dump cache-path logits
python benchmarks/parity.py benchmarks/minillm_logits.json # vs transformersMeasured on a Ryzen 5 5600G, CPU, fp32, openai-community/gpt2 (124M):
| prefill (64 tok) | ~485–510 tok/s |
| decode (128 steps, seq 64→192), no cache | ~4.5 tok/s |
| decode (128 steps, seq 64→192), KV cache | ~49–50 tok/s (≈11×) |
| peak RSS | ~977 MiB |
| parity vs HF Transformers (24 prompts), forward | mean cos 1.00000, top-1 24/24, mean MSE 1.8e-9 |
| parity vs HF Transformers (24 prompts), KV-cache path | mean cos 1.00000, top-1 24/24, mean MSE 2.2e-9 |
Numbers come from real runs, not estimates. They predate the execution plan, preallocated cache and grouped attention, which together made cached decode 1.5–1.9× faster on the same CPU (measured under WSL2, so not directly comparable with the table above); see benchmarks §9.
cargo testCovers the samplers (greedy = argmax, top_k = 1 is deterministic, nucleus
keeps the dominant token, ids stay in range), the analytic parameter /
KV-cache-byte counts, and the KvCache container (append grows the sequence
axis, reset, length tracking, int8/int4 round-trip error bounds). These do not
require downloading a model.
The end-to-end KV-cache checks need weights in benchmarks/gpt2/,
benchmarks/qwen3-0.6b/ and/or benchmarks/qwen3.5-0.8b/ (each architecture's
cases skip if its directory is absent) and are #[ignore]d by default:
cargo test --release --test cache_parity -- --ignored --test-threads=1--test-threads=1 keeps one model in memory at a time; run in parallel, the
fp32 Qwen copies need well over 13 GiB.
cache_matches_no_cache asserts that greedy decoding with the cache produces the
exact same token ids as the full-recompute forward; quantized_caches_stay_close_to_fp32
bounds how far int8/int4 prefill logits drift, and that int8 stays closer than int4.
CI (.github/workflows/ci.yml) runs cargo fmt --check,
cargo clippy --all-targets -D warnings, the test suite and cargo doc on every
push and PR.
KV cache— done (forward_with_cache, ~11× faster CPU decode, parity-checked)int8 / int4 KV-cache quantization— done (per-token, perplexity + latency + memory ablation)A second architecture— done (Qwen3: RoPE, RMSNorm, GQA, QK-norm, SwiGLU, bf16, sharded checkpoints)- GPU numbers — the CPU tables stop at 512 tokens because that is where CPU
decode becomes unusable, which is also where a KV cache stops being
interesting. See
benchmarks/GPU-RUNBOOK.md. - Confirm the int8 architecture gap on perplexity. The Qwen3 sensitivity result is currently a single-prompt logit measurement; 60k tokens of WikiText-2 would either confirm it or show it up as an artifact.
- Pre-RoPE and/or per-channel key quantization — the known fix for the drift that section 8 of the benchmarks measures
Preallocated cache buffer— done for full-precision storage (quantized modes stillcat)Hardware-aware execution plan— done (dtype probe, threads, device-side argmax)- CPU mixed precision — f16 weight storage with fp32 activations for bf16 checkpoints: half the memory and bandwidth of the current fp32 fallback
- Low-precision attention matmul (turn the memory saving into a peak-RSS + speed win)
A hybrid architecture— done (Qwen3.5: Gated DeltaNet, per-layer recurrent state)- Chunked DeltaNet prefill — prefill runs the recurrence token by token, 25–40% slower than Qwen3's attention prefill on CPU
- MTP speculative decoding — Qwen3.5 checkpoints carry a multi-token- prediction head, a ready-made drafter
- Verify and benchmark
gpt2-medium/large/xl,Qwen3-1.7B/4B/8B,Qwen3.5-2B/4B/9B - Batched generation
MIT — see LICENSE.
BM Monjur Morshed — @bmqube
- candle for tensors and nn primitives
- HuggingFace for model weights and the tokenizers library
- Inspired by Andrej Karpathy's educational GPT implementations