8000 GitHub - bmqube/minillm: A lightweight, efficient transformer inference engine written in Rust. MiniLLM provides a clean, well-documented implementation of GPT-2 style transformer models with support for text generation. · GitHub
[go: up one dir, main page]

Skip to content
bmqubePublic

About

A lightweight, efficient transformer inference engine written in Rust. MiniLLM provides a clean, well-documented implementation of GPT-2 style transformer models with support for text generation.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Latest commit

 

History

53 Commits

Folders and files

Repository files navigation

MiniLLM

CI

A small transformer inference engine written in Rust. It loads decoder-only checkpoints from the HuggingFace Hub and runs autoregressive text generation on CPU or CUDA, with a KV cache that can store keys and values at full precision, per-token int8, or per-token packed int4.

Three architectures, one interface:

GPT-2 Qwen3 Qwen3.5
checkpoints openai-community/gpt2{,-medium,-large,-xl} Qwen/Qwen3-{0.6B,1.7B,4B,8B,14B} Qwen/Qwen3.5-{0.8B,2B,4B,9B,27B}, Qwen3.6-27B, Qwen3.8-27B
token mixer multi-head attention grouped-query attention hybrid: Gated DeltaNet ×3, gated GQA ×1
decode state KV cache KV cache KV cache in 1 layer of 4, fixed-size recurrent state in the rest
positions learned wpe table RoPE (rope_theta 1e6) RoPE on 25% of each head (rope_theta 1e7)
normalization LayerNorm RMSNorm RMSNorm, zero-centred gain x̂·(1+w)
Q/K scaling — per-head RMSNorm (QK-norm) before RoPE QK-norm; L2-norm in DeltaNet
MLP 4×, tanh-GELU SwiGLU SwiGLU
weight layout Conv1D [in, out] nn.Linear [out, in] nn.Linear, under model.language_model.
context 1024 40960 262144
published dtype fp32 bf16 bf16

All three are verified against HuggingFace Transformers to floating-point noise, on the forward path and the KV-cache path (see Benchmarks).

Two implementations

Branch What it is
main (this branch) Built on candle for tensor ops and the nn primitives. GPU support, less low-level code to maintain. This is the base for ongoing work.
scratch The original from-scratch version: a hand-written tensor library (src/tensor.rs, ~1D–4D, matmul / softmax / layernorm) with no ML dependencies, plus the full GPT-2 forward pass on top of it. CPU only. Kept as a reference for the ground-up implementation.

If you want to see the transformer built without a tensor framework, read the scratch branch. If you want to run models, use main.

Status and scope

  • Works: GPT-2, Qwen3 and Qwen3.5 (dense) weight loading (single-file and sharded safetensors), forward pass, greedy / temperature / top-k / top-p sampling, CPU and CUDA execution.
  • No weight quantization. Weights load at the checkpoint's own torch_dtype — fp32 for GPT-2, bf16 for Qwen3 — or at an explicit Precision. candle has no CPU bf16 gemm, so a bf16 checkpoint on CPU falls back to fp32 with a note; on CUDA it stays bf16.
  • KV cache. forward_with_cache decodes one token per step against a per-layer key/value cache — O(n) instead of the O(n²) full-recompute forward. ~11× faster greedy decode on CPU at seq 64→192 (up to ~31× by seq 512→544, see the sweep in benchmarks), and the cached path still matches HuggingFace to fp32 noise. The plain forward is kept as the reference and pre-cache baseline; both it and the cached path skip the LM head on positions the caller won't use (forward_last / forward_with_cache_last).
  • Grouped-query attention is respected in the cache and in the math. Only the narrow n_kv_head tensors are stored, and attention folds each KV head's group of query heads into one matmul instead of materialising an expanded copy of K/V every step. On Qwen3-0.6B (16 query heads over 8 KV heads) that is a 2× smaller cache before any quantization.
  • Preallocated cache buffer. At full precision K/V are written in place into a buffer that doubles when full, and attention reads a strided view of it, so a decode step no longer copies the whole cache.
  • Hardware-aware execution plan. At startup the engine probes the device (which half-precision dtypes actually run there), the CPU (physical cores, SIMD features available vs. compiled in) and picks thread count, weight precision and where greedy argmax runs. Every binary prints the plan and sweep records it in each CSV row. See Hardware.
  • Quantized KV cache. KvQuant::Int8 (per-token symmetric) is 3.76× smaller and KvQuant::Int4 (per-token asymmetric, packed) 6.4× smaller. On GPT-2 int8 costs effectively nothing (Δppl ≈ +0.008 on WikiText-2) — but that result does not transfer to Qwen3, which is ~7× more sensitive to int8 and ~15× more to int4, to the point where int8 changes the top-1 token on prompts GPT-2 handles unaffected. Qwen3.5, which rotates only a quarter of each head and caches K/V in only 6 of its 24 layers, sits far closer to GPT-2 (int8 drift 0.054σ against Qwen3's 0.266σ, on one prompt). See Benchmarks.
  • Hybrid linear attention (Qwen3.5). Three of every four layers are Gated DeltaNet, whose decode state is a fixed-size matrix per head, so the cache holds per-layer state, either a growing KV cache or a constant recurrent one. On Qwen3.5-0.8B the KV cache is 9.3× smaller per token than Qwen3-0.6B's, and cached decode loses 8% of its speed from 64 to 960 tokens of context against Qwen3's 27%. The vision tower and MTP head ship in the checkpoint but are not loaded; the MoE variants (35B-A3B) are rejected.
  • Larger sizes (gpt2-medium/large/xl, Qwen3-1.7B and up, Qwen3.5-2B and up) share their family's architecture and load through the same path, but only gpt2 124M, Qwen3-0.6B and Qwen3.5-0.8B are regularly exercised.
  • Inference only — no training.

Architecture

Architecture-specific code lives under models/, behind the CausalLM trait. Everything above it — the decode loop, the cache, the benchmark harnesses — is generic and never names a concrete model type.

src/
├── lib.rs          library root
├── main.rs         CLI demo: [MODEL] [PROMPT], KV-cache decode
├── loader.rs       config parsing, architecture dispatch, sharded safetensors, precision choice
├── dtype.rs        Precision (f32/f16/bf16) + what each device can actually run
├── device.rs       pick CUDA if built with --features cuda, else CPU
├── hardware.rs     probe: cores, CPU SIMD (available vs. compiled in), half-precision support
├── plan.rs         ExecutionPlan: the hardware-dependent choices for a run
├── generation.rs   Generator (decode loop) + SamplingConfig / sample(): greedy / temp / top-k / top-p
├── kv_cache.rs     KvCache / LayerKvCache: per-layer K/V, model precision or per-token int8 / int4
│
├── layers/         primitives shared across architectures
│   ├── activation.rs  tanh-approx GELU (GPT-2's gelu_new) + SiLU (Qwen3's SwiGLU gate)
│   ├── mask.rs        additive causal + sliding-window masks, dtype-aware
│   ├── rope.rs        rotary embeddings, offset-aware for cached decode
│   └── mod.rs         grouped_attention (GQA without expanding K/V), repeat_kv reference
│
└── models/
    ├── mod.rs      CausalLM trait + ModelMeta (layers, q/kv heads, head_dim, ctx, params)
    ├── gpt2/       config, attention (fused QKV, MHA), block (pre-LN + GELU MLP), model
    ├── qwen3/      config, attention (GQA + QK-norm + RoPE), block (RMSNorm + SwiGLU), model
    └── qwen3_5/    config, delta_net (Gated DeltaNet), attention (gated GQA, partial RoPE),
                    norm (zero-centred + gated RMSNorm), block, model

src/bin/
├── bench.rs        throughput + size benchmark, no-cache vs KV-cache decode
├── sweep.rs        seq-len sweep (cache off/on, quant, --repeats N, --precision) → CSV
├── ppl.rs          sliding-window perplexity on a text file (--kv-quant off|int8|int4)
├── memprobe.rs     peak-RSS probe for one decode path (KV-cache memory cost)
└── parity_dump.rs  dump logits for the parity check (--cache exercises the cache path)

examples/
└── generate.rs     minimal library-usage example (KV-cache decode loop)

tests/
└── cache_parity.rs cache == full-recompute, incremental == bulk prefill, GQA cache
                    shape, quantization drift — per architecture (ignored; needs weights)

benchmarks/         prompt set, parity scripts, methodology + results, GPU runbook

Adding an architecture

  1. src/models/<name>/ with config.rs (deserialize config.json, expose a ModelMeta), attention.rs, block.rs, model.rs.
  2. impl CausalLM for <Name>Model.
  3. One arm in loader::Architecture and its config.json probe.

Reuse layers/ for anything that isn't specific to the architecture. Nothing else in the crate needs to change — the binaries, the cache and the decode loop are already generic.

Build

CPU build (default, no CUDA toolkit needed):

cargo build --release

GPU build (requires a CUDA toolkit and a compatible driver):

cargo build --release --features cuda

Rust 1.91+ required. Some dependencies compile native code (ring, aws-lc-rs, onig_sys), so a C toolchain is required: on Linux/macOS the system compiler is enough; on Windows use the MSVC build tools, or MSYS2 mingw-w64-gcc with the x86_64-pc-windows-gnu Rust target.

On x86_64-pc-windows-gnu, aws-lc-sys (pulled in via hf-hub) compiles against the MSYS2 winpthreads headers but rustup's bundled self-contained mingw libs are older and lack nanosleep64, so the link fails. .cargo/config.toml sets -C link-self-contained=no for that target, which hands CRT + winpthreads resolution to the MSYS2 gcc driver (whose libwinpthread has the symbol). Keep MSYS2 mingw-w64-x86_64-toolchain on PATH. See the comments in that file for the MSVC link.exe-shadowing case.

Hardware

.cargo/config.toml builds with -C target-cpu=native, so a local build uses every SIMD extension the build machine has. This matters more than it sounds: candle compiles its AVX2/F16C kernels (f16 and bf16 dot products, the quantized matmuls) only when the target enables them, and a default x86-64 build leaves them out. fp32 matmul is unaffected either way, because the gemm crate picks its SIMD at run time.

The flip side is that the binary only runs on CPUs with the same features. To build one for distribution, set RUSTFLAGS (which replaces the config file's flags), e.g. RUSTFLAGS="-C target-cpu=x86-64-v2" cargo build --release.

Everything else is decided at run time by ExecutionPlan, printed at startup:

plan   : device=cpu threads=6 (6C/12T) cpu_simd=built:avx2+fma+f16c available:avx2+fma+f16c half=f16 argmax=host
choice CPU CUDA
threads physical cores (decode is bandwidth-bound; SMT siblings add little). Override with RAYON_NUM_THREADS or sweep --threads —
weight precision checkpoint dtype if the device runs it (probed), else fp32 bf16 on Ampere+, fp32 fallback
greedy argmax on the host on the device; only the token id is copied back

If the CPU supports more than the binary was built for, the plan prints a warning saying so.

Usage

CLI

cargo run --release                                        # gpt2, CPU
cargo run --release --features cuda                         # gpt2, GPU
cargo run --release -- Qwen/Qwen3-0.6B "Once upon a time"   # qwen3

Arguments are [MODEL] [PROMPT], where MODEL is a Hub id or a local directory. Downloads on first run and prints a 50-token completion, followed by a tok/s line on stderr.

Library

loader::load returns a Box<dyn CausalLM> — the architecture is picked from the checkpoint's config.json, so the calling code is identical either way. Generator owns the KV cache and drives the decode loop: prefill the prompt once, then take one token per step.

use minillm::generation::{Generator, SamplingConfig};
use minillm::{device, loader};

fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let dev = device::best();
    // Swap for "Qwen/Qwen3-0.6B" and nothing below changes.
    let (model, tokenizer) = loader::load("openai-community/gpt2", &dev)?;

    let ids = tokenizer.encode("The future of AI is", true)?.get_ids().to_vec();
    let cfg = SamplingConfig { temperature: 0.8, top_k: Some(40), top_p: Some(0.95) };

    let mut generator = Generator::new(model.as_ref());
    // Holds back a character split across tokens until it is whole.
    let mut text = tokenizer.decode_stream(false);
    generator.prefill(&ids)?;

    for _ in 0..40 {
        if let Some(piece) = text.step(generator.next_token(&cfg)?)? {
            print!("{piece}");
        }
    }
    Ok(())
}

model.meta() exposes the shape facts a caller might need — layer count, query and KV head counts, head_dim, context window, parameter count, and kv_cache_bytes_per_token_with(quant, precision) for cache accounting.

Precision

use minillm::dtype::Precision;
let (model, tok) = loader::load_with("Qwen/Qwen3-0.6B", &dev, Some(Precision::BF16))?;

Omit the override and the checkpoint's own torch_dtype is used. candle has no CPU bf16 gemm, so on CPU a bf16 checkpoint falls back to fp32 with a note, while an explicit Precision::BF16 on CPU is refused outright rather than failing later inside a matmul.

Run the same thing as an example:

cargo run --release --example generate -- "Once upon a time"

Other entry points:

Generator::with_quant(model, KvQuant::Int8) quantized KV cache
generation::greedy(model, &ids, n, quant) one-shot greedy decode
generation::greedy_no_cache(model, &ids, n) the O(n²) reference path
generation::sample_with(&logits, &cfg, &mut rng) seeded, reproducible sampling
Generator::next_token_with(&cfg, &mut rng) same, inside the decode loop
CausalLM::forward_last / forward_with_cache_last only the final position's logits, skipping the LM head on the rest — what Generator and greedy_no_cache use internally
loader::load_with(spec, &dev, Some(Precision::BF16)) pin the weight dtype
model.meta() layers, q/kv heads, head_dim, context, params, KV bytes/token

Greedy decoding (temperature <= 1e-6; top_k/top_p cannot change the argmax, so they are ignored) consumes no randomness, so every benchmark in this repo is deterministic. For reproducible sampled output, pass a seeded RNG:

use rand::SeedableRng;
let mut rng = rand::rngs::StdRng::seed_from_u64(42);
let next = generator.next_token_with(&cfg, &mut rng)?;

Sampling options

SamplingConfig controls sample(&logits, &cfg):

field effect
temperature: f64 divides logits before softmax; <= 1e-6 selects greedy (argmax)
top_k: Option<usize> keep only the k highest-scoring tokens
top_p: Option<f64> nucleus: keep the shortest ranked prefix whose probability mass reaches p

Sampling runs on the CPU over a Vec<f32> — one token per step is cheap and the scalar code is easy to verify.

Model loading

Weights, config.json and tokenizer.json are fetched via hf-hub and cached under ~/.cache/huggingface. For gated or private repos, provide a token:

echo "HF_TOKEN=hf_your_token_here" > .env      # loaded via dotenvy
# or: export HF_TOKEN=hf_your_token_here

Benchmarks

See benchmarks/ for the harness, methodology and full results.

# one-time: fetch GPT-2 weights into a local dir (see benchmarks/README.md)
cargo run --release --bin bench -- benchmarks/gpt2 64 128        # no-cache vs KV-cache decode
cargo run --release --bin parity_dump -- --cache                 # dump cache-path logits
python benchmarks/parity.py benchmarks/minillm_logits.json       # vs transformers

Measured on a Ryzen 5 5600G, CPU, fp32, openai-community/gpt2 (124M):

prefill (64 tok) ~485–510 tok/s
decode (128 steps, seq 64→192), no cache ~4.5 tok/s
decode (128 steps, seq 64→192), KV cache ~49–50 tok/s (≈11×)
peak RSS ~977 MiB
parity vs HF Transformers (24 prompts), forward mean cos 1.00000, top-1 24/24, mean MSE 1.8e-9
parity vs HF Transformers (24 prompts), KV-cache path mean cos 1.00000, top-1 24/24, mean MSE 2.2e-9

Numbers come from real runs, not estimates. They predate the execution plan, preallocated cache and grouped attention, which together made cached decode 1.5–1.9× faster on the same CPU (measured under WSL2, so not directly comparable with the table above); see benchmarks §9.

Tests

cargo test

Covers the samplers (greedy = argmax, top_k = 1 is deterministic, nucleus keeps the dominant token, ids stay in range), the analytic parameter / KV-cache-byte counts, and the KvCache container (append grows the sequence axis, reset, length tracking, int8/int4 round-trip error bounds). These do not require downloading a model.

The end-to-end KV-cache checks need weights in benchmarks/gpt2/, benchmarks/qwen3-0.6b/ and/or benchmarks/qwen3.5-0.8b/ (each architecture's cases skip if its directory is absent) and are #[ignore]d by default:

cargo test --release --test cache_parity -- --ignored --test-threads=1

--test-threads=1 keeps one model in memory at a time; run in parallel, the fp32 Qwen copies need well over 13 GiB.

cache_matches_no_cache asserts that greedy decoding with the cache produces the exact same token ids as the full-recompute forward; quantized_caches_stay_close_to_fp32 bounds how far int8/int4 prefill logits drift, and that int8 stays closer than int4.

CI (.github/workflows/ci.yml) runs cargo fmt --check, cargo clippy --all-targets -D warnings, the test suite and cargo doc on every push and PR.

Roadmap

  • KV cache — done (forward_with_cache, ~11× faster CPU decode, parity-checked)
  • int8 / int4 KV-cache quantization — done (per-token, perplexity + latency + memory ablation)
  • A second architecture — done (Qwen3: RoPE, RMSNorm, GQA, QK-norm, SwiGLU, bf16, sharded checkpoints)
  • GPU numbers — the CPU tables stop at 512 tokens because that is where CPU decode becomes unusable, which is also where a KV cache stops being interesting. See benchmarks/GPU-RUNBOOK.md.
  • Confirm the int8 architecture gap on perplexity. The Qwen3 sensitivity result is currently a single-prompt logit measurement; 60k tokens of WikiText-2 would either confirm it or show it up as an artifact.
  • Pre-RoPE and/or per-channel key quantization — the known fix for the drift that section 8 of the benchmarks measures
  • Preallocated cache buffer — done for full-precision storage (quantized modes still cat)
  • Hardware-aware execution plan — done (dtype probe, threads, device-side argmax)
  • CPU mixed precision — f16 weight storage with fp32 activations for bf16 checkpoints: half the memory and bandwidth of the current fp32 fallback
  • Low-precision attention matmul (turn the memory saving into a peak-RSS + speed win)
  • A hybrid architecture — done (Qwen3.5: Gated DeltaNet, per-layer recurrent state)
  • Chunked DeltaNet prefill — prefill runs the recurrence token by token, 25–40% slower than Qwen3's attention prefill on CPU
  • MTP speculative decoding — Qwen3.5 checkpoints carry a multi-token- prediction head, a ready-made drafter
  • Verify and benchmark gpt2-medium/large/xl, Qwen3-1.7B/4B/8B, Qwen3.5-2B/4B/9B
  • Batched generation

License

MIT — see LICENSE.

Author

BM Monjur Morshed — @bmqube

Acknowledgments

  • candle for tensors and nn primitives
  • HuggingFace for model weights and the tokenizers library
  • Inspired by Andrej Karpathy's educational GPT implementations

About

A lightweight, efficient transformer inference engine written in Rust. MiniLLM provides a clean, well-documented implementation of GPT-2 style transformer models with support for text generation.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0