Use extra drives when model reads limit inference. COLI_MODEL_MIRROR points
to read-only copies of the primary model's shards; no RAID setup is required.
These examples target the GLM-5.2 engine. Other engines have their own I/O
paths, so check their model guides before applying the same settings.
Run from c/ in a source checkout, or the directory containing coli in an
unpacked release. Replace the paths with your existing model copies.
Bash, with one primary and two mirrors:
COLI_MODEL_MIRROR='/mnt/nvme2/glm52_i4;/mnt/nvme3/glm52_i4' \
python3 ./coli chat --model /mnt/nvme1/glm52_i4PowerShell, with the same three-drive layout:
$env:COLI_MODEL_MIRROR = 'E:\models\glm52_i4;F:\models\glm52_i4'
python ./coli chat --model 'D:\models\glm52_i4'For two drives, supply just one mirror directory. Quote the list: an unquoted
semicolon separates shell commands. The primary is supplied by --model
(or COLI_MODEL); do not repeat it in the mirror list.
With COLI_DISK_WEIGHTS unset, the engine probes bandwidth at startup and prints
the measured split. An optional override is a comma-separated list of positive
relative weights, primary first, followed by each usable mirror in list order.
For example, 9,3,3 requests shares of 60%, 20%, and 20% across three drives;
it is not a measured speedup. Check the startup log for skipped mirrors and
probe failures before trusting the split. See the
environment reference for the variables.
Missing or divergent mirror shards fall back to the primary. For a smaller
drive, use the partial-mirror planner
to stage a usage-ranked subset. COLI_MODEL_DIRS is a different layout: it
spreads distinct shards across drives to combine capacity, rather than adding
replicas of the same shards.
Details worth knowing:
- the mirror is validated at startup (per-file size + safetensors header must be byte-identical to the primary); divergent or missing files stay on the primary, so a partial mirror is fine: a smaller second SSD can serve the shards it holds;
- the mirror is never written:
.coli_usage,.coli_kvand all sidecars stay on the primary; - a read error on the mirror falls back to the primary (one warning, no crash), so unplugging the second drive mid-run degrades instead of killing the server;
- buffered reads use deterministic expert routing; eligible direct reads can stripe one expert across replicas;
- routing never changes tokens, because both copies are byte-identical; enable
PROF=1for theMIRROR:profile counters showing GB served per drive.
For a second drive that cannot hold the whole model, Colibri can rank a partial
mirror from the expert history it already learns. Run a few representative
prompts first so .coli_usage reflects the workload, then plan, stage, and
verify the mirror:
./c/coli mirror plan --model /fast/glm52_i4 --mirror /second/glm52_i4 \
--budget-gib 200 --reserve-gib 20
./c/coli mirror stage --model /fast/glm52_i4 --mirror /second/glm52_i4 \
--budget-gib 200 --reserve-gib 20
./c/coli mirror verify --model /fast/glm52_i4 --mirror /second/glm52_i4The planner reads safetensors headers directly, follows split-model directories
from COLI_MODEL_DIRS, and prioritizes shards that can serve the hottest routed
experts. Staging never changes the primary model: it copies through temporary
files, preserves the requested free-space reserve, verifies every shard with
SHA-256, never deletes an existing mirror shard, and atomically publishes a
receipt only after the selected mirror is ready.
More storage bandwidth does not translate directly into more tokens per second. Compute, cache residency, and shared PCIe/controller bandwidth can dominate. Compare the complete inference workload, not just the drive probe.
- Two independent NVMe drives helped. The Threadripper PRO 7965WX report in
#1249, also recorded in
the benchmark table, measured 0.80 to 1.10 tok/s (+37.5%)
with
DIRECT=1; the buffered comparison gained about 16%. - Mixed-speed drives exposed a bug that has since been fixed. #1270 replaced equal-sized stripes with bandwidth-weighted chunks. Before that fix, adding a SATA drive to two NVMe drives reduced GLM-5.2 decode from 0.900 to 0.666 tok/s in the reported experiment. This is historical evidence, not a current slowdown attributed to every SATA mirror.
- The post-fix third drive was neutral on that host. The contributor's
follow-up on
devreported 1.030 tok/s with two NVMe drives and 1.006 with the added SATA drive (five interleaved runs per arm, reported standard deviations 0.041 and 0.034). The author judged the difference within run-to-run variation. That Windows 11 / i9-14900K / RTX 3090 result does not promise a gain from a slower disk on another machine.
These are community measurements on specific configurations, not new results from this guide. #1137 contains the multi-disk discussion and an earlier mixed-speed report.
- Record the commit, model/container, hardware and drive/controller layout. Keep the prompt, seed, output-token limit, RAM/VRAM budgets, and I/O mode fixed. Keep background disk work idle. Use the same learned usage history and cache preparation for each arm, and report cold and warm runs separately.
- Start a fresh process for each arm. For the primary-only baseline, clear
COLI_MODEL_MIRROR, its legacy aliasSNAP_MIRROR, and anyCOLI_DISK_WEIGHTSoverride. For the mirror arm, set onlyCOLI_MODEL_MIRRORto the copies being tested and let the probe choose the weights. Keep anyCOLI_MODEL_DIRSlayout fixed; if it spans drives, label this a split-layout baseline rather than a single-drive baseline. - Set
PROF=1in both arms. Save startup[MIRROR]messages, throughput, TTFT, expert hit rate, and per-driveMIRROR:byte/read counters. The GLM serve loop emits cumulative profile counters on clean shutdown; do not assume they appear after every chat response. - Interleave at least three repetitions per arm. Report the median and
spread, output/correctness checks, and raw logs, including excluded runs.
If testing
DIRECT=1, run it as a separate comparison and verify that the platform and filesystem actually use direct I/O. See the benchmarking protocol for cache-state and reporting rules.
To return to the primary-only configuration in your shell:
unset COLI_MODEL_MIRROR SNAP_MIRROR COLI_DISK_WEIGHTSRemove-Item Env:COLI_MODEL_MIRROR, Env:SNAP_MIRROR, Env:COLI_DISK_WEIGHTS -ErrorAction SilentlyContinuePowerShell's $env: assignments persist in that session; remove PROF too if
you enabled it only for this comparison.