More MoEs: large mixture-of-experts models on one graphics card plus system RAM
its own CUDA and Metal engines, and Strata's · tested on Linux with NVIDIA and on Apple Silicon Macs · free and open source (MIT)
You need Linux with an NVIDIA card (RTX 20 series or newer) or an Apple Silicon Mac, Python 3, and enough RAM for the model. Setup checks the rest and says what is missing.
git clone https://github.com/aflin/momoe.git
cd momoe
./momoe setup # pick a model: setup compiles the engine, downloads the model and starts itSetup picks the model's size for your machine and writes a config for it (for example flint-ud-iq3_xxs); to skip
the menu, name the family: ./momoe setup --family flint. Then:
./momoe run claude # Claude Code connected to the model (also: codex, opencode, unsloth, lmstudio)
./momoe status # what is running, and its address (http://127.0.0.1:8080/v1 by default)
./momoe stop
./momoe start flint-ud-iq3_xxs # start a config again later (./momoe list shows them)
./update.sh # the newest MoMoE; compiles again only what changedAny OpenAI- or Anthropic-compatible app can use the address momoe status shows. More in Install,
Use and Models.
MoMoE began as a fork of Strata, by Niko1221 and the Strata contributors, and Strata is still its base. It has four engines:
| Engine | Runs on | Models |
|---|---|---|
Strata's engine (momoe) |
Linux + NVIDIA | Qwen3.8-Flash-Next and its variants, the Qwen3.6 family |
Strata's DeepSeek engine (strata-dsv41) |
Linux + NVIDIA | DeepSeek-V4.1-Flash on a PC (docs/DEEPSEEK41.md) |
momoe-cuda (MoMoE's own) |
Linux + NVIDIA, the DGX Spark, a Jetson | every family below (DeepSeek-V4.1-Flash on the Spark); on a PC: Nemotron 3.5 B 8000 olt and Lightning, GLM-5.3-Flash, Ornith-1.5-122B, Ling-3.0-flash, MiMo-V2.6 and V2.5, MiniMax M3.1 Flash (two cards), Flint, Gemma 4, LFM2.5 |
momoe-metal (MoMoE's own) |
Apple Silicon Macs | every family below |
-
Strata's engine is Strata's, with MoMoE's additions (the Qwen3.6 family's path began as #1402): experts spread over VRAM, RAM and the SSD, an expert cache filled by predictions, and a multi-token draft layer. How it works: Strata's paper and DETAILS.md. The server and the installer grew from Strata's too. It is Strata 0.2.9's engine. Three of Strata's newer defaults are off here, after measuring them on MoMoE's machines: the routing that prefers experts already on the card (faster, but it hurt the answers; off in Strata too since 0.2.8), and the int8 prompt attention on RTX 40 cards and 0.2.9's GDN prompt kernels (they changed the answers and were no faster here). Measured in docs/ROUTE_PRIOR.md.
-
MoMoE's engines (src/metal) are written for MoMoE on ggml, llama.cpp's library. They read a model's GGUF as it is and keep the experts they can in a pool on the GPU. The rest are, per model and machine:
- read from the SSD (on the Spark past the page cache, at 10-13 GB/s);
- copied in from RAM;
- or computed by the CPU while the GPU computes its own (for Ornith-122B, Flint, Ling-3.0-flash and MiMo, a trial while the model runs picks the faster way).
Their graphs are kept and replayed (as CUDA graphs on NVIDIA), and two cards can share a model (the layer split or the helper tier).
For Qwen3.8-Flash-Next on Windows, an AMD or Intel card, or with a desktop app, use Strata. It has a maintainer, ready-made engines and far more testing on far more hardware. Problems with Strata's engine that also happen in Strata belong in Strata's issues.
Why a separate project: MoMoE started as a pull request to Strata (#1402) that added a second model family, and some of what followed doesn't fit Strata, which takes one change per pull request and keeps its default behaviour unchanged. MoMoE's aims:
- more MoE models, each installed with
momoe setup; - agents first:
momoe run claude(or Codex, OpenCode, Unsloth Studio, LM Studio) starts the model and the tool, already connected; there is no web page; - Linux with NVIDIA, and Apple Silicon Macs (Windows and AMD or Intel cards may work, but need help from people who run them);
- built from source: setup compiles the engine on your machine.
Status: early, and maintained by one person.
Unlike my other projects, MoMoE's changes, since the fork and in the pull request it grew from (#1402), are written entirely with Claude Code. Strata's own code, which is most of what is here, is not covered by this note.
The machines:
| Name | Machine |
|---|---|
| PC | Debian 12, RTX 4070 Ti (12 GB), Ryzen 9 5950X, 63 GB of RAM |
| Spark | NVIDIA DGX Spark, 128 GB shared by CPU and GPU |
| Jetson | Jetson Orin Nano Super, 8 GB shared (about 5.9 GiB free beside its desktop), 15 W mode, NVMe SSD |
| M1 Ultra | Mac Studio M1 Ultra, 64 GB |
| M4 Air | MacBook Air M4, 32 GB ("as an 8 GB Mac": setup's 8 GB config with 10 GiB locked away) |
| Mac mini | Mac mini M1, 8 GB, macOS 11 |
Writing speed in tokens per second, with setup's config on each machine; "-" means not tried. * = drafting with the model's own multi-token-prediction layer. These are short tests: a short prompt (a few short questions, or a few hundred tokens) on setup's config, whose context is small (16K to 64K tokens). Real use is slower, for two reasons. A larger context leaves less memory for the expert cache: Ornith-1.5-35B on the PC wrote 128.8 tok/s with setup's 32K config (a 7.6 GiB cache) and 99.0 with a 256K one (5.1 GiB). And the attention over a long conversation costs more for each new token: in a Claude Code session on that 256K config it wrote about 85 tok/s at 40K tokens of conversation and 55-60 at 100K, with the same share of experts found in the cache (about 75-80%) at both.
| Model, size | PC | Spark | Jetson | M1 Ultra | M4 Air |
|---|---|---|---|---|---|
| Qwen3.8-Flash-Next IQ3_XXS | 57.9* | 35.5 | - | 38.2 | 7.9* |
| Flash-Next Coder IQ1_M | 54.8* | 33.3 | - | 48.9* | 12.4* |
| Swift 1.5 IQ3_XXS | 63.4* | 34.6 | - | 37.9 | 11.4* (IQ2_XS) |
| Flash-Next (Unsloth) UD-IQ4_XS | 18.4* | 30.2 | - | 19.0* | 6.1* |
| Qwen3.6-35B-A3B (Unsloth) UD-IQ4_XS | 158.7* | 125.2* | - | 129.3* | 56.5* |
| Qwen3.6-35B-A3B (bartowski) IQ4_XS | 157.5* | 123.6* | - | 117.1* | 60.2* |
| Ornith-1.5-35B-A3B IQ4_XS | 128.8* | 104.1* | - | 92.7* | 49.3* (a 14 GiB pool: 31.2*) |
| KAT-Coder-V2.5-Dev i-quality-v2 | 126.2* | 99.4* | - | 104.9* | 41.1* |
| Nemotron 3.5 Bolt UD-IQ3_XXS | 29.7 | 28.4 | - | 37.5 (UD-Q3_K_XL) | 8.6 |
| Ornith-1.5-122B-A10B UD-IQ3_XXS | 19.2 | 33.2* | - | 19.8* | 4.5 |
| GLM-5.3-Flash UD-IQ3_XXS | does not fit a 12 GB card | 19.7* | - | 4.6 (a 25 GiB pool) | 1.4 |
| GLM-5.3-Flash UD-IQ2_XXS | 4.7 (two cards, from the SSD) | - | - | 5.3 (a 26 GiB pool) | - |
| Ling-3.0-flash IQ3_XXS | 35.6 | - | - | - | - |
| Ling-3.0-flash Sante IQ3_XXS | 33.5 | - | - | 27.0* | - |
| MiMo-V2.6-Flash-MOPD BPW2.5 | 5.4 (from the SSD) | TBD | - | 7.0 (a 25 GiB pool) | - |
| MiMo-V2.5 UD-IQ2_XXS | 5.9 (from the SSD) | TBD | - | 6-9 (single runs) | - |
| MiniMax M3.1 Flash UD-IQ2_XXS | 2.8 (two cards, from the SSD) | TBD | - | 2.7 (a 20 GiB pool) | - |
| Flint 40B-A2.5B | 89.9 (UD-IQ3_XXS) / 65.7 (UD-Q4_K_XL) | 73.2 (UD-Q4_K_XL) | - | 84.4 (UD-Q4_K_XL) | 42.2 (UD-IQ3_XXS) |
| Nemotron 3.5 Lightning 30B-A3B | 71.1 (UD-IQ3_XXS) / 38.7 (UD-Q4_K_XL) | 66.9 (UD-Q4_K_XL) | - | 89.9 (UD-Q4_K_XL) | 25.9 (UD-IQ3_XXS) |
| Gemma 4 26B-A4B | 92.4 (UD-IQ3_XXS) / 75.2 (UD-Q4_K_XL) | 59.3 (UD-Q4_K_XL) | - | 85.2 (UD-Q4_K_XL) | 28.8 (UD-IQ3_XXS) |
| LFM2.5-24B-A2B | 75.9 (UD-Q4_K_XL) | 134.0 | 6.9 | 104.3 | 59.0 (11.9 as an 8 GB Mac) |
| LFM2.5-8B-A1B UD-Q4_K_XL | 305.5 | 161.7 | 16.6 | 185.3 | 43.1 (as an 8 GB Mac) |
| DeepSeek-V4.1-Flash IST-DAS 2.00 bpw | 3.8 (strata-dsv41) |
19.2 (DSpark) | - | 7.4 | - |
| Flash-Next NVFP4 | Strata's own packs: a PC with an NVIDIA card only, not tested in MoMoE |
- Engines: on the PC, Strata's engine runs the Flash-Next and Qwen3.6 families, and
momoe-cudaruns the rest (Ornith-35B onmomoe-cudawould be 63.9). Elsewhere it'smomoe-cudaon the Spark and the Jetson, andmomoe-metalon the Macs. - Checked: every model's output against llama.cpp's, and the server's chat and tool calls (whole, streamed, after the result). For the newer ports (Ling-3.0-flash, MiMo, MiniMax) the check is the KL of the next-token probabilities on llama.cpp's own answer, in their bench/results folders.
- Reading prompts on the PC (Strata's engine, 4K / 16K tokens): Flash-Next IQ3_XXS 1,178 / 2,133 tok/s, the Coder 1,931 / 2,615, Swift 1,196 / 2,163, Unsloth's UD-IQ4_XS 213 / 265 (88 GB, more than the RAM: with ~52 GB free; 108 / 157, writing 14.2, with 36 GB free), the Qwen3.6 family 3,081-3,767 / 3,974-4,521.
- Ling-3.0-flash (and its Sante fine-tune, the same engine): on the PC the CPU computes the experts the card does not hold, and prompts read at 1,189 / 1,494 tok/s (4K / 16K tokens). Its 54-56 GB file has to be in the RAM's file cache: the first answer after another large model ran is slow (9-12 tok/s) until it is again. On the M1 Ultra setup drafts with its MTP layer (without drafting 16.7 tok/s); the plain model there is not measured yet.
- MiMo-V2.6-Flash-MOPD (Xiaomi, September 2026; V2.5's architecture trained further) and MiMo-V2.5 (310B-A15B, a 97 GB file each): on the PC, with 63 GB of RAM, most of the experts are read from the SSD as tokens need them: 5.4-5.9 tok/s writing and 33-35 tok/s reading real text (2048-token chunks read ~66, but leave a 12 GB card too little VRAM: setup keeps 1024). They want ~110 GB of memory to hold the file. Both checked against llama.cpp on the PC, V2.5 on the Spark and the M1 Ultra too, V2.6 on the M1 Ultra (7.0 tok/s writing there, the Spark TBD): bench/results/2026-11-03-mimo25, bench/results/2026-11-03-mimo26.
- GLM-5.3-Flash UD-IQ2_XXS (a 102 GB file) on the PC: its layers split across the two cards, the CPU computing the experts they lack: 4.7 tok/s writing in the server (4.5-5.8 on its own; llama.cpp 4.0-5.4), prompts 82-122 tok/s. Setup picks it on separate cards and on Macs under 100 GB (on the M1 Ultra 25-30% faster than UD-IQ3_XXS in the same memory, server 5.3 tok/s); the Spark keeps UD-IQ3_XXS. Its MTP drafting pays on neither (only where memory holds nearly every expert, as on the Spark): bench/results/2026-11-04-glm53-pc.
- MiniMax M3.1 Flash (428B-A23B, a 135 GB file, sparse attention): its 9.2 GB of other weights do not fit one 12 GB card, so on the PC setup splits the layers across two (the 4070 Ti and a 3060, a 16K context) and the CPU computes the experts the cards lack: 2.8 tok/s writing, as llama.cpp's 2.8-3.1 with every expert on the CPU. The M1 Ultra writes 2.7. Its DSpark drafter would not pay on either (measured: a check of several tokens costs nearly as much as that many single ones). Its license asks a product built on it to say "Built with MiniMax M3.1-Flash". Checked against llama.cpp on both, the Spark TBD: bench/results/2026-11-03-minimax31.
- DeepSeek-V4.1-Flash (249 GB of files with its n-gram table): on a PC upstream Strata's DeepSeek engine, on the Spark and a Mac MoMoE's; checked against each other, the measurements and the memory it needs: docs/DEEPSEEK41.md.
- Details: how each column was measured, drafting per model, and the engines' speed-ups: design/benchmark-notes.md. The raw numbers are in bench/results.
Writing speed in tokens per second. llama.cpp is shown at the best of the placements tried: "N on CPU" is
--n-cpu-moe N (the experts of N layers in RAM); "MoE cache G" is --cpu-moe --moe-cache-mib (every expert in RAM, a
cache of G on the GPU, llama.cpp's own expert cache since 2026-10-07). * = drafting, both sides with the same MTP
layer. "MoMoE" is with setup's config, on the same prompt as llama.cpp's, not the one in the table above, so with
drafting (*) it can differ from it: how many drafted tokens are kept depends on the text (on this chat 85-89%, so
the PC's Flash-Next wrote 65.5-76.7 here against 57.9 above).
| Model | MoMoE | llama.cpp |
|---|---|---|
| PC (RTX 4070 Ti 12 GB) | ||
| Qwen3.8-Flash-Next IQ3_XXS | 65.5-76.7* | 23.1 (MoE cache 5 GiB; its default fit 22.5) |
| Ornith-1.5-35B-A3B IQ4_XS | 131.6-153.5* | 77.8 (MoE cache 8 GiB; its default fit 57.2; drafting 63.5*) |
| Ornith-1.5-122B-A10B UD-IQ3_XXS | 19.2 | 4.62 (48 on CPU) |
| Ling-3.0-flash Sante IQ3_XXS | 33.5 | 27.4 (MoE cache 6 GiB; 38 on CPU 26.7) |
| MiMo-V2.6-Flash-MOPD BPW2.5 | 5.4 | 5.1-5.8 (every expert on CPU; other placements not tried) |
| MiMo-V2.5 UD-IQ2_XXS | 5.9 | 5.7-6.4 (every expert on CPU; other placements not tried) |
| GLM-5.3-Flash UD-IQ2_XXS (+ RTX 3060) | 4.5-5.8 (the 4070 Ti alone: 3.8) | 4.0-5.4 (every expert on CPU, the 4070 Ti; other placements not tried) |
| MiniMax M3.1 Flash UD-IQ2_XXS (+ RTX 3060) | 2.9-3.8 | 2.8-3.1 (every expert on CPU; other placements not tried) |
| Nemotron 3.5 Bolt UD-IQ3_XXS | 29.7 (server 30.0, sampled 30.6) | 26.2 (26.1) |
| Nemotron 3.5 Lightning UD-IQ3_XXS / UD-Q4_K_XL | 71.1 / 38.7 | 65.3 (24 on CPU) / 40.2 (36) |
| Gemma 4 26B-A4B UD-IQ3_XXS / UD-Q4_K_XL | 92.4 / 75.2 | 90.5 at 32K context (4 on CPU) / 54.8 (11) |
| Flint UD-IQ3_XXS / UD-Q4_K_XL | 89.9 / 65.7 | 102.6 (15 on CPU) / 58.5 (30) |
| LFM2.5-24B-A2B UD-Q4_K_XL | 75.9 | 105.1 (10 on CPU) |
| LFM2.5-8B-A1B UD-Q4_K_XL | 305.5 | 305.8 |
| Spark (128 GB shared) | ||
| GLM-5.3-Flash UD-IQ3_XXS | 19.7* (13.3 without) | 10.85 (5 on CPU; --no-host 1 --repack 0 -lm mmap) |
| GLM-5.3-Flash UD-IQ4_XS | 11.7* (9.3 without) | 3.52 (20 on CPU, the same settings; larger than the memory) |
| Flint / Gemma 4 / Lightning UD-Q4_K_XL | 73.2 / 59.3 / 66.9 | 76.0 / 57.5 / 69.6 |
| Jetson (8 GB shared) | ||
| LFM2.5-24B-A2B UD-IQ3_XXS | 6.9 | 4.8 (16 layers' experts on the GPU, the most that fit) |
| LFM2.5-8B-A1B UD-Q4_K_XL / UD-IQ3_XXS | 16.6 / 20.3 | 17.2 / 20.5 |
| M1 Ultra (64 GB) | ||
| Qwen3.6-35B-A3B (Unsloth) UD-IQ4_XS | 129.3* / 92.0 | 150.1* / 90.3 |
| Qwen3.6-35B-A3B (bartowski) IQ4_XS | 117.1* / 95.8 | 146.1* / 96.1 |
| KAT-Coder-V2.5-Dev i-quality-v2 | 104.9* / 91.0 | 106.8* / 89.6 |
| Ornith-1.5-35B-A3B IQ4_XS | 92.7* / 96.1 | 96.2* / 96.1 |
| Nemotron 3.5 Bolt UD-Q3_K_XL | 37.5 | 38.5 (cannot read a prompt: no Metal kernel for 9 experts a token) |
| Flint / Gemma 4 / Lightning UD-Q4_K_XL | 84.4 / 85.2 / 89.9 | 77.8 / 86.7 / 92.1 |
| LFM2.5-24B-A2B UD-IQ3_XXS / LFM2.5-8B-A1B UD-Q4_K_XL | 104.3 / 185.3 | 108.7 / 169-190 |
| M4 Air (32 GB) | ||
| Qwen3.6-35B-A3B (Unsloth) UD-IQ4_XS | 60.8* / 33.2 | 47.6* / 28.9 |
| Ornith-1.5-35B-A3B IQ4_XS | 49.3* / 40.9 | 36.1* / 35.3 |
| Flint UD-IQ3_XXS / UD-Q4_K_XL | 42.2 / 16.7-19.1 (a pool + the SSD) | 41.0 / 20.8 (all on CPU: larger than Metal's limit) |
| Gemma 4 UD-IQ3_XXS / UD-Q4_K_XL | 28.8 / 24.0 | 26.8 / 24.1 |
| Lightning UD-IQ3_XXS / LFM2.5-24B-A2B UD-IQ3_XXS | 25.9 / 59.0 | 26.0 / 58.1 |
| Mac mini (8 GB, macOS 11) | ||
| Flint UD-IQ3_XXS | 8.7 | 2.96 (all on CPU: on the GPU it aborts) |
| Gemma 4 UD-IQ3_XXS | 8.4 (a 2.15 GiB pool) | 6.4 (30 on CPU) |
| LFM2.5-24B-A2B UD-IQ3_XXS | 11.7 | 8.8-10.6 (30-38 on CPU) |
| LFM2.5-8B-A1B UD-Q4_K_XL | 26.0-28.4 | 43.8 (3 on CPU) |
In short:
- Where the experts don't fit the GPU, MoMoE is well ahead. On the PC that holds even against llama.cpp's MoE cache: Flash-Next 3x, Ornith-35B 2x, Ornith-122B 4x, Ling-3.0-flash 1.2x. MoMoE is also ahead with GLM on the Spark (13.3 against 10.85 without drafting, 19.7 with it), LFM2.5-24B on the Jetson, and the three larger models on the Mac mini.
- Where every expert fits in memory, the two are level. That's Flint, Gemma 4 and Lightning on the Spark and the Macs, LFM2.5 on the Jetson, the M1 Ultra and the M4 Air, LFM2.5-8B on the PC, and Qwen3.6 on the M1 Ultra without drafting.
- Where the model is larger than the RAM too, the two are level. MiMo on the PC (97 GB against 63 GB of RAM): both wait for the SSD (MoMoE 5.4-5.9, llama.cpp 5.1-6.4).
- llama.cpp is ahead in a few places. It computes some experts on the CPU from RAM while the GPU holds the rest, which wins for Flint IQ3_XXS and LFM2.5-24B on the PC and for LFM2.5-8B on the Mac mini. Its drafting gains more for Qwen3.6 on the M1 Ultra.
How each was measured (contexts, prompts, runs, llama.cpp builds) and the per-model notes are in design/benchmark-notes.md.
Each with the number it was measured at; details and switches in design/benchmark-notes.md.
- The CPU computes missing experts while the GPU computes its own (
momoe-cuda), instead of copying them over PCIe. Ornith-122B on the PC: 19.2 tok/s, against 11.4 with the copies. - Kept graphs, built once per shape and replayed, as CUDA graphs on NVIDIA. LFM2.5-8B: 120 -> 185 tok/s on the M1 Ultra, 255 -> 305.5 on the PC.
- A wait-for-event thread on Macs, so macOS doesn't park the engine on efficiency cores at a low GPU clock. LFM2.5-8B through the server: 64-69 -> 120-128 tok/s.
- Fewer copies: the delta net's state is written in place and the router's choice is used as it is. Qwen3.6 on Metal: 76.5 -> 92.2 tok/s (llama.cpp 90.3).
- Drafting's embedding rows looked up on the CPU, not the whole table copied to the GPU per draft. Qwen3.6 on the Spark: 59.5 -> 125.2 tok/s.
You need:
- an NVIDIA card, RTX 20 series or newer, or an Apple Silicon Mac;
- enough RAM for the model;
- Python 3 (3.10 or newer on a Mac:
brew install python); - a C++ compiler and, with NVIDIA, the CUDA toolkit (on a Mac:
xcode-select --install). Setup checks your machine and offers to install what is missing.
git clone https://github.com/aflin/momoe.git
cd momoe
./momoe setup # check the machine, choose a model, compile the engine, download, start
sudo ln -s "$PWD/momoe" /usr/local/bin/momoe # optional: run momoe from any folderSetup asks for a name for the model's config (for example ornith-8gb) and starts the server in the background.
./setup.sh --check lists what fits your PC. The Qwen3.6-family files are described in
docs/QWEN36.md and docs/QWEN35MOE_GGUFS.md.
Setup also installs a GGUF it does not list, when its architecture is one the engines run: a fine-tune, or another uploader's quantiza 8BD0 tion of a listed model. Name the Hugging Face repository and the quantization, or give a file you have:
momoe setup --gguf hf:bartowski/Qwen3.8-Flash-Next-GGUF:IQ3_XXS # bartowski's quantization of Flash-Next
momoe setup --gguf hf:bartowski/Qwen3.8-Flash-Next-GGUF # lists the repository's quantizations
momoe setup --gguf /data/models/Model-IQ3_XXS-00001-of-00002.gguf- Setup reads each file's header before downloading anything. It stops early if the architecture is not one the engines run, a part is missing, or (on Strata's engine) the experts are in a format it has no kernels for.
- A gated repository (its page asks you to accept terms) needs a Hugging Face read token:
HF_TOKEN,hf auth loginor--hf-token-file; without one, setup asks for it. The token is sent to huggingface.co only. - A Flash-Next fine-tune with an MTP draft file beside it (
...-MTP-draft.gguformtp-<model>-Q8_0.gguf) drafts with its own head; without one, or when the head is the original's, it uses the original's draft layer. Files with the head built in (-MTPquantizations) are refused: take the same quantization without-MTP. - It runs with the settings of the nearest listed family, and a fine-tune with no image encoder of its own uses the original's.
Architectures: Qwen3.8-Flash-Next (qwen4exp) and the Qwen3.6-35B-A3B family (qwen35moe) on a PC with its own NVIDIA or AMD card; nemotron_h_moe, glm5-next, lfm2moe, flint, gemma4, bailingmoe3, mimo2 and minimax-m3 everywhere. Details: docs/INSTALL.md.
momoe list # the configs in config/
momoe start ornith-8gb # start one (.json can be left off)
momoe status # what is running, and where
momoe log # follow the server's and the engine's logs (Ctrl+C ends)
momoe run claude # Claude Code, connected to it (also: codex, opencode)
momoe run unsloth # Unsloth Studio with a "MoMoE" provider
momoe run lmstudio # LM Studio, with the plugin it needs installed and set up
momoe run claude --config ornith-8gb # start that config first if nothing is running
momoe stop
momoe remove ornith-8gb # delete a model: its GGUF, its pack, the config and its logs (asks first)momoe runstarts a server if none runs, asking which config if there are several. It gives the tool its settings for that run only and installs nothing: a missing tool is named, with how to install it.momoe remove(ordelete) lists what it will delete and asks first (-y: no question;--dry-run: only the list).- A model or pack that other configs also use stays, and it asks whether those go too (
--wipe: they do, without asking). - It touches nothing outside
models/,packs/,config/andlogs/. - Given a folder in
models/(a download without a config, say), it deletes that folder and its pack.
- A model or pack that other configs also use stays, and it asks whether those go too (
- Pictures (off unless asked for):
momoe setup --vision yesdownloads the model's image encoder and builds it, for every family that has one (all but Flint, Lightning and LFM2.5). Pictures work in a message and in a tool result (Claude Code's Read of an image). The encoder runs on the GPU on a Mac and a DGX Spark (1-3 s a picture) and on the CPU with a separate card (1-2 s; ~9 s for the Qwen3.6 family, which needs 1,024 tokens a picture to read text right). Checked on the RTX 4070 Ti (against llama.cpp's answers), the M1 Ultra and the Spark. On a Mac the encoder's memory comes out of the expert pool. - Other apps: any OpenAI- or Anthropic-compatible client can use
http://127.0.0.1:<port>/v1;momoe statusshows the port. The server listens on 127.0.0.1 only. To reach it from another device, give--hosttogether with--api-key, for examplemomoe start ornith-8gb --host 0.0.0.0 --api-key KEY. They apply to that start only; add--saveto keep them in the config. - Every option of
momoe, setup, the config file and the engines: docs/OPTIONS.md.
Every log is in logs/:
logs/<config>.logis the engine's: loading, the expert cache, one block of numbers per request. The engine logs keep growing. Itsexpertslines say how often the experts a request needed were already in GPU memory: docs/EXPERT_STATS.md.logs/server.logis the server's, started again with eachmomoe start. It writes a line a second while it answers, and adone:line per answer with its speed and the expert cache's hit rate.
momoe log # the last 20 lines of the server's and the engine's logs, then follow them
momoe log server # one log: server, engine, unsloth or lmstudio (all: the four)
momoe log -n 100 # start from the last 100 lines
momoe log --no-follow # print the last lines and stopTwo NVIDIA cards can share a model in two ways. Setup writes either with --gpus a,b (and --helper-tier), or by
hand in the config: both cards in "gpu", the main card first.
-
The layer split (
"gpu": [a, b],"layer_split": "auto"or the first layer of each later card): each card runs its own layers, the CPU computing the experts neither card holds. Setup splits GLM-5.3-Flash and MiniMax M3.1 Flash this way by itself across two cards under 16 GB each (measured on an RTX 4070 Ti + RTX 3060: GLM-5.3-Flash UD-IQ2_XXS 4.5 tok/s against 3.8 on the 4070 Ti alone). -
The helper tier (
"args": [..., "--helper-tier"]): the second card is a second level of expert cache under the first.- The main card runs the model as it would alone.
- The helper card holds the next experts in line and computes them itself.
- The RAM holds the rest.
Each expert is in one place only: what the main card lets go moves down to the helper, and the helper's evictions move to RAM. With Flash-Next it writes and reads faster, and with IQ3_XXS the PC keeps more memory free (the measurements below).
--helper-gib Glimits the helper's VRAM for experts; every setting it turns on can be changed on its own (docs/SECOND_GPU.md).
Measured with Flash-Next on an RTX 4070 Ti (main) + RTX 3060 12 GB (helper, also driving the desktop), Ryzen 9 5950X,
64 GB of RAM (~52 GB of it available before each run), tools/strata_bench.py, three runs per test, 2026-10-24:
| IQ3_XXS, 64K: 4070 Ti alone | --helper-tier |
UD-IQ4_XS, 256K: 4070 Ti alone | --helper-tier |
|
|---|---|---|---|---|
| experts on the 3060 | 5,525 (8.90 GiB) | 3,939 (8.89 GiB) | ||
| the PC's available memory, loaded | 8.6 GB | 19.0 GB | 10.4 GB | 8.8 GB |
| writes, short chat | 55.6 | 70.0 | 18.4 | 21.7 |
| writes, agent session | 63.9 | 78.7 | 18.9 | 25.3 |
| reads, 4K-token prompt | 1,145 | 1,479 | 213 | 407 |
| reads, 16K-token prompt | 2,113 | 1,856 | 265 | 576 |
| agent turn, first token | 3.37 s | 2.48 s | 18.4 s | 8.4 s |
With the helper Flash-Next writes 18-34% faster; IQ3_XXS reads long prompts ~12% slower, and UD-IQ4_XS (88 GB, the
larger share of it read from the SSD on one card: the helper holds experts that would otherwise come from there) reads
prompts about twice as fast.
The answers are not bit-identical to one card's. Until 2026-10-24 the helper tier ran without the route tail skip
(missed experts at the bottom of a token's routing are skipped, on by default on one card; docs/DETAILS.md); with it,
agent turns on UD-IQ4_XS went from 19.2 to 23.7 tok/s (tools/agent_turns_bench.py, 8 turns, 2 rounds).
The Qwen3.6 family (Qwen3.6, Ornith-1.5-35B, KAT-Coder) runs on two cards too, but only when asked: --gpus a,b
(a layer split) or --gpus a,b --helper-tier. A second card is no clear gain there, because one 12 GB card already
holds most of its experts. Measured with Ornith-1.5-35B IQ4_XS on the same two cards (the 3060 also holding ~3.9 GB of
another program), tools/strata_bench.py, 2026-10-23, tok/s:
| one card | layer split | --helper-tier |
|
|---|---|---|---|
| 32K context: short chat / agent session | 126.3 / 136.6 | 128.3 / 106.6 | 130.8 / 130.8 |
| 32K context: reads a 16K-token prompt | 4,475 | 3,569 | 3,513 |
| 256K context: short chat / agent session | 103.5 / 114.3 | 121.7 / 110.3 | 127.2 / 118.0 |
| 256K context: reads a 16K-token prompt | 4,459 | 3,712 | 3,497 |
With a large context (where the KV cache takes the card's room for experts) the helper writes short chats ~20% faster; agent sessions are about the same and long prompts ~20% slower. (256K: the English draft vocabulary, for the split's room.) Its helper does not compute prompt rows for this family: the primary card reads them.
With Ornith-1.5-122B the helper tier also works with the CPU computing the experts neither card holds, but it is slower: 12.0-12.2 tok/s against 19-21 on the 4070 Ti alone. More in design/benchmark-notes.md.
| Folder | What |
|---|---|
config/ |
the model configs setup writes (yours; not in git) |
models/, packs/, mtp/ |
the downloaded models and what setup prepares from them (not in git) |
engine/ |
Strata's compiled engine (momoe, momoe-vision) |
build-cuda/, build-metal/ |
MoMoE's compiled engines (momoe-cuda, momoe-metal) |
build-vision-momoe*/ |
their image encoder (momoe-vision), when pictures are on |
logs/ |
every log (above) |
.momoe/ |
the server's state (which config runs) |
design/ |
MoMoE's design notes: what was measured, chosen or rejected |
bench/results/ |
the measurements behind the tables, with the scripts |
docs/ |
Strata's documentation. Most of it applies; it says "Strata" where MoMoE has its own name |
- MoMoE: its CUDA and Metal engines (
momoe-cuda,momoe-metal: src/metal), built on ggml, and the changes to Strata's engine, server and installer (the Qwen3.6 family's path began as #1402). - Strata, by Niko1221 and the Strata contributors: the engine
momoeruns (Strata's, with MoMoE's additions), and the server, the installer, the documentation and the paper MoMoE's grew from. - Changes taken from other Strata forks and open pull requests, credited to their authors:
- Asofwar/Strata by Asofwar: the Qwen3.6 decode-path speedups, the fast 256 x 8 router on CUDA and the control-vector fix (three commits, under their own name).
- #2674 by Scott Page (page1): the server logs a reply that ended after its thinking with no text and no tool call.
- llama.cpp / ggml, by the ggml authors: the ggml library under
momoe-cudaandmomoe-metal(in src/metal/ggml, with MoMoE's additions listed in itsREADME.momoe), the model code their graphs for Nemotron, GLM-5.3, LFM2.5, Flint, Gemma 4, Ling-3.0-flash, DeepSeek-V4.1, MiMo and MiniMax M3 follow, and the GGUF format, the quantization kernels and the image encoder Strata builds on. - The models' authors and quantizers listed above: the Qwen team, ISTA-DASLab, ukisai, Unsloth, bartowski, Ornith AI, Myric, NVIDIA, Z.ai, Liquid AI, Recursive, Google, inclusionAI, DeepSeek, Xiaomi, AesSedai and MiniMax.
MIT: see LICENSE, which holds Strata's copyright notice and MoMoE's own. The copy of ggml in src/metal/ggml keeps its own MIT license (the ggml authors), beside it.