A local AI server for low-bit, very high quality Sushi quants, using custom Sushi Trellis Quant (STQ) or EXL3-Affine hybrid converter. It is a detached fork of ddalcu's mlx-serve. Sushi runs stand-alone and also as a guest engine inside mlx-serve, Apple Silicon M1+ and macOS 26.2 or later are required.
Usage see recommended launch commands and coding agent usage. Which pack suits your Mac, and running a bigger one from the SSD: Choosing a pack and streaming.
- DeepSeek-V4.1-Flash-1017-Sushi-STQ-1.75bpw (pack revision 2.0, requires 96 GB+ and sushi 1.5.0 or later, experts streamed, text and image input)
- Qwen3.8-Flash-Next-Sushi-2bpw (requires 48 GB+, or 32 GB streamed)
- Qwen3.8-Flash-Next-Sushi-STQ-2.75bpw (pack revision 2.0, requires 64 GB+ and sushi 1.5.1 or later)
- Qwen3.8-Flash-Next-Sushi-4bpw (requires 96 GB+, or 64 GB streamed)
- GLM-5.3-Flash-Sushi-2bpw (requires 96 GB+, text, image and video input)
- GLM-5.3-Flash-Sushi-2.4bpw (requires 128 GB, text, image and video input)
Unsupported: MiMo-V2.6-Flash-Sushi-2.3bpw still loads, but from 1.5.1 releases do not test, benchmark or tune it. 1.5.0 is the last release that supports it.
- DeepSeek-V4.1-Flash-1017-Sushi-STQ-1.75bpw, pack revision 2.0, scores KLD 0.1576 (top-1 agreement 89.13%) on the eight-category reading, down from 0.1843 in revision 1.0 and below oQ2.5e's 0.1717 at 2.96 bits per weight
- MiMo-V2.6-Flash-Sushi-2.3bpw scores KLD 0.0860 (top-1 agreement 91.95%)
- GLM-5.3-Flash-Sushi-2.4bpw scores KLD 0.0742 (top-1 agreement 90.33%)
- GLM-5.3-Flash-Sushi-2bpw scores KLD 0.1037 (top-1 agreement 88.18%)
brew install beamivalice/tap/sushiUpdate with brew upgrade sushi.
sushibar puts the server in the menu bar: the generation speed in tok/s as its title, and a menu with a live status card, Start/Stop and Load/Unload buttons (Start brings the server up empty; Load brings back the last model), a port switcher, its pages, and Log, Update and Quit buttons (Log follows the server log live; hold ⌥ for Quit & Stop).
- The server listens on
127.0.0.1:12345(Settings has a Default port). - Support speculative decoding: MTP for Qwen and MiMo, DFlash2 for GLM.
- 8-bit KV cache are on by default.
GPU memory in GiB to serve one prompt that fills the whole context (8-bit KV, MTP on with --mtp-head-kv-quant; GLM
with its DFlash2 assistant and vision tower). The prompt cache and the n-gram table stay on the SSD and are not counted.
The Qwen rows are at the default f32 GatedDeltaNet state; --qwen-gdn 16 (bf16) takes about 0.2 GiB off each Qwen cell.
| pack | weights only | 128k | 256k | 512k | 1M |
|---|---|---|---|---|---|
| Qwen3.8-Flash-Next-Sushi-2bpw | 35.2 | 40.3 | 43.2 | 48.2 | 58.4 |
| Qwen3.8-Flash-Next-Sushi-STQ-2.75bpw | 44.9 | 50.0 | 52.8 | 57.9 | 68.1 |
| Qwen3.8-Flash-Next-Sushi-4bpw | 63.9 | 69.0 | 71.9 | 76.9 | 87.1 |
| GLM-5.3-Flash-Sushi-2bpw | 78.3 | 82.6 | 83.8 | 86.0 | 90.5 |
| GLM-5.3-Flash-Sushi-2.4bpw | 89.8 | 94.1 | 95.2 | 97.5 | 102.0 |
| MiMo-V2.6-Flash-Sushi-2.3bpw | 89.7 | 93.7 | 95.8 | 99.9 | 108.2 |
| DeepSeek-V4.1-Flash-1017-Sushi-STQ-1.75bpw ‡ | 108.0 | 110.0 | 110.3 | 111.0 | 112.5 |
A context fits when its number is below the GPU limit you set with sudo sysctl iogpu.wired_limit_mb=<N> vm.global_user_wire_limit=<N × 1048576> vm.user_wire_limit=<N × 1048576> (all three, always: why). Max context is
the largest multiple of 8192 tokens that fits, at 8-bit / 4-bit KV, with 256 MiB spare and capped at 1M (GLM has no
4-bit KV):
| pack | 48 GB † (43,000 MB) | 64 GB † (59,000 MB) | 96 GB (90,000 MB) | 128 GB (120,000 MB) |
|---|---|---|---|---|
| Qwen3.8-Flash-Next-Sushi-2bpw | 200k / 304k | 976k / 1M | 1M / 1M | 1M / 1M |
| Qwen3.8-Flash-Next-Sushi-STQ-2.75bpw | — | 488k / 824k | 1M / 1M | 1M / 1M |
| Qwen3.8-Flash-Next-Sushi-4bpw | — | — | 1M / 1M | 1M / 1M |
| GLM-5.3-Flash-Sushi-2bpw | — | — | 696k / — | 1M / — |
| GLM-5.3-Flash-Sushi-2.4bpw | — | — | — | 1M / — |
| MiMo-V2.6-Flash-Sushi-2.3bpw | — | — | — | 1M / 1M |
| DeepSeek-V4.1-Flash-1017-Sushi-STQ-1.75bpw ‡ | — | — | 1M / — | 1M / — |
† At --qwen-gdn 16 (bf16 GatedDeltaNet state), which the 32, 48 and 64 GB launch lines use. The f32 default gives
less: 192k / 288k at 48 GB, and 968k / 1M (Sushi-2bpw) and 480k / 800k (Sushi-STQ-2.75bpw) at 64 GB.
‡ DeepSeek-V4.1-Flash streams its routed experts from the SSD on 96 and 128 GB Macs. It runs 1M at --ssd-budget-gb 108 on a 128 GB Mac with sudo sysctl iogpu.wired_limit_mb=125000 vm.global_user_wire_limit=131072000000 vm.user_wire_limit=131072000000 and at 80 on a 96 GB Mac with sudo sysctl iogpu.wired_limit_mb=93000 vm.global_user_wire_limit=97517568000 vm.user_wire_limit=97517568000 (usage). Always raise the two user-wire limits with the GPU limit: without them the system slows down and can become unresponsive.
A pack bigger than your Mac's GPU memory can still run: --ssd-budget-gb N keeps N GiB of the model on the GPU and
reads the rest from the SSD as it runs, with the same answers and lower speed. Which pack to pick for your Mac, what
the GPU limit is, how to size the budget and when streaming is worth it, with measured examples:
Choosing a pack and streaming.
export SUSHI_KEY=<pick a secret>
sushi serve --model <pack> --host 0.0.0.0 --api-key-env SUSHI_KEYOther computers then reach http://<your-mac-ip>:12345/v1 (ipconfig getifaddr en0 prints the IP) and send the key
as Authorization: Bearer <key>. If SUSHI_KEY is unset or empty, the server runs with no key.
| Sushi 🍣 1.4.0 | M5 Max 128 GB | 4k | 8k | 16k | 32k | 64k | 128k |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 Flash-Next-STQ-2.75bpw | Prefill | 1,122 | 1,993 | 2,049 | 1,976 | 1,906 | 1,877 |
| Decode | 168.6 | 151.1 | 146.1 | 146.8 | 151.0 | 138.2 | |
| Qwen 3.8 Flash-Next-4bpw | Prefill | 2,079 | 2,379 | 2,326 | 2,231 | 2,147 | 2,120 |
| Decode | 146.2 | 150.1 | 136.2 | 149.8 | 153.5 | 122.0 | |
| GLM 5.3 Flash-2.4bpw | Prefill | 928 | 775 | 814 | 805 | 688 | 573 |
| Decode | 71.5 | 69.4 | 67.3 | 65.8 | 71.8 | 63.8 | |
DeepSeek 4.1 Flash-STQ-1.75bpw (--ssd-budget-gb 108) |
Prefill | 707 | 1,024 | 998 | 993 | 950 | 829 |
| Decode | 76.6 | 84.2 | 76.2 | 85.3 | 78.1 | 63.7 |
| Pack | Thinking | Score | Points | Rating | Max weights in RAM |
|---|---|---|---|---|---|
| Qwen STQ 2.75bpw | medium | 94 | 182/194 | 5★ | 48 GB |
| DeepSeek STQ 1.75bpw | high | 92 | 178/194 | 5★ | 129 GB (108 stream) |
| Qwen 4bpw | medium | 90 | 174/194 | 5★ | 68 GB |
| GLM 2.4bpw | high | 90 | 175/194 | 5★ | 98 GB |
| MiMo 2.3bpw | on | 89 | 172/194 | 4★ | 103 GB |
| Qwen 2bpw | medium | 88 | 170/194 | 3★ | 38 GB |
MIT, for sushi and the mlx-serve code it forks (LICENSE); ported kernels and vendored code are listed in NOTICE. The Qwen packs follow the Qwen Community License and the MiMo pack Xiaomi's MIT license, stated on each Hugging Face page.