Engines we build
Eighteen engines,written from scratch.
Most LocalAI backends wrap somebody else's engine. These were written from scratch, each one because the thing we needed was a multi-gigabyte Python install, or closed, or nobody had built it yet. What ships instead is a binary and a GGUF file, checked against the reference implementation in CI, running on the machine you already own.
Hearing
Turning sound into something a model can act on, words first and then everything else in the room.
C++17
parakeet.cpp
Transcribe a meeting on a laptop CPU and be finished before whisper.cpp has cleared the first minute.
- About 27x faster than whisper.cpp turbo on CPU, and about 12x on GPU
- WER 0 against NVIDIA NeMo on every published checkpoint, so the transcript is identical
- Cache-aware streaming with end-of-utterance detection, for live audio
- Ten checkpoints, from 110M to 1.1B, and 40 or more locales on the streaming multilingual model
C++17
moss-transcribe.cpp
Get the transcript, the speaker labels and the timestamps out of a single pass, then export straight to srt or json.
- 1.58x to 1.78x faster than PyTorch on CPU, on about 1.5x less memory
- Byte-identical transcript against the reference, cosine 1.0 component by component
- 3.4 GB at f32 down to 511 MB at q4_k, still byte-identical through q5_k
C++17
ced.cpp
Let the model hear a smoke alarm, a dog or breaking glass, not only the words somebody typed.
- 527 AudioSet sound classes, multi-label, tagged in about 55 ms
- 6 MB on disk at ced-tiny q8_0, 111 MB at ced-base
- About 1.25x faster than PyTorch at f32 on half the memory
- Works over REST and live over the realtime websocket
C++
LocalVQE
Keep a voice loop usable in a real room, with the echo, the noise and the reverb removed before the model ever hears it.
- Echo cancellation, noise suppression and dereverberation in one pass
- Streaming and causal, 16 ms latency, 5x realtime on a desktop CPU
- From a 17 KB linear filter to a 19 MB joint model, pick what your CPU can afford
- About 21x realtime on a single Raspberry Pi 5 core
Voice
Speech coming back out, in a voice you chose or one you cloned from a few seconds of audio.
C++17
moss-tts.cpp
Clone a voice from a short reference clip and have it read anything back at 48 kHz stereo.
- About 1.9x faster per frame than PyTorch on CPU, both at fp32
- Codec decode matches the reference at 114.9 dB SNR
- 48 kHz stereo out of MOSS-TTS-Local v1.5, 12 codebooks
- Flat C API as well as a CLI, so it embeds anywhere
C++17
magpie-tts.cpp
Ship multilingual speech from one GGUF that already carries the codec, the tokenizer and the pronunciation dictionaries.
- 63x faster than the NeMo reference at f32, 73x at q8_0
- 5 named voices, 9 languages plus 3 Arabic variants
- 541 MB at q4_k, everything bundled in the single file
- Deterministic from a seed, and parity holds to 3.6e-5 on the full decode
C++
vibevoice.cpp
Read a multi-speaker script in cloned voices, and transcribe long recordings back with speaker labels, from the same binary.
- Voice cloning from roughly 5 seconds of reference audio
- Long-form ASR with diarization on the same engine as the synthesis
- 11 GB down to 6.8 GB at Q8_0 with no measurable recall loss
- Realtime 0.5B, 1.5B and a 7B ASR model
CExperimental
voxtral-tts.c
Run a 4B speech model with nothing but a C compiler and libm, reading the weights straight off the safetensors file.
- Pure C, no dependency beyond the C standard library and math
- BF16 weights read from mmap, no conversion step
- 20 preset voices across 9 languages, 24 kHz output
- Optional BLAS, Apple Accelerate, NEON and CUDA paths
Identity
Working out who is in front of the microphone or the camera, and whether they are really there.
C++17
voice-detect.cpp
Tell who is speaking, and read their age, gender and mood, without an onnxruntime install anywhere near it.
- Embedding cosine 0.9999 or better against the reference, often exactly 1.0
- 5.4x lower peak memory than the Python path, 62 MB against 334 MB
- Six model families, ECAPA-TDNN and WeSpeaker through ERes2Net and CAM++
- Verification, identification against a registry, plus age, gender and emotion
C++17
face-detect.cpp
Detect, recognise and verify a face, and catch a photo held up to the camera, all from one shared library.
- Boxes and landmarks land within 1 pixel of insightface
- Recognition embedding cosine 1.000000 against the reference
- Detect, align, recognise, demographics and anti-spoofing in one pipeline
- The yunet-sface pack is Apache-2.0, so it is usable commercially
Vision
Finding things in an image and naming them, including things nobody trained a class for.
C++17
locate-anything.cpp
Ask for the red mug on the left in plain words and get coordinates back, not a caption.
- 1.66x to 3.09x faster than the official PyTorch on CPU
- Identical detections, IoU 1.000 against the reference
- At q8_0 it is about 4.8x faster than PyTorch f32 and still box-identical
- 9.2 GB at f16, 4.7 GB at q4_k, and an annotated PNG out of the box
C++17
rf-detr.cpp
Get boxes and instance masks at COCO quality out of any image, with no PyTorch anywhere in the process.
- 11 variants, 5 detection and 6 segmentation, from Nano to 2XLarge
- About 9% faster than PyTorch on CPU at F16, and 1.86x smaller
- Mean mask IoU 0.99 against PyTorch on the small segmentation variants
- 44 published GGUFs, F32 through Q4_K
Space
Reading distance, camera pose and shape out of ordinary photos, with no rig and no capture setup.
C++17
depth-anything.cpp
Turn one ordinary photo into distance in metres, a camera pose and a point cloud you can open in a 3D viewer.
- 1.31x faster than PyTorch on CPU at q8_0, in half the memory
- Loads about 6.7x faster, 112 ms against 749 ms
- 99 MB at q4_k, and correlation 1.0 with the reference component by component
- Exports to glb, COLMAP and PLY, plus confidence and a sky mask
C++
free-splatter.cpp
Turn a handful of snapshots into a 3D Gaussian scene with no camera poses, no rig and no GPU.
- 0.22 s per forward pass on Vulkan, against 1.37 s for the PyTorch reference on CUDA
- 14 s on 12 CPU threads, roughly 4x the reference, with no GPU at all
- Pose-free, so ordinary photos are enough
- One 3D Gaussian per pixel, ready for any splat viewer
C++
trellis2.cpp
Drop in one image and get back a watertight textured mesh you can hand straight to a 3D tool.
- Single image to GLB with PBR materials, all inference in C++
- Prebuilt f16 GGUFs, so no safetensors download and no conversion
- Flat C ABI plus a Go demo server with a browser mesh viewer
- Metal on by default on Apple, CUDA and CPU elsewhere
Text
Serving language models, and cleaning what goes into them before it leaves the machine.
C++20In development
vllm.cpp
Serve a language model with vLLM's throughput from a 66 MiB binary instead of a 9 GB virtualenv.
- 66 MiB to install, against 9.1 GiB for a vLLM environment
- Continuous batching, paged KV cache, prefix caching and speculative decoding
- 25 or more architectures, gated token for token against a pinned vLLM oracle
- CPU, CUDA, Metal and Vulkan from the same source
C++
privacy-filter.cpp
Catch names, addresses and card numbers on the machine, before any of it reaches a model or a log.
- 7.7x faster than HF Transformers on an 8k token document, on CPU
- Runs flat to 131k tokens on GPU where HF runs out of memory at about 16k
- Exact UTF-8 byte offsets for every span it finds
- 360 tokens a second on a Raspberry Pi 5, on-device
Data
The storage and quantization work that decides what actually fits on your hardware.
Go
local-store
Get vector search inside LocalAI with nothing to deploy, nothing to configure and no second service to run.
- Ships in-tree and is the default, so embeddings work on a fresh install
- Exact cosine similarity, zero configuration
- Backs RAG, the face and voice registries and the semantic router cache
- Swap in valkey-store per request when you need durability
Shell
apex-quant
Fit a 35B mixture-of-experts model on a card you already own, and watch it run faster than the full-size build.
- 64.6 GB down to 12.2 GB, at 74.4 tokens a second against 30.4
- APEX Quality beats F16 perplexity at a third of the size
- Ordinary GGUF files, so stock llama.cpp opens them unpatched
- 201 builds already sitting in the LocalAI gallery
Nothing in that modality yet.
The rule we hold them to
Every engine here is gated against the framework it replaces, on the same input, on the same machine. That means a transcript identical to the reference, boxes that land on the same pixels, or a waveform inside a stated tolerance. Speed work comes after that, and the numbers on this page come out of each engine's own benchmark suite.