8000 GitHub - flyingnobita/llml: Terminal UI for discovering local GGUF and safetensors models, detecting llama.cpp or vLLM runtimes, and launching them with saved presets. Β· GitHub
[go: up one dir, main page]

Skip to content
flyingnobitaPublic

About

Terminal UI for discovering local GGUF and safetensors models, detecting llama.cpp or vLLM runtimes, and launching them with saved presets.

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

398 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

LLM Launcher (llml)

Go License: MIT

LLM Launcher demo

LLM Launcher (llml) is a TUI for people who already have models on disk and are tired of reconstructing launch commands from shell history.

It scans your local filesystem for GGUF, Hugging Face-style safetensors, and NInfer (.ninfer) models, plus oMLX and Splash models on Apple Silicon, detects installed runtimes (llama.cpp, vLLM, Ollama, KoboldCpp, NInfer, mlx-lm, mlx-vlm, oMLX, and Splash), plus the Strata engine packs it finds in a Strata install, and lets you save named parameter profiles per model β€” so the command that worked last time is always one keystroke away.

Browse local models. Detect the right runtime. Launch with one key.

Works alongside llama.cpp, vLLM, Ollama, KoboldCpp, NInfer, mlx-lm, mlx-vlm, oMLX, Splash, and Strata β€” not a replacement for them. llml is the profile manager for whatever backend you already run.

Note

Importable profiles are available at llml.dev. The catalog is actively curated β€” new profiles are added regularly. Watch the catalog repo (Watch β†’ Custom β†’ Releases) to get notified when new ones land.

✨ Features

  • Model discovery β€” auto-scans common paths for GGUF files, safetensors model directories, and NInfer artifacts (including the models/ directory of a configured NInfer checkout), models in oMLX's model directories, Splash bundles in the Hugging Face cache, and Strata engine configs (strata-<model>.json) in a Strata install; add extra roots via LLML_MODEL_PATHS and/or config.toml. Results are cached under {UserConfigDir}/llml/cache/models.toml so the next launch can skip the filesystem walk when the cache is still valid.
  • Runtime detection β€” finds installed llama-server, vllm, koboldcpp, ninfer-serve, mlx_lm.server, mlx_vlm.server, omlx, and splash binaries, a Strata install's serve/server.py, and installed ollama plus the configured Ollama host, then maps each model to its compatible runtime. GGUF models can use llama.cpp or KoboldCpp, safetensors models vLLM, mlx-lm, or mlx-vlm, via profile selection, and Strata engine packs run on Strata.
  • Named parameter profiles β€” save multiple profiles per model (e.g. fast-laptop, quality, api-8080), each storing runtime args, env vars, port, and context settings. The active profile is always one key away.
  • Profile export β€” share your parameter profiles with others via the TUI (E) or CLI (llml export). Profiles are written to a portable TOML file matching the same schema the llml-import skill reads. Filter by model or profile name, toggle individual profiles or entire model groups, and handle file collisions (overwrite or auto-suffixed save).
  • Portable profile format β€” the shared import/export contract is documented in docs/profile-format.md and used by the repo-managed llml-import agent skill.
  • One-keystroke launch β€” s 10BC8 elect a model, select a profile, press R. The generated command is shown before execution and server output streams directly in the TUI.
  • Persistent status and alert history β€” long-running work such as Ollama preloads stays visible in a persistent status line, while warnings and errors remain inspectable in a dedicated alert-history pane.
  • Ollama preload flow β€” Ollama models are discovered via the Ollama API. Pressing R starts ollama serve if needed, then preloads the selected model into the shared Ollama service with keep_alive: -1.
  • Zero required setup β€” common model directories and binary locations are checked automatically; configure only what differs from the defaults.

πŸš€ Quick start

Runtime Requirements

  • Runtime engine (at least one): llama.cpp (llama-server) or KoboldCpp (koboldcpp) for GGUF models, vLLM (vllm), mlx-lm (mlx_lm.server), or mlx-vlm (mlx_vlm.server) for safetensors models (the MLX runtimes on Apple Silicon and Linux), NInfer (ninfer-serve) for .ninfer artifacts on Linux, oMLX (omlx) or Splash (splash) on Apple Silicon, Strata for its installed engine packs on Windows and Linux (NVIDIA / AMD), and/or Ollama (ollama) for Ollama models are installed (see Runtime Engines).
  • Models in default scan locations, or configure custom roots with LLML_MODEL_PATHS (see Model Discovery).

Install

Pick one path; you only need a single install method.

One-line installer (macOS/Linux)

Install the latest version of llml directly to $HOME/.local/bin/ (or /usr/local/bin/ if run with sudo):

curl -fsSL https://llml.dev/install.sh | sh

Go (go install)

Requires Go 1.26+. Ensure $(go env GOPATH)/bin is on your PATH.

go install github.com/flyingnobita/llml/cmd/llml@latest

Homebrew

brew tap flyingnobita/tap
brew install --cask llml

Upgrade later with brew upgrade --cask llml.

The macOS binary is not signed or notarized by Apple. The cask clears the quarantine flag Homebrew sets on downloads, so macOS opens it without asking. If you installed 0.8.0 or earlier and macOS says "llml" Not Opened, run brew upgrade --cask llml, or clear the flag yourself:

xattr -d com.apple.quarantine "$(brew --prefix)/Caskroom/llml/$(brew list --cask --versions llml | cut -d' ' -f2)/llml"

Scoop

llml is published to the maintainer bucket, not the default Scoop main bucket.

scoop bucket add flyingnobita https://github.com/flyingnobita/scoop-bucket
scoop install flyingnobita/llml

Upgrade later with scoop update llml.

Winget

After the package manifest merges into the public Windows Package Manager catalog:

winget install --id FlyingNobita.llml

Upgrade later with:

winget upgrade --id FlyingNobita.llml

Pre-built binaries

For each GitHub release, archives are published for Linux and macOS (tar.gz) plus Windows (zip). Names follow GoReleaser’s pattern, for example llml_1.2.3_Linux_x86_64.tar.gz, llml_1.2.3_Darwin_arm64.tar.gz, or llml_1.2.3_Windows_x86_64.zip (adjust version and OS/arch to match your download). Extract the llml binary or llml.exe.

# Example: Linux x86_64 β€” use the archive name from the release you downloaded
tar -xzf llml_1.2.3_Linux_x86_64.tar.gz
chmod +x llml

On macOS, an archive downloaded with a browser is quarantined, and macOS refuses to open the unsigned binary ("llml" Not Opened). Clear the flag after extracting with xattr -d com.apple.quarantine llml. Archives fetched with curl, including by the install script, are not quarantined.

Install on your PATH if you like (Linux/macOS/WSL):

sudo mv llml /usr/local/bin/llml

No-sudo option:

mkdir -p "$HOME/.local/bin"
mv llml "$HOME/.local/bin/llml"

(Ensure ~/.local/bin is on your PATH.)

Verify the download against llml_<version>_checksums.txt on the release page if you rely on checksums (for example llml_1.2.3_checksums.txt).

Build from source

Build Requirements

git clone https://github.com/flyingnobita/llml.git
cd llml
go build -o llml ./cmd/llml

To install a development build from your clone, run mise run install-dev. It builds to ~/.local/bin/llml-dev, so it never shadows a released llml installed with Homebrew or the install script.

llml --version (and the footer of the ? shortcuts panel) names the running build. A release prints its version, such as 0.7.1. Any other build prints its commit, the commit's date, and modified if the tree had uncommitted changes, such as dev (16728e6, 2026-09-28, modified), so a stale build is easy to spot.

Start

llml

llml will automatically scan common locations for models and binaries. If your setup is non-standard (e.g., binaries not on PATH or models in custom folders), see the Configuration section to point the app to the right directories. Select a model in the UI and press R to launch.

⌨️ Usage

Key Action
hjkl/↑↓←→ Move selection; horizontal scroll when the path column is wider than the terminal
E Open profile export modal β€” select profiles by model group, filter by name or backend, and save to a portable TOML file you can share across machines
r Reload [runtime] from config.toml and re-detect binaries (no model rescan)
S Full model filesystem rescan; refresh the discovery cache (cache/models.toml)
R Run server (split view: table + log pane)
ctrl+R Run server full-screen
c Edit runtime environment (paths, ports) and turn runtimes on or off
p Edit parameter profiles for the selected model
m Edit extra model search paths (saved in config.toml)
, / . Change sort column / reverse sort direction
enter Copy the launch command for the selected row to the clipboard
a Toggle alert history pane
t Cycle theme (dark β†’ light β†’ auto β†’ …)
? Toggle the full shortcut help overlay
q Quit

Server output

R runs the server in a split layout: the model table stays in the upper half and server logs stream into a scrollable pane below. tab switches focus between the two panes; s stops the server. You can still quit with q or ctrl+c while a model is running, but llml will warn first and require explicit y confirmation before closing.

ctrl+R runs full-screen: the TUI is suspended and the server process is attached directly to your terminal. On Linux/macOS, after the server exits you are prompted to press Enter before the TUI redraws. On Windows there is no Enter prompt; you return when the server process exits.

For Ollama rows, R and ctrl+R do not start a dedicated per-model port. Instead, llml ensures the shared Ollama daemon is running on the configured host and preloads the selected model into memory with keep_alive: -1. The selected row still matters, but the service endpoint remains the shared Ollama host.

Status and alerts

llml separates active work from alert history:

  • a persistent status line shows in-flight operations such as starting Ollama or loading a model
  • a opens a bottom alert-history pane with timestamped INFO, WARN, and ERROR entries
  • when the pane is closed, the footer shows an unread alert count

Minor confirmations such as copy-to-clipboard remain transient. Important failures and lifecycle events remain inspectable in alert history until you dismiss or replace them by later work.

Parameter profiles (p)

Each model path can have multiple named profiles. Each profile stores:

  • Environment variables (KEY=value per line).
  • Extra arguments (--flag value per line).

R / ctrl+R use the active profile (the highlighted row in the p profile list is prefixed with (active) in the name column). Changes persist automatically. tab cycles: profile list β†’ env β†’ extra args. On the profile list: a add profile, c clone (duplicate) the highlighted profile, d delete (not the last), r rename. esc closes the panel (and n cancels a delete confirmation).

A profile's Backend chooses the runtime a model launches on. GGUF models choose llama.cpp or KoboldCpp. Safetensors models choose vLLM, mlx-lm, or mlx-vlm, and also oMLX for models inside one of oMLX's model directories. (none) keeps the default: llama.cpp for GGUF, oMLX inside oMLX's model directories, vLLM elsewhere. The table's Runtime column shows the choice. llml does not check that the weights suit the runtime: MLX weights launched on vLLM fail when the server starts. A runtime that is turned off in c is still offered, labelled (off), so you can set a profile up before turning it on.

Profiles are stored in model-params.json (see Storage & Locations).

Agent skill files

This repo keeps the canonical llml-import skill in:

  • .agents/skills/llml-import/SKILL.md

That workspace path is already consumed directly by many agentic tools. The repo also tracks a Claude workspace compatibility copy at:

  • .claude/skills/llml-import/SKILL.md

Keep the canonical .agents file as the source of truth. When you update it, refresh the Claude compatibility copy with:

./scripts/sync-skill --workspace --tool claude

To install user-level copies for local agent tools, use:

./scripts/sync-skill --user --all-detected

llml-import skill

The llml-import skill imports launch settings from a URL or local file into llml parameter profiles.

Typical flow:

  1. Make sure your coding agent can see the skill from .agents/skills/llml-import/ or from a user-level install created by ./scripts/sync-skill.
  2. Invoke the skill with a source URL or file path, for example:
/llml-import https://huggingface.co/...
  1. Review the extracted profiles, choose which ones to keep, and attach them to one of your local models.
  2. Open llml, press p, and inspect or activate the imported profiles.

What it imports:

  • portable runtime args and env vars for llama.cpp, vLLM, Ollama, KoboldCpp, NInfer, mlx-lm, mlx-vlm, oMLX, Splash, or Strata
  • profile names, model hints, and optional one-line descriptions

What it does not keep:

  • model-location parameters such as --model, -hf, HF_HOME, or similar path/cache settings, because llml supplies the model path itself at launch time

Imported profiles are written into model-params.json. The shared portable contract used by the skill is documented in docs/profile-format.md.

βš™οΈ Configuration

llml follows a strict precedence for settings:
Environment Variables (highest) β†’ config.toml β†’ Default/Auto-detected paths (lowest).

Storage & Locations

User data and settings are stored in a dedicated folder. Routine app upgrades do not delete these files.

Platform Path
Linux (XDG) $XDG_CONFIG_HOME/llml/ (usually ~/.config/llml/)
macOS ~/Library/Application Support/llml/
Windows %AppData%\llml\

Key Files:

  • config.toml: Stores your global settings (ports, binary paths, extra model roots). llml rewrites it only when you save a changed setting from the c or m panels, so comments and formatting survive model scans and runtime on/off toggles.
  • cache/models.toml: The model discovery cache. Machine-owned and safe to delete; llml rebuilds it on the next scan.
  • model-params.json: Stores your named parameter profiles (args/env) for each model.
  • runtimes.toml: Which runtimes are on, as set with the checkboxes in the c panel. Owned by llml, but not disposable like the cache: it is backed up before every overwrite. No environment variable overrides it.
  • backups/: Automatic timestamped snapshots created before the app overwrites configuration.

Runtime Engines

Configure how llml finds and launches servers. You can edit these interactively in the UI (c). The panel lists the runtimes this machine can run, grouped by model format, each marked running, found, or not found. ↑/↓ pick a runtime, tab or β†’ move into its fields, enter saves, and esc discards. Under each field, an in use line shows the value llml runs with and where it came from: the environment variable's name, config, default, or detected (a program found with no path configured). An environment variable wins over the saved value until it is unset.

Each runtime has a checkbox. space (in the list only, never in a field) turns the highlighted runtime on or off, and enter saves it with your field edits. A runtime you turn off shows off in the list and "Off" in its detail pane: llml stops probing it and leaves it out of the missing-runtime warning in the footer, and its fields stay editable. A runtime you tick back on shows no status mark until you save, since llml has not probed it. Saving re-detects runtimes without rescanning models. The on/off choices are stored in runtimes.toml, not config.toml, and no environment variable overrides them. The first time llml sees a runtime, it turns it on only if it finds the program or its server answers, and turns it off otherwise; a single alert lists any runtimes turned off this way. After that only your choice changes it, so a runtime you install later stays off until you tick it.

Models whose runtime is off stay in the table, dimmed and marked (off) after the runtime name, so you can still see what is taking up disk space. You can select them, copy their command, and open their p panel, but R / ctrl+R will not launch them; an alert names the runtime to turn on in c, and a note under the launch preview says the runtime is off. A GGUF model follows its active profile: it is dimmed only when that profile's runtime (llama.cpp or KoboldCpp) is off, so switching profiles can make it launchable again.

When Ollama is off, llml does not contact the Ollama daemon at all: startup and S neither start ollama serve nor call its API. The Ollama models from the last scan stay in the table, dimmed, and may be out of date until you turn Ollama back on and scan again.

Feature Environment Variable config.toml key (under [runtime]) Default
llama.cpp path LLAMA_CPP_PATH default_llama_cpp_path (auto)
llama.cpp port LLAMA_SERVER_PORT default_llama_server_port 8080
vLLM path VLLM_PATH default_vllm_path (auto)
vLLM venv VLLM_VENV default_vllm_venv (auto)
vLLM port VLLM_SERVER_PORT default_vllm_server_port 8000
Ollama path OLLAMA_PATH default_ollama_path (auto)
Ollama host OLLAMA_HOST default_ollama_host 127.0.0.1:11434
KoboldCpp path KOBOLDCPP_PATH default_koboldcpp_path (auto)
KoboldCpp port KOBOLDCPP_PORT default_koboldcpp_port 5001
NInfer path NINFER_PATH default_ninfer_path (auto)
NInfer port NINFER_SERVER_PORT default_ninfer_server_port 8080
NInfer host NINFER_SERVER_HOST default_ninfer_server_host 127.0.0.1
oMLX path OMLX_PATH default_omlx_path (auto)
oMLX port OMLX_PORT default_omlx_port 8000
oMLX host OMLX_HOST default_omlx_host 127.0.0.1
oMLX model dirs OMLX_MODEL_DIRS - (from oMLX app)
Splash path SPLASH_PATH default_splash_path (auto)
Splash port SPLASH_PORT default_splash_port 8000
Splash host SPLASH_HOST default_splash_host 127.0.0.1
mlx-lm path MLX_LM_PATH default_mlx_lm_path (auto)
mlx-lm port MLX_LM_PORT default_mlx_lm_port 8080
mlx-lm host MLX_LM_HOST default_mlx_lm_host 127.0.0.1
mlx-vlm path MLX_VLM_PATH default_mlx_vlm_path (auto)
mlx-vlm port MLX_VLM_PORT default_mlx_vlm_port 8080
mlx-vlm host MLX_VLM_HOST default_mlx_vlm_host 127.0.0.1
Strata path STRATA_PATH default_strata_path (auto)
Strata port STRATA_PORT default_strata_port 8080
Strata host STRATA_HOST default_strata_host 127.0.0.1
TUI Theme LLML_THEME - auto

Detection Logic:

  1. Explicitly configured paths (Env/TOML).
  2. Common system directories (e.g., /usr/local/bin, /opt/homebrew/bin, ~/.local/bin).
  3. Binary names available on your system PATH.
  4. Probing each runtime's port for an already-running server. Several runtimes share a default port (llama.cpp, NInfer, mlx-lm, and mlx-vlm on 8080; vLLM, oMLX, and Splash on 8000), so llml checks which server answers: NInfer, oMLX, and Splash by the owner they report in /v1/models, mlx-lm by a /health status of ok from Python's HTTP server (a Server header starting BaseHTTP/), mlx-vlm by a /health status of healthy, llama.cpp by a /health answer from a server that is not NInfer, mlx-lm, or mlx-vlm, and vLLM by an answer from a server that is not oMLX, Splash, mlx-lm, or mlx-vlm. Strata shares 8080 too and names itself in its /health body ("service": "strata").
  5. (vLLM only) Common venv locations (e.g., ~/.venv-vllm-metal/bin on macOS).
  6. (KoboldCpp only) Platform-specific name variants with CUDA preference on Linux.
  7. (NInfer only) NInfer has no install target, so NINFER_PATH may point at the checkout root (llml looks in build/apps/), the build directory, or ninfer-serve itself. The checkout's models/ directory is added to the scan roots. The default port is 8080, ninfer-serve's own default, which llama-server shares; only one of them can listen on it at a time.
  8. (oMLX only) llml uses the CLI shim the oMLX app installs (~/.omlx/bin/omlx) and reads the app's model directories from its settings.json, so models placed there appear as oMLX rows. R runs omlx serve --model-dir <dir>; oMLX serves every model in that directory and applies the per-model settings (draft model, sampling) saved in the app. Stop the app's own background server first if it is using the same port. A profile can launch such a model on vLLM instead (see Parameter profiles).
  9. (Splash only) Splash bundles are found in the Hugging Face cache and launched with splash serve --model <org/repo>.
  10. (mlx-lm only) llml looks for the mlx_lm.server script at MLX_LM_PATH (the script, or the folder holding it) and then on PATH. It does not search for venvs, and there is no common-directory step. See Setting up mlx-lm.
  11. (mlx-vlm only) The same for the mlx_vlm.server script and MLX_VLM_PATH. See Setting up mlx-vlm.
  12. (Strata only) STRATA_PATH may be the Strata install directory, its serve/server.py, its engine/strata, or one of its engine config files. When unset, llml reads the installs from Strata's own settings.json and also checks ~/Applications/strata and ~/Strata. Each strata-<model>.json engine config becomes a row; R runs the install's venv Python with serve/server.py --engine strata --config <config>.

NInfer is shown only on Linux, oMLX and Splash only on Apple Silicon macOS, mlx-lm and mlx-vlm only on Apple Silicon macOS and Linux (x86_64 and aarch64), and Strata only on Linux and Windows (the platforms its NVIDIA / AMD engines ship for); the c panel hides runtimes the platform cannot run. A runtime turned off in c gets no server probe, but llml still looks for its program on disk so the panel can show whether it is installed.

Setting up mlx-lm

mlx-lm runs a safetensors model directory with Apple's MLX. To use it, choose mlx-lm as the Backend of a safetensors model's profile in p. R then runs:

mlx_lm.server --model <absolute model path> --host <host> --port <port> <profile args>

llml adds nothing else, and passes the profile's env vars as for any runtime.

  • macOS (Apple Silicon): install mlx-lm with pip, uv, or mise into any Python environment, for example pip install mlx-lm or uv tool install mlx-lm. If mlx_lm.server is not on your PATH, set MLX_LM_PATH (or the Path field in c) to the script or the folder that holds it, such as a venv's bin/. The venv does not need to be activated: the script starts its own Python.
  • Linux (x86_64 and aarch64): a plain pip install mlx-lm installs no MLX backend on Linux, so the server cannot start. Install a backend next to it: pip install mlx-lm "mlx[cuda12]" for CUDA 12, "mlx[cuda13]" for CUDA 13, or "mlx[cpu]" without a GPU. The CUDA build needs an NVIDIA GPU of compute capability 7.5 or newer and a recent driver. The CPU build works but is far too slow for real models. MLX compiles its CUDA kernels at run time against the toolkit that CUDA_PATH (or CUDA_HOME) points to, so that toolkit must match the extra: with mlx[cuda12] and a CUDA 13 toolkit on CUDA_PATH, the server starts and answers /health, but the first request hangs while kernels fail to compile. Install the matching extra, or set CUDA_PATH to a CUDA 12 toolkit in the profile's env (for example CUDA_PATH=/opt/cuda-12.9.1).

Things to know:

  • Answer length: mlx-lm stops each answer at 512 tokens unless the profile passes --max-tokens (for example --max-tokens 8192) or the client sends max_tokens.
  • Model id: the launch preview shows the model id clients must send, which is the model's absolute path. mlx-lm loads whatever model a request names, and downloads it from Hugging Face if it is not a local folder, so a client that sends another id makes it switch models. Leaving model out of the request also uses the launched model.
  • Missing folders: llml will not launch mlx-lm on a model folder that no longer exists, since the server would take the path for a Hugging Face repo id. Press S to rescan.
  • Port: mlx-lm's default port is 8080, the same as llama.cpp's and NInfer's. Only one server can listen on it at a time; set MLX_LM_PORT to run them side by side.

Setting up mlx-vlm

mlx-vlm runs a safetensors model directory with Apple's MLX, including vision and audio models. To use it, choose mlx-vlm as the Backend of a safetensors model's profile in p. R then runs:

mlx_vlm.server --model <absolute model path> --host <host> --port <port> <profile args>

llml adds nothing else, and passes the profile's env vars as for any runtime. It always passes --host (127.0.0.1 unless you set MLX_VLM_HOST), because mlx-vlm otherwise listens on all interfaces.

  • Version: mlx-vlm 0.6.5 or later is needed to serve text-only models. Older releases serve only the vision model families they know.
  • macOS (Apple Silicon): install mlx-vlm with pip, uv, or mise into any Python environment, for example pip install mlx-vlm or uv tool install mlx-vlm. If mlx_vlm.server is not on your PATH, set MLX_VLM_PATH (or the Path field in c) to the script or the folder that holds it, such as a venv's bin/. The venv does not need to be activated.
  • Linux (x86_64 and aarch64): a plain pip install mlx-vlm installs MLX without a backend, so import mlx.core fails and the server cannot start. Install a backend next to it: pip install mlx-vlm "mlx[cuda12]" for CUDA 12, "mlx[cuda13]" for CUDA 13, or "mlx[cpu]" without a GPU. Do not use pip install "mlx-vlm[cuda]": it pulls an outdated mlx-cuda (0.30.0) beside a newer mlx. The CUDA and CPU notes for mlx-lm apply here too.

Things to know:

  • Model id: the launch preview shows the model id clients must send, which is the model's absolute path. Clients must send exactly that id: mlx-vlm loads whatever model a request names, and downloads it from Hugging Face if it is not a local folder, so any other id makes it swap or download a model. Unlike mlx-lm, leaving model out of the request does not use the launched model.
  • Missing folders: llml will not launch mlx-vlm on a model folder that no longer exists, since the server would take the path for a Hugging Face repo id. Press S to rescan.
  • Port: mlx-vlm's default port is 8080, the same as llama.cpp's, NInfer's, and mlx-lm's. Set MLX_VLM_PORT to run them side by side.

Setting up Strata

Strata runs its own large Qwen3.8-Flash-Next model on an NVIDIA or AMD card, on Windows or Linux. Install it with its START-HERE.bat / ./setup.sh; setup writes one engine config per model (strata-<model>.json) in the Strata folder. llml discovers each config as a row and R runs:

<install>/.venv/bin/python <install>/serve/server.py \
  --engine strata --config <install>/strata-<model>.json \
  --host <host> --port <port> <profile args>
  • Path: when STRATA_PATH (or the c panel's Path field) is unset, llml reads the installs from Strata's own settings.json and also checks ~/Applications/strata and ~/Strata. Set it to the install directory (or its serve/server.py, its engine/strata, or an engine config) when your install lives elsewhere.
  • Port: Strata's setup default is 8080, which its own /health body distinguishes (it names "service": "strata"). Set STRATA_PORT to move it.
  • Platform: shown on Windows and Linux only, the platforms Strata ships NVIDIA / AMD engines for.
  • Profiles: the engine config already carries the model, engine args, tokenizer, and data paths, so a profile for a Strata row usually just adds server flags. Model-location flags (--config, --engine, --pack, --native, --mtp, …) are stripped on import.

Model Discovery

llml scans several common directories (e.g., ~/models, ~/.cache/llama.cpp, ~/.cache/lm-studio/models) by default.

Custom Search Paths

Add extra directories to scan via the UI (m), config.toml (discovery.extra_model_paths), or environment variables:

Environment Variable Role
LLML_MODEL_PATHS Comma-separated list of extra roots to scan.
Example: export LLML_MODEL_PATHS="/data/models,/opt/weights"
HUGGINGFACE_HUB_CACHE Specific root for Hugging Face hub cache (overrides HF_HOME/hub).
HF_HOME Root for HF home (cache resolves to $HF_HOME/hub).

Example config.toml entry:

[discovery]
extra_model_paths = ["/data/models", "/opt/weights"]

Discovery Rules

The app recognizes the following model types:

  • GGUF: Files ending in .gguf.
  • Safetensors: Directories containing config.json and *.safetensors files (Hugging Face style).
  • NInfer: Files ending in .ninfer.
  • Splash: Bundles in the Hugging Face cache.
  • Strata: strata-<model>.json engine configs in a Strata install (read directly, not walked).

Data Integrity & Backups

To protect your settings and cache, llml maintains a history of your configuration:

  • Atomic Writes: Files are written to a temporary location before being moved, preventing corruption.
  • Automatic Backups: The newest 10 versions of each file are kept in the backups/ directory.
  • Upgrade Snapshots: When the llml version changes, a snapshot of config.toml, model-params.json, and runtimes.toml is created automatically so you can roll back if needed. The file .last-run-version in the same directory records the last run version for that behavior.

πŸ’» Development

Development Tooling Requirements

  • mise β€” manages Go, Node.js, and tasks. Install mise first.
  • Go 1.26+ and Node.js (LTS) β€” both installed automatically by mise install.

Set up tooling

Clone the repository and install tooling:

mise install   # installs Go, Node.js, pre-commit (pipx), and other tools from mise.toml
npm ci         # installs Prettier + markdownlint
pre-commit install   # optional: enable git pre-commit / pre-push hooks (see .pre-commit-config.yaml)

Common Tasks

mise run run      # go run ./cmd/llml
mise run build    # build to bin/llml
mise run install-dev  # build to ~/.local/bin/llml-dev
mise run format   # auto-fix: gofmt + prettier + markdownlint
mise run lint     # check only: gofmt + vet + prettier + markdownlint
mise run test     # go test -race ./...
mise run check    # lint + test (run before opening a PR)

Layout

  • cmd/llml β€” entrypoint.
  • internal/config β€” config.toml read/write and cache helpers.
  • internal/tui β€” Bubble Tea UI.
  • internal/models β€” discovery, metadata, runtime detection.
  • scripts/ β€” gofmt-check.sh, precommit-docs-fix.sh.

Contributions are welcome. Please run mise run check before opening a pull request.

❀️ Support This Project

You can support this project in a few simple ways:

πŸ“„ License

MIT Β© Flying Nobita

About

Terminal UI for discovering local GGUF and safetensors models, detecting llama.cpp or vLLM runtimes, and launching them with saved presets.

Topics

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0