Repository navigation
feat(llm): run a GGUF chat model locally with llama.cpp - #127
Merged
Merged
Conversation
Settings > Local model lists GGUF chat models with the memory and disk each needs; the ones this machine can run can be picked. Lens downloads the model from Hugging Face (pinned revision, SHA-256 checked, resumed), fetches llama.cpp's server for the platform unless llama-server is installed, runs it with its own API key under a lease, and makes it the LLM provider, putting the previous provider back when turned off. Any GGUF on Hugging Face works as hf:owner/repo/file.gguf. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Fki157VgxFETzSSpwFwYe
…oviders-iuvzld # Conflicts: # fastapi_backend/app/api/v1/routes/admin.py # fastapi_backend/app/core/runtime.py # nextjs-frontend/app/openapi-client/index.ts # nextjs-frontend/app/openapi-client/sdk.gen.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: Lens needed a separate model server (Ollama, LM Studio, a cloud API) for summaries, templates and chat.
After: Settings → Local model lists 17 GGUF chat models (Qwen2.5/Qwen3, Llama 3.x, Gemma 3, Phi-4 mini, SmolLM2, Mistral Nemo) with each model's download size and the memory it needs. Models this machine can't run, or has no disk for, are greyed out. Pick one and turn it on, and Lens:
llama-serverif it's already installed; otherwise it fetches llama.cpp's latest release build for Linux x64/arm64 or macOS intodata_dir/bin/llama.cpp. A path set in archive.yaml (local_llm.server) takes priority; for security that one can't be changed from the web app;data_dir/models/gguf, pinned to a revision, checked against its SHA-256, and able to resume after an interruption;Any other GGUF works as
hf:owner/repo/file.gguf. The page shows progress (fetching llama.cpp, downloading %, starting, running), the server's last log lines, and the downloaded files with a Delete button.How:
app/domain/local_llm.py(catalog, download, fetch, Runner),local_llmsettings section,GET /settings/local-llm/status,DELETE /settings/local-llm/models,docs/local-models.md. Off by default, so nothing changes until an admin turns it on.Motivation and Context
The second part of Adeel's voice/providers ask: "allow local llm (via llama.cpp and hugging face gguf download from a list of runnable models)". The first part, cloud speech providers, merged as #116.
Types of changes
!in the title)Checklist:
tests/domain/test_local_llm.py: catalog fit, settings, resumed and checksummed downloads against a fake Hugging Face, release fetch from a fake GitHub API including a path-escape entry, and a full run with a fake llama-server that becomes the provider and gives the old one back;__tests__/settings-local-model.test.tsx).docs/accordingly.## UnreleasedinCHANGELOG.md.Local runs on current main: backend 564 passed, 1 failed (the known environmental LibreOffice test); frontend 82 suites, 673 tests passed. Not tried with a real llama.cpp download here, because this sandbox can't reach GitHub.
🤖 Generated with Claude Code
https://claude.ai/code/session_014Fki157VgxFETzSSpwFwYe
Generated by Claude Code