Repository navigation
feat(describe): describe pictures on this machine with Moondream - #518
Merged
Merged
Conversation
Settings → LLM provider → Describing pictures can also have M87 Labs' Moondream caption each page and shot (llm.describe_engine moondream) through transformers and PyTorch, the moondream extra fetched on first use: Moondream 3 Preview (BSL 1.1, about 19 GB, an NVIDIA GPU or a Mac with 32 GB) or Moondream 2 (Apache-2.0, about 3.7 GB, on a CPU, slowly), pinned to a revision. moondream.cannot_run keeps the model from being fetched where it can't run and the step says why. Agents choose it with set_picture_descriptions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
…-ogtvpr # Conflicts: # README.md # docs/requirements.md # threat-model.yaml
…-ogtvpr # Conflicts: # threat-model.yaml
…-ogtvpr # Conflicts: # threat-model.yaml
…-ogtvpr # Conflicts: # threat-model.yaml
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013kCtgLNMjPQYdZpV5gbwUW
This was referenced Oct 18, 2026
adeelahmad
pushed a commit
that referenced
this pull request
Oct 18, 2026
The threat model workflow's validate job wants a change to threat-model.md to change a companion too (#507): the Kokoro row is documented, so the JSON's documented count goes up by one, as #518 did for Moondream. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
adeelahmad
added a commit
that referenced
this pull request
Oct 18, 2026
…ine (#538) * feat(voice): read answers and podcasts aloud with Kokoro on this machine Kokoro v1.0 (hexgrad's Kokoro-82M, Apache-2.0) as a text-to-speech provider, voice.tts_provider "kokoro": spoken answers and podcast hosts are read on the server's own CPU through kokoro-onnx on ONNX Runtime, in any of its 54 voices across nine languages, with nothing sent anywhere and nothing billed. It's opt-in and fetched once it's picked: the `kokoro` extra (espeak-ng and phonemizer are GPL-3.0, so they stay out of the default install), and the model (voice.kokoro_model: fp32, fp16 or int8) and voices files from kokoro-onnx's release, checked against their SHA-256. Settings → AI assistant → Voice offers it with the model size and the voice picker; a podcast's second host gets a voice in the first host's language. kokoro.yml fetches and reads with each model on x64 and arm64. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * test(settings): carry ai.auto_title in the Kokoro settings fixture main's conversation titles (#531) added ai.auto_title to the section the Kokoro settings test renders, so its fixture names it too; without it Save sent an ai update beside the voice one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * docs(threat-model): count the Kokoro row in the JSON companion The threat model workflow's validate job wants a change to threat-model.md to change a companion too (#507): the Kokoro row is documented, so the JSON's documented count goes up by one, as #518 did for Moondream. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * fix(voice): offer Kokoro in full precision and 8-bit only On GitHub's arm64 runner, ONNX Runtime's CPU provider returned an empty waveform for kokoro-onnx's fp16 export, and kokoro-onnx's pause insertion then failed on a zero-size array. On x64 it was no faster than fp32, so voice.kokoro_model is now fp32 (the default) or int8, and the fp16 file is no longer pinned. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * test(settings): carry the research section in the Kokoro settings fixture Main's deep research (#517) added a research section to Settings › AI assistant, so the fixture carries its defaults and Save sends only the voice change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * test(settings): carry ai.compact_at in the Kokoro settings fixture Main added ai.compact_at (summarise long conversations early), so the fixture carries its default and Save sends only the voice change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * docs(readme): list answers about Lens itself as built, now that #400 is released Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * docs(changelog): name the Kokoro fragment after its pull request Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * docs(threat-model): count the Kokoro entries against main's newest model Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH * docs(readme): take Kokoro out of In progress, now that it's built Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH --------- Co-authored-by: Adeel Ahmad <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: pictures were described by the provider's vision model or, since #477, by Florence-2 on this server.
After: Settings → LLM provider → Describing pictures also offers Moondream on this server (
llm.describe_enginemoondream), with a Moondream model choice (llm.moondream_model): Moondream 3 Preview (the default when Moondream is picked, about 19 GB, needs an NVIDIA GPU with 20 GB or a Mac with 32 GB; BSL 1.1, not open source) or Moondream 2 (Apache-2.0, about 3.7 GB, runs on a CPU, slowly). No picture leaves the machine. The default engine stays the provider's model.How:
moondream.pyloads the model with transformers at a pinned revision (trust_remote_code, since Moondream's code ships with its weights) and calls itscaption(image, length="normal"), the same for both models. Themoondreamextra (PyTorch and transformers) and the weights are a component fetched the first time a job waits at the describe step, and only where the chosen model can run:moondream.cannot_runreads the machine (GPUs and their memory, Apple silicon, RAM) and, where it can't, the step's log and thepicture_descriptionsMCP tool say why and suggest Moondream 2 or Florence-2, instead of fetching 19 GB. Each picture is amodel.describeledger row. PyTorch's threads are capped atOMP_THREAD_LIMITwhen that is lower, since PyTorch's CPU kernels otherwise write nonsense (found by the real-model test).Checked here: Moondream 2 at the pinned revision describes a photo well through Lens's own code but took 3 to 6 minutes a picture on four shared CPU cores (Florence-2 base: about 3 s). Moondream 3 Preview was not run here (no GPU); its code at the pinned revision was read and uses the same call. An opt-in test (
LENS_TEST_MOONDREAM=1) fetches PyTorch, transformers and Moondream 2 through the component and describes a picture.Motivation and Context
Adeel asked to use Moondream 3 and/or Microsoft Florence for visual descriptions. Florence-2 landed in #477; this adds Moondream as an opt-in engine, since Moondream 3's weights are BSL 1.1 and need a GPU or a large Mac.
Types of changes
!in the title)Checklist:
make hookschecks it on commit).docs/accordingly.## UnreleasedinCHANGELOG.md.Done questions (AGENTS.md)
set_picture_descriptionstakesmoondream_model;picture_descriptionssays whether it can run here and why not.llm.describe_engineandllm.moondream_modelin Settings with defaults; nothing in the wizard.model.describeledger row.docs/requirements.md.make openapigave no changes).Generated by Claude Code