Repository navigation
feat(voice): read answers and podcasts aloud with Kokoro on this machine - #538
Merged
Merged
Conversation
Kokoro v1.0 (hexgrad's Kokoro-82M, Apache-2.0) as a text-to-speech provider, voice.tts_provider "kokoro": spoken answers and podcast hosts are read on the server's own CPU through kokoro-onnx on ONNX Runtime, in any of its 54 voices across nine languages, with nothing sent anywhere and nothing billed. It's opt-in and fetched once it's picked: the `kokoro` extra (espeak-ng and phonemizer are GPL-3.0, so they stay out of the default install), and the model (voice.kokoro_model: fp32, fp16 or int8) and voices files from kokoro-onnx's release, checked against their SHA-256. Settings → AI assistant → Voice offers it with the model size and the voice picker; a podcast's second host gets a voice in the first host's language. kokoro.yml fetches and reads with each model on x64 and arm64. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
main's conversation titles (#531) added ai.auto_title to the section the Kokoro settings test renders, so its fixture names it too; without it Save sent an ai update beside the voice one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Keeps both the Kokoro and the Moondream components, both changelog, devlog and requirements entries, and the README sentence with Kokoro and read-any-message-aloud; uv.lock's provides-extras lists both extras. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
The threat model workflow's validate job wants a change to threat-model.md to change a companion too (#507): the Kokoro row is documented, so the JSON's documented count goes up by one, as #518 did for Moondream. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
On GitHub's arm64 runner, ONNX Runtime's CPU provider returned an empty waveform for kokoro-onnx's fp16 export, and kokoro-onnx's pause insertion then failed on a zero-size array. On x64 it was no faster than fp32, so voice.kokoro_model is now fp32 (the default) or int8, and the fp16 file is no longer pinned. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Keeps both changelog entries (Kokoro and deep research) and counts the Kokoro threat-model row on top of main's documented count (255, now 256). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
…ture Main's deep research (#517) added a research section to Settings › AI assistant, so the fixture carries its defaults and Save sends only the voice change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Keeps both changelog entries (Kokoro and the upgrade agent starting the code sandbox's gateway). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Keeps Kokoro in the README's chat sentence alongside main's automatic summarising of long conversations. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Main added ai.compact_at (summarise long conversations early), so the fixture carries its default and Save sends only the voice change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Owner
Author
|
Clicks and keystrokes per feature is red on So this looks like a slow answer on the runner, not this PR's. I'm re-running the failed job once; if it fails again I'll treat it as real and dig in. Generated by Claude Code |
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
…is released Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
…entry as a changelog fragment Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
…-hky2fo # Conflicts: # fastapi_backend/app/domain/speech.py
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
…-hky2fo # Conflicts: # docs/components.md # fastapi_backend/pyproject.toml # fastapi_backend/uv.lock # threat-model.json
…-hky2fo # Conflicts: # fastapi_backend/uv.lock
…-hky2fo # Conflicts: # README.md # fastapi_backend/pyproject.toml # fastapi_backend/uv.lock
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH
21 of 23 tasks
adeelahmad
pushed a commit
that referenced
this pull request
Oct 18, 2026
…swers setting #622 added ai.answer_comments after #538's test was written, so the test's view lacked it and Save also saved the AI section, failing on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QFUNzy1QZy5HhKAyJxTAYJ
20 of 23 tasks
adeelahmad
pushed a commit
that referenced
this pull request
Oct 18, 2026
#622 added ai.answer_comments (answer when tagged in a comment) after #538 wrote the Kokoro settings test's view of the section, so Save sent an ai update beside the voice one and main's Next.js check went red at 2deac7a. The fixture names the default, the same line #621 and #624 carry. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NY27LFMbLezbvQQQeHT19a
20 of 23 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: spoken answers and podcast episodes needed a speech provider (OpenAI-compatible, ElevenLabs, Deepgram, Gemini), a Mac for
say, or the separate StyleTTS2 service; a fully local install with no extra container had no voice.After: Kokoro (54 voices, nine languages) is a text-to-speech provider in Settings → AI assistant → Voice. Pick it and save: a worker fetches it in the background (Settings → Components shows how far it got), then answers and podcast hosts are read on the server's own CPU, with nothing sent anywhere and nothing billed. The voice picker lists all 54 voices before the fetch, with a Hear button; the model size (full 326 MB or 8-bit 114 MB) is a setting. A podcast's second host gets a voice in the first host's language (
am_michaelbesideaf_heart,em_alexbesideef_dora) unlesspodcasts.voice_bnames one.How:
kokoro.pyruns Kokoro v1.0 (hexgrad's Kokoro-82M, Apache-2.0) through kokoro-onnx (MIT) on ONNX Runtime, in-process, asvoice.tts_providerkokoro; the model is loaded on first use and let go after ten minutes of quiet.components.pygains aKokorocomponent that fetches thekokoroextra atuv.lock's versions and the model and voices files from kokoro-onnx'smodel-files-v1.1release, checked against their SHA-256, as soon as the provider is picked; podcast jobs wait while it's being fetched. Each read is amodel.speechrow in the activity ledger (kokoro:kokoro-v1.0).espeak-ng and phonemizer (which kokoro-onnx phonemizes with) are GPL-3.0, so they live only in the opt-in
kokoroextra, never in the default install.Motivation and Context
Adeel asked for Kokoro v1.0's model and voice assets for TTS and podcast generation, "making it 100% local supported". StyleTTS2 already read locally but as its own 2 GB service; Kokoro is a 330 MB download into the data folder with no service to start, so a Raspberry Pi with cloud models can read aloud on its own.
Screenshots (if appropriate):
n/a (a select and a paragraph in the existing Voice settings).
Steps to reproduce (if appropriate):
Measured (one 5-second sentence, fp32): 2.8 s on a 4-core x64 container (int8 4.6 s), 3.2 s on GitHub's arm64 runner in
kokoro.yml, loading included; on x64 loading the model takes 1.8–5 s. kokoro-onnx's fp16 export isn't offered: on arm64, ONNX Runtime's CPU provider read nothing with it (an empty waveform), and on x64 it was no faster than fp32.Types of changes
!in the title)Checklist:
make hookschecks it on commit).docs/accordingly.## UnreleasedinCHANGELOG.md.Done questions (AGENTS.md)
PUT /api/v1/settings/voice(tts_provider,kokoro_model,tts_voice) and the assistant'schange_settingstool pick it;GET /api/v1/voice/voices?provider=kokorolists the voices;POST /api/v1/voice/speakreads; podcasts through the API and MCP as before.voice.tts_provider,voice.kokoro_model(fp32, the default, or int8) andvoice.tts_voicein the web app; nothing in the setup wizard.openai); about 450 MB fetched only when picked; int8 for a small disk; the model is unloaded after ten minutes of quiet.threat-model.mdhas a row for it (the voices.npzis read with pickles off).model.speechledger row.voice.kokoro_modelis a new setting with a default).tests/domain/test_kokoro.py(7, on a tiny Kokoro-shaped ONNX model so no download),settings-kokoro.test.tsx, a Kokoro case insettings-voice-picker.test.tsx;kokoro.ymlfetches the real files and reads with both models on x64 and arm64;make openapiregenerated;mkdocs build --strictclean.DEVLOG.md.components.py's fetch-by-URL-with-checksum,speech.py's provider dispatch, the voice picker and the podcast audio step are reused unchanged in shape.Local CI (
ci/local-ci.shon3ded53f, main through #534 merged in; no Docker here, so the SurrealDB-server and upgrade jobs run on GitHub):After main through #549 was merged in (a changelog-only conflict), the Kokoro, podcast, speech and components backend tests, the voice settings frontend tests,
ci/check_requirements.py,ci/validate_threat_model.py --base origin/mainanduv lock --checkwere run again. The worker's own fetch path was replayed earlier against a bare venv with only the base dependencies:components.pip_install(cfg, "kokoro")put 10 packages intodata_dir/python,kokoro_onnximported from there, and Kokoro read an English and a Spanish sentence.🤖 Generated with Claude Code
https://claude.ai/code/session_01Dzm4YHdHn2qZy7aXer3KeH