8000 feat(voice): read aloud with Gemini TTS or macOS say, and pick the voice in Settings by adeelahmad · Pull Request #268 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,22 @@

Plans and progress for work in flight. Newest first.

## 2026-10-10 · More voices for spoken answers

Goal (Adeel): read answers and podcasts aloud with ElevenLabs, macOS `say`, Google Gemini TTS or StyleTTS2, with
the voice picked in Settings. ElevenLabs and Deepgram were already there (#116); the providers stay shared with
podcasts through `speech.speak()`.

Todo:

- [x] Gemini TTS and macOS `say` as text-to-speech providers; `GET /voice/voices` lists each provider's voices;
`POST /voice/speak` takes a voice to try
- [ ] StyleTTS2 as an opt-in local service (off by default; heavy)
- [x] Settings → AI assistant → Voice: pick a voice from the list and hear it; Gemini in Speech providers

Refine later: pick a second `say` voice for podcast hosts from what the Mac has; Gemini's multi-speaker mode for a
whole podcast in one call; ElevenLabs voice settings (stability, style); cache the voice lists.

## 2026-10-10 · Podcasts from picked knowledge

Goal (Adeel): pick documents, search results, topics, notes or graph concepts and get a two-host, Recast-style
Expand Down
9 changes: 5 additions & 4 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,13 @@ Voice goes through one hook, `useVoice()` in `nextjs-frontend/lib/voice.ts` (`li
listening. Without a server engine, or with `voice.input: browser`, the browser's own recognition listens (Chrome,
Edge and Safari; Chrome sends the audio to its own speech service).
- **Speaking.** Answers are read by a speech model when one is set (`voice.tts_model`: any OpenAI-compatible
`/audio/speech`, such as Kokoro-FastAPI, LocalAI or OpenAI's `tts-1`, on `voice.tts_base_url` or the LLM provider);
else by the browser.
`/audio/speech`, such as Kokoro-FastAPI, LocalAI or OpenAI's `tts-1`, on `voice.tts_base_url` or the LLM provider),
or by ElevenLabs, Deepgram, Google Gemini or macOS `say` ([Speech providers](speech-providers.md#voices)); else by
the browser.

Settings → AI assistant → Voice holds these. The API: `GET /voice` (what the server does; it also starts loading the
engine), `POST /voice/transcribe` (a clip as the raw body; nothing is kept), `POST /voice/speak` (`{text}`; 204 when
the browser should read it).
engine), `POST /voice/transcribe` (a clip as the raw body; nothing is kept), `POST /voice/speak` (`{text, voice?}`;
204 when the browser should read it), `GET /voice/voices` (the voices a provider reads in).

## The assistant's tools

Expand Down
19 changes: 18 additions & 1 deletion docs/speech-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ summaries, chapters and entities as optional imports, per-minute prices in the a
| ElevenLabs | Scribe | yes | yes | no | laughter, applause, … | yes |
| AssemblyAI | Universal, Slam-1 | yes | detected | from sentiment | no | no |
| Deepgram | Nova | yes | detected | from sentiment (English) | no | Aura |
| Google Gemini | no | no | no | no | no | `gemini-2.5-flash-preview-tts`, 30 prebuilt voices |
| macOS `say` | no | no | no | no | no | the Mac's own voices; no key, nothing leaves the Mac |

Sentiment becomes emotion as positive → Happy, negative → Sad, neutral → Neutral (turn it off with
`speech.sentiment`). SenseVoice's own emotion tags stay richer; a provider only reports sentiment.
Expand All @@ -35,18 +37,33 @@ Then pick where each task goes:
- **Speaker separation → Method**: *Speech provider* uses the speakers the provider found; *Auto* uses them when a
transcript has some, and otherwise works as before. Voice IDs across recordings still come from local voiceprints.
- **AI assistant → Voice**: *Engine that hears it* (default: the transcription engine) and *Spoken answers by* (an
OpenAI-compatible speech server, ElevenLabs or Deepgram Aura).
OpenAI-compatible speech server, ElevenLabs, Deepgram Aura, Google Gemini or macOS `say`) and *Voice*.

## Voices

`GET /voice/voices?provider=…` lists the voices a provider reads in, as `{id, name, detail}`: ElevenLabs lists the
voices in your account (premade, cloned and from the library), Deepgram its Aura models, `say` the voices installed on
the Mac (System Settings → Accessibility → Spoken Content adds more), and an OpenAI-compatible server its own when it
has `/audio/voices` (Kokoro-FastAPI does); otherwise OpenAI's eleven. Gemini's are a fixed list. The voice picked is
`voice.tts_voice`; podcast hosts use it too (`podcasts.voice_a`, `voice_b`). `POST /voice/speak` takes `{text,
voice}` to try a voice before keeping it.

`say` only works when Lens runs on the Mac itself (`make start-backend`, not Docker): a container has no `say`. It
writes WAV, and both podcast hosts get the same voice unless `podcasts.voice_b` names another, since no voice is on
every Mac.

| Setting | Default |
| --- | --- |
| `speech.openai_base_url`, `openai_model` | `https://api.openai.com/v1`, `whisper-1` |
| `speech.elevenlabs_base_url`, `elevenlabs_model` | `https://api.elevenlabs.io`, `scribe_v1` |
| `speech.assemblyai_base_url`, `assemblyai_model` | `https://api.assemblyai.com`, `universal` |
| `speech.deepgram_base_url`, `deepgram_model` | `https://api.deepgram.com`, `nova-3` |
| `speech.gemini_base_url`, `gemini_model` | `https://generativelanguage.googleapis.com`, `gemini-2.5-flash-preview-tts` |
| `speech.sentiment` | on |
| `speech.timeout` | 1800 seconds |
| `voice.stt` | `same` |
| `voice.tts_provider` | `openai` |
| `voice.tts_voice` | the provider's default (alloy, a premade ElevenLabs voice, aura-2-thalia-en, Kore, the Mac's) |

## What is sent

Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ def test_embeddings(user: AdminWriter, cfg: Cfg, db: Db) -> EmbedTestResult:

@router.post("/settings/speech/test")
def test_speech(
user: AdminWriter, cfg: Cfg, provider: str = Query(..., pattern="^(openai|elevenlabs|assemblyai|deepgram)$")
user: AdminWriter, cfg: Cfg, provider: str = Query(..., pattern="^(openai|elevenlabs|assemblyai|deepgram|gemini)$")
) -> SpeechTestResult:
"""Check a speech provider's address and key with the saved settings (lists its models; nothing is billed)."""
t0 = time.time()
Expand Down
19 changes: 15 additions & 4 deletions fastapi_backend/app/api/v1/routes/voice.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@

from __future__ import annotations

from fastapi import APIRouter, HTTPException, Request, Response
from fastapi import APIRouter, HTTPException, Query, Request, Response
from starlette.concurrency import run_in_threadpool

from app.api.deps import Cfg, CurrentUser
from app.domain import voice
from app.schemas.voice import Heard, SpeakRequest, VoiceInfo
from app.domain import speech, voice
from app.schemas.voice import Heard, SpeakRequest, VoiceInfo, VoiceList

router = APIRouter(prefix="/voice", tags=["voice"])
CLIP = {"requestBody": {"required": True, "content": {"application/octet-stream": {"schema": {"type": "string", "format": "binary"}}}}}
Expand Down Expand Up @@ -47,8 +47,19 @@ async def transcribe_clip(request: Request, user: CurrentUser, cfg: Cfg) -> Hear
async def speak_text(body: SpeakRequest, user: CurrentUser, cfg: Cfg) -> Response:
"""The text read aloud by the text-to-speech model (Settings → AI assistant → Voice). 204 when there's none, or it
failed: the browser reads it instead."""
out = await run_in_threadpool(voice.speak, cfg, body.text)
out = await run_in_threadpool(voice.speak, cfg, body.text, body.voice)
if out is None:
return Response(status_code=204)
audio, media = out
return Response(content=audio, media_type=media, headers={"Cache-Control": "no-store"})


@router.get("/voices")
async def list_voices(
user: CurrentUser, cfg: Cfg, provider: str | None = Query(None, pattern="^(" + "|".join(speech.TTS_PROVIDERS) + ")$")
) -> VoiceList:
"""The voices a text-to-speech provider (else the one in the voice settings) reads in, from the provider when it
lists them (ElevenLabs, Deepgram, macOS say, OpenAI-compatible servers that do), else the ones it's known to have."""
p = provider or speech.tts_provider(cfg)
found, error = await run_in_threadpool(speech.voices, cfg, p)
return VoiceList(provider=p, current=speech.tts_voice(cfg) if p == speech.tts_provider(cfg) else None, voices=found, error=error)
13 changes: 6 additions & 7 deletions fastapi_backend/app/domain/podcasts.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
AFTER = ["analyze", "embed", "summarize"] # what any new transcript gets
STATUS = {"gather": "gathering", "plan": "planning", "write": "writing", "check": "checking", "audio": "rendering", "publish": "publishing"}
# the second host's voice when podcasts.voice_b isn't set, per text-to-speech provider (the first's is voice.tts_voice)
SECOND_VOICE = {"openai": "onyx", "elevenlabs": "EXAVITQu4vr4xnSDxMKL", "deepgram": "aura-2-orion-en"}
SECOND_VOICE = {"openai": "onyx", "elevenlabs": "EXAVITQu4vr4xnSDxMKL", "deepgram": "aura-2-orion-en", "gemini": "Puck"}
PAUSE_MS = {"turn": 450, "same": 250} # silence before a line: when the other host speaks, and when the same one goes on
LENGTHS = (5, 10, 20) # minutes offered; anything from 1 to podcasts.max_minutes goes
STYLES = {
Expand Down Expand Up @@ -718,14 +718,13 @@ def timed(lines, wpm=150, pause_ms=400):

def voices(cfg, req=None):
"""{a, b}: the hosts' voices for the text-to-speech provider: what the episode asked for, else podcasts.voice_a /
voice_b, else voice.tts_voice and a second voice of the provider's."""
voice_b, else voice.tts_voice and a second voice of the provider's (say has no voice every Mac is sure to have, so
both hosts get the same one unless podcasts.voice_b names another)."""
from . import speech

v, p = cfg.get("voice") or {}, cfg.get("podcasts") or {}
provider = v.get("tts_provider") or "openai"
first = {"openai": v.get("tts_voice") or "alloy", "elevenlabs": v.get("tts_voice") or speech.ELEVENLABS_VOICE}.get(
provider, v.get("tts_model") or speech.DEEPGRAM_TTS_MODEL
)
p = cfg.get("podcasts") or {}
provider = speech.tts_provider(cfg)
first = speech.tts_voice(cfg) # None for say: the Mac's own voice
asked = (req or {}).get("voices") or {}
return {"a": asked.get("a") or p.get("voice_a") or first, "b": asked.get("b") or p.get("voice_b") or SECOND_VOICE.get(provider, first)}

Expand Down
4 changes: 2 additions & 2 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@
"embeddings": ("api_key",),
"decisions": ("api_key",),
"voice": ("tts_api_key",),
"speech": ("openai_api_key", "elevenlabs_api_key", "assemblyai_api_key", "deepgram_api_key"),
"speech": ("openai_api_key", "elevenlabs_api_key", "assemblyai_api_key", "deepgram_api_key", "gemini_api_key"),
"mail": ("password",),
"bridge": ("token",),
"matterbridge": ("slack_token", "discord_token", "telegram_token", "matrix_password"),
Expand All @@ -137,7 +137,7 @@
("decisions", "engine"): {"auto", "jev", "laya", "llm", "off"},
("voice", "input"): {"auto", "server", "browser"},
("voice", "stt"): {"same", "sensevoice", "whisper", "mlx-whisper", "openai", "elevenlabs", "assemblyai", "deepgram"},
("voice", "tts_provider"): {"openai", "elevenlabs", "deepgram"},
("voice", "tts_provider"): {"openai", "elevenlabs", "deepgram", "gemini", "say"},
("mail", "security"): {"starttls", "ssl", "none"},
("bridge", "answer"): {"mention", "all"},
("bridge", "approve"): {"off", "low_risk"},
Expand Down
Loading
Loading
0