adeelahmad · Oct 4, 2026 · Oct 4, 2026 · Oct 4, 2026 · Oct 4, 2026
diff --git a/DEVLOG.md b/DEVLOG.md
@@ -2,6 +2,39 @@
 
 Plans and progress for work in flight. Newest first.
 
+## 2026-10-04 · Cloud speech providers and local GGUF models
+
+Goal (Adeel): transcription, speaker separation and emotion stay local by default (SenseVoice and friends, unchanged),
+but each voice task can be sent to a provider instead: an OpenAI-compatible Whisper endpoint, ElevenLabs (speech to
+text and text to speech), AssemblyAI and Deepgram (every capability that fits). Every service takes a custom base URL,
+for proxies and compatible servers. Also: run a local LLM with llama.cpp from a list of GGUF models this machine can run,
+downloaded from Hugging Face.
+
+Model:
+
+- Settings → Speech providers (`speech` section): per service a base URL, a model and an API key (sealed like every
+  other key). The defaults point at each vendor; change the URL for a proxy or a compatible server.
+- `transcribe.engine` gains openai, elevenlabs, assemblyai, deepgram. Their transcripts carry the provider's speaker
+  labels, language, audio events and sentiment (as emotion); `diarize.engine` gains provider, and auto uses those
+  labels when a transcript has them. Voice IDs across recordings still come from the local voiceprints.
+- Voice chat: `voice.stt` picks its own engine (default: the transcription engine); `voice.tts_provider` adds
+  ElevenLabs and Deepgram Aura next to the OpenAI-compatible speech server.
+- Provider calls go through the activity ledger once it lands (seconds of audio per call).
+- Local LLM: a catalog of GGUF chat models with the memory each needs; the ones this machine can run are offered,
+  downloaded from Hugging Face into data_dir/models/gguf, and served by llama.cpp's server, which becomes the LLM
+  provider.
+
+Todo:
+
+- [x] Speech providers: settings, OpenAI-compatible, ElevenLabs, AssemblyAI and Deepgram transcription with speakers,
+      language, events and sentiment; provider speaker labels in speaker separation
+- [x] Voice chat: own speech-to-text engine; ElevenLabs and Deepgram text to speech
+- [x] Settings → Speech providers in the web app; new engines in Transcription and Speaker separation
+- [ ] Local LLM: GGUF catalog filtered by this machine, download from Hugging Face, llama.cpp server as the provider
+
+Refine later: per-minute prices for provider calls in the ledger; AssemblyAI and Deepgram summaries, chapters and
+entities as optional imports next to Lens's own analysis; several endpoints per service; speaker labels kept across
+chunks of long OpenAI-compatible transcriptions.
 ## 2026-10-04 · Topics: a controlled vocabulary apart from entities
 
 Goal (Adeel): entities and topics are mixed (a topic is an entity of type TERM, which is also the fallback type for

diff --git a/docs/speech-providers.md b/docs/speech-providers.md
@@ -0,0 +1,57 @@
+# Speech providers
+
+Lens transcribes, tells speakers apart and reads emotion on your own machine by default (SenseVoice, Whisper,
+SpeechBrain). Nothing changes until an admin picks a provider. Any of these voice tasks can go to a speech provider
+instead, which helps on a small machine such as a Raspberry Pi, or when you want a provider's accuracy.
+
+Status: built (transcription, speakers, language, sentiment, audio events, text to speech). Planned: provider
+summaries, chapters and entities as optional imports, per-minute prices in the activity ledger.
+
+## Providers
+
+| Provider | Speech to text | Speakers | Language | Emotion | Sounds | Text to speech |
+| --- | --- | --- | --- | --- | --- | --- |
+| OpenAI-compatible | `/audio/transcriptions` (whisper-1, gpt-4o-transcribe) | models named `…diarize` | yes | no | no | `/audio/speech` |
+| ElevenLabs | Scribe | yes | yes | no | laughter, applause, … | yes |
+| AssemblyAI | Universal, Slam-1 | yes | detected | from sentiment | no | no |
+| Deepgram | Nova | yes | detected | from sentiment (English) | no | Aura |
+
+Sentiment becomes emotion as positive → Happy, negative → Sad, neutral → Neutral (turn it off with
+`speech.sentiment`). SenseVoice's own emotion tags stay richer; a provider only reports sentiment.
+
+## Settings
+
+**Settings → Speech providers** has, for each provider, an address, a speech-to-text model and an API key. Keys are
+stored encrypted like every other key. The Test button checks the address and key without billing anything.
+
+Every address can be changed: point it at a proxy, a regional endpoint (`https://api.eu.assemblyai.com`), a
+self-hosted Deepgram, or any server that speaks the OpenAI audio API (speaches, LocalAI, Groq's
+`https://api.groq.com/openai/v1`). An OpenAI-compatible server on your own network needs no key.
+
+Then pick where each task goes:
+
+- **Transcription → Engine**: SenseVoice, Whisper, mlx-whisper, or a provider. A provider's transcript carries its
+  speakers, language, sounds and sentiment.
+- **Speaker separation → Method**: *Speech provider* uses the speakers the provider found; *Auto* uses them when a
+  transcript has some, and otherwise works as before. Voice IDs across recordings still come from local voiceprints.
+- **AI assistant → Voice**: *Engine that hears it* (default: the transcription engine) and *Spoken answers by* (an
+  OpenAI-compatible speech server, ElevenLabs or Deepgram Aura).
+
+| Setting | Default |
+| --- | --- |
+| `speech.openai_base_url`, `openai_model` | `https://api.openai.com/v1`, `whisper-1` |
+| `speech.elevenlabs_base_url`, `elevenlabs_model` | `https://api.elevenlabs.io`, `scribe_v1` |
+| `speech.assemblyai_base_url`, `assemblyai_model` | `https://api.assemblyai.com`, `universal` |
+| `speech.deepgram_base_url`, `deepgram_model` | `https://api.deepgram.com`, `nova-3` |
+| `speech.sentiment` | on |
+| `speech.timeout` | 1800 seconds |
+| `voice.stt` | `same` |
+| `voice.tts_provider` | `openai` |
+
+## What is sent
+
+Only the audio of the recording (or the clip said into the mic), encoded as Ogg Opus at 32 kbit/s (FLAC when ffmpeg
+has no Opus). Recordings longer than ten minutes go to OpenAI-compatible servers in ten-minute parts, since they take
+at most 25 MB a request; speaker labels then belong to each part. Text to speech sends the answer's text.
+
+Code: `fastapi_backend/app/domain/speech.py`.
diff --git a/fastapi_backend/app/api/v1/routes/admin.py b/fastapi_backend/app/api/v1/routes/admin.py
@@ -13,7 +13,7 @@
 from app import email
 from app.api.deps import AdminReader, AdminWriter, Cfg, Db, domain_errors
 from app.core.middleware import host_name
-from app.domain import auth, bridge, jobs, llm, semantic, settings, sources, store, telemetry, tunnel
+from app.domain import auth, bridge, jobs, llm, semantic, settings, sources, speech, store, telemetry, tunnel
 from app.schemas.admin import (
     AuditEntry,
     BridgeStatus,
@@ -24,6 +24,7 @@
     LlmTestResult,
     MailTestResult,
     SemanticStatus,
+    SpeechTestResult,
     Started,
     TelemetryStatus,
     TelemetryTestResult,
@@ -122,6 +123,16 @@ def test_embeddings(user: AdminWriter, cfg: Cfg, db: Db) -> EmbedTestResult:
     return EmbedTestResult(ok=True, dimension=len(vec), ms=int((time.time() - t0) * 1000), model=semantic.endpoint(cfg)[2])
 
 
+@router.post("/settings/speech/test")
+def test_speech(
+    user: AdminWriter, cfg: Cfg, provider: str = Query(..., pattern="^(openai|elevenlabs|assemblyai|deepgram)$")
+) -> SpeechTestResult:
+    """Check a speech provider's address and key with the saved settings (lists its models; nothing is billed)."""
+    t0 = time.time()
+    ok, detail = speech.check(cfg, provider)
+    return SpeechTestResult(ok=ok, error=None if ok else detail, detail=detail if ok else None, ms=int((time.time() - t0) * 1000))
+
+
 @router.get("/admin/semantic")
 def semantic_status(user: AdminReader, cfg: Cfg, db: Db) -> SemanticStatus:
     """Search by meaning: whether it's set up, its model, and how many recordings are indexed with it."""

diff --git a/fastapi_backend/app/domain/ingest.py b/fastapi_backend/app/domain/ingest.py
@@ -15,7 +15,7 @@
 
 import numpy as np
 
-from . import store
+from . import speech, store
 
 TAG_RX = re.compile(r"<\|([^|]*)\|>")
 SR = 16000
@@ -359,6 +359,11 @@ def _make_engine(cfg, e):
         return SenseVoice(cfg)
     if e in ("whisper", "mlx-whisper"):
         return Whisper(cfg, mlx=e == "mlx-whisper")
+    if e in speech.PROVIDERS:
+        try:
+            return speech.Engine(cfg, e)
+        except speech.ProviderError as err:
+            raise EngineMissing(str(err)) from None
     raise SystemExit(f"unknown transcribe.engine {e!r}")
 
 
@@ -404,6 +409,7 @@ def segment_rows(rid, nid, segs):
                     "t1": t1,
                     "dur": t1 - t0,
                     "local_speaker": s.get("speaker"),
+                    "provider_speaker": s.get("provider_speaker"),  # a speech provider's label (domain/speech.py)
                     "text": s["text"],
                     "raw_text": s.get("raw_text"),
                     "emotion": store.norm_emotion(s.get("emotion")),

diff --git a/fastapi_backend/app/domain/settings.py b/fastapi_backend/app/domain/settings.py
@@ -38,6 +38,7 @@
     "decisions": None,
     "components": None,
     "voice": None,
+    "speech": None,
     "mail": None,
     "bridge": None,
     "notifications": None,
@@ -91,16 +92,17 @@
     "embeddings": ("api_key",),
     "decisions": ("api_key",),
     "voice": ("tts_api_key",),
+    "speech": ("openai_api_key", "elevenlabs_api_key", "assemblyai_api_key", "deepgram_api_key"),
     "mail": ("password",),
     "bridge": ("token",),
     "telemetry": ("headers",),
     "fedora": ("password",),
     "tunnel": ("token", "api_token"),
 }
 ENUMS = {
-    ("transcribe", "engine"): {"sensevoice", "whisper", "mlx-whisper"},
+    ("transcribe", "engine"): {"sensevoice", "whisper", "mlx-whisper", "openai", "elevenlabs", "assemblyai", "deepgram"},
     ("transcribe", "device"): {"auto", "cpu", "cuda", "mps"},
-    ("diarize", "engine"): {"auto", "channels", "cluster", "pyannote", "none"},
+    ("diarize", "engine"): {"auto", "channels", "cluster", "pyannote", "provider", "none"},
     ("speakers", "embedder"): {"speechbrain", "none"},
     ("speakers", "cross_namespace"): {"suggest", "off"},
     ("analysis", "entities"): {"rules", "spacy"},
@@ -111,6 +113,8 @@
     ("video", "object_engine"): {"yolox", "ultralytics", "off"},
     ("decisions", "engine"): {"auto", "jev", "llm", "off"},
     ("voice", "input"): {"auto", "server", "browser"},
+    ("voice", "stt"): {"same", "sensevoice", "whisper", "mlx-whisper", "openai", "elevenlabs", "assemblyai", "deepgram"},
+    ("voice", "tts_provider"): {"openai", "elevenlabs", "deepgram"},
     ("mail", "security"): {"starttls", "ssl", "none"},
     ("bridge", "answer"): {"mention", "all"},
     ("tunnel", "mode"): {"off", "quick", "token", "managed"},
@@ -376,6 +380,8 @@ def _check(section, key, value, default):
         if not (isinstance(value, str) and len(value.strip()) <= 500):
             raise ValueError(f"voice.{key} is text")
         return value.strip()
+    if section == "speech":
+        return _speech_setting(key, value)
     if section == "decisions" and key != "engine":
         return _decision_setting(key, value)
     if (section, key) == ("documents", "attachment_resources"):
@@ -423,6 +429,26 @@ def _check(section, key, value, default):
     return value
 
 
+def _speech_setting(key, value):
+    if key.endswith("_base_url"):
+        if not (isinstance(value, str) and VIEWER_URL.match(value.strip())):
+            raise ValueError(f"speech.{key} is an http(s) address (the provider's, a proxy's or a compatible server's)")
+        return value.strip().rstrip("/")
+    if key.endswith("_model"):
+        if not (isinstance(value, str) and value.strip() and len(value.strip()) <= 200):
+            raise ValueError(f"speech.{key} is a model's name")
+        return value.strip()
+    if key == "sentiment":
+        if not isinstance(value, bool):
+            raise ValueError("speech.sentiment is true or false")
+        return value
+    if key == "timeout":
+        if not (isinstance(value, int) and not isinstance(value, bool) and 30 <= value <= 86400):
+            raise ValueError("speech.timeout is a whole number of seconds from 30 to 86400")
+        return value
+    raise ValueError(f"unknown setting speech.{key}")
+
+
 EMAIL_RX = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
 
 

diff --git a/fastapi_backend/app/domain/speakers.py b/fastapi_backend/app/domain/speakers.py
@@ -280,12 +280,15 @@ def assign_labels(db, nid, rid, mapping):
 
 def diarize_one(db, cfg, rid, log=print):
     r = db.one("SELECT record::id(id) AS id, space, path, source, channels, title, remote FROM $r", r=R("recording", rid))
-    segs = db.rows("SELECT record::id(id) AS id, idx, t0, t1 FROM segment WHERE recording = $r ORDER BY idx", r=rid)
+    segs = db.rows("SELECT record::id(id) AS id, idx, t0, t1, provider_speaker FROM segment WHERE recording = $r ORDER BY idx", r=rid)
     engine = cfg["diarize"]["engine"]
     path = ingest.audio_path(db, cfg, r) if r.get("source") == "audio" and engine != "none" and segs else None
     labels, how, mono, stereo = [None] * len(segs), "none", None, None
     if segs and r.get("source") == "audio" and engine != "none":
-        if engine in ("auto", "channels") and (r.get("channels") or 1) >= 2:
+        if engine in ("auto", "provider") and any(s.get("provider_speaker") for s in segs):
+            # the speech provider that transcribed it told speakers apart (domain/speech.py)
+            labels, how = [s.get("provider_speaker") for s in segs], "provider"
+        if how == "none" and engine in ("auto", "channels") and (r.get("channels") or 1) >= 2:
             stereo = ingest.decode(path, channels=2)
             if engine == "channels" or looks_dual_channel(stereo):
                 labels, how = channel_labels(stereo, segs), "channels"