8000 feat(voice): add audio workspace and podcast output modes by adeelahmad · Pull Request #727 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations 8000
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
server](docs/mcp.md#one-namespaces-mcp-server), [chat completions](docs/api.md#chat-completions-for-any-app),
[API](docs/api.md), [the graph as files](docs/file-tree.md), [authentication](docs/authentication.md);
[checks](docs/requirements.md#connect-your-agents))
- **Use speech directly.** The Voice workspace turns raw text into downloadable audio and short media into text,
using the configured speech engines. Its API works at the same local or remote Lens address; anonymous raw
access is off until an admin enables it, with bounded admission and clip sizes.
([Voice service](docs/voice-service.md); [checks](docs/requirements.md#use-speech-directly))
- **Ask, and get cited answers.** A chat assistant on every page, and by voice, searches your archive by words (with
"phrases", OR, and -word to leave lines out) and by meaning, once turned on, reads it, cites it and acts on it with
tools: changes wait for your approval. A search result opens the asset with Find showing the words. Ask it about Lens itself ("what's the MCP address?", "is Telegram on?") and it
Expand Down Expand Up @@ -433,3 +437,4 @@ licenses (for example `extract-msg`, GPL-3.0, for Outlook `.msg` files); none is
Built on the [Next.js FastAPI Template](https://github.com/vintasoftware/nextjs-fastapi-template) (MIT).

<img src="https://www.google-analytics.com/g/collect?v=2&amp;tid=G-RKFB16BPXJ&amp;cid=555.lense&amp;en=page_view&amp;dl=https%3A%2F%2Fgithub.com%2Fadeelahmad%2Flense&amp;dt=lense&amp;ep.repo=lense" width="1" height="1" alt="">

1 change: 1 addition & 0 deletions changelog.d/voice-workspace.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add a Voice workspace inside Lens for raw text-to-audio and media-to-text using existing speech engines. Raw requests reject oversized text instead of silently truncating it, bound media decoding and concurrent work, and provide explicit server-only errors. Administrators can enable anonymous raw speech on demand, off by default and without opening archive or podcast access. Existing deployments are not changed automatically.
8 changes: 8 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -2055,3 +2055,11 @@ many more there are. For someone signed in without permission,
it is open to everyone); asking again replaces the request. `GET /access-requests` lists the requests waiting for an
answer in the namespaces the caller owns. Assets the caller may not see at all (restricted ones
to visitors who aren't signed in, private ones to anyone without permission) answer 404, as missing ones do.


## Raw voice service

The [Voice service](voice-service.md) reuses `/api/v1/voice/speak` and `/api/v1/voice/transcribe` for
direct audio and text, locally or at the existing remote origin. Authentication remains on by default;
`voice.public_enabled` opens only the stateless raw service on demand. Use `server_only:true` for explicit
synthesis errors instead of the assistant's brows 8 er fallb 10BC8 ack.
8 changes: 8 additions & 0 deletions docs/podcasts.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# Podcasts

The create dialog supports four output modes: **Original** reads the complete selected source text without rewriting,
**Summary** and **Explanation** generate a cited single-narrator episode, and **Dialogue** uses two hosts. Choose the
narrator or host voices from the configured speech provider, or retain the podcast defaults. Import historical text,
documents or URLs through the existing Library import flow, then select those assets to make episodes. Original mode
rejects text above the configured context limit rather than silently shortening it. Audio still requires a working
speech provider; a script-only episode is not playable audio.

**Status: phase 1 built.** Lens writes a cited, fact-checked script, records it in two voices, and serves it over the
API, MCP and the web app. Picking search hits, topics, notes and graph concepts comes next (see the
[dev log](https://github.com/adeelahmad/lense/blob/main/DEVLOG.md)).
Expand Down Expand Up @@ -97,3 +104,4 @@ the job when it's over.
- Choose the hosts' voices in the dialog; the API takes them already.
- Write long episodes in sections, for small local models.
- Check claims in batches when a script has many.

9 changes: 9 additions & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -452,3 +452,12 @@ request, to another check that proves the same thing; loosening a requirement or
| The web app works on a phone: every journey is surveyed on one, and none may take more than its baseline | [clicks.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/clicks.yml) `Survey on a phone`; [baseline-phone.json](https://github.com/adeelahmad/lense/blob/main/ci/clicks/baseline-phone.json); [phone-layout.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/phone-layout.test.tsx) |
| Heavy steps kept to their hours and to so many at a time, off by default | [test_quiet.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_quiet.py) `test_the_window_and_its_checks`, `test_heavy_steps_wait_for_their_hours_and_their_turn` |
| The default install runs on a Raspberry Pi (arm64) with a cloud model | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |


## Use speech directly

| Requirement | Checks |
|---|---|
| Raw speech is authenticated by default; anonymous opt-in is narrowly scoped, bounded and fails explicitly | [raw voice API tests](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_raw.py) |
| Decoding bounds media duration and wall time before inference | [raw voice domain tests](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voice_raw.py) |
| The workspace preserves input, supports downloads and ignores cancelled results | [workspace tests](https://github.com/adeelahmad/lense/tree/main/nextjs-frontend/components/voice/__tests__) |
3 changes: 3 additions & 0 deletions docs/speech-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ instead, which helps on a small machine such as a Raspberry Pi, or when you want
Status: built (transcription, speakers, language, sentiment, audio events, text to speech, prices per minute and per
character in the activity ledger). Planned: provider summaries, chapters and entities as optional imports.

For direct text-to-audio and media-to-text work, use the [Voice workspace and API](voice-service.md).

## Providers

| Provider | Speech to text | Speakers | Language | Emotion | Sounds | Text to speech |
Expand Down Expand Up @@ -155,3 +157,4 @@ has no Opus), in pieces when it is longer or larger than the provider takes at o
to speech sends the answer's text.

Code: `fastapi_backend/app/domain/speech.py`.

90 changes: 90 additions & 0 deletions docs/voice-service.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Voice service

The **Voice** workspace in Lens turns raw text into downloadable audio and an audio or video clip into text.
It uses the same speech providers as the assistant and podcasts. There is no second service to install, no new
queue, and no model download until an administrator selects a local engine. Open **Voice** beside **Chat**, or use
the link in **Settings → AI assistant → Voice**.

## Text and media

- Paste up to 4,000 characters, choose one of the configured provider's voices, and generate audio. The text is sent
exactly as entered, including punctuation and Markdown. The workspace plays and downloads the returned audio.
A missing provider or failed synthesis is an error, not a silent browser-speech substitute or truncated recording.
- Upload an audio or video clip, up to 25 MiB and five minutes, to get a transcript you can copy or download. This
does not create a Library asset. Use normal Library import for longer recordings and background transcription.
- Cancel stops the browser waiting and discards late results. A model call already running on the server can finish;
cancellation does not promise to stop provider billing.

Readiness, transcription engine, and provider voice-list errors are shown in the workspace. Configure the engines in
**Settings → AI assistant → Voice** and keys in **Settings → Speech providers**. See [speech providers](speech-providers.md)
for real capability and platform requirements. Kokoro and StyleTTS2 are supported existing integrations. Piper is
not a native Lens backend: an independently operated server that actually implements the OpenAI audio API can be
configured as OpenAI-compatible, but Lens does not install or validate a Piper deployment here. macOS `say` cannot
run in a Linux Proxmox container.

## The same API locally and remotely

Use your existing Lens origin, including its existing HTTPS/Cloudflare Tunnel address, with `/api/v1`:

```sh
# An ordinary Lens API token, kept in the shell environment, not in the URL.
curl --fail-with-body "$LENS_URL/api/v1/voice/speak" \
-H "Authorization: Bearer $LENS_TOKEN" -H 'Content-Type: application/json' \
--data '{"text":"Read this exact text.","server_only":true}' --output speech.audio

curl --fail-with-body "$LENS_URL/api/v1/voice/transcribe" \
-H "Authorization: Bearer $LENS_TOKEN" -H 'Content-Type: application/octet-stream' \
--data-binary @clip.wav
```

The speech response's `Content-Type` identifies its format; do not assume every provider returns MP3.
Transcription returns `text`, `language`, `engine` and `seconds`. `GET /voice` reports capability, and authenticated
`GET /voice/voices` lists selectable voices. A namespace-only key deliberately cannot use these global endpoints.
The workspace asks for `server_only:true`. Existing assistant clients can still omit it and receive 204 for browser
fallback. Blank or over-limit text returns 422; malformed media 400, oversized/long media 413, unavailable speech 503,
and an unconfigured transcription engine 409. Busy/rate-limited requests return 429 with `Retry-After`.

This feature does not create a tunnel, change firewall rules, install models on a live host, or upgrade an existing
v0.20.0 instance. Those remain operator actions after the change is reviewed and released.

## Anonymous access, only when needed

Authentication stays required by default. An administrator can enable **Allow anonymous raw voice API** in the
Voice settings (`voice.public_enabled`) and switch it off again afterward. This opens only capability, raw speech
and raw transcription. It does not open the Voice web page, provider voice lists, settings, the archive, podcasts,
or private feeds. Anonymous callers omit the Authorization header and cannot override the configured voice.
Invalid or namespace-scoped credentials do not downgrade to anonymous access.

Enabling this on an internet-reachable instance lets anyone who can reach it consume its configured speech service,
including paid providers. The raw service has no waiting queue: at most two expensive requests run per API process,
and all anonymous callers share ten provider requests per minute per process. A worker keeps its slot until it
finishes, even when the HTTP request is cancelled. At most two media uploads are admitted concurrently.
Forwarding headers do not create a fresh quota.
Uploads time out after 30 seconds; decoding is bounded to 301 seconds of output, rejects anything over 300 seconds,
and has a 30-second wall-clock timeout. Decoding accepts only media formats and local file/pipe protocols, not URL
playlists. Successful audio and transcript responses use `Cache-Control: no-store`.

These are process-local safeguards, not a global internet abuse firewall. Multiple API workers multiply the limits.
Keep anonymous mode off when not in use, configure global budgets and provider spending limits, and use an edge
rate limit for a publicly exposed deployment. Raw requests are not archive assets, but the activity ledger still
records speech use; a configured cloud provider receives the supplied text/audio under its own retention policy.

## Sources, podcasts and Apple Podcasts

Import a web URL through Lens's guarded web importer, paste text through text import, or upload a document through
the Library. Wait for its text to be extracted, then select the asset and choose **Podcast**. This preserves the
existing URL network guard, source citations, namespace/vault permissions and podcast jobs rather than making an
anonymous URL-fetching service. Do not send URLs to the raw transcription endpoint: it accepts media bytes.

Episodes use Lens's existing [podcast flow](podcasts.md). Follow a saved collection or a published private podcast
[feed](feeds.md) in Apple Podcasts using **Follow a Show by URL**. An episode must have audio and be in that feed's
selection. A script without a working TTS provider is not an audio episode. Feed URLs are bearer secrets: anyone who
has the URL can read that feed until its token is rotated. Do not publish a private feed to a podcast directory.

## Alexa

Alexa playback is not implemented by this change. A realistic follow-up is a private Alexa skill using Amazon's
AudioPlayer interface and an HTTPS audio URL reachable by Amazon, with suitable authentication/linking and verified
audio requirements. The Lens API must not be treated as an Alexa Skill endpoint, and a private Apple Podcasts feed
does not automatically become available on Alexa. Building or publishing such a skill, and exposing its media,
requires separate configuration and verification.
8 changes: 6 additions & 2 deletions fastapi_backend/app/api/v1/routes/podcasts.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Podcasts: two-host learning episodes made from picked resources and passages, every claim cited
"""Podcasts: original readings and cited learning episodes made from picked resources and passages
(app/domain/podcasts.py, docs/podcasts.md). An episode is a recording, so its audio, transcript, sharing and deletion
also go through /recordings; these routes start, show and remake episodes."""

Expand Down Expand Up @@ -41,6 +41,7 @@ def start(acl: Access, user: Principal, db: DB, cfg: dict, body: PodcastCreate)
style=body.style,
title=body.title,
voices=body.voices.model_dump(exclude_none=True) if body.voices else None,
mode=body.mode,
)
except podcasts.Problem as e:
raise HTTPException(400, str(e)) from None
Expand All @@ -54,7 +55,10 @@ def start(acl: Access, user: Principal, db: DB, cfg: dict, body: PodcastCreate)
def create_podcast(body: PodcastCreate, acl: Acl, user: Writer, db: Db, cfg: Cfg) -> PodcastStarted:
"""Make an episode about these resources and lines, in the background (watch its job on /events). It goes in the
podcasts namespace when everyone there can read every source, else in the sources' own namespace (400 when they
span several); you need editor access where it goes. Audited as `podcast.create`."""
span several); you need editor access where it goes. Original reads the complete selected text without a model,
refusing text over podcasts.context_chars; summary and explanation narrate in voice a; dialogue uses a and b.
Import text, documents and URLs through the existing import routes before selecting their asset ids here.
Audited as `podcast.create`."""
return start(acl, user, db, cfg, body)


Expand Down
Loading
Loading
0