8000 feat(files): translate an asset's transcript into another language, when asked by adeelahmad · Pull Request #684 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,15 +181,17 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
(noise, volume, echo, and silence left out, with the transcript's times kept), and transcripts cleaned up before
analysis (stutters, "um"s, spacing and, with a local model, punctuation put back, and grammar by a LanguageTool server
you run), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each
off by default. Words transcription wasn't sure of are underlined, with how sure, to check and correct in place.
off by default. Words transcription wasn't sure of are underlined, with how sure, to check and correct in place. Translate a transcript into another language when you ask, with the estimate shown first: it's kept
beside the asset as a translation whose lines keep their times, found by search like an uploaded one.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
[enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings),
[cleaning up audio](docs/processing.md#cleaning-up-audio),
[your own names](docs/processing.md#your-own-names-in-transcription),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates),
[unsure words](docs/processing.md#unsure-words); [checks](docs/requirements.md#understand-it))
[unsure words](docs/processing.md#unsure-words),
[translations](docs/api.md#translating-a-transcript); [checks](docs/requirements.md#understand-it))
- **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its
source, read aloud in two voices, by Kokoro on this machine or a speech provider. Turn a voice session or a meeting into a technical spec, in a note linked to it,
that keeps every decision, every no and every constraint in the speaker's own words, linked to when they said them,
Expand Down
9 changes: 9 additions & 0 deletions changelog.d/translate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
- **Translate a transcript.** The Files tab's **Translate…** has the configured model translate an asset's transcript
into another language (the one your browser reads is filled in), showing what it would take first, always labelled
an estimate. It runs only when asked, in the background, sending the lines in batches marked as data so instructions
in a transcript are never followed, and keeps the result as a translation file whose lines keep their times and
speaker, read by search, the Files tab and IIIF like an uploaded one. Every call is in the activity ledger; a budget
that would go over holds it, and one used up stops it. Agents use `GET` and
`POST /api/v1/resources/{id}/files/translate`, the MCP tools `translate_transcript` and `get_translation`, and the
assistant's `translate_transcript`, which asks first
([Translating a transcript](docs/api.md#translating-a-transcript)).
31 changes: 31 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1009,6 +1009,8 @@ PATCH /api/v1/resources/{rid}/files/{fid}
DELETE /api/v1/resources/{rid}/files/{fid}
GET /api/v1/resources/{rid}/files/{fid}/download
GET /api/v1/resources/{rid}/files/{fid}/lines
GET /api/v1/resources/{rid}/files/translate?language=
POST /api/v1/resources/{rid}/files/translate {"language": "fr"}
```

A resource's files: its primary file (`primary`: the audio or video its pipeline runs on, none for text
Expand Down Expand Up @@ -1041,6 +1043,35 @@ stay with a resource that moves to another namespace and go when it's deleted.
([Access](access.md#files)): transcripts, captions and translations with its text, indexes with its index,
thumbnails with its media. Attachments always need permission.

### translating a transcript

Lens can make a translation as well as keep an uploaded one. `POST …/files/translate` with a `language` (a code such
as `fr`, `de` or `pt-BR`) has the configured model translate the resource's transcript, and keeps the result as a
supplementary file with the role `translation` and that language, exactly like an uploaded one: search finds its
lines, the Files tab shows them and IIIF publishes it, and it downloads as Lens's own transcript JSON (`lens/1`, which
imports straight back in), named after the resource (`Interview (fr).json`), with a description that says the model
made it. Each line keeps its original's times and speaker, so a timed transcript gives a timed translation.

It runs only when someone asks for it, never on its own: in the web app from the Files tab's **Translate…** (the
language you read, from your browser, is filled in, unless the transcript is already in it), through the API or the
MCP tool `translate_transcript`, or by the assistant's `translate_transcript` once the person approves it. It runs as
a background job (202 with the `job`; asking again while it runs gives the same job) after what the resource's job
still has to do. The lines go to the model in batches of up to 40 lines and 6,000 characters, numbered and marked as
data ([prompt injection](assistant.md#third-party-content-and-prompt-injection)), so instructions written in a
transcript are translated, never followed. A line the model doesn't give back is asked for once more on its own, then
kept as it was, and the file's description says how many were.

Every call is in the [activity ledger](activity.md), counted for the job, the resource and its namespace. A
[budget](budgets.md) the run would go over holds the job until someone picks, as for any job; before each batch Lens
looks again, so a budget used up part-way stops the translation (the job fails saying so, and no file is kept).

`GET …/files/translate` says beforehand what it would take: the transcript's `lines`, the model calls and tokens in
`llm`, and the cost at the model's price in Settings (`llm.cost`, null without a price), always an estimate
(`estimate` is true, with its `basis`); `problem` says why it can't run now (no model configured, no transcript, a
budget used up), and with `language`, `job` is the one already translating it into that language. `POST` needs the
right to process the resource: 400 without a model or a transcript, for a language that isn't a code, or one the
transcript is already in; 409 over budget. Asking is audited (`file.translate`).

## imports

```
Expand Down
3 changes: 2 additions & 1 deletion docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,8 @@ When the configured model supports function calling, chat becomes an agent.
- **Citations:** every moment a tool returns is numbered, so answers cite [n] across everything the assistant read.
- **Visible steps:** each tool call streams as a step, for example 'Searched for "refund": 42 matches'.
- **Approvals:** running a template on assets, writing a [spec](notes.md#specs-from-transcripts) from an asset's
transcript (`write_spec`), changing entities (merging, renaming, retyping, describing one
transcript (`write_spec`), [translating](api.md#translating-a-transcript) an asset's transcript into another
language (`translate_transcript`, with its estimate), changing entities (merging, renaming, retyping, describing one
and its other names, hiding one, or adding one to a namespace's fixed list) and changing topics (merging,
renaming, describing or deleting one; [Topics](topics.md)) don't happen directly.
They become approval cards (with the batch estimate) that the person approves, approves on a sample, or declines
Expand Down
4 changes: 3 additions & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `create_feed` | make an RSS feed or a podcast feed in a namespace, for feed readers and podcast apps. Needs a write-scope token and editor access |
| `publish_to_feed` | add resources, episodes, timeline activities and a text item to a feed; what's there already isn't added twice, and what may not show there comes back refused, with why. Needs a write-scope token and editor access |
| `write_spec` | write a technical [spec](notes.md#specs-from-transcripts) from a recording's transcript as a note linked to it: every decision, no and constraint in the speaker's own words with links to when they said them, and the gaps left to decide later; it's written in the background. Needs a write-scope token and the right to process the recording |
| `translate_transcript` | [translate](api.md#translating-a-transcript) a recording's transcript into another language (`fr`, `pt-BR`) with the configured model, kept as a translation file beside it whose lines keep their times and speaker; it runs in the background, only when asked, within budgets. Needs a write-scope token and the right to process the recording |
| `get_translation` | before translating: the transcript's lines, the model calls, tokens and cost it would take (always an estimate) and why it can't be translated now, if it can't; after: its translation files and the latest job translating it, with its status |
| `get_spec` | how a recording's spec is coming (`writing`, `ready` or `failed` with why), its note's id and link, and how many decisions, things turned down, constraints and gaps it holds and quotes the transcript doesn't |
| `email_replies` | email that waits for an answer (not a mailing list's, a no-reply sender's or the person's own) and every draft, newest first, or one [reply](email-replies.md) status |
| `draft_reply` | draft a [reply](email-replies.md) to an email from what its namespace holds, each fact cited to its excerpt and what the archive doesn't say marked to confirm; `instructions` says what it should say. It's drafted in the background and nothing is sent. Needs a write-scope token and the right to process the email |
Expand Down Expand Up @@ -205,7 +207,7 @@ which records an undoable change (proposed by default), `undo_graph_change`, `ed
app (merges and deletes can be undone), `write_note`, which adds a note, `create_podcast`, which
queues an episode, `create_feed` and `publish_to_feed`, which make and fill a [published feed](feeds.md),
`save_circle`, `share_with_circle` and `undo_share`, which change your [circles](circles.md) and what you share with them,
`write_spec`, which queues a spec, `draft_reply` and `edit_reply`, which draft and change [email replies](email-replies.md) (nothing is sent), `send_reply`, which asks to send one (it waits in Approvals), `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and
`write_spec`, which queues a spec, `translate_transcript`, which queues a translation, `draft_reply` and `edit_reply`, which draft and change [email replies](email-replies.md) (nothing is sent), `send_reply`, which asks to send one (it waits in Approvals), `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and
editors), `enhance`, which enhances an image and reads it again (the process permission),
`set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `set_secret_reference`, which points a secret field at a password manager's item (admins), `install_plugin`, which
installs a plugin from a marketplace as your own extension, `add_mcp_server`, which adds an
Expand Down
3 changes: 3 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -821,6 +821,9 @@ audio skip themselves for imported text. Steps:
which changes nothing anyone relied on, since search didn't look at outputs before.
- `spec`: writes a technical spec from the transcript into a note linked to the asset, with the Spec from transcript
template unless the step names another prompt template ([Specs from transcripts](notes.md#specs-from-transcripts)).
- `translate` (not a pipeline step: it runs only when someone asks): translates the transcript into a language with
the configured model, in batches of lines that keep their times, and keeps it as a translation file
([Translating a transcript](api.md#translating-a-transcript)).
- `report` with a template: renders an HTML page into the namespace's reports. These pages may not run scripts.
- `export`: renders a template to a file (Markdown, text, HTML...) and can copy it to a storage source.
- `convert`: makes new assets from one, a video's soundtrack as audio or a document's pages as images
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Grammar fixed by a LanguageTool server you run, when its address is set: grammar fixes only, never spelling or a word with a capital | [test_cleanup.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_cleanup.py) `test_grammar_fixes_from_languagetool_leave_names_and_spelling`, `test_without_its_server_the_rules_run_alone`, `test_the_grammar_server_is_an_address_without_a_password` |
| Entities found by spaCy, fetched the first time it's picked, with the defined entities first and the rules until it's there | [test_spacy_entities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_spacy_entities.py) `test_defined_entities_come_first_then_spacy_s_names`, `test_the_analyze_step_and_workflows_use_it`, `test_without_spacy_the_rules_find_them`, `test_spacy_is_fetched_only_when_it_is_picked` |
| Dates said in an asset read as days ("next Tuesday", "last week"), off by default, with dates that read two ways left out | [test_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_dates.py) `test_dates_are_read_as_days`, `test_what_could_be_two_dates_is_left_out`, `test_off_by_default_reads_none` |
| A transcript translated into another language only when someone asks, the estimate first: in batches of lines that keep their times and speaker, the transcript marked as data, every call in the activity ledger and within budgets, kept as a translation file like an uploaded one; through the web app's Files tab, the API, MCP and the assistant (asking first) | [test_translations.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_translations.py) `test_translating_writes_a_translation_file`, `test_the_transcript_reaches_the_model_as_data`, `test_lines_go_in_batches_and_keep_their_place`, `test_a_spent_budget_refuses`, `test_the_estimate_comes_first`, `test_what_stops_a_translation`, `test_agents_translate`, `test_the_assistant_asks_first`; [recording-translate.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-translate.test.tsx) `is the first the browser reads that the transcript isn't in`, `translates into the browser's language with Enter, after showing the estimate` |

## Learn from it

Expand Down
57 changes: 57 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -2503,6 +2503,63 @@ def get_spec(ctx: Context, recording_id: int) -> dict[str, Any]:
return store.clean({**s, "note_id": note, "url": _note_url(ctx, note) if note else None, "recording_url": ctx.link(recording_id)})


# ---------- translating a transcript (docs/api.md#translating-a-transcript) ----------
@tool(
"translate_transcript",
"Translate a transcript",
"Translate a recording's transcript into another language with the configured model, in the background, and keep it "
"as a supplementary file with the role translation and that language, each line with its original times and "
"speaker, so search and the recording's page read it like an uploaded translation. It costs model calls: look at "
"get_translation's estimate first, and translate only when the person asked for it. Asking again while it runs "
"gives the same job. Check on it with get_translation. Needs a token with the write scope and the right to process "
"the recording; refused without a model or a transcript, or when a budget is used up.",
Arg("recording_id", "integer", "the recording", required=True),
Arg("language", "string", "the language to translate into, as a code such as fr, de or pt-BR", required=True),
writes=True,
)
def translate_transcript(ctx: Context, recording_id: int, language: str) -> dict[str, Any]:
from app.api.v1.routes import files as file_routes
from app.schemas.files import TranslationCreate

if not ctx.user.can_write:
raise ToolError("this token is read-only: ask for a token with the write scope to translate transcripts")
try:
body = TranslationCreate(language=language)
except ValueError:
raise ToolError("give the language as a code such as fr or pt-BR") from None
out = file_routes.translate_transcript(recording_id, body, ctx.acl, ctx.user, ctx.db, ctx.cfg)
return {"job": out.job, "status": "translating", "language": body.language, "recording_url": ctx.link(recording_id)}


@tool(
"get_translation",
"Read a transcript's translations",
"Before translating a recording's transcript: what it would take (its lines, model calls, tokens and the cost at "
"the model's price in Settings, always an estimate, with how it was worked out) and why it can't be translated now, "
"if it can't. After: its translation files (uploaded or made: id, name, language, lines, whether they're timed) and "
"the latest job that was to translate it (into language, when given), with its status and, if it failed, why.",
Arg("recording_id", "integer", "the recording", required=True),
Arg("language", "string", "only the job translating it into this language (a code such as fr)"),
)
def get_translation(ctx: Context, recording_id: int, language: str | None) -> dict[str, Any]:
from app.domain import translations

ctx.acl.recording(recording_id)
try:
code = translations.language(language) if language else None
except translations.Problem as e:
raise ToolError(str(e)) from None
est = translations.estimate(ctx.db, ctx.cfg, recording_id)
return store.clean(
{
**translations.status(ctx.db, recording_id, code),
"estimate": {k: est[k] for k in ("lines", "llm", "basis", "estimate")},
"problem": translations.problem(ctx.db, ctx.cfg, recording_id),
"recording_url": ctx.link(recording_id),
}
)


# ---------- email replies (docs/email-replies.md) ----------
def _reply_out(ctx: Context, rid: int, r: dict[str, Any]) -> dict[str, Any]:
"""A reply as agents get it: its sources linked into the archive, and where the person reviews it."""
Expand Down
Loading
Loading
0