8000 feat(language): set or correct an asset's language as a BCP 47 tag with a region by adeelahmad · Pull Request #693 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
(noise, volume, echo, and silence left out, with the transcript's times kept), and transcripts cleaned up before
analysis (stutters, "um"s, spacing and, with a local model, punctuation put back, and grammar by a LanguageTool server
you run), entities found by spaCy instead of the rules, and the dates people say read as days on the calendar, each
off by default. Words transcription wasn't sure of are underlined, with how sure, to check and correct in place. Translate a transcript into another language when you ask, with the estimate shown first: it's kept
off by default. Words transcription wasn't sure of are underlined, with how sure, to check and correct in place. The language
transcription hears is kept on the asset, and you can correct it with a region (en-GB or en-US, pt-BR or pt-PT), which
transcribing again keeps. Translate a transcript into another language when you ask, with the estimate shown first: it's kept
beside the asset as a translation whose lines keep their times, found by search like an uploaded one.
A date said ("let's meet next Tuesday at 3") becomes a calendar event in one click,
prefilled from the line and added only once you approve it ([propose an event](docs/processing.md#propose-an-event)).
Expand All @@ -195,7 +197,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[cleaning up audio](docs/processing.md#cleaning-up-audio),
[your own names](docs/processing.md#your-own-names-in-transcription),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates),
[unsure words](docs/processing.md#unsure-words),
[unsure words](docs/processing.md#unsure-words), [an asset's language](docs/api.md#an-assets-language),
[translations](docs/api.md#translating-a-transcript), [captions](docs/video.md#captions);
[checks](docs/requirements.md#understand-it))
- **Learn from it.** Make a two-host podcast episode about what you pick, every claim cited and checked against its
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/asset-language.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
- **Correct an asset's language, with its region.** The language transcription hears stays on the asset, and editors
can set or correct it as a BCP 47 tag with a region when it matters (`en-GB` or `en-US`, `pt-BR` or `pt-PT`) in the
asset's Details tab, with `PATCH /api/v1/resources/<id> {"language": "en-gb"}` or the MCP tool `set_language`. The
tag is checked and written the usual way (`en-GB`, `zh-Hant-TW`), the IIIF manifest and Dublin Core record give it,
and transcribing the asset again keeps a person's language instead of the detected one; `null` forgets it. A
pipeline step's `languages` condition of `en` now takes `en-GB` and `en-US` too. Languages already stored stay as
they are ([An asset's language](docs/api.md#an-assets-language)).
16 changes: 15 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -728,7 +728,7 @@ many changed. `GET /resources/tags` (`ns` for one namespace) lists the tags in u
most used first.

`PATCH /resources/{rid}` with `{"title": …}` renames an asset (editors; whitespace is collapsed, at most 200
characters). It is audited as `recording.rename`; the asset's report page follows the new title and is rebuilt,
characters), and with `{"language": …}` sets its language ([An asset's language](#an-assets-language)). It is audited as `recording.rename`; the asset's report page follows the new title and is rebuilt,
and a published asset shows up as an Update in the IIIF change feed.

`DELETE /resources/{rid}` deletes an asset (owners). Everything Lens made from it goes: its text and analysis
Expand Down Expand Up @@ -788,6 +788,20 @@ all of the asset (owners). `PUT …/{gid}` opens the asset to a group that opens
that opens `everything` already), `DELETE …/{gid}` closes it again (404 when it wasn't open). Both answer with the list
and are audited as `recording.ip_group.open` and `recording.ip_group.close`.

### An asset's language

Transcription keeps the language it hears on the asset (`language`): SenseVoice's, the one Whisper detects when
Settings → Transcription leaves the language on auto, or a speech provider's, as a short code such as `en`. A person
can set or correct it as a BCP 47 tag, with a region when it matters (`en-GB` or `en-US`, `pt-BR` or `pt-PT`):
`PATCH /resources/{rid}` with `{"language": "en-gb"}` (editors) checks the tag and writes it the usual way (`en-GB`,
`zh-Hant-TW`, `es-419`); one that isn't a tag is a 400. `GET /resources/{rid}` gives `language` and `language_set`,
true when a person set it: transcribing it again, importing a transcript over it or reading it again keeps their
language instead of the detected one. `{"language": null}` forgets it, so the next transcription says again. Changes
are audited as `recording.language`, and a published asset shows up as an Update in the IIIF change feed (its
manifest and Dublin Core record give the tag). The MCP tool `set_language` does the same, and the asset's Details tab
in the web app shows the language by name with its tag, which editors change there. The Language filter counts `en-GB`
apart from `en`; a pipeline step's `languages` condition of `en` takes `en-GB` and `en-US` too.

### Documents and images

A document (a PDF; a Word, PowerPoint or spreadsheet file, OpenDocument or RTF; text or Markdown; a saved web page; an
Expand Down
4 changes: 3 additions & 1 deletion docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,9 @@ as missing on older rows: a chat message's record (`meta`,
none. [Deep research](assistant.md#deep-research) is another: its tables (`research`, and `research_run` for the claim
of the process running one) are new, a message's `kind` is missing on older ones, and notes accept `chat` links
without rewriting any. [Import runs](api.md#import-runs) are a third: the `import_run` table and the index on a
recording's `import_run` are new, and assets imported before have no run. Anything else, such as rewriting stored values, moving data between
recording's `import_run` are new, and assets imported before have no run. [An asset's
language](api.md#an-assets-language) is a fourth: `language_set` is new, assets from before have none (their language
reads as detected), and the codes already stored (`en`, `de`, `zh`) stay valid as they are. Anything else, such as rewriting stored values, moving data between
tables, or changing or removing an index that already exists, is a named step in `app/domain/migrations.py`:

* Every process that opens the database (the API, workers, `lens watch`, `lens` commands) runs the steps the database
Expand Down
3 changes: 2 additions & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `nsfw_settings` | how [NSFW classification](nsfw.md) works in a namespace: on or off, the threshold, how flagged resources show, and how many are classified, flagged and not classified yet |
| `set_nsfw` | turn NSFW classification on or off, set its threshold and display, for a namespace (owners) or the archive (admins), and optionally classify what's there. Needs a write-scope token |
| `mark_nsfw` | mark a resource NSFW or not by hand, or hand it back to the model's score. Needs a write-scope token and editor access |
| `set_language` | set or correct the [language an asset is in](api.md#an-assets-language) as a BCP 47 tag with a region if it matters (`en-GB`, `pt-BR`), written the usual way and kept over the one transcription detects; left out, it's forgotten. Needs a write-scope token and editor access |
| `enhance` | [enhance an image](enhance.md) whatever the settings, keeping the original, and read it again from the enhanced copy. Needs a write-scope token and the process permission |
| `namespace_integrations` | whether a namespace has [its own MCP server](#one-namespaces-mcp-server) and [chat completions](api.md#chat-completions-for-any-app) on, their addresses, and the tools offered there |
| `set_namespace_integrations` | turn a namespace's own MCP server or chat completions on or off, and pick the read tools its chat completions may use. Owners, with a write-scope token |
Expand Down Expand Up @@ -212,7 +213,7 @@ app (merges and deletes can be undone), `write_note`, which adds a note, `create
queues an episode, `create_feed` and `publish_to_feed`, which make and fill a [published feed](feeds.md),
`save_circle`, `share_with_circle` and `undo_share`, which change your [circles](circles.md) and what you share with them,
`write_spec`, which queues a spec, `translate_transcript`, which queues a translation, `draft_reply` and `edit_reply`, which draft and change [email replies](email-replies.md) (nothing is sent), `send_reply`, which asks to send one (it waits in Approvals), `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and
editors), `enhance`, which enhances an image and reads it again (the process permission),
editors), `set_language`, which sets an asset's language (editors), `enhance`, which enhances an image and reads it again (the process permission),
`set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `set_secret_reference`, which points a secret field at a password manager's item (admins), `install_plugin`, which
installs a plugin from a marketplace as your own extension, `add_mcp_server`, which adds an
MCP server (admins), `subscribe_feed` and `unsubscribe_feed`, which follow and stop following news feeds (admins), `set_mail_routing`, which changes where an email account's messages go (admins), `add_mesh_node`,
Expand Down
6 changes: 4 additions & 2 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ How assets move through Lens: where they come from, what each step does, and how
file). What one namespace's scan brings in is one [import run](api.md#import-runs), undone whole.
- `clean_audio` (only in a pipeline that has it) cleans up the sound before it's transcribed: noise, volume, rumble,
echo, and silence left out of what's transcribed ([Cleaning up audio](#cleaning-up-audio)).
- `transcribe` uses SenseVoice, faster-whisper or mlx-whisper. When the configured engine isn't installed on a worker (or
- `transcribe` uses SenseVoice, faster-whisper or mlx-whisper, and keeps the language it heard on the asset, unless a
person set one ([An asset's language](api.md#an-assets-language)). When the configured engine isn't installed on a worker (or
doesn't import there), it uses the next one that is and says so in the job log; with none at all the step fails,
saying how to add one. A file that fails is marked and the batch carries on. For a
document or an image it draws the pages and reads their text instead: a PDF's own text, and OCR for scans and images
Expand Down Expand Up @@ -864,7 +865,8 @@ audio skip themselves for imported text. Steps:
- `convert`: makes new assets from one, a video's soundtrack as audio or a document's pages as images
([Converting assets](#converting-assets)).

Any step can carry a condition: `min_minutes`, `max_minutes`, `source` (audio or transcript), `languages`.
Any step can carry a condition: `min_minutes`, `max_minutes`, `source` (audio or transcript), `languages` (the
asset's [language](api.md#an-assets-language), ignoring case; `en` takes `en-GB` and `en-US` too).

Templates are versioned (publish, history, diff) and rendered in a sandboxed Jinja environment. It can't reach Python
internals, caps output size, and escapes HTML in reports. Templates see `recording`, `speakers`, `segments`,
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Transcription by a local engine (SenseVoice, Whisper), run in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| One asset's voice split off from the person it was matched to, with Undo, and a voice never matched or suggested again as someone it was said not to be | [test_speakers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_speakers.py) `test_detach_one_assets_voice`; [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_a_voice_never_goes_to_someone_it_was_said_not_to_be` |
| How sure transcription was of each word kept (Whisper, Deepgram, AssemblyAI, ElevenLabs), words from before still loading, the unsure ones marked in the transcript with how sure, a threshold in Settings (0 for none), and `unsure` words per line over MCP | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_providers_keep_how_sure_they_were_of_each_word`, `test_whisper_keeps_how_sure_it_was_of_each_word`; [test_split_merge.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_split_merge.py) `test_words_keep_how_sure_transcription_was`, `test_words_from_before_how_sure_was_kept_still_load`; [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_get_transcript_says_which_words_transcription_was_unsure_of`; [recording-split-words.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-split-words.test.ts) `keeps how sure transcription was of each word, and words from before it was kept`, `marks the words below the threshold, and none when it's off`; [recording-unsure-words.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-unsure-words.test.tsx) `are underlined with how sure, and a click still plays the line` |
| The language transcription hears kept on the asset, and a person sets or corrects it as a BCP 47 tag with a region (en-GB, pt-BR), checked and written the usual way, kept over the detected one when it is transcribed again; through the Details tab, the API and MCP | [test_language.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_language.py) `test_tags_are_written_the_usual_way`, `test_what_isnt_a_tag_is_refused`, `test_editors_set_and_correct_an_assets_language`, `test_transcribing_again_keeps_the_language_a_person_set`, `test_a_pipeline_condition_on_en_takes_its_regions`, `test_agents_set_it_over_mcp`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_whisper_keeps_the_language_it_heard`; [recording-language.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/recording-language.test.tsx) `sends a tidy tag, null to forget it, and nothing when it's the same`, `lets an editor correct it with Enter, and refuses what isn't a tag` |
| People recognised by voice; long recordings sent to a speech provider in pieces that fit its limits | [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_match_suggest_merge_undo_link`, `test_merging_keeps_who_they_are_not`, `test_upgrade_gives_pairs_lost_to_merges_to_who_they_were_merged_into`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_a_long_recording_goes_in_pieces_and_comes_back_whole`, `test_a_piece_too_large_to_send_is_halved` |
| An owner turns voice recognition off for a namespace: voices told apart in each recording only, its voiceprints deleted | [test_voice_mode.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_mode.py) `test_voices_can_be_told_apart_without_being_recognised` |
| Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` |
Expand Down
32 changes: 31 additions & 1 deletion fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -920,7 +920,10 @@ def list_speakers(ctx: Context, namespace: str | None, query: str | None, limit:
def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]:
rec = recording_routes.get_recording(recording_id, ctx.acl, ctx.db, ctx.cfg)
raw = (
ctx.db.one("SELECT recorded_at, duration_ms, source, media, language, pages, analyzed_at FROM $r", r=R("recording", recording_id))
ctx.db.one(
"SELECT recorded_at, duration_ms, source, media, language, language_set, pages, analyzed_at FROM $r",
r=R("recording", recording_id),
)
or {}
)
sections = ctx.db.rows("SELECT idx, t0, t1, title FROM section WHERE recording = $r ORDER BY idx", r=recording_id)
Expand All @@ -937,6 +940,7 @@ def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]:
"media": render.kind(raw),
"pages": raw.get("pages"),
"language": raw.get("language"),
"language_set": bool(raw.get("language_set")) or None, # a person set it; transcription leaves it
"status": rec.status,
"tags": rec.tags or None,
"your_role": rec.role,
Expand Down Expand Up @@ -2960,6 +2964,32 @@ def mark_nsfw(ctx: Context, recording_id: int, verdict: str) -> dict[str, Any]:
return {"recording_id": recording_id, "nsfw": got, "url": ctx.link(recording_id)}


# ---------- an asset's language ----------
@tool(
"set_language",
"Set an asset's language",
"Set or correct the language an asset is in, as a BCP 47 tag with a region if it matters (en-GB, en-US, pt-BR, "
"pt-PT); it is written the usual way (en-gb becomes en-GB) and kept over the language transcription detects. "
"Left out, it forgets it, so the next transcription says again. get_recording gives its language and whether a person "
"set it (language_set). Editors, with a token that has the write scope.",
Arg("recording_id", "integer", "the asset", required=True),
Arg("language", "string", "a BCP 47 tag such as en, en-GB or pt-BR; left out, it's forgotten", max_length=40),
writes=True,
)
def set_language(ctx: Context, recording_id: int, language: str | None) -> dict[str, Any]:
if not ctx.user.can_write:
raise ToolError("this token is read-only: ask for a token with the write scope to set languages")
ctx.acl.recording(recording_id, "edit")
try:
before, after = library.set_language(ctx.db, ctx.cfg, recording_id, language)
except ValueError as e:
raise ToolError(str(e)) from None
if before != after:
detail = {"from": before, "to": after, "via": "mcp"}
auth.audit(ctx.db, ctx.user.as_audit(), "recording.language", f"recording:{recording_id}", detail)
return {"recording_id": recording_id, "language": after, "language_set": after is not None, "url": ctx.link(recording_id)}


# ---------- enhancing low quality images ----------
@tool(
"enhance",
Expand Down
12 changes: 9 additions & 3 deletions fastapi_backend/app/api/v1/routes/recordings.py
Original file line number Diff line number Diff line change
Expand Up @@ -387,11 +387,17 @@ def _attached_to(acl: Access, db: DB, link: dict[str, Any] | None) -> dict[str,

@router.patch("/{rid}")
def update_recording(rid: int, body: RecordingUpdate, acl: Acl, user: Writer, db: Db, cfg: Cfg) -> Recording:
"""Rename an asset or replace its tags (editors). A renamed asset's report is rebuilt with the new title."""
"""Rename an asset, replace its tags or set its language (editors). A renamed asset's report is rebuilt with the
new title. The language is a BCP 47 tag, written the usual way (en-gb → en-GB), kept over the one transcription
detects; null forgets it. Audited as `recording.rename` and `recording.language`."""
rec = acl.recording(rid, "edit")
if body.title is None and body.tags is None:
raise HTTPException(400, "send a title or tags")
if body.title is None and body.tags is None and "language" not in body.model_fields_set:
raise HTTPException(400, "send a title, tags or a language")
with domain_errors():
if "language" in body.model_fields_set:
before, after = library.set_language(db, cfg, rid, body.language)
if before != after:
auth.audit(db, user.as_audit(), "recording.language", f"recording:{rid}", {"from": before, "to": after})
if body.tags is not None:
library.set_tags(db, rid, body.tags)
if body.title is not None:
Expand Down
Loading
Loading
0