8000 feat(speakers): let an owner turn voice recognition off for a namespace by adeelahmad · Pull Request #644 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
([Web clipper](docs/web-clipper.md), [setting it up](docs/web-clipper.md#setting-it-up),
[the assistant in your browser](docs/browser-assistant.md); [checks](docs/requirements.md#clip-from-the-browser))
- **Understand it.** Transcription, which can expect the names your namespace knows (off by default), people
recognised by voice (long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
recognised by voice (an owner can keep voices to each recording instead; long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
server (off by default, per namespace), low quality images enhanced before they're read (off by default), entities
and topics in a knowledge graph you can explore, query with Cypher or
SPARQL, and see as it was at any version, where every source, sensor, person and agent is a node linked to what it
Expand Down
3 changes: 3 additions & 0 deletions changelog.d/voices-off.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
- **Voice recognition off, per namespace.** An owner can turn off People › Recognise voices across recordings: voices
are still told apart in each recording, but no voiceprint is kept and the namespace's voiceprints are deleted
([People and namespaces](docs/processing.md#people-and-namespaces)).
7 changes: 7 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,13 @@ as the same person (and unlink them), and with `speakers.cross_namespace: sugges
as dashed edges and People lists them with how alike the voices are. Saying a suggested pair is "not the same", in one
namespace or across two, removes the suggestion for good.

**Voice recognition off.** Voiceprints are biometric data. An owner can turn off People › Recognise voices across
recordings for their namespace (`PUT /api/v1/namespaces/<ns>/voices/mode {"mode": "separate"}`; `GET /speakers` says
which is on as `voices`). Voices are then still told apart inside each recording, but no voiceprint is kept and nobody
is matched across recordings or namespaces. Turning it off deletes the namespace's voiceprints (each person's, each
appearance's and those kept for undoing a merge) and the matches waiting for review, and is audited as `voices.mode`;
names, transcripts and who spoke when stay. Turning it back on recognises people in recordings diarized after that.

## Knowledge graph

Nodes are people and named things. Edges are:
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Transcription that expects the names the namespace knows (custom vocabulary, named people, defined entities, names people fixed), off by default | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_the_namespaces_own_names_go_to_the_engine` |
| Transcription by a local engine (SenseVoice, Whisper), run in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| People recognised by voice; long recordings sent to a speech provider in pieces that fit its limits | [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_match_suggest_merge_undo_link`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_a_long_recording_goes_in_pieces_and_comes_back_whole`, `test_a_piece_too_large_to_send_is_halved` |
| An owner turns voice recognition off for a namespace: voices told apart in each recording only, its voiceprints deleted | [test_voice_mode.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_mode.py) `test_voices_can_be_told_apart_without_being_recognised` |
| Text on screen and in sca 8000 ns | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` |
| Faces, off by default | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_people_on_screen`, `test_shots_text_search_chat_and_iiif`; [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_video.py) `test_face_modes` |
| NSFW pictures flagged by a model on your own server, off by default, per namespace | [test_nsfw.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_nsfw.py) `test_off_by_default_nothing_is_fetched_or_classified`, `test_a_namespace_classifies_flags_blurs_and_hides`; [nsfw.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/nsfw.test.tsx) |
Expand Down
19 changes: 18 additions & 1 deletion fastapi_backend/app/api/v1/routes/speakers.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
SpeakerPair,
SpeakerRecording,
SpeakerRename,
VoiceMode,
)

router = APIRouter(tags=["speakers"])
Expand Down Expand Up @@ -98,10 +99,26 @@ def list_speakers(ns: str, acl: Acl, user: CurrentUser, db: Db, cfg: Cfg) -> Spe
if nid in (x["asp"], x["bsp"])
]
return SpeakerDirectory.model_validate(
{"speakers": spk.list_speakers(db, nid), "merges": merges, "links": links, "cross": _cross(db, cfg, nid, set(acl.roles))}
{
"speakers": spk.list_speakers(db, nid),
"merges": merges,
"links": links,
"cross": _cross(db, cfg, nid, set(acl.roles)),
"voices": spk.voice_mode(db, nid),
}
)


@router.put("/namespaces/{name}/voices/mode")
def set_namespace_voice_mode(name: str, body: VoiceMode, user: Writer, acl: Acl, db: Db) -> Ok:
"""Recognise voices across the namespace's recordings (the default), or only tell them apart in each one
(separate), which deletes its voiceprints and the matches waiting for review. Owners only."""
nid = acl.namespace(name, "manage")
spk.set_voice_mode(db, nid, body.mode, user.email)
auth.audit(db, user.as_audit(), "voices.mode", name, {"mode": body.mode})
return Ok()


@router.get("/speakers/{sid}/recordings")
def list_speaker_recordings(sid: int, acl: Acl, user: CurrentUser, db: Db) -> list[SpeakerRecording]:
acl.need(speaker_space(db, sid))
Expand Down
34 changes: 34 additions & 0 deletions fastapi_backend/app/domain/speakers.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
Speaker ids belong to one namespace and are never merged across namespaces. Across
namespaces the archive only records links: ones you declare, and (optionally) voice
matches it suggests as graph edges.

A namespace's own 8BD0 er can turn voice recognition off (`voice_mode` separate): voices are still told apart inside each
recording, but no voiceprint is kept and nobody is matched across recordings. Turning it off deletes the namespace's
voiceprints and the matches waiting for review.
"""

from __future__ import annotations
Expand Down Expand Up @@ -184,6 +188,33 @@ def voice_print(signal, segs, embed, seconds=60):

# ---------- the namespace registry ----------
R = store.R
VOICE_MODES = ("recognize", "separate")


def voice_mode(db, nid):
"""recognize (the default: voiceprints kept, people matched across recordings) or separate (told apart in each
recording only)."""
return (db.one("SELECT voices_mode FROM $s", s=R("space", int(nid))) or {}).get("voices_mode") or "recognize"


def set_voice_mode(db, nid, mode, user=None):
"""Recognise voices across recordings, or only tell them apart in each one. Separate deletes the namespace's
voiceprints (people's, each appearance's and those kept for undoing a merge) and its suggested matches: there is
nothing left to match with. Recognising again starts from the recordings diarized after it."""
if mode not in VOICE_MODES:
raise ValueError(f"voice mode is one of {', '.join(VOICE_MODES)}")
db.q("UPDATE $s MERGE $p", s=R("space", int(nid)), p={"voices_mode": mode, "voices_set_by": user, "voices_set_at": store.now()})
if mode == "separate":
db.run(
[
"UPDATE speaker SET embedding = NONE, n_obs = 0 WHERE space = $s",
"UPDATE appearance SET embedding = NONE WHERE space = $s",
"DELETE suggestion WHERE space = $s",
# the voiceprints kept for undoing a merge too
"UPDATE merge SET snapshot.speaker.embedding = NONE, snapshot.dst.embedding = NONE WHERE space = $s",
],
s=int(nid),
)


def _vec(v):
Expand Down Expand Up @@ -316,6 +347,9 @@ def diarize_one(db, cfg, rid, log=print):
vp = voice_print(sig, g, emb, cfg["speakers"]["sample_seconds"])
if vp is not None:
prints[l] = vp
keep = voice_mode(db, r["space"]) == "recognize"
if not keep: # told apart in this recording only: no voiceprint kept, nobody matched
prints = {}
ids = match(db, cfg, r["space"], prints) if prints else {}
for l in groups:
if l not in ids:
Expand Down
13 changes: 12 additions & 1 deletion fastapi_backend/app/schemas/speakers.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from __future__ import annotations

from typing import Any
from typing import Any, Literal

from pydantic import Field

Expand Down Expand Up @@ -69,6 +69,17 @@ class SpeakerDirectory(ResponseModel):
description="likely the same person in another shared namespace you can read (speakers.cross_namespace: suggest), "
"best first; not linked, and not marked as different",
)
voices: Literal["recognize", "separate"] = Field(
"recognize",
description="recognize: voiceprints kept and people matched across recordings; separate: voices told apart in "
"each recording only (PUT /namespaces/{name}/voices/mode)",
)


class VoiceMode(RequestModel):
mode: Literal["recognize", "separate"] = Field(
description="recognize, or separate: voices told apart in each recording only, the namespace's voiceprints deleted"
)


class SpeakerRecording(ResponseModel):
Expand Down
66 changes: 66 additions & 0 deletions fastapi_backend/tests/api/test_voice_mode.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
"""Voice recognition per namespace (speakers.voice_mode): on by default; off, voices are told apart in each recording
only, and the namespace's voiceprints are deleted."""

from __future__ import annotations

import numpy as np

from app.domain import ingest, speakers, store
from tests.helpers import login, make_user, unit

R = store.R


def _recording(db, nid, n):
rid = db.next_id("recording")
db.q(
"CREATE $r CONTENT $d", r=R("recording", rid), d={"space": nid, "fingerprint": f"fp{n}", "status": "transcribed", "source": "audio"}
)
segs = [{"t0": 0, "t1": 1000, "text": "hello"}, {"t0": 1000, "t1": 2000, "text": "hi"}]
db.q("INSERT INTO segment $s", s=ingest.segment_rows(rid, nid, segs))
for i, who in enumerate(("A", "B")):
db.q("UPDATE segment SET provider_speaker = $p WHERE recording = $r AND idx = $i", p=who, r=rid, i=i)
return rid


def test_voices_can_be_told_apart_without_being_recognised(client, db, cfg, monkeypatch):
rng = np.random.default_rng(5)
prints = {"A": unit(rng.normal(size=192)), "B": unit(rng.normal(size=192))}
monkeypatch.setattr(ingest, "audio_path", lambda *a, **k: "unused.wav")
monkeypatch.setattr(ing 10BC8 est, "decode", lambda *a, **k: np.zeros(16000, dtype=np.float32))
monkeypatch.setattr(speakers, "get_embedder", lambda cfg, log=None: object())
monkeypatch.setattr(speakers, "voice_print", lambda sig, g, emb, s: (prints[g[0]["provider_speaker"]], 30.0))
nid = store.ns_id(db, "pods")

def who(rid):
return sorted(db.values("SELECT VALUE speaker FROM appearance WHERE recording = $r", r=rid))

first, second = _recording(db, nid, 1), _recording(db, nid, 2)
speakers.diarize_one(db, cfg, first, log=lambda *a: None)
speakers.diarize_one(db, cfg, second, log=lambda *a: None)
assert who(first) == who(second) # recognised: the same two people
assert db.values("SELECT VALUE embedding FROM speaker WHERE space = $s AND embedding != NONE", s=nid)

make_user(db, "own@x.io", "owner password 1", roles={"pods": "owner"})
make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"})
ho, he = login(client, "own@x.io", "owner password 1"), login(client, "ed@x.io", "editor password 1")
assert client.get("/api/v1/speakers", params={"ns": "pods"}, headers=ho).json()["voices"] == "recognize"
url = "/api/v1/namespaces/pods/voices/mode"
assert client.put(url, headers=he, json={"mode": "separate"}).status_code == 403 # owners only
assert client.put(url, headers=ho, json={"mode": "never"}).status_code == 422
assert client.put(url, headers=ho, json={"mode": "separate"}).status_code == 200
assert client.get("/api/v1/speakers", params={"ns": "pods"}, headers=ho).json()["voices"] == "separate"
assert db.values("SELECT VALUE embedding FROM speaker WHERE space = $s AND embedding != NONE", s=nid) == []
assert db.values("SELECT VALUE embedding FROM appearance WHERE space = $s AND embedding != NONE", s=nid) == []
assert db.one("SELECT detail FROM audit_log WHERE action = 'voices.mode'")["detail"] == {"mode": "separate"}

third = _recording(db, nid, 3)
speakers.diarize_one(db, cfg, third, log=lambda *a: None)
assert len(who(third)) == 2 and not set(who(third)) & set(who(first)) # told apart, matched with nobody
assert db.values("SELECT VALUE embedding FROM speaker WHERE space = $s AND embedding != NONE", s=nid) == []

assert client.put(url, headers=ho, json={"mode": "recognize"}).status_code == 200
fourth, fifth = _recording(db, nid, 4), _recording(db, nid, 5)
speakers.diarize_one(db, cfg, fourth, log=lambda *a: None)
speakers.diarize_one(db, cfg, fifth, log=lambda *a: None)
assert who(fourth) == who(fifth) # recognising again, from then on
6 changes: 6 additions & 0 deletions nextjs-frontend/app/openapi-client/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3569,6 +3569,11 @@ export type {
SetNamespaceFacesModeErrors,
SetNamespaceFacesModeResponse,
SetNamespaceFacesModeResponses,
SetNamespaceVoiceModeData,
SetNamespaceVoiceModeError,
SetNamespaceVoiceModeErrors,
SetNamespaceVoiceModeResponse,
SetNamespaceVoiceModeResponses,
SetPolicyData,
SetPolicyError,
SetPolicyErrors,
Expand Down Expand Up @@ -4339,6 +4344,7 @@ export type {
VoiceInfoResponse,
VoiceInfoResponses,
VoiceList,
VoiceMode,
VoiceOption,
Watch,
WatchCreate,
Expand Down
22 changes: 22 additions & 0 deletions nextjs-frontend/app/openapi-client/sdk.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1717,6 +1717,9 @@ import type {
SetNamespaceFacesModeData,
SetNamespaceFacesModeErrors,
SetNamespaceFacesModeResponses,
SetNamespaceVoiceModeData,
SetNamespaceVoiceModeErrors,
SetNamespaceVoiceModeResponses,
SetPolicyData,
SetPolicyErrors,
SetPolicyResponses,
Expand Down Expand Up @@ -7852,6 +7855,25 @@ export class Speakers {
});
}

/**
* Set Namespace Voice Mode
*
* Recognise voices across the namespace's recordings (the default), or only tell them apart in each one
* (separate), which deletes its voiceprints and the matches waiting for review. Owners only.
*/
public static setNamespaceVoiceMode<ThrowOnError extends boolean = false>(
options: Options<SetNamespaceVoiceModeData, ThrowOnError>,
): RequestResult<SetNamespaceVoiceModeResponses, SetNamespaceVoiceModeErrors, ThrowOnError> {
return (options.client ?? client).put<SetNamespaceVoiceModeResponses, SetNamespaceVoiceModeErrors, ThrowOnError>({
url: "/api/v1/namespaces/{name}/voices/mode",
...options,
headers: {
"Content-Type": "application/json",
...options.headers,
},
});
}

/**
* List Speaker Recordings
*/
Expand Down
48 changes: 48 additions & 0 deletions nextjs-frontend/app/openapi-client/types.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25754,6 +25754,12 @@ export type SpeakerDirectory = {
* likely the same person in another shared namespace you can read (speakers.cross_namespace: suggest), best first; not linked, and not marked as different
*/
cross?: Array<SpeakerCandidate>;
/**
* Voices
*
* recognize: voiceprints kept and people matched across recordings; separate: voices told apart in each recording only (PUT /namespaces/{name}/voices/mode)
*/
voices?: "recognize" | "separate";
[key: string]: unknown;
};

Expand Down Expand Up @@ -29008,6 +29014,18 @@ export type VoiceList = {
[key: string]: unknown;
};

/**
* VoiceMode
*/
export type VoiceMode = {
/**
* Mode
*
* recognize, or separate: voices told apart in each recording only, the namespace's voiceprints deleted
*/
mode: "recognize" | "separate";
};

/**
* VoiceOption
*/
Expand Down Expand Up @@ -40070,6 +40088,36 @@ export type ListSpeakersResponses = {

export type ListSpeakersResponse = ListSpeakersResponses[keyof ListSpeakersResponses];

export type SetNamespaceVoiceModeData = {
body: VoiceMode;
path: {
/**
* Name
*/
name: string;
};
query?: never;
url: "/api/v1/namespaces/{name}/voices/mode";
};

export type SetNamespaceVoiceModeErrors = {
/**
* Validation Error
*/
422: HttpValidationError;
};

export type SetNamespaceVoiceModeError = SetNamespaceVoiceModeErrors[keyof SetNamespaceVoiceModeErrors];

export type SetNamespaceVoiceModeResponses = {
/**
* Successful Response
*/
200: Ok;
};

export type SetNamespaceVoiceModeResponse = SetNamespaceVoiceModeResponses[keyof SetNamespaceVoiceModeResponses];

export type ListSpeakerRecordingsData = {
body?: never;
path: {
Expand Down
4 changes: 3 additions & 1 deletion nextjs-frontend/components/speakers/speakers-page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { useNamespaceSpeakers } from "@/components/speakers/parts";
import { lastHeard, reviewPairs } from "@/components/speakers/derive";
import { Registry } from "@/components/speakers/registry";
import { ReviewQueue } from "@/components/speakers/review";
import { VoiceMode } from "@/components/speakers/voice-mode";
import { Banner } from "@/components/ui/banner";
import { Button } from "@/components/ui/button";
import { Select } from "@/components/ui/field";
Expand All @@ -26,7 +27,7 @@ export function SpeakersPage() {
const params = useSearchParams();
const router = useRouter();
const pathname = usePathname();
const { namespaces, namespace: topNs } = useArchive();
const { namespaces, namespace: topNs, can } = useArchive();
const ns =
params.get("ns") && namespaces.some((n) => n.name === params.get("ns"))
? params.get("ns")!
Expand Down Expand Up @@ -141,6 +142,7 @@ export function SpeakersPage() {
Voiceprints are built from clean audio (Settings → Voice IDs). People without one are still matched, less
reliably.
</p>
<VoiceMode ns={ns} mode={dir.data.voices ?? "recognize"} canManage={can("owner", ns)} />
</div>
)}
{dir.data && ns && tab === "review" && <ReviewQueue pairs={pairs} ns={ns} />}
Expand Down
Loading
Loading
0