8000 feat(speakers): split one asset's voice off the person it was matched to by adeelahmad · Pull Request #663 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
SPARQL, and see as it was at any version, where every source, sensor, person and agent is a node linked to what it
brought in. Every entity has its own page at a wiki address
(`/wiki/<namespace>/<Name>`), and entities, topics and people are edited and merged from their lists, the graph or an
asset, with Undo instead of a confirm. Names get their real types (people, places, organisations, events, works) by
asset, with Undo instead of a confirm (and a voice matched to the wrong person split off). Names get their real types (people, places, organisations, events, works) by
rules, without a model, and a decision model types the ones the rules can't. Find what is most like a passage, an
asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos
by what they show. Pages and shots are described in words by the model provider's vision model or, off by default, by
Expand Down
4 changes: 4 additions & 0 deletions changelog.d/detach-voice.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
- **Not them here.** When a voice in one asset was matched to the wrong person, Not them here on their card in the
asset's Speakers tab (or `POST /api/v1/speakers/{sid}/detach`) gives its lines there to a new person, works out the
first person's voice again without it, and never suggests the two as the same; Undo merges them back
([People and namespaces](docs/processing.md#people-and-namespaces)).
8 changes: 6 additions & 2 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1316,6 +1316,7 @@ GET /api/v1/speakers
GET /api/v1/speakers/{sid}/recordings
POST /api/v1/speakers/{sid}
POST /api/v1/speakers/{sid}/merge
POST /api/v1/speakers/{sid}/detach
POST /api/v1/merges/{mid}/undo
POST /api/v1/speakers/{sid}/link
POST /api/v1/speakers/{sid}/unlink
Expand All @@ -1328,8 +1329,11 @@ People (everyone recognised by voice or face) live under `/speakers`.
`recordings` and `segments` that moved, and `undone_by`/`undone_at`), its links to other namespaces, and `cross`:
voices in other shared namespaces you can read that are likely the same person (`score`, best first). `not-same
{with}` says two entries aren't the same person (one namespace or two): the suggestion goes and isn't made again.
`unlink {with}` removes a link (404 when there's none). Linking, unlinking, "not the same" and undoing a merge need
editor access to both people's namespaces and are audited (`speaker.link`, `speaker.unlink`, `speaker.not_same`,
`detach {recording}` says the voice heard as this person in one asset is someone else: its lines there go to a new
person (`speaker` in the answer), this person's voice is worked out again from the assets they're still in, and the two
are marked not the same; merging the new person back undoes it. It's a 400 when they aren't heard in that asset, or
are heard only there (rename them instead). `unlink {with}` removes a link (404 when there's none). Linking, unlinking, "not the same" and undoing a merge need
editor access to both people's namespaces and are audited (`speaker.link`, `speaker.unlink`, `speaker.not_same`, `speaker.detach`,
`speaker.merge.undo`).

## entities
Expand Down
4 changes: 3 additions & 1 deletion docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,9 @@ You can rename and merge people in the web app or the CLI (`lens speakers …`),
keep who made them and how many assets moved. People in different namespaces are never merged. You can link them
as the same person (and unlink them), and with `speakers.cross_namespace: suggest` the graph shows likely voice matches
as dashed edges and People lists them with how alike the voices are. Saying a suggested pair is "not the same", in one
namespace or across two, removes the suggestion for good.
namespace or across two, removes the suggestion for good. When a voice was matched to the wrong person in one asset,
**Not them here** on that person's card in the asset's Speakers tab gives its lines there to a new person and works out
the first person's voice again without it, with Undo; the two aren't suggested as the same again.

**Voice recognition off.** Voiceprints are biometric data. An owner can turn off People › Recognise voices across
recordings for their namespace (`PUT /api/v1/namespaces/<ns>/voices/mode {"mode": "separate"}`; `GET /speakers` says
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Transcription | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_an_openai_compatible_server_transcribes`, `test_deepgram_transcribes_with_speakers_and_sentiment` |
| Transcription that expects the names the namespace knows (custom vocabulary, named people, defined entities, names people fixed), off by default | [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_the_namespaces_own_names_go_to_the_engine` |
| Transcription by a local engine (SenseVoice, Whisper), run in CI | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| One asset's voice split off from the person it was matched to, with Undo | [test_speakers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_speakers.py) `test_detach_one_assets_voice` |
| People recognised by voice; long recordings sent to a speech provider in pieces that fit its limits | [test_voices.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_voices.py) `test_match_suggest_merge_undo_link`; [test_speech.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_speech.py) `test_a_long_recording_goes_in_pieces_and_comes_back_whole`, `test_a_piece_too_large_to_send_is_halved` |
| An owner turns voice recognition off for a namespace: voices told apart in each recording only, its voiceprints deleted | [test_voice_mode.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_mode.py) `test_voices_can_be_told_apart_without_being_recognised` |
| Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` |
Expand Down
19 changes: 19 additions & 0 deletions fastapi_backend/app/api/v1/routes/speakers.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
from app.domain.store import DB
from app.schemas.common import Ok
from app.schemas.speakers import (
SpeakerDetached,
SpeakerDetachRequest,
SpeakerDirectory,
SpeakerLinkRequest,
SpeakerMerged,
Expand Down Expand Up @@ -157,6 +159,23 @@ def merge_speaker(sid: int, body: SpeakerMergeRequest, request: Request, acl: Ac
return SpeakerMerged(merge_id=mid)


@router.post("/speakers/{sid}/detach")
def detach_speaker(sid: int, body: SpeakerDetachRequest, request: Request, acl: Acl, user: Writer, db: Db) -> SpeakerDetached:
"""The voice heard as this person in one asset is someone else: its lines there go to a new person, this person's
voice is worked out again without them, and the two are never suggested as the same. Undo by merging the new person
back (POST /speakers/{speaker}/merge with ``into`` this one). Audited as speaker.detach."""
acl.need(speaker_space(db, sid), "curate")
try:
new = spk.detach(db, sid, body.recording, by=user.email)
except KeyError:
raise HTTPException(404, "not found") from None
except ValueError as e:
raise HTTPException(400, str(e)) from None
auth.audit(db, user.as_audit(), "speaker.detach", f"speaker:{sid}", {"recording": body.recording, "to": new})
request.app.state.graph_cache.clear()
return SpeakerDetached(speaker=new)


@router.post("/merges/{mid}/undo")
def undo_speaker_merge(mid: int, request: Request, acl: Acl, user: Writer, db: Db) -> Ok:
m = db.one("SELECT space FROM $r", r=R("merge", mid))
Expand Down
52 changes: 52 additions & 0 deletions fastapi_backend/app/domain/speakers.py
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,58 @@ def undo(db, merge_id, by=None):
db.run(stmts, **params)


def _recentre(db, sid):
"""A person's voice again from the voices of the recordings they're still in (each weighted by how long they
talk there); kept as it was when none of them has one."""
rows = db.rows("SELECT recording, embedding FROM appearance WHERE speaker = $s AND embedding != NONE", s=sid)
if not rows:
return
secs = {
r["recording"]: r["ms"] / 1000
for r in db.rows("SELECT recording, math::sum(dur) AS ms FROM segment WHERE speaker = $s GROUP BY recording", s=sid)
}
w = [max(secs.get(r["recording"], 0.0), 1.0) for r in rows]
v = sum(np.asarray(r["embedding"], dtype=np.float64) * x for r, x in zip(rows, w))
db.q(
"UPDATE $r SET embedding = $e, n_obs = $n",
r=R("speaker", sid),
e=_vec(v / (np.linalg.norm(v) + 1e-9)),
n=float(sum(w)),
)


def detach(db, sid, rid, by=None):
"""The voice heard as `sid` in recording `rid` is someone else: their lines there go to a new person, `sid`'s
voice is worked out again without them, and the two are never suggested as the same. Returns the new person's id
(merge it back into `sid` to undo)."""
sp = db.one("SELECT space FROM $r", r=R("speaker", sid))
if not sp:
raise KeyError("speaker")
here = db.rows("SELECT record::id(id) AS id, embedding FROM appearance WHERE speaker = $s AND recording = $rid", s=sid, rid=rid)
segs = db.values("SELECT VALUE record::id(id) FROM segment WHERE speaker = $s AND recording = $rid", s=sid, rid=rid)
if not segs:
raise ValueError("they aren't heard in that asset")
if not db.values("SELECT VALUE id FROM segment WHERE speaker = $s AND recording != $rid LIMIT 1", s=sid, rid=rid):
raise ValueError("they're only heard in this asset: rename them instead")
emb = next((a["embedding"] for a in here if a.get("embedding") is not None), None)
new = new_speaker(db, sp["space"], emb=emb, weight=1.0 if emb is not None else 0.0)
db.run(
[
"UPDATE $segs SET speaker = $new",
"UPDATE $apps SET speaker = $new, method = 'detached', score = NONE",
"UPDATE mentions SET speaker = $new WHERE in IN $segs",
"UPDATE $rec SET embedded = NONE",
],
segs=[R("segment", i) for i in segs],
apps=[R("appearance", a["id"]) for a in here],
new=new,
rec=R("recording", rid),
)
_recentre(db, sid)
not_same(db, sid, new, by=by)
return new


def link(db, a, b):
ra, rb = db.one("SELECT space FROM $r", r=R("speaker", a)), db.one("SELECT space FROM $r", r=R("speaker", b))
if not ra or not rb:
Expand Down
8 changes: 8 additions & 0 deletions fastapi_backend/app/schemas/speakers.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,10 @@ class SpeakerMergeRequest(RequestModel):
into: int = Field(description="the person that remains")


class SpeakerDetachRequest(RequestModel):
recording: int = Field(description="the asset where the voice heard as this person is someone else")


class SpeakerLinkRequest(RequestModel):
# ``with`` is a Python keyword. (FastAPI re-wraps aliased body fields and pydantic warns about it; harmless.)
with_: int = Field(alias="with", description="a person in another namespace: the same person")
Expand All @@ -109,3 +113,7 @@ class SpeakerPair(RequestModel):

class SpeakerMerged(Ok):
merge_id: int


class SpeakerDetached(Ok):
speaker: int = Field(description="the new person heard in that asset (merge them back to undo)")
40 changes: 40 additions & 0 deletions fastapi_backend/tests/api/test_speakers.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,3 +110,43 @@ def test_not_the_same_unlink_and_merge_history(app, client, db, cfg, folder):
assert m["undone"] is True and m["undone_by"] == "ed@x.io" and m["undone_at"]
acts = set(db.values("SELECT VALUE action FROM audit_log"))
assert {"speaker.not_same", "speaker.link", "speaker.unlink", "speaker.merge", "speaker.merge.undo"} <= acts


def test_detach_one_assets_voice(client, db, cfg, folder):
seed(db, cfg, folder)
make_user(db, "ed@x.io", "editor password 1", roles={"pods": "editor"})
make_user(db, "vi@x.io", "viewer password 1", roles={"pods": "viewer"})
he, hv = login(client, "ed@x.io", "editor password 1"), login(client, "vi@x.io", "viewer password 1")
by = {}
for r in db.rows(
"SELECT speaker, recording FROM segment WHERE space = $s AND speaker > 0 GROUP BY speaker, recording", s=store.ns_id(db, "pods")
):
by.setdefault(r["speaker"], []).append(r["recording"])
sid, (keep, wrong, *_) = next((s, rs) for s, rs in by.items() if len(rs) >= 2)
rng = np.random.default_rng(5)
voice, other = (v / np.linalg.norm(v) for v in rng.normal(size=(2, 192)))
for rid, e in ((keep, voice), (wrong, other)):
db.q("DELETE appearance WHERE recording = $r AND speaker = $s", r=rid, s=sid)
db.q(
"CREATE appearance CONTENT $a",
a={"recording": rid, "speaker": sid, "local_label": "S0", "method": "voice", "embedding": list(e)},
)
db.q("UPDATE $r SET embedding = $e", r=store.R("speaker", sid), e=list((voice + other) / np.linalg.norm(voice + other)))

path = f"/api/v1/speakers/{sid}/detach"
assert client.post(path, json={"recording": wrong}, headers=hv).status_code == 403
r = client.post(path, json={"recording": wrong}, headers=he)
assert r.status_code == 200, r.text
new = r.json()["speaker"]
assert set(db.values("SELECT VALUE speaker FROM segment WHERE recording = $r AND speaker IN [$a, $b]", r=wrong, a=sid, b=new)) == {new}
assert db.values("SELECT VALUE speaker FROM appearance WHERE recording = $r AND local_label = 'S0'", r=wrong) == [new]
centre = np.asarray(db.one("SELECT embedding FROM $r", r=store.R("speaker", sid))["embedding"])
assert float(np.dot(centre, voice)) > 0.99 # their voice without the other one's
assert db.rows("SELECT a, b FROM not_same") == [{"a": min(sid, new), "b": max(sid, new)}] # never suggested again
assert client.post(path, json={"recording": wrong}, headers=he).status_code == 400 # not heard there any more
if len(by[sid]) == 2:
assert client.post(path, json={"recording": keep}, headers=he).status_code == 400 # only heard here: rename
assert client.post("/api/v1/speakers/99999/detach", json={"recording": wrong}, headers=he).status_code == 404

assert client.post(f"/api/v1/spe 10BC8 akers/{new}/merge", json={"into": sid}, headers=he).status_code == 200 # Undo
assert sid in db.values("SELECT VALUE speaker FROM segment WHERE recording = $r", r=wrong)
7 changes: 7 additions & 0 deletions nextjs-frontend/app/openapi-client/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1007,6 +1007,11 @@ export type {
Described,
Description,
Desired,
DetachSpeakerData,
DetachSpeakerError,
DetachSpeakerErrors,
DetachSpeakerResponse,
DetachSpeakerResponses,
DetectLlmData,
DetectLlmResponse,
DetectLlmResponses,
Expand Down Expand Up @@ -3742,6 +3747,8 @@ export type {
SparqlQuery,
Speaker,
SpeakerCandidate,
SpeakerDetached,
SpeakerDetachRequest,
SpeakerDirectory,
SpeakerFacet,
SpeakerLink,
Expand Down
23 changes: 23 additions & 0 deletions nextjs-frontend/app/openapi-client/sdk.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -470,6 +470,9 @@ import type {
DeleteWatchData,
DeleteWatchErrors,
DeleteWatchResponses,
DetachSpeakerData,
DetachSpeakerErrors,
DetachSpeakerResponses,
DetectLlmData,
DetectLlmResponses,
DiffTemplateVersionsData,
Expand Down Expand Up @@ -8021,6 +8024,26 @@ export class Speakers {
});
}

/**
* Detach Speaker
*
* The voice heard as this person in one asset is someone else: its lines there go to a new person, this person's
* voice is worked out again without them, and the two are never suggested as the same. Undo by merging the new person
* back (POST /speakers/{speaker}/merge with ``into`` this one). Audited as speaker.detach.
*/
public static detachSpeaker<ThrowOnError extends boolean = false>(
options: Options<DetachSpeakerData, ThrowOnError>,
): RequestResult<DetachSpeakerResponses, DetachSpeakerErrors, ThrowOnError> {
return (options.client ?? client).post<DetachSpeakerResponses, DetachSpeakerErrors, ThrowOnError>({
url: "/api/v1/speakers/{sid}/detach",
...options,
10BC8 headers: {
"Content-Type": "application/json",
...options.headers,
},
});
}

/**
* Undo Speaker Merge
*/
Expand Down
59 changes: 59 additions & 0 deletions nextjs-frontend/app/openapi-client/types.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25996,6 +25996,35 @@ export type SpeakerCandidate = {
[key: string]: unknown;
};

/**
* SpeakerDetachRequest
*/
export type SpeakerDetachRequest = {
/**
* Recording
*
* the asset where the voice heard as this person is someone else
*/
recording: number;
};

/**
* SpeakerDetached
*/
export type SpeakerDetached = {
/**
* Ok
*/
ok?: boolean;
/**
* Speaker
*
* the new person heard in that asset (merge them back to undo)
*/
speaker: number;
[key: string]: unknown;
};

/**
* SpeakerDirectory
*/
Expand Down Expand Up @@ -40703,6 +40732,36 @@ export type MergeSpeakerResponses = {

export type MergeSpeakerResponse = MergeSpeakerResponses[keyof MergeSpeakerResponses];

export type DetachSpeakerData = {
body: SpeakerDetachRequest;
path: {
/**
* Sid
*/
sid: number;
};
query?: never;
url: "/api/v1/speakers/{sid}/detach";
};

export type DetachSpeakerErrors = {
/**
* Validation Error
*/
422: HttpValidationError;
};

export type DetachSpeakerError = DetachSpeakerErrors[keyof DetachSpeakerErrors];

export type DetachSpeakerResponses = {
/**
* Successful Response
*/
200: SpeakerDetached;
};

export type DetachSpeakerResponse = DetachSpeakerResponses[keyof DetachSpeakerResponses];

export type UndoSpeakerMergeData = {
body?: never;
path: {
Expand Down
25 changes: 25 additions & 0 deletions nextjs-frontend/components/recording/hooks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -574,6 +574,30 @@ export function useRecordingActions(id: number) {
},
onError: fail("Couldn't merge the people"),
});
// "Not them here": this asset's voice heard as them goes to a new person; Undo merges it back.
const detachSpeaker = useMutation({
mutationFn: (v: { sid: number; name: string }) =>
data(Speakers.detachSpeaker({ client, path: { sid: v.sid }, body: { recording: id } })),
onSuccess: (r, v) => {
const done = () => {
void refresh();
void qc.invalidateQueries({ queryKey: ["speakers"] });
};
done();
toast({
title: `Not ${v.name} here: a new person now`,
tone: "green",
action: {
label: "Undo",
onClick: () =>
void data(Speakers.mergeSpeaker({ client, path: { sid: r.speaker }, body: { into: v.sid } }))
.then(done)
.catch(fail("Couldn't undo it")),
},
});
},
onError: fail("Couldn't separate the voice"),
});
return {
reprocess,
writeSpec,
Expand All @@ -584,6 +608,7 @@ export function useRecordingActions(id: number) {
rename,
renameSpeaker,
mergeSpeaker,
detachSpeaker,
refresh,
};
}
Expand Down
Loading
Loading
0