8000 feat(faces): name a namespace's unnamed faces in a row by adeelahmad · Pull Request #694 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
([Web clipper](docs/web-clipper.md), [setting it up](docs/web-clipper.md#setting-it-up),
[the assistant in your browser](docs/browser-assistant.md); [checks](docs/requirements.md#clip-from-the-browser))
- **Understand it.** Transcription, which can expect the names your namespace knows (off by default), people
recognised by voice (an owner can keep voices to each recording instead; long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
recognised by voice (an owner can keep voices to each recording instead; long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default; the ones nobody has named yet are named in a row, the most seen first, with Undo), NSFW pictures flagged by a model on your own
server (off by default, per namespace), low quality images enhanced before they're read (off by default), entities
and topics in a knowledge graph you can explore, query with Cypher or
SPARQL, and see as it was at any version, where every source, sensor, person and agent is a node linked to what it
Expand All @@ -192,7 +192,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
itself shows (its captions menu, full screen, picture in picture), in the workspace and on public pages.
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
[topics](docs/topics.md), [video](docs/video.md), [naming faces](docs/video.md#naming-faces), [NSFW classification](docs/nsfw.md),
[enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings),
[cleaning up audio](docs/processing.md#cleaning-up-audio),
[your own names](docs/processing.md#your-own-names-in-transcription),
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/unnamed-faces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
- **Name the unnamed faces in a row.** Where a namespace recognises faces, Settings → People has an Unnamed faces tab
(and a video's People tab links to it): the faces nobody has named yet, the largest group first, each with a few
crops and the assets it's in. The first name field has focus, filled with the name Lens offers when it has one (the
voice the face is linked to, or a named face suggested for it); Enter names it and moves on to the next, with Undo.
A name another face already has joins that person, the same merge as "Same person". Agents do it with
`GET /api/v1/namespaces/<ns>/faces/unnamed` and `POST /api/v1/faces/<id>/name`, or the MCP tools
`list_unnamed_faces` and `name_face` ([Naming faces](docs/video.md#naming-faces)).
10 changes: 10 additions & 0 deletions docs/api.md
8000
Original file line number Diff line number Diff line change
Expand Up @@ -1550,10 +1550,12 @@ GET /api/v1/resources/{rid}/frames/{name}
PATCH /api/v1/resources/{rid}/ocr/{span}
DELETE /api/v1/resources/{rid}/faces/{track}
GET /api/v1/namespaces/{name}/faces
GET /api/v1/namespaces/{name}/faces/unnamed
DELETE /api/v1/namespaces/{name}/faces
PUT /api/v1/namespaces/{name}/faces/mode
POST /api/v1/faces/merges/{mid}/undo
POST /api/v1/faces/{fid}
POST /api/v1/faces/{fid}/name
DELETE /api/v1/faces/{fid}
POST /api/v1/faces/{fid}/merge
POST /api/v1/faces/{fid}/speaker
Expand All @@ -1567,6 +1569,14 @@ say them for a resource. `GET /resources/{rid}/frames/{name}` (frames, keyframes
image's pages and thumbnails) serves a request without a role in the namespace the faces found on it pixelated, where
the namespace says so; a role, or a link the API signed for a member (`full=1`), gets the picture as it is.

`GET /namespaces/{name}/faces/unnamed?limit=` lists the faces nobody has named yet (recognize mode; none otherwise),
the largest group first: in the most assets, then the longest on screen (a page counts five seconds). Each has a few
signed crops (`thumbnails`, from different assets first), the assets it's in (`recordings`, where it's seen longest
first), how often it's seen, and `guess`, a name to offer: the voice it's linked to, else a named face or voice
suggested for it. `POST /faces/{fid}/name {name}` (editors) names one: a name another face of the namespace has (any
case) joins it to that face, a merge whose id comes back as `merge` to undo with `POST /faces/merges/{mid}/undo`; any
other name becomes its own, undone with `POST /faces/{fid} {"name": ""}` ([Naming faces](video.md#naming-faces)).

### Objects

The `objects` step finds the people, vehicles, animals and everyday things (the 80 kinds of the COCO dataset) on a
Expand Down
2 changes: 2 additions & 0 deletions docs/mcp.md
8000
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `budgets` | where spending stands: each namespace you can read this period against its budget, and for admins all of Lens and every other budget, each ok, near or over ([Asking the assistant or an agent](budgets.md#asking-the-assistant-or-an-agent)) |
| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) |
| `list_speakers` | the people of your namespaces, the ones with the most talk time first |
| `list_unnamed_faces` | the faces nobody has named yet in your namespaces that recognise faces, the largest group first: a name to offer, how often each is seen and the assets it's in, with links ([Naming faces](video.md#naming-faces)) |
| `name_face` | say who a face is: a name another face of its namespace has joins that person, any other becomes its own. Needs a write-scope token and editor access |
| `get_recording` | one asset: when, how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW |
| `get_transcript` | an asset's lines in order (who, time, text, link, and the words transcription wasn't sure of as `unsure`), a page at a time, from a line or a time window |
| `fetch` | an asset's whole text at once, as `[m:ss] Speaker: …` lines |
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ request, to another check that proves the same thing; loosening a requirement or
| An owner turns voice recognition off for a namespace: voices told apart in each recording only, its voiceprints deleted | [test_voice_mode.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_voice_mode.py) `test_voices_can_be_told_apart_without_being_recognised` |
| Text on screen and in scans | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_shots_text_search_chat_and_iiif`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_scans_and_images_are_read_by_ocr`; [test_ocr.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_ocr.py) `test_a_video_read_by_doctr_and_without_it` |
| Faces, off by default | [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_video.py) `test_people_on_screen`, `test_shots_text_search_chat_and_iiif`; [test_video.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_video.py) `test_face_modes` |
| The faces nobody has named yet, the largest group first with crops, the assets they're in and a name to offer, named one after another (Enter for the next), a name someone has joining that person, with Undo; through the API, MCP and the People page | [test_unnamed_faces.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_unnamed_faces.py) `test_unnamed_faces_largest_first_named_in_a_row_with_undo`, `test_only_where_faces_are_recognised`, `test_agents_list_and_name_faces`; jest [name-faces.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/name-faces.test.tsx) `focuses the largest group first; Enter names it and moves 8BD0 on to the next, with Undo`, `joins a face to the person who has the name, and Undo takes the merge back` |
| NSFW pictures flagged by a model on your own server, off by default, per namespace | [test_nsfw.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_nsfw.py) `test_off_by_default_nothing_is_fetched_or_classified`, `test_a_namespace_classifies_flags_blurs_and_hides`; [nsfw.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/nsfw.test.tsx) |
| Low quality images enhanced before they're read, off by default | [test_enhance.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_enhance.py) `test_off_by_default_the_original_is_read`, `test_a_low_quality_image_is_read_from_its_enhanced_copy`, `test_a_good_image_is_left_as_it_is`, `test_agents_and_automations_enhance_too`, `test_the_copy_is_encrypted_where_the_files_are`; [enhance.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/enhance.test.tsx) |
| Entities and topics in a knowledge graph to explore and query with Cypher or SPARQL | [test_graph.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_graph.py) `test_query`; [test_rdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rdf.py) `test_sparql`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `graph-find` |
Expand Down
20 changes: 19 additions & 1 deletion docs/video.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ soundtrack, and three more steps look at the picture. Each step skips itself for
- Where a namespace allows it, faces are grouped per recording and matched against that namespace's people, the way
voices are: auto-match, review, or new.
- When a face is on screen while one speaker talks, the app suggests they're the same person.
- Faces can be renamed, merged (with undo), linked to a person in People, marked "not a face" or deleted.
- Faces can be renamed, merged (with undo), linked to a person in People, marked "not a face" or deleted, and the
ones nobody has named yet are [named in a row](#naming-faces).
- Engines: OpenCV's YuNet + SFace (`pip install -e ".[faces]"`, then point `video.yunet_model` and
`video.sface_model` at the ONNX files from the OpenCV Zoo), InsightFace, whose pretrained models are licensed for
non-commercial research only, or `anytopdf`: [anytopdf](configuration.md#anytopdf)'s plugins find faces with
Expand Down Expand Up @@ -66,6 +67,23 @@ recording's media, even when its transcript came with it:

Recordings transcribed before the setting was on get their copy when their transcribe step runs again.

## Naming faces

In recognize mode, the faces nobody has named yet wait in **Settings → People → Unnamed faces** (the tab shows only
where faces are recognised; a video's People tab links to it while any are unnamed). The largest group comes first:
the face in the most assets, then the longest on screen. Each shows a few crops, how often it's seen and the assets
it's in.

- The first face's name field has focus, filled with the name Lens offers when it has one (the voice the face is
linked to, or a named face or voice suggested for it) and selected, so typing replaces it.
- Enter names it and moves on to the next. The notice that follows has Undo.
- A name another face of the namespace already has (any case) joins that person: it's the same merge as "Same
person", and Undo takes it back. Any other name becomes the face's own, everywhere in the namespace.
- Naming needs the curate permission (editors); members who can't see the reason on the button.

Agents do the same with the MCP tools `list_unnamed_faces` and `name_face`, or `GET
/api/v1/namespaces/<ns>/faces/unnamed` and `POST /api/v1/faces/<id>/name` ([API](api.md#video)).

## The video recording page

Video recordings open in their own layout.
Expand Down
54 changes: 54 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -909,6 +909,60 @@ def list_speakers(ctx: Context, namespace: str | None, query: str | None, limit:
return {"total": len(rows), "speakers": rows[:limit]}


@tool(
"list_unnamed_faces",
"List unnamed faces",
"The faces nobody has named yet in the namespaces you have a role in that recognise faces, the largest group "
"first: each face's id, what it's called until named, a name to offer (`guess`: the voice it's linked to or one "
"suggested), how often it's seen and the assets it's in, with links. Name them with name_face.",
Arg("namespace", "string", "only this namespace"),
Arg("limit", "integer", "how many", default=50, minimum=1, maximum=500),
)
def list_unnamed_faces(ctx: Context, namespace: str | None, limit: int) -> dict[str, Any]:
from app.domain import faces

spaces = [ctx.acl.namespace(namespace)] if namespace else ctx.acl.spaces()
names = store.space_names(ctx.db)
rows = [
store.clean(
{
"face_id": g["id"],
"label": g["label"],
"guess": g["guess"],
"namespace": names.get(sid),
"appearances": g["appearances"],
"screen_minutes": round(g["screen_ms"] / 60000, 1) if g["screen_ms"] else None,
"pages": g["pages"] or None,
"assets": [{"recording_id": r["id"], "title": r["title"], "url": ctx.link(r["id"])} for r in g["recordings"][:10]],
"more_assets": max(0, len(g["recordings"]) - 10) or None,
}
)
for sid in spaces
if faces.mode(ctx.db, sid) == "recognize"
for g in faces.unnamed(ctx.db, sid, thumbs=0)
]
return {"total": len(rows), "faces": rows[:limit]}


@tool(
"name_face",
"Name a face",
"Say who a face is (ids from list_unnamed_faces): a name another face of its namespace has joins it to that "
"person (a merge, which the web app can undo), any other name becomes its own. Needs a token with the write scope "
"and editor access.",
Arg("face_id", "integer", "the face", required=True),
Arg("name", "string", "who it is", required=True, max_length=80),
writes=True,
)
def name_face(ctx: Context, face_id: int, name: str) -> dict[str, Any]:
from app.api.v1.routes import video as video_routes
from app.schemas.video import FaceName

_writer(ctx, "name faces")
out = video_routes.name_face(face_id, FaceName(name=name), ctx.user, ctx.acl, ctx.db)
return store.clean({"face_id": face_id, "name": out.name, "joined_face_id": out.into, "merge_id": out.merge})


# ---------- reading ----------
@tool(
"get_recording",
Expand Down
34 changes: 33 additions & 1 deletion fastapi_backend/app/api/v1/routes/video.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
import re
from typing import Any

from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import FileResponse, Response

from app.api.deps import Access, Acl, Cfg, CurrentUser, Db, Writer, domain_errors
Expand All @@ -27,12 +27,15 @@
FaceDismiss,
FaceMerge,
FaceMerged,
FaceName,
FaceNamed,
FaceRename,
FacesMode,
FacesModeSet,
FaceSpeaker,
NamespaceFaces,
OcrFix,
UnnamedFaces,
)

router = APIRouter(tags=["video"])
Expand Down Expand Up @@ -190,6 +193,21 @@ def get_namespace_faces(name: str, user: CurrentUser, acl: Acl, db: Db) -> Names
return NamespaceFaces.model_validate(sign_urls(out, full=True))


@router.get("/namespaces/{name}/faces/unnamed")
def get_unnamed_faces(name: str, user: CurrentUser, acl: Acl, db: Db, limit: int = Query(100, ge=1, le=500)) -> UnnamedFaces:
"""The faces nobody has named yet, grouped across the namespace (recognize mode), the largest group first: each
with a few signed crops, the assets it's in and a name to offer. Name them with POST /faces/{fid}/name."""
sid = acl.namespace(name)
mode = facemod.mode(db, sid)
groups = facemod.unnamed(db, sid) if mode == "recognize" else []
for g in groups:
g["thumbnails"] = [
{"recording": t["recording"], "cover_url": f"{API}/recordings/{t['recording']}/frames/{t['file']}"} for t in g["thumbnails"]
]
out = {"mode": mode, "total": len(groups), "groups": groups[:limit]}
return UnnamedFaces.model_validate(sign_urls(out, full=True))


@router.put("/namespaces/{name}/faces/mode")
def set_namespace_faces_mode(name: str, body: FacesMode, user: Writer, acl: Acl, db: Db, cfg: Cfg) -> FacesModeSet:
"""off, detect or recognize (which needs a purpose), and whether faces are pixelated for visitors. Owners only.
Expand Down Expand Up @@ -242,6 +260,20 @@ def rename_face(fid: int, body: FaceRename, user: Writer, acl: Acl, db: Db) -> O
return Ok()


@router.post("/faces/{fid}/name")
def name_face(fid: int, body: FaceName, user: Writer, acl: Acl, db: Db) -> FaceNamed:
"""Name a face: a name another face of the namespace has joins it to that person (a merge: undo it with POST
/faces/merges/{merge}/undo); any other name is its own (undo: POST /faces/{fid} with no name)."""
_face_space(db, acl, fid, "curate")
with domain_errors():
out = facemod.name_face(db, fid, body.name, user.email)
if out["merge"]:
auth.audit(db, user.as_audit(), "face.merge", f"face:{fid}", {"into": out["into"]})
else:
auth.audit(db, user.as_audit(), "face.rename", f"face:{fid}")
return FaceNamed(**out)


@router.post("/faces/{fid}/merge")
def merge_face(fid: int, body: FaceMerge, user: Writer, acl: Acl, db: Db) -> FaceMerged:
"""Merge this face into another (the same person)."""
Expand Down
Loading
Loading
0