8000 feat(video): blur faces in a copy of a video for sharing by adeelahmad · Pull Request #699 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -321,8 +321,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
Open signed in), encryption at rest with a key per namespace, roles per namespace (viewer, editor, owner, or custom
roles made of permissions), who sees each asset, set for many at once from the Library, access that ends on a date
you pick (a role in a namespace or on a collection, or permission on one asset), a downloads switch per asset (off,
it still plays and reads, but only its owners save its files), email invitations, removing someone for good, and an
audit log. Prompt injection meets
it still plays and reads, but only its owners save its files), a video's copy with faces blurred for sharing (made
when you ask, and played to visitors instead of the original if you pick that), email invitations, removing someone
for good, and an audit log. Prompt injection meets
several layers: what the assistant reads (recordings, documents, web pages, tool results) reaches the model marked as
data, not instructions, and text that reads like instructions to an AI is flagged (by free patterns, or an optional
classifier model that runs on your machine and is fetched only when you pick it). Once an answer has read such text,
Expand All @@ -336,7 +337,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
field in Settings can refer to an item there, and Lens reads it without anyone, agents included, seeing it.
([Encryption](docs/encryption.md), [secrets](docs/secrets.md), [access](docs/access.md),
[prompt injection](docs/assistant.md#third-party-content-and-prompt-injection), [HTTPS](docs/https.md),
[remote access](docs/remote-access.md); [checks](docs/requirements.md#stay-private-and-safe))
[remote access](docs/remote-access.md), [blurring faces](docs/video.md#blurring-faces-for-sharing);
[checks](docs/requirements.md#stay-private-and-safe))
- **Use open standards.** RDF and Dublin Core for every record, OAuth for apps and OpenID Connect sign-in, MCP, MQTT,
IIIF for publishing (with your team's shared notes and comments as W3C Web Annotations, for people with a role on the
asset only), OpenTelemetry and Prometheus for monitoring, an optional Fedora mirror, and a CSV catalogue of the assets
Expand Down
8 changes: 8 additions & 0 deletions changelog.d/blur-faces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- **Blur faces for sharing.** A video's People tab has **Blur faces for sharing**: every face found is ticked, and
Enter makes a copy of the video with those faces blurred, by ffmpeg over the faces' tracks, in the background as a
heavy step (quiet processing hours apply), with its progress shown. The copy is kept with the video's files, the
original stays as it is, and an owner can have visitors (public pages, embeds, share links, IIIF, feeds) play the
copy instead. Nothing changes for a video until someone asks. Agents use `GET` and
`POST /api/v1/recordings/{id}/blur-faces` (all, unnamed or chosen faces), `PUT .../blur-faces/visitors`, and the
MCP tools `blur_faces`, `get_face_blur` and `set_face_blur_for_visitors`
([Blurring faces for sharing](docs/video.md#blurring-faces-for-sharing)).
13 changes: 13 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1574,6 +1574,9 @@ DELETE /api/v1/faces/{fid}
POST /api/v1/faces/{fid}/merge
POST /api/v1/faces/{fid}/speaker
POST /api/v1/faces/{fid}/dismiss
GET /api/v1/resources/{rid}/blur-faces
POST /api/v1/resources/{rid}/blur-faces
PUT /api/v1/resources/{rid}/blur-faces/visitors
```

`PUT /namespaces/{name}/faces/mode {mode?, purpose?, pixelate?, reprocess?}` (owners) sets the namespace's face mode
Expand All @@ -1591,6 +1594,16 @@ suggested for it. `POST /faces/{fid}/name {name}` (editors) names one: a name an
case) joins it to that face, a merge whose id comes back as `merge` to undo with `POST /faces/merges/{mid}/undo`; any
other name becomes its own, undone with `POST /faces/{fid} {"name": ""}` ([Naming faces](video.md#naming-faces)).

`POST /resources/{rid}/blur-faces {who?, tracks?}` (the process permission) makes a copy of a video with faces
blurred for sharing, in the background: every face found (`who` `all`, the default), the ones nobody has named
(`unnamed`), or the face tracks in `tracks` (`chosen`); 202 with the job, 400 for anything but a video or with no faces
to blur. The copy is kept as one of the resource's files (an attachment labelled "Faces blurred"); asking again
replaces it. `GET /resources/{rid}/blur-faces` (members) gives `{blurred, job}`: the copy (`file`, `name`, `size`, a
signed `download`, `faces`, `who`, `tracks`, `made_at`, `visitors`) and the latest job making one (`status`, `progress`
in percent while ffmpeg runs, `waits` while quiet processing holds it, `error`). `PUT .../blur-faces/visitors {on}`
(owners) has visitors play the copy from `/audio` and `/media` (and IIIF and feeds) instead of the original
([Blurring faces for sharing](video.md#blurring-faces-for-sharing)).

### Objects

The `objects` step finds the people, vehicles, animals and everyday things (the 80 kinds of the COCO dataset) on a
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ jobs), drain it (it also hands its job back after the step it's on) and resume i
record, so it holds across restarts. `GET /api/v1/events` streams job progress as server-sent events.

**Quiet processing** (Settings › Workers, both off by default) keeps the heavy steps (`workers.heavy_steps`: by
default transcribe, read, diarize, shots, ocr, faces, objects and nsfw, the ones that work through media on this
machine) to when it suits a small machine at home:
default transcribe, read, diarize, shots, ocr, faces, objects, nsfw and blur_faces, the ones that work through media
on this machine) to when it suits a small machine at home:

- `workers.heavy_hours`, a window like `01:00-06:00` in `workers.timezone` (UTC by default; it may pass midnight).
Outside it no job starts a heavy step: a job that reaches one goes back on the queue, its log saying why, and waits
Expand Down
7 changes: 6 additions & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,9 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `mark_nsfw` | mark a resource NSFW or not by hand, or hand it back to the model's score. Needs a write-scope token and editor access |
| `set_language` | set or correct the [language an asset is in](api.md#an-assets-language) as a BCP 47 tag with a region if it matters (`en-GB`, `pt-BR`), written the usual way and kept over the one transcription detects; left out, it's forgotten. Needs a write-scope token and editor access |
| `enhance` | [enhance an image](enhance.md) whatever the settings, keeping the original, and read it again from the enhanced copy. Needs a write-scope token and the process permission |
| `get_face_blur` | a video's faces (track ids and names, to pick from) and its [copy with faces blurred](video.md#blurring-faces-for-sharing): the file and where to download it, whose faces it blurs and whether visitors play it; and the latest job making one, with how far it got, why it waits or why it failed. Members of its namespace |
| `blur_faces` | make a [copy of a video with faces blurred](video.md#blurring-faces-for-sharing) for sharing: all faces, the unnamed ones or the tracks given; it runs in the background and is kept with the video's files, the original unchanged. Needs a write-scope token and the process permission |
| `set_face_blur_for_visitors` | have visitors (public pages, embeds, share links, IIIF, feeds) play a video's copy with faces blurred instead of the original, or not. Needs a write-scope token and owner access |
| `namespace_integrations` | whether a namespace has [its own MCP server](#one-namespaces-mcp-server) and [chat completions](api.md#chat-completions-for-any-app) on, their addresses, and the tools offered there |
| `set_namespace_integrations` | turn a namespace's own MCP server or chat completions on or off, and pick the read tools its chat completions may use. Owners, with a write-scope token |
| `list_files` | a folder of [the graph as files](file-tree.md): your namespaces, then their collections, topics (nested by narrower topics) and entities, each holding its assets' files |
Expand Down Expand Up @@ -215,7 +218,9 @@ app (merges and deletes can be undone), `write_note`, which adds a note, `create
queues an episode, `create_feed` and `publish_to_feed`, which make and fill a [published feed](feeds.md),
`save_circle`, `share_with_circle` and `undo_share`, which change your [circles](circles.md) and what you share with them,
`write_spec`, which queues a spec, `translate_transcript`, which queues a translation, `draft_reply` and `edit_reply`, which draft and change [email replies](email-replies.md) (nothing is sent), `send_reply`, which asks to send one (it waits in Approvals), `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and
editors), `set_language`, which sets an asset's language (editors), `enhance`, which enhances an image and reads it again (the process permission),
editors), `set_language`, which sets an asset's language (editors), `enhance`, which enhances an image and reads it again (the process permission), `blur_faces` and
`set_face_blur_for_visitors`, which make a video's copy with faces blurred and choose whether visitors play it (the
process permission, and owners),
`set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `set_secret_reference`, which points a secret field at a password manager's item (admins), `install_plugin`, which
installs a plugin from a marketplace as your own extension, `add_mcp_server`, which adds an
MCP server (admins), `subscribe_feed` and `unsubscribe_feed`, which follow and stop following news feeds (admins), `set_mail_routing`, which changes where an email account's messages go (admins), `add_mesh_node`,
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -410,6 +410,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Roles per namespace: viewer, editor, owner, or custom roles made of permissions | [test_roles.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_roles.py) `test_builtin_roles_keep_their_meaning`, `test_custom_roles_allow_exactly_their_permissions`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `person-add` |
| Access that ends on a date: a role in a namespace or on a collection, or permission on an asset, opens nothing after its end and lists show it as expired | [test_grant_ends.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_grant_ends.py) `test_asset_permission_holds_until_it_ends`, `test_namespace_role_holds_until_it_ends`, `test_collection_role_holds_until_it_ends`; [test_setup_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_setup_assistant.py) `test_the_assistant_gives_a_role_until_a_date`; [access-until.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/access-until.test.ts) `sends the end of the day picked, and nothing for an empty field` |
| A downloads switch per asset: off, only its owners save its files (its own file, its PDF, supplementary files) while everyone who sees it still plays it and reads its text | [test_downloads.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_downloads.py) `test_downloads_off_refuses_a_viewers_download_but_still_plays`, `test_downloads_off_keeps_iiif_playing`; [access-model.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/access-model.test.ts) `sends the downloads switch only when it changes, on by default`, `says why the files can't be saved, only when they can't` |
| A video's copy with faces blurred for sharing, made by ffmpeg over the faces' tracks only when asked (all, unnamed or chosen faces; API, MCP and the People tab), kept with its files, and played to visitors instead of the original only when an owner picks that | [test_face_blur.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_face_blur.py) `test_regions_cover_each_track_between_and_around_its_samples`, `test_the_filter_graph_blurs_each_region_for_its_time`, `test_blurring_faces_for_sharing`; [face-blur.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/face-blur.test.ts) `asks for every face when all are ticked, else for the ones ticked in order`, `says how far it got, why it waits and why it failed` |
| Email invitations, and removing someone for good | [test_users.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_users.py) `test_invites_are_emailed_when_mail_is_set_up`, `test_admins_remove_people_for_good` |
| An audit log | [test_settings.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_settings.py) `test_settings_are_editable_and_secret`; [test_files.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_files.py) `test_change_a_file_and_the_audit_log` |
| Outside text reaches the model marked as data; text that reads like instructions is flagged, by patterns or an optional local classifier | [test_prompt_injection.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_prompt_injection.py) `test_what_tools_read_is_marked_as_data`, `test_a_result_that_reads_like_instructions_is_flagged`; [test_injection_model.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_injection_model.py) `test_until_the_model_is_here_the_patterns_stand_alone` |
Expand Down
35 changes: 35 additions & 0 deletions docs/video.md
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,41 @@ videos, documents and images.
A published video recording becomes a IIIF Video on a Canvas with its width, height and duration. It gets a thumbnail,
a "Text on screen" annotation layer targeting `#xywh=…&t=…`, and its shots as a Range.

## Blurring faces for sharing

A person who may process a video can ask for a **copy with faces blurred**, to share it without showing who is in it.
Nothing changes for a video until someone asks; the original stays as it is.

- **Asking:** the People tab's **Blur faces for sharing** button opens a list of the faces found, all ticked; Enter (or
the button) starts it. Agents ask with `POST /api/v1/recordings/<id>/blur-faces` (`who`: `all`, the default,
`unnamed` for the faces nobody has named, or `chosen` with `tracks`, the face tracks' ids) or the MCP tool
`blur_faces`. Asking while a copy is being made gives the same job. It needs faces found first (the faces step, with
detection or recognition on).
- **How it's made:** ffmpeg, which Lens already uses, draws a strong blur over each face track's boxes over time, as a
background job step (`blur_faces`). The faces step saw each face only on the frames it sampled, so between two
samples that saw a face the blur covers both of its boxes, it starts a sample before the first and ends a sample
after the last, and a margin goes round each box: the copy errs on blurring too much, never too little. Many boxes
are merged into fewer, larger ones (at most 300 at a time). The copy is H.264 video with AAC sound in MP4, and what
the file says about itself (its title, where a phone recorded it) is left out. Faces the detector missed aren't
blurred: check the copy before sharing it.
- **Heavy work:** `blur_faces` is one of the heavy steps (`workers.heavy_steps`), so Settings › Workers › Quiet
processing keeps it to its hours and to so many at a time. The People tab shows the job: waiting (and why), how far
ffmpeg got, or why it failed.
- **The copy:** it's kept with the video's files (the Files tab), an attachment labelled "Faces blurred", encrypted
like the rest when the namespace is, and deleted with the video. Asking again replaces it. `GET /api/v1/recordings/<id>/blur-faces` (members of its namespace) and the MCP tool
`get_face_blur` say what there is and how the job is doing, with a link to save the copy.
- **Visitors:** an owner can have visitors play the copy instead of the original: the People tab's **Visitors see
this copy** switch, `PUT /api/v1/recordings/<id>/blur-faces/visitors` with `{"on": true}`, or the MCP tool
`set_face_blur_for_visitors`. Then a request without a role in the namespace (public pages, embeds, share links,
IIIF and podcast feeds) gets the copy from `/audio` and `/media`; members still get the original. If the copy goes
missing while this is on, visitors get nothing rather than the faces; deleting it in the Files tab turns the choice
off with it. Pictures (frames, keyframes, pages) are pixelated for visitors by the namespace's own setting above.
- Asking for a copy is audited as `recording.blur_faces`, and the visitors' choice as `recording.blur_faces.visitors`.

Refine later: a document's or an image's pages; following a face frame by frame between samples instead of covering
both boxes; IIIF manifests and podcast feeds giving the copy's own type and size; the in-app assistant asking for a
copy; a copy made on a mesh worker node reaching the primary.

## Playable copies

Phones and cameras write files many browsers can't play: HEVC video in a .mov, AVI, MPEG transport streams, AMR voice
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/api/iiif.py
Original file line number Diff line number Diff line change
Expand Up @@ -508,7 +508,7 @@ def iiif_audio(rid: int, request: Request, user: OptionalUser, db: Db, cfg: Cfg)
if full.get("source") in documents.KINDS: # a document or an image: its file, to save
_may_save(request, db, user, rid)
return serve_document(db, cfg, full, request)
return serve_audio(db, cfg, full, rid, request)
return serve_audio(db, cfg, full, rid, request, visitor=not _member(request, db, user, full["space"]))


@router.get("/iiif/{rid}/pdf", response_class=FileResponse, responses={200: {"content": {"application/pdf": {}}}})
Expand Down
Loading
Loading
0