8000 feat(nsfw): classify NSFW pictures with a local model, per namespace by adeelahmad · Pull Request #306 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **NSFW classification, with a model on your server.** Turn it on in Settings → NSFW for every namespace, or on a
namespace's page for that one (off by default). The new `nsfw` step scores a video's sampled frames, a document's
pages or an image with an Apache-2.0 model (fetched on first use, about 105 MB with ONNX Runtime) and flags the
resource when a picture scores at least the threshold. Flagged resources are blurred until someone chooses to see
them, left out of the Library, or shown with a label; editors can mark one NSFW or not by hand, and **Classify
existing** classifies what was there before. Agents use it through the API and MCP (`nsfw_settings`, `set_nsfw`,
`mark_nsfw`) ([NSFW classification](docs/nsfw.md)).

- **Apache Tika.** Lens can use an Apache Tika server (`--profile tika` in the compose files, or one of your own, set
in Settings → Documents). Documents nothing here can make into a PDF, such as Word files on a lean server or a
Raspberry Pi, are read by Tika as text, and every document's own title, authors, language and dates are read as
Expand Down
30 changes: 29 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,9 @@ DELETE /api/v1/namespaces/{name}/notifications/{tid}
POST /api/v1/namespaces/{name}/notifications/{tid}/test
POST /api/v1/namespaces/{name}/notifications/{tid}/secret
GET /api/v1/namespaces/{name}/notifications/{tid}/deliveries
GET /api/v1/namespaces/{name}/nsfw
PUT /api/v1/namespaces/{name}/nsfw
POST /api/v1/namespaces/{name}/nsfw/classify
GET /api/v1/namespaces/{name}/collections
POST /api/v1/namespaces/{name}/collections
GET /api/v1/namespaces/{name}/collections/{cid}
Expand Down Expand Up @@ -294,6 +297,15 @@ removes it). `POST …/{tid}/test` sends a test message now and answers `{ok, co
lists the latest 50 messages with their `status` (`pending`, `sending`, `sent`, `failed`, `dropped`), `attempts`,
answer `code` and `error`. Changes are audited as `namespace.notification.create`, `.update`, `.delete` and `.secret`.

`/namespaces/ 10BC8 {name}/nsfw` says how [NSFW classification](nsfw.md) works in a namespace (viewers): whether it's
`enabled`, the `threshold` and `display` that apply, what the namespace chose itself (`own`, `null` where it follows
the archive) and the archive's settings (`archive`), and how many resources are `classified`, `flagged` and
`unclassified`. `PUT` with `{enabled?, threshold?, display?, classify?}` changes the namespace's own choices (owners;
`null` follows the archive again, a new threshold flags or unflags what was classified at once), and `classify: true`
also queues the `nsfw` step for its videos, documents and images not classified yet, answering how many runs were
`queued`. Audited as `namespace.nsfw`. `POST …/nsfw/classify` queues them on its own (editors) and answers the `jobs`.
Both are a 409 where classification is off in the namespace.

### Collections of a namespace

Every recording lives in exactly one collection of its namespace, and collections nest (at most 8 deep, 5000 per
Expand Down Expand Up @@ -340,6 +352,7 @@ GET /api/v1/resources/tags
GET /api/v1/resources/origins
GET /api/v1/resources/languages
GET /api/v1/resources/objects
GET /api/v1/resources/nsfw
POST /api/v1/resources/tags
POST /api/v1/resources/collection
GET /api/v1/resources/{rid}
Expand All @@ -358,6 +371,7 @@ GET /api/v1/resources/{rid}/ip-groups
PUT /api/v1/resources/{rid}/ip-groups/{gid}
DELETE /api/v1/resources/{rid}/ip-groups/{gid}
GET /api/v1/resources/{rid}/player
PUT /api/v1/resources/{rid}/nsfw
GET /api/v1/resources/{rid}/embed-link
GET /api/v1/resources/{rid}/audio
GET /api/v1/resources/{rid}/wordcloud.svg
Expand Down Expand Up @@ -398,6 +412,7 @@ repeat a parameter that takes several values (`?status=new&status=error`) to mat
| `language` | language codes, ignoring case; `none` for recordings whose language isn't known |
| `object` | kinds of object seen in them (`person`, `car` …; [Objects](#objects)), ignoring case |
| `edited_by` | `me`: recordings you edited (corrected a line of the transcript, changed the catalogue record, or renamed) |
| `sensitive` | where [NSFW classification](nsfw.md) is on: `only` the ones flagged NSFW, `exclude` them, or `include` them. Without it, flagged ones are left out where their namespace hides them (`display: hide`) |
| `sort` | `date`, `title`, `duration`, `speakers`, `status` or `importance`; `-` in front for descending (default `-date`). Recordings without the value come last either way |
| `limit`, `offset` | one page (default 500 rows, at most 1000) |

Expand Down Expand Up @@ -1043,12 +1058,25 @@ and each place it was found (`[t, x, y, w, h, score]`, at most 500). The kinds a
`GET /resources?object=` filters on; search finds them by name (`source: object`), and its `object` filter and facet
use them ([search](#search)). Moving or deleting a resource moves or deletes its objects.

### NSFW

The `nsfw` step scores a video's sampled frames, a document's pages or an image for NSFW pictures with a model on the
server, where [NSFW classification](nsfw.md) is on in the resource's namespace (off by default); it comes after
`objects` in the standard pipeline, and is skipped, saying why, where it's off. A resource, its player and its Library
row carry `nsfw`: `{flagged, score, manual, blur, display, threshold, checked, of, at}`, and on the resource and its
player `hits`, its highest-scoring pictures (`{t, score, frame}`, a signed link); `null` where it's off in the
namespace or the resource wasn't classified. `PUT /resources/{rid}/nsfw {flagged}` (editors) marks it NSFW (`true`)
or not (`false`) by hand, over the model's score, or hands it back to the score (`null`); a 409 where it's off.
Audited as `recording.nsfw`. `GET /resources/nsfw` (`ns` for one namespace) says whether it's `on` in any namespace
you read, how many resources are `flagged`, and how many of those the Library leaves out (`hidden`). Search hits and
photo search results carry `nsfw: true` for the ones the web app blurs.

### Descriptions

The `describe` step has a model that can see images, the one an admin chose as `llm.vision_model`, say in a few
sentences what each page of a document or an image shows, and each shot of a video (from its keyframe): the setting,
the people (by what they do and wear, never who they might be), the things and any text in it. It comes after
`objects` in the standard pipeline, and describes at most `llm.describe_max` pages or shots of a resource. Without an
`nsfw` in the standard pipeline, and describes at most `llm.describe_max` pages or shots of a resource. Without an
LLM, or with no `llm.vision_model`, it's skipped, saying why; a model that can't see makes it fail with what the server
said ([Configuration](configuration.md#descriptions)). The player's `descriptions` lists them in order:
`{idx, t0, t1, text, model, frame, paged}`: the shot or page, where it starts and ends (ms; on pages, the page from 0
Expand Down
3 changes: 2 additions & 1 deletion docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ never fetch a speech or voice model.
| Face models | `video.face_engine` is `opencv` (the default) | OpenCV, YuNet and SFace |
| Face models | `video.face_engine` is `anytopdf` | SFace, for anytopdf to describe faces with (anytopdf too) |
| Object model | `video.object_engine` is `yolox` (the default) or `anytopdf` | YOLOX-s, and ONNX Runtime for `yolox` |
| NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| yt-dlp | `documents.video_urls` is on | the yt-dlp program, release 2026.08.19 (Unlicense), to download videos and podcasts imported by address |
| Chat model, embedding model | the LLM provider or embeddings server is Ollama | pulled on that server (`/api/pull`) |
Expand All @@ -27,7 +28,7 @@ never fetch a speech or voice model.
- Python packages go into `data_dir/python`, at the versions in `uv.lock`. Packages the image already has win.
- Models go into `data_dir/models`: the Hugging Face, ModelScope and PyTorch caches point there (unless `HF_HOME`,
`MODELSCOPE_CACHE` or `TORCH_HOME` say otherwise). They outlast container rebuilds and upgrades.
- Model files fetched by URL (YOLOX, YuNet, SFace) are checked against their SHA-256.
- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model) are checked against their SHA-256.
- A job step that needs something still being fetched waits for it instead of falling back or failing. If a fetch fails,
the step runs as before (for transcription, falling back to another engine), and the fetch is tried again an hour
later, when the settings change, or on **Check again now**.
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ Start from `fastapi_backend/archive.example.yaml`, which documents every key. Th

Admins can change, in Settings: transcription, speaker separation, speech providers, voice IDs, analysis, the LLM
provider and key, a local model, the AI assistant, search (word search and search by meaning), reports and graph,
video, OCR, faces and objects, workers, sign-in, components, access and embedding (embed frame ancestors, transcript
video, OCR, faces and objects, [NSFW classification](nsfw.md), workers, sign-in, components, access and embedding (embed frame ancestors, transcript
upload limit, allowed hosts, trusted proxies, session length), remote access, email, chat rooms, notifications,
telemetry, the Fedora repository, sensors, file shares, uploads, storage, documents, API keys, and IIIF and metadata. The API
refuses an allowed-host list that leaves out the address you are using.
Expand Down
10 changes: 7 additions & 3 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,9 @@ front of the web app also needs `TRUST_PROXY_HEADERS=true` ([Authentication](aut
| `search` | moments where words are said, shown on screen or written, best first: the recording, speaker, time, the matching text (matches in **bold**) and a link to that moment. Words, `"phrases"`, `OR`; by namespace, speaker or recording |
| `find_similar` | the recordings, documents and images most like a text, a recording, a topic or an entity: why each is alike (meaning, key words, topics, entities in common) and the moments that show it, with links |
| `list_namespaces` | the namespaces you can read, with their recordings, hours, speakers and your role |
| `list_recordings` | recordings, newest first, by namespace, title words, dates, speaker, tag or kind |
| `list_recordings` | recordings, newest first, by namespace, title words, dates, speaker, tag or kind, and whether they're flagged NSFW (`sensitive`) |
| `list_speakers` | the speakers of your namespaces, the ones who talk most first |
| `get_recording` | one recording: when, how long, speakers, summary and key points, chapters, the entities and keywords most mentioned, its link |
| `get_recording` | one recording: when, how long, speakers, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW |
| `get_transcript` | a recording's lines in order (speaker, time, text, link), a page at a time, from a line or a time window |
| `fetch` | a recording's whole text at once, as `[m:ss] Speaker: …` lines |
| `cite` | a citation for a moment (a line or a second, and a few lines from there): the words, who said them, the recording and date, the time, the link, and all of it as Markdown |
Expand All @@ -68,6 +68,9 @@ front of the web app also needs `TRUST_PROXY_HEADERS=true` ([Authentication](aut
| `write_note` | write a note or a thing's page, marked as written by an assistant. Needs a write-scope token and editor access |
| `create_podcast` | make a two-host learning [episode](podcasts.md) about recordings or lines in them, with an angle, a length and a style; it's made in the background. Needs a write-scope token and editor access where it goes |
| `get_podcast` | an episode: how far it has got, its script with each line's speaker and the sources it cites (links), and what the fact-check changed |
| `nsfw_settings` | how [NSFW classification](nsfw.md) works in a namespace: on or off, the threshold, how flagged resources show, and how many are classified, flagged and not classified yet |
| `set_nsfw` | turn NSFW classification on or off, set its threshold and display, for a namespace (owners) or the archive (admins), and optionally classify what's there. Needs a write-scope token |
| `mark_nsfw` | mark a resource NSFW or not by hand, or hand it back to the model's score. Needs a write-scope token and editor access |
| `list_files` | a folder of [the graph as files](file-tree.md): your namespaces, then their collections, topics (nested by narrower topics) and entities, each holding its assets' files |
| `read_file` | a text file of that tree (an asset's `.txt` text or `.json` metadata), a page at a time; other files' download link |
| `file_storage` | where Lens keeps its own files (notes' images and attachments) and the connections it could use; `check` tests it. Admins only |
Expand All @@ -82,7 +85,8 @@ expect (`results` with `id`, `title`, `url`; `fetch` by that `id`).

Every tool reads, except `suggest_topic`, which adds suggestions people accept or dismiss, `propose_graph_change`,
which records an undoable change (proposed by default), `write_note`, which adds a note, `create_podcast`, which
queues an episode, and `set_file_storage`, which changes where new files are kept (admins).
queues an episode, `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and
editors), and `set_file_storage`, which changes where new files are kept (admins).
Importing, editing and curating stay in the web app and the API.

### What the archive says is data
Expand Down
90 changes: 90 additions & 0 deletions docs/nsfw.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# NSFW classification

Lens can tell which videos, documents and images have NSFW pictures in them (nudity and sexual content), with a model
that runs on your own server. It's **off** until an admin turns it on for every namespace, or a namespace's owner for
theirs. Pictures never leave the server for it, and no model provider is called, so it costs nothing per picture.

## Turning it on

- **Every namespace:** Settings → NSFW, **Classify NSFW pictures** (`nsfw.enabled`).
- **One namespace:** its page (Admin → Namespaces → the namespace → NSFW), or `PUT /api/v1/namespaces/{name}/nsfw`
(owners). A namespace can also stay off while the rest is on, and have its own threshold and display.

Once it's on, the **nsfw** step in the standard pipeline classifies what's added from then on. Resources that were
there before are classified when someone asks: **Classify existing** on the namespace's page, `classify: true` with the
`PUT`, or `POST /api/v1/namespaces/{name}/nsfw/classify` (editors). A pipeline of your own runs it when you add the
**nsfw** step to it.

The first time a resource waits at the step, the worker fetches the model and ONNX Runtime (about 105 MB together,
[Components](components.md)); until then nothing is downloaded, so a Raspberry Pi that never turns it on stays small.

## What it does

The nsfw step scores each of a video's sampled frames (from the `shots` step), each page of a document, or the image,
from 0 (safe for work) to 1, and keeps the highest. A long video or document has at most `nsfw.max_frames` pictures
scored, spread evenly over it. A resource is **flagged** when its highest score is at least the namespace's
threshold. Changing the threshold flags or unflags what was already classified at once, without running the model
again.

The model is [AdamCodd/vit-base-nsfw-detector](https://huggingface.co/AdamCodd/vit-base-nsfw-detector) (Apache-2.0):
a Vision Transformer fine-tuned to tell NSFW from SFW pictures, as its author's 8-bit ONNX export, pinned to one
revision and checked against its SHA-256. It was trained to be strict: pictures with a lot of skin count as NSFW. It
is less sure of generated pictures. So a person can **mark a resource NSFW or not** by hand (editors), and their call
wins over the score until they hand it back to the model.

## How flagged resources show

`nsfw.display`, or the namespace's own:

| Display | In the web app |
|---|---|
| `blur` (default) | its pictures (the video, frames, faces, pages and photo search results) are blurred until someone chooses to see them |
| `hide` | left out of the Library unless you ask for them, and blurred when shown |
| `show` | shown as they are, with a label |

A flagged resource has an **NSFW** label in the Library, and its page says what was found, with **Show pictures**
and, for editors, **Not NSFW**. Its Details tab says what classification found on any classified resource, with
**Mark NSFW**. Every mark comes with Undo. Where it's on in a namespace you can read, the Library has an **NSFW**
filter: as each namespace shows them, including hidden ones, only flagged ones, or leaving them out
(`/library?namespace=…&nsfw=only` opens it so).

Blurring and hiding are display preferences, not access control: anyone who may read a resource can still open it,
see its pictures and read its classification.

## Settings

| Setting | Default | |
|---|---|---|
| `nsfw.enabled` | `false` | classify in every namespace that doesn't say otherwise |
| `nsfw.threshold` | `0.8` | flag a resource when one of its pictures scores at least this (0.5 to 0.99) |
| `nsfw.display` | `blur` | `blur`, `hide` or `show` |
| `nsfw.max_frames` | `300` | the most pictures scored per resource (1 to 5000) |

A namespace's own `enabled`, `threshold` and `display` win over these; `null` follows them again.

## API and agents

- `GET /api/v1/namespaces/{name}/nsfw`: what applies in a namespace, what it chose itself, the archive's settings, and
how many resources are classified, flagged and not classified yet.
- `PUT /api/v1/namespaces/{name}/nsfw {enabled?, threshold?, display?, classify?}` (owners). Audited as `namespace.nsfw`.
- `POST /api/v1/namespaces/{name}/nsfw/classify` (editors): queue the nsfw step for what isn't classified yet.
- `PUT /api/v1/resources/{rid}/nsfw {flagged}` (editors): `true`, `false`, or `null` to let the score decide. Audited
as `recording.nsfw`.
- Resources, their player and Library rows carry `nsfw`: `{flagged, score, manual, blur, display, threshold, checked,
of, at}`, and on a resource and its player `hits`, its highest-scoring pictures with signed links. It's `null` where
classification is off in the namespace or the resource wasn't classified.
- `GET /api/v1/resources?sensitive=only|exclude|include` filters the Library, and `GET /api/v1/resources/nsfw` says
whether it's on in any namespace you read, how many are flagged, and how many of those are hidden. Search hits and
photo search results carry `nsfw: true` for the ones the web app blurs.
- [MCP](mcp.md): `nsfw_settings`, `set_nsfw` (owners for a namespace, admins for the archive) and `mark_nsfw`
(editors); `get_recording` and `list_recordings` say what's flagged.

The resource's job log says what the step found: `flagged NSFW: highest score 0.93 (threshold 0.8) over 1 page(s)`.

## Refine later

- Blur flagged pictures for visitors on public pages, embeds and IIIF, as faces can be pixelated.
- Keep the Library's NSFW filter in saved views.
- Skip sending flagged pictures to a cloud vision model (the `describe` step).
- Other models (a smaller one for a Raspberry Pi, one trained on generated pictures) to choose from.
- Scores per picture in the player, to jump to what was flagged.
Loading
Loading
0