8000 feat(find): find assets by face, voice, entity, topic, type, content type and date by adeelahmad · Pull Request #398 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
registries or the public web. Files attached to the conversation go in; a file it makes reaches a namespace once the
person approves. Off by default; starting the gateway from Settings is in progress
([Code sandbox](docs/assistant.md#code-sandbox)).
- **Find anything by who, what and when.** Agents and the API can find assets by the faces seen in them, the people
heard, the entities they mention, the topics they're about, their kind and content type, when they were made (a
day, a month or a year), tags, language, objects seen, emotions heard, length and words in them, together or alone,
naming people and things instead of looking up ids. Each result says why it matched, with moments to cite
(`POST /find`, the MCP tool `find`, the assistant's `find_resources`; [Finding by any
attribute](docs/processing.md#finding-by-any-attribute)). The Library list takes faces, entities, topics, content
types and emotions as filters too.

## 0.13.0 <small>October 16, 2026</small> {id="0.13.0"}

Expand Down
20 changes: 20 additions & 0 deletions docs/api.md
10000
Original file line number Diff line number Diff line change
Expand Up @@ -600,6 +600,11 @@ repeat a parameter that takes several values (`?status=new&status=error`) to mat
| `origin` | where they came from: `source:<id>` (a connected source), `upload`, `paste`, `iiif`, `folder` (the archive's own folders) or `file` (another file imported by path) |
| `language` | language codes, ignoring case; `none` for assets whose language isn't known |
| `object` | kinds of object seen in them (`person`, `car` …; [Objects](#objects)), ignoring case |
| `face` | face identity ids: the assets they're seen in ([Video](video.md)) |
| `entity` | entity ids: the assets that mention them |
| `topic` | topic ids: the assets accepted as about them (not their narrower topics; `POST /find` brings those) |
| `content_type` | content type keys (`podcast`, `scan`, `email` …; [content types](#content-types-and-custom-nodes)) |
| `emotion` | emotions heard in them, ignoring case |
| `edited_by` | `me`: assets you edited (corrected a line of the text, changed the catalogue record, or renamed) |
| `sensitive` | where [NSFW classification](nsfw.md) is on: `only` the ones flagged NSFW, `exclude` them, or `include` them. Without it, flagged ones are left out where their namespace hides them (`display: hide`) |
| `sort` | `date`, `title`, `duration`, `speakers`, `status` or `importance`; `-` in front for descending (default `-date`). Assets without the value come last either way |
Expand Down Expand Up @@ -1105,6 +1110,7 @@ asset later leaves its uploaded file in place, like any other media file.
```
GET /api/v1/search
GET /api/v1/search/terms
POST /api/v1/find
GET /api/v1/similar
GET /api/v1/graph
GET /api/v1/mentions
Expand Down Expand Up @@ -1145,6 +1151,20 @@ key words marked), the key `words` it says, and the `topics` and `entities` in c
when an embedding model compared them too, else `words`; `terms` are the key words looked for and `meaning` says why
it wasn't by meaning when search by meaning is set up. An asset is never listed as like itself.

`POST /find` finds the assets you can read by any attribute ([finding by any attribute](processing.md#finding-by-any-attribute)).
The body takes `faces`, `speakers`, `entities` and `topics` (ids or names), `types`, `content_types` (keys or labels),
`date_from` and `date_to` (`2026-09-30`, `2026-09` or `2026`), `min_duration` and `max_duration` (seconds), `tags`,
`languages`, `objects`, `emotions`, `status`, `origins`, `collection`, `sensitive`, `title` (words in the title, the
namespace or a person's name), `text` (words said or written, as `GET /search` takes them) and `namespace`; `sort`
(`relevance`, the default with `text`, else `-date`; or `date`, `title`, `duration`, `-` for descending), `limit`
(default 20, at most 100) and `offset`. Filters combine with AND, the values of one with OR. The reply has `total`,
the page's `items` (rows as `GET /resources` lists them, each with `why`: `[{by, name, moments, count}]`, `by` being
`words`, `face`, `voice`, `entity`, `topic`, `content_type`, `object` or `emotion`, and each moment a `t0` and `t1` in
ms or a `page` from 0, with its `text` when it has words), `resolved` (what each name matched: `[{id, name, namespace}]`
per filter, and `narrower_topics`) and `text` (how the words were matched, as `GET /search` says, and whether the
moments looked through were `capped`). A name that matches nothing in the namespaces you can read is a 400 that names
the closest ones; an unknown field is a 422.

Search has no prefix search (`interp*` looks for the word "interp"). `GET /search/terms?prefix=interp` lists whole
words found in the namespaces you can read (`ns` for one) that start with it, the most frequent first, with how often and
in how many assets (`limit`, default 8, at most 20); the web app offers them as "Try …".
Expand Down
5 changes: 4 additions & 1 deletion docs/assistant.md
8000
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,10 @@ When the configured model supports function calling, chat becomes an agent.
- **What it can look things up with:** assets' text (transcripts and documents, and on-screen text), assets, passages,
outputs, entities and their mentions and timelines, the knowledge graph, people, and content similar to a text,
an asset, a topic or an entity (`find_similar`), and what happened in Lens, newest first (`timeline`, the
[stream of activities](automations.md#timeline-reads); `unreviewed` for what the person hasn't marked reviewed).
[stream of activities](automations.md#timeline-reads); `unreviewed` for what the person hasn't marked reviewed),
and assets by any attribute: faces seen, people heard, entities, topics, kind, content type, dates and the rest,
each with the moments that show why it matched (`find_resources`,
[Finding by any attribute](processing.md#finding-by-any-attribute)).
- **The person's review marks:** `mark_activities` marks activities of the timeline reviewed, important or hidden for
the person ([Timeline](timeline.md#review-marks)). After it has read text from outside Lens, hiding or marking
reviewed waits for their yes; marking important or bringing things back acts at once.
Expand Down
1 change: 1 addition & 0 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
|---|---|
| `search` | moments where words appear (written, spoken or shown on screen), best first: the asset, person, time, the matching text (matches in **bold**) and a link to that moment. Words, `"phrases"`, `OR`; by namespace, person or asset |
| `find_similar` | the assets (recordings, documents and images) most like a text, an asset, a topic or an entity: why each is alike (meaning, key words, topics, entities in common) and the moments that show it, with links |
| `find` | assets by any attribute: faces seen, people heard, entities, topics, kind, content type, dates (a day, month or year), tags, language, objects, emotions, length, collection and words in them; ids or names. Each says why it matched, with moments and links to cite ([Finding by any attribute](processing.md#finding-by-any-attribute)) |
| `list_namespaces` | the namespaces you can read, with their assets, hours, people and your role |
| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) |
| `list_speakers` | the people of your namespaces, the ones with the most talk time first |
Expand Down
32 changes: 32 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,38 @@ In the web app:
* a topic's and an entity's drawer, and a resource, topic or entity selected on the graph canvas, list similar
content too.

### Finding by any attribute

The API (`POST /find`), the assistant (`find_resources`) and the MCP server (`find`) find assets by what is known
about them, one filter for all three:

* who is **seen** in them (`faces`: face identities, [Video](video.md)) and **heard** (`speakers`: people, by voice);
* the **entities** they mention and the **topics** they were accepted as about (a topic brings its narrower topics);
* their **kind** (`types`: audio, video, transcript, document, image) and **content type** (`content_types`: podcast,
interview, scan, photo, email … [Content types](#content-types));
* **when** they were made (`date_from`, `date_to`: a day, a month or a year, so `2025` is the whole year) and how
long they are;
* their tags, language, the kinds of object seen in them, the emotions heard, their status, collection and where they
came from;
* and words said or written in them (`text`, matched as a search is, by meaning too when that is set up).

Filters combine with AND, the values of one filter with OR. Faces, people heard, entities and topics take an id or a
name: a name matches every face, person, entity (or one of the other ways it's said) or topic label called that in the
namespaces you can read, ignoring case, and `resolved` says what it matched. A name that matches nothing is an error
that lists the closest names, so an agent can correct itself.

Each asset found says **why** it matched, with up to three moments per reason to cite: where the words are, when the
face is on screen (or the pages it's on), the person's first lines, the lines that mention the entity, and the topics,
content type, kinds of object and emotions that matched. With `text`, the asset whose moment matches best comes
first; otherwise the newest. Nothing here calls a model, apart from `text` searching by meaning when it is set up
(one embedding request, logged in the activity ledger).

The Library list (`GET /resources`) takes the same attributes as ids: `face`, `entity`, `topic` (as it is, without its
narrower topics), `content_type` and `emotion`, besides its other filters ([API](api.md#resources)).

Planned, not built yet: these filters in the Library's filter bar and in saved collections, finding by what a photo
shows (`GET /search/photos`) inside the same filter, and ranking by more than the best matching moment.

## Background work

Imports, pipeline runs and folder scans return at once and run as jobs stored in SurrealDB. The server runs
Expand Down
102 changes: 102 additions & 0 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
variables,
)
from app.domain import entities as ents
from app.domain import find as findmod
from app.domain import speakers as spk
from app.domain import topics as topicmod
from app.domain.store import DB
Expand Down Expand Up @@ -648,6 +649,107 @@ def list_recordings(
return out


FIND_SORTS = ("relevance", "-date", "date", "-title", "title", "-duration", "duration")


@tool(
"find",
"Find assets by any attribute",
"Assets (recordings, videos, documents and images) you can read by who is seen in them (faces) and heard (speakers), the "
"entities they mention, the topics they're about, their kind and content type, when they were made, tags, language, "
"objects seen, emotions heard, length, collection, and words said or written in them. Filters combine with AND, the "
"values of one with OR. Faces, people heard, entities and topics are ids or names (ignoring case); `resolved` says what "
"each name matched, and one that matches nothing comes back as an error naming the closest. Each result says `why` "
"it matched, with up to three moments per reason (a time or a page, the words there, and a url that opens it), to cite.",
Arg("namespace", "string", "only this namespace"),
Arg("text", "string", 'words said or written in them (words, "phrases", OR); by meaning too when that is set up', max_length=300),
Arg("title", "string", "words that must all appear in the title, the namespace or a person's name", max_length=200),
Arg("faces", "string[]", "people seen in them: face ids or names"),
Arg("speakers", "string[]", "people heard in them: speaker ids or names"),
Arg("entities", "string[]", "entities they mention (people, organisations, places …): ids or names"),
Arg("topics", "string[]", "topics they're about: ids or labels; narrower topics count too"),
Arg("types", "string[]", "their kind", enum=MEDIA),
Arg("content_types", "string[]", "content types by key or label: podcast, interview, meeting_audio, scan, photo, email, document …"),
Arg("date_from", "string", "made on or after this day, month or year (2026-09-30, 2026-09, 2026)"),
Arg("date_to", "string", "made on or before this day, month or year"),
Arg("min_duration", "number", "at least this many seconds long", minimum=0),
Arg("max_duration", "number", "shorter than this many seconds", minimum=0),
Arg("tags", "string[]", "tags (ignoring case)"),
Arg("languages", "string[]", "language codes (en, de …); none for those whose language isn't known"),
Arg("objects", "string[]", "kinds of object seen in them (person, car, dog …)"),
Arg("emotions", "string[]", "emotions heard in them (happy, sad, angry …)"),
Arg("collection_id", "integer", "only what's in this collection (and the collections inside it)"),
Arg(
"sensitive",
"string",
"where NSFW classification is on: only those flagged NSFW, exclude them, or include them",
enum=("include", "only", "exclude"),
),
Arg("sort", "string", "relevance (the default with text), else date, title or duration; - first for descending", enum=FIND_SORTS),
Arg("limit", "integer", "how many", default=20, minimum=1, maximum=100),
Arg("offset", "integer", "skip this many (for the next page)", default=0, minimum=0, maximum=10000),
)
def find(
ctx: Context,
namespace: str | None,
limit: int,
offset: int,
sort: str | None,
collection_id: int | None,
**spec: Any,
) -> dict[str, Any]:
spaces, within = ctx.acl.scope(namespace)
if collection_id is not None:
spec["collection"] = collection_id
res = findmod.find(ctx.db, ctx.cfg, spaces, within, spec, sort=sort, limit=limit, offset=offset)
items = []
for r in res["items"]:
rid = r["id"]
why = []
for w in r["why"]:
moments = [
store.clean(
{
**({"page": m["page"] + 1} if m.get("page") is not None else {}),
**({"to_page": m["last_page"] + 1} if m.get("last_page") is not None else {}),
**({"time": store.tc(m["t0"])} if m.get("t0") is not None else {}),
**({"until": store.tc(m["t1"])} if m.get("t1") is not None and w["by"] == "face" else {}),
"speaker": m.get("speaker"),
"text": _cut(m["text"], 400) if m.get("text") else None,
"url": ctx.link(rid, m.get("t0")),
}
)
for m in w.get("moments") or []
]
why.append(store.clean({"by": w["by"], "name": w.get("name"), "count": w.get("count"), "moments": moments or None}))
items.append(
store.clean(
{
"recording_id": rid,
"title": r.get("title"),
"namespace": r.get("namespace"),
"recorded_at": r.get("recorded_at"),
"duration": store.tc(r.get("duration_ms")) if r.get("duration_ms") else None,
"media": r.get("media_kind"),
"speakers": r.get("speakers") or None,
"tags": r.get("tags") or None,
"language": r.get("language"),
"nsfw": True if (r.get("nsfw") or {}).get("flagged") else None,
"url": ctx.link(rid),
"why": why,
}
)
)
out: dict[str, Any] = {"total": res["total"], "results": items}
if res["resolved"]:
out["resolved"] = res["resolved"]
if res["text"]:
out["text"] = res["text"]
if offset + len(items) < res["total"]:
out["next_offset"] = offset + len(items)
return out


@tool(
"list_speakers",
"List people",
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
fields,
files,
filetree,
find,
graph,
graph_history,
hierarchy,
Expand Down Expand Up @@ -107,6 +108,7 @@
imports,
uploads,
search,
find,
searches,
speakers,
entities,
Expand Down
29 changes: 29 additions & 0 deletions fastapi_backend/app/api/v1/routes/find.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
"""Find resources by any attribute: faces, voices, entities, topics, kind, content type, dates and the rest (domain/find.py)."""

from __future__ import annotations

from fastapi import APIRouter

from app.api.deps import Acl, Cfg, CurrentUser, Db, domain_errors
from app.api.media import sign_urls
from app.domain import find as findmod
from app.schemas.find import FindQuery, FindResults

router = APIRouter(tags=["search"])


@router.post("/find")
def find_resources(body: FindQuery, acl: Acl, user: CurrentUser, db: Db, cfg: Cfg) -> FindResults:
"""Assets you can read that match every filter given: the people seen in them (faces) and heard (speakers), the
entities they mention, the topics they're about, their kind and content type, when they were made, their tags,
language, objects seen, emotions heard, length, status, collection and origin, and words said or written in them.
Faces, people heard, entities and topics can be named instead of numbered; `resolved` says what each name matched, and
a name that matches nothing is a 400 naming the closest ones. Each asset says `why` it matched, with up to three
moments per reason to cite (a time, or a page). With `text`, the best matching come first."""
spaces, within = acl.scope(body.namespace)
spec = body.model_dump(exclude={"namespace", "sort", "limit", "offset"}, exclude_none=True)
with domain_errors():
res = findmod.find(db, cfg, spaces, within, spec, sort=body.sort, limit=body.limit, offset=body.offset)
for x in res["items"]:
x["role"] = acl.role_in(x["space"], x.get("collection"))
return FindResults.model_validate(sign_urls(res, full=True))
14 changes: 14 additions & 0 deletions fastapi_backend/app/api/v1/routes/recordings.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,15 @@ def list_recordings(
description="where NSFW classification is on (docs/nsfw.md): only those flagged NSFW, exclude them, or include them. "
"Without it, flagged ones are left out where their namespace hides them (display hide)",
),
face: list[int] | None = Query(None, description="face identity ids: the assets they're seen in; repeat for several"),
entity: list[int] | None = Query(None, description="entity ids: the assets mentioning them; repeat for several"),
topic: list[int] | None = Query(
None, description="topic ids: the assets accepted as about them (not their narrower topics); repeat for several"
),
content_type: list[str] | None = Query(
None, description="content type keys (podcast, scan, email …; GET /content-types): their assets; repeat for several"
),
emotion: list[str] | None = Query(None, description="emotions heard in them (ignoring case); repeat for several"),
collection: int | None = Query(None, description="a collection: the assets in it and in the collections inside it"),
field: int | None = Query(None, description="a custom field (of resources): the assets with a value for it, or with `value`"),
value: str | None = Query(
Expand Down Expand Up @@ -206,6 +215,11 @@ def list_recordings(
field=by_field,
objects=object,
sensitive=sensitive,
faces=face,
entities=entity,
topics=topic,
content_types=content_type,
emotions=emotion,
)
response.headers["X-Total-Count"] = str(total)
for r in rows:
Expand Down
Loading
Loading
0