8000 feat(search): filter by date, tag, origin and language by adeelahmad · Pull Request #695 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
8000
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
rules, without a model, and a decision model types the ones the rules can't. Find what is most like a passage, an
asset, a topic or an entity, in a Similar tab on every asset or by selecting text anywhere, and, with anytopdf, photos
by what they show. Search filters moments by person, emotion or the sound heard in them (laughter, applause,
music), with a count for each. Pages and shots are described in words by the model provider's vision model or, off by
music), with a count for each, and keeps to the assets recorded between two days, with a tag, from a source or in a
language, in the link and for agents too. Pages and shots are described in words by the model provider's vision model or, off by
default, by Florence-2 or Moondream on your own server. A recording's sound can be cleaned up before it's transcribed
(noise, volume, echo, and silence left out, with the transcript's times kept), and transcripts cleaned up before
analysis (stutters, "um"s, spacing and, with a local model, punctuation put back, and grammar by a LanguageTool server
Expand Down
8 changes: 8 additions & 0 deletions changelog.d/search-filters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- **Search by when, tag, source and language.** `GET /api/v1/search` takes `from` and `to` (days: when the asset was
recorded, or added when its file doesn't say), `tag`, `origin` and `language`, the Library's filters of the same
names, and keeps every kind of hit to the assets that have them; the MCP `search` tool takes `date_from` and
`date_to` (a day, a month or a year), `tags`, `origins` and `languages`. On the search page they're compact controls
at the top of the filter panel, empty until picked, typed in the box too (`tag:interview`, `from:2026-01-01`), shown
as chips, kept in the link and in saved searches. A search's object filter now also keeps text on screen, file lines
and descriptions to the assets the object is seen in, not only what was said
([search](docs/api.md#search)).
17 changes: 14 additions & 3 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1351,7 +1351,12 @@ namespaces you can read, best first: every word
(English stemming), "quoted phrases" as written, `OR` between alternatives, `NOT` or `-` before a word or phrase to leave out the lines
with it; `ns`, `speaker`, `emotion`, `event` (a sound heard, such as laughter, applause or music, typed any way; music
is stored as `BGM`) and `recording` narrow it, and `object` keeps to the assets a kind of object is seen in
([Objects](#objects));
([Objects](#objects)); `from` and `to` (days, both included: when the asset was recorded, or added when its file
doesn't say), `tag`, `origin` (`upload`, `paste`, `iiif`, `folder`, `file` or `source:<id>`, as `GET
/recordings/origins` lists them) and `language` (a code, or `none` for not known) keep to the assets that have them,
exactly as the Library's filters of the same names do, on every kind of hit (said, on screen, in files …); repeat
`tag`, `origin` or `language` for several (any of them). None of them is set unless asked for, so a search without
them is as it was;
`limit`/`offset` page through it. `total` counts the moments ranked so far (`capped` when there may be more).
With `facets=true` it also counts all the matching moments, whatever the page, by namespace, person, emotion, sound
(`events`; plain speech isn't one) and asset (`facets`: up to 50 values each, most first, and `moments`), and lists the kinds of object seen in the
Expand All @@ -1360,7 +1365,11 @@ assets they're in, with how many of those assets each is in (`objects`); past 20

In the web app, a result links to the asset with the words searched for (`/resources/<id>?t=<seconds>&q=<words>`,
without filters, left-out words or `OR`): it opens with Find in its text showing them, on the match at that moment,
or on the longest of the words when the text doesn't have them together.
or on the longest of the words when the text doesn't have them together. The search page's filter panel starts with
Assets: From and To days, and a tag, a source and a language to pick (a list is left out when there's nothing to
choose between). Each is empty until picked, one pick searches again, and each can be typed in the box too
(`from:2026-01-01`, `to:2026-03-31`, `tag:interview`, `origin:upload` or a source's name, `language:de` or
`language:German`). They're kept in the address, so a link carries them, and in a saved search.

Each hit's `source` says where it was found: `said` (a line of the asset's text or transcript), `screen` (text on screen), `page` (a document's or an
image's text: its `page`, from 0, and its `box` on it), `object` (a kind of object, where it's first seen: its time, or
Expand Down Expand Up @@ -1942,7 +1951,9 @@ DELETE /api/v1/searches/{sid}
```

Saved searches: `POST` with a `name` (unique among yours), the search's `q`, and any of `namespace`, `speaker`,
`emotion`, `event` and `recording`, as `GET /search` takes them. They follow the rules of saved views (above): yours, or
`emotion`, `event`, `recording`, `object`, `date_from` and `date_to` (`YYYY-MM-DD`), `tag`, `origin` and `language`,
as `GET /search` takes them (one value each; `date_from` and `date_to` are its `from` and `to`). They follow the rules
of saved views (above): yours, or
`shared` with the namespace they search (editors there; a search of every namespace can't be shared); only their maker
renames or shares them (`PATCH` with `name` or `shared`); their maker, or an owner of the namespace for a shared one,
deletes them. `GET` lists yours, then the shared ones, with `speaker_name` and `recording_title` where you can read
Expand Down
2 changes: 1 addition & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| Tool | |
|---|---|
| `lens_help` | Lens itself rather than the archive: the sections of Lens's docs that match a question (or one page read whole), each with its link, and this server's own facts: version, addresses (web app, MCP, API), what is turned on and your access ([Questions about Lens itself](assistant.md#questions-about-lens-itself)) |
| `search` | moments where words appear (written, spoken or shown on screen), best first: the asset, person, time, the matching text (matches in **bold**) and a link to that moment. Words, `"phrases"`, `OR`; by namespace, person, asset or sound heard (laughter, applause, music …) |
| `search` | moments where words appear (written, spoken or shown on screen), best first: the asset, person, time, the matching text (matches in **bold**) and a link to that moment. Words, `"phrases"`, `OR`; by namespace, person, asset or sound heard (laughter, applause, music …), and by when the assets were recorded (a day, month or year), their tags, where they came from and their language |
| `find_similar` | the assets (recordings, documents and images) most like a text, an asset, a topic or an entity: why each is alike (meaning, key words, topics, entities in common) and the moments that show it, with links |
| `find` | assets by any attribute: faces seen, people heard, entities, topics, kind, content type, dates (a day, month or year), tags, language, objects, emotions, length, collection and words in them; ids or names. Each says why it matched, with moments and links to cite ([Finding by any attribute](processing.md#finding-by-any-attribute)) |
| `classify` | which of some labels fits an asset or a text, how sure, and every label ranked: labels you give, or Lens's content types or the namespace's topics; the decision model answers, else the language model. Read only: it changes nothing ([Classifying](processing.md#classifying)) |
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,7 @@ request, to another check that proves the same thing; loosening a requirement or
| A decision model (Jev or Laya) types the names the rules leave as Terms, in one call, keeping only types it is sure of; Settings › Analysis can leave them Terms | [test_name_types.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_name_types.py) `test_the_decision_model_types_what_the_rules_cant`, `test_a_type_the_model_isnt_sure_of_stays_a_term`, `test_by_default_the_language_model_isnt_asked`, `test_with_typing_rules_nothing_is_asked` |
| What is most like a passage, an asset, a topic or an entity | [test_similar.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_similar.py) `test_like_a_text_by_its_key_words`, `test_like_a_resource_by_words_topics_and_entities`, `test_like_a_topic_and_an_entity`; [similar.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/similar.test.tsx) |
| Search finds and filters moments by the sound heard in them (laughter, applause, music), typed any way, and counts each | [test_search.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_search.py) `test_sound_events_filter_and_count`; [test_semantic.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_semantic.py) `test_search_by_meaning_keeps_to_a_sound`; [search.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/search.test.ts) `reads sound events typed any way` |
| Search keeps to the assets recorded between two days, with a tag, from a source or in a language, by the API, MCP and the search page (empty until picked, kept in the link and in saved searches) | [test_search.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_search.py) `test_dates_tags_origins_and_languages_narrow_search`; [search-asset-filters.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/search-asset-filters.test.tsx) `starts with none set, so the search is as it was, and picking one puts it in the address`, `asks with the ones in the address, shows each as a chip, and says which to clear when nothing matches`; [search.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/search.test.ts) `reads dates, tags, sources and languages, and keeps them in the address` |
| With anytopdf, photos found by what they show | [test_photos.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photos.py) `test_photos_found_by_what_they_show`, `test_off_by_default` |
| Pages and shots described by the provider's vision model or, off by default, by Florence-2 on this server, fetched pinned and checked when picked | [test_florence.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_florence.py) `test_florence_describes_pictures_without_a_model_provider`, `test_florence_is_fetched_pinned_and_checked`, `test_admins_and_agents_choose_what_describes_pictures`, `test_a_picture_is_read_as_the_model_was_trained` |
| Or by Moondream on this server, off by default, fetched at a pinned revision only where the chosen model can run, and saying why where it can't | [test_moondream.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_moondream.py) `test_moondream_describes_pictures_without_a_model_provider`, `test_where_each_moondream_can_run`, `test_on_a_cpu_moondream_3_is_never_fetched_and_the_step_says_why`, `test_moondream_is_fetched_pinned`, `test_admins_and_agents_choose_moondream` |
Expand Down
23 changes: 22 additions & 1 deletion fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -464,12 +464,23 @@ def _recording(ctx: Context, rid: int) -> dict[str, Any]:
"pipelines' model steps wrote about assets (found: generated, with the step in output). Words are all required "
'(matched after stemming), "quoted phrases" must appear as written, OR separates alternatives, NOT or a leading - leaves out lines with a word or phrase. Results are the best '
"moments first, each with the asset, the person (who said or wrote it), where in it, the matching text (matches in **bold**) "
"and a url that opens the asset at that moment. Use get_transcript for the lines around a moment.",
"and a url that opens the asset at that moment. A date range, tags, origins or languages keep to the assets that have "
"them. Use get_transcript for the lines around a moment.",
Arg("query", "string", 'what to look for: words, "a phrase", or alternatives separated by OR', required=True, max_length=500),
Arg("namespace", "string", "only this namespace (see list_namespaces)"),
Arg("speaker_id", "integer", "only lines from this person (ids from list_speakers or get_recording)"),
Arg("recording_id", "integer", "only within this asset"),
Arg("event", "string", "only lines with this sound: laughter, applause, music, cough …", max_length=40),
Arg("date_from", "string", "only assets recorded (or added) on or after this day, month or year (2026-09-30, 2026-09, 2026)"),
Arg("date_to", "string", "only assets recorded (or added) on or before this day, month or year"),
Arg("tags", "string[]", "only assets with any of these tags (ignoring case)"),
Arg(
"origins",
"string[]",
"only assets that came from here: source:<id> (a connected source), upload, paste, iiif, folder (the archive's own "
"folders) or file",
),
Arg("languages", "string[]", "only assets in these languages (en, de …); none for those whose language isn't known"),
Arg("limit", "integer", "how many moments", default=10, minimum=1, maximum=50),
Arg("offset", "integer", "skip this many (for the next page)", default=0, minimum=0, maximum=10000),
)
Expand All @@ -480,6 +491,11 @@ def search(
speaker_id: int | None,
recording_id: int | None,
event: str | None,
date_from: str | None,
date_to: str | None,
tags: list[str] | None,
origins: list[str] | None,
languages: list[str] | None,
limit: int,
offset: int,
) -> dict[str, Any]:
Expand All @@ -497,6 +513,11 @@ def search(
event=event,
recording=recording_id,
object=None,
date_from=findmod.day(date_from, name="date_from"),
date_to=findmod.day(date_to, end=True, name="date_to"),
tag=tags or None,
origin=origins or None,
language=languages or None,
limit=limit,
offset=offset,
facets=False,
Expand Down
39 changes: 36 additions & 3 deletions fastapi_backend/app/api/v1/routes/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,15 @@

from __future__ import annotations

from datetime import date
from typing import Literal

from fastapi import APIRouter, HTTPException, Query, Request

from app.api.deps import Acl, Cfg, CurrentUser, Db
from app.api.deps import Acl, Cfg, CurrentUser, Db, domain_errors
from app.api.media import sign_urls
from app.domain import graph as graphmod
from app.domain import graph_history, hierarchy, nsfw, photos, render, similar, store
from app.domain import graph_history, hierarchy, library, nsfw, photos, render, similar, store
from app.domain import search as searchmod
from app.schemas.search import Graph, Mention, PhotoResults, SearchResults, SimilarResults, TermSuggestion

Expand All @@ -31,6 +32,19 @@ def search_transcripts(
event: str | None = Query(None, max_length=40, description="only lines with this sound (laughter, applause, music …), ignoring case"),
recording: int | None = None,
object: str | None = Query(None, max_length=60, description="only assets this kind of object is seen in (person, car …)"),
date_from: date | None = Query(
None, alias="from", description="only assets recorded on or after this day (added, when the file doesn't say when)"
),
date_to: date | None = Query(None, alias="to", description="only assets recorded on or before this day"),
tag: list[str] | None = Query(None, description="only assets with these tags (ignoring case); repeat for several (any of them)"),
origin: list[str] | None = Query(
None,
description="only assets that came from here: source:<id> (a connected source), upload, paste, iiif, folder (the "
"archive's own folders) or file (GET /recordings/origins); repeat for several",
),
language: list[str] | None = Query(
None, description="only assets in these languages (codes, ignoring case; none: not known); repeat for several"
),
limit: int = Query(50, ge=1, le=200),
offset: int = Query(0, ge=0),
facets: bool = Query(
Expand All @@ -48,10 +62,28 @@ def search_transcripts(
a resource's supplementary transcripts, captions, translations and indexes, the kinds of object seen in videos,
documents and images, what a model that can see said they show, or what a pipeline's model step wrote about them),
best first, in the namespaces you can read and the collections you were given a role on. A person, emotion or
sound filter keeps to their lines. By meaning, a moment is a passage about what was asked, though its words may differ."""
sound filter keeps to their lines. A date, tag, origin or language keeps to the assets that have it, as the Library
filters them. By meaning, a moment is a passage about what was asked, though its words may differ."""
if ns:
acl.scope(ns) # 404 unless they see some of it
also = acl.partial_recordings()
kept = None
if date_from or date_to or tag or origin or language:
spaces, within = acl.scope(ns)
with domain_errors():
cond, p = library.where(
db,
spaces,
within=within,
cfg=cfg,
sensitive="include", # search blurs what's flagged rather than leaving it out
date_from=date_from,
date_to=date_to,
tags=tag,
origins=origin,
languages=language,
)
kept = set(db.values(f"SELECT VALUE record::id(id) FROM recording WHERE {cond}", **p))
res = searchmod.search(
db,
q,
Expand All @@ -71,6 +103,7 @@ def search_transcripts(
cfg=cfg,
mode=mode,
event=event,
recordings=kept,
)
blur = nsfw.blurred(db, cfg, [h.get("recording_id") for h in res.get("hits") or []])
for h in res.get("hits") or []:
Expand Down
8 changes: 4 additions & 4 deletions fastapi_backend/app/api/v1/routes/searches.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from app.domain import auth, store, views
from app.domain.store import DB
from app.schemas.common import Ok
from app.schemas.views import SavedSearch, SearchCreate, SearchUpdate
from app.schemas.views import SavedSearch, SearchCreate, SearchState, SearchUpdate

router = APIRouter(prefix="/searches", tags=["searches"])

Expand Down Expand Up @@ -67,14 +67,14 @@ def list_searches(user: CurrentUser, acl: Acl, db: Db) -> list[SavedSearch]:

@router.post("")
def create_search(body: SearchCreate, user: Writer, acl: Acl, db: Db) -> SavedSearch:
"""Save a search (its words, namespace, person, emotion, sound, asset and object) under a name unique among yours. Sharing it
with its namespace needs editor access there."""
"""Save a search (its words, namespace, person, emotion, sound, asset, object, dates, tag, origin and language) under a
name unique among yours. Sharing it with its namespace needs editor access there."""
sid = acl.namespace(body.namespace) if body.namespace else None
if body.recording is not None:
acl.recording(body.recording)
if body.shared:
check_share(acl, sid, "saved search")
state = body.model_dump(include={"q", "speaker", "emotion", "event", "recording", "object"})
state = body.model_dump(include=set(SearchState.model_fields))
with domain_errors():
vid = views.create(db, user.id, body.name, sid, state, body.shared, kind="search")
if body.shared:
Expand Down
Loading
Loading
0