8000 feat(find): find assets by face, voice, entity, topic, type, content type and date by adeelahmad · Pull Request #398 · adeelahmad/lense · GitHub
[go: up one dir, main page]
Skip to content

feat(find): find assets by face, voice, entity, topic, type, content type and date - #398

Merged
adeelahmad merged 6 commits into
mainfrom
claude/project-thread-26l27n
Oct 17, 2026
Merged

feat(find): find assets by face, voice, entity, topic, type, content type and date#398
adeelahmad merged 6 commits into
mainfrom
claude/project-thread-26l27n

Conversation

@adeelahmad
@adeelahmad adeelahmad commented Oct 15, 2026
Copy link
Copy Markdown
Owner

Requested by Adeel · project thread

Description

Before: an agent could search words (search), list recordings by a few attributes (list_recordings: namespace, title, dates, speaker, tag, kind), or ask for similar content. It couldn't ask "videos with Adeel's face from 2025", "podcasts that mention Dyno Therapeutics", or "what is about Health", and it had to look up ids first.

After: one core find service, used by the API (POST /api/v1/find), the MCP server (find) and the assistant (find_resources). It finds resources by:

  • faces seen in them and speakers (voices) heard;
  • entities mentioned and topics they're about (a topic brings its narrower topics);
  • types (audio, video, transcript, document, image) and content types (podcast, interview, scan, photo, email …);
  • dates: a day, a month or a year (2025 is the whole year); and length;
  • tags, language, objects seen, emotions heard, status, collection, origin, sensitive;
  • and text: words said or written, matched as search matches them (by meaning too when that's set up).

Faces, speakers, entities and topics take ids or names (ignoring case; an entity's other names and any topic label count). resolved says what each name matched. A name that matches nothing returns an error naming the closest ones, so an agent can correct itself.

Each result says why it matched (why), with up to three moments per reason to cite: where the words are, when the face is on screen (or which pages), the speaker's first lines, the lines that mention the entity, and the topic, content type, object or emotion that matched. MCP moments carry links that open at that moment. Assistant moments are numbered [n], so answers cite them.

The Library list (GET /resources) also takes face, entity, topic, content_type and emotion ids, ready for the web app's filter bar.

Motivation and Context

Adeel asked for core services that agents and workflows can use: classify, plus search by face, voice, entity, type, content type, date and other attributes. This PR is the search half. Classify (decision model) follows in its own PR.

How: domain/find.py resolves names inside the asker's namespaces, then builds on the existing library.where filters, which gained faces, entities, topics, content types, emotions, a recording-id set and a list of kinds. It reuses search.search for text, ranks by the best matching moment, and builds why from face_track, segment, mentions and topic_about. Content types are worked out the way content_types.of_recording does it (keys_of), so nothing new is stored.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change; ! in the title)

Checklist:

  • The title is a Conventional Commit.
  • My code follows the code style of this project (make hooks checks it on commit).
  • I added or updated tests.
  • My change requires documentation updates, and I have updated docs/ accordingly.
  • Users need more than the title to understand it, so I wrote it under ## Unreleased in CHANGELOG.md.

Done questions (AGENTS.md)

  • Goal: delivers find by face, voice, entity, topic, type, content type, date and other attributes; merged on green CI.
  • Agents: POST /find, MCP find, assistant find_resources. Assistant tool graphs reach it through call_tool.
  • Zero setup: works on any install; faces need face detection on, as before.
  • Fewest clicks and keystrokes: names instead of ids, so no lookups between calls.
  • Configurable: n/a, no new settings.
  • Light: no new dependency. Content-type and emotion filters run in Python over the asker's recordings.
  • Private and secure: reads only what the asker can read (acl.scope); name hints come only from their namespaces. No new attack surface, so the threat model is unchanged.
  • Cost: no model call, except text by meaning when it's set up (one embedding, already in the activity ledger).
  • User decides: read only; defaults unchanged.
  • Upgrades: n/a, nothing new is stored.
  • Honest: every result cites moments; docs label the planned parts.
  • Tests, docs and generated files updated: tests/api/test_find.py, docs/processing.md, docs/api.md, docs/mcp.md, docs/assistant.md, make openapi.
  • Open standards; dependencies keep Lens MIT (none added).
  • Result reported with evidence. Refine later: these filters in the Library's filter bar and saved collections; finding by what a photo shows inside the same filter; ranking by more than the best matching moment.

Local checks: ruff check, ruff format --check, mypy, full backend suite (1089 passed, 20 skipped), mkdocs build --strict.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GKv2oVRvFRdiSf4USLsM1N

Generated by Claude Code

@claude
feat(find): find resources by face, voice, entity, topic, type, conte…
f00cf77 
…nt type and date

One filter for the API (POST /find), the MCP server (find) and the
assistant (find_resources): who is seen in a resource (faces) and heard
(speakers), the entities it mentions, the topics it is about (narrower
topics included), its kind and content type, when it was made (a day, a
month or a year), tags, language, objects seen, emotions heard, length,
status, collection, origin and words said in it. Faces, speakers,
entities and topics take ids or names; `resolved` says what each name
matched, and a name that matches nothing is an error naming the closest.
Each result says why it matched, with up to three moments per reason to
cite. The Library list (GET /resources) takes face, entity, topic,
content_type and emotion ids as filters too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKv2oVRvFRdiSf4USLsM1N
@adeelahmad adeelahmad self-assigned this Oct 15, 2026
@adeelahmad adeelahmad mentioned this pull request Oct 15, 2026
Merged
16 of 22 tasks
adeelahmad added a commit that referenced this pull request Oct 15, 2026
@adeelahmad @claude
docs: list analytics, per-namespace MCP and chat completions, entity …
5e55fa4 
…pages and notes as built (#424)

Move usage analytics (#274) and assets and people (#273) out of the
README's In progress list, and add what merged since #383: a namespace's
own MCP server, chat completions and one-namespace API keys; editing and
merging entities and topics with undo, from agents too; entity pages at
wiki addresses; Connect with Google and rclone sign-in on the server;
the notes tree, today's note, pins, search and Recently deleted; backups
that keep Lens's files; and grouped, searchable settings. In progress now
lists the open feature PRs (#398, #400, #402, #408, #419), and the docs
index links usage analytics.


Claude-Session: https://claude.ai/code/session_01HDx7PPUauyjMn1LZs8JAsS

Co-authored-by: Adeel Ahmad <noreply@anthropic.com>
claude added 2 commits October 17, 2026 02:39
@claude
Merge remote-tracking branch 'origin/main' into claude/project-thread…
5364590 
…-26l27n

# Conflicts:
#	CHANGELOG.md
#	docs/api.md
#	docs/assistant.md
#	docs/mcp.md
#	fastapi_backend/app/api/v1/routes/recordings.py
#	fastapi_backend/app/domain/ai_tools.py
#	nextjs-frontend/app/openapi-client/index.ts
@claude
docs(changelog): find is unreleased
e6aa9c6 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKv2oVRvFRdiSf4USLsM1N
@adeelahmad adeelahmad changed the title feat(find): find resources by face, voice, entity, topic, type, content type and date feat(find): find assets by face, voice, entity, topic, type, content type and date Oct 17, 2026
@adeelahmad
adeelahmad merged commit 0b80d8a into main Oct 17, 2026
10 of 11 checks passed
@adeelahmad
adeelahmad deleted the claude/project-thread-26l27n branch October 17, 2026 03:20
This was referenced Oct 17, 2026
Merged
Merged
adeelahmad pushed a commit that referenced this pull request Oct 17, 2026
@claude
docs(requirements): list finding and classifying as built, with their…
a0c72c6 
… checks

The README's Connect your agents and Automate it items now name find,
classify and the workflow Classify node, docs/requirements.md backs each
with its tests, and the merged #398 leaves In progress.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKv2oVRvFRdiSf4USLsM1N
@adeelahmad adeelahmad mentioned this pull request Oct 17, 2026
Merged
21 of 23 tasks
adeelahmad added a commit that referenced this pull request Oct 17, 2026
@adeelahmad @claude
feat(workflows): a classify node on the canvas (#498)
c442e9e 
* feat(workflows): a classify node on the canvas

A Classify node picks one of your labels, a content type or a topic for the
recording (or for what came in) and passes on the choice, how sure and every
label ranked; set to act, a sure answer sets the content type, suggests the
topic or adds the tag, so the node keeps something by itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKv2oVRvFRdiSf4USLsM1N

* docs(requirements): list finding and classifying as built, with their checks

The README's Connect your agents and Automate it items now name find,
classify and the workflow Classify node, docs/requirements.md backs each
with its tests, and the merged #398 leaves In progress.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKv2oVRvFRdiSf4USLsM1N

---------

Co-authored-by: Adeel Ahmad <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@adeelahmad adeelahmad

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@adeelahmad @claude
0