Repository navigation
feat(search): find photos by what they show, with anytopdf's CLIP plugin - #230
Merged
Merged
Conversation
Settings › Search › Search photos by what they show (embeddings.photos, off by default). Lens keeps anytopdf's CLIP plugin beside the program and fetches its model (about 600 MB) as a component for the embed step, which then gives each image resource a CLIP vector, once per file, offline. GET /api/v1/search/photos compares a query with the photos the person may see, and the search page shows the best matches above the moments. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018GMFApXZSk7XERM3J5gNTy
The node answered without reading the request, so on some runners the connection was reset before Lens saw the answer was too long. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018GMFApXZSk7XERM3J5gNTy
…-kgtpxt # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: search found photos only by words on them (OCR) or detected objects.
After: with Settings › Search › Search photos by what they show on (
embeddings.photos, off by default), a search like "a bus on a city street" also shows the photos most like it, in a strip above the moments, limited to the namespaces and collections the person can see.GET /api/v1/search/photosserves it.Motivation and Context
Last part of Phase 3 of the anytopdf integration (CLIP image search). Follows #229.
How:
anytopdf.PLUGINSnow keepsanytopdf-plugin-clip, and aplugins/.askedmarker refetches once when Lens keeps more plugins. Aclipcomponent fetches the CLIP ViT-B/32 model (about 600 MB, intodata_dir/models/clip) for theembedstep only when the switch is on. Theembedstep gives each image resource one CLIP vector (photo_vector, once per file, offline, network blocked), kept apart from search by meaning's passages. It is removed with the recording. Queries use brute-force cosine. The floor is 0.24 with a 0.05 spread: measured with the real model, a plain blue picture scores 0.21 to 0.24 against most queries, while a bus photo scored 0.30 for "a bus on a city street".Types of changes
Checklist:
tests/api/test_photos.pywith a fake plugin; a real-model test underLENS_TEST_ANYTOPDF+LENS_TEST_CLIP_DATA, passed locally).docs/accordingly.## UnreleasedinCHANGELOG.md.🤖 Generated with Claude Code
https://claude.ai/code/session_018GMFApXZSk7XERM3J5gNTy
Generated by Claude Code