8000 feat(library): show photos and videos on a map by where they were taken by adeelahmad · Pull Request #697 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,18 +144,20 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
Google accounts connect with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google
Drive can sign in through rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs,
Office files, photos, JSON and zip or tar archives (photos and videos dated by when they were taken, from the file
itself, and photos placed by their GPS), and transcripts made elsewhere (SRT, WebVTT, text, Word, PDF), each line
credited to the speaker the file names; and, with an Apache Tika server (an opt-in extra), documents nothing else
here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking
the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off
by default), taking a video's own captions as its transcript when people wrote some (off by default too). Say what a
file or a page is as it comes in (a podcast, an interview, a scan), in Import, by API or to the assistant, and its
content type's pipeline runs from the start. A file, page or video the namespace already has links to the one there
instead of being imported again. Media browsers can't play, such as an AVI or HEVC from a phone, can get a copy they
can (off by default). A bulk import (a folder scan, an archive's files, a link list, a source's files) is one run:
the toast after it, and the Import screen's list of recent runs, undo it whole.
itself, and photos placed by their GPS on the Library's map, by API and MCP too), and transcripts made elsewhere
(SRT, WebVTT, text, Word, PDF), each line credited to the speaker the file names; and, with an Apache Tika server (an
opt-in extra), documents nothing else here can read; capture web pages one at a time, from a link list or a
bookmarks file, by API or by asking the assistant, from private networks you allow in Settings too; and import
videos and podcasts by their address (off by default), taking a video's own captions as its transcript when people
wrote some (off by default too). Say what a file or a page is as it comes in (a podcast, an interview, a scan), in
Import, by API or to the assistant, and its content type's pipeline runs from the start. A file, page or video the
namespace already has links to the one there instead of being imported again. Media browsers can't play, such as an
AVI or HEVC from a phone, can get a copy they can (off by default). A bulk import (a folder scan, an archive's
files, a link list, a source's files) is one run: the toast after it, and the Import screen's list of recent runs,
undo it whole.
([Sensors](docs/sensors.md), [processing](docs/processing.md), [Apache Tika](docs/configuration.md#apache-tika),
[import webhooks](docs/api.md#import-webhooks), [playable copies](docs/video.md#playable-copies),
[where an asset was taken](docs/processing.md#where-an-asset-was-taken),
[import runs](docs/api.md#import-runs);
[checks](docs/requirements.md#bring-in-everything))
- **Clip from the browser.** The Lens web clipper for Chrome, Edge, Firefox and Safari turns a page, a selection or a
Expand Down
9 changes: 9 additions & 0 deletions changelog.d/photo-map.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
- **Photos and videos on a map.** Library → **Map** (beside Table and List, on a phone too) shows each asset whose own
file said where it was taken as a dot, with the Library's filters, framed to show them all; a dot links to its
asset. Besides a photo's EXIF GPS fix, a video's location is now read as it arrives (the ISO 6709 tag iPhones and
Android phones write). Agents get the same through `GET /api/v1/resources?located=true&bbox=west,south,east,north`,
which also gives every row its `place`, and the MCP tool `list_recordings` (`located`, `bbox`); only members see
where an asset was taken. The tiles under the map come from Settings → **Maps** (OpenStreetMap's by default); each
viewer's browser fetches them, so the tile server sees the area looked at: point it at your own tile server, or leave
it empty for none ([Where an asset was taken](docs/processing.md#where-an-asset-was-taken),
[Maps](docs/configuration.md#maps)). Assets that arrived before keep what they had: nothing needs upgrading.
7 changes: 6 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -612,6 +612,7 @@ GET /api/v1/resources/origins
GET /api/v1/resources/languages
GET /api/v1/resources/objects
GET /api/v1/resources/nsfw
GET /api/v1/resources/map-tiles
POST /api/v1/resources/tags
POST /api/v1/resources/collection
GET /api/v1/resources/{rid}
Expand Down Expand Up @@ -680,13 +681,17 @@ repeat a parameter that takes several values (`?status=new&status=error`) to mat
| `emotion` | emotions heard in them, ignoring case |
| `edited_by` | `me`: assets you edited (corrected a line of the text, changed the catalogue record, or renamed) |
| `sensitive` | where [NSFW classification](nsfw.md) is on: `only` the ones flagged NSFW, `exclude` them, or `include` them. Without it, flagged ones are left out where their namespace hides them (`display: hide`) |
| `located` | `true`: only assets known to have been taken somewhere (a `place`, [Where an asset was taken](processing.md#where-an-asset-was-taken)) |
| `bbox` | `west,south,east,north` in degrees (GeoJSON's order): only assets taken inside that box; west past east crosses the antimeridian |
| `sort` | `date`, `title`, `duration`, `speakers`, `status` or `importance`; `-` in front for descending (default `-date`). Assets without the value come last either way |
| `limit`, `offset` | one page (default 500 rows, at most 1000) |

The body is the page's rows; the `X-Total-Count` header says how many assets match on all pages.

Rows also say where each asset came from (`origin`, and `origin_name`: the source's name, or e.g. "Uploaded"),
its `language` when known and the collection it lives in (`collection`, `collection_name`). `GET /resources/{rid}`
its `language` when known, the collection it lives in (`collection`, `collection_name`) and where it was taken
(`place`: `{lat, lon}`, or null). `GET /resources/map-tiles` (anyone signed in) gives the tiles the web app's map draws
on: `url`, an XYZ address with `{z}`, `{x}` and `{y}` (null for none), and its `attribution` (Settings → Maps). `GET /resources/{rid}`
adds `collection_path`: the collections from the top of the namespace down to its own, `[{id, name}]`.
`POST /resources/collection {recordings, collection}` moves assets into a collection of their namespace (editors
of each); an asset of another namespace is a 400 (move it to that namespace first). It answers how many `moved`;
Expand Down
19 changes: 18 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ Start from `fastapi_backend/archive.example.yaml`, which documents every key. Th

Admins can change, in Settings: transcription, speaker separation, speech providers, voice IDs, analysis, the LLM
provider and key, a local model, the AI assistant, the [code sandbox](assistant.md#code-sandbox), search (word
search and search by meaning), reports and graph,
search and search by meaning), reports and graph, [maps](#maps),
video, OCR, faces and objects, [NSFW classification](nsfw.md), [enhancing
images](enhance.md), workers, sign-in, components (and the edition, full or
lean), access and embedding (whether other devices on the network reach Lens, embed frame ancestors, text and transcript
Expand Down Expand Up @@ -290,6 +290,23 @@ resource are described. Each call is in the [activity ledger](activity.md) (`mod

Agents read and change these with the MCP tools `picture_descriptions` and `set_picture_descriptions` (admins).

## Maps

The Library's Map shows where photos and videos were taken
([Where an asset was taken](processing.md#where-an-asset-was-taken)) on tiles that each viewer's browser fetches
itself. Settings → Maps:

| Setting | Default | |
|---|---|---|
| `maps.tile_url` | `https://tile.openstreetmap.org/{z}/{x}/{y}.png` | where the tiles come from: an http(s) address with `{z}`, `{x}` and `{y}` (and `{s}` for a subdomain, if the server has them), with no password in it. Empty draws no tiles, only the dots |
| `maps.attribution` | `© OpenStreetMap contributors` | the credit the tiles' terms ask to be shown on the map, one line |

The tile server sees which tiles are asked for, so the area each person looks at (around where their photos were
taken) and the address they look from, as any web map's does; nothing else about the assets goes to it. To keep that
on your own hardware, run a tile server of your own and give its address, or leave the address empty. OpenStreetMap's
tiles are for light use under its [tile usage policy](https://operations.osmfoundation.org/policies/tiles/); a busy
install should use its own or a provider's.

## Uploads

Audio, video, documents (PDF, Office and OpenDocument files, text, Markdown, saved web pages and emails) and images
Expand Down
4 changes: 2 additions & 2 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,11 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `classify` | which of some labels fits an asset or a text, how sure, and every label ranked: labels you give, or Lens's content types or the namespace's topics; the decision model answers, else the language model. Read only: it changes nothing ([Classifying](processing.md#classifying)) |
| `list_namespaces` | the namespaces you can read, with their assets, hours, people and your role |
| `budgets` | where spending stands: each namespace you can read this period against its budget, and for admins all of Lens and every other budget, each ok, near or over ([Asking the assistant or an agent](budgets.md#asking-the-assistant-or-an-agent)) |
| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, and whether they're flagged NSFW (`sensitive`) |
| `list_recordings` | assets, newest first, by namespace, title words, dates, person, tag or kind, whether they're flagged NSFW (`sensitive`), and where they were taken: each with its `place` (`lat`, `lon`) when its file had a GPS fix, `located` for only those and `bbox` (`west,south,east,north`) for those inside a box ([Where an asset was taken](processing.md#where-an-asset-was-taken)) |
| `list_speakers` | the people of your namespaces, the ones with the most talk time first |
| `list_unnamed_faces` | the faces nobody has named yet in your namespaces that recognise faces, the largest group first: a name to offer, how often each is seen and the assets it's in, with links ([Naming faces](video.md#naming-faces)) |
| `name_face` | say who a face is: a name another face of its namespace has joins that person, any other becomes its own. Needs a write-scope token and editor access |
| `get_recording` | one asset: when, how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW |
| `get_recording` | one asset: when, where it was taken (`place`), how long, people, summary and key points, chapters, the entities and keywords most mentioned, its link, and whether it's flagged NSFW |
| `get_transcript` | an asset's lines in order (who, time, text, link, and the words transcription wasn't sure of as `unsure`), a page at a time, from a line or a time window |
| `fetch` | an asset's whole text at once, as `[m:ss] Speaker: …` lines |
| `cite` | a citation for a moment (a line or a second, and a few lines from there): the words, who said or wrote them, the asset and date, the time, the link, and all of it as Markdown |
Expand Down
29 changes: 26 additions & 3 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,32 @@ to an import webhook), from the first of these that says it:
Nothing is guessed: a clock that was never set (a year before 1971, such as QuickTime's 1904) or a date in the future
counts as no date. A date an imported transcript gives wins over all three, as an email's or a document's own date
does once it is read, and a file uploaded again into the namespace that has it keeps its asset's date. A file on a
[storage source](#storage-sources) is dated by the source (the date it lists, else when the file last changed there). A photo's GPS fix from its EXIF is kept as the asset's `place` (`{lat, lon}`, in degrees);
`GET /api/v1/recordings/{rid}` and the MCP tool `get_recording` give it to the namespace's members only, never to
visitors. It is read on this machine and nothing is sent anywhere. Assets that arrived before keep their dates.
[storage source](#storage-sources) is dated by the source (the date it lists, else when the file last changed there).
Assets that arrived before keep their dates.

### Where an asset was taken

As it arrives, an asset's own file can say where it was taken, and that is kept as its `place` (`{lat, lon}`, WGS 84
degrees to six places): a photo's EXIF GPS fix (`GPSLatitude`, `GPSLongitude` and their `Ref`s), or a video's location
as ffprobe reads it, in the ISO 6709 form phones write (`+48.8582+002.2945/`: an iPhone's
`com.apple.quicktime.location.ISO6709`, an Android phone's `location`). A fix of 0, 0 is a device with no fix, and
counts as none. It is read on this machine with Pillow and ffprobe, which Lens has anyway, and nothing is sent
anywhere. A photo that arrived before this was kept, but that [anytopdf](configuration.md#anytopdf) read with its GPS
fix, is placed by that fix; other assets that arrived before have no place, and nothing about them changes.

Only the namespace's members see where an asset was taken, never visitors (public pages, embeds and share links):

- **The web app**: Library → **Map** (beside Table and List; on a phone, beside List) puts each asset with a place on a
map as a dot, with the Library's filters and tab, framed to show them all; a dot's popup links to the asset. It
draws the newest 1,000 that match and says when there are more. The tiles under the map are fetched by the browser
from Settings → Maps (`maps.tile_url`, OpenStreetMap's by default), so the tile server sees the area being looked at
and the address it is looked at from: point it at a tile server of your own, or leave it empty to draw only the
dots ([Configuration](configuration.md#maps)).
- **The API**: `GET /api/v1/resources?located=true` lists only assets with a place, and
`bbox=west,south,east,north` (degrees, in GeoJSON's order; west past east crosses the antimeridian) those taken
inside a box. Both combine with the list's other filters, and each row carries its `place`; so does
`GET /api/v1/resources/{rid}`. `GET /api/v1/resources/map-tiles` gives the tile address and its credit.
- **Agents**: the MCP tools `list_recordings` (with `located` and `bbox`) and `get_recording` give `place` too.

### Cleaning up transcripts

Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Transcripts made elsewhere (text, Markdown, Word, PDF, SRT, WebVTT, JSON) imported with each line credited to the speaker the file names, WebVTT voice spans (`<v Name>`, `<v.loud Name>`) included | [test_importing.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_importing.py) `test_formats_and_speakers_by_name`, `test_webvtt_voice_spans_name_each_line` |
| Media browsers can't play (an AVI, HEVC, AMR) gets a copy they can, for the player only, off by default; files they play get none | [test_playable.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_playable.py) `test_a_camera_clip_gets_a_copy_every_browser_plays`, `test_a_file_browsers_play_gets_no_copy` |
| Photos and videos dated by when they were taken, from the file itself (a photo's EXIF date and GPS fix as `place`, a video's creation time) when no name or source dates them; a date already set is kept and none is invented | [test_file_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_file_dates.py) `test_a_photo_is_dated_and_placed_by_its_exif`, `test_a_date_already_set_is_kept`, `test_a_video_is_dated_by_its_creation_time`, `test_a_scanned_video_is_dated_by_its_creation_time` |
| Assets placed by their file's GPS fix (a photo's EXIF, a video's ISO 6709 location) on the Library's map, listed by API (`located`, `bbox`) and MCP to members only, on tiles from a setting | [test_photo_map.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_photo_map.py) `test_located_assets_are_listed_inside_a_box`, `test_an_agent_finds_what_was_taken_in_a_place`, `test_a_video_is_placed_by_its_location`, `test_the_map_tiles_are_a_setting`; [library-map.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/library-map.test.tsx) `placed assets are dots on the tiles from Settings, framed to show them all` |
| An uploaded video, end to end | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| Apache Tika (opt-in) reads documents nothing else here can | [test_tika.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_tika.py) `test_off_by_default_nothing_changes`, `test_what_nothing_here_can_convert_tika_reads_as_text` |
| Web pages captured one at a time, from a link list or a bookmarks file, by API or by asking; private networks only when allowed | [test_webcapture.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webcapture.py) `test_a_web_page_becomes_a_document`, `test_an_admin_allows_a_private_network_in_settings`, `test_private_addresses_are_refused`; [test_web_links.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_web_links.py) `test_one_or_several_addresses_as_a_list`, `test_a_bookmarks_file_becomes_pages_to_capture`; [test_setup_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_setup_assistant.py) `test_the_assistant_captures_web_pages` |
Expand Down
2 changes: 2 additions & 0 deletions docs/technology-selection.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
- [shadcn/ui](https://ui.shadcn.com/) and [Tailwind CSS](https://tailwindcss.com/)
- [BlockSuite](https://blocksuite.io/): the notes editor; [@xyflow/react](https://reactflow.dev/): the canvas
pipelines and workflows are drawn on
- [Leaflet](https://leafletjs.com/) (BSD-2-Clause): the Library's map of where assets were taken, on XYZ tiles from
Settings → Maps ([OpenStreetMap](https://www.openstreetmap.org/)'s by default), loaded only when the map is opened
- Processing: ffmpeg, [SenseVoice](https://github.com/FunAudioLLM/SenseVoice) / [faster-whisper](https://github.com/SYSTRAN/faster-whisper) /
[mlx-whisper](https://github.com/ml-explore/mlx-examples), [SpeechBrain](https://speechbrain.github.io/) ECAPA voiceprints,
optional [pyannote](https://github.com/pyannote/pyannote-audio), Tesseract / Apple Vision / RapidOCR, OpenCV YuNet + SFace,
Expand Down
Loading
Loading
0