8000 feat(imports): date photos and videos by when they were taken, from the file itself by adeelahmad · Pull Request #674 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
8000
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,9 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
emails a light server can't convert itself.
Google accounts connect with Connect with Google once Lens has a Google client, and Dropbox, OneDrive and Google
Drive can sign in through rclone on the Lens server, with no app of your own to register. Upload audio, video, PDFs,
Office files, photos, JSON and zip or tar archives, and, with an Apache Tika server (an opt-in extra), documents
nothing else here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking
Office files, photos, JSON and zip or tar archives (photos and videos dated by when they were taken, from the file
itself, and photos placed by their GPS), and, with an Apache Tika server (an opt-in extra), documents nothing else
here can read; capture web pages one at a time, from a link list or a bookmarks file, by API or by asking
the assistant, from private networks you allow in Settings too; and import videos and podcasts by their address (off
by default), taking a video's own captions as its transcript when people wrote some (off by default too). Say what a
file or a page is as it comes in (a podcast, an interview, a scan), in Import, by API or to the assistant, and its
Expand Down
6 changes: 6 additions & 0 deletions changelog.d/file-dates.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
- **Photos and videos are dated by when they were taken.** An asset uploaded, scanned or imported is now dated by its
own file when its name gives no date: a photo by its EXIF (`DateTimeOriginal`, else `DateTime`), a video or a
recording by its creation time (QuickTime's `creationdate`, else `creation_time`), before the file's last change. A
photo's GPS fix is kept as its `place` (`lat`, `lon`), which `GET /recordings/{rid}` and MCP `get_recording` give its
namespace's members. A date in its name, or one its source gives, still wins, and a clock never set (1970, 1904) dates
nothing ([When an asset was made](docs/processing.md#when-an-asset-was-made)).
3 changes: 2 additions & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1216,7 +1216,8 @@ runs instead of the namespace's once a new asset is made, `content_type` is the
type](processing.md#content-types) (one of the file's base type, else 400; its pipeline runs) and `collection` is
where it goes (none of them with `recording`, which keeps its own). The same file uploaded again with a content type
gives the asset it already is that one. The name loses any folders and its extension is lowercased; `modified`
(the file's last-modified time in milliseconds) dates the asset when its name doesn't. 400 for a type not in
(the file's last-modified time in milliseconds) dates the asset when neither its name nor the file itself does
([When an asset was made](processing.md#when-an-asset-was-made)). 400 for a type not in
`uploads.extensions`, 413 over `uploads.max_mb`, 507 when the server's disk can't hold it with 512 MB to spare.

`PUT /uploads/{uid}?offset=` sends the next piece as the raw request body (`application/octet-stream`), starting at
Expand Down
18 changes: 18 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,24 @@ key terms with Universal-3 and Slam-1, or boosted words with the others. SenseVo
they transcribe as before. The words go only to the engine the recording was going to anyway, and a provider's row in
the [activity ledger](activity.md) says how many it was given. Run Transcribe again on an asset to use new names.

### When an asset was made

An asset's date (`recorded_at`) is set once, when it arrives (scanned, uploaded, shared into a mounted folder or sent
to an import webhook), from the first of these that says it:

1. A date in its file's name, such as `2024-03-05 interview.mp3` or `IMG_20240305_101500.jpg`.
2. The file itself: a photo's EXIF (`DateTimeOriginal`, with its `OffsetTimeOriginal` where the camera wrote one, else
`DateTime`), or a video's or a recording's creation time as ffprobe reads it (QuickTime's
`com.apple.quicktime.creationdate`, which keeps the phone's local time, else `creation_time`, in UTC).
3. When the file last changed (for an upload, the last-modified time the browser sends).

Nothing is guessed: a clock that was never set (a year before 1971, such as QuickTime's 1904) or a date in the future
counts as no date. A date an imported transcript gives wins over all three, as an email's or a document's own date
does once it is read, and a file uploaded again into the namespace that has it keeps its asset's date. A file on a
[storage source](#storage-sources) is dated by the source (the date it lists, else when the file last changed there). A photo's GPS fix from its EXIF is kept as the asset's `place` (`{lat, lon}`, in degrees);
`GET /api/v1/recordings/{rid}` and the MCP tool `get_recording` give it to the namespace's members only, never to
visitors. It is read on this machine and nothing is sent anywhere. Assets that arrived before keep their dates.

### Cleaning up transcripts

Speech comes out of transcription as it was said, and the extractor, chapters, keywords and search all read it. With
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Connect with Google; Dropbox, OneDrive and Google Drive sign in through rclone on the server | [test_gmail_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_gmail_source.py) `test_connecting_the_account_in_the_browser`; [test_drive_google.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_drive_google.py) `test_drive_connects_with_google_for_rclone`; [test_rclone_signin.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_rclone_signin.py) `test_drive_signs_in_through_rclone_with_no_google_app`, `test_a_browser_on_the_server_reaches_rclone_itself`, `test_onedrive_finds_the_accounts_own_drive` |
| Uploads of audio, PDFs, Office files, photos, JSON and archives | [test_uploads.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_uploads.py) `test_uploading_audio_in_chunks`, `test_json_uploaded_is_read_as_records`; [test_documents.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_documents.py) `test_a_pdf_becomes_a_document_with_its_pages_and_text`, `test_scans_and_images_are_read_by_ocr`; [test_conversion.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_conversion.py) `test_office_and_text_files_become_documents`; [test_anytopdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_anytopdf.py) `test_an_uploaded_zip_keeps_its_files`; [test_archives.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_archives.py) `test_reading_an_archive`; [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `import-file`, `import-paste` |
| Media browsers can't play (an AVI, HEVC, AMR) gets a copy they can, for the player only, off by default; files they play get none | [test_playable.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_playable.py) `test_a_camera_clip_gets_a_copy_every_browser_plays`, `test_a_file_browsers_play_gets_no_copy` |
| Photos and videos dated by when they were taken, from the file itself (a photo's EXIF date and GPS fix as `place`, a video's creation time) when no name or source dates them; a date already set is kept and none is invented | [test_file_dates.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_file_dates.py) `test_a_photo_is_dated_and_placed_by_its_exif`, `test_a_date_already_set_is_kept`, `test_a_video_is_dated_by_its_creation_time`, `test_a_scanned_video_is_dated_by_its_creation_time` |
| An uploaded video, end to end | No check yet ([#486](https://github.com/adeelahmad/lense/issues/486)) |
| Apache Tika (opt-in) reads documents nothing else here can | [test_tika.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_tika.py) `test_off_by_default_nothing_changes`, `test_what_nothing_here_can_convert_tika_reads_as_text` |
| Web pages captured one at a time, from a link list or a bookmarks file, by API or by asking; private networks only when allowed | [test_webcapture.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webcapture.py) `test_a_web_page_becomes_a_document`, `test_an_admin_allows_a_private_network_in_settings`, `test_private_addresses_are_refused`; [test_web_links.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_web_links.py) `test_one_or_several_addresses_as_a_list`, `test_a_bookmarks_file_becomes_pages_to_capture`; [test_setup_assistant.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_setup_assistant.py) `test_the_assistant_captures_web_pages` |
Expand Down
5 changes: 3 additions & 2 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -905,8 +905,8 @@ def list_speakers(ctx: Context, namespace: str | None, query: str | None, limit:
@tool(
"get_recording",
"Get an asset",
"One asset: what it is, its date, the people in it, summary, chapters (sections), the entities and "
"keywords most mentioned in it, and a url to it. Read its text with get_transcript.",
"One asset: what it is, its date, where a photo was taken (place), the people in it, summary, chapters (sections), "
"the entities and keywords most mentioned in it, and a url to it. Read its text with get_transcript.",
Arg("recording_id", "integer", "the asset", required=True),
)
def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]:
Expand All @@ -924,6 +924,7 @@ def get_recording(ctx: Context, recording_id: int) -> dict[str, Any]:
"namespace": rec.namespace,
"collection": " / ".join(c.name for c in rec.collection_path) or None,
"recorded_at": raw.get("recorded_at"),
"place": rec.place.model_dump() if rec.place else None, # a photo's GPS fix, for members
"duration": store.tc(raw["duration_ms"]) if raw.get("duration_ms") else None,
"media": render.kind(raw),
"pages": raw.get("pages"),
Expand Down
4 changes: 3 additions & 1 deletion fastapi_backend/app/api/v1/routes/recordings.py
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ def retag_recordings(body: RecordingsRetag, acl: Acl, user: Writer, db: Db) -> T
def get_recording(rid: int, acl: Acl, db: Db, cfg: Cfg) -> Recording:
r = acl.recording(rid)
space = db.one("SELECT name FROM $s", s=R("space", r["space"])) or {}
d = {k: v for k, v in r.items() if k not in ("envelope", "stats", "summary", "access_parts", "enrichment")}
d = {k: v for k, v in r.items() if k not in ("envelope", "stats", "summary", "access_parts", "enrichment", "place")}
a = acc.of(db, rid)
d.update(access=a["access"], open=a["open"], featured=a["featured"], access_inherited=a["inherited"])
d.update(
Expand All @@ -336,6 +336,8 @@ def get_recording(rid: int, acl: Acl, db: Db, cfg: Cfg) -> Recording:
d["converted_from"] = _attached_to(acl, db, r.get("converted_from"))
if r.get("enrichment") and acl.member(r): # where a photo was taken: members only
d["enrichment"] = {k: v for k, v in r["enrichment"].items() if k != "photo_fp"}
if r.get("place") and acl.member(r): # its file's GPS fix, likewise
d["place"] = r["place"]
d["nsfw"] = _nsfw(db, cfg, r, rid)
d["enhanced"] = enhance.brief(r.get("enhanced"), rid)
waiting = r.get(&quo 10000 t;awaits_pdf") or None
Expand Down
104 changes: 84 additions & 20 deletions fastapi_backend/app/domain/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import importlib.util
import html
import json
import math
import pathlib
import re
import subprocess
Expand Down Expand Up @@ -41,32 +42,30 @@ def parse_sv(raw):
MEDIA_DEMUXERS = "mp3,mov,wav,flac,ogg,aac,matroska,amr,avi,asf,mpeg,mpegts,aiff"


def probe(path):
ch = 0
def _ffprobe(path, entries):
"""What ffprobe says of an audio or video file (the -show_entries asked for), as JSON; {} when it can't read it."""
try:
out = subprocess.run(
[
"ffprobe",
"-v",
"error",
"-format_whitelist",
MEDIA_DEMUXERS,
"-show_entries",
"format=duration:stream=channels,codec_type",
"-of",
"json",
str(path),
],
["ffprobe", "-v", "error", "-format_whitelist", MEDIA_DEMUXERS, "-show_entries", entries, "-of", "json", str(path)],
capture_output=True,
text=True,
timeout=120,
)
j = json.loads(out.stdout or "{}")
except (OSError, ValueError, subprocess.TimeoutExpired):
return {}
return j if isinstance(j, dict) else {}


def probe(path):
ch = 0
j = _ffprobe(path, "format=duration:stream=channels,codec_type")
try:
dur = float(j.get("format", {}).get("duration") or 0)
ch = max([s.get("channels", 0) for s in j.get("streams", []) if s.get("codec_type") == "audio"] or [0])
if dur:
return int(dur * 1000) or None, int(ch) or None
except (OSError, ValueError, subprocess.TimeoutExpired):
except (TypeError, ValueError):
pass
dur, wav_ch = _probe_wav(path)
return dur, (int(ch) or None) or wav_ch
Expand Down Expand Up @@ -110,14 +109,78 @@ def _fingerprint(f, size, block):
DATE_RX = re.compile(r"((?:19|20)\d{2})[-_.]?(\d{2})[-_.]?(\d{2})(?:[ T_-]?(\d{2})[-_.:h]?(\d{2})(?:[-_.:m]?(\d{2}))?)?")


def recorded_at(path, mtime):
def recorded_at(path, mtime, taken_at=None):
"""When an asset was made: a date in its file's name (someone named it so), else the date its file itself says it
was taken (taken()), else when the file last changed."""
for m in DATE_RX.finditer(pathlib.Path(path).stem):
y, mo, d, h, mi, s = (int(x) if x else 0 for x in m.groups())
try:
return dt.datetime(y, mo, d, h, mi, s).isoformat(timespec="seconds")
except ValueError:
continue
return dt.datetime.fromtimestamp(mtime).isoformat(timespec="seconds")
return taken_at or dt.datetime.fromtimestamp(mtime).isoformat(timespec="seconds")


OLDEST_YEAR = 1971 # a clock never set writes 1970, or 1904 (QuickTime's zero): not when anything was taken


def _taken_when(s):
"""A date and time a file gives (ISO 8601, or EXIF's "2024:05:01 12:20:30"), as ISO 8601 to the second with its
offset where it has one; None for one that isn't a real date (unset, before 1971 or in the future)."""
s = str(s or "").strip().strip("\x00").strip()
if re.match(r"\d{4}:\d{2}:\d{2}", s):
s = s[:10].replace(":", "-") + s[10:]
try:
t = dt.datetime.fromisoformat(s)
except ValueError:
return None
if t.year < OLDEST_YEAR or t > dt.datetime.now(t.tzinfo) + dt.timedelta(days=1):
return None
return t.isoformat(timespec="seconds")


def _degrees(v, ref):
"""An EXIF GPS coordinate (degrees, minutes, seconds) with its N/S or E/W, as signed degrees; None if unreadable."""
try:
d, m, s = (float(x) for x in v)
except (TypeError, ValueError, ZeroDivisionError):
return None
x = d + m / 60 + s / 3600
return -x if str(ref or "").upper().startswith(("S", "W")) else x


def _exif(path):
"""A photo's EXIF: when it was taken (DateTimeOriginal, else DateTime) and its GPS fix, as taken() gives them."""
from PIL import ExifTags, Image

try:
with Image.open(path) as img:
exif = img.getexif()
sub, gps = exif.get_ifd(ExifTags.IFD.Exif), exif.get_ifd(ExifTags.IFD.GPSInfo)
except (OSError, ValueError, SyntaxError, Image.DecompressionBombError):
return None, None
orig = sub.get(ExifTags.Base.DateTimeOriginal)
when = _taken_when(f"{orig}{sub.get(ExifTags.Base.OffsetTimeOriginal) or ''}") if orig else None
when = when or _taken_when(exif.get(ExifTags.Base.DateTime))
lat = _degrees(gps.get(ExifTags.GPS.GPSLatitude), gps.get(ExifTags.GPS.GPSLatitudeRef))
lon = _degrees(gps.get(ExifTags.GPS.GPSLongitude), gps.get(ExifTags.GPS.GPSLongitudeRef))
ok = lat is not None and lon is not None and math.isfinite(lat + lon) and abs(lat) <= 90 and abs(lon) <= 180
place = {"lat": round(lat, 6), "lon": round(lon, 6)} if ok and (lat, lon) != (0, 0) else None # 0, 0: no fix
return when, place


def taken(path):
"""When and where a photo, a video or a recording was taken, as its own file says: a photo's EXIF (DateTimeOriginal,
else DateTime, and its GPS fix), a video's or a recording's creation time (QuickTime's creationdate, which keeps
the local time, else creation_time). (ISO 8601 or None, {lat, lon} or None): nothing is guessed. Read it before
the file is encrypted."""
ext = pathlib.Path(path).suffix.lower()
if ext in store.IMAGE_EXT:
return _exif(path)
if ext in store.MEDIA_EXT:
tags = (_ffprobe(path, "format_tags=creation_time,com.apple.quicktime.creationdate").get("format") or {}).get("tags") or {}
return _taken_when(tags.get("com.apple.quicktime.creationdate")) or _taken_when(tags.get("creation_time")), None
return None, None


DECODE_SECONDS = 6 * 3600 # a recording is decoded far faster than it plays; past this ffmpeg is stuck
Expand Down Expand Up @@ -230,7 +293,7 @@ def scan(db, cfg, log=print):
d=store.clean({"path": str(p), "size": st.st_size, "mtime": st.st_mtime, "duration_ms": dur, "channels": ch}),
)
continue
rid = db.next_id("recording")
rid, (when, place) = db.next_id("recording"), taken(p)
db.q(
"CREATE $r CONTENT $d",
r=store.R("recording", rid),
Expand All @@ -243,7 +306,8 @@ def scan(db, cfg, log=print):
"fingerprint": fp,
"fp_key": f"{nid}:{fp}",
"title": p.stem,
"recorded_at": recorded_at(p, st.st_mtime),
"recorded_at": recorded_at(p, st.st_mtime, when),
"place": place,
"duration_ms": dur,
"channels": ch,
"size": st.st_size,
Expand Down Expand Up @@ -1077,7 +1141,7 @@ def import_transcript(db, cfg, ns, tpath, audio=None, title=None, speaker_names=
title or (t.get("title") if t.get("form") == "chat" else None) or pathlib.Path(tpath).stem, # a Telegram chat's name
fingerprint(src),
str(src.resolve()),
recorded_at(src, st.st_mtime),
recorded_at(src, st.st_mtime, taken(src)[0]),
audio,
speaker_names,
"import:" + pathlib.Path(tpath).suffix.lstrip("."),
Expand Down
Loading
Loading
0