8000 feat(automations): new assets as emails, calendar events, files, uploads, web pages and imports by adeelahmad · Pull Request #435 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading 8000
Diff view
Diff view
47 changes: 40 additions & 7 deletions docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ merged is called built.
| Match trees: one condition language for triggers, an automation's If and workflow nodes | built |
| Activities, the kinds catalog, the event log, following the stream, timeline reads (`app/domain/events.py`) | built |
| New assets as `asset.added` (the `recordings` source) | built |
| More kinds: emails, files, calendar events, jobs, deletions, comments, notes, graph changes, sensors, chat | planned |
| New assets by kind: emails received, calendar events, files from storage, uploads, pages saved from the web, imports | built |
| More kinds: jobs, deletions, calendar moves, comments, notes, graph changes, sensors, chat | planned |
| The timeline API (`GET /api/v1/activities`), the MCP tools `timeline` and `event_kinds`, the assistant's `timeline` | built |
| Review marks on the timeline (reviewed, important, hidden), their filters and `mark_activities` ([Timeline](timeline.md)) | built |
| The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built |
Expand Down Expand Up @@ -73,9 +74,32 @@ registered), and whether it is `noisy` (left out of the timeline unless named).
then its own), `events.descendants(name)` the kind and every kind under it, and `events.field_at(kind, path)` the
schema of one field, whose `title` names it in words (`match.describe(tree, kind)` says "the title contains “call”").

| Kind | Parent | Source | Object |
| --- | --- | --- | --- |
| `asset.added` | | `recordings` | `{type: Audio, Video, Image or Document, id: lens:recording/<id>, name: its title, url: /resources/<id>, media: audio, video, image, document or transcript, content_type}` |
Every kind's object has `type`, `id`, `name` and `url`; the table lists what each adds. Its fields' titles, a sample
and the paths holding outside text are in `GET /api/v1/activities/kinds` (`event_kinds` over MCP).

| Kind | Parent | Source | Object | Example summary |
| --- | --- | --- | --- | --- |
| `asset.added` | | `recordings` | `{type: Audio, Video, Image or Document, id: lens:recording/<id>, name: its title, url: /resources/<id>, media: audio, video, image, document or transcript, content_type}` | Lens added “Scan 12” |
| `email.received` | `asset.added` | `recordings` | `from`, `to`, `cc` (addresses), `subject`, `list`, `mailbox`, `date` (sent), `snippet` (the start of its text), `attachments` (how many) | Work mail received “School trip on Friday” from office@stmarys.org |
| `calendar_event.added` | `asset.added` | `recordings` | `startTime`, `endTime`, `location`, `organizer`, `attendees` (addresses), `calendar` | Family calendar added the event “Parents' evening” on 2026-10-20 |
| `file.added` | `asset.added` | `recordings` | `path` (in its source), `mediaType`, `size` (bytes), `duration` (seconds) | Office share added “Minutes” |
| `upload.added` | `asset.added` | `recordings` | `path` (the file's name), `mediaType`, `size`, `duration` | Sam uploaded “Call with the school” |
| `web_page.saved` | `asset.added` | `recordings` | `href` (the page's address), `site` | Sam saved “Museum opening hours” from museum.example.org |
| `import.received` | `asset.added` | `recordings` | `via`: hook (an import hook), paste (pasted text) or file (a file read as text) | Phone shortcut imported “Shopping list” |

A new asset is one activity, of the most specific kind its row tells, decided by its content type and where it came
from, never by reading its file: an email (content type email, or from an IMAP or Gmail mailbox), a calendar event
(content type calendar_event, or from an iCal feed or Google Calendar), an import hook's, one saved from the web
(webcapture.py, mediaurl.py), a file from a storage source or Drive, an upload, an import (pasted text, a transcript
imported), else `asset.added`. A file an email or archive held, made a resource of its own, stays `asset.added`: the
email is what arrived. A trigger on `asset.added` matches every one of them.

Fields come from what the row keeps, so reading them costs nothing on the ingest path. A message's addressing is
kept as it is listed (`recording.mail`: who sent it to whom, its subject, list, mailbox, and Gmail's snippet), an
event's times, place and people as it is imported (`recording.event`), and an email's attachments and the start of
its text once it has been read (`recording.email`). Emails and events imported before these were kept have only the
fields their rows had. Outside text (titles, subjects, addresses, snippets, places, file names, page addresses) is
listed in `lens.untrusted`.

### Sources and the event log

Expand All @@ -93,9 +117,13 @@ so nothing is stored twice:
A kind belongs to the source named in its `source`, and a source may list more in `kinds`. Two are built:

- `recordings` (`app/domain/events_content.py`) reads `recording` by `created_at`, the id breaking ties, as
`asset.added`. The actor is whoever added it: an automation (its first job was queued by `automation:<id>`), the
storage source a watched file came from (`Service`, `lens:source/<id>`), the person whose upload it was (`Person`,
`lens:account/<id>`), or Lens (a folder scan).
`asset.added` and its children (Kinds, above). The actor is whoever added it: an automation (its first job was
queued by `automation:<id>`), the storage source or mailbox it came from (`Service`, `lens:source/<id>`), an import
hook (`Service`, `lens:import_hook/<id>`), the person whose upload, clip or import it was (`Person`,
`lens:account/<id>`, or `lens:email/<address>` for someone who isn't an account), a sensor's digest (`Service`,
`lens:sensor/<id>`), or Lens (a folder scan). `events_content.actors(db, labels)` names them the way Lens writes
them down (an account id, an email, `watch:<id>`, `routine:<id>`, `import_hook:<id>`, `automation:<id>`, ...),
in a few queries for a whole page, and keeps names for 30 seconds.
- `log` reads `event_log`: activities no row of their own records (a deletion, a webhook call), written by
`events.emit(db, kind, *, actor, obj, space, summary, target=None, tag=None, key=None, published=None, spaces=None,
audience=None, owner=None, untrusted=None)`, which gives the id (`log-<n>`). Emitting a `key` again changes
Expand Down Expand Up @@ -282,6 +310,11 @@ room needs the bridge on and answering as the automation's person (or the person
no row records; pushing instead of polling.
- People given only some collections of a namespace see that namespace's activities about those collections.
- Name an automation actor by its automation's name once automations are stored.
- A timeline page filtered to one child of `asset.added` (only emails, say) reads every new asset and keeps those of
that kind, up to five source pages; narrow it in the query (path, `remote`, `web`) if that proves slow.
- Folder scans of a namespace's own folders as `file.added`; uploaded `.ics` files with their event's fields; a
snippet for IMAP messages before they are read (it needs the message body).
- Per-recipient threading of email conversations; entity tags on new assets once extraction runs at ingest.
- Telling a person: buttons on chat approvals (Telegram inline keyboards, Slack Block Kit); email as a `notify`
channel; web push; private chats through Matterbridge (it can't tell them from rooms); with several people let in,
"me" is whoever last wrote privately, until each chat user has a Lens account of their own; telling which links in a
Expand Down
5 changes: 4 additions & 1 deletion docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,10 @@ kept anyway, such as `recording.created_at`. What no row records goes in `event_
of the stream keeps where its next look starts in `event_cursor:<consumer>` (a position per source), and claims what
it handles in `event_claim:<hash of consumer and key>`; a claim is a `CREATE` only one process wins, and claims go
after a day. Each person's review marks on the timeline ([Timeline](timeline.md#review-marks)) are
`event_mark:<hash of account and activity>`, a row only while one of its flags is set.
`event_mark:<hash of account and activity>`, a row only while one of its flags is set. So that it can be read
without opening files, a resource made from an email keeps the message's addressing as it was listed
(`recording.mail`) and one made from a calendar event its times, place and people (`recording.event`); both are
optional, and older resources simply don't have them.

## Upgrades

Expand Down
25 changes: 25 additions & 0 deletions fastapi_backend/app/domain/calendars.py
Original file line number Diff line number Diff line change
Expand Up @@ -378,6 +378,31 @@ def event_text(event):
return "\n".join([f"# {title}", "", *head, *([""] + [desc] if desc else [])]) + "\n"


def 10000 _address(value):
return re.sub(r"^mailto:", "", (value or "").strip(), flags=re.I).strip().lower() or None


def details(raw, calendar=None):
"""What an automation or the timeline reads of a calendar's first event (an event's own .ics), kept on its
resource as `event`: {uid, startTime, endTime, location, organizer, attendees, calendar}; None for no event."""
events = parse(raw)["events"]
if not events:
return None
ev = events[0]
p, v = first(ev, "DTEND")
_, org = first(ev, "ORGANIZER")
out = {
"uid": key(ev)[:300],
"startTime": iso(start(ev)),
"endTime": iso(when(p, v, ev.get("zones"))),
"location": unescape(first(ev, "LOCATION")[1] or "").strip()[:300] or None,
"organizer": _address(org),
"attendees": [a for a in (_address(v) for _, v in all_of(ev, "ATTENDEE")) if a][:50] or None,
"calendar": (calendar or "")[:200] or None,
}
return {k: v for k, v in out.items() if v is not None}


def text(raw):
"""A whole calendar as Markdown, its events in order of when they start (for a .ics file read as a transcript)."""
cal = parse(raw)
Expand Down
16 changes: 14 additions & 2 deletions fastapi_backend/app/domain/convert.py
Original file line number Diff line number Diff line change
Expand Up @@ -841,6 +841,19 @@ def _fetch_for(cfg, ext, say):
fetch_program(cfg, "chromium", say)


def email_info(e):
"""What a resource keeps of an email it was read from (read_eml's shape): its subject, who sent it to whom, when,
how many files were attached, and the start of its text (for the timeline and automations)."""
from .ingest import _html_text

info = {k: e[k] for k in ("subject", "from", "to", "cc", "date") if e.get(k)}
info["attachments"] = sum(1 for a in e.get("parts") or [] if a.get("attached"))
body = e.get("text") if e.get("text") is not None else _html_text(e.get("html"))
if body and body.strip():
info["snippet"] = " ".join(body.split())[:300]
return info


def to_pdf(cfg, src, out, say=None):
"""Make `out`, the PDF of the document at `src`. Returns what was learnt on the way: {by, title?, email?,
attachments?}. Unavailable when the server can't; ValueError when the file can't be read. A program installed
Expand All @@ -856,8 +869,7 @@ def to_pdf(cfg, src, out, say=None):
if ext in (".eml", ".msg"):
e = read_msg(path) if ext == ".msg" else read_eml(path)
page, attachments = email_page(e)
info = {k: e[k] for k in ("subject", "from", "to", "cc", "date") if e.get(k)}
return {"by": page_pdf(cfg, page, out, via), "title": e["subject"] or None, "email": info, "attachments": attachments}
return {"by": page_pdf(cfg, page, out, via), "title": e["subject"] or None, "email": email_info(e), "attachments": attachments}
if ext in store.ARCHIVE_EXT:
from . import archives

Expand Down
Loading
Loading
0