8000 feat(automations): a webhook address per automation trigger by adeelahmad · Pull Request #442 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,25 @@ Each person marks activities `reviewed`, `important` or `hidden` for themselves
and `POST /activities/mark` with `ids` (up to 500) answers `{marks, unknown}`. `GET /activities` has your marks on
the page in `marks` and filters by them with `reviewed`, `important` and `hidden` (true or false).

## automations

```
POST /api/v1/automations/hooks/{token}
```

An automation's webhook ([Webhook trigger](automations.md#webhook-trigger)): calling the address of a trigger
`{"on": "webhook"}` sets that automation off. No session or API token: the token in the address (`lah_...`, shown
once when the automation is saved) is the credential. Send JSON, a form (text fields) or text, up to 64 KB; the answer
(202) is `{"id"}`, the `webhook.received` activity the call made, before the automation runs. Sign the call the
Standard Webhooks way (`webhook-id`, `webhook-timestamp`, `webhook-signature`, with the secret shown with the
address) and the signature is checked; a `webhook-id` or `Idempotency-Key` makes a retried call count once. 401 for a
token that isn't a webhook's or a signature that doesn't match, 403 while the automation is switched off, 413 over 64
KB, 415 for other bodies, 429 past 60 calls a minute.

```
curl -H "Content-Type: application/json" -d '{"event": "doorbell"}' https://lens.example/api/v1/automations/hooks/lah_...
```

## vaults

```
Expand Down
41 changes: 40 additions & 1 deletion docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ merged is called built.
| New assets by kind: emails received, calendar events, files from storage, uploads, pages saved from the web, imports | built |
| Runs that finished or failed, deletions and calendar events that moved | built |
| Comments, notes written and saved, and changes to the entity graph | built |
| More kinds: sensors, chat, webhooks, system events | planned |
| More kinds: sensors, chat, system events | planned |
| The timeline API (`GET /api/v1/activities`), the MCP tools `timeline` and `event_kinds`, the assistant's `timeline` | built |
| Review marks on the timeline (reviewed, important, hidden), their filters and `mark_activities` ([Timeline](timeline.md)) | built |
| The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built |
Expand All @@ -26,6 +26,7 @@ merged is called built.
| Running Lens work: the `run_workflow`, `run_pipeline` and `run_routine` actions | built |
| Changing assets and writing notes: the `set_field`, `add_to_collection` and `write_note` actions | built |
| Settings → Automations: how often followers look and how long the event log is kept | built |
| A webhook address per automation trigger, and calls to it as `webhook.received` ([Webhook trigger](#webhook-trigger)) | built |
| The Timeline page with filters, grouping, review keys and new items, and Home's Latest card ([Timeline](timeline.md)) | built |
| Storing and running automations, schedules and timers, joins, approvals, variables, the studio | planned |

Expand Down Expand Up @@ -101,6 +102,7 @@ and the paths holding outside text are in `GET /api/v1/activities/kinds` (`event
| `note.created` | | `note_pages` | a `Page`: `summary`, `about` (what it's the page of), `place`, `author` (person or assistant) | Assistant wrote the note “St Mary's School” |
| `note.saved` | | `note_versions` | as `note.created`, with `version` (the page as it was before) and `why` | Sam edited the note “School trip plans” |
| `entity.changed` | | `graph` | `op` (created, renamed, merged, split, linked, unlinked, deleted, rolled back, changed), `change` (the graph's own op), `entityType`, `before` (a renamed entity's name), `version`, `count`; tagged with its entities | Sam renamed “St Marys” to “St Mary's School” |
| `webhook.received` | | `log` | `{type: Note, hook: the webhook's id, payload: what was sent (JSON, a form's fields or text), content_type, query}`; the actor is the webhook (`Service`, `lens:automation_hook/<id>`, "Webhook for <automation>") | Webhook for Front door was called |

A new asset is one activity, of the most specific kind its row tells, decided by its content type and where it came
from, never by reading its file: an email (content type email, or from an IMAP or Gmail mailbox), a calendar event
Expand Down Expand Up @@ -265,6 +267,41 @@ gives a cleaned copy, or a ValueError naming the first problem.
- `check_policy(p, nodes, allow_auto=False)`: `{action node id: ask or auto}`. Only a person in a signed-in session
lets a node run on its own; everywhere else (agents, drafts, the workflow API) `auto` is refused.

### Webhook trigger

Built (`app/domain/automation_hooks.py`, `POST /api/v1/automations/hooks/{token}`). A trigger `{"on": "webhook"}`
is a secret address anything that can send HTTP calls to set the automation off: Home Assistant's `rest_command`, a
phone shortcut, another service. No webhook exists until an automation with such a trigger is saved.

- `automation_hooks.token_for(db, cfg, automation_id, alias, signed=False)` makes the hook for one trigger when the
automation is saved: `{"hook": id, "url": address, "path": address without the server, "secret": signing secret}`.
The address and secret are shown this once; the row (`automation_hook {automation, alias, token_hash, secret,
signed, created_at, last_called_at, calls}`) keeps only the token's hash and the secret sealed with the server's
key. Calling it again for the same trigger replaces the hook, and the old address stops working. The trigger keeps
the hook's id in `hook`. `automation_hooks.revoke(db, hook)` deletes it (when the trigger or the automation goes);
`for_automation(db, id)` lists an automation's hooks.
- A call needs only the token in the address (`lah_...`). It takes JSON, a form (text fields; files are refused) or
text, up to 64 KB, at most 60 calls a minute per hook (in the API process; 429 with `Retry-After` past it), and
answers 202 `{"id": activity}` at once. 401 for a token that isn't a hook's, 403 while the automation is switched
off, 413 over 64 KB, 415 for anything but JSON, a form or text.
- What was sent becomes a `webhook.received` activity in the event log, with the payload, its content type and the
query as outside text. Its namespace is the automation's (admins see it while the automation isn't stored).
- Signed calls: a caller that signs the Standard Webhooks way (`webhook-id`, `webhook-timestamp` in seconds,
`webhook-signature: v1,<base64 HMAC-SHA256 of "<id>.<timestamp>.<body>">` keyed with the secret, as Lens signs its
own [notification webhooks](notifications.md#webhooks)) has the signature checked, and the time must be within five minutes.
A hook made with `signed=True` refuses calls that aren't signed. A call naming its message (`webhook-id`, or an
`Idempotency-Key` header) is logged once however often the sender retries it.
- In Docker the API runs no background work, so the worker's automations see the call on their next look (5
seconds, `events.poll_seconds`).

Lens has three kinds of webhook address; they don't share tokens or do each other's jobs:

| Address | Made by | What a call does |
| --- | --- | --- |
| `POST /api/v1/automations/hooks/{token}` | saving an automation with a webhook trigger | sets that automation off with what was sent (`webhook.received`); nothing is kept but the activity |
| `POST /api/v1/sensors/push/{token}` ([Sensors](sensors.md)) | an admin adding a webhook sensor | keeps readings (numbers, on/off, text) for charts, with the sensor's handling |
| `POST /api/v1/hooks/import/{token}` ([Import webhooks](api.md#import-webhooks)) | a namespace's owner | imports files, web pages or text as assets |

### Run context

What an automation's If, its nodes and its templates see (`automation_spec.context_paths()` lists the keys, with
Expand Down Expand Up @@ -454,3 +491,5 @@ the agent can draft an event as a note.
- Changing assets: asking first instead of failing when a move would show an asset to new people (risk that reads the
database); fields on collections and files; tags; roles on collections counting for `edit`.
- Writing calendar events (a new Google consent, so it changes existing installs).
- Webhook trigger: signed answers to the caller (what the run did); a rate limit shared by every API process (today
each counts its own); showing a hook's last call in the studio.
3 changes: 2 additions & 1 deletion docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,8 @@ kept anyway, such as `recording.created_at`. What no row records goes in `event_
of the stream keeps where its next look starts in `event_cursor:<consumer>` (a position per source), and claims what
it handles in `event_claim:<hash of consumer and key>`; a claim is a `CREATE` only one process wins, and claims go
after a day. Each person's review marks on the timeline ([Timeline](timeline.md#review-marks)) are
`event_mark:<hash of account and activity>`, a row only while one of its flags is set. So that it can be read
`event_mark:<hash of account and activity>`, a row only while one of its flags is set. An automation trigger's
webhook is `automation_hook:<n>` (its automation and trigger, the hash of its token, its signing secret sealed). So that it can be read
without opening files, a resource made from an email keeps the message's addressing as it was listed
(`recording.mail`) and one made from a calendar event its times, place and people (`recording.event`); both are
optional, and older resources simply don't have them.
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
activity,
admin,
auth,
automation_hooks,
backups,
batches,
browser,
Expand Down Expand Up @@ -83,6 +84,7 @@
updates,
activity,
activities,
automation_hooks,
budgets,
fedora,
setup,
Expand Down
100 changes: 100 additions & 0 deletions fastapi_backend/app/api/v1/routes/automation_hooks.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
"""An automation's webhook (docs/automations.md, Webhook trigger): the address anything that can send HTTP calls to set
an automation off. The token in the address is all a caller needs; the hooks themselves are made and revoked when
automations are saved and deleted (domain/automation_hooks.py)."""

from __future__ import annotations

from typing import Any

from fasta 8000 pi import APIRouter, HTTPException, Request
from starlette.concurrency import run_in_threadpool
from starlette.formparsers import MultiPartException, MultiPartParser

from app.api.deps import Cfg, Db
from app.domain import automation_hooks
from app.schemas.automation_hooks import HookCalled

router = APIRouter(tags=["automations"])
CALL_BODY = {
"requestBody": {
"required": False,
"content": {
"application/json": {"schema": {}},
"application/x-www-form-urlencoded": {"schema": {"type": "object", "additionalProperties": {"type": "string"}}},
"multipart/form-data": {"schema": {"type": "object", "additionalProperties": {"type": "string"}}},
"text/plain": {"schema": {"type": "string"}},
},
}
}


async def _body(request: Request) -> bytes:
"""The request's body, refused (413) past automation_hooks.MAX_BODY without reading the rest."""
most = automation_hooks.MAX_BODY
told = request.headers.get("content-length") or ""
if told.isdigit() and int(told) > most:
raise HTTPException(413, f"send at most {most // 1024} KB")
got, n = [], 0
async for piece in request.stream():
n += len(piece)
if n > most:
raise HTTPException(413, f"send at most {most // 1024} KB")
got.append(piece)
return b"".join(got)


async def _multipart(request: Request, body: bytes) -> dict[str, Any] | None:
"""A multipart form's fields ({field: value, or a list of values}); files are refused."""
if (request.headers.get("content-type") or "").split(";")[0].strip().lower() != "multipart/form-data":
return None

async def once() -> Any:
yield body

try:
form = await MultiPartParser(request.headers, once(), max_files=0, max_fields=automation_hooks.MAX_FIELDS).parse()
except MultiPartException as e:
raise HTTPException(400, f"a form of text fields only: {e.message}") from None
out: dict[str, Any] = {}
for k in form:
vals = [str(v) for v in form.getlist(k)]
out[k] = vals[0] if len(vals) == 1 else vals
return out


@router.post(
"/automations/hooks/{token}",
status_code 9946 =202,
openapi_extra=CALL_BODY,
responses={
401: {"description": "not a webhook's token, or a signature that doesn't match"},
403: {"description": "the automation is switched off"},
413: {"description": "over 64 KB"},
415: {"description": "neither JSON, a form nor text"},
429: {"description": "over 60 calls a minute to this webhook"},
},
)
async def call_automation_hook(token: str, request: Request, db: Db, cfg: Cfg) -> HookCalled:
"""Set an automation off: what is sent (JSON, a form or text, up to 64 KB) becomes a `webhook.received` activity
for the automation whose trigger this address is. No session: the token in the address is the credential. Signed
the Standard Webhooks way (webhook-id, webhook-timestamp, webhook-signature), the signature must match the webhook's
secret; `webhook-id` or `Idempotency-Key` makes a retried call count once. Answers at once, before the automation
runs."""
body = await _body(request)
form = await _multipart(request, body)
try:
aid = await run_in_threadpool(
automation_hooks.receive,
db,
cfg,
token,
body,
request.headers.get("content-type") or "",
request.query_params.multi_items(),
dict(request.headers),
form,
)
except automation_hooks.Refused as e:
headers = {"Retry-After": "60"} if e.status == 429 else None
raise HTTPException(e.status, str(e), headers=headers) from None
return HookCalled(id=aid)
Loading
Loading
0