8000 feat(automations): routine runs, budget warnings, backups and releases as activities by adeelahmad · Pull Request #449 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ merged is called built.
| Comments, notes written and saved, and changes to the entity graph | built |
| Sensors: readings a sensor keeps, alert and notable log patterns, devices that report for the first time ([Sensors](#sensors-as-activities)) | built |
| What people say to Lens, in the web app and chat apps, as `chat.message` ([Chat](#chat-as-activities)) | built |
| More kinds: system events | planned |
| Lens itself: routine runs, budget warnings, backups and new releases ([Lens itself](#lens-itself-as-activities)) | built |
| The timeline API (`GET /api/v1/activities`), the MCP tools `timeline` and `event_kinds`, the assistant's `timeline` | built |
| Review marks on the timeline (reviewed, important, hidden), their filters and `mark_activities` ([Timeline](timeline.md)) | built |
| The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built |
Expand Down Expand Up @@ -112,6 +112,10 @@ and the paths holding outside text are in `GET /api/v1/activities/kinds` (`event
| `sensor.alert` | | `log` | `{type: Note, id: lens:sensor_pattern/<id>}`, `label` (alert or notable), `pattern`, `example` (a line it fits), `count` (lines in all), `lines` (in the batch that raised it), `hour` | 192.168.1.1 logged an alert: wlan<n>: authentication failed for <mac> |
| `sensor.joined` | | `sensors` | `{type: Service, id: lens:sensor/<id>, url: /sensors/<id>}`, `sensor_type` (mqtt, syslog, ...), `device`, `status` | A new MQTT device reported: zigbee2mqtt/kitchen |
| `chat.message` | | `chat_messages` | `{type: Note, id: lens:chat_message/<id>, name: the start of it, url: /chat/<id>}`, `text`, `channel` (web, telegram, slack, bridge, mailbox), `room`, `chat`, `direct` (a private chat or the web app), `attachments` (how many files) | sam wrote in Telegram: “groceries: milk, eggs” |
| `routine.ran` | | `routine_runs` | `{type: Note, id: lens:routine_run/<id>, url: /routines/<id>}`, `status` (done, partly, error, skipped), `trigger` (schedule, manual), `startTime`, `endTime`, `actions`, `failed`, `error`, `changes` (graph changes applied and proposed), `cost_usd`, `tokens` | Nightly tidy ran |
| `budget.warned` | | `log` | `{type: Note, id: lens:budget/<resource>, url: /costs}`, `resource` (all, space:<id>, routine:<id>, ...), `state` (near, over), `share`, `usd`, `tokens`, `estimate`, `cap_usd`, `cap_tokens`, `period`, `since` | The pods namespace has used 82% of its budget |
| `backup.finished` | | `log` | `{type: Document, id: lens:backup/<name>, url: /settings/backups}`, `label` (manual, scheduled), `ok`, `error`, `kind` (export, folder), `bytes`, `files`, `copied` (to the storage connection) | A scheduled backup finished: scheduled-20261015-020000.surql.gz |
| `lens.release` | | `log` | `{type: Note, id: lens:release/<tag>, url: its release notes}`, `tag`, `title`, `current` (what this server runs), `published_at` | Lens v0.7.0 is out (this server runs 0.6.0) |
| `webhook.received` | | `log` | `{type: Note, hook: the webhook's id, payload: what was sent (JSON, a form's fields or text), content_type, query}`; the actor is the webhook (`Service`, `lens:automation_hook/<id>`, "Webhook for <automation>") | Webhook for Front door was called |

A new asset is one activity, of the most specific kind its row tells, decided by its content type and where it came
Expand Down Expand Up @@ -180,6 +184,22 @@ said to Lens in a room is never kept, so it is never an activity. Its text and n
- A question an automation's `ask_agent` asks (`origin` on the message) isn't a person's, so it is never a
`chat.message`, and an automation can't set itself off that way.

#### Lens itself as activities

Built (`app/domain/events_streams.py`). What Lens does on its own is for admins, unless it is about one namespace.
Each is logged once, however many processes see it happen.

- `routine.ran` is a routine run that ended: done, partly (some actions failed), error, or skipped over budget once
someone picked to skip it. Its actor is the routine (`Application`, `lens:routine/<id>`). A run held over budget is
one when it ends; one someone let run anyway runs again as a new run, and that run is the activity. A routine of
one namespace makes activities in it.
- `budget.warned` is a budget crossing its warning share (`warn_at`) or its cap, once per period and state, logged by
the same check that alerts on it ([Budgets](budgets.md#warnings)). A namespace's budget makes it in that namespace.
- `backup.finished` is a backup taken in Settings or through the API, or on the schedule, which says whether it
worked (a scheduled one that failed says why) and whether it was copied to the storage connection.
- `lens.release` is a newer Lens release, once per release, when Lens tells the admins and notification targets
([Updates](updates.md#being-told)).

### Sources and the event log

Built. A source (`events.Source`, added with `events.register_source`) reads rows Lens already keeps as activities,
Expand Down Expand Up @@ -212,6 +232,8 @@ A kind belongs to the source named in its `source`, and a source may list more i
- `sensor_readings` (`app/domain/events_streams.py`) reads `sensor_reading` by `at` (microseconds), the id breaking
ties, one sensor at a time, as `sensor.reading`; `sensors` reads the `storage_source` rows discovery made by
`created_at`, as `sensor.joined`.
- `routine_runs` (`app/domain/events_streams.py`) reads `routine_run` rows that ended by `finished_at`, the id
breaking ties, as `routine.ran`.
- `chat_messages` (`app/domain/events_streams.py`) reads `chat_message` rows with role user by `created_at` over the
index on (role, created_at), the id breaking ties, as `chat.message`.
- `log` reads `event_log`: activities no row of their own records (a deletion, a webhook call), written by
Expand Down Expand Up @@ -632,3 +654,5 @@ def _(db, cfg, payload): ... # payload + late (true
- Chat: recording every message in a room as an activity (opt-in per channel, off by default); a chat user who is
also a Lens account as that person; a timeline page of others' private messages is read and left out, up to five
source pages, so narrow it by account if that proves slow.
- Lens itself: a run held over budget, waiting for someone's pick, as an activity of its own; backups taken before
upgrades or with `lens backup` (only those from Settings, the API and the schedule are activities today).
3 changes: 2 additions & 1 deletion docs/budgets.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,8 @@ Asking a routine to run from the app or the API when it's over budget is refused
## Warnings

Every five minutes the scheduler looks at every budget. One that crosses `warn_at` or its cap gets an alert, once per
period: on the budget (`alert`) and as a `budget.near` or `budget.over` row in its activity history.
period: on the budget (`alert`) and as a `budget.near` or `budget.over` row in its activity history. Automations
see it too, as a `budget.warned` activity ([Automations](automations.md#lens-itself-as-activities)).

## The meter and the Costs page

Expand Down
3 changes: 3 additions & 0 deletions docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,9 @@ first ([Updates and backups](updates.md)). Each is one entry in `<data_dir>/back
downloads as one `.tar.gz` holding `<name>` and `<name>.files/`. Backups from before files were kept show *database
only* in Settings.

A backup taken in Settings, through the API or on the schedule is also a `backup.finished` activity for automations,
and a scheduled one that failed says why ([Automations](automations.md#lens-itself-as-activities)).

**Restoring** replaces the whole database, and the files under `data_dir`, with the backup's (a *database only*
backup leaves the files as they are). What was there is backed up first (`before-restore`) and
put back if the restore fails; a backup from an older Lens is brought up to date (schema and upgrade steps) afterwards.
Expand Down
1 change: 1 addition & 0 deletions docs/updates.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ When a check finds a release newer than the one running, Lens says so once per r
* **by email** to every admin (`updates.email`, on by default; needs an email server in Settings › Email);
* **to notification targets** (webhooks, Matterbridge, Slack, Discord) whose namespace owners chose the
**A new Lens release is out** event (`lens.release`, [Notifications](notifications.md));
* to automations, as a `lens.release` activity ([Automations](automations.md#lens-itself-as-activities));
* in the server log.

## Upgrading
Expand Down
20 changes: 19 additions & 1 deletion fastapi_backend/app/domain/backups.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,20 @@ def take(db):
dst = migrations.backup(db, "manual")
if dst is None:
return None
return next((b for b in listing(db) if b["name"] == dst.name), None)
got = next((b for b in listing(db) if b["name"] == dst.name), None)
if got:
_finished(db, "manual", got)
return got


def _finished(db, label, got=None, error=None, copied=None, at=None):
"""backup.finished for automations (events_streams.py); a backup is never undone for want of it."""
try:
from . import events_streams

events_streams.backup_finished(db, label, got, error, copied, at)
except Exception as e: # noqa: BLE001 - the backup stands without it
log.info("backups: couldn't log %s: %s", (got or {}).get("name") or label, e)


def delete(db, name):
Expand Down Expand Up @@ -341,9 +354,14 @@ def run_due(db, cfg, log=None, now=None):
)
if log:
log(f"scheduled backup: {type(e).__name__}: {e}")
_finished(db, "scheduled", error=f"{type(e).__name__}: {e}", at=st["at"])
return True
copied = None
if got and remote(cfg):
send_quietly(db, cfg, got["name"], log)
copied = not remote_state(db).get("error")
if got:
_finished(db, "scheduled", got, copied=copied)
return True


Expand Down
6 changes: 6 additions & 0 deletions fastapi_backend/app/domain/budgets.py
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,12 @@ def sweep(db, cfg, now=None, every=None):
activity.record(
"run", f"budget.{st['state']}", [res], cfg, db, detail={"share": st["share"], "estimate": st["spent"]["estimate"] or None}
)
try: # for automations: budget.warned (events_streams.py)
from . import events_streams

events_streams.budget_warned(db, b, alert, st)
except Exception as e: # noqa: BLE001 - the alert stands without it
log.info("budgets: couldn't log the warning for %s: %s", res, e)
made.append({"resource": res, **alert})
return made

Expand Down
6 changes: 3 additions & 3 deletions fastapi_backend/app/domain/events_content.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,13 @@

def _lookup(db, table, keys, fields="name", by="id"):
"""{key: row} for these keys of a table (record ids, or the values of the indexed field `by`), one query for the
ones not looked up in the last NAME_SECONDS."""
ones not looked up (with these `fields`) in the last NAME_SECONDS."""
keys = [k for k in dict.fromkeys(keys) if k is not None]
now, out, todo = time.monotonic(), {}, []
with _names_lock:
mine = _names.setdefault(db, {})
for k in keys:
hit = mine.get((table, by, k))
hit = mine.get((table, by, fields, k))
if hit and now - hit[0] < NAME_SECONDS:
if hit[1] is not None:
out[k] = hit[1]
Expand All @@ -83,7 +83,7 @@ def _lookup(db, table, keys, fields="name", by="id"):
if len(mine) > 20000:
mine.clear()
for k in todo:
mine[(table, by, k)] = (now, got.get(k))
mine[(table, by, fields, k)] = (now, got.get(k))
out.update(got)
return out

Expand Down
Loading
Loading
0