adeelahmad · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026 · Oct 18, 2026
diff --git a/README.md b/README.md
@@ -155,7 +155,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
   [checks](docs/requirements.md#learn-from-it))
 - **Automate it.** Pipelines and workflows on a canvas (with Convert steps that make a video's soundtrack an audio
   asset and a document's pages image assets), routines on a schedule, batch runs over a collection, and
-  notifications to webhooks, Slack, Discord and Matterbridge. Everything that happens in Lens (new assets, emails,
+  notifications to webhooks, Slack, Discord and Matterbridge, including when a routine's run fails or leaves graph
+  changes to review. Everything that happens in Lens (new assets, emails,
   calendar events, files, feed posts, what people say to Lens, sensor readings, notes, graph changes, runs, budget
   warnings) is kept as a stream of activities in W3C Activity Streams form, which agents read through the API, MCP and
   the assistant, and people review on the Timeline page (and Latest on Home), marking what they've seen. Automations act

diff --git a/changelog.d/560.md b/changelog.d/560.md
@@ -0,0 +1,5 @@
+- **Routine run alerts.** A routine's run that fails (all its actions, some of them, or one that stops without
+  finishing) or leaves graph changes to review can now be sent through Matterbridge, to a signed webhook, or to Slack
+  and Discord: pick the new `routine.failed` and `routine.proposed` events for a namespace's notification target. Each
+  namespace the routine runs over gets its own message, linking the routine or the run's changes; new targets' default
+  events are unchanged ([Notifications](docs/notifications.md#events)).
@@ -1,7 +1,7 @@
 # Notifications
 
 Lens can tell a chat room or another app when something happens in a namespace: a run finishes or fails, a batch
-run finishes, something is added. Each namespace's owners choose where its notifications go (its **targets**) and
+run finishes, something is added, a routine's run fails or leaves graph changes to review. Each namespace's owners choose where its notifications go (its **targets**) and
 which events each target gets, on the namespace's page under **Notifications**.
 
 ## Targets
@@ -93,14 +93,16 @@ check that the timestamp is recent too, to refuse replays.
 | `recording.added` | an asset (a recording, document, image or web page) was added, however it came | `recording`, `title`, `source`, `created_at` |
 | `lens.release` | a new Lens release is out, once per release ([Updates](updates.md)) | `tag`, `name`, `current`, `published_at`, `notes_url` |
 | `approval.waiting` | an automation's action waits for its person's pick ([Approvals](automations.md#approvals)); one decided before the notifier saw it isn't sent | `approval`, `automation`, `automation_name`, `summary`, `action`, `effect`, `expires_at`, `created_at` |
+| `routine.failed` | a [routine](processing.md#routines)'s run failed: all its actions, some of them (`text` says how many), or it stopped without finishing; each namespace the routine runs over gets it | `routine`, `routine_name`, `run`, `trigger`, `status` (`error` or `partly`), `failed` (each failed action's `action` number, `type` and `error`), `actions`, `error`, `started_at`, `finished_at` |
+| `routine.proposed` | a routine's run left graph changes to accept or dismiss, counted in each namespace they touch; one whose changes were all decided before the notifier saw it isn't sent | `routine`, `routine_name`, `run`, `trigger`, `status`, `proposed`, `kinds` (`merge`, `link`), `started_at`, `finished_at` |
 
 New targets get `job.failed` and `batch.finished`. A batch run's own runs aren't sent one by one: the batch run is,
 when it finishes. More than five things added to a namespace at once (a watched folder's new files, an import) are one
 message (`data.recordings`, `data.count`).
 
-`text` is the message chat targets get, and `url` links to the run, batch run, asset or approval in the web app. What
-an approval would do holds text its run read from outside Lens, so web addresses in its `text` are broken up
-(`hxxps://evil[.]example`), except the web app's own.
+`text` is the message chat targets get, and `url` links to the run, batch run, asset, approval, routine or a routine
+run's graph changes in the web app. What an approval would do, and a routine's errors, can hold text read from outside
+Lens, so web addresses in their `text` are broken up (`hxxps://evil[.]example`), except the web app's own.
 
 ## How it's sent
 

@@ -790,7 +790,9 @@ The schedule is a five-field cron expression (minute hour day-of-month month day
 `@hourly`, `@daily`, `@weekly`, `@monthly`, in a time zone; without one a routine runs only when someone presses Run
 now. Every 30 seconds, `lens worker` checks (and so does the API process when it runs background work, and `lens watch`); a routine is
 never started twice at once, and one started in two processes runs once. Each run keeps what every action did and a
-log.
+log. A run that fails, or leaves graph changes to review, can be sent to a chat room through
+Matterbridge or to a webhook: pick `routine.failed` or `routine.proposed` for a namespace's target
+([Notifications](notifications.md#events)).
 
 The [Schedules](schedules.md) page (Automate › Schedules) lists every routine next to the automations' schedules and
 the backup, to switch, re-time and run each from one place; the Routines page and New routine open from its top.

@@ -192,6 +192,7 @@ request, to another check that proves the same thing; loosening a requirement or
 | Everything that runs on a clock (routines, automations' schedule and at triggers, the backup) in one Schedules list, switched, re-timed and run from it, by people and by agents over the API and MCP | [test_schedules.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_schedules.py) `test_one_list_holds_every_routine_automation_schedule_and_the_backup`, `test_schedules_are_changed_switched_and_run_from_the_list`, `test_agents_list_change_and_run_schedules_over_mcp`; [schedules-page.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/schedules-page.test.tsx) `lists every routine, automation schedule and the backup, each linking to its own page`, `switches one off, runs one now and changes when one runs, from the list` |
 | Batch runs over a collection | [test_batches.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_batches.py) `test_collections_and_batches` |
 | Notifications to webhooks, Slack, Discord and Matterbridge | [test_notifications.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_notifications.py) `test_owners_manage_targets_and_a_test_is_signed`, `test_runs_that_end_are_sent_once_to_matterbridge` |
+| A routine's run that fails or leaves graph changes to review is told through Matterbridge and webhooks, to each namespace it runs over | [test_notifications.py](https://github.com/adeelahmad/lense/blob/main/fastapi_back
8000
end/tests/api/test_notifications.py) `test_routine_runs_that_fail_or_leave_changes_to_review_are_told` |
 | Everything that happens is kept as W3C Activity Streams activities, read by agents through the API, MCP and the assistant | [test_events.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_events.py) `test_activities_have_one_shape`; [test_activities.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_activities.py) `test_the_timeline_lists_new_assets_newest_first`, `test_agents_read_the_timeline_too`; [test_news_feed_source.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_news_feed_source.py) `test_a_news_feed_as_a_source` |
 | The Timeline page and Latest on Home, marking what you've seen | [test_timeline_marks.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_timeline_marks.py) `test_marks_are_set_cleared_and_filter_the_timeline`; [timeline-ui.test.tsx](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/timeline-ui.test.tsx) `Home's Latest card lists what you haven't reviewed` |
 | Automations act on the stream, on a schedule, at a time or when their own webhook is called | [test_automation_runs.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_automation_runs.py) `test_each_new_activity_runs_each_automation_it_matches_once`, `test_a_schedule_runs_its_automation_when_its_timer_fires`, `test_a_call_to_a_webhook_runs_its_automation`; [test_timer_triggers.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/domain/test_timer_triggers.py) `test_an_at_trigger_fires_once_then_reads_done` |

@@ -10,8 +10,9 @@
   {"content"} for Discord).
 
 Nothing in the rest of the app calls in here when something happens. A notifier thread in every process that runs
-background work (the API's, `lens worker`) reads what changed since it last looked: runs that ended (job.finished_at)
-and recordings added (recording.created_at). Each event is claimed once across every process (notify_event:<hash of
+background work (the API's, `lens worker`) reads what changed since it last looked: runs that ended (job.finished_at),
+recordings added (recording.created_at), automations' approvals (approval.created_at) and routines' runs that failed or
+left graph changes to review (routine_run.finished_at). Each event is claimed once across every process (notify_event:<hash of
 its key>, which only one CREATE wins, in the same transaction that queues it per target in notify_delivery), then
 sent, with retries. Each source is read in (time, id) order from where the last look stopped, a page at a time; once
 caught up, a look also reads the last minute again, so a row written just as it looked isn't missed, and the claims
@@ -50,6 +51,8 @@
     "recording.added": "Something was added",
     "lens.release": "A new Lens release is out",
     "approval.waiting": "An automation's action waits for someone's approval",
+    "routine.failed": "A routine's run failed",
+    "routine.proposed": "A routine's run left graph changes to review",
 }
 DEFAULT_EVENTS = ["job.failed", "batch.finished"]
 KINDS = {
@@ -569,22 +572,31 @@ def _event(kind, key, space, ns, text, url, data):
 JOB_FIELDS = "recording, space, batch, steps, step_index, status, error, finished_at"
 REC_FIELDS = "space, title, source, created_at"
 APPROVAL_FIELDS = "space, summary, automation, origin, intent.action AS action, intent.effect AS effect, expires_at, created_at"
+RUN_FIELDS = "routine, trigger, status, results, error, started_at, finished_at"
 SOURCES = {
     # source: (table, its time column, the fields read, more to match)
-    "jobs": ("job", "finished_at", JOB_FIELDS, "AND status IN ['succeeded', 'failed', 'cancelled']"),
-    "recordings": ("recording", "created_at", REC_FIELDS, ""),
-    "approvals": ("approval", "created_at", APPROVAL_FIELDS, "AND intent != NONE AND status = 'pending'"),  # automations' (approvals.py)
+    "jobs": ("job", "finished_at", JOB_FIELDS, "AND space IN $sp AND status IN ['succeeded', 'failed', 'cancelled']"),
+    "recordings": ("recording", "created_at", REC_FIELDS, "AND space IN $sp"),
+    # automations' (approvals.py)
+    "approvals": ("approval", "created_at", APPROVAL_FIELDS, "AND space IN $sp AND intent != NONE AND status = 'pending'"),
+    # routines' (routines.py), which run over namespaces rather than in one: _routine_events says which
+    "routines": ("routine_run", "finished_at", RUN_FIELDS, "AND (status IN ['error', 'partly'] OR changes.proposed > 0)"),
 }
 JOB_EVENTS = {"job.succeeded", "job.failed", "job.cancelled", "batch.finished"}
-SOURCE_EVENTS = {"jobs": JOB_EVENTS, "recordings": {"recording.added"}, "approvals": {"approval.waiting"}}
+SOURCE_EVENTS = {
+    "jobs": JOB_EVENTS,
+    "recordings": {"recording.added"},
+    "approvals": {"approval.waiting"},
+    "routines": {"routine.failed", "routine.proposed"},
+}
 
 
 def _rows(db, source, pos, floor, spaces, since):
     """A source's rows past `pos` ([time, id]) in (time, id) order, at most PAGE; and whether that page was full.
     A page that isn't full has caught up, so the rows of the last LOOKBACK seconds (`since`) are read again too, for
     one written as the last look ran; their claims keep them from being sent twice."""
     table, col, fields, more = SOURCES[source]
-    head = f"SELECT id AS key, record::id(id) AS id, {fields} FROM {table} WHERE {col} > $f AND space IN $sp {more}"
+    head = f"SELECT id AS key, record::id(id) AS id, {fields} FROM {table} WHERE {col} > $f {more}"
     page = db.rows(
         f"{head} AND ({col} > $t OR ({col} = $t AND id > $i)) ORDER BY {col}, key LIMIT {PAGE}",
         f=floor,
@@ -710,6 +722,71 @@ def _approval_events(cfg, rows, names):
     return out
 
 
+def _plural(n, one, many):
+    return f"{n} {one if n == 1 else many}"
+
+
+def _routine_events(db, cfg, rows, spaces, names):
+    """routine.failed (a routine's run whose actions all or partly failed, or that stopped without finishing) and
+    routine.proposed (one that left graph changes for someone to accept or dismiss), told to each namespace with
+    targets that the routine runs over (every one, for a routine over all of them). Changes are counted per namespace
+    they touch, as they stand now: a run whose proposals were all decided before the notifier saw it isn't sent. Its
+    errors can hold what the run read from outside Lens, so web addresses in the message are broken up, but Lens's own."""
+    from . import untrusted
+
+    owners = {
+        r["id"]: r
+        for r in db.rows(
+            "SELECT record::id(id) AS id, name, namespaces FROM routine WHERE id IN $ids",
+            ids=[R("routine", i) for i in {r["routine"] for r in rows if isinstance(r.get("routine"), int)}],
+        )
+    }
+    waiting: dict = {}  # run: {space: {kind: proposed changes}}
+    for c in db.rows("SELECT run, kind, spaces FROM graph_change WHERE run IN $ids AND status = 'proposed'", ids=[r["id"] for r in rows]):
+        for s in c.get("spaces") or []:
+            kinds = waiting.setdefault(c["run"], {}).setdefault(s, {})
+            kinds[c["kind"]] = kinds.get(c["kind"], 0) + 1
+    out = []
+    for r in rows:
+        ro = owners.get(r.get("routine"))
+        if not ro:
+            continue  # deleted with its runs as the notifier looked
+        name, over = ro.get("name") or f"Routine {ro['id']}", spaces if ro.get("namespaces") is None else ro["namespaces"]
+        results = r.get("results") or []
+        bad = [
+            {"action": k, "type": a.get("type"), "error": a.get("error")} for k, a in enumerate(results, 1) if a.get("status") == "error"
+        ]
+        base = {
+            "routine": ro["id"],
+            "routine_name": name,
+            "run": r["id"],
+            "trigger": r.get("trigger"),
+            "status": r["status"],
+            "started_at": r.get("started_at"),
+            "finished_at": r["finished_at"],
+        }
+        for sp in sorted(set(over) & set(spaces)):
+            ns = names.get(sp)
+            if r["status"] in ("error", "partly"):
+                why = "; ".join(f"{b['type']}: {b['error']}" for b in bad[:3]) or r.get("error") or "no reason given"
+                if r["status"] == "partly":
+                    head = f"{len(bad)} of {_plural(len(results), 'action', 'actions')} of routine “{name}” failed in {ns}"
+                else:
+                    head = f"Routine “{name}” failed in {ns}"
+                key = f"routine_run-{r['id']}-{sp}-failed"
+                data = {**base, "failed": bad, "actions": len(results), "error": r.get("error")}
+                text = untrusted.defang(f"{head}: {why}", own=[app_url(cfg)])
+                out.append(([key], _event("routine.failed", key, sp, ns, text, _link(cfg, f"/routines/{ro['id']}"), data)))
+            kinds = (waiting.get(r["id"]) or {}).get(sp)
+            if kinds:
+                n = sum(kinds.values())
+                key = f"routine_run-{r['id']}-{sp}-proposed"
+                data = {**base, "proposed": n, "kinds": kinds}
+                text = f"Routine “{name}” left {_plural(n, 'graph change', 'graph changes')} to review in {ns}."
+                out.append(([key], _event("routine.proposed", key, sp, ns, text, _link(cfg, f"/routines/changes?run={r['id']}"), data)))
+    return out
+
+
 _CLAIMED: dict[str, None] = {}  # keys this process knows are claimed, so it doesn't ask the database again
 _CL = threading.Lock()
 
@@ -810,6 +887,8 @@ def scan(db, cfg):
                 events += _batch_events(db, cfg, bids, spaces, names)
         elif source == "approvals":
             events = _approval_events(cfg, rows, names)
+        elif source == "routines":
+            events = _routine_events(db, cfg, rows, spaces, names) if rows else []
         else:
             fresh = _unclaimed(db, [f"recording-{r['id']}-added" for r in rows])
             events = _added_events(cfg, [r for r in rows if f"recording-{r['id']}-added" in fresh], names)

@@ -9,7 +9,17 @@
 from app.schemas.common import RequestModel, ResponseModel
 
 TargetKind = Literal["webhook", "matterbridge", "slack", "discord"]
-EventType = Literal["job.succeeded", "job.failed", "job.cancelled", "batch.finished", "recording.added", "lens.release", "approval.waiting"]
+EventType = Literal[
+    "job.succeeded",
+    "job.failed",
+    "job.cancelled",
+    "batch.finished",
+    "recording.added",
+    "lens.release",
+    "approval.waiting",
+    "routine.failed",
+    "routine.proposed",
+]
 
 
 class LastSend(ResponseModel):