8000 feat(automations): timers that outlast a restart and fire once by adeelahmad · Pull Request #446 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content 7FFF
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 45 additions & 2 deletions docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,12 @@ merged is called built.
| Calling a web address: the `http_request` action, with credentials in headers kept sealed | built |
| Running Lens work: the `run_workflow`, `run_pipeline` and `run_routine` actions | built |
| Changing assets and writing notes: the `set_field`, `add_to_collection` and `write_note` actions | built |
| Settings → Automations: how often followers look and how long the event log is kept | built |
| Settings → Automations: how often followers look, how long the event log is kept, and how late a timer may fire | built |
| A webhook address per automation trigger, and calls to it as `webhook.received` ([Webhook trigger](#webhook-trigger)) | built |
| Timers that outlast a restart and fire once across the API and the worker (`app/domain/timers.py`, [Time](#time)) | built |
| Schedule, `at` and `relative` triggers, and actions that wait ([Time](#time)) | planned |
| The Timeline page with filters, grouping, review keys and new items, and Home's Latest card ([Timeline](timeline.md)) | built |
| Storing and running automations, schedules and timers, joins, approvals, variables, the studio | planned |
| Storing and running automations, joins, approvals, variables, the studio | planned |

## Contract

Expand Down Expand Up @@ -450,6 +452,45 @@ everyone who reads the asset sees.
Calendar events aren't written: Lens signs in to Google Calendar read-only, and writing needs a new consent. Meanwhile
the agent can draft an event as a note.

## Time

Time is a trigger and a step: "every weekday at 7:00", "at 9:00 on the 20th", "30 minutes before a calendar event",
"wait 2 hours, then...", and the clock behind a join's "if no reply within 2 days". All of it rests on timers.

### Timers

Built (`app/domain/timers.py`). A timer is one row per key in the `timer` table: when it is due, what runs then (its
kind), what that gets (its payload), and the automation it is for. Status: `waiting`, then `fired`, or `failed`,
`cancelled` or `missed`.

```python
timers.set(db, key, at, kind, payload, automation=None) # makes it, or replaces the one with that key
timers.cancel(db, key) # True when one was waiting
timers.cancel_for(db, automation) # every waiting one of an automation (deleted, switched off)
timers.get(db, key)

@timers.handler("join.absent") # what runs when one of this kind fires
def _(db, cfg, payload): ... # payload + late (true or false) + timer {key, at, automation}
```

- **Once.** A thread started with the routines' threads (so in the API's background work and in `lens worker`) looks
every 5 seconds, and at once after `events.wake()`. Each due timer is claimed with a compare-and-set update that
only one process wins, then its handler runs. A process that stops while a handler runs leaves that firing done:
a timer fires at most once, never twice.
- **Repeating.** A handler registered with `again(payload, after)` (a schedule's next time) is re-armed in the same
update that claims it, so a repeating timer is never lost; its row keeps waiting, with `last` saying how the last
firing went.
- **Lens was off.** A timer due within `automations.catch_up` (1 hour; Settings → Automations) fires late, with
`late: true`; an older one is `missed` and counted on its automation (`missed`). A repeating one that missed
several times fires once, for its latest time if that is within the window, then waits for its next. A timer fired
within 2 minutes of its time is on time.
- **Kept small.** An automation has at most 1000 timers waiting; setting more is refused, saying so. Timers that
ended are dropped after 7 days, by a tidy `routines.run_due` calls (once an hour). A kind nothing handles fails
with a note.
- **Light.** No timer exists until an automation uses time. While none waits the thread doesn't query the table: it
checks once a minute, and at once when this process sets one. A timer set by the other process (the API, when the
worker fires them) can so fire up to a minute late.

## Refine later

- More operators: `in`, `matches` (bounded regular expressions), `between`, times against now, `changed`.
Expand Down Expand Up @@ -486,6 +527,8 @@ the agent can draft an event as a note.
- Calling a web address: secrets in the address itself (a Slack or Discord webhook URL) sealed like headers; PATCH and
DELETE (DELETE would count as deleting); OAuth sign-in for an API; form-encoded bodies; retries with backoff;
networks allowed for automations apart from notifications.
- Time: sub-second timers woken by `events.wake()` alone; telling the other process (the worker) at once when the
API sets a timer, rather than within a minute.
- Running Lens work: a routine run's id passed on (today the scheduler starts it later); waiting for the job and passing
on its outputs; roles on collections counting for `process`; running a pipeline on every asset a join collected.
- Changing assets: asking first instead of failing when a move would show an asset to new people (risk that reads the
Expand Down
4 changes: 3 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -463,12 +463,14 @@ Where namespaces may send notifications ([Notifications](notifications.md)). Set

## Automations

The stream of activities automations follow ([Automations](automations.md#contract)). Settings → Automations:
The stream of activities automations follow ([Automations](automations.md#contract)), and their timers. Settings →
Automations:

| Setting | Default | |
|---|---|---|
| `events.poll_seconds` | 5 | how often followers look for new activities when nothing wakes them, 1–3600 |
| `events.keep_days` | 30 | how long activities with no row of their own (the event log) are kept, 1–3650 days |
| `automations.catch_up` | `1h` | how late a timer (a schedule, an action that waits) may still fire after Lens was off, `0s` to `7d`; older ones are missed ([Time](automations.md#time)) |

## Trusted proxies

Expand Down
4 changes: 3 additions & 1 deletion docs/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,9 @@ of the stream keeps where its next look starts in `event_cursor:<consumer>` (a p
it handles in `event_claim:<hash of consumer and key>`; a claim is a `CREATE` only one process wins, and claims go
after a day. Each person's review marks on the timeline ([Timeline](timeline.md#review-marks)) are
`event_mark:<hash of account and activity>`, a row only while one of its flags is set. An automation trigger's
webhook is `automation_hook:<n>` (its automation and trigger, the hash of its token, its signing secret sealed). So that it can be read
webhook is `automation_hook:<n>` (its automation and trigger, the hash of its token, its signing secret sealed).
Automations' timers ([Time](automations.md#time)) are `timer:<hash of key>` (by `status` and `at`, and by
`automation`); one that ended goes after 7 days. So that it can be read
without opening files, a resource made from an email keeps the message's addressing as it was listed
(`recording.mail`) and one made from a calendar event its times, place and people (`recording.event`); both are
optional, and older resources simply don't have them.
Expand Down
1 change: 1 addition & 0 deletions fastapi_backend/app/domain/ops_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
"reports",
"notifications",
"events",
"automations",
"workers",
"decisions",
"voice",
Expand Down
12 changes: 10 additions & 2 deletions fastapi_backend/app/domain/routines.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
sources,
store,
telemetry,
timers,
workflows,
)

Expand Down Expand Up @@ -609,6 +610,11 @@ def run_due(db, cfg, log=print, now=None):
except Exception as e: # noqa: BLE001 - the schedule carries on
if log:
log(f"events: {type(e).__name__}: {e}")
try:
timers.tidy(db, cfg) # timers that ended more than timers.KEEP_DAYS ago, once an hour
except Exception as e: # noqa: BLE001 - the schedule carries on
if log:
log(f"timers: {type(e).__name__}: {e}")
try:
analytics.tick(db, cfg, now) # once a day, while Google Analytics is on (docs/analytics.md)
except Exception as e: # noqa: BLE001 - the schedule carries on
Expand Down Expand Up @@ -651,8 +657,9 @@ def run_due(db, cfg, log=print, now=None):

def start(db, cfg_fn, stop, log=None):
"""The scheduling threads, until `stop` is set: watched folders scanned when due (every sources.check_seconds),
routines run when due (every CHECK_SECONDS) and the scheduled backup taken when due. The API runs them with its background work; `lens worker` runs them too,
since Docker and the packages run the API without it."""
routines run when due (every CHECK_SECONDS), the scheduled backup taken when due, and automations' timers fired
when due (timers.py). The API runs them with its background work; `lens worker` runs them too, since Docker and
the packages run the API without it."""

def every(seconds, fn, what):
def loop():
Expand All @@ -673,6 +680,7 @@ def loop():
every(lambda: cfg_fn()["fedora"]["sync_seconds"], fedora.sync_due, "fedora"),
every(lambda: notebook.QUIET_SECONDS / 2, notebook.organise_due, "notes"),
every(lambda: CHECK_SECONDS, backups.run_due, "backups"),
timers.start(db, cfg_fn, stop, log=log),
]


Expand Down
22 changes: 22 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
"updates": ("check", "check_hours", "auto_install", "email"),
"activity": None,
"events": None,
"automations": None,
"backups": None,
"encryption": None,
"fedora": None,
Expand Down Expand Up @@ -479,6 +480,8 @@ def _check(section, key, value, default):
return _activity_setting(key, value)
if section == "events":
return _events_setting(key, value)
if section == "automations":
return _automations_setting(key, value)
if section == "backups":
return _backup_setting(key, value)
if section == "fedora":
Expand Down Expand Up @@ -1028,6 +1031,25 @@ def _events_setting(key, value):
return value


CATCH_UP_MAX = 7 * 86400


def _automations_setting(key, value):
"""Automations (docs/automations.md): catch_up is a duration like 1h, from 0s (a timer missed while Lens was off
is missed) to 7d."""
from . import automation_spec

if key != "catch_up":
raise ValueError(f"unknown setting automations.{key}")
try:
n = automation_spec.check_duration(value, what="automations.catch_up")
except ValueError:
n = -1
if not 0 <= n <= CATCH_UP_MAX:
raise ValueError("automations.catch_up is a duration from 0s to 7d, like 1h")
return value.strip()


def _backup_setting(key, value):
from . import schedule

Expand Down
8 changes: 8 additions & 0 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -622,6 +622,9 @@ def labels():
# the stream of activities automations follow (docs/automations.md): keep_days: how long activities with no row
# of their own (the event log) are kept; poll_seconds: how often followers look when nothing wakes them
"events": {"keep_days": 30, "poll_seconds": 5},
# automations (docs/automations.md): catch_up: how late a timer (a schedule, a waiting action) may still fire
# after Lens was off; older ones are missed
"automations": {"catch_up": "1h"},
# database backups (docs/database.md, Backups): schedule is a cron expression in timezone, or none (the default)
# for none; the newest `keep` scheduled backups are kept. connection: a storage connection (sources.py) every new
# backup is also copied to, under folder, encrypted with a key from secret.key first (none: kept here only)
Expand Down Expand Up @@ -1268,6 +1271,11 @@ def close(self):
"DEFINE TABLE IF NOT EXISTS automation_hook SCHEMALESS",
"DEFINE INDEX IF NOT EXISTS automation_hook_token ON automation_hook FIELDS token_hash UNIQUE",
"DEFINE INDEX IF NOT EXISTS automation_hook_automation ON automation_hook FIELDS automation",
# timers (domain/timers.py, docs/automations.md, Time): timer:<hash of key>, what runs when, for which automation
"DEFINE TABLE IF NOT EXISTS timer SCHEMALESS",
"DEFINE INDEX IF NOT EXISTS timer_due ON timer FIELDS status, at",
"DEFINE INDEX IF NOT EXISTS timer_automation ON timer FIELDS automation, status",
"DEFINE INDEX IF NOT EXISTS timer_done ON timer FIELDS done_at",
"DEFINE INDEX IF NOT EXISTS upload_recording ON upload FIELDS recording",
# review marks on the timeline (domain/event_marks.py, docs/timeline.md): each person's reviewed, important and
# hidden flags on an activity, event_mark:<hash of account and activity> (the id keeps the pair unique)
Expand Down
Loading
Loading
0