8000 feat(automations): templates in params and prompts, with outside text kept marked by adeelahmad · Pull Request #454 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 102 additions & 5 deletions docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ merged is called built.
| The action registry with risk from each action's effect, and one action, `log_line` (`app/domain/actions.py`) | built |
| Telling a person: the `notify` action and `channels.tell` (their own chat, a notification target, a Matterbridge room) | built |
| The model and the agent: the `ask_model`, `call_tool` and `ask_agent` actions, and prompts with outside text marked | built |
| Templates in params and prompts: the run context, one expression keeping its type, errors that name the nearest variable, saved prompt templates, filters for times, text and addresses ([Variables and templates](#variables-and-templates)) | built |
| Saved variables (`vars`, the `set_variable` action), the variable picker's catalog and a preview of what a template gives | planned |
| Calling a web address: the `http_request` action, with credentials in headers kept sealed | built |
| Running Lens work: the `run_workflow`, `run_pipeline` and `run_routine` actions | built |
| Changing assets and writing notes: the `set_field`, `add_to_collection` and `write_note` actions | built |
Expand Down Expand Up @@ -411,7 +413,7 @@ titles, for the variable picker):
| `run` | `{id, dry}` |
| `namespace` | `{id, name}` |
| `user` | who the automation acts for: `{id, name, email}` |
| `now`, `timezone` | when the run started (ISO 8601 in UTC) and the person's time zone |
| `now`, `now_local`, `timezone` | when the run started, in UTC and in the run's time zone, and that time zone |
| `vars` | the automation's and its namespace's saved variables |
| `steps` | what each node passed on so far, by node id, such as `steps.n3` |

Expand Down Expand Up @@ -537,10 +539,11 @@ count in the activity ledger under the run, and stop once a budget they count fo
is used up.

Params marked `"format": "prompt"` in an action's schema (`ask_model`'s `prompt`, `ask_agent`'s `instructions`) are
prompts: they are rendered with `actions.render_prompt(text, ctx, cfg)`, which marks as data (`<untrusted-…>`) what
the run read from outside Lens before the model sees it: each activity's `lens.untrusted` fields (an email's subject
and body) and everything earlier steps passed on (a model's reply, a web page). Your own words in the prompt stay as
written. `ask_agent` passes the activity it is about the same way.
prompts: they are rendered with `variables.prompt` ([Variables and templates](#prompts)), which marks as data
(`<untrusted-…>`) what the run read from outside Lens before the model sees it: each activity's `lens.untrusted`
fields (an email's subject and body), everything earlier steps passed on (a model's reply, a web page) and saved
variables. Your own words in the prompt stay as written. A prompt param may name a saved prompt template instead of
its text. `ask_agent` passes the activity it is about the same way.

#### Calling a web address

Expand Down Expand Up @@ -746,6 +749,96 @@ are built; until then it waits, saying why). The run hears of each step through
with what it gave back, failed with why, waiting, or expired when the automation was switched off or removed first,
the timer was cancelled, or Lens was off past the catch-up window.

## Variables and templates

Built (`app/domain/variables.py`). An action's params and prompts are templates over the [run
context](#run-context): sandboxed Jinja, the same sandbox as Lens's other templates (`templates.py`: no reach into
Python, no huge strings, a 2 MB cap on what one renders). A template reads everything the run knows, so one line can
join two sources: `{{ events.mail.object.subject }} on {{ events.soon.object.startTime | date('%a %H:%M') }}`.

```python
variables.context(db, cfg, automation, run, trigger, event, events, user, now=None) # the run context
variables.render(value, ctx, untrusted_paths=None, where="") # params, dicts and lists walked
variables.prompt(template, ctx, untrusted_paths=None, cfg=None, db=None) # {text, rules, system, schema, template}
```

### The run context

The keys are those of [Run context](#run-context). `now` is when the run started in UTC (`2026-10-15T16:22:09Z`) and
`now_local` the same moment in the run's time zone (`2026-10-15T17:22:09+01:00`). The time zone is the trigger's
(a schedule's or an `at`'s), else the first one the automation's triggers name, else UTC. `namespace` is the event's
namespace when the automation works there, else its first.

### Params

- A param that is exactly one expression keeps its type: `"{{ steps.n3.count }}"` gives the number 3,
`"{{ event.object.from }}"` a list, `"{{ event }}"` the whole activity (a copy). Anything else renders as text.
- Dicts and lists are walked, so a param such as `{"to": ["{{ user.email }}"]}` is filled in where it stands.
- `a.b` reads a dict's key `b` first, then a method of that name, so an action's output `{"items": [...]}` reads as
`steps.n3.items` (Lens's other templates keep Jinja's own order).
- An error names the param and the variable that isn't there, with the nearest name there is, and the template's
line: `text: events.mail.object.subjct is not defined; did you mean subject?`, or `to: user.zzz is not defined; it
has email, id, name`. A path guarded by `default` or `is defined` doesn't count.
- Text from outside Lens keeps its words in params (a message forwards the subject as it was written), without the
invisible characters that would smuggle instructions to a model that reads it later. The intent still says it
read outside text (`untrusted`), and outbound actions treat it so.

### Prompts

A prompt (a param with `"format": "prompt"`) renders with every value read from outside Lens marked as data with
`untrusted.wrap`: the source is the activity's kind and actor (`email.received from Work Gmail`), and anything that
looks like a marker or a chat model's control token is broken up. That covers each activity's `lens.untrusted`
fields, everything earlier nodes passed on (`steps`) and saved variables (`vars`), which a run may have set from
outside text. The person's own words, Lens's own fields (a summary, ids, times) and numbers stay as written.
Filters and string methods work on the words of marked text and give back marked text, so `| plain | truncate(4000)`
or `.upper()` can't drop or cut the markers; comparisons (`==`, `in`, `length`) read the words themselves, so `{% if
event.object.subject == "Trip" %}` works as written.
`rules` is the system prompt's paragraph about marked text, to add to the system prompt (the AI actions add it
themselves); with `ai.untrusted_marking` off nothing is marked and `rules` is empty.

A prompt param may name a saved prompt template (Settings → Templates, `POST /api/v1/templates`) instead of its text:
`{"template": 5}` for its current version, `{"template": 5, "version": 2}` to pin one. It is checked when the
workflow is saved (the template must be there and be a prompt template). The template's own instructions and output
schema fill `ask_model`'s `system` and `json_schema` when the node doesn't give them, so the same prompt serves
pipelines and automations.

### Filters

Every template in Lens has these, on top of Jinja's own (`truncate`, `first`, `join`, `default`...) and Lens's `json`
and `tc`:

| Filter | Gives |
| --- | --- |
| `date(fmt, tz=None)` | a time written with a `strftime` format, in `tz` (default the run's time zone): `{{ x \| date('%a %d %b, %H:%M') }}` |
| `local` | a time in the run's time zone, as ISO 8601 with its offset |
| `ago` | how long ago a time was, from the run's `now`: `3 hours ago`, `in 2 days`, `just now` |
| `plain` | HTML as plain text, a line per block |
| `emails` | the email addresses in text (or a list of texts), lowercased, each once |
| `defang` | web addresses broken up so a chat app doesn't fetch them (`hxxps://evil[.]example/`) |

Times may be ISO 8601 text (a `Z`, an offset, or none for UTC; a date alone stays a date), epoch seconds or
milliseconds.

### Examples

A message joining two triggers (a `notify` node's `text`):

```text
{{ events.mail.object.subject }} from {{ events.mail.object.from | emails | first }}: you have
{{ events.soon.object.name }} {{ events.soon.object.startTime | ago }} ({{ events.soon.object.startTime | date('%a %H:%M') }}).
```

A prompt with outside text marked (`ask_model`'s `prompt`, with a `json_schema` asking for `{trip, date}`):

```text
Does this email from {{ event.object.from | join(', ') }} ask us to sign anything for a school trip?
Subject: {{ event.object.subject }}
{{ event.object.content | plain | truncate(4000) }}
```

The model sees the subject and the body between `<untrusted-…>` markers, with the rules that say marked text is
material to read, not instructions to follow; your question stays as written.

## Refine later

- More operators: `in`, `matches` (bounded regular expressions), `between`, times against now, `changed`.
Expand Down Expand Up @@ -778,6 +871,10 @@ the timer was cancelled, or Lens was off past the catch-up window.
message the person's template wrote, so they stay when the run read outside text.
- The canvas draws and edits event workflows (today they are saved and tried through the API).
- An action's outputs in the variable picker, typed from its `output` schema.
- Variables and templates: secrets as variables (sealed, never rendered into prompts); typed variable declarations
with defaults per automation; marking only the saved variables that were set from outside text (today every saved
variable's text is marked in prompts); a time zone of each person's own, rather than the triggers'; wrapping
outside text the same way in `tool_nodes.ask_model` (pipeline workflows) and extensions' prompt tools.
- The model and the agent: per-node model picks in the studio; streaming an agent's steps into the run's log as they
happen; a namespace assistant's memory for automations (`remember` is left out today); approving an agent's
proposals from the chat app it told you in; count the automation's own budget once budgets take `automation:<id>`.
Expand Down
68 changes: 17 additions & 51 deletions fastapi_backend/app/domain/actions.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,20 @@
an intent out, or with `dry` says what it would do and changes nothing. An action acts for the automation's person
(`person`), with their permissions as they are when it runs (`reach`), and what it writes is by the automation itself
(`actor`). Params marked `"format": "prompt"` are prompts: the engine renders them with `render_prompt`, which wraps
what the run read from outside Lens as data. `log_line` ships here; the other actions are registered by a module per
group, loaded with this one: actions_messages.py (notify), actions_ai.py (ask_model, call_tool, ask_agent),
what the run read from outside Lens as data; such a param may name a saved prompt template ({"template": 5}) instead.
`log_line` ships here; the other actions are registered by a module per group, loaded with this one: actions_messages.py (notify), actions_ai.py (ask_model, call_tool, ask_agent),
actions_web.py (http_request), actions_run.py (run_workflow, run_pipeline, run_routine), actions_lens.py (set_field, add_to_collection,
write_note).
"""

from __future__ import annotations

import copy
import logging
import re
from collections.abc import Callable
from dataclasses import dataclass

from . import auth, store, templates, untrusted
from . import auth, store, templates
from . import automation_spec as spec_

NAME_RX = re.compile(r"^[a-z][a-z0-9_]{0,40}$")
Expand Down Expand Up @@ -131,7 +130,12 @@ def check_params(spec, params):
if k in spec.required:
raise ValueError(f"{spec.name} needs {p.get('title') or k} ({k})")
continue
if not _templated(v):
if p.get("format") == PROMPT and isinstance(v, dict):
try:
templates.reference(v) # a saved prompt template in place of the text
except templates.TemplateProblem as e:
raise ValueError(f"{spec.name}: {k}: {e}") from None
elif not _templated(v):
want = TYPES.get(p.get("type"))
if want and (not isinstance(v, want) or (isinstance(v, bool) and p.get("type") in ("integer", "number"))):
raise ValueError(f"{spec.name}: {k} is {'an' if p.get('type')[0] in 'aeiou' else 'a'} {p.get('type')}")
Expand Down Expand Up @@ -299,54 +303,16 @@ def prompt_params(spec):
return [k for k, p in (spec.params.get("properties") or {}).items() if p.get("format") == PROMPT]


def _wrap(v, source, cfg):
if isinstance(v, str):
return untrusted.wrap(v, source, cfg)
if isinstance(v, list):
return [_wrap(x, source, cfg) for x in v]
if isinstance(v, dict):
return {k: _wrap(x, source, cfg) for k, x in v.items()}
return v


def _wrapped(a, cfg):
"""An activity with what its `lens.untrusted` paths hold marked as data."""
out = copy.deepcopy(a)
source = f"{a.get('kind') or 'activity'} from {(a.get('actor') or {}).get('name') or 'a source'}"
for path in (a.get("lens") or {}).get("untrusted") or []:
*head, last = str(path).split(".")
node = out
for k in head:
node = (
node[int(k)]
if isinstance(node, list) and k.isdigit() and int(k) < len(node)
else node.get(k)
if isinstance(node, dict)
else None
)
if node is None:
break
if isinstance(node, dict) and last in node:
node[last] = _wrap(node[last], source, cfg)
elif isinstance(node, list) and last.isdigit() and int(last) < len(node):
node[int(last)] = _wrap(node[int(last)], source, cfg)
return out

def render_prompt(text, ctx, cfg=None, db=None):
"""A prompt param rendered against the run context (variables.prompt), with what the run read from outside Lens
marked as data first (untrusted.wrap): each activity's `lens.untrusted` paths, everything earlier steps passed on
(a model's reply, a web page) and saved variables. The person's own words stay as written. `text` may name a saved
prompt template ({"template": 5, "version": 2}), read with `db`. ValueError naming the problem."""
from . import variables

def render_prompt(text, ctx, cfg=None):
"""A prompt param rendered against the run context (templates.render_body), with what the run read from outside
Lens marked as data first (untrusted.wrap): each activity's `lens.untrusted` paths and everything earlier steps
passed on (a model's reply, a web page). The person's own words stay as written. ValueError naming the problem."""
safe = dict(ctx or {})
if isinstance(safe.get("event"), dict):
safe["event"] = _wrapped(safe["event"], cfg)
if isinstance(safe.get("events"), dict):
safe["events"] = {k: _wrapped(a, cfg) if isinstance(a, dict) else a for k, a in safe["events"].items()}
if isinstance(safe.get("steps"), dict):
safe["steps"] = _wrap(safe["steps"], "an earlier step of this automation", cfg)
try:
return templates.render_body(str(text or ""), safe, "prompt")
except templates.TemplateProblem as e:
return variables.prompt(text, ctx, None, cfg, db)["text"]
except ValueError as e:
raise ValueError(f"the prompt: {e}") from None


Expand Down
Loading
Loading
0