8000 feat(automations): match trees for conditions by adeelahmad · Pull Request #425 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions docs/automations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Automations

Automations are the workflow studio: Lens follows what happens in every source as a stream of time-coded activities,
and an automation watches that stream and the clock, tests its conditions, and acts (directly for routine things,
through the agent, or by asking you first). The timeline is the same stream, filtered and reviewed.

Status: in progress. The studio is built in modules; this page labels each part built or planned, and only what is
merged is called built.

| Part | Status |
| --- | --- |
| Match trees: one condition language for triggers, an automation's If and workflow nodes | built |
| Activities, the kinds catalog, the event log, following the stream, the timeline API | planned |
| Automations, triggers, joins, schedules, actions, approvals, variables, the timeline page, the studio | planned |

## Contract

Every part of the studio builds on the shapes below. A change to one of them comes in a small pull request of its
own that changes the code, this section and the tests together.

### Match trees

Built (`app/domain/match.py`). One condition language for a trigger's `where`, an automation's `if`, and the
condition, filter and switch nodes of a workflow:

```
M := {"all": [M, …]} | {"any": [M, …]} | {"not": M} | {"path": "a.b.0", "op": OP, "value": V}
```

- A test (`{path, op, value}`) reads the part of the value at a dotted path, list items by number
(`object.from.0`); no path tests the value itself. The tests are the workflow engine's: `exists`, `empty`,
`equals`, `not_equals`, `contains` (text in any case, or an item of a list), `gt` and `lt` (numbers).
- `all` holds when every condition in it does (an empty `all` always holds), `any` when one does (it needs at least
one), `not` when its condition doesn't.
- Values are literals, never templates, so matching stays cheap and safe.
- A tree nests at most 6 deep and holds at most 50 tests. A problem is named with where it is: `the condition,
all.1.not: compare with a number`.
- `match.describe(tree)` reads a tree in words: `object.from contains “@stmarys.org” and (object.subject contains
“trip” or object.subject contains “outing”)`.

In a workflow, a condition, filter or switch case takes either one test, as before, or a tree in `match`; with a
tree, `path` (when given) is the part of the value the tree tests. Graphs saved before keep running unchanged.

```json
{"id": "school", "type": "condition",
"config": {"path": "object", "match": {"all": [{"path": "from.0", "op": "contains", "value": "@stmarys.org"},
{"not": {"path": "attachments", "op": "gt", "value": 5}}]}}}
```

## Refine later

- More operators: `in`, `matches` (bounded regular expressions), `between`, times against now, `changed`.
- The canvas shows and edits match trees on condition, filter and switch nodes (today it edits one test; a tree
saved through the API is kept).
3 changes: 2 additions & 1 deletion docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -489,7 +489,8 @@ node passes what it makes along its connections.
corrects or adds to any passed in; Merge joins lists; Save entities makes them the asset's entities (people's
corrections kept) and redoes keywords and chapters.
- **AI and logic:** LLM prompt (a prompt template, with what came in as `{{ input }}`), Pick (a path like
`action_items.0.text`), Condition (yes and no branches) and Merge.
`action_items.0.text`), Condition (yes and no branches; one test, or a [match tree](automations.md#match-trees) of
several) and Merge.
- **Keeping results:** Save output (`outputs.<name>`), Set field (a custom field, kept in the metadata history) and
Save entities.

Expand Down
69 changes: 53 additions & 16 deletions fastapi_backend/app/domain/flow.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,16 @@

- `input`: where a workflow starts (the recording, or the namespaces a routine runs over).
- `pick`: one part of the value, by a dotted path (`action_items.0.text`).
- `condition`: tests the value (or a path in it) and passes it on along `yes` or `no`.
- `condition`: tests the value (or a path in it) and passes it on along `yes` or `no`. The test is one
`{path, op, value}`, or a match tree in `match` (match.py: all, any and not over such tests).
- `switch`: tests the cases in turn and passes the value on along the first that matches (each case is a port), or
along `default`.
along `default`. A case is a test or a match tree, like a condition's.
- `merge`: joins what reaches it: lists into one list (entities found twice once), objects into one object.
- `set`: makes an object from fields, each a path into what came in, a value, or a template. It can take several
inputs (`inputs`, its ports): a field's path then starts with the input's name.
- `template`: renders a template with what came in as `input` (and the recording or namespaces); `json` reads the
result as JSON.
- `filter`: keeps the items of a list that pass a test.
- `filter`: keeps the items of a list that pass a test (or a match tree).
- `for_each`: runs its `body` (a graph) once per item of a list and passes on the list of what the body returned;
an item the body returned nothing for is left out (so it maps and filters). The body starts from `arg` nodes:
`item`, `index` and `input` (all of what came in).
Expand Down Expand Up @@ -59,12 +60,12 @@
CONFIG = {
"input": set(),
"pick": {"path"},
"condition": {"path", "op", "value"},
"condition": {"path", "op", "value", "match"},
"switch": {"path", "cases"},
"merge": set(),
"set": {"inputs", "fields", "keep"},
"template": {"template", "json"},
"filter": {"path", "op", "value"},
"filter": {"path", "op", "value", "match"},
"for_each": {"path", "body", "max_items"},
"repeat": {"body", "max_rounds", "until"},
"group": {"body"},
Expand Down Expand Up @@ -345,13 +346,43 @@ def keeps(db, nodes, kit):
return False


def check_test(nid, kind, cfg):
def test_problem(cfg):
"""What is wrong with a test ({path, op, value}), or None."""
if cfg.get("op") not in OPS:
raise ValueError(f"{kind} {nid}: the test is one of {', '.join(OPS)}")
return f"the test is one of {', '.join(OPS)}"
if cfg.get("path") and not PATH_RX.match(str(cfg["path"])):
raise ValueError(f"{kind} {nid}: give a path like summary.importance, or none")
return "give a path like summary.importance, or none"
if cfg["op"] in ("gt", "lt") and not _num(cfg.get("value")):
raise ValueError(f"{kind} {nid}: compare with a number")
return "compare with a number"
return None


def check_test(nid, kind, cfg):
problem = test_problem(cfg)
if problem:
raise ValueError(f"{kind} {nid}: {problem}")


def _check_match(nid, kind, cfg):
"""A condition, filter or switch case that tests with a match tree (match.py) in `match` instead of one test: the
tree, cleaned. `path`, when given, is the part of the value the tree tests."""
from . import match

if "op" in cfg or "value" in cfg:
raise ValueError(f"{kind} {nid}: give one test (op, value) or a match tree, not both")
if cfg.get("path") and not PATH_RX.match(str(cfg["path"])):
raise ValueError(f"{kind} {nid}: give a path like summary.importance, or none")
return match.check(cfg["match"], f"{kind} {nid}")


def passes(cfg, value, env=None):
"""Whether a value passes a node's test: one test ({path, op, value}), or a match tree (`match`) on the part of
the value at `path`."""
if "match" in cfg:
from . import match

return match.test(cfg["match"], dig(value, cfg["path"]) if cfg.get("path") else value, env)
return test(cfg, value)


def _whole(v, lo, hi):
Expand All @@ -363,7 +394,10 @@ def _check_primitive(db, kit, nid, t, cfg, body, depth, here):
if not PATH_RX.match(str(cfg.get("path") or "")):
raise ValueError(f"pick node {nid}: give a path like action_items.0.text")
elif t in ("condition", "filter"):
check_test(nid, t, cfg)
if "match" in cfg:
cfg["match"] = _check_match(nid, t, cfg)
else:
check_test(nid, t, cfg)
elif t == "switch":
cases = cfg.get("cases")
if not isinstance(cases, list) or not 1 <= len(cases) <= MAX_CASES:
Expand All @@ -373,12 +407,15 @@ def _check_primitive(db, kit, nid, t, cfg, body, depth, here):
names, clean = set(), []
for k in cases:
k = dict(k or {})
if set(k) - {"port", "path", "op", "value"}:
raise ValueError(f"switch {nid}: a case is {{port, op, value}}, and a path if it tests its own")
if set(k) - {"port", "path", "op", "value", "match"}:
raise ValueError(f"switch {nid}: a case is {{port, op, value}} or {{port, match}}, and a path if it tests its own")
if not NAME_RX.match(str(k.get("port") or "")) or k["port"] == "default" or k["port"] in names:
raise ValueError(f"switch {nid}: name each case differently, in lowercase (not default)")
names.add(k["port"])
check_test(nid, "switch", k)
if "match" in k:
k["match"] = _check_match(nid, "switch", k)
else:
check_test(nid, "switch", k)
clean.append(k)
cfg["cases"] = clean
elif t == "set":
Expand Down Expand Up @@ -590,12 +627,12 @@ def _step(run, f, node, vals, args, returned, trace):
if t == "pick":
return {"out": dig(v, c["path"])}
if t == "condition":
yes = test(c, v)
yes = passes(c, v, run.extra.get("env"))
run.say(f"{name}: {'yes' if yes else 'no'}")
return {"yes" if yes else "no": v}
if t == "switch":
for k in c["cases"]:
if test({**k, "path": k.get("path") or c.get("path")}, v):
if passes({**k, "path": k.get("path") or c.get("path")}, v, run.extra.get("env")):
run.say(f"{name}: {k['port']}")
return {k["port"]: v}
run.say(f"{name}: default")
Expand All @@ -613,7 +650,7 @@ def _step(run, f, node, vals, args, returned, trace):
raise ValueError(f"{name}: what it rendered isn't JSON") from None
return {"out": text}
if t == "filter":
kept = [x for x in v if test(c, x)] if isinstance(v, list) else []
kept = [x for x in v if passes(c, x, run.extra.get("env"))] if isinstance(v, list) else []
run.say(f"{name}: kept {len(kept)}")
return {"out": kept}
if t == "for_each":
Expand Down
140 changes: 140 additions & 0 deletions fastapi_backend/app/domain/match.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
"""Match trees (docs/automations.md, Match trees): the one condition language for a trigger's `where`, an
automation's `if`, and the condition, filter and switch nodes of a workflow.

M := {"all": [M, ...]} | {"any": [M, ...]} | {"not": M} | {"path": "a.b.0", "op": OP, "value": V}

A leaf tests the part of a value at a dotted path (list items by number, as flow.dig does; no path tests the value
itself) with one of the workflow engine's tests (flow.OPS: exists, empty, equals, not_equals, contains, gt, lt), the
same way a condition node does. `all` holds when every tree in it does (an empty one always does), `any` when one
does, `not` when its tree doesn't. Values are literals: templates belong to actions, which keeps matching cheap and
safe. A tree nests at most MAX_DEPTH deep and holds at most MAX_LEAVES tests.

`env` carries what tests about time and state need (`now`, `timezone`, a `state(key)` reader). None of the tests here
reads it; the operators added later do, so that tests and dry runs can fix the clock instead of reading a global one.
"""

from __future__ import annotations

from . import flow

MAX_DEPTH = 6
MAX_LEAVES = 50
GROUPS = ("all", "any", "not")
LEAF = {"path", "op", "value"}
WORDS = { # each test in words, and its opposite (for a test under not)
"exists": ("is set", "is not set"),
"empty": ("is empty", "is not empty"),
"equals": ("is", "is not"),
"not_equals": ("is not", "is"),
"contains": ("contains", "doesn't contain"),
"gt": ("is more than", "is not more than"),
"lt": ("is less than", "is not less than"),
}


def check(tree, what="the condition"):
"""The tree, cleaned: ValueError naming the first problem, and where it is (`what`, then the path into the tree,
such as "the condition, all.1: ...")."""
return _check(tree, what, 0, [0], "")


def _check(m, what, depth, count, at):
where = f"{what}, {at}" if at else what
if depth >= MAX_DEPTH:
raise ValueError(f"{where}: conditions nest at most {MAX_DEPTH} deep")
if not isinstance(m, dict) or not m:
raise ValueError(f"{where}: a condition is {{all: [...]}}, {{any: [...]}}, {{not: ...}} or {{path, op, value}}")
groups = [k for k in GROUPS if k in m]
if groups:
if len(m) != 1:
raise ValueError(f"{where}: {groups[0]} stands alone, with nothing next to it")
k = groups[0]
if k == "not":
return {"not": _check(m["not"], what, depth + 1, count, f"{at}.not" if at else "not")}
items = m[k]
if not isinstance(items, list):
raise ValueError(f"{where}: {k} is a list of conditions")
if k == "any" and not items:
raise ValueError(f"{where}: any needs at least one condition")
return {k: [_check(x, what, depth + 1, count, f"{at}.{k}.{n}" if at else f"{k}.{n}") for n, x in enumerate(items)]}
extra = sorted(set(m) - LEAF)
if extra:
raise ValueError(f"{where}: a test is {{path, op, value}}, not {extra[0]}")
count[0] += 1
if count[0] > MAX_LEAVES:
raise ValueError(f"{what} has at most {MAX_LEAVES} tests")
problem = flow.test_problem(m)
if problem:
raise ValueError(f"{where}: {problem}")
out = {"op": m["op"]}
if m.get("path"):
out["path"] = str(m["path"])
if "value" in m:
out["value"] = m["value"]
return out


def test(tree, value, env=None):
"""Whether `value` passes the tree (checked with `check` first)."""
if "all" in tree:
return all(test(m, value, env) for m in tree["all"])
if "any" in tree:
return any(test(m, value, env) for m in tree["any"])
if "not" in tree:
return not test(tree["not"], value, env)
return flow.test(tree, value)


def leaves(tree):
"""The tests in a tree, in order."""
if "all" in tree or "any" in tree:
return [x for m in tree.get("all", tree.get("any")) for x in leaves(m)]
if "not" in tree:
return leaves(tree["not"])
return [tree]


def paths(tree):
"""The paths a tree tests, each once, in order (for checking them against a kind's fields)."""
return list(dict.fromkeys(m.get("path") or "" for m in leaves(tree)))


def _label(path, kind):
"""A path in words (`kind` is for naming fields by their titles, once kinds have schemas)."""
return path or "the value"


def _value(v):
if isinstance(v, str):
return f"“{v}”"
if isinstance(v, bool):
return "true" if v else "false"
if v is None:
return "nothing"
return str(v)


def describe(tree, kind=None):
"""The tree in plain words, as the builder and the agent show it: "the sender contains “@stmarys.org” and the
subject is “Trip”". `kind` is the kind of activity the tree tests (events.py), for naming its fields."""
return _describe(tree, kind, True)


def _describe(m, kind, top):
for k, word in (("all", " and "), ("any", " or ")):
if k in m:
if not m[k]:
return "always"
parts = [_describe(x, kind, False) for x in m[k]]
text = word.join(parts)
return text if top or len(parts) == 1 else f"({text})"
if "not" in m:
inner = m["not"]
return _leaf(inner, kind, True) if "op" in inner else f"not ({_describe(inner, kind, True)})"
return _leaf(m, kind, False)


def _leaf(m, kind, negated):
said = WORDS.get(m["op"], (m["op"], f"not {m['op']}"))[1 if negated else 0]
words = f"{_label(m.get('path'), kind)} {said}"
return words if m["op"] in ("exists", "empty") else f"{words} {_value(m.get('value'))}"
2 changes: 1 addition & 1 deletion fastapi_backend/app/domain/organize.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
CONFIG = {
"candidates": {"kind", "min_confidence", "limit", "types"},
"llm_judge": {"model", "instructions", "batch"},
"filter": {"path", "op", "value"},
"filter": {"path", "op", "value", "match"},
"apply_changes": {"apply_above", "max_apply"},
}
KINDS = ("merge", "link")
Expand Down
Loading
Loading
0