8000 feat(automations): a decide node for judgement calls by adeelahmad · Pull Request #549 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,11 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
a time (for the same sender, say), when one came a number of times, when one didn't come in time after another (no
reply within two days), or once for a batch collected over a time. What it waits on outlasts a restart, and switching
the automation off drops it ([Joins](docs/automations.md#joins)).
- **A decide node for judgement calls.** An automation's workflow can ask the decision model a question with a few
options ("Does this need me today?") and go on along the option it picked, or along `unsure` when it isn't sure
enough, none is set up or the budget is spent. The email it reads is marked as outside text, the answer is kept per
activity so a retry doesn't pay again, and the call shows in the activity ledger under the run
([The decide node](docs/automations.md#the-decide-node)).
- **Specs from transcripts.** Turn a voice session or a meeting into a technical spec, saved as a note linked to it:
the model writes the spec, then lists every named decision, every "no" and "not that, this", every constraint and
every gap the speaker left to decide later, each with its quote and a link to the moment it was said; a quote the
Expand Down
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
on that stream, on a schedule, at a time or when their own webhook is called, alone or with their triggers joined (all
of them, a number of times, one that doesn't come in time, or a batch): they test conditions (text, patterns,
numbers, lists, times, days, whether a value changed and whether something like it happened before in their
namespaces), then tell you in
namespaces) and let a decision model pick the branch, then tell you in
your own chat app, ask the model or the agent, use a tool, call a web address (its credentials kept sealed), run a
workflow, pipeline or routine, or change assets and notes. A Classify node in a workflow sorts an asset into labels,
content types or topics, acting only on a sure answer. Anything that reaches outside Lens or changes more than what
Expand Down Expand Up @@ -234,8 +234,7 @@ These are open pull requests, not yet part of Lens:

Next, and not in a pull request yet; each links to where it is tracked:

- **The automations studio:** drawing automations in the web app, and the decide node, where a decision model
picks the branch ([automations](docs/automations.md)).
- **The automations studio:** drawing automations in the web app ([automations](docs/automations.md)).
- **Timeline:** what each item caused, and Approve and Decline on waiting approvals ([timeline](docs/timeline.md)).
- **Notes:** nesting a note under its project or area without a click, pages in the graph, and wiki addresses for
topics, recordings and the other things that have pages ([notes](docs/notes.md)).
Expand Down
33 changes: 30 additions & 3 deletions docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ merged is called built.
| Review marks on the timeline (reviewed, important, hidden), their filters and `mark_activities` ([Timeline](timeline.md)) | built |
| The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built |
| The event workflow scope: what an automation does, drawn with action nodes and tried on an activity (`app/domain/automation_nodes.py`) | built |
| The decide node (the decision model picks a branch) | planned |
| The decide node: the decision model picks a branch, one port per option and `unsure` ([The decide node](#the-decide-node)) | built |
| The action registry with risk from each action's effect, and one action, `log_line` (`app/domain/actions.py`) | built |
| Telling a person: the `notify` action and `channels.tell` (their own chat, a notification target, a Matterbridge room) | built |
| The model and the agent: the `ask_model`, `call_tool` and `ask_agent` actions, and prompts with outside text marked | built |
Expand Down Expand Up @@ -546,8 +546,7 @@ its own:
nothing on, so the nodes after it are skipped and the run's log says where it stopped and why. An action that
fails is marked failed on its node, and the rest of the graph goes on.
- **Decide** (`decide`, configured `{question, options, input?, min_confidence?}`): the decision model picks one of
its options, with an output port per option and `unsure`. Planned with the decision model; until then a decide node
is refused when the workflow is saved, as coming soon. Use a condition or switch node.
its options, with an output port per option and `unsure` ([The decide node](#the-decide-node)).

Checked when it is saved, on every path (create, a new version, a body):

Expand All @@ -568,6 +567,34 @@ lists each one with its params, effect, risk, status (`done`, `waiting`, `schedu
back. Text from outside Lens (`lens.untrusted` on the activity, or what a node passed on when its output says
`untrusted`, like a web page's answer) that reaches an action's params marks its intent `untrusted`.

#### The decide node

Built (`match.check_decide`, `match.run_decide`). A decide node asks the decision model (Jev, then Laya, then the
language model as a fallback, as in [Routine choices](assistant.md#routine-choices)) a judgement call, and the run
goes on along the port of the option it chose:

```json
{"id": "n4", "type": "decide", "config": {"question": "Does this need me today?",
"options": {"urgent": "Needs an answer today.", "later": "Can wait."}, "min_confidence": 0.8}}
```

- `options` names 2 to 10 options, in lowercase letters, digits and `_`, each with what it means. The node's ports
are the options, then `unsure`; wire an edge from each port you want to act on.
- `input` is a template of what the model reads, rendered as [prompts](#prompts) are (each value from outside Lens
marked as data), with `input` holding what reached the node: `Subject: {{ event.object.subject }}`. With none, it
reads the event's summary and its outside text (the fields in `lens.untrusted`), marked as outside text
(`untrusted.wrap`). Either way instructions in an email are read as content.
- The run goes along the chosen option when the model is sure enough: `min_confidence`, else
`decisions.act_above` (default 0.8). Otherwise it goes along `unsure`, as it does when no decision model or
language model is set up, or the budget's hard limit is reached. Along the port passes the decision, `{choice,
confidence, ranked, by, sure}`, and `steps.<node id>` gets `{port, choice, confidence, sure}` (`choice` is its best
guess when it went along `unsure`): `{{ steps.n4.choice }}`.
- The call is made inside the run's activity scope, so it shows in the activity ledger under the automation and
the run, and counts towards budgets. An answer is kept for 7 days per activity and question (in
`automation_state`), so a retry or a dry run (Try it) of the same activity doesn't pay again; a dry run does call
the model the first time. No model is called unless an automation has a decide node, and then only for activities
that reach it.

### Actions and intents

Built: the registry (`app/domain/actions.py`). An action is registered with `actions.register(ActionSpec(name,
Expand Down
9 changes: 5 additions & 4 deletions fastapi_backend/app/domain/automation_nodes.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,8 @@
back passes on and goes in `steps.<node id>`. One that asks first waits for the person's pick, and one with a delay
(or a time, `when`) waits for its time: such a node passes nothing on, so the nodes after it are skipped and the
run says why.
- `decide`: asks the decision model which of its options fits, with an output port per option and `unsure`. It comes
with the conditions module (match.check_decide, match.run_decide); until then a decide node is refused as coming
soon.
- `decide`: asks the decision model which of its options fits, with an output port per option and `unsure`
(match.check_decide, match.run_decide); `steps.<node id>` gets {port, choice, confidence, sure}.

The leaf rule (`check_graph`, on every path a graph is saved by): an action that reaches outside Lens or other people
(effect `out`, worked out from its params when it is saved) or that waits has nothing wired after it, since it can't
Expand Down Expand Up @@ -278,7 +277,9 @@ def run_node(r, f, node, value, vals):
if run_decide is None:
raise ValueError(f"{flow._label(node)}: deciding with the model is coming soon")
out = run_decide(r, f, node, value)
_steps(r)[node["id"]] = {"choice": next(iter(out), None)}
port, got = next(iter(out.items()), (None, None))
d = got if isinstance(got, dict) and "sure" in got else {}
_steps(r)[node["id"]] = {"port": port, "choice": d.get("choice", port), "confidence": d.get("confidence"), "sure": d.get("sure")}
return out
return _action(r, f, node)

Expand Down
120 changes: 120 additions & 0 deletions fastapi_backend/app/domain/match.py
Original file line number Diff line number Diff line change
Expand Up @@ -1084,6 +1084,126 @@ def candidates(index, activity):
return list(index["by_kind"].get((activity or {}).get("kind"), []))


# ---------- the decide node ----------
DECIDE_CONFIG = {"question", "options", "input", "min_confidence"}
MAX_OPTIONS = 10
DECIDE_DAYS = 7 # an answer is kept this long, so a retry or a dry run of the same activity doesn't pay again
UNSURE = "unsure"


def check_decide(nid, cfg):
"""A decide node's settings ({question, options: {name: what it means}, input, min_confidence}), cleaned:
ValueError naming the node and the problem."""
from . import templates

q = cfg.get("question")
if not isinstance(q, str) or not q.strip() or len(q) > 1000:
raise ValueError(f"decide {nid}: ask a question, up to 1000 characters")
opts = cfg.get("options")
if not isinstance(opts, dict) or not 2 <= len(opts) <= MAX_OPTIONS:
raise ValueError(f"decide {nid}: give 2 to {MAX_OPTIONS} options, each a name and what it means")
for name, meaning in opts.items():
if not isinstance(name, str) or not flow.NAME_RX.match(name) or name in (UNSURE, "default"):
raise ValueError(f"decide {nid}: name each option in lowercase letters, digits and _ (not unsure)")
if not isinstance(meaning, str) or not meaning.strip() or len(meaning) > 300:
raise ValueError(f"decide {nid}: say what option {name} means, up to 300 characters")
out = {"question": q.strip(), "options": {k: v.strip() for k, v in opts.items()}}
if cfg.get("input") is not None:
body = cfg["input"]
if not isinstance(body, str) or not body.strip() or len(body) > 20000:
raise ValueError(f"decide {nid}: input is a template of up to 20000 characters, or none")
try:
templates.check(body)
except templates.TemplateProblem as e:
raise ValueError(f"decide {nid}: {e}") from None
out["input"] = body
mc = cfg.get("min_confidence")
if mc is not None:
if not _num(mc) or not 0 <= mc <= 1:
raise ValueError(f"decide {nid}: min_confidence is from 0 to 1")
out["min_confidence"] = float(mc)
return out


def decide_ports(cfg):
"""A decide node's output ports: one per option, then unsure."""
return [*(cfg.get("options") or {}), UNSURE]


def _untrusted_text(a):
"""An activity's summary and the outside text at its lens.untrusted paths, as one text."""
lines = [str(a.get("summary") or "")]
for p in (a.get("lens") or {}).get("untrusted") or []:
v = flow.dig(a, p)
if isinstance(v, list):
v = ", ".join(str(x) for x in v if x is not None)
if v not in (None, ""):
lines.append(f"{p}: {v}")
return "\n".join(x for x in lines if x)


def run_decide(run, frame, node, value):
"""Run a decide node: ask decide.choose (Jev, Laya, or the language model) which option fits what came in, inside
the run's activity.scope. It reads the node's input template, rendered as prompts are (variables.prompt: each
value from outside Lens marked, and `input`, what reached the node, counted as outside), or else the event's
summary and outside text, wrapped with untrusted.wrap. Passes the decision ({choice, confidence, ranked, by, sure}) on along the chosen option's port, or
along unsure when no option is sure enough (min_confidence, else decisions.act_above), nothing could answer, or
the budget's hard limit is reached. Answers are kept per activity and question, so a retry or a dry run doesn't
pay twice."""
from . import activity, decide, untrusted, variables

c, name = node["config"], node.get("label") or f"decide {node['id']}"
ctx = run.ctx if isinstance(run.ctx, dict) else {}
event = ctx.get("event") if isinstance(ctx.get("event"), dict) else None
if c.get("input"):
seen = {**ctx, "input": None if value is run.ctx else value}
try:
state = variables.prompt(c["input"], seen, [*variables.outside_paths(ctx), "input"], run.cfg, run.db)["text"]
except variables.Problem as e:
raise ValueError(f"{name}: input: {e}") from None
else:
text = _untrusted_text(event) if event else flow.preview(value, 20000)
source = f"activity {event['id']}" if event and event.get("id") else "the run's input"
state = untrusted.wrap(text, source, run.cfg)
aid = (ctx.get("automation") or {}).get("id")
< 2360 span class='blob-code-inner blob-code-marker ' data-code-marker="+"> rid = (ctx.get("run") or {}).get("id")
asked = _canon({"q": c["question"], "o": c["options"], "i": c.get("input")})
key = f"{event['id']}|{hashlib.sha1(asked.encode()).hexdigest()}" if event and event.get("id") else None
ref = state_ref(aid, "decide", "", key) if key else None
kept = run.db.one("SELECT decision FROM $r", r=ref) if ref is not None and run.db is not None else None
if kept and isinstance(kept.get("decision"), dict):
got = dict(kept["decision"], cached=True)
else:
refs = [r for r in (f"automation:{aid}" if aid else None, f"automation_run:{rid}" if rid else None) if r]
try:
with activity.scope(run.db, *refs, cfg=run.cfg):
got = decide.choose(run.cfg, c["question"], c["options"], state)
except (decide.Undecided, activity.OverBudget) as e:
why = "over budget" if isinstance(e, activity.OverBudget) else str(e)
run.say(f"{name}: unsure ({why})")
return {UNSURE: {"choice": None, "confidence": 0.0, "ranked": [], "by": None, "sure": False, "why": why}}
if ref is not None and run.db is not None:
if not run.db.one("SELECT id FROM $r", r=ref):
make_room(run.db, aid)
run.db.q(
"UPSERT $r SET automation = $a, kind = 'decide', alias = '', key = $k, decision = $d, at = $t, expires_at = $x",
r=ref,
a=aid,
k=key[:300],
d=got,
t=_stamp(),
x=_later(DECIDE_DAYS * 86400),
)
least = c.get("min_confidence")
if least is None:
least = float((run.cfg.get("decisions") or {}).get("act_above") or 0.8)
sure = got.get("choice") in c["options"] and float(got.get("confidence") or 0) >= least
out = {**got, "sure": sure}
port = got["choice"] if sure else UNSURE
run.say(f"{name}: {port} ({float(got.get('confidence') or 0):.2f}{', kept from before' if got.get('cached') else ''})")
return {port: out}


# ---------- reading trees ----------
def leaves(tree):
"""The tests in a tree, in order (not the lookbacks: `lookbacks`)."""
Expand Down
13 changes: 12 additions & 1 deletion fastapi_backend/app/domain/variables.py
Original file line number Diff line number Diff line change
Expand Up @@ -758,7 +758,18 @@ def _outputs(graph, ids):
else:
out[i] = (n.get("label") or spec.label, spec.output or {"type": "object"})
elif n.get("type") == "decide":
out[i] = (n.get("label") or "Decide", {"type": "object", "properties": {"choice": {"type": "string", "title": "Choice"}}})
out[i] = (
n.get("label") or "Decide",
{
"type": "object",
"properties": {
"port": {"type": "string", "title": "Branch taken (an option, or unsure)"},
"choice": {"type": "string", "title": "Option chosen (its best guess when unsure)"},
"confidence": {"type": "number", "title": "Confidence, 0 to 1"},
"sure": {"type": "boolean", "title": "Sure enough to follow it"},
},
},
)
return out


Expand Down
Loading
Loading
0