8000 feat(automations): what people say to Lens as chat.message activities by adeelahmad · Pull Request #448 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ merged is called built.
| Runs that finished or failed, deletions and calendar events that moved | built |
| Comments, notes written and saved, and changes to the entity graph | built |
| Sensors: readings a sensor keeps, alert and notable log patterns, devices that report for the first time ([Sensors](#sensors-as-activities)) | built |
| More kinds: chat, system events | planned |
| What people say to Lens, in the web app and chat apps, as `chat.message` ([Chat](#chat-as-activities)) | built |
| More kinds: system events | planned |
| The timeline API (`GET /api/v1/activities`), the MCP tools `timeline` and `event_kinds`, the assistant's `timeline` | built |
| Review marks on the timeline (reviewed, important, hidden), their filters and `mark_activities` ([Timeline](timeline.md)) | built |
| The shapes of triggers, joins, an If, policy and intents, checked (`app/domain/automation_spec.py`) | built |
Expand Down Expand Up @@ -106,6 +107,7 @@ and the paths holding outside text are in `GET /api/v1/activities/kinds` (`event
| `sensor.reading` | | `sensor_readings` | `{type: Note, id: lens:sensor_reading/<id>, name: "<stream>: <what it said>", url: /sensors/<id>}`, `stream`, `value`, `text`, `fields` (a JSON reading's numbers), `level` (syslog), `pattern`, `label` (the line's pattern and its label); noisy | Kitchen sensor temperature: 31.2 |
| `sensor.alert` | | `log` | `{type: Note, id: lens:sensor_pattern/<id>}`, `label` (alert or notable), `pattern`, `example` (a line it fits), `count` (lines in all), `lines` (in the batch that raised it), `hour` | 192.168.1.1 logged an alert: wlan<n>: authentication failed for <mac> |
| `sensor.joined` | | `sensors` | `{type: Service, id: lens:sensor/<id>, url: /sensors/<id>}`, `sensor_type` (mqtt, syslog, ...), `device`, `status` | A new MQTT device reported: zigbee2mqtt/kitchen |
| `chat.message` | | `chat_messages` | `{type: Note, id: lens:chat_message/<id>, name: the start of it, url: /chat/<id>}`, `text`, `channel` (web, telegram, slack, bridge, mailbox), `room`, `chat`, `direct` (a private chat or the web app), `attachments` (how many files) | sam wrote in Telegram: “groceries: milk, eggs” |
| `webhook.received` | | `log` | `{type: Note, hook: the webhook's id, payload: what was sent (JSON, a form's fields or text), content_type, query}`; the actor is the webhook (`Service`, `lens:automation_hook/<id>`, "Webhook for <automation>") | Webhook for Front door was called |

A new asset is one activity, of the most specific kind its row tells, decided by its content type and where it came
Expand Down Expand Up @@ -158,6 +160,22 @@ outside text.
- `sensor.joined` is a device or host that made itself a sensor by reporting for the first time (MQTT, syslog),
read from its `storage_source` row by `created_at`; a sensor added by hand isn't news.

#### Chat as activities

Built (`app/domain/events_streams.py`). `chat.message` is what a person said to Lens: a question in the web app (Chat,
the setup conversation) or a message to Lens in a chat app (Telegram, Slack, Matterbridge, the assistant's mailbox).
It reads the messages Lens already keeps (`chat_message` with role user), so nothing new is stored, and what isn't
said to Lens in a room is never kept, so it is never an activity. Its text and names are outside text.

- In the web app the actor is the person (`lens:account/<id>`), and the activity is theirs alone (`lens.audience`
owner): not even admins or others in the namespace see it. Its namespace is the conversation's scope.
- In a chat app the actor is who wrote (`Person`, `lens:chat_user/<channel>:<their id there>`, their name there),
since a chat user isn't a Lens account. A room given to a namespace's assistant belongs to that namespace; any
other conversation (a private chat, an open room) is the answering account's alone. The conversation keeps where
it is (`chat.via`: the channel, the room, who, a private chat or not) from its next message on.
- A question an automation's `ask_agent` asks (`origin` on the message) isn't a person's, so it is never a
`chat.message`, and an automation can't set itself off that way.

### Sources and the event log

Built. A source (`events.Source`, added with `events.register_source`) reads rows Lens already keeps as activities,
Expand Down Expand Up @@ -190,6 +208,8 @@ A kind belongs to the source named in its `source`, and a source may list more i
- `sensor_readings` (`app/domain/events_streams.py`) reads `sensor_reading` by `at` (microseconds), the id breaking
ties, one sensor at a time, as `sensor.reading`; `sensors` reads the `storage_source` rows discovery made by
`created_at`, as `sensor.joined`.
- `chat_messages` (`app/domain/events_streams.py`) reads `chat_message` rows with role user by `created_at` over the
index on (role, created_at), the id breaking ties, as `chat.message`.
- `log` reads `event_log`: activities no row of their own records (a deletion, a webhook call), written by
`events.emit(db, kind, *, actor, obj, space, summary, target=None, tag=None, key=None, published=None, spaces=None,
audience=None, owner=None, untrusted=None)`, which gives the id (`log-<n>`). Emitting a `key` again changes
Expand Down Expand Up @@ -523,3 +543,6 @@ the agent can draft an event as a note.
- Sensors: a threshold per reading in the intake (only readings over a value become activities, for sensors that
keep everything); a reading's unit, which sensors don't record yet; an alert as soon as a pattern is labelled, for
the lines already seen that hour.
- Chat: recording every message in a room as an activity (opt-in per channel, off by default); a chat user who is
also a Lens account as that person; a timeline page of others' private messages is read and left out, up to five
source pages, so narrow it by account if that proves slow.
4 changes: 4 additions & 0 deletions docs/chat-rooms.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ That's all. Without an account given, it answers as the admin who turned it on.
"Matterbridge · room · person", listed in Chat with the account's other conversations. Follow-ups keep their context,
and older messages are folded into a running summary ([Long conversations](assistant.md)). `Lens /clear` starts
the conversation afresh and `Lens /compact` summarises it now.
- **Automations hear what's said to Lens.** Each message said to Lens (and only those: the room's other talk is never
kept) is a `chat.message` activity an automation can trigger on, "when someone messages Lens 'groceries' in
Telegram" ([Automations](automations.md#chat-as-activities)). A room given to a namespace belongs to it; any other
conversation is the answering account's alone.
- **A namespace's own assistant.** **Rooms for a namespace's assistant** gives a room to a namespace
(`team = pods`, or one channel: `team/general = pods`). Conversations there are scoped to that namespace, so when its
assistant is on ([A namespace's own assistant](assistant.md#a-namespaces-own-assistant)) it answers with its
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/app/domain/actions_ai.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ def _ask_agent(db, cfg, intent, user, dry):
if about:
url = (about.get("object") or {}).get("url") if isinstance(about.get("object"), dict) else None
page = {"url": str(url or f"/timeline?activity={about.get('id') or ''}"), "title": _short(happened, 300)}
chat.add(db, cid, "user", instructions, context=page)
chat.add(db, cid, "user", instructions, context=page, origin=box.origin)
# reading the activity counts as reading outside text for this conversation from now on (ai_tools.read_before)
steps = (
[{"tool": "automation_event", "args": {"id": about.get("id")}, "summary": f"Read what happened: {_short(happened, 200)}"}]
Expand Down
25 changes: 20 additions & 5 deletions fastapi_backend/app/domain/bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,23 +176,38 @@ def _account(db, cfg):
return {"id": u["id"], "email": u["email"], "admin": bool(u.get("admin"))}


def conversation(db, account, m, namespace=None, label="Matterbridge"):
def conversation(db, account, m, namespace=None, label="Matterbridge", channel="bridge"):
"""The conversation for this person in this room, started on their first message; scoped to `namespace` (the
room's namespace assistant), which follows bridge.rooms when it changes. In a room no namespace is given to, the
scope is routing's (route). `label` names the channel in its title."""
scope is routing's (route). `label` names the channel in its title; the chat keeps where it is (`via`: the
channel's key and label, the room, who is talking and whether it's a private chat) for its messages as activities
(events_streams.py, chat.message)."""
who = m["thread"] if m.get("thread") else [m.get("gateway"), m.get("channel"), m.get("account"), m.get("username")]
key = hashlib.sha1(json.dumps(who).encode()).hexdigest()
scope = {"namespaces": [namespace]} if namespace else {}
cid = db.one("SELECT VALUE record::id(id) FROM chat WHERE bridge = $k AND account = $a LIMIT 1", k=key, a=account["id"])
via = store.clean(
{
"channel": channel,
"label": label,
"room": str(m.get("channel") or "")[:200] or None,
"user": (m.get("username") or "").strip("<> ")[:200] or None,
"userid": str(m.get("userid") or "")[:200] or None,
"private": bool(m.get("private")),
}
)
row = db.one("SELECT record::id(id) AS id, via FROM chat WHERE bridge = $k AND account = $a LIMIT 1", k=key, a=account["id"])
cid = row["id"] if row else None
if cid:
if row.get("via") != via: # a chat from before chats kept it, or someone renamed
db.q("UPDATE $r SET via = $v", r=R("chat", cid), v=via)
if namespace:
db.q("UPDATE $r SET scope = $s, room = 'given' WHERE (scope ?? {}) != $s", r=R("chat", cid), s=scope)
else: # the room was given to a namespace and no longer is: routing starts again
db.q("UPDATE $r SET scope = {}, room = NONE, pinned = NONE WHERE room = 'given'", r=R("chat", cid))
return cid
where = " · ".join(x for x in [m.get("channel"), (m.get("username") or "").strip("<> ")] if x)
cid = chat.create(db, account["id"], title=f"{label} · {where}" if where else label, scope=scope, kind="bridge")
db.q("UPDATE $r SET bridge = $k, room = $g", r=R("chat", cid), k=key, g="given" if namespace else None)
db.q("UPDATE $r SET bridge = $k, room = $g, via = $v", r=R("chat", cid), k=key, g="given" if namespace else None, v=via)
return cid


Expand Down Expand Up @@ -513,7 +528,7 @@ def handle(db, cfg, base, m, ch=None):
if not q:
return False
account = _account(db, cfg)
cid = conversation(db, account, m, ns if sid is not None else None, ch.label)
cid = conversation(db, account, m, ns if sid is not None else None, ch.label, ch.key)
try: # a private chat with someone the channel vouches for is where the account's person is told things
channels.remember(db, cfg, ch, account, m, cid)
except Exception: # noqa: BLE001 - answering comes first
Expand Down
18 changes: 16 additions & 2 deletions fastapi_backend/app/domain/chat.py
Original file line number Diff line number Diff line change
Expand Up @@ -351,10 +351,23 @@ def history(db, cid):


def add(
db, cid, role, content, passages=None, stopped=False, steps=None, notice=None, error=None, model=None, attachments=None, context=None
db,
cid,
role,
content,
passages=None,
stopped=False,
steps=None,
notice=None,
error=None,
model=None,
attachments=None,
context=None,
origin=None,
):
"""Save a message; an answer keeps the tool steps it took, any notice (e.g. the model can't use tools) and error; a
question asked from a page keeps what it shared of it (shared_context)."""
question asked from a page keeps what it shared of it (shared_context). A question an automation asked keeps its
`origin` ({automation, run, node}), so it isn't taken for a person's message (`chat.message`)."""
mid = db.next_id("chat_message")
db.q(
"CREATE $r CONTENT $d",
Expand All @@ -373,6 +386,7 @@ def add(
"error": error,
"model": model,
"context": context,
"origin": origin or None,
}
),
)
Expand Down
Loading
Loading
0