8000 feat(assistant): set up a namespace's assistant in the app and give it chat rooms by adeelahmad · Pull Request #164 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,10 +115,14 @@ It is the same assistant with the same tools, extensions and approvals; what it
`PATCH .../memories/<id>` to correct or pin one, `DELETE .../memories/<id>` to forget it, and owners `DELETE
.../memories` to forget everything.

Owners find it on the namespace's page (Admin → Namespaces → the namespace → Assistant): the switch, its name and
instructions, and what it remembers, with where each memory came from, to add to, pin or forget.

It makes no model calls of its own and keeps no index, so it costs nothing extra on a Raspberry Pi.

Refine later: a decision model choosing what's worth remembering, chat rooms answered by a namespace's assistant,
its settings and memories in the web app, similar memories merged.
A chat room can be given to a namespace's assistant too ([chat rooms](chat-rooms.md)).

Refine later: a decision model choosing what's worth remembering, similar memories merged.

## Extending the assistant: tools, skills, hooks and plugins

Expand Down
6 changes: 5 additions & 1 deletion docs/chat-rooms.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,16 @@ That's all. Without an account given, it answers as the admin who turned it on.
links to the conversation.
- **Conversations.** Each person in each room is a conversation of that account's, titled
"Matterbridge · room · person", listed in Chat with the account's other conversations. Follow-ups keep their context.
- **A namespace's own assistant.** **Rooms for a namespace's assistant** gives a room to a namespace
(`team = pods`, or one channel: `team/general = pods`). Conversations there are scoped to that namespace, so when its
assistant is on ([A namespace's own assistant](assistant.md#a-namespaces-own-assistant)) it answers with its
instructions and what it remembers, and a message starting with its name is for it too.
- **One reader.** Matterbridge hands out each message once, so only one server process reads it: the API and the
workers all run the bridge thread, and whichever holds the lease (renewed every look, given up when it stops or the
bridge is turned off) does the reading. Settings → Chat rooms says whether it is listening and how many messages it answered.

Settings (`bridge`): `enabled`, `url`, `token` (a secret), `account`, `name` (default Lens), `answer` (`mention` or
`all`), `gateway`, `users`, `poll_seconds` (default 2). The assistant can change them too ("connect Lens to
`all`), `gateway`, `users`, `rooms`, `poll_seconds` (default 2). The assistant can change them too ("connect Lens to
Matterbridge at …").

Refine later: approving in the room itself, voice notes and files posted in rooms, one Lens account per chat user.
47 changes: 37 additions & 10 deletions fastapi_backend/app/domain/bridge.py
10BC8
Original file line number Diff line numberDiff line change
Expand Up @@ -7,6 +7,10 @@
it reads what that account can read, and changes anything only after someone approves it in the web app, like any
chat. Each person in each room is a conversation of that account's, listed with its other chats.

A room can be given to a namespace's own assistant (bridge.rooms, "gateway = namespace" or "gateway/channel =
namespace"): conversations there are scoped to that namespace, so its assistant answers with its instructions and
memory (ns_assistant.py), and a message naming the assistant is for it too.

Refine later: approving in the room itself, voice notes, files posted in the room, one Lens account per chat user.
"""

Expand All @@ -22,7 +26,7 @@
import urllib.error
import urllib.request

from . import ai_tools, auth, chat, llm, store
from . import ai_tools, auth, chat, llm, ns_assistant, store
from .store import R

log = logging.getLogger(__name__)
Expand Down Expand Up @@ -101,14 +105,30 @@ def check(db, cfg):


# ---------- who's talking to it ----------
def addressed(cfg, m):
"""The question in a message, when it's for Lens; None when it isn't (or is Lens's own)."""
def room_namespace(cfg, m):
"""The namespace whose assistant a room is given to (bridge.rooms), or None."""
gw, ch = m.get("gateway") or "", m.get("channel") or ""
rooms = {}
for line in cfg["bridge"].get("rooms") or []:
where, _, ns = line.partition("=")
rooms.setdefault(where.strip().lower(), ns.strip())
return rooms.get(f"{gw}/{ch}".lower()) or rooms.get(gw.lower())


def addressed(cfg, m, aliases=()):
"""The question in a message, when it's for Lens (or one of `aliases`: the room's namespace assistant's name); None
when it isn't (or is Lens's own)."""
b = cfg["bridge"]
name = (b.get("name") or "Lens").strip()
text = (m.get("text") or "").strip()
who = (m.get("username") or "").strip()
if not text or (m.get("event") or "") not in ("", "user_action") or who.strip("<> ").lower() == name.lower():
return None
for alias in aliases:
if alias.lower() != name.lower():
got = addressed({**cfg, "bridge": {**b, "name": alias, "answer": "mention"}}, m)
if got:
return got
gw = b.get("gateway")
if gw and m.get("gateway") != gw:
return None
Expand All @@ -132,14 +152,17 @@ def _account(db, cfg):
return {"id": u["id"], "email": u["email"], "admin": bool(u.get("admin"))}


def conversation(db, account, m):
"""The conversation for this person in this room, started on their first message."""
def conversation(db, account, m, namespace=None):
"""The conversation for this person in this room, started on their first message; scoped to `namespace` (the
room's namespace assistant), which follows bridge.rooms when it changes."""
key = hashlib.sha1(json.dumps([m.get("gateway"), m.get("channel"), m.get("account"), m.get("username")]).encode()).hexdigest()
scope = {"namespaces": [namespace]} if namespace else {}
cid = db.one("SELECT VALUE record::id(id) FROM chat WHERE bridge = $k AND account = $a LIMIT 1", k=key, a=account["id"])
if cid:
db.q("UPDATE $r SET scope = $s WHERE (scope ?? {}) != $s", r=R("chat", cid), s=scope)
return cid
where = " · ".join(x for x in [m.get("channel"), (m.get("username") or "").strip("<> ")] if x)
cid = chat.create(db, account["id"], title=f"Matterbridge · {where}" if where else "Matterbridge", kind="bridge")
cid = chat.create(db, account["id"], title=f"Matterbridge · {where}" if where else "Matterbridge", scope=scope, kind="bridge")
db.q("UPDATE $r SET bridge = $k", r=R("chat", cid), k=key)
return cid

Expand All @@ -152,13 +175,14 @@ def answer(db, cfg, base, account, cid, q):
readable = set(roles)
editable = {s for s in roles if auth.allows(roles, s, "editor")}
past = chat.history(db, cid)
scope = (db.one("SELECT scope FROM $r", r=R("chat", cid)) or {}).get("scope") or None
chat.add(db, cid, "user", q)
passages = chat.retrieve(db, q, readable, None, cfg=cfg)
passages = chat.retrieve(db, q, readable, scope, cfg=cfg)
steps, approvals, notice = [], [], None
wrote = cfg["llm"].get("model") if llm.configured(cfg) else None
if llm.configured(cfg) and cfg["ai"].get("tools"):
box = ai_tools.Toolbox(
db, cfg, {"id": account["id"], "email": account["email"]}, readable, editable, None, cid, base, account["admin"], said=q
db, cfg, {"id": account["id"], "email": account["email"]}, readable, editable, scope, cid, base, account["admin"], said=q
)
try:
text = None
Expand Down Expand Up @@ -200,11 +224,14 @@ def reply_text(cfg, text, approvals, cid):

def handle(db, cfg, base, m):
"""One message from a room: answered when it's for Lens. True when it was."""
q = addressed(cfg, m)
ns = room_namespace(cfg, m)
sid = {v: k for k, v in store.space_names(db).items()}.get(ns) if ns else None
home = ns_assistant.profile(db, sid) if sid is not None else None
q = addressed(cfg, m, [home["name"]] if home and home["enabled"] else ())
if not q:
return False
account = _account(db, cfg)
cid = conversation(db, account, m)
cid = conversation(db, account, m, ns if sid is not None else None)
try:
text, approvals = answer(db, cfg, base, account, cid, q[:MAX_CHARS])
except Exception: # noqa: BLE001 - still say something in the room
Expand Down
14 changes: 14 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,20 @@ def _bridge_setting(key, value):
if not (isinstance(value, list) and all(isinstance(v, str) and len(v) <= 200 for v in value) and len(value) <= 500):
raise ValueError("bridge.users is a list of chat usernames")
return list(dict.fromkeys(v.strip() for v in value if v.strip()))
if key == "rooms":
bad = "bridge.rooms is a list of rooms and namespaces, like team = pods or team/general = pods"
if not (isinstance(value, list) and len(value) <= 200 and all(isinstance(v, str) and len(v) <= 300 for v in value)):
raise ValueError(bad)
out = []
for v in value:
if not v.strip():
continue
where, eq, ns = v.partition("=")
where, ns = where.strip(), ns.strip().lower()
if not eq or not where or not store.NS_RX.match(ns):
raise ValueError(bad)
out.append(f"{where} = {ns}")
return list(dict.fromkeys(out))
if key not in ("url", "token", "gateway", "account", "name"):
raise ValueError(f"unknown setting bridge.{key}")
if value in (None, ""):
Expand Down
4 changes: 3 additions & 1 deletion fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,8 @@ def labels():
},
# the assistant in chat rooms through Matterbridge (bridge.py): url is its API (http://matterbridge:4242), token its
# API token; it answers as `account` (an email), when a message names it (answer "mention") or to every message
# ("all"), from anyone or only the chat usernames in `users`, in every gateway or only `gateway`
# ("all"), from anyone or only the chat usernames in `users`, in every gateway or only `gateway`; `rooms` gives a
# gateway (or gateway/channel) to a namespace's own assistant, as "gateway = namespace" (ns_assistant.py)
"bridge": {
"enabled": False,
"url": None,
Expand All @@ -322,6 +323,7 @@ def labels():
"name": "Lens",
"answer": "mention",
"users": [],
"rooms": [],
"poll_seconds": 2,
},
# what Lens fetches for itself (components.py): auto fetches what the settings need, on first use unless ahead;
Expand Down
28 changes: 28 additions & 0 deletions fastapi_backend/tests/api/test_bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -179,3 +179,31 @@ def test_the_thread_answers_on_its_own(db, cfg, room):
stop.set()
th.join(5)
assert bridge.status(db, cfg)["state"] == "starting" # it let go when it stopped


def test_a_room_given_to_a_namespace_assistant(client, db, cfg, room, llm):
from app.domain import ns_assistant, store

pods = store.ns_id(db, "pods")
ns_assistant.save_profile(db, pods, enabled=True, name="Podpal", instructions="Be brief.")
ns_assistant.remember(db, pods, "Episodes ship on Fridays.", 1, author="person")
a = login(client, "root@x.io", "root password 1")
bad = client.put("/api/v1/settings/bridge", headers=a, json={"rooms": ["team"]})
assert bad.status_code == 400
got = client.put("/api/v1/settings/bridge", headers=a, json={"rooms": ["Team/General=Pods", " ", "other = calls"]})
assert got.status_code == 200
cfg["bridge"]["rooms"] = settings.Settings(db, cfg).current()["bridge"]["rooms"]
assert cfg["bridge"]["rooms"] == ["Team/General = pods", "other = calls"]
room.waiting = [said("Podpal, when do episodes ship?"), said("Podpal, hi", gateway="elsewhere")]
assert run(db, cfg) == 1 # the assistant's name works only in its room
asked = [m for m in llm.seen if m.get("tools")][-1]
system = next(m["content"] for m in asked["messages"] if m["role"] == "system")
assert "You are Podpal, the assistant of the pods namespace" in system and "Episodes ship on Fridays." in system
rows = db.rows("SELECT scope FROM chat WHERE kind = 'bridge'")
assert [r["scope"] for r in rows] == [{"namespaces": ["pods"]}]
# Lens's own name still works there, and the room follows the setting when it changes
cfg["bridge"]["rooms"] = []
room.waiting = [said("Lens, and Mondays?")]
assert run(db, cfg) == 1
assert [r["scope"] for r in db.rows("SELECT scope FROM chat WHERE kind = 'bridge'")] == [{}]
assert bridge.room_namespace({"bridge": {"rooms": ["team = pods", "team/x = calls"]}}, {"gateway": "team", "channel": "x"}) == "calls"
94 changes: 94 additions & 0 deletions nextjs-frontend/__tests__/ns-assistant.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { fireEvent, render, screen, waitFor } from "@testing-library/react";
import "@testing-library/jest-dom";

import { Namespaces } from "@/app/openapi-client";
import { NsAssistantSection } from "@/components/assistant/ns-assistant-section";

jest.mock("@/app/openapi-client", () => ({
Namespaces: {
getNamespaceAssistant: jest.fn(),
listAssistantMemories: jest.fn(),
updateNamespaceAssistant: jest.fn(),
addAssistantMemory: jest.fn(),
updateAssistantMemory: jest.fn(),
forgetAssistantMemory: jest.fn(),
},
}));
jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) }));

const ok = (data: unknown) => Promise.resolve({ data, response: { ok: true, status: 200 } });
const m = (f: unknown) => f as jest.Mock;
const OFF = { enabled: false, name: "Assistant", instructions: "", memories: 2 };
const MEMORIES = [
{ id: 2, text: "Episodes ship on Fridays.", author: "person", pinned: true, created_at: "2026-10-05T01:00:00Z" },
{
id: 1,
text: "Dyno designs capsids.",
author: "assistant",
pinned: false,
recording: 7,
t0: 65000,
title: "Episode 1",
time: "1:05",
created_at: "2026-10-04T01:00:00Z",
},
];

function show(isOwner = true) {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
return render(
<QueryClientProvider client={qc}>
<NsAssistantSection ns="pods" isOwner={isOwner} />
</QueryClientProvider>,
);
}

beforeEach(() => {
jest.clearAllMocks();
m(Namespaces.getNamespaceAssistant).mockReturnValue(ok(OFF));
m(Namespaces.listAssistantMemories).mockReturnValue(ok(MEMORIES));
});

test("lists memories with where they came from, and owners turn the assistant on", async () => {
m(Namespaces.updateNamespaceAssistant).mockReturnValue(ok({ ...OFF, enabled: true }));
show();
expect(await screen.findByText("Dyno designs capsids.")).toBeInTheDocument();
expect(screen.getByRole("link", { name: "Episode 1 at 1:05" })).toHaveAttribute("href", "/resources/7?t=65");
expect(screen.getByText("Written here", { exact: false })).toBeInTheDocument();
expect(screen.getByText("Pinned")).toBeInTheDocument();
fireEvent.click(screen.getByRole("switch", { name: "pods’s assistant" }));
await waitFor(() =>
expect(Namespaces.updateNamespaceAssistant).toHaveBeenCalledWith(
expect.objectContaining({ body: { enabled: true } }),
),
);
});

test("saving a name and forgetting a memory", async () => {
m(Namespaces.updateNamespaceAssistant).mockReturnValue(ok({ ...OFF, name: "Pod pal" }));
m(Namespaces.forgetAssistantMemory).mockReturnValue(ok({ ok: true }));
show();
const name = await screen.findByLabelText("Name");
expect(screen.getByRole("button", { name: "Save" })).toBeDisabled();
fireEvent.change(name, { target: { value: "Pod pal" } });
fireEvent.click(screen.getByRole("button", { name: "Save" }));
await waitFor(() =>
expect(Namespaces.updateNamespaceAssistant).toHaveBeenCalledWith(
expect.objectContaining({ body: { name: "Pod pal", instructions: "" } }),
),
);
fireEvent.click(screen.getByRole("button", { name: "Forget: Dyno designs capsids." }));
await waitFor(() =>
expect(Namespaces.forgetAssistantMemory).toHaveBeenCalledWith(
expect.objectContaining({ path: { name: "pods", mid: 1 } }),
),
);
});

test("only owners change it", async () => {
show(false);
expect(await screen.findByLabelText("Name")).toBeDisabled();
expect(screen.getByRole("switch", { name: "pods’s assistant" })).toBeDisabled();
expect(screen.queryByRole("button", { name: "Save" })).not.toBeInTheDocument();
});
2 changes: 2 additions & 0 deletions nextjs-frontend/components/admin/namespaces.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { useState } from "react";

import { Namespaces, Pipelines, Users } from "@/app/openapi-client";
import { IpGroupsSection } from "@/components/access/ip-groups";
import { NsAssistantSection } from "@/components/assistant/ns-assistant-section";
import { AdminFrame, usePeople } from "@/components/admin/admin-frame";
import { roleLabel, type Role } from "@/components/admin/people-model";
import { ActivityPanel, BudgetPanel, CostCell, useBudgets, useCosts } from "@/components/costs/costs";
Expand Down Expand Up @@ -472,6 +473,7 @@ export function NamespaceDetail({ ns }: { ns: string }) {
</ul>
)}
</section>
<NsAssistantSection ns={ns} isOwner={isOwner} />
<IpGroupsSection ns={ns} isOwner={isOwner} admin={admin} />
<NotificationsSection ns={ns} isOwner={isOwner} admin={admin} />
{known && <VaultSection ns={ns} isOwner={isOwner} />}
Expand Down
Loading
Loading
0