8000 feat(chat): /clear and /compact commands, and early compaction near the context limit by adeelahmad · Pull Request #269 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,16 @@ then.
behind them, they're folded into a running summary kept on the conversation (one model call, after an answer), and
every answer gets that summary plus the messages it doesn't cover yet. Room conversations (chat-rooms.md) go on for
good, so they rely on it most. Editing a question the summary covers drops the summary; it's made again from what's
left. Settings › AI assistant › Summarise long conversations (`ai.compact`, on) turns it off; off, or with no model,
only the last six messages are read.
left. A conversation that fills half of the model's context window (**Chat model's context**, `ai.context_tokens`,
16000 when empty; about four characters a token) is folded in early, before the next answer, whatever the count.
**Model for summaries** (`ai.compact_model`) writes them with a smaller, cheaper model on the same provider; empty,
the chat's model does. Settings › AI assistant › Summarise long conversations (`ai.compact`, on) turns it off; off,
or with no model, only the last six messages are read.
- **/clear and /compact:** typed as a message, in the app or in a chat room (`Lens /clear`). `/clear` makes the
assistant start afresh: the messages stay to read, but the model no longer sees them or their summary.
`/compact` folds the whole conversation into its summary now, and `/compact keep the dates and owners` tells the
summary what to keep; the reply shows the summary. Both work with automatic summaries off. Neither command, nor its
reply, is read by the model afterwards.
- **Changed access:** old citations are filtered by the person's current access when a conversation is reopened.
- **Choosing the model:** a conversation can use any model an admin offers (`llm.chat_models`, else whatever the model
server lists); Try another model asks a question again with a different one. Each answer records the model that
Expand Down
3 changes: 2 additions & 1 deletion docs/chat-rooms.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,8 @@ That's all. Without an account given, it answers as the admin who turned it on.
said yes.
- **Conversations.** Each person in each room is a conversation of that account's, titled
"Matterbridge · room · person", listed in Chat with the account's other conversations. Follow-ups keep their context,
and older messages are folded into a running summary ([Long conversations](assistant.md)).
and older messages are folded into a running summary ([Long conversations](assistant.md)). `Lens /clear` starts
the conversation afresh and `Lens /compact` summarises it now.
- **A namespace's own assistant.** **Rooms for a namespace's assistant** gives a room to a namespace
(`team = pods`, or one channel: `team/general = pods`). Conversations there are scoped to that namespace, so when its
assistant is on ([A namespace's own assistant](assistant.md#a-namespaces-own-assistant)) it answers with its
Expand Down
20 changes: 19 additions & 1 deletion fastapi_backend/app/api/v1/routes/chats.py
Original file line number Diff line number Diff line change
Expand Up @@ -180,14 +180,28 @@ async def send_message(cid: int, body: MessageCreate, user: Writer, acl: Acl, db

With `edit`, one of your earlier questions is edited: it and everything after it are replaced by this question and
its new answer (404 if it isn't a question in this conversation). Without `context`, it keeps the page it was asked
from (where, and the highlighted text; not the page's text, which isn't kept)."""
from (where, and the highlighted text; not the page's text, which isn't kept).

`/clear` makes the model start afresh (the messages stay to read) and `/compact [what to keep]` folds the
conversation so far into its summary now; each streams cleared or compacted, then its reply as an answer."""
c = await run_in_threadpool(_own_chat, db, cid, user)
q = body.content.strip()[:4000]
files = await run_in_threadpool(_attachments, db, body.attachments, user)
if not q and files:
q = "I attached " + ("this file." if len(files) == 1 else "these files.")
if not q:
raise HTTPException(400, "ask something")
cmd = chat.command(q) if not files and body.edit is None else None
if cmd: # /clear or /compact: done here, answered like a question, without asking the model anything else

def run() -> Iterator[str]:
reply, mid = chat.run_command(db, cfg, cid, *cmd)
yield _ev({"clear": "cleared", "compact": "compacted"}[cmd[0]], {"message": mid})
yield _ev("passages", [])
yield _ev("token", {"text": reply})
yield _ev("done", {"message": mid})

return StreamingResponse(run(), media_type="text/event-stream", headers=SSE_HEADERS)
# this answer's model: the one asked for (400 if it isn't offered), else the conversation's while it's still offered
model = await run_in_threadpool(_model, cfg, body.model) if body.model else c.get("model")
if model and not body.model and model not in await run_in_threadpool(chat.model_choices, cfg):
Expand All @@ -211,6 +225,10 @@ async def send_message(cid: int, body: MessageCreate, user: Writer, acl: Acl, db

def prepare() -> tuple[list[dict[str, Any]], list[dict[str, Any]], Any]:
nonlocal summary
try: # a conversation that fills the model's context is folded into its summary before it's answered
chat.compact(db, cfg, cid, full_only=True)
except Exception: # noqa: BLE001 - answered with what fits; tried again after the answer
log.warning("chat %s: couldn't update the summary", cid, exc_info=True)
past = chat.history(db, cid)
summary = chat.memory(db, cfg, cid)
chat.add(db, cid, "user", q, attachments=files, context=kept)
Expand Down
13 changes: 13 additions & 0 deletions fastapi_backend/app/domain/bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,10 @@ def answer(db, cfg, base, account, cid, q):
roles = auth.roles(db, account)
readable = set(roles)
editable = {s for s in roles if auth.allows(roles, s, "editor")}
try: # a conversation that fills the model's context is folded into its summary before it's answered
chat.compact(db, cfg, cid, full_only=True)
except Exception: # noqa: BLE001 - answered with what fits; tried again after the answer
log.warning("bridge: couldn't update the summary of chat %s", cid, exc_info=True)
past = chat.history(db, cid)
summary = chat.memory(db, cfg, cid)
scope = (db.one("SELECT scope FROM $r", r=R("chat", cid)) or {}).get("scope") or None
Expand Down Expand Up @@ -415,6 +419,15 @@ def handle(db, cfg, base, m):
account = _account(db, cfg)
cid = conversation(db, account, m, ns if sid is not None else None)
note = None
cmd = chat.command(q)
if cmd: # /clear or /compact, for this room's conversation
try:
said = chat.run_command(db, cfg, cid, *cmd)[0]
except Exception: # noqa: BLE001 - still say something in the room
log.exception("bridge: /%s failed", cmd[0])
said = f"Couldn't {cmd[0]} this conversation. Try again."
post(cfg, m.get("gateway"), said, m.get("channel"))
return True
try:
decided = decide_in_room(db, cfg, base, account, cid, m, q)
if decided:
Expand Down
99 changes: 86 additions & 13 deletions fastapi_backend/app/domain/chat.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,10 +265,11 @@ def with_context(question, ctx):
def past_turns(history, n=6, summary=None):
"""The conversation's last messages for the model, with the files sent with each; a question asked about
highlighted text keeps (the start of) that text, so a follow-up still knows what "it" was. With a `summary`
(compaction is on), every message it doesn't cover yet, up to RECENT + COMPACT_AFTER of them."""
(compaction is on, or /clear or /compact was used), every message it doesn't cover yet, up to RECENT + COMPACT_AFTER of
them (its "window" when automatic compaction is off)."""
if summary is not None:
history = [m for m in history if m["id"] > (summary.get("upto") or 0)]
n = RECENT + COMPACT_AFTER
n = summary.get("window") or RECENT + COMPACT_AFTER
out = []
for m in list(history)[-n:]:
sel = ((m.get("context") or {}).get("selection") or "")[:500] if m["role"] == "user" else ""
Expand Down Expand Up @@ -383,6 +384,8 @@ def rewind(db, cid, mid):
RECENT = 6 # the latest messages, always given word for word
COMPACT_AFTER = 8 # how many more build up before they're folded into the summary (one model call per that many)
TURN_CHARS = 1500 # of each message, for the summary
CONTEXT_TOKENS = 16000 # the model's context window when ai.context_tokens isn't set
FULL = 0.5 # the share of it the conversation may take before it's folded in early (the rest: excerpts, tools, answer)
COMPACT_SYSTEM = (
"You keep the running summary of a conversation between a person and their assistant, so the assistant still "
"knows what was said once the messages themselves are out of view. Fold the new messages into the summary so far: "
Expand All @@ -393,12 +396,20 @@ def rewind(db, cid, mid):
)


def _stored(db, cid):
return (db.one("SELECT summary FROM $r", r=R("chat", cid)) or {}).get("summary") or {"text": "", "upto": 0}


def memory(db, cfg, cid):
"""What compaction knows of a conversation: {"text", "upto": the last message it covers, "at"}, empty before the
first summary; None when compaction is off (the model then sees the last 6 messages only, as before)."""
if not cfg["ai"].get("compact", True) or not llm.configured(cfg):
first summary; None when compaction is off and neither /clear nor /compact was used (the model then sees the last 6
messages only, as before)."""
if not llm.configured(cfg):
return None
return (db.one("SELECT summary FROM $r", r=R("chat", cid)) or {}).get("summary") or {"text": "", "upto": 0}
had = _stored(db, cid)
if cfg["ai"].get("compact", True):
return had
return {**had, "window": RECENT} if had.get("upto") else None


def summary_note(summary):
Expand All @@ -414,23 +425,85 @@ def _said(m):
return f"{who}: {text}" + (f"\n(What it did: {did[:500]})" if did else "")


def compact(db, cfg, cid):
def tokens(messages):
"""About how many tokens the messages take (four characters each), with the files noted on them."""
return sum(len(m.get("content") or "") + len(attached_note(m.get("attachments"))) for m in messages) // 4


def full(cfg, messages):
"""True when the messages fill more than FULL of the model's context window (ai.context_tokens)."""
return tokens(messages) > (cfg["ai"].get("context_tokens") or CONTEXT_TOKENS) * FULL


def compact(db, cfg, cid, force=False, focus=None, full_only=False):
"""Fold the messages older than the latest RECENT into the conversation's summary, once COMPACT_AFTER of them have
built up (ai.compact, on by default; needs a model). Returns the new summary, or None when nothing changed."""
had = memory(db, cfg, cid)
if had is None:
built up or they fill FULL of the model's context window (`full_only`: only then; ai.compact, on by default; needs a
model). `force` (/compact) folds every message in now, whatever the setting, with the person's `focus` (what to
keep) if any. The summary is written by ai.compact_model when set, a cheaper model than the chat's. Returns the new
summary (with "folded": how many messages), or None when nothing changed."""
if not llm.configured(cfg) or not (force or cfg["ai"].get("compact", True)):
return None
had = _stored(db, cid)
fresh = [m for m in history(db, cid) if m["id"] > (had.get("upto") or 0)]
if len(fresh) < RECENT + COMPACT_AFTER:
old = fresh if force else fresh[:-RECENT]
if not old:
return None
if not force and not full(cfg, fresh) and (full_only or len(fresh) < RECENT + COMPACT_AFTER):
return None
old = fresh[:-RECENT]
ask = f"Summary so far:\n{had.get('text') or '(none yet)'}\n\nNew messages:\n\n" + "\n\n".join(_said(m) for m in old)
text = llm.chat(cfg, [{"role": "system", "content": COMPACT_SYSTEM}, {"role": "user", "content": ask}], max_tokens=800).strip()
if focus:
ask += f"\n\nThe person asked the summary to keep: {focus[:500]}"
msgs = [{"role": "system", "content": COMPACT_SYSTEM}, {"role": "user", "content": ask}]
text = llm.chat(cfg, msgs, model=cfg["ai"].get("compact_model") or None, max_tokens=800).strip()
if not text:
return None
out = {"text": text, "upto": old[-1]["id"], "at": store.now()}
db.q("UPDATE $r SET summary = $s", r=R("chat", cid), s=out)
return out
return {**out, "folded": len(old)}


# ---------- commands: /clear and /compact, typed in the app or in a chat room ----------
COMMAND = re.compile(r"^/(clear|compact)(?:\s+(.*))?$", re.S | re.I)


def command(text):
"""(name, the rest) when the message is /clear or /compact [what to keep], else None."""
m = COMMAND.match((text or "").strip())
return (m.group(1).lower(), (m.group(2) or "").strip()) if m else None


def run_command(db, cfg, cid, name, rest=""):
"""Do a chat command and save it with its reply, which the model won't see afterwards. /clear: the model starts
afresh (the messages stay to read); /compact: everything so far is folded into the summary now. Returns (reply,
the reply's message id)."""
if name == "clear":
reply = "Cleared. I'll start afresh from here; the messages above stay for you to read, but I won't use them."
keep = None
elif not llm.configured(cfg):
reply = "There's no language model to write a summary with. /clear starts afresh instead."
keep = None
else:
try:
done, failed = compact(db, cfg, cid, force=True, focus=rest or None), None
except llm.LLMError as e:
done, failed = None, str(e)
keep = _stored(db, cid)
if failed:
reply, keep = f"I couldn't write the summary, so nothing changed: {failed}", None
elif done:
n = done["folded"]
reply = f"Compacted {n} message{'s' if n != 1 else ''} into a summary; from here on I read it instead:\n\n{done['text']}"
elif any(m["id"] > (keep.get("upto") or 0) for m in history(db, cid)):
reply, keep = "The model didn't write a summary, so nothing changed. Try again, or /clear to start afresh.", None
else:
reply = "Nothing new to compact yet."
add(db, cid, "user", f"/{name}" + (f" {rest}" if rest else ""))
mid = add(db, cid, "assistant", reply, [])
if name == "clear":
keep = {"text": "", "upto": mid, "at": store.now()}
if keep is not None: # the command and its reply are covered too, so the model doesn't read them
db.q("UPDATE $r SET summary = $s", r=R("chat", cid), s={**keep, "upto": mid})
return reply, mid


def model_choices(cfg):
Expand Down
8 changes: 8 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,14 @@ def _check(section, key, value, default):
if value is not None and not (isinstance(value, str) and len(value.strip()) <= 200):
raise ValueError("llm.vision_model is a model's name")
return (value or "").strip() or None
if (section, key) == ("ai", "compact_model"):
if value is not None and not (isinstance(value, str) and len(value.strip()) <= 200):
raise ValueError("ai.compact_model is a model's name")
return (value or "").strip() or None
if (section, key) == ("ai", "context_tokens"):
if value is not None and not (isinstance(value, int) and not isinstance(value, bool) and 1024 <= value <= 2_000_000):
raise ValueError("ai.context_tokens is a whole number of tokens from 1024 to 2000000")
return value
if (section, key) == ("llm", "describe_max"):
if not (isinstance(value, int) and not isinstance(value, bool) and 1 <= value <= 1000):
raise ValueError("llm.describe_max is a whole number from 1 to 1000")
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,8 @@ def labels():
"extensions": True, # tools, skills, hooks and plugins people add (extensions.py)
"refine_notes": True, # the model keeps notes' titles and one-line summaries true to the whole note (notebook.py)
"compact": True, # a long conversation keeps a summary of the messages the model no longer sees (chat.compact)
"compact_model": None, # a cheaper model to write those summaries (None: the chat's model)
"context_tokens": None, # the chat model's context window; summarised early before it fills (None: 16000)
"organise_notes": True, # notes nobody filed are filed in PARA, or a suggestion waits on them (notebook.py)
"disabled_tools": [],
"max_steps": 6,
Expand Down
16 changes: 16 additions & 0 deletions fastapi_backend/tests/api/test_bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -314,3 +314,19 @@ def ask(text):
assert ask("Lens, use the force")[0].startswith("OK\n") # not a namespace: a question, answered
cid = db.one("SELECT VALUE record::id(id) FROM chat WHERE kind = 'bridge' LIMIT 1")
assert "use pods" in [m["content"] for m in chat.history(db, cid)] # kept in the conversation like any message


def test_clear_and_compact_in_a_room(db, cfg, room, llm):
from app.domain import chat

cfg["ai"]["tools"] = False
room.waiting = [said("Lens, what leaves on Friday?")]
run(db, cfg)
room.waiting = [said("Lens /compact keep the shipment")]
run(db, cfg)
assert room.posted[-1]["text"].startswith("Compacted 2 messages")
room.waiting = [said("@Lens /clear")]
run(db, cfg)
assert room.posted[-1]["text"].startswith("Cleared")
cid = db.one("SELECT VALUE record::id(id) FROM chat WHERE kind = 'bridge' LIMIT 1")
assert chat.memory(db, cfg, cid)["text"] == ""
Loading
Loading
0