8000 feat(notes): keep note titles and summaries up to date with the model by adeelahmad · Pull Request #125 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,12 @@ Todo:

- [x] Pages: create, read, change, move in the tree, delete; a page per resource made on first open
- [x] @ and # mentions: links and backlinks, a search to pick what to link
- [ ] Web app: tree explorer in the left navigation, page view with title, summary, date, place and backlinks; the
- [x] Web app: tree explorer in the left navigation, page view with title, summary, date, place and backlinks; the
page of a recording or entity reached from its detail view
- [ ] Editor: BlockSuite (AFFiNE's block editor, OctoBase/Yjs documents) behind one component; plain text kept for
search and the assistant. Waits on Adeel's link for the exact editor
- [ ] AI title and summary refinement (notes.refine, on by default, can be switched off); costs to the activity ledger
- [x] Editor: BlockSuite (AFFiNE's block editor on Yjs documents, what OctoBase stores) behind one component; plain
text kept for search and the assistant
- [x] AI title and summary refinement (ai.refine_notes, on by default, can be switched off)
- [ ] Costs of refinement to the activity ledger (once "Activity history and budgets" lands)
- [ ] Assistant and MCP tools: find, read, write, link and file notes; changes by the assistant are undoable
- [ ] Self-organising: a routine files notes into PARA and links them to entities and topics, proposing what it isn't
sure of
Expand Down
15 changes: 12 additions & 3 deletions docs/notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
Every resource, entity and topic in Lens has a page of its own, and next to them sit free notes that people and the
assistant write. The assistant is the main writer and organiser; people can do everything it can.

Status: **built**: pages over the API (free notes in a tree, a page per recording, entity, collection or speaker),
@ and # links with backlinks. **In progress**: the tree in the left navigation, the page view and its editor, AI
refinement of titles and summaries. **Planned**: assistant and MCP tools, a routine that files notes (PARA and the
Status: **built**: pages (free notes in a tree, a page per recording, entity, collection or speaker), @ and # links
with backlinks, the tree in the left navigation, the page view with the BlockSuite editor, and the model keeping
titles and summaries up to date. **Planned**: assistant and MCP tools, a routine that files notes (PARA and the
ontology), attachments on encrypted object storage, pages in the graph, # links to SKOS topics.

## A page
Expand Down Expand Up @@ -40,6 +40,15 @@ vocabulary of their own; see [the graph](graph.md)). `GET /api/v1/notes/targets?
Links are kept as rows (`note_link`), so every page lists its backlinks; links to things that are gone or that the
reader can't see show without a name.

## Titles and summaries

With **Settings → Assistant → Keep note titles and summaries up to date** (`ai.refine_notes`, on by default), a note
whose title or text changed is sent to the model once nobody has touched it for two minutes. The model rewrites the
one-line summary to fit the whole note, and the title only when it is empty, "Untitled" or no longer fits. A note being
typed is never sent on each keystroke, empty notes are never sent, and nothing is sent without a model configured. The
API and `lens worker` processes run this pass with the routines. Model calls are measured by telemetry; their costs go
to the activity ledger once it exists.

## Access

Pages belong to a namespace: anyone with a role there reads them, editors write, move and delete them. The pages of
Expand Down
80 changes: 80 additions & 0 deletions fastapi_backend/app/domain/notebook.py
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,7 @@ def create(db, sid, account, title, body="", summary=None, date=None, place=None
"text": plain(body),
"doc": _doc(doc),
"author": author,
"refine_pending": True,
"created_by": account,
"updated_by": account,
"created_at": t,
Expand Down Expand Up @@ -253,6 +254,8 @@ def update(db, pid, account, title=None, body=None, summary=UNSET, date=UNSET, p
if place is not UNSET:
sets.append("place = $place")
args["place"] = _place(place)
if title is not None or body is not None:
sets.append("refine_pending = true")
if body is not None:
sets += ["body = $body", "text = $text"]
args["body"] = _body(body)
Expand Down Expand Up @@ -398,3 +401,80 @@ def follow(db, thing, dst):
for p in db.rows("SELECT record::id(id) AS id FROM note_page WHERE about = $a", a=thing):
db.q("UPDATE $r SET space = $s, about_key = $k", r=R("note_page", p["id"]), s=dst, k=f"{dst}:{kind}:{key}")
db.q("UPDATE note_link SET space = $s WHERE page = $p", s=dst, p=p["id"])


# ---------- refining titles and summaries (ai.refine_notes) ----------
QUIET_SECONDS = 120 # a page is refined once nobody has changed it for this long
REFINE_BATCH = 5 # pages refined per pass
REFINE_CHARS = 12_000 # of a page's text the model reads
REFINE_SCHEMA = {
"type": "object",
"properties": {"title": {"type": "string"}, "summary": {"type": "string"}},
"required": ["title", "summary"],
}
REFINE_SYSTEM = (
"You keep a note's title and its one-line summary true to the whole note. The summary is the context an assistant "
"reads before deciding whether to open the note, so it says what the note holds, specifically, in one sentence of "
"at most 200 characters. Keep the title as it is unless it is empty, 'Untitled' or no longer describes the note; "
"then give a short, specific title of at most 80 characters. Write in the note's language."
' Reply with JSON: {"title": ..., "summary": ...}.'
)


def refine(db, cfg, pid):
"""Ask the model for the page's title and summary; saved unless someone changed the page meanwhile. Returns what
changed: {"title"?, "summary"?}."""
from . import llm

p = db.one("SELECT record::id(id) AS id, title, summary, text, updated_at FROM $r", r=R("note_page", int(pid)))
if not p:
raise KeyError(pid)
if not (p.get("text") or "").strip():
db.q("UPDATE $r SET refine_pending = false, refine_claim = NONE", r=R("note_page", p["id"]))
return {}
user = f"Title: {p['title']}\nSummary: {p.get('summary') or '(none)'}\n\nNote:\n{p['text'][:REFINE_CHARS]}"
out = llm.json_out(cfg, REFINE_SYSTEM, user, REFINE_SCHEMA)
title = " ".join(str(out.get("title") or "").split())[:TITLE_MAX] or p["title"]
summary = " ".join(str(out.get("summary") or "").split())[:SUMMARY_MAX] or p.get("summary")
now = db.one("SELECT updated_at FROM $r", r=R("note_page", p["id"]))
if not now or now.get("updated_at") != p.get("updated_at"):
db.q("UPDATE $r SET refine_claim = NONE", r=R("note_page", p["id"]))
return {} # changed while the model was thinking: it's still pending, so a later pass looks again
changed = {k: v for k, v in (("title", title), ("summary", summary)) if v and v != p.get(k)}
sets = ["refine_pending = false", "refine_claim = NONE", "refined_at = $t"] + [f"{k} = ${k}" for k in changed]
if "summary" in changed:
sets.append("summary_by = 'assistant'")
db.q(f"UPDATE $r SET {', '.join(sets)}", r=R("note_page", p["id"]), t=store.now(), **changed)
return changed


def refine_due(db, cfg, log=None):
"""Refine the pages whose title or text changed since they were last refined and that nobody has touched for a
while (so a page being typed isn't sent on every keystroke). Off with ai.refine_notes, or without a model."""
from . import llm

if not cfg["ai"].get("refine_notes", True) or not llm.configured(cfg):
return 0
t = dt.datetime.now(dt.timezone.utc)
quiet = (t - dt.timedelta(seconds=QUIET_SECONDS)).isoformat(timespec="seconds")
rows = db.rows(
"SELECT record::id(id) AS id, refine_claim FROM note_page WHERE refine_pending = true AND updated_at < $q LIMIT $n",
q=quiet,
n=REFINE_BATCH * 4,
)
done = 0
for p in rows:
if done >= REFINE_BATCH:
break
if (p.get("refine_claim") or "") > quiet:
continue # another process took it a moment ago
db.q("UPDATE $r SET refine_claim = $c", r=R("note_page", p["id"]), c=store.now())
try:
refine(db, cfg, p["id"])
except Exception as e: # noqa: BLE001 - one page failing doesn't stop the others
if log:
log(f"notes: couldn't refine page {p['id']}: {type(e).__name__}: {e}")
# not again until it changes
db.q("UPDATE $r SET refine_pending = false, refine_claim = NONE", r=R("note_page", p["id"]))
done += 1
return done
17 changes: 16 additions & 1 deletion fastapi_backend/app/domain/routines.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,21 @@
import datetime as dt
import threading

from . import fedora, jobs, organize, schedule, semantic, sensor_digests, sensor_patterns, sensors, sources, store, telemetry, workflows
from . import (
fedora,
jobs,
notebook,
organize,
schedule,
semantic,
sensor_digests,
sensor_patterns,
sensors,
sources,
store,
telemetry,
workflows,
)

R = store.R
ACTIONS = ("sync", "pipeline", "workflow", "sensors")
Expand Down Expand Up @@ -524,6 +538,7 @@ def loop():
every(lambda: cfg_fn()["sources"]["check_seconds"], sources.poll_due, "watched folders"),
every(lambda: CHECK_SECONDS, run_due, "routines"),
every(lambda: cfg_fn()["fedora"]["sync_seconds"], fedora.sync_due, "fedora"),
every(lambda: notebook.QUIET_SECONDS / 2, notebook.refine_due, "notes"),
]


Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,7 @@ def labels():
"ai": {
"tools": True,
"extensions": True, # tools, skills, hooks and plugins people add (extensions.py)
"refine_notes": True, # the model keeps notes' titles and one-line summaries true to the whole note (notebook.py)
"disabled_tools": [],
"max_steps": 6,
"max_transcript_reads": 20,
Expand Down Expand Up @@ -899,6 +900,7 @@ def close(self):
"DEFINE INDEX IF NOT EXISTS note_page_space ON note_page FIELDS space",
"DEFINE INDEX IF NOT EXISTS note_page_parent ON note_page FIELDS parent",
"DEFINE INDEX IF NOT EXISTS note_page_about ON note_page FIELDS about_key UNIQUE",
"DEFINE INDEX IF NOT EXISTS note_page_refine ON note_page FIELDS refine_pending",
# the links in a page's body (@ and # mentions), for backlinks and the graph
"DEFINE TABLE IF NOT EXISTS note_link SCHEMALESS",
"DEFINE INDEX IF NOT EXISTS note_link_page ON note_link FIELDS page",
Expand Down
48 changes: 48 additions & 0 deletions fastapi_backend/tests/api/test_notebook.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,51 @@ def test_domain_helpers():
body = "**Hi** @[Ada](entity:5) and #[Caps](entity:9), @[Ada](entity:5) twice, @[x](nope:1)\n\n- [ ] [site](http://x)"
assert notebook.mentions(body) == [("@", "entity", 5, "Ada"), ("#", "entity", 9, "Caps")]
assert notebook.plain(body) == "Hi Ada and Caps, Ada twice, x\n\nsite"


@pytest.fixture
def llm(cfg):
from tests import fake_llm

srv, url = fake_llm.start()
cfg["llm"].update(base_url=url, model="fake")
fake_llm.Handler.seen = []
yield fake_llm.Handler
srv.shutdown()


def _age(db, pid, seconds=600):
import datetime as dt

old = (dt.datetime.now(dt.UTC) - dt.timedelta(seconds=seconds)).isoformat(timespec="seconds")
db.q("UPDATE $r SET updated_at = $t", r=R("note_page", pid), t=old)


def test_refining_titles_and_summaries(client, env, db, cfg, llm):
he = env["he"]
p = _new(client, he, title="Untitled", body="Capsid samples ship on Friday.\n\nAlice sends them.")
# nothing happens while someone may still be typing
assert notebook.refine_due(db, cfg) == 0
_age(db, p["id"])
assert notebook.refine_due(db, cfg) == 1
got = client.get(f"/api/v1/notes/{p['id']}", headers=he).json()
assert (got["title"], got["summary"], got["summary_by"]) == ("Capsid plan", "About Capsid samples ship on Friday.", "assistant")
# done until the note changes; a person's title change makes it due again, and their own title stays
assert notebook.refine_due(db, cfg) == 0
client.patch(f"/api/v1/notes/{p['id']}", headers=he, json={"title": "Shipping"})
_age(db, p["id"])
assert notebook.refine_due(db, cfg) == 1
assert client.get(f"/api/v1/notes/{p['id']}", headers=he).json()["title"] == "Shipping"
# switched off, nothing is sent
client.patch(f"/api/v1/notes/{p['id']}", headers=he, json={"body": "New text"})
_age(db, p["id"])
cfg["ai"]["refine_notes"] = False
n = len(llm.seen)
assert notebook.refine_due(db, cfg) == 0 and len(llm.seen) == n
# empty notes aren't sent
cfg["ai"]["refine_notes"] = True
e = _new(client, he, title="Empty")
_age(db, e["id"])
_age(db, p["id"])
notebook.refine_due(db, cfg)
assert len(llm.seen) == n + 1
6 changes: 6 additions & 0 deletions fastapi_backend/tests/fake_llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,12 @@ def do_POST(self):
)
elif set(schema.get("properties", {})) == {"names"}:
content = json.dumps({"names": Handler.names or ["travel", "Family Trips", "calls"]})
elif set(schema.get("properties", {})) == {"title", "summary"}:
# a note's title and summary (notebook.refine): the title kept, the first line of the note as its summary
text = body["messages"][-1]["content"]
title = re.search(r"^Title: (.*)$", text, re.M).group(1)
first = text.split("Note:\n", 1)[1].strip().splitlines()[0]
content = json.dumps({"title": "Capsid plan" if title == "Untitled" else title, "summary": f"About {first}"})
elif set(schema.get("properties", {})) == {"choice", "confidence"}:
content = json.dumps(Handler.decision or {"choice": schema["properties"]["choice"]["enum"][0], "confidence": 0.9})
elif body.get("response_format"):
Expand Down
1 change: 1 addition & 0 deletions nextjs-frontend/components/settings/bodies.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -951,6 +951,7 @@ function AiBody({ ctx }: { ctx: BodyCtx }) {
The assistant uses the model set in LLM provider; it needs one that supports tool calls.
</p>
<F ctx={ctx} id="ai.tools" />
<F ctx={ctx} id="ai.refine_notes" />
<div className="flex flex-col gap-2">
<span className="text-[13px] font-bold leading-tight text-fg-strong">Tools</span>
<ul className="flex flex-col gap-2">
Expand Down
7 changes: 7 additions & 0 deletions nextjs-frontend/components/settings/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -613,6 +613,13 @@ export const FIELDS: FieldSpec[] = [
hint: "Without tools, answers use search results only.",
},
{ section: "ai", key: "disabled_tools", label: "Tools", kind: "checks" },
{
section: "ai",
key: "refine_notes",
label: "Keep note titles and summaries up to date",
kind: "switch",
hint: "A couple of minutes after a note changes, the model rewrites its one-line summary, and its title when that no longer fits.",
},
{
section: "ai",
key: "max_steps",
Expand Down
Loading
0