8000 feat(notes): count note refinement and filing in the activity ledger by adeelahmad · Pull Request #133 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ Todo:
- [x] Editor: BlockSuite (AFFiNE's block editor on Yjs documents, what OctoBase stores) behind one component; plain
text kept for search and the assistant
- [x] AI title and summary refinement (ai.refine_notes, on by default, can be switched off)
- [ ] Costs of refinement to the activity ledger (once "Activity history and budgets&qu 8000 ot; lands)
- [x] Costs of refinement and filing to the activity ledger (`note_page:id` and its namespace; shown on the page)
- [x] Assistant tools: find, read, write and update notes (at once, no approval: notes are its notebook)
- [ ] MCP tools to read notes; page history so the assistant's changes can be undone
- [x] Self-organising, first step: notes nobody filed are filed in PARA by a decision model (or the LLM), with a
Expand Down
1 change: 1 addition & 0 deletions docs/activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Each row names every resource it counts for, as `table:id`:
| A job, its recording, namespace and pipeline | `job:40`, `recording:7`, `space:1`, `pipeline:2` |
| A workflow | `workflow:4` |
| A chat | `chat:9` |
| A note (its refinement and filing, and the requests that changed it) | `note_page:5` |
| The person who asked | `account:1` |

A model call inside a workflow step of a job that a routine queued counts for the workflow, the job, the recording,
Expand Down
5 changes: 3 additions & 2 deletions docs/notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,9 @@ With **Settings → Assistant → Keep note titles and summaries up to date** (`
whose title or text changed is sent to the model once nobody has touched it for two minutes. The model rewrites the
one-line summary to fit the whole note, and the title only when it is empty, "Untitled" or no longer fits. A note being
typed is never sent on each keystroke, empty notes are never sent, and nothing is sent without a model configured. The
API and `lens worker` processes run this pass with the routines. Model calls are measured by telemetry; their costs go
to the activity ledger once it exists.
API and `lens worker` processes run this pass with the routines. Each model call, and each filing decision, counts in
the [activity ledger](activity.md) for the note (`note_page:5`) and its namespace; a note's page shows its costs and
activity at the bottom.

## Filing (PARA)

Expand Down
10 changes: 8 additions & 2 deletions fastapi_backend/app/api/v1/routes/activity.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""The activity ledger (docs/activity.md): a resource's history (its calls in and out, runs and changes) and what it
cost. Admins see every resource's; others see their own account's, their chats', and the recordings and namespaces they
can read."""
cost. Admins see every resource's; others see their own account's, their chats', and the recordings, notes and namespaces
they can read."""

from __future__ import annotations

Expand Down Expand Up @@ -40,6 +40,12 @@ def _may_see(db: Db, user: CurrentUser, acl: Acl, resource: str) -> None:
raise HTTPException(404, "not found")
acl.need(e["space"])
return
if table == "note_page":
page = db.one("SELECT space FROM $r", r=activity.R("note_page", int(key)))
if not page:
raise HTTPException(404, "not found")
acl.need(page["space"])
return
if table == "chat" and db.one("SELECT id FROM $r WHERE account = $a", r=activity.R("chat", int(key)), a=user.id):
return
raise HTTPException(403, "only admins see this resource's activity")
Expand Down
16 changes: 12 additions & 4 deletions fastapi_backend/app/domain/activity.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
"watches": "watch_path",
"sensors": "sensor",
"templates": "template",
"notes": "note",
"notes": "note", # a recording's notes; /api/v1/notes/{id} are note pages (_table)
"comments": "comment",
"entities": "entity",
"speakers": "speaker",
Expand Down Expand Up @@ -120,13 +120,20 @@ def current():
return list(dict.fromkeys(r for r in out if REF.match(r)))


def _table(parts, i):
"""The table a key after path segment i names: /api/v1/notes/{id} are note pages, a recording's notes/{id} its notes."""
if parts[i] == "notes" and i and parts[i - 1] == "v1":
return "note_page"
return TABLES.get(parts[i])


def request_refs(asgi):
"""An API request's resources: each number (or path parameter) after a known path segment, and who called."""
out = []
params = {str(v) for v in (asgi.get("path_params") or {}).values()}
parts = [p for p in (asgi.get("path") or "").split("/") if p]
for a, b in zip(parts, parts[1:]):
table = TABLES.get(a)
for i, b in enumerate(parts[1:]):
table = _table(parts, i)
if table and (b.isdigit() or b in params) and b not in TABLES:
out.append(ref(table, b))
p = (asgi.get("state") or {}).get("principal")
Expand Down Expand Up @@ -322,7 +329,8 @@ def _log(self, scope_, app, db, status, error, t0, head=b""):

def created(path, head):
"""What a POST to a collection made, from its reply ({"id": n}): `routine:7` for POST /api/v1/routines."""
table = TABLES.get(path.rstrip("/").rsplit("/", 1)[-1])
parts = [p for p in path.split("/") if p]
table = _table(parts, len(parts) - 1) if parts else None
if not table or not head:
return None
try:
Expand Down
17 changes: 12 additions & 5 deletions fastapi_backend/app/domain/notebook.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
import datetime as dt
import re

from . import store
from . import activity, store

R = store.R
PLACES = ("project", "area", "resource", "archive")
Expand Down Expand Up @@ -450,19 +450,25 @@ def follow(db, thing, dst):
)


def _refs(p):
"""What the model calls made for a page count for in the activity ledger: the page and its namespace."""
return (f"note_page:{p['id']}", f"space:{p['space']}")


def refine(db, cfg, pid):
"""Ask the model for the page's title and summary; saved unless someone changed the page meanwhile. Returns what
changed: {"title"?, "summary"?}."""
from . import llm

p = db.one("SELECT record::id(id) AS id, title, summary, text, updated_at FROM $r", r=R("note_page", int(pid)))
p = db.one("SELECT record::id(id) AS id, space, title, summary, text, updated_at FROM $r", r=R("note_page", int(pid)))
if not p:
raise KeyError(pid)
if not (p.get("text") or "").strip():
db.q("UPDATE $r SET refine_pending = false, refine_claim = NONE", r=R("note_page", p["id"]))
return {}
user = f"Title: {p['title']}\nSummary: {p.get('summary') or '(none)'}\n\nNote:\n{p['text'][:REFINE_CHARS]}"
out = llm.json_out(cfg, REFINE_SYSTEM, user, REFINE_SCHEMA)
with activity.scope(db, *_refs(p), cfg=cfg):
out = llm.json_out(cfg, REFINE_SYSTEM, user, REFINE_SCHEMA)
title = " ".join(str(out.get("title") or "").split())[:TITLE_MAX] or p["title"]
summary = " ".join(str(out.get("summary") or "").split())[:SUMMARY_MAX] or p.get("summary")
now = db.one("SELECT updated_at FROM $r", r=R("note_page", p["id"]))
Expand Down Expand Up @@ -528,15 +534,16 @@ def file_due(db, cfg, log=None):
if not cfg["ai"].get("organise_notes", True) or not decide.engine(cfg):
return 0
rows = db.rows(
"SELECT record::id(id) AS id, title, summary, text, updated_at FROM note_page "
"SELECT record::id(id) AS id, space, title, summary, text, updated_at FROM note_page "
"WHERE about = NONE AND filed != true AND refine_pending != true LIMIT $n",
n=FILE_BATCH,
)
done = 0
for p in rows:
state = {"title": p["title"], "summary": p.get("summary"), "text": (p.get("text") or "")[:4000]}
try:
d = decide.choose(cfg, "Where does this note belong in PARA (projects, areas, resources, archives)?", PARA, state)
with activity.scope(db, *_refs(p), cfg=cfg):
d = decide.choose(cfg, "Where does this note belong in PARA (projects, areas, resources, archives)?", PARA, state)
except decide.Undecided as e:
if log:
log(f"notes: couldn't file page {p['id']}: {e}")
Expand Down
5 changes: 5 additions & 0 deletions fastapi_backend/tests/api/test_notebook.py
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,11 @@ def test_refining_titles_and_summaries(client, env, db, cfg, llm):
assert notebook.refine_due(db, cfg) == 1
got = client.get(f"/api/v1/notes/{p['id']}", headers=he).json()
assert (got["title"], got["summary"], got["summary_by"]) == ("Capsid plan", "About Capsid samples ship on Friday.", "assistant")
# what it cost counts for the page and its namespace, and the page's readers see it with the requests that changed it
led = client.get("/api/v1/activity", headers=he, params={"resource": f"note_page:{p['id']}"}).json()
kinds = {(e["kind"], e["action"]) for e in led}
assert ("out", "model.chat") in kinds and any(k == "in" for k, _ in kinds), kinds
assert any(e["kind"] == "out" and f"space:{store.ns_id(db, got['namespace'])}" in e["resources"] for e in led)
# done until the note changes; a person's title change makes it due again, and their own title stays
assert notebook.refine_due(db, cfg) == 0
client.patch(f"/api/v1/notes/{p['id']}", headers=he, json={"title": "Shipping"})
Expand Down
2 changes: 2 additions & 0 deletions nextjs-frontend/components/notes/note-page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { useCallback, useEffect, useRef, useState } from "react";

import { Notes } from "@/app/openapi-client";
import type { NotePage as Page, NotePageDraft as PageDraft } from "@/app/openapi-client/types.gen";
import { ActivityPanel } from "@/components/costs/costs";
import type { EditorChange, LinkTarget } from "@/components/notes/block-editor";
import { PLACES, hrefFor } from "@/components/notes/links";
import { Button } from "@/components/ui/button";
Expand Down Expand Up @@ -327,6 +328,7 @@ export function NotePage({ id, about }: { id?: number; about?: string }) {
>
/>
<Backlinks items={page.backlinks ?? []} />
{saved && <ActivityPanel resource={`note_page:${saved.id}`} title="Costs and activity" />}
</article>
);
}
Expand Down
Loading
0