8000 feat(graph): a versioned, replayable entity graph with history, as-of, diffs, rollback and a History tab by adeelahmad · Pull Request #119 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension All 6 file types selected

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **The entity graph keeps its history.** Every change to entities, their other names, links across namespaces and
"not the same" pairs is recorded as a numbered version: what changed, who (a person, a routine, the assistant,
analysis), through what (the web app, a token, an app, MCP) and why. `GET /api/v1/graph/history` lists versions
(one entity's with `entity`), `/graph/as-of/{version}` shows the graph as it was, `/graph/diff` what changed
between two versions, and `/graph/tags` names versions to come back to; the graph API and Cypher take `as_of` to read the graph at a version. `POST /graph/rollback` takes the graph back to a version, after a
preview, undoing merges with their mentions; a rollback can be rolled back too. Checkpoints let the graph be replayed
from its history, and `lens history verify` (or `/graph/verify`) checks the replay matches today's graph. See
docs/graph-history.md.
In the web app, Routines › History shows every change and what it did, and names or rolls back to a version; an
entity's History button shows its own.
- **Apps and AI assistants in the setup wizard.** A new optional step, Apps and AI, asks whether apps and AI
assistants (Claude, ChatGPT, Cursor and other MCP clients) may sign people in with their Lens account, how long
their tokens last, and shows the MCP server's address to add to an assistant. Skipping it keeps OAuth on, as
Expand Down
37 changes: 37 additions & 0 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,43 @@

Plans and progress for work in flight. Newest first.

## 2026-10-04 · Graph history: versions, replay, rollback

Goal (Adeel): make the graph replayable and versioned.

Model (fastapi_backend/app/domain/graph_history.py, docs/graph-history.md):

- Every change to the curated entity graph is one append-only `graph_event`: its number is the graph's version. It
says when, who (a person, a routine, the assistant, analysis), through what (web app, API token, OAuth app,
assistant, MCP, routine, analysis), why, and for every record it touched (entity, alias, cross-namespace link,
"not the same" pair) the row before and after.
- Built on what is there: graph changes (organize.py: proposed, applied, dismissed, undone) and merges (entity_merge)
are recorded through the same entity functions, and their events point back at them.
- As of: the graph at any version, rebuilt by walking back from today's rows. Diff between two versions. An entity's
history. Versions can be named ("before the cleanup").
- Rollback to a version: a preview first, then later changes are undone as new events, so a rollback can itself be
undone. Merges are undone with their mentions.
- Replay: checkpoints (the graph at a version) plus the events after them rebuild the graph; verify compares the
replay with the live rows and records any difference as an event, so nothing stays unrecorded.
- Not versioned here: mentions (analysis makes them from transcripts, whose edits have their own history). As-of views
use today's mention counts.

Todo:

- [x] Event log with who, through what and why; every curation path records (entity page, assistant, MCP, routines
and graph changes, analysis, the entity map)
- [x] Versions API: list, the graph as of a version, diff, an entity's history, named versions
- [x] Rollback to a version, with a preview
- [x] Checkpoints, replay and verify (`lens graph verify`, nightly checkpoint)
- [x] The graph API and Cypher as of a version (`as_of`); the graph cache is keyed on the graph's version
- [ ] An as-of picker in the explorer (with the graph explorer thread)
- [x] Web app: a history timeline with diffs and rollback (Routines › History; an entity's History button)
- [ ] Graph events in each resource's activity history (with the activity history thread)

Refine later: speakers (same-person links, names) in the history; a mention-level history.



## 2026-10-04 · Activity history and budgets

Goal (Adeel): every change and every call coming in or going out is tracked, with its cost, down to the resource, so
Expand Down
114 changes: 114 additions & 0 deletions docs/graph-history.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Graph history

Every change to the entity graph is kept, in order, as an event that is never changed or deleted. Its number is the
graph's version. So the graph can be seen as it was at any version, two versions compared, and one entity's history
read: who renamed it, who merged what into it, which routine linked it, and why.

Built: the event log, versions, as-of, diffs, entity history, named versions, rollback to a version, checkpoints with
replay and verify, the History tab, the graph API and Cypher as of a version. Planned: an as-of picker in the
explorer (DEVLOG.md).

In the web app, Routines › History lists every version; open one to see what changed, name it, or roll back to it
(with a preview). An entity's History button shows only its changes.

## What is versioned

What people and agents curate:

- **Entities**: name, type, description, hidden (and why), on the fixed list, Unknown and Unlabeled.
- **Other names** (`entity_alias`): the ways a name is said that point at an entity, from merges, renames, the
entity page and names the model placed.
- **Links**: the same thing in two namespaces.
- **Distinct pairs**: two entities someone said are not one, so they aren't suggested again.

Mentions are not versioned: analysis makes them from transcripts, and transcript edits have their own history. An
as-of view uses today's mentions. A moved or removed mention is still an event (`mention.move`, `mention.remove`), so
it shows in both entities' histories.

## An event

| Field | What it holds |
| --- | --- |
| `version` | The graph's version after this change. |
| `at` | When. |
| `op` | What happened: `entity.rename`, `entity.describe`, `entity.retype`, `entity.hide`, `entity.show`, `entity.define`, `entity.undefine`, `entity.aliases`, `entity.delete`, `entity.merge`, `entity.unmerge`, `link.add`, `link.remove`, `distinct.add`, `mention.move`, `mention.remove`, `merge.apply`, `link.apply`, `merge.accept`, `link.accept`, `merge.dismiss`, `merge.undo`, `link.undo`, `analysis`. |
| `actor` | Who: the person's email, `routine:<id>`, `analysis`, or `system`. |
| `via` | Through what: `web`, `token` (an API token), `oauth` (an app), `assistant`, `mcp`, `routine`, `workflow`, `analysis`, `cli`, `system`. |
| `why` | The reason given: a graph change's reason, the model's verdict, why an entity was hidden. |
| `origin` | What it came from: `graph_change`, `merge` (the entity_merge to undo), `routine`, `run`, `workflow`, `recording`, `approval` (an assistant approval), `tool` (an MCP tool), `mention`. |
| `spaces`, `entities` | The namespaces and entities it touched. |
| `ops` | Every record it changed: `{t: table, k: key, b: the row before, a: the row after, s: namespaces}`; no `b` means it was made, no `a` that it went. |

One action is one event: a merge (the other entity going, its names becoming other names of the one kept, its links
moving) is one `entity.merge`, and a routine's merge is one `merge.apply` that points at its graph change.

## The API

All calls see only the namespaces the caller can read (`namespace` narrows it to one). An event that touched a
namespace the caller can't read (a link across namespaces) isn't listed for them. A version is a number, a version's
name, or `head`.

| Call | What it does |
| --- | --- |
| `GET /api/v1/graph/history?namespace=&entity=&before=&limit=` | Versions, newest first, with what changed, who, through what, why and the entities' names; `entity` gives one entity's hi 10000 story; `next` pages back. |
| `GET /api/v1/graph/history/{version}` | One version in full, every record before and after. |
| `GET /api/v1/graph/as-of/{version}?namespace=` | The graph at a version: entities with their other names, links and distinct pairs. |
| `GET /api/v1/graph/diff?from=&to=&namespace=` | What changed between two versions: entities added, removed and changed field by field; other names, links and distinct pairs added and removed. |
| `GET /api/v1/graph/tags` | Named versions. |
| `POST /api/v1/graph/tags` `{name, version, note}` | Name a version (default: the current one), such as "before the cleanup". Editors and admins. |
| `DELETE /api/v1/graph/tags/{name}` | Drop a name; the version stays. |
| `POST /api/v1/graph/rollback` `{to, namespace, dry_run}` | Take the graph back to a version. A preview by default: the changes it would undo, newest first, and the entities, names, links and pairs that would change. `dry_run: false` does it. |

## Rolling back

A rollback undoes every change made since a version, in one namespace or in all the namespaces you can edit, newest
first, and records the whole of it as one new version (`graph.rollback`, with `rollback_to`). It is a change like
any other: rolling back to the version just before it brings everything back, merges included.

- Merges come undone with their mentions; a merge undone since is merged again. Graph changes it takes back are
marked undone. Moved mentions go back to the entity they were on, and removed ones are said again.
- Then every record is set to how it was at that version.
- What analysis found since stays: it comes from the transcripts, and mentions point at it. Changes people made to
those entities are still taken back.
- An entity made since that is mentioned now stays (hide it instead); the reply lists it under `skipped`.
- It needs editor access to every namespace the changes touched: a link to a namespace you can't edit stops it.

## Replay and verify

A checkpoint keeps the whole graph as it was at one version. The first is taken with the first recorded change (the
graph as it was before it, so an archive that had entities before this feature replays too), then one every 1000
changes. Replaying starts from the newest checkpoint and applies the events after it, without reading today's rows.

Verify replays the history and compares it with the graph. Anything that differs was written without being recorded
(by an older version of Lens, or by hand in the database). Fixing records it as one `graph.drift` change, so the
history matches again.

| Call | What it does |
| --- | --- |
| `GET /api/v1/graph/verify` | `{same, differences, from, head}` and what differs. Admins. |
| `POST /api/v1/graph/verify` | Record what differs as one change. Admins. |
| `GET /api/v1/graph/checkpoints`, `POST /api/v1/graph/checkpoints` | List checkpoints; keep one now. Admins. |

On the command line: `lens history verify [--fix]`, `lens history checkpoint`, `lens history list`.

## How it works

Code that changes the graph wraps its writes in `graph_history.change(db, op, entities=...)`. It reads the rows of
those entities (and the other names, links and distinct pairs that name them) before and after, and records what
differs as one event; a change made inside another joins it. Who and why come from `graph_history.acting(...)`: each
API request sets the caller and whether they came through the web app, a token or an app; the assistant, MCP tools,
routines, workflows and analysis set their own.

The graph as of a version is today's rows with every later event walked back, newest first. A diff composes the
events between two versions: for each record, how it was before the first and after the last.

The graph API (`/graph/schema`, `/graph/related`, `/graph/paths`) and Cypher (`POST /graph/query`) take `as_of`, a
version or a version's name: entities and their cross-namespace links are as they were then; mentions, recordings and
speakers are today's, so an entity merged away since shows only if it is still mentioned. The explorer's graph is
cached per graph version, so any recorded change shows at once.

## Refine later

- Speakers (same-person links, names) in the history.
- A mention-level history.
- As-of from the nearest checkpoint, so very old versions don't walk back through every later event.
9 changes: 8 additions & 1 deletion fastapi_backend/app/api/deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@

from app.core import security
from app.domain import access as acc
from app.domain import auth, hierarchy, ipgroups, oauth, store
from app.domain import auth, graph_history, hierarchy, ipgroups, oauth, store
from app.domain.store import DB

Config = dict[str, Any]
Expand Down Expand Up @@ -64,6 +64,13 @@ def as_audit(self) -> dict[str, Any]:


def _principal(request: Request, db: DB) -> Principal | None:
p = _resolve(request, db)
if p: # graph changes made in this request are recorded as theirs (graph_history.py)
graph_history.note(actor=p.email, via={"access": "web", "token": "token", "oauth": "oauth"}[p.via])
return p


def _resolve(request: Request, db: DB) -> Principal | None:
if hasattr(request.state, "principal"):
return request.state.principal
p: Principal | None = None
Expand Down
5 changes: 3 additions & 2 deletions fastapi_backend/app/api/mcp_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from app.api.v1.routes import namespaces as namespace_routes
from app.api.v1.routes import recordings as recording_routes
from app.api.v1.routes import search as search_routes
from app.domain import analyze, library, rdf, render, store
from app.domain import analyze, graph_history, library, rdf, render, store
from app.domain import entities as ents
from app.domain import speakers as spk
from app.domain.store import DB
Expand Down Expand Up @@ -210,7 +210,8 @@ def call(ctx: Context, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
if unknown:
raise ToolError(f"unknown argument {', '.join(unknown)}; {name} takes {', '.join(a.name for a in t.args) or 'none'}")
values = {a.name: a.parse(arguments.get(a.name)) for a in t.args}
return _text(t.run(ctx, **values))
with graph_history.acting(via="mcp", tool=name):
return _text(t.run(ctx, **values))
except ToolError as e:
return _text({"error": str(e)}, error=True)
except HTTPException as e:
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
fields,
files,
graph,
graph_history,
hierarchy,
iiif,
imports,
Expand Down Expand Up @@ -86,6 +87,7 @@
entities,
topics,
graph,
graph_history,
entity_setup,
metadata,
rdf,
Expand Down
5 changes: 3 additions & 2 deletions fastapi_backend/app/api/v1/routes/entities.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from fastapi import APIRouter, HTTPException, Request

from app.api.deps import Access, Acl, CurrentUser, Db, Writer, domain_errors
from app.domain import auth, entity_map, entity_setup, hierarchy, jobs
from app.domain import auth, entity_map, entity_setup, graph_history, hierarchy, jobs
from app.domain import entities as ents
from app.domain.store import DB, R
from app.schemas.common import Ok
Expand Down Expand Up @@ -258,7 +258,8 @@ def update_entity(eid: int, body: EntityUpdate, request: Request, user: Writer,
if body.defined is not None:
if db.one("SELECT builtin FROM $r", r=R("entity", eid)).get("builtin"):
raise ValueError("Unknown and Unlabeled are always there.")
db.q("UPDATE $r SET defined = $d", r=R("entity", eid), d=body.defined)
with graph_history.change(db, "entity.define" if body.defined else "entity.undefine", entities=[eid]):
db.q("UPDATE $r SET defined = $d", r=R("entity", eid), d=body.defined)
if body.defined:
entity_map.builtins(db, sid)
auth.audit(db, user.as_audit(), "entity.update", f"entity:{eid}", changes)
Expand Down
Loading
Loading
0