8000 feat(topics): a controlled vocabulary of topics, apart from entities by adeelahmad · Pull Request #126 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Topics, apart from entities.** Each namespace has a controlled vocabulary of topics (SKOS): a label, other
labels, a definition, and broader, narrower and related topics. Recordings are about topics, said by a person or
brought over when a topic-like entity (type TERM) becomes a topic; the entity is hidden until the topic is deleted.
Editors create, edit, merge and delete topics through `/api/v1/topics`. Topics are in the graph (`Topic`, with
`ABOUT`, `NARROWER` and `RELATED`; TERM entities are now labelled `Term` there) and in RDF (`/id/topic/<id>`). See
docs/topics.md.

- **Query the graph in Cypher, and walk it.** The archive is now a property graph of namespaces, collections,
recordings, speakers and entities. `POST /api/v1/graph/query` runs read-only Cypher (the language of Neo4j and ISO
GQL) over the namespaces you can read; `/graph/related` gives a node's parents, children, ancestors, descendants or
Expand Down
23 changes: 22 additions & 1 deletion DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,27 @@

Plans and progress for work in flight. Newest first.

## 2026-10-04 · Topics: a controlled vocabulary apart from entities

Goal (Adeel): entities and topics are mixed (a topic is an entity of type TERM, which is also the fallback type for
anything unclassified). Separate them: topics become a SKOS controlled vocabulary per namespace (preferred and
alternative labels, a definition, broader, narrower and related topics), and recordings are about topics. Entities
stay the named things (people, organisations, places...) and later become authority records. Decided: after the graph
PRs (done, #111 #121).

Todo:

- [x] Vocabulary in the backend: topics with labels, definition, broader/narrower/related; recordings about topics;
create, edit, merge, delete; turn a TERM entity into a topic (and back, by deleting the topic); API with viewer
reads and editor changes; topics in the property graph (Topic, ABOUT, NARROWER, RELATED) and in RDF as SKOS
- [ ] Topics page: the tree of broader and narrower topics, a topic's recordings, edit and merge; a recording's topics
- [ ] Analysis suggests topics (summary topics and keywords matched to the vocabulary; new ones as suggestions to
accept); a namespace can keep its vocabulary fixed or open, as entities do
- [ ] # links in notes point at topics; assistant and MCP tools for topics

Refine later: shared vocabularies across namespaces and imported schemes (LCSH, Wikidata) with exactMatch; entities as
authority records (variant names, external identifiers); topic history and undo for merges.

## 2026-10-04 · anytopdf as the conversion engine (not started: wait for Adeel's go)

Goal (Adeel): use the sister project [anytopdf-rs](https://github.com/adeelahmad/anytopdf-rs) (README on its sprint2
Expand Down Expand Up @@ -73,7 +94,7 @@ Todo:
ancestors, descendants, neighbours and paths
- [x] Questions in plain language: the question becomes Cypher (shown, editable), the answer lights up on the canvas
- [x] Assistant and MCP tools: graph schema, query, related, paths; proposing changes behind an approval
- [ ] Topics as a controlled vocabulary (SKOS), apart from entities (authority records); next, after the graph PRs (Adeel)
- [ ] Topics as a controlled vocabulary (SKOS), apart from entities (authority records): started, see "Topics" above

## 2026-10-03 · Assistant extensions: tools, skills, hooks, plugins

Expand Down
18 changes: 11 additions & 7 deletions docs/graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ live from the database (`app/domain/graph_model.py`), so it is never out of step

Status: **built**: the graph model, walking it (parents, children, ancestors, descendants, neighbours, paths),
read-only Cypher and asking for changes over the API; the explorer canvas (drag, one-click layouts, a node menu,
touch); questions in plain language; graph tools for the assistant and MCP. **Planned**: topics as a controlled
vocabulary (SKOS), apart from entities.
touch); questions in plain language; graph tools for the assistant and MCP; topics as each namespace's controlled
vocabulary (SKOS), apart from entities. **Planned**: topics on the explorer canvas as their own kind of node.

## The explorer

Expand Down Expand Up @@ -36,8 +36,9 @@ Graph in the sidebar shows the graph of the scope picked at the top (one namespa
| Recording | `r<id>` | name (the title), date, media, namespace |
| Speaker | `s<id>` | name, seconds (talk time), namespace |
| Entity | `e<id>`, or `e:<key>` in the global scope | name, type, key, mentions, namespaces, ids |
| Topic | `t<id>`, or `t:<key>` in the global scope | name (the label), alt, definition, recordings, namespaces, ids |

An entity has its type as a second label: `Person`, `Organisation`, `Product`, `Place`, `Event`, `Work`, `Topic`, or
An entity has its type as a second label: `Person`, `Organisation`, `Product`, `Place`, `Event`, `Work`, `Term`, or
a namespace's own type (`MY_TYPE` becomes `MyType`). Every node also has `id`, `name` and `namespace`. Hidden entities
and quiet types (dates and numbers) are left out, as in the overview graph.

Expand All @@ -51,9 +52,13 @@ and quiet types (dates and numbers) are left out, as in the overview graph.
| SPOKE_WITH | Speaker → Speaker (in the same recording) | recordings |
| SAME_AS | Speaker → Speaker (said to be one person) | |
| SAME_THING | Entity → Entity (linked across namespaces) | |
| ABOUT | Recording → Topic | source, weight |
| NARROWER | Topic → Topic (from the broader one) | |
| RELATED | Topic → Topic | |

CONTAINS, HAS_SPEAKER, MENTIONS and SAID are the hierarchy: an entity's parents are the recordings that mention it and
the speakers who said it; its ancestors go on up to collections and the namespace.
CONTAINS, HAS_SPEAKER, MENTIONS, SAID, ABOUT and NARROWER are the hierarchy: an entity's parents are the recordings that
mention it and the speakers who said it; its ancestors go on up to collections and the namespace. A topic's parents
are its broader topics and the recordings about it. Topics are each namespace's vocabulary ([Topics](topics.md)).

## Scope and rights

Expand Down Expand Up @@ -147,8 +152,7 @@ MATCH (c:Collection {name: 'Interviews'})-[:CONTAINS*1..8]->(r:Recording) RETURN

## Refine later

- Topics as SKOS concepts in schemes (broader, narrower, related), apart from entities, which become authority records
with preferred and variant names and external identifiers.
- Entities as authority records, with preferred and variant names and external identifiers.
- More change kinds for agents (rename, retype, hide, unlink), and Cypher write clauses mapped to proposed changes.
- Collections people were given a role on (without a namespace role) aren't in the graph yet.
- `EXISTS { }` subqueries and pattern predicates in `WHERE`.
Expand Down
1 change: 1 addition & 0 deletions docs/rdf.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ URIs use the address Lens publishes (`iiif.base_url`, or the address a request a
| `/id/collection/<id>` | a collection (`dcmitype:Collection`), `dcterms:isPartOf` its parent or its namespace |
| `/id/namespace/<name>` | a namespace: a `dcmitype:Collection` and the `skos:ConceptScheme` of its entities |
| `/id/entity/<id>` | a `skos:Concept`, also `foaf:Person`, `foaf:Organization`, `dcterms:Location` or `dcmitype:Event` depending on its type |
| `/id/topic/<id>` | a topic: a `skos:Concept` in `/id/namespace/<name>#topics` (the namespace's `skos:ConceptScheme` of topics), with `skos:broader`, `skos:narrower` and `skos:related`; recordings about it name it as `dcterms:subject` ([Topics](topics.md)) |
| `/id/speaker/<id>` | a `foaf:Person`, `owl:sameAs` the speakers declared to be the same person |
| `/id/field/<id>` | a custom field, used as the property its values are given with |
| `/ns` | the Lens vocabulary (`lens:`), in Turtle |
Expand Down
81 changes: 81 additions & 0 deletions docs/topics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Topics

Each namespace keeps a controlled vocabulary of topics: what its recordings are about. Topics are separate from
entities. Entities are the named things a transcript mentions (people, organisations, products, places); a topic is a
subject someone chose for the vocabulary, like "Gene therapy" or "Funding".

Status: **built**: the vocabulary (labels, definitions, broader, narrower and related topics), recordings about
topics, topics made from topic-like entities, the API, topics in the graph and in RDF. **Planned**: a Topics page,
topic suggestions from analysis, `#` links in notes pointing at topics, and assistant and MCP tools for topics.

## A topic

A topic follows SKOS, the W3C standard for vocabularies:

| Field | SKOS | What it is |
| --- | --- | --- |
| label | `skos:prefLabel` | Its name. Unique in the namespace. |
| alt | `skos:altLabel` | Other labels it goes by (synonyms, abbreviations, spellings). Unique in the namespace too. |
| definition | `skos:definition` | What it covers, in your words. |
| broader | `skos:broader` | The wider topics it belongs under. A topic can have several; it can't end up under itself. |
| narrower | `skos:narrower` | The topics that have it as broader (kept from their side). |
| related | `skos:related` | Topics that are related but not broader or narrower. Kept on both sides. |

Recordings are **about** topics. A person says so, a topic made from an entity brings the recordings that mention
it, and (planned) analysis suggests topics for someone to accept. Each link records where it came from (`person`,
`entity`, `analysis`) and whether it holds (`accepted`) or waits (`suggested`).

## Topics and entities

Until now a topic was an entity of type TERM, which is also the type analysis gives anything it can't classify. Turn
such an entity into a topic (`POST /api/v1/entities/{id}/topic`): its name becomes the label, its aliases the other
labels, its description the definition, and the recordings that mention it are about the topic. The entity is hidden
while the topic exists; deleting the topic shows it again. If the namespace already has a topic of that name, the
recordings join it.

Deleting a topic moves its narrower topics up to its broader ones. Merging topics makes the others' labels the kept
topic's other labels and brings their recordings over.

## Rights

Reading needs a role in the namespace; changing the vocabulary or which recordings are about a topic needs the editor
role there. Every change is in the audit log (`topic.create`, `topic.update`, `topic.delete`, `topic.merge`,
`topic.tag`, `topic.untag`, `topic.from_entity`).

## The API

| Call | What it does |
| --- | --- |
| `GET /api/v1/topics?ns=&q=&top=&broader=` | Topics by label, with how many recordings are about each and how many narrower topics it has. |
| `GET /api/v1/topics/{id}` | A topic with its broader, narrower and related topics and the recordings about it. |
| `POST /api/v1/namespaces/{name}/topics` | Add a topic `{label, alt, definition, broader, related}`. |
| `PATCH /api/v1/topics/{id}` | Change any of those; an empty definition clears it. |
| `DELETE /api/v1/topics/{id}` | Delete it. |
| `POST /api/v1/topics/merge` `{keep, others}` | Merge topics into one. |
| `POST /api/v1/topics/{id}/recordings` `{recordings, remove}` | Say recordings are about it, or aren't. |
| `GET /api/v1/resources/{id}/topics` | The topics a recording is about, and those suggested. |
| `POST /api/v1/entities/{id}/topic` | Make a TERM entity a topic. |

## In the graph and in RDF

In the property graph ([The graph](graph.md)) a topic is a `Topic` node `t<id>`, and recordings link to it with
`ABOUT`; broader topics link to narrower ones with `NARROWER`, and related ones with `RELATED`. ABOUT and NARROWER are
part of the hierarchy, so a topic's ancestors are its broader topics and the recordings about it. TERM entities are
labelled `Term` there, so they aren't confused with topics.

```cypher
// recordings about gene therapy or anything narrower
MATCH (t:Topic)-[:NARROWER*0..6]->(n:Topic)<-[:ABOUT]-(r:Recording)
WHERE toLower(t.name) = 'gene therapy'
RETURN DISTINCT r.name, n.name
```

In RDF ([Linked data](rdf.md)) a topic is `/id/topic/<id>`, a `skos:Concept` in the namespace's vocabulary
`/id/namespace/<name>#topics` (a `skos:ConceptScheme`), and recordings name the topics they are about as
`dcterms:subject`.

## Refine later

- Vocabularies shared by several namespaces, and imported schemes (LCSH, Wikidata) linked with `skos:exactMatch`.
- Entities as authority records: variant names and external identifiers.
- History and undo for topic merges and deletes.
7 changes: 7 additions & 0 deletions fastapi_backend/app/api/linked_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,13 @@ def entity(eid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, format: str |
return rdf_response(rdf.entity_graph(db, base, eid), fmt) if fmt else _see(f"{base}/entities/{eid}")


@router.get("/id/topic/{tid}")
def topic(tid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, format: str | None = FORMAT) -> Response:
_member_of(acl, (db.one("SELECT space FROM $r", r=R("topic", tid)) or {}).get("space"))
base, fmt = base_url(request, cfg), wanted(request, format)
return rdf_response(rdf.topic_graph(db, base, tid), fmt) if fmt else _see(f"{base}/topics/{tid}")


@router.get("/id/speaker/{sid}")
def speaker(sid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, format: str | None = FORMAT) - 10BC8 > Response:
space = (db.one("SELECT space FROM $r", r=R("speaker", sid)) or {}).get("space")
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
sources,
speakers,
templates,
topics,
uploads,
users,
vaults,
Expand Down Expand Up @@ -79,6 +80,7 @@
searches,
speakers,
entities,
topics,
graph,
entity_setup,
metadata,
Expand Down
162 changes: 162 additions & 0 deletions fastapi_backend/app/api/v1/routes/topics.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
"""Topics: each namespace's controlled vocabulary (docs/topics.md), and which recordings are about which topic.

Reads cover the namespaces you have a role in; changes need the editor role there. Each change drops the cached
knowledge graph, which shows topics too.
"""

from __future__ import annotations

from fastapi import APIRouter, HTTPException, Request

from app.api.deps import Access, Acl, CurrentUser, Db, Writer, domain_errors
from app.domain import auth, topics
from app.domain.store import DB, R
from app.schemas.common import Ok
from app.schemas.topics import (
RecordingTopic,
TopicCreate,
TopicDetail,
TopicList,
TopicMerge,
TopicRecordings,
TopicUpdate,
)

router = APIRouter(tags=["topics"])


def _topic_space(db: DB, acl: Access, tid: int, role: str = "viewer") -> int:
row = db.one("SELECT space FROM $r", r=R("topic", int(tid)))
if not row:
raise HTTPException(404, "not found")
acl.need(row["space"], role)
return row["space"]


def _changed(request: Request) -> None:
request.app.state.graph_cache.clear()


def _detail(db: DB, acl: Access, tid: int) -> TopicDetail:
return TopicDetail.model_validate(topics.detail(db, tid))


@router.get("/ 10BC8 topics")
def list_topics(
user: CurrentUser,
acl: Acl,
db: Db,
q: str = "",
ns: str = "",
top: bool = False,
broader: int | None = None,
limit: int = 200,
offset: int = 0,
) -> TopicList:
"""Topics of the namespaces you can read (or of `ns`), by label. `q` matches labels and other labels, `top` keeps
those with no broader topic, `broader` lists the narrower topics of one."""
spaces = set(acl.roles)
if ns:
sid = acl.nsid(ns)
acl.need(sid)
spaces = {sid}
if broader is not None:
_topic_space(db, acl, broader)
return TopicList.model_validate(topics.list_topics(db, spaces, q, top, broader, limit, offset))


@router.get("/topics/{tid}")
def get_topic(tid: int, user: CurrentUser, acl: Acl, db: Db) -> TopicDetail:
"""A topic with its broader, narrower and related topics and the recordings about it."""
_topic_space(db, acl, tid)
return _detail(db, acl, tid)


@router.post("/namespaces/{name}/topics", status_code=201)
def create_topic(name: str, body: TopicCreate, request: Request, user: Writer, acl: Acl, db: Db) -> TopicDetail:
"""Add a topic to the namespace's vocabulary."""
sid = acl.nsid(name)
acl.need(sid, "editor")
with domain_errors():
tid = topics.create(db, sid, body.label, body.alt, body.definition, body.broader, body.related, user=user.email)
auth.audit(db, user.as_audit(), "topic.create", f"topic:{tid}", body.model_dump())
_changed(request)
return _detail(db, acl, tid)


@router.patch("/topics/{tid}")
def update_topic(tid: int, body: TopicUpdate, request: Request, user: Writer, acl: Acl, db: Db) -> TopicDetail:
"""Rename it, change its other labels or definition, or what it is narrower than or related to."""
_topic_space(db, acl, tid, "editor")
changes = body.model_dump(exclude_unset=True)
with domain_errors():
topics.update(
db,
tid,
body.label,
body.alt,
body.definition or None,
body.broader,
body.related,
user=user.email,
clear_definition="definition" in changes and not body.definition,
)
auth.audit(db, user.as_audit(), "topic.update", f"topic:{tid}", changes)
_changed(request)
return _detail(db, acl, tid)


@router.delete("/topics/{tid}")
def delete_topic(tid: int, request: Request, user: Writer, acl: Acl, db: Db) -> Ok:
"""Delete it: its narrower topics move up, recordings stop being about it, and an entity it was made from shows again."""
_topic_space(db, acl, tid, "editor")
with domain_errors():
eid = topics.delete(db, tid)
auth.audit(db, user.as_audit(), "topic.delete", f"topic:{tid}", {"entity": eid} if eid else None)
_changed(request)
return Ok()


@router.post("/topics/merge")
def merge_topics(body: TopicMerge, request: Request, user: Writer, acl: Acl, db: Db) -> TopicDetail:
"""Fold topics into one: their labels become its other labels and their recordings are about it."""
for t in [body.keep, *body.others]:
_topic_space(db, acl, t, "editor")
with domain_errors():
topics.merge(db, body.keep, body.others, user=user.email)
auth.audit(db, user.as_audit(), "topic.merge", f"topic:{body.keep}", {"others": body.others})
_changed(request)
return _detail(db, acl, body.keep)


@router.post("/topics/{tid}/recordings")
def tag_recordings(tid: int, body: TopicRecordings, request: Request, user: Writer, acl: Acl, db: Db) -> TopicDetail:
"""Say recordings of the topic's namespace are about it (accepting suggestions too), or with `remove` that they aren't."""
_topic_space(db, acl, tid, "editor")
with domain_errors():
topics.tag(db, tid, body.recordings, body.remove, user=user.email)
auth.audit(db, user.as_audit(), "topic.untag" if body.remove else "topic.tag", f"topic:{tid}", {"recordings": body.recordings})
_changed(request)
return _detail(db, acl, tid)


@router.get("/recordings/{rid}/topics")
def recording_topics(rid: int, user: CurrentUser, acl: Acl, db: Db) -> list[RecordingTopic]:
"""The topics a recording is about, and those suggested for it."""
acl.recording(rid)
return [RecordingTopic(**t) for t in topics.of_recording(db, rid)]


@router.post("/entities/{eid}/topic", status_code=201)
def entity_to_topic(eid: int, request: Request, user: Writer, acl: Acl, db: Db) -> TopicDetail:
"""Make a topic-like entity (type TERM) a topic: its names become the topic's labels and the recordings that
mention it are about the topic. The entity is hidden; deleting the topic shows it again."""
row = db.one("SELECT space FROM $r", r=R("entity", int(eid)))
if not row:
raise HTTPException(404, "not found")
acl.need(row["space"], "editor")
with domain_errors():
tid = topics.from_entity(db, eid, user=user.email)
auth.audit(db, user.as_audit(), "topic.from_entity", f"topic:{tid}", {"entity": eid})
_changed(request)
return _detail(db, acl, tid)
1 change: 1 addition & 0 deletions fastapi_backend/app/domain/deletion.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
OWN = (
"mentions",
"term",
"topic_about",
"section",
"appearance",
"segment",
Expand Down
Loading
Loading
0