8000 feat(rdf): describe the archive as linked data with Dublin Core by adeelahmad · Pull Request #79 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions docs/rdf.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Linked data (RDF)

Everything in the archive is also linked data. Each recording, collection, namespace, entity and speaker has a URI,
and that URI returns its description in RDF. The description uses Dublin Core
([DCMI Metadata Terms](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/)), SKOS, FOAF and OWL, plus
a small Lens vocabulary for what those don't cover.

The RDF is generated from what Lens already stores: the recording's metadata (see [IIIF](iiif.md)), its entities and
its speakers. It can't go out of date, and nothing needs migrating.

## URIs

URIs use the address Lens publishes (`iiif.base_url`, or the address a request arrived at):

| URI | What it is |
| --- | --- |
| `/id/recording/<id>` | a recording, typed `lens:Item` and a DCMI type (`Sound`, `MovingImage`, `Text`, `StillImage`) |
| `/id/collection/<id>` | a collection (`dcmitype:Collection`), `dcterms:isPartOf` its parent or its namespace |
| `/id/namespace/<name>` | a namespace: a `dcmitype:Collection` and the `skos:ConceptScheme` of its entities |
| `/id/entity/<id>` | a `skos:Concept`, also `foaf:Person`, `foaf:Organization`, `dcterms:Location` or `dcmitype:Event` depending on its type |
| `/id/speaker/<id>` | a `foaf:Person`, `owl:sameAs` the speakers declared to be the same person |
| `/id/field/<id>` | a custom field, used as the property its values are given with |
| `/ns` | the Lens vocabulary (`lens:`), in Turtle |

Each URI uses content negotiation. An `Accept` of `text/turtle`, `application/ld+json`, `application/n-triples` or
`application/rdf+xml`, or `?format=turtle|json-ld|nt|xml`, returns RDF. A browser is sent to the page instead
(303 See Other).

A public recording's URI is open to everyone, just like its IIIF manifest. Visitors get its metadata and published
custom fields. Members also get the entities it mentions (`dcterms:references`), its access and its internal fields.
Collection, namespace, entity and speaker URIs are for the namespace's members only. Anyone else gets a 404, as if
they weren't there.

## Mapping

| Lens metadata | RDF |
| --- | --- |
| label, summary | `dcterms:title`, `dcterms:description` (language-tagged) |
| creators, contributors | `dcterms:creator`, `dcterms:contributor`: the speaker, the person's URI, or a node with `foaf:name` |
| subjects | `dcterms:subject`: the entity, the subject's URI, or its label |
| navDate | `dcterms:created` (`xsd:dateTime`) |
| language | `dcterms:language` (`dcterms:RFC5646`) |
| rights, attribution | `dcterms:license`, `dcterms:rights` |
| provider | `dcterms:publisher`, a `foaf:Organization` |
| identifiers | `dcterms:identifier` (typed ones also as `lens:identifier`) |
| homepage, related | `foaf:homepage`, `dcterms:relation` |
| label/value pairs | `lens:property` (`rdfs:label`, `rdf:value`) |
| terms | the DCMI term itself (`dcterms:spatial`, `dcterms:alternative`, …) |
| duration, file type | `dcterms:extent` (`xsd:duration`), `dcterms:format` (`dcterms:IMT`) |
| where it is | `dcterms:isPartOf` its collection or namespace |

`terms` holds every other DCMI term a recording can have: alternative, abstract, audience, coverage, spatial,
temporal, source, issued, modified, rightsHolder, provenance, isVersionOf, replaces, requires and the rest. Each
term holds a list of values. A value that is an http(s) address becomes a link. The oai_dc record (`/iiif/<id>/dc.xml`)
includes those of them that refine one of its 15 elements (spatial becomes `dc:coverage`, alternative becomes
`dc:title`, and so on).

## API

- `GET /api/v1/recordings/{id}/rdf` returns a recording with the entities it mentions.
- `GET /api/v1/namespaces/{name}/rdf` returns a whole namespace: its collections, recordings, entities and speakers.
Add `download=true` to save it as a file.

Both accept `?format=` or an `Accept` header, and return Turtle when neither is given.

## Refine later

- A permanent vocabulary URI shared by every Lens. Today `lens:` is `<address>/ns#`, so two archives use different
terms.
- Roles qualified per recording. Today a speaker's role is left out, because the speaker is the same resource in
every recording.
- Collections and entities that visitors can see in public namespaces.
100 changes: 100 additions & 0 deletions fastapi_backend/app/api/linked_data.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
"""Linked data: the URIs the archive's RDF uses (domain/rdf.py), each answering with RDF or a page.

`/id/<kind>/<id>` is what a recording, collection, namespace, entity or speaker is called in RDF. An RDF client
(Accept: text/turtle, application/ld+json, application/n-triples or application/rdf+xml, or `?format=`) gets its
description; a browser is sent to its page (303 See Other, as linked data does). A recording is open to everyone when it
is public (its IIIF manifest is), and to whoever may see it otherwise; the rest are for the namespace's members, and
look absent to anyone else. `/ns` is the Lens vocabulary.
"""

from __future__ import annotations

import urllib.parse

from fastapi import APIRouter, HTTPException, Query, Request
from fastapi.responses import RedirectResponse, Response

from app.api.deps import Acl, Cfg, Db, OptionalUser
from app.api.iiif import _rec, base_url
from app.domain import rdf
from app.domain.store import R

router = APIRouter(include_in_schema=False, tags=["linked-data"])
FORMAT = Query(None, description="turtle, json-ld, nt or xml (or ttl, jsonld, rdf); else the Accept header decides")


def rdf_response(g, fmt: str, filename: str | None = None) -> Response:
headers = {"Vary": "Accept", "Access-Control-Allow-Origin": "*"}
if filename:
ext = {v: k for k, v in rdf.SUFFIX.items()}[fmt]
headers["Content-Disposition"] = f'attachment; filename="{filename}.{ext}"'
return Response(rdf.serialize(g, fmt), media_type=f"{rdf.FORMATS[fmt]}; charset=utf-8", headers=headers)


def wanted(request: Request, fmt: str | None) -> str | None:
try:
return rdf.negotiate(request.headers.get("accept"), fmt)
except ValueError as e:
raise HTTPException(400, str(e)) from None


def _see(url: str) -> RedirectResponse:
return RedirectResponse(url, status_code=303, headers={"Vary": "Accept"})


@router.get("/ns")
def vocabulary(request: Request, cfg: Cfg, format: str | None = FORMAT) -> Response:
return rdf_response(rdf.vocabulary(base_url(request, cfg)), wanted(request, format) or "turtle")


@router.get("/id/recording/{rid}")
def recording(rid: int, request: Request, user: OptionalUser, acl: Acl, db: Db, cfg: Cfg, format: str | None = FORMAT) -> Response:
rec, a = _rec(request, db, cfg, user, rid)
row = db.one("SELECT space, collection FROM $r", r=R("recording", rid)) or {}
member = bool(user and acl.rank_in(row["space"], row.get("collection")))
base = base_url(request, cfg)
fmt = wanted(request, format)
if not fmt:
return _see(f"{base}/recordings/{rid}" if member else f"{base}/explore/recordings/{rid}")
return rdf_response(rdf.recording_graph(db, cfg, base, rid, member), fmt)


def _member_of(acl: Acl, space: int | None) -> None:
if space is None or not acl.user or not acl.rank_in(space, None):
raise HTTPException(404, "not found")


@router.get("/id/collection/{cid}")
def collection(cid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, format: str | None = FORMAT) -> Response:
c = db.one("SELECT space FROM $r", r=R("collection", cid))
if not c or not acl.user or not acl.rank_in(c["space"], cid):
raise HTTPException(404, "not found")
base, fmt = base_url(request, cfg), wanted(request, format)
return rdf_response(rdf.collection_graph(db, base, cid), fmt) if fmt else _see(f"{base}/collections/{cid}")


@router.get("/id/entity/{eid}")
def entity(eid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, format: str | None = FORMAT) -> Response:
_member_of(acl, (db.one("SELECT space FROM $r", r=R("entity", eid)) or {}).get("space"))
base, fmt = base_url(request, cfg), wanted(request, format)
return rdf_response(rdf.entity_graph(db, base, eid), fmt) if fmt else _see(f"{base}/entities/{eid}")


@router.get("/id/speaker/{sid}")
def speaker(sid: int, request: Request, acl: Acl, db: Db, cfg: Cfg, format: str | None = FORMAT) -> Response:
space = (db.one("SELECT space FROM $r", r=R("speaker", sid)) or {}).get("space")
_member_of(acl, space)
base, fmt = base_url(request, cfg), wanted(request, format)
if not fmt:
return _see(f"{base}/speakers/{sid}")
return rdf_response(rdf.speaker_graph(db, base, sid), fmt)


@router.get("/id/namespace/{name}")
def namespace(name: str, request: Request, acl: Acl, db: Db, cfg: Cfg, format: str | None = FORMAT) -> Response:
sid = acl.nsid(name)
_member_of(acl, sid)
base, fmt = base_url(request, cfg), wanted(request, format)
if not fmt:
return _see(f"{base}/library?ns={urllib.parse.quote(name)}")
return rdf_response(rdf.namespace_graph(db, cfg, base, sid), fmt)
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
oauth,
pipelines,
public,
rdf,
recordings,
requests,
routines,
Expand Down Expand Up @@ -65,6 +66,7 @@
entities,
entity_setup,
metadata,
rdf,
video,
iiif,
jobs,
Expand Down
45 changes: 45 additions & 0 deletions fastapi_backend/app/api/v1/routes/rdf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
"""The archive as RDF (domain/rdf.py) for the app and API clients: a recording's description, or a whole namespace's
graph (its collections, recordings, entities and speakers), in Turtle, JSON-LD, N-Triples or RDF/XML."""

from __future__ import annotations

from typing import Any

from fastapi import APIRouter, Query, Request
from fastapi.responses import Response

from app.api.deps import Acl, Cfg, CurrentUser, Db
from app.api.iiif import base_url
from app.api.linked_data import rdf_response, wanted
from app.domain import rdf

router = APIRouter(tags=["rdf"])
RDF_CONTENT: dict[int | str, dict[str, Any]] = {
200: {"content": {m: {"schema": {"type": "string"}} for m in rdf.FORMATS.values()}, "description": "RDF"}
}
FORMAT = Query(None, description="turtle, json-ld, nt or xml; else the Accept header decides (Turtle when it doesn't)")


@router.get("/recordings/{rid}/rdf", response_class=Response, responses=RDF_CONTENT)
def get_recording_rdf(rid: int, request: Request, user: CurrentUser, acl: Acl, db: Db, cfg: Cfg, format: str | None = FORMAT) -> Response:
"""The recording described with Dublin Core, with the entities it mentions."""
rec = acl.recording(rid)
g = rdf.recording_graph(db, cfg, base_url(request, cfg), rid, member=acl.member(rec))
return rdf_response(g, wanted(request, format) or "turtle")


@router.get("/namespaces/{name}/rdf", response_class=Response, responses=RDF_CONTENT)
def get_namespace_rdf(
name: str,
request: Request,
user: CurrentUser,
acl: Acl,
db: Db,
cfg: Cfg,
format: str | None = FORMAT,
download: bool = Query(False, description="as a file to save"),
) -> Response:
"""Everything in the namespace as one graph: itself, its collections, recordings, entities and speakers."""
sid = acl.namespace(name)
g = rdf.namespace_graph(db, cfg, base_url(request, cfg), sid)
return rdf_response(g, wanted(request, format) or "turtle", filename=name if download else None)
80 changes: 80 additions & 0 deletions fastapi_backend/app/domain/metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,55 @@
"identifiers",
"homepage",
"related",
"terms",
"access",
"open",
"featured",
)
LANG_RX = re.compile(r"^(none|[a-zA-Z]{2,3}(-[A-Za-z0-9]{2,8})*)$")
# The DCMI Metadata Terms a recording can be given directly (`terms`: {term: [text or http(s) address]}), beyond the
# ones its other fields already say (title, description, creator, contributor, subject, created, language, license,
# rights, publisher, identifier, relation, isPartOf, type, format, extent). See rdf.py.
DC_TERMS = (
"abstract",
"accessRights",
"accrualMethod",
"accrualPeriodicity",
"accrualPolicy",
"alternative",
"audience",
"available",
"bibliographicCitation",
"conformsTo",
"coverage",
"dateAccepted",
"dateCopyrighted",
"dateSubmitted",
"educationLevel",
"hasFormat",
"hasPart",
"hasVersion",
"instructionalMethod",
"isFormatOf",
"isReferencedBy",
"isReplacedBy",
"isRequiredBy",
"issued",
"isVersionOf",
"mediator",
"medium",
"modified",
"provenance",
"replaces",
"requires",
"rightsHolder",
"source",
"spatial",
"tableOfContents",
"temporal",
"valid",
)
TERM_MAX, TERM_VALUES = 2000, 50
RIGHTS_RX = re.compile(r"^https?://(creativecommons\.org/(licenses|publicdomain)/|rightsstatements\.org/vocab/)")


Expand Down Expand Up @@ -91,6 +135,24 @@ def _uri(v, field):
return v


def _terms(v):
if not isinstance(v, dict):
raise MetaProblem('terms is an object of Dublin Core terms and their values, like {"spatial": ["Berlin"]}')
out = {}
for term, vals in v.items():
if term not in DC_TERMS:
raise MetaProblem(f"'{term}' isn't one of the Dublin Core terms a recording can be given here")
vals = [vals] if isinstance(vals, str) else vals
if vals is None:
continue
if not isinstance(vals, list) or not all(isinstance(x, str) for x in vals):
raise MetaProblem(f"{term}: each value is text or an http(s) address")
vals = list(dict.fromkeys(x.strip()[:TERM_MAX] for x in vals if x.strip()))[:TERM_VALUES]
if vals:
out[term] = vals
return out


def clean(patch):
"""Validate and normalise fields someone is saving. None clears a field (it won't fall back to a default)."""
out = {}
Expand Down Expand Up @@ -187,6 +249,8 @@ def clean(patch):
)
for x in (v if isinstance(v, list) else [v])
]
elif k == "terms":
out[k] = _terms(v) or None
elif k == "open":
try:
out[k] = acc.parts(v)
Expand Down Expand Up @@ -516,6 +580,21 @@ def schema_org(meta, rec, urls):
)


# the one of the 15 Dublin Core elements each of those terms refines, for oai_dc (which has only those)
DC_ELEMENT = {
**{t: "date" for t in ("available", "dateAccepted", "dateCopyrighted", "dateSubmitted", "issued", "modified", "valid")},
**{t: "relation" for t in ("conformsTo", "hasFormat", "hasPart", "hasVersion", "isFormatOf", "isReferencedBy", "isReplacedBy")},
**{t: "relation" for t in ("isRequiredBy", "isVersionOf", "replaces", "requires")},
**{t: "coverage" for t in ("coverage", "spatial", "temporal")},
**{t: "description" for t in ("abstract", "tableOfContents")},
**{t: "rights" for t in ("accessRights", "rightsHolder")},
"alternative": "title",
"source": "source",
"medium": "format",
"bibliographicCitation": "identifier",
}


def dublin_core(meta, rec, urls):
def el(tag, text, lang=None):
attr = f" xml:lang={quoteattr(lang)}" if lang and lang != "none" else ""
Expand All @@ -538,6 +617,7 @@ def el(tag, text, lang=None):
el("rights", first(meta.get("attribution"))),
]
lines += [el("publisher", (meta.get("provider") or {}).get("name")), el("relation", urls["collection"])]
lines += [el(DC_ELEMENT[t], x) for t, vals in (meta.get("terms") or {}).items() if t in DC_ELEMENT for x in vals]
return (
'<?xml version="1.0" encoding="UTF-8"?>\n<oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" '
'xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" '
Expand Down
Loading
Loading
0