8000 feat(fedora): keep a copy of the archive in a Fedora 6 repository by adeelahmad · Pull Request #88 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ x-backend: &backend
SURREAL_PASS: ${SURREAL_PASS:?set SURREAL_PASS}
RUN_BACKGROUND: "false"
FRONTEND_URL: ${FRONTEND_URL:-http://localhost:3000}
# Fedora (docs/fedora.md): empty keeps it off; with the fedora profile, http://fedora:8080/fcrepo/rest
LENS_FEDORA_URL: ${LENS_FEDORA_URL:-}
LENS_FEDORA_USER: ${LENS_FEDORA_USER:-}
LENS_FEDORA_PASSWORD: ${LENS_FEDORA_PASSWORD:-}
CORS_ORIGINS: '["${FRONTEND_URL:-http://localhost:3000}"]'
ARCHIVE_ALLOWED_HOSTS: ${ARCHIVE_ALLOWED_HOSTS:-backend,localhost,127.0.0.1}
MAIL_SERVER: ${MAIL_SERVER:-}
Expand Down Expand Up @@ -103,9 +107,25 @@ services:
restart: unless-stopped
networks: [lens]

# Fedora 6 (docs/fedora.md), the repository Lens can keep a copy of the archive in. Opt-in, and not needed on small
# machines: `docker compose --profile fedora up`, with LENS_FEDORA_URL=http://fedora:8080/fcrepo/rest (and
# LENS_FEDORA_USER / LENS_FEDORA_PASSWORD) in .env, or set it in Settings → Fedora.
fedora:
image: fcrepo/fcrepo:${FEDORA_VERSION:-6.5.1-tomcat9}
profiles: [fedora]
environment:
CATALINA_OPTS: >-
-Dfcrepo.home=/fcrepo-home -Djava.awt.headless=true -Dfcrepo.autoversioning.enabled=true
ports:
- "127.0.0.1:8081:8080" # its web UI and REST API, for this machine only
volumes:
- fedora-data:/fcrepo-home
networks: [lens]

volumes:
surreal-data:
archive-data:
fedora-data:

networks:
lens:
Expand Down
16 changes: 16 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,25 @@ services:
- "127.0.0.1:8025:8025" # the mailbox
networks: [lens]

# Fedora 6 (docs/fedora.md), the repository Lens can keep a copy of the archive in. Opt-in, and not needed on small
# machines: `docker compose --profile fedora up`, with LENS_FEDORA_URL=http://fedora:8080/fcrepo/rest (and
# LENS_FEDORA_USER / LENS_FEDORA_PASSWORD) in .env, or set it in Settings → Fedora.
fedora:
image: fcrepo/fcrepo:${FEDORA_VERSION:-6.5.1-tomcat9}
profiles: [fedora]
environment:
CATALINA_OPTS: >-
-Dfcrepo.home=/fcrepo-home -Djava.awt.headless=true -Dfcrepo.autoversioning.enabled=true
ports:
- "127.0.0.1:8081:8080" # its web UI and REST API, for this machine only
volumes:
- fedora-data:/fcrepo-home
networks: [lens]

volumes:
surreal-data:
archive-data:
fedora-data:

networks:
lens:
Expand Down
72 changes: 72 additions & 0 deletions docs/fedora.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Fedora repository

Lens can keep a copy of the archive in [Fedora](https://wiki.lyrasis.org/display/FF) 6, the repository many libraries
and archives use for preservation. Every namespace, collection, recording (with its file), entity and speaker becomes
an LDP resource in Fedora, described in RDF with Dublin Core ([Linked data](rdf.md)). SurrealDB is still the working
database, and Fedora holds a copy that is kept up to date with it.

The copy is off until you give Lens Fedora's address, so installs without Fedora run exactly as before. Fedora is a
Java server that needs about 1 GB of memory, so leave it off on a Raspberry Pi.

## Running it

```sh
docker compose --profile fedora up -d # Fedora 6.5 at http://127.0.0.1:8081/fcrepo (user fedoraAdmin)
```

Then do one of these:

- Set the address in **Settings → Fedora repository**: `http://fedora:8080/fcrepo/rest`, with the user and password.
- Put `LENS_FEDORA_URL`, `LENS_FEDORA_USER` and `LENS_FEDORA_PASSWORD` in `.env`. These override the app's
settings, which then show as locked.

The Fedora image's default account is `fedoraAdmin` / `fedoraAdmin`. Change it (Tomcat's `tomcat-users.xml`) before
the server is reachable from anywhere but this machine. You can also point Lens at an existing Fedora 6 instead.

## What goes where

Under `<url>/<root>` (the root is `lens` by default):

| Path | What it is |
| --- | --- |
| `<namespace>` | the namespace (`dcmitype:Collection`) |
| `<namespace>/collections/<id>` | a collection |
| `<namespace>/recordings/<id>` | a recording: dcterms title, creator, subject, created, license and the rest |
| `<namespace>/recordings/<id>/file` | its file (a binary), unless files are off or it is larger than the limit |
| `<namespace>/entities/<id>` | an entity (`skos:Concept`) |
| `<namespace>/speakers/<id>` | a speaker (`foaf:Person`) |

Each resource gets the triples Lens has about it. Blank nodes become hash URIs (`…/recordings/12#b1`), and each
resource also gets `owl:sameAs` pointing to its Lens URI. Links to other resources use their Lens URIs
(`/id/entity/7`), which don't change wherever Fedora is.

## Keeping it up to date

- A change to a recording's metadata is queued and sent at the next sync, every `sync_seconds` (60).
- Every `full_hours` (24), everything is compared with what was last sent, using a hash for each resource. This
catches what analysis changed, new recordings, and deletions: a resource Lens no longer has is deleted from Fedora
along with its tombstone.
- Only what changed is sent. A file is sent again only when its size or modification time changes.
- **Compare and send now** in Settings, or `POST /api/v1/admin/fedora/sync`, runs a full comparison straight away.
`GET /api/v1/admin/fedora` shows the state: resources kept, changes waiting, the last sync and the last problem.

The workers send the changes, and so does the API when it runs its own background work. A problem such as an
unreachable server or a refused resource is shown in Settings and tried again at the next sync. The rest still
goes.

## Testing

`tests/api/test_fedora.py` runs against a stand-in. To run it against a real Fedora:

```sh
docker run -d -p 8080:8080 fcrepo/fcrepo:6.5.1-tomcat9
LENS_TEST_FEDORA_URL=http://127.0.0.1:8080/fcrepo/rest uv run pytest tests/api/test_fedora.py
```

## Refine later

- One process sending the changes. Today the API and the worker may both send them, which is safe because the PUTs
are idempotent.
- Fedora transactions, so a sync lands as a whole.
- Files Fedora reads in place (external content), instead of a copy.
- Reading Fedora back (restoring an archive from it), and checks on its OCFL storage.
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
entities,
entity_setup,
extensions,
fedora,
fields,
files,
hierarchy,
Expand Down Expand Up @@ -54,6 +55,7 @@
for module in (
users,
admin,
fedora,
setup,
namespaces,
hierarchy,
Expand Down
52 changes: 52 additions & 0 deletions fastapi_backend/app/api/v1/routes/fedora.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
"""The copy of the archive in Fedora (domain/fedora.py, docs/fedora.md): how it stands, and sending everything now.
Admins only; the connection itself is set in Settings (the `fedora` section)."""

from __future__ import annotations

from typing import Any

from fastapi import APIRouter, HTTPException
from starlette.concurrency import run_in_threadpool

from app.api.deps import AdminReader, AdminWriter, Cfg, Db
from app.domain import auth, fedora
from app.schemas.common import ResponseModel

router = APIRouter(prefix="/admin/fedora", tags=["fedora"])


class FedoraStatus(ResponseModel):
enabled: bool
url: str | None = None
root: str
pending: int
resources: int
last_sync: str | None = None
last_full: str | None = None
last_counts: dict[str, Any] | None = None
last_error: str | None = None
full_requested: bool


class FedoraSyncResult(ResponseModel):
sent: int
files: int
unchanged: int
deleted: int
failed: int
errors: list[str]


@router.get("")
def get_fedora_status(user: AdminReader, db: Db, cfg: Cfg) -> FedoraStatus:
return FedoraStatus.model_validate(fedora.summary(db, cfg))


@router.post("/sync")
async def sync_fedora(user: AdminWriter, db: Db, cfg: Cfg) -> FedoraSyncResult:
"""Compare everything with Fedora now and send what differs (the background sync does this every fedora.full_hours)."""
if not fedora.enabled(cfg):
raise HTTPException(409, "Fedora isn't set up: set fedora.url and turn it on in Settings first")
out = await run_in_threadpool(fedora.sync, db, cfg)
auth.audit(db, user.as_audit(), "fedora.sync", None, {k: v for k, v in out.items() if k != "errors"})
return FedoraSyncResult.model_validate(out)
Loading
Loading
0