8000 feat(shares): the graph over S3 and WebDAV, read-only by adeelahmad · Pull Request #324 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
inside their broader topics, entities by type. Each folder holds its assets' files, with a `.json` sidecar of
metadata and a `.txt` of the text; everyone sees what the web app shows them. Browse and read it through the API
(`/api/v1/tree`) and the MCP tools `list_files` and `read_file` ([The graph as files](docs/file-tree.md)).
S3, WebDAV, SMB and NFS servers that share it are in progress.
SMB and NFS shares are in progress.

- **HTTPS with automatic certificates.** Lens can serve itself at `https://` your domain with a certificate it gets
and renews on its own from Let's Encrypt or ZeroSSL (or any ACME authority), proving the domain over HTTP, TLS-ALPN
Expand All @@ -29,6 +29,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
month. **Costs** in the left menu shows spend by day, model and resource, and sets the namespace's budget
([Budgets](docs/budgets.md#the-meter-and-the-costs-page)).

- **The graph over S3 and WebDAV.** Turn on Settings, File shares, and any S3 client (rclone, Cyberduck, the AWS CLI) or
WebDAV client (Finder, Windows Explorer, file managers) lists and reads the graph as folders and files,
read-only, with access keys each person makes on their API tokens page ([S3](docs/file-tree.md#s3),
[WebDAV](docs/file-tree.md#webdav)). Docker publishes S3 on port 9555 (`LENS_S3_PORT`) and WebDAV on 9556
(`LENS_WEBDAV_PORT`), where the web app is.

- **Ask Lens from your browser.** The web clipper extension has the Lens assistant in a side panel: **Ask Lens** in
its popup or on the right-click menu. It signs in with one click through Lens's own sign-in page (OAuth, nothing to
copy), and the conversations are the same ones Lens's chat shows, with cited answers, the steps taken and approvals.
Expand Down
4 changes: 4 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,10 @@ services:
- "${LENS_SENSOR_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_MQTT_PORT:-1883}:1883"
- "${LENS_SENSOR_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_SYSLOG_PORT:-5514}:5514/udp"
- "${LENS_SENSOR_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_SYSLOG_PORT:-5514}:5514/tcp"
# the graph as files over S3 and WebDAV (docs/file-tree.md): read-only, listened on only once File shares are turned
# on in the app; published where the web app is. Something else using 9555 or 9556? Set LENS_S3_PORT / LENS_WEBDAV_PORT.
- "${LENS_SHARE_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_S3_PORT:-9555}:9555"
- "${LENS_SHARE_BIND:-${LENS_BIND:-127.0.0.1}}:${LENS_WEBDAV_PORT:-9556}:9556"
healthcheck:
disable: true

Expand Down
3 changes: 3 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ services:
- "${LENS_SENSOR_BIND:-127.0.0.1}:${LENS_MQTT_PORT:-1883}:1883"
- "${LENS_SENSOR_BIND:-127.0.0.1}:${LENS_SYSLOG_PORT:-5514}:5514/udp"
- "${LENS_SENSOR_BIND:-127.0.0.1}:${LENS_SYSLOG_PORT:-5514}:5514/tcp"
# the graph as files over S3 and WebDAV (docs/file-tree.md), once File shares are on; LENS_S3_PORT / LENS_WEBDAV_PORT
- "${LENS_SHARE_BIND:-127.0.0.1}:${LENS_S3_PORT:-9555}:9555"
- "${LENS_SHARE_BIND:-127.0.0.1}:${LENS_WEBDAV_PORT:-9556}:9556"
healthcheck:
disable: true

Expand Down
6 changes: 6 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -599,10 +599,16 @@ removes one.
```
GET /api/v1/tree?path=
GET /api/v1/tree/file?path=
GET /api/v1/shares/s3
POST /api/v1/shares/s3/keys
DELETE /api/v1/shares/s3/keys/{key_id}
GET /api/v1/shares/s3/keys
```

The archive as folders and files ([The graph as files](file-tree.md)): `tree` lists a folder (an empty path lists your
namespaces; in each, `Collections`, `Topics` and `Entities`), `tree/file` reads a file with byte ranges. Read-only; everyone sees what they can read in the web app.
`shares/s3` is the [S3](file-tree.md#s3) and [WebDAV](file-tree.md#webdav) servers' state and your access keys; keys are
made only while signed in.

## comments

Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ Admins can change, in Settings: transcription, speaker separation, speech provid
provider and key, a local model, the AI assistant, search (word search and search by meaning), reports and graph,
video, OCR, faces and objects, workers, sign-in, components, access and embedding (embed frame ancestors, transcript
upload limit, allowed hosts, trusted proxies, session length), remote access, email, chat rooms, notifications,
telemetry, the Fedora repository, sensors, uploads, storage, documents, API keys, and IIIF and metadata. The API
telemetry, the Fedora repository, sensors, file shares, uploads, storage, documents, API keys, and IIIF and metadata. The API
refuses an allowed-host list that leaves out the address you are using.

## Word search
Expand Down
56 changes: 54 additions & 2 deletions docs/file-tree.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@

Lens shows its graph as an ordinary tree of folders and files: graph nodes (collections, topics, entities) are
folders, nested along the graph's links, and every asset (a recording, document or image) is a few files inside the
folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read Lens without learning its API.
folders it belongs to, its metadata in a sidecar beside it. Agents and tools that already understand files can read
Lens without learning its API.

| Where | Status |
| --- | --- |
| The API (`GET /api/v1/tree`, `GET /api/v1/tree/file`) and the MCP tools `list_files` and `read_file` | built |
| S3 and WebDAV servers (read-only, off until turned on) | in progress |
| An [S3 server](#s3) and a [WebDAV server](#webdav) (read-only, off until turned on) | built |
| SMB and NFS shares | in progress |

Every way in serves the same tree, built by `app/domain/filetree.py`.
Expand Down Expand Up @@ -54,8 +55,59 @@ content_type}]}`. An empty path lists your namespaces. `GET /api/v1/tree/file?pa
The MCP server has `list_files` and `read_file` ([MCP server](mcp.md)): `read_file` returns the text of `.txt` and
`.json` files, a page at a time, and the download link of everything else.

## S3

Off until an admin turns it on, in Settings, File shares (`shares.enabled`). Then Lens's worker serves the tree
read-only over S3 on port 9555 (`shares.s3_port`; with Docker, `LENS_S3_PORT` publishes it on another port of the
host), where the web app is: this machine only unless Lens was opened to the network (`LENS_BIND`;
`LENS_SHARE_BIND` sets it apart). Each namespace you can read is a bucket; keys are the paths above.

Each person makes their own access keys on their API tokens page (File share access keys), while signed in to the web app;
API tokens can't make them. A key reads as its owner and sees exactly what they see. Its secret is shown once, kept
sealed with the server's key (S3's signatures are made with the secret itself, so it can't be hashed), expires like
an API key (Settings, API keys), stops working when its owner is disabled, and can be revoked at once. Making and
revoking keys is audited. `GET /api/v1/shares/s3` says whether the S3 and WebDAV servers are on and lists your keys;
`POST /api/v1/shares/s3/keys` and `DELETE /api/v1/shares/s3/keys/{id}` make and revoke them (admins list everyone's
with `GET /api/v1/shares/s3/keys`).

Any S3 client works with path-style addressing and any region. With rclone, the remote the web app shows when you
make a key:

```ini
[lens]
type = s3
provider = Other
access_key_id = LK…
secret_access_key = …
endpoint = http://lens.local:9555
force_path_style = true
```

then `rclone lsd lens:`, `rclone copy "lens:pods/Topics/Gene therapy" ./gene-therapy`, or `rclone mount lens: ~/lens`.
Supported: listing buckets, listing objects (v1 and v2, by folder or recursively, a page at a time), reading objects whole or
in ranges, and presigned links. Anything that writes answers AccessDenied. Requests must be signed with Signature
Version 4.

The server speaks plain HTTP. On a network you don't trust, keep it on this machine or behind TLS (a reverse proxy),
since files travel in clear.

## WebDAV

Off until an admin turns it on, with S3 (Settings, File shares; `shares.webdav`, on by default once shares are).
Lens's worker serves the same tree read-only over WebDAV on port 9556 (`shares.webdav_port`; with Docker,
`LENS_WEBDAV_PORT`), so it mounts as a drive: in Finder, Go, Connect to Server; in Windows Explorer, Map network
drive; in GNOME Files or Dolphin, `dav://lens.local:9556/`; or `rclone` with `type = webdav`.

Sign in with an access key: its id as the user name and its secret as the password. The folders at the top are your
namespaces. Listing (PROPFIND, one level at a time) and reading (GET, with ranges) work; anything that writes answers
403. Eight failed sign-ins from one address within 15 minutes are refused for a while.

Like S3, it speaks plain HTTP, and Basic sign-in sends the secret with every request: keep it on this machine or
behind TLS on a network you don't trust.

## Refine later

- TLS by itself for S3 and WebDAV, folders' real dates, and WebDAV locks for clients that insist on them.
- Show supplementary files (captions, translations, attachments) beside each asset.
- Cache an asset's text and metadata so large folders list faster.
- A sidecar for each collection, topic and entity, and a folder of notes.
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/api/v1/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
searches,
sensors,
setup,
shares,
sources,
speakers,
templates,
Expand Down Expand Up @@ -125,6 +126,7 @@
notifications,
routines,
sensors,
shares,
import_hooks,
voice,
vaults,
Expand Down
51 changes: 51 additions & 0 deletions fastapi_backend/app/api/v1/routes/shares.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
"""File shares (docs/file-tree.md): whether the S3 and WebDAV servers are on, and people's access keys for them.

Admins turn the server on in the settings (`shares`). Anyone signed in makes keys for themselves, which act as them;
a key's secret is shown once. Making and revoking keys is audited.
"""

from __future__ import annotations

from fastapi import APIRouter, HTTPException

from app.api.deps import AdminReader, Cfg, CurrentUser, Db, Writer, domain_errors
from app.domain import auth, s3, shares
from app.schemas.common import Ok
from app.schemas.filetree import S3Key, S3KeyCreate, S3KeyCreated, S3Status, ShareServer

router = APIRouter(prefix="/shares", tags=["shares"])


@router.get("/s3")
def s3_status(user: CurrentUser, db: Db, cfg: Cfg) -> S3Status:
"""Whether the S3 server is on and on which port, what the worker running it says, the same for WebDAV, and your
access keys."""
st = shares.status(db, cfg)
return S3Status(**st["s3"], webdav=ShareServer(**st["webdav"]), keys=[S3Key(**k) for k in s3.keys_of(db, user.id)])


@router.post("/s3/keys")
def create_s3_key(body: S3KeyCreate, user: Writer, db: Db, cfg: Cfg) -> S3KeyCreated:
"""An access key that reads the file tree as you, over S3 and WebDAV. Only while signed in to the web app (with your passkey),
not with an API token. The secret is in this answer only."""
if user.via != "access":
raise HTTPException(403, "make access keys while signed in")
with domain_errors():
kid, secret = s3.create_key(db, cfg, user.id, body.name, body.days)
auth.audit(db, user.as_audit(), "s3_key.create", f"file_key:{kid}")
st = shares.status(db, cfg)
return S3KeyCreated(id=kid, secret=secret, port=st["s3"]["port"], webdav_port=st["webdav"]["port"])


@router.delete("/s3/keys/{key_id}")
def revoke_s3_key(key_id: str, user: Writer, db: Db) -> Ok:
"""Revoke one of your keys (admins: anyone's). It stops working at once."""
if s3.revoke_key(db, key_id, None if user.admin and user.via != "oauth" else user.id):
auth.audit(db, user.as_audit(), "s3_key.revoke", f"file_key:{key_id}")
return Ok()


@router.get("/s3/keys")
def all_s3_keys(user: AdminReader, db: Db) -> list[S3Key]:
"""Everyone's access keys, with their owners (admins)."""
return [S3Key(**k) for k in s3.keys_of(db)]
2 changes: 2 additions & 0 deletions fastapi_backend/app/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,7 @@ def platform_main(argv, config):
routines,
sensors,
settings,
shares,
sources,
telemetry,
tunnel,
Expand Down Expand Up @@ -379,6 +380,7 @@ def password():
if not (a.steps or a.no_schedule): # and scans watched folders and runs routines, as `lens watch` does
routines.start(db, C, stop, log=print)
sensors.start(db, C, stop, log=print, name=wk.name) # and the sensor hub, while sensors are on
shares.start(db, C, stop, log=print, name=wk.name) # and the file shares (S3, WebDAV), while they're on
bridge.start(db, C, stop, lambda: cfg, name=wk.name, log_fn=print) # and the chat-room bridge, while it's on
tunnel.start(db, C, stop, name=wk.name, log=print) # and the Cloudflare tunnel, while it's on
https.start(db, C, stop, name=wk.name, log=print) # and HTTPS, where the ports are (LENS_HTTPS_LISTEN)
Expand Down
4 changes: 3 additions & 1 deletion fastapi_backend/app/domain/ops_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,10 @@
"matterbridge",
"telemetry",
"sensors",
"shares",
)
ALWAYS_ASK = {"telemetry", "sensors", "files"} # sensors open ports on the network; files moves where new files go
# sensors and shares open ports on the network; files moves where new files go
ALWAYS_ASK = {"telemetry", "sensors", "shares", "files"}

ADMIN_TOOLS = [
(
Expand Down
Loading
Loading
0