8000 feat(shares): reach S3 and WebDAV through remote access, at /s3 and /dav by adeelahmad · Pull Request #402 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content 7FFF
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .json  (2)

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
`automation_actions`, `save_automation`, `test_automation` and `run_automation`, with problems in plain words. The
assistant saves them switched off and asks before switching one on or running it
([Agents](docs/automations.md#agents)).
- **File shares through Remote access.** S3 and WebDAV can now be reached at the web app's own address, `/s3` and
`/dav`, so they work through the Cloudflare tunnel and HTTPS as well as on the network, and on Cloudron, which
publishes only the web app. It's off until an admin turns on **At the web app's address too** in Settings › File
shares; then a new access key's dialog gives those addresses. Signing in is the same: a person's own access key,
read-only. MQTT, syslog, SMB and NFS aren't HTTP and stay on their own ports
([The graph as files](docs/file-tree.md#at-the-web-apps-address), [Remote access](docs/remote-access.md#what-it-reaches)).
- **Versions in conversations.** A conversation is now a tree: editing a question makes a new version beside it
instead of deleting what followed, regenerating an answer (`regenerate`, with the same model or another) writes one
beside it, and a question can go on from any message (`parent`). `GET /api/v1/chats/<id>` shows each message's
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.cloudron
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ RUN UV_PROJECT_ENVIRONMENT=/app/code/venv UV_PYTHON_DOWNLOADS=never UV_COMPILE_B

# The web app, built in the stage above; Next writes its cache to /run (the code is read-only on Cloudron)
WORKDIR /app/code/frontend
COPY --from=frontend /app/package.json /app/next.config.mjs ./
COPY --from=frontend /app/package.json /app/next.config.mjs /app/lens-relay.js ./
COPY --from=frontend /app/node_modules ./node_modules
COPY --from=frontend /app/.next ./.next
COPY --from=frontend /app/public ./public
Expand Down
2 changes: 2 additions & 0 deletions cloudron/supervisord.conf
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ redirect_stderr=true
priority=40
user=cloudron
directory=/app/code/frontend
; lens-relay.js: the file shares at the app's address, /s3 and /dav, once allowed in Settings › File shares
environment=NODE_OPTIONS="--require /app/code/frontend/lens-relay.js"
command=/app/code/frontend/node_modules/.bin/next start --hostname 0.0.0.0 --port 3000
autorestart=true
stopasgroup=true
Expand Down
3 changes: 3 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,9 @@ services:
AUTH_TRUST_HOST: "true"
# the containers running the Cloudflare tunnel: their X-Forwarded-For names the visitor (lens-server.js)
LENS_WEB_PROXY_HOSTS: backend,worker
# the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares
LENS_S3_URL: http://worker:9555
LENS_WEBDAV_URL: http://worker:9556
# no AUTH_URL: sign-in follows the address the browser is on (its LAN name, https:// address or Cloudflare tunnel)
ports:
# this machine only by default; LENS_BIND=0.0.0.0 to reach it from the network (install.sh does on a server)
Expand Down
4 changes: 4 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,10 @@ services:
AUTH_TRUST_HOST: "true"
# no AUTH_URL: sign-in follows the address the browser is on (its LAN name, https:// address or Cloudflare tunnel)
OPENAPI_OUTPUT_FILE: ./shared-data/openapi.json
# the file shares at the web app's address, /s3 and /dav, once allowed in Settings › File shares (lens-relay.js)
NODE_OPTIONS: --require /app/lens-relay.js
LENS_S3_URL: http://worker:9555
LENS_WEBDAV_URL: http://worker:9556
volumes:
- ./nextjs-frontend:/app
- /app/node_modules # the image's, renewed with it (see the backend's /opt/venv)
Expand Down
1 change: 1 addition & 0 deletions docs/configuration.md
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ whether the wizard is still pending (`wizard_pending`).
| `AUTH_TRUST_HOST` | `true` behind a proxy or in Docker |
| `TRUST_PROXY_HEADERS` | `true` when a reverse proxy in front of the web app sets `X-Forwarded-Host`, `-Proto` and `-For`: they're passed on to the API, which names that address in OAuth discovery and reads the visitor's address for IP groups and throttles. Off, the web app reports the `Host` the browser sent and the address it was reached from |
| `LENS_WEB_PROXY_HOSTS` | hosts whose `X-Forwarded-For` the web app passes on with `TRUST_PROXY_HEADERS` off (`backend,worker` in the Docker Compose files, for the Cloudflare tunnel) |
| `LENS_S3_URL`, `LENS_WEBDAV_URL` | where the web app finds the file shares it relays at `/s3` and `/dav` once Settings › File shares allows it (`http://worker:9555` and `http://worker:9556` in the Docker Compose files; unset, this machine's ports 9555 and 9556). See [The graph as files](file-tree.md#at-the-web-apps-address) |
| `LENS_HTTPS`, `LENS_HTTPS_PORT`, `LENS_HTTP_PORT`, `LENS_HTTPS_BIND` | install.sh: `LENS_HTTPS=1` publishes the HTTPS and HTTP ports (443 and 80 by default) for [HTTPS](https.md) |
| `LENS_HTTPS_LISTEN` | `1` in the process that has the HTTPS ports and runs Caddy (the worker in `docker-compose.https.yml`); `LENS_CADDY`: a caddy to use instead of fetching one |

Expand Down
34 changes: 34 additions & 0 deletions docs/file-tree.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,43 @@ namespaces. Listing (PROPFIND, one level at a time) and reading (GET, with range
Like S3, it speaks plain HTTP, and Basic sign-in sends the secret with every request: keep it on this machine or
behind TLS on a network you don't trust.

## At the web app's address

The web app can relay both shares at its own address: S3 at `/s3` and WebDAV at `/dav`. They then work wherever the web
app opens, including through [Remote access](remote-access.md) and [HTTPS](https.md), which serve the web app over
`https://` and reach nothing else. It's off until an admin turns on **At the web app's address too** in Settings, File
shares (`shares.remote`), on top of the shares themselves; until then requests that come that way are turned away
before anyone signs in, and the shares' own ports work as before.

Once it's on, the dialog that shows a new access key gives these addresses:

```ini
[lens]
type = s3
provider = Other
access_key_id = LK…
secret_access_key = …
endpoint = https://lens.example.com/s3
force_path_style = true
```

and `https://lens.example.com/dav/` for WebDAV (Finder, Windows Explorer, `davs://lens.example.com/dav/` in GNOME Files
or Dolphin, rclone's `type = webdav`). The S3 endpoint has a path, so it needs a client that takes one: rclone, the AWS
CLI and boto3 do.

How it works: the web app hands requests under `/s3` and `/dav` to the worker's share servers before Next.js sees them
(`nextjs-frontend/lens-relay.js`; the servers are at `LENS_S3_URL` and `LENS_WEBDAV_URL`, set in the Docker Compose
files, else this machine's ports 9555 and 9556). The path and `Host` stay as the client sent them, since S3 clients
sign both, and `X-Forwarded-Prefix` tells the share server the request came through the web app. Failed WebDAV
sign-ins are counted per visitor, from `X-Forwarded-For` as the web app and the proxies Lens trusts report it
(Settings, Access & embedding, Trusted proxies). Signing in is the same as at the ports: a person's own access key,
read-only.

## Refine later

- TLS by itself for S3 and WebDAV, folders' real dates, and WebDAV locks for clients that insist on them.
- Virtual-host S3 addressing at the web app's address, for clients that can't take an endpoint with a path (MinIO's
`mc`).
- Show supplementary files (captions, translations, attachments) beside each asset.
- Cache an asset's text and metadata so large folders list faster.
- A sidecar for each collection, topic and entity, and a folder of notes.
Expand Down
3 changes: 3 additions & 0 deletions docs/https.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,9 @@ All in Settings › HTTPS (`PUT /api/v1/settings/https`; `POST /api/v1/settings/
Passkeys and sign-in follow the https:// address, and email links use the first domain unless
`notifications.app_url` says otherwise.
- **Changes**: a saved change restarts Caddy with the new configuration within a few seconds.
- **What it serves**: the web app, and what the web app serves at its own address: the API, the MCP server at `/mcp`,
and the file shares at `/s3` and `/dav` once Settings › File shares allows them there
([The graph as files](file-tree.md#at-the-web-apps-address)). MQTT, syslog, SMB and NFS keep their own ports.

By hand, with Docker:

Expand Down
12 changes: 12 additions & 0 deletions docs/remote-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,18 @@ command (the long string after `--token`).
- The tunnel's address counts as one Lens is served at: passkeys and sign-in redirects use `https://<that host>`,
and email links use the fixed hostname unless Settings › Notifications sets another address.

## What it reaches

The tunnel serves the web app and what the web app serves at its own address:

| | Through the tunnel |
| --- | --- |
| The web app, the API (`/api/v1`) and sign-in | Yes |
| The MCP server (`/mcp`, with OAuth) | Yes ([MCP server](mcp.md)) |
| Sensors that push over HTTP (`/api/v1/sensors/push/…`) | Yes ([Sensors](sensors.md)) |
| The file shares: S3 at `/s3` and WebDAV at `/dav` | Once an admin turns on **At the web app's address too** in Settings › File shares ([The graph as files](file-tree.md#at-the-web-apps-address)) |
| MQTT, syslog, SMB and NFS | No: they aren't HTTP, so they stay on their own ports, for the local network |

## Where it sends visitors

cloudflared reaches the web app at, in order: the **web app address** in Settings › Remote access, the
Expand Down
26 changes: 1 addition & 25 deletions fastapi_backend/app/api/deps.py
Original file line number Diff line number Diff line change
Expand Up @@ -381,33 +381,9 @@ def client_ip(request: Request) -> str:
return str(addr) if addr else (request.client.host if request.client else "")


_PROXY_HOSTS: dict[str, tuple[float, tuple[str, ...]]] = {}


def _proxy_hosts() -> tuple[str, ...]:
"""The addresses of the hosts named in LENS_TRUSTED_PROXY_HOSTS (the web app's container, `frontend` in the
Docker Compose files, whose address changes when it's recreated), looked up at most every 30 seconds."""
import os
import socket
import time

names = [n.strip() for n in os.environ.get("LENS_TRUSTED_PROXY_HOSTS", "").split(",") if n.strip()]
out: list[str] = []
for name in names:
hit = _PROXY_HOSTS.get(name)
if not hit or time.monotonic() - hit[0] > 30:
try:
found = tuple(sorted({str(i[4][0]) for i in socket.getaddrinfo(name, None)}))
except OSError:
found = ()
hit = _PROXY_HOSTS[name] = (time.monotonic(), found)
out += hit[1]
return tuple(out)


def trusted_proxies(request: Request) -> tuple[str, ...]:
"""server.trusted_proxies, and the web app's own container (LENS_TRUSTED_PROXY_HOSTS)."""
return tuple(request.app.state.settings.current()["server"].get("trusted_proxies") or ()) + _proxy_hosts()
return tuple(request.app.state.settings.current()["server"].get("trusted_proxies") or ()) + ipgroups.proxy_hosts()


def visitor_address(request: Request) -> ipaddress.IPv4Address | ipaddress.IPv6Address | None:
Expand Down
7 changes: 4 additions & 3 deletions fastapi_backend/app/api/v1/routes/shares.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,11 @@

@router.get("/s3")
def s3_status(user: CurrentUser, db: Db, cfg: Cfg) -> S3Status:
"""Whether the S3 server is on and on which port, what the worker running it says, the same for WebDAV, and your
access keys."""
"""Whether the S3 server is on and on which port, what the worker running it says, the same for WebDAV, whether
both are open at the web app's address too (/s3, /dav), and your access keys."""
st = shares.status(db, cfg)
return S3Status(**st["s3"], webdav=ShareServer(**st["webdav"]), keys=[S3Key(**k) for k in s3.keys_of(db, user.id)])
keys = [S3Key(**k) for k in s3.keys_of(db, user.id)]
return S3Status(**st["s3"], webdav=ShareServer(**st["webdav"]), remote=shares.remote(cfg), keys=keys)


@router.post("/s3/keys")
Expand Down
23 changes: 23 additions & 0 deletions fastapi_backend/app/domain/ipgroups.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@

import functools
import ipaddress
import os
im 8 port soc 10BC8 ket
import time
from typing import NamedTuple

from . import store
Expand Down Expand Up @@ -83,6 +86,26 @@ def address(raw):
return (ip.ipv4_mapped or ip) if isinstance(ip, ipaddress.IPv6Address) else ip


_PROXY_HOSTS: dict[str, tuple[float, tuple[str, ...]]] = {}


def proxy_hosts() -> tuple[str, ...]:
"""The addresses of the hosts named in LENS_TRUSTED_PROXY_HOSTS (the web app's container, `frontend` in the
Docker Compose files, whose address changes when it's recreated), looked up at most every 30 seconds."""
names = [n.strip() for n in os.environ.get("LENS_TRUSTED_PROXY_HOSTS", "").split(",") if n.strip()]
out: list[str] = []
for name in names:
hit = _PROXY_HOSTS.get(name)
if not hit or time.monotonic() - hit[0] > 30:
try:
found = tuple(sorted({str(i[4][0]) for i in socket.getaddrinfo(name, None)}))
except OSError:
found = ()
hit = _PROXY_HOSTS[name] = (time.monotonic(), found)
out += hit[1]
return tuple(out)


@functools.lru_cache(maxsize=256)
def _networks(values):
out = []
Expand Down
11 changes: 11 additions & 0 deletions fastapi_backend/app/domain/s3.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@

Path-style addressing (http://host:port/<bucket>/<key>), any region name. Supported: ListBuckets, HeadBucket,
GetBucketLocation, ListObjects (v1 and v2, with a delimiter or recursive), HeadObject and GetObject (with Range).

With `shares.remote` on, the web app relays it at its own address too (https://<web app>/s3/<bucket>/<key>), path and
all: clients sign that path, and the bucket and key come after /s3 (shares.web_path()).
"""

from __future__ import annotations
Expand Down Expand Up @@ -423,12 +426,20 @@ def _write(self):
do_PUT = do_POST = do_DELETE = _write # noqa: N815 - http.server's names

def _handle(self, head):
from . import shares

raw, _, query = self.path.partition("?")
try:
pairs = urllib.parse.parse_qsl(query, keep_blank_values=True)
headers = {k.lower(): v for k, v in self.headers.items()}
db, cfg = self.server.db, self.server.cfg_fn()
prefix = shares.web_path(headers, "s3")
if prefix and not shares.remote(cfg):
raise S3Error(
403, "AccessDenied", "S3 isn't open at the web app's address: an admin turns that on in Settings, File shares"
)
acct = verify(db, cfg, self.command, raw, pairs, headers)
raw = shares.share_path(raw, prefix)
tree = filetree.Tree(db, cfg, filetree.Viewer.of(db, acct))
path = urllib.parse.unquote(raw)
bucket, _, key = path.lstrip("/").partition("/")
Expand Down
4 changes: 2 additions & 2 deletions < 10BC8 span class="Truncate"> fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@
"triage",
),
# bind is a startup setting only
"shares": ("enabled", "s3", "s3_port", "webdav", "webdav_port"),
"shares": ("enabled", "s3", "s3_port", "webdav", "webdav_port", "remote"),
"video": (
"sample_seconds",
"scene_threshold",
Expand Down Expand Up @@ -1390,7 +1390,7 @@ def _web_setting(key, value):


def _share_setting(key, value):
if key in ("enabled", "s3", "webdav"):
if key in ("enabled", "s3", "webdav", "remote"):
if not isinstance(value, bool):
raise ValueError(f"shares.{key} is true or false")
return value
Expand Down
41 changes: 40 additions & 1 deletion fastapi_backend/app/domain/shares.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@
Off until an admin turns them on (`shares.enabled`, then `s3` and `webdav`); the process that runs routines (`lens
worker`) follows the settings as they change, as the sensor hub does, and says what each server is doing in
`app_service:share-<protocol>`.

The web app also relays them at its own address, /s3 and /dav (nextjs-frontend/lens-relay.js), so they can be reached
wherever the web app is, Settings › Remote access included. Those requests keep their path and carry X-Forwarded-Prefix,
and the servers turn them away until an admin turns on `shares.remote`.
"""

from __future__ import annotations
Expand All @@ -15,7 +19,7 @@
from collections.abc import Callable
from typing import Any

from . import s3, store, webdav
from . import ipgroups, s3, store, webdav

R = store.R
CHECK_SECONDS = 5
Expand All @@ -27,6 +31,11 @@
}


# protocol -> the path the web app relays it at (nextjs-frontend/lens-relay.js)
WEB_PATHS = {"s3": "/s3", "webdav": "/dav"}
LOOPBACK = ("127.0.0.0/8", "::1/128")


def _row(proto):
return R("app_service", f"share-{proto}")

Expand Down Expand Up @@ -55,6 +64,36 @@ def settings_of(cfg):
return {**store.DEFAULTS["shares"], **(cfg.get("shares") or {})}


def web_path(headers, proto):
"""The path the web app relays this protocol at, when the request came that way (its X-Forwarded-Prefix), else "".
Anyone on the network could send the header too; it only gets their request turned away, or signed and linked
under that path."""
prefix = (headers.get("x-forwarded-prefix") or "").strip()
return prefix if prefix == WEB_PATHS[proto] else ""


def share_path(path, prefix):
"""The path a request asks for within the share: without the web app's prefix (the web app relays the path as the
client sent it, which is also the path S3 clients sign)."""
if prefix and (path == prefix or path.startswith(prefix + "/")):
return path[len(prefix) :] or "/"
return path


def remote(cfg):
"""Whether the shares may be reached at the web app's address (shares.remote)."""
return bool(settings_of(cfg).get("remote"))


def visitor(handler, cfg):
"""The address a request comes from, for throttles: what the web app (and proxies it trusts: server.trusted_proxies,
LENS_TRUSTED_PROXY_HOSTS, this machine) report in X-Forwarded-For, else the connection's peer."""
peer, port = handler.client_address[0], handler.client_address[1]
trusted = tuple((cfg.get("server") or {}).get("trusted_proxies") or ()) + ipgroups.proxy_hosts() + LOOPBACK
addr = ipgroups.client_address(peer, port or 1, handler.headers.get("x-forwarded-for", ""), trusted)
return str(addr) if addr else peer


def wanted(cfg):
"""{protocol: (bind, port) it should listen on, or None while it's off}."""
s = settings_of(cfg)
Expand Down
13 changes: 11 additions & 2 deletions fastapi_backend/app/domain/store.py
Original file line number Diff line number Diff line change
Expand Up @@ -587,8 +587,17 @@ def labels():
"account": None,
},
# the graph as files over S3 and WebDAV (shares.py, docs/file-tree.md): off until an admin turns it on. Then the process
# that runs routines (`lens worker`) serves it, read-only, on these ports. bind is a startup setting only.
"shares": {"enabled": False, "bind": "0.0.0.0", "s3": True, "s3_port": 9555, "webdav": True, "webdav_port": 9556},
# that runs routines (`lens worker`) serves it, read-only, on these ports. bind is a startup setting only. remote: also at
# the web app's address (/s3, /dav), so through Settings › Remote access and HTTPS too; off until an admin turns it on.
"shares": {
"enabled": False,
"bind": "0.0.0.0",
"s3": True,
"s3_port": 9555,
"webdav": True,
"webdav_port": 9556,
"remote": False,
},
# notifications to webhooks and Matterbridge (docs/notifications.md): targets reach public addresses only, and the
# private networks listed here (a Matterbridge on the LAN or the Docker network); app_url is where links in messages
# point (null: FRONTEND_URL)
Expand Down
Loading
Loading
0