8000 feat(sensors): reach the MQTT hub through remote access, over WebSockets at /mqtt by adeelahmad · Pull Request #508 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
step in the pipeline editor. Agents write one with `POST /api/v1/recordings/{id}/spec`, the MCP tools `write_spec` and
`get_spec`, and the assistant's `write_spec` (which asks first); pipelines and batches run it as a `spec` step
([Specs from transcripts](docs/notes.md#specs-from-transcripts)).
- **MQTT through Remote access.** Devices away from home can publish to the sensor hub, and subscribe to it, at the
web app's own address: MQTT over WebSockets at `/mqtt` (`wss://` under HTTPS), through the Cloudflare tunnel and
HTTPS too. It's off until an admin turns on **MQTT at the web app's address too** in Settings › Sensors, and there
every client signs in with a hub login, with too many wrong passwords from one address turned away for a while. It's
the same broker as on port 1883. SMB, NFS and syslog stay on the local network, and Remote access now says why
([Sensors](docs/sensors.md#at-the-web-apps-address), [Remote access](docs/remote-access.md#what-it-reaches)).
- **Export a conversation.** `GET /api/v1/chats/<id>/export.md`, `.pdf` or `.json` gives the version showing, one
message (`?message=`) or, as JSON, every version (`?all=true`). The Markdown has each question and answer, its
citations as footnotes linking back to the moment in Lens and the tools it used; the PDF is that Markdown printed by
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
agents over MCP get the archive as data, and what Lens caught goes into the audit log. Serve Lens over HTTPS, set
up in Settings, with certificates from Let's Encrypt or ZeroSSL that renew on their own, answering only for its own
domains and, if you choose, only to devices holding a client certificate; or reach it from anywhere through a
Cloudflare tunnel with nothing opened on your router. ([Encryption](docs/encryption.md), [access](docs/access.md),
Cloudflare tunnel with nothing opened on your router. Either way, the S3 and WebDAV shares and the MQTT hub answer at
that same address once you turn them on. ([Encryption](docs/encryption.md), [access](docs/access.md),
[prompt injection](docs/assistant.md#third-party-content-and-prompt-injection), [HTTPS](docs/https.md),
[remote access](docs/remote-access.md); [checks](docs/requirements.md#stay-private-and-safe))
- **Use open standards.** RDF and Dublin Core for every record, OAuth for apps and OpenID Connect sign-in, MCP, MQTT,
Expand Down
3 changes: 2 additions & 1 deletion cloudron/supervisord.conf
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@ redirect_stderr=true
priority=40
user=cloudron
directory=/app/code/frontend
; lens-relay.js: the file shares at the app's address, /s3 and /dav, once allowed in Settings › File shares
; lens-relay.js: the file shares at the app's address, /s3 and /dav, once allowed in Settings › File shares, and
; MQTT over WebSockets at /mqtt, once allowed in Settings › Sensors
environment=NODE_OPTIONS="--require /app/code/frontend/lens-relay.js"
command=/app/code/frontend/node_modules/.bin/next start --hostname 0.0.0.0 --port 3000
autorestart=true
Expand Down
4 changes: 3 additions & 1 deletion docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,9 +155,11 @@ services:
AUTH_TRUST_HOST: "true"
# the containers running the Cloudflare tunnel: their X-Forwarded-For names the visitor (lens-server.js)
LENS_WEB_PROXY_HOSTS: backend,worker
# the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares
# the file shares at the web app's address, /s3 and /dav (lens-relay.js), once allowed in Settings › File shares,
# and MQTT over WebSockets at /mqtt, once allowed in Settings › Sensors
LENS_S3_URL: http://worker:9555
LENS_WEBDAV_URL: http://worker:9556
LENS_MQTT_WS_URL: http://worker:9883
# no AUTH_URL: sign-in follows the address the browser is on (its LAN name, https:// address or Cloudflare tunnel)
ports:
# this machine only by default; LENS_BIND=0.0.0.0 to reach it from the network (install.sh does on a server)
Expand Down
4 changes: 3 additions & 1 deletion docker-compose.yml
8000
Original file line number Diff line number Diff line change
Expand Up @@ -122,10 +122,12 @@ services:
AUTH_TRUST_HOST: "true"
# no AUTH_URL: sign-in follows the address the browser is on (its LAN name, https:// address or Cloudflare tunnel)
OPENAPI_OUTPUT_FILE: ./shared-data/openapi.json
# the file shares at the web app's address, /s3 and /dav, once allowed in Settings › File shares (lens-relay.js)
# the file shares at the web app's address, /s3 and /dav, once allowed in Settings › File shares, and MQTT over
# WebSockets at /mqtt, once allowed in Settings › Sensors (lens-relay.js)
NODE_OPTIONS: --require /app/lens-relay.js
LENS_S3_URL: http://worker:9555
LENS_WEBDAV_URL: http://worker:9556
LENS_MQTT_WS_URL: http://worker:9883
volumes:
- ./nextjs-frontend:/app
- /app/node_modules # the image's, renewed with it (see the backend's /opt/venv)
Expand Down
1 change: 1 addition & 0 deletions docs/configuration.md
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ whether the wizard is still pending (`wizard_pending`).
| `TRUST_PROXY_HEADERS` | `true` when a reverse proxy in front of the web app sets `X-Forwarded-Host`, `-Proto` and `-For`: they're passed on to the API, which names that address in OAuth discovery and reads the visitor's address for IP groups and throttles. Off, the web app reports the `Host` the browser sent and the address it was reached from |
| `LENS_WEB_PROXY_HOSTS` | hosts whose `X-Forwarded-For` the web app passes on with `TRUST_PROXY_HEADERS` off (`backend,worker` in the Docker Compose files, for the Cloudflare tunnel) |
| `LENS_S3_URL`, `LENS_WEBDAV_URL` | where the web app finds the file shares it relays at `/s3` and `/dav` once Settings › File shares allows it (`http://worker:9555` and `http://worker:9556` in the Docker Compose files; unset, this machine's ports 9555 and 9556). See [The graph as files](file-tree.md#at-the-web-apps-address) |
| `LENS_MQTT_WS_URL` | where the web app finds the sensor hub's MQTT over WebSockets, which it relays at `/mqtt` once Settings › Sensors allows it (`http://worker:9883` in the Docker Compose files; unset, this machine's port 9883). See [Sensors](sensors.md#at-the-web-apps-address) |
| `LENS_HTTPS`, `LENS_HTTPS_PORT`, `LENS_HTTP_PORT`, `LENS_HTTPS_BIND` | install.sh: `LENS_HTTPS=1` publishes the HTTPS and HTTP ports (443 and 80 by default) for [HTTPS](https.md) |
| `LENS_HTTPS_LISTEN` | `1` in the process that has the HTTPS ports and runs Caddy (the worker in `docker-compose.https.yml`); `LENS_CADDY`: a caddy to use instead of fetching one |

Expand Down
6 changes: 4 additions & 2 deletions docs/https.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,10 @@ All in Settings › HTTPS (`PUT /api/v1/settings/https`; `POST /api/v1/settings/
`notifications.app_url` says otherwise.
- **Changes**: a saved change restarts Caddy with the new configuration within a few seconds.
- **What it serves**: the web app, and what the web app serves at its own address: the API, the MCP server at `/mcp`,
and the file shares at `/s3` and `/dav` once Settings › File shares allows them there
([The graph as files](file-tree.md#at-the-web-apps-address)). MQTT, syslog, SMB and NFS keep their own ports.
the file shares at `/s3` and `/dav` once Settings › File shares allows them there
([The graph as files](file-tree.md#at-the-web-apps-address)), and MQTT over WebSockets at `/mqtt` once Settings ›
Sensors does ([Sensors](sensors.md#at-the-web-apps-address)). MQTT on port 1883, syslog, SMB and NFS keep their own
ports.

By hand, with Docker:

Expand Down
8 changes: 7 additions & 1 deletion docs/remote-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,13 @@ The tunnel serves the web app and what the web app serves at its own address:
| The MCP server (`/mcp`, with OAuth) | Yes ([MCP server](mcp.md)) |
| Sensors that push over HTTP (`/api/v1/sensors/push/…`) | Yes ([Sensors](sensors.md)) |
| The file shares: S3 at `/s3` and WebDAV at `/dav` | Once an admin turns on **At the web app's address too** in Settings › File shares ([The graph as files](file-tree.md#at-the-web-apps-address)) |
| MQTT, syslog, SMB and NFS | No: they aren't HTTP, so they stay on their own ports, for the local network |
| The sensor hub's MQTT broker, over WebSockets at `/mqtt` | Once an admin turns on **MQTT at the web app's address too** in Settings › Sensors ([Sensors](sensors.md#at-the-web-apps-address)) |
| MQTT on port 1883, syslog, SMB and NFS | No: they aren't HTTP, so they stay on their own ports, for the local network |

SMB and NFS stay on the local network: neither travels over HTTP, and both are made for a LAN (NFS has no sign-in at
all). To open the graph's files from elsewhere, use WebDAV at `/dav`, which macOS, Windows and Linux mount as a drive,
or S3 at `/s3`. Syslog has no sign-in either, so it stays local too; devices elsewhere send readings to a
[webhook](sensors.md#ways-in) or publish over MQTT at `/mqtt`.

## Where it sends visitors

Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ request, to another check that proves the same thing; loosening a requirement or
| Chat-room answers have the links they built broken up; agents over MCP get the archive as data | [test_prompt_injection.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_prompt_injection.py) `test_a_room_gets_the_links_the_answer_built_broken_up`; [test_mcp.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_mcp.py) `test_what_the_archive_says_reaches_agents_as_data` |
| HTTPS set up in Settings, with certificates that renew, only for its own domains and optionally only for client certificates | [test_https.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_https.py) `test_https_is_set_up_in_the_app`, `test_caddys_configuration_follows_the_settings`, `test_strict_sni_client_certificates_and_tls_13` |
| Remote access through a Cloudflare tunnel | [test_tunnel.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_tunnel.py) `test_remote_access_is_set_up_in_the_app`, `test_a_quick_tunnel_runs_in_one_process_and_stops_when_turned_off`, `test_a_managed_tunnel_is_made_routed_and_given_a_dns_record` |
| The S3 and WebDAV shares and the MQTT hub at the web app's address (`/s3`, `/dav`, `/mqtt` over WebSockets), each off until turned on | [test_s3.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_s3.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_webdav.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_webdav.py) `test_at_the_web_apps_address_once_an_admin_allows_it`; [test_sensors.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_sensors.py) `test_broker_over_websockets`, `test_mqtt_over_websockets_takes_hub_logins_only`; [lens-relay.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/lens-relay.test.ts) `relays any method, path and body to the share, and its answer back`, `passes the handshake on with the visitor and the prefix, then joins the two connections` |
| The threat model's three files agree, and a change to the prose comes with its companions, without a line every change edits | [threat-model.yml](https://github.com/adeelahmad/lense/blob/main/.github/workflows/threat-model.yml) `validate`, `Validate the threat model`, `Test the validator`; [validate_threat_model.py](https://github.com/adeelahmad/lense/blob/main/ci/validate_threat_model.py); [test_validate_threat_model.py](https://github.com/adeelahmad/lense/blob/main/ci/test_validate_threat_model.py) `test_prose_change_without_companions_fails`, `test_two_changes_to_the_prose_merge_cleanly` |

## Use open standards
Expand Down
24 changes: 23 additions & 1 deletion docs/sensors.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,27 @@ network reach them. Run natively, it runs in `lens worker` (and in
the API when it runs background work). Several processes can run it: the first to get the port serves it, and the
others try again every minute.

## At the web app's address

Devices away from home (a phone, a sensor at another site) can reach the hub wherever the web app opens, through
[Remote access](remote-access.md) or [HTTPS](https.md): MQTT over WebSockets at `/mqtt`. An admin turns it on with
**MQTT at the web app's address too** in Settings → Sensors (`sensors.mqtt_web`, off by default, and only with MQTT on).

- The address is the web app's with `wss://` and `/mqtt` (`ws://` on plain http), such as
`wss://lens.example.com/mqtt`. The sensors page shows it next to the MQTT port once it's on. Any client with a
WebSocket transport takes it, for example paho-mqtt (`transport="websockets"`, path `/mqtt`) or mqtt.js in a browser.
- Every client signs in with a hub login. Clients without one are turned away here even when `sensors.mqtt_anonymous`
lets them in on port 1883, and eight wrong passwords from one address in 15 minutes turn that address away for a
while, as sign-in does.
- It is the same broker: what a client publishes over WebSockets reaches subscribers on port 1883 and becomes a sensor
as it would there (in the login's namespace), and the other way round.

How it works: the worker listens for MQTT over WebSockets on `sensors.mqtt_web_port` (9883, on `sensors.bind` like
1883), and the web app passes `/mqtt` on to it (`nextjs-frontend/lens-relay.js`, at `LENS_MQTT_WS_URL`:
`http://worker:9883` in the Docker Compose files, else this machine's port 9883). The broker reads each visitor's address
from the web app's `X-Forwarded-For`, which it trusts from the hosts in `LENS_TRUSTED_PROXY_HOSTS` and this machine. In
Docker, port 9883 isn't published: devices reach it through the web app.

## Organising itself

A device publishing to the hub, or a host sending syslog, becomes a sensor the first time it does, marked **new**. Its
Expand Down Expand Up @@ -137,6 +158,7 @@ folders feeding their namespaces):

## Refine later

Notifications for alert patterns, sensors for namespace members (admins only for now), MQTT 5, the port mappings of
Notifications for alert patterns, sensors for namespace members (admins only for now), MQTT 5, per-topic access for
hub logins (any login can subscribe to `#`, at `/mqtt` too), the port mappings of
the Synology, QNAP, Proxmox and Cloudron packages; in the web app, a chart with axes and zoom, paging back through
readings, and labelling many log patterns at once.
12 changes: 12 additions & 0 deletions fastapi_backend/app/domain/ipgroups.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,18 @@ def client_address(peer, port, forwarded, trusted):
return None


LOOPBACK = ("127.0.0.0/8", "::1/128")


def visitor(cfg, peer, port, forwarded):
"""The address a connection to one of Lens's own servers (the file shares, MQTT over WebSockets) comes from, for
throttles: what the web app relaying it (and proxies it trusts: server.trusted_proxies, LENS_TRUSTED_PROXY_HOSTS,
this machine) report in X-Forwarded-For, else the connection's peer."""
trusted = tuple((cfg.get("server") or {}).get("trusted_proxies") or ()) + proxy_hosts() + LOOPBACK
addr = client_address(peer, port or 1, forwarded or "", trusted)
return str(addr) if addr else peer


# ---------- what an address opens ----------
class Network(NamedTuple):
"""What the visitor's address opens: whole namespaces and chosen recordings, each with the name of the IP group
Expand Down
Loading
Loading
0