8000 feat(mesh): run Nebula on the primary in an optional mesh container by adeelahmad · Pull Request #553 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ These are open pull requests, not yet part of Lens:
([#406](https://github.com/adeelahmad/lense/pull/406)).
- **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
- **Nodes:** Lens's own private network between the machines it runs on, a Nebula mesh it issues the certificates
for ([#546](https://github.com/adeelahmad/lense/pull/546)).
for ([#553](https://github.com/adeelahmad/lense/pull/553)).

## Planned

Expand Down
5 changes: 5 additions & 0 deletions changelog.d/mesh-container.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **Nebula on the primary** (the private network between Lens's machines, in progress). On the primary Nebula runs in
the optional `mesh` container, which the upgrade agent starts when a node is added: it makes its own key pair, checks
Nebula's download against the kept checksums and relays the database and the API on the mesh address only. The
install line that joins a machine and the Settings pages follow in their own pull requests
([Nodes and the mesh](docs/mesh.md)).
36 changes: 36 additions & 0 deletions docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ x-backend: &backend
LENS_GRAFANA_URL: ${LENS_GRAFANA_URL:-}
# the volume shared with the matterbridge service (its config, written from Settings → Chat rooms)
LENS_MATTERBRIDGE_DIR: /matterbridge
# the volume shared with the mesh service (its config, written while the mesh is on; docs/mesh.md)
LENS_MESH_DIR: /mesh
# the MCP runner (the optional `mcp` service, docs/assistant.md): MCP servers that run as a program, sandboxed there
LENS_MCP_RUNNER_URL: http://mcp:8090
LENS_MCP_TOKEN_FILE: /mcp-key/token
Expand All @@ -98,6 +100,7 @@ x-backend: &backend
- ./docs:/docs:ro
- ${AUDIO_DIR:-./local-audio}:/audio:ro
- matterbridge-data:/matterbridge
- mesh-data:/mesh
- mcp-key:/mcp-key:ro
- sandbox-key:/sandbox-key:ro
depends_on:
Expand Down Expand Up @@ -451,9 +454,42 @@ services:
restart: unless-stopped
networks: [lens]

# The mesh (docs/mesh.md): Nebula, the private network between Lens's machines, on this machine (mesh/sidecar.py).
# Lens's server writes its config into the volume the two share; it makes this machine's key pair itself and holds
# NET_ADMIN for the mesh's network device, which nothing else in Lens does. It relays the database and the API to
# the machines the mesh's firewall lets in. Started by the upgrade agent when an admin adds a node (Settings ›
# Nodes), or `docker compose --profile mesh up -d`. Other machines reach it on UDP 4242 (LENS_MESH_PORT).
mesh:
image: python:3.12-slim # the plain image: the script is the repo's, Nebula is fetched with its checksum checked
profiles: [mesh]
command: ["python", "-I", "/lens-mesh/sidecar.py"]
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
cap_add: [NET_ADMIN]
devices: ["/dev/net/tun:/dev/net/tun"]
read_only: true
tmpfs: [/tmp]
mem_limit: 256m
pids_limit: 256
environment:
LENS_MESH_DIR: /mesh
LENS_MESH_STATE: /state
LENS_MESH_LISTEN_PORT: "4242"
LENS_MESH_RELAYS: 8000=surrealdb:8000,8080=backend:8000
ports:
- "${LENS_MESH_BIND:-0.0.0.0}:${LENS_MESH_PORT:-4242}:4242/udp"
volumes:
- ./mesh/sidecar.py:/lens-mesh/sidecar.py:ro
- mesh-data:/mesh
- mesh-state:/state
restart: unless-stopped
networks: [lens]

volumes:
surreal-data:
matterbridge-data:
mesh-data:
mesh-state:
mcp-data:
mcp-key:
sandbox-key:
Expand Down
35 changes: 35 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ x-backend: &backend
OPENAPI_OUTPUT_FILE: ./shared-data/openapi.json
# the volume shared with the matterbridge service (its config, written from Settings → Chat rooms)
LENS_MATTERBRIDGE_DIR: /matterbridge
# the volume shared with the mesh service (its config, written while the mesh is on; docs/mesh.md)
LENS_MESH_DIR: /mesh
# the MCP runner (the optional `mcp` service, docs/assistant.md): MCP servers that run as a program, sandboxed there
LENS_MCP_RUNNER_URL: http://mcp:8090
LENS_MCP_TOKEN_FILE: /mcp-key/token
Expand All @@ -65,6 +67,7 @@ x-backend: &backend
- ./docs:/docs:ro
- ${AUDIO_DIR:-./local-audio}:/audio:ro
- matterbridge-data:/matterbridge
- mesh-data:/mesh
- mcp-key:/mcp-key:ro
- sandbox-key:/sandbox-key:ro
depends_on:
Expand Down Expand Up @@ -337,9 +340,41 @@ services:
- fedora-data:/fcrepo-home
networks: [lens]

# The mesh (docs/mesh.md): Nebula, the private network between Lens's machines, on this machine (mesh/sidecar.py).
# Lens's server writes its config into the volume the two share; it makes this machine's key pair itself and holds
# NET_ADMIN for the mesh's network device, which nothing else in Lens does. It relays the database and the API to
# the machines the mesh's firewall lets in. Started by the upgrade agent when an admin adds a node (Settings ›
# Nodes), or `docker compose --profile mesh up -d`. Other machines reach it on UDP 4242 (LENS_MESH_PORT).
mesh:
image: python:3.12-slim # the plain image: the script is the repo's, Nebula is fetched with its checksum checked
profiles: [mesh]
command: ["python", "-I", "/lens-mesh/sidecar.py"]
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
cap_add: [NET_ADMIN]
devices: ["/dev/net/tun:/dev/net/tun"]
read_only: true
tmpfs: [/tmp]
mem_limit: 256m
pids_limit: 256
environment:
LENS_MESH_DIR: /mesh
LENS_MESH_STATE: /state
LENS_MESH_LISTEN_PORT: "4242"
LENS_MESH_RELAYS: 8000=surrealdb:8000,8080=backend:8000
ports:
- "${LENS_MESH_BIND:-0.0.0.0}:${LENS_MESH_PORT:-4242}:4242/udp"
volumes:
- ./mesh/sidecar.py:/lens-mesh/sidecar.py:ro
- mesh-data:/mesh
- mesh-state:/state
networks: [lens]

volumes:
surreal-data:
matterbridge-data:
mesh-data:
mesh-state:
mcp-data:
mcp-key:
sandbox-key:
Expand Down
5 changes: 3 additions & 2 deletions docs/assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -898,8 +898,9 @@ not [the assistant in your browser](browser-assistant.md), which uses your own b

### Optional containers

The MCP runner, the browser, the [code sandbox](#the-gateway)'s gateway and monitoring
([Prometheus and Grafana](prometheus.md)) are containers Lens can run next to it, off until an admin turns one on. The web app can't start containers itself, so the machine's upgrade agent does
The MCP runner, the browser, the [code sandbox](#the-gateway)'s gateway, monitoring
([Prometheus and Grafana](prometheus.md)) and the mesh ([Nodes and the mesh](mesh.md#on-this-machine), started when an
admin adds a node) are containers Lens can run next to it, off until an admin turns one on. The web app can't start containers itself, so the machine's upgrade agent does
([Updates](updates.md)): it adds the profile to `COMPOSE_PROFILES` in Lens's `.env`, or takes it out, and starts or
stops that profile's containers, one change at a time. Without an agent (Lens wasn't installed with `install.sh`), the
switch shows the line to run on the machine instead. Agents use the same switches: `GET /api/v1/admin/services` and
Expand Down
43 changes: 38 additions & 5 deletions docs/mesh.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Nodes and the mesh

Status: **in progress**. Built: Lens as the mesh's certificate authority, join tokens, joining without sending the
token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings and the API
under `/api/v1/mesh`. Coming in their own pull requests: Nebula running on the primary and relaying the database,
`lens node` and the install line that joins another machine, Settings › Network and Nodes in the web app, and running
code on compute nodes.
token, each machine's Nebula config, a node's renewals and reports, removing a node, the `mesh` settings, the API
under `/api/v1/mesh`, and Nebula running on the primary in the `mesh` container, relaying the database and the API.
Coming in their own pull requests: `lens node` and the install line that joins another machine, Settings › Network and
Nodes in the web app, and running code on compute nodes.

Lens runs on one machine until you add another: a Mac with a GPU to transcribe on, a box to run the assistant's code
on, a server with more disk. Those machines talk to this install over a private network of Lens's own, a
Expand Down Expand Up @@ -65,6 +65,37 @@ Over the mesh, with its node token (`Authorization: Bearer lensnode1.…`):

A removed node is refused on all three.

## On this machine

Lens's server can't create network devices, and nothing in Lens but Nebula needs to, so on the primary Nebula runs in
a container of its own, `mesh` (`mesh/sidecar.py`, in the plain Python image). It's an optional service like the
browser or monitoring: adding a node asks the machine's upgrade agent to start it, and turning the mesh off
(`mesh.enabled`) stops it. Without an upgrade agent, start it with the line Settings › Network shows:

```sh
docker compose -f docker-compose.prod.yml --profile mesh up -d
```

- **Its own key.** The container makes this machine's Nebula key pair the first time and keeps the private key in its
own volume. It shares a folder with Lens's server (`LENS_MESH_DIR`, `/mesh`): it puts the public key there, Lens
signs it and writes the config back (`lens.json`), and the container adds the private key and runs Nebula. Lens
signs it again when a third of its life is left, or when the container's key changes.
- **Checked before it runs.** It fetches Nebula's release for the machine on first use, checked against the same
checksums Lens keeps, and runs each new config past `nebula -test` first. A change Nebula can take while running
(a removed node's certificate on the blocklist, the firewall) is reloaded without dropping tunnels; a new network
device or port restarts it.
- **Relays.** It listens on this machine's mesh address only, and passes port 8000 to the database and port 8080 to
the API: what a worker node and every node reach here, as Nebula's firewall allows (below).
- **The port.** Other machines reach it on UDP 4242, which Docker publishes (`LENS_MESH_PORT` in `.env` publishes
another; `mesh.port` is the port nodes are told to use, so change both, or set `mesh.hosts` to `host:port` when a
router forwards another port to it).
- **What it says.** It writes how it runs (`status.json`): Nebula on or not, since when, the relays and Nebula's last
lines. `GET /api/v1/mesh` shows it under `service`, and the primary's own row in `/api/v1/mesh/nodes` carries it as
a node's report does.

It holds `NET_ADMIN` and `/dev/net/tun` and nothing else: no other capability, a read-only image, 256 MB of memory,
none of Lens's data or keys but its own key pair and the config.

## The config

Each machine's config is rendered by Lens, from the settings and the node's row; the node adds only its private key.
Expand All @@ -91,7 +122,7 @@ Everything is in **Settings › Network** (`mesh`), editable in the app with def
| --- | --- | --- |
| `enabled` | off | on when a node is added; off stops the mesh |
| `network` | `10.73.0.0/16` | the mesh's addresses, a private IPv4 range from /8 to /28; the primary is its first. It can't change once machines have addresses in it |
| `port` | `4242` | the UDP port the primary listens on; nodes must reach it |
| `port` | `4242` | the UDP port nodes reach the primary at: the one Docker publishes (`LENS_MESH_PORT`), unless a router forwards another |
| `hosts` | none | where nodes reach that port from outside the mesh (`host` or `host:port`, up to 8). Empty: the address each node joined this install at |
| `cipher` | `aes` | or `chachapoly` (faster on machines without AES hardware, such as some Raspberry Pis); the same on every machine |
| `cert_days` | 30 | how long a machine's certificate lasts; it renews at two thirds |
Expand All @@ -107,6 +138,8 @@ removes it (undo with `POST /nodes/{id}/restore` while its certificate is good).
## What keeps it safe

- Nothing listens until an admin adds a node, and single-machine installs never do.
- On the primary only the `mesh` container holds `NET_ADMIN`; it makes its own key pair, runs only Nebula's release with
the checksum kept in Lens, and its relays listen on the mesh address alone.
- Only a machine with a token an admin made joins; the token is spent on use, expires, and never crosses the wire.
- A machine's certificate names only its own address, role and namespaces; it can't widen them by renewing.
- Nebula's firewall, rendered by Lens, opens only what each role needs. Lens itself never trusts the mesh's address:
Expand Down
6 changes: 3 additions & 3 deletions docs/updates.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,9 @@ The same agent turns Lens's [optional containers](assistant.md#optional-containe
`action=services`, a `profile` and `turn=on` or `off` (in the same inbox file, the one every agent watches) adds or
removes that profile in `COMPOSE_PROFILES` in `.env` and starts or stops its containers, and nothing else. The
profiles are fixed in the agent: `mcp` (the MCP runner), `browser` (the browser and its proxy), `sandbox` (the
[code sandbox](assistant.md#the-gateway)'s gateway, its certificates and the client's key) and `monitoring`
([Charts and history](prometheus.md#prometheus-and-grafana-ready-made): Prometheus, node-exporter and Grafana); it
refuses any other. It reports in `services-status` and `services.log`. A request from an older Lens, with
[code sandbox](assistant.md#the-gateway)'s gateway, its certificates and the client's key), `monitoring`
([Charts and history](prometheus.md#prometheus-and-grafana-ready-made): Prometheus, node-exporter and Grafana) and
`mesh` ([Nodes and the mesh](mesh.md#on-this-machine): Nebula on the primary); it refuses any other. It reports in `services-status` and `services.log`. A request from an older Lens, with
`monitoring=on` or `off` and no profile, still works.

With no agent (Lens wasn't installed with install.sh, the machine has neither systemd nor launchd, or install.sh ran
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,7 @@ def platform_main(argv, config):
jobs,
local_llm,
logs,
mesh,
nfs_share,
notify,
passkeys,
Expand Down Expand Up @@ -406,6 +407,7 @@ def password():
shares.start(db, C, stop, log=say, name=wk.name) # and the file shares (S3, WebDAV), while they're on
channels.start_all(db, C, stop, lambda: cfg, name=wk.name, log_fn=say) # and the chat channels, while they're on
tunnel.start(db, C, stop, name=wk.name, log=say) # and the Cloudflare tunnel, while it's on
mesh.start(db, C, stop, name=wk.name, log=say) # and the mesh container's config, while the mesh is on
https.start(db, C, stop, name=wk.name, log=say) # and HTTPS, where the ports are (LENS_HTTPS_LISTEN)
local_llm.start(db, C, stop, name=wk.name, log=say) # and the local chat model, while it's on
scaling.start(db, C, stop, name=wk.name, log_=say) # and auto scaling (docs/scaling.md), while it's on
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/core/runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
https,
jobs,
local_llm,
mesh,
notify,
routines,
scaling,
Expand Down Expand Up @@ -101,6 +102,7 @@ def start_background(self) -> None:
sensors.start(self.db, self.current, self.stop, log=log.info, name=f"api-{os.getpid()}")
channels.start_all(self.db, self.current, self.stop, lambda: self.base, name=f"api-{os.getpid()}", log_fn=log.warning)
tunnel.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info)
mesh.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info)
https.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info)
local_llm.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log=log.info)
scaling.start(self.db, self.current, self.stop, name=f"api-{os.getpid()}", log_=log.info)
Expand Down
Loading
Loading
0