8000 feat(https): set HTTPS up in Settings and open its ports from the web app by adeelahmad · Pull Request #326 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,8 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
- **HTTPS with automatic certificates.** Lens can serve itself at `https://` your domain with a certificate it gets
and renews on its own from Let's Encrypt or ZeroSSL (or any ACME authority), proving the domain over HTTP, TLS-ALPN
or DNS, with Cloudflare, Route 53 and the other caddy-dns providers, so wildcards and machines the internet can't
reach work too. Run the install line with `LENS_HTTPS=1` to open ports 443 and 80, then set it up with
`PUT /api/v1/settings/https`; the Settings page comes next. Off by default ([HTTPS](docs/https.md)).
reach work too. Set it up in **Settings › HTTPS**, whose **Open the HTTPS ports** has the machine open 443 and 80
and restart Lens (or run the install line with `LENS_HTTPS=1`). Off by default ([HTTPS](docs/https.md)).

- **Budget meter and Costs page.** The top bar shows what the namespace picked spent against its budget, used and
left, highlighted when near or over, and follows the namespace switcher; it opens the details for a day, week or
Expand Down
15 changes: 8 additions & 7 deletions docs/https.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,18 @@ Lens can serve itself at `https://lens.example.com` with a certificate it gets a
renews it about a month before it runs out and passes visitors on to the web app. Passkeys work there, since browsers
allow them on https:// addresses.

Status: **built** in the API and the installer; the **Settings › HTTPS** page in the web app is in progress (until
then, `PUT /api/v1/settings/https` sets it). Off by default.
Status: **built**, set up in **Settings › HTTPS**. Off by default.

For a machine the internet can't reach at all, [Remote access](remote-access.md) (a Cloudflare tunnel) is the
other way to an https:// address.

## Turning it on

1. Point the domain's DNS at this machine (an A or AAAA record), unless you use the DNS challenge.
2. Give Lens the HTTPS ports: run the install line again with `LENS_HTTPS=1`. It publishes ports 443 and 80 to the
worker, where Caddy runs, and remembers it in `.env`, so upgrades keep them.
2. Give Lens the HTTPS ports: **Open the HTTPS ports** in Settings › HTTPS has the machine's upgrade agent
([Updates](updates.md)) run install.sh again for the version running, with `LENS_HTTPS=1`, and Lens restarts. Or
run the install line again with it yourself. Either way it publishes ports 443 and 80 to the worker, where Caddy
runs, and remembers it in `.env`, so upgrades keep them.

```sh
curl -fsSL https://raw.githubusercontent.com/adeelahmad/lense/main/install.sh | LENS_HTTPS=1 sh
Expand Down Expand Up @@ -63,7 +64,7 @@ its fields as JSON. `dns_package` names a plugin outside caddy-dns (`github.com/

## Settings

All in Settings › HTTPS (the `https` section of `PUT /api/v1/settings/https`):
All in Settings › HTTPS (`PUT /api/v1/settings/https`; `POST /api/v1/settings/https/ports` opens the ports):

| Key | Default | |
| --- | --- | --- |
Expand Down Expand Up @@ -106,6 +107,6 @@ docker compose -f docker-compose.prod.yml -f docker-compose.https.yml up -d

## Refine later

- A button in Settings that opens the ports through the upgrade agent, so step 2 needs no terminal.
- The install wizard asking for the domain.
- Closing the ports again from Settings (`LENS_HTTPS=0` on the install line does it now).
- The install wizard asking for the domain (left out on purpose: it asks only what's needed to start).
- Caddy in the image, for machines that can't download it.
27 changes: 26 additions & 1 deletion fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
telemetry,
tika,
tunnel,
updates,
)
from app.schemas.admin import (
AuditEntry,
Expand All @@ -54,6 +55,7 @@
FileStoreTestResult,
FileStoreTry,
Health,
HttpsPorts,
HttpsProvider,
HttpsStatus,
IndexQueued,
Expand Down Expand Up @@ -217,7 +219,30 @@ def tunnel_status(user: AdminReader, cfg: Cfg, db: Db) -> TunnelStatus:
def https_status(user: AdminReader, cfg: Cfg, db: Db) -> HttpsStatus:
"""How HTTPS (Settings › HTTPS) is doing: whether a process has the HTTPS ports and runs Caddy, the certificates it
holds and when they run out, and Caddy's last lines."""
return HttpsStatus(**https.status(db, cfg))
return _https_status(db, cfg)


def _https_status(db, cfg) -> HttpsStatus:
run = updates.run_state(db)
return HttpsStatus(
**https.status(db, cfg),
agent=updates.agent(),
manual=updates.manual_https(cfg, True),
reconfigure=run if run and run.get("https") is not None else None,
)


@router.post("/settings/https/ports", status_code=202)
def https_ports(user: AdminWriter, cfg: Cfg, db: Db, body: HttpsPorts | None = None) -> HttpsStatus:
"""Open (or close) the HTTPS ports: this machine's upgrade agent runs install.sh again for the version running,
with LENS_HTTPS=1 (or 0), and Lens restarts. Without an agent, `manual` 8000 is the line to run there instead."""
body = body or HttpsPorts()
try:
updates.reconfigure(db, cfg, body.open, by=user.email)
except updates.UpdateError as e:
raise HTTPException(409, str(e)) from None
auth.audit(db, user.as_audit(), "https.ports", None, {"open": body.open})
return _https_status(db, cfg)


@router.get("/settings/https/providers")
Expand Down
26 changes: 25 additions & 1 deletion fastapi_backend/app/domain/updates.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,29 @@ def upgrade(db, cfg, ref=None, by=None, backup=True, wait=False):
return run_state(db)


def reconfigure(db, cfg, https, by=None, wait=False):
"""Have this machine run install.sh again for the version it runs, with the HTTPS ports opened (https=True) or
closed (docs/https.md). Lens restarts; nothing else changes. Returns the run."""
if not agent():
raise UpdateError("this machine has no upgrade agent; run the line shown instead")
run = run_state(db) or {}
if run.get("state") in ACTIVE and not run.get("error"):
raise UpdateError("an upgrade is already under way; try again when it's done")
run = {"id": secrets.token_hex(8), "ref": "", "https": bool(https), "by": by, "state": "queued", "asked_at": _ts(), "error": None}
_put(db, upgrade=run)
t = threading.Thread(target=_hand_over, args=(db, run, False), daemon=True, name="reconfigure")
t.start()
if wait:
t.join()
return run_state(db)


def manual_https(cfg, https):
"""The line that opens (or closes) the HTTPS ports by hand, on the machine Lens runs on."""
line = manual(cfg)
return f"{line[: -len(' sh')]} LENS_HTTPS={1 if https else 0} sh"


def _hand_over(db, run, backup):
try:
if backup:
Expand All @@ -256,7 +279,8 @@ def _hand_over(db, run, backup):
run["backup"] = dst.name
inbox = control_dir() / "inbox"
part = inbox / f".upgrade.{run['id']}"
part.write_text(f"id={run['id']}\nref={run['ref']}\nby={run.get('by') or ''}\nat={_ts()}\n", encoding="utf-8")
extra = f"https={1 if run['https'] else 0}\n" if run.get("https") is not None else ""
part.write_text(f"id={run['id']}\nref={run['ref']}\n{extra}by={run.get('by') or ''}\nat={_ts()}\n", encoding="utf-8")
part.replace(inbox / "upgrade")
run.update(state="queued", queued_at=_ts())
except Exception as e: # noqa: BLE001 - the admin reads why
Expand Down
8 changes: 8 additions & 0 deletions fastapi_backend/app/schemas/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from pydantic import Field

from app.schemas.common import Ok, RequestModel, ResponseModel
from app.schemas.updates import UpgradeRun


class Started(Ok):
Expand Down Expand Up @@ -101,6 +102,13 @@ class HttpsStatus(ResponseModel):
log: list[str] = Field(default_factory=list, description="Caddy's last lines")
origin: str = Field(description="where Caddy sends visitors: the web app, as the server reaches it")
process: str | None = Field(None, description="the server process with the HTTPS ports")
agent: str | None = Field(None, description="what can open the ports from here (the upgrade agent: systemd, launchd)")
manual: str = Field(description="the line that opens the ports by hand, on the machine Lens runs on")
reconfigure: UpgradeRun | None = Field(None, description="the last run that opened or closed the ports (as in /admin/updates)")


class HttpsPorts(RequestModel):
open: bool = Field(True, description="open the HTTPS ports (true) or close them")


class HttpsProviderField(ResponseModel):
Expand Down
3 changes: 2 additions & 1 deletion fastapi_backend/app/schemas/updates.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ class Release(ResponseModel):

class UpgradeRun(ResponseModel):
id: str
ref: str = Field(description="the release tag (or branch) being installed")
ref: str = Field(description="the release tag (or branch) being installed; empty: the one running, set up again")
https: bool | None = Field(None, description="the HTTPS ports opened (true) or closed (false) by this run (docs/https.md)")
state: Literal["backing-up", "queued", "running", "done", "failed"]
by: str | None = None
asked_at: str | None = None
Expand Down
62 changes: 62 additions & 0 deletions fastapi_backend/tests/api/test_https.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
import datetime as dt
import json
import pathlib
import shutil
import subprocess
import sys
import textwrap
import time
Expand Down Expand Up @@ -288,3 +290,63 @@ def test_passkeys_and_email_links_use_the_https_address(app, client, db, cfg, mo
db.next_id("settings") # the app's settings are read again
o = client.post("/api/v1/auth/passkey/options", headers={"x-forwarded-host": "lens.example.com", "x-forwarded-proto": "https"})
assert o.json()["options"]["rpId"] == "localhost"


@pytest.fixture
def control(tmp_path, monkeypatch):
"""The control folder install.sh shares with the containers, with an upgrade agent watching it."""
d = tmp_path / "control"
(d / "inbox").mkdir(parents=True)
(d / "agent").write_text("kind=systemd\n")
monkeypatch.setenv("LENS_CONTROL_DIR", str(d))
return d


def test_the_ports_are_opened_by_the_upgrade_agent(client, db, control, monkeypatch):
make_user(db, "root@x.io", "root password 1", admin=True)
a = login(client, "root@x.io", "root password 1")
s = client.get("/api/v1/settings/https/status", headers=a).json()
assert s["agent"] == "systemd" and "LENS_HTTPS=1 sh" in s["manual"] and s["reconfigure"] is None
r = client.post("/api/v1/settings/https/ports", headers=a, json={"open": True})
assert r.status_code == 202, r.text
for _ in range(100): # handed over after the response
if (control / "inbox" / "upgrade").exists():
break
time.sleep(0.02)
req = dict(line.split("=", 1) for line in (control / "inbox" / "upgrade").read_text().splitlines())
assert req["ref"] == "" and req["https"] == "1" # the version running, set up again with the ports
run = client.get("/api/v1/settings/https/status", headers=a).json()["reconfigure"]
assert run["https"] is True and run["state"] == "queued" and run["id"] == req["id"]
assert client.post("/api/v1/settings/https/ports", headers=a, json={"open": False}).status_code == 409 # one at a time
monkeypatch.delenv("LENS_CONTROL_DIR")
r = client.post("/api/v1/settings/https/ports", headers=a, json={"open": True})
assert r.status_code == 409 and "run the line shown" in r.json()["detail"]


AGENT = pathlib.Path(__file__).resolve().parents[3] / "packaging" / "host" / "lens-upgrade.sh"


@pytest.mark.skipif(not shutil.which("git"), reason="needs git")
def test_the_agent_sets_up_the_version_running_with_the_ports(tmp_path):
"""packaging/host/lens-upgrade.sh, with an install.sh that writes down what it was given."""
d = tmp_path / "lens"
d.mkdir()
(d / "install.sh").write_text('echo "ref=$LENS_REF https=$LENS_HTTPS" > "$LENS_DIR/ran"\n')
(d / ".env").write_text("LENS_HTTPS=0\n")
git = ["git", "-C", str(d), "-c", "user.name=t", "-c", "user.email=t@x.io"]
subprocess.run([*git, "init", "-q", "-b", "main"], check=True)
subprocess.run([*git, "add", "."], check=True)
subprocess.run([*git, "commit", "-q", "-m", "x"], check=True)
(d / ".lens-control" / "inbox").mkdir(parents=True)

def run(request):
(d / ".lens-control" / "inbox" / "upgrade").write_text(request)
subprocess.run(["sh", str(AGENT), str(d)], check=True, capture_output=True)
return (d / "ran").read_text().strip(), (d / ".lens-control" / "upgrade-status").read_text()

ran, st = run("id=a1\nref=\nhttps=1\n")
assert ran == "ref=main https=1" and "state=done" in st
ran, _ = run("id=a2\nref=main\n") # an upgrade keeps what .env says
assert ran == "ref=main https=0"
ran, _ = run("id=a3\nref=\nhttps=yes; rm -rf /\n") # nothing but 0 or 1
assert ran == "ref=main https=0"
152 changes: 152 additions & 0 deletions nextjs-frontend/__tests__/httpsSettings.test.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { fireEvent, render, screen, waitFor } from "@testing-library/react";
import "@testing-library/jest-dom";

import { Admin } from "@/app/openapi-client";
import { crossErrors, type SettingsView } from "@/components/settings/model";
import { SettingsSection } from "@/components/settings/section";
import { TooltipProvider } from "@/components/ui/tooltip";

jest.mock("@/app/openapi-client", () => ({
Admin: { updateSettings: jest.fn(), httpsStatus: jest.fn(), httpsProviders: jest.fn(), httpsPorts: jest.fn() },
Metadata: {},
}));
jest.mock("next-auth/react", () => ({ useSession: () => ({ data: { accessToken: "t" } }) }));
jest.mock("next/navigation", () => ({ useRouter: () => ({ push: jest.fn() }) }));
jest.mock("@/lib/hooks/session", () => ({ useArchive: () => ({ namespaces: [] }) }));
jest.mock("@/components/ui/toast", () => ({ useToast: () => jest.fn() }));

const ok = (data: unknown) => Promise.resolve({ data, response: { ok: true, status: 200 } });
const STATUS = {
enabled: true,
running: false,
ports_open: true,
url: "https://lens.example.com",
certificates: [],
error: null,
log: [],
origin: "http://frontend:3000",
process: "worker-1",
agent: null,
manual:
"curl -fsSL https://raw.githubusercontent.com/adeelahmad/lense/main/install.sh | LENS_REF=main LENS_HTTPS=1 sh",
reconfigure: null,
};
const PROVIDERS = [
{ name: "cloudflare", label: "Cloudflare", fields: [{ name: "api_token", label: "API token", secret: true }] },
];
const OFF = {
enabled: false,
domains: [],
email: "",
ca: "both",
ca_url: "",
staging: false,
challenge: "http",
dns_provider: "",
dns_package: "",
dns_credentials: { set: false },
dns_resolvers: [],
propagation: "",
eab_key_id: "",
eab_hmac_key: { set: false },
redirect_http: true,
hsts: false,
origin: "",
https_port: 8443,
http_port: 8080,
};

function show(values: Record<string, unknown>) {
const view = { https: { values } } as unknown as SettingsView;
const qc = new QueryClient({ defaultOptions: { queries: { retry: false }, mutations: { retry: false } } });
return render(
<QueryClientProvider client={qc}>
<TooltipProvider>
<SettingsSection id="https" view={view} => undefined} />
</TooltipProvider>
</QueryClientProvider>,
);
}

describe("Settings › HTTPS", () => {
beforeEach(() => {
(Admin.updateSettings as jest.Mock).mockReset().mockImplementation(() => ok({ ok: true }));
(Admin.httpsStatus as jest.Mock).mockReset();
(Admin.httpsProviders as jest.Mock).mockReset().mockImplementation(() => ok(PROVIDERS));
(Admin.httpsPorts as jest.Mock).mockReset();
});

it("shows the address and the certificate once it has one", async () => {
(Admin.httpsStatus as jest.Mock).mockImplementation(() =>
ok({
...STATUS,
running: true,
certificates: [{ names: ["lens.example.com"], expires: "2027-01-08T00:00:00+00:00", issuer: "Let's Encrypt" }],
}),
);
show({ ...OFF, enabled: true, domains: ["lens.example.com"] });
expect(await screen.findByText("Serving over HTTPS.")).toBeInTheDocument();
expect(screen.getByRole("link", { name: "https://lens.example.com" })).toBeInTheDocument();
expect(screen.getByText(/Let's Encrypt · until/)).toBeInTheDocument();
});

it("gives the line to run when nothing has the ports and nothing can open them", async () => {
(Admin.httpsStatus as jest.Mock).mockImplementation(() => ok({ ...STATUS, ports_open: false }));
show({ ...OFF, enabled: true, domains: ["lens.example.com"] });
expect(await screen.findByText("No server process has the HTTPS ports yet.")).toBeInTheDocument();
expect(screen.getByText(/LENS_HTTPS=1 sh/)).toBeInTheDocument();
});

it("opens the ports through the upgrade agent", async () => {
(Admin.httpsStatus as jest.Mock).mockImplementation(() => ok({ ...STATUS, ports_open: false, agent: "systemd" }));
(Admin.httpsPorts as jest.Mock).mockImplementation(() =>
ok({
...STATUS,
ports_open: false,
agent: "systemd",
reconfigure: { id: "r1", ref: "", https: true, state: "queued", log: [] },
}),
);
show({ ...OFF, enabled: true, domains: ["lens.example.com"] });
fireEvent.click(await screen.findByRole("button", { name: "Open the HTTPS ports" }));
expect(await screen.findByText("Opening the HTTPS ports.")).toBeInTheDocument();
expect((Admin.httpsPorts as jest.Mock).mock.calls[0][0]).toMatchObject({ body: { open: true } });
});

it("shows why there's no certificate", async () => {
(Admin.httpsStatus as jest.Mock).mockImplementation(() =>
ok({ ...STATUS, running: true, error: "could not get certificate: no valid A records found", log: ["ERROR x"] }),
);
show({ ...OFF, enabled: true, domains: ["lens.example.com"] });
expect(await screen.findByText("could not get certificate: no valid A records found")).toBeInTheDocument();
expect(screen.getByText(/What Caddy said/)).toBeInTheDocument();
});

it("sets up the DNS challenge with Cloudflare", async () => {
(Admin.httpsStatus as jest.Mock).mockImplementation(() => ok({ ...STATUS, enabled: false }));
show(OFF);
expect(await screen.findByText("Off.")).toBeInTheDocument();
fireEvent.click(screen.getByText("DNS"));
await screen.findByRole("option", { name: "Cloudflare" });
fireEvent.change(screen.getByLabelText("DNS provider"), { target: { value: "cloudflare" } });
fireEvent.change(await screen.findByLabelText("API token"), { target: { value: "cf-secret" } });
fireEvent.click(screen.getByRole("button", { name: "Save" }));
await waitFor(() => expect(Admin.updateSettings).toHaveBeenCalledTimes(1));
expect((Admin.updateSettings as jest.Mock).mock.calls[0][0]).toMatchObject({
path: { section: "https" },
body: { challenge: "dns", dns_provider: "cloudflare", dns_credentials: '{"api_token":"cf-secret"}' },
});
});

it("checks the domains, the email and the authority's address", () => {
expect(crossErrors({ "https.domains": ["not a domain"] })["https.domains"]).toMatch(/domain/);
expect(crossErrors({ "https.domains": ["*.example.com"], "https.challenge": "http" })["https.domains"]).toMatch(
/DNS/,
);
expect(crossErrors({ "https.domains": ["*.example.com"], "https.challenge": "dns" })).toEqual({});
expect(crossErrors({ "https.enabled": true, "https.domains": [] })["https.domains"]).toMatch(/domain/);
expect(crossErrors({ "https.email": "nobody" })["https.email"]).toMatch(/email/);
expect(crossErrors({ "https.ca": "custom", "https.ca_url": "http://x" })["https.ca_url"]).toMatch(/https/);
});
});
Loading
Loading
0