8000 feat(settings): private networks for web pages and feeds in Settings → Documents by adeelahmad · Pull Request #378 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -785,7 +785,7 @@ http(s), a user name or password in it, another port, or a host with an address
the page makes is checked again by a proxy inside Lens, which resolves each host itself and connects only to the
address it checked; redirects are followed to http and https addresses only, each checked. Chromium has no other way
out: it looks up no names itself, reaches no D-Bus, and WebRTC, which a page's script can start, may only go through
the proxy (never UDP straight to an address). `documents.web_networks` (startup) adds networks that may be reached too, for an
the proxy (never UDP straight to an address). `documents.web_networks` (Settings → Documents) adds networks that may be reached too, for an
intranet. Without Chromium on the server it answers 400 (`GET /uploads/limits` → `convert.web`). Audited as
`import.web`. The PDF is the resource's file (`GET /resources/{rid}/media`), named after the link
(`annual-report.pdf`) or the page's title (`harbour-news.pdf`).
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,7 @@ each the first time it's needed; without them, PDFs and images are read as befor
| `documents.convert_seconds` | 300 | how long making one PDF may take before its job fails, 10–3600 seconds |
| `documents.attachment_resources` | true | whether an email's attachments that Lens can read (documents, images, audio, video, emails) also become resources of their own; they're kept as its files either way |
| `documents.video_urls` | false | whether videos and podcasts can be imported by their address ([API](api.md#videos-and-podcasts)): their media is downloaded with yt-dlp, which Lens fetches on first use |
| `documents.web_networks` | `[]` | private networks (CIDR, such as `192.168.1.0/24`) that web pages, calendar feeds and videos may be fetched from besides the public internet: for an intranet, or a calendar server at home. The web app can't list this machine (loopback) or link-local addresses, where clouds answer with the server's own credentials; archive.yaml or `LENS_WEB_NETWORKS` in `.env` (comma-separated, added to archive.yaml's) can. Those stay whatever Settings saves: the networks listed in Settings are added to them |
| `documents.converter` | `auto` | who makes the PDFs: `auto` uses [anytopdf](#anytopdf) only for what LibreOffice and Chromium can't do here, `anytopdf` uses it for every document and image (and fetches it), `lens` never uses it |
| `documents.anytopdf_url` | none | a conversion node: another machine running `anytopdf queue serve` and `anytopdf queue work`, such as `https://convert.home:8640`. Set, it converts instead of the program here |
| `documents.anytopdf_token` | none | the node's bearer token (its `ANYTOPDF_QUEUE_TOKEN`), kept encrypted |
Expand All @@ -257,7 +258,6 @@ Set at startup only (the config file; the web app can't choose what the server r
| `documents.soffice` | `soffice` or `libreoffice` on PATH | LibreOffice |
| `documents.chromium` | the first of `chromium`, `chromium-browser`, `google-chrome`, `google-chrome-stable`, `chrome` on PATH | Chromium or Chrome (a headless shell works too); it also captures web pages ([API](api.md#web-pages)) |
| `documents.anytopdf` | the one Lens downloaded, else `anytopdf` on PATH | the anytopdf program |
| `documents.web_networks` | `[]` | networks (CIDR, such as `10.20.0.0/16`) that web pages and calendar feeds may be fetched from besides the public internet: for an intranet, or a calendar server at home; loopback and cloud metadata addresses stay out unless listed. `LENS_WEB_NETWORKS` in `.env` (comma-separated) adds to it, for Docker and the packages |

### Apache Tika

Expand Down
1 change: 0 additions & 1 deletion fastapi_backend/app/domain/convert.py
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,6 @@ def bootstrap(cfg):
"soffice": d.get("soffice") or (shutil.which("soffice") or shutil.which("libreoffice") or "not installed"),
"chromium": d.get("chromium") or (chromium(cfg) or "not installed"),
"anytopdf": anytopdf.binary(cfg) or "not installed",
"web_networks": [str(n) for n in d.get("web_networks") or []],
}


Expand Down
4 changes: 2 additions & 2 deletions fastapi_backend/app/domain/feeds.py
Original file line number Diff line number Diff line change
Expand Up @@ -416,8 +416,8 @@ def _download(cfg, src):
if "public address" in str(e):
host = urllib.parse.urlsplit(url).hostname
raise RuntimeError(
f"that calendar can't be fetched: {host} is on a private network. An admin can allow that network: "
"LENS_WEB_NETWORKS in .env (e.g. 192.168.1.0/24), or documents.web_networks in archive.yaml"
f"that calendar can't be fetched: {host} is on a private network. An admin can allow that network in "
"Settings → Documents (Private networks Lens may fetch from), e.g. 192.168.1.0/24"
) from None
raise RuntimeError(f"that calendar can't be fetched: {e}") from None
req = urllib.request.Request(url, headers={"User-Agent": "Lens calendar source", "Accept": "text/calendar, */*;q=0.5"})
Expand Down
42 changes: 42 additions & 0 deletions fastapi_backend/app/domain/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@
"tika_url",
"tika_text",
"tika_metadata",
"web_networks",
),
}
MAIL_SECURITY = ("ssl", "starttls", "none")
Expand Down Expand Up @@ -312,6 +313,9 @@ def effective(db, base, reveal=True):
continue
allowed = EDITABLE[sec]
cfg[sec] = store._merge(cfg[sec], {k: v for k, v in _data(row).items() if allowed is None or k in allowed})
if sec == "documents": # the web app adds networks to the server's own (archive.yaml, LENS_WEB_NETWORKS)
server = (base.get("documents") or {}).get("web_networks") or []
cfg[sec]["web_networks"] = list(dict.fromkeys([*server, *(cfg[sec].get("web_networks") or [])]))
for k, sealed in (row.get("sealed") or {}).items():
if not reveal:
cfg[sec][k] = {"secret": True, "set": True}
Expand Down Expand Up @@ -538,6 +542,8 @@ def _check(section, key, value, default):
if not (u and VIEWER_URL.match(value.strip()) and u.hostname and u.username is None and u.password is None):
raise ValueError("documents.tika_url is a Tika server's http(s) address with no password in it, such as http://tika:9998")
return value.strip().rstrip("/")
if (section, key) == ("documents", "web_networks"):
return _web_networks(value)
if (section, key) == ("documents", "anytopdf_token"):
if value in (None, ""):
return None
Expand Down Expand Up @@ -599,6 +605,38 @@ def _check(section, key, value, default):
return value


# What the web app can't open web pages, feeds and downloads to: this machine (loopback), and link-local addresses,
# where clouds answer with the server's own credentials (169.254.169.254). archive.yaml or LENS_WEB_NETWORKS, on the
# server, can still list them.
SERVER_ONLY_NETWORKS = [ipaddress.ip_network(n) for n in ("0.0.0.0/8", "127.0.0.0/8", "169.254.0.0/16", "::/127", "fe80::/10")]


def _network(value):
try:
return str(ipaddress.ip_network(str(value).strip(), strict=False))
except ValueError:
return str(value)


def _web_networks(value):
"""documents.web_networks from the web app: private networks pages and feeds may be fetched from."""
if not isinstance(value, list):
raise ValueError("documents.web_networks is a list of networks like 192.168.1.0/24")
out = []
for v in value:
try:
net = ipaddress.ip_network(str(v).strip(), strict=False)
except ValueError:
raise ValueError(f"documents.web_networks: {v} isn't a network like 192.168.1.0/24") from None
if any(net.version == bad.version and net.overlaps(bad) for bad in SERVER_ONLY_NETWORKS):
raise ValueError(
f"documents.web_networks: {net} reaches this server itself or a cloud's metadata address, which only "
"archive.yaml or LENS_WEB_NETWORKS on the server can allow"
)
out.append(str(net))
return list(dict.fromkeys(out))


LOCAL_LLM_RANGES = {"context": (512, 262144), "threads": (1, 512), "gpu_layers": (0, 999), "port": (1024, 65535)}


Expand Down Expand Up @@ -1370,6 +1408,10 @@ def save(db, base, section, changes, user=None):
if k not in defaults:
raise ValueError(f"unknown setting {section}.{k}")
before = data.get(k, (base.get(section) or {}).get(k))
if (section, k) == ("documents", "web_networks") and isinstance(v, list):
# the server's own networks are shown with the rest and stay whatever the web app sends: keep only the others
server = {_network(x) for x in (base.get("documents") or {}).get("web_networks") or []}
v = [x for x in v if _network(x) not in server]
data[k] = _check(section, k, v, defaults[k])
new_headers = isinstance((changes or {}).get("headers"), str) # the mask echoed back is no new value
if (section, k) == ("telemetry", "endpoint") and not new_headers and _origin(before) != _origin(data[k]):
Expand Down
2 changes: 1 addition & 1 deletion fastapi_backend/tests/api/test_mail_calendar_sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,7 @@ def health(url, password="feed password 1", **params):
# what isn't public is refused, unless an admin allowed its network
cfg["documents"]["web_networks"] = []
refused = health(feed, user="lens")
assert not refused["ok"] and "127.0.0.1 is on a private network" in refused["error"] and "LENS_WEB_NETWORKS" in refused["error"]
assert not refused["ok"] and "127.0.0.1 is on a private network" in refused["error"] and "Settings → Documents" in refused["error"]
cfg["documents"]["web_networks"] = ["127.0.0.0/8"]
assert health(feed, user="lens")["ok"]
# its password isn't sent on to another server it redirects to
Expand Down
48 changes: 48 additions & 0 deletions fastapi_backend/tests/api/test_webcapture.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import time

import pytest
from fastapi.testclient import TestClient

from app.domain import convert, jobs, netguard, store, webcapture
from tests import fake_chromium, fake_llm, fake_programs
Expand Down Expand Up @@ -121,6 +122,53 @@ def test_only_public_addresses_on_web_ports(cfg):
assert webcapture.placeholder("https://example.org/news/item?id=4") == "example.org/news/item?id=4"


def test_an_admin_allows_a_private_network_in_settings(app, client, db):
"""documents.web_networks is in Settings → Documents; this server (loopback) and the metadata address stay for
archive.yaml and LENS_WEB_NETWORKS on the server."""
make_user(db, "root@x.io", "root password 1", admin=True)
h = login(client, "root@x.io", "root password 1")
with pytest.raises(ValueError, match="isn't a public address"):
webcapture.check_url(app.state.settings.current(), "http://192.168.1.5/")
r = client.put("/api/v1/settings/documents", json={"web_networks": [" 192.168.1.9/24 ", "fd00::/8", "192.168.1.0/24"]}, headers=h)
assert r.status_code == 200, r.text
assert client.get("/api/v1/settings", headers=h).json()["documents"]["values"]["web_networks"] == ["192.168.1.0/24", "fd00::/8"]
assert webcapture.check_url(app.state.settings.current(), "http://192.168.1.5/") == "http://192.168.1.5/"
for bad, why in (
(["127.0.0.0/8"], "this server itself"),
(["169.254.169.254/32"], "metadata address"),
(["0.0.0.0/0"], "this server itself"),
(["::1/128"], "this server itself"),
(["fe80::/64"], "metadata address"),
(["192.168.1.0/33"], "isn't a network"),
("192.168.1.0/24", "a list of networks"),
):
r = client.put("/api/v1/settings/documents", json={"web_networks": bad}, headers=h)
assert r.status_code == 400 and why in r.json()["detail"], (bad, r.text)
assert app.state.settings.current()["documents"]["web_networks"] == ["192.168.1.0/24", "fd00::/8"]


def test_the_servers_own_networks_stay_whatever_the_web_app_saves(cfg, db):
"""Networks archive.yaml or LENS_WEB_NETWORKS list show in Settings with the rest, and the web app only adds to
them: one it can't allow (loopback) saves when it is echoed back, and leaving one out doesn't drop it."""
from app.main import create_app

cfg["documents"]["web_networks"] = ["127.0.0.0/8", "10.0.0.0/8"]
client = TestClient(create_app(cfg, db, background=False), base_url="http://127.0.0.1")
make_user(db, "root@x.io", "root password 1", admin=True)
h = login(client, "root@x.io", "root password 1")

def shown():
return client.get("/api/v1/settings", headers=h).json()["documents"]["values"]["web_networks"]

assert shown() == ["127.0.0.0/8", "10.0.0.0/8"]
r = client.put("/api/v1/settings/documents", json={"web_networks": ["127.0.0.0/8", "10.0.0.0/8", "192.168.1.0/24"]}, headers=h)
assert r.status_code == 200, r.text
assert shown() == ["127.0.0.0/8", "10.0.0.0/8", "192.168.1.0/24"]
r = client.put("/api/v1/settings/documents", json={"web_networks": []}, headers=h)
assert r.status_code == 200, r.text
assert shown() == ["127.0.0.0/8", "10.0.0.0/8"]


def test_file_names():
assert webcapture.file_name("https://example.org/files/Annual%20Report.PDF", "Annual") == "annual-report.pdf"
assert webcapture.file_name("https://example.org/news/", "Harbour news") == "harbour-news.pdf"
Expand Down
17 changes: 17 additions & 0 deletions nextjs-frontend/__tests__/settings-web-networks.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import { crossErrors, FIELDS, parse } from "@/components/settings/model";

const spec = FIELDS.find((f) => f.section === "documents" && f.key === "web_networks")!;

describe("private networks Lens may fetch from", () => {
it("is a list in Settings → Documents, one network per line", () => {
expect(spec.kind).toBe("lines");
expect(parse(spec, "192.168.1.0/24\n\nfd00::/8\n")).toEqual({ value: ["192.168.1.0/24", "fd00::/8"] });
});

it("says which line isn't a network", () => {
expect(crossErrors({ "documents.web_networks": ["192.168.1.0/24"] })["documents.web_networks"]).toBeUndefined();
expect(crossErrors({ "documents.web_networks": ["intranet"] })["documents.web_networks"]).toMatch(
/“intranet” isn’t a network/,
);
});
});
5 changes: 1 addition & 4 deletions nextjs-frontend/components/settings/bodies.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,7 @@ export function SectionBody({ ctx }: { ctx: BodyCtx }) {
<F ctx={ctx} id="documents.convert_seconds" />
<F ctx={ctx} id="documents.attachment_resources" />
<F ctx={ctx} id="documents.video_urls" />
<F ctx={ctx} id="documents.web_networks" />
<F ctx={ctx} id="documents.converter" />
<F ctx={ctx} id="documents.anytopdf_url" />
<F ctx={ctx} id="documents.anytopdf_token" />
Expand Down Expand Up @@ -2897,10 +2898,6 @@ function Startup({ view }: { view: SettingsView }) {
["rclone", b.rclone ?? "—"],
["LibreOffice", b.soffice ?? "—"],
["Chromium", b.chromium ?? "—"],
[
"Web capture",
b.web_networks?.length ? `public addresses and ${b.web_networks.join(" · ")}` : "public addresses only",
],
["YOLOX model", b.yolox_model ?? "—"],
["Watchable folders", b.local_roots?.length ? b.local_roots.join(" · ") : "none: local folders can’t be watched"],
];
Expand Down
13 changes: 11 additions & 2 deletions nextjs-frontend/components/settings/model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,6 @@ export type SettingsView = Record<string, SectionView> & {
chromium?: string;
/** The anytopdf program (documents.anytopdf), or "not installed". */
anytopdf?: string;
/** Networks web pages may be captured from besides the public internet (documents.web_networks). */
web_networks?: string[];
/** The YOLOX model the objects step uses, or "not found". */
yolox_model?: string;
} & Record<string, unknown>;
Expand Down Expand Up @@ -3096,6 +3094,14 @@ export const FIELDS: FieldSpec[] = [
kind: "switch",
hint: "Its title, authors, language and dates, as Dublin Core",
},
{
section: "documents",
key: "web_networks",
label: "Private networks Lens may fetch from",
kind: "lines",
mono: true,
hint: "One per line, like 192.168.1.0/24: web pages, calendar feeds and videos there can be imported. Public addresses always can, and networks set on the server stay",
},
// IIIF & metadata
{
section: "iiif",
Expand Down Expand Up @@ -3411,6 +3417,9 @@ export function crossErrors(values: Record<string, unknown>): Record<string, str
const badMNet = mNets?.find((x) => !/^[0-9a-f.:]+(\/\d{1,3})?$/i.test(x));
if (badMNet) e["smb.allowed_networks"] = `“${badMNet}” isn’t a network like 192.168.1.0/24`;
else if (mNets && mNets.length === 0) e["smb.allowed_networks"] = "List at least one network";
const wNets = values["documents.web_networks"] as string[] | undefined;
const badWNet = wNets?.find((x) => !/^[0-9a-f.:]+(\/\d{1,3})?$/i.test(x));
if (badWNet) e["documents.web_networks"] = `“${badWNet}” isn’t a network like 192.168.1.0/24`;
const sNets = values["sensors.syslog_networks"] as string[] | undefined;
const badSNet = sNets?.find((x) => !/^[0-9a-f.:]+(\/\d{1,3})?$/i.test(x));
if (badSNet) e["sensors.syslog_networks"] = `“${badSNet}” isn’t a network like 192.168.1.0/24`;
Expand Down
Loading
Loading
0