8000 feat(documents): harden the anytopdf conversion node and add Check it by adeelahmad · Pull Request #224 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version
it then also reads images, flattening a photographed page before OCR. Or set a conversion node, another machine
running `anytopdf queue serve`, and a small server reads Word, PowerPoint and spreadsheet files too
([anytopdf](docs/configuration.md#anytopdf)).
- **A conversion node gets the token and nothing more.** Lens refuses a node that redirects (so its token never
goes on to another server), caps what the node answers and refuses a PDF that isn't one. Settings › Documents ›
Check it says whether the node answers and takes the token ([anytopdf](docs/configuration.md#anytopdf)).
- **Web pages and emails keep their look when anytopdf makes the PDF.** Where Chromium and `pdftoppm` are installed,
anytopdf draws the page as page images, offline, with its text as the search layer. Without them it reads the text,
as before.
Expand Down
4 changes: 4 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,10 @@ that makes documents, photos and media into searchable PDFs. Lens uses it as a c
a job (only the file, named `document.<type>`), and Lens fetches the PDF. The node has LibreOffice and whatever else
it needs, so a small server (a Raspberry Pi) reads Word, PowerPoint and spreadsheet files with nothing installed.
Run it behind TLS off loopback (`anytopdf queue serve --tls-cert … --tls-key …`).
Lens sends the file and token only to the address set: an http(s) address with no user name or password, and a
node that answers with a redirect is refused, so the token never goes on to another server. A job's answer is read
to 1 MB at most and its PDF to 2 GB, and a PDF that doesn't start like one is refused. **Check it** in Settings →
Documents asks the node for a job with Lens's token and says whether it answered and took it.

It runs with no network, no runtime plugins and no config file of its own, and its PDF has no provenance page and
holds only the document's text, so what Lens reads is what the document says. Faces, objects and speech stay Lens's
Expand Down
10 changes: 10 additions & 0 deletions fastapi_backend/app/api/v1/routes/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from app.api.deps import AdminReader, AdminWriter, Cfg, Db, domain_errors
from app.core.middleware import host_name
from app.domain import (
anytopdf,
auth,
blobs,
bridge,
Expand All @@ -35,6 +36,7 @@
AuditEntry,
BridgeStatus,
BridgeTestResult,
ConversionNodeTestResult,
DecisionStatus,
DecisionTestResult,
EmbedTestResult,
Expand Down Expand Up @@ -183,6 +185,14 @@ def test_bridge(user: AdminWriter, cfg: Cfg, db: Db) -> BridgeTestResult:
return BridgeTestResult(ok=error is None, error=error)


@router.post("/settings/documents/node/test")
def test_conversion_node(user: AdminWriter, cfg: Cfg) -> ConversionNodeTestResult:
"""Check that the conversion node (documents.anytopdf_url) answers at its address and takes Lens's token."""
t0 = time.time()
error = anytopdf.check(cfg)
return ConversionNodeTestResult(ok=error is None, error=error, ms=None if error else int((time.time() - t0) * 1000))


@router.get("/settings/decisions/status")
def decision_status(user: AdminReader, cfg: Cfg) -> DecisionStatus:
"""Who takes routine decisions, and whether a local Laya model can here: on this machine (Apple Silicon, once
Expand Down
96 changes: 88 additions & 8 deletions fastapi_backend/app/domain/anytopdf.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
stay Lens's own steps) and no config file of its own; its PDF has no provenance page, and its search layer holds the
document's text only, so what Lens reads from it is what the document says.

A conversion node gets the document and its token at the address the admin set and nowhere else: only http(s), no
user name or password in the address, and no redirect is followed, so the token never goes on to another server.
What it answers is bounded (MAX_JSON for a job, MAX_PDF for its PDF, which must start like one) before Lens reads it.

analyze() asks its face and object plugins, kept beside the program, what is in a picture: sandboxed, with no network,
reading only the picture and the models Lens names.
"""
Expand All @@ -26,6 +30,7 @@
import os
import pathlib
import platform
import re
import shutil
import tarfile
import tempfile
Expand All @@ -49,6 +54,9 @@
# its search layer: the document's own text, not the dates, colours and places it would add
QUIET = ["--no-provenance-page", "--no-entities", "--colors", "off", "--location", "off"]
POLL_SECONDS = 1.0
MAX_JSON = 1 << 20 # the most of a conversion node's job answer that is read
MAX_PDF = 2 << 30 # the largest PDF taken from a conversion node
JOB_ID = re.compile(r"[A-Za-z0-9_.:-]{1,128}")
# the plugins Lens keeps beside the program, for analyze(): the rest of the release's plugins stay behind
PLUGINS = ("anytopdf-plugin-faces", "anytopdf-plugin-face-id", "anytopdf-plugin-objects")

Expand Down Expand Up @@ -207,49 +215,121 @@ def _local(cfg, src, out, scan, browser=None):
shutil.move(str(made), out)


class _NoRedirect(urllib.request.HTTPRedirectHandler):
"""A conversion node that answers with a redirect is refused: the token goes to the address set, never on."""

def redirect_request(self, req, fp, code, msg, headers, newurl):
raise urllib.error.HTTPError(req.full_url, code, "it redirected elsewhere, and Lens doesn't follow", headers, fp)


_OPENER = urllib.request.build_opener(_NoRedirect)


def _url(cfg):
"""The node's address, refused unless it is http(s) with a host and no user name or password."""
url = node(cfg)
u = urllib.parse.urlsplit(url or "")
if u.scheme not in ("http", "https") or not u.hostname or u.username is not None or u.password is not None:
raise ValueError("documents.anytopdf_url is a conversion node's http(s) address, such as https://convert.home:8640")
return url


def _call(cfg, method, path, body=None, query=None):
url = node(cfg) + path + (f"?{urllib.parse.urlencode(query)}" if query else "")
url = _url(cfg) + path + (f"?{urllib.parse.urlencode(query)}" if query else "")
token = _opts(cfg).get("anytopdf_token") or ""
headers = {"Authorization": f"Bearer {token}"} if token else {}
if body is not None:
headers["Content-Length"] = str(len(body))
req = urllib.request.Request(url, data=body, method=method, headers=headers)
return urllib.request.urlopen(req, timeout=60)
return _OPENER.open(req, timeout=60)


def _job(r):
"""A job the node answered with, read no further than MAX_JSON, with an id that is only an id."""
raw = r.read(MAX_JSON + 1)
if len(raw) > MAX_JSON:
raise ValueError("the conversion node's answer was too long")
try:
job = json.loads(raw)
except ValueError:
raise ValueError("the conversion node didn't answer like anytopdf queue serve") from None
if not isinstance(job, dict) or not JOB_ID.fullmatch(str(job.get("job_id") or "")):
raise ValueError("the conversion node didn't answer like anytopdf queue serve")
return job


def _save_pdf(r, part):
"""The node's PDF, written to `part`: refused when it doesn't start like a PDF or runs past MAX_PDF."""
size, head = 0, b""
with open(part, "wb") as f:
while chunk := r.read(1 << 20):
if not size:
head = chunk[:5]
size += len(chunk)
if size > MAX_PDF:
raise ValueError(f"the conversion node's PDF ran past {MAX_PDF >> 30} GB")
f.write(chunk)
if head != b"%PDF-":
raise ValueError("the conversion node sent something that isn't a PDF")


def _said(e):
if 300 <= e.code < 400:
return e.reason
try:
return json.loads(e.read() or b"{}").get("error") or e.reason
return str(json.loads(e.read(MAX_JSON) or b"{}").get("error") or e.reason)[:300]
except (ValueError, AttributeError):
return getattr(e, "reason", str(e))


def _remote(cfg, src, out):
"""The file sent to the conversion node as a job, waited for, and its PDF fetched."""
seconds = _opts(cfg).get("convert_seconds") or 300
part = out.with_suffix(".part")
try:
with _call(cfg, "POST", "/v1/jobs", src.read_bytes(), {"filename": f"document{src.suffix.lower()}"}) as r:
job = json.loads(r.read())
job = _job(r)
jid, end = job["job_id"], time.monotonic() + seconds
while job.get("state") not in ("succeeded", "failed"):
if time.monotonic() > end:
raise ValueError(f"the conversion node took longer than {seconds} s (documents.convert_seconds)")
time.sleep(POLL_SECONDS)
with _call(cfg, "GET", f"/v1/jobs/{urllib.parse.quote(jid)}") as r:
job = json.loads(r.read())
job = _job(r)
if job["state"] == "failed":
raise ValueError(f"the conversion node couldn't read it (exit {job.get('exit_code')})")
out.parent.mkdir(parents=True, exist_ok=True)
part = out.with_suffix(".part")
with _call(cfg, "GET", f"/v1/jobs/{urllib.parse.quote(jid)}/output") as r, open(part, "wb") as f:
shutil.copyfileobj(r, f)
with _call(cfg, "GET", f"/v1/jobs/{urllib.parse.quote(jid)}/output") as r:
_save_pdf(r, part)
part.replace(out)
except urllib.error.HTTPError as e:
if e.code == 401:
raise ValueError("the conversion node refused Lens's token (documents.anytopdf_token)") from None
raise ValueError(f"the conversion node said {e.code}: {_said(e)}") from None
except (urllib.error.URLError, OSError) as e:
raise ValueError(f"the conversion node can't be reached ({getattr(e, 'reason', e)})") from None
finally:
part.unlink(missing_ok=True)


def check(cfg):
"""Whether the conversion node answers at its address and takes Lens's token: None when it does, else what's
wrong. It asks for a job that isn't there, which a node that took the token answers with 404."""
if not node(cfg):
return "set the conversion node's address first (documents.anytopdf_url)"
try:
with _call(cfg, "GET", "/v1/jobs/lens-check"):
return None
except urllib.error.HTTPError as e:
if e.code == 404:
return None
if e.code == 401:
return "it refused Lens's token (documents.anytopdf_token)"
return f"it said {e.code}: {_said(e)}"
except ValueError as e:
return str(e)
except (urllib.error.URLError, OSError) as e:
return f"it can't be reached ({getattr(e, 'reason', e)})"


def analyze(cfg, picture, models=None, seconds=None):
Expand Down
6 changes: 6 additions & 0 deletions fastapi_backend/app/schemas/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,12 @@ class BridgeTestResult(ResponseModel):
error: str | None = None


class ConversionNodeTestResult(ResponseModel):
ok: bool
error: str | None = None
ms: int | None = None


class EncryptionProgress(ResponseModel):
"""Converting the files already kept after encryption.files changed: `to` "encrypted" or "plain" (vaults stay
encrypted), how many changed and how many were skipped (a locked vault, a damaged file), and why a run stopped
Expand Down
93 changes: 93 additions & 0 deletions fastapi_backend/tests/api/test_anytopdf.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,99 @@ def test_a_conversion_node_converts_office_files_too(lean, client, env, db, cfg,
assert job["status"] == "failed" and "can't be reached" in job["error"]


def test_check_it_says_whether_the_node_answers_and_takes_the_token(client, env):
ha = env["ha"]
check = lambda: client.post("/api/v1/settings/documents/node/test", headers=ha).json() # noqa: E731
assert not check()["ok"] and "set the conversion node's address" in check()["error"]
srv, url = fake_anytopdf.node(TOKEN)
try:
assert client.put("/api/v1/settings/documents", headers=ha, json={"anytopdf_url": url, "anytopdf_token": TOKEN}).status_code == 200
got = check()
assert got["ok"] and got["error"] is None and got["ms"] >= 0, got
assert client.put("/api/v1/settings/documents", headers=ha, json={"anytopdf_token": "another token 0123456789"}).status_code == 200
got = check()
assert not got["ok"] and "refused Lens's token" in got["error"]
finally:
srv.shutdown()
srv.server_close()
got = check()
assert not got["ok"] and "can't be reached" in got["error"]


class _Hostile(fake_anytopdf.Node):
"""A conversion node that misbehaves the way `mode` says."""

mode = ""

def do_POST(self): # noqa: N802
if _Hostile.mode == "redirect":
self.send_response(307)
self.send_header("Location", f"http://127.0.0.1:{_Hostile.elsewhere}/v1/jobs")
self.send_header("Content-Length", "0")
self.end_headers()
return
if _Hostile.mode == "long":
return self._send(202, b"{" + b" " * (anytopdf.MAX_JSON + 10) + b"}")
if _Hostile.mode == "odd id":
return self._send(202, {"job_id": "../../admin", "state": "queued"})
super().do_POST()

def do_GET(self): # noqa: N802
if _Hostile.mode == "not a pdf" and self.path.endswith("/output") and self._authorized():
return self._send(200, b"<html>not a PDF</html>", "application/pdf")
super().do_GET()


def test_a_misbehaving_node_gets_nothing_more_and_gives_nothing_unchecked(cfg, tmp_path):
import http.server
import threading

seen = []

class Elsewhere(http.server.BaseHTTPRequestHandler):
def do_POST(self): # noqa: N802
seen.append(self.headers.get("Authorization"))
self.send_response(500)
self.end_headers()

def log_message(self, *a):
pass

other = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Elsewhere)
threading.Thread(target=other.serve_forever, daemon=True).start()
srv, url = fake_anytopdf.node(TOKEN)
srv.RequestHandlerClass = _Hostile
_Hostile.elsewhere = other.server_address[1]
cfg["documents"].update(anytopdf_url=url, anytopdf_token=TOKEN, convert_seconds=10)
src = tmp_path / "notes.txt"
src.write_text("Notes\n")
out = tmp_path / "out.pdf"
try:
for mode, said in (
("redirect", "doesn't follow"),
("long", "answer was too long"),
("odd id", "didn't answer like anytopdf queue serve"),
("not a pdf", "isn't a PDF"),
):
_Hostile.mode = mode
with pytest.raises(ValueError, match=said):
anytopdf.to_pdf(cfg, src, out)
assert not out.exists() and not out.with_suffix(".part").exists()
assert seen == [] # the token never went on to where the node pointed
_Hostile.mode = ""
anytopdf.to_pdf(cfg, src, out)
assert out.read_bytes().startswith(b"%PDF-")
cfg["documents"]["anytopdf_url"] = url.replace("http://", "http://user:pw@")
with pytest.raises(ValueError, match="http\\(s\\) address"):
anytopdf.to_pdf(cfg, src, out)
cfg["documents"]["anytopdf_url"] = "file:///etc"
assert "http(s) address" in anytopdf.check(cfg)
finally:
for s in (srv, other):
s.shutdown()
s.server_close()


def test_with_anytopdf_chosen_it_reads_images_too(here, client, env, db, cfg, folder):
he, ha = env["he"], env["ha"]
assert client.put("/api/v1/settings/documents", headers=ha, json={"converter": "anytopdf"}).status_code == 200
Expand Down
4 changes: 4 additions & 0 deletions nextjs-frontend/app/openapi-client/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,7 @@ export type {
ControlWorkerErrors,
ControlWorkerResponse,
ControlWorkerResponses,
ConversionNodeTestResult,
Converters,
CreateBatchData,
CreateBatchError,
Expand Down Expand Up @@ -2435,6 +2436,9 @@ export type {
TestBridgeData,
TestBridgeResponse,
TestBridgeResponses,
TestConversionNodeData,
TestConversionNodeResponse,
TestConversionNodeResponses,
TestDecisionsData,
TestDecisionsResponse,
TestDecisionsResponses,
Expand Down
16 changes: 16 additions & 0 deletions nextjs-frontend/app/openapi-client/sdk.gen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1176,6 +1176,8 @@ import type {
TelemetryStatusResponses,
TestBridgeData,
TestBridgeResponses,
TestConversionNodeData,
TestConversionNodeResponses,
TestDecisionsData,
TestDecisionsResponses,
TestEmbeddingsData,
Expand Down Expand Up @@ -2444,6 +2446,20 @@ export class Admin {
});
}

/**
* Test Conversion Node
*
* Check that the conversion node (documents.anytopdf_url) answers at its address and takes Lens's token.
*/
public static testConversionNode<ThrowOnError extends boolean = false>(
options?: Options<TestConversionNodeData, ThrowOnError>,
): RequestResult<TestConversionNodeResponses, unknown, ThrowOnError> {
return (options?.client ?? client).post<TestConversionNodeResponses, unknown, ThrowOnError>({
url: "/api/v1/settings/documents/node/test",
...options,
});
}

/**
* Decision Status
*
Expand Down
Loading
Loading
0