8000 feat(documents): make PDFs with anytopdf, here or on a conversion node by adeelahmad · Pull Request #210 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **anytopdf can make Lens's PDFs.** Settings › Documents › Make PDFs with anytopdf. On `auto` (the default) nothing
changes unless anytopdf is here: then it reads the emails, web pages and text a server without Chromium couldn't.
Choose "For every document and image" and Lens downloads anytopdf 0.3.0 (checked against its release checksum);
it then also reads images, flattening a photographed page before OCR. Or set a conversion node, another machine
running `anytopdf queue serve`, and a small server reads Word, PowerPoint and spreadsheet files too
([anytopdf](docs/configuration.md#anytopdf)).
- **Connect Slack, WhatsApp, Telegram, Discord and Matrix from the web app.** Start the optional `matterbridge`
service, then turn on Settings › Chat rooms › Run Matterbridge here and add a bot token and rooms. To link
WhatsApp, scan the QR code the page shows. Lens writes Matterbridge's config, a gateway per room, and restarts it
Expand Down
23 changes: 23 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,15 +205,38 @@ makes of them, or by LibreOffice where there's no Chromium. Outlook `.msg` email
| `documents.max_pages` | 2000 | the most pages of one document that are drawn and read, 1–50000 |
| `documents.convert_seconds` | 300 | how long making one PDF may take before its job fails, 10–3600 seconds |
| `documents.attachment_resources` | true | whether an email's attachments that Lens can read (documents, images, audio, video, emails) also become resources of their own; they're kept as its files either way |
| `documents.converter` | `auto` | who makes the PDFs: `auto` uses [anytopdf](#anytopdf) only for what LibreOffice and Chromium can't do here, `anytopdf` uses it for every document and image (and fetches it), `lens` never uses it |
| `documents.anytopdf_url` | none | a conversion node: another machine running `anytopdf queue serve` and `anytopdf queue work`, such as `https://convert.home:8640`. Set, it converts instead of the program here |
| `documents.anytopdf_token` | none | the node's bearer token (its `ANYTOPDF_QUEUE_TOKEN`), kept encrypted |

Set at startup only (the config file; the web app can't choose what the server runs):

| Setting | Default | |
|---|---|---|
| `documents.soffice` | `soffice` or `libreoffice` on PATH | LibreOffice |
| `documents.chromium` | the first of `chromium`, `chromium-browser`, `google-chrome`, `google-chrome-stable`, `chrome` on PATH | Chromium or Chrome (a headless shell works too); it also captures web pages ([API](api.md#web-pages)) |
| `documents.anytopdf` | the one Lens downloaded, else `anytopdf` on PATH | the anytopdf program |
| `documents.web_networks` | `[]` | networks (CIDR, such as `10.20.0.0/16`) that web pages and calendar feeds may be fetched from besides the public internet: for an intranet, or a calendar server at home; loopback and cloud metadata addresses stay out unless listed. `LENS_WEB_NETWORKS` in `.env` (comma-separated) adds to it, for Docker and the packages |

### anytopdf

[anytopdf](https://github.com/adeelahmad/anytopdf-rs) (MIT or Apache-2.0) is Lens's sister project: one static program
that makes documents, photos and media into searchable PDFs. Lens uses it as a converter, two ways:

- **Here.** With `documents.converter` set to `anytopdf`, Lens downloads release 0.3.0 for this machine on first use
(Linux x86-64 and arm64, macOS), checks it against the release's checksum and keeps only the program in
`data_dir/models/anytopdf-0.3.0/`. It makes the PDF of text, Markdown, web pages and emails from the same cleaned page
Lens would print with Chromium, so a server without Chromium reads them, and it reads images too: a photographed page
is found, straightened and flattened before OCR. Office files still need LibreOffice beside it.
- **On a conversion node.** With `documents.anytopdf_url` and its token set, every conversion goes to that machine as
a job (only the file, named `document.<type>`), and Lens fetches the PDF. The node has LibreOffice and whatever else
it needs, so a small server (a Raspberry Pi) reads Word, PowerPoint and spreadsheet files with nothing installed.
Run it behind TLS off loopback (`anytopdf queue serve --tls-cert … --tls-key …`).

It runs with no network, no runtime plugins and no config file of its own, and its PDF has no provenance page and
holds only the document's text, so what Lens reads is what the document says. Faces, objects and speech stay Lens's
own steps. The resource's rendition says `anytopdf` made it.

Neither may reach anything while converting: Chromium goes through a proxy inside Lens that serves the page and refuses
every other request (the page also allows no scripts), and LibreOffice is given a proxy address that isn't there.
Chromium runs with its sandbox where it can, and without it as root or where the container lacks what the sandbox
Expand Down
220 changes: 220 additions & 0 deletions fastapi_backend/app/domain/anytopdf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
"""anytopdf (github.com/adeelahmad/anytopdf-rs), the sister project, as a converter: documents and images made into
searchable PDFs that Lens then reads like any other (docs/api.md#documents-and-images).

It runs one of two ways, chosen in Settings → Documents:
- here: the anytopdf program, a single static binary that components.py downloads on first use (checked against its
release checksum) unless documents.anytopdf names one. It reads emails, web pages, text and photographed pages with no
Chromium, and flattens a photographed page before OCR; Office files still need LibreOffice beside it.
- on a conversion node: another machine running `anytopdf queue serve` and `anytopdf queue work` (documents.anytopdf_url,
with its bearer token, documents.anytopdf_token). The node has LibreOffice and whatever else it needs, so a small
server (a Raspberry Pi) reads Office files without installing anything.

documents.converter says when: `auto` (the default) uses it only for what Lens can't convert itself, `anytopdf` for
every document and image, `lens` never. Run here, it has no network, no runtime plugins (faces, objects and speech
stay Lens's own steps) and no config file of its own; its PDF has no provenance page, and its search layer holds the
document's text only, so what Lens reads from it is what the document says.
"""

from __future__ import annotations

import json
import os
import pathlib
import platform
import shutil
import tarfile
import tempfile
import time
import urllib.error
import urllib.parse
import urllib.request

from . import netguard

VERSION = "0.3.0"
RELEASES = "https://github.com/adeelahmad/anytopdf-rs/releases/download"
# the release archives Lens fetches, by (system, machine), with the sha256 its release published (SHA256SUMS)
ARCHIVES = {
("Linux", "x86_64"): ("x86_64-unknown-linux-musl", "2a789bf099cd14b5f68a147b44170d3f243299565b09b93b3a3329811a2492c0"),
("Linux", "aarch64"): ("aarch64-unknown-linux-musl", "9c6cc307a3e6dc7e7e44c390176ce62e7b77ebad640ea74c569272104a79f6dd"),
("Darwin", "x86_64"): ("x86_64-apple-darwin", "71b9170b75ac6b1783e91f15aaef9fba826d0bda5af7dbb6bbc0efcc9445f2bc"),
("Darwin", "arm64"): ("aarch64-apple-darwin", "29751b5a916a328cafdad52af19a6415a9b72c36ddc8784c8c11bbd936d435c3"),
}
MACHINES = {"amd64": "x86_64", "arm64": "arm64", "aarch64": "aarch64"}
# its search layer: the document's own text, not the dates, colours and places it would add
QUIET = ["--no-provenance-page", "--no-entities", "--colors", "off", "--location", "off"]
POLL_SECONDS = 1.0


class Unavailable(RuntimeError):
"""anytopdf isn't here, nor a conversion node set."""


def _opts(cfg):
return cfg.get("documents") or {}


def archive(system=None, machine=None):
"""This machine's release archive: (file name, sha256), or None where anytopdf has no build."""
system, machine = system or platform.system(), machine or platform.machine()
found = ARCHIVES.get((system, MACHINES.get(machine.lower(), machine)))
if system == "Darwin" and not found and machine.lower() == "aarch64":
found = ARCHIVES.get(("Darwin", "arm64"))
return (f"anytopdf-{VERSION}-{found[0]}.tar.gz", found[1]) if found else None


def fetched_path(cfg):
"""Where components.py keeps the downloaded program."""
return pathlib.Path(cfg["data_dir"]) / "models" / f"anytopdf-{VERSION}" / "anytopdf"


def binary(cfg):
"""The anytopdf program to run here: documents.anytopdf, else the one downloaded, else one on PATH."""
own = _opts(cfg).get("anytopdf")
if own:
return own if os.path.isfile(own) and os.access(own, os.X_OK) else None
got = fetched_path(cfg)
if got.is_file() and os.access(got, os.X_OK):
return str(got)
return shutil.which("anytopdf")


def node(cfg):
"""The conversion node's address, or None."""
url = (_opts(cfg).get("anytopdf_url") or "").strip().rstrip("/")
return url or None


def available(cfg):
"""Whether documents can be made into PDFs by anytopdf, and where: "node", "here" or None."""
if node(cfg):
return "node"
return "here" if binary(cfg) else None


def mode(cfg):
return _opts(cfg).get("converter") or "auto"


def fetch(cfg, say=None):
"""Download this machine's anytopdf release, check it against its checksum, and keep only the program."""
import hashlib

found = archive()
if not found:
raise Unavailable(f"anytopdf has no build for {platform.system()} {platform.machine()}")
name, sha = found
dest = fetched_path(cfg)
if dest.is_file():
return str(dest)
dest.parent.mkdir(parents=True, exist_ok=True)
if say:
say(f"downloading anytopdf {VERSION}")
with tempfile.TemporaryDirectory(dir=dest.parent) as tmp:
part, h = pathlib.Path(tmp) / name, hashlib.sha256()
with urllib.request.urlopen(f"{RELEASES}/v{VERSION}/{name}", timeout=60) as r, open(part, "wb") as f:
while chunk := r.read(1 << 20):
h.update(chunk)
f.write(chunk)
if h.hexdigest() != sha:
raise RuntimeError(f"{name} didn't match its checksum")
want = f"{name.removesuffix('.tar.gz')}/anytopdf"
with tarfile.open(part) as t:
member = t.getmember(want) # the program only: its plugins stay behind, and nothing else is unpacked
if not member.isfile():
raise RuntimeError(f"{want} in {name} isn't a file")
src = t.extractfile(member)
out = pathlib.Path(tmp) / "anytopdf"
with open(out, "wb") as f:
shutil.copyfileobj(src, f)
out.chmod(0o755)
out.replace(dest)
return str(dest)


def _args(cfg, scan=False):
"""What every conversion asks for: no OCR when Lens reads no text from pictures, a photographed page flattened."""
args = list(QUIET)
if (cfg.get("video") or {}).get("ocr_engine") == "none":
args += ["--ocr", "off"]
if scan:
args += ["--scan-mode", "auto"]
return args


def to_pdf(cfg, src, out, scan=False):
"""Make `out`, the PDF of the file at `src`, on the conversion node if one is set, else here. Returns who made it:
"anytopdf"."""
where = available(cfg)
if where == "node":
_remote(cfg, pathlib.Path(src), pathlib.Path(out))
elif where == "here":
_local(cfg, pathlib.Path(src), pathlib.Path(out), scan)
else:
raise Unavailable("anytopdf isn't installed here, and no conversion node is set (documents.anytopdf_url)")
return "anytopdf"


def _local(cfg, src, out, scan):
from . import convert

exe = binary(cfg)
seconds = _opts(cfg).get("convert_seconds") or 300
with tempfile.TemporaryDirectory(prefix="lens-anytopdf-") as tmp:
doc = pathlib.Path(tmp) / f"document{src.suffix.lower()}"
shutil.copyfile(src, doc)
made = pathlib.Path(tmp) / "out.pdf"
env = netguard.nowhere_env({**os.environ, "HOME": tmp, "ANYTOPDF_DATA_DIR": f"{tmp}/data", "TMPDIR": tmp})
env = {k: v for k, v in env.items() if not k.startswith("ANYTOPDF_") or k == "ANYTOPDF_DATA_DIR"}
argv = [exe, "--no-plugins", "--no-config", "convert", str(doc), "-o", str(made), *_args(cfg, scan)]
r = convert._run(argv, seconds, env=env, cwd=tmp)
if r.returncode != 0 or not made.is_file() or made.stat().st_size == 0:
said = convert._last_said(r.stderr or r.stdout)
raise ValueError(f"anytopdf couldn't read it ({said or f'exit {r.returncode}'})")
out.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(made), out)


def _call(cfg, method, path, body=None, query=None):
url = node(cfg) + path + (f"?{urllib.parse.urlencode(query)}" if query else "")
token = _opts(cfg).get("anytopdf_token") or ""
headers = {"Authorization": f"Bearer {token}"} if token else {}
if body is not None:
headers["Content-Length"] = str(len(body))
req = urllib.request.Request(url, data=body, method=method, headers=headers)
return urllib.request.urlopen(req, timeout=60)


def _said(e):
try:
return json.loads(e.read() or b"{}").get("error") or e.reason
except (ValueError, AttributeError):
return getattr(e, "reason", str(e))


def _remote(cfg, src, out):
"""The file sent to the conversion node as a job, waited for, and its PDF fetched."""
seconds = _opts(cfg).get("convert_seconds") or 300
try:
with _call(cfg, "POST", "/v1/jobs", src.read_bytes(), {"filename": f"document{src.suffix.lower()}"}) as r:
job = json.loads(r.read())
jid, end = job["job_id"], time.monotonic() + seconds
while job.get("state") not in ("succeeded", "failed"):
if time.monotonic() > end:
raise ValueError(f"the conversion node took longer than {seconds} s (documents.convert_seconds)")
time.sleep(POLL_SECONDS)
with _call(cfg, "GET", f"/v1/jobs/{urllib.parse.quote(jid)}") as r:
job = json.loads(r.read())
if job["state"] == "failed":
raise ValueError(f"the conversion node couldn't read it (exit {job.get('exit_code')})")
out.parent.mkdir(parents=True, exist_ok=True)
part = out.with_suffix(".part")
with _call(cfg, "GET", f"/v1/jobs/{urllib.parse.quote(jid)}/output") as r, open(part, "wb") as f:
shutil.copyfileobj(r, f)
part.replace(out)
except urllib.error.HTTPError as e:
if e.code == 401:
raise ValueError("the conversion node refused Lens's token (documents.anytopdf_token)") from None
raise ValueError(f"the conversion node said {e.code}: {_said(e)}") from None
except (urllib.error.URLError, OSError) as e:
raise ValueError(f"the conversion node can't be reached ({getattr(e, 'reason', e)})") from None
27 changes: 27 additions & 0 deletions fastapi_backend/app/domain/components.py
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,25 @@ def fetch(self, cfg, m, say):
fetch_file(cfg, "yolox_s.onnx", say)


class Anytopdf(Component):
"""The anytopdf program (anytopdf.py), fetched when Settings → Documents asks for it to run here."""

def needed(self, cfg, m):
from . import anytopdf

return anytopdf.mode(cfg) == "anytopdf" and not anytopdf.node(cfg) and bool(anytopdf.archive())

def present(self, cfg):
from . import anytopdf

return bool(anytopdf.binary(cfg))

def fetch(self, cfg, m, say):
from . import anytopdf

anytopdf.fetch(cfg, say)


class Msg(Component):
def needed(self, cfg, m):
return False # only when asked for (components.also): it's GPL-3.0
Expand Down Expand Up @@ -526,6 +545,14 @@ def detail(self, cfg):
check=_soffice,
hint="in the full Lens image (LENS_TARGET=full, the default)",
),
Anytopdf(
"anytopdf",
"anytopdf",
"makes documents and photographed pages into searchable PDFs (documents.converter)",
steps={"transcribe"},
size_mb=50,
license="MIT OR Apache-2.0",
),
PYTORCH,
SENSEVOICE,
WHISPER,
Expand Down
Loading
Loading
0