8000 feat(pipelines): a Read step picks its own converter, such as anytopdf by adeelahmad · Pull Request #610 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 19 additions & 14 deletions DEVLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -472,29 +472,34 @@ authority records (variant names, external identifiers); topic history and undo
to accept analysis suggestions without asking, or to stop offering new topics; matching labels of three or more words
in transcripts (only summaries match them now).

## 2026-10-04 · anytopdf as the conversion engine (not started: wait for Adeel's go)
## 2026-10-04 · anytopdf as the conversion engine (started 2026-10-06; status 2026-10-18)

Goal (Adeel): use the sister project [anytopdf-rs](https://github.com/adeelahmad/anytopdf-rs) (README on its sprint2
branch) to turn any source into one searchable, cited PDF: OCR, captions, transcripts, keyframes and metadata as an
invisible text layer, with an embedded manifest, chunks, source anchors and provenance. Do not start until Adeel says go.
Goal (Adeel): use the sister project [anytopdf-rs](https://github.com/adeelahmad/anytopdf-rs) to turn any source into
one searchable, cited PDF: OCR, captions, transcripts, keyframes and metadata as an invisible text layer, with an
embedded manifest, chunks, source anchors and provenance.

Why it fits: one static Rust binary (Linux x86_64 and arm64 musl, so a Raspberry Pi) that Lens can download as a
component, instead of LibreOffice and Chromium in the full image; provenance per fact matches "trusted, cited memory".

Todo (when started):
Todo:

- [ ] Wait for anytopdf's JSON output (`--json`, schemas, exit codes) and `extract --json` manifest to land
- [ ] anytopdf as a component Lens installs itself (settings: auto, on, off), with its version recorded
- [ ] A pipeline / workflow node "Make evidence PDF": inputs a recording or resource, outputs the PDF as a rendition
and its manifest chunks and anchors as Lens chunks (citations jump to time span, box or byte range)
- [ ] Use it where Lens has no converter first (images, captions, mixed folders); keep LibreOffice and Chromium as
the default for Office, HTML and email until anytopdf reads them (its roadmap: PDF, HTML, EML, Office)
- [ ] Reuse Lens's own OCR and transcripts as sidecars rather than running them twice
- [ ] Run it with no network, size and time caps, plugins off unless an admin allows them (its plugins run with full
user rights until its sandbox lands)
- [x] anytopdf as a component Lens fetches itself, checked against its release checksum (`documents.converter` auto,
anytopdf, lens), or a conversion node (#210 #220 #224); release 0.4.0 pinned in anytopdf.py
- [x] Use it where Lens has no converter first; LibreOffice and Chromium stay the default for Office, HTML and email
on `auto` (#210); web pages and emails drawn as they look, with its OCR word boxes used as page text (#222 #223)
- [x] Run it with no network, no plugins and no config file for conversions; face and object plugins only sandboxed
(#221 #227 #228); camera RAW and HEIC photos (#233)
- [x] Evidence PDF of an asset or a collection, with Lens's transcripts handed over as sidecars (#226 #370)
- [x] Places, dates and photo GPS as graph hints, CLIP photo search (#229 #230)
- [x] A pipeline's Read step picks its own converter, so one pipeline can use anytopdf while the server's choice stays
- [ ] An evidence PDF as a pipeline step (today an endpoint and an Export menu item)
- [ ] Read anytopdf's structured output (manifest chunks and anchors as Lens chunks, citations jumping to a time span
or a box) once it ships a portable graph bundle
- [ ] Hand Lens's own OCR to anytopdf as a sidecar where Lens read a page first, so it isn't run twice

Refine later: anytopdf's intake channels (IMAP, webhooks, watched folders) and printing overlap Lens's sources and
sensors; decide which side owns them. MCP server mode could be an extension.

## 2026-10-04 · Notes: a page for everything

Goal (Adeel): every resource, entity and topic has its own page (its note), next to free notes written by people or
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[evidence PDFs](docs/api.md#evidence-pdfs), [published feeds](docs/feeds.md);
[checks](docs/requirements.md#learn-from-it))
- **Automate it.** Pipelines and workflows on a canvas (with Convert steps that make a video's soundtrack an audio
asset and a document's pages image assets), routines on a schedule, batch runs over a collection, and
asset and a document's pages image assets, and Read steps that pick their own converter, such as anytopdf), routines on a schedule, batch runs over a collection, and
notifications to webhooks, Slack, Discord and Matterbridge, including when a routine's run fails or leaves graph
changes to review. Everything that happens in Lens (new assets, emails,
calendar events, files, feed posts, what people say to Lens, sensor readings, notes, graph changes, runs, budget
Expand Down
5 changes: 5 additions & 0 deletions changelog.d/read-step-converter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
- **A pipeline's Read step picks its own converter.** One pipeline can make its documents and photographed pages into
PDFs with anytopdf while the rest keep the server's choice in Settings → Documents: set **Make PDFs with** on its Read
step, or `converter` (`auto`, `anytopdf` or `lens`) in the step over the API and MCP. An image read again by Lens
itself no longer keeps pointing at a PDF an earlier anytopdf run made of it
([processing](docs/processing.md#choosing-the-converter-for-a-read-step)).
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ text (Word, text, Markdown) are imported as text. Settings → Documents:
| `documents.attachment_resources` | true | whether an email's attachments that Lens can read (documents, images, audio, video, emails) also become resources of their own; they're kept as its files either way |
| `documents.video_urls` | false | whether videos and podcasts can be imported by their address ([API](api.md#videos-and-podcasts)): their media is downloaded with yt-dlp, which Lens fetches on first use |
| `documents.web_networks` | `[]` | private networks (CIDR, such as `192.168.1.0/24`) that web pages, calendar and news feeds and videos may be fetched from besides the public internet: for an intranet, or a calendar server at home. The web app can't list this machine (loopback) or link-local addresses, where clouds answer with the server's own credentials; archive.yaml or `LENS_WEB_NETWORKS` in `.env` (comma-separated, added to archive.yaml's) can. Those stay whatever Settings saves: the networks listed in Settings are added to them |
| `documents.converter` | `auto` | who makes the PDFs: `auto` uses [anytopdf](#anytopdf) only for what LibreOffice and Chromium can't do here, `anytopdf` uses it for every document and image (and fetches it), `lens` never uses it |
| `documents.converter` | `auto` | who makes the PDFs: `auto` uses [anytopdf](#anytopdf) only for what LibreOffice and Chromium can't do here, `anytopdf` uses it for every document and image (and fetches it), `lens` never uses it; a pipeline's Read step can choose its own ([processing](processing.md#choosing-the-converter-for-a-read-step)) |
| `documents.anytopdf_url` | none | a conversion node: another machine running `anytopdf queue serve` and `anytopdf queue work`, such as `https://convert.home:8640`. Set, it converts instead of the program here |
| `documents.anytopdf_token` | none | the node's bearer token (its `ANYTOPDF_QUEUE_TOKEN`), kept encrypted |
| `documents.tika_url` | none (off) | an [Apache Tika](#apache-tika) server, such as `http://tika:9998` for the compose files' `tika` service |
Expand Down
10 changes: 10 additions & 0 deletions docs/processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -739,6 +739,16 @@ New pipelines, workflows, templates and routines don't need a name typed: the we
steps, what a workflow runs on, a template's kind, a routine's schedule and actions), numbered when it's taken
("Prompt template 2"). The pencil next to the name on its page renames it later, as does `PATCH` on its address.

### Choosing the converter for a Read step

A Read step makes an Office file, email, web page or photographed page into a PDF before it reads it. Which program
does that is the server's choice, `documents.converter` in Settings → Documents ([anytopdf](configuration.md#anytopdf)),
unless the step names its own: `{"type": "read", "converter": "anytopdf"}` makes every document and image in that
pipeline into a PDF with anytopdf (flattening a photographed page first), `"lens"` uses Lens's own converters only, and
`"auto"` uses anytopdf only for what Lens can't convert. Other pipelines keep the server's choice. In the pipeline
editor it's the Read step's **Make PDFs with** setting. Where a step asks for anytopdf but it isn't installed here and
no conversion node is set, Lens converts the file itself and the run's log says so.

### Converting assets

A Convert step makes new assets from the one it runs on. Default pipelines have none; add one in the pipeline editor
Expand Down
1 change: 1 addition & 0 deletions docs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ request, to another check that proves the same thing; loosening a requirement or
|---|---|
| Pipelines and workflows on a canvas | [test_workflows.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_workflows.py) `test_workflows_run_in_pipelines_drawn_as_graphs`; [workflow-canvas.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-canvas.test.ts); [journeys.mjs](https://github.com/adeelahmad/lense/blob/main/ci/clicks/journeys.mjs) `pipeline-create`, `workflow-create` |
| Convert steps make new assets: a video's soundtrack as audio, a document's pages as images, each run through its own pipeline | [test_conversions.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_conversions.py) `test_a_videos_soundtrack_becomes_an_audio_asset`, `test_a_documents_pages_become_image_assets`, `test_settings_are_checked`; [pipeline-convert.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/pipeline-convert.test.ts) `is added from the library ready to run: to audio unless images are picked` |
| A pipeline's Read step picks its own converter (auto, anytopdf or Lens's own) in place of the server's, checked as it's saved | [test_anytopdf.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_anytopdf.py) `test_a_read_step_picks_its_own_converter`; [pipeline-convert.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/pipeline-convert.test.ts) `is kept only on a read step, and said in its summary` |
| A Classify node in workflows chooses from labels, content types or topics, and acting on a sure answer keeps something | [test_classify.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_classify.py) `test_a_workflow_classifies`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `chooses from labels, content types or topics, and acting on a sure answer keeps something` |
| An Entity filter node keeps a list's entities by type, stop list, length, pattern or how often they're said | [test_flow.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_flow.py) `test_entity_filter_keeps_the_entities_asked_for`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `an entity filter says what it keeps and refuses lowercase types` |
| An HTTP call node calls a web address through the web tools' guard, with its host written out, its credentials sealed to their host, and a try that only reads | [test_http_nodes.py](https://github.com/adeelahmad/lense/blob/main/fastapi_backend/tests/api/test_http_nodes.py) `test_it_calls_the_address_and_passes_the_answer_on`, `test_where_it_goes_is_written_out_and_guarded`, `test_a_try_only_reads`; [workflow-flow.test.ts](https://github.com/adeelahmad/lense/blob/main/nextjs-frontend/__tests__/workflow-flow.test.ts) `an HTTP call node's host is written out` |
Expand Down
2 changes: 2 additions & 0 deletions fastapi_backend/app/domain/documents.py
Original file line number Diff line number Diff line change
Expand Up @@ -577,6 +577,8 @@ def transcribe(db, cfg, rid, say):
patch = {"status": "transcribed", "engine": how, "media": media, "transcribed_at": store.now()}
if learnt.get("by"):
patch["rendition"] = None if read else {"from": convert.ext_of(path), "by": learnt["by"]}
elif source == "image": # read from the picture itself: a PDF an earlier run made of it isn't where its pages are
patch["rendition"] = None
if source == "document":
patch["described"] = learnt.get("described") or None # what it says about itself (Tika), else nothing kept
patch.update(_named(rec, path, learnt))
Expand Down
18 changes: 17 addi 8 tions &a 10BC8 mp; 1 deletion fastapi_backend/app/domain/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,23 @@ def _read(db, cfg, rid, say, spec=None):
raise Skip("only documents and images are read; audio and video are transcribed")
from . import documents

return documents.transcribe(db, cfg, rid, say)
return documents.transcribe(db, _converter(cfg, spec, say), rid, say)


def _converter(cfg, spec, say):
"""The settings a read step runs with: its own `converter` (auto, anytopdf or lens) in place of documents.converter,
so one pipeline can make its PDFs with anytopdf while the rest keep the server's choice."""
how = (spec or {}).get("converter")
if how is None:
return cfg
if how not in pipelines.CONVERTERS:
raise ValueError(f"read step: the converter is one of {', '.join(pipelines.CONVERTERS)}")
if how == "anytopdf":
from . import anytopdf

if not anytopdf.available(cfg):
say("this step asks for anytopdf, but it isn't installed here and no conversion node is set: Lens converts it")
return {**cfg, "documents": {**(cfg.get("documents") or {}), "converter": how}}


def _transcribe(db, cfg, rid, say, spec=None):
Expand Down
6 changes: 5 additions & 1 deletion fastapi_backend/app/domain/pipelines.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,10 @@
"to",
"format",
"pipeline",
"converter",
}
"convert", "format": "convert", "pipeline": "convert"} # settings only one type of step has
"convert", "format": "convert", "pipeline": "convert", "converter": "read"} # settings only one type of step has
CONVERTERS = ("auto", "anytopdf", "lens") # what a read step makes PDFs with, as documents.converter says
NODE_RX = re.compile(r"^[A-Za-z0-9_-]{1,40}$")
WHEN = {"min_minutes": (int, float), "max_minutes": (int, float), "source": str, "languages": list}
KEY_RX = re.compile(r"^[a-z][a-z0-9_]{0,40}$")
Expand Down Expand Up @@ -111,6 +113,8 @@ def validate_steps(db, steps):
get(db, s["pipeline"])
except KeyError:
raise ValueError("convert step: choose a pipeline for the new assets") from None
if t == "read" and s.get("converter") is not None and s["converter"] not in CONVERTERS:
raise ValueError(f"read step: the converter is one of {', '.join(CONVERTERS)}")
need = {"llm": "prompt", "spec": "prompt", "reply": "prompt", "export": "export", "report": "report"}.get(t)
if need and (s.get("template") or t not in ("report", "spec", "reply")): # these have a built-in one
try:
Expand Down
35 changes: 35 additions & 0 deletions fastapi_backend/tests/api/test_anytopdf.py
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,41 @@ def test_with_anytopdf_chosen_it_reads_images_too(here, client, env, db, cfg, fo
assert any("read without anytopdf: anytopdf couldn't read it (error: it broke)" in x for x in log), log


def test_a_read_step_picks_its_own_converter(here, client, env, db, cfg):
"""A pipeline's read step can name its converter: one pipeline makes its PDFs with anytopdf (a photographed page
flattened) while the server's choice (auto here) stays as it is for the rest."""
he, ha = env["he"], env["ha"]
for bad in ({"type": "read", "converter": "pdf"}, {"type": "analyze", "converter": "anytopdf"}):
r = client.post("/api/v1/pipelines", headers=ha, json={"name": "x", "steps": [bad]})
assert r.status_code == 400, bad
up = _upload(client, he, _png(scan(["Galway harbour"])), "receipt.png")
drain(db, settings.effective(db, cfg))
assert _rec(db, up["recording"])["rendition"] is None # auto: an image Lens reads itself
assert fake_anytopdf.runs(here) == []

step = {"type": "read", "converter": "anytopdf"}
r = client.post("/api/v1/pipelines", headers=ha, json={"name": "Scans", "steps": [step]})
assert r.status_code == 200, r.text
pid = r.json()["id"]
assert client.get(f"/api/v1/pipelines/{pid}", headers=ha).json()["steps"] == [step]
r = client.post(f"/api/v1/pipelines/{pid}/run", headers=ha, json={"recording": up["recording"]})
assert r.status_code == 200, r.text
drain(db, settings.effective(db, cfg))
rec = _rec(db, up["recording"])
assert rec["rendition"] == {"from": ".png", "by": "anytopdf"}, rec.get("error")
assert "--scan-mode" in fake_anytopdf.runs(here)[-1]["args"]
assert client.get("/api/v1/settings", headers=ha).json()["documents"]["values"]["converter"] == "auto"

# asked for where anytopdf isn't: Lens reads it, and the run says why
cfg["documents"]["anytopdf"] = None
r = client.post("/api/v1/jobs", headers=ha, json={"recordings": [up["recording"]], "steps": [step]})
assert r.status_code == 200, r.text
drain(db, settings.effective(db, cfg))
assert _rec(db, up["recording"])["rendition"] is None
log = client.get(f"/api/v1/jobs/{r.json()['jobs'][0]}/log", headers=he).json()["lines"]
assert any("asks for anytopdf, but it isn't installed here" in x for x in log), log


def test_camera_raw_photos_are_read_by_anytopdf(here, client, env, db, cfg):
"""A RAW photo is one Lens can't open: anytopdf reads it, whatever documents.converter says but `lens`."""
from app.domain import content_types
Expand Down
10 changes: 10 additions & 0 deletions nextjs-frontend/__tests__/pipeline-convert.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,13 @@ describe("the convert step in a pipeline", () => {
expect(r.error).toBe("Convert needs Read pages’s output, so it has to come after it.");
});
});

describe("the read step's converter", () => {
it("is kept only on a read step, and said in its summary", () => {
expect(cleanSpec({ type: "read", converter: "anytopdf" })).toEqual({ type: "read", converter: "anytopdf" });
expect(cleanSpec({ type: "read" })).toBe("read");
expect(cleanSpec({ type: "analyze", converter: "anytopdf" })).toBe("analyze");
expect(stepSummary({ type: "read", converter: "anytopdf" })).toBe("with anytopdf");
expect(stepSummary({ type: "read" })).toBe("");
});
});
13 changes: 13 additions & 0 deletions nextjs-frontend/components/pipelines/pipeline-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,17 @@ export type StepSpec = {
to?: "audio" | "images";
format?: string;
pipeline?: number | "none";
/** A read step: what makes its PDFs, in place of the server's choice (Settings › Documents). */
converter?: "auto" | "anytopdf" | "lens";
};

/** What a read step can make its PDFs with; unset follows Settings › Documents. */
export const CONVERTERS: { value: NonNullable<StepSpec["converter"]>; label: string }[] = [
{ value: "auto", label: "anytopdf for what Lens can’t convert" },
{ value: "anytopdf", label: "anytopdf for every document and image" },
{ value: "lens", label: "Lens’s own converters only" },
];

/** The audio formats a convert step makes (the first is the default). */
export const AUDIO_FORMATS = ["m4a", "mp3", "opus", "flac"];

Expand Down Expand Up @@ -86,6 +95,7 @@ export function cleanSpec(s: StepSpec): StepSpec | string {
if (s.to === "audio" && s.format) out.format = s.format;
if (s.pipeline != null) out.pipeline = s.pipeline;
}
if (s.type === "read" && s.converter) out.converter = s.converter;
return Object.keys(out).length === 1 ? s.type : (out as StepSpec);
}

Expand Down Expand Up @@ -221,6 +231,9 @@ export function stepSummary(s: StepSpec, templateName?: (id: number) => string |
parts.push(s.to === "images" ? "pages to images" : `to audio · ${s.format ?? AUDIO_FORMATS[0]}`);
if (s.pipeline === "none") parts.push("not processed");
}
if (s.type === "read" && s.converter) {
parts.push({ auto: "anytopdf when needed", anytopdf: "with anytopdf", lens: "Lens converters" }[s.converter]);
}
if (s.model) parts.push(s.model);
if (s.force) parts.push("forced");
const w = whenText(s.when);
Expand Down
Loading
Loading
0