8000 feat(enhance): enhance low quality images before they're read by adeelahmad · Pull Request #479 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
c0ed7f5
feat(enhance): enhance low quality images before they're read
claude Oct 17, 2026
081d1b5
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
a0e0246
test(imports): a transcript's import skips enhance too
claude Oct 17, 2026
4365af5
docs(threat-model): the Real-ESRGAN model the enhance step can fetch
claude Oct 17, 2026
c5b013a
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
9934c3f
docs(configuration): rewrap the Settings paragraph
claude Oct 17, 2026
c9c1929
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
9497184
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
c022936
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
c38357d
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
597c131
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
3947972
docs(readme): enhancing images is built, not in progress
claude Oct 17, 2026
643e1d7
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
d90df50
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
94ef74e
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
a16794c
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
c211011
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
dbae17b
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
594ee38
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
12c511e
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
b7c9576
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 17, 2026
8e3ae8a
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 18, 2026
3cc0f74
docs(threat-model): list Real-ESRGAN among the dependencies
claude Oct 18, 2026
11d98f0
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 18, 2026
b118092
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 18, 2026
bc7a024
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 18, 2026
df70dca
docs(readme): answering about Lens itself is built, not in progress
claude Oct 18, 2026
9bb0ba9
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 18, 2026
7ef1d3d
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 18, 2026
356d170
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 18, 2026
924b85f
docs(readme): keep the enhance step in the built list after merging main
claude Oct 18, 2026
753a7b6
Merge main into enhance (changelog entry moves to changelog.d/479.md)
claude Oct 18, 2026
c58ec30
Merge remote-tracking branch 'origin/main' into claude/project-thread…
claude Oct 18, 2026
328b248
Merge main into enhance (README Understand-it paragraph)
claude Oct 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8000
2 changes: 0 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@

The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are versioned together.

## Unreleased

## 0.16.0 <small>October 18, 2026</small> {id="0.16.0"}

### Features
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,8 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
[the assistant in your browser](docs/browser-assistant.md); [checks](docs/requirements.md#clip-from-the-browser))
- **Understand it.** Transcription, which can expect the names your namespace knows (off by default), people
recognised by voice (long recordings go to a speech provider in pieces that fit its limits), text on screen and in scans, faces (off by default), NSFW pictures flagged by a model on your own
server (off by default, per namespace), entities and topics in a knowledge graph you can explore, query with Cypher or
server (off by default, per namespace), low quality images enhanced before they're read (off by default), entities
and topics in a knowledge graph you can explore, query with Cypher or
SPARQL, and see as it was at any version. Every entity has its own page at a wiki address
(`/wiki/<namespace>/<Name>`), and entities, topics and people are edited and merged from their lists, the graph or an
asset, with Undo instead of a confirm. Names get their real types (people, places, organisations, events, works) by
Expand All @@ -145,7 +146,7 @@ listed under [In progress](#in-progress), and what comes next under [Planned](#p
([Graph](docs/graph.md), [entity pages](docs/notes.md#entity-pages),
[similar content](docs/processing.md#similar-content), [graph history](docs/graph-history.md),
[topics](docs/topics.md), [video](docs/video.md), [NSFW classification](docs/nsfw.md),
[long recordings](docs/speech-providers.md#long-recordings),
[enhancing images](docs/enhance.md), [long recordings](docs/speech-providers.md#long-recordings),
[your own names](docs/processing.md#your-own-names-in-transcription),
[cleaning up transcripts](docs/processing.md#cleaning-up-transcripts), [dates](docs/processing.md#dates);
[checks](docs/requirements.md#understand-it))
Expand Down Expand Up @@ -258,7 +259,6 @@ still loading. ([Components](docs/components.md),
These are open pull requests, not yet part of Lens:

- **Chat:** a conversation exported as Markdown, PDF or JSON ([#474](https://github.com/adeelahmad/lense/pull/474)).
- **Pictures:** low-quality images enhanced before they're read ([#479](https://github.com/adeelahmad/lense/pull/479)).
- **Web clipper:** signing in, or using your API token, not only an import webhook's
([#406](https://github.com/adeelahmad/lense/pull/406)).
- **Automations:** more operators for conditions ([#434](https://github.com/adeelahmad/lense/pull/434)).
Expand Down Expand Up @@ -313,8 +313,8 @@ The guides are on the [docs site](https://adeelahmad.github.io/lense/) and in th
[the assistant by email](docs/assistant-email.md), [podcasts](docs/podcasts.md), [notes](docs/notes.md),
[web clipper](docs/web-clipper.md), [the assistant in your browser](docs/browser-assistant.md),
[sensors](docs/sensors.md), [processing](docs/processing.md), [video](docs/video.md),
[NSFW classification](docs/nsfw.md), [notifications](docs/notifications.md), [automations](docs/automations.md),
[timeline](docs/timeline.md)
[NSFW classification](docs/nsfw.md), [enhancing images](docs/enhance.md), [notifications](docs/notifications.md),
[automations](docs/automations.md), [timeline](docs/timeline.md)
- **Knowledge:** [graph and Cypher](docs/graph.md), [graph history](docs/graph-history.md), [topics](docs/topics.md),
[the graph as files](docs/file-tree.md), [NFS share](docs/file-shares.md), [SMB share](docs/smb.md),
[linked data (RDF)](docs/rdf.md), [IIIF](docs/iiif.md), [Fedora](docs/fedora.md)
Expand Down
7 changes: 7 additions & 0 deletions changelog.d/479.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
- **Enhance low quality images before they're read.** The new `enhance` step, first in the standard pipeline,
measures each image (size, sharpness, noise, contrast, and face size where the namespace finds faces) and keeps an
enhanced copy of a low quality one, which OCR, faces, objects and descriptions then read; the original stays as it
is. Off by default: Settings → Enhance. The light `classic` engine (Pillow) is the default; Real-ESRGAN general
x4v3 (BSD-3-Clause, 5 MB) is fetched the first time it's chosen. The Details tab says what was found, saves the
copy, and enhances an image on request; agents use `POST /api/v1/resources/{rid}/enhance`, the MCP tool `enhance`
or the `enhance` automation action ([Enhancing images](docs/enhance.md)).
13 changes: 13 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -574,6 +574,8 @@ PUT /api/v1/resources/{rid}/ip-groups/{gid}
DELETE /api/v1/resources/{rid}/ip-groups/{gid}
GET /api/v1/resources/{rid}/player
PUT /api/v1/resources/{rid}/nsfw
POST /api/v1/resources/{rid}/enhance
GET /api/v1/resources/{rid}/enhanced
GET /api/v1/resources/{rid}/embed-link
GET /api/v1/resources/{rid}/audio
GET /api/v1/resources/{rid}/wordcloud.svg
Expand Down Expand Up @@ -1397,6 +1399,17 @@ Audited as `recording.nsfw`. `GET /resources/nsfw` (`ns` for one namespace) says
you read, how many resources are `flagged`, and how many of those the Library leaves out (`hidden`). Search hits and
photo search results carry `nsfw: true` for the ones the web app blurs.

### Enhancing images

The `enhance` step comes first in the standard pipeline: where [enhancing images](enhance.md) is on (off by
default), it measures an image and, when it's small, blurry, noisy, dull or has small faces, keeps an enhanced copy
that `read` and the steps after it work from; the original stays as it is. Other resources skip it. A resource
carries `enhanced`: `{kind, low, measured, enhanced, engine, scale, width, height, forced, at, download}` (`download`
a signed link to the copy), or `null` until the step looked at it. `POST /resources/{rid}/enhance` (the process
permission) enhances an image whatever the settings and its quality and runs its pipeline's later steps again,
answering the `job`; 400 for anything but an image. Audited as `recording.enhance`. `GET /resources/{rid}/enhanced`
is the copy, for members of its namespace.

### Descriptions

The `describe` step has a model that can see images say in a few sentences what each page of a document or an image
Expand Down
1 change: 1 addition & 0 deletions docs/automations.md
Original file line number Diff line number Diff line change
Expand Up @@ -658,6 +658,7 @@ or removed does nothing. What it writes is by the automation itself, `automation
| `http_request` | sends a `method` (GET, POST or PUT) to a `url`, with `headers` and a `body` (text, or an object sent as JSON) | out | `{status, ok, body, json, content_type, truncated, method, url, headers}`: the answer, as outside text |
| `run_workflow` | runs a recording `workflow` (its published `version`, or the one named) on an `asset` as a job | lens | `{job, workflow, version, asset}` |
| `run_pipeline` | processes an `asset` again with a `pipeline` (or the one its namespace uses), keeping its outputs | lens | `{job, pipeline, asset}` |
| `enhance` | [enhances](enhance.md) an image `asset` whatever the settings, keeping the original, and reads it again from the copy | lens | `{job, asset}` |
| `run_routine` | asks the scheduler to run a `routine` now (`propose_only` to only propose its changes) | lens, so it asks | `{routine, requested, propose_only}` |
| `set_field` | sets one of an `asset`'s custom fields (`field`: its id or label) to `value` (empty clears it) | lens | `{asset, field, label, value, changed}` |
| `add_to_collection` | moves an `asset` into a `collection` of its namespace | lens | `{asset, collection, name, moved, newly_see}` |
Expand Down
3 changes: 2 additions & 1 deletion docs/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ never fetch a speech or voice model.
| NSFW model | [NSFW classification](nsfw.md) is on for the archive or a namespace (off by default) | the 8-bit ONNX export of AdamCodd/vit-base-nsfw-detector (Apache-2.0, 88 MB), and ONNX Runtime |
| Florence-2 | `llm.describe_engine` is `florence-2` | Florence-2-base-ft or -large-ft (MIT, `llm.florence_model`; about 280 or 830 MB), their 8-bit ONNX exports, the tokenizers library and ONNX Runtime, to [describe pictures](configuration.md#descriptions) on this machine |
| Moondream | `llm.describe_engine` is `moondream`, and this machine can run the chosen one | Moondream 3 Preview (BSL 1.1, about 19 GB, an NVIDIA GPU or a Mac with 32 GB) or Moondream 2 (Apache-2.0, about 3.7 GB), `llm.moondream_model`, at a pinned revision, with its tokenizer, PyTorch (the CPU build without an NVIDIA GPU) and transformers, to [describe pictures](configuration.md#descriptions) on this machine |
| Real-ESRGAN | [enhancing images](enhance.md) is on and `enhance.image_engine` is `realesrgan` (off by default) | Real-ESRGAN general x4v3 as an ONNX model (BSD-3-Clause, 5 MB), and ONNX Runtime |
| Prompt-injection classifier | `ai.injection_check` is `classifier` | ProtectAI's deberta-v3-base-prompt-injection-v2 (Apache-2.0, about 740 MB), the tokenizers library and ONNX Runtime ([third-party content](assistant.md#third-party-content-and-prompt-injection)) |
| Punctuation model | `analysis.cleanup` is on and `analysis.cleanup_punctuation` isn't `off` ([Cleaning up transcripts](processing.md#cleaning-up-transcripts)) | kredor/punctuate-all at a pinned revision (MIT, about 1.1 GB), with Transformers and PyTorch's CPU build |
| spaCy | `analysis.entities` is `spacy` ([Entities](processing.md#entities-and-the-graph-explorer)) | spaCy (MIT) and the model in `analysis.spacy_model`: `en_core_web_sm`, `de_core_news_sm` or `xx_ent_wiki_sm` (MIT, 11 to 15 MB each), about 60 MB in all |
Expand All @@ -35,7 +36,7 @@ never fetch a speech or voice model.
- Python packages go into `data_dir/python`, at the versions in `uv.lock`. Packages the image already has win.
- Models go into `data_dir/models`: the Hugging Face, ModelScope and PyTorch caches point there (unless `HF_HOME`,
`MODELSCOPE_CACHE` or `TORCH_HOME` say otherwise). They outlast container rebuilds and upgrades.
- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model, Florence-2, spaCy's models) are checked against their
- Model files fetched by URL (YOLOX, YuNet, SFace, the NSFW model, Florence-2, spaCy's models, Real-ESRGAN) are checked against their
SHA-256.
- A job step that needs something still being fetched waits for it instead of falling back or failing. If a fetch fails,
the step runs as before (for transcription, falling back to another engine), and the fetch is tried again an hour
Expand Down
3 changes: 2 additions & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,8 @@ Start from `fastapi_backend/archive.example.yaml`, which documents every key. Th
Admins can change, in Settings: transcription, speaker separation, speech providers, voice IDs, analysis, the LLM
provider and key, a local model, the AI assistant, the [code sandbox](assistant.md#code-sandbox), search (word
search and search by meaning), reports and graph,
video, OCR, faces and objects, [NSFW classification](nsfw.md), workers, sign-in, components (and the edition, full or
video, OCR, faces and objects, [NSFW classification](nsfw.md), [enhancing
images](enhance.md), workers, sign-in, components (and the edition, full or
lean), access and embedding (whether other devices on the network reach Lens, embed frame ancestors, text and transcript
upload limit, allowed hosts, trusted proxies, session length), remote access, email, chat rooms, notifications,
automations, telemetry, the Fedora repository, sensors, file shares, uploads, storage, documents, API keys,
Expand Down
92 changes: 92 additions & 0 deletions docs/enhance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Enhancing images

Lens can look at each image before it's read and, when it's low quality, keep an enhanced copy that reading (OCR),
faces, objects, NSFW classification and descriptions work from instead. The original stays as it is. It's **off**
until an admin turns it on, and it runs on your own server: pictures aren't sent anywhere, and no model provider is
called.

Audio isn't enhanced yet.

## Turning it on

Settings → Enhance, **Enhance low quality images** (`enhance.images`). The **enhance** step comes first in the
standard pipeline, so it looks at images added from then on. A pipeline of your own runs it when you add the
**enhance** step to it; a step with **force** set enhances every image it runs on, whatever the settings and its
quality.

Any image can also be enhanced on request, even with it off: **Enhance** in its Details tab (editors),
`POST /api/v1/resources/{rid}/enhance`, the `enhance` tool for agents, or the `enhance` automation action. That makes
the copy and runs the rest of its pipeline again on it.

## What counts as low quality

The step measures the image and keeps why it looked low quality:

| Reason | When |
|---|---|
| small | its shorter side is under `enhance.min_side` pixels |
| blurry | its sharpness (the variance of its Laplacian, where it's sharpest) is under `enhance.min_sharpness` |
| noisy | its noise, estimated where it's plainest so edges and texture don't count, is over 6 levels out of 255 |
| low contrast | the spread of its tones is under 30 levels out of 255: flat, too dark or too bright |
| faces too small | a face on it is under `enhance.min_face_px` pixels tall, so recognition has little to go on |

Faces are looked for only where the namespace finds faces (its faces mode isn't off), with the face engine in
Settings → Video. With **Only images with faces** (`enhance.only_faces`), an image with no face is left alone.

An image with none of these is left as it is, and its job log says so: `good enough as it is (900×760, sharpness
312, noise 1.8, contrast 61)`.

## How it's enhanced

A small image, or one with small faces, is made bigger: enough to bring its shorter side to `enhance.min_side` and
its smallest face to `enhance.min_face_px`, at most `enhance.max_scale` times and never past 4000 pixels a side. The
engine (`enhance.image_engine`):

| Engine | What it does |
|---|---|
| `classic` (default) | Pillow, nothing to fetch: a noisy image is smoothed with a median filter, enlarged with Lanczos, a dull one stretched to its full range of tones, and a soft or enlarged one sharpened (unsharp mask) |
| `realesrgan` | [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN) general x4v3 (BSD-3-Clause, about 5 MB, as Qualcomm AI Hub exports it to ONNX), run with ONNX Runtime on the CPU in tiles. It enlarges four times while taking out blur, noise and JPEG blocks, and the result is brought to the size wanted |

Real-ESRGAN and ONNX Runtime are fetched the first time the step runs with it chosen ([Components](components.md));
until then nothing is downloaded, so a Raspberry Pi that keeps `classic` stays small. An image over 1.5 megapixels is
enhanced by `classic` instead, so a small machine isn't swamped, and the job log says why.

## The enhanced copy

The copy is a JPEG kept beside the original (`data_dir/renditions/<id>-enhanced.jpg`), encrypted where the
namespace's files are, re-keyed with them, and deleted with the resource. Its Details tab says what was found and
made (`Small, blurry · 240×180 to 960×720 by the classic engine`) with **Save the copy**. Only members of the
namespace can download it; visitors to a shared resource see its pages as before.

Turning it off leaves copies alone until an image is processed again: then a copy the step made by itself is removed,
and one made on request is kept.

## Settings

| Setting | Default | |
|---|---|---|
| `enhance.images` | `false` | enhance low quality images |
| `enhance.image_engine` | `classic` | `classic` or `realesrgan` |
| `enhance.only_faces` | `false` | only images with a face |
| `enhance.min_side` | `720` | small below this many pixels on the shorter side (64 to 4000) |
| `enhance.min_sharpness` | `100` | blurry below this sharpness (0 to 10000; 0 never counts one as blurry) |
| `enhance.min_face_px` | `64` | faces smaller than this many pixels tall are too small (0 to 1024; 0 turns it off) |
| `enhance.max_scale` | `4` | enlarge at most this many times (1 to 4) |

## API and agents

- `POST /api/v1/resources/{rid}/enhance` (needs the process permission on the resource): enhance an image now and run
its pipeline's later steps again; answers the `job`. 400 for anything but an image. Audited as `recording.enhance`.
- A resource carries `enhanced`: `{kind, low, measured, enhanced, engine, scale, width, height, forced, at, download}`,
where `measured` is `{width, height, sharpness, noise, contrast, brightness, faces, smallest_face}` and `download` a
signed link to the copy; `null` until the step has looked at it.
- `GET /api/v1/resources/{rid}/enhanced`: the copy (members of its namespace).
- [MCP](mcp.md): `enhance` (a write-scope token); `get_recording` says what the step found.
- [Automations](automations.md): the `enhance` action, on an asset.

## Refine later

- Enhance audio before it's transcribed: noise, level and clipping.
- Enhance a video's keyframes and a document's scanned pages, not only images.
- A face restorer for small faces, with a permissive license.
- Show the original and the copy side by side in the asset page.
4 changes: 3 additions & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ For an app that speaks OpenAI's API rather than MCP, the namespace can also answ
| `nsfw_settings` | how [NSFW classification](nsfw.md) works in a namespace: on or off, the threshold, how flagged resources show, and how many are classified, flagged and not classified yet |
| `set_nsfw` | turn NSFW classification on or off, set its threshold and display, for a namespace (owners) or the archive (admins), and optionally classify what's there. Needs a write-scope token |
| `mark_nsfw` | mark a resource NSFW or not by hand, or hand it back to the model's score. Needs a write-scope token and editor access |
| `enhance` | [enhance an image](enhance.md) whatever the settings, keeping the original, and read it again from the enhanced copy. Needs a write-scope token and the process permission |
| `namespace_integrations` | whether a namespace has [its own MCP server](#one-namespaces-mcp-server) and [chat completions](api.md#chat-completions-for-any-app) on, their addresses, and the tools offered there |
| `set_namespace_integrations` | turn a namespace's own MCP server or chat completions on or off, and pick the read tools its chat completions may use. Owners, with a write-scope token |
| `list_files` | a folder of [the graph as files](file-tree.md): your namespaces, then their collections, topics (nested by narrower topics) and entities, each holding its assets' files |
Expand Down Expand Up @@ -171,7 +172,8 @@ which records an undoable change (proposed by default), `undo_graph_change`, `ed
app (merges and deletes can be undone), `write_note`, which adds a note, `create_podcast`, which
queues an episode, `create_feed` and `publish_to_feed`, which make and fill a [published feed](feeds.md),
`write_spec`, which queues a spec, `set_nsfw` and `mark_nsfw`, which change [NSFW classification](nsfw.md) (owners or admins, and
editors), `set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `set_secret_reference`, which points a secret field at a password manager's item (admins), `add_mcp_server`, which adds an
editors), `enhance`, which enhances an image and reads it again (the process permission),
`set_namespace_integrations`, which turns a namespace's own MCP server or chat completions on or off (owners), `set_picture_descriptions`, which changes what describes pictures (admins), `set_file_storage`, which changes where new files are kept (admins), `set_secret_reference`, which points a secret field at a password manager's item (admins), `add_mcp_server`, which adds an
MCP server (admins), `subscribe_feed` and `unsubscribe_feed`, which follow and stop following news feeds (admins), and `set_monitoring`, `set_service`,
`set_sandbox_gateway` (the [code sandbox](assistant.md#code-sandbox)'s gateway) and `set_install_settings`, which have the machine change how Lens runs (admins).
Importing stays in the web app and the API.
Expand Down
Loading
Loading
0