Repository navigation
feat(documents): read documents with an Apache Tika server - #292
Merged
Merged
Conversation
An opt-in Tika server (`--profile tika` in the compose files, or any Tika server set in Settings → Documents) reads, as text, the documents nothing here can make into a PDF, and what every document says about itself as Dublin Core terms (`described` on the resource; its authors and language feed the resource's metadata). Off until documents.tika_url is set. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
…talled Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
Owner
Author
|
Blocked on merge conflicts with main in Generated by Claude Code |
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
…older The survey passed, but rm -rf failed with 'Directory not empty' while the just-killed API was still writing, failing the job. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
Owner
Author
|
The "Clicks and keystrokes per feature" check failed after every journey passed. The cause was the job's cleanup step: Generated by Claude Code |
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
…able converters Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
Merged
16 of 22 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Description
Before: a Word, PowerPoint or spreadsheet file on a lean server or a Raspberry Pi (no LibreOffice, no anytopdf node) was refused, and Lens knew nothing of a document's own title, authors or dates.
After: Lens can use an Apache Tika server. The compose files get an opt-in
tikaprofile (docker compose --profile tika up -d, orCOMPOSE_PROFILES=tika). In Settings → Documents, one button fills inhttp://tika:9998, and a Test button checks it. Then:documents.tika_text, on): when LibreOffice, Chromium and anytopdf can't make a document into a PDF here, or one of them fails on it, Tika reads it as text with no pages. The text is searched, analyzed and asked about like any other. The engine istext:tika, and uploads accept Office files.documents.tika_metadata, on): every document is described by Tika as Dublin Core terms (describedon the resource: title, creators, subject, language, created, modified, pages, words, format). The title and creation date replace the file-name defaults. The authors and language flow into the resource's metadata, and from there into RDF and schema.org. The Details tab shows them. If Tika is down, the document is still read.Off until
documents.tika_urlis set, so nothing changes by default.How:
domain/tika.pycallsPUT /rmeta/text(Tika 4.xtk:keys and the 2.x/3.xX-TIKA:keys) andGET /version. Requests go only to the address the admin set: http(s), no credentials, no redirects. The file is sent asdocument.<ext>. Answers are capped at 64 MB and text at 20M characters.convert.unavailablelets Tika cover what nothing else can, anddocuments.transcribefalls back to it. The compose service runsapache/tika:4.0.0-1, read-only, as a non-root user, with no published port and a 512 MB heap (TIKA_HEAP,TIKA_VERSION).Motivation and Context
Adeel asked to integrate and set up Apache Tika: text and metadata for many formats, alongside anytopdf.
Steps to reproduce
Tested against a real
apache/tika:4.0.0-1container (LENS_TEST_TIKA=http://127.0.0.1:9998 pytest tests/api/test_tika.py). The image starts read-only in about 1 s and uses about 220 MB.Types of changes
Checklist:
tests/api/test_tika.pywith a fake Tika server, plus a real-server test; frontenddescribedRows).Done questions
POST /settings/documents/tika/test, anddescribedonGET /resources/{id}.describedis an additive, schemaless field. Older documents get it when they are re-transcribed.make openapi, 779 backend and 728 frontend tests passed locally.-fullimage.🤖 Generated with Claude Code
https://claude.ai/code/session_01X6p3skuCAfa2nTxSmDKvxq
Generated by Claude Code