8000 Tags · mehanahmed/opendataloader-pdf · GitHub
[go: up one dir, main page]

Skip to content

Tags

Tags

v2.5.7

Toggle v2.5.7's commit message
chore(deps): bring every manager up to date

Java (java/pom.xml):

- verapdf 1.31.160 -> 1.31.169. One property, five artifacts, because
  the released validation pom this project imports as a BOM pins the
  modules beneath it: validation-model, wcag-validation and
  metadata-fixer to 1.31.169, parser 1.31.48 -> 1.31.51, core and
  verapdf-xmp-core 1.31.35 -> 1.31.37
- okhttp 5.4.0 -> 5.5.0
- jackson-databind 2.22.1 -> 2.22.2. jackson-annotations stays at 2.22,
  its own release line, where no 2.22.2 exists
- junit-jupiter 6.1.2 -> 6.1.3
- pdfbox stays at 3.0.4, the version wcag-algorithms asks for

The comments in <properties> are cut down to what prevents a mistake.
The entry-point and security-floor notes go entirely: they said not to
go below a version, which guides nothing when the policy is to pin the
latest and only move forward. What is left is three two-line notes on
the pins that are not obvious — a range that needs freezing, a version
line that does not track its sibling, and one that has to move in step
with veraPDF.

Node: 9 devDependencies. commander, the only runtime dependency, is
unchanged. Manifest moves with the lockfile because reverting it alone
fails pnpm install --frozen-lockfile, which is what CI runs.

Python: 45 transitive packages in opendataloader-pdf and 22 in
opendataloader-pdf-mcp, lockfiles only. Held back so the tier-3 set does
not move here: docling and its family, since 2.124.0 pulls docling-parse
across two majors; transformers, safetensors, accelerate,
huggingface-hub, rapidocr, imageio, pypdfium2, which reach the extracted
text; opencv-python 4.13 -> 5.0.0, rpds-py and pywin32, all major.

The docling floor also moves >=2.91.0 -> >=2.94.0. Everything below
2.94.0 carries CVE-2026-47214, unsafe URI and path handling in the HTML
backend. The lockfile already pinned 2.94.0, so a resolved install was
not exposed, but a fresh resolution could have been — which is also why
no alert fired.

Examples: opendataloader-pdf floor >=2.2.1 -> >=2.5.6.

Verified: 641 tests; samples/pdf byte-identical in both -f json (20/20)
and -f pdf (20/20); pnpm --frozen-lockfile, build and 33 tests;
uv lock --check on both packages; clean-venv install of both examples.

v2.5.6

Toggle v2.5.6's commit message
test(filter): let @tempdir clean up after the background filter test

The directory was created with Files.createTempDirectory and removed on the
last line, so a throw anywhere in between left it behind. @tempdir is what
the other 18 test classes here use and it deletes on failure too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

v2.5.5

Toggle v2.5.5's commit message
feat(hybrid): retry transient hancom-ai failures and bound the call

The client had no retry and no effective timeout: a backend restart lost
the layout for the pages in flight, and the default timeout of 0 meant a
hung request waited forever, which in a batch cannot be told apart from a
hung client.

Timeouts. An explicit --hybrid-timeout keeps its exact meaning, 0
included, so a deliberate setting is unaffected. Only the unset case
gains a ceiling: connect 10s, and one whole call capped at 1 hour via
callTimeout. Not readTimeout — the backend computes for minutes and then
answers at once, so an idle-byte limit would kill healthy requests. The
cap sits far above real work (a 20-page OCR chunk measured 268s); its job
is to end a hang, not to pace normal calls. The health probes override
callTimeout rather than inherit it: an hour-long health check is not one.

Retries. Three attempts, waiting 5s then 45s. The second wait clears a
restart rather than a blip — scan.sh records restarts taking 45-60s, so a
retry sooner than that just spends an attempt on a server still starting.

What is retried is deliberately narrow:
- 5xx, unusable body, connect failure, timeout — transient
- 4xx, and a module the server does not have — permanent, one attempt
- SUCCESS=false — one retry only. It arrives with HTTP 500 and
  RESULT:[[]] both from a restarting backend and from a document the
  backend genuinely rejects, and the two are indistinguishable; measured,
  one 2.2MB PDF returned it three times running. Each attempt costs GPU
  time, so it does not get the full budget. Carried as a field on the
  attempt rather than re-derived from the log wording.

Layout, page images and the per-region enrichment calls all share the
budget: the enrichment calls are the numerous ones, so a restart lands
hardest there — without a retry every table on the page comes back
structurally empty and nothing reports it.

That breadth needs bounding, because the budget is per request and a
backend refusing everything makes every page pay it. Measured on a
document whose pdf2img calls all failed: 50s per page, past ten minutes
of pure waiting. Two streak limits now cap it — after 3 consecutive
exhausted enrichment calls, and after 3 layout slices that returned no
pages, the rest are sent once each. The same document finishes in 325s
with 6 retries instead of one per page.

Every retry logs at WARNING with the attempt and the wait, so a retried
hang is visible rather than silent.

865 tests pass, 11 new. Each retry decision is pinned by exact request
count, including one that fails at 37 requests instead of 19 if the
breaker is removed. Output on a healthy backend is unchanged and no retry
fires.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

v2.5.4

Toggle v2.5.4's commit message
feat(hybrid): retry transient hancom-ai failures and bound the call

The client had no retry and no effective timeout: a backend restart lost
the layout for the pages in flight, and the default timeout of 0 meant a
hung request waited forever, which in a batch cannot be told apart from a
hung client.

Timeouts. An explicit --hybrid-timeout keeps its exact meaning, 0
included, so a deliberate setting is unaffected. Only the unset case
gains a ceiling: connect 10s, and one whole call capped at 1 hour via
callTimeout. Not readTimeout — the backend computes for minutes and then
answers at once, so an idle-byte limit would kill healthy requests. The
cap sits far above real work (a 20-page OCR chunk measured 268s); its job
is to end a hang, not to pace normal calls. The health probes override
callTimeout rather than inherit it: an hour-long health check is not one.

Retries. Three attempts, waiting 5s then 45s. The second wait clears a
restart rather than a blip — scan.sh records restarts taking 45-60s, so a
retry sooner than that just spends an attempt on a server still starting.

What is retried is deliberately narrow:
- 5xx, unusable body, connect failure, timeout — transient
- 4xx, and a module the server does not have — permanent, one attempt
- SUCCESS=false — one retry only. It arrives with HTTP 500 and
  RESULT:[[]] both from a restarting backend and from a document the
  backend genuinely rejects, and the two are indistinguishable; measured,
  one 2.2MB PDF returned it three times running. Each attempt costs GPU
  time, so it does not get the full budget. Carried as a field on the
  attempt rather than re-derived from the log wording.

Layout, page images and the per-region enrichment calls all share the
budget: the enrichment calls are the numerous ones, so a restart lands
hardest there — without a retry every table on the page comes back
structurally empty and nothing reports it.

That breadth needs bounding, because the budget is per request and a
backend refusing everything makes every page pay it. Measured on a
document whose pdf2img cal
8000
ls all failed: 50s per page, past ten minutes
of pure waiting. Two streak limits now cap it — after 3 consecutive
exhausted enrichment calls, and after 3 layout slices that returned no
pages, the rest are sent once each. The same document finishes in 325s
with 6 retries instead of one per page.

Every retry logs at WARNING with the attempt and the wait, so a retried
hang is visible rather than silent.

865 tests pass, 11 new. Each retry decision is pinned by exact request
count, including one that fails at 37 requests instead of 19 if the
breaker is removed. Output on a healthy backend is unchanged and no retry
fires.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

v2.5.3

Toggle v2.5.3's commit message
feat(hybrid): split long documents across several layout requests

The Hancom AI layout module takes a whole PDF and offers no page-selection
parameter, and rejects requests past roughly 30 pages. HancomAIClient never
read request.getPageNumbers(), so the 50-page chunking in
HybridDocumentProcessor was a no-op for this backend: every chunk re-uploaded
the entire document and got every page back. Long documents failed outright
and shorter ones did the same work twice.

The client now slices the document into shorter PDFs of
hybrid.layoutPageChunk pages (default 20, leaving headroom under the limit)
and merges the replies. Slicing keeps the PDF form rather than rendering
pages to images, which would strip the text layer and push a digital
document down the OCR path.

The correctness hinge is page numbering: the module numbers response pages
from 0 relative to the request, while the transformer reads page_number as an
absolute document index. Without translation, page 30's content lands on
page 0 — nothing throws and the output looks well formed. Requested pages are
filtered against the real page count before slicing so the slice list and the
uploaded PDF cannot disagree, records numbered outside the slice are dropped
rather than placed, and merged records are sorted by absolute page so the
evidence JSON is not left shuffled.

Every requested page that comes back without a layout record is reported for
the Java pipeline to retry, on both the sliced and whole-document paths. Left
unreported it becomes an empty page in the output, which reads as a page that
genuinely had no content. That covers a slice that errors, one answered [[]]
(empty but well-formed, which an array-size check passes as success), one
that comes back short, and one whose page tree is damaged, since PDFBox
throws IllegalStateException there rather than IOException. Only a total
failure raises.

The document goes up whole only when every page is wanted: 10 pages of a
100-page document fit under the limit, but sending the file would upload all
100 and make the backend process them. Turning slicing off means "do not
divide the pages", not "ignore the selection", and a selection naming no page
that exists raises rather than answering with pages nobody asked for.

hancom-ai no longer chunks in HybridDocumentProcessor, since the client does
it: the outer split only re-uploaded the file and, because just the last
call's raw JSON is kept, dropped the earlier pages' evidence.

The chunk size is settable as --hybrid-hancom-ai-layout-page-chunk. The
backend's real limit is not measured — the servers were unreachable while
this was built — so it needs to be adjustable without a code change, and
verifying the behaviour needs it from the command line.

Verified against a page-aware probe that reports which source page each
record carried: 1, 20, 21, 30, 31, 82 and 255-page documents place every page
exactly once, zero misplaced, and no request exceeds the chunk size. A mutant
that ignores the slice mapping collapses a 32-page document to 20 pages, and
one that skips the missing-page accounting fails five tests. Core suite: 822
tests, no failures; CLI verification 61 passed.

v2.5.2

Toggle v2.5.2's commit message
feat(hybrid): support DLA figure subdivision and English captions

The DLA model split its "figure" class into Figure(10)/Chart(18)/Image(19).
The transformer only knew labels 0-17, so charts and photos fell through to
a branch that drops objects with no OCR text — they vanished from output
entirely. A 200-document corpus scan puts that at 58% of visual objects.

- Map labels 18/19 through label 10's existing picture route
- Switch captioning to IMAGE_CAPTIONING_EN; both engines write the same
  response field, so the module name alone decides the language
- Caption all three visual classes rather than gating on the class: a
  rectangular photo was observed classified Figure(10) at confidence 0.900,
  so gating drops alt text for real photos with nothing to show it happened
- Add FORMULA_RECOGNITION; equations carry LaTeX instead of OCR text, with
  the crop rectangle so evidence tooling can point at the region. Blank or
  whitespace output falls back to the OCR text rather than replacing it
- Drop duplicate visual detections (IoU >= 0.8, keep the confident one).
  Measured: 3 duplicate pairs, all IoU 0.964-0.989, weaker always wrong.
  Identity is the object's array position, not object_id — DLA omits that
  field on some objects and repeats it across others on one page. Survivors
  are walked confidence-first and compared only against those already kept,
  so the cost stays linear in the detections a response claims
- Run formulas before captioning so a page with both is rendered once
- Ignore a page the server marked IS_SUCCESS:false, and detect "not existed
  engine": an unknown module returns HTTP 200 with SUCCESS:true, so a
  module-name typo would otherwise be silent
- Keep per-conversion state local; a cached client serves many documents and
  convertAsync lets calls overlap
- Add scripts/dla-corpus-scan/ to measure label distribution against a real
  corpus, since the published spec does not match what the model emits

Verified end to end against the backend: English alt text reaches the JSON
and the PDF struct tree, 35 equations emit LaTeX, and the overlap cases
collapse to one picture each. 769 core tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

v2.5.1

Toggle v2.5.1's commit message
fix(release): let pnpm version run on the version-injected tree

The Release workflow for v2.5.1 failed at "Build and test all packages",
so Maven Central, PyPI, npm, the GitHub Release and the reference-docs
push were all skipped. The tag exists but nothing shipped: `gh release
view v2.5.1` is "not found", npm is still 2.5.0, PyPI 2.5.1 is a 404.

build-all.sh injects the release version into each ecosystem before
building it -- `mvn versions:set` for Java, `sed` for pyproject.toml,
`pnpm version` for Node. Java and Python come first, so by the time the
Node stage runs the tree necessarily has modifications. pnpm 11 added a
clean-working-tree gate to `version` (`opts.gitChecks !== false` guarding
isWorkingTreeClean) and aborts:

    [ERR_PNPM_UNCLEAN_WORKING_TREE] Working tree is not clean.

pnpm 10 had no such gate, which is why this only appeared once a7789b8
moved CI from pnpm 10 to 11. Release only runs on tag push, so v2.5.1 was
the first tag to exercise that combination.

c9de934 fixed a real instance of this -- a stale samples/json/lorem.json
rewritten by every Java build -- and that fix is in v2.5.1 and holds: a
full `mvn clean install` now leaves the tree clean. What remains is the
version injection itself. `mvn versions:set` rewrites java/pom.xml plus
the cli and core poms, and no amount of sample regeneration makes that go
away, so the gate has to be opted out of rather than satisfied.

--no-git-checks is the narrowest fix. The gate guards against creating a
version commit or tag on top of uncommitted work, and --no-git-tag-version
already disables both, so nothing it protects applies here. The dirty tree
is this script's own deliberate output, three lines earlier.

Verified locally on pnpm 11.21.0, java/python manifests dirtied to match
the CI state: `pnpm version 2.5.1 --no-git-tag-version --allow-same-version`
reproduces ERR_PNPM_UNCLEAN_WORKING_TREE, the same command with
--no-git-checks bumps 0.0.0 -> 2.5.1, and build-node.sh then builds and
passes all 33 tests. Confirmed `pnpm install --frozen-lockfile` has no
such gate and succeeds on a dirty tree, and that this is the only
`pnpm version` call site in the repo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

v2.5.0

Toggle v2.5.0's commit message
chore(deps): bump junit-jupiter to 6.1.2 and spotbugs-maven-plugin to…

… 4.10.3.0

Objective: The build should track the latest stable releases of its test
and static-analysis tooling so contributors get current bug fixes and
avoid drifting onto unsupported versions.

Approach: Audited every version property and plugin in java/pom.xml
against the authoritative sources — Maven Central release metadata for
Central-hosted artifacts and the vera-dev artifactory for the veraPDF
family. Only the two artifacts that had a newer *stable* release were
bumped: junit-jupiter 6.1.1 -> 6.1.2 and spotbugs-maven-plugin
4.10.2.0 -> 4.10.3.0. Everything else (veraPDF 1.31.99 / wcag-algorithms
1.31.33, jackson, okhttp, commons-cli, and the remaining plugins) is
already at its latest stable, so it was left untouched. Pre-release
versions that exist for assertj (4.0.0-M1), maven-compiler (4.0.0-beta-4),
surefire (3.6.0-M1), source, and deploy were intentionally skipped — a
published library should not ship on milestones/betas.

Evidence: Ran the real build and full test suite after the bump.
Before: pom pinned junit 6.1.1 / spotbugs 4.10.2.0.
After: `mvn clean package` succeeds and `mvn test` passes with 0
failures / 0 errors across all modules (exit 0).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

v2.4.7

Toggle v2.4.7's commit message
fix(hybrid/docling): collapse spanning cells so docling tables pass P…

…DF/UA §7.2

Objective: When the docling-fast backend produces a table with merged cells
(colspan>1 or rowspan>1), every header row that contains a merged cell
fails veraPDF rule PDF/UA-1 §7.2 (test 43) / PDF/UA-2 §8.2.5.26 "Table rows
shall have the same number of columns (taking into account column spans)".
In a 200-PDF batch, 9 documents fail for this reason — heuristic and
hancom-ai backends are unaffected because they emit cells differently.

Approach: docling reports the merged cell at its top-left start position
AND emits redundant empty 1x1 placeholder cells at every slot the merged
cell already covers. The previous loop wrote each cell into the grid
independently, so the placeholders survived and inflated the row's
effective column count. Mirror what HancomAISchemaTransformer already
does for the same backend pathology: in a second pass, fill every slot
inside a spanning cell's area with the same cell instance so
AutoTaggingProcessor.addTableRow's identity check skips them. Clamp
spans to the declared table bounds defensively.

Evidence: 200-PDF batch with `--hybrid docling-fast`, `--conformance ua1,ua2`.

| Scenario | Before | After |
|----------|--------|-------|
| ua1 veraPDF pass | 191 / 200 | 200 / 200 |
| ua2 veraPDF pass | 191 / 200 | 200 / 200 |
| §7.2 test 43 failed checks across batch | 9 | 0 |
| Wall clock (200 docs) | ~3m27s | ~3m13s |

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

v2.4.6

Toggle v2.4.6's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(processors): surface clean error for corrupted or truncated PDFs (o…

…pendataloader-project#527)

Objective: A PDF whose %PDF- header is present but whose body is corrupt or truncated (download interrupted near the end, missing trailing xref) currently surfaces a 17-line veraPDF IOException stack trace on stdout.

The user has no way to tell whether their file is broken or whether opendataloader-pdf is.

Approach: Wrap only the new PDDocument(pdfName) call in DocumentProcessor.preprocessing with a try-catch on IOException and rethrow as InvalidPdfFileException with a distinct message ending in "corrupted or truncated content".

InvalidPasswordException is re-thrown as-is so the existing password-handling branch in CLIMain still wins.

Other failure modes (parseStructureTreeRoot, parseChunks, hybrid path) remain on the existing SEVERE path — only the header-present-body-broken case is reclassified.

Because InvalidPdfFileException is already handled by CLIMain.processFile, no CLI changes are needed: CLI-argument input prints "Error: ..." to stdout and exits 1; a corrupt file inside a directory logs WARNING and continues (preserves batch-folder semantics).

The original veraPDF exception is preserved as getCause() so Java API consumers can still inspect the root cause for diagnostics.

Evidence: mvn -pl opendataloader-pdf-cli -am test runs DocumentProcessorMagicNumberTest at 4/4 pass and CLIMainTest at 23/23 pass.

Manual end-to-end against the built jar:

| Scenario | Expected | Actual |
|----------|----------|--------|
| Real PDF with last 2000 bytes cut off | friendly stdout, exit 1, no stack trace | "Error: 'real_truncated.pdf' is not a valid PDF file (corrupted or truncated content)." exit 1 |
| JPEG renamed to .pdf | distinct missing-header wording | "Error: 'fake.pdf' is not a valid PDF file (missing %PDF- header)." exit 1 |
| Intact PDF | exit 0, JSON written | exit 0, JSON generated |
| Directory with 1 truncated + 1 intact | intact processed, truncated skipped with WARNING, exit 0 | WARNING "'truncated.pdf' ... Skipping.", good.json written, exit 0 |

npm run sync not required — no CLI option definitions changed.

Python/Node wrappers pass Java stdout and exit code through unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0