8000 Tags · gkgy/opendataloader-pdf · GitHub
[go: up one dir, main page]

Skip to content

Tags

Tags

v2.4.7

Toggle v2.4.7's commit message
fix(hybrid/docling): collapse spanning cells so docling tables pass P…

…DF/UA §7.2

Objective: When the docling-fast backend produces a table with merged cells
(colspan>1 or rowspan>1), every header row that contains a merged cell
fails veraPDF rule PDF/UA-1 §7.2 (test 43) / PDF/UA-2 §8.2.5.26 "Table rows
shall have the same number of columns (taking into account column spans)".
In a 200-PDF batch, 9 documents fail for this reason — heuristic and
hancom-ai backends are unaffected because they emit cells differently.

Approach: docling reports the merged cell at its top-left start position
AND emits redundant empty 1x1 placeholder cells at every slot the merged
cell already covers. The previous loop wrote each cell into the grid
independently, so the placeholders survived and inflated the row's
effective column count. Mirror what HancomAISchemaTransformer already
does for the same backend pathology: in a second pass, fill every slot
inside a spanning cell's area with the same cell instance so
AutoTaggingProcessor.addTableRow's identity check skips them. Clamp
spans to the declared table bounds defensively.

Evidence: 200-PDF batch with `--hybrid docling-fast`, `--conformance ua1,ua2`.

| Scenario | Before | After |
|----------|--------|-------|
| ua1 veraPDF pass | 191 / 200 | 200 / 200 |
| ua2 veraPDF pass | 191 / 200 | 200 / 200 |
| §7.2 test 43 failed checks across batch | 9 | 0 |
| Wall clock (200 docs) | ~3m27s | ~3m13s |

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

v2.4.6

Toggle v2.4.6's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(processors): surface clean error for corrupted or truncated PDFs (o…

…pendataloader-project#527)

Objective: A PDF whose %PDF- header is present but whose body is corrupt or truncated (download interrupted near the end, missing trailing xref) currently surfaces a 17-line veraPDF IOException stack trace on stdout.

The user has no way to tell whether their file is broken or whether opendataloader-pdf is.

Approach: Wrap only the new PDDocument(pdfName) call in DocumentProcessor.preprocessing with a try-catch on IOException and rethrow as InvalidPdfFileException with a distinct message ending in "corrupted or truncated content".

InvalidPasswordException is re-thrown as-is so the existing password-handling branch in CLIMain still wins.

Other failure modes (parseStructureTreeRoot, parseChunks, hybrid path) remain on the existing SEVERE path — only the header-present-body-broken case is reclassified.

Because InvalidPdfFileException is already handled by CLIMain.processFile, no CLI changes are needed: CLI-argument input prints "Error: ..." to stdout and exits 1; a corrupt file inside a directory logs WARNING and continues (preserves batch-folder semantics).

The original veraPDF exception is preserved as getCause() so Java API consumers can still inspect the root cause for diagnostics.

Evidence: mvn -pl opendataloader-pdf-cli -am test runs DocumentProcessorMagicNumberTest at 4/4 pass and CLIMainTest at 23/23 pass.

Manual end-to-end against the built jar:

| Scenario | Expected | Actual |
|----------|----------|--------|
| Real PDF with last 2000 bytes cut off | friendly stdout, exit 1, no stack trace | "Error: 'real_truncated.pdf' is not a valid PDF file (corrupted or truncated content)." exit 1 |
| JPEG renamed to .pdf | distinct missing-header wording | "Error: 'fake.pdf' is not a valid PDF file (missing %PDF- header)." exit 1 |
| Intact PDF | exit 0, JSON written | exit 0, JSON generated |
| Directory with 1 truncated + 1 intact | intact processed, truncated skipped with WARNING, exit 0 | WARNING "'truncated.pdf' ... Skipping.", good.json written, exit 0 |

npm run sync not required — no CLI option definitions changed.

Python/Node wrappers pass Java stdout and exit code through unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v2.4.5

Toggle v2.4.5's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(processors): surface clean error for corrupted or truncated PDFs (o…

…pendataloader-project#527)

Objective: A PDF whose %PDF- header is present but whose body is corrupt or truncated (download interrupted near the end, missing trailing xref) currently surfaces a 17-line veraPDF IOException stack trace on stdout.

The user has no way to tell whether their file is broken or whether opendataloader-pdf is.

Approach: Wrap only the new PDDocument(pdfName) call in DocumentProcessor.preprocessing with a try-catch on IOException and rethrow as InvalidPdfFileException with a distinct message ending in "corrupted or truncated content".

InvalidPasswordException is re-thrown as-is so the existing password-handling branch in CLIMain still wins.

Other failure modes (parseStructureTreeRoot, parseChunks, hybrid path) remain on the existing SEVERE path — only the header-present-body-broken case is reclassified.

Because InvalidPdfFileException is already handled by CLIMain.processFile, no CLI changes are needed: CLI-argument input prints "Error: ..." to stdout and exits 1; a corrupt file inside a directory logs WARNING and continues (preserves batch-folder semantics).

The original veraPDF exception is preserved as getCause() so Java API consumers can still inspect the root cause for diagnostics.

Evidence: mvn -pl opendataloader-pdf-cli -am test runs DocumentProcessorMagicNumberTest at 4/4 pass and CLIMainTest at 23/23 pass.

Manual end-to-end against the built jar:

| Scenario | Expected | Actual |
|----------|----------|--------|
| Real PDF with last 2000 bytes cut off | friendly stdout, exit 1, no stack trace | "Error: 'real_truncated.pdf' is not a valid PDF file (corrupted or truncated content)." exit 1 |
| JPEG renamed to .pdf | distinct missing-header wording | "Error: 'fake.pdf' is not a valid PDF file (missing %PDF- header)." exit 1 |
| Intact PDF | exit 0, JSON written | exit 0, JSON generated |
| Directory with 1 truncated + 1 intact | intact processed, truncated skipped with WARNING, exit 0 | WARNING "'truncated.pdf' ... Skipping.", good.json written, exit 0 |

npm run sync not required — no CLI option definitions changed.

Python/Node wrappers pass Java stdout and exit code through unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v2.4.4

Toggle v2.4.4's commit message
Update verapdf version

v2.4.3

Toggle v2.4.3's commit message
fix: pass headless JVM flags so macOS does not show Dock icon

Objective: On macOS, every CLI run pops a Java icon in the Dock and
steals focus from the active window — so any batch or repeated PDF
conversion turns into a flickering, focus-thrashing experience.

Approach: Pass -Djava.awt.headless=true and -Dapple.awt.UIElement=true
as JVM args ahead of -jar at every spawn site (Node wrapper, Python
wrapper, scripts/run-cli.sh, npm export-options). The CLI never opens
a UI — it only manipulates BufferedImages and renders PDFBox pages off
screen — so headless is safe everywhere. apple.awt.UIElement is the
macOS-specific flag the launcher reads to skip Dock registration; it
is harmless on other OSes.

Evidence: Ran the built jar and the Node wrapper against a sample PDF
on macOS.

Before: every invocation surfaced a Java Dock icon and stole focus.
After (java -D... -jar):    PDF processed, JSON written, no Dock icon.
After (Node convert() API): variant resolved successfully, no Dock icon.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v2.4.2

Toggle v2.4.2's commit message
fix(runner): remove dead locale import and harden encoding robustness

- Remove unused `import locale` (dead after encoding fix, fails ruff F401)
- Add `errors="replace"` to both subprocess calls for resilience against
  unexpected non-UTF-8 bytes in JVM error output
- Guard `sys.stdout.buffer` access with hasattr for redirected-stdout
  environments (test fixtures, io.StringIO, etc.)
- Add `sys.stdout.buffer.flush()` to ensure real-time streaming output

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

v2.4.1

Toggle v2.4.1's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
refactor(core): move CLIOptions to core api.cli package (opendataload…

…er-project#475)

* refactor(core): move CLIOptions to core api.cli package

Objective: Downstream tools (opendataloader-pdfua) need CLI option
parsing logic to populate Config from a CommandLine. They had to
depend on opendataloader-pdf-cli, which is not published to Maven
Central — forcing a local `mvn install` step before any build.

Approach: Move CLIOptions to core under org.opendataloader.pdf.api.cli
so downstream tools depend only on core (which is already on Central).
The cli module keeps CLIMain as the executable entry point and adds
an import to the new package. Stable members for downstream use are
documented on the class: defineOptions, addAllTo, applyAllTo, FOLDER_OPTION.

Evidence:
- mvn test in core: 634/634 pass (incl. moved CLIOptionsTest,
  CLIOptionsContentSafetyTest)
- mvn test in cli: 5/5 pass
- npm test (vitest, node wrapper): 32/32 pass
- pytest (python wrapper, cli_options + convert_integration): 13/13 pass
- `--export-options` byte-level diff vs existing options.json: identical
- shaded jar manifest: Main-Class still org.opendataloader.pdf.cli.CLIMain

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* review: tighten CLIOptions stability contract + cover addAllTo edge cases

Objective: Code review on opendataloader-project#475 flagged that the class-level Javadoc
declared four "stable" members but did nothing to discourage downstream
from depending on the dozens of other public statics (option-name
constants, createConfigFromCommandLine, exportOptionsAsJson). Also
asked for direct test coverage of the addAllTo external-Options pattern
that pdfua relies on.

Approach:
- Rewrite the class Javadoc to explicitly mark all non-listed public
  members as internal — public visibility for cross-package access
  only, may change in any release. Add a usage example so the
  intended pattern (custom Options + addAllTo + applyAllTo) is visible
  at the top of the file.
- Add two tests covering the contract pdfua actually depends on:
  addAllTo preserves pre-existing downstream options, and addAllTo
  is idempotent (commons-cli's silent-replace-by-long-name behavior
  is pinned so a future commons-cli upgrade that changes it surfaces
  here).

Evidence:
- mvn test CLIOptionsTest: 61/61 pass (incl. 2 new)
- mvn install: BUILD SUCCESS
- mvn javadoc:jar -P release: BUILD SUCCESS (doclint passes new Javadoc)
- Probe of commons-cli 1.11 confirms Options.addOption silently replaces
  by long-name (count stays 1, last description wins) — test pins this.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v2.4.0

Toggle v2.4.0's commit message
docs(readme): mark auto-tagging as shipped and stop overpromising com…

…pliance

Objective: Two messaging problems in README confuse the target audience
(accessibility teams at organizations seeking open-source remediation
tools):

1. Auto-tagging is presented as "Coming Q2 2026" in 14 places, but it
   has shipped (--format tagged-pdf, --format json,tagged-pdf, etc.).
   The Python sample showed a fake auto_tag=True parameter that does
   not exist in CLIOptions.java — the real API uses format="tagged-pdf".
2. The Problems table marks "PDF accessibility compliance" as Shipped,
   and the AI-AGENT-SUMMARY says we "automate PDF accessibility
   compliance". Auto-tagging alone is not compliance — PDF/UA requires
   alt text, language metadata, validation, and PDF/UA export
   (enterprise-only). Marking compliance itself as shipped overpromises
   what the open-source tier delivers.

Approach:
1. Replace every "Q2 2026" / "Coming Q2 2026" / "preview" reference
   with the shipped state. Rewrite the Auto-Tagging Preview section as
   "Auto-Tagging" with the real Python + CLI examples. Update the
   Which-Mode table, Capability Matrix, Accessibility Pipeline table,
   workflow diagram, and FAQ. Remove auto-tagging from the Roadmap.
2. Tone down the Problems table:
   "PDF accessibility compliance ... Shipped" →
   "Manual PDF remediation cost ... Auto-tag: Shipped. PDF/UA export:
   Enterprise", explicitly framing auto-tagging as "Foundation for
   PDF/UA workflows".
   Update AI-AGENT-SUMMARY: "automate PDF accessibility compliance" →
   "accelerate PDF accessibility remediation ... as foundation for
   PDF/UA". This positions the OSS tier honestly while keeping the
   commercial value clear.

Evidence:
- grep "Q2 2026|coming Q2|auto_tag\s*=" README.md → 0 hits.
- The fake API auto_tag=True is fully replaced by format="tagged-pdf"
  (Python/Node) and --format tagged-pdf (CLI), matching the actual
  values in CLIOptions.applyFormatOption (cli/CLIOptions.java:514-545).
- The "Coming Soon" reference for Hancom Data Loader integration
  (L279) is intentionally preserved — it is unrelated to auto-tagging.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v2.3.0

Toggle v2.3.0's commit message
fix(python): include JAR and license files in sdist for conda-forge

Objective: conda-forge packagers must bundle Maven + OpenJDK as
build-time dependencies because PyPI only ships a wheel — there is
no sdist with the pre-compiled JAR they can build from. Attempting
to produce one silently fails: hatch drops gitignored files (JAR,
LICENSE, NOTICE, THIRD_PARTY) from the sdist even when they are
listed in 'include', yielding a 32KB tarball that installs but is
missing the CLI it wraps.

Approach: Promote 'artifacts' to top-level [tool.hatch.build] so
both wheel and sdist force-include the gitignored build outputs.
Drop '--wheel' from build-python.sh so 'uv build' produces both
artifacts. Add verify-python-sdist.sh as a standalone check that
runs inside build-python.sh and can also be run locally — this
guards against silent regressions if .gitignore or hatch config
drifts in future. README.md is copied by build-python.sh before
'uv build' because hatchling validates [project.readme] during
metadata parsing, which runs before build hooks.

Evidence: Built from a fresh clean checkout (no prior artifacts
in the package dir) and installed the resulting sdist in a fresh
venv.

| Scenario | Expected | Actual |
|----------|----------|--------|
| uv build on clean checkout | sdist + wheel | 21MB sdist + 21MB wheel |
| sdist contains JAR | yes | yes (tar -tzf confirmed) |
| sdist contains LICENSE/NOTICE/THIRD_PARTY | yes | yes |
| verify-python-sdist.sh | pass | "OK: sdist contains all required files" |
| verify-python-sdist.sh with multiple tarballs | fail with list | fails cleanly |
| pip install from sdist | no mvn call | wheel built in ~1s, mvn not invoked |
| JAR in site-packages | yes | 23MB at .../jar/opendataloader-pdf-cli.jar |
| import opendataloader_pdf | success | success |

Fixes opendataloader-project#435

v2.2.1

Toggle v2.2.1's commit message
fix: preserve whitespace chunk positions to restore missing spaces (o…

…pendataloader-project#358)

TextLineProcessor dropped whitespace chunks then relied solely on
physical gap detection to re-insert spaces. PDFs with tight kerning
(gap < threshold) lost all word spacing. Now the position of each
dropped whitespace chunk is recorded via an IdentityHashMap-backed Set
and used as an additional signal for space insertion alongside the
existing gap-based logic. Space BoundingBox coordinates are clamped
with min/max to handle overlapping chunks from tight kerning.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
0