8000 GitHub - pypi-ahmad/DocLayout: From scans and images to structured Markdown. · GitHub
[go: up one dir, main page]

Skip to content
pypi-ahmadPublic

About

From scans and images to structured Markdown.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

74 Commits

Folders and files

DocLayout

DocLayout turns PDFs, scans, images, and office documents into structured Markdown and other outputs. In the browser app, you can inspect pages, copy or download results, and ask questions about the extracted text. The CLI, Python library, and local HTTP API provide other ways to convert documents.

Features

  • PP-DocLayoutV3 attempts local layout detection before GPT-6.1 Sol reads each whole page. The page model supplies text and HTML; accepted V3 matches supply mask-derived contours (or V3 rectangles when contours are unusable) and relative reading order. Unmatched content keeps the page model's geometry; V3-only regions remain diagnostic evidence. If V3 cannot run, conversion continues with the page model and records the fallback.
  • Markdown has a native rendered view and an exact raw view. The separate HTML preview uses a white page, serif typography, embedded crops, and MathML.
  • Download Markdown, HTML, JSON, chunks, annotated page images, or an annotated PDF. The ZIP contains document outputs and excludes the source upload and chat history.
  • GPT-6 Luna answers document questions with source-quote checks and a second verification request before displaying accepted answers.
  • Preview navigation, copying, and downloads reuse completed session results without repeating OCR.
  • Upload multiple files with three active document jobs; batches process all pages.
  • Extract authorization fields from raw Markdown with Sol/medium. The default is one field call. The sidebar checkbox Retry evidence quotes, or DOCLAYOUT_GROUNDING_RETRY_ENABLED=true, may add one quote-only call. Results are stored as JSON and SQLite records. Classification is explicitly off by default. A separate persistent Extracted information page presents grouped summaries and service tables. Its source view links fields to PDF regions in both directions. See the field extraction guide.

How DocLayout works

The data-flow diagram reflects the current parsed-text and layout path; the system architecture is a high-level snapshot. The layout integration record describes the v3.1.0 contour integration, pipeline v4 diagnostics, and validation limits. In the GUI, authorization-field extraction starts after conversion and saves its results in Extracted information. The CLI and HTTP API remain conversion-only. See the architecture guide for the complete workflow, including optional classification.

Installation

Install uv. Git is also needed for GitHub-source installs. DocLayout v3.1.0 requires Python >=3.11,<4; Windows checks use Python 3.14. PyPI publication is deferred. The Git commands below install the current main branch. The release wheel installs v3.1.0; see its release notes. Dependencies still need a reachable package index or a populated local cache. Before running conversion, configure API access.

Install as a uv tool

For the CLI and all export formats:

uv tool install "git+https://github.com/pypi-ahmad/DocLayout.git@main"
doclayout input.pdf output --all

To include the browser app, use this installation instead:

uv tool install "doclayout[gui] @ git+https://github.com/pypi-ahmad/DocLayout.git@main"
doclayout_gui

If uv reports that its executable directory is missing from PATH, run uv tool update-shell and open a new terminal. Reinstalling with different extras may require uv tool install --force with the desired installation specifier.

Manual clone

git clone https://github.com/pypi-ahmad/DocLayout.git
cd DocLayout
uv sync --locked --group dev --extra full

This installs the GUI, API, development tools, and optional document converters. After configuring credentials, run:

uv run doclayout input.pdf output --all
.\launch.cmd

The launcher opens http://localhost:8471 and stops an existing DocLayout listener on that port before starting. For another application, it asks for confirmation. Restarting resets the browser session but preserves saved results. The launcher disables file watching, so restart the app after edits. doclayout_gui also binds loopback and leaves existing listeners alone. The API requires a separate bearer token; see security limits.

pip or uv pip

Use the Python environment where you want the commands installed. With uv, create one first if needed using uv venv. Choose one installer:

uv pip install "doclayout[gui] @ git+https://github.com/pypi-ahmad/DocLayout.git@main"
python -m pip install "doclayout[gui] @ git+https://github.com/pypi-ahmad/DocLayout.git@main"

The v3.1.0 release wheel avoids the Git requirement. Either installer can use this URL:

uv pip install "https://github.com/pypi-ahmad/DocLayout/releases/download/v3.1.0/doclayout-3.1.0-py3-none-any.whl"

The wheel command above installs th 8000 e base CLI. To add extras to a downloaded wheel, use its local path, for example uv pip install ".\doclayout-3.1.0-py3-none-any.whl[gui]".

Installation Includes
Base package CLI/library and every CLI export format
[gui] Streamlit workbench
[server] Local HTTP API, launched with doclayout_server
[full] Office/HTML/EPUB document-format converters
[gui,server,full] All of the above

At first preparation, DocLayout checks for the pinned V3 model and downloads it if missing. Auto mode tests CUDA execution when available and uses CPU if it fails. Sol still reads the whole page. See layout setup and the implementation and evidence record. Office/HTML/EPUB conversion also requires native WeasyPrint libraries; see format prerequisites. Tool-installed commands run directly as doclayout .... Inside a clone, uv run doclayout ... uses the project's environment.

Configure API access

DocLayout reads environment variables first. If a value is absent, it reads .env in the launch folder. Each variable follows that order independently. The GUI, CLI, Python library, and API server share this behavior.

Either set these in the terminal that will launch DocLayout:

$env:OPENAI_API_KEY = "your-api-key"
# Optional: set this only for a custom compatible endpoint.
$env:OPENAI_BASE_URL = "https://your-endpoint.example/v1"

Or copy .env.example to .env in your launch folder and edit it:

OPENAI_API_KEY=your-api-key
# Optional for a custom compatible endpoint:
# OPENAI_BASE_URL=https://your-endpoint.example/v1

With launch.cmd, the launch folder is the repository root. For an installed doclayout_gui or doclayout command, it is the current working directory. Keep .env private; it is excluded from Git and package builds. Restart the app after credential changes. After changing Windows User/Machine environment variables, also open a fresh terminal before launching it.

The endpoint must support the app's models and request formats. See credentials and persistent Windows setup.

Command-line examples

doclayout input.pdf output
doclayout input.pdf output --all
doclayout input.pdf output --markdown --html
doclayout input.pdf output --json --chunks
doclayout input.pdf output --annotated-pdf --annotated-images
doclayout input.pdf output --zip
doclayout input.pdf output --all --page_range 0,2-4
doclayout --help

The default is Markdown with its image crops. --all writes every conversion export and a ZIP after one extraction per selected page. --zip alone writes only the complete archive. Individual format flags can be combined; see the full output reference, including --metadata and --images.

CLI files and GUI downloads use originalfilename_YYYYMMDD_HHMMSS.ext, with one UTC timestamp per extraction. JSON variants, crops, and annotated pages add a type or page suffix. ZIP entries use the same names as loose exports. File outputs go directly into the requested directory; later runs get new names. See filename details for examples.

Folder conversion and the legacy single-file command remain available:

doclayout documents --output_dir output --workers 1
doclayout_single input.pdf --output_dir output --output_format markdown

GUI Start/End pages are one-based and inclusive for single-file uploads; multi-file batches process all pages without page selectors. CLI/API page ranges are zero-based. Extra refinement is optional; OCR always runs through Sol.

Documentation

Guide Contents
Usage Installation, GUI, CLI, Python, API, troubleshooting
Configuration Settings, defaults, limits, environment, precedence
Development Setup, tests, builds, extension and prompt-change practices
Python API reference Conversion, layout, renderers, and saved-field entry points
Onboarding First offline checks and a source-code map
Tutorial Hands-on path from coordinates and tests to a bounded live check
Contributor runbook Change workflow, review checklist, and failure handling
Python documentation audit Focused docstring pass and remaining coverage gaps
Changelog Current changes and compatibility history
Architecture Extraction pipeline, model calls, renderers, prompts, session state
Validation Dated live observations, offline checks, known limitations
Benchmarks Running a bounded extraction evaluation
Deployment example Optional Modal deployment and its verification limits

Interactive diagrams

Accuracy, privacy, and cost

DocLayout sends page images to the configured API endpoint. Chat sends extracted page text and recent accepted conversation turns. Requests use store=False, which does not replace the endpoint provider's own data-handling policy.

The sidebar shows estimated session API cost for OCR, refinement, and chat. CLI commands print each conversion's estimate, and document metadata includes token usage and costs. See rates and limitations.

Extraction and chat can incur API charges. Model-estimated boxes are not confidence scores. Small text, complex tables, equations, and header/footer classification need review. Document chat verification can also miss mistakes. The project does not report a general benchmark ranking or claim perfect accuracy.

The browser keeps conversion results in session memory. Download them before closing the session. CLI conversion writes files. The GUI and file command generate HTML from Markdown. The legacy single-file command, folder conversion, and API render HTML from document blocks.

Development

See the development guide for environment setup, offline and browser checks, explicit live evaluation, package builds, and prompt-change practices. The changelog records additions and retired interfaces.

Report problems through DocLayout issues, after removing secrets and private document contents.

References and attribution

DocLayout includes code derived from Marker, originally developed by Vik Paruchuri and contributors. DocLayout adapts that foundation for its extraction service, document workbench, chat, and exports. It is an independent application and does not imply upstream endorsement.

Distributed under the Apache License 2.0. See NOTICE for attribution and a description of modifications. Optional local benchmark data has its own source attribution and setup.

About

From scans and images to structured Markdown.

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0