Dolphin (Document Image Parsing via Heterogeneous Anchor Prompting) is a novel multimodal document image parsing model following an analyze-then-parse paradigm. This repository contains the demo code and pre-trained models for Dolphin.
Document image parsing is challenging due to its complexly intertwined elements such as text paragraphs, figures, formulas, and tables. Dolphin addresses these challenges through a two-stage approach:
- 🔍 Stage 1: Comprehensive page-level layout analysis by generating element sequence in natural reading order
- 🧩 Stage 2: Efficient parallel parsing of document elements using heterogeneous anchors and task-specific prompts
Dolphin achieves promising performance across diverse page-level and element-level parsing tasks while ensuring superior efficiency through its lightweight architecture and parallel parsing mechanism.
Try our demo on Demo-Dolphin.
- 🔥 2025.07.10 Released the Fox-Page Benchmark, a manually refined subset of the original Fox dataset. Download via: Baidu Yun | Google Drive.
- 🔥 2025.06.30 Added TensorRT-LLM support for accelerated inference!
- 🔥 2025.06.27 Added vLLM support for accelerated inference!
- 🔥 2025.06.13 Added multi-page PDF document parsing capability.
- 🔥 2025.05.21 Our demo is released at link. Check it out!
- 🔥 2025.05.20 The pretrained model and inference code of Dolphin are released.
- 🔥 2025.05.16 Our paper has been accepted by ACL 2025. Paper link: arXiv.
-
Clone the repository:
git clone https://github.com/ByteDance/Dolphin.git cd Dolphin -
Install the dependencies:
pip install -r requirements.txt
-
Download the pre-trained models using one of the following options:
Option A: Original Model Format (config-based)
Download from Baidu Yun or Google Drive and put them in the
./checkpointsfolder.Option B: Hugging Face Model Format
Visit our Huggingface model card, or download model by:
# Download the model from Hugging Face Hub git lfs install git clone https://huggingface.co/ByteDance/Dolphin ./hf_model # Or use the Hugging Face CLI pip install huggingface_hub huggingface-cli download ByteDance/Dolphin --local-dir ./hf_model
Dolphin provides two inference frameworks with support for two parsing granularities:
- Page-level Parsing: Parse the entire document page into a structured JSON and Markdown format
- Element-level Parsing: Parse individual document elements (text, table, formula)
# Process a single document image
python demo_page.py --config ./config/Dolphin.yaml --input_path ./demo/page_imgs/page_1.jpeg --save_dir ./results
# Process a single document pdf
python demo_page.py --config ./config/Dolphin.yaml --input_path ./demo/page_imgs/page_6.pdf --save_dir ./results
# Process all documents in a directory
python demo_page.py --config ./config/Dolphin.yaml --input_path ./demo/page_imgs --save_dir ./results
# Process with custom batch size for parallel element decoding
python demo_page.py --config ./config/Dolphin.yaml --input_path ./demo/page_imgs --save_dir ./results --max_batch_size 8# Process a single document image
python demo_page_hf.py --model_path ./hf_model --input_path ./demo/page_imgs/page_1.jpeg --save_dir ./results
# Process a single document pdf
python demo_page_hf.py --model_path ./hf_model --input_path ./demo/page_imgs/page_6.pdf --save_dir ./results
# Process all documents in a directory
python demo_page_hf.py --model_path ./hf_model --input_path ./demo/page_imgs --save_dir ./results
# Process with custom batch size for parallel element decoding
python demo_page_hf.py --model_path ./hf_model --input_path ./demo/page_imgs --save_dir ./results --max_batch_size 16