Wenzheng Zeng, Siyi Jiao, Chen Gao, Hwee Tou Ng, Mike Zheng Shou
National University of Singapore
Paper | HF Daily Paper | Demo | News | Overview
Click here to watch the demo video in MP4 format
- [2026-08] The code and model have been released!
- [2026-07] Our work is featured by DailyPapers!
- [2026-07] Our paper is available on arXiv.
- [2026-06] Our work is accepted by ECCV 2026!
We propose PadCaptioner, a 3B model for omni-modal dense video captioning that achieves high efficiency and strong grounded caption quality, outperforming 7B counterparts.
The core idea is to exploit the weak local dependencies among temporally distinct events and restructure the causal token dependency, enabling lossless parallel generation.
We design a latent planning mechanism that automatically determines parallelizable units with non-local awareness, guiding subsequent parallel decoding and improving event grounding and caption quality.
- Clone this repository:
git clone https://github.com/showlab/PadCaptioner.git
cd PadCaptioner- Create the conda environment:
conda create -n padcaptioner python=3.12
conda activate padcaptioner- Install dependencies. FFmpeg is needed by torchcodec for video/audio decoding, and Java by pycocoevalcap for the METEOR metric:
conda install -c conda-forge ffmpeg openjdk
pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txtPlease refer to DATASET.md for raw dataset links, annotation preparation, the full JSON schema, and task examples.
-
Model Preparation
-
Prepare the video-SALMONN 2+ 3B checkpoint.
-
Set
MODEL,MODEL_BASE, andLORA_CKPT(download from here) inscripts_run/train.shto the downloaded paths. At training start, the LoRA checkpoint is automatically merged into the base model to form the video-SALMONN 2+ starting point.
- We notice that Video-SALMONN 2+ was subsequently updated, so its newer, already-merged model may potentially serve as a starting point. However, we have not verified the correctness of this setup.
-
-
Fill in the path block at the top of
scripts_run/train.sh(DATASET,MODEL,MODEL_BASE,LORA_CKPT,OUTPUT_ROOT), or pass them as flags. -
Run:
bash scripts_run/train.sh
-
Download the pretrained model.
-
Download the test annotations from PadCaptioner-Data (
ChronusAV/test_dvc2000.jsonfor dense video captioning, orChronusAV/test.jsonfor all seven tasks) and the ChronusAV test videos, then point the annotations at your videos:python my_tools/relocate_video_paths.py \ --anno /path/to/test_dvc2000.json \ --video_dir /path/to/videos \ --out_dir /path/to/annotations -
Fill in the path block at the top of
scripts_run/test.sh(DATASET,MODEL,OUTPUT_ROOT), or pass them as flags:bash scripts_run/test.sh \ --dataset /path/to/annotations/test_dvc2000.json \ --model /path/to/PadCaptioner-3B \ --output_root /path/to/output \ --run_name dvc2000 \ --gpus 0,1Predictions are written to
/path/to/output/dvc2000/test_results_rank*.json. -
Compute the metrics:
python my_tools/etbench_dvc_metrics.py \ --run_dir /path/to/output/dvc2000 \ --gt /path/to/annotations/test_dvc2000.json
We list below the related works that inspired PadCaptioner:
If you find our work useful, please kindly cite:
@inproceedings{zeng2026parallelized,
title={Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning},
author={Zeng, Wenzheng and Jiao, Siyi and Gao, Chen and Ng, Hwee Tou and Shou, Mike Zheng},
booktitle={European Conference on Computer Vision},
pages={189--209},
year={2026},
organization={Springer}
}