Utility scripts for lmms-eval development and dataset preparation.
Split large ZIP files into smaller parts for hosting services with file size limits.
# Split into 5GB parts (default)
python tools/get_split_zip.py dataset.zip ./output/
# Split into 2GB parts
python tools/get_split_zip.py dataset.zip ./output/ --max-size 2GBRun regression tests across git branches to compare model performance.
python tools/regression.py --tasks ocrbench,mmmu_val --limit 8Build MINERVA Lance dataset from metadata and local videos.
python tools/minerva_to_lance.py \
--metadata-json data/minerva/minerva.json \
--videos-dir data/minerva/videos \
--output data/minerva_hf_package/data/train.lance \
--mode overwriteBenchmark absolute latency of MINERVA video resolution (minerva_doc_to_visual).
python tools/bench_minerva_video_resolution.py \
--metadata-json data/minerva/minerva.json \
--mode lance \
--lance-uri hf://datasets/lmms-lab-eval/minerva/data/train.lance \
--limit 200 \
--sample-unique-videoBenchmark end-to-end MINERVA pipeline latency for local raw storage vs Lance storage using the same decode path.
python tools/bench_minerva_pipeline_latency.py \
--local-video-dir /absolute/path/to/minerva/videos \
--lance-uri hf://datasets/lmms-lab-eval/minerva/data/train.lance \
--limit 100 \
--batch-size 1 \
--decode-num-frames 8This benchmark keeps decode behavior fixed and compares only storage mode. On local disks it is often near-parity; Lance tends to help more in remote/object-storage and reproducible multi-machine setups.
Benchmark video decode backend latency and throughput on the same local video file across pyav, torchcodec, and dali.
python tools/bench_video_decode_backends.py \
--video /absolute/path/to/video.mp4 \
--backends pyav,torchcodec,dali \
--num-frames 8 \
--iterations 30 \
--output /tmp/video_decode_backends.jsonpyav is the default reference. If optional backends are not installed, the script records backend-specific errors in the JSON report without failing the whole run.
Tutorial for creating properly formatted Hugging Face datasets with images. Demonstrates the complete workflow from raw data to HF Hub upload.
The following modules were removed during cleanup. They can be found in the main branch if needed:
lite/- LMMs-Eval Lite for dataset core-set selection using embedding-based samplinglive_bench/- Separate package for LiveBench data generation and website screenshot capture