VideoGAIA is a benchmark for agentic video understanding. This repository provides the open-source agent-loop inference and evaluation framework for evaluating OpenAI-compatible multimodal models with a unified ReAct harness.
Instead of answering from a fixed set of frames, an evaluated model can iteratively inspect selected video segments, search the web, and read relevant pages before producing its final answer.
- 🎬 Agentic Video Analysis: Select and resample arbitrary temporal windows for focused visual inspection
- 🌐 Tool-Augmented Reasoning: Combine video analysis with web search and webpage extraction
- 🔁 Unified ReAct Loop: Evaluate different multimodal models under the same interaction protocol
- ⚖️ Robust LLM Judging: Support independent primary and fallback judge models
- ⚡ Parallel Evaluation: Run multiple models concurrently with per-model concurrency controls
- 💾 Safe Resumption: Persist every judged sample and resume interrupted runs automatically
- 📊 Detailed Reports: Track overall, category-level, and difficulty-level accuracy
| Tool | Description |
|---|---|
VideoAnalysisAction |
Resamples a selected time window and asks the evaluated model to inspect its frames |
WebSearchAction |
Retrieves web search results through Serper |
ExtractUrlContentAction |
Reads a selected webpage through Jina Reader |
The evaluated model summarizes each tool result before the runner inserts that summary as a genuine <observation> message. The parser rejects <observation> tags authored by the model, preventing model-generated content from being mistaken for real tool output.
videogaia/
├── run_eval.py # 🚀 Evaluation entry point
├── pyproject.toml # Package metadata and dependencies
├── requirements.txt # Runtime dependencies
├── .env.example # Environment variable template
├── videogaia/ # 📦 Core evaluation package
│ ├── data.py # Dataset loading and validation
│ ├── evaluator.py # Parallel evaluation runner
│ ├── judge.py # Primary and fallback LLM judges
│ ├── io_utils.py # Output and resume utilities
│ ├── openai_compat.py # OpenAI-compatible request helpers
│ └── agent_loop/ # 🤖 ReAct agent implementation
│ ├── agent.py # Agent loop, prompt, and response parser
│ ├── config.py # Agent configuration
│ ├── data.py # Agent-loop sample data model
│ ├── llm_client.py # Multimodal model client
│ ├── media.py # Video decoding and frame sampling
│ ├── summarizer.py # Tool-result summarization
│ ├── tools.py # Video, search, and webpage tools
│ └── trace.py # Trajectory recording
└── tests/ # ✅ Unit tests
The ReAct system prompt is defined in videogaia/agent_loop/agent.py. The benchmark data is released separately on 🤗 Hugging Face and is not duplicated in this repository.
Python 3.10 or newer is required.
git clone https://github.com/zfkarl/VideoGAIA.git
cd VideoGAIA
python -m venv .venv
source .venv/bin/activate
pip install -e .Copy the environment template and add your credentials:
cp .env.example .env| Variable | Description |
|---|---|
VIDEOGAIA_API_BASE_URL |
OpenAI-compatible chat-completions endpoint for evaluated models |
VIDEOGAIA_API_KEY |
API key for the evaluated-model endpoint |
VIDEOGAIA_JUDGE_API_BASE_URL |
Optional endpoint for the primary judge |
VIDEOGAIA_JUDGE_API_KEY |
Optional API key for the primary judge |
VIDEOGAIA_FALLBACK_JUDGE_API_BASE_URL |
Optional endpoint for the fallback judge |
VIDEOGAIA_FALLBACK_JUDGE_API_KEY |
Optional API key for the fallback judge |
SERPER_API_KEY |
Serper API key for web search |
JINA_API_KEY |
Jina API key for webpage extraction |
VIDEOGAIA_MODELS |
Optional comma-separated default model list |
The model endpoint must accept OpenAI-style multimodal messages containing base64 image_url content. Blank judge settings automatically fall back to the evaluated-model endpoint.
Important
Never commit your .env file. API keys are read only from environment variables and are redacted from run_config.json.
Download the VideoGAIA dataset from Hugging Face:
hf download Karl28/VideoGAIA \
--repo-type dataset \
--local-dir ../videogaia_dataPlace the downloaded dataset next to this repository:
videogaia-release/
├── videogaia/
74BC
# This repository
└── videogaia_data/
├── videogaia_271.jsonl
└── videos/
Validate the dataset before making any API calls:
python run_eval.py \
--input-jsonl ../videogaia_data/videogaia_271.jsonl \
--validate-input-onlyValidation checks sample IDs, English questions, answers, video paths, category counts, and the L1/L2 difficulty distribution.
Models run in parallel with an independent concurrency limit for each model:
python run_eval.py \
--input-jsonl ../videogaia_data/videogaia_271.jsonl \
--models "MODEL_A,MODEL_B,MODEL_C" \
--judge-primary-model "PRIMARY_JUDGE" \
--judge-fallback-model "FALLBACK_JUDGE" \
--concurrency-per-model 5 \
--model-max-attempts 30 \
--judge-primary-max-attempts 30 \
--judge-fallback-max-attempts 30 \
--output-dir outputs/full_runThe runner performs a minimal model preflight before scheduling samples. Use --skip-preflight only when an endpoint does not support text-only checks. GPT-5-family endpoints automatically use max_completion_tokens and omit unsupported temperature values; Kimi-family endpoints use their supported temperature behavior.
outputs/full_run/
├── run.log # Runtime logs
├── run_config.json # Redacted run configuration
├── preflight.json # Endpoint preflight results
├── results.jsonl # Per-sample judged results
├── summary.json # Machine-readable metrics
├── summary.md # Human-readable report
└── artifacts/<model>/<sample>/
├── inference.json # Full agent trajectory
└── result.json # Final judged result
results.jsonl is appended immediately after each judged sample. Re-running the same command with the same --output-dir skips completed (model, sample) pairs and continues pending work. Reports are refreshed after every completed sample.
Generated frames are deleted after each sample by default, while the complete textual trajectory is retained. Add --keep-generated-frames when frame-level debugging is needed.
- OpenAI Python SDK — OpenAI-compatible model communication
- PyAV and OpenCV — video decoding and frame sampling
- Serper — web search
- Jina Reader — webpage content extraction
If you find Orchestra-o1 useful, please cite our paper:
@misc{zhang2026videogaiabenchmarkgeneralai,
title={VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding},
author={Fan Zhang and Guangming Yao and Jinyang Wu and Hao Wu and Zheng Lian and Xinyu Geng and Jingdong Chen and Yi Yuan and Pheng-Ann Heng},
year={2026},
eprint={2608.14718},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.14718},
}