📜 Paper | 🌐 Project Page | 🤗 Dataset
WMEval runs the same action trajectory in a Unity scene and in a world model, then scores the generated rollout against simulator ground truth (camera poses, object states, momentum and energy, causal event traces) rather than perceptual quality alone. It provides:
- 9 Unity scenes (
BallRoller,BeamDomino,BilliardGame,ChessBoard,LiquidContainer,MirrorRollingBall,RacingCar,SunRisesAndSets,WanderingGarden) with designed action trajectories. - WMEval Lite, a Unity-free suite: checksummed case bundles (conditioning frame, prompt, actions, reference rollout, Unity quantities) and one CLI that validates, evaluates and aggregates. The paper's Table 4/7 cells are Lite cases.
- Standardized trajectories compiled onto each backend's native interface, with explicit action targets, calibration and unsupported-control reporting.
Models ranked against their Unity engine twin (live version): results from the paper's tables, plus Odyssey scored with WMEval Lite under the same protocol.
| Rank | Model | Score ↑ | Control | Consistency | Kinematics | Fluid | Lighting | Prompt |
|---|---|---|---|---|---|---|---|---|
| 1 | Project Genie | 0.767 | 1/6 | 1/6 | 1/6 | 2/6 | 4/6 | 1/6 |
| 2 | LingBot-World (Act) | 0.467 | 3/6 | 3/6 | 3/6 ‡ | 3/6 ‡ | 2/6 | 1/6 |
| 2 | Odyssey | 0.467 | 4/6 | 1/6 | 3/6 | 1/6 | 5/6 | 3/6 |
| 4 | Matrix-Game 2.0 | 0.433 | 1/6 | 4/6 | 2/6 | 3/6 ‡ | 3/6 | 6/6 ‡ |
| 5 | LingBot-World v2 (Cam) | 0.267 | 5/6 ‡ | 5/6 | 3/6 ‡ | 3/6 ‡ | 1/6 | 3/6 |
| 6 | Waypoint-1.5 | 0.033 | 5/6 ‡ | 6/6 ‡ | 3/6 ‡ | 3/6 ‡ | 6/6 | 5/6 |
Each dimension is ranked on one metric, scene by scene: models are ordered by their mean rank across the dimension's scenes (equal means share a rank). Score = mean over dimensions of the share of other models ranked below. ‡ = not supported (the interface cannot express the dimension's actions): ranked tied last, scoring 0.
Per-dimension rankings
| Dimension | Ranking metric | Rank · model | Not ranked |
|---|---|---|---|
| Control | Camera-compensated cosine | 1 Project Genie 1 Matrix-Game 2.0 3 LingBot-World (Act) 4 Odyssey |
LingBot-World v2 (Cam) ‡, Waypoint-1.5 ‡ |
| Consistency | Character consistency | 1 Project Genie 1 Odyssey 3 LingBot-World (Act) 4 Matrix-Game 2.0 5 LingBot-World v2 (Cam) |
Waypoint-1.5 ‡ |
| Kinematics | Final displacement error (RacingCar only) | 1 Project Genie 2 Matrix-Game 2.0 |
LingBot-World (Act) ‡, Odyssey, LingBot-World v2 (Cam) ‡, Waypoint-1.5 ‡ |
| Fluid | Surface-tilt error | 1 Odyssey 2 Project Genie |
LingBot-World (Act) ‡, Matrix-Game 2.0 ‡, LingBot-World v2 (Cam) ‡, Waypoint-1.5 ‡ |
| Lighting | Sun disagreement | 1 LingBot-World v2 (Cam) 2 LingBot-World (Act) 3 Matrix-Game 2.0 4 Project Genie 5 Odyssey 6 Waypoint-1.5 |
— |
| Prompt | Overall rubric score | 1 Project Genie 1 LingBot-World (Act) 3 Odyssey 3 LingBot-World v2 (Cam) 5 Waypoint-1.5 |
Matrix-Game 2.0 ‡ |
Built by tools/build_leaderboard.py from experiments/leaderboard/paper_scores.yaml and Lite results for models added after the paper; Causality and cLPIPS are not included; HY-WorldPlay and Matrix-Game 3.0 are left out. --output lists every metric value and rank.
git clone https://github.com/hao-ai-lab/WMEval.git && cd WMEval
# macOS (Apple silicon)
uv venv --python 3.11 .venv && uv pip install --python .venv/bin/python -r requirements/macos.lock
# Linux + NVIDIA GPU: creates .venv-gpu and checks it; runs every dimension
bash scripts/bootstrap_gpu.sh
Case metadata (case.yaml, checksums) lives in this repo; case data is on the Hub at lmgame/WMEval-Lite (public, no token), pinned to lite/DATASET_REVISION:
python tools/sync_lite_data.py download # add --split full for the 220 synthetic trajectories
python tools/run_wmeval_lite.py validate --suite lite
VLM-judged dimensions need OPENAI_API_KEY; CV dimensions fetch pinned checkpoints on first use (experiments/protocol/evaluator_assets.lock.json). The data is git-ignored; git config core.hooksPath .githooks warns when a checkout needs a re-download.
Generate one video per case and seed at <model_root>/<Scene>/<variant>/<seed>.mp4 (or list them in a JSONL --candidate-manifest), then:
python tools/run_wmeval_lite.py list-cases --suite lite
python tools/run_wmeval_lite.py run --suite lite \
--model-root runs/<model> --output-dir reports/wmeval_lite/<model> --resume
--dimensions selects a subset; --split full adds the synthetic trajectories, reported apart from the core headline. Outputs: per_case_results.jsonl (one row per dimension × case × seed, with the case's evaluator lock), aggregate_summary.json (seed means per case, then across cases) and aggregate_summary.csv. A headline needs 5 seeds per applicable case, otherwise it is kept as provisional_value. Models that draw a HUD set shadow_hud_preset (lingbot, mg2, mg3, genie) for the lighting case. On Modal, modal run tools/run_wmeval_lite_modal.py::main --model-root /cand/<model>; tools/generation/generate_waypoint15_candidates_modal.py generates candidates from the bundles without Unity.
| Task | Command |
|---|---|
| Jobs and N/A cases for a new model | tools/expected_jobs.py --model <name> --output runs/<name> · TEMPLATE.md |
| Environment check | tools/preflight_lite.py --device <cuda|mps|cpu> |
| Compare with the paper's Table 4/7 | tools/paper_tables/export_lite_table_results.py → tools/paper_tables/compare_table_results.py compare |
| Hosted judge via a gateway | tools/judge_gateway.py --key-file <key> -- <args> |
| Rebuild the leaderboard | tools/build_leaderboard.py ... --readme README.md (full command in its docstring) |
Each dimension compares quantities recovered from the candidate video with those logged by the Unity twin. Pipelines, flags and caveats: docs/evaluation_dimensions.md.
| Dimension | Question | Headline metrics | Cases | Needs |
|---|---|---|---|---|
| Consistency | Do objects, layout and character identity persist as the camera leaves and returns? | Depth AbsRel ↓, pairwise-distance MAE ↓, DINOv2 character consistency ↑ | 16 | CV |
| Action Controllability | Does motion follow the commanded action? | GT-RAFT relative EPE ↓ / cosine ↑; synthetic-flow cosine ↑ | 3 | CV |
| Kinematics & Mechanics | Do matched controls give simulator-consistent trajectories? | DTW ↓, FDE ↓, momentum/energy robust error ↓ | 3 | CV |
| Fluid Mechanics | Does liquid in a moving bottle slosh like the reference? | Area jitter ↓, surface tilt ↓, slosh-spectrum L1 ↓ | 4 | CV |
| Physical Causality | Does an action propagate through a multi-stage causal chain? | Causal success rate ↑, ICRA, DUA (deltas from Unity) | 3 | VLM |
| Lighting & Shading | Do shadows stay consistent with one light source? | Multi-object sun disagreement ↓, shadow jitter / area / shape drift ↓ | 1 | CV |
| Prompt-Scene Consistency | Does the rollout match its scene description? | Environment / object / overall rubric ↑ (ρ = 0.90 with humans) | 6 | VLM |
| Perceptual Similarity | How close is the rollout around matched contact events? | Causal LPIPS ↓ | 7 | CV + VLM |
Unity self-match floors: SELF_MATCH_BOUNDS.md; differences from the paper: DIFFERENCES_FROM_PAPER.md. Optional CLIP4Clip, DiffusionLight and GroundingDINO tools: docs/additional_evaluators.md.
lite/suite.yaml is the single source of truth for case wiring. The test_core split has 32 cases across 9 scenes and 8 dimensions (43 case–dimension pairs), including the 14 Table 4/7 cells (<Scene>/scene_<variant>, <Scene>/kinematics_<view>).
cases/<Scene>/<variant>/
case.yaml # prompt, modules, protocol, evaluator lock (in repo)
reference/ # from the Hub, except checksums.sha256
reference.mp4, ref.png # reference video; conditioning frame if not frame 0
artifacts/ # actions.txt, action.npy/npz, trajectory.yaml, rubrics, QA, liquid masks/flow
tools/dataset/build_lite_case.py (re)builds a case and its lock; tools/sync_lite_data.py upload publishes data (maintainers).
Action trajectories. Table 4/7 cases carry a canonical semantic trajectory.yaml (typed channels, units, segments) and its compiled forms; tools/dataset/build_table_lite_cases.py rebuilds them from the hash-locked trajectories in experiments/protocol/trajectories/. Backends are declared once in experiments/protocol/action_backends.yaml: the entity each trajectory actuates (camera, ego_vehicle, cue_ball, ...), each backend's native timeline and calibration, and which targets it cannot express. Input preparation (tools/trajectories/prepare_table_trajectories.py, prepare_matrix_game_inputs.py, prepare_lingbot_table7_inputs.py) refuses an unsupported target/backend pair rather than substitute another control.
| Backend | Native input | Camera | Vehicle / ball |
|---|---|---|---|
| Unity | runtime arguments | reference | reference |
| Project Genie | browser keys | nominal (left turn 51.47 °/s) | nominal key rules |
| Matrix-Game 2.0 | 129 frames @ 12 fps, keys + mouse | 10.1 °/mouse unit/frame | thresholded WASD |
| Matrix-Game 3.0 | 177 frames @ 17 fps, keys + mouse | regime-specific gains | out of scope |
| LingBot-World Base Cam | 161 camera poses @ 16 fps | nominal poses | unsupported |
Synthetic trajectories (test_full). 220 trajectories (252 rows with the core cases) built text first: each is an action description from a per-target step vocabulary ("the camera turns left at 30°/s for 2 s, then holds still") that completes a core case's prompt and compiles to a schema-v1 trajectory.yaml, starting from that case's conditioning frame. Each has a Unity reference rendered by replaying the actions; plans whose reference cannot give a meaningful score are screened out and replaced deterministically (audit/). Synthetic rows are scored like their conditioning case and aggregated in a separate synthetic block.
| Family | Target | # | Steps (core case) |
|---|---|---|---|
camera_yaw |
camera | 45 | turn, hold, sweep, half turn (9 scene views) |
camera_orbit |
camera | 20 | circle cw / ccw, pause (SunRisesAndSets/lighting) |
ego_vehicle |
vehicle | 28 | drive, steer, lane change, brake (RacingCar/kinematics_third) |
cue_ball |
cue ball | 25 | strikes in 8 directions × 3 strengths (BilliardGame/kinematics_first) |
ball_force |
ball | 28 | pushes in 8 directions × 2 strengths (BallRoller, MirrorRollingBall) |
beam_domino |
ball | 25 | rolls forward / angled / sideways (BeamDomino/causal) |
liquid_container |
bottle | 25 | jolts, slides, shakes, rests (LiquidContainer/fluid_rest) |
character |
character | 24 | walk in 8 directions, sprint, jump (ChessBoard/walk_third) |
Maintainer commands for rendering references and reference inputs (Unity editor required) are documented in tools/dataset/render_synthetic_references.py and tools/dataset/build_synthetic_reference_inputs.py.
The Unity benchmark scenes, and any output they produce (including the reference rollouts and Unity-logged quantities in the dataset), are licensed under the Unity Benchmark Scene License v1.0 (LICENSE-UNITY.txt). They may be used only to produce or validate the benchmark's metrics, and in particular not as training, fine-tuning or other input to any AI model, except as the reference against which a model's independently generated output is compared. Third-party evaluator code and weights are listed in THIRD_PARTY_NOTICES.md.
@article{hu2026wmeval,
title = {WMEval: Evaluating Interactive World Models Grounded on Physical Simulations},
author = {Hu, Lanxiang and Huang, Yixin and Shankarampeta, Abhilash and Dai, Zilin and
Wu, Yuyuan and Qian, Kun and Liu, Junze and Srinivasan, Arvind and Huo, Mingjia and
Lin, Will and Zhong, Kai and Zhang, Weiqi and Wang, Tian and Cui, Qingjun and
Rosing, Tajana and Zhang, Hao},
journal = {arXiv preprint arXiv:2610.33894},
year = {2026}
}