This repo contains ToyBlocks: an open simulation framework based on IsaacSim/IsaacLab, built around a robot arm manipulating blocks on a tabletop. The setting is deliberately simple, but the robot must still reason about how the blocks interact and how to arrange them while pushing, flipping, and stacking.
ToyBlocks automatically and scalably generates demonstration data for policy training, without requiring time-consuming and expensive teleoperation. Researchers can easily vary these training experiences. For example, a team could generate more pushing or flipping demonstrations, then evaluate performance on tasks kept out of training.
The generated datasets are recorded in the LeRobot format and can be used directly to train VLAs or other policies. ToyBlocks ships with training infrastructure for ManiFlow, a low-level visuomotor policy conditioned on language and on a target heatmap, and serves it either on its own or under a high-level policy (such as Molmo2, Astra, Claude) that decides what to do next.
Policies are evaluated in closed loop in simulation, either on freshly generated unseen scenes and tasks or on the exact initial states of episodes recorded in a dataset. Evaluation is GPU-parallel. Because every task's goal is expressed in PDDL, ToyBlocks automatically measures success and partial progress, with no task-specific metric code. Either level of a hierarchical policy can be replaced by an oracle to isolate where failures come from, and the states where a policy failed can be sent back to data generation to plan new expert demonstrations and recoveries from them.
The overall goal of ToyBlocks is to provide a controlled world for robot learning and evaluation. By providing full control over the data generation process, ToyBlocks provides tools for researchers to trace the outcomes of trained policies back to the training data.
Install, fetch the assets, and generate a small dataset on one GPU:
git submodule update --init --recursive
uv venv --python 3.11
uv pip install -e ".[dev,isaac,torch-cu128,vecgen]"get-toyblocks-assets --asset-manifest asset_manifest.jsonpython scripts/datagen/run_datagen_pipeline.py \
--experiment-id my_first_run --output-root data \
--task-gen-script scripts/task_gen/stack_three.py --task-gen-count 2 \
--env-per-problem 8 --num-envs 8 --exec-num-envs 8 --vecgen-problems-per-env 1 \
--render-pass src/toyblocks/domain_randomization/flags/render_flags_plain.yamlThat plans and executes 2 stacking problems, renders the recorded episodes, and merges them
into one LeRobot dataset under data/datasets/merged/train/my_first_run.
Execution records no camera images, so the video comes from the render stage.
--render-pass FLAGS.yaml[:N] is the only flag that turns it on: each one renders every
episode under one flags YAML into its own output directory, and :N asks for N differently
randomized copies. Repeat the flag for more looks, and the merge takes them all:
--render-pass src/toyblocks/domain_randomization/flags/render_flags_plain.yaml \
--render-pass src/toyblocks/domain_randomization/flags/render_flags_allrand_texcache.yaml:3That ships 4 episodes per executed one: the plain look once, the randomized look three times. See Automatic Data Generation and Rendering.
The docs walk through each workflow:
- Installation and Setup
- Automatic Data Generation
- Training ManiFlow Policies on Automatically Generated Data
- Policy Evaluation
All source code in src/toyblocks/
src/toyblocks/tasks: all hand created tasks as well as templates for PDDL based task generationsrc/toyblocks/vecgen: the datagen engine (FF planning, batched skills, cuRobo planner pool, LeRobot recording)src/toyblocks/robots: all robot assets and configurations, including controllerssrc/toyblocks/serving: policy serving infrastructure (BasePolicy,PolicyServer, WebSocket client, obs transforms)src/toyblocks/eval: evaluation of learned models (with or without oracles) and replaysrc/toyblocks/wrappers: gymWrappersubclasses (recording, stats, obs flattening)src/toyblocks/cli: command line utilitiessrc/toyblocks/data_annotation: dataset annotation utilitiessrc/toyblocks/utils: utilities for ToyBlockssrc/toyblocks/utils/rigid_objects: utilities for working with rigid objects (e.g. creating them more easily)
The ManiFlow and Molmo2 policy code is in the maniflow-tb/ and molmo2/ submodules; the agentic
robotics loop (LLM agents with the skills as MCP tools) is the agentic_robotics/ submodule, see
Agentic Robotics.
ToyBlocks is licensed under the Apache License 2.0.
The curobo, maniflow-tb, molmo2 and agentic_robotics submodules and the downloaded assets (e.g. the HDRI pools) are
distributed under their own licenses.