Any-to-any forward-dynamics cotraining for robot learning from video.
FloMo fine-tunes the Wan 2.2 TI2V-5B video model (LoRA by default) to predict any
subset of {robot action, future video, motion tokens} from {image, language}.
Action is only one output, so one model learns from robot data and from action-free
human video. The paper trains on EgoVerse (egocentric human video), MolmoAct2, and
RealYAM (real robot), then fine-tunes on RoboTwin 2.0 (simulation).
Contents: Install · Pretrained checkpoints · Inference · Fine-tune on RoboTwin 2.0 · Train on your own data · Preprocessing · Motion tokens · Code layout
Needs a CUDA GPU and Python 3.10. Run all commands from the repo root.
conda create -n flomo python=3.10 -y && conda activate flomo
git clone https://github.com/pairlab/flomo.git && cd flomo
# Backbone code, imported from the repo root
git clone https://github.com/Wan-Video/Wan2.2.git
git clone https://github.com/dreamzero0/dreamzero.git
# PyTorch (cu128 shown; Blackwell needs CUDA 12.8+), then FloMo
pip install "torch==2.10.*" "torchvision==0.25.*" --index-url https://download.pytorch.org/whl/cu128
pip install -e .
# Wan VAE and text-encoder weights
hf download Wan-AI/Wan2.2-TI2V-5B Wan2.2_VAE.pth models_t5_umt5-xxl-enc-bf16.pth \
--local-dir ./wan_checkpointsThe Wan DiT and the umT5 tokenizer download on first use.
To preprocess your own videos, also install the 3D trackers in the repo root:
git clone https://github.com/henry123-boy/SpaTrackerV2.git
git clone https://github.com/facebookresearch/vggt.git
pip install "numpy<2" kornia==0.8.1 moviepy==1.0.0 pycolmap==3.11.1 pyceres==2.4 \
scikit-learn flow_vis jaxtyping \
"utils3d @ git+https://github.com/EasternJournalist/utils3d.git@d3a577acf0a9ad7e513a1416449a07b6f47d967f"- Keep
torchvision<=0.25: SpaTrackerV2 usestorchvision.io.write_video, removed in 0.26. - Keep
numpy<2in the same command, or pip installs numpy 2. - Skip SpaTrackerV2's
requirements.txt: itsxformerspin needs an older torch.
The paper base models are at shitijgovil/FloMo. They share training data and differ only in outputs:
| File | Predicts |
|---|---|
ma/flomo_ma.pt |
motion tokens + action |
va/flomo_va.pt |
video + action |
vma/flomo_vma.pt |
video + motion tokens + action |
hf download shitijgovil/FloMo vma/flomo_vma.pt --local-dir ./flomo_ckptsMA and VMA include the per-dataset motion-token stats that decode_positions needs.
Copies downloaded bef
8000
ore 2026-11-19 lack them; download again.
infer_example.py maps camera images, a task, and the current end-effector pose to a
(T, 14) action chunk: x, y, z, yaw, pitch, roll, gripper per arm.
python infer_example.py --checkpoint ./flomo_ckpts/vma/flomo_vma.pt \
--wan-ckpt-dir ./wan_checkpoints \
--image primary.png --image left.png --image right.png \
--task "pick up the croissant and place it in the oven" \
--ee-pose="x,y,z,yaw,pitch,roll,grip,x,y,z,yaw,pitch,roll,grip"- One
--imageper camera, in orderprimary,left,right. - Use
--ee-pose=...with=; otherwise a negative first value reads as a flag. --normalizer-type:realyam(default) for base checkpoints,robotwinfor RoboTwin fine-tunes.
The script only predicts actions; camera capture and robot control are up to you.
Fine-tunes one task (open_laptop) from a base checkpoint, as in the paper: 50
Aloha-AgileX demo_clean demos, 5,000 updates at batch size 256. Needs the trackers
from Install.
# 1. Download demos
hf download TianxingChen/RoboTwin2.0 dataset/open_laptop/demo_clean.zip \
--repo-type dataset --local-dir <RoboTwin>/data/hf
unzip <RoboTwin>/data/hf/dataset/open_laptop/demo_clean.zip -d <RoboTwin>/data/demo_clean
# 2. Preprocess: 3D tracks and text embeddings
for mode in tracks text_emb; do
python -m preprocessing.preprocess_robotwin mode=$mode task_names=[open_laptop] \
source_dir=<RoboTwin>/data/demo_clean dest_dir=<RoboTwin>/data/flomo
done
# 3. Fine-tune (resume=false: weights only, fresh optimizer and step count)
python train.py --config-name train_robotwin data_root=<RoboTwin>/data \
robotwin_task=open_laptop checkpoint=./flomo_ckpts/vma/flomo_vma.pt \
resume=false run_name=robotwin_open_laptopCheckpoints go to checkpoints/forward_dynamics/robotwin_open_laptop/<epoch>.pt. VA
and VMA warn about unused video_primary keys; this is expected, since the RoboTwin
config does not predict video. Run the result with Inference and
--normalizer-type robotwin; score it with RoboTwin's policy evaluator.
cfg/datasets/ configs choose the data and each dataset's inputs and outputs.
cfg/train.yaml holds training settings; its comments explain every key.
datasets= |
Use |
|---|---|
template (default) |
Annotated example; copy it and fill in the ??? fields. |
flomo |
Paper mixture (EgoVerse + MolmoAct2 + RealYAM). |
robotwin |
One RoboTwin task (with --config-name train_robotwin). |
python train.py datasets=my_dataset run_name=my_run
# From a base checkpoint (weights only)
python train.py datasets=my_dataset checkpoint=./flomo_ckpts/vma/flomo_vma.pt resume=false run_name=my_ft
# Paper mixture, 8 GPUs
torchrun --nproc_per_node=8 train.py datasets=flomo data_root=/path/to/flomo_data run_name=flomo_repro- Resume: with
resume=true(default), a run with the samerun_namecontinues from its last checkpoint. - Overrides: set any key on the command line, e.g.
optim.lr=5e-5 batch_size=64 gpu_max_bs=16.batch_sizeis global;gpu_max_bsis the per-GPU micro-batch. - Cache: the first run caches per-dataset motion stats in
~/.cache/flomo/track_stats(override withFLOMO_CACHE_DIR). flomo.yamldefaults: thedataset_defaultsanchor expands before command-line overrides, sodatasets.0.args.sample_stride=2changes only the first dataset.
Each dataset has preprocessing/preprocess_<dataset>.py and
cfg/preprocessing/preprocess_<dataset>.yaml. mode= selects a stage:
| Dataset | Modes |
|---|---|
| RoboTwin | tracks, text_emb, vae_latent_h16 |
| RealYAM | tracks, text_emb, vae_latent_h16 |
| MolmoAct2 | tracks, video, text, text_emb, vae_latent, vae_latent_h16, actions |
| EgoVerse | tracks, video, text, text_emb, meta, vae_latent_h16 |
-
tracksmakes the 3D tracks for motion tokens;text_embcaches umT5 embeddings. -
EgoVerse: run
videobeforetracksandvae_latent_h16, andtextbeforetext_emb.datasets=flomoneedsvideo,tracks,vae_latent_h16,text, andtext_emb. Set trainingdata_rootto the preprocessingdest_dir; dataset names are<output_name>_<split>. -
MolmoAct2 subsets: build an episode index, select hours, then preprocess:
python -m preprocessing.preprocess_molmoact2 index --index-dir /idx build --source-dir /raw python -m preprocessing.preprocess_molmoact2 index --index-dir /idx select --name subset --hours 10 python -m preprocessing.preprocess_molmoact2 index_path=/idx/subset/index.jsonl mode=video
Motion tokens are dense 3D scene flow, encoded by the frozen Wan VAE as a 3-channel video:
- Track (
preprocessing/SpatialTracker.py): SpatialTrackerV2 tracks a 64×64 grid (4,096 points) in 3D, in the camera frame of each window's first frame, restarting every 16 frames. - Load (
BaseDataset.load_tracks): withsupervision=cumulative_disp, each 16-frame window becomes displacement from frame 0:(1, 16, 4096, 3). - Encode (
MotionTokensWanPretrainedModality): an XYZ grid(3, 15, 64, 64)is resized to 256, scaled per dataset to[-1, 1](flomo/utils/motion_stats.py), and VAE-encoded to a(48, 5, 16, 16)latent, which the Wan patch embedding tokenizes.
supervision=point_tracks and optical_flow encode per-frame changes instead. All
datasets that output motion tokens must use the same mode.
train.py # training entry point (Hydra)
infer_example.py # single-step action inference example
cfg/
train.yaml, train_robotwin.yaml
datasets/ # template, flomo, robotwin
preprocessing/ # one config per preprocess_<dataset>.py
flomo/
models/FloMo.py # Wan trunk, token order, flow-matching timesteps, _build_modality
models/modalities.py # Modality contract + image/video/text/action/motion-token modalities
loaders/ # BaseDataset, per-dataset readers, cotrain_loader (sampling, batching)
utils/ # normalizers, motion stats, Wan encoding, training, visualization
preprocessing/ # trackers (SpatialTracker.py, VGGT.py), writers, per-dataset pipelines
- New dataset: subclass
BaseDatasetand register it incotrain_loader._dataset_class. - New modality: implement the
Modalitycontract inmodels/modalities.pyand add akindbranch to_build_modality.
MIT. See LICENSE.