8000 GitHub - andylin-hao/RLinf: RLinf is a flexible, scalable and open-source infrastructure designed for reinforcement-learning (RL) post-training of foundation models — including large language models (LLMs), vision-language models (VLMs), and vision-language-action (VLAs) models. · GitHub
[go: up one dir, main page]
Skip to content
andylin-hao
RLinf
Public
forked from RLinf/RLinf

About

RLinf is a flexible, scalable and open-source infrastructure designed for reinforcement-learning (RL) post-training of foundation models — including large language models (LLMs), vision-language models (VLMs), and vision-language-action (VLAs) models.

Resources

Readme
License

Code of conduct

Code of conduct

Contributing

Contributing
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags
 
 

Repository files navigation

RLinf-logo
Hugging Face Ask DeepWiki
Slack Channel

English 简体中文

RLinf: Reinforcement Learning Infrastructure for Embodied and Agentic AI

RLinf is a flexible and scalable open-source RL infrastructure designed for Embodied and Agentic AI. The 'inf' in RLinf stands for Infrastructure, highlighting its role as a robust backbone for next-generation training. It also stands for Infinite, symbolizing the system’s support for open-ended learning, continuous generalization, and limitless possibilities in intelligence development.

RLinf-overview

What's NEW!

  • [2026/11] 🔥 RLinf supports RL on π₀.₅ in GenieSim with PPO and GRPO. Doc: RL with GenieSim.
  • [2026/11] 🔥 RLinf supports SO-101 joint-space demonstration collection, OpenPI π₀.₅ supervised fine-tuning, and local evaluation. Doc: SO-101.
  • [2026/11] 🔥 RLinf supports Psi0 VLA and the SIMPLE humanoid simulator. Docs: Psi0, SIMPLE.
  • [2026/11] 🔥 RLinf supports PhyAI as an in-process rollout backend for π₀.₅ PPO training on LIBERO, including hot weight synchronization. Doc: PhyAI rollout backend.
  • [2026/10] 🎉 RLinf v0.4 is released, taking RL out of simulation and into agile control of the physical world. Highlights: a modular real-robot system (rlinf.robotics), real-robot SOTA post-training algorithms (RLT, STEAM, OPD, RTC), full-modality models (native PyTorch OpenPI π₀ / π₀.₅, Cosmos3, FastWAM, π₀-FAST, Evo-1, MolmoAct2, SD3 / Wan2.2), and eight accelerator backends. Release notes: RLinf v0.4.
  • [2026/10] 🎉 RLinf introducing CEL (Collective Experience Learning): robot fleets improve a shared policy from their own deployment experience without additional expe 8000 rt demonstrations or corrective actions, validated on up to 96 real robots across 5 real-world tasks. Project page: CEL, paper: arXiv:2610.25736. Code will release soon.
  • [2026/10] 🔥 RLinf supports MetaX (MACA) GPUs. PR: MetaX GPU support (#1656).
  • [2026/10] 🔥 RLinf supports RL on LingbotVLA 2.0 in RoboTwin with PPO. Doc: LingbotVLA 2.0.
  • [2026/09] 🔥 RLinf refactors real-world robots and environments, using a unified robotics interface to manage hardware, tasks, and rollout wrappers separately. Docs: Robotics Interface, Robotics Architecture, Real-World Tasks and Environments, Add a Robot, Add a Real-World Task, Teleoperation.
  • [2026/09] 🔥 RLinf supports RL on π₀-FAST. Doc: π₀-FAST.
  • [2026/09] 🔥 RLinf supports Biren GPUs. PR: Biren GPU support (#1581).
  • [2026/09] 🔥 RLinf supports ApxInf-accelerated evaluation of π₀.₅ on LIBERO. RLinf keeps the native OpenPI transform chain and enters the engine through its L1 inference interface. Doc: ApxInf LIBERO eval.
  • [2026/09] 🔥 RLinf supports FSDP supervised fine-tuning and batched LIBERO evaluation for FastWAM. Doc: FastWAM Evaluation and SFT.
  • [2026/08] 🔥 RLinf supports SFT and SGLang-based evaluation of NVIDIA's omnimodal world model, Cosmos3. Docs: Cosmos3 SFT, SGLang eval.
  • [2026/08] 🎉 RLinf is officially welcomed into the PyTorch Ecosystem! We will continue to bring scalable embodied and agentic RL to PyTorch users, pushing model intelligence into the real world. Blog: PyTorch Ecosystem Landscape Q3 Update.
  • [2026/08] 🎉 Isaac Lab v3.0.0 officially adopts RLinf as its reinforcement learning (RL) training infrastructure. Doc: RLinf on Isaac Lab.
  • [2026/08] 🔥 RLinf integrates Diffusion-NFT for SD3 and Wan2.2 video generation models. Doc: RL for Video Generation Models.
  • [2026/08] 🔥 RLinf supports Moore Threads (MUSA), Huawei Ascend (CANN), and AMD (ROCm). See the hardware support matrix for model/environment combinations and setup instructions.
  • [2026/08] 🔥 RLinf supports GRPO training for Moonlight-16B-A3B (DeepSeek-V3 MLA + MoE). Doc: Moonlight-16B GRPO.
  • [2026/08] 🔥 RLinf supports MolmoAct2 evaluation on LIBERO. Doc: MolmoAct2.
  • [2026/08] 🎉 Two papers accepted to OSDI 2026: RLinf and DynaRL. DynaRL dynamically reallocates compute, memory, and communication across heterogeneous RL components to improve end-to-end training throughput. Doc: DynaRL.
  • [2026/08] 🎉 Two papers accepted to RSS 2026: RLinf-VLA and USER. RLinf-VLA is a unified and efficient framework for scalable RL training of VLA models across diverse architectures, algorithms, and simulators. Doc: RLinf-VLA. USER is a unified and extensible system for real-world online policy learning that treats robots as first-class resources alongside GPUs. Doc: USER.
  • [2026/08] 🎉 One paper accepted to NSDI 2027: FUSCO. FUSCO accelerates MoE All-to-All communication by fusing data transformation and communication for high-performance distributed data shuffling. Doc: FUSCO.
  • [2026/07] 🔥 RLinf supports RTC in both simulation (LIBERO) and real-world (Franka). Doc: RTC.
  • [2026/07] 🔥 RLinf supports Evo-1 full-parameter SFT and GRPO fine-tuning on the LIBERO simulator. Doc: Evo-1.
  • [2026/07] 🔥 RLinf reimplements π₀ and π₀.₅ in PyTorch with numerical behavior aligned with the JAX reference implementations. Doc: OpenPI.
  • [2026/07] 🔥 RLinf supports OPD for online policy distillation of OpenVLA-OFT on LIBERO. Doc: OPD.
  • [2026/07] 🎉 RLinf v0.3 is released with major upgrades in the real-world RL full pipeline (data collection → SFT → RL → deployment), more simulators and SOTA models, and system-level optimizations. Release notes: RLinf v0.3.
  • [2026/07] 🔥 RLinf supports RLT for online RL fine-tuning of VLA policies. Doc: RLT.
More updates

Key Features

RLinf has high flexibility to support diverse RL training workflows (PPO, GRPO, SAC and so on), while hiding the complexity of distributed programming. Users can easily scale RL training to a large number of GPU nodes without modifying code, meeting the increasing demand of computation for RL training.

The high flexibility allows RLinf to explore more efficient scheduling and execution. The hybrid execution mode for embodied RL achieves up to 2.434× throughput compared to existing frameworks.

Training and Inference Engines

RLinf updates models with a training engine and runs rollout and evaluation with an inference engine. FSDP adapts quickly to new models and algorithms, and Megatron-LM and Megatron-Bridge are built for large-scale training. Agentic AI includes reasoning tasks such as math and VQA. PhyAI and ApxInf currently support π₀.₅ only.

Training Inference
FSDP Megatron-LM Megatron-Bridge HuggingFace SGLang vLLM PhyAI ApxInf
Embodied AI ✅—✅ ✅✅—✅✅
Agentic AI ✅✅✅ —✅✅——

Examples

Embodied AI

RLinf supports SFT, simulation RL, and real-world RL for World Action Models (WAM) and Vision-Language-Action Models (VLA). The current support list is as follows:

Simulators Models Algorithms
Real-world Robotics Data Collection

Hardware Support

RLinf hides the differences between heterogeneous hardware behind one unified low-level abstraction, so the same training stack runs seamlessly on NVIDIA and AMD GPUs and on domestic accelerators such as Huawei Ascend, Moore Threads, and Kunlunxin. Moving a training job between compute nodes takes zero changes to models and algorithms. Accelerator vendors, in turn, only have to adapt a minimal interface to bring the complete embodied and agentic RL training ecosystem onto their own hardware, with none of the burden of maintaining a customized fork of the framework.

Choose the model and environment combination you need, then follow the corresponding link for detailed hardware instructions. Every hardware backend fully supports all the environments listed in its row.

Model Environment NVIDIA CUDA Huawei Ascend CANN Moore Threads MUSA AMD ROCm
OpenVLA-OFT LIBERO · ManiSkill ✅ ✅ ✅ ✅
OpenVLA-OFT Wan world model ✅ ✅ — —
GR00T N1.5 LIBERO · ManiSkill ✅ ✅ ✅ ✅
π₀ / π₀.₅ (OpenPI) LIBERO · ManiSkill ✅ ✅ ✅ ✅
StarVLA (QwenOFT) LIBERO ✅ ✅ — —
DreamZero SFT (LeRobot datasets) ✅ ✅ — —

Agentic AI

Single-Agent Multi-Agent

Quick Start

Installation: Users can refer to our installation guide to install RLinf. We recommend users to use our provided docker image (i.e., Installation Method 1), as the environment and dependencies of embodied RL are complex.

Run a simple example: After setting up the environment, users can run a simple example of embodied RL with ManiSkill3 simulator following this document.

SOTA RL Training Reproduction: RLinf provides end-to-end recipes that reproduce or match state-of-the-art (SOTA) RL results out of the box—users can directly run our configs and scripts to obtain SOTA performance without custom engineering. Check out our example gallery for more details.

Awesome Community Projects with RLinf

We are excited to see a growing ecosystem of projects building on top of or integrate with RLinf, spanning embodied AI, robotics, and long-horizon agentic systems. Here are some awesome community projects:

  • Qwen-VLA: A unified vision-language-action model that uses RLinf for reinforcement learning to optimize closed-loop task success.
  • i4h-workflows: NVIDIA team open sourced RL-based workflow built on Isaac ecosystem, integrating RLinf for healthcare-oriented embodied intelligence.
  • pi-StepNFT: Extends RLinf for step-level training and optimization of π-series VLA models.
  • Dexbotic: A robotics + RL system integrating RLinf for scalable training and deployment of embodied agents.
  • RoboTwin: A digital twin + robotics platform leveraging RLinf for large-scale embodied RL training.
  • IsaacLab: Official integration of RLinf within IsaacLab, enabling seamless reinforcement learning workflows on top of NVIDIA Isaac Sim based robotics environments.
  • RISE: A robot reinforcement learning framework based on a compositional world model, using RLinf for online reinforcement learning.

💡 Want to feature your project here? Open a PR and we’ll be happy to include it!

Adoption

RLinf is a production-grade, open-source reinforcement learning framework for embodied AI. It is being adopted by leading companies and startups across AI infrastructure and robotics, including AgiBot, X Square Robot, PsiBot, Dexmal, Moore Threads, D-Robotics, DexForce, YinWang, Robbyant and GigaAI.

adoption

✨ If your organization is using RLinf, feel free to reach out or submit a PR to be listed here.

CI Test Status

RLinf has comprehensive CI tests for both the core components (via unit tests) and end-to-end RL training workflows of embodied, agent, and reasoning scenarios. Below is the summary of the CI test status of the main branch:

Test Name Status
unit-tests GitHub Actions Workflow Status
agent-reason-e2e-tests GitHub Actions Workflow Status
embodied-e2e-tests GitHub Actions Workflow Status
scheduler-tests GitHub Actions Workflow Status

Contribution Guidelines

We welcome contributions to RLinf. Before opening a pull request, read the contribution guide and add Signed-off-by to each commit to certify compliance with the Developer Certificate of Origin (DCO). Thank you to everyone who contributes to the project.

Citation and Acknowledgement

If you find RLinf helpful, please cite the paper:

@inproceedings{yu2026rlinf,
  author = {Chao Yu and Yuanqing Wang and Zhen Guo and Hao Lin and Si Xu and Hongzhi Zang and Quanlu Zhang and Yongji Wu and Chunyang Zhu and Junhao Hu and Zixiao Huang and Mingjie Wei and Yuqing Xie and Ke Yang and Bo Dai and Zhexuan Xu and Jiakun Du and Xiangyuan Wang and Xu Fu and Letong Shi and Zhihao Liu and Kang Chen and Weilin Liu and Gang Liu and Boxun Li and Jianlei Yang and Zhi Yang and Guohao Dai and Yu Wang},
  title = {{RLinf}: Flexible and Efficient {Large-Scale} Reinforcement Learning via {Macro-to-Micro} Flow Transformation},
  booktitle = {20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26)},
  year = {2026},
  isbn = {978-1-939133-55-7},
  address = {Seattle, WA},
  pages = {829--846},
  url = {https://www.usenix.org/conference/osdi26/presentation/yu-chao},
  publisher = {USENIX Association},
  month = jul
}

If you use dynamic scheduling in RLinf, please also cite DynaRL:

@inproceedings{wang2026dynarl,
  author = {Yuanqing Wang and Hao Lin and Junhao Hu and Chunyang Zhu and Quanlu Zhang and Zhen Guo and Yuchen Zhang and Xu Fu and Si Xu and Bo Dai and Zixiao Huang and Chao Yu and Boxun Li and Guohao Dai and Zhi Yang and Yu Wang},
  title = {{DynaRL}: Flexible and Dynamic Scheduling of {Large-Scale} Reinforcement Learning Training},
  booktitle = {20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26)},
  year = {2026},
  isbn = {978-1-939133-55-7},
  address = {Seattle, WA},
  pages = {847--862},
  url = {https://www.usenix.org/conference/osdi26/presentation/wang-yuanqing},
  publisher = {USENIX Association},
  month = jul
}

If you use RL+VLA in RLinf, you can also cite our technical report and empirical study paper:

@INPROCEEDINGS{ZangH2-RSS-26,
    AUTHOR    = {Hongzhi Zang AND Mingjie Wei AND Si Xu AND Yongji Wu AND Zhen Guo AND Yuanqing Wang AND Hao Lin AND Peihong Wang AND Hua Yuan AND Yixian Zhang AND Liangzhi Shi AND Yuqing Xie AND Zhexuan Xu AND Zhihao Liu AND Kang Chen AND Wenhao Tang AND Quanlu Zhang AND Weinan Zhang AND Chao Yu AND Yu Wang},
    TITLE     = {{RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models}},
    BOOKTITLE = {Proceedings of Robotics: Science and Systems},
    YEAR      = {2026},
    ADDRESS   = {Sydney, Australia},
    MONTH     = {July},
    DOI       = {10.15607/RSS.2026.XXII.089}
}
@inproceedings{NEURIPS2025_8c0fabe3,
 author = {Liu, Jijia and Gao, Feng and Wei, Bingwen and Chen, Xinlei and Liao, Qingmin and WU, YI and Yu, Chao and Wang, Yu},
 booktitle = {Advances in Neural Information Processing Systems},
 doi = {10.52202/085713-3248},
 editor = {D. Belgrave and C. Zhang and H. Lin and R. Pascanu and P. Koniusz and M. Ghassemi and N. Chen},
 pages = {97121--97151},
 publisher = {Curran Associates, Inc.},
 title = {What Can RL Bring to VLA Generalization? An Empirical Study},
 url = {https://proceedings.neurips.cc/paper_files/paper/2025/file/8c0fabe372177d2aded596be2d3b4544-Paper-Conference.pdf},
 volume = {38, Main Conference},
 year = {2025}
}
@article{chen2025pi_,
  title={$\pi_{\texttt{RL}}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models},
  author={Chen, Kang and Liu, Zhihao and Zhang, Tonghe and Guo, Zhen and Xu, Si and Lin, Hao and Zang, Hongzhi and Li, Xiang and Zhang, Quanlu and Yu, Zhaofei and Fan, Guoliang and Huang, Tiejun and Wang, Yu and Yu, Chao},
  journal={arXiv preprint arXiv:2510.25889},
  year={2025}
}

If you train your policies in physical world with RLinf, you can cite our paper:

@INPROCEEDINGS{ZangH1-RSS-26,
    AUTHOR    = {Hongzhi Zang AND Shu'ang Yu AND Hao Lin AND Tianxing Zhou AND Zefang Huang AND Zhen Guo AND Xin Xu AND Yuze Sheng AND Jiakai Zhou AND Shizhe Zhang AND Feng Gao AND Wenhao Tang AND Yufeng Yue AND Quanlu Zhang AND Xinlei Chen AND Chao Yu AND Yu Wang},
    TITLE     = {{USER: A Unified and Extensible System for Online Real-World Policy Learning in Embodied AI}},
    BOOKTITLE = {Proceedings of Robotics: Science and Systems},
    YEAR      = {2026},
    ADDRESS   = {Sydney, Australia},
    MONTH     = {July},
    DOI       = {10.15607/RSS.2026.XXII.037}
}

If you use World Model + VLA + RL in RLinf, you can cite our paper:

@article{jiang2026wovr,
  title={WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL},
  author={Zhennan Jiang and Shangqing Zhou and Yutong Jiang and Zefang Huang and Mingjie Wei and Yuhui Chen and Tianxing Zhou and Zhen Guo and Hao Lin and Quanlu Zhang and Yu Wang and Haoran Li and Chao Yu and Dongbin Zhao},
  year={2026},
  journal={arXiv preprint arXiv:2602.13977},
  url={https://arxiv.org/abs/2602.13977},
}

If you use RL-based sim-real co-training in RLinf, you can cite our paper:

@article{shi2026rlinf,
  title={Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models},
  author={Shi, Liangzhi and Chen, Shuaihang and Gao, Feng and Chen, Yinuo and Chen, Kang and Zhang, Tonghe and Zang, Hongzhi and Zhou, Jiakai and Zhang, Weinan and Yu, Chao and Wang, Yu},
  journal={arXiv preprint arXiv:2602.12628},
  year={2026},
  url={https://arxiv.org/abs/2602.12628},
}

If you use WideSeek-R1 in RLinf, you can cite our paper:

@article{xu2026wideseek,
  title={WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning},
  author={Xu, Zelai and Xu, Zhexuan and Zhang, Ruize and Zhu, Chunyang and Yu, Shi and Liu, Weilin and Zhang, Quanlu and Ding, Wenbo and Yu, Chao and Wang, Yu},
  journal={arXiv preprint arXiv:2602.04634},
  year={2026},
}

If you use FUSCO for MoE communication in RLinf, you can cite our paper:

@article{zhu2025fusco,
  title={FUSCO: High-Performance Distributed Data Shuffling via Transformation-Communication Fusion},
  author={Zhu, Zhuoran and Zhu, Chunyang and Lin, Hao and Fu, Xu and Zhou, Yiming and Zhang, Quanlu and Li, Zhenhua and Qian, Feng and Yu, Chao and Li, Boxun and Dai, Guohao and Wang, Yu},
  journal={arXiv preprint arXiv:2512.22036},
  year={2025}
}

Acknowledgements RLinf has been inspired by, and benefits from, the ideas and tooling of the broader open-source community. In particular, we would like to thank the teams and contributors behind veRL, vLLM, AReaL, Megatron-LM, SGLang, and PyTorch Fully Sharded Data Parallel (FSDP), and if we have inadvertently missed your project or contribution, please open an issue or a pull request so we can properly credit you.

Contact: We welcome applications from Postdocs, PhD/Master's students, and interns. Join us in shaping the future of RL infrastructure and embodied AI!

About

RLinf is a flexible, scalable and open-source infrastructure designed for reinforcement-learning (RL) post-training of foundation models — including large language models (LLMs), vision-language models (VLMs), and vision-language-action (VLAs) models.

Resources

Readme
License

Code of conduct

Code of conduct

Contributing

Contributing
Activity

Stars

2 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages
0