verl is a flexible, efficient and production-ready RL training library for large language models (LLMs).
verl is the open-source version of HybridFlow: A Flexible and Efficient RLHF Framework paper.
verl is flexible and easy to use with:
-
Easy extension of diverse RL algorithms: The hybrid-controller programming model enables flexible representation and efficient execution of complex Post-Training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code.
-
Seamless integration of existing LLM infra with modular APIs: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc
-
Flexible device mapping: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.
-
Readily integration with popular HuggingFace models
verl is fast with:
-
State-of-the-art throughput: SOTA LLM training and inference engine integrations and SOTA RL throughput.
-
Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.
- [2025/03] DAPO is the open-sourced SOTA RL algorithm that achieves 50 points on AIME 2024 based on the Qwen2.5-32B pre-trained model, surpassing the previous SOTA achieved by DeepSeek's GRPO (DeepSeek-R1-Zero-Qwen-32B). 8000 DAPO's training is fully powered by verl and the reproduction code is publicly available now.
- [2025/03] We will present verl(HybridFlow) at EuroSys 2025. See you in Rotterdam!
- [2025/03] We introduced the programming model of verl at the vLLM Beijing Meetup and verl intro and updates at the LMSys Meetup in Sunnyvale mid March.
- [2025/02] verl v0.2.0.post2 is released! See release note for details.
- [2025/01] Doubao-1.5-pro is released with SOTA-level performance on LLM & VLM. The RL scaling preview model is trained using verl, reaching OpenAI O1-level performance on math benchmarks (70.0 pass@1 on AIME).
more...
- [2025/02] We presented verl in the Bytedance/NVIDIA/Anyscale Ray Meetup. See you in San Jose!
- [2024/12] verl is presented at Ray Forward 2024. Slides available here
- [2024/10] verl is presented at Ray Summit. Youtube video available.
- [2024/12] The team presented Post-training LLMs: From Algorithms to Infrastructure at NeurIPS 2024. Slides and video available.
- [2024/08] HybridFlow (verl) is accepted to EuroSys 2025.
- FSDP and Megatron-LM for training.
- vLLM, SGLang(experimental) and HF Transformers for rollout generation.
- Compatible with Hugging Face Transformers and Modelscope Hub: Qwen-2.5, Llama3.1, Gemma2, DeepSeek-LLM, etc
- Supervised fine-tuning.
- Reinforcement learning with PPO, GRPO, ReMax, Reinforce++, RLOO, PRIME, etc.
- Support model-based reward and function-based reward (verifiable reward)
- Support vision-language models (VLMs) and multi-modal RL
- Flash attention 2, sequence packing, sequence parallelism support via DeepSpeed Ulysses, LoRA, Liger-kernel.
- Scales up to 70B models and hundreds of GPUs.
- Experiment tracking with wandb, swanlab, mlflow and tensorboard.
- DeepSeek 671b optimizations with Megatron v0.11
- Multi-turn rollout optimizations
Quickstart:
Running a PPO example step-by-step:
- Data and Reward Preparation
- Understanding the PPO Example
Reproducible algorithm baselines:
For code explanation and advance usage (extension):
- PPO Trainer and Workers
- Advance Usage and Extension
Blogs from the community
The performance is essential for on-policy RL algorithm. We write a detailed performance tuning guide to allow people tune the performance. See here for more details.
veRL now supports vLLM>=0.8.0 when using FSDP as the training backend. Please refer to this document for installation guide and more information.
If you find the project helpful, please cite:
- HybridFlow: A Flexible and Efficient RLHF Framework
- A Framework for Training Large Language Models for Code Generation via Proximal Policy Optimization
@article{sheng2024hybridflow,
title = {HybridFlow: A Flexible and Efficient RLHF Framework},
author = {Guangming Sheng and Chi Zhang and Zilingfeng Ye and Xibin Wu and Wang Zhang and Ru Zhang and Yanghua Peng and Haibin Lin and Chuan Wu},
year = {2024},
journal = {arXiv preprint arXiv: 2409.19256}
}verl is inspired by the design of Nemo-Aligner, Deepspeed-chat and OpenRLHF. The project is adopted and supported by Anyscale, Bytedance, LMSys.org, Shanghai AI Lab, Tsinghua University, UC Berkeley, UCLA, UIUC, University of Hong Kong, and many more.
- TinyZero: a reproduction of DeepSeek R1 Zero recipe for reasoning tasks
- DAPO: the fully open source SOTA RL algorithm that beats DeepSeek-R1-zero-32B
- SkyThought: RL training for Sky-T1-7B by NovaSky AI team.
- Easy-R1: Multi-modal RL training framework
- OpenManus-RL: LLM Agents RL tunning framework for multiple agent environments.
- deepscaler: iterative context scaling with GRPO
- PRIME: Process reinforcement through implicit rewards
- RAGEN: a general-purpose reasoning agent training framework
- Logic-RL: a reproduction of DeepSeek R1 Zero on 2K Tiny Logic Puzzle Dataset.
- Search-R1: RL with reasoning and searching (tool-call) interleaved LLMs
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
- DeepRetrieval: Hacking Real Search Engines and retrievers with LLMs via RL for information retrieval
- cognitive-behaviors: Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
- MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
- DeepEnlighten: Reproduce R1 with social reasoning tasks and analyze key findings
- Code-R1: Reproducing R1 for Code with Reliable Rewards
- self-rewarding-reasoning-LLM: self-rewarding and correction with generative reward models
- critic-rl: LLM critics for code generation
- DQO: Enhancing multi-Step reasoning abilities of language models through direct Q-function optimization
- FIRE: Flaming-hot initiation with regular execution sampling for large language models
Contributions from the community are welcome! Please checkout our roadmap and release plan.
Code formatting
< 5A66 /a>We use yapf (Google style) to enforce strict code formatting when reviewing PRs. To reformat you code locally, make sure you installed latest yapf
pip3 install yapf --upgradeThen, make sure you are at top level of verl repo and run
bash scripts/format.shWe are HIRING! Send us an email if you are interested in internship/FTE opportunities in MLSys/LLM reasoning/multimodal alignment.