verl for a single consumer GPU.
Train with PPO, GRPO or on-policy distillation on one NVIDIA GPU. Bring your verl configs, follow the training, resume from a checkpoint and take the resulting models into your next workflow.
Start training{ .md-button .md-button--primary } Choose a workflow{ .md-button }
Install the matching CUDA-enabled build from the PyTorch installer, then:
python -m pip install "miniverl[train,hydra]"
miniverl data sample --reward-profile target-length --rows 8 --out data/rl-prompts.parquet
miniverl run --example hydra-ppo --bind reward.provider=target_length --dry-run
miniverl run --example hydra-ppo --bind reward.provider=target_length --run-id local-ppoThis Qwen3-0.6B example uses a simple length reward you can inspect in
rewards.jsonl. The dry run checks the configuration before downloading models.
Continue with inspect → resume → export.
Run PPO or GRPO using your verl config and local data.
miniverl run --example hydra-ppo \
--bind reward.provider=target_length --dry-runArtifact: training recipe and configuration report.
Next: Bring your own config
Train a student on its own rollouts with feedback from a teacher.
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
--config builtin:qwen3-0.6b-1.7b-opdArtifact: a memory and execution plan for the student/teacher pair.
Export your model, data and config for handoff to verl.
miniverl export-verl --run runs/my-run \
--target-verl v0.9.0 --out scaleoutArtifact: PEFT, Parquet and config files with a setup checklist.
Next: Review the bundle
miniVERL schedules training roles in phases so actor, critic, reference, reward and teacher can share one GPU. Choose a model and placement strategy with the single-GPU guide and hardware planner. Native recipes also offer SFT, DPO and offline KD.
- OPD on an RTX 4080: update timing, memory and exact recovery.
- Rollout Runtime v2: 24 measurements across rollout settings.
- Alignment Lab: a saturated tool-policy case study.
- RecoveryBench: tool recovery outcomes.
- External Alignment Gate: frozen external evaluation results.
miniVERL is an independent project for single-GPU NVIDIA CUDA training. The compatibility matrix lists supported verl versions, profiles and handoff stages. Limitations collects hardware, scientific and security scope. See comparisons for help choosing a training workflow.