Run common resolved verl PPO/GRPO configs directly on one NVIDIA GPU. The local runtime produces policy-bound trajectories, transactional checkpoints and portable PEFT/Parquet artifacts.
Run PPO or GRPO locally{ .md-button .md-button--primary } Choose a workflow{ .md-button }
resolved verl-shaped config
↓ versioned compatibility compiler
field report + validated native recipe
↓ temporal role scheduler
actor rollout → reward / reference / teacher → actor update
↘ PPO critic/value update
↓
PEFT + Parquet + config + typed provenance
The current RL profile targets official verl v0.9.0 at commit
483b8a009ba3a97563edee3a19887e4862b8094a. It runs PPO/GAE, GRPO, Dr.GRPO,
RLOO and REINFORCE++ with grouped samples, task rewards, trained reward-model
inference and optional fixed reference-policy KL. The existing verl v0.8.0 profiles run direct GKD and
sampled-k1 OPD with explicit teacher targets.
python -m pip install torch --index-url https://download.pytorch.org/whl/cu130
python -m pip install "miniverl[train]"
miniverl data sample --reward-profile target-length --rows 8 --out data/rl-prompts.parquet
miniverl run --example ppo --bind reward.provider=target_length --dry-run
miniverl run --example ppo --bind reward.provider=target_length --run-id local-ppoReplace --example ppo with your resolved verl YAML to use an existing experiment.
--dry-run checks semantic compatibility without model downloads; execution
resolves snapshots, filters prompts, derives the epoch schedule and checks GPU
capacity. Continue with direct run → inspect → resume → handoff.
Bring a resolved v0.9-shaped config and run PPO or a grouped critic-free algorithm on one CUDA device.
miniverl run verl-rl.yaml --dry-runArtifact: native recipe plus field-by-field compatibility report.
Next: Direct verl configs
Compile a direct-GKD or sampled-k1 OPD profile, then inspect teacher targets, trajectory provenance and the final adapter.
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
--config verl-opd.yaml --out plan.jsonArtifact: immutable plan, teacher cache, trajectories and PEFT adapter.
Next: OPD quickstart
Package a local run into standard PEFT, safetensors, Parquet and config artifacts with a separate readiness report.
miniverl export-verl --run runs/my-run --target-verl v0.9.0 --out scaleoutArtifact: checksummed portable bundle and compatibility states.
Next: Inspect and hand off
| Surface | Local status |
|---|---|
| PPO/GAE | independent actor/critic updates with exact dual-role resume |
| GRPO, Dr.GRPO, RLOO, REINFORCE++ | verl v0.9-conformant estimator and policy-loss semantics |
| Grouped rollout, task rewards, trained RM, actor KL/entropy | supported and provenance-bound |
| Direct GKD and sampled-k1 OPD | supported through pinned verl v0.8 profiles |
| SFT, DPO, offline KD, tool environments | native recipe workflows |
| Ray, FSDP/FSDP2, Megatron, TP/PP/DP > 1 | distributed-only |
Read compatibility for the complete field and semantic contract. Limitations collects measurement, architecture, security and generalization boundaries.
The RTX 4080 systems record covers current-policy OPD updates, exact resume and rollout backends. The v0.9 RL path adds upstream numerical/gradient conformance, CPU integration and exact-wheel GPU qualification. Runtime qualification is kept separate from task-quality claims.