PyPI · Stable docs · Development docs · 中文
miniVERL runs a validated subset of verl experiment semantics on one NVIDIA GPU. Give it a resolved verl-shaped config and Parquet prompts; its versioned compiler produces a reviewable local plan, executes actor/reference/teacher/ reward roles in phases, and publishes portable PEFT and data artifacts.
The current development line covers PPO/GAE, GRPO, Dr.GRPO, RLOO and
REINFORCE++ against official verl v0.9.0 (483b8a00). PPO uses an independent
trainable critic with its own optimizer and checkpoint state; actor KL, entropy
regularization, grouped rollouts, task rewards and a pinned sequence-classifier
reward role share the same provenance model. The established verl v0.8.0 OPD
profiles remain available for direct GKD and sampled-k1 distillation.
PyPI v0.13.0 is stable; main is development.
Install the CUDA-enabled PyTorch build that matches your machine, then:
python -m pip install "miniverl[train,cuda]"
miniverl data sample --task-rewards --rows 8 --out data/rl-prompts.parquet
miniverl import-verl --profile verl-rl-v0.9-single-gpu-v2 \
--config examples/verl-rl-v0.9-single-gpu-ppo.yaml --out local-ppo.yaml
miniverl validate local-ppo.yaml
miniverl train local-ppo.yaml --dry-runThe importer writes local-ppo.import-report.json beside the native recipe.
It accounts for every accepted source field and records how distributed
resource settings lower to one process and one device. Remove --dry-run to
load the pinned Qwen3-0.6B actor and execute the two-iteration example.
The [train,cuda] extra installs the ML and quantization stack; select PyTorch's
CUDA build separately with the PyTorch installer.
The single-GPU guide covers memory planning and the
maintainer-measured RTX 4080 environment.
- A field-by-field compiler report. Experiment fields retain their meaning; physical distribution fields receive an explicit one-GPU lowering.
- Policy-bound trajectories. Group/sample identity, generated-token spans, behavior log-probabilities and policy version travel together.
- Auditable rewards and objectives. Reward components, group advantages, reference KL, clipping, entropy and update metrics are structured data.
- Exact recovery. Transactional manifests and checkpoints restore policy, optimizer, cursor, RNG and reward identities before another rollout.
- Portable outputs. PEFT adapters, safetensors, Parquet, resolved config and typed provenance can move into a larger workflow.
One GPU is treated as a temporal scheduler for logical roles. RL generates complete prompt groups, scores outcomes, evaluates optional reference and reward-model roles, computes the pinned advantage estimator, and updates the actor. PPO adds a separate critic phase with clipped value updates. OPD uses the same trajectory and checkpoint foundations, with teacher targets supplying the learning signal.
| Goal | First command | Primary artifact | Next step |
|---|---|---|---|
| Run local RL | miniverl import-verl --profile verl-rl-v0.9-single-gpu-v2 --config verl-ppo.yaml --out local.yaml |
native recipe + compatibility report | RL quickstart |
| Run local OPD | miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml --out plan.json |
immutable execution plan | OPD quickstart |
| Fit your GPU | miniverl plan --config verl-opd.yaml --probe |
measured placement plan | Hardware planning |
| Hand off artifacts | miniverl export-verl --run runs/my-run --target-verl v0.9.0 --out scaleout |
actor, critic, Parquet + config bundle | Compatibility contract |
Native recipes also support SFT, DPO and offline KD, plus calculator, JSON navigation, read-only SQLite and custom tool environments.
| Experiment surface | Local status | Contract |
|---|---|---|
| PPO / GAE | semantically conformant | independent critic, clipped value loss, actor/critic exact resume |
| GRPO / Dr.GRPO | semantically conformant | verl v0.9 group statistics and vanilla clipped policy loss |
| RLOO / REINFORCE++ | semantically conformant | verl v0.9 advantage and masking rules |
Grouped n > 1 rollouts |
supported | complete groups, stable sample seeds and behavior-policy identity |
| Task rewards and trained RM | supported | built-ins, environment verifier, trusted Python API, or pinned HF sequence classifier |
| Actor KL and entropy | semantically conformant | sampled-token reference KL and entropy regularization |
| Direct GKD / sampled-k1 OPD | supported | pinned verl v0.8 profiles with teacher targets |
| Ray, FSDP/FSDP2, Megatron, TP/PP/DP > 1 | distributed-only | these change physical scale, not the local objective |
The generated v0.9 PPO compatibility record
binds every example field to its source value, local target, classification and
compiler-rule digest. miniverl import-verl accepts a resolved documented
subset, rejects unknown algorithm-changing fields, and never substitutes a
dataset or reward implementation.
The published Qwen3-0.6B/1.7B OPD workload consumed 32 distinct prompts and completed eight current-policy updates at 3.1914 GiB peak reserved VRAM on an RTX 4080. A matched interruption after update four reproduced byte-identical trajectories, adapter and optimizer tensors. The SmolLM2-360M/1.7B workload completed the same shape at 1.4961 GiB. System records contain configs, hashes and phase timings.
Rollout Runtime v2 measured hf_cached and managed vLLM across 24 RTX 4080
cells spanning response length, sampling mode and n=1/4. See the
runtime report. Evidence for the new
RL family is published through the exact-wheel release qualification rather
than presented as a task-quality comparison.
The scientific reports preserve their original outcomes and frozen inputs: calculator protocol, RecoveryBench, Alignment Lab, and the External Alignment Gate. They are scoped studies, separate from runtime qualification.
miniVERL targets one local process and one NVIDIA CUDA GPU. The compiler distinguishes exact or conformant semantics, local physical lowering, distributed-only settings, feasible work not yet implemented, and technically unsupported inputs. The compatibility policy is the complete matrix; limitations collects architecture, measurement, security and generalization boundaries in one place. miniVERL is an independent Apache-2.0 project.
git clone https://github.com/DaoyuanLi2816/mini-verl.git
cd mini-verl
python -m pip install -e ".[dev,train]"
pytest -q -m "not gpu and not network"See CONTRIBUTING.md, CHANGELOG.md, CITATION.cff, the guide for verl users, SECURITY.md and the Apache-2.0 license.