8000 Release v0.13.0: executable single-GPU PPO by DaoyuanLi2816 · Pull Request #115 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@ jobs:
"environments/known-good-rtx4080-wsl2-cu130.txt",
"scripts/run_v011_profile_qualification.py",
"scripts/run_v012_rl_qualification.py",
"scripts/run_v013_ppo_qualification.py",
"examples/verl-rl-v0.9-single-gpu-ppo.yaml",
"docs/generated/verl-rl-v0.9-ppo-compatibility.json",
"LICENSE",
"README.md",
):
Expand Down
12 changes: 10 additions & 2 deletions .github/workflows/gpu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
workflow_dispatch:
inputs:
qualification_level:
description: "Release smoke, or the complete v0.12 WSL2 qualification"
description: "Release smoke, or the complete WSL2 release qualification"
type: choice
required: true
default: smoke
Expand Down Expand Up @@ -193,6 +193,13 @@ jobs:
--output "$full_dir/v012-rl.json" \
--work "$full_work/v012-rl" \
"${offline[@]}"
"$QUALIFICATION_PYTHON" scripts/run_v013_ppo_qualification.py \
--commit "$GITHUB_SHA" \
--wheel "$QUALIFICATION_WHEEL" \
--wheel-sha256 "$QUALIFICATION_WHEEL_SHA256" \
--output "$full_dir/v013-ppo.json" \
--work "$full_work/v013-ppo" \
"${offline[@]}"
test -z "$(git status --porcelain)"
"$QUALIFICATION_PYTHON" scripts/benchmark_rollout_runtime_v2_selected.py \
--backend hf_cached \
Expand All @@ -215,7 +222,8 @@ jobs:
--hf-cached-runtime "$full_dir/hf-cached-runtime.json" \
--vllm-runtime "$full_dir/vllm-runtime.json" \
--hf-reference benchmarks/results/rollout-runtime-v2-hf-reference.json \
--v012-rl "$full_dir/v012-rl.json"
--v012-rl "$full_dir/v012-rl.json" \
--v013-ppo "$full_dir/v013-ppo.json"
- name: Upload full qualification records
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: ${{ success() && inputs.qualification_level == 'full' }}
Expand Down
32 changes: 31 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,35 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

## [0.13.0] - 2026-09-08

### PPO and full local RL

- Added the immutable `verl-rl-v0.9-single-gpu-v2` compiler for PPO/GAE while
preserving the released v1 compiler and generated artifact bytes.
- Added an independent trainable critic with a scalar value head, clipped value
updates, dedicated optimizer/schedule, physical-microbatch OOM retry and
transactional checkpoint state.
- Connected actor-loss reference KL and entropy regularization to the sampled
policy objective, with scalar, gradient and optimizer-step conformance against
official verl v0.9.0 at `483b8a00`.
- Added a pinned Hugging Face sequence-classifier reward role with deterministic
batching, typed identity and phased CPU/GPU residency.
- Extended exact resume to reproduce actor and critic tensors and counters, and
extended the exact-wheel RTX 4080 release gate with PPO, reward-model and
v0.9 handoff evidence.

### Compatibility and architecture

- Added v0.9 RL export for actor PEFT, independent critic safetensors, Parquet,
resolved semantics and checksummed provenance. The launch template remains
inactive until upstream reward, base snapshots and distributed settings are
supplied.
- Taught `bridge doctor` to identify and locally inspect the v0.9 PPO bundle,
including its pinned requirement file, critic checkpoint and reward interface.
- Extracted PPO value/update phase ownership from the trainer facade into
`PPOPhaseRuntime`, preserving public trainer and checkpoint compatibility.

## [0.12.0] - 2026-09-07

### Single-GPU verl RL
Expand Down Expand Up @@ -1177,7 +1206,8 @@ Same-tokenizer only; one trajectory per forward pass; `swap` unavailable for
quantized models; only Qwen3 and Qwen2 architectures tested; single-seed GPU
results. The full list is in `docs/limitations.md`.

[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.12.0...HEAD
[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.13.0...HEAD
[0.13.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.12.0...v0.13.0
[0.12.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.11.0...v0.12.0
[0.11.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.10.1...v0.11.0
[0.10.1]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.10.0...v0.10.1
Expand Down
2 changes: 1 addition & 1 deletion CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ cff-version: 1.2.0
title: "miniVERL: A bounded single-GPU runtime for verl-style OPD"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.12.0
version: 0.13.0
date-released: 2026-09-08
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
Expand Down
31 changes: 16 additions & 15 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,26 @@ current product and evidence state rather than repeating release history.

Last updated: 2026-09-08.

Canonical release state: stable `v0.12.0` (`94216828ecab77d5f87a541563f5c2d8951d5a33`), development `0.12.1.dev0`.
Canonical release state: releasing `v0.13.0`.

## Release state

- Stable release: `v0.12.0` at `94216828ecab77d5f87a541563f5c2d8951d5a33`.
- Development line: `0.12.1.dev0`.
- Release candidate: `v0.13.0`; the exact release commit remains pending.
- Development line during release finalization: `0.13.0`.
- Stable docs: <https://daoyuanli2816.github.io/mini-verl/>.
- Development docs: <https://daoyuanli2816.github.io/mini-verl/dev/>.
- Historical build log: [v0.1-v0.9 archive](docs/history/project-state-v0.1-v0.9.md).

## Current product boundary

Development `0.12.1.dev0` has a closed typed profile compiler and adds
`verl-rl-v0.9-single-gpu-v1`, a typed resolved
subset of official verl `v0.9.0` at commit
`483b8a009ba3a97563edee3a19887e4862b8094a`. It compiles GRPO, Dr.GRPO, RLOO
and REINFORCE++ experiments into grouped one-GPU actor/reward/reference phases,
with behavior-policy identity, fixed reference reward KL, transactional resume
and a machine-readable field report.
Release `0.13.0` has a closed typed profile compiler: it retains the v1 critic-free contract and adds
`verl-rl-v0.9-single-gpu-v2`, a typed resolved PPO subset of official verl
`v0.9.0` at commit `483b8a009ba3a97563edee3a19887e4862b8094a`.
It compiles PPO/GAE into phased one-GPU actor, critic, reference and reward
roles. The critic is independently trainable and owns its optimizer, schedule,
checkpoint state and exact-resume identity. The same release line connects
actor KL, entropy regularization and a pinned Hugging Face sequence-classifier
reward provider.

The established `verl-opd-v0.8-single-gpu-v1` family remains the teacher-based
distillation path. It targets official verl `v0.8.0` at
Expand Down Expand Up @@ -52,12 +53,12 @@ Releases use a canonical, explicit evidence layout with four principal JSON
records and one deterministic subordinate-evidence archive; historical v0.10.1
asset names remain immutable.

PPO/GAE execution remains unimplemented because the runtime has no trainable
critic model/optimizer/checkpoint lifecycle; its mathematical primitives are
conformance-tested. Ray, FSDP/FSDP2, Megatron, multi-GPU and multi-node
execution are distributed-only. Arbitrary unresolved verl YAML and unknown
The v0.9 export path preserves actor, critic, Parquet, recipe and provenance
artifacts in a checksummed bundle. Its launch template stays inactive because
miniVERL does not translate local optimizer state into upstream distributed
checkpoints or run Ray/FSDP. Arbitrary unresolved verl YAML and unknown
algorithm-changing fields remain outside the versioned compiler. The legacy
environment/PPO artifact bridge is migration-only.
v0.8 environment/PPO artifact bridge is migration-only.

## Current evidence

Expand Down
Loading
Loading
0