8000 Build the one-GPU Alignment Lab by DaoyuanLi2816 · Pull Request #33 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,37 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

### Added

- `miniverl align` for explicit base → SFT checkpoint → teacher/reference →
alignment → evaluation → Alignment Card workflows, plus an
uncertainty-aware, versioned `miniverl pilot` decision aid.
- Policy-conditioned and frozen aligned-adapter teachers, pinned TRL DPO
provenance, a versioned verifier-gated selector, deterministic tool-policy
evaluation and privacy-safe JSON/Markdown Alignment Cards.
- A preregistered three-seed Alignment Lab result, 864 task-level records, a
matched State × Supervision diagnostic, four data-bound figures, technical
report, article and reproducible short demo.

### Changed

- Public positioning now treats OPD as a post-SFT teacher-student mechanism to
justify with alignment, over-alignment, retained utility and cost evidence,
rather than as a generic replacement for SFT.
- DPO Alignment Cards include the external pinned TRL training time, peak VRAM,
optimizer updates and exact provenance instead of counting evaluation only.

### Results

- The common Qwen3-0.6B SFT checkpoint saturated the deterministic Minipolicy
suite at 100% alignment and 100% retained tool utility across all three
seeds. No continuation method improved it; completed regressions from
continued SFT, standard OPD and verifier-gated OPD are retained.
- Verifier gating reduced queried positions from 100% to 46.8% and mean GPU
time from 76.7 to 66.0 seconds without improving quality. The matched signal
diagnostic found only 0.0251% fresh soft probability mass beyond argmax, so
the pilot recommends not spending online teacher-query cost for this recipe.

## [0.4.0] - 2026-08-02

### Added
Expand Down
23 changes: 23 additions & 0 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,29 @@ and what it printed.

Last updated: 2026-08-02.

## v0.5.0 One-GPU Alignment Lab release

| item | current state |
| --- | --- |
| phase boundary | v0.4.0 was released and state-synced to `0.5.0.dev0` before final Alignment Lab execution; v0.5 must merge, release and advance main to `0.6.0.dev0` before the verified verl bridge begins |
| public workflow | `miniverl align` resolves base → SFT checkpoint → teacher/reference → alignment → evaluation → Alignment Card; `miniverl pilot` exposes versioned evidence, uncertainty, cost assumptions and a bounded method recommendation |
| alignment roles | policy-conditioned self-distillation, frozen aligned-adapter teaching and shared-backbone execution are supported; continued SFT, pinned TRL DPO, offline soft distillation, standard OPD and verifier-gated OPD are explicit methods |
| policy suite | Minipolicy v1 uses deterministic sandbox tools to check authorization, confirmation, instruction hierarchy, secret exclusion, safe refusal, benign completion and safe error recovery; no real destructive action is executed |
| preregistration | public revision 1.4 digest `71307dbfe9a5bb20c686307cafce8bd254c07af8b69c1bf1c6ec0dbf53a8cde0` freezes three seeds, the shared SFT checkpoint, four continuation updates, 48 final-test tasks, the gate and the disjoint seed-1234 recovery rule |
| final artifact audit | all 18 arms completed with 48 ordered paired final-test tasks each; 864 task-level records are retained; all arms share starting checkpoint `7304922281268a687dd1c75ba918e26c64c8207b5701db78c368afd20d80ae89`; strict OPD freshness, every verifier-gate decision, DPO provenance/cost and the two disjoint baseline segments pass publication checks |
| final result | the SFT checkpoint scored 100% alignment and 100% tool utility in every seed; DPO and offline soft distillation tied it; continued SFT averaged 94.4% / 88.9%, standard OPD 98.6% / 97.2%, and verifier-gated OPD 97.9% / 95.8% alignment / utility |
| preserved negative results | verifier-gated OPD seed 20260727 failed 3 safe-error-recovery tasks; continued SFT seed 20260801 failed 8; standard OPD seed 20260801 failed 2; no completed final arm was rerun, and the interrupted pre-evaluation continued-SFT construction remains preserved outside the headline result |
| State × Supervision | measured signal diagnostic `9e08129ba4cd9e460c189b94b4e421d881ba69e3938f02eac95d251f50c88788` finds 100% teacher-argmax/student-token agreement and 0.0251% fresh soft probability mass beyond argmax; it is not a separately trained hard-target result and no soft-target advantage is claimed |
| verifier-gated OPD | frozen `policy-critical-span-v1` reduced mean queried positions from 100% to 46.8% and GPU time from 76.7 to 66.0 seconds, but did not improve alignment or retained utility; selected positions are not teacher-backbone FLOPs |
| pilot decision | `alignment-pilot-v1` returns `insufficient_evidence` and the operational decision not to spend online teacher-query cost because the starting policy is already saturated and no continuation method improves it |
| frozen public artifacts | result `584752dccb91654109c357b8ebb12681a12a9c1476a9ba539dd35e4d860a22ef`; task rows `8d7fc723436d7377d196fc44046d960e3cb7f0aa81e03d49ef05b627eb84630f`; technical PDF `adbffa967f6b9a25d2cdb0cc4464a93c13db4615a1e91499585fb199285d980b`; 18 JSON plus 18 Markdown Alignment Cards |
| public teacher | the common Qwen3-0.6B SFT adapter is immutable at `DaoyuanLi/mini-verl-qwen3-0.6b-tool-policy-sft@7b98164f73e493c51f2ed3fca3169fea078f47f0`; starting checkpoint content digest is recorded separately above |
| demonstration assets | reviewed short CPU run completed the actual alignment path in 17.6 seconds; the committed 75-second recording plan includes pilot, stage graph, real update, typed token inspection, Alignment Card and export-ready checkpoint without hiding the toy 0% result |
| presentation | four data-bound dark SVGs passed native and 820-pixel inspection; banner and social preview passed raster inspection; the report PDF is deterministic, six pages, metadata-valid and visually inspected page by page |
| local release gates | ruff, format, mypy, actionlint, generated-artifact checks, Markdown links and package/twine checks pass; the full non-GPU/non-network suite passes 1508 tests with 6 deselected at 86.13% branch coverage; the RTX 4080 gate passes 5 and the network gate passes 3; fresh core and CPU-training wheel installs, a real no-network toy demo, and all 1508 tests from an isolated extracted sdist pass |
| immutable baseline | calculator benchmark remains byte-identical at `53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc`; v0.3/v0.4 tags, RecoveryBench artifacts and public adapter revisions remain unchanged |
| integration state | development branch `v0.5-alignment-lab` contains the frozen execution commit `8ffb2d9c3094fb6580e5afd89c861c8e0511ef78`; implementation and local release gates are complete, while PR review, merge, release and state sync remain pending |

## v0.4.0 Consumer Runtime release

| item | current state |
Expand Down
82 changes: 70 additions & 12 deletions PYPI.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<p align="center">
<img src="https://raw.githubusercontent.com/DaoyuanLi2816/mini-verl/main/docs/banner.svg" alt="miniVERL — auditable online post-training on one GPU" width="880">
<img src="https://raw.githubusercontent.com/DaoyuanLi2816/mini-verl/main/docs/banner.svg" alt="miniVERL — online alignment and distillation on one GPU" width="880">
</p>

<div align="center">
Expand All @@ -16,21 +16,25 @@
<a href="https://pypi.org/project/miniverl/"><strong>PyPI package</strong></a> ·
<a href="#single-gpu-quickstart">Install &amp; train</a> ·
<a href="https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/single-gpu-guide.md">Bring your own GPU</a> ·
<a href="#recoverybench-do-fresh-on-policy-states-justify-their-cost">Measured result</a>
<a href="#alignment-lab-when-to-turn-opd-off">Alignment Lab result</a>
</p>

**The independent one-GPU companion for prototyping, diagnosing and validating
online post-training workflows before scaling selected artifacts to verl.**
**Online alignment and distillation on one GPU.**

Post-train an SFT model from a policy-conditioned, preference-trained or
safety-aligned teacher—without Ray or a cluster.

**Measure alignment, over-refusal, retained utility and cost before choosing
SFT, DPO or OPD.**

PyPI `v0.4.0` is the stable release; `main` is development and may be ahead.

miniVERL is a compact, auditable lab for online teacher-student training on
multi-turn tool trajectories. SFT establishes task and protocol competence;
OPD then exposes an online mechanism for transferring the teacher's reasoning,
policy, style or other behavior. They are not interchangeable stages, and the
teacher must be qualified for the behavior being transferred. miniVERL runs
without Ray or a cluster and has no CUDA device-name allowlist; fit still
depends on model size, sequence budget and available VRAM.
miniVERL is an independent one-GPU companion for prototyping, diagnosing and
validating online post-training workflows before scale-out. SFT establishes
task and protocol competence; OPD transfers behavior from an explicit teacher
on student-visited states. They are not interchangeable stages. The local CUDA
path has no device-name allowlist; fit still depends on model size, sequence
budget and available VRAM.

```bash
python -m pip install miniverl # lightweight core
Expand All @@ -55,10 +59,12 @@ validate artifacts without downloading a multi-gigabyte ML stack; use
can enter the loss.
- **Budget truth:** exact full-vocabulary objectives and compressed
`top-k + tail` objectives are named and reported separately.
- **Decision truth:** every Alignment Card reports policy quality, retained
utility, teacher-query ratio, GPU time, VRAM and limitations together.

[Run the local demo](#local-toy-demo) ·
[Train on your GPU](#single-gpu-quickstart) ·
[Inspect the measured result](#recoverybench-do-fresh-on-policy-states-justify-their-cost) ·
[Inspect the measured result](#alignment-lab-when-to-turn-opd-off) ·
[Read the math](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/math.md)

## Why miniVERL exists
Expand Down Expand Up @@ -97,11 +103,63 @@ keeps the whole lifecycle in one readable single-GPU process.
| `resident` and `swap` memory strategies, `auto` resolution | yes, with an equivalence test |
| Versioned, checksummed, pickle-free teacher-target cache | yes |
| SFT / offline KD / strict OPD / explicitly labeled replay behind one trainer | yes |
| `align` / `pilot`, policy-conditioned and aligned-adapter teachers, DPO provenance | yes |
| Versioned verifier gate, AlignmentBench metadata adapters and privacy-safe Alignment Cards | yes; external suites are metadata-only in the v0.5 measurement |
| Calculator, JSON-navigation and SQLite environments | yes, deterministic with exact verifiers |
| Exact checkpoint/resume | yes, asserted parameter-for-parameter |
| Self-contained offline HTML report with token-level divergence | yes |
| Ray, FSDP, DeepSpeed, vLLM, VLMs, cross-tokenizer, PPO/GRPO | **no** — see [limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/limitations.md) |

## Alignment Lab: when to turn OPD off

> [!IMPORTANT]
> **The starting SFT policy already saturated this deterministic test.** Across
> three preregistered seeds, no continuation method improved its 100% alignment
> and 100% tool-utility result. The completed regressions from continued SFT,
> standard OPD and verifier-gated OPD remain in the headline result.

All six methods start from the same checksummed Qwen3-0.6B SFT checkpoint and
use the same 48 ordered final-test tasks per seed. This is a small deterministic
tool-policy suite—not a broad safety benchmark—and the only measured GPU is an
RTX 4080.

| method | alignment | tool utility | teacher query | continuation GPU time |
| --- | ---: | ---: | ---: | ---: |
| SFT checkpoint | **100.0%** | **100.0%** | n/a | 0.0 s |
| continued SFT | 94.4% | 88.9% | n/a | 3.9 s |
| DPO | **100.0%** | **100.0%** | n/a | 8.6 s |
| offline soft distillation | **100.0%** | **100.0%** | 100.0% | 26.6 s |
| standard OPD | 98.6% | 97.2% | 100.0% | 76.7 s |
| verifier-gated OPD | 97.9% | 95.8% | 46.8% | 66.0 s |

![Alignment quality versus utility retention](https://raw.githubusercontent.com/DaoyuanLi2816/mini-verl/main/docs/alignment-lab/quality-vs-utility.svg)

Harmful-compliance and over-refusal rates were both 0% for every method, yet
safe-error-recovery utility regressed in three completed arms. Those two safety
axes alone therefore missed a real policy-utility failure in this suite. The
matched State × Supervision diagnostic found only 0.0251% mean teacher
probability mass beyond argmax on fresh states; it is a signal diagnostic, not
a separately trained hard-target result, and no soft-target advantage is
claimed.

`miniverl pilot` returns `insufficient_evidence` and tells this recipe not to
spend online teacher-query cost. That is the intended product behavior: OPD is
an option to justify with evidence, not a default replacement for SFT.

```bash
miniverl pilot recipes/alignment_tool_policy_toy.yaml --json
miniverl align recipes/alignment_tool_policy_toy.yaml --dry-run
```

Read the [data-bound report](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-lab/alignment-lab-v1.md),
[technical PDF](https://github.com/DaoyuanLi2816/mini-verl/blob/main/paper/alignment-lab-v1/alignment-lab-v1.pdf),
[public article](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-lab/when-opd-should-follow-sft.md),
[demo script](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/alignment-lab/demo.md), or the
[18 privacy-safe Alignment Cards](https://github.com/DaoyuanLi2816/mini-verl/blob/main/benchmarks/alignment-cards/alignment-lab-v1/sft_checkpoint-seed-1234.md).
The [preregistration](https://github.com/DaoyuanLi2816/mini-verl/blob/main/benchmarks/preregistration/alignment-lab-v1.yaml),
[machine-readable result](https://github.com/DaoyuanLi2816/mini-verl/blob/main/benchmarks/results/alignment-lab-v1.json) and all 864
task-level records bind the claims above.

## Consumer Runtime: batch speed without a cluster

> A low-memory one-GPU runtime for actor rollout, teacher/reference scoring and
Expand Down
Loading
0