8000 Add verified verl bridge and community launch surface by DaoyuanLi2816 · Pull Request #36 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: docs

on:
workflow_dispatch:
push:
branches: [main]
paths:
- "docs/**"
- "mkdocs.yml"
- ".github/workflows/docs.yml"

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: pages
cancel-in-progress: false

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1
with:
python-version: "3.12"
cache: pip
- uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b
- run: python -m pip install "mkdocs>=1.6,<2"
- run: mkdocs build
- uses: actions/upload-pages-artifact@7b1f4a764d45c48632c6b24a0339c27f5614fb0b
with:
path: site
deploy:
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
needs: build
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e
49 changes: 49 additions & 0 deletions .github/workflows/verl-bridge.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: pinned verl bridge

on:
workflow_dispatch:
pull_request:
paths:
- ".github/workflows/verl-bridge.yml"
- "pyproject.toml"
- "scripts/*verl_bridge*"
- "src/miniverl/bridge/**"
- "tests/**/*verl_bridge*"
push:
tags:
- "v0.6.*"

permissions:
contents: read

jobs:
pinned-profile-smoke:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1
with:
python-version: "3.12"
cache: pip
- name: Install miniVERL bridge environment
run: >-
python -m pip install --upgrade pip &&
python -m pip install ".[bridge,train]" "hydra-core>=1.3,<2"
- name: Install the exact official verl source without its distributed stack
run: >-
python -m pip install --no-deps
"git+https://github.com/verl-project/verl.git@7aed6b230776f963fa09509c10d9c3a767d1102c"
- name: Generate and export the standards smoke bundle
run: |
python scripts/prepare_verl_bridge_smoke.py --out _verl-smoke-source
miniverl export-verl --run _verl-smoke-source --target-verl v0.8.0 --out _verl-smoke-bundle
- name: Verify official config, standard artifacts, privacy and exact VCS pin
run: >-
python scripts/verify_verl_bridge_smoke.py _verl-smoke-bundle
--out verl-bridge-smoke-report.json
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: verl-bridge-smoke-report
path: verl-bridge-smoke-report.json
if-no-files-found: error
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ __pycache__/
.eggs/
/build/
/dist/
/site/
/.venv/
/venv/
.python-version
Expand Down
26 changes: 26 additions & 0 deletions CHANGELOG.md
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,32 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

### Added

- A compatibility Level-3 bridge for the fail-closed
`single-gpu-online-distillation-v1` profile, pinned to official verl `v0.8.0`
commit `7aed6b230776f963fa09509c10d9c3a767d1102c`.
- `import-verl`, bidirectional prompt-Parquet conversion, `export-verl` standard
artifact bundles and `bridge doctor`, with exact pin, PEFT/safetensors,
tokenizer, data, reward-scaffold, privacy and hash checks.
- Five versioned, evidence-bound community recipe records plus
`benchmark --export-community`, schema/privacy/digest validation, a static
documentation site and launch materials.

### Changed

- The README now describes miniVERL as single-GPU prototyping for one
documented subset of verl-style online post-training and distinguishes
artifact/config interoperability from untested distributed execution.

### Verified

- A Python 3.12 smoke installed the exact pinned verl source (observed package
version `0.8.0.dev0`), parsed the official and exported OmegaConf shapes,
loaded standard PEFT/safetensors and both Parquet splits, imported the safe
reward scaffold, and verified privacy plus every bundle hash. Ray,
FSDP/Megatron, vLLM/SGLang and distributed training were not run.

## [0.5.0] - 2026-08-02

### Added
Expand Down
12 changes: 12 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,18 @@ Read `docs/math.md` first. Then:

## Contributing a benchmark result

For the versioned community hardware matrix, start with a privacy-safe template:

```bash
miniverl benchmark --export-community submission.json
```

Fill measured fields only from retained artifacts and include their SHA-256
digests. CI validates the schema, packaged recipe digest, hardware/software
provenance, privacy and artifact hashes. Negative and failed runs are useful;
label them rather than removing them. See the
[community submission guide](docs/community-benchmarks.md).

```bash
miniverl train <recipe>
miniverl export-benchmark runs/<run-id> --out benchmarks/results/<gpu>-<recipe>.json
Expand Down
14 changes: 14 additions & 0 deletions PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,20 @@ and what it printed.

Last updated: 2026-08-02.

## v0.6.0 Verified verl Bridge development state

| item | current state |
| --- | --- |
| audited upstream | official stable verl `v0.8.0`, source commit `7aed6b230776f963fa09509c10d9c3a767d1102c`, tested with Python 3.12; the installed source reports `0.8.0.dev0` |
| compatibility contract | Levels 0-3 are explicit; Level 2 is a fail-closed 14-field whitelist for `single-gpu-online-distillation-v1`; Level 3 exchanges standard HF/PEFT/safetensors/tokenizer/Parquet artifacts and a checksummed scale-out bundle |
| command surface | `import-verl`, `convert-dataset`, `export-verl`, `bridge doctor` and `benchmark --export-community` are implemented without importing torch in the core path |
| exact compatibility smoke | the pinned source installed successfully; OmegaConf parsed official and exported config shapes; PEFT, safetensors, train/val Parquet, reward import, privacy and all bundle hashes passed; the checksummed record is `docs/generated/verl-bridge-smoke.json` |
| distributed boundary | Ray, FSDP/Megatron, vLLM/SGLang and a full distributed run were not installed or launched; distributed execution remains explicitly `not tested` |
| community surface | four versioned recipe records bind existing measurements; the preference-teacher distillation entry is explicitly unmeasured and links an aligned-adapter scaffold that still requires a pinned preference-trained teacher; no external adoption is claimed |
| documentation | a static MkDocs site, bridge architecture diagram, launch article/demo/card copy/announcement/social preview and community submission guide are part of the release candidate |
| immutable evidence | the frozen calculator JSON remains required byte-identical at SHA-256 `53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc`; prior tags and published adapter revisions remain immutable |
| release state | development PR in progress from `0.6.0.dev0`; publication evidence is recorded only after tag, OIDC, public install and state sync complete |

## v0.5.0 One-GPU Alignment Lab release

| item | current state |
Expand Down
72 changes: 52 additions & 20 deletions PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,27 +14,32 @@

<p align="center">
<a href="https://pypi.org/project/miniverl/"><strong>PyPI package</strong></a> ·
<a href="https://daoyuanli2816.github.io/mini-verl/">Documentation</a> ·
<a href="#single-gpu-quickstart">Install &amp; train</a> ·
<a href="https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/single-gpu-guide.md">Bring your own GPU</a> ·
<a href="#alignment-lab-when-to-turn-opd-off">Alignment Lab result</a>
<a href="https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md">Verified verl bridge</a> ·
<a href="#alignment-lab-when-to-turn-opd-off">Alignment Lab</a>
</p>

**Online alignment and distillation on one GPU.**
**Single-GPU prototyping for a documented subset of verl-style online
post-training.**

Post-train an SFT model from a policy-conditioned, preference-trained or
safety-aligned teacher—without Ray or a cluster.
Develop, diagnose and validate an alignment or distillation recipe locally,
then export standard model, dataset, recipe and provenance artifacts to a
pinned verl release for scale-out.

**Measure alignment, over-refusal, retained utility and cost before choosing
SFT, DPO or OPD.**

PyPI `v0.5.0` is the stable release; `main` is development and may be ahead.

miniVERL is an independent one-GPU companion for prototyping, diagnosing and
validating online post-training workflows before scale-out. SFT establishes
task and protocol competence; OPD transfers behavior from an explicit teacher
on student-visited states. They are not interchangeable stages. The local CUDA
path has no device-name allowlist; fit still depends on model size, sequence
budget and available VRAM.
miniVERL is independent from verl and implements one verified Level-3 profile,
`single-gpu-online-distillation-v1`, against official verl `v0.8.0`. This is a
standard-artifact and config-subset bridge—not generic YAML compatibility or a
claim that distributed execution was tested. The local CUDA path has no
device-name allowlist; fit still depends on model size, sequence budget and
available VRAM.

![miniVERL one-GPU workflow and pinned verl scale-out bridge](https://raw.githubusercontent.com/DaoyuanLi2816/mini-verl/main/docs/verl-bridge-architecture.svg)

```bash
python -m pip install miniverl # lightweight core
Expand Down Expand Up @@ -108,7 +113,32 @@ keeps the whole lifecycle in one readable single-GPU process.
| Calculator, JSON-navigation and SQLite environments | yes, deterministic with exact verifiers |
| Exact checkpoint/resume | yes, asserted parameter-for-parameter |
| Self-contained offline HTML report with token-level divergence | yes |
| Ray, FSDP, DeepSpeed, vLLM, VLMs, cross-tokenizer, PPO/GRPO | **no** — see [limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/limitations.md) |
| Pinned verl `v0.8.0` Level-3 bridge | yes; one fail-closed profile, Parquet round trips and standard PEFT/safetensors export |
| Native Ray/FSDP/Megatron/vLLM execution, VLMs, cross-tokenizer, PPO/GRPO | **no** — see [limitations](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/limitations.md) |

## Verified verl bridge

The bridge imports 14 named fields from one pinned profile, converts the
official prompt Parquet schema in both directions, exports a self-checking
scale-out bundle and diagnoses it without pretending that miniVERL teacher
targets are PPO reference log-probabilities.

```bash
python -m pip install "miniverl[bridge]"
miniverl import-verl verl.yaml --profile single-gpu-online-distillation-v1 \
--target-verl v0.8.0 --out recipes/imported.yaml
miniverl convert-dataset --from verl-parquet train.parquet --out local.parquet
miniverl export-verl --run runs/my-alignment --target-verl v0.8.0 \
--out exports/my-alignment-verl
miniverl bridge doctor exports/my-alignment-verl --json
```

The release smoke pins commit
`7aed6b230776f963fa09509c10d9c3a767d1102c`, parses the generated OmegaConf
profile, loads standard PEFT/safetensors and both Parquet splits, imports the
reward scaffold, verifies privacy plus every hash, and records distributed
execution as **not tested**. See the [contract, whitelist and evidence](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/verl-bridge.md)
or the [community recipe registry](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/community-benchmarks.md).

## Alignment Lab: when to turn OPD off

Expand Down Expand Up @@ -615,21 +645,23 @@ See [`docs/reproducibility.md`](https://github.com/DaoyuanLi2816/mini-verl/blob/
Not implemented, not promised, listed so the scope is unambiguous:
cross-tokenizer distillation, batched or engine-backed rollout decoding,
entropy-aware divergence mixing (arXiv:2603.07079), additional model families,
more environments, and multi-GPU. For anything at cluster scale, use verl.
more environments, and native multi-GPU execution. The Level-3 bridge exports
one documented profile to pinned verl; it does not execute that distributed job.

## Acknowledgement and disclaimer

> miniVERL is an independent project and is not affiliated with or endorsed by
> the verl project, ByteDance, or Volcano Engine. It is not a drop-in
> replacement for verl.

The name is a nod to the problem space, not a claim of compatibility. verl is an
excellent, much larger system that also implements on-policy distillation and
multi-turn tool use — at cluster scale, with Ray. If you have a cluster, use it.
miniVERL exists for the case where you have one personal GPU and want to read
every line of what is happening. That can be an older 12 GiB card or a current
high-end card; the repository claims measured performance only for hardware it
actually ran. See [`docs/comparisons.md`](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/comparisons.md).
The name is a nod to the problem space, not a claim of generic compatibility.
The verified bridge is deliberately limited to one pinned profile. verl is an
excellent, much larger system for cluster-scale execution with Ray. miniVERL
exists for the case where you have one personal GPU and want to read every line
of what is happening, then hand standard artifacts to verl for scale-out. That
can be an older 12 GiB card or a current high-end card; the repository claims
measured performance only for hardware it actually ran. See
[`docs/comparisons.md`](https://github.com/DaoyuanLi2816/mini-verl/blob/main/docs/comparisons.md).

## Citation

Expand Down
72 changes: 52 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,27 +14,32 @@

<p align="center">
<a href="https://pypi.org/project/miniverl/"><strong>PyPI package</strong></a> ·
<a href="https://daoyuanli2816.github.io/mini-verl/">Documentation</a> ·
<a href="#single-gpu-quickstart">Install &amp; train</a> ·
<a href="docs/single-gpu-guide.md">Bring your own GPU</a> ·
<a href="#alignment-lab-when-to-turn-opd-off">Alignment Lab result</a>
<a href="docs/verl-bridge.md">Verified verl bridge</a> ·
<a href="#alignment-lab-when-to-turn-opd-off">Alignment Lab</a>
</p>

**Online alignment and distillation on one GPU.**
**Single-GPU prototyping for a documented subset of verl-style online
post-training.**

Post-train an SFT model from a policy-conditioned, preference-trained or
safety-aligned teacher—without Ray or a cluster.
Develop, diagnose and validate an alignment or distillation recipe locally,
then export standard model, dataset, recipe and provenance artifacts to a
pinned verl release for scale-out.

**Measure alignment, over-refusal, retained utility and cost before choosing
SFT, DPO or OPD.**

PyPI `v0.5.0` is the stable release; `main` is development and may be ahead.

miniVERL is an independent one-GPU companion for prototyping, diagnosing and
validating online post-training workflows before scale-out. SFT establishes
task and protocol competence; OPD transfers behavior from an explicit teacher
on student-visited states. They are not interchangeable stages. The local CUDA
path has no device-name allowlist; fit still depends on model size, sequence
budget and available VRAM.
miniVERL is independent from verl and implements one verified Level-3 profile,
`single-gpu-online-distillation-v1`, against official verl `v0.8.0`. This is a
standard-artifact and config-subset bridge—not generic YAML compatibility or a
claim that distributed execution was tested. The local CUDA path has no
device-name allowlist; fit still depends on model size, sequence budget and
available VRAM.

![miniVERL one-GPU workflow and pinned verl scale-out bridge](docs/verl-bridge-architecture.svg)

```bash
python -m pip install miniverl # lightweight core
Expand Down Expand Up @@ -108,7 +113,32 @@ keeps the whole lifecycle in one readable single-GPU process.
| Calculator, JSON-navigation and SQLite environments | yes, deterministic with exact verifiers |
| Exact checkpoint/resume | yes, asserted parameter-for-parameter |
| Self-contained offline HTML report with token-level divergence | yes |
| Ray, FSDP, DeepSpeed, vLLM, VLMs, cross-tokenizer, PPO/GRPO | **no** — see [limitations](docs/limitations.md) |
| Pinned verl `v0.8.0` Level-3 bridge | yes; one fail-closed profile, Parquet round trips and standard PEFT/safetensors export |
| Native Ray/FSDP/Megatron/vLLM execution, VLMs, cross-tokenizer, PPO/GRPO | **no** — see [limitations](docs/limitations.md) |

## Verified verl bridge

The bridge imports 14 named fields from one pinned profile, converts the
official prompt Parquet schema in both directions, exports a self-checking
scale-out bundle and diagnoses it without pretending that miniVERL teacher
targets are PPO reference log-probabilities.

```bash
python -m pip install "miniverl[bridge]"
miniverl import-verl verl.yaml --profile single-gpu-online-distillation-v1 \
--target-verl v0.8.0 --out recipes/imported.yaml
miniverl convert-dataset --from verl-parquet train.parquet --out local.parquet
miniverl export-verl --run runs/my-alignment --target-verl v0.8.0 \
--out exports/my-alignment-verl
miniverl bridge doctor exports/my-alignment-verl --json
```

The release smoke pins commit
`7aed6b230776f963fa09509c10d9c3a767d1102c`, parses the generated OmegaConf
profile, loads standard PEFT/safetensors and both Parquet splits, imports the
reward scaffold, verifies privacy plus every hash, and records distributed
execution as **not tested**. See the [contract, whitelist and evidence](docs/verl-bridge.md)
or the [community recipe registry](docs/community-benchmarks.md).

## Alignment Lab: when to turn OPD off

Expand Down Expand Up @@ -615,21 +645,23 @@ See [`docs/reproducibility.md`](docs/reproducibility.md) and the concise
Not implemented, not promised, listed so the scope is unambiguous:
cross-tokenizer distillation, batched or engine-backed rollout decoding,
entropy-aware divergence mixing (arXiv:2603.07079), additional model families,
more environments, and multi-GPU. For anything at cluster scale, use verl.
more environments, and native multi-GPU execution. The Level-3 bridge exports
one documented profile to pinned verl; it does not execute that distributed job.

## Acknowledgement and disclaimer

> miniVERL is an independent project and is not affiliated with or endorsed by
> the verl project, ByteDance, or Volcano Engine. It is not a drop-in
> replacement for verl.

The name is a nod to the problem space, not a claim of compatibility. verl is an
excellent, much larger system that also implements on-policy distillation and
multi-turn tool use — at cluster scale, with Ray. If you have a cluster, use it.
miniVERL exists for the case where you have one personal GPU and want to read
every line of what is happening. That can be an older 12 GiB card or a current
high-end card; the repository claims measured performance only for hardware it
actually ran. See [`docs/comparisons.md`](docs/comparisons.md).
The name is a nod to the problem space, not a claim of generic compatibility.
The verified bridge is deliberately limited to one pinned profile. verl is an
excellent, much larger system for cluster-scale execution with Ray. miniVERL
exists for the case where you have one personal GPU and want to read every line
of what is happening, then hand standard artifacts to verl for scale-out. That
can be an older 12 GiB card or a current high-end card; the repository claims
measured performance only for hardware it actually ran. See
[`docs/comparisons.md`](docs/comparisons.md).

## Citation

Expand Down
Loading
0