8000 mini-verl/docs/opd-quickstart.md at v0.9.0 · gitlick/mini-verl · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
76 lines (61 loc) · 3.87 KB

opd-quickstart.md

File metadata and controls

76 lines (61 loc) · 3.87 KB

Run verl-style OPD locally

miniVERL v0.8 supports one pinned, fail-closed subset of verl v0.8: one actor, one teacher, one generation per prompt, reward-free direct GKD with forward_kl_topk, token-mean aggregation and a LoRA/QLoRA student. It is local single-GPU execution, not Ray/FSDP or distributed verl execution.

Install a CUDA build of PyTorch that matches your machine first, then:

python -m pip install "miniverl[train,cuda,bridge]"
miniverl data sample --format verl-parquet --out data/opd-smoke.parquet
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
  --config builtin:qwen3-0.6b-1.7b-opd
miniverl run --profile verl-opd-v0.8-single-gpu-v1 \
  --config builtin:qwen3-0.6b-1.7b-opd

The built-in recipe downloads pinned Qwen3-0.6B and Qwen3-1.7B snapshots when they are not cached. Allow roughly 6 GiB of download/cache space and 0.25 GiB for run artifacts. The plan command itself is CPU-only and weight-free; use --offline for a zero-network compiler smoke.

Measured reference

The v0.9 developer workload extends this install smoke to 32 distinct consumed prompts, 64 response tokens and eight optimizer updates. See its phase-time, throughput, VRAM and resume evidence.

GPU student / teacher strategy limits / top-k peak reserved first update status
RTX 4080 16 GiB Qwen3-0.6B / Qwen3-1.7B, both NF4 dual resident 128 + 16 tokens / 32 3.176 GiB 12.02 s measured
12 GiB CUDA GPU same built-in recipe planner-selected same — — not measured
24 GiB CUDA GPU same built-in recipe planner-selected same — — not measured

The measured run completed one current-policy rollout/scoring/update cycle, exported a loadable PEFT adapter, and used one RTX 4080. It demonstrates runtime and artifact correctness only; it did not evaluate alignment quality. The checksummed record is rtx4080-verl-opd-runtime-v1.json.

What plan means

plan reports estimates separately from measurements. Auto placement uses model metadata plus the configured VRAM headroom; it never branches on a GPU product name. Unknown model sizes conservatively select swap. --probe is reserved but fails closed in v0.8.0 rather than loading weights unexpectedly.

Unsupported settings—including policy-gradient OPD, task rewards, reference KL, multiple teachers, multiple generations, multimodal inputs and every distributed dimension—are rejected rather than reinterpreted.

Import and export the same bounded profile

miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 \
  --config verl-opd.yaml --out local-opd.yaml
miniverl run --profile verl-opd-v0.8-single-gpu-v1 \
  --config local-opd.yaml --output runs
miniverl export-verl --run runs/<run-id> --target-verl v0.8.0 --out scaleout
miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --require-verl

A compatible export contains the standard student PEFT adapter, tokenizer metadata, exact student/teacher identities, original Parquet bytes, the source config, compiled plan and pure OPD overrides. No reward scaffold is generated. Validation data is exported only when the source declared it; an empty data.val_files remains empty rather than duplicating training rows.

The bundle stays launchable: false until the exact student and teacher base snapshots are materialized. A local teacher adapter adds an explicit merge requirement because the pinned upstream profile does not consume that adapter path directly. Materialization requires the exact installed verl pin and emits launch.sh only after model/tokenizer/data/config checks pass. The bridge never claims a distributed verl job ran. See the materialization contract.

0