8000 mini-verl/docs/index.md at main · thinglinks/mini-verl · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
134 lines (92 loc) · 4.77 KB

index.md

File metadata and controls

134 lines (92 loc) · 4.77 KB

miniVERL

Run a documented subset of verl v0.8 on-policy distillation on one consumer GPU. Bring a typed OPD profile and Parquet prompts, inspect rollout → teacher scoring → update locally, then export standard artifacts. miniVERL is independent; distributed execution and full verl compatibility are not claimed.

Install and run locally{ .md-button .md-button--primary } Read the compatibility boundary{ .md-button }

Pip-only OPD quickstart

python -m pip install torch --index-url https://download.pytorch.org/whl/cu130
python -m pip install "miniverl[train,cuda]"
miniverl doctor
miniverl data sample --format verl-parquet --out prompts.parquet
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
  --config builtin:qwen3-0.6b-1.7b-opd \
  --set 'data.train_files=["prompts.parquet"]'
miniverl run --profile verl-opd-v0.8-single-gpu-v1 \
  --config builtin:qwen3-0.6b-1.7b-opd \
  --set 'data.train_files=["prompts.parquet"]' --dry-run

Planning loads no weights. Remove --dry-run on one CUDA GPU to execute the pinned Qwen3 recipe and export a standard PEFT adapter. Install the matching CUDA-enabled PyTorch wheel first; the [cuda] extra does not select one. For the exact maintainer-measured versions, use the machine-readable known-good stack.

After inspection, write plan.json with plan --out, run it without --dry-run, then use inspect, cache stats, export-verl and bridge doctor; the quickstart carries the complete copyable path.

Runtime and compatibility boundary

miniVERL runs one local CPU process or one NVIDIA CUDA GPU. Fit depends on the model pair, context, kernels and VRAM; there is no GPU-name allowlist. Ray, FSDP, Megatron, PPO, GRPO and distributed launch are outside the runtime.

The executable profile pins verl v0.8.0 at 7aed6b23: one actor, one teacher, n=1, GKD forward_kl_topk, token-mean and no reward/KL penalty. Unsupported algorithm or distributed semantics fail closed. Exports remain unlaunchable until exact base snapshots are materialized and checked under the pinned verl commit. See scale-out materialization.

Measured runtime evidence

The v0.9 RTX 4080 developer workload consumed 32 distinct prompts and completed 8 current-policy updates at 3.1914 GiB peak reserved VRAM. Median steady-state rollout, teacher-scoring and update times were 9.7200, 0.4864 and 2.3260 seconds; a matched interruption/resume reproduced byte-identical trajectories, adapter and optimizer tensors. No quality endpoint or method comparison ran.

Measured workload evidence{ .md-button }

Choose a path

Run OPD locally

Plan, execute, inspect and resume one local pure-OPD run.

miniverl plan --profile verl-opd-v0.8-single-gpu-v1 --config verl-opd.yaml

Artifact: a checksummed local execution plan, trajectories and PEFT adapter.

Next: Plan and run

Bring a verl config

Import the resolved, documented OPD subset with field-by-field classifications.

miniverl import-verl --profile verl-opd-v0.8-single-gpu-v1 \
  --config verl-opd.yaml --out local-opd.yaml

Artifact: a round-trippable profile and local-opd.import-report.json.

Next: Supported field boundary

Move data and artifacts

Convert Parquet, export standard artifacts and inspect the unsupported boundary.

miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout

Artifact: PEFT + Parquet + OPD overrides and separate readiness flags.

Next: current scale-out contract

Research Notes

The v0.7 external study stopped at its first preregistered gate: 0 selected checkpoints, 0 qualified teachers, 0 continuation arms and 0 final-test tasks accessed. All eight candidates scored 0/64 retained JSONNav utility against the unchanged 20% floor.

miniverl pilot --builtin-study alignment-external-v1 --json

Read the early-stop study{ .md-button .md-button--primary }

Alignment Lab starts from a saturated SFT checkpoint. No continuation method improves it; measured regressions and unexecuted external safety endpoints stay visible. RecoveryBench and the calculator study likewise preserve their negative and mixed results rather than turning them into product claims.

0