8000 mini-verl/docs/for-verl-users.md at main · pikaipi/mini-verl · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
150 lines (124 loc) · 7.25 KB

for-verl-users.md

File metadata and controls

150 lines (124 loc) · 7.25 KB

For verl users

miniVERL brings five closed verl v0.8 OPD profiles into a local, inspectable workflow; two have measured systems records and three are conformance-only. Familiar field names and Parquet data flow through a typed compiler, which turns resource intent into sequential phases on one CUDA GPU and records every local mapping in the resulting plan.

Inspect the closed, versioned registry with miniverl profiles list; use miniverl compat explain before assuming that an accepted field is effective. The profile registry guide documents the identity carried by plans, caches, checkpoints and exports.

verl-shaped YAML, overrides and Parquet prompts pass through a typed compiler; one CUDA GPU runs actor rollout, teacher scoring and actor update; portable artifacts form a pinned scale-out handoff.

Supported input shape

  • A resolved YAML using the verl-opd-v0.8-single-gpu-v1 field subset.
  • Verl-style Parquet prompts with structured chat messages; the rewarded profile additionally requires bound deterministic exact-answer metadata.
  • One actor, one teacher, token-mean aggregation, and either direct forward-top-k GKD or sampled-k1 vanilla policy loss. The measured profiles remain n=1; their grouped counterparts accept independent n>1 samples.
  • Immutable Hugging Face revisions, PEFT adapters and tokenizer snapshots.
  • Familiar fields such as actor_rollout_ref.model.path, distillation.teacher_models.teacher_model.model_path, response bounds, learning rate and LoRA configuration.

The profile compiler accepts a resolved, documented subset and classifies every field before execution. Compatibility profiles contains the field contract; limitations collects the algorithms and distributed runtime surfaces covered by other tools.

Command mapping

verl action miniVERL action
compose or capture a resolved config provide resolved YAML to --config
add a Hydra-style override repeat --set, use --overrides-file, or place tokens after --
inspect resolved intent miniverl plan --json
launch OPD miniverl run
read prompt Parquet use the file directly
allocate resource pools compile to sequential local phases
hand artifacts back miniverl export-verl
miniverl plan --profile verl-opd-v0.8-single-gpu-v1 \
  --config verl-opd.yaml \
  --set 'data.train_files=["data/train.parquet"]' \
  --set actor_rollout_ref.actor.optim.lr=1e-5

Planning is weight-free and offline. Use --json to retain the complete field matrix. The CLI records repeated --set, override files and trailing tokens with deterministic precedence. Resolve ${...} composition in the trusted source workflow first; see Config overrides.

For a serious run, add --accept-local-reinterpretations --out plan.json, inspect the immutable artifact, then execute miniverl run --plan plan.json. This binds the source YAML, overrides and scanned Parquet bytes to the exact native config; see Immutable execution plans.

Same fields, different placement

An upstream-shaped fragment can stay recognizable:

actor_rollout_ref:
  model: {path: Qwen/Qwen3-0.6B}
  rollout: {name: vllm, n: 1}
distillation:
  teacher_models:
    teacher_model:
      model_path: Qwen/Qwen3-1.7B
      inference: {name: vllm}

The vllm values remain visible as source intent. The compiler classifies them as local reinterpretations, runs Hugging Face generation and teacher scoring sequentially, and preserves the source value, local meaning and risk in the compatibility report.

Data mapping

data.train_files, data.val_files, data.prompt_key, prompt and response bounds, shuffle and seed feed the native Parquet source directly. Each prompt row preserves its structured messages and source metadata. Run miniverl data sample --out prompts.parquet for a valid small file, or use miniverl convert-dataset when crossing the native trajectory boundary.

Parquet paths are direct execution inputs, so a missing file produces an actionable error when execution begins. Dataset bytes and schema are copied into export provenance.

Actor, teacher and reference roles

The actor is the trainable PEFT policy. In the direct-GKD profile the teacher produces top-k token IDs and log-probabilities on actor-generated tokens; top-k mass and overlap are diagnostics, not an explicit tail bucket. In the PG k1 profile the target is only the sampled token's teacher log-probability, bound to the rollout actor log-probability and policy version; current actor log-probability is recomputed at update time. The rewarded PG profile adds a separately logged task advantage from the closed exact-answer provider; it has no critic, value model or GRPO semantics. Local runtime strategies may keep quantized roles resident, swap movable unquantized roles, or share a compatible backbone while preserving distinct role identities in provenance.

Export boundary

miniverl export-verl --run runs/my-opd --target-verl v0.8.0 --out scaleout
miniverl bridge materialize scaleout --download --offline
miniverl bridge doctor scaleout --json

The export bundle carries PEFT, Parquet, config and provenance artifacts, but is reported as launchable: false until exact base snapshots are materialized and the pinned upstream checks pass. Read the materialization workflow. Upstream parse/load smoke, artifact completeness, launchability and distributed execution are separate statuses, so the report shows exactly how far a bundle has progressed.

Common errors

  • Unknown field: capture a resolved config and remove fields outside the documented profile; inspect-only compilation can still explain known unsupported values.
  • Algorithm field rejected: select the explicit PG-k1 profile for its narrow sampled policy-loss path, or a grouped profile for Parquet n>1. Arbitrary reward code, critics, value models, KL penalties, multiple teachers and distributed counts remain unsupported.
  • Interpolation rejected: resolve Hydra/OmegaConf in your trusted verl environment first. miniVERL will not execute ${...}.
  • High-risk reinterpretation not accepted: inspect miniverl plan, then pass --accept-local-reinterpretations for an external config. Packaged profiles carry reviewed acceptance metadata.
  • Prompt schema mismatch: validate the Parquet prompt column as structured role/content messages, or convert it explicitly.
  • Tokenizer mismatch: actor and teacher scoring require structural identity for the shared token space; a legacy behavioral fingerprint is not proof.
  • CUDA out of memory: reduce context, response length or physical batches; keep logical update semantics unchanged. See single-GPU planning.
  • Bundle not launchable: materialize exact snapshots and install the pinned verl commit; inspect the reported blocker. Read scale-out materialization.

Next: follow the OPD quickstart, inspect the compatibility policy, or review all limitations.

0