Living build log for miniVERL (mini-verl / miniverl / CLI miniverl).
A checkbox is not evidence: every completed item names the command that was run
and what it printed.
Last updated: 2026-07-28.
| item | current state |
|---|---|
| public repository | https://github.com/DaoyuanLi2816/mini-verl |
| release-code merge | public main advanced to 4287682873e3e1f4858827822befc714f5478296 through PR #6; local main, origin/main and ls-remote matched after the merge |
| previous integration | PR #4 and PR #5 are merged |
| release branch | v0.2-final-release was merged and deleted |
| final integration | PR #6 merged on 2026-07-28; all eight PR checks and both post-merge main workflows passed |
| lifecycle fix | merged on main: destructive, idempotent cleanup and explicit cold-start/arm context boundaries are implemented |
| public default-branch health | post-merge ci run 30417663806 and build run 30417663821 completed successfully |
| final-pass status | complete locally: 1004 CPU tests, 5 GPU tests, 129 focused offline tests, dual Transformers compatibility, build, clean-wheel, README-command, link, schema, JSON/JSONL, privacy and release-workflow gates pass |
| version | 0.2.0; no public v0.2.0 tag, GitHub Release or PyPI publication is claimed |
| PyPI | https://pypi.org/pypi/miniverl/json returned HTTP 404 on 2026-07-28; a pending publisher can create the project on first OIDC upload, but its registration has not been verified |
| GitHub environment | pypi created through the repository API on 2026-07-28; custom deployment policy permits only tags matching v* |
external publication blocker | PyPI pending publisher not yet registered; both available browser contexts reached the PyPI login page, so account-level pending state could not be verified |
| Hugging Face adapter | public at https://huggingface.co/DaoyuanLi/mini-verl-qwen3-1.7b-protocol-teacher, immutable head 23323751318135484c06c043b1f9b9e7016dd89f |
| scientific artifact | schema-v2 JSON remains byte-identical at SHA-256 53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc |
Focused final-pass evidence:
pytest -q tests/unit/test_offline_contract.py tests/integration/test_hf_backend_offline.py tests/unit/test_config.py-- 129 passed. Actualtrain,benchmark,evalandexport-adaptercommands also completed with--offline; the export loaded the cached pinned Qwen3 revision and wrote a standard PEFT adapter.- Socket denial exposed a Transformers 5.x PEFT auto-detection request even
with the top-level local-only flag. Version-compatible PEFT probe kwargs now
carry
local_files_onlyexplicitly on 5.x without duplicating the parameter on 4.x; missing model, tokenizer, full adapter snapshot and partial adapter snapshot tests all record zero socket attempts. - Hub adapter validation returns public provenance plus one exact local snapshot and its config, weights and manifest paths. PEFT receives only that local snapshot; public manifests retain no cache path.
release.ymlpassed actionlint 1.7.12. Static tests prove manual dispatch and branch pushes cannot publish, av*tag push can, distributions build once, and PyPI verification precedes GitHub Release creation.ruff check .,ruff format --check .andmypy srcpass. The complete CPU suite reports 1004 passed, 5 deselected; the CUDA suite reports 5 passed, 1004 deselected on the RTX 4080.- Independent Python 3.12 environments with Transformers 4.51.3 and 5.14.1 each report 124 passed for the offline Qwen3/PEFT/config compatibility bundle.
- A fresh isolated build produced the 0.2.0 wheel and sdist and both passed
twine check. A wheel-only environment confirmed torch absent and core commands ready; a second[train]environment completeddemo --fast,inspectandreport. - All 117 local Markdown targets, 47 Markdown anchors and 14 external Markdown URLs passed. Ten representative runtime JSONL files (115 records) parsed strictly; package/schema/SVG/privacy/release tests report 88 passed.
- The frozen benchmark remains byte-identical at SHA-256
53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc.
Currently failing commands:
- None. The only remaining blocker is external account state: the PyPI pending publisher has not been verifiably registered.
Exact next actions:
- Register and verify the exact PyPI pending publisher documented in
docs/release-checklist.md. - Re-check PyPI name availability immediately before publication.
- Do not create
v0.2.0while the pending publisher remains unverified; once verified and publication is explicitly authorized, the tag-only workflow publishes and verifies PyPI before creating the GitHub Release.
This section records the final local evidence snapshot prepared for integration through PR #4. The PR link is authoritative for its later merge state.
| item | evidence snapshot |
|---|---|
| public repository | https://github.com/DaoyuanLi2816/mini-verl; fetched origin/main is 3383f2b9a3c595e0fa143fecdc27522ab368b27f |
| working branch | v0.2-protocol-aligned-opd; foundational commits are pushed and the measured-result integration passes the final local gates |
| version | 0.2.0 release candidate; no tag or publication has been created |
| current PR | draft #4 |
| GPU runs | protocol teacher, adapter export/gate, 4 GPU tests and all 10 benchmark arms completed locally on the RTX 4080 |
| PyPI/release | not published, no tag, no upload authorized |
Completed v0.2 work in the current worktree:
- Benchmark schema/config v2 separates common and cold-start overrides, performs a declared structured diff before model loading, records resolved hashes and cumulative accounting, and retains v1 read/JSON-schema compatibility.
- Strict OPD freshness is the default and rejects multiple optimizer updates
from one rollout batch; explicit replay is labeled
online_distillation_with_replay. - The ambiguous public
loss.ce_weightname is replaced bysampled_token_nll_weightfor distillation; a nonzero legacy name is rejected. - Frozen standard PEFT teacher adapters can be validated, loaded and exported with checksums and provenance. The no-network local Qwen-compatible round trip, freeze checks, wrong-base/tokenizer checks and missing-file checks pass.
recipes/qwen3_1.7b_protocol_teacher_sft.yamlvalidates and its exactminiverl train ... --dry-run --jsoncommand succeeds.- Policy evaluation now records strict success as primary plus lenient
diagnostic success, valid tool-call rate/count, final-format validity and
average turns. Protocol-token accuracy is explicitly
null/not_applicablefor unaligned free-running trajectories. k == Vbypasses tail smoothing; temperature-squared claims are narrowed and a deterministic 48-cell gradient sweep covers three divergences, four temperatures and two logit regimes.- README source-install and CUDA installation order are corrected in English and Chinese.
- Candidate A reached 100% strict held-out tool-policy success, passed its prespecified 50% gate and was exported with complete checksums; candidates B and C were not run by design.
- The five-arm equal-update GPU benchmark completed at both prespecified seeds. Cold start was 75% twice; SFT and protocol-teacher OPD were 100% twice; raw-teacher and privileged-context OPD were 0% twice.
- Published schema-v2 provenance is made portable before hashing. The result JSON, Markdown and data-bound SVG pass schema, privacy and consistency tests.
Evidence executed on this branch:
ruff check .andruff format --check .— clean, 148 files formatted.mypy src/miniverl— clean, 73 source files.- complete offline/CPU suite — 966 passed, 4 GPU tests deselected, 86.55% branch coverage over 6405 statements.
- Transformers 4.51.3 boundary environment — offline Qwen3/PEFT/config bundle, 117 passed.
python -m buildandpython -m twine check dist/*— 0.2.0 sdist and wheel built, both metadata checks passed.- committed benchmark schema exactly matches
miniverl schema; every recipe validates and the protocol-teacher dry run reports 24 optimizer steps and 192 oracle traces. - focused benchmark/config/cache suite — 129 passed before the adapter slice.
- no-network PEFT adapter suite — 14 passed.
- policy metrics, benchmark v2, packaging, toy pipeline and offline HF bundle — 119 passed; its only failure was a duplicate test tokenizer ID, corrected and rerun as 1 passed.
- loss/property/temperature sweep suite — 45 passed.
- protocol-teacher recipe
validate --jsonandtrain --dry-run --json— valid, 24 planned optimizer steps and 192 oracle traces. pytest -q -m gpu— 4 passed on the RTX 4080.- Candidate A training — 24 optimizer steps in 554.9 s; final strict/lenient success, valid-call rate and final-format validity all 100% on 24 tasks.
- Adapter export/validation — checkpoint tree
e9c42893b861e371dd48e2c151940a198e22eff2f91649ca6a5303c525c5ee4c, adapter weights8df7e7bc1b8283b910aa13bc4173083ae20c838bcacb366d7dbcabc7b310b994, manifest502bca7489c6fe161ebf198d2a1b4622123d4f958885a7e4714c6a02a2e1ac43. - Complete GPU benchmark — 5 arms x 2 seeds, all 10 completed; published JSON
SHA-256
53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc. - Fresh 0.2.0 build — sdist and wheel passed
twine check; a wheel-only venv ran--help,--versionanddoctor --jsonwith torch absent, while a second[train]venv completeddemo --fastand inspected its trajectories.
Measured scientific conclusion: protocol competence prevents the observed raw/privileged-teacher collapse, but protocol-teacher OPD only ties SFT at 100% on both seeds. There is no measured OPD-over-SFT advantage.
Exact integration steps at this snapshot:
- Commit and push the portable artifacts and updated interpretation, then make draft PR #4 ready once remote checks are green.
- Merge, verify
mainand its public artifacts, then close superseded dependency PRs without creating a tag, release or PyPI upload. - Preserve the two-seed result as the v0.2 baseline for a future, less-saturated task-family experiment.
At this snapshot, phase 7 is at v0.2 release-candidate integration. The protocol-teacher experiment, both benchmark seeds and all local gates are complete; PR #4 contains the integration state.
| item | value | how |
|---|---|---|
| OS | Windows 11 Pro 10.0.22631 | platform.platform() |
| GPU | NVIDIA GeForce RTX 4080, 16376 MiB, driver 596.49 | nvidia-smi --query-gpu=name,memory.total,driver_version |
| dev interpreter | CPython 3.12.13 (.venv, uv-managed) |
uv venv --python 3.12 |
| torch | 2.13.0+cu130, torch.cuda.is_available() == True |
python -c "import torch; ..." |
| transformers | 5.14.1 | importlib.metadata.version |
| peft | 0.19.1 | idem |
| bitsandbytes | 0.50.0, bnb.nn.Linear4bit present |
idem |
PyPI name miniverl |
AVAILABLE (HTTP 404 on /pypi/miniverl/json, 2026-07-27) |
Invoke-WebRequest |
Confirmed twice against huggingface.co/api/models/... with a negative control
(an all-zero SHA returns 404, so the endpoint really validates revisions):
| role | model | revision | notes |
|---|---|---|---|
| student | Qwen/Qwen3-0.6B |
c1899de289a04d12100db370d81485cdf75e47ca |
Apache-2.0, 28 layers, hidden 1024 |
| teacher | Qwen/Qwen3-1.7B |
70d244cc86ccca08cf5af4e1e306ecf908b1ad5e |
Apache-2.0, 28 layers, hidden 2048 |
tokenizer.json is byte-identical across the pair
(sha256 aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4).
config.vocab_size is 151936 while len(tokenizer) is 151669 -- the vocabulary
is padded. miniVERL sizes the cache and top_k from the model's output
dimension, never from the tokenizer; a GPU test asserts both numbers.
This table is retained as the v0.1 evidence baseline. Current v0.2 evidence is recorded in the takeover checkpoint above and will replace this table after the GPU experiment and clean-install verification finish.
| gate | command | result |
|---|---|---|
| lint | ruff check . |
clean |
| format | ruff format --check . |
clean, 138 files |
| types | mypy src/miniverl |
clean, 71 source files |
| tests | pytest -q -m "not gpu and not network" --cov=miniverl |
934 passed, 4 deselected, 85% branch coverage over 5911 statements |
| GPU tests | pytest -q -m gpu |
4 passed |
| build | python -m build |
sdist + wheel built |
| metadata | python -m twine check dist/* |
PASSED for both |
| wheel completeness | enumerate src/miniverl against the wheel |
all 77 entries present |
| clean install (core) | wheel into a fresh venv | --help, --version, doctor --json work; torch absent |
| clean install (train) | wheel + [train] into a second fresh venv |
doctor resolves a CPU-only torch and reports GPU training / QLoRA unavailable with the right remedy; both examples/ scripts run to completion |
| determinism | two miniverl demo runs in the clean venv, artifact-by-artifact |
checkpoints and trajectories bitwise identical; eval.json differs only in seconds, throughput, timestamp and run id |
| artifact portability | every *.json / *.jsonl parsed with parse_constant set to reject |
strict-valid (this gate found 16 bare NaN tokens, now fixed) |
| published results | jsonschema validate against benchmarks/schema/ + personal-information scan |
both result files valid and clean |
| docs links | every relative markdown link and #anchor resolved |
37 files, 0 broken |
| CLI/docs parity | every miniverl <cmd> in the docs matched against the Typer app |
all 11 commands documented, no phantom commands |
| unfinished markers | rg "TODO|FIXME|XXX|HACK|NotImplementedError" src tests examples scripts |
none |
tests/unit/test_losses_exact.py-- 21 passed. Brute-force Python references for forward KL, reverse KL and beta-JSD; an orientation test that catches an accidental KL reversal; thelog 2bound on symmetric JSD; finiteness at logit scale 1e4; finite gradients; fp16/bf16 upcast; theT^2contract.tests/unit/test_losses_bucketed.py-- 30 passed.log1mexpagainst a reference; tail-mass validity;k == Vconverging to the exact loss; the data-processing-inequality bound for k in {1,2,8,32} across all three divergences; monotonicity in k; a bounded reverse-KL tail penalty.tests/unit/test_chunked_equivalence.py-- 22 passed. The two-stage detach/backward reproduces the unchunked gradient for chunk sizes 1, 5 and 37 across all three divergences; zero-weight positions provably contribute nothing; CE mixing is convex;loss_scaleaffects gradients only.tests/property/-- 14 passed under Hypothesis, including the data-processing inequality evaluated in float64 to1e-9.
tests/unit/test_token_provenance.py-- 30 passed. A mask that marks atool_resulttoken trainable is rejected; spans must tile the sequence; position 0 can never be a target; privileged-context alignment computes per-segment offsets and rejects a target-token mismatch, a length-changed segment, a missing segment, a duplicate key and a tokenizer mismatch.
Oracle solves 6/6 train tasks for every (environment, difficulty) pair;
train/eval/test prompt sets are pairwise disjoint. SQLite lockdown, all
BLOCKED: DROP, INSERT, UPDATE, ATTACH, PRAGMA,
SELECT * FROM sqlite_master (authorizer), SELECT randomblob(10) (function
whitelist), two statements in one call.
tests/integration/test_toy_pipeline.py-- SFT, offline KD and OPD each run to completion and write every documented artifact; OPD increments its policy version once per cycle; offline KD announces that it is reusing fixed targets; no stored trajectory ever marks tool output trainable.tests/integration/test_resume_and_swap.py-- an interrupted-and-resumed run matches an uninterrupted one to1e-6parameter-for-parameter;swapmatchesresidentto the same tolerance; checkpoints contain no pickle; the OOM retry only halves the chunk and gives up with actionable advice.tests/integration/test_hf_backend_offline.py-- a realQwen3ForCausalLMbuilt from a config (no download): adapter resolution through PEFT, selected-position projection matching a full forward to1e-4, KV-cache generation matching a cacheless reference token for token, and gradients reaching only the LoRA tensors.
-
Full recipe,
recipes/qwen_consumer_gpu_calc.yaml, run idrtx4080-calc-opd: 16 optimizer steps in 481.1 s; held-out greedy success 0.0% -> 100.0% on 12 tasks. The 8-cycle SFT cold start did most of that: the first OPD rollout batch already scored 83.3%. -
One-cycle smoke: peak 4.251 GiB allocated / 4.762 GiB reserved; strategy resolved to
residentbecause a quantized model cannot be moved off the accelerator; projection chunk 256; 0 OOM retries; student 10,092,544 trainable LoRA parameters; teacher 1,720,574,976 parameters. -
Throughput probe: NF4 11.19 tok/s, bf16 LoRA 12.84 tok/s (14.12 with determinism off). A 14-token prefill costs 37.0 ms and a cached one-token step costs 30.9 ms, so decoding here is kernel-launch bound, not compute bound.
-
Protocol-teacher GPU benchmark v2 (
gpu-calc-hard-equal-update-v2, five arms x two prespecified seeds, 12 matched optimizer updates per continuation):arm seed 1234 seed 20260727 mean train s cold-start-only 75.0% 75.0% 0.1 sft-continued 100.0% 100.0% 86.4 opd-raw-teacher 0.0% 0.0% 444.0 opd-privileged-context 0.0% 0.0% 531.5 opd-protocol-sft-teacher 100.0% 100.0% 523.8 The protocol teacher prevents the collapse and ties SFT; it does not beat SFT. The complete portable schema-v2 artifact is
benchmarks/results/gpu-calc-hard-equal-update-v2.json. -
Matched-budget GPU benchmark (
benchmarks/configs/gpu_calc_hard.yaml, chainedhardcalculator split, legacy schema v1, single seed, 12 matched optimizer steps from one shared cold start). Result published inbenchmarks/results/rtx4080-calc-hard-matched.json:arm steps success cold-start-only 0 62.5% sft-continued 12 100.0% 6B5Eopd-bucketed-k64 12 0.0% opd-privileged-context 12 0.0% On-policy distillation lost, and the privileged-context prediction was wrong. Decoding the final evaluations showed the two OPD arms fail differently: the standard teacher drops the opening
<tool_call>tag and wraps answers in its own<answer>tags (83.3% invalid calls), while the privileged teacher emits clean output and no tool calls at all (1.00 turns, 0.0% invalid) because it knows the answer and so never needs one.scripts/attribute_failures.pyre-scores the collected trajectories with a lenient parser and shows formatting explains 4 of 24 failures for the standard arm and none for the privileged arm, so neither number is a verifier artifact. Full write-up indocs/rtx4080-baselines.md; both executions of the benchmark are reported and nothing was discarded. -
Matched-budget CPU benchmark (
recipes/benchmark_calc.yaml, 7 arms x 2 seeds, all 14 completed): published inbenchmarks/results/cpu-toy-calc-matched.json. Changing the seed moves one arm by up to 58 points (cold-start-only: 66.7% at seed 1234, 8.3% at seed 20260727) while changing the objective moves it by at most 4 points at a fixed seed. This is a parity check -- it shows all seven arms includingexact_full_vocabrun to completion under identical budgets -- and the table is inbenchmarks/README.mdso a reader can see that no ranking is available. The first attempt of this benchmark restarted the cosine schedule and damaged every arm; that attempt is recorded but not published, because it measured a harness bug rather than any arm.
Recorded because each one is a class of bug worth remembering.
| defect | how it was found | fix |
|---|---|---|
src/miniverl/models/ excluded from git and the wheel by an unanchored models/ rule in .gitignore |
installing the wheel into a clean venv | anchored every directory rule; CI now enumerates the source tree against the wheel and fails on any git-ignored source file |
Rich markup ate [...], so the missing-extra hint printed pip install "miniverl" |
reading actual CLI output | every dynamic string goes through rich.markup.escape |
A bare install raised ModuleNotFoundError instead of naming the extra |
a CLI test | the scorer import moved inside the constructor; the CLI pre-checks the stack |
exact_full_vocab rejected a recipe that omitted top_k -- the remedy its own error suggested -- and then wrote a config.resolved.yaml that could not be re-read |
an agent-written config test | consult model_fields_set, then normalize to 1 |
weighted_mean rescaled the result for a tiny-but-positive weight sum |
Hypothesis | floor only an exactly-zero sum |
EnvironmentConfig.name was a hard-coded regex, making register() useless |
writing the custom-environment example | validate against the registry |
render_final could emit a block that parsed back truncated |
an agent-written protocol test | escape closing tags in JSON, refuse them in plain text |
configure_logging(None) silently downgraded --log-level DEBUG |
an agent-written utils test | only an explicit level changes the level |
The report dropped stored rollouts and printed 0.000 GiB for CPU runs on a GPU machine |
an agent-written reporting test | fill the display budget; decide on the run's device |
| Sampling drew from a CUDA tensor with a CPU generator | the first GPU smoke test | always draw on the CPU, which also makes a seed reproduce across devices |
| Resume replayed completed cycles with a different LR schedule | the resume-equivalence test | train() continues at the checkpointed cycle; the teacher fit is wrapped in fork_rng |
| The toy model could not learn to copy | measuring, not assuming | learned absolute positions -> RoPE; 48 traces -> 256 traces |
ArmResult.run_dir wrote an absolute path, carrying a username and home directory into a file meant for pull requests |
a new privacy test over benchmarks/results/ |
a validator reduces any path to its final component |
| Schema-v2 resolved configs and invocation still carried local adapter/run paths | inspecting the first complete GPU result before publication | recursively replace absolute provenance paths with portable <local>/<name> values before hashing or publishing; privacy and chart-source tests cover the result |
eval.enabled: false did not suppress cycle-triggered evaluation |
timing the first GPU benchmark attempt | periodic evaluation now checks both the interval and the enabled flag; the incomplete attempt is preserved under runs/benchmarks/_aborted/ and the complete benchmark was rerun |
The committed JSON schema was never checked against the model it is generated from, though schema.py claimed the two could not drift |
writing the schema-validation test | a test asserts the committed file equals json_schema() |
16 bare NaN tokens in metrics.jsonl and eval.json, which strict JSON parsers reject |
diffing two identical demo runs | tokens_per_solved_task is null when nothing was solved, via schema.finite_or_none; a test parses every artifact with parse_constant set to reject |
TerminationReason.ENVIRONMENT_ERROR was declared but never assigned |
auditing for dead public API | a raising tool now ends the policy rollout with it; the oracle path still raises, because a truncated oracle trace would silently degrade every SFT target |
log1mexp clamped at a fixed -1e-7, silently saturating tail mass below float32 resolution regardless of dtype |
the documentation pass, then Hypothesis in float64 | the clamp derives from finfo(dtype).eps; the data-processing-inequality property now models the 1 / q_tail error amplification instead of using a flat tolerance |
Both example scripts printed a bare 0.0% with no context, which reads as a broken example |
running them in a clean install | each prints the same "expected, not a capability claim" note that miniverl demo prints |
None. Every gate in the table above passes.
- Protocol alignment removes collapse but has not produced an OPD-over-SFT advantage. The protocol-trained teacher reached 100% strict competence before downstream selection and its OPD arm then reached 100% on both seeds, versus 0% twice for raw and privileged teachers. SFT also reached 100% twice. The result supports teacher-protocol competence as the explanation for the collapse, but the small saturated evaluation cannot rank protocol OPD above verified SFT.
- The toy backend cannot rank methods. It solves only the
easysplit, where SFT saturates. Measured:mediumandhardstay at 0% even after 700 SFT steps. The CPU benchmark is therefore a parity check, not a ranking. - Only two GPU seeds on one task and machine. The primary v0.2 comparison repeats all arms twice, while legacy GPU artifacts remain single-seed. No significance or cross-task/model/hardware generalization is claimed.
- Transformers API drift. CI now tests the Qwen3 introduction boundary
(4.51.x) and the supported 5.x major. A future rename beyond the explicit
<6cap would still need a reviewed compatibility change.
- Finish the clean-wheel/core and clean-source-training install gates, plus every executable README command that does not require rerunning the full GPU experiment.
- Commit and push the measured artifacts/docs, make draft PR #4 ready, and repair any remote check that is an actual failure rather than an approval gate.
- Merge only after green CI, verify public
mainand artifact hashes, then close superseded dependency PRs #1-#3. Do not tag or publish to PyPI.
- Publishing to PyPI or creating a tag/release. Not authorized for this v0.2 iteration. The OIDC job is hard-disabled until a trusted publisher is registered and a separate reviewed change explicitly enables publication.
- A GPU CI runner.
.github/workflows/gpu.ymlis manual-dispatch only and targets a self-hostedcudarunner that does not exist for this repository. The prespecified GPU tests and benchmark therefore run locally on the RTX 4080, with artifacts and checksums recorded for review.
See the v0.2 release-hardening status for
current evidence and the historical table for the v0.1 baseline. Known
limitations are enumerated in docs/limitations.md.