8000 v0.6.3 security, artifact integrity and release-state hardening by DaoyuanLi2816 · Pull Request #45 · DaoyuanLi2816/mini-verl · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
181aeaf
Stop executing bundle reward code and validate safetensors payloads
DaoyuanLi2816 Aug 5, 2026
64b3c9e
Reject source/output aliases and fail closed on ambiguous conversions
DaoyuanLi2816 Aug 5, 2026
9e40484
Enforce Parquet bounds while reading instead of afterwards
DaoyuanLi2816 Aug 5, 2026
b2343bc
Give stable and development version claims one canonical source
DaoyuanLi2816 Aug 5, 2026
72d33ad
Escape literal dots in release-state test match patterns
DaoyuanLi2816 Aug 5, 2026
72fd0f2
Give the four legacy figures a real narrow layout and drop the exemption
DaoyuanLi2816 Aug 5, 2026
e0b0b66
Narrow the transactional claim and document the v0.6.3 contracts
DaoyuanLi2816 Aug 5, 2026
e72ec9d
Record the v0.6.3 changelog and measured quality evidence
DaoyuanLi2816 Aug 5, 2026
85a7eb3
Add hostile-bundle and rollback-failure adversarial coverage
DaoyuanLi2816 Aug 5, 2026
55ad463
Record the v0.6.3 takeover audit and measured gate evidence
DaoyuanLi2816 Aug 5, 2026
1269998
Ship release-state.yaml in the sdist so its gate runs outside the che…
DaoyuanLi2816 Aug 5, 2026
8f2d0ce
Record the GitHub repository metadata change
DaoyuanLi2816 Aug 5, 2026
c0f1377
Distinguish an unavailable safetensors reader from a rejected file
DaoyuanLi2816 Aug 5, 2026
447d7f9
Require the official reader for --require-adapter-payload
DaoyuanLi2816 Aug 5, 2026
bfc3dc7
Accept either safetensors level where the official reader may be absent
DaoyuanLi2816 Aug 5, 2026
1d2b43f
Give release state an explicit phase and put numpy in the bridge extra
DaoyuanLi2816 Aug 5, 2026
d07ac0f
Prepare the v0.6.3 release
DaoyuanLi2816 Aug 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ jobs:
- name: CLI smoke (base install)
run: |
python scripts/build_pypi_readme.py --check
python scripts/release_state.py --check
python scripts/check_markdown_links.py
miniverl --help
miniverl --version
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ jobs:
ruff format --check .
mypy src/miniverl
python scripts/build_pypi_readme.py --check
python scripts/release_state.py --check
python scripts/check_markdown_links.py
echo "53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc benchmarks/results/gpu-calc-hard-equal-update-v2.json" | sha256sum --check
pytest -q -m "not gpu and not network" \
Expand Down
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ htmlcov/

# Local working archives. Preserved on disk for provenance, never published.
/.local-archive-*/
/.takeover-archive-*/

# Browser screenshots from scripts/check_docs_visual.py. CI uploads them as a
# workflow artifact; they are never committed.
/docs-visual-screenshots/

# Editors / OS
.vscode/
Expand Down
73 changes: 72 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,76 @@ All notable changes to miniVERL are recorded here. The format follows

## [Unreleased]

## [0.6.3] - 2026-08-05

### Security

- `miniverl bridge doctor` no longer imports the inspected bundle's reward
scaffold. Up to 0.6.2 it used `importlib` `exec_module`, so any top-level
statement in an untrusted bundle ran with the user's privileges as soon as
they asked for a diagnosis, while the report described the result as a
"side-effect-free import". The scaffold is now parsed with `ast.parse` and
verified statically, reporting `not_present`, `syntax_valid`,
`interface_statically_verified` or `trusted_dynamic_import_verified`. Top-level
calls, non-literal assignments, decorators, class-body statements and
call-valued default arguments are rejected because each runs at import time.
- Added `miniverl bridge doctor --trust-and-import-reward-code` for bundles you
produced yourself. It warns before executing anything, reports
`untrusted_code_executed: true`, and does not claim to be a sandbox.
- `import-verl` and `convert-dataset` reject any overlap between an input file
and their intended output family before taking a reservation, covering exact,
relative, symlink, hard-link and Windows case aliases. Previously `--out`
could name the source config: a successful import overwrote it and a rejected
import deleted it while publishing the rejection report. `--overwrite`
replaces a previous output family and never authorizes destroying an input.

### Added

- Adapter safetensors are validated past the header. dtype and shape byte
arithmetic, offset ordering, contiguity and full coverage of the data segment
are checked, then every tensor is materialized through the official reader.
Levels are `not_present`, `header_only`, `payload_structure_validated` and
`tensor_materialization_validated`. A file declaring a 4x4 F32 tensor with no
payload previously passed. The structural pass needs no optional dependency,
so a torch-free install reaches `payload_structure_validated` and reports
`official_reader_status: dependency_missing` instead of implying the file is
broken; `--require-adapter-payload` demands the strongest level and is
therefore not satisfied without the official reader.
- `miniverl convert-dataset --allow-rejected-rows` opts into a partial dataset.
Conversion is otherwise complete-or-nothing, and a partial report carries
`complete_dataset_conversion: false`, `lossless_for_accepted_rows: true` and
the output-row-to-source-row index map.
- `release-state.yaml` and `scripts/release_state.py` give every public
stable/development version claim one canonical source, gated in CI.
- Dedicated 390px layouts for `consumer-runtime-v1-pareto`,
`cost-quality-pareto`, `fresh-vs-frozen` and `recovery-success`.

### Changed

- Conflicting miniVERL extension data across `miniverl_extensions`, the
conversion sidecar and `extra_info.miniverl` now fails the conversion instead
of silently preferring one source. Canonical-equal duplicates are accepted and
recorded. Diagnostics name the row and the locations, never the values.
- Parquet bounds are enforced while reading. The dataset scan streams row groups
through `iter_batches` restricted to string-bearing columns and stops at
`max_rows`/`max_bytes`; schema validation reads the footer only. Both
previously materialized the whole table first. The scan reports
`files_inspected`, `row_groups_read`, `rows_scanned`, `rows_total` and
`bytes_scanned`.
- The bridge output guarantee is stated as transactional publication with
in-process rollback rather than atomic multi-file publication, which it never
provided across `kill -9`, kernel panic or power loss.
- `CITATION.cff` describes the current scope: an auditable single-GPU alignment
and distillation runtime with comparable SFT, DPO, KD and OPD arms,
shared-backbone role switching and a bounded verl artifact bridge.

### Fixed

- The mobile readability exemption list is empty: every public figure is
enforced at the 11px floor at 390px.
- The docs version selector, both READMEs, `PYPI.md` and the quality record no
longer disagree about which release is stable.

## [0.6.2] - 2026-08-05

### Added
Expand Down Expand Up @@ -651,7 +721,8 @@ Same-tokenizer only; one trajectory per forward pass; `swap` unavailable for
quantized models; only Qwen3 and Qwen2 architectures tested; single-seed GPU
results. The full list is in `docs/limitations.md`.

[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.2...HEAD
[Unreleased]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.3...HEAD
[0.6.3]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.2...v0.6.3
[0.6.2]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.1...v0.6.2
[0.6.1]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.6.0...v0.6.1
[0.6.0]: https://github.com/DaoyuanLi2816/mini-verl/compare/v0.5.0...v0.6.0
Expand Down
47 changes: 31 additions & 16 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -1,36 +1,51 @@
cff-version: 1.2.0
title: "miniVERL: On-policy distillation for tool-using agents on one GPU"
title: "miniVERL: Auditable single-GPU alignment and distillation runtime"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.6.2
version: 0.6.3
date-released: 2026-08-05
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
url: "https://github.com/DaoyuanLi2816/mini-verl"
abstract: >-
miniVERL is a single-GPU laboratory for multi-turn, tool-aware on-policy
distillation. A student language model samples its own tool-using
trajectories against deterministic local environments; a teacher scores
exactly the states the student visited; and the student is updated with
token-level distributional supervision on its own generated tokens only.
It implements exact full-vocabulary forward KL, reverse KL and
Jensen-Shannon divergences as well as a compressed top-k plus tail
coarse-graining, enforces per-token provenance so that tool output can never
become a training label, and stores teacher targets in a versioned,
checksummed, pickle-free cache with policy-version enforcement. It is
designed for one personal CUDA GPU, automatically selects bf16 or fp16, and
requires neither Ray nor a cluster. Published performance is measured on one
RTX 4080; other GPU models use the same code path but remain unmeasured.
miniVERL is a single-GPU runtime for comparing alignment and distillation
methods under conditions strict enough that the comparison means something.
Its core is multi-turn, tool-aware on-policy distillation: a student language
model samples its own tool-using trajectories against deterministic local
environments, a teacher scores exactly the states the student visited, and
the student is updated with token-level distributional supervision on its own
generated tokens only. It implements exact full-vocabulary forward KL,
reverse KL and Jensen-Shannon divergences as well as a compressed top-k plus
tail coarse-graining, enforces per-token provenance so that tool output can
never become a training label, and stores teacher targets in a versioned,
checksummed, pickle-free cache with policy-version enforcement.
Around that core it provides supervised fine-tuning, recorded-provenance
preference optimization and knowledge distillation as directly comparable
arms; shared-backbone execution that switches student, teacher and reference
roles across adapters on one set of base weights to fit low-memory hardware;
padded update batching; deterministic replay; transactional checkpoints and
cross-process run locks; and mechanism studies such as RecoveryBench that are
published with their negative and mixed results intact. A fail-closed
artifact bridge exchanges configs, LoRA adapters, tokenizers and Parquet
datasets with one pinned verl profile, verifying what it can and refusing to
imply the rest: it does not launch distributed jobs and does not establish
algorithmic parity with PPO. miniVERL is designed for one personal CUDA GPU,
automatically selects bf16 or fp16, and requires neither Ray nor a cluster.
Published performance is measured on one RTX 4080; other GPU models use the
same code path but remain unmeasured.
authors:
- family-names: Li
given-names: Daoyuan
email: lidaoyuan2816@gmail.com
keywords:
- on-policy distillation
- knowledge distillation
- llm alignment
- preference optimization
- large language models
- tool use
- agents
- PEFT
- QLoRA
- consumer GPU
- single GPU
Expand Down Expand Up @@ -98,4 +113,4 @@ references:
url: "https://arxiv.org/abs/2603.07079"
notes: >-
Motivates recording per-token teacher entropy. Entropy-aware divergence
mixing is a roadmap item and is not implemented in miniVERL v0.6.2.
mixing is a roadmap item and is not implemented in miniVERL.
29 changes: 28 additions & 1 deletion PROJECT_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,34 @@ Living build log for **miniVERL** (`mini-verl` / `miniverl` / CLI `miniverl`).
A checkbox is not evidence: every completed item names the command that was run
and what it printed.

Last updated: 2026-08-04.
Last updated: 2026-08-05.

Canonical release state: releasing `v0.6.3`.
Every public version claim is generated from `release-state.yaml` and gated by
`python scripts/release_state.py --check`.

## v0.6.3 Security, artifact integrity and release-state hardening

| item | current state |
| --- | --- |
| starting state | `origin/main` and local `main` both at `c62e14ba4229f32b3db06b293c615e94654debc6`; tag `v0.6.2` at `bef9f0878eb3280f450aee3868b43d61f0726557`; working tree clean; no open pull request; source version `0.6.3.dev0`; branch `v0.6.3-security-artifact-hardening` cut from that commit |
| immutable baseline | all ten frozen result artifacts plus `docs/generated/verl-bridge-smoke.json` re-hashed at takeover and unchanged; calculator remains `53fc1d4d5b7adee09618d77ad62d4086ba56b78569832d6fc7c3bcd5c2695bbc` |
| reward-code execution | reproduced: a scaffold whose top level wrote `PWNED.txt` created the marker during `_check_reward`, and the report still said `side-effect-free import; scaffold intentionally not executed`. Fixed by `src/miniverl/bridge/reward_static.py`, an AST walk that never imports; levels are `not_present`, `syntax_valid`, `interface_statically_verified`, `trusted_dynamic_import_verified`, and the last one requires `--trust-and-import-reward-code` |
| safetensors validation | reproduced: a header declaring a 4x4 F32 tensor with zero payload bytes returned `(True, '1 tensor header(s)')` while the official reader rejected it with `file not fully covered`. Fixed by `src/miniverl/bridge/safetensors_check.py`; levels are `not_present`, `header_only`, `payload_structure_validated`, `tensor_materialization_validated`, with `--require-adapter-payload` for strict callers |
| source/output aliasing | reproduced: `import-verl --out <source> --overwrite` overwrote the source on success and **deleted** it on rejection while publishing the report. Fixed by `reject_source_output_alias`, which runs before any transaction and covers exact, relative, symlink, hard-link and Windows case aliases |
8080 | conversion semantics | extension data in `miniverl_extensions`, the sidecar and `extra_info.miniverl` is now reconciled: equal content deduplicates, different content fails closed naming row and locations but never values. Invalid rows fail the conversion unless `--allow-rejected-rows` is given, and a partial report carries `complete_dataset_conversion: false` |
| Parquet bounds | reproduced by inspection: the scan called `pq.read_table().to_pylist()` before applying bounds and the schema check read every row. Both now use footer metadata and `iter_batches` per row group; an instrumented test asserts the requested row groups are exactly `[0]` when the bound is two rows into an eight-group file |
| release-state drift | `python scripts/release_state.py --check` reported ten disagreements at takeover, including README stable `v0.6.1`, docs selector `Stable 0.6.1 / Development 0.6.2.dev0` and `quality_floor ... at v0.6.1` inside the `release: 0.6.2` record. All ten are resolved and the gate runs in `ci.yml` and `release.yml` |
| canonical release state | `release-state.yaml` drives the package `__version__`, both READMEs, `PYPI.md`, the docs channel selector, `CITATION.cff`, the changelog comparison link, the release-checklist section, this file's header line and the quality record; a test drives the full `release 0.6.3 → stable 0.6.3 → main 0.6.4.dev0` transition and still reports the prose sections a person must write |
| mobile visual debt | the four-figure exemption list is empty: `consumer-runtime-v1-pareto`, `cost-quality-pareto`, `fresh-vs-frozen` and `recovery-success` each ship a dedicated 390 px layout selected by a `<picture>` media query, and the desktop SVG bytes are unchanged at their pinned SHA-256 values |
| transactional scope | wording is narrowed to transactional publication with in-process rollback; multi-file crash atomicity across `kill -9`, kernel panic or power loss is explicitly not claimed, and the versioned-directory design that would provide it is named as out of scope |
| adversarial audit | tampered bundles were re-sealed with self-consistent `SHA256SUMS` and still rejected: hostile reward code stops at `syntax_valid`, a truncated adapter stops at `header_only`, and conflicting pinned-verl or adapter-versus-config fields fail closed. `--require-verl` is not a back door into execution, and a failing rollback re-raises the original cause rather than masking it |
| defect introduced and fixed in this branch | the first safetensors implementation treated a *dependency gap* as evidence the file was invalid. The torch-free environment installs `safetensors` but not `numpy`, and `safe_open(framework="np")` needs `numpy`; the resulting `ModuleNotFoundError` was caught by a generic handler and reported as `the official safetensors reader rejected adapter_model.safetensors`, failing the model check and nine tests against structurally sound files. This is the mirror of the defect being fixed: the old code read "only a header" as "valid", the new code read "cannot check" as "invalid". `_materialize` now returns `materialized`, `unavailable` or `rejected`; `unavailable` keeps `payload_structure_validated` with `official_reader_status: dependency_missing` and a passing status, because the structural pass is dependency-free. The exact condition is a regression test |
| strict adapter option | `--require-adapter-payload` demands `tensor_materialization_validated`, so an absent official reader does not satisfy it; the same file still passes without the flag |
| local gates | ruff, ruff format and mypy clean; `pytest -m "not gpu and not network"` passes 1,757 with 2 skipped, 6 deselected at 86.02% branch coverage on commit `85a7eb3aee46109ced405a7cbfb2106edeb82516`; `pytest -m gpu` passes 5 on the RTX 4080; `pytest -m network` passes 3; `mkdocs build --strict` clean; the browser gate checks 28 rendered SVG instances across 1440x900, 1024x768, 820x1000 and 390x844 with zero exemptions; `python -m build` and `twine check` both pass. The two skips are symlink creation, which needs privileges on Windows; the hard-link and case-alias cases cover the same guard |
| external metadata | the GitHub repository description was changed from `Auditable on-policy distillation for tool-using agents on one personal GPU.` to `Auditable one-GPU alignment and distillation runtime with shared-backbone training and a fail-closed verl artifact bridge.`, and topics `alignment`, `llm-alignment` and `peft` were added; no existing topic was removed |
| integration | PR [#45](https://github.com/DaoyuanLi2816/mini-verl/pull/45) |
| release state | in progress on `v0.6.3-security-artifact-hardening` |

## v0.6.2 Bridge correctness and responsive visual hardening

Expand Down
Loading
0