Repository navigation
Expand file tree
/
Copy pathCITATION.cff
More file actions
124 lines (124 loc) · 5.05 KB
/
Copy pathCITATION.cff
File metadata and controls
124 lines (124 loc) · 5.05 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
8000
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
cff-version: 1.2.0
title: "miniVERL: A bounded single-GPU runtime for verl-style OPD"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.9.0
date-released: 2026-08-13
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
url: "https://github.com/DaoyuanLi2816/mini-verl"
abstract: >-
miniVERL runs a documented subset of verl-style on-policy distillation on one
consumer GPU. It is a local single-GPU runtime for a documented subset of verl v0.8
on-policy distillation. It consumes typed verl-shaped configuration and
Parquet prompts, executes actor rollout, teacher scoring and actor update in
one inspectable process, and exports standard PEFT, Parquet and configuration
artifacts through a fail-closed pinned bridge. Its native core also supports
multi-turn, tool-aware on-policy distillation: a student language
model samples its own tool-using trajectories against deterministic local
environments, a teacher scores exactly the states the student visited, and
the student is updated with token-level distributional supervision on its own
generated tokens only. It implements exact full-vocabulary forward KL,
reverse KL and Jensen-Shannon divergences as well as a compressed top-k plus
tail coarse-graining, enforces per-token provenance so that tool output can
never become a training label, and stores teacher targets in a versioned,
checksummed, pickle-free cache with policy-version enforcement.
Around that core it provides supervised fine-tuning, recorded-provenance
preference optimization and knowledge distillation; shared-backbone
execution that switches student, teacher and reference
roles across adapters on one set of base weights to fit low-memory hardware;
padded update batching; deterministic replay; transactional checkpoints and
cross-process run locks; and mechanism studies such as RecoveryBench that are
published with their negative and mixed results intact. A fail-closed
artifact bridge exchanges configs, LoRA adapters, tokenizers and Parquet
datasets with one pinned verl profile, verifying what it can and refusing to
imply the rest: it does not launch distributed jobs or establish broad
algorithmic parity. miniVERL is designed for one personal CUDA GPU,
automatically selects bf16 or fp16, and requires neither Ray nor a cluster.
Published performance is measured on one RTX 4080; other GPU models use the
same code path but remain unmeasured. The v0.7 external-alignment study
terminated at its preregistered checkpoint-selection gate: two declared
lineages and eight candidates all scored 0/64 retained JSONNav utility, so
no teacher qualification, continuation method comparison or reserved final
test was run.
authors:
- family-names: Li
given-names: Daoyuan
email: lidaoyuan2816@gmail.com
keywords:
- on-policy distillation
- knowledge distillation
- llm alignment
- preference optimization
- large language models
- tool use
- agents
- PEFT
- QLoRA
- consumer GPU
- single GPU
- post-training
- reproducibility
references:
- type: article
title: "Distilling the Knowledge in a Neural Network"
authors:
- family-names: Hinton
given-names: Geoffrey
- family-names: Vinyals
given-names: Oriol
- family-names: Dean
given-names: Jeff
year: 2015
url: "https://arxiv.org/abs/1503.02531"
notes: "Source of the high-temperature forward-KL correction; reverse-KL and JSD use it only as an explicit heuristic."
- type: article
title: >-
On-Policy Distillation of Language Models: Learning from Self-Generated
Mistakes
authors:
- family-names: Agarwal
given-names: Rishabh
year: 2024
url: "https://arxiv.org/abs/2306.13649"
notes: "Generalized JSD and training on student-sampled sequences."
- type: article
title: "On-Policy Context Distillation for Language Models"
authors:
- family-names: Ye
given-names: Tianzhu
- family-names: Dong
given-names: Li
- family-names: Wu
given-names: Xun
- family-names: Huang
given-names: Shaohan
- family-names: Wei
given-names: Furu
year: 2026
url: "https://arxiv.org/abs/2602.12275"
notes: "The privileged-context teacher mode follows this formulation."
- type: article
title: "Entropy-Aware On-Policy Distillation of Language Models"
authors:
- family-names: Jin
given-names: Woogyeol
- family-names: Min
given-names: Taywon
- family-names: Yang
given-names: Yongjin
- family-names: Wei
given-names: Dennis
- family-names: Zhou
given-names: Yi
- family-names: Kadhe
given-names: Swanand Ravindra
- family-names: Baracaldo
given-names: Nathalie
- family-names: Lee
given-names: Kimin
year: 2026
url: "https://arxiv.org/abs/2603.07079"
notes: >-
Motivates recording per-token teacher entropy. Entropy-aware divergence
mixing is a roadmap item and is not implemented in miniVERL.