Repository navigation
Expand file tree
/
Copy pathCITATION.cff
More file actions
102 lines (102 loc) · 3.46 KB
/
Copy pathCITATION.cff
File metadata and controls
102 lines (102 loc) · 3.46 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
cff-version: 1.2.0
title: "miniVERL: Run verl experiment semantics on one GPU"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.15.0
date-released: 2026-09-10
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
url: "https://github.com/DaoyuanLi2816/mini-verl"
abstract: >-
miniVERL compiles supported verl experiment semantics for one NVIDIA GPU.
Its versioned profiles consume resolved configuration and Parquet prompts,
run PPO with an independent critic or critic-free reward-driven algorithms,
and support on-policy distillation with explicit teachers. Typed trajectories,
role identities, transactional checkpoints and exact replay make local runs
inspectable and resumable. Portable PEFT, critic, Parquet and provenance
bundles support handoff toward pinned upstream verl. Installed PPO and GRPO
walkthroughs exercise the complete workflow. Runtime qualification is measured
on one RTX 4080 and kept separate from task-quality research; distributed
execution is outside the local runtime. Frozen studies preserve their original
negative and mixed results.
authors:
- family-names: Li
given-names: Daoyuan
email: lidaoyuan2816@gmail.com
keywords:
- on-policy distillation
- knowledge distillation
- llm alignment
- preference optimization
- large language models
- tool use
- agents
- PEFT
- QLoRA
- consumer GPU
- single GPU
- post-training
- reproducibility
references:
- type: article
title: "Distilling the Knowledge in a Neural Network"
authors:
- family-names: Hinton
given-names: Geoffrey
- family-names: Vinyals
given-names: Oriol
- family-names: Dean
given-names: Jeff
year: 2015
url: "https://arxiv.org/abs/1503.02531"
notes: "Source of the high-temperature forward-KL correction; reverse-KL and JSD use it only as an explicit heuristic."
- type: article
title: >-
On-Policy Distillation of Language Models: Learning from Self-Generated
Mistakes
authors:
- family-names: Agarwal
given-names: Rishabh
year: 2024
url: "https://arxiv.org/abs/2306.13649"
notes: "Generalized JSD and training on student-sampled sequences."
- type: article
title: "On-Policy Context Distillation for Language Models"
authors:
- family-names: Ye
given-names: Tianzhu
- family-names: Dong
given-names: Li
- family-names: Wu
given-names: Xun
- family-names: Huang
given-names: Shaohan
- family-names: Wei
given-names: Furu
year: 2026
url: "https://arxiv.org/abs/2602.12275"
notes: "The privileged-context teacher mode follows this formulation."
- type: article
title: "Entropy-Aware On-Policy Distillation of Language Models"
authors:
- family-names: Jin
given-names: Woogyeol
- family-names: Min
given-names: Taywon
- family-names: Yang
given-names: Yongjin
- family-names: Wei
given-names: Dennis
- family-names: Zhou
given-names: Yi
- family-names: Kadhe
given-names: Swanand Ravindra
- family-names: Baracaldo
given-names: Nathalie
- family-names: Lee
given-names: Kimin
year: 2026
url: "https://arxiv.org/abs/2603.07079"
notes: >-
Motivates recording per-token teacher entropy. Entropy-aware divergence
mixing is a roadmap item and is not implemented in miniVERL.