A small, honest eval harness for small code models. You hand it a model checkpoint; it hands back a number you can trust.
Built for one experiment: can a ~366M-parameter model trained on curated reasoning data beat Microsoft's phi-1-small? Answering that requires a measuring instrument that can't fool you — so this harness runs the two standard code benchmarks, executes every answer in a sandbox instead of judging it by eye, uses the unbiased pass@k estimator, and locks away half the questions before anyone sees a single result.
One command in, one table out:
### humaneval — 164 tasks, 20 samples/task
| metric | value |
|---|---|
| pass@1 | 0.4512 |
| pass@10 | 0.6100 |
flowchart LR
A["Task:<br/>a function signature<br/>+ description"] --> B["Model writes<br/>n candidate answers"]
B --> C["Sandbox runs each<br/>answer against<br/>the task's tests"]
C --> D["pass / fail<br/>for every answer"]
D --> E["pass@k:<br/>one honest number"]
E --> F["Report:<br/>markdown table +<br/>results/*.json"]
The pieces, in plain terms:
- The benchmarks. HumanEval is 164 short Python problems: here is a function signature and a description, write the body. MBPP is 500 beginner-level Python tasks phrased as one-sentence instructions. Both come with hidden tests that decide pass or fail.
- Execution, not opinion. An answer counts as correct only if it actually runs and passes the tests. No human judgment, no model-grading- a-model. The interpreter cannot be talked into agreeing with you.
- pass@k. The model gets
ntries per problem (say 20). pass@1 answers the question "if you took one answer at random, how often would it work?" — pass@10, the same for ten tries. Computing this naively from samples is biased (it flatters the model), so the harness uses the unbiased estimator from OpenAI's Codex paper, verified here against brute-force enumeration.
git clone https://github.com/wilsonwu-ai/smoleval.git
cd smoleval
python -m venv .venv && source .venv/bin/activate
pip install -e . # harness only (dryrun + openai backends)
pip install -e ".[hf]" # + torch/transformers for local checkpointsWorks on macOS (Apple Silicon via MPS) and Linux (CUDA or CPU). Execution sandboxing is POSIX-only; on Windows use WSL.
The core discipline: seal half the exam before anyone studies. If you
run an eval, look at the score, tweak the model, and run it again, you are
slowly tuning the model to those exact questions — the score inflates and
stops meaning anything. The fix is to split the tasks in two up front:
iterate as much as you want on one half (dev), and touch the other half
(heldout) exactly once, at the very end, for the number you report.
flowchart TD
L["1 — Lock the split.<br/>Half the tasks are sealed away<br/>before anyone sees any result."] --> DEV
subgraph DEV["2 — Iterate freely on the dev half"]
direction LR
B1["base checkpoint"] --> B2["after SFT"] --> B3["next experiment ..."]
end
DEV --> H["3 — Report the held-out half<br/>once, at the very end.<br/>That number is the result."]
1. Lock the held-out set first, before anyone sees a single result:
smoleval lock --tasks humaneval,mbpp --fraction 0.5 --seed 1234
git add heldout.lock.json && git commit -m "Lock held-out eval split"The lockfile records dataset sha256 fingerprints, so it refuses to run if the data underneath it ever changes. Committing it makes the seal public and time-stamped — nobody, including you, can quietly redraw the split later.
2. Evaluate the base model when pretraining finishes:
A base model is what pretraining produces: it continues text, it does not answer questions — nobody has taught it that a question is something you respond to. You might assume it therefore can't score on a benchmark at all. Not so, and this is worth internalizing:
smoleval smoke --backend hf --model /path/to/checkpoint
smoleval run --backend hf --model /path/to/checkpoint --tasks humaneval \
--heldout heldout.lock.json --split devDon't wait for SFT to run HumanEval. It is a completion benchmark —
signature + description in, function body out — and continuing text is
natively what a base model does; --mode complete (the default) exists
for exactly this. phi-1-base scored 29% pass@1 with no fine-tuning of any
kind (arXiv:2306.11644, Table 1).
Calibrate expectations down from there: that was 1.3B on a Python-only
diet, so at 366M with a split data mix the signal to watch is
"meaningfully above zero," not "close to 29" — and pass@1 is noisy at low
absolute scores, so treat small deltas between checkpoints as noise until
they persist. Read the number alongside the loss curve and the smoke
completions (does it produce syntactically plausible Python? does it stay
on topic?). Add --ppl somefile.txt for perplexity on held-out text.
3. After SFT, run the real thing:
# quick greedy check (n=1)
smoleval run --backend hf --model /path/to/ckpt --tasks humaneval \
--heldout heldout.lock.json --split dev
# proper pass@1 (Codex-paper settings: n=20, temp 0.2)
smoleval run --backend hf --model /path/to/ckpt --tasks humaneval,mbpp \
--n-samples 20 --temperature 0.2 --k 1 \
--heldout heldout.lock.json --split dev
# proper pass@10 needs more sampling diversity: temp 0.8
smoleval run --backend hf --model /path/to/ckpt --tasks humaneval,mbpp \
--n-samples 20 --temperature 0.8 --k 1,10 \
--heldout heldout.lock.json --split dev
# the final, report-once number
...same command... --split heldout(Temperature is the randomness dial: 0 means the model always picks its single most likely next token; higher values let it explore. pass@10 only means something if the ten tries actually differ, hence the higher temperature.)
Each run writes results/<name>/summary.json (config + metrics, so every
number is reproducible) and samples.jsonl (every completion with its
execution outcome, for error analysis).
| backend | use case |
|---|---|
hf |
local checkpoint dir or hub id; auto-picks cuda > mps > cpu |
openai |
any OpenAI-compatible server (vLLM, llama.cpp) — eval a model that lives on a rented GPU box from your laptop |
dryrun |
no model; canonical replays reference solutions (must score ~1.0), empty must score 0.0, mixed checks the pass@k math |
The checkpoint never has to move. If the model lives on a rented GPU instance, serve it there and point the harness at it through an SSH tunnel:
flowchart LR
subgraph LAPTOP["Your laptop"]
S["smoleval<br/>--backend openai<br/>--base-url localhost:8000"]
end
subgraph GPUBOX["Rented GPU box"]
V["vLLM serving<br/>the checkpoint<br/>on port 8000"]
end
S -- "ssh -L 8000:localhost:8000" --> V
# on the GPU instance
vllm serve /path/to/checkpoint --port 8000
# on your laptop (ssh -L 8000:localhost:8000 gpu-box)
smoleval run --backend openai --model ckpt --base-url http://localhost:8000/v1 ...--mode complete (default) is for base models: HumanEval prompts are the
bare signature+docstring, MBPP uses the paper's 3-shot [BEGIN]/[DONE]
format. --mode instruct phrases tasks as instructions and extracts
fenced python code blocks from the reply — use it for post-SFT chat
models (with --chat on the openai backend if the server applies a chat
template).
- Context length. The model can only see 1024 tokens at once, and
MBPP's 3-shot prompt (three worked examples before the real task) can
approach that limit by itself. The hf backend clamps generation to the
space available and warns; if you see the warning, drop to
--mbpp-shots 1(record the change — shots affect scores). - Greedy + pass@10 don't mix. Temperature 0 returns n identical samples; the harness warns. Use temp 0.8 for pass@10.
- phi-1 numbers for calibration (from the paper): phi-1 (1.3B) 50.6% HumanEval pass@1, 55.5% MBPP; phi-1-small (350M) 45% HumanEval. Note phi-1 was Python-only — a three-way Python/math/STEM data mix at 366M is fighting for capacity phi-1-small spent entirely on code.
The code being executed was written by a language model — treat it as untrusted. Every candidate runs inside layered containment, and a pass requires proof the tests actually ran, not just a clean exit:
flowchart TD
subgraph P["Fresh process — whole process group killed on wall-clock timeout"]
subgraph R["CPU, memory, and file-size limits"]
subgraph I["python -I — isolated interpreter, minimal env, scratch dir, destructive os/shutil/subprocess calls stubbed out"]
C1["candidate code loads"] --> C2["tests run"] --> C3["success sentinel<br/>written after tests"]
end
end
end
P --> V{"exit code 0<br/>AND sentinel present?"}
V -- "yes" --> PASS["pass"]
V -- "no" --> FAIL["fail"]
The sentinel matters because of a real failure mode: a completion that
calls sys.exit(0) exits cleanly before the tests ever run. Exit code
alone would score that as a pass; requiring a marker written after the
tests closes the hole. The candidate is also compiled as its own file, so
from __future__ imports in generated code stay legal.
That's solid against accidents (infinite loops, memory bombs, stray
os.system) but it is not a hardened boundary against an adversarial
payload — don't run as root; use a container if you want belt and
suspenders.
How do you trust the measuring instrument itself? By feeding it inputs whose scores are known in advance:
- 59 unit/integration tests (
pytest), including an end-to-end CLI run on fixture data with exact expected pass@k values, and regression tests for the fun failure modes:sys.exit(0)masquerading as a pass,from __future__imports, retry storms on deterministic HTTP 4xx. - Canonical replay: feed every task's official reference solution
through the full pipeline as if a model had written it. Every one
should pass — and on Linux x86-64, all 164 HumanEval and all 500 MBPP
do. (This caught a real quirk: Mbpp/367's
test_setup_codeuses a class the solution defines, so setup must execute after the candidate code.) - The platform footnote: on macOS arm64 the replay is 498/500. Two
MBPP tasks (180, 493) fail their own reference asserts because Apple's
math library and Linux's disagree about the last decimal place of a
trigonometric chain — one ulp, the smallest possible difference between
two floating-point numbers — and those tasks check the result with
exact
==. That is a dataset property, not a harness bug, but the consequence is real: MBPP scores from different machines can disagree by up to 0.4%. Compare checkpoints on one platform, and report which. - pass@k estimator verified against brute-force enumeration of all C(n,k) subsets.
- HumanEval from
openai/human-eval(MIT), fetched once and sha256-pinned in your cache (~/.cache/smoleval). - MBPP (full, 974 tasks) from
google-research(CC-BY-4.0), using the paper's splits: test = ids 11–510, few-shot examples from ids 2–4. - Execution guard adapted in spirit from OpenAI's human-eval harness.