8000 GitHub - wilsonwu-ai/smoleval: Eval harness for small code models: HumanEval + MBPP, sandboxed execution, unbiased pass@k, held-out set locking. · GitHub
[go: up one dir, main page]

Skip to content
wilsonwu-aiPublic

About

Eval harness for small code models: HumanEval + MBPP, sandboxed execution, unbiased pass@k, held-out set locking.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

smoleval

A small, honest eval harness for small code models. You hand it a model checkpoint; it hands back a number you can trust.

Built for one experiment: can a ~366M-parameter model trained on curated reasoning data beat Microsoft's phi-1-small? Answering that requires a measuring instrument that can't fool you — so this harness runs the two standard code benchmarks, executes every answer in a sandbox instead of judging it by eye, uses the unbiased pass@k estimator, and locks away half the questions before anyone sees a single result.

One command in, one table out:

### humaneval — 164 tasks, 20 samples/task

| metric | value |
|---|---|
| pass@1 | 0.4512 |
| pass@10 | 0.6100 |

How a score gets made

flowchart LR
    A["Task:<br/>a function signature<br/>+ description"] --> B["Model writes<br/>n candidate answers"]
    B --> C["Sandbox runs each<br/>answer against<br/>the task's tests"]
    C --> D["pass / fail<br/>for every answer"]
    D --> E["pass@k:<br/>one honest number"]
    E --> F["Report:<br/>markdown table +<br/>results/*.json"]
Loading

The pieces, in plain terms:

  • The benchmarks. HumanEval is 164 short Python problems: here is a function signature and a description, write the body. MBPP is 500 beginner-level Python tasks phrased as one-sentence instructions. Both come with hidden tests that decide pass or fail.
  • Execution, not opinion. An answer counts as correct only if it actually runs and passes the tests. No human judgment, no model-grading- a-model. The interpreter cannot be talked into agreeing with you.
  • pass@k. The model gets n tries per problem (say 20). pass@1 answers the question "if you took one answer at random, how often would it work?" — pass@10, the same for ten tries. Computing this naively from samples is biased (it flatters the model), so the harness uses the unbiased estimator from OpenAI's Codex paper, verified here against brute-force enumeration.

Install

git clone https://github.com/wilsonwu-ai/smoleval.git
cd smoleval
python -m venv .venv && source .venv/bin/activate
pip install -e .          # harness only (dryrun + openai backends)
pip install -e ".[hf]"    # + torch/transformers for local checkpoints

Works on macOS (Apple Silicon via MPS) and Linux (CUDA or CPU). Execution sandboxing is POSIX-only; on Windows use WSL.

The protocol (do these in order)

The core discipline: seal half the exam before anyone studies. If you run an eval, look at the score, tweak the model, and run it again, you are slowly tuning the model to those exact questions — the score inflates and stops meaning anything. The fix is to split the tasks in two up front: iterate as much as you want on one half (dev), and touch the other half (heldout) exactly once, at the very end, for the number you report.

flowchart TD
    L["1 — Lock the split.<br/>Half the tasks are sealed away<br/>before anyone sees any result."] --> DEV
    subgraph DEV["2 — Iterate freely on the dev half"]
        direction LR
        B1["base checkpoint"] --> B2["after SFT"] --> B3["next experiment ..."]
    end
    DEV --> H["3 — Report the held-out half<br/>once, at the very end.<br/>That number is the result."]
Loading

1. Lock the held-out set first, before anyone sees a single result:

smoleval lock --tasks humaneval,mbpp --fraction 0.5 --seed 1234
git add heldout.lock.json && git commit -m "Lock held-out eval split"

The lockfile records dataset sha256 fingerprints, so it refuses to run if the data underneath it ever changes. Committing it makes the seal public and time-stamped — nobody, including you, can quietly redraw the split later.

2. Evaluate the base model when pretraining finishes:

A base model is what pretraining produces: it continues text, it does not answer questions — nobody has taught it that a question is something you respond to. You might assume it therefore can't score on a benchmark at all. Not so, and this is worth internalizing:

smoleval smoke --backend hf --model /path/to/checkpoint
smoleval run --backend hf --model /path/to/checkpoint --tasks humaneval \
    --heldout heldout.lock.json --split dev

Don't wait for SFT to run HumanEval. It is a completion benchmark — signature + description in, function body out — and continuing text is natively what a base model does; --mode complete (the default) exists for exactly this. phi-1-base scored 29% pass@1 with no fine-tuning of any kind (arXiv:2306.11644, Table 1). Calibrate expectations down from there: that was 1.3B on a Python-only diet, so at 366M with a split data mix the signal to watch is "meaningfully above zero," not "close to 29" — and pass@1 is noisy at low absolute scores, so treat small deltas between checkpoints as noise until they persist. Read the number alongside the loss curve and the smoke completions (does it produce syntactically plausible Python? does it stay on topic?). Add --ppl somefile.txt for perplexity on held-out text.

3. After SFT, run the real thing:

# quick greedy check (n=1)
smoleval run --backend hf --model /path/to/ckpt --tasks humaneval \
    --heldout heldout.lock.json --split dev

# proper pass@1 (Codex-paper settings: n=20, temp 0.2)
smoleval run --backend hf --model /path/to/ckpt --tasks humaneval,mbpp \
    --n-samples 20 --temperature 0.2 --k 1 \
    --heldout heldout.lock.json --split dev

# proper pass@10 needs more sampling diversity: temp 0.8
smoleval run --backend hf --model /path/to/ckpt --tasks humaneval,mbpp \
    --n-samples 20 --temperature 0.8 --k 1,10 \
    --heldout heldout.lock.json --split dev

# the final, report-once number
...same command... --split heldout

(Temperature is the randomness dial: 0 means the model always picks its single most likely next token; higher values let it explore. pass@10 only means something if the ten tries actually differ, hence the higher temperature.)

Each run writes results/<name>/summary.json (config + metrics, so every number is reproducible) and samples.jsonl (every completion with its execution outcome, for error analysis).

Backends

backend use case
hf local checkpoint dir or hub id; auto-picks cuda > mps > cpu
openai any OpenAI-compatible server (vLLM, llama.cpp) — eval a model that lives on a rented GPU box from your laptop
dryrun no model; canonical replays reference solutions (must score ~1.0), empty must score 0.0, mixed checks the pass@k math

The checkpoint never has to move. If the model lives on a rented GPU instance, serve it there and point the harness at it through an SSH tunnel:

flowchart LR
    subgraph LAPTOP["Your laptop"]
        S["smoleval<br/>--backend openai<br/>--base-url localhost:8000"]
    end
    subgraph GPUBOX["Rented GPU box"]
        V["vLLM serving<br/>the checkpoint<br/>on port 8000"]
    end
    S -- "ssh -L 8000:localhost:8000" --> V
Loading
# on the GPU instance
vllm serve /path/to/checkpoint --port 8000
# on your laptop (ssh -L 8000:localhost:8000 gpu-box)
smoleval run --backend openai --model ckpt --base-url http://localhost:8000/v1 ...

--mode complete (default) is for base models: HumanEval prompts are the bare signature+docstring, MBPP uses the paper's 3-shot [BEGIN]/[DONE] format. --mode instruct phrases tasks as instructions and extracts fenced python code blocks from the reply — use it for post-SFT chat models (with --chat on the openai backend if the server applies a chat template).

Things that will bite a 366M GPT-2-tokenizer model

  • Context length. The model can only see 1024 tokens at once, and MBPP's 3-shot prompt (three worked examples before the real task) can approach that limit by itself. The hf backend clamps generation to the space available and warns; if you see the warning, drop to --mbpp-shots 1 (record the change — shots affect scores).
  • Greedy + pass@10 don't mix. Temperature 0 returns n identical samples; the harness warns. Use temp 0.8 for pass@10.
  • phi-1 numbers for calibration (from the paper): phi-1 (1.3B) 50.6% HumanEval pass@1, 55.5% MBPP; phi-1-small (350M) 45% HumanEval. Note phi-1 was Python-only — a three-way Python/math/STEM data mix at 366M is fighting for capacity phi-1-small spent entirely on code.

Sandbox

The code being executed was written by a language model — treat it as untrusted. Every candidate runs inside layered containment, and a pass requires proof the tests actually ran, not just a clean exit:

flowchart TD
    subgraph P["Fresh process — whole process group killed on wall-clock timeout"]
        subgraph R["CPU, memory, and file-size limits"]
            subgraph I["python -I — isolated interpreter, minimal env, scratch dir, destructive os/shutil/subprocess calls stubbed out"]
                C1["candidate code loads"] --> C2["tests run"] --> C3["success sentinel<br/>written after tests"]
            end
        end
    end
    P --> V{"exit code 0<br/>AND sentinel present?"}
    V -- "yes" --> PASS["pass"]
    V -- "no" --> FAIL["fail"]
Loading

The sentinel matters because of a real failure mode: a completion that calls sys.exit(0) exits cleanly before the tests ever run. Exit code alone would score that as a pass; requiring a marker written after the tests closes the hole. The candidate is also compiled as its own file, so from __future__ imports in generated code stay legal.

That's solid against accidents (infinite loops, memory bombs, stray os.system) but it is not a hardened boundary against an adversarial payload — don't run as root; use a container if you want belt and suspenders.

Validation

How do you trust the measuring instrument itself? By feeding it inputs whose scores are known in advance:

  • 59 unit/integration tests (pytest), including an end-to-end CLI run on fixture data with exact expected pass@k values, and regression tests for the fun failure modes: sys.exit(0) masquerading as a pass, from __future__ imports, retry storms on deterministic HTTP 4xx.
  • Canonical replay: feed every task's official reference solution through the full pipeline as if a model had written it. Every one should pass — and on Linux x86-64, all 164 HumanEval and all 500 MBPP do. (This caught a real quirk: Mbpp/367's test_setup_code uses a class the solution defines, so setup must execute after the candidate code.)
  • The platform footnote: on macOS arm64 the replay is 498/500. Two MBPP tasks (180, 493) fail their own reference asserts because Apple's math library and Linux's disagree about the last decimal place of a trigonometric chain — one ulp, the smallest possible difference between two floating-point numbers — and those tasks check the result with exact ==. That is a dataset property, not a harness bug, but the consequence is real: MBPP scores from different machines can disagree by up to 0.4%. Compare checkpoints on one platform, and report which.
  • pass@k estimator verified against brute-force enumeration of all C(n,k) subsets.

Provenance

  • HumanEval from openai/human-eval (MIT), fetched once and sha256-pinned in your cache (~/.cache/smoleval).
  • MBPP (full, 974 tasks) from google-research (CC-BY-4.0), using the paper's splits: test = ids 11–510, few-shot examples from ids 2–4.
  • Execution guard adapted in spirit from OpenAI's human-eval harness.

About

Eval harness for small code models: HumanEval + MBPP, sandboxed execution, unbiased pass@k, held-out set locking.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

0