Repository navigation
evals: --repeat N runs the cases N times and reports which results hold - #10
Merged
Merged
Conversation
One run of ten cases cannot tell a real change from the model's own variance: one flip moves the pass rate by ten points. With --repeat, each case gets a pass count, the report names the cases that pass every time, never, or only sometimes, and the pass rate gets a range. --lessons compare pools the repeats: a case counts as changed when its pass count moves (1/3 -> 3/3), not when one run happened to differ. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The README already said it: with ten cases run once each, one flip is ten points of noise, so
--lessons comparewas a hint and not evidence. Every later change to prompts, memory or lessons needs a number that is not noise. This is that instrument.Change
run --repeat Nruns the selected cases N times as runsNAME-t1 … NAME-tN. Each run keeps its own table, results file and failure report.pass(every run),flaky(some runs) orfail(no run), and the pass rate as mean with min and max.--lessons compare --repeat Npools the runs. A case counts as changed when its pass count moves (1/3 → 3/3), not when one run differed.--repeat 1(the default) the output is the same as before, except that the comparison now says "runs" where it said "cases".Tests
Four new tests in
tests/test_evals_lessons.py. Full suite: 441 tests OK,ruff check .clean.First real use: a baseline
run --local --repeat 3ondeepseek-v4-pro, on this branch:file-edit-guarded(change one word in a two-line file),edit_filerejected the model's first edit in 3 of 3 runs withinvalid_args. The model gaveold_textwithout the line's trailing newline. I replayed the three recorded calls against the real tool and got the same rejection. The model then calledexecseveral times (a silent stub in the sandbox) and finished withwrite_file, which replaces the whole file. The case took 7, 7 and 13 model calls and 49k to 112k tokens, and it scored as a pass each time. The fix foredit_fileis not in this pull request.Not done
run_experiment) was not run with--repeat; only--localwas. The code path is the same loop, but it is unverified against the real service.search_history, schedules or the heartbeat.🤖 Generated with Claude Code