8000 evals: --repeat N runs the cases N times and reports which results hold by Jiminyang1 · Pull Request #10 · Jiminyang1/minibot · GitHub
[go: up one dir, main page]

Skip to content

evals: --repeat N runs the cases N times and reports which results hold - #10

Merged
Jiminyang1 merged 1 commit into
mainfrom
feat/eval-repeat
Nov 17, 2026
Merged

Jiminyang1 merged 1 commit into
mainfrom
feat/eval-repeat

Conversation

@Jiminyang1
@Jiminyang1 Jiminyang1 commented Nov 15, 2026 •
Copy link
Copy Markdown
Owner

Why

The README already said it: with ten cases run once each, one flip is ten points of noise, so --lessons compare was a hint and not evidence. Every later change to prompts, memory or lessons needs a number that is not noise. This is that instrument.

Change

  • run --repeat N runs the selected cases N times as runs NAME-t1 … NAME-tN. Each run keeps its own table, results file and failure report.
  • A stability report follows: per case pass (every run), flaky (some runs) or fail (no run), and the pass rate as mean with min and max.
  • --lessons compare --repeat N pools the runs. A case counts as changed when its pass count moves (1/3 → 3/3), not when one run differed.
  • With --repeat 1 (the default) the output is the same as before, except that the comparison now says "runs" where it said "cases".

Tests

Four new tests in tests/test_evals_lessons.py. Full suite: 441 tests OK, ruff check . clean.

First real use: a baseline

run --local --repeat 3 on deepseek-v4-pro, on this branch:

  • 30 of 30 runs pass. No case is flaky. The golden set is saturated: it can show a regression, but it cannot show an improvement.
  • A pass hides a tool failure. In file-edit-guarded (change one word in a two-line file), edit_file rejected the model's first edit in 3 of 3 runs with invalid_args. The model gave old_text without the line's trailing newline. I replayed the three recorded calls against the real tool and got the same rejection. The model then called exec several times (a silent stub in the sandbox) and finished with write_file, which replaces the whole file. The case took 7, 7 and 13 model calls and 49k to 112k tokens, and it scored as a pass each time. The fix for edit_file is not in this pull request.
  • A turn with no tool call sends about 6,140 input tokens. That is the fixed size of the system prompt plus the tool definitions.

Not done

  • Cost (calls, tokens) is not part of the stability report.
  • The Langfuse path (run_experiment) was not run with --repeat; only --local was. The code path is the same loop, but it is unverified against the real service.
  • No new cases. The set still has one memory case and none for search_history, schedules or the heartbeat.

🤖 Generated with Claude Code

One run of ten cases cannot tell a real change from the model's own
variance: one flip moves the pass rate by ten points. With --repeat,
each case gets a pass count, the report names the cases that pass every
time, never, or only sometimes, and the pass rate gets a range.

--lessons compare pools the repeats: a case counts as changed when its
pass count moves (1/3 -> 3/3), not when one run happened to differ.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@Jiminyang1
Jiminyang1 merged commit ad098cd into main Nov 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0