Repository navigation
Reject invalid eval scores, trials, and budgets #243
Description
Evaluation inputs currently accept boolean dimension scores because bool is an int subclass, and zero or negative trial counts produce an empty successful run plan. Additionally, --budget-usd nan passes the current comparisons because NaN is unordered; a mocked local repro completed all 3 default trials with a NaN remaining budget instead of rejecting before runner invocation. Reject booleans, require positive trial counts, and require a finite positive budget no greater than $25. Reproduction: python scripts/run_evals.py plan --trials 0; python scripts/run_evals.py run --runner claude --condition baseline --trials 0 --output responses.jsonl; python scripts/run_evals.py run --runner claude --condition baseline --budget-usd nan --output responses.jsonl.
Activity
[-]Reject boolean scores and non-positive eval trial counts[/-][+]Reject invalid eval scores, trials, and budgets[/+]