Repository navigation
--allow-unmetered turns off the budget check #181
Description
So I was running evals with a Codex runner and passed --budget-usd 5 --allow-unmetered, thinking the budget would still hold things back. It didn't. The thing just ran every case × trial without stopping and printed Reported cost: $0.0000 at the end. Kind of a problem if someone does this against a paid API.
The budget logic looks at reported_cost, which only goes up when the runner returns a dollar amount:
reported_cost += float(cost or 0)
But with --allow-unmetered, _parse_response always returns cost = None. So reported_cost stays at zero, remaining never hits zero, and this check:
remaining = args.budget_usd - reported_cost
if remaining <= 0:
return 2
never fires. The flag basically makes --budget-usd decorative.
Reacted by Mobina and Eric Li
Activity
MobinaSafarirad commentedon Sep 10, 2026
what do you say?