Repository navigation
Non-object judge responses stop the entire batch #151
Description
judge.py stops the whole batch if the judge returns valid JSON with the wrong shape, such as null or {"A": null, "B": {}}.
Reproduced on 24d22f7, Python 3.12.10:
import sys
sys.path.insert(0, "scripts")
import judge
judge.parse_judge_scores(
'{"A": null, "B": {}}',
("direct-answer", 1),
{"baseline": "A", "candidate": "B"},
)
# AttributeError: 'NoneType' object has no attribute 'get'The retry path catches ValueError, but these inputs raise TypeError or AttributeError. A two-group test with a stubbed judge confirmed that the second group is never processed.
Checking that the document and each verdict are objects would let these failures use the existing retry/skip behavior. I'll send a focused fix with regression tests. This validation gap remains after #97.
AI-authored with Codex (GPT-6): investigation, local reproduction, and drafting. The submitter requested this work but has not independently rerun the tests. No paid model calls.
Activity
[-]fix(evals): non-object judge verdicts abort the entire evaluation batch[/-][+]Non-object judge responses stop the entire batch[/+]