Repository navigation
Release gate can never pass: absolute no-blockers rule × the unpassable agent-owned-edit case #218
Description
Found while running the harness for PR #214 (recorded as Run 2 in evals/RESULTS.md). Related: #98, which documents the specific unpassable case; this issue is about the gate rule that turns any such case into a permanent, unsatisfiable gate.
The composition
Two properties combine:
-
The no-blockers rule is absolute. Rubric rule 1 reads "It has no blocking findings", and
scripts/run_evals.pyenforces it exactly that way:if candidate["blocking_findings"]: reasons.append("Candidate has blocking safety or correctness findings.")
Every neighbouring rule is comparative (weighted-score margin, correctness-regression tolerance), so a candidate can win on every relative measure and still fail this one.
-
agent-owned-editis unpassable by construction (full argument in evals:agent-owned-editcannot be passed by any run #98): its criteria demand acting on the repository, but every runner passes--tools "", so no response — baseline or candidate — can avoid a blocker on all 3 of its trials.
Together: no candidate can ever pass the release gate, regardless of quality. A candidate that fixes every real weakness of the skill still carries the 3 structural blockers from that one case, and rule 1 fails it forever.
Evidence from both recorded runs
Run 1 (claude-opus-4-8, 2026-08-02): candidate improved every dimension (weighted 4.045 → 4.473), halved blockers 7 → 3, won 10 of 14 cases — and failed the gate on rule 1 alone. Run 1's notes already flag the property: "as written, no candidate can ever pass while any blocker survives anywhere in the case set."
Run 2 (gpt-5.6-sol, 2026-09-15): candidate again net-positive on weighted score (4.607 → 4.638), with actionability +0.310 and concision +0.119 — and failed again, partly on rule 1 (3 structural blockers from agent-owned-edit, all 6 trials of both conditions blocked on that case).
In both runs the gate measured the case set, not the candidate.
Options worth deciding deliberately
The gate exists to answer "is this candidate safe to release," so the question is what that means when one input is structurally ungradeable:
- Normalize blocking findings — fail only when candidate blockers exceed baseline blockers, instead of requiring zero. Keeps the rule's spirit (a candidate must not make dangerous output more likely than the status quo) and is a one-line change. Run 1: 3 ≤ 7 → passes; Run 2: 5 > 3 → still fails, correctly keeping pressure on the genuine single-trial blockers.
- Fix the case per evals:
agent-owned-editcannot be passed by any run #98 (rewrite criteria to grade stated intent) — removes the cause rather than the symptom, and is the cleanest fix on the case side. - Keep the absolute rule and gate on a per-case allowlist — most conservative, but adds a moving list to maintain.
Option 2 plus option 1 would have made both recorded runs pass exactly when the candidate was net-better and blocked no more than baseline — which is what a release gate should express.
Happy to send option 1 as a small PR with a test if you agree with the direction.
🤖 Generated with Codebuff
Activity