Repository navigation
Tags
Tags
Restore latest.json and the web data from the run's own snapshot Run 42256415982 published results/runs/2026-11-06T17-24-53-000Z.json, which matches the raw artifacts of the runs it merges. The results/latest.json and apps/web/src/data/eval-results.json written in the same commit (8716254) do not: eleven Codex rows carry the attempt counts from run 42044644850 rather than 42220222687, and two alerting-001 rows carry docs and skills fields from run 42250228870. Pass and fail agree in every row; first-attempt passes read 104 where the snapshot says 102. Both files are now the snapshot's content, unchanged. The cause is two merge runs publishing over each other, filed separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Publish the 1 September matrix: 101/114, one execution date The first clean full matrix under the current base prompt. 114 cells, six experiments, nineteen scenarios, every row executed on 1 September — which is what #66 asks for before publishing resumes, and the export confirms it: no carried rows, no provenance warnings, one execution date across the file. Replaces a snapshot from 24 August that was measured under the previous base prompt and produced by a local sweep rather than a CI run, so the published numbers now link to a workflow run a reader can open. Published by hand from the run's own artifacts rather than by the workflow. `publish-results` was skipped on 1 September because `EVALS_PUBLISH` was `false`; re-running that job today exported correctly — the log shows the same 114 rows and the same 101/13 split — and then failed to push. A re-run checks out the SHA the original run used, and main has moved six merges since, so the push was a non-fast-forward and the rebase-and-retry loop did not recover it. The artifacts are the same either way; only the commit path differs. Filed separately. What the run says: claude-code-sonnet-5 19/19 codex-gpt-5.6 18/19 codex-gpt-5.6-no-skills 18/19 claude-code-sonnet-5-no-skills 17/19 codex-gpt-5.4-mini-no-skills 15/19 codex-gpt-5.4-mini 14/19 Skills delta +2, 0 and -1. The weak model's figure has now read -3, -2 and -1 on successive clean measurements, so the sign has replicated and the magnitude has not. Thirteen failures, and the largest cluster is real: four of six arms failed `verification-001-stripe-express`, three of them on the same check. Triaged before publishing, because that is the shape a scorer defect makes. It is not one — exactly one run configured the Hookdeck source with the secret sitting in its own `.env`, and the four that failed used placeholders like `whsec_xxx` and `whsec_replace_me`. That is the failure this scenario was built to catch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK
Publish the corrected snapshot: 78/90 Two scenarios were mis-scoring and both are fixed, so investigate-001 and alerting-001 were re-run across all six experiments at three attempts. Eight cells move from fail to pass and no agent behaved differently. compare-snapshots reports all eight under "not attributable to the agent", which is the correct reading: these are corrections, not improvements. Refs #13, #22, #28 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
Publish cumulative run totals for the page's top counter The counter is a claim about the whole record, not the latest run, and neither latest.json nor a sum over index.json can answer it: snapshots overlap, so summing rows counts the same run several times (600 for a record holding 180). Identity is the pairing plus ranAt, which is stamped at execution and copied verbatim on republish. Not runId: only 6 of 600 rows carry one, so including it splits older rows by snapshot and inflates the count. Refs #21 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA