8000 Tags · hookdeck/evals · GitHub
[go: up one dir, main page]

Skip to content

Tags

Tags

v0.5.0

Toggle v0.5.0's commit message

Verified

This commit was signed with the committer’s verified signature.
leggetter Phil Leggetter
Restore latest.json and the web data from the run's own snapshot

Run 42256415982 published results/runs/2026-11-06T17-24-53-000Z.json,
which matches the raw artifacts of the runs it merges. The
results/latest.json and apps/web/src/data/eval-results.json written in
the same commit (8716254) do not: eleven Codex rows carry the attempt
counts from run 42044644850 rather than 42220222687, and two alerting-001
rows carry docs and skills fields from run 42250228870. Pass and fail
agree in every row; first-attempt passes read 104 where the snapshot
says 102.

Both files are now the snapshot's content, unchanged. The cause is two
merge runs publishing over each other, filed separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

v0.4.0

Toggle v0.4.0's commit message

Verified

This commit was signed with the committer’s verified signature.
leggetter Phil Leggetter
Publish the 1 September matrix: 101/114, one execution date

The first clean full matrix under the current base prompt. 114 cells, six
experiments, nineteen scenarios, every row executed on 1 September — which is
what #66 asks for before publishing resumes, and the export confirms it: no
carried rows, no provenance warnings, one execution date across the file.

Replaces a snapshot from 24 August that was measured under the previous base
prompt and produced by a local sweep rather than a CI run, so the published
numbers now link to a workflow run a reader can open.

Published by hand from the run's own artifacts rather than by the workflow.
`publish-results` was skipped on 1 September because `EVALS_PUBLISH` was
`false`; re-running that job today exported correctly — the log shows the same
114 rows and the same 101/13 split — and then failed to push. A re-run checks
out the SHA the original run used, and main has moved six merges since, so the
push was a non-fast-forward and the rebase-and-retry loop did not recover it.
The artifacts are the same either way; only the commit path differs. Filed
separately.

What the run says:

  claude-code-sonnet-5              19/19
  codex-gpt-5.6                     18/19
  codex-gpt-5.6-no-skills           18/19
  claude-code-sonnet-5-no-skills    17/19
  codex-gpt-5.4-mini-no-skills      15/19
  codex-gpt-5.4-mini                14/19

Skills delta +2, 0 and -1. The weak model's figure has now read -3, -2 and -1
on successive clean measurements, so the sign has replicated and the magnitude
has not.

Thirteen failures, and the largest cluster is real: four of six arms failed
`verification-001-stripe-express`, three of them on the same check. Triaged
before publishing, because that is the shape a scorer defect makes. It is not
one — exactly one run configured the Hookdeck source with the secret sitting in
its own `.env`, and the four that failed used placeholders like `whsec_xxx` and
`whsec_replace_me`. That is the failure this scenario was built to catch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQzUoMAwEBJWpEGVvVzSjK

v0.3.0

Toggle v0.3.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Merge pull request #53 from hookdeck/release-0.3.0

Publish the Outpost snapshot: 104/114

v0.2.0

Toggle v0.2.0's commit message

Verified

This commit was signed with the committer’s verified signature.
leggetter Phil Leggetter
Publish the corrected snapshot: 78/90

Two scenarios were mis-scoring and both are fixed, so investigate-001 and
alerting-001 were re-run across all six experiments at three attempts. Eight
cells move from fail to pass and no agent behaved differently.

compare-snapshots reports all eight under "not attributable to the agent",
which is the correct reading: these are corrections, not improvements.

Refs #13, #22, #28

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA

v0.1.0

Toggle v0.1.0's commit message

Verified

This commit was signed with the committer’s verified signature.
leggetter Phil Leggetter
Publish cumulative run totals for the page's top counter

The counter is a claim about the whole record, not the latest run, and neither
latest.json nor a sum over index.json can answer it: snapshots overlap, so
summing rows counts the same run several times (600 for a record holding 180).

Identity is the pairing plus ranAt, which is stamped at execution and copied
verbatim on republish. Not runId: only 6 of 600 rows carry one, so including it
splits older rows by snapshot and inflates the count.

Refs #21

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nt2Zgjw7STjrnFXYKRRVAA
0