Release 0.2.0: runs as first-class rows by ynishi · Pull Request #44 · ynishi/evalhub

ynishi · 2026-09-26T04:55:54Z

0.2.0 makes an Eval's runs first-class. An Eval is now a versioned header plus a set of runs that belong to the record, not to a version: each run is a row written by run_id, with its own conditions (the header's facets as defaults), status, metrics and attachments, and it can be overwritten, archived or deleted without appending a header version. The verdict on a run moves to the Card (run_results), and GET /evals/{ns}/{name}/runs reads runs × metrics × the chosen Cards' judgements in one page. Four hashes keep it verifiable: the header's content_hash, each run's content_hash, the record's runs_hash, and a Card's used-set hash, from which changed_since_card is derived. This is the 9-stage change of #24, landed as one release.

Upgrading a hosted database

evalhub migrate for 0.2.0 applies the 0003 DDL and then a one-shot data migration, 0003_runs_split, recorded in data_migrations so it never runs twice. serve refuses to start until both have run.

Runs are collected from every live 1.0 version by run_id (highest seq wins) and become rows. pass / fail → ok, error → error with error.kind: unrecorded, skipped → skipped with skip_reason: unrecorded; the dropped outcome is kept as meta.v0.2.0_migration.outcome; the source version's header facets are copied onto each run.
Every live 1.0 version body becomes its 2.0 header (no runs, schema 2.0). Header content_hash values change; the old and new hash of every version are in the audit log (migration.runs_split). Versions that differed only in runs become identical header versions.
Each Card's used runs are rebuilt from the Eval version it points at, so a run that changed since shows as changed at once. A Card's relations keep resolving because they point at version ids, not hashes. Used run ids with no run are skipped and recorded as migration.card_runs_unmatched (0.1.x never checked attrs.runs); these are expected, not errors.
The step is one transaction: a failure (or the release command's 5-minute timeout) rolls it back and leaves it pending. A run_id it cannot accept is named in the error, and nothing is written.

Production deploy (docs/hosting.md, "Upgrading to 0.2.0"; not the plain fly deploy of other releases), from the checkout of the v0.2.0 tag:

Snapshot: fly volumes snapshots create <volume-id> --app evalhub-db (and a pg_dump).
Stop writes: fly machine update <machine-id> --autostart=false --skip-start --app evalhub, then fly machine stop <machine-id> --app evalhub; /api/v1/healthz must fail. Otherwise the old binary can store a 1.0 body after the migration and nothing would split it.
Deploy: fly deploy --ha=false. The release command logs applied data migration 0003_runs_split: …. If the Machine is still stopped afterwards, fly machine start <machine-id> --app evalhub.
Confirm: fly releases --app evalhub and fly ssh console --app evalhub -C "evalhub --version" (→ evalhub 0.2.0); /api/v1/healthz and /openapi.json (info.version stays the contract's 1.0.0) confirm liveness only.
Smoke: bash deploy/fly/smoke.sh (header POST, two run PUTs, a Card with run_results, the projection, anonymous reads after publishing).

Upgrading a client

0.1.x	0.2.0
`POST /evals/{ns}/{name}` with `runs[]` in the body	`POST` the header without `runs`, then `PUT …/runs/{run_id}` per run or `POST …/runs:batch`
`runs[].outcome` (`pass` / `fail` / `error` / `skipped`)	the run's `status` (`ok` / `error` with `error.kind` / `skipped` with `skip_reason`), and the pass / fail in the Card's `run_results[]`
`"schema": "evalhub.eval/1.0"`	`"schema": "evalhub.eval/2.0"` (`GET /schemas/eval-2`)
`"schema": "evalhub.card/1.0"`	`"schema": "evalhub.card/1.1"` (adds `run_results`); `evalhub.card/1.0` is still accepted

An evalhub.eval/1.0 body with runs[] is accepted and converted in 0.2.0 only: stored as a 2.0 header plus run rows, with a Deprecation: @1790294400 header (RFC 9745), converted_from: "evalhub.eval/1.0" and converted_runs in the response. 0.3.0 rejects it with 422 runs_moved, as 0.2.0 already does for a 2.0 header carrying runs. The issue for that removal is opened when 0.2.0 lands, with its date.
Two things a 1.0 client notices already in 0.2.0: the returned content_hash is the converted 2.0 header's, not the hash of the posted body; and re-posting a body that omits a run no longer drops that run (runs are rows now; archive or delete it).
A run PUT must carry run_id in the body, equal to the path's.

API

Runs under /api/v1/evals/{ns}/{name}: PUT /runs/{run_id} (201 created / 200 updated / 200 unchanged), POST /runs:batch (all or nothing; a refusal lists every failing element), GET /runs (the projection: cards=, where, sort, limit 1–200, cursor, include=archived,deleted for members), GET / PATCH (archived) / DELETE (tombstone) /runs/{run_id}. Runs follow the Eval's visibility; every route is 404 for an Eval the caller may not see.
Projection paths, bracket form: run_id, status, error.kind, started_at, ended_at, facet keys (model.id, …), fingerprint.{facet}, metrics[{ns}/{name}], results[{card}][{metric}].value / .label, ext.*. Per Card the page reports runs_used, used_set_hash, posted_used_set_hash and changed_since_card.
GET /evals/{ns}/{name}[@…] carries runs: { count, by_status, archived, deleted, runs_hash } (archived / deleted for members only). The comparison view (GET /evals/…/cards) compares same_harness / same_model against every run a Card used and adds runs_used, used_set_hash, changed_since_card.
Redaction: a public Card's run_results entries that name an Eval the reader may not see are removed and counted in withheld.run_results (GET and POST /cards/query); a writer naming a run of an Eval they may not see gets run_unknown, as for a missing run.
Limits ([limits]): body_bytes 16 MiB (413 body_too_large, every endpoint; was axum's 2 MiB default), batch_runs 1,000 (413 batch_too_large), run_results 100,000 per Card version (422 too_many_run_results). Refused, never truncated.
Schemas: GET /schemas/eval-2 (the header) and GET /schemas/run are new; GET /schemas/eval stays the 1.0 document until 0.3.0.
Error codes (new): runs_moved, run_status_detail, run_id_invalid, run_id_mismatch, run_result_value_or_label, run_results_eval_unknown, run_unknown, run_not_in_used_set, too_many_run_results, batch_duplicate_run_id (422); run_deleted (409); batch_too_large, body_too_large (413).
Audit: run.create / run.update / run.archive / run.unarchive / run.delete, migration.runs_split, migration.card_runs_unmatched.
info.version of /openapi.json stays 1.0.0.

Crates

evalhub-schema, evalhub-core and evalhub-query are published at 0.2.0 (the store and the server ship as ghcr.io/ynishi/evalhub:0.2.0). just semver-check against 0.1.4 on crates.io, before the bump:

Crate	Verdict
`evalhub-schema`	requires a new major (0.2 on 0.x): 5 lints — `Eval.runs` removed, `eval::Run` and `eval::Outcome` removed, `Card.run_results` added to an exhaustively constructible struct, 13 `ErrorCode` variants added
`evalhub-core`	no semver update required (196 checks: 196 pass, 58 skip); bumped with the workspace
`evalhub-query`	requires a new major (0.2 on 0.x): 1 lint — `Column::Run` and `SortKey::Run` added to exhaustive enums

At 0.2.0 the same command reports "no semver update required" for all three (0 checks, 254 skipped: a major bump skips the lints).

Changes by stage

Schema: Run leaves the Eval body; per-run facets, status and metrics; Card.run_results; evalhub.eval/2.0 #36 Schema: Run leaves the Eval body; per-run facets, status and metrics; Card.run_results; evalhub.eval/2.0 (Schema: Run leaves the Eval body; per-run facets, status and metrics; Card.run_results; evalhub.eval/2.0 #27)
Core: validate a Run, refuse runs in an Eval body, check run_results, fingerprint a run, publish the run hash formulas #37 Core: validate a Run, refuse runs in an Eval body, check run_results, fingerprint a run, publish the run hash formulas (Core: validate a Run, refuse runs in an Eval body, check run_results, fingerprint a run, publish the run hash formulas #28)
Query: a run projection target — path table, run columns, sort keys and cursor for GET /evals/{ns}/{name}/runs #38 Query: a run projection target — path table, run columns, sort keys and cursor for GET /evals/{ns}/{name}/runs (Query: a run projection target — path table, run columns, sort keys and cursor for GET /evals/{ns}/{name}/runs #29)
Store: run tables, run put/batch/archive/delete, runs_hash, audit actions, GC and download references for run attachments #39 Store: run tables, run put/batch/archive/delete, runs_hash, audit actions, GC and download references for run attachments (Store: run tables, run put/batch/archive/delete, runs_hash, audit actions, GC and download references for run attachments #30)
Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs #40 Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs (Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs #31)
Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has #41 Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has (Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has #32)
Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI #42 Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI (Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI #33)
Web UI and deploy scripts: the Eval page reads /runs and shows status, error kind, metrics and withheld counts; the Fly smoke test PUTs runs #43 Web UI and deploy scripts: the Eval page reads /runs and shows status, error kind, metrics and withheld counts; the Fly smoke test PUTs runs (Web UI and deploy scripts: the Eval page reads /runs and shows status, error kind, metrics and withheld counts; the Fly smoke test PUTs runs #34)
This PR: README (run paths, [limits], schema ids, Upgrading from 0.1.x), the crate docs' final pass, the 0.2.0 bump (Release 0.2.0: README endpoints and limits, crate docs, semver check, bump, release notes with the 0.1.x → 0.2.0 client mapping #35)

Verified

Command	Outcome
`cargo fmt --check`	clean
`cargo clippy --workspace --all-targets -- -D warnings`	clean
`cargo test -p evalhub-schema`	20 passed
`cargo test -p evalhub-core`	62 passed
`cargo test -p evalhub-query`	27 passed
`cargo test -p evalhub-store` (Docker)	96 passed
`cargo test -p evalhub-server` (Docker)	82 passed, no `.snap.new`
`just sqlx-check` (throwaway `postgres:16-alpine`, migrations 0001–0003)	exit 0
`cargo deny check`	advisories ok, bans ok, licenses ok, sources ok
`RUSTDOCFLAGS=-D warnings cargo doc --no-deps` (five crates)	clean
`just semver-check` (before the bump)	schema: major, 5 lints; core: none; query: major, 1 lint
`just semver-check` (at 0.2.0, on the commit)	all three: "no semver update required" (0 checks, 254 skipped: a major bump skips the lints)
`just package` (on the commit)	`evalhub-schema` 66422, `evalhub-core` 75555, evalhub-query 68769 bytes, gate ok
`just e2e` (on the commit)	`e2e: 18 passed, 0 failed`; Playwright 6 passed
`Cargo.lock`	only the five `evalhub-*` versions changed

Not run: deploy/fly/smoke.sh (targets the hosted service; step 5 above, after the deploy).

Closes #24
Refs #35

…0.2.0 client mapping (0.2.0) Stage 9 of #24, the last: documentation and the bump, no production code. README gains the run projection's path table in bracket form with the query rules and an example, the [limits] keys with their env form and defaults, the schema ids as served (eval-2 and run are new, eval stays the 1.0 document until 0.3.0, card covers 1.1 and 1.0), and a section on upgrading from 0.1.x: a POST with runs becomes a header POST plus run PUTs, outcome becomes the run's status plus the Card's run_results, evalhub.eval/1.0 with runs is accepted and converted in 0.2.0 only with a Deprecation header and 0.3.0 refuses it with runs_moved, the returned content_hash of a converted body is the 2.0 header's, re-posting a body that omits a run no longer drops that run, header hashes change on migration with old and new in the audit log, versions that differed only in runs become identical header versions, and Card relations keep resolving because they point at version ids. The five crate docs each name the model as it applies to the crate: header plus runs, the used set, the four hashes, the visibility of runs and run_results, where the limits live, and the data migration as the one exception to write-once versions. Two stale claims went: the store doc offered export?format=bundle, which answers 501, and the server doc described 0.3.0's refusal differently from README. cargo semver-checks against 0.1.4 on crates.io: evalhub-schema 5 major lints (Eval.runs, eval::Run and eval::Outcome removed, Card.run_results and the ErrorCode variants added), evalhub-query 1 major (Column::Run and SortKey::Run on exhaustive enums), evalhub-core no update required. On 0.x that is the minor bump: [workspace.package] version and the four internal pins go to 0.2.0 and cargo update -w moves the five workspace entries in Cargo.lock and nothing else. Verified: cargo fmt --check; cargo clippy --workspace --all-targets -D warnings; cargo test -p evalhub-schema (20), -core (62), -query (27), -store (96, Docker), -server (82, Docker); just sqlx-check against a migrated throwaway Postgres; cargo deny check (advisories, bans, licenses, sources ok); cargo doc --no-deps with -D warnings for the five crates; just semver-check before the bump (above) and after (all lints skipped at the new major). just package and just e2e run on this commit are in the pull request body. Refs #35 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>

ynishi merged commit e8d8b6d into main Sep 26, 2026
4 checks passed

This was referenced Sep 26, 2026

Release 0.2.0: README endpoints and limits, crate docs, semver check, bump, release notes with the 0.1.x → 0.2.0 client mapping #35

Closed

0.3.0: remove the evalhub.eval/1.0 compatibility window; a 1.0 body answers 422 runs_moved #45

Closed

Navigation Menu

Repository navigation

Release 0.2.0: runs as first-class rows - #44

Release 0.2.0: runs as first-class rows#44
ynishi merged 1 commit into
mainfrom
docs/release-0.2.0

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Upgrading a hosted database

Upgrading a client

API

Crates

Changes by stage

Verified

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant