Repository navigation
Releases
evalhub 0.4.0
Close the evalhub.eval/1.0 window: a 1.0 body is 422 runs_moved, GET /schemas/eval is gone (0.4.0) (#54)
0.4.0: the evalhub.eval/1.0 compatibility window is closed
Closes #45 and #51. Releases 0.2.0 and 0.3.0 accepted an evalhub.eval/1.0 body (header and runs[] in one document) on POST /evals/{ns}/{name} and converted it at ingest into a 2.0 header and its runs, answering with a Deprecation header and converted_from / converted_runs in the envelope. The schema crate's rule is that the previous major is accepted for one release and then refused; the README and the crate docs said 0.3.0 would refuse it, and 0.3.0 shipped with the window open (#51). 0.4.0 closes it.
Behaviour
POST /evals/{ns}/{name}withschema: evalhub.eval/1.0answers422with exactly one error,runs_movedat/runs, with or withoutrunsand whatever else is wrong with the body. It is the error a 2.0 header carryingrunsalready got. NoDeprecationheader, noconverted_from, nothing is written. The hint namesPUT /evals/{ns}/{name}/runs/{run_id}andPOST /evals/{ns}/{name}/runs:batch.- A 2.0 header carrying
runswhoseattachments[].sha256is malformed also answers that singleruns_moved(previously the sha error was appended after it). GET /schemas/eval(the 1.0 document) is gone:404.GET /schemas/eval-2andGET /schemas/runare unchanged.- Stored data is untouched. A 0.1.x database still migrates:
0003_runs_splitkeeps its own reading of the 1.0 shape insideevalhub-store(unpublished), byte-for-byte the conversionevalhub-coreused to expose. - Producers still sending 1.0 bodies get
422after this deploy; the README's 0.1.x → 0.2.0 mapping stays as the record of what to change.
SDK (why 0.4.0)
just semver-check reports major for evalhub-schema and evalhub-core; evalhub-query is unchanged.
evalhub-schema:eval::v1(Eval,Run,Outcome) andschema_for_eval_v1()removed;SCHEMA_NAMESis[&str; 5](evaldropped; cargo-semver-checks does not flag the array length);RecordKind::Eval.schema_ids()is["evalhub.eval/2.0"],schema_for("evalhub.eval/1.0")isNone,all_schemas()no longer yieldseval;schemas/eval.jsonremoved;ErrorCode::RunsMoveddoc covers the 1.0 declaration.evalhub-core:EvalSchemaanddeclared_schemaremoved (the 1.0 arm of the validator). Theevalmodule went with them:split_v1,SplitV1,UNRECORDED,V1_MIGRATION_META_KEYnow live inevalhub-storefor the data migration only.validate(Eval, …)on a 1.0 body returns[runs_moved].- OpenAPI:
ConvertedRunsDto,converted_from,converted_runsremoved; the web client (schema.d.ts) regenerated.
Docs
Every sentence that promised the removal for 0.3.0 (README §API, §Upgrading from 0.1.x; rustdoc in schema, core, store, server) now records that 0.2.0 and 0.3.0 accepted 1.0 and 0.4.0 closed the window. web/README.md names eval-2 instead of the removed eval.
Verification
cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test -p evalhub-schema (20) / -core (56) / -query (27) / -store (104, Docker) / -server (86, Docker), cargo insta test -p evalhub-schema and -p evalhub-server (no pending snapshots), RUSTDOCFLAGS=-D warnings cargo doc for schema / core / store, just semver-check (major for schema and core before the bump; "no semver update required" for all three at 0.4.0), just package (three crates ok), corepack pnpm check / pnpm lint in web/. RED observed before each behaviour change: schema_ids() still listed 1.0; a 1.0 body validated with zero or unrelated errors; POST answered 201 with converted_from; GET /schemas/eval answered 200; a malformed sha256 produced two errors.
Fly is deployed separately after the release (fly deploy --ha=false); nothing stored changes, so no pre-deploy data check is needed. Producers still posting 1.0 bodies get 422 runs_moved from then on.
SDK crates: evalhub-schema evalhub-core evalhub-query
Image: ghcr.io/ynishi/evalhub:0.4.0
evalhub 0.3.0
Validate ext keys as {ns}/{name} on records, runs and facets: ext_key_invalid (0.3.0) (#49)
0.3.0: ext keys are checked as {ns}/{name}
Fixes #46. The schema crate and the served JSON Schema have always said that ext is keyed by namespace ({ns}/{name}), and nothing checked it: a record or a run with "ext": {"notes": {}} was accepted, stored and served.
Behaviour
- Every key of
extis validated with the same rule as a metric id, at the top level of a record or a run and inside each facet that carries anextof its own (model,task,harness,generation,trial,grading,env). - A violation is
422with oneext_key_invalidper key, at/ext/{key}or/{facet}/ext/{key}. Values are still not looked into. - Stored bodies are not rewritten. The rule applies to writes; a 0.2.0 body with an un-namespaced key keeps being read.
SDK (why 0.3.0)
evalhub_schema::error::ErrorCodegainsExtKeyInvalidand is now#[non_exhaustive], as its doc already asked of clients ("treat an unknown code as a generic 422"). Amatchon it in another crate needs a fallback arm.schemas/error.json, the OpenAPI document and the web client'sErrorCodeunion are regenerated.
Deploy
Before fly deploy, run bash deploy/fly/ext-keys-check.sh (read-only) to list the stored keys a 0.2.0 hub accepted that 0.3.0 refuses on write, and record the result on #46. See docs/hosting.md, "Upgrading to 0.3.0".
Verification
cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test -p evalhub-schema / -core / -query / -server (Docker), cargo insta test -p evalhub-server --test boot, just semver-check, just package, pnpm check / pnpm lint. Store tests not run (no Rust change there).
SDK crates: evalhub-schema evalhub-core evalhub-query
Image: ghcr.io
8000
/ynishi/evalhub:0.3.0
evalhub 0.2.0
Release 0.2.0: runs as first-class rows (#44)
0.2.0 makes an Eval's runs first-class. An Eval is now a versioned header plus a set of runs that belong to the record, not to a version: each run is a row written by run_id, with its own conditions (the header's facets as defaults), status, metrics and attachments, and it can be overwritten, archived or deleted without appending a header version. The verdict on a run moves to the Card (run_results), and GET /evals/{ns}/{name}/runs reads runs × metrics × the chosen Cards' judgements in one page. Four hashes keep it verifiable: the header's content_hash, each run's content_hash, the record's runs_hash, and a Card's used-set hash, from which changed_since_card is derived. This is the 9-stage change of #24, landed as one release.
Upgrading a hosted database
evalhub migrate for 0.2.0 applies the 0003 DDL and then a one-shot data migration, 0003_runs_split, recorded in data_migrations so it never runs twice. serve refuses to start until both have run.
- Runs are collected from every live 1.0 version by
run_id(highestseqwins) and become rows.pass/fail→ok,error→errorwitherror.kind: unrecorded,skipped→skippedwithskip_reason: unrecorded; the droppedoutcomeis kept asmeta.v0.2.0_migration.outcome; the source version's header facets are copied onto each run. - Every live 1.0 version body becomes its 2.0 header (no
runs,schema2.0). Headercontent_hashvalues change; the old and new hash of every version are in the audit log (migration.runs_split). Versions that differed only inrunsbecome identical header versions. - Each Card's used runs are rebuilt from the Eval version it points at, so a run that changed since shows as changed at once. A Card's relations keep resolving because they point at version ids, not hashes. Used run ids with no run are skipped and recorded as
migration.card_runs_unmatched(0.1.x never checkedattrs.runs); these are expected, not errors. - The step is one transaction: a failure (or the release command's 5-minute timeout) rolls it back and leaves it pending. A
run_idit cannot accept is named in the error, and nothing is written.
Production deploy (docs/hosting.md, "Upgrading to 0.2.0"; not the plain fly deploy of other releases), from the checkout of the v0.2.0 tag:
- Snapshot:
fly volumes snapshots create <volume-id> --app evalhub-db(and apg_dump). - Stop writes:
fly machine update <machine-id> --autostart=false --skip-start --app evalhub, thenfly machine stop <machine-id> --app evalhub;/api/v1/healthzmust fail. Otherwise the old binary can store a 1.0 body after the migration and nothing would split it. - Deploy:
fly deploy --ha=false. The release command logsapplied data migration 0003_runs_split: …. If the Machine is still stopped afterwards,fly machine start <machine-id> --app evalhub. - Confirm:
fly releases --app evalhubandfly ssh console --app evalhub -C "evalhub --version"(→evalhub 0.2.0);/api/v1/healthzand/openapi.json(info.versionstays the contract's1.0.0) confirm liveness only. - Smoke:
bash deploy/fly/smoke.sh(header POST, two run PUTs, a Card withrun_results, the projection, anonymous reads after publishing).
Upgrading a client
| 0.1.x | 0.2.0 |
|---|---|
POST /evals/{ns}/{name} with runs[] in the body |
POST the header without runs, then PUT …/runs/{run_id} per run or POST …/runs:batch |
runs[].outcome (pass / fail / error / skipped) |
the run's status (ok / error with error.kind / skipped with skip_reason), and the pass / fail in the Card's run_results[] |
"schema": "evalhub.eval/1.0" |
"schema": "evalhub.eval/2.0" (GET /schemas/eval-2) |
"schema": "evalhub.card/1.0" |
"schema": "evalhub.card/1.1" (adds run_results); evalhub.card/1.0 is still accepted |
- An
evalhub.eval/1.0body withruns[]is accepted and converted in 0.2.0 only: stored as a 2.0 header plus run rows, with aDeprecation: @1790294400header (RFC 9745),converted_from: "evalhub.eval/1.0"andconverted_runsin the response. 0.3.0 rejects it with422 runs_moved, as 0.2.0 already does for a 2.0 header carryingruns. The issue for that removal is opened when 0.2.0 lands, with its date. - Two things a 1.0 client notices already in 0.2.0: the returned
content_hashis the converted 2.0 header's, not the hash of the posted body; and re-posting a body that omits a run no longer drops that run (runs are rows now; archive or delete it). - A run
PUTmust carryrun_idin the body, equal to the path's.
API
- Runs under
/api/v1/evals/{ns}/{name}:PUT /runs/{run_id}(201 created / 200updated/ 200unchanged),POST /runs:batch(all or nothing; a refusal lists every failing element),GET /runs(the projection:cards=,where,sort,limit1–200,cursor,include=archived,deletedfor members),GET/PATCH(archived) /DELETE(tombstone)/runs/{run_id}. Runs follow the Eval's visibility; every route is404for an Eval the caller may not see. - Projection paths, bracket form:
run_id,status,error.kind,started_at,ended_at, facet keys (model.id, …),fingerprint.{facet},metrics[{ns}/{name}],results[{card}][{metric}].value/.label,ext.*. Per Card the page reportsruns_used,used_set_hash,posted_used_set_hashandchanged_since_card. GET /evals/{ns}/{name}[@…]carriesruns: { count, by_status, archived, deleted, runs_hash }(archived / deleted for members only). The comparison view (GET /evals/…/cards) comparessame_harness/same_modelagainst every run a Card used and addsruns_used,used_set_hash,changed_since_card.- Redaction: a public Card's
run_resultsentries that name an Eval the reader may not see are removed and counted inwithheld.run_results(GETandPOST /cards/query); a writer naming a run of an Eval they may not see getsrun_unknown, as for a missing run. - Limits (
[limits]):body_bytes16 MiB (413 body_too_large, every endpoint; was axum's 2 MiB default),batch_runs1,000 (413 batch_too_large),run_results100,000 per Card version (422 too_many_run_results). Refused, never truncated. - Schemas:
GET /schemas/eval-2(the header) andGET /schemas/runare new;GET /schemas/evalstays the 1.0 document until 0.3.0. - Error codes (new):
runs_moved,run_status_detail,run_id_invalid,run_id_mismatch,run_result_value_or_label,run_results_eval_unknown,run_unknown,run_not_in_used_set,too_many_run_results,batch_duplicate_run_id(422);run_deleted(409);batch_too_large,body_too_large(413). - Audit:
run.create/run.update/run.archive/run.unarchive/run.delete,migration.runs_split,migration.card_runs_unmatched. info.versionof/openapi.jsonstays1.0.0.
Crates
evalhub-schema, evalhub-core and evalhub-query are published at 0.2.0 (the store and the server ship as ghcr.io/ynishi/evalhub:0.2.0). just semver-check against 0.1.4 on crates.io, before the bump:
| Crate | Verdict |
|---|---|
evalhub-schema |
requires a new major (0.2 on 0.x): 5 lints — Eval.runs removed, eval::Run and eval::Outcome removed, Card.run_results added to an exhaustively constructible struct, 13 ErrorCode variants added |
evalhub-core |
no semver update required (196 checks: 196 pass, 58 skip); bumped with the workspace |
evalhub-query |
requires a new major (0.2 on 0.x): 1 lint — Column::Run and SortKey::Run added to exhaustive enums |
At 0.2.0 the same command reports "no semver update required" for all three (0 checks, 254 skipped: a major bump skips the lints).
Changes by stage
- #36 Schema: Run leaves the Eval body; per-run facets, status and metrics; Card.run_results; evalhub.eval/2.0 (#27)
- #37 Core: validate a Run, refuse runs in an Eval body, check run_results, fingerprint a run, publish the run hash formulas (#28)
- #38 Query: a run projection target — path table, run columns, sort keys and cursor for GET /evals/{ns}/{name}/runs (#29)
- #39 Store: run tables, run put/batch/archive/delete, runs_hash, audit actions, GC and download references for run attachments (#30)
- #40 Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs (#31)
- #41 Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has (#32)
- #42 Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI (#33)
- #43 Web UI and deploy scripts: the Eval page reads /runs and shows status, error kind, metrics and withheld counts; the Fly smoke test PUTs runs (#34)
- This PR: README (run paths,
[limits], schema ids, Upgrading from 0.1.x), the crate docs' final pass, the 0.2.0 bump (#35)
Verified
| Command | Outcome |
|---|---|
cargo fmt --check |
clean |
cargo clippy --workspace --all-targets -- -D warnings |
clean |
cargo test -p evalhub-schema |
20 passed |
cargo test -p evalhub-core |
62 passed |
cargo test -p evalhub-query |
27 passed |
cargo test -p evalhub-store (Docker) |
96 passed |
cargo test -p evalhub-server (Docker) |
82 passed, no .snap.new |
just sqlx-check (throwaway postgres:16-alpine, migrations 0001–0003) |
exit 0 |
cargo deny check |
advisories ok, bans ok, licenses ok, sources ok |
RUSTDOCFLAGS=-D warnings cargo doc --no-deps (five crates) |
clean |
just semver-check (before the bump) |
schema: major, 5 lints; core: none; query: major, 1 lint |
just semver-check (at 0.2.0, on the commit) |
all three: "no semver update required" (0 checks, 254 skipped: a major bump skips the lints) |
just package (on the commit) |
evalhub-schema 66422, `... |
evalhub 0.1.4
Relations: a reader or writer learns nothing of a version it may not see (0.1.4) (#26)
For #25. Releases 0.1.4.
evalhub_schema::eval promises that when a public Card points at a private Eval, a reader without access learns that the Eval exists, its version_id and its content_hash, and nothing else. The hub kept that promise only in GET …/relations. Three other paths gave the Eval away:
- Reads returned the Card body verbatim.
GET /cards/…andPOST /cards/queryreturnedrelations[].to, which is the privatens/name@seqas text. - Query conditions matched every edge. A
relations.tocondition (exact or prefix) was evaluated against targets the caller could not see, so a caller could confirm a private name by whether a public Card was hit. - Writes resolved targets regardless of visibility. Posting a Card or
POST …/relationsthat named someone else's private version resolved it, so the response and therefs_resolvedbadge told the writer that the version existed.
Export was not a path: hf-model-index is rendered from results[] and carries no relation.
What changes
-
Reads withhold what the reader may not see. Every read that returns a body removes the
relations[]elements whose target the reader may not see, and lists them in the envelope as the commitment:"withheld": { "relations": [ { "type": "core/uses_eval", "version_id": "…", "content_hash": "…" } ] }
A body with elements withheld is not the stored body, so its
content_hashdoes not match it. Verify the hash only on responses that have nowithheld. A member of the Eval's namespace gets the body unchanged and nowithheld. -
Query conditions see only visible targets. A
relations.tocondition matches only edges whose target the caller may see. Negated, the hidden edge is just as absent. -
A write learns no more than a read. At ingest, in
POST …/relationsand in the deferred resolver, a target the writer may not see is handled exactly like one that does not exist: it is stored unresolved, the response is the same and norefs_resolvedbadge is given. Once the target is visible to a reader, the edge resolves for them. The writer's own namespace and their organisations' namespaces are visible, so legitimate private references resolve as before. -
One visibility predicate (
evalhub_store::relations::visible_to) serves the relation DTO, body redaction and resolution.
Contract
withheldis new and optional on the version envelope and on query hits. The OpenAPI snapshot and theweb/client are regenerated.- The SDK crates' public API is unchanged:
just semver-checkreports no semver update required forevalhub-schema,-coreand-query. The schema change is limited to the crate doc (eval§ "Privacy across the relation"). .sqlxis regenerated for the changed queries.
Verification
crates/evalhub-server/tests/privacy.rs: five HTTP tests covering the body onGET, the body on query, therelations.tocondition, the badge onPOST, andPOST …/relations. All five were first run againstmainand failed on the defect itself:alice/secret@1appeared in the body, bob's condition hit, the badge differed from that of a nonexistent target, and the relation response revealed the target.- Store tests:
relations_resolve_only_to_versions_the_writer_may_see,resolution_never_reveals_a_private_versionandrelation_filters_see_only_targets_the_caller_may_seewere checked against this commit with the three visibility checks disabled, and each failed on the leak. cargo fmt --check;cargo clippy --workspace --all-targets -- -D warnings;cargo test -p evalhub-schema / -core / -query / -store / -server(Docker);just sqlx-check;pnpm check;just e2e(18 passed);just package+package-gate;just semver-check.
Test image: MinIO from Chainguard
The first CI run of this PR failed in evalhub-store tests/objects.rs before any test code ran: quay.io now answers 401 to anonymous pulls of quay.io/minio/minio (and minio/mc), which passed on main the day before. The store and server test harnesses and compose.yml now use cgr.dev/chainguard/minio, pinned by digest (it publishes latest only). It ships mc and a shell, so the bucket setup is unchanged. The tests mount a tmpfs on /data and wait for the API: http banner, because on the overlay root MinIO first logs a rename error that also starts with API:.
SDK crates: evalhub-schema evalhub-core evalhub-query
Image: ghcr.io/ynishi/evalhub:0.1.4
evalhub 0.1.3
README: the UI ships in the binary, and /whoami returns no roles (0.1.3) (#23)
Release 0.1.3. What changed since 0.1.2:
Organisations in the web UI (#17)
Settings creates an organisation (the creator is its first admin); the organisation's namespace page shows its roster and adds, updates and removes members. Browser tests drive the flow against the release binary.
The roster rule, documented as the token rule it is (#20)
GET /orgs/{org}/members asks that the presented token names the organisation, like every other read. The module doc, the handler, /openapi.json and the README said membership was enough; they now say what the code does. docs/hosting.md gains a section on making an organisation in two steps: create it, then issue a token that names it.
The organisation page after creating one (#21)
A signed-in token that does not name the organisation now gets a line saying why the roster is missing and a link to Settings with the token form filled in; Settings fills the same form right after creating an organisation. Member controls appear only for an admin token of an admin member, which is what the hub accepts.
This pull request
README.md: §Status no longer says the UI is being built, and the /whoami row names the fields the response has (user, scope, namespaces — no organisation roles). Carries the version bump to 0.1.3.
Verification
cargo fmt --check; cargo clippy --workspace --all-targets -- -D warnings; cargo test -p evalhub-schema, -core, -query; just package (all three crates at 0.1.3, gate ok); just semver-check: no semver update required for any of the three. No Rust source changed in this pull request; evalhub-store / evalhub-server tests were not run here.
The UI changes of #21 were deployed to the hosted service from main before this release and passed deploy/fly/smoke.sh.
Refs #22
SDK crates: evalhub-schema evalhub-core evalhub-query
Image: ghcr.io/ynishi/evalhub:0.1.3
evalhub 0.1.2
Release from the tag: crates, image and GitHub Release by one workflow (0.1.2) (#15)
The tag push becomes the only release action a person takes; the release workflow does the rest.
What changes
.github/workflows/release.ymlon av*tag:check: the tag names the version inCargo.tomland the tagged commit is onmain; otherwise nothing runs.crates:just package,cargo semver-checksagainst the versions on crates.io, thencargo publishfor whichever ofevalhub-schema/evalhub-core/evalhub-queryis not yet on crates.io at that version. Authentication is crates.io Trusted Publishing (OIDC,rust-lang/crates-io-auth-action); no token is stored. A rerun publishes nothing twice.image: as before (ghcr.io/ynishi/evalhub:<version>,:latest).release: a GitHub Release for the tag, notes from the merged pull request's body plus crate and image links.
deploy/release/land.sh <pr> <issue>: merge with a merge commit (never squash), close the issue, fast-forwardmain, thentag.sh.deploy/release/tag.sh: onmain, clean, HEAD isorigin/main, CI for HEAD green (waits), version fromCargo.toml, tag once, push, follow the workflow withgh run watch.just semver-check; CONTRIBUTING §Releases; pointers in AGENTS.md anddocs/hosting.md.
Why not release-plz (dsl-kit's flow)
Its version and changelog are computed from Conventional Commit headers, which this repository does not use (prose subjects, bump riding with the last change), and its release PR carries a bot commit without Signed-off-by. What it is for (no hand-typed publish, one tag as the trigger, a semver check, a release page) is taken; the rest is left.
0.1.2
This PR is the release that exercises the workflow it adds: the version bump rides in its one commit, and landing it (deploy/release/land.sh 15 14) tags v0.1.2 from a commit that contains release.yml. No Rust code changed since 0.1.1.
Verified
bash -non both scripts;release.ymlparses; the PR lookup by merge sha returns #13 for the 0.1.1 merge commit;gh run watch --exit-statusexists.cargo fmt --check,cargo clippy --workspace --all-targets -D warnings,cargo test -p evalhub-schema / -core / -query,just packageat 0.1.2,just semver-checkagainst 0.1.1.- Trusted Publishing is enabled on crates.io for the three crates (repository
ynishi/evalhub, workflowrelease.yml). - Not run before merge: the workflow itself; that is what the tag does.
Refs #14
SDK crates: evalhub-schema evalhub-core evalhub-query
Image: ghcr.io/ynishi/evalhub:0.1.2