Repository navigation
Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI - #42
Merged
Conversation
…ion, the Eval runs summary, comparison counts, OpenAPI Stage 7 of #24. The run rows of stages 4 and 5 get their API. Under /api/v1/evals/{ns}/{name}: PUT /runs/{run_id} (201 created, 200 updated or unchanged, 409 run_deleted / attachment_missing, 422 validation with run_id_mismatch when the body disagrees with the path), POST /runs:batch (all or nothing, 413 batch_too_large, a refusal lists every failing element at /runs/{i}/..., 409 only when every entry is a conflict, else 422), GET /runs (the projection: repeatable cards=, where, sort, limit 1-200, a signed cursor bound to the Eval and the sort so a cursor from another Eval, another sort or a record listing is 400, include=archived,deleted for members; rows carry metrics and per-Card judgements, the envelope carries each Card's used_set_hash and changed_since_card), GET /runs/{run_id} (a tombstoned run answers { run_id, content_hash, tombstone }), PATCH { archived } and DELETE { reason, note? }. Every run route of an Eval the caller cannot see, or with no live header, is 404 with the same empty body as a missing Eval, checked before anything that depends on the body; a Card named in cards= that the caller cannot see, that does not exist or that does not use this Eval is the same 404. POST /evals: a 2.0 body carrying runs gets the single runs_moved error ahead of schema errors. A 1.0 body is accepted for this release: the store converts it, the response carries a Deprecation header and the envelope says converted_from with the per-run results as runs:batch returns them; README and the crate doc say 0.3.0 removes this. GET /evals/{ns}/{name}[@seq] gains runs { count, by_status, archived, deleted, runs_hash }, the last two for members. POST /cards refuses more than the configured run_results with 422 too_many_run_results before validation. [limits] body_bytes (16 MiB, up from axum's 2 MiB default), batch_runs (1,000) and run_results (100,000) in config; the body limit sits on the router and a non-JSON 413 is rewritten into the body_too_large envelope. Redaction: withhold also drops run_results entries whose Eval is in hidden_evals, on GET and on POST /cards/query, and Withheld counts them. may_download counts an archived run's reference only for a member. The comparison DTO carries runs_used, used_set_hash and changed_since_card. run.* and migration.* actions read back from GET /audit. The server crate doc names the migration as the exception to a version's immutability. The stage-1 fixture card-run-results.json had counts that core rejects as counts_inconsistent; this stage posts it, so the test was written first, seen fail on exactly that code, and completed set to 1. OpenAPI snapshot and the web client regenerated in this commit; 413 is documented on operations that take a request body. Verified: cargo fmt --check; cargo clippy --workspace --all-targets -D warnings; cargo test -p evalhub-server (82 passed, Docker; runs 18, boot 7); cargo test -p evalhub-schema (20); cargo test -p evalhub-store (96, Docker); cargo doc --no-deps -p evalhub-server (0 warnings); corepack pnpm check (0 errors); the regenerated client diffed against a fresh generator run, identical. Refs #33 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
8DCA
This was referenced Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 7 of 9 of #24 (Runs as first-class rows), on top of stages 1–6 (#36–#41).
evalhub-server: the run endpoints and everything the API exposes of the run rows. Mergeable on its own; the UI is stage 8 (#34), the version bump stage 9 (#35).What changed
/api/v1/evals/{ns}/{name}(api/runs.rs, new), write scopewriteon the namespace:PUT /runs/{run_id}→ 201created, 200updated, 200unchanged; body{ run_id, content_hash, status, runs_hash, result }(resultadded so a 200 says which it was); 409run_deleted/attachment_missing; 422 validation incl.run_id_mismatch(the body must carryrun_id, asrun.jsonrequires).POST /runs:batch{ runs: [ { run_id, …Run } ] }→ all or nothing; 413batch_too_large; a refusal lists every failing element at/runs/{i}/…, 409 when every entry is a conflict, else 422; success 200{ runs: [ { run_id, content_hash, result } ], runs_hash }.GET /runs→ the projection: repeatablecards=({ns}/{name}),where(filter JSON), repeatablesort({path}[:asc|:desc]),limit1–200 default 50 (clamped, as the listings),cursor,include=archived,deleted(honoured for members, ignored otherwise); rows carrymetricsand per-Card judgements; the envelope carries per Cardused_set_hashandchanged_since_card(a list of run ids). The cursor is signed with the existingCursorSignerover{ k: "runs", e: record_id, s: sort, c: RunCursor }, so a cursor from another Eval, another sort, or a record listing / query is 400 (tested in all directions).GET /runs/{run_id}→ the run; archived → 404 for non-members; tombstoned →{ run_id, content_hash, tombstone }.PATCH /runs/{run_id}{ archived };DELETE /runs/{run_id}{ reason, note? }.where, cursor or body validation (visible but nowrite→ 403). A Card incards=the caller cannot see, that does not exist, or that does not use this Eval, and a malformed card ref, are the same empty 404.POST /evals/{ns}/{name}: a 2.0 body withruns→ 422 with the singleruns_movederror ahead of schema errors. A 1.0 body is accepted this release:Deprecation: @1790294400(RFC 9745; 2026-09-25T00:00:00Z, the day 2.0 replaced 1.0) on the 201 and the 200, envelopeconverted_from: evalhub.eval/1.0andconverted_runs { runs, runs_hash, duplicate_run_ids? }shaped like the batch response; the stored 2.0 header'scontent_hash. README and the crate doc say 0.3.0 removes this.GET /evals/{ns}/{name}[@seq|@label]envelope gainsruns: { count, by_status, archived, deleted, runs_hash },archived/deletedfor members only, on every version incl. tombstoned ones (runs belong to the record).POST /cards/{ns}/{name}:run_resultscounted on the parsed body, 422too_many_run_resultsat/run_resultsabove the limit, before validation and the store.config.rs[limits]):body_bytes16 MiB (replaces axum's 2 MiB default; no route streams a body, uploads are presigned),batch_runs1,000,run_results100,000; inentries()and the documented keys.DefaultBodyLimiton the router; a non-JSON 413 is rewritten into thebody_too_largeenvelope, handlers' own JSON 413 passes through.api/relations.rs,api/records.rs,api/query.rs):Withheld { relations, run_results: u64 }, both always present;withhold(body, hidden_targets, hidden_evals)removesrun_results[]entries whose Eval the caller cannot see, onGETand onPOST /cards/query; the trigger is any hidden relation or hidden Eval. Export unchanged (no body projection ofrun_results).may_download: a reference from an archived run counts only for a member; tombstoned runs have none.runs_used,used_set_hash,changed_since_card. Audit:run.*andmigration.*read back fromGET /audit. Schemas:eval-2,run,evalserved.error.rs):ApiError::Conflict(409 envelope),ApiError::TooLarge(413 envelope),runs_rejected(409 iff every entry isrun_deleted/attachment_missing),CardRunsRejected→ 422,RunNotFound→ 404,RunDeleted→ 409 at/run_id,RunCardsUnknown→ empty 404, storeQueryUnsupportedfrom the projection → 422type_mismatchat/where,Canonical→ 500. A 1.0 body whoseruns[].run_idfails the run format is now 422 (was the stage-4 interim 500). 413 is documented on every operation that takes a request body.lib.rs): API table, run lifecycle, limits, comparability; the migration named as the exception to a version's immutability.crates/evalhub-schema/fixtures/card-run-results.jsonhadcountsattempted 2 < completed 2 + failed 1. The server test that posts it was written first and failed withcounts_inconsistentat/counts(422 vs 201), thencompletedwas set to 1.cargo test -p evalhub-schemaunchanged (20).web/src/lib/api/schema.d.tsregenerated (diffed against a fresh generator run in review: identical). No.sveltechange.tests/runs.rs(18) covers every Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI #33 Acceptance bullet: PUT →GET /runswith metrics → Card undercards=→ overwrite inchanged_since_card; batch all-or-nothing listing every failure; 409-only-for-conflicts and mixed 422 batches;attachment_missingon PUT; 2.0runs_movedahead ofunknown_key; 1.0 conversion withDeprecation,converted_from, runs onGET /runs, re-postunchanged; public Card citing a private Eval redacted onGETandPOST /cards/querywithwithheld.run_results; every run route (GET / PUT / PATCH / DELETE / batch / list) 404 for an outsider incl. afterPATCH …/settings; private Card incards=→ 404; 413 envelope for body and batch;too_many_run_results; archived runs members-only incl. attachment download; tombstoned read shape; runs summary; cursor paging with cross-Eval / cross-endpoint replay → 400; audit entries incl. a realmigration.runs_splitrow.tests/boot.rs+1 (schemas served).tests/common/mod.rs:EVAL_V1,RUN,CARD_RUN_RESULTS,start_with_config,call_full,put_run,post_card_with_runs.Verified
cargo fmt --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p evalhub-server(Docker)cargo test -p evalhub-schemacargo test -p evalhub-store(Docker)cargo doc --no-deps -p evalhub-servercorepack pnpm check(web).snap.newNot run:
cargo deny check(no dependency change),just e2e(the web build is stage 8; the API surface is covered by the server tests), core / query tests (untouched).Decisions recorded here
resulton the PUT response and theconverted_runsname are additions (Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI #33 names neither). TheDeprecationdate is the implementer's choice; noSunset/Linkheader.GET /evals/{ns}/{name}[@…]only (the Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI #33 Changes text); the listing'slatestcarries noruns. If the Acceptance bullet "onGET /evals" meant the listing too, that is a small follow-up.cards/where/include, the same as record cursors.changed_since_cardis passed through as a list.GET …/runsdoes not document 400 in OpenAPI (nor doGET /cards/GET /audittoday).Refs #33