8000 Close the evalhub.eval/1.0 window: a 1.0 body is 422 runs_moved, GET /schemas/eval is gone (0.4.0) by ynishi · Pull Request #54 · ynishi/evalhub · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/*"]

[workspace.package]
version = "0.3.0"
version = "0.4.0"
edition = "2024"
# sqlx 0.9 declares rust-version 1.94.0; the workspace cannot claim less.
rust-version = "1.94"
Expand All @@ -21,10 +21,10 @@ categories = ["web-programming::http-server", "science"]

[workspace.dependencies]
# internal
evalhub-schema = { path = "crates/evalhub-schema", version = "0.3.0" }
evalhub-core = { path = "crates/evalhub-core", version = "0.3.0" }
evalhub-query = { path = "crates/evalhub-query", version = "0.3.0" }
evalhub-store = { path = "crates/evalhub-store", version = "0.3.0" }
evalhub-schema = { path = "crates/evalhub-schema", version = "0.4.0" }
evalhub-core = { path = "crates/evalhub-core", version = "0.4.0" }
evalhub-query = { path = "crates/evalhub-query", version = "0.4.0" }
evalhub-store = { path = "crates/evalhub-store", version = "0.4.0" }

# serde
serde = { version = "1", features = ["derive"] }
Expand Down
36 changes: 19 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,11 @@ start until `evalhub migrate` has brought the schema current.
## API

Everything is under `/api/v1`; the contract is `GET /openapi.json` (OpenAPI
3.1) and the record schemas are `GET /schemas/{card|eval-2|run|eval|error|query}`.
3.1) and the record schemas are `GET /schemas/{card|eval-2|run|error|query}`.
`eval-2` is the Eval header (`evalhub.eval/2.0`) and `run` one run;
`card` covers `evalhub.card/1.1` and the `evalhub.card/1.0` it still
accepts; `eval` stays the 0.1.x document (`evalhub.eval/1.0`, with
`runs[]`) that 0.2.0 still accepts, until 0.3.0 removes it.
accepts. `GET /schemas/eval`, the 0.1.x document (`evalhub.eval/1.0`, with
`runs[]`), was served by 0.2.0 and 0.3.0 and was removed in 0.4.0.
Writes need `Authorization: Bearer <token>` with `write` on the namespace;
reads of public records need nothing, and private records are `404` to
anyone the token does not cover. A token acts only in the namespaces it
Expand All @@ -104,19 +104,17 @@ covers it.

| Method | Path | Does |
| -------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `POST` | `/{cards\|evals}/{ns}/{name}?label=` | Validate, canonicalise and append a version. `201` with `{ id, version_id, seq, label, content_hash, created_at, changed[], badges[] }`; `200` and the existing version when the canonical body equals the latest; `422 { errors[] }` with every violation; `409 attachment_missing` or `label_in_use`. An Eval header carrying `runs` is `422 runs_moved` (runs are written below); a Card with more `run_results` than `limits.run_results` is `422 too_many_run_results`. |
| `POST` | `/{cards\|evals}/{ns}/{name}?label=` | Validate, canonicalise and append a version. `201` with `{ id, version_id, seq, label, content_hash, created_at, changed[], badges[] }`; `200` and the existing version when the canonical body equals the latest; `422 { errors[] }` with every violation; `409 attachment_missing` or `label_in_use`. An Eval header carrying `runs`, and any body declaring `evalhub.eval/1.0`, is `422 runs_moved` (runs are written below); a Card with more `run_results` than `limits.run_results` is `422 too_many_run_results`. |
| `GET` | `/{cards\|evals}/{ns}/{name}[@{seq}\|@{label}]` | The latest live version, or one addressed by sequence number or label, with the canonical `record`. `?expand=fingerprints,badges,changed` adds the hub's derived facts. A tombstoned version comes back with `tombstone` and no `record`. An Eval also carries `runs: { count, by_status, archived, deleted, runs_hash }` (the same at every `@seq`; `archived` / `deleted` for members of the namespace only). Card judgements of an Eval the reader may not see are removed and counted in `withheld.run_results`. |
| `GET` | `/{cards\|evals}/{ns}/{name}/versions` | Every version, oldest first, tombstones included, without bodies. |
| `PATCH` | `/{cards\|evals}/{ns}/{name}@{seq}/label` | Point a label at that version. Labels are unique within the name and never purely numeric. |
| `PATCH` | `/{cards\|evals}/{ns}/{name}/settings` | `{ "visibility": "public" \| "private" }`. |
| `DELETE` | `/{cards\|evals}/{ns}/{name}@{seq}` | Tombstone with `{ "reason": "withdrawn\|duplicate\|takedown\|other", "note": … }`. The body goes; the version id, the content hash and `changed[]` stay. |
| `GET` | `/{cards\|evals}?ns&search&sort&cursor&limit` | Page over names with a live version. `sort` is `created_desc` (default), `created_asc` or `name_asc`; paging is by opaque cursor. |

**The `evalhub.eval/1.0` body is deprecated.** 0.2.0 still accepts a
0.1.x Eval body (header and `runs[]` in one) on `POST /evals/{ns}/{name}`:
it is stored as a 2.0 header plus run rows, and the response carries a
`Deprecation` header, `converted_from: "evalhub.eval/1.0"` and
`converted_runs`. **0.3.0 removes this**; post the header, then the runs
**The `evalhub.eval/1.0` body is refused.** A 0.1.x Eval body (header
and `runs[]` in one) on `POST /evals/{ns}/{name}` is `422` with the single
error `runs_moved`, with or without `runs`; post the header, then the runs
(see Upgrading from 0.1.x, below).

### Runs
Expand Down Expand Up @@ -268,16 +266,20 @@ verdict on a run is the Card's. What a 0.1.x client changes:
| `"schema": "evalhub.eval/1.0"` | `"schema": "evalhub.eval/2.0"` (`GET /schemas/eval-2`) |
| `"schema": "evalhub.card/1.0"` | `"schema": "evalhub.card/1.1"`, which adds `run_results`; `evalhub.card/1.0` is still accepted |

A 1.0 Eval body with `runs[]` is accepted by 0.2.0 only: it is converted
into a 2.0 header and run rows, and the response carries a `Deprecation`
header. 0.3.0 refuses it with `422 runs_moved`, as 0.2.0 already refuses a
2.0 header that carries `runs`. Two things a 1.0 client will notice in
0.2.0 already:
A 1.0 Eval body was accepted for a compatibility window, by 0.2.0 and
0.3.0: it was converted into a 2.0 header and run rows, and the response
carried a `Deprecation` header, `converted_from: "evalhub.eval/1.0"` and
`converted_runs`. Two things a 1.0 client noticed in that window:

- the `content_hash` it gets back is the converted 2.0 header's, not the
- the `content_hash` it got back was the converted 2.0 header's, not the
hash of the body it posted;
- re-posting a body that leaves a run out no longer drops that run. Runs
are rows now; archive (`PATCH …/runs/{run_id}`) or delete it.
- re-posting a body that left a run out no longer dropped that run. Runs
are rows; archive (`PATCH …/runs/{run_id}`) or delete it.

The window closed in 0.4.0: a body declaring `evalhub.eval/1.0` is refused
with `422 runs_moved`, with or without `runs`, as a 2.0 header that carries
`runs` is, and `GET /schemas/eval` is gone (`404`). The mapping above
stays as the record of what changed.

On a hosted database, `evalhub migrate` for 0.2.0 does the same
conversion to every stored Eval once (`docs/hosting.md`, "Upgrading to
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
[
{"path": "/runs/0/artifacts/1", "code": "attachment_ref_unknown"},
{"path": "/runs/1/calls", "code": "attachment_ref_unknown"}
{"path": "/runs", "code": "runs_moved"}
]
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
[]
[
{"path": "/runs", "code": "runs_moved"}
]

This file was deleted.

This file was deleted.

36 changes: 14 additions & 22 deletions crates/evalhub-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,11 @@
//!
//! There are two record kinds, a Card and an Eval, each an append-only
//! sequence of immutable versions under a name (the one exception is the
//! 0.2.0 data migration of stored 1.0 bodies, which runs through
//! [`eval::split_v1`] here and is `evalhub_store`'s to perform and
//! audit). An Eval's versioned body is
//! a *header*; its runs are rows of the Eval record, outside every version.
//! Writing a run does not append a header version, and a header version
//! does not rewrite a run.
//! 0.2.0 data migration of stored 1.0 bodies, which is `evalhub_store`'s
//! to perform and audit). An Eval's versioned body is a *header*; its runs
//! are rows of the Eval record, outside every version. Writing a run does
//! not append a header version, and a header version does not rewrite a
//! run.
//!
//! ```text
//! Eval record {ns}/{name}
Expand Down Expand Up @@ -61,7 +60,8 @@
//! facts around it and never rewrites it.
//!
//! One exception to "all errors": an `evalhub.eval/2.0` header that carries
//! `runs` gets exactly one error, `runs_moved`, naming the run endpoints
//! `runs`, and any body that declares `evalhub.eval/1.0`, gets exactly one
//! error, `runs_moved`, naming the run endpoints
//! (`PUT /evals/{ns}/{name}/runs/{run_id}`,
//! `POST /evals/{ns}/{name}/runs:batch`).
//!
Expand All @@ -78,16 +78,14 @@
//! ──▶ (store) the row, then run::runs_hash over the record's runs
//! ```
//!
//! # The 1.0 arm
//! # `evalhub.eval/1.0`
//!
//! Release 0.2.0 still accepts an `evalhub.eval/1.0` body, with `runs[]` in
//! it, for one release. [`eval::declared_schema`] tells the caller which
//! arm a body declares; [`validate()`] checks a 1.0 body as 0.1.x did,
//! including that its runs' `calls` / `artifacts[]` name the body's
//! attachments; and [`eval::split_v1`] converts it into the 2.0 header and
//! runs a 2.0 producer would have written. The ingest of a 1.0 body and the
//! data migration of stored 1.0 versions both use that one function.
//! Release 0.3.0 removes the arm.
//! Releases 0.2.0 and 0.3.0 accepted an `evalhub.eval/1.0` body, with
//! `runs[]` in it, and converted it at ingest. Release 0.4.0 closed that
//! window: [`validate()`] answers a body declaring `evalhub.eval/1.0` with
//! the single `runs_moved` error, and this crate no longer knows the 1.0
//! shape or its conversion. The conversion of stored 0.1.x bodies belongs
//! to `evalhub_store`'s data migration, which keeps its own copy.
//!
//! # Hashes
//!
Expand Down Expand Up @@ -123,9 +121,6 @@
//! - **Validation is the same everywhere.** The server, a future CLI
//! `evalhub check`, and a harness embedding this crate all reject the same
//! records and runs for the same codes.
//! - **One conversion from 1.0.** A 1.0 run converted at ingest and one
//! converted by the migration are the same bytes and the same hash,
//! because both go through [`eval::split_v1`].
//!
//! # What this crate refuses to do
//!
Expand Down Expand Up @@ -155,7 +150,6 @@
//! records and runs.
//! - [`mod@validate`] — structural and semantic checks, error collection,
//! for records and runs.
//! - [`eval`] — the two Eval schema arms and the 1.0 → 2.0 conversion.
//! - [`run`] — facet materialisation, the run content hash and `runs_hash`.
//! - [`badge`] — the badge rules and their inputs.
//! - [`registry`] — the `core/` registry entries (metrics, relation types)
Expand All @@ -164,7 +158,6 @@

pub mod badge;
pub mod canonical;
pub mod eval;
pub mod fingerprint;
pub mod id;
pub mod registry;
Expand All @@ -173,7 +166,6 @@ pub mod validate;

pub use badge::{Badge, BadgeInput, compute as compute_badges};
pub use canonical::{ContentHash, canonicalize, content_hash};
pub use eval::{EvalSchema, SplitV1, declared_schema, split_v1};
pub use fingerprint::{Facet, Fingerprints, fingerprints};
pub use id::{RecordId, VersionId};
pub use run::{materialise, run_content_hash, runs_hash};
Expand Down
5 changes: 3 additions & 2 deletions crates/evalhub-core/src/run.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,9 @@
//! is the run's claim and wins.
//!
//! Materialisation is a pure function here, rather than a step inside the
//! store, so that the run write path and the 1.0 → 2.0 conversion
//! ([`crate::eval::split_v1`]) share one implementation.
//! store, so that the run write path and the 1.0 → 2.0 conversion of the
//! store's data migration (`evalhub_store::data_migrations::v1`) share one
//! implementation.
//!
//! # Hashes
//!
Expand Down
Loading
Loading
0