8000 Release 0.2.0: runs as first-class rows by ynishi · Pull Request #44 · ynishi/evalhub · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/*"]

[workspace.package]
version = "0.1.4"
version = "0.2.0"
edition = "2024"
# sqlx 0.9 declares rust-version 1.94.0; the workspace cannot claim less.
rust-version = "1.94"
Expand All @@ -21,10 +21,10 @@ categories = ["web-programming::http-server", "science"]

[workspace.dependencies]
# internal
evalhub-schema = { path = "crates/evalhub-schema", version = "0.1.4" }
evalhub-core = { path = "crates/evalhub-core", version = "0.1.4" }
evalhub-query = { path = "crates/evalhub-query", version = "0.1.4" }
evalhub-store = { path = "crates/evalhub-store", version = "0.1.4" }
evalhub-schema = { path = "crates/evalhub-schema", version = "0.2.0" }
evalhub-core = { path = "crates/evalhub-core", version = "0.2.0" }
evalhub-query = { path = "crates/evalhub-query", version = "0.2.0" }
evalhub-store = { path = "crates/evalhub-store", version = "0.2.0" }

# serde
serde = { version = "1", features = ["derive"] }
Expand Down
99 changes: 91 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ does not rewrite what it receives, and never calls anything "verified".
## Status

Pre-alpha. The API is complete: records with validation, fingerprints and
badges, attachments, relations, the query language, the registry and the
badges, an Eval's runs and their projection with Cards' judgements,
attachments, relations, the query language, the registry and the
audit log. The web UI is compiled into the binary; a debug build without it
serves the API and a placeholder page. Read the design in the crate docs:

Expand Down Expand Up @@ -60,24 +61,37 @@ Configuration is layered: defaults, then a TOML file (`--config` or
`EVALHUB_DATABASE__URL`), then flags. `config show --origin` prints every
key with the layer that set it; secrets are redacted.

What one request may carry is configured under `limits`; a request over a
limit is refused, never truncated:
What one request may carry is configured in the `[limits]` table of the
file (or `EVALHUB_LIMITS__BODY_BYTES` and so on); a request over a limit
is 8000 refused, never truncated:

| Key | Default | Over it |
| -------------------- | ------------------- | -------------------------------- |
| `limits.body_bytes` | `16777216` (16 MiB) | `413 body_too_large`, any endpoint |
| `limits.batch_runs` | `1000` | `413 batch_too_large` on `runs:batch` |
| `limits.run_results` | `100000` | `422 too_many_run_results` on a Card |

```toml
[limits]
body_bytes = 16777216
batch_runs = 1000
run_results = 100000
```

Read pages are not configurable: `limit` on the record, run and audit
listings is 1–200 (default 50), clamped rather than refused.

`serve` refuses to start without a database, and with one it refuses to
start until `evalhub migrate` has brought the schema current.

## API

Everything is under `/api/v1`; the contract is `GET /openapi.json` (OpenAPI
3.1) and the record schemas are `GET /schemas/{card|eval-2|run|eval|error|query}`
(`eval-2` is the Eval header, `run` one run, `eval` the 0.1.x Eval that
0.2.0 still accepts).
3.1) and the record schemas are `GET /schemas/{card|eval-2|run|eval|error|query}`.
`eval-2` is the Eval header (`evalhub.eval/2.0`) and `run` one run;
`card` covers `evalhub.card/1.1` and the `evalhub.card/1.0` it still
accepts; `eval` stays the 0.1.x document (`evalhub.eval/1.0`, with
`runs[]`) that 0.2.0 still accepts, until 0.3.0 removes it.
Writes need `Authorization: Bearer <token>` with `write` on the namespace;
reads of public records need nothing, and private records are `404` to
anyone the token does not cover. A token acts only in the namespaces it
Expand All @@ -102,7 +116,8 @@ covers it.
0.1.x Eval body (header and `runs[]` in one) on `POST /evals/{ns}/{name}`:
it is stored as a 2.0 header plus run rows, and the response carries a
`Deprecation` header, `converted_from: "evalhub.eval/1.0"` and
`converted_runs`. **0.3.0 removes this**; post the header, then the runs.
`converted_runs`. **0.3.0 removes this**; post the header, then the runs
(see Upgrading from 0.1.x, below).

### Runs

Expand All @@ -115,11 +130,46 @@ for an Eval the caller may not see. Writes need `write` on the namespace.
| -------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `PUT` | `/evals/{ns}/{name}/runs/{run_id}` | Write one run (`GET /schemas/run`; its `run_id` must equal the path's). `201` created, `200` with `result: updated` or `unchanged`; body `{ run_ 8000 id, content_hash, status, result, runs_hash }`. `409 run_deleted` for a deleted id, `409 attachment_missing`, `422` otherwise. |
| `POST` | `/evals/{ns}/{name}/runs:batch` | `{ runs: [Run, …] }`, all or nothing: on failure nothing is written and every failing element is listed (`/runs/{index}/…`). `200 { runs: [{ run_id, content_hash, result }], runs_hash }`. More than `limits.batch_runs` is `413 batch_too_large`. |
| `GET` | `/evals/{ns}/{name}/runs?cards&where&sort&limit&cursor&include` | Runs × metrics × each named Card's judgements. `cards={ns}/{name}` repeats; `where` is the query grammar as JSON text over run paths (`status`, `error.kind`, `model.id`, `metrics[{id}]`, `results[{card}][{metric}].value`, …); `sort={path}[:asc\|:desc]` repeats; `limit` 1–200; `include=archived,deleted` for members. Per Card: `runs_used`, `used_set_hash`, `changed_since_card`. A Card that is unknown, private to the caller or not using the Eval is `404`. |
| `GET` | `/evals/{ns}/{name}/runs?cards&where&sort&limit&cursor&include` | Runs × metrics × each named Card's judgements. `cards={ns}/{name}` repeats; `where` is the query grammar as JSON text over the run paths below; `sort={path}[:asc\|:desc]` repeats, `run_id` is always the last key; `limit` 1–200; `include=archived,deleted` for members. Per Card: `runs_used`, `used_set_hash`, `changed_since_card`. A Card that is unknown, private to the caller or not using the Eval is `404`. |
| `GET` | `/evals/{ns}/{name}/runs/{run_id}` | One run. Archived runs are `404` to non-members; a deleted run is `{ run_id, content_hash, tombstone }`. |
| `PATCH` | `/evals/{ns}/{name}/runs/{run_id}` | `{ "archived": true \| false }`. Hides a run from non-members and the counts; no hash changes. |
| `DELETE` | `/evals/{ns}/{name}/runs/{run_id}` | Tombstone with `{ "reason", "note" }`. The body and attachment references go; the id, content hash and metrics stay, and the id is never written again. |

The paths `where` and `sort` accept on `GET …/runs` are the run's, not a
record's. Parameters are in brackets, because a metric id or a Card
name contains `/` and may contain `.`; the dotted forms
(`metrics.core/tokens_out`) are `422 unknown_path` with a hint.

| Path | Type | Is |
| -------------------------------------- | ----------------- | ------------------------------------------------------ |
| `run_id`, `status`, `error.kind` | string | the run's id, `ok` / `error` / `skipped`, the error slug |
| `started_at`, `ended_at` | string (RFC 3339) | compared as timestamps; `prefix` / `contains` refused |
| `{facet}.{key…}` (`model.id`, `generation.temperature`, `env.git.commit`, …) | as in the schema | the run's own facets (six, no `grading`), header defaults copied in |
| `fingerprint.{facet}` | string | the run's fingerprint of that facet |
| `metrics[{ns}/{name}]` | number | a metric the run measured; registered or not |
| `results[{card}][{metric}].value` | number | that Card's judgement of the run; `{card}` must be in `cards=` |
| `results[{card}][{metric}].label` | string | the same, as a label |
| `ext.*` | as for records | the run's own `ext`; an unregistered key answers `eq` / `exists` only |

Every typed path takes every operator its type allows, sorting
included: the projection is scoped to one Eval, so facet keys range and
sort here although they do not in a record query. The one exception is
an unregistered `ext` key, which is `422 not_indexed` for anything but
`eq` / `exists`, sorting included. There are no array paths, so no
`any`.

```bash
curl -G https://hub.example/api/v1/evals/alice/loop/runs \
--data-urlencode 'cards=alice/judge' \
--data-urlencode 'where={"and":[{"path":"status","op":"eq","value":"ok"},{"path":"results[alice/judge][core/pass].value","op":"gte","value":1}]}' \
--data-urlencode 'sort=metrics[core/tokens_out]:desc' \
--data-urlencode 'limit=100'
```

A page is `{ cards, items, next_cursor }`; pass `next_cursor` back as
`cursor`. A cursor is bound to the Eval and the `sort` it was issued for
and is `400` anywhere else.

### Identity

| Method | Path | Does |
Expand Down Expand Up @@ -205,6 +255,39 @@ their `metric_registered` / `harness_registered` badge, on the next sweep.
stream them through the hub, which the presigned-URL design exists to
avoid, and the alternative has not been chosen yet.

## Upgrading from 0.1.x

0.2.0 moves an Eval's runs out of its body. The versioned body is now a
header; the runs are rows of the Eval, written by `run_id`, and the
verdict on a run is the Card's. What a 0.1.x client changes:

| 0.1.x | 0.2.0 |
| ------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `POST /evals/{ns}/{name}` with `runs[]` in the body | `POST` the header without `runs`, then `PUT …/runs/{run_id}` per run or `POST …/runs:batch` |
| `runs[].outcome` (`pass` / `fail` / `error` / `skipped`) | the run's `status` (`ok` / `error` with `error.kind` / `skipped` with `skip_reason`), and the pass / fail on the Card, in `run_results[]` |
| `"schema": "evalhub.eval/1.0"` | `"schema": "evalhub.eval/2.0"` (`GET /schemas/eval-2`) |
| `"schema": "evalhub.card/1.0"` | `"schema": "evalhub.card/1.1"`, which adds `run_results`; `evalhub.card/1.0` is still accepted |

A 1.0 Eval body with `runs[]` is accepted by 0.2.0 only: it is converted
into a 2.0 header and run rows, and the response carries a `Deprecation`
header. 0.3.0 refuses it with `422 runs_moved`, as 0.2.0 already refuses a
2.0 header that carries `runs`. Two things a 1.0 client will notice in
0.2.0 already:

- the `content_hash` it gets back is the converted 2.0 header's, not the
hash of the body it posted;
- re-posting a body that leaves a run out no longer drops that run. Runs
are rows now; archive (`PATCH …/runs/{run_id}`) or delete it.

On a hosted database, `evalhub migrate` for 0.2.0 does the same
conversion to every stored Eval once (`docs/hosting.md`, "Upgrading to
0.2.0"). Header `content_hash` values change, with the old and the new
hash of every version in the audit log (`migration.runs_split`).
Versions that differed only in `runs` become identical header versions.
A Card's relations keep resolving, because they point at version ids,
not hashes, and each Card's used runs are rebuilt from the Eval version
it points at.

## Web UI

The single-page application in `web/` is one more client of `/api/v1`: it
Expand Down
18 changes: 15 additions & 3 deletions crates/evalhub-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@
//! # The model: records, and the runs of an Eval
//!
//! There are two record kinds, a Card and an Eval, each an append-only
//! sequence of immutable versions under a name. An Eval's versioned body is
//! sequence of immutable versions under a name (the one exception is the
//! 0.2.0 data migration of stored 1.0 bodies, which runs through
//! [`eval::split_v1`] here and is `evalhub_store`'s to perform and
//! audit). An Eval's versioned body is
//! a *header*; its runs are rows of the Eval record, outside every version.
//! Writing a run does not append a header version, and a header version
//! does not rewrite a run.
Expand All @@ -33,7 +36,11 @@
//! and a run is fingerprinted from its own facets ([`fingerprint::for_run`]).
//! A run records what happened, never whether it passed; the verdict is a
//! Card's `run_results`, whose form this crate checks and whose references
//! to stored runs the store checks.
//! to stored runs the store checks. The runs a Card judged, per Eval, are
//! its *used set*: the `attrs.runs` of its `core/uses_eval` relations, or
//! every run neither archived nor deleted when it is posted; this crate
//! hashes a used set ([`run::runs_hash`]) but never computes one, because
//! that needs the stored runs.
//!
//! # What the hub derives from a record
//!
Expand Down Expand Up @@ -134,7 +141,12 @@
//! - It does not know which runs exist. Whether a `run_results[].run_id`
//! names a stored run, and whether it is in the Card's used set, is the
//! store's check (`run_unknown`, `run_not_in_used_set`); how many
//! `run_results` a Card may carry is the server's (`too_many_run_results`).
//! `run_results` a Card may carry is the server's (`too_many_run_results`),
//! as are the request body and batch limits.
//! - It does not know who is asking. Visibility (runs follow their Eval,
//! `run_results` follow their Card, and entries naming an Eval the
//! reader may not see are withheld) is decided by the store and the
//! server; every function here answers the same for every caller.
10000 //!
//! # Modules
//!
Expand Down
20 changes: 20 additions & 0 deletions crates/evalhub-query/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,26 @@
//! ──▶ ir::RunQuery { cards, filter, sort, limit, cursor: RunCursor }
//! ```
//!
//! A second target because a run is not a record. An Eval is a versioned
//! header plus runs that belong to the record, not to a version, so the
//! header's schema (what the record table is projected from) has no
//! `runs` key, and a record query cannot reach a run. A Card judges runs
//! in `run_results`, within its *used set* (the runs of the Eval it
//! used); the projection is where the two meet, one row per run with the
//! judgements of each named Card. What the page reports beside the rows
//! (per Card the used-set hash, the one recorded when the Card was
//! posted, and the runs changed since; each row's `content_hash`) is
//! computed by the store with `evalhub_core::run` and is not a path of
//! this table, and neither is the record's `runs_hash`, which the Eval's
//! own `GET` and every run write report. Nor is anything
//! about who is asking: runs follow their Eval's visibility, judgements
//! their Card's, and the limits (page size 1–200, the batch size, the
//! number of `run_results` per Card) are the server's; a query that
//! compiles here says nothing about what the caller may see. None of it
//! writes: header versions are write-once (the one exception, the 0.2.0
//! data migration, is `evalhub_store`'s) and runs are written through
//! the run endpoints, not through this crate.
//!
//! [`PathTable::for_runs`] is built by hand, not from a record schema: the
//! run's columns (`run_id`, `status`, `error.kind`, `started_at`,
//! `ended_at`), the keys of its six facets (the Eval header's key set,
Expand Down
17 changes: 16 additions & 1 deletion crates/evalhub-schema/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,9 @@
//! A `card::Card` says *what was measured, how, and what the score was*. An
//! `eval::Eval` is *the material*: a set of runs, prompts, tasks or traces
//! that a Card was measured from. Both are published under a name,
//! `{ns}/{name}`, as an append-only sequence of immutable versions. The
//! `{ns}/{name}`, as an append-only sequence of immutable versions (the one
//! exception, the 0.2.0 data migration that rewrote stored
//! `evalhub.eval/1.0` bodies into 2.0 headers, is `evalhub_store`'s). The
//! relation between them (`core/uses_eval`) is what makes comparison possible:
//! two Cards that point at the same Eval version measured the same thing.
//!
Expand All @@ -49,6 +51,19 @@
//! a header version. A run records what happened and what was measured
//! directly; the verdict on it is the Card's (`card::Card::run_results`).
//!
//! The runs a Card judged are its *used set*, per Eval it uses: the
//! `attrs.runs` of its `core/uses_eval` relations, or, without them, every
//! run neither archived nor deleted when the Card is posted ([`common`],
//! [`card`]). Four hashes tie the pieces together, all computed by the
//! hub and none sent by the client: the header's `content_hash`, each
//! run's `content_hash`, the record's `runs_hash` over every run, and a
//! used-set hash over the runs a Card used; the formulas are
//! `evalhub_core`'s. Runs have no visibility of their own and follow their
//! Eval; a Card's `run_results` follow the Card, except that a reader who
//! may not see an Eval is never shown the entries that name it. How many
//! runs a batch and how many `run_results` a Card may carry are the
//! server's limits, not constraints of these types.
//!
//! A Card has no `kind` field. Whether a score came from a judge, an
//! aggregation, or was transcribed from a paper is not a kind of Card; those
//! are properties of how it was graded (`facet::Grading`), how it was
Expand Down
5 changes: 3 additions & 2 deletions crates/evalhub-server/src/api/records.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,9 @@
//! `converted_runs`, the runs as a batch reports them. Posting the same
//! 1.0 body again is `200` with every run `unchanged`. A refused run
//! refuses the whole post, its entries prefixed `/runs/{index}` of the
//! posted `runs[]`. **0.3.0 removes this**: a 1.0 body will then be
//! refused like any unknown `schema`.
//! posted `runs[]`. **0.3.0 removes this**: a 1.0 body with `runs` will
//! then be refused with `422 runs_moved`, as a 2.0 header with `runs` is
//! now.
//!
//! # A Card's `run_results`
//!
Expand Down
Loading
Loading
0