Repository navigation
Store: run tables, run put/batch/archive/delete, runs_hash, audit actions, GC and download references for run attachments - #39
Merged
Conversation
…ions, GC and download references for run attachments Stage 4 of #24. Runs become rows of the Eval record: 0003_runs.sql adds runs, run_metrics, run_attachment_refs, run_fingerprints, the two Card-side tables stage 5 fills (run_results, card_eval_runs), data_migrations for stage 6, and records.runs_hash. runs.rs writes them. Every write locks the record row as the ingest path does, then: materialise the facets the run omits from the latest live header, validate, check every referenced attachment is ready, hash, upsert the four tables, recompute runs_hash over every row (archived and tombstoned included), write the audit row. Outcomes are created / updated / unchanged; unchanged writes no row and no audit row. put_batch is one transaction under one lock, all or nothing, and reports every failing element with its index; a run_id repeated in the batch is batch_duplicate_run_id. A tombstoned id answers run_deleted. Overwriting an archived run leaves it archived. A tombstone drops the body and the attachment references and keeps the content hash, the metrics and the reason. An Eval whose header versions are all tombstoned makes run reads and writes not-found, and its runs no longer count as download references (the GC still counts them). The 0.2.0 compatibility window: an evalhub.eval/1.0 body is split by evalhub_core::eval::split_v1 after validation; the 2.0 header goes through the existing ingest steps and the runs through the same code as put_batch, in the one transaction under the one lock. So a 1.0 client re-posting the same body is a no-op on both, and one that appends a run gets a run row and no new header version. The stored body is the 2.0 header and the returned content_hash is that header's; ingest() reports converted_from and the per-run results for the server's envelope (stage 7), while create_or_append keeps its signature. Audit gains run.create / run.update / run.archive / run.unarchive / run.delete with subject eval/{ns}/{name}/runs/{run_id}. runs_summary(record_id, member) gives the Eval envelope its count, by_status, archived, deleted and runs_hash. objects::referencing() walks run_attachment_refs, Referencing.version_id becomes optional and the row carries run_archived; gc_candidates and the gc re-check count a run reference. .sqlx regenerated (20 added, 2 removed). Two things this stage records rather than changes. A run body must carry run_id (run.json requires it; #28's wording reads as optional), so PUT with a body that omits the path id fails the schema pass; stage 7 decides which side is authoritative. And a started_at / ended_at that is not RFC 3339 is accepted, kept in the body and stored NULL in the column, because core validates Draft 2020-12 without format assertions; documented and tested here, the fix is in core and touches records too. Verified: cargo fmt --check; cargo clippy --workspace --all-targets -D warnings; cargo test -p evalhub-store (76 passed, Docker; the referencing fix was written test-first and seen red); cargo test -p evalhub-server (60, Docker); cargo sqlx prepare --check against a migrated throwaway Postgres; cargo doc --no-deps -p evalhub-store (0 warnings). Refs #30 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
This was referenced Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 4 of 9 of #24 (Runs as first-class rows), on top of stages 1–3 (#36, #37, #38).
evalhub-storeonly: the DDL, the run rows, the 1.0 ingest conversion, audit,runs_summary, and the GC / download references. The Card side (used set,run_results, projection) is stage 5; the data migration is stage 6. No server code changes; the branch is mergeable on its own. The version bump is stage 9 (#35).What changed
migrations/0003_runs.sql: the DDL from Store: run tables, run put/batch/archive/delete, runs_hash, audit actions, GC and download references for run attachments #30 verbatim (diffed against the issue text with comments stripped):runs,run_metrics,run_attachment_refs,run_fingerprints,run_resultsandcard_eval_runs(populated in stage 5),data_migrations(stage 6),records.runs_hash.runs.rs(new):put,put_batch,get,archive/unarchive,tombstone(reason, note). Every write takeslock_record(the ingest path'sFOR UPDATE) and the latest live header, then: materialise omitted facets (evalhub_core::run::materialise) →evalhub_core::validate::run→ every referenced attachmentready→ content hash → upsert 8000 the four tables (stale metric / ref / fingerprint rows deleted first) → recomputerecords.runs_hashover every row, archived and tombstoned included → audit rows. Outcomescreated/updated/unchanged(unchanged: no rows, no audit row).put_batch: one transaction, one lock, all or nothing, every failing element with its index (StoreError::RunsRejected), duplicate id →batch_duplicate_run_idon each later occurrence.run_deletedfor a tombstoned id. Overwriting an archived run leaves it archived. Tombstone dropsbodyandrun_attachment_refs, keepscontent_hash,run_metrics,run_fingerprints,archived_atand the reason; a second tombstone isRunDeleted. No live header → reads and writes not-found.gethides an archived run from non-members; a deleted run is returned with its tombstone to anyone who may see the Eval unless it is also archived.records.rs, the 0.2.0 compatibility window):ingest()(new;create_or_appenddelegates and keeps its signature) splits anevalhub.eval/1.0body withsplit_v1after validation; the 2.0 header goes through the existing steps and the runs through the same code asput_batch, in the one transaction under the one lock. Re-posting the same 1.0 body is a no-op on both; appending a run adds one row and no header version. Stored body = 2.0 header; returnedcontent_hash= the header's;Ingestedcarriesconverted_from: evalhub.eval/1.0, the per-run results andduplicate_run_idsfor stage 7's envelope.run.create/run.update/run.archive/run.unarchive/run.delete, subjecteval/{ns}/{name}/runs/{run_id}; a batch writes one row per created or updated run, after theruns_hashrecompute, same transaction.runs_summary(record_id, member)→{ count, by_status, archived, deleted, runs_hash }:count/by_statusover rows neither archived nor tombstoned;archived/deletedfor members only;runs_hashfor every reader (NULL reported as the hash of[]).objects.rs:referencing()also walksrun_attachment_refs → runs → records(only for Evals with a live header);Referencing.version_idisOption<Uuid>, the row carriesrun: Option<(record_id, run_id)>andrun_archived: boolfor stage 7'smay_download.gc_candidatesand thegcre-check count arun_attachment_refsrow as a reference.tests/objects.rsadjusted.StoreError::{RunNotFound, RunDeleted, RunsRejected(Vec<RunRejection>), Canonical}. The server's catch-all maps them to 500 with no body until stage 7 maps them (see below).runs.rsmodule doc with the write ordering and a Timestamps section..sqlx: 20 added, 2 removed (the oldreferencing/gcqueries), regenerated against a migrated throwaway Postgres.tests/runs.rs, 10): created → unchanged (same hash, no audit row) → updated (runs_hashmoves; a dropped metric's row goes); archive keepsruns_hash, hides from non-members, same body again isUnchangedand still archived; tombstone keeps hash / metrics, drops refs, later put →run_deleted; batch with one invalid element writes nothing and lists every failure; duplicate id in a batch; a materialised facet survives a later header POST; a 1.0 body with two runs → 2.0 header + two rows, re-post no-op, re-post with one more run → one row and no header version; GC keeps an object referenced only by a run and collects it after tombstone;referencing()yields the run's Eval andrun_archived, and nothing once every header version is tombstoned (written test-first, seen red); a private Eval; an unparseable timestamp; the audit rows.Verified
cargo fmt --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p evalhub-store(Docker)cargo test -p evalhub-server(Docker)cargo sqlx prepare --check(throwawaypostgres:16-alpine, migrations 0001–0003).sqlxfilescargo doc --no-deps -p evalhub-storeNot run:
cargo deny check(no dependency change),just e2e, schema / core / query tests (untouched crates; the server run links them).Recorded here, decided later
run_idin the body.run.jsonrequiresrun_id(Run.run_id: String, Schema: Run leaves the Eval body; per-run facets, status and metrics; Card.run_results; evalhub.eval/2.0 #27), while Core: validate a Run, refuserunsin an Eval body, check run_results, fingerprint a run, publish the run hash formulas #28'srun_id_mismatchwording reads as if the body may omit it. The store validates the body as given, soPUT …/runs/r1with a body lackingrun_idfails the schema pass; every test here carries it. Stage 7 chooses: keep it required (body repeats the path id) or have the handler /runs::putinsert the path id before validation, aspreparealready does after it.started_at/ended_atdeclareformat: date-time, but core validates Draft 2020-12 without format assertions, so"started_at": "yesterday"is accepted, kept in the body and stored NULL in the column (sorts as NULL). Documented and tested; the fix is inevalhub-coreand applies to records too.runs[].run_idcontains/or exceeds 200 bytes passes 1.0 validation (eval.jsonhas norun_idconstraint), then failsvalidate::runafter the split; the server's catch-all answers 500 with an empty body and a log line. Such a body was accepted before this stage. Stage 7 mapsRunsRejectedto 422.may_downloaddoes not readrun_archivedyet, so on a public Eval an archived run's reference counts as public. No endpoint can archive a run before stage 7.Refs #30