Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has by ynishi · Pull Request #41 · ynishi/evalhub

ynishi · 2026-09-25T12:01:10Z

Stage 6 of 9 of #24 (Runs as first-class rows), on top of stages 1–5 (#36, #37, #38, #39, #40). The data migration from 0.1.x bodies to run rows, the serve gate on it, and the hosting runbook. Mergeable on its own; the version bump is stage 9 (#35).

What changed

pool::migrate (evalhub-store): after MIGRATOR.run, applies every data migration whose name is not in data_migrations, each in one transaction that first inserts its data_migrations row (ON CONFLICT DO NOTHING; a conflicting concurrent migrate waits, then skips). A failure midway rolls the whole step back and leaves it pending; the 0003 DDL, applied by sqlx in its own transaction, stays. pending_migrations now returns Vec<PendingMigration> (SQL or data entries, printable; was Vec<i64>) and reports a data step that has not run, so serve refuses a database with the DDL applied and the data step pending. migrate returns Vec<Applied>.
data_migrations.rs (new): 0003_runs_split, per Eval record in records.id order under FOR UPDATE, memory bounded to one Eval at a time:
1. runs[] from every live 1.0 version by run_id, highest seq wins; tombstoned versions contribute nothing; a repeated id within one body warns and the last wins.
2. Each run converted with evalhub_core::eval::spli 8000 t_v1 applied to its source version (outcome mapping, meta.v0.2.0_migration.outcome, that version's header facets materialised, the header attachments[] named by calls / artifacts copied into the run) and run_attachment_refs written. A run_id failing the stage-1 format, or an element with no string run_id (which split_v1 would drop silently), makes the migration scan to the end, write nothing, and return DataMigrationRefused listing every offender; the database is untouched and the step stays pending (tested on a full snapshot).
3. Run hashes and fingerprints through the run write path's own helpers (runs::prepare / upsert / recompute_runs_hash, now pub(crate); no logic change), records.runs_hash set.
4. Every live 1.0 version body replaced by the header split_v1 returns (schema 2.0, no runs), content_hash recomputed, one audit row migration.runs_split { version_id, old_content_hash, new_content_hash } per version (subject eval/{ns}/{name}@{seq}, actor columns NULL). Versions already on 2.0 are left alone. Two live versions that differed only in runs[] end up with the same hash; nothing constrains (record_id, content_hash), so that is fine. The store crate doc names this migration as the one exception to the write-once rule for versions.
5. card_eval_runs for every Card version with resolved uses_eval into the Eval, from the versions it points at (union of attrs.runs, else every run of the pointed-at live versions; a shared id takes the higher seq's hash). A used id with no row is skipped with one audit row migration.card_runs_unmatched { card_version_id, eval, run_ids } per (Card version, Eval), under the Card's namespace. An attrs.runs id that is a row but in none of the pointed-at versions records the current row's hash (documented and tested). Edges to tombstoned versions contribute nothing; non-string attrs.runs elements are dropped (documented; only 0.1.x data can carry them). Card bodies are not rewritten.
evalhub migrate prints one line per data migration applied with its counts, or no data migration pending. The serve gate message lists the pending entries, e.g. pending migrations [data 0003_runs_split].
docs/hosting.md: "Upgrading to 0.2.0": snapshot the volume and pg_dump; fly machine update <id> --autostart=false --skip-start then fly machine stop (the old binary, restarted by autostart after release_command = "migrate" ran, would store a 1.0 body with runs unsplit); fly deploy --ha=false; confirm the release with fly releases and evalhub --version over fly ssh console; /healthz and /openapi.json as liveness / contract checks only (the doc's older claim that info.version shows the release is corrected: it is the API contract version 1.0.0); deploy/fly/smoke.sh. The release command's 5 minute default timeout rolls the step back and leaves it pending.
.sqlx: 6 added.
Tests: crates/evalhub-store/tests/data_migrations.rs (3) seeds 0.1.x-shaped rows by SQL (two live versions with overlapping ids, different bodies and facets; a tombstoned version; a Card on the older version with attrs.runs including a dangling id and an id only the newer version has; a Card without attrs.runs; a Card on the tombstoned version) and after pool::migrate asserts every Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has #32 Acceptance item (no live body has runs, every live schema is 2.0, each run row's status / detail / meta / source-version facets and hashes, run_attachment_refs for calls, one audit row per version with old and new hash and new hash == hash of the stored body, card_eval_runs from the pointed-at version with its hashes, the dangling id absent with its audit row, the tombstoned Card empty, the differing run in changed_since_card, cards_using_eval still lists every Card, one data_migrations row, a second migrate changes nothing); pending_migrations names the data step when only the DDL is applied; a refused migration leaves the database byte-identical. crates/evalhub-server/tests/boot.rs: serve_refuses_a_database_with_the_data_step_pending runs the real evalhub binary against a DDL-only database and sees serve refuse, migrate apply, migrate apply nothing. tests/common/mod.rs gains db_unmigrated().

Verified

Command	Outcome
`cargo fmt --check`	clean
`cargo clippy --workspace --all-targets -- -D warnings`	clean
`cargo test -p evalhub-store` (Docker)	96 passed (unit 27; data_migrations 3, cards_runs 10, runs 10, records 17, …)
`cargo test -p evalhub-server` (Docker)	61 passed (boot 6)
`cargo sqlx prepare --check` (throwaway `postgres:16-alpine`, migrations 0001–0003)	exit 0; no orphan `.sqlx` files
`cargo doc --no-deps -p evalhub-store`	0 warnings

Not run: cargo deny check (no dependency change), just e2e, schema / core / query tests (untouched). The Fly steps in the runbook were checked against the local flyctl --help (flag names, --release-command-timeout default) only, not against a deploy: how fly deploy treats a stopped Machine and whether it resets autostart is written up as both outcomes.

Recorded here, decided later

Refusing an element with no run_id goes beyond Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has #32 (format failures only); on hosted data it cannot fire (the 0.1.2 schema required run_id and every body was validated at ingest). Left in, documented.
crates/evalhub-server/src/lib.rs:18, 61 still say a version is immutable; the store crate doc carries the exception. Stage 7 (server) or the owner updates the server doc.
StoreError::MigrationsPending(Vec<i64>) was already unused and is left as is.

Refs #32

…ord that it ran, refuse to serve until it has Stage 6 of #24. A 0.1.x database holds runs inside Eval version bodies; 0.2.0 reads them from the runs table. evalhub migrate now runs the SQL migrations and then every data migration whose name is not in data_migrations, each in one transaction that also inserts its row, so a step either lands whole or stays pending. sqlx runs each SQL file in its own transaction, so DDL and data cannot share one; the 0003 DDL staying applied while the data step is pending is exactly what pending_migrations now reports, and serve refuses such a database. 0003_runs_split, per Eval record in id order under FOR UPDATE: collect runs[] from every live 1.0 version, highest seq winning a run_id; convert each with evalhub_core::eval::split_v1 applied to the version it came from (outcome mapping, meta.v0.2.0_migration, that version's header facets materialised, the header attachments its calls / artifacts name copied in); a run_id that fails the stage-1 format, or an element with no run_id, stops the migration with a report and leaves the database untouched; write the run rows with the same helpers the run write path uses and set runs_hash; replace every live version body with the header split_v1 returns, recompute content_hash, and write migration.runs_split with the old and new hashes per version, the one exception to the write-once rule for versions, which the crate doc now names; build card_eval_runs for every Card version with resolved uses_eval into the Eval from the versions it points at, skipping a used id with no row with one migration.card_runs_unmatched row per (Card version, Eval). Card bodies are not rewritten. evalhub migrate prints each data migration it applied with its counts; a second run prints that nothing is pending. The store and server test databases go through pool::migrate, so the step runs there as a no-op. docs/hosting.md gains the 0.2.0 upgrade: snapshot the database, turn autostart off and stop the serving Machine (after the release command has run, the old binary would still store a 1.0 body with runs unsplit), fly deploy --ha=false, confirm the release with fly releases and evalhub --version (neither /healthz nor /openapi.json shows the crate version; the doc's older claim that they do is corrected), then smoke.sh. The release command's 5 minute default timeout rolls the step back and leaves it pending. Verified: cargo fmt --check; cargo clippy --workspace --all-targets -D warnings; cargo test -p evalhub-store (96 passed, Docker; the migration test seeds 0.1.x rows by SQL and checks every acceptance item of #32); cargo test -p evalhub-server (61, Docker; the boot test runs the real binary against a DDL-only database and sees serve refuse, migrate apply, migrate apply nothing); cargo sqlx prepare --check against a migrated throwaway Postgres; cargo doc --no-deps -p evalhub-store (0 warnings). The Fly steps were checked against flyctl --help only, not against a deploy. Refs #32 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>

ynishi merged commit 23bf692 into main Sep 25, 2026
4 checks passed

Navigation Menu

Repository navigation

Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has - #41

Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has#41
ynishi merged 1 commit into
mainfrom
feat/migration-runs

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

What changed

Verified

Recorded here, decided later

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant