8000 Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs by ynishi · Pull Request #40 · ynishi/evalhub · GitHub
[go: up one dir, main page]
Skip to content

Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs - #40

Merged
ynishi merged 1 commit into
mainfrom
feat/store-cards
Sep 25, 2026
Merged

Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs#40
ynishi merged 1 commit into
mainfrom
feat/store-cards

Conversation

@ynishi
@ynishi ynishi commented Sep 25, 2026
Copy link
Copy Markdown
Owner

Stage 5 of 9 of #24 (Runs as first-class rows), on top of stages 1–4 (#36, #37, #38, #39). evalhub-store only: everything that joins a Card to runs. No server code changes; the branch is mergeable on its own. The version bump is stage 9 (#35).

What changed (all under crates/evalhub-store/)

  • Card ingest (records.rs, new step 5b fix_card_runs, Cards only): after relation resolution, FOR SHARE on each referenced Eval's record row in id order (after the Card's own FOR UPDATE; run writes lock only the Eval FOR UPDATE, so a concurrent run write waits for the Card and the hashes recorded are the committed ones; no lock cycle). The used set per Eval record: the union of attrs.runs over the Card's core/uses_eval relations resolved to that record, else every run neither archived nor tombstoned at that moment. Eval visibility is re-read under the lock, so an Eval that turned private between resolution and the lock is treated as unresolved (no used set, edge stored as text, refs_resolved not awarded). External / unresolved targets get no used set and their attrs.runs are not checked. A run of an invisible Eval, a missing Eval and a missing run answer byte-identical run_unknown entries (tested on serialised bytes). run_results[] outside the used set is run_not_in_used_set; archived and tombstoned runs are rows and inside it. The store reads body["run_results"] itself; NewVersion gains no field. resolve_version now returns ResolvedVersion { version_id, record_id, record_type }. Step 8 inserts run_results (one UNNEST) and card_eval_runs (each used run with its current content_hash).
  • relations::add (a later uses_eval): same fixing, grow-only (ON CONFLICT DO NOTHING): a version's run_results were checked against the set it had and a version never changes, so the set keeps its posting-time hashes and gains runs; documented and tested.
  • runs::project(pool, record_id, &RunQuery, include, caller_ns) -> Option<RunPage> with the SQL builder in run_sql.rs (query_sql::build stays bound to records JOIN versions): runs joined with run_metrics, run_fingerprints and, per Card in RunQuery::cards, the run_results of that Card's latest live version; per Card used_set_hash (core's runs_hash over current hashes), posted_used_set_hash, changed_since_card; keyset paging on (sort key, run_id) with RunCursor, NULL-aware (NULLS LAST asc / NULLS FIRST desc, tie-break follows the last key's direction); archived and tombstoned rows only when asked via include and only for a namespace member, plus every run in a requested Card's used set whatever its state, marked state (archived looks like deleted to a non-member; masked rows expose no status / timestamps / body / metrics / fingerprints, judgements follow the Card); judgement filters use EXISTS, judgement sorts take the best non-null value as query_sql does for results; timestamp literals must be RFC 3339 (else QueryUnsupported), prefix / contains refused on timestamps; every dynamic value bound or escaped; all reads in one REPEATABLE READ, READ ONLY transaction. A Card the caller cannot see, that does not exist, has no live version or has no uses_eval into this Eval is answered identically as RunCardsUnknown([names]). An invisible / missing / header-less Eval is None, as runs::get.
  • relations::hidden_evals(card_version_ids, caller_ns) → per Card version the Evals its run_results name that the caller cannot see (eval_record_id, ns, name, count); no run ids in the return. Second input for the server's redaction beside hidden_targets.
  • Comparison view (relations::cards_using_eval): same_harness / same_model compare the Card's fingerprint with every run in its used set, false if any differs or is missing; an empty used set (no runs, or an explicit attrs.runs: []) compares with the header as before 0.2.0, which is what keeps evalhub-server/tests/relations.rs green as Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs #31 requires. Rows gain runs_used, used_set_hash, changed_since_card (a list of run ids).
  • Errors: StoreError::CardRunsRejected(Vec<ErrorEntry>), StoreError::RunCardsUnknown(Vec<String>).
  • used_set.rs (new, private): lock_evals, fix, check_run_results, insert, read, summarise. query_sql.rs: nine helpers pub(crate) for reuse, no behaviour change (diffed against main). push_key_value takes a what argument so a bad filter literal no longer reports "cursor timestamp".
  • Docs: records.rs ingest steps, "A Card's runs", the lock order; relations.rs; runs.rs "The projection"; run_sql.rs module doc mirroring query_sql's; crate doc (Runs, Visibility, Modules).
  • .sqlx: 17 added, 1 removed (the old resolve_version query).
  • Tests: tests/cards_runs.rs (10): used set with / without attrs.runs and as a union over two uses_eval to different seqs; run_unknown byte-identical for a missing run and an invisible Eval's run; run_not_in_used_set; archived / tombstoned still count; card_eval_runs holds the posting-time hash, an overwrite lands in changed_since_card and moves used_set_hash; projection filter / sort / page on status, error.kind, a facet key, metrics[x], results[c][m].value / .label, started_at, with NULL cursors and tie-breaks; include=archived member-only; a private Card named by an outsider, a missing Card and a Card not using this Eval are unknown; hidden_evals; same_model false when one used run differs; a later uses_eval edge only grows the set. tests/relations.rs: the add case with attrs.runs ["r9"] seeds a run row and checks card_eval_runs; a second case asserts run_unknown at /attrs/runs/1 with no edge written. run_sql unit tests (5) incl. a non-RFC 3339 literal.

Verified

Command Outcome
cargo fmt --check clean
cargo clippy --workspace --all-targets -- -D warnings clean
cargo test -p evalhub-store (Docker) 91 passed (lib 25, audit 1, auth 3, cards_runs 10, objects 3, query_sql 11, records 17, registry 4, relations 7, runs 10)
cargo test -p evalhub-server (Docker) 60 passed, no server change
cargo sqlx prepare --check (throwaway postgres:16-alpine, migrations 0001–0003) exit 0; no orphan .sqlx files
cargo doc --no-deps -p evalhub-store 0 warnings

Not run: cargo deny check (no dependency change), just e2e, schema / core / query tests (untouched crates).

Recorded here, decided later

Refs #31

@ynishi
Store: the Card's used set and run_results at ingest, the run project…
b44e7a1 
…ion query, hidden Evals for redaction, comparison over used runs

Stage 5 of #24. Everything in evalhub-store that joins a Card to
runs.

At Card ingest, after relation resolution, the store takes FOR SHARE
on each referenced Eval's record row (in id order, after the Card's
own FOR UPDATE; run writes lock only the Eval, so there is no cycle)
and fixes the used set per Eval record: the union of attrs.runs over
the Card's uses_eval relations resolved to that record, or, when none
carries attrs.runs, every run neither archived nor tombstoned at
that moment. The visibility of each Eval is re-read under the lock,
so an Eval that turned private between resolution and the lock is
treated as unresolved. Existence is checked only among Evals the
writer can see: a run of an invisible Eval, a missing Eval and a
missing run answer with byte-identical run_unknown entries.
run_results[] must lie inside the used set (run_not_in_used_set);
archived and tombstoned runs are rows and therefore inside it. The
store reads body["run_results"] itself, so NewVersion gains no field.
Step 8 inserts run_results and card_eval_runs, each used run with its
content hash at posting time. relations::add does the same fixing
for a later uses_eval edge, grow-only: a version's run_results were
checked against the set it had, and a version never changes.

runs::project answers GET /evals/{ns}/{name}/runs (stage 7): runs
joined with metrics, fingerprints and, per Card named in the
request, the run_results of that Card's latest live version; per
Card the used_set_hash over current hashes and changed_since_card
from the posting-time hashes; keyset paging on (sort key, run_id),
NULL-aware; archived and tombstoned rows only when asked and only
for a member, a used run that is archived or tombstoned marked as
such, archived looking like tombstoned to a non-member; every read
in one REPEATABLE READ read-only transaction. run_sql.rs builds the
SQL, query_sql stays bound to records JOIN versions. Timestamp
literals must be RFC 3339. Cards the caller cannot see, that do not
exist or that do not use this Eval are answered identically as
unknown.

relations::hidden_evals gives the server's redaction its second
input: per Card version, the Evals its run_results name that the
caller cannot see, with the count to withhold. The comparison view
compares the Card's fingerprint with every run in its used set,
false if any differs or is missing; an empty used set compares with
the header as before 0.2.0, which is what keeps the server's
comparison tests green. Rows gain runs_used, used_set_hash and
changed_since_card.

.sqlx regenerated (17 added, 1 removed). Until stage 7 maps them,
CardRunsRejected and RunCardsUnknown reach the server's catch-all;
no server path posts run_results or attrs.runs today.

Verified: cargo fmt --check; cargo clippy --workspace --all-targets
-D warnings; cargo test -p evalhub-store (91 passed, Docker);
cargo test -p evalhub-server (60, Docker); cargo sqlx prepare --check
against a migrated throwaway Postgres; cargo doc --no-deps
-p evalhub-store (0 warnings).

Refs #31

Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
@ynishi
ynishi merged commit d1b7f97 into main Sep 25, 2026
4 checks passed
This was referenced Sep 25, 2026
Closed
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ynishi
0