Repository navigation
Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs #31
Description
Part of #24, stage 5 of 9. Depends on stages 3 and 4; stage 7 depends on it.
Scope
evalhub-store: everything that joins Cards to runs. Today the server extracts results and passes them as NewVersion.results; for run_results the store reads body["run_results"] itself (a new pattern, chosen so NewVersion gains no field and the server's struct literal does not change in this stage). The run_results count limit is the server's (stage 7).
Changes
- Card ingest (
records.rs,create_or_append): after relation resolution (step 5), takeFOR SHAREon each referenced Eval's record row, then fix the used set per Eval record: the union ofattrs.runsover the Card'score/uses_evalrelations resolved to that record; if none carriesattrs.runs, every run of the record that is neither archived nor tombstoned at that moment. A relation that resolved to an external or not-yet-resolvable target has no used set and gets nocard_eval_runsrows. Existence is checked only among Evals the writer can see (relations::visible_to); a run of an Eval the writer cannot see, and arun_results.evalthat resolved to nothing, answer exactly as a missing run,run_unknown.run_results[]entries must lie inside the used set (run_not_in_used_set); archived and tombstoned runs are rows and therefore inside it.resolve_versionreturns only a version id today; it must also yield the Eval's record id. At step 8, besideresults, insertrun_resultsandcard_eval_runs(each used run with its currentcontent_hash). relations::add(POST …@{seq}/relationsaddinguses_evallater): the same used-set fixing andcard_eval_runsinsert; that path carries norun_results.- Projection:
runs::project(record_id, RunQuery, cards, include, caller_ns) -> page:runsjoined withrun_metrics,run_fingerprintsand, per chosen Card, therun_resultsof that Card's latest live version; per Card,used_set_hash(stage 2 formula overcard_eval_runs) andchanged_since_cardfromcard_eval_runsrows whose stored hash differs from the currentruns.content_hash; keyset paging on(sort key, run_id); archived and tombstoned rows only when asked and only for a member of the namespace; a used run that is archived or tombstoned appears marked as such (for a non-member, archived looks like tombstoned).cardsare resolved withvisible_tofor the caller and must have auses_evalinto this Eval; a Card the caller cannot see, that does not exist, or that does not use this Eval is answered identically as unknown. SQL builder in a newrun_sql.rs:query_sql::buildis bound torecords JOIN versions. relations::hidden_evals(card_version_ids, caller_ns): for each Card version, the Eval record ids named by itsrun_resultsthat the caller cannot see.hidden_targetscovers only resolvedrelations[]; this is the second input the server's redaction needs.- Comparison view (
relations::cards_using_eval):same_harness/same_modelcompare the Card's fingerprint with every run in its used set (false if any used run's fingerprint differs or is missing); each row gainsruns_used,used_set_hashandchanged_since_card. - Existing tests touched:
tests/relations.rs(therelations::addcase withattrs.runs ["r9"]now seeds a run row, and a second case assertsrun_unknownwithout one; the rawINSERTedges withattrs.runsstay as they are). Thetests/records.rsNewVersionwithattrs.runs ["r1"]targets an external Eval, so it has no used set and needs nothing. The server fixturecard-complete.jsoncarries noattrs.runssince stage 1, socargo test -p evalhub-serverneeds no change here. .sqlxregenerated in the same commit.
Acceptance
cargo test -p evalhub-store(Docker): used set with and withoutattrs.runs, and as a union over twouses_evalto different seqs of one Eval;run_unknownfor a missing run and, byte-identical, for a run of an Eval the writer cannot see;run_not_in_used_set; an archived and a tombstoned run still count as rows;card_eval_runsholds the hash at posting time and overwriting a run afterwards puts it inchanged_since_cardand changesused_set_hash; projection filter, sort and page onstatus,error.kind, a facet key,metrics[x]andresults[c][m].value;include=archivedhonoured only for a member; a private Card named incardsby an outsider is unknown;hidden_evalsfor a public Card citing a private Eval;same_modelfalse when one used run's model differs.just sqlx-checkgreen;cargo test -p evalhub-serverstill green.
Activity
ynishi commentedon Sep 25, 2026
Shipped in #40.