Repository navigation
Query: a run projection target — path table, run columns, sort keys and cursor for GET /evals/{ns}/{name}/runs - #38
Merged
Merged
Conversation
…sort keys and cursor Stage 3 of #24. GET /evals/{ns}/{name}/runs (stage 7) needs a query target that is not a record: rows are runs of one Eval, joined with the run_results of the Cards the caller names. The grammar (where / sort / limit / cursor) is shared; the projection is a second target with a path table of its own, built by hand rather than from a RecordKind schema. PathTable::for_runs(cards) resolves run_id, status, error.kind, started_at, ended_at, the run's facet leaves (the same key set as the Eval header's facets, taken from the eval-2 document, so the two cannot drift), fingerprint.{facet}, metrics[{ns}/{name}] (a number, any well-formed id, the registry is not consulted), and results[{card}][{metric}].value / .label for the Cards in the request; ext.* goes through with_ext as before. Parameters use the bracket form the results[{metric}].value sort key already uses, because slugs carry `.` and `/`; the dotted forms are refused with a hint that names the bracket form. A CardRef is {ns}/{name}, joined at the Card's latest live version. ir gains Column::Run(RunColumn), SortKey::Run, RunQuery and RunCursor { keys, run_id }: Cursor hard-codes version_id as the tie-breaker and a record cursor must not deserialise as a run cursor. compile_runs mirrors compile. Which Cards a caller may name and include=archived,deleted are server concerns, not part of the DSL. Timestamps are strings in the DSL; the backend compares them as timestamps (written where stage 5 reads it). Column and SortKey are exhaustive public enums, so evalhub-store's query_sql.rs gains three arms that refuse a run column with QueryUnsupported; no store behaviour is added. The grammar doc no longer lists `runs` among the array paths, which typecheck.rs had contradicted since the path table was built. Verified: cargo fmt --check; cargo clippy --workspace --all-targets -D warnings; cargo test -p evalhub-query (27 passed: compile 15, runs 12; record snapshots unchanged, one new run snapshot); cargo test -p evalhub-store (67, Docker); cargo test -p evalhub-server (60, Docker); cargo doc --no-deps -p evalhub-query (0 warnings). Refs #29 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
This was referenced Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 3 of 9 of #24 (Runs as first-class rows), on top of stages 1–2 (#27 / #36, #28 / #37).
evalhub-querygains a second target, the run projection behindGET /evals/{ns}/{name}/runs(stage 7). The grammar does not change. Mergeable on its own; the version bump is stage 9 (#35).What changed
PathTable::for_runs(cards: &[CardRef])(typecheck.rs), built by hand:run_id,status,error.kind,started_at,ended_at, the run facet leaves (the same key set as the Eval header's facets, read from the eval-2 document so the two cannot drift: 58 leaves overmodel/task/harness/generation/trial/env),fingerprint.{facet}for the six,metrics[{ns}/{name}](number; any well-formed id, registry not consulted),results[{card}][{metric}].value(number) /.label(string) for the Cards named in the request,ext.*through the existingwith_ext. Bracket form for parameters because slugs carry.and/;metrics.x,results.c.m.value, baremetrics/results, a malformed id, a Card not in the request, an@seqcard ref and a field other thanvalue/labelareunknown_pathwith a hint naming the bracket form.anyon a run path istype_mismatch(no array paths). Every run path is fully indexed (the projection is scoped to one Eval); an unregisteredextkey keepseq/existsonly.ir.rs:Column::Run(RunColumn)withRunColumn { RunId, Status, ErrorKind, StartedAt, EndedAt, Facet(path), Fingerprint(facet), Metric(id), Result { card, metric, field } },ResultField { Value, Label },CardRef({ns}/{name}, validated withis_valid_id, joined at the Card's latest live version),SortKey::Run(RunColumn),RunQuery { cards, filter, sort, limit, cursor }(default orderrun_idascending),RunCursor { keys, run_id }sinceCursorhard-codesversion_idas the tie-breaker; a record cursor does not deserialise as a run cursor.Column::Ext/Column::Bodyin aRunQueryare paths below the run's body.compile_runs(request, table, limit, cursor) -> RunQuery, mirroringcompile;expand/versionare record concepts and are not read.include=archived,deletedand which Cards a caller may name are server concerns.grammar.rsdoc no longer listsrunsamong the array pathsanyapplies to (it contradictedtypecheck.rs); record queries do not gainruns. The record table is still built fromRecordKind::Eval.schema()(2.0); nothing reads 1.0. Record snapshots are unchanged.started_at/ended_atare strings in the DSL; the backend compares them as timestamps. Written onRunColumn::StartedAtand in thetypecheck.rstable for stage 5 to read.evalhub-store/src/query_sql.rs:ColumnandSortKeyare exhaustive public enums, so three match arms now refuse a run column withStoreError::QueryUnsupported. No store behaviour added.lib.rs): the second target and its path table.ir.rs/typecheck.rsmodule docs cover the run table.tests/runs.rs, 12): typecheck of each path kind; an exact assertion of the 69 fixed paths; rejection ofresults[{card}]for an unnamed Card, a malformed metric id, the dotted forms;CardRefparsing;RunCursorround-trip and key-count mismatch; hints free of double spaces; and an insta snapshot (runs__runs.snap) of the IR for a query filtering onstatus,metrics[a/b]andresults[c/d][a/b].valueand sorting onmetrics[a/b].Verified
cargo fmt --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p evalhub-query.snap.newcargo test -p evalhub-store(Docker)cargo test -p evalhub-server(Docker)cargo doc --no-deps -p evalhub-queryNot run:
cargo deny check(no dependency change),cargo sqlx prepare --check(no SQL change),just e2e,just semver-check(adding variants toColumn/SortKeyis a break inevalhub-query's public API; the 0.2.0 bump of stage 9 covers it, as #24 states).Notes for the reviewer
GET /evals/{ns}/{name}/runs#29 does not ask for them and the record path has no equivalent: a closed-set check onstatusliterals, and refusingexpand/versionincompile_runs. If either is wanted it is a one-line decision on Query: a run projection target — path table, run columns, sort keys and cursor forGET /evals/{ns}/{name}/runs#29.unknown_pathhints for the bracket forms had literal runs of spaces from a broken line continuation in the first draft; fixed, andthe_hints_are_clean_proseasserts no hint carries two consecutive spaces. That test was written after the fix, so its failing state was not observed; by inspection the original strings (14–18 spaces) would have trippedcontains(" ").CursorSigner(server) MACs the payload without a domain tag, for record cursors today and run cursors tomorrow; Query: a run projection target — path table, run columns, sort keys and cursor forGET /evals/{ns}/{name}/runs#29 leaves signing to the server, so stage 7 decides whether to bind the cursor to its target.Refs #29