Repository navigation
Core: validate a Run, refuse runs in an Eval body, check run_results, fingerprint a run, publish the run hash formulas - #37
Merged
Conversation
… fingerprint a run, publish the run hash formulas Stage 2 of #24. Stage 1 gave the hub the shapes (a Run type, a header-only Eval under evalhub.eval/2.0, Card.run_results); this stage gives evalhub-core the rules and the formulas, and nothing else: store and server behaviour is unchanged, and no signature they call moves. `validate::run(run_id, body)` is a new entry point, not a RecordKind variant (a run is not a record, and the kind is matched exhaustively in six places). It runs run.json and then the semantic rules: run_id_invalid / run_id_mismatch, run_status_detail (error needs `error`, skipped needs `skip_reason`, ok needs neither), the attachment reference checks for calls / artifacts[] / error.log against the run's own attachments[], and metric_id_invalid on every metrics key. Records have no ext namespace rule in this crate, so runs get none either; the module doc says so. Eval now has two accepted schema ids. `validate` keeps its signature and picks the document by the body's `schema` through RecordKind::schema_for, with one compiled validator per accepted id. `eval::declared_schema` tells the caller which arm applied. A 2.0 body carrying `runs` gets the single error runs_moved, naming the run endpoints, before the schema pass; a 1.0 body validates as before, including its runs[] attachment checks. `eval::split_v1` is the one 1.0 -> 2.0 conversion, pure, for the ingest of a 1.0 body (stage 4) and the data migration (stage 6): outcome pass / fail -> ok, error -> error with kind `unrecorded`, skipped -> skipped with skip_reason `unrecorded`, the dropped outcome kept under meta.v0.2.0_migration, the header's facets materialised onto each run, the header attachments a run names copied into it, duplicate run_ids last-wins and reported. Returning a struct rather than the tuple in the issue keeps the duplicate list named. Card run_results[] get metric_id_invalid, run_result_value_or_label and run_results_eval_unknown (compared on {ns}/{name}, ignoring @seq); the rules that need the store or a count are listed in the table with their owner. `fingerprint::for_run`, `run::materialise` (pure, idempotent, only an absent facet is copied) and the hashes are public with their formulas in the crate doc: run_content_hash = SHA-256 of the RFC 8785 form of the materialised run with its run_id; runs_hash = SHA-256 of the RFC 8785 form of [[run_id, hex], ...] sorted by run_id bytes. Archive and delete change none of them. The hash tests carry literal hex a client can reproduce. Known and left for the owner: the stage-1 fixture crates/evalhub-schema/fixtures/card-run-results.json has counts that fail counts_inconsistent (attempted 2 < completed 2 + failed 1); core keeps a corrected copy, the schema-crate file is out of this stage's scope. And the schema doc calls ext namespacing mandatory while no crate enforces it, for records or runs. Verified: cargo fmt --check; cargo clippy --workspace --all-targets -D warnings; cargo test -p evalhub-schema (20 passed), -p evalhub-core (62), -p evalhub-query (15), -p evalhub-store (63, Docker), -p evalhub-server (60, Docker); cargo doc --no-deps -p evalhub-core (0 warnings); the five literal hash values recomputed independently from the canonical strings and matching. Refs #28 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
This was referenced Sep 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 2 of 9 of #24 (Runs as first-class rows), on top of stage 1 (#27, PR #36).
evalhub-coreonly: the rules and the formulas. Store and server behaviour is unchanged and their tests are untouched; the branch is mergeable on its own. The version bump is stage 9 (#35).What changed (all under
crates/evalhub-core/)validate::run(run_id, &Value), a new entry point rather than aRecordKindvariant:run.json, thenrun_id_invalid/run_id_mismatch(at/run_id),run_status_detail(errorneedserror,skippedneedsskip_reason,okneeds neither; reported at/erroror/skip_reason,nullcounts as absent),attachment_ref_unknown/attachment_path_invalidforcalls,artifacts[]anderror.logagainst the run's ownattachments[],metric_id_invalidon everymetricskey. Records have noextnamespace rule in this crate, so runs get none; the module doc says so (see "For the owner" below).validatekeeps its signature and picks the JSON Schema by the body'sschemathroughRecordKind::schema_for; the compiled-validator cache is one entry per accepted id.eval::declared_schema(&Value) -> Option<EvalSchema { V1, V2 }>is public. A 2.0 body with anyrunskey gets the single errorruns_moved, whose message namesPUT /evals/{ns}/{name}/runs/{run_id}andPOST /evals/{ns}/{name}/runs:batch, before the schema pass. A 1.0 body validates as before,runs[]attachment checks included (1.0 arm only). An unknownschemavalue keeps today's/schemaerror.eval::split_v1(&Value) -> SplitV1 { header, runs, duplicate_run_ids }, pure, the one 1.0 → 2.0 conversion for the ingest of a 1.0 body (stage 4) and the data migration (stage 6):runsremoved andschemaset to 2.0 on the header; per runpass/fail→ok,error→errorwithkind: unrecorded,skipped→skippedwithskip_reason: unrecorded, the droppedoutcomekept verbatim undermeta.v0.2.0_migration.outcome, the header's facets materialised onto the run, the header attachments the run names copied into itsattachments[](copied, not moved, so the header's hash equals that of the same header posted as 2.0), duplicaterun_idlast wins and is reported. A struct instead of the issue's tuple, so the duplicate list has a name.run_results[]:metric_id_invalid,run_resul 8000 t_value_or_label,run_results_eval_unknown(evalcompared on{ns}/{name}against the Card'score/uses_evaltargets with@seqstripped;external:/hf:targets never match).run_unknown/run_not_in_used_set(store, stage 5) andtoo_many_run_results(server, stage 7) are in the rule table with their owner.fingerprint::for_runandrun::materialise(run, header)(pure, idempotent; only an absent facet is copied, whole facet; a run's explicitnullis its own claim).run_content_hash(run_id, &Value)= SHA-256 of the RFC 8785 form of{ …run, "run_id": run_id }(the caller passes the materialised body);runs_hash(&[(run_id, hash)])= SHA-256 of the RFC 8785 form of[[run_id, hex], …]sorted byrun_idbytes; the used-set hash is the same function. Archive and delete change none of them. Tests carry literal hex a client can reproduce (recomputed independently in review).validate.rsmodule doc rewritten for the model: runs as rows, the two Eval arms, the run rule table, the formula table.fixtures/validate/, 48 new files):valid-run,run-status-detail-{valid,ok,error,error-null,skipped-absent,skipped-null,skipped-error},run-refs-{unknown,path-invalid},run-id-{empty,slash,max,too-long,mismatch}(byte lengths 200 / 201),run-metric-id,eval-runs-moved,eval-run-set-v1,eval-schema-unknown,card-run-results-{valid,metric,value-or-label,eval-unknown}.eval-run-refs.jsonis back to its 1.0 body with run pointers (stage 1 had left it testing nothing).tests/run.rs(19 tests):materialise,split_v1for every outcome,meta, facet and attachment copy, duplicate ids, header-hash equality, the literal hashes,for_run.Verified
cargo fmt --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p evalhub-schemacargo test -p evalhub-corecargo test -p evalhub-querycargo test -p evalhub-store(Docker)cargo test -p evalhub-server(Docker)cargo doc --no-deps -p evalhub-coreNot run:
cargo deny check(no dependency change),cargo sqlx prepare --check(no SQL change),just e2e.Observable effect through the server (a consequence of the issue text, no server code changed)
validate, so until stage 4 adds the conversion at ingest the server stores it withrunsin the body;runsnow gets422 runs_movedinstead ofschemaerrors.For the owner (out of this stage's scope, not changed here)
crates/evalhub-schema/fixtures/card-run-results.json(stage 1) hascountsattempted: 2, completed: 2, failed: 1, which failscounts_inconsistent. Core keeps a corrected copy (completed: 1). A later stage that posts the schema-crate fixture will get a 422 until it is fixed.evalhub-schema'scommon.rscallsextnamespacing mandatory, and Core: validate a Run, refuserunsin an Eval body, check run_results, fingerprint a run, publish the run hash formulas #28 says "theextnamespace rule as for records", but no crate enforces it for records or runs. Either a rule for both (new code,is_valid_idon each key) or a softer doc; a separate change.Refs #28