Repository navigation
Comparison view: an empty used set agrees with nothing, same_harness / same_model false - #53
Merged
Merged
Conversation
…/ same_model false The comparison view judges `same_harness` / `same_model` against every run in a Card's used set (#24, #31), but an empty used set fell back to the Eval version's header fingerprint, the pre-0.2.0 rule. A Card posted against an Eval with no runs, with an explicit `attrs.runs: []`, or through an edge that did not resolve then, got `true` from the header although it was compared with no run, while a Card whose one used run differed got `false`: two Cards could be "comparable" for different reasons (#48). An empty used set now gives false for both flags; the row's `runs_used: 0` says why. `cards_using_eval` no longer reads the Eval version's fingerprints, and the fallback sentences go from the store's module doc, `ComparisonRow` and the function doc, and from the server's comparison doc and `same_harness` field doc. The OpenAPI snapshot and the web client's generated `schema.d.ts` carry the new field text; the web comparison table's tooltips said "equals this Eval's" and now name the runs the Card used. README already states the run-level rule. Tests: the store's comparison_view seeds a used run for c1 and expects the textual-edge c2 (no used set) to read false; cards_runs covers an Eval without runs and an explicit `runs: []` on an Eval with runs. The server's comparison_view_lines_cards_up PUTs one run before the Cards post, so `first` agrees with a run and `second` differs from it, and a new case posts a Card on an Eval with no runs and expects both flags false with `runs_used: 0`. Closes #48 Signed-off-by: Yutaka Nishimura <ytk.nishimura@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Comparison view: an empty used set agrees with nothing
Closes #48 (Option A).
GET /evals/{ns}/{name}/cardsreportssame_harness/same_modelper Card as "the Card's fingerprint equals that of every run in its used set" (#24, #31). When the used set was empty, the view fell back to the Eval version's header fingerprint, the pre-0.2.0 rule, so a Card posted against an Eval with no runs, or with an explicitattrs.runs: [], could readsame_model: truealthough it had been compared with no run. The fallback was kept in #31 to leave a pre-#24 server test green; this PR makes the test say what the model says.Behaviour
attrs.runsis an explicit[], or its edge did not resolve) givessame_harness: falseandsame_model: false. The row'sruns_used: 0says why. The Eval version's header fingerprint is never compared.truewhen every used run's fingerprint equals the Card's,falsewhen any differs or is missing, or the Card has none.yesin the comparison table through the header; after this it showsnowithruns used: 0. Nothing stored changes.same_harness(insta snapshot, one line), the generated web client (schema.d.ts, byte-identical to the generator's output) and the web tooltip say the run-level rule including the empty case. README's comparison row already did.Verification
cargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings,cargo test -p evalhub-store(97, Docker),-p evalhub-server(85, Docker, after rebase onto #52),cargo insta test -p evalhub-server(one description line accepted),pnpm check/pnpm lintonweb/. RED observed before the fix: the store tests with a header-matching Card gotsame_model: truefor an empty used set; the new server test gotruns_used: 0, same_harness: true, same_model: trueon the unmodified base. Core / schema / query untouched. No version bump in this PR.