Repository navigation
Seed Search images: Jev image relevance and reference labels - #612
Conversation
run_relevance asks Jev how good each Jev-safe candidate (NSFW score below 1.0) is as an image result for its query. label_batches writes reference-label batches for Sonnet judges: image need for every train query, text relevance for a sample stratified by Jev need, and a vision spot-check of images every local safety layer passed. labels checks the judges' output and joins it to the batches' manifests. Part of #610 No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
json.loads gives True for true, and True is an int equal to 1, so a judge writing true passed as a level 1 label. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In CI, `test` resolves to Python's own test package, so `from test.test_seed_images_eval` fails there. pytest puts this directory on the path, so the module imports directly both locally and in CI. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
| ITEM_MARK = "### " | ||
|
|
||
| NEED_RUBRIC = """\ | ||
| You are helping evaluate a UK web search engine. For each query below, picture its results page: the ten web \ |
There was a problem hiding this comment.
I don't think we should mention UK. As far as I know, Mwmbl aims to be a global search engine. There's nothing UK-specific about it.
There was a problem hiding this comment.
Evaluation is with respect to a specific locale, and at the moment, we are using the UK-English locale. At some point we will be more general.
| first web results for the query are shown to tell you which meaning of an ambiguous query is meant. | ||
|
|
||
| 0 - Wrong. A different subject, the wrong meaning of the query, or nothing anyone searching this would want. \ | ||
| Example: for "jaguar" (the car), a photo of the cat; for "anna friel", a photo of a different actress. |
There was a problem hiding this comment.
What do we do about ambiguous cases that require context?
For example, if someone searches for something ambiguous, like jaguar, what then? Do we show the cat, the car, neither, both?
There was a problem hiding this comment.
Suggestion: if the first query result clearly and unambiguously resolves this, go with that (e.g. jaguar the cat? show the cat image).
There was a problem hiding this comment.
Reply from Claude: The relevance labels already work this way. Each query's batch shows its first three web results, and the rubric says they are there "to tell you which meaning of an ambiguous query is meant". The rubric's 0 example is exactly this case: for "jaguar" (the car), a photo of the cat.
The gap is on the Jev side. Jev's image-relevance question sees only the query, not the web results. In the holistic comparison (#615), wrong-sense strips were the main losses: people with the same name ("boone helm", "isak"), and the palace for "versailles series". Your suggestion is the natural fix. It's now in the production recommendations (#614): give Jev's relevance question the top web results, re-tune the relevance minimum with that prompt, and drop images that don't match. That means relevance waits for Staan. Only the Commons fetch and the NSFW check can run in parallel with it.
| def stratified_sample(scores: dict[str, float], strata: int, total: int, rng: random.Random) -> list[str]: | ||
| """`total` keys, an equal share from each of `strata` equal-sized bands of their score.""" | ||
| ranked = sorted(scores, key=lambda key: (scores[key], key)) | ||
| bands = [ranked[band * len(ranked) // strata : (band + 1) * len(ranked) // strata] for band in range(strata)] |
There was a problem hiding this comment.
Dividing by strata works with larger sets. If we get a smaller data set, sample will raise a value error. Should we check or require a specific minimum data-set size here?
There was a problem hiding this comment.
Presumably if this was an issue it would have been found when running the evaluation.
Mwmbl is a global search engine, so the reference judges shouldn't be told the searcher is in the UK. The #610 labels were made with the old wording and are not redone; the module docstring says so. stratified_sample now raises a clear ValueError when asked for more keys than it has, rather than letting random.sample fail on a short band. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sample size" This reverts commit 5e053c4. The evaluation is for the UK-English locale, so the rubrics keep saying so, and the sample size was never a problem in the runs, so it needs no check. No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No functional changes to the crawler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stack
PR 2 of 7 in the #610 evaluation stack: #611 ← #612 ← #613 ← #616 ← #615 ← #617 ← #614. Merge in that order.
What changed
The labelling half of the query-gate and image-ranking evaluation, in
mwmbl/rankeval/seed_images/:run_relevanceasks Jev, in separate mode, how good each candidate is as an image result. One request per query, train and en-gb. Candidates are those with Jev NSFW below 1.0. Results are cached, with each request's latency.label_batcheswrites self-contained batch files for Sonnet judges. Their rubrics are worded separately from the Jev questions, so the reference stays independent. It covers image need, text relevance and a vision spot-check.labelsvalidates the judges'out_*.txtfiles and joins them to each batch's manifest. It fails loudly, naming the batch, on a malformed line, an unknown or repeated id, a score outside 0–3 or an unlabelled item.gate_dataholds the shared Jev-safe filter.jev_imagesgainsrelevance_request.Results
Jev relevance:
Reference labels (Sonnet judges):
Validation:
How verified
make checkpasses, as dotest/test_seed_images_labels.py(new) andtest/test_seed_images_eval.py.truepassed as a level 1 label, becauseTrue == 1. Booleans are now rejected.test.test_seed_images_eval, which fails in CI, wheretestis Python's own package.mwmbl/rankeval/seed_images/*is omitted from coverage (set in Add a harness to evaluate images in Seed Search #611), because it is offline evaluation code.Decisions already made
Nothing outside
mwmbl/rankevalimports this code, so production is unchanged.Part of #610
🤖 Generated with Claude Code