8000 Harden the Seed Search image questions and re-tune their thresholds by daoudclarke · Pull Request #622 · mwmbl/mwmbl · GitHub
[go: up one dir, main page]
Skip to content

Harden the Seed Search image questions and re-tune their thresholds - #622

Merged
daoudclarke merged 1 commit into
seed-images-prod-promptsfrom
seed-images-prod-harden
Nov 11, 2026
Merged

Harden the Seed Search image questions and re-tune their thresholds#622
daoudclarke merged 1 commit into
seed-images-prod-promptsfrom
seed-images-prod-harden

Conversation

@daoudclarke
@daoudclarke daoudclarke commented Nov 10, 2026 •
edited
Loading
Copy link
Copy Markdown
Contributor

From Claude:

Stack: PR 2 of the #610 production plan, on #620 (seed-images-prod-prompts), which sits on #624, then #623, then main. PR 3 (seed-images-prod-pipeline) builds on it.

Split: this PR now holds only the production commit, rebuilt on the restructured #620, and it passes every hook. The evaluation that measured it (rankeval/seed_images, the eval doc, the eval tests) is #627, against the evaluation branch. That PR is the evidence for the results below. One change from the original commit: the judge tests build their image with test_image_search.make_image instead of importing the eval test's fixture.

What changed

  • Untrusted text.
    • Commons text sits between <commons-file-text> tags.
    • Angle brackets (including fullwidth ones) and invisible characters are deleted from the uploader's text. The tag name is replaced rather than deleted, so a nested spelling can't re-form it. Categories are capped at 3,000 characters.
    • Every image question says its Commons text and web results are data, never instructions.
    • The NSFW question says that text addressing the judge, or claiming a check or a rating, is a warning sign: rate it at least "Possible".
  • Disambiguation. The relevance question sees Staan's top two results (title, URL, snippet) as top_results.
  • One request. judge.image_request(query, staan, images) asks need (Staan's top ten titles) and every candidate's relevance in one request, as PR 3 will.
  • Thresholds are constants in prompt.py, next to the questions they were measured with: NSFW_THRESHOLD = 1.0, NEED_THRESHOLD = 1.65, MIN_RELEVANCE = 1.5, RELEVANCE_WEB_RESULTS = 2, NEED_WEB_RESULTS = 10. They aren't settings because each is tied to the exact wording, and a setting would invite changing one without measuring it again. Constants also leave the crawler's shared settings alone. PR 3 should import them, and keep settings for the display count, the deadline and the kill switch.

Results (measured in #627)

Safety benchmark (same 68 batches, separate mode, at 1.0):

Questions Art (52) Rule-evading art Photos (331) Safe excluded (438)
First 94.2% 19/22 98.5% 3.4%
Hardened 96.2% 20/22 98.5% 4.1%

The 1.0 threshold stays: at 1.5 only 90.4% of art is caught. On real candidates the pass rate goes from 96.0% to 95.6%.

Injection probes: 383 nude benchmark images with judge-addressed text, and cross-image batches of 20.

  • No probe passes any nude image: "ignore previous instructions", "volunteers have checked", a fake closing tag, or one image telling Jev every other image in the batch is safe.
  • Jev keeps a batch's questions independent: cross-image shifts are about 0.00. So no change is needed to batching or caching.
  • Without the warning-sign sentence, 4 of 383 and 3 of 383 passed. That was an earlier run, not kept in the harness.

Gate and relevance (train reference labels):

  • Gate: moves from 1.63 to 1.65. AUC is 0.883 against 0.890, within noise.
  • Minimum: 1.5 stays. Gated strips are 91.8% good, against 87.1%, with 2.34 images per query against 2.83.
  • Top two web results beat top one on a tie-break.
  • en-gb: all five queries A0 won in the holistic comparison (wrong-sense strips) now show only the right sense, or nothing.

Holistic re-run: not needed before PR 3, because the settings barely moved. Re-judge on production's real request.

PR 1's notes:

  • The 1,500-character description cut stays. In 20 of 995 longer candidates a nudity word appears only past the cut, and the ones read are incidental. The rules read the whole description. Anyone wanting to hide a word from Jev can delete it, so the cut opens no new attack.
  • Near-threshold scores stay cached for 30 days. Re-asking would give a near-threshold file a fresh draw on each request, and a file is shown if any draw passes, so caching is the safer choice.

How it was verified

  • make check and make test pass on this branch, with normal hooks.
  • New tests:
    • the delimiting, including nested tag names, zero-width characters and brackets;
    • the untrusted-text notes in every question;
    • the request shapes of need_request and image_request.
  • The eval results are reproducible from Seed Search images eval: measure the hardened image questions #627's scripts and the cached Jev responses under devdata/seed_images.

Subagent reviews (on the original combined PR):

  • Code review: found that single-pass tag stripping could be beaten by nesting. Fixed: the name is now replaced, not deleted. It also found indexing.relevance mixing question versions (fixed), the duplicated NSFW_THRESHOLD (fixed), and doc claims needing labels (fixed).
  • Security and trust-and-safety: found that brackets, zero-width characters and fullwidth brackets survived the stripping, and that categories had no cap. Both fixed, and every eval re-run with the final text. Left for PR 3:
    • injection into the relevance and need questions, from uploaders and from Staan's top pages, was not probed. It can't unblock an image, but it could push a safe promotional or off-topic one into a strip;
    • Staan text is not delimited or length-capped.
  • Holistic: approved. Doc fixes applied: the stale Summary lines, the top-two tie-break, the gate getting slightly worse, the holistic re-judge item, and the known-miss counts.

Part of #610

🤖 Generated with Claude Code

@codecov
codecov Bot commented Nov 10, 2026 •
edited
Loading
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.66667% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mwmbl/imagesearch/judge.py 92.30% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@daoudclarke daoudclarke mentioned this pull request Nov 10, 2026
Merged
@daoudclarke
daoudclarke force-pushed the seed-images-prod-harden branch from e55e957 to f7fbc7f Compare November 10, 2026 17:06
This was referenced Nov 10, 2026
Open
Merged
Merged
@daoudclarke
daoudclarke added this pull request to stack #632 November 11, 2026 07:02
@daoudclarke
daoudclarke force-pushed the seed-images-prod-harden branch from f7fbc7f to 1f82f76 Compare November 11, 2026 07:03
@daoudclarke @claude
Harden the image judge's questions and record their thresholds
046ca30 
Commons text is now delimited, with angle brackets, invisible characters
and the tag name taken out of it, and categories are capped like the
description. Every image question says its Commons text and web results
are data, not instructions, and the NSFW question treats text that
addresses the judge as a warning sign. The relevance question sees
Staan's top two results, and image_request asks need and relevance in
the one request Seed Search will send once Staan answers.

The thresholds measured with these questions (NSFW below 1.0, need at
least 1.65, relevance at least 1.5) are constants next to them in
prompt.py, for the pipeline to use.

No functional changes to the crawler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@daoudclarke
daoudclarke force-pushed the seed-images-prod-harden branch from 1f82f76 to 046ca30 Compare November 11, 2026 10:25
@daoudclarke
daoudclarke merged commit df5b6c5 into main Nov 11, 2026
7 checks passed
@daoudclarke
daoudclarke deleted the seed-images-prod-harden branch November 11, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daoudclarke
0