Repository navigation
Mission layer M9.1 research and M9.5 judges - #73
Merged
Merged
Conversation
Research (M9.1): `aloop mission research new / archive / claim / check / resolve / show / list / retire`. Sources are archived as content-addressed snapshots (research/snapshots/<sha256>); every claim cites a quote that must occur in its snapshot (normalized), or earlier claims; check re-hashes every snapshot and re-finds every quote. Citation refs are rsr_<id>#cN, resolved by research.cite_problems for journeys and rubrics. Judges (M9.5): `aloop mission judge new / activate / check / run / verdict / show / list / retire`. A rubric pins a model id (no alias, route or latest), at least 3 runs and a majority agreement, cites research claims, and is frozen on activation. `judge run` executes the pinned judge command and records every run; the verdict is computed from the first N valid runs on the rubric's current sha256, so re-running cannot shop for a verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EX5zT25VQABa7psjEmHLNS
Review (3 lenses per module, 44 findings, each fixed with a regression test): quotes match as plain text (no stripped markup can smuggle words), whole words, linear normalization, malformed files block instead of crashing, locked concurrent writes, --file confined to the project, final_url and source provenance recorded; rubrics require their seal, status agrees with fields and events, records match their judge.run events, plain-text or mismatched served models never count, the subject is one token, the prompt template is recorded and gates the verdict, ALOOP_JUDGE_TIMEOUT. The event log is git-ignored, so a report, active rubric or run record with no local event counts when HEAD holds it exactly (a fresh clone or a pull); an uncommitted re-sealed edit or forged record still blocks. verdict() takes rubric_sha256 (an approved journey pins its rubric's content; agreed with the M9.3 owner) and input_sha256; each input of a subject has its own first N runs, so re-assessment after a sprint works and re-running never changes a verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EX5zT25VQABa7psjEmHLNS
… follow-ups Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EX5zT25VQABa7psjEmHLNS
The core now decodes argv itself (invalid UTF-8 becomes U+FFFD), so the research test checks the module's own non-UTF-8 guard in process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EX5zT25VQABa7psjEmHLNS
…judges-aj204m # Conflicts: # docs/ROADMAP.md
adeelahmad
marked this pull request as ready for review
October 20, 2026 04:31
This was referenced Oct 20, 2026
adeelahmad
pushed a commit
that referenced
this pull request
Oct 20, 2026
adeelahmad
pushed a commit
that referenced
this pull request
Oct 20, 2026
The role runs use the real research and judge commands: the Scout's report is written through the CLI, and research and judge checks get the artifact's id. Gap rows move to M9.40/M9.41 (M9.28-M9.35 are taken by open PRs); M9.42 records the shared research dir and judge runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0197nSutKR1QAAqgJhFZ8fPj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Adeel · project thread
Before: the mission layer could seal and approve a mission, but research and LLM judging had no structure, so "best" could rest on unsourced claims and a single judge call.
After:
aloop mission research …records a sourced landscape report where every source is an archived, content-addressed snapshot and every claim quotes text that must occur, as whole words, in its snapshot.aloop mission judge …grades subjective UX with a frozen rubric on a pinned model, runs it at least 3 times, and computes an agreement verdict from the recorded runs.The gates check the setup and the evidence, never whether a judgment is right:
research checkre-hashes every snapshot and re-finds every quote;judge checkrefuses alias/route/latest/[1m]models, fewer than 3 runs, non-majority agreement and unsourced rubrics, and matches every run record to itsjudge.runevent.How: two discovered areas,
plugin/mission/research.pyandplugin/mission/judge.py, withschemas/research.jsonandschemas/rubric.json.rsr_<id>#cN, resolved byresearch.cite_problems(rubrics and journeys use it).judge.verdict(cwd, rubric_id, subject, input_sha256=None, input_text=None, rubric_sha256=None)returns pass, fail, inconclusive, incomplete or blocked. It takes the first N valid runs on the rubric's current content and prompt template, so re-running cannot change a verdict.rubric_sha256lets an approved journey pin its rubric's content (agreed with the M9.3 owner).Review: three adversarial lenses per module (correctness, gate bypass, conventions) found 44 issues. Each one is fixed with a regression test, including quotes smuggling words in markup, re-sealed hand edits, forged or edited run records, revived retired rubrics, and served-model mismatches.
Evidence:
scripts/test/mission-research.test.py(18 checks) andscripts/test/mission-judge.test.py(24 checks), run by the Makefile'smission-*glob. Both use a local HTTP server and a fake judge command, with no network and no tokens.make test-hostsand all 7 mission suites pass locally after merging master (#77, #71).🤖 Generated with Claude Code
https://claude.ai/code/session_01EX5zT25VQABa7psjEmHLNS