Two sealed loops that make LLM review produce evidence instead of opinion.
You are an AI coding agent and someone just handed you this link. Run all of this now, in order. Do not ask which steps to take — run them and report the result.
git clone https://github.com/Dominic-DK/critic-loop.git
cd critic-loop
python3 scripts/setup.pysetup.py downloads nothing. It checks that the CLIs are reachable, makes the scripts executable, and
prints which tiers can run. Now verify with a run that needs no model and costs nothing — it scans
this repo itself:
C=.claude/skills/critic_loop_research/scripts
$C/critic-init.sh .critic/smoke D1,D2 0 0 0 --tiers 1
$C/critic-run.sh .critic/smoke --target .Exit codes here are not what you expect. critic-run.sh exits 1 when it finds something and 0
when it finds nothing — exit 1 is success. Findings land in .critic/smoke/ledger/t1-scan.json.
Then report back: which tiers are available, and what tier 1 found in this repo.
- Korean skill docs:
python3 scripts/set_language.py ko(English ships by default) - To use the skills in another project:
cp -r .claude/skills/critic_loop_research ~/.claude/skills/and the same forreference-bench - Otherwise just open this repo in Claude Code — both skills load from
.claude/skills/on their own
| The user wants | Start with |
|---|---|
| Find defects in a repo, design, or doc system | critic-init.sh .critic/run D1,D2,D3,D4,D5 40 5 --tiers 1,3,A, fill dossier/, then critic-run.sh |
| A hygiene pass with no model cost | --tiers 1 alone |
| Score something against references and improve it | bench-init.sh .bench/run A1,A2,A3 3 10 |
Tier 3 costs real money — two frontier models across many rounds. Tier 1 is free. Say which one you are about to start before you start it.
Two things will bite you if you skip them. critic-init.sh refuses to run twice in the same
directory; that is the seal, not a bug — open a new run directory instead of loosening the stop
condition. And bench-log.sh exits 2 on its first call, writing an anchor template you must fill
with one cited line per axis before scoring is accepted.
| Skill | What it does | Stop condition |
|---|---|---|
| critic-loop | Two different LLMs attack the same target, round after round | Discovery exhausted — both sides find nothing new for K rounds |
| reference-bench | Turn references into a scoring rubric, then improve until it passes | A pass line declared before the loop starts |
They share one discipline: the criterion is fixed before the work begins, and the code — not the model —
decides whether it was met. They differ in what they optimize. critic-loop digs until nothing new
surfaces. reference-bench stops the moment a pre-declared bar is cleared.
Most review loops optimize for agreement. That is the wrong objective. When two models are told to converge, they spend their tokens persuading each other instead of finding defects.
This tool optimizes for discovery. A round with no new findings is a failed round. Disagreement is a normal, recorded outcome — not a blocker.
The target is not limited to code. Repositories, designs, documentation systems, operating models, and other agent skills are all valid targets.
One real run against an evidence-ontology project produced 186 findings in 17 rounds, with zero duplicates. Five of them could not have been found by reading the code for errors:
- "The validator lets the thing it validates set its own grade." No syntax error, no broken logic. The validator read a flag that the generator wrote. Finding it required asking why the validator exists.
- "An iOS app's market evidence is Android reviews." The data was clean and the aggregation was correct. The conclusion was circular. You have to inspect the argument, not the data.
- "61 exercises are permanently silent." The source data used
middle back; the graph usedmiddle_back. Nothing errored. They just never matched. - "Only 1 of 7.1 billion branches was validated." 300 regression cases all passed — under one random seed.
- "Rules the audit marked wrong are still shipping as prescriptions." 252 repairs had been produced and never merged.
S=.claude/skills/critic_loop_research/scripts
# 1) Seal the run. Axes, minimum rounds, exhaustion rule, and which tiers to run.
# Re-running this is refused — that is the point.
$S/critic-init.sh .critic/myrun D1,D2,D3,D4,D5 40 5 --tiers 1,3,A
# 2) Fill the dossier so both critics argue from the same facts.
# dossier/state.md — deterministic snapshot. MUST include declared-vs-implemented diff
# dossier/refs/ — primary sources, read directly
# dossier/focus.txt — 12~20 deep-dive targets, rotated one per round
# dossier/scan/*.py — optional target-specific deterministic checks
# 3) Run the sealed tiers.
$S/critic-run.sh .critic/myrun --target /path/to/targetProgress lives in one file: .critic/myrun/ledger/discovery.md.
Stop and resume any time — completed rounds are skipped.
| Tier | What it catches | Share of one real run | Cost |
|---|---|---|---|
| 1 Deterministic scan | Orphan references, duplicate IDs, declared-vs-actual mismatches, empty fields | 41% | No model |
| 2 Narrow fan-out (optional) | Doc-vs-implementation drift, two sources of truth | — | Cheap model, parallel |
| 3 Adversarial loop | Trust boundaries, circular arguments, validation scope | 20% | Two frontier models |
| A Amplify | Every other instance of a pattern tier 3 already found | 46% of findings fall here | Cheap model, parallel |
--tiers 1 # hygiene check, minutes
--tiers 1,2 # single review pass
--tiers 1,3,A # full audit (default)
--tiers A # amplify a prior run's patterns against a new targetTier order is fixed and each tier hands its findings to the next, so later tiers do not re-discover what earlier ones already found. Skipping tier 1 means frontier models will burn rounds reporting "1,022 orphan edges" as a discovery.
The first version of this loop used consensus as its stop condition. Measured result: from round 2 onward, responses outnumbered new claims (net-new 36→7→9→3→4→4 while responses went 0→43→18→11→4→7). The run ended by forbidding further discovery to force closure.
Changing the objective back to discovery, with the same two models on the same target, moved net-new findings from 4 to 12 in a single round.
So: verdict decides nothing. The aggregator does not read it. The stop condition is both sides
producing zero net-new findings for K consecutive rounds, and only after the minimum round count.
In that same 186-finding run, 87 findings (46%) were repeats of six patterns, and those 87 were spread across up to 17 rounds.
| Pattern | Findings | Rounds spanned |
|---|---|---|
| Mixed units in one slot | 22 | 1–17 |
| Information dropped during build | 19 | 1–16 |
| Claim broader than validation | 14 | 1–15 |
| Filter applied on one path only | 14 | 1–16 |
| Two sources of truth | 10 | 1–16 |
| Depends on the audited party's own report | 8 | 2–16 |
68 of those appeared after the pattern's first instance. Sweeping immediately after the first instance would have caught them in one pass.
Amplification is also where a cheap model earns its cost: the judgment was already made by tier 3, and the cheap model only has to find more instances of it.
Its risk is false confidence — a narrow pattern description makes you believe you found everything.
So the amplify prompt is required to answer "what would this description miss?", and those answers are
appended to focus.txt as deep-dive targets for the next tier-3 round.
Critics write prose; the aggregator scores with code. critic-ledger.py rejects, by exit code:
accept without a citation, reject without a citation, evidence with no file path or URL, duplicate
claim IDs, invalid axis/grade/kind.
LLMs grading LLM output grade each other generously. Qualitative critique, deterministic scoring.
There is a cost boundary worth naming: adding cheap models raises verification cost too. Past some point, verification costs more than discovery and the cheap model is a net loss. Build the gate first, then scale the parallelism. Doing it in the other order gets you a pile of unverifiable claims.
critic-init.sh writes loop.conf and refuses to run again. Minimum rounds, exhaustion rule, axes, and
tiers cannot be overridden by environment variable.
This is not bureaucracy. In the first run, the loop was "closed" by loosening the stop condition once the findings became inconvenient. Loosening the stop condition is how you manufacture false consensus.
Four points from that failure are encoded as rules:
- A few low-yield rounds are not exhaustion. Exhaustion requires zero net-new from both sides for K consecutive rounds, after the minimum.
- Critics read the target and write only their own ledger. Enforced by permissions, not by asking nicely.
- Everything is logged — withdrawals, rejections, duplicate candidates, discarded rounds. A rejected attempt stops the next round from hitting the same wall.
- One backend failure discards that round and continues. A 100-round unattended run without a watchdog is dead by morning.
"Do not admire the reference. Promote it into a scoring criterion, and fix only until that criterion passes."
LLM review without a rubric is self-appreciation. This skill derives axes from three or more references, attaches a quotable real artifact to every axis (a sentence, a number, a screenshot crop, a layout dimension), and refuses to score any axis that has no anchor.
S=.claude/skills/reference-bench/scripts
# Declare the axes and the pass line BEFORE looping. Re-running is refused.
# bench-init.sh <run-dir> <axes-csv> <min-per-axis> <sum-required>
$S/bench-init.sh .bench/myrun A1,A2,A3 3 10
# Each round: score every axis, and cite evidence for each one.
# bench-log.sh <run-dir> <round> <axis-worked-on> <scores-csv> [note]
$S/bench-log.sh .bench/myrun R0 - 3,3,4 "baseline"
# → writes anchors/R0.md as a template and stops. Fill one cited line per axis:
# A1: opening names the decision it serves — R-A "the first line names the decision"
# A2: every figure cites a source — [measured] 12 of 14 now do
# A3: no unsupported claim left — refs/notes.md:31
# Then run the same command again. Exit 0 = KEEP, 1 = REVERT, 2 = usage error.The rules that give it force:
- The pass line is declared first and never adjusted afterward. A pass line you can move after seeing the score is decoration, not a gate.
- Every axis carries a quoted anchor. No anchor means the axis cannot be scored, which means it is not an axis.
- Three references minimum. One becomes imitati 5F3C on, two becomes a compromise, three is where the axes emerge.
- Full-score is not "reproduce the reference." The reference is an upper anchor, not the answer. Define full marks as reproduction and you have built a plagiarism machine.
- Declare a readonly zone. If the scope is unbounded, the loop does not converge.
The two skills stack. reference-bench fixes how good is good enough; critic-loop asks what is still
wrong that nobody has noticed. Running bench first and critic second is the common order — bench gets you
to a defensible bar, critic finds what the bar never measured.
claudeCLI (tier 2, 3, A)codexCLI (tier 3, one of the two critics) — falls back toclaudeif unavailablepython3,bash- Tier 1 needs neither
Models, effort, timeouts, and backends are environment-overridable
(CRITIC_GPT_MODEL, CRITIC_OPUS_MODEL, CRITIC_EFFORT, CRITIC_TIMEOUT, CRITIC_FANOUT_CONCURRENCY, …).
MIN_ROUNDS, EXHAUST_AFTER, DIMS, and TIERS are sealed and have no override.
MIT