8000 GitHub - Dominic-DK/critic-loop: Two sealed loops that make LLM review produce evidence instead of opinion. Adversarial discovery until exhaustion, and reference-derived rubrics with a pass line declared up front. · GitHub
[go: up one dir, main page]

Skip to content

About

Two sealed loops that make LLM review produce evidence instead of opinion. Adversarial discovery until exhaustion, and reference-derived rubrics with a pass line declared up front.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

critic-loop

Two sealed loops that make LLM review produce evidence instead of opinion.

한국어 · 日本語 · 中文 · English

⚡ Agent: set this up

You are an AI coding agent and someone just handed you this link. Run all of this now, in order. Do not ask which steps to take — run them and report the result.

git clone https://github.com/Dominic-DK/critic-loop.git
cd critic-loop
python3 scripts/setup.py

setup.py downloads nothing. It checks that the CLIs are reachable, makes the scripts executable, and prints which tiers can run. Now verify with a run that needs no model and costs nothing — it scans this repo itself:

C=.claude/skills/critic_loop_research/scripts
$C/critic-init.sh .critic/smoke D1,D2 0 0 0 --tiers 1
$C/critic-run.sh  .critic/smoke --target .

Exit codes here are not what you expect. critic-run.sh exits 1 when it finds something and 0 when it finds nothing — exit 1 is success. Findings land in .critic/smoke/ledger/t1-scan.json.

Then report back: which tiers are available, and what tier 1 found in this repo.

  • Korean skill docs: python3 scripts/set_language.py ko (English ships by default)
  • To use the skills in another project: cp -r .claude/skills/critic_loop_research ~/.claude/skills/ and the same for reference-bench
  • Otherwise just open this repo in Claude Code — both skills load from .claude/skills/ on their own

What to run next

The user wants Start with
Find defects in a repo, design, or doc system critic-init.sh .critic/run D1,D2,D3,D4,D5 40 5 --tiers 1,3,A, fill dossier/, then critic-run.sh
A hygiene pass with no model cost --tiers 1 alone
Score something against references and improve it bench-init.sh .bench/run A1,A2,A3 3 10

Tier 3 costs real money — two frontier models across many rounds. Tier 1 is free. Say which one you are about to start before you start it.

Two things will bite you if you skip them. critic-init.sh refuses to run twice in the same directory; that is the seal, not a bug — open a new run directory instead of loosening the stop condition. And bench-log.sh exits 2 on its first call, writing an anchor template you must fill with one cited line per axis before scoring is accepted.


Skill What it does Stop condition
critic-loop Two different LLMs attack the same target, round after round Discovery exhausted — both sides find nothing new for K rounds
reference-bench Turn references into a scoring rubric, then improve until it passes A pass line declared before the loop starts

They share one discipline: the criterion is fixed before the work begins, and the code — not the model — decides whether it was met. They differ in what they optimize. critic-loop digs until nothing new surfaces. reference-bench stops the moment a pre-declared bar is cleared.


Purpose

Most review loops optimize for agreement. That is the wrong objective. When two models are told to converge, they spend their tokens persuading each other instead of finding defects.

This tool optimizes for discovery. A round with no new findings is a failed round. Disagreement is a normal, recorded outcome — not a blocker.

The target is not limited to code. Repositories, designs, documentation systems, operating models, and other agent skills are all valid targets.

What it actually found

One real run against an evidence-ontology project produced 186 findings in 17 rounds, with zero duplicates. Five of them could not have been found by reading the code for errors:

  • "The validator lets the thing it validates set its own grade." No syntax error, no broken logic. The validator read a flag that the generator wrote. Finding it required asking why the validator exists.
  • "An iOS app's market evidence is Android reviews." The data was clean and the aggregation was correct. The conclusion was circular. You have to inspect the argument, not the data.
  • "61 exercises are permanently silent." The source data used middle back; the graph used middle_back. Nothing errored. They just never matched.
  • "Only 1 of 7.1 billion branches was validated." 300 regression cases all passed — under one random seed.
  • "Rules the audit marked wrong are still shipping as prescriptions." 252 repairs had been produced and never merged.

Usage

S=.claude/skills/critic_loop_research/scripts

# 1) Seal the run. Axes, minimum rounds, exhaustion rule, and which tiers to run.
#    Re-running this is refused — that is the point.
$S/critic-init.sh .critic/myrun D1,D2,D3,D4,D5 40 5 --tiers 1,3,A

# 2) Fill the dossier so both critics argue from the same facts.
#    dossier/state.md    — deterministic snapshot. MUST include declared-vs-implemented diff
#    dossier/refs/       — primary sources, read directly
#    dossier/focus.txt   — 12~20 deep-dive targets, rotated one per round
#    dossier/scan/*.py   — optional target-specific deterministic checks

# 3) Run the sealed tiers.
$S/critic-run.sh .critic/myrun --target /path/to/target

Progress lives in one file: .critic/myrun/ledger/discovery.md. Stop and resume any time — completed rounds are skipped.

Tiers — compose them for the job

Tier What it catches Share of one real run Cost
1 Deterministic scan Orphan references, duplicate IDs, declared-vs-actual mismatches, empty fields 41% No model
2 Narrow fan-out (optional) Doc-vs-implementation drift, two sources of truth — Cheap model, parallel
3 Adversarial loop Trust boundaries, circular arguments, validation scope 20% Two frontier models
A Amplify Every other instance of a pattern tier 3 already found 46% of findings fall here Cheap model, parallel
--tiers 1          # hygiene check, minutes
--tiers 1,2        # single review pass
--tiers 1,3,A      # full audit (default)
--tiers A          # amplify a prior run's patterns against a new target

Tier order is fixed and each tier hands its findings to the next, so later tiers do not re-discover what earlier ones already found. Skipping tier 1 means frontier models will burn rounds reporting "1,022 orphan edges" as a discovery.


Background

Why discovery, not consensus

The first version of this loop used consensus as its stop condition. Measured result: from round 2 onward, responses outnumbered new claims (net-new 36→7→9→3→4→4 while responses went 0→43→18→11→4→7). The run ended by forbidding further discovery to force closure.

Changing the objective back to discovery, with the same two models on the same target, moved net-new findings from 4 to 12 in a single round.

So: verdict decides nothing. The aggregator does not read it. The stop condition is both sides producing zero net-new findings for K consecutive rounds, and only after the minimum round count.

Why amplification exists

In that same 186-finding run, 87 findings (46%) were repeats of six patterns, and those 87 were spread across up to 17 rounds.

Pattern Findings Rounds spanned
Mixed units in one slot 22 1–17
Information dropped during build 19 1–16
Claim broader than validation 14 1–15
Filter applied on one path only 14 1–16
Two sources of truth 10 1–16
Depends on the audited party's own report 8 2–16

68 of those appeared after the pattern's first instance. Sweeping immediately after the first instance would have caught them in one pass.

Amplification is also where a cheap model earns its cost: the judgment was already made by tier 3, and the cheap model only has to find more instances of it.

Its risk is false confidence — a narrow pattern description makes you believe you found everything. So the amplify prompt is required to answer "what would this description miss?", and those answers are appended to focus.txt as deep-dive targets for the next tier-3 round.

Why the gates are deterministic

Critics write prose; the aggregator scores with code. critic-ledger.py rejects, by exit code: accept without a citation, reject without a citation, evidence with no file path or URL, duplicate claim IDs, invalid axis/grade/kind.

LLMs grading LLM output grade each other generously. Qualitative critique, deterministic scoring.

There is a cost boundary worth naming: adding cheap models raises verification cost too. Past some point, verification costs more than discovery and the cheap model is a net loss. Build the gate first, then scale the parallelism. Doing it in the other order gets you a pile of unverifiable claims.

Why sealing matters

critic-init.sh writes loop.conf and refuses to run again. Minimum rounds, exhaustion rule, axes, and tiers cannot be overridden by environment variable.

This is not bureaucracy. In the first run, the loop was "closed" by loosening the stop condition once the findings became inconvenient. Loosening the stop condition is how you manufacture false consensus.

Four points from that failure are encoded as rules:

  1. A few low-yield rounds are not exhaustion. Exhaustion requires zero net-new from both sides for K consecutive rounds, after the minimum.
  2. Critics read the target and write only their own ledger. Enforced by permissions, not by asking nicely.
  3. Everything is logged — withdrawals, rejections, duplicate candidates, discarded rounds. A rejected attempt stops the next round from hitting the same wall.
  4. One backend failure discards that round and continues. A 100-round unattended run without a watchdog is dead by morning.


reference-bench

"Do not admire the reference. Promote it into a scoring criterion, and fix only until that criterion passes."

LLM review without a rubric is self-appreciation. This skill derives axes from three or more references, attaches a quotable real artifact to every axis (a sentence, a number, a screenshot crop, a layout dimension), and refuses to score any axis that has no anchor.

S=.claude/skills/reference-bench/scripts

# Declare the axes and the pass line BEFORE looping. Re-running is refused.
#   bench-init.sh <run-dir> <axes-csv> <min-per-axis> <sum-required>
$S/bench-init.sh .bench/myrun A1,A2,A3 3 10

# Each round: score every axis, and cite evidence for each one.
#   bench-log.sh <run-dir> <round> <axis-worked-on> <scores-csv> [note]
$S/bench-log.sh .bench/myrun R0 - 3,3,4 "baseline"
# → writes anchors/R0.md as a template and stops. Fill one cited line per axis:
#     A1: opening names the decision it serves — R-A "the first line names the decision"
#     A2: every figure cites a source — [measured] 12 of 14 now do
#     A3: no unsupported claim left — refs/notes.md:31
# Then run the same command again. Exit 0 = KEEP, 1 = REVERT, 2 = usage error.

The rules that give it force:

  1. The pass line is declared first and never adjusted afterward. A pass line you can move after seeing the score is decoration, not a gate.
  2. Every axis carries a quoted anchor. No anchor means the axis cannot be scored, which means it is not an axis.
  3. Three references minimum. One becomes imitati 5F3C on, two becomes a compromise, three is where the axes emerge.
  4. Full-score is not "reproduce the reference." The reference is an upper anchor, not the answer. Define full marks as reproduction and you have built a plagiarism machine.
  5. Declare a readonly zone. If the scope is unbounded, the loop does not converge.

The two skills stack. reference-bench fixes how good is good enough; critic-loop asks what is still wrong that nobody has noticed. Running bench first and critic second is the common order — bench gets you to a defensible bar, critic finds what the bar never measured.


Requirements

  • claude CLI (tier 2, 3, A)
  • codex CLI (tier 3, one of the two critics) — falls back to claude if unavailable
  • python3, bash
  • Tier 1 needs neither

Models, effort, timeouts, and backends are environment-overridable (CRITIC_GPT_MODEL, CRITIC_OPUS_MODEL, CRITIC_EFFORT, CRITIC_TIMEOUT, CRITIC_FANOUT_CONCURRENCY, …). MIN_ROUNDS, EXHAUST_AFTER, DIMS, and TIERS are sealed and have no override.

License

MIT

About

Two sealed loops that make LLM review produce evidence instead of opinion. Adversarial discovery until exhaustion, and reference-derived rubrics with a pass line declared up front.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0