A benchmark for MCP servers, skills and agent harnesses in the TYPO3 context.
It answers four questions with numbers instead of anecdote:
- Does installing this help? — success rate against a control arm that has the same model and the same file access, but no server and no skills.
- Which one helps most? — several packages side by side, each installed the way its own installer installs it, and two refs of one package as two subjects.
- What does it cost? — per task, and as a fixed context tax paid on every request whether anything uses it or not.
- Which tools earn their place? — calls, bytes returned, success rate of the runs that used them, and the list of tools that never fired at all.
Targets are built beside this repository, never inside it, and every agent
process runs with its own HOME. So the machine's own configuration can neither
influence a measurement nor be changed by one, and nothing the agent can read —
its working directory included — tells it that its answers are being scored.
composer install
cd web && npm install && npm run build && cd ..
# Optional. Every subject names a repository or a published package, so nothing
# machine-specific is required — this is for overriding where a target is built
# or for measuring your own working tree. Never committed.
cp bench.local.yaml.example bench.local.yaml
./bin/bench doctor # what is configured, and what is missing
./bin/bench run --target smoke --case smoke --reps 1 # prove the pipeline, ~5s
./bin/bench target:setup typo3-14 # build the TYPO3 target (~20 min, once)
./bin/bench run --label pilot # the sweep
./bin/bench judge # grade what checkers cannot
./bin/bench tax # what the provisions cost per request
./bin/bench report # this submission: claim table, per-case, tool ledger
./bin/bench report --suite # the whole suite: every submission that answered it, merged
./bin/bench serve # the results, in a browserHost requirements: PHP 8.3, Composer, git. ddev for TYPO3 targets, Node for the
web app, and at least one agent CLI — opencode and claude are supported out
of the box.
A measurement variant — an arm — is four things:
harness × model × provision × target
claude opus-5 dev-companion typo3-14
A provision is what the arm is equipped with, and normally it is a subject: a package, put in place by its own install command rather than by a reconstruction of what that command does. The provision that installs nothing is the control arm, which is why nothing in the runner needs a special case for it — it is a subject like any other, declared by having no source.
What a package provides is never written down. It is read back per run as the difference between the workspace and the master it was copied from, so a release that ships a new skill is measured as shipping it, without anyone updating a list.
bench.yaml declares what can be run; its matrix: section decides what does.
The cross product is expanded into arms, and a model that declares it only runs
under certain harnesses silently drops the impossible combinations.
The control arm is not blindfolded. It is a normal agent session with read,
grep and shell access in a real TYPO3 project. It can grep Configuration/TCA/,
run vendor/bin/typo3, and read settings.php and additional.php. A control
that cannot do those things measures "tools versus nothing" and proves nothing.
The target is host-mounted. Agent CLIs run on the host, so a container-only
project would leave the control arm reaching files through ddev exec cat —
crippling the control and inflating everything else.
The tools under test are never the oracle. Ground truth comes from
vendor/bin/typo3, direct SQL and reading config files. The judge model runs
from an empty workspace with no provision at all.
A mis-provisioned arm is refused, not measured. Provisioning reads back which MCP servers the harness configuration actually names, and a control arm with a server attached — or an arm whose install silently failed — voids rather than producing a null result that looks like a finding.
One run per arm is noise. Five repetitions by default, reported as medians: agent runs have a long right tail, and one runaway run would otherwise swallow the signal.
The target is built once and copied per run. A master is recorded before any
arm touches it; every run starts as an exact copy of it, database included. Git
tracks 43 of the target's 16,000 files, so a git-based reset would have left
vendor/, var/ and the database exactly as the last agent left them.
The agent cannot tell it is being measured. The project it is dropped into is
an ordinary customer site called Riverside — package name, namespace, table
prefix, backend labels, page titles, extension icon, DDEV project and first git
commit all agree on that story. No BENCH_* variable reaches an agent process,
and $HOME carries nothing of the arm's name — an agent that could read it
would learn not just that it is in an experiment but which side of it. An agent
that recognises a benchmark hedges and double-checks, and stops predicting the
user it stands in for.
What a tool provides is measured, not declared. A subject in bench.yaml
says only where its code comes from and which command switches it on. Every run
starts as an exact copy of a master, so what the install command wrote is simply
the difference afterwards, recorded per run. Listing a package's servers and
skills by hand describes the transcription rather than the package, and goes out
of date with the next release.
Every tool gets its own master. The target is built once, then each subject
is installed into its own copy. A run restores the master for its arm — so the
control arm starts from a workspace where the package was never present, rather
than one where it sits in composer.json switched off. It also means a second
tool can be added without rebuilding anything the first one depends on.
Cheap and wrong is not a win. Every claim table carries cost-per-success next to success, and the context tax next to the benefit.
bench.yaml |
the experiment: harnesses, models, subjects, provisions, targets, matrix |
bench.local.yaml |
machine-specific overrides, deep-merged, never committed; see bench.local.yaml.example |
cases/<family>/<id>.yaml |
one question per file: prompt, ground truth, grading rule |
fixtures/<name>/ |
package/ (the TYPO3 extension), seeder/ (build-time only) and seed.sh |
../riverside/ |
built systems under test and sandbox homes — outside this checkout, because the agent can read both |
runs/<timestamp>--<label>/ |
manifest, ledger, answers, summary — committed |
var/ |
subject checkouts, target masters, raw event streams, scratch — never committed |
web/ |
the results explorer; static files over the JSON in runs/ |
src/ |
the runner |
bench doctor |
configuration, harnesses, targets, containment |
bench cases:list / cases:validate / cases:import |
curate the questions |
bench subject:list / subject:build / subject:inspect |
the tools under test: what is defined, install one into the target, report what it produced |
bench target:setup / target:status / target:reset |
the system under test |
bench run |
a sweep; --case, --provision, --model, --target, --reps, --dry-run |
bench judge |
grade the rows a checker could not decide |
bench report |
claim table, per-case deltas, tool ledger; --suite merges every submission that answered the same suite, --json for the raw summary |
bench suite:list |
the named sets of cases a run can ask |
bench runs:list |
the measurement history |
bench baseline:pin / baseline:show |
the reference every later run is read against |
bench mcp:inspect |
what a server registers, and what its schemas cost — no model involved |
bench tax |
the fixed context cost each provision adds |
bench web:export / bench serve |
publish and browse the results |
Options on bench run replace matrix axes rather than filtering the result,
so a one-off question can reach a target or provision the committed matrix leaves
out without editing it.
- docs/architecture.md — how the pieces fit, and which invariants the design exists to protect
- docs/case-format.md — writing a case
- docs/adding-a-run.md — measuring an arm and having its rows land in the right suite
- docs/adding-a-harness.md — supporting another agent CLI
This replaces the single-purpose dev-companion-benchmark, which measured one
MCP server on one harness with two hard-coded arms, in bash. Its 17 tasks import
with bench cases:import and are the starting case set here. Its own numbers do
not carry over — that sweep never finished, and its report says so.