8000 GitHub - benjaminkott/typo3-mcp-bench · GitHub
[go: up one dir, main page]

Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

typo3-mcp-bench

A benchmark for MCP servers, skills and agent harnesses in the TYPO3 context.

It answers four questions with numbers instead of anecdote:

  1. Does installing this help? — success rate against a control arm that has the same model and the same file access, but no server and no skills.
  2. Which one helps most? — several packages side by side, each installed the way its own installer installs it, and two refs of one package as two subjects.
  3. What does it cost? — per task, and as a fixed context tax paid on every request whether anything uses it or not.
  4. Which tools earn their place? — calls, bytes returned, success rate of the runs that used them, and the list of tools that never fired at all.

Targets are built beside this repository, never inside it, and every agent process runs with its own HOME. So the machine's own configuration can neither influence a measurement nor be changed by one, and nothing the agent can read — its working directory included — tells it that its answers are being scored.


Quick start

composer install
cd web && npm install && npm run build && cd ..

# Optional. Every subject names a repository or a published package, so nothing
# machine-specific is required — this is for overriding where a target is built
# or for measuring your own working tree. Never committed.
cp bench.local.yaml.example bench.local.yaml

./bin/bench doctor                     # what is configured, and what is missing
./bin/bench run --target smoke --case smoke --reps 1   # prove the pipeline, ~5s

./bin/bench target:setup typo3-14      # build the TYPO3 target (~20 min, once)
./bin/bench run --label pilot          # the sweep
./bin/bench judge                      # grade what checkers cannot
./bin/bench tax                        # what the provisions cost per request
./bin/bench report                     # this submission: claim table, per-case, tool ledger
./bin/bench report --suite             # the whole suite: every submission that answered it, merged
./bin/bench serve                      # the results, in a browser

Host requirements: PHP 8.3, Composer, git. ddev for TYPO3 targets, Node for the web app, and at least one agent CLI — opencode and claude are supported out of the box.

The idea

A measurement variant — an arm — is four things:

harness  ×  model  ×  provision  ×  target
claude      opus-5     dev-companion   typo3-14

A provision is what the arm is equipped with, and normally it is a subject: a package, put in place by its own install command rather than by a reconstruction of what that command does. The provision that installs nothing is the control arm, which is why nothing in the runner needs a special case for it — it is a subject like any other, declared by having no source.

What a package provides is never written down. It is read back per run as the difference between the workspace and the master it was copied from, so a release that ships a new skill is measured as shipping it, without anyone updating a list.

bench.yaml declares what can be run; its matrix: section decides what does. The cross product is expanded into arms, and a model that declares it only runs under certain harnesses silently drops the impossible combinations.

Why it is built this way

The control arm is not blindfolded. It is a normal agent session with read, grep and shell access in a real TYPO3 project. It can grep Configuration/TCA/, run vendor/bin/typo3, and read settings.php and additional.php. A control that cannot do those things measures "tools versus nothing" and proves nothing.

The target is host-mounted. Agent CLIs run on the host, so a container-only project would leave the control arm reaching files through ddev exec cat — crippling the control and inflating everything else.

The tools under test are never the oracle. Ground truth comes from vendor/bin/typo3, direct SQL and reading config files. The judge model runs from an empty workspace with no provision at all.

A mis-provisioned arm is refused, not measured. Provisioning reads back which MCP servers the harness configuration actually names, and a control arm with a server attached — or an arm whose install silently failed — voids rather than producing a null result that looks like a finding.

One run per arm is noise. Five repetitions by default, reported as medians: agent runs have a long right tail, and one runaway run would otherwise swallow the signal.

The target is built once and copied per run. A master is recorded before any arm touches it; every run starts as an exact copy of it, database included. Git tracks 43 of the target's 16,000 files, so a git-based reset would have left vendor/, var/ and the database exactly as the last agent left them.

The agent cannot tell it is being measured. The project it is dropped into is an ordinary customer site called Riverside — package name, namespace, table prefix, backend labels, page titles, extension icon, DDEV project and first git commit all agree on that story. No BENCH_* variable reaches an agent process, and $HOME carries nothing of the arm's name — an agent that could read it would learn not just that it is in an experiment but which side of it. An agent that recognises a benchmark hedges and double-checks, and stops predicting the user it stands in for.

What a tool provides is measured, not declared. A subject in bench.yaml says only where its code comes from and which command switches it on. Every run starts as an exact copy of a master, so what the install command wrote is simply the difference afterwards, recorded per run. Listing a package's servers and skills by hand describes the transcription rather than the package, and goes out of date with the next release.

Every tool gets its own master. The target is built once, then each subject is installed into its own copy. A run restores the master for its arm — so the control arm starts from a workspace where the package was never present, rather than one where it sits in composer.json switched off. It also means a second tool can be added without rebuilding anything the first one depends on.

Cheap and wrong is not a win. Every claim table carries cost-per-success next to success, and the context tax next to the benefit.

Layout

bench.yaml the experiment: harnesses, models, subjects, provisions, targets, matrix
bench.local.yaml machine-specific overrides, deep-merged, never committed; see bench.local.yaml.example
cases/<family>/<id>.yaml one question per file: prompt, ground truth, grading rule
fixtures/<name>/ package/ (the TYPO3 extension), seeder/ (build-time only) and seed.sh
../riverside/ built systems under test and sandbox homes — outside this checkout, because the agent can read both
runs/<timestamp>--<label>/ manifest, ledger, answers, summary — committed
var/ subject checkouts, target masters, raw event streams, scratch — never committed
web/ the results explorer; static files over the JSON in runs/
src/ the runner

Commands

bench doctor configuration, harnesses, targets, containment
bench cases:list / cases:validate / cases:import curate the questions
bench subject:list / subject:build / subject:inspect the tools under test: what is defined, install one into the target, report what it produced
bench target:setup / target:status / target:reset the system under test
bench run a sweep; --case, --provision, --model, --target, --reps, --dry-run
bench judge grade the rows a checker could not decide
bench report claim table, per-case deltas, tool ledger; --suite merges every submission that answered the same suite, --json for the raw summary
bench suite:list the named sets of cases a run can ask
bench runs:list the measurement history
bench baseline:pin / baseline:show the reference every later run is read against
bench mcp:inspect what a server registers, and what its schemas cost — no model involved
bench tax the fixed context cost each provision adds
bench web:export / bench serve publish and browse the results

Options on bench run replace matrix axes rather than filtering the result, so a one-off question can reach a target or provision the committed matrix leaves out without editing it.

Documentation

Predecessor

This replaces the single-purpose dev-companion-benchmark, which measured one MCP server on one harness with two hard-coded arms, in bash. Its 17 tasks import with bench cases:import and are the starting case set here. Its own numbers do not carry over — that sweep never finished, and its report says so.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0