Open starter agent framework for Terminal-Bench.
blobfish is designed so anyone can fork the repo, set their agent name to their GitHub username, and run benchmark jobs with shared org identity:
Agent: your GitHub username (for example,cchuter)Agent Org:teamblobfish.com
- The Harbor agent implementation benchmark adapter:
harbor/src/blobfish_harbor/agent.py - Prompt/
CLAUDE.mdtemplate setup:harbor/src/blobfish_harbor/templates/prompt.md.j2(full)harbor/src/blobfish_harbor/templates/prompt-slim.md.j2(slim)harbor/src/blobfish_harbor/templates/prompt-minimax.md.j2(MiniMax local-model profile)harbor/src/blobfish_harbor/templates/claude-project-default.md(default project instructions)harbor/src/blobfish_harbor/templates/claude-project-minimax.md(MiniMax project instructions)
- Scripts for agent identity scaffolding and benchmark runs
- A submission metadata helper to keep leaderboard fields consistent
Agent profiles:
blobfish_harbor:BlobfishAgent: experimental agent with hooks, rules, and skillsblobfish_harbor:BlobfishSimpleAgent: minimal Claude runner intended to stay close to the baseline agent behavior
- Fork this repository.
- Clone your fork.
- Create your local agent profile:
./scripts/new-agent.sh <your-github-username>- Install Harbor and the Blobfish adapter:
brew install uv
uv tool install terminal-bench
uv pip install --python ~/.local/share/uv/tools/harbor/bin/python -e harbor- Run a sample Terminal-Bench job:
./scripts/run-terminal-bench.sh \
--agent-name <your-github-username> \
--backend claude \
--model anthropic/claude-sonnet-4-5 \
-k 1Single-task quick example:
./run-blobfish.sh anthropic/claude-sonnet-4-5 fix-git- Run the full benchmark for leaderboard submission:
./run-blobfish.sh anthropic/claude-sonnet-4-5 --leaderboardThis runs all tasks with 5 attempts each (-k 5) and 4 concurrent trials. Your agent name defaults to your system username ($USER) and org is set to teamblobfish.com. To override:
./scripts/run-terminal-bench.sh \
--agent-name <your-github-username> \
--backend claude \
--model anthropic/claude-sonnet-4-5 \
-k 5 -n 4You can run benchmarks against a local inference server (llama-server, vllm-mlx, etc.) instead of the Anthropic API. The server must expose an OpenAI-compatible or Anthropic-compatible /v1/messages endpoint.
Set ANTHROPIC_BASE_URL to point at your local server (or proxy) and provide a dummy API key:
ANTHROPIC_BASE_URL=http://localhost:8081 \
ANTHROPIC_API_KEY=no-key \
./run-blobfish.sh minimax/minimax-m2.5 fix-gitFull leaderboard run against a local model (-n 1 since you're limited to one GPU):
ANTHROPIC_BASE_URL=http://localhost:8081 \
ANTHROPIC_API_KEY=no-key \
./scripts/run-terminal-bench.sh \
--backend claude \
--model minimax/minimax-m2.5 \
-k 5 -n 1When ANTHROPIC_BASE_URL is set, the script automatically rewrites localhost to host.docker.internal, enables Docker host networking (network_mode=host), and passes the env vars into the container.
MiniMax M2.5 runs now use a model-specific CLAUDE.md and prompt profile automatically. That tuning is limited to agent instructions; it does not modify benchmark tasks, resources, or timeout behavior.
To run the minimal baseline-style agent instead of the full Blobfish control stack:
ANTHROPIC_BASE_URL=http://localhost:8081 \
ANTHROPIC_API_KEY=no-key \
./scripts/run-terminal-bench.sh \
--agent-profile simple \
--claude-code-version 2.1.63 \
--backend claude \
--model minimax/minimax-m2.5 \
-k 5 -n 1This repo includes a sample username-mapped agent profile and class:
- Profile:
agents/cchuter/agent.env - Class:
blobfish_harbor:CchuterAgent(same behavior asBlobfishAgent)
Run it directly:
./scripts/run-terminal-bench.sh \
--agent-import-path blobfish_harbor:CchuterAgent \
--agent-name cchuter \
--backend claude \
--model anthropic/claude-sonnet-4-5 \
-k 1Prompt variants:
# Auto prompt selection (default)
./scripts/run-terminal-bench.sh --agent-name cchuter --backend claude -k 1
# Explicit full prompt
./scripts/run-terminal-bench.sh --agent-name cchuter --backend claude --prompt-variant full -k 1
# Slim prompt
./scripts/run-terminal-bench.sh --agent-name cchuter --backend claude --slim-prompt -k 1
# Explicit MiniMax prompt
./scripts/run-terminal-bench.sh --agent-name cchuter --backend claude --prompt-variant minimax --model minimax/minimax-m2.5 -k 1
# Explicit DeepSeek prompt with a tighter thinking budget
./scripts/run-terminal-bench.sh --agent-name cchuter --backend claude --prompt-variant deepseek --model deepseek-v4-flash --max-thinking-tokens 2048 -k 1
# No prompt template
./scripts/run-terminal-bench.sh --agent-name cchuter --backend claude --no-prompt -k 1
# Single task
./scripts/run-terminal-bench.sh --agent-name cchuter --backend claude -t "fix-git*" -k 1 -n 1When prompt_variant=auto, Blobfish uses the MiniMax-specific prompt + CLAUDE.md profile for any model with "minimax" in its name, the Qwen-specific profile for any model with "qwen" in its name, the DeepSeek-specific profile for any model with "deepseek" in its name, and the default profile for everything else.
Runner switches:
--agent-profile blobfish|simple--claude-code-version VERSION
--claude-code-version passes Harbor agent kwarg version, which the Claude install template uses to install a specific Claude Code CLI version.
Blobfish standardizes metadata like this:
agent_name: your GitHub usernameagent_org:teamblobfish.commodel_name: from your run (for exampleanthropic/claude-sonnet-4-5)model_org: model provider (for exampleAnthropic)
Generate a clean metadata payload from a Harbor job:
./scripts/prepare-submission.py \
--job-dir jobs/<job-id> \
--agent-name <your-github-username> \
--agent-org teamblobfish.comThis writes jobs/<job-id>/blobfish-submission.json.
Leaderboard submissions must keep the benchmark configuration unchanged:
- keep
timeout_multiplier=1.0 - do not add agent/verifier timeout overrides
- do not add CPU, memory, or storage overrides
- do not modify benchmark tasks or verifier behavior
Blobfish model-specific prompt tuning is intended to stay within those rules by changing only agent instructions. See the leaderboard docs: https://www.tbench.ai/docs/submitting-to-leaderboard and the Terminal-Bench 2.0 leaderboard validation rules: https://huggingface.co/datasets/harborframework/terminal-bench-2-leaderboard
harbor/
src/blobfish_harbor/ # Harbor adapter package
scripts/
new-agent.sh # Bootstrap local agent identity
run-terminal-bench.sh # Run Harbor against terminal-bench@2.0
prepare-submission.py # Build leaderboard metadata payload from job results
docs/
OSS_MIGRATION_PLAN.md # Migration checklist from private platform to OSS
agents/
README.md # Fork workflow for per-user agent profiles
- Project:
blobfish - Organization for submissions:
teamblobfish.com - Public website: teamblobfish.com