Repository navigation
Make the executing model swappable; fix results sort direction and plugin hook schema - #2
Open
ChipNChamp wants to merge 5 commits into
Open
ChipNChamp wants to merge 5 commits into
ChipNChamp wants to merge 5 commits into
Conversation
…schema
`autolab results` hardcoded a "maximize" default and had no way to see the
campaign's declared metrics.direction (it lives in the campaign YAML and is
not stored in the DB). For a minimize metric such as latency it sorted
descending and printed the WORST experiment first under a "Top N" heading,
contradicting the campaign summary's own "Best latency" line.
db.best_by_metric() already honored direction correctly — it was simply never
told the right one. --direction now defaults to None and resolves from the
campaign YAML, so an explicit flag still wins. Only the primary metric is
inferred (a campaign declares one direction, for one metric); unfiltered
queries infer only when all campaigns declaring the metric agree; maximize
remains the fallback, so existing campaigns are unaffected.
Also fixes tests/test_plugin.py, which asserted a flat [{event, command}]
hooks schema that Claude Code silently ignores — meaning the Stop hook never
fires and the research loop cannot continue past one iteration. The test was
green while asserting a format that made the loop a no-op. hooks.json already
carried the correct nested schema as an uncommitted working-tree change (not
authored by me — preserved here because the test fix depends on it).
134 tests pass (126 + 8 new).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… backend Makes autolab drivable by both agents on this machine: Claude Code via the plugin, and Hermes/OpenClaw via DeepSeek V4. DeepSeek V4 defaults thinking ON when extra_body.thinking is unset, returns reasoning_content, and then requires later turns to echo it back. AgentHarness replays the entire message history every tool round, and OpenAIAgent dropped reasoning_content entirely — so the loop would have died with HTTP 400 "reasoning_content must be passed back" immediately after its first tool call, which is the first thing a research iteration does. - AgentMessage gains reasoning_content, round-tripped by OpenAIAgent on replay for both tool-call and plain assistant turns. - extra_body.thinking is now always sent explicitly for V4+ models (either value) rather than left to the server default, plus reasoning_effort mapping (xhigh/max/ultra -> max). Detection is substring-based so routed ids such as deepseek/deepseek-v4-pro over OpenRouter behave like native ones; deepseek-v3 is excluded so the V3 wire format is untouched. Mirrors the profile in Hermes' plugins/model-providers/deepseek so both agents speak the same wire format. - New `deepseek` backend preset (deepseek-v4-pro, DEEPSEEK_API_KEY, api.deepseek.com/v1); explicit config always overrides preset defaults. - `backend: claude-code` no longer dies with "Unknown backend" under `autolab loop` — it explains that the plugin drives that path and names the CLI alternatives. The scaffold template ships this backend by default, so every new project hit it. Wire format pinned by tests/test_deepseek_wire.py using a fake client. NOT verified against the live DeepSeek API — no key is configured on this box. 157 tests pass (134 + 23 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Autolab's research architecture (campaigns, runners, metrics, journal, state) is independent of which model drives the loop, but backend selection was a hardcoded if/elif chain in cli.py plus DeepSeek-specific wire logic inside OpenAIAgent. Swapping or adding a model meant editing core files. Introduces agents/providers/: a declarative ProviderProfile (api_mode, env vars, base_url, default/fallback models, request quirks) and a registry with name/alias lookup. _create_agent now resolves a profile and picks the backend class from api_mode; OpenAIAgent takes its wire extras from the profile instead of hardcoding DeepSeek. The shape mirrors Hermes' providers.base.ProviderProfile so profiles port between the two agents. Adding a provider requires no autolab edit: a .py file calling register_provider() in ~/.autolab/providers/ or <project>/providers/ is loaded at startup, and may deliberately override a built-in. A profile that raises is skipped with a warning rather than failing the run. Quirks are matched on model id via owns_model(), not on the configured backend, so a routed model keeps its own provider's handling: deepseek + deepseek-v4-pro -> DeepSeek thinking flags openrouter + deepseek/deepseek-v4-pro -> DeepSeek thinking flags openrouter + meta-llama/llama-3-70b -> untouched Ships anthropic, deepseek, openai, openai-compatible, openrouter, ollama, and claude-code profiles, plus an `autolab providers` command listing defaults and key status. Also refreshes stale Anthropic defaults: the model was claude-sonnet-4-20250514 (deprecated, retires 2026-06-15) -> claude-opus-5, and max_tokens 4096 -> 16000, since current Claude models think by default and max_tokens caps thinking plus response text together. 182 tests pass (157 + 25 new). Wire formats are pinned against fake clients; no live provider API was called. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The claude-code backend was already the right architecture for any model-routed agent — api_mode: external, no key, no model, no wire format, because the harness owns the loop and picks the model. Only the name presumed Claude Code. Renames it to `agent-driven`, with claude-code/hermes/external as aliases, and makes it the scaffold default. Its guidance now names both harnesses and their loop-continuation mechanisms (Claude Code's Stop hook; a Hermes cron job), and says plainly that model selection lives in the harness, not in autolab.yaml. Docs now lead with the distinction that actually matters: an agent-driven project needs nothing from the provider registry, which exists only for `autolab loop`, where autolab itself owns the tool-use loop. Verified Hermes already reads the project's CLAUDE.md (hermes-agent/agent/coding_context.py:86 _CONTEXT_FILES), so no project-file work is needed for it to pick up the research guide. 182 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dark-mode tree work (2509a72) changed the best-path green to #1B5E20 and the global-best border to penwidth=5, but test_render_dot still asserted the old #2E7D32 / penwidth=3. main has been red since that commit. Not authored by me — this fix was sitting uncommitted in the shared working tree; preserved here so this branch lands green rather than inheriting the existing failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five commits, split into two independent groups. All 182 tests pass;
mainis currently red (see the last commit).Bug fixes
autolab resultsignored the campaign's declared metric direction. It hardcoded amaximizedefault and had no way to seemetrics.direction, which lives in the campaign YAML and isn't stored in the DB — so for a minimize metric like latency it sorted descending and printed the worst run first under a "Top N" heading, contradicting the campaign summary's own "Best latency" line.db.best_by_metric()was honoring direction correctly all along; it was simply never told the right one.--directionnow defaults toNoneand resolves from the campaign YAML, so an explicit flag still wins. Only the primary metric is inferred; unfiltered queries infer only when all campaigns declaring the metric agree;maximizeremains the fallback.The plugin hook schema was wrong, and the test asserted the wrong shape.
hooks.jsonused a flat[{event, command}]list, which Claude Code silently ignores — meaning the Stop hook never fires and the Ralph Wiggum research loop cannot continue past one iteration. The test was green while asserting a format that made the loop a no-op. Both now match Claude Code's real nested schema.Swappable models
Backend selection was a hardcoded
if/elifchain incli.py, andbackend: claude-code— the value the scaffold ships by default — died withUnknown backendunderautolab loop, so every new project hit it.Adds
agents/providers/: a declarativeProviderProfile(api_mode, env vars, base_url, default/fallback models, request quirks) plus a registry with name/alias lookup._create_agentresolves a profile and picks the backend class fromapi_mode.The distinction the docs now lead with:
RESEARCH_PROMPT.mdand calls theautolabCLIautolab loopautolab.yamlclaude-codewas already the right architecture for any model-routed agent (api_mode: external, no key/model/wire format); only the name presumed Claude Code. It's nowagent-driven, withclaude-code/hermes/externalas aliases, and it's the scaffold default.Adding a provider needs no core edit: a
.pyfile callingregister_provider()in~/.autolab/providers/or<project>/providers/is loaded at startup and may deliberately override a built-in. A profile that raises is skipped with a warning rather than failing the run. Newautolab providerscommand lists backends, defaults, and key status.DeepSeek V4 support. V4 defaults thinking ON when
extra_body.thinkingis unset, returnsreasoning_content, then requires later turns to echo it back.AgentHarness.run_iterationreplays the entire message history every tool round andOpenAIAgentdropped the field entirely — so the loop would have died with HTTP 400reasoning_content must be passed backimmediately after its first tool call, which is the first thing a research iteration does.AgentMessagenow round-tripsreasoning_content, and the thinking flag is always sent explicitly.Quirks match on model id via
owns_model(), not on the configured backend, so a routed model keeps its own provider's handling:Keyed to the backend name instead, DeepSeek V4 over a router would 400 on the first tool call while the native path worked.
Refreshes stale Anthropic defaults: model was
claude-sonnet-4-20250514(deprecated, retires 2026-06-15) →claude-opus-5, andmax_tokens4096 → 16000, since current Claude models think by default andmax_tokenscaps thinking plus response text together.mainis redtest_render_dothas been failing since 2509a72, which changed the best-path green to#1B5E20and the global-best border topenwidth=5without updating the test. The last commit fixes it so this branch lands green.Not verified
No live provider API was called — there are no API keys on the machine this was written on. The 33 new wire-format tests use fake clients: they prove autolab sends what each provider's spec documents, not that the provider accepts it. First real DeepSeek run is the actual test.
🤖 Generated with Claude Code