Repository navigation
Add Hermes Agent as a CLI Base Agent (acting agent and hermes Evolver) - #107
Conversation
An independent audit of 6c76cca (general, test and sandbox checklists; findings reproduced with the real hermes 0.19.0 against a scripted model) found no blocker and these, now fixed: - A kept home holds only Hermes's sessions, hermit's ledger and the logs between calls (HERMES_KEPT): a profile Hermes would switch its home to (active_profile, profiles/), a .env/.op.env, credentials and skills a session left are removed; a home hermit did not make (no marker) is refused, never cleared. - context_files false (the Evolver's): the plugin turns off Hermes's subdirectory hints, which --ignore-rules leaves on (a touched folder's AGENTS.md added to a tool result), and reports it; a session whose hints stayed on fails. - A context file Hermes's prompt-injection scan replaced with a [BLOCKED: ...] notice fails the session, naming it (benign rules match). - The reply is the model's own: Hermes's verifier footer and completion explainer are off. - The output limit is the model's max_tokens (Hermes ignores the provider's); a fresh empty models.dev catalog in the home, so no catalog request; extra_body documented as chat-completions only. - Readers stop at the turn's end, so a reply printed after it is never read as events; an incomplete turn with no reply is no turn limit; the plugin's payload cap raised; --no-restore-cwd on resume; the plugin's database connection closed. - Tests: each fix, the "failed request then served" rule, the context-tool and isolate checks for the host; integration tests for hints, the blocked context and the footer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rollouts count every usage: the acting agent kept only usage[0], dropping Hermes's auxiliary calls from cost, tokens and model calls. - A reply cut at its output limit, which Hermes continues before its hook reports the request, is counted: the plugin writes hermes.request_start for every attempt, and a request that neither answered nor failed is an unreported call (cost unknown); a 400 policy rejection is a failed one. - A failed turn's auxiliary calls are read from the home's state.db (no session_end reports them). - A request after a 503 counts as recovery: a continued reply that ended at the turn limit is no outage to retry. - ANTHROPIC_AUTH_TOKEN is dropped; the bound key is kept whatever its name (HERMES_API_KEY). - A context file Hermes's scan blocked is reported from the scan itself, on the Responses and Messages APIs too. - Docs: hermes-agent[mcp,anthropic]; max_output_tokens on the model; Hermes loads its own source's .env and a managed .env. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for this. The plugin stream is a clean way to read Hermes, and the recorded fixtures made the review much easier. We reviewed e20ebf4 with two independent reviewers over four rounds, verified every finding against the code and Hermes 0.19.0's source, and ran the real hermes against a scripted model where a case could be reproduced. The fixes are pushed to your branch as f114385; each code fix has a test that fails without it.
Usage and cost
- Auxiliary calls in rollouts. The acting agent kept only
usage[0]inRunOutput, so Hermes's auxiliary calls (a compression's summary) were left out of a rollout's cost, tokens and model calls. It now sums every usage. - A reply cut at its output limit. Hermes asks for the continuation before its
post_api_requesthook reports the request. With the realhermes, 2 requests were counted as 1, with a known cost. The plugin now writeshermes.request_startfor every attempt.parse_hermescounts a request that neither answered nor failed as an unreported call, so its cost is unknown. A content filter's refusal of a 200 counts the same way; a 400 safety rejection is a failed request. - A failed turn. A turn that failed fires no
on_session_end, so its auxiliary calls were lost.adjust_usagenow reads them from a copy of the home'sstate.db. - A recovered 503. A 503, then a continued request, then the turn limit was read as an outage, and the acting agent ran it again. A request started after the error now counts as recovery, in
outcome()and inhermes_steps.
Environment
ANTHROPIC_AUTH_TOKENnow joins the dropped keys. Underisolation: none, the Anthropic SDK of a Messages-API session sends it beside the bound key, to that backend.- The bound key is never dropped. For a backend named
hermes, the key isHERMES_API_KEY, which theHERMES_*filter removed.
Blocked context
- On the Responses API the system prompt is
instructions, which the hook never sees. On the Messages API the hook's copy ofsystemis cut at 8,000 characters. So a context file that Hermes's scan blocked could pass unnoticed. The plugin now records whatagent.prompt_builder._scan_context_contentblocks, and the scan of the system prompt is kept for resumed sessions.
Docs
- The installation line is
hermes-agent[mcp,anthropic]==0.19.0: a Messages-API backend needs the SDK, and Hermes cannot install it inside the sandbox. max_output_tokensgoes on the model (max_tokens), not the provider (model-backends).docs/framework/task.mdnow notes that Hermes still loads the.envbeside its own source and a machine-wide managed.env, over what Hermit passes.
Noted, not changed
- If a compression rotates the session id and the turn then fails, that compression's usage is still missed. The same happens for a preflight compression in a resumed session. Both are rare, and the fix would re-key the usage ledger by session id.
- An auxiliary response without usage writes no row, so a relay that leaves usage out of non-streamed responses hides that call.
Tests. python -m pytest --changed --base origin/main: 1563 passed, 121 skipped, 0 failed. tests/core/test_hermes_agent.py --integration with Hermes 0.19.0: 40 passed.
🤖 Generated with Claude Code
There was a problem hiding this comment.
Approving with the fixes in f114385 (details in the review above). CI passed on f114385.
🤖 Generated with Claude Code
Change
Hermit could not use Nous Research's Hermes Agent (
hermes, PyPIhermes-agent0.19.0) as an acting agent or alearner. It now can, in the same three layers as Pi (#89), OpenCode (#94) and DeepSeek Harness (#90): a Base Agent
hermes, the acting agentagent.base: hermes(agent.name: hermes), thehermesEvolver, andhermit deploy --host hermes.The one design decision: a stream from Hermes's plugin API.
hermes chat -Qprints only its final reply (no JSONevent mode). Rather than reading Hermes's internal SQLite transcript after the run, the Base Agent copies a plugin of
hermit's own (hermit/agent/base/hermes_plugin/, standard library only) into each
session's home and enables it. It writes every hook Hermes fires as one JSON line on the session's stdout, the stream
hermit keeps (
agent-hermes.jsonl): requests with their text, reasoning, tool calls and usage; failed requests withHermes's own classification (
reason,status_code,retryable); tool results; the first request's tool names andany context file Hermes's scan blocked; the turn's end with Hermes's recorded auxiliary usage. It uses Hermes's public
plugin API, is live (a stopped session keeps what it did), and lets
outcome()decide "provider unavailable" fromHermes's classification, never from words. Readers stop at the turn's end:
chat -Qprints the raw reply after it, anda reply that reads as an event is the model's text.
Binding.hermes_provider()makes a<backend>/<model>reference Hermes's config provider
hermit:api_modechat_completions, else anthropic_messages (at the Messagesroot), else codex_responses, the first the backend serves (as Pi picks its dialect); the key by its variable's name
(
key_env), never its value;extra_bodyon the provider (Hermes sends it with chat-completions requests). Thebackend's
context_windowandmax_output_tokensbecome the model'scontext_lengthandmax_tokens. Hermes'scost figure is ignored: the backend's price applies.
Hermes):hermes chat -Q --yolo --source tool -t <toolsets> -q <prompt>in a private home (HERMES_HOME = HOME), config.yaml rewritten every call.providers, no tirith download (46 MB), no update check (a cache of one for the installed version), no models.dev
catalog (a fresh empty one in the home), no memory, skill-creation nudges, curator or title request. Approvals are
never asked (Hermes's hardline blocks stay). The reply is the model's own (Hermes's verifier footer and completion
explainer off).
skillswith a skills folder, and the call's tools as an MCP server (streamableHTTP, 1800 s a call, waited for up to 60 s before the first request). No web search, delegation, clarify, memory.
HERMES_*,TERMINAL_*,TIRITH_*variables and other providers' keys never reach a session(
HERMES_SAFE_MODEwould load no plugin, a stray key would be a fallback provider).logs between calls (
HERMES_KEPT): a profile Hermes would switch its home to, a.env/.op.envit loads, itsidentity text, plugins, hooks, credentials and skills a session left are removed. A home hermit did not make (a
marker file) is refused, never cleared.
context_files: false(the Evolver's):--ignore-rules, and the plugin turns off Hermes's subdirectory hints(which add a touched folder's
AGENTS.mdto a tool's result whatever--ignore-rulessays) and reports whether itcould; a session whose hints stayed on fails.
outcome(): served when the turn completed with a reply;unavailableonly for Hermes's reasons rate_limit,upstream_rate_limit, overloaded, server_error, timeout (or 429/5xx) on the last failed request Hermes did not then
serve;
rejectedfor auth, billing, model_not_found, format_error, ... A session fails although Hermes exits 0 whenthe call's MCP tools were missing from its first request (Hermes goes on without a server it cannot reach), when
Hermes's prompt-injection scan replaced a context file with a
[BLOCKED: ...]notice (benign rules can match), orwhen it hit Hermes's turn limit (
agent.hermes.max_turns, default 90: the summary Hermes then asks for is thereply, finish_reason
length).parse_hermes): per request (input = input + cache reads + cachewrites; output holds reasoning); the turn-limit summary request, which no hook reports, is an unreported call (cost
unknown); Hermes's auxiliary calls (compression, ...) come from its state database at the turn's end, a resumed
session counting only this call's share (
_this_callledger, as OpenCode's compactions).hermes_steps): commands with exit codes, filechanges, tool calls, the failed request the session ended with (not the ones Hermes retried).
--resume --no-restore-cwd) only when the home's state database, read as a privatecopy with its WAL, keeps it: Hermes fails a turn on an unknown id.
text_only, an effort Hermes has not(checked before the first session in
isolate.validateandlogin.evolver_problemsviahermes_problems), and aprompt over one argument (
ARG_MAX, 120,000 bytes:chat -Qtakes no prompt on stdin).from its home's config, so the adapter's tools are served by the session's own tool server (as for OpenCode and
mini-swe-agent) and the deployment's skills folder is the session's
skills_dir..hermes.md(searched up to the git root), AGENTS.md, CLAUDE.md, .cursorrules, so the context goes to.hermes.md; skills to.hermes/skills/; and.hermes/hermit.config.yaml, the part of Hermes's own config thetools (stdio server, 120 s) and skills need, for the user to merge (a deployment never changes
~/.hermes).AgentEvolver): signs in through its modelreference, copies nothing of
~/.hermes, reads no context file of the evolver workspace, takesmax_turns. MWM andRRSI already refuse a learner other than Codex, Claude Code or Pi.
defaults.yaml,CLI_KINDS,NO_LOGIN,deploy --host, the AHE debugger'sstream texts and a
hermeshook wire (no-hooks host like Pi: hooks listed in.hermes.md), package data for theplugin's manifest.
config-fields, installation (
uv tool install 'hermes-agent[mcp]==0.19.0'), compatibility, methods/agent, AHE, FAQ,CHANGELOG.
Not in this PR:
evolver.base: hermesmaking the run's model Hermes for methods that call it (llm: {cli: ...}; onlyPi does that), a real AHE hook wire over Hermes's shell hooks, roles through Hermes's delegation, container isolation.
Validation
Hermes Agent 0.19.0 installed with
uv tool install 'hermes-agent[mcp,anthropic]==0.19.0'; Linux, Python 3.12.9,bubblewrap. No real model provider was called (no key on this machine): every Hermes run used a scripted local
chat-completions model.
python -m pytest tests/core/test_hermes_agent.py --integration: 34 passed (25 unit tests, ~9 s; 9 integrationcases with the installed
hermes, ~55 s: a session under isolation none and in the run's sandbox with an isolatednetwork and no update check made, a resumed kept home, a timeout that leaves no command running, the acting agent
deploying an adapter's context, skill and tools inside a git repository whose own
.hermes.mdstays out, no folder'sAGENTS.mdgiven withcontext_files: false(and given withtrue), a context file Hermes's scan blocks failing thesession, the model's own reply after a failed patch).
tests/core/test_context_tool.py,test_isolate.py,test_packaging.py,test_selection.py,test_layers.pyand
tests/methods/ahe/test_ahe_hosts.py,--integration: 164 passed.python -m pytest tests/core/test_sandbox.py --integration -k "installed_cli_starts and hermes": 1 passed.python -m pytest --changed: 1371 passed, 165 skipped, 6 failed. The 6 fail identically onmainwith this branchstashed (environment, not this PR):
examples/clbench_database/utils/tests/test_parity_golden.py(3),tests/core/test_sandbox.py::test_inside_the_sandbox_the_hidden_paths_are_empty,tests/methods/harness2/test_harness2_on_pi.py::test_what_a_pi_session_spent_is_kept_when_it_times_out_or_is_stopped(2).
(unavailable), 400 (rejected), turn limit, interrupted, tools down, from the real Hermes run by the Base Agent.
Tests for cases the recordings do not hold edit one event of a recording (
editedin the test file).hermit deploy --host hermes --adapter docs/_examples/file_adapter, the fragment merged into a Hermes home,then the real
hermesin the workspace: the adapter's stdio tool server answered, its skill and.hermes.mdwere inthe system prompt.
ruff check --select F401,F841,F811on the new and changed modules andgit diff --check: clean.Unverified: a real provider (usage, cost and Hermes's error classification on real 429/5xx); the Anthropic Messages and
Responses
api_modes (only chat completions was run); a fullhermit trainwith thehermesEvolver; Hermes releasesother than 0.19.0 (the plugin's hook payloads, the subdirectory-hint switch and the config keys are Hermes's, pinned
there; a Hermes without the hint tracker fails an Evolver session rather than give it an
AGENTS.md).Review
Tests and documentation cover the affected behavior, where applicable. Tests: see Validation; docs: see Change.
Commands, defaults, and public interfaces are consistent.
hermesis added to every list that namesdsh(config bases and sections, defaults,
CLI_KINDS,NO_LOGIN,--host, the AgentEvolver's backends, stream readersand markers, AHE wires and texts);
tests/core/test_selection.pyand the sandbox CLI test enumerate it.Shared logs and examples contain no credentials or private data. Fixtures use a mock key variable; workspace
paths are written
/wand the model URL127.0.0.1:9(README).Documentation-only: not applicable.
Core changes. Shared code gains a new subclass and registry entries;
CliAgent(the shared runner) isunchanged. Touched shared modules:
cli.py(new class and helpers only),records.py,trajectory.py,backends.py,config.py,isolate.py(a settings check only),login.py,workspace.py,evolver/agent.py,pyproject.toml(package data).Implementation correctness and failures. Failed, cut and stopped sessions are reported as such, never as
success: exit 1 with Hermes's classification (unavailable vs rejected), the turn limit, missing tools, a blocked
context file and hints left on although Hermes exits 0, a stop (interrupted, no orphan command), a run that wrote no
plugin event (
started: false). The audit's findings on this item are fixed (below).Real execution paths and tests. Unit tests go through
make("hermes").run, the acting agent'srun,Config.load/layeredand the Evolver loader, with a fakehermesthat replays recorded streams; integration testsrun the installed CLI.
Documentation. See Change; the claims the audit found inaccurate are corrected.
This description. Read back against the code at the PR's head.
Independent review (post-PR audit). A separate agent audited
6c76cca(before publication) against thegeneral, test and sandbox checklists: it read the diff, callers, tests and Hermes 0.19.0's source, ran the suites and
reproduced its findings with the real
hermesagainst a scripted model. No blocker; its findings and theirresolution:
active_profile+profiles/), and.op.env, auth files and skills a session left survivedHERMES_KEPT) and a marker; test plants each and checks the next call's home--ignore-rulesleaves Hermes's subdirectory hints on, so the Evolver was given the adapter'sAGENTS.md.hermes.mdsilently (benign rules match)hermit deployby hand; unit and integration testsmax_output_tokenson the provider is not applied;extra_bodyis not on every request; a models.dev fetch for known hostsmodel.max_tokens; docs say chat-completions requests; a fresh empty catalog in the home; docs and CHANGELOG correctedlengthand "turn limit" only with the summary as reply; testhermes_events); test. Writing to/proc/<pid>/fd/1is open to every CLI whose stdout is the stream: left as isparse_hermesHERMES_PLUGIN_PAYLOAD_MAX_CHARSset high in the session--resumemoves to the session's old directory--no-restore-cwdThe fixes are in the PR's second commit, which the audit did not review; they are covered by the tests above.
Group and selection.
tests/core/test_hermes_agent.pyis in core;tests/fixtures/hermes/*is intests/selection.py's CORE and in tests/README.md.Names / docstrings. Each test is a sentence of the behaviour; the file's docstring names the code and the
stand-ins (a fake
hermesreplaying 0.19.0 recordings; the installedhermesagainst a scripted loopback model).What it asserts. Statuses, records, files, stream fields, the config Hermes was given; error text matched only
where it is the contract. The new behaviour has no code before this PR; the audit mutation-checked four mutations
(one survived and now has a test).
Isolation. Unit tests write only to
tmp_path, use no CLI, sign-in or network; integration tests are markedand skip without
hermes,gitor a working bub 8000 blewrap; the sandbox test's workspace is under~/.cache/hermit-testsand removed.Time and order. No fixed timestamps that expire (the update check's is in 2100 by design); no shared state.
Cost. Unit tests ~9 s, integration ~55 s (the stop test 18 s of it, waiting on its clock).
Hygiene. Lint clean (Validation); the integration tests' servers and workspaces are closed and removed, and
the stop test asserts no command outlives the session.
Reuse the framework's sandbox. Hermes runs through
CliAgent's isolation (none, or the run's sandbox viaSandbox.wrap, as Pi, OpenCode and dsh); no new process type. The audit's record of the sandbox integration run(
agent-1.json): role agent, network isolated, probe passed (outside hosts blocked, model and tool ports forwarded),enforced; read-only the hermes uv-tool directory and its interpreter's base; read-write only the workspace and the
home. No
hermit trainrun with it exists yet (no model key).Sandbox changes are core changes. The sandbox itself is unchanged (
hermit/isolate.pygains only thehermes_problemsconfig check, beside dsh's).🤖 Generated with Claude Code