8000 Add Hermes Agent as a CLI Base Agent (acting agent and hermes Evolver) by veoery · Pull Request #107 · Infini-AI-Lab/hermit · GitHub
[go: up one dir, main page]

Skip to content

Add Hermes Agent as a CLI Base Agent (acting agent and hermes Evolver) - #107

Merged
haizhongzheng merged 3 commits into
Infini-AI-Lab:mainfrom
veoery:feat/hermes
Nov 16, 2026
Merged

haizhongzheng merged 3 commits into
Infini-AI-Lab:mainfrom
veoery:feat/hermes

Conversation

@veoery
@veoery veoery commented Nov 15, 2026
Copy link
Copy Markdown
Contributor

Change

Hermit could not use Nous Research's Hermes Agent (hermes, PyPI hermes-agent 0.19.0) as an acting agent or a
learner. It now can, in the same three layers as Pi (#89), OpenCode (#94) and DeepSeek Harness (#90): a Base Agent
hermes, the acting agent agent.base: hermes (agent.name: hermes), the hermes Evolver, and hermit deploy --host hermes.

The one design decision: a stream from Hermes's plugin API. hermes chat -Q prints only its final reply (no JSON
event mode). Rather than reading Hermes's internal SQLite transcript after the run, the Base Agent copies a plugin of
hermit's own (hermit/agent/base/hermes_plugin/, standard library only) into each
session's home and enables it. It writes every hook Hermes fires as one JSON line on the session's stdout, the stream
hermit keeps (agent-hermes.jsonl): requests with their text, reasoning, tool calls and usage; failed requests with
Hermes's own classification (reason, status_code, retryable); tool results; the first request's tool names and
any context file Hermes's scan blocked; the turn's end with Hermes's recorded auxiliary usage. It uses Hermes's public
plugin API, is live (a stopped session keeps what it did), and lets outcome() decide "provider unavailable" from
Hermes's classification, never from words. Readers stop at the turn's end: chat -Q prints the raw reply after it, and
a reply that reads as an event is the model's text.

  • Layer 1 (backends.py): Binding.hermes_provider() makes a <backend>/<model>
    reference Hermes's config provider hermit: api_mode chat_completions, else anthropic_messages (at the Messages
    root), else codex_responses, the first the backend serves (as Pi picks its dialect); the key by its variable's name
    (key_env), never its value; extra_body on the provider (Hermes sends it with chat-completions requests). The
    backend's context_window and max_output_tokens become the model's context_length and max_tokens. Hermes's
    cost figure is ignored: the backend's price applies.
  • Layer 2 (cli.py, class Hermes): hermes chat -Q --yolo --source tool -t <toolsets> -q <prompt> in a private home (HERMES_HOME = HOME), config.yaml rewritten every call.
    • What Hermes would reach besides the run's model is off: every auxiliary task routed to the same model, no fallback
      providers, no tirith download (46 MB), no update check (a cache of one for the installed version), no models.dev
      catalog (a fresh empty one in the home), no memory, skill-creation nudges, curator or title request. Approvals are
      never asked (Hermes's hardline blocks stay). The reply is the model's own (Hermes's verifier footer and completion
      explainer off).
    • Toolsets: terminal, file, todo, skills with a skills folder, and the call's tools as an MCP server (streamable
      HTTP, 1800 s a call, waited for up to 60 s before the first request). No web search, delegation, clarify, memory.
    • The caller's HERMES_*, TERMINAL_*, TIRITH_* variables and other providers' keys never reach a session
      (HERMES_SAFE_MODE would load no plugin, a stray key would be a fallback provider).
    • A kept home (the Evolver's) holds only Hermes's sessions (its state database), hermit's usage ledger and Hermes's
      logs between calls (HERMES_KEPT): a profile Hermes would switch its home to, a .env/.op.env it loads, its
      identity text, plugins, hooks, credentials and skills a session left are removed. A home hermit did not make (a
      marker file) is refused, never cleared.
    • context_files: false (the Evolver's): --ignore-rules, and the plugin turns off Hermes's subdirectory hints
      (which add a touched folder's AGENTS.md to a tool's result whatever --ignore-rules says) and reports whether it
      could; a session whose hints stayed on fails.
    • outcome(): served when the turn completed with a reply; unavailable only for Hermes's reasons rate_limit,
      upstream_rate_limit, overloaded, server_error, timeout (or 429/5xx) on the last failed request Hermes did not then
      serve; rejected for auth, billing, model_not_found, format_error, ... A session fails although Hermes exits 0 when
      the call's MCP tools were missing from its first request (Hermes goes on without a server it cannot reach), when
      Hermes's prompt-injection scan replaced a context file with a [BLOCKED: ...] notice (benign rules can match), or
      when it hit Hermes's turn limit (agent.hermes.max_turns, default 90: the summary Hermes then asks for is the
      reply, finish_reason length).
    • Usage (records.py parse_hermes): per request (input = input + cache reads + cache
      writes; output holds reasoning); the turn-limit summary request, which no hook reports, is an unreported call (cost
      unknown); Hermes's auxiliary calls (compression, ...) come from its state database at the turn's end, a resumed
      session counting only this call's share (_this_call ledger, as OpenCode's compactions).
    • Trajectory (trajectory.py hermes_steps): commands with exit codes, file
      changes, tool calls, the failed request the session ended with (not the ones Hermes retried).
    • Resume: a handle is resumed (--resume --no-restore-cwd) only when the home's state database, read as a private
      copy with its WAL, keeps it: Hermes fails a turn on an unknown id.
    • Refused rather than dropped: roles, a login, a container, web search, text_only, an effort Hermes has not
      (checked before the first session in isolate.validate and login.evolver_problems via hermes_problems), and a
      prompt over one argument (ARG_MAX, 120,000 bytes: chat -Q takes no prompt on stdin).
  • Layer 3:
    • Acting agent (workspace.py): Hermes reads MCP servers and skill folders only
      from its home's config, so the adapter's tools are served by the session's own tool server (as for OpenCode and
      mini-swe-agent) and the deployment's skills folder is the session's skills_dir.
    • Deployment (hosts/hermes.py): Hermes reads one project context file, the first of
      .hermes.md (searched up to the git root), AGENTS.md, CLAUDE.md, .cursorrules, so the context goes to
      .hermes.md; skills to .hermes/skills/; and .hermes/hermit.config.yaml, the part of Hermes's own config the
      tools (stdio server, 120 s) and skills need, for the user to merge (a deployment never changes ~/.hermes).
    • Evolver (methods/hermes/, AgentEvolver): signs in through its model
      reference, copies nothing of ~/.hermes, reads no context file of the evolver workspace, takes max_turns. MWM and
      RRSI already refuse a learner other than Codex, Claude Code or Pi.
    • Registries: config bases and sections, defaults.yaml, CLI_KINDS, NO_LOGIN, deploy --host, the AHE debugger's
      stream texts and a hermes hook wire (no-hooks host like Pi: hooks listed in .hermes.md), package data for the
      plugin's manifest.
  • Docs: agent-layers, adapter (deployment table column, workspace tree), commands, task, model-backends,
    config-fields, installation (uv tool install 'hermes-agent[mcp]==0.19.0'), compatibility, methods/agent, AHE, FAQ,
    CHANGELOG.

Not in this PR: evolver.base: hermes making the run's model Hermes for methods that call it (llm: {cli: ...}; only
Pi does that), a real AHE hook wire over Hermes's shell hooks, roles through Hermes's delegation, container isolation.

Validation

Hermes Agent 0.19.0 installed with uv tool install 'hermes-agent[mcp,anthropic]==0.19.0'; Linux, Python 3.12.9,
bubblewrap. No real model provider was called (no key on this machine): every Hermes run used a scripted local
chat-completions model.

  • python -m pytest tests/core/test_hermes_agent.py --integration: 34 passed (25 unit tests, ~9 s; 9 integration
    cases with the installed hermes, ~55 s: a session under isolation none and in the run's sandbox with an isolated
    network and no update check made, a resumed kept home, a timeout that leaves no command running, the acting agent
    deploying an adapter's context, skill and tools inside a git repository whose own .hermes.md stays out, no folder's
    AGENTS.md given with context_files: false (and given with true), a context file Hermes's scan blocks failing the
    session, the model's own reply after a failed patch).
  • With tests/core/test_context_tool.py, test_isolate.py, test_packaging.py, test_selection.py, test_layers.py
    and tests/methods/ahe/test_ahe_hosts.py, --integration: 164 passed.
  • python -m pytest tests/core/test_sandbox.py --integration -k "installed_cli_starts and hermes": 1 passed.
  • python -m pytest --changed: 1371 passed, 165 skipped, 6 failed. The 6 fail identically on main with this branch
    stashed (environment, not this PR): examples/clbench_database/utils/tests/test_parity_golden.py (3),
    tests/core/test_sandbox.py::test_inside_the_sandbox_the_hidden_paths_are_empty,
    tests/methods/harness2/test_harness2_on_pi.py::test_what_a_pi_session_spent_is_kept_when_it_times_out_or_is_stopped
    (2).
  • Recorded streams (tests/fixtures/hermes/, README says how): success, resume, 503
    (unavailable), 400 (rejected), turn limit, interrupted, tools down, from the real Hermes run by the Base Agent.
    Tests for cases the recordings do not hold edit one event of a recording (edited in the test file).
  • By hand: hermit deploy --host hermes --adapter docs/_examples/file_adapter, the fragment merged into a Hermes home,
    then the real hermes in the workspace: the adapter's stdio tool server answered, its skill and .hermes.md were in
    the system prompt.
  • ruff check --select F401,F841,F811 on the new and changed modules and git diff --check: clean.

Unverified: a real provider (usage, cost and Hermes's error classification on real 429/5xx); the Anthropic Messages and
Responses api_modes (only chat completions was run); a full hermit train with the hermes Evolver; Hermes releases
other than 0.19.0 (the plugin's hook payloads, the subdirectory-hint switch and the config keys are Hermes's, pinned
there; a Hermes without the hint tracker fails an Evolver session rather than give it an AGENTS.md).

Review

  • Tests and documentation cover the affected behavior, where applicable. Tests: see Validation; docs: see Change.

  • Commands, defaults, and public interfaces are consistent. hermes is added to every list that names dsh
    (config bases and sections, defaults, CLI_KINDS, NO_LOGIN, --host, the AgentEvolver's backends, stream readers
    and markers, AHE wires and texts); tests/core/test_selection.py and the sandbox CLI test enumerate it.

  • Shared logs and examples contain no credentials or private data. Fixtures use a mock key variable; workspace
    paths are written /w and the model URL 127.0.0.1:9 (README).

  • Documentation-only: not applicable.

  • Core changes. Shared code gains a new subclass and registry entries; CliAgent (the shared runner) is
    unchanged. Touched shared modules: cli.py (new class and helpers only), records.py, trajectory.py,
    backends.py, config.py, isolate.py (a settings check only), login.py, workspace.py, evolver/agent.py,
    pyproject.toml (package data).

  • Implementation correctness and failures. Failed, cut and stopped sessions are reported as such, never as
    success: exit 1 with Hermes's classification (unavailable vs rejected), the turn limit, missing tools, a blocked
    context file and hints left on although Hermes exits 0, a stop (interrupted, no orphan command), a run that wrote no
    plugin event (started: false). The audit's findings on this item are fixed (below).

  • Real execution paths and tests. Unit tests go through make("hermes").run, the acting agent's run,
    Config.load/layered and the Evolver loader, with a fake hermes that replays recorded streams; integration tests
    run the installed CLI.

  • Documentation. See Change; the claims the audit found inaccurate are corrected.

  • This description. Read back against the code at the PR's head.

  • Independent review (post-PR audit). A separate agent audited 6c76cca (before publication) against the
    general, test and sandbox checklists: it read the diff, callers, tests and Hermes 0.19.0's source, ran the suites and
    reproduced its findings with the real hermes against a scripted model. No blocker; its findings and their
    resolution:

    # Finding (audit's severity) Resolution
    1 Should-fix, reproduced: a kept home could be redirected (active_profile + profiles/), and .op.env, auth files and skills a session left survived Allowlist (HERMES_KEPT) and a marker; test plants each and checks the next call's home
    2 Should-fix, reproduced: --ignore-rules leaves Hermes's subdirectory hints on, so the Evolver was given the adapter's AGENTS.md The plugin turns them off and reports it; a session whose hints stayed on fails; unit and integration tests (both settings)
    3 Should-fix, reproduced: Hermes's prompt-injection scan can replace the whole .hermes.md silently (benign rules match) The plugin reports a blocked context file; the session fails naming it; documented for hermit deploy by hand; unit and integration tests
    4 Should-fix, reproduced: Hermes appends its verifier footer / completion explainer to the graded reply Both off in the session's config; integration test of a failed patch
    5 Should-fix: max_output_tokens on the provider is not applied; extra_body is not on every request; a models.dev fetch for known hosts model.max_tokens; docs say chat-completions requests; a fresh empty catalog in the home; docs and CHANGELOG corrected
    6 Nit: an incomplete turn with no reply was called the turn limit length and "turn limit" only with the summary as reply; test
    7 Nit, reproduced: a reply printed after the turn could be read as events (and a command can write to Hermes's stdout) Readers stop at the turn's end (hermes_events); test. Writing to /proc/<pid>/fd/1 is open to every CLI whose stdout is the stream: left as is
    8 Nit: the "failed request then served" rule was untested Test; mutation-checked
    9 Nit: an empty turn-limit summary is retried once by Hermes (2 unreported calls, counted 1) Documented in parse_hermes
    10 Nit, plausible: Hermes caps a hook's payload at 50,000 characters, which would hide the first request's tools HERMES_PLUGIN_PAYLOAD_MAX_CHARS set high in the session
    11 Nit: --resume moves to the session's old directory --no-restore-cwd
    12 Nit: context-tool and isolate tests not extended to the host Extended
    13 Nit: the caller's PYTHONPATH reaches Hermes's interpreter Documented (the Evolver needs it for its commands; no clashing names at the repository root)
    14 Nits: the plugin's database connection not closed; stale docstring Closed; docstrings corrected
    Q The bound model's key is readable by the session's commands As for every CLI Base Agent: the session needs it, the run's records are scrubbed of it

    The fixes are in the PR's second commit, which the audit did not review; they are covered by the tests above.

  • Group and selection. tests/core/test_hermes_agent.py is in core; tests/fixtures/hermes/* is in
    tests/selection.py's CORE and in tests/README.md.

  • Names / docstrings. Each test is a sentence of the behaviour; the file's docstring names the code and the
    stand-ins (a fake hermes replaying 0.19.0 recordings; the installed hermes against a scripted loopback model).

  • What it asserts. Statuses, records, files, stream fields, the config Hermes was given; error text matched only
    where it is the contract. The new behaviour has no code before this PR; the audit mutation-checked four mutations
    (one survived and now has a test).

  • Isolation. Unit tests write only to tmp_path, use no CLI, sign-in or network; integration tests are marked
    and skip without hermes, git or a working bub 8000 blewrap; the sandbox test's workspace is under
    ~/.cache/hermit-tests and removed.

  • Time and order. No fixed timestamps that expire (the update check's is in 2100 by design); no shared state.

  • Cost. Unit tests ~9 s, integration ~55 s (the stop test 18 s of it, waiting on its clock).

  • Hygiene. Lint clean (Validation); the integration tests' servers and workspaces are closed and removed, and
    the stop test asserts no command outlives the session.

  • Reuse the framework's sandbox. Hermes runs through CliAgent's isolation (none, or the run's sandbox via
    Sandbox.wrap, as Pi, OpenCode and dsh); no new process type. The audit's record of the sandbox integration run
    (agent-1.json): role agent, network isolated, probe passed (outside hosts blocked, model and tool ports forwarded),
    enforced; read-only the hermes uv-tool directory and its interpreter's base; read-write only the workspace and the
    home. No hermit train run with it exists yet (no model key).

  • Sandbox changes are core changes. The sandbox itself is unchanged (hermit/isolate.py gains only the
    hermes_problems config check, beside dsh's).

🤖 Generated with Claude Code

veoery and others added 3 commits November 14, 2026 12:36
An independent audit of 6c76cca (general, test and sandbox checklists; findings reproduced
with the real hermes 0.19.0 against a scripted model) found no blocker and these, now fixed:

- A kept home holds only Hermes's sessions, hermit's ledger and the logs between calls
  (HERMES_KEPT): a profile Hermes would switch its home to (active_profile, profiles/), a
  .env/.op.env, credentials and skills a session left are removed; a home hermit did not
  make (no marker) is refused, never cleared.
- context_files false (the Evolver's): the plugin turns off Hermes's subdirectory hints,
  which --ignore-rules leaves on (a touched folder's AGENTS.md added to a tool result), and
  reports it; a session whose hints stayed on fails.
- A context file Hermes's prompt-injection scan replaced with a [BLOCKED: ...] notice fails
  the session, naming it (benign rules match).
- The reply is the model's own: Hermes's verifier footer and completion explainer are off.
- The output limit is the model's max_tokens (Hermes ignores the provider's); a fresh empty
  models.dev catalog in the home, so no catalog request; extra_body documented as
  chat-completions only.
- Readers stop at the turn's end, so a reply printed after it is never read as events; an
  incomplete turn with no reply is no turn limit; the plugin's payload cap raised;
  --no-restore-cwd on resume; the plugin's database connection closed.
- Tests: each fix, the "failed request then served" rule, the context-tool and isolate
  checks for the host; integration tests for hints, the blocked context and the footer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rollouts count every usage: the acting agent kept only usage[0], dropping
  Hermes's auxiliary calls from cost, tokens and model calls.
- A reply cut at its output limit, which Hermes continues before its hook
  reports the request, is counted: the plugin writes hermes.request_start
  for every attempt, and a request that neither answered nor failed is an
  unreported call (cost unknown); a 400 policy rejection is a failed one.
- A failed turn's auxiliary calls are read from the home's state.db (no
  session_end reports them).
- A request after a 503 counts as recovery: a continued reply that ended at
  the turn limit is no outage to retry.
- ANTHROPIC_AUTH_TOKEN is dropped; the bound key is kept whatever its name
  (HERMES_API_KEY).
- A context file Hermes's scan blocked is reported from the scan itself, on
  the Responses and Messages APIs too.
- Docs: hermes-agent[mcp,anthropic]; max_output_tokens on the model; Hermes
  loads its own source's .env and a managed .env.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@haizhongzheng haizhongzheng left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The plugin stream is a clean way to read Hermes, and the recorded fixtures made the review much easier. We reviewed e20ebf4 with two independent reviewers over four rounds, verified every finding against the code and Hermes 0.19.0's source, and ran the real hermes against a scripted model where a case could be reproduced. The fixes are pushed to your branch as f114385; each code fix has a test that fails without it.

Usage and cost

  • Auxiliary calls in rollouts. The acting agent kept only usage[0] in RunOutput, so Hermes's auxiliary calls (a compression's summary) were left out of a rollout's cost, tokens and model calls. It now sums every usage.
  • A reply cut at its output limit. Hermes asks for the continuation before its post_api_request hook reports the request. With the real hermes, 2 requests were counted as 1, with a known cost. The plugin now writes hermes.request_start for every attempt. parse_hermes counts a request that neither answered nor failed as an unreported call, so its cost is unknown. A content filter's refusal of a 200 counts the same way; a 400 safety rejection is a failed request.
  • A failed turn. A turn that failed fires no on_session_end, so its auxiliary calls were lost. adjust_usage now reads them from a copy of the home's state.db.
  • A recovered 503. A 503, then a continued request, then the turn limit was read as an outage, and the acting agent ran it again. A request started after the error now counts as recovery, in outcome() and in hermes_steps.

Environment

  • ANTHROPIC_AUTH_TOKEN now joins the dropped keys. Under isolation: none, the Anthropic SDK of a Messages-API session sends it beside the bound key, to that backend.
  • The bound key is never dropped. For a backend named hermes, the key is HERMES_API_KEY, which the HERMES_* filter removed.

Blocked context

  • On the Responses API the system prompt is instructions, which the hook never sees. On the Messages API the hook's copy of system is cut at 8,000 characters. So a context file that Hermes's scan blocked could pass unnoticed. The plugin now records what agent.prompt_builder._scan_context_content blocks, and the scan of the system prompt is kept for resumed sessions.

Docs

  • The installation line is hermes-agent[mcp,anthropic]==0.19.0: a Messages-API backend needs the SDK, and Hermes cannot install it inside the sandbox.
  • max_output_tokens goes on the model (max_tokens), not the provider (model-backends).
  • docs/framework/task.md now notes that Hermes still loads the .env beside its own source and a machine-wide managed .env, over what Hermit passes.

Noted, not changed

  • If a compression rotates the session id and the turn then fails, that compression's usage is still missed. The same happens for a preflight compression in a resumed session. Both are rare, and the fix would re-key the usage ledger by session id.
  • An auxiliary response without usage writes no row, so a relay that leaves usage out of non-streamed responses hides that call.

Tests. python -m pytest --changed --base origin/main: 1563 passed, 121 skipped, 0 failed. tests/core/test_hermes_agent.py --integration with Hermes 0.19.0: 40 passed.

🤖 Generated with Claude Code

@haizhongzheng haizhongzheng left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with the fixes in f114385 (details in the review above). CI passed on f114385.

🤖 Generated with Claude Code

@haizhongzheng
haizhongzheng merged commit 616e818 into Infini-AI-Lab:main Nov 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0