8000 Expand AgentDyn TEST and use Lightning in both EvoSafe recipes by andylinx · Pull Request #103 · Infini-AI-Lab/hermit · GitHub
[go: up one dir, main page]

Skip to content

Expand AgentDyn TEST and use Lightning in both EvoSafe recipes - #103

Merged
haizhongzheng merged 2 commits into
Infini-AI-Lab:mainfrom
andylinx:fix/agentdyn-test-lightning
Nov 13, 2026
Merged

haizhongzheng merged 2 commits into
Infini-AI-Lab:mainfrom
andylinx:fix/agentdyn-test-lightning

Conversation

@andylinx
@andylinx andylinx commented Nov 11, 2026 •
Copy link
Copy Markdown
Contributor

Change

Use vllm/nvidia/nvidia/nemotron-3.5-lightning as the victim for both the Codex and Claude EvoSafe recipes. Both use full-test-v1 and keep 25 search iterations. The Claude recipe uses the 54-case validation split for candidate confirmation.

Add prepare_all_groups.py --full-test to include all frozen variants in the 18 TEST-owned groups (18 clean, 168 injected). TRAIN has 57 cases. The preparation step preserves group ownership and frozen payloads.

Apply the documented patch to pinned AgentDyn revision 5353cf7 so newly created invitation pages remain available and their action handlers are validated. This fixes the simulated GitHub invitation 404.

Validation

The Lightning comparison uses Codex roles on 186 TEST cases. No-defense recorded 75.0% clean utility (12/16) and 32.3% ASR (50/155). EvoSafeHarness recorded 62.5% clean utility (10/16) and 0.0% observed ASR (0/150), with 2 clean and 18 injected episode errors (due to exhausting the tool-call turn limit). Rates use completed episodes.

For the squashed head e6cc7e7, the AgentDyn tests reported 39 passed, 22 skipped. The docs build and git diff --check passed. The AgentDyn run skipped tests that require optional dependencies.

Original implementation checks:

  • Five new offline regressions pass (~1.6 seconds). All four native webpage tests fail on unpatched AgentDyn and pass with the patch; invitation tasks 5 and 7 pass native utility judging. Native tests require the optional patched AgentDyn dependency and otherwise skip.
  • Actual preparation CLI: 57 TRAIN / 54 validation / 186 TEST; TRAIN and validation ID files are byte-identical to the original protocol. Existing rows and payloads are unchanged. The 186 TEST IDs and task fields match the previously used expanded experiment set. Config.load accepts the updated recipe and generated data.
  • Patch application was checked against the pristine pinned dependency. This update uses the recorded Lightning experiment and does not rerun it.

Review

  • Tests and documentation cover the affected behavior, where applicable. Tests are in the selected evosafe group; fixtures use frozen local cases or temporary data, no model calls, network or user credentials.
  • Commands, defaults, and public interfaces are consistent. Both recipes use the documented victim and dataset paths. Each keeps its candidate selection settings.
  • Shared logs and examples contain no credentials or private data. Native test accounts are public benchmark fixtures; generated artifacts are not committed.
  • Documentation-only: links the issue where it was discussed with the core team. Not applicable: this changes code, configuration and tests.
  • Core and sandbox changes: none. Existing runtime boundaries are unchanged.
  • Independent review (post-PR audit): reviewed e6cc7e7, both recipes, split handling, tests, and result accounting. No actionable findings.

…nce only

- The webpage tests only checked that the module imports, which the pinned
  5353cf7 does without the patch, so the four tests failed there instead of
  skipping. The fixture now posts a probe page and skips unless it is kept.
- The Lightning comparison ran with the learner's sandbox off, so the README,
  REPRODUCTION.md and the recipe page mark it as for reference only, not a
  reported result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@haizhongzheng haizhongzheng left a comment
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dual review of e6cc7e7 (this session and a separate Codex gpt-6.1-sol review, 2 rounds). I pushed one commit, dad93a2.

Changes in dad93a2:

Before After
test_agentdyn_webpages.py's fixture only importorskip'd dynamic_web_client, which AgentDyn at the pinned 5353cf7 has without the patch. With that install (the one the README documented before this PR) all four tests failed rather than skipped, so pytest --changed went red. The fixture posts a probe page to an empty WebDatabase and skips unless the page is kept. On unpatched 5353cf7: 4 skipped (was 4 failed). With the patch applied: 4 passed.
The README presented the Lightning comparison as the result, although the run had sandbox.enabled=false; the reproduction checklist does not count an unconfined run as a reported result. evosaf 8000 e/README.md, REPRODUCTION.md and docs/recipes/agentdyn.md mark it as for reference only, not a reported result. The numbers are unchanged.

Checked, no issue: Codex rebuilt the datasets from the frozen collection, and the counts (57 / 54 / 186), both SHA-256 values in DATA.md, the recipes' search and validation IDs and the table's percentages and error totals all match. The patch applies to 5353cf7. Both recipes resolve the repeated nvidia/ model ID through Config.load.

Tests on dad93a2: python -m pytest --changed --base origin/main (core + evosafe): 953 passed, 109 skipped, 0 failed. tests/methods/evosafe: 216 passed, 9 skipped. bash docs/build.sh with docs/requirements.txt: passed.

Left for a follow-up (@andylinx):

  • Nothing checks at run time that the webpage patch is applied: a run on an unpatched AgentDyn still scores the GitHub invitation tasks with the 404, and the manifest records only the constant AGENTDYN_REVISION. The runner could refuse such an install, or record the patch in the manifest.
  • The README says the run was at framework revision 9c45a65, but the runner snapshots HEAD and requires its committed config, and the recipes at 9c45a65 still name Super 49B and all-groups-v1. Please check framework_commit in the run's artifacts/manifest.json and name that commit, or say it was a pre-squash commit on 9c45a65.

🤖 Generated with Claude Code

@haizhongzheng
haizhongzheng merged commit ad64ea7 into Infini-AI-Lab:main Nov 13, 2026
1 check passed
haizhongzheng added a commit to andylinx/hermit that referenced this pull request Nov 13, 2026
- evolver.base: pi works: call_model takes pi, as Infini-AI-Lab#92 documents for every method.
- setup() refuses a bad proposer, debugger, debugger_batch_size or gate_mode, or a
  missing learner model, before any TRAIN episode.
- Each round must measure the whole TRAIN set (is_epoch_end) before the official
  selector compares it; an error before the round's records still persists the stop flag.
- Case IDs follow upstream: `<id>::<condition>`, and a user task's clean case
  `task_<suite>-<user_task>::clean`, so upstream's clean-failure guidance and
  regression risks reach the learner.
- Hook staging makes every file readable, so SHE's hooks start under umask 077.
- CI installs --group she; run.yaml reads full-test-v1, names its llm and enables
  the semantic judge.
- Docs: SHE in the Pi tables; the method index, shell-tool and launch statements
  corrected; results marked for reference only (the learner ran without Hermit's
  sandbox), as for Infini-AI-Lab#103.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
haizhongzheng added a commit to andylinx/hermit that referenced this pull request Nov 13, 2026
…ightning

# Conflicts:
#	docs/recipes/agentdyn.md
#	examples/agentdyn/evosafe/README.md
#	examples/agentdyn/evosafe/REPRODUCTION.md
#	tests/methods/evosafe/test_agentdyn_webpages.py
haizhongzheng added a commit that referenced this pull request Nov 13, 2026
* Expand AgentDyn TEST and use Nemotron 3.5 Lightning in both EvoSafe recipes

* Add official SHE evolution on AgentDyn with Lightning

* SHE on AgentDyn: fixes from the dual review

- evolver.base: pi works: call_model takes pi, as #92 documents for every method.
- setup() refuses a bad proposer, debugger, debugger_batch_size or gate_mode, or a
  missing learner model, before any TRAIN episode.
- Each round must measure the whole TRAIN set (is_epoch_end) before the official
  selector compares it; an error before the round's records still persists the stop flag.
- Case IDs follow upstream: `<id>::<condition>`, and a user task's clean case
  `task_<suite>-<user_task>::clean`, so upstream's clean-failure guidance and
  regression risks reach the learner.
- Hook staging makes every file readable, so SHE's hooks start under umask 077.
- CI installs --group she; run.yaml reads full-test-v1, names its llm and enables
  the semantic judge.
- Docs: SHE in the Pi tables; the method index, shell-tool and launch statements
  corrected; results marked for reference only (the learner ran without Hermit's
  sandbox), as for #103.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Haizhong Zheng <hzzheng@umich.edu>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0