Repository navigation
Expand AgentDyn TEST and use Lightning in both EvoSafe recipes - #103
Conversation
…nce only - The webpage tests only checked that the module imports, which the pinned 5353cf7 does without the patch, so the four tests failed there instead of skipping. The fixture now posts a probe page and skips unless it is kept. - The Lightning comparison ran with the learner's sandbox off, so the README, REPRODUCTION.md and the recipe page mark it as for reference only, not a reported result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Dual review of e6cc7e7 (this session and a separate Codex gpt-6.1-sol review, 2 rounds). I pushed one commit, dad93a2.
Changes in dad93a2:
| Before | After |
|---|---|
test_agentdyn_webpages.py's fixture only importorskip'd dynamic_web_client, which AgentDyn at the pinned 5353cf7 has without the patch. With that install (the one the README documented before this PR) all four tests failed rather than skipped, so pytest --changed went red. |
The fixture posts a probe page to an empty WebDatabase and skips unless the page is kept. On unpatched 5353cf7: 4 skipped (was 4 failed). With the patch applied: 4 passed. |
The README presented the Lightning comparison as the result, although the run had sandbox.enabled=false; the reproduction checklist does not count an unconfined run as a reported result. |
evosaf
8000
e/README.md, REPRODUCTION.md and docs/recipes/agentdyn.md mark it as for reference only, not a reported result. The numbers are unchanged. |
Checked, no issue: Codex rebuilt the datasets from the frozen collection, and the counts (57 / 54 / 186), both SHA-256 values in DATA.md, the recipes' search and validation IDs and the table's percentages and error totals all match. The patch applies to 5353cf7. Both recipes resolve the repeated nvidia/ model ID through Config.load.
Tests on dad93a2: python -m pytest --changed --base origin/main (core + evosafe): 953 passed, 109 skipped, 0 failed. tests/methods/evosafe: 216 passed, 9 skipped. bash docs/build.sh with docs/requirements.txt: passed.
Left for a follow-up (@andylinx):
- Nothing checks at run time that the webpage patch is applied: a run on an unpatched AgentDyn still scores the GitHub invitation tasks with the 404, and the manifest records only the constant
AGENTDYN_REVISION. The runner could refuse such an install, or record the patch in the manifest. - The README says the run was at framework revision
9c45a65, but the runner snapshotsHEADand requires its committed config, and the recipes at9c45a65still name Super 49B andall-groups-v1. Please checkframework_commitin the run'sartifacts/manifest.jsonand name that commit, or say it was a pre-squash commit on9c45a65.
🤖 Generated with Claude Code
- evolver.base: pi works: call_model takes pi, as Infini-AI-Lab#92 documents for every method. - setup() refuses a bad proposer, debugger, debugger_batch_size or gate_mode, or a missing learner model, before any TRAIN episode. - Each round must measure the whole TRAIN set (is_epoch_end) before the official selector compares it; an error before the round's records still persists the stop flag. - Case IDs follow upstream: `<id>::<condition>`, and a user task's clean case `task_<suite>-<user_task>::clean`, so upstream's clean-failure guidance and regression risks reach the learner. - Hook staging makes every file readable, so SHE's hooks start under umask 077. - CI installs --group she; run.yaml reads full-test-v1, names its llm and enables the semantic judge. - Docs: SHE in the Pi tables; the method index, shell-tool and launch statements corrected; results marked for reference only (the learner ran without Hermit's sandbox), as for Infini-AI-Lab#103. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ightning # Conflicts: # docs/recipes/agentdyn.md # examples/agentdyn/evosafe/README.md # examples/agentdyn/evosafe/REPRODUCTION.md # tests/methods/evosafe/test_agentdyn_webpages.py
* Expand AgentDyn TEST and use Nemotron 3.5 Lightning in both EvoSafe recipes * Add official SHE evolution on AgentDyn with Lightning * SHE on AgentDyn: fixes from the dual review - evolver.base: pi works: call_model takes pi, as #92 documents for every method. - setup() refuses a bad proposer, debugger, debugger_batch_size or gate_mode, or a missing learner model, before any TRAIN episode. - Each round must measure the whole TRAIN set (is_epoch_end) before the official selector compares it; an error before the round's records still persists the stop flag. - Case IDs follow upstream: `<id>::<condition>`, and a user task's clean case `task_<suite>-<user_task>::clean`, so upstream's clean-failure guidance and regression risks reach the learner. - Hook staging makes every file readable, so SHE's hooks start under umask 077. - CI installs --group she; run.yaml reads full-test-v1, names its llm and enables the semantic judge. - Docs: SHE in the Pi tables; the method index, shell-tool and launch statements corrected; results marked for reference only (the learner ran without Hermit's sandbox), as for #103. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Haizhong Zheng <hzzheng@umich.edu> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Change
Use
vllm/nvidia/nvidia/nemotron-3.5-lightningas the victim for both the Codex and Claude EvoSafe recipes. Both usefull-test-v1and keep 25 search iterations. The Claude recipe uses the 54-case validation split for candidate confirmation.Add
prepare_all_groups.py --full-testto include all frozen variants in the 18 TEST-owned groups (18 clean, 168 injected). TRAIN has 57 cases. The preparation step preserves group ownership and frozen payloads.Apply the documented patch to pinned AgentDyn revision
5353cf7so newly created invitation pages remain available and their action handlers are validated. This fixes the simulated GitHub invitation 404.Validation
The Lightning comparison uses Codex roles on 186 TEST cases. No-defense recorded 75.0% clean utility (12/16) and 32.3% ASR (50/155). EvoSafeHarness recorded 62.5% clean utility (10/16) and 0.0% observed ASR (0/150), with 2 clean and 18 injected episode errors (due to exhausting the tool-call turn limit). Rates use completed episodes.
For the squashed head
e6cc7e7, the AgentDyn tests reported 39 passed, 22 skipped. The docs build andgit diff --checkpassed. The AgentDyn run skipped tests that require optional dependencies.Original implementation checks:
Config.loadaccepts the updated recipe and generated data.Review
evosafegroup; fixtures use frozen local cases or temporary data, no model calls, network or user credentials.e6cc7e7, both recipes, split handling, tests, and result accounting. No actionable findings.