Repository navigation
ASPIRE on LIBERO (Object, Spatial, Goal, Long, LIBERO-90) - #100
Draft
ruiling-smartbear wants to merge 8 commits into
Draft
ruiling-smartbear wants to merge 8 commits into
ruiling-smartbear wants to merge 8 commits into
Conversation
examples/aspire_libero/ adds LIBERO's Object, Spatial, Goal, Long (libero_10) and LIBERO-90 suites as tasks of ASPIRE's harness (cap-harness --benchmark libero-pro): 130 task rows, run.yaml (the four ten-task suites), run_90.yaml, smoke.yaml and a launcher that starts the Robosuite example's on this folder. The Robosuite example's engine now takes the benchmark from the task row: the venv that holds cap-harness, the init mode (LIBERO: seeded, so a seed is a scene of its own), the simulator's environment and the benchmark's docs page (LIBERO's without the section that names ASPIRE's worked programs per task). A row without the field is Robosuite's, as before. prepare.py's seeds are presets (--preset aspire|smoke) and its row and split writing is shared.
…out's pages The LIBERO configs granted $ASPIRE_ROOT/docs/libero-pro.md to the task sandbox, whose "Task examples" section names ASPIRE's worked programs per task; only the workspace copy was cut. launch.sh now stages the docs outside the checkout (stage_docs.py) and the Codex configs of both examples grant those (agent.params.aspire_docs). A benchmark page without its cut is refused rather than copied whole. LIBERO's demonstration datasets are hidden from the simulator's sandbox.
smoke_luna.yaml is smoke.yaml with GPT-5.6-Luna acting and coordinating on the Codex ChatGPT login. The launcher asks for FIREWORKS_API_KEY only when the config's models are Fireworks'.
This was referenced Nov 11, 2026
…t on an Object task
…ep's init mode is tested
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
Adds LIBERO as a second benchmark for the ASPIRE method, beside Robosuite.
examples/aspire_libero/: LIBERO's five suites (libero_object,libero_spatial,libero_goal,libero_10,libero_90; 130 tasks) as tasks of ASPIRE's harness (cap-harness --benchmark libero-pro), withrun.yaml(the four ten-task suites),run_90.yaml,smoke.yaml, and alaunch.shthat starts the Robosuite example's launcher on this folder.examples/aspire_robosuite/(no core code changes): the engine, simulator service, judge and task functions take the benchmark from the task row ("benchmark": "libero-pro") and with it ASPIRE's settings for it:.venv-libero,--init-mode seededon every run,LIBERO_CONFIG_PATH, anddocs/libero-pro.mdin the actor's workspace without its "Task examples" section (it names ASPIRE's worked programs per task). A row without the field is Robosuite's, as before.prepare.pytakes--preset aspire|smoke; its row and split writing is shared with LIBERO's.launch.shstages them outside the checkout (utils/agent/stage_docs.py,agent.params.aspire_docs), not the checkout's own pages; a benchmark page without its cut is refused. LIBERO's demonstration datasets and ASPIRE'sdocs/are hidden from the simulator's sandbox.smoke_luna.yaml: the smoke with GPT-5.6-Luna on the machine's ChatGPT login; the launcher asks forFIREWORKS_API_KEYonly for a Fireworks config.Validation
python -m pytest tests/methods/aspire -qon Linux with bubblewrap: 36 passed. CI (tests.yml) passes on the head.scripts/bootstrap_libero.sh: a read-only probe program run throughsim.Engine(benchmark="libero-pro")in its sandbox onlibero_90:33,libero_object:0andlibero_10:9, seeds 51 and 60: programs complete,init_modeisseeded, the task language is read. The registry reports 10 / 10 / 10 / 10 / 90 tasks for the five suites.sim.Engine:examples/libero/close_microwave.pyonlibero_90:33, seeds 1-3: 3/3task_success;examples/libero/object_swap_seed1.pyonlibero_object:0, seeds 1-3 and 51-53: 6/6.ASPIRE_CONFIG=smoke_luna.yaml ./launch.sh, 44 minutes, 9.0M tokens): the run completes (two steps, actor, judge, evolver,finaltag). Both tasks score 0/3 at the judge on seeds 1-3 and the library is unchanged. The sessions ran only seeds 51-53 and 66; their sandbox records show enforcement on, the network isolated, and no path of the other task, the judge's runs, the simulator service or ASPIRE's examples; the judged programs call only the public API. The learner was given the finished task's record, trajectory and files.run.yamlandrun_90.yamlare scaled from Robosuite's by task count, not measured.Review
tests/methods/aspire/test_aspire.py; docs:examples/aspire_libero/README.md,examples/README.md,docs/methods/aspire.md,CHANGELOG.md.confine(hermit.sandbox). The smoke'ssandbox/rollout-<n>.jsonandsandbox/learner-<n>.jsonare as described in Validation.50754c4against the general, sandbox, test and reproduction checklists. It found no way for a session to run held-out seeds or choose its init mode, no path to the judge, the task rows, ASPIRE's examples or another session's output from the task sandbox, and no regression for Robosuite rows. Its findings, fixed ind529101: the simulator's sandbox showed the checkout's uncutdocs/(now hidden, tested); the sweep's--init-modeand the hidden datasets had no test;docs/methods/aspire.mdlisted LIBERO-Pro as not ported; a second launch rewrote the staged docs under a running one. Unchanged frommain: a generated program runs in cap-harness's process and could write its own outcome (the Robosuite README's stated limit).d529101itself has not been reviewed again.