Repository navigation
ASPIRE on LIBERO-Pro (Goal, Object, Spatial, Long: swap and task) - #101
Closed
ruiling-smartbear wants to merge 23 commits into
Closed
ruiling-smartbear wants to merge 23 commits into
ruiling-smartbear wants to merge 23 commits into
Conversation
examples/aspire_libero/ adds LIBERO's Object, Spatial, Goal, Long (libero_10) and LIBERO-90 suites as tasks of ASPIRE's harness (cap-harness --benchmark libero-pro): 130 task rows, run.yaml (the four ten-task suites), run_90.yaml, smoke.yaml and a launcher that starts the Robosuite example's on this folder. The Robosuite example's engine now takes the benchmark from the task row: the venv that holds cap-harness, the init mode (LIBERO: seeded, so a seed is a scene of its own), the simulator's environment and the benchmark's docs page (LIBERO's without the section that names ASPIRE's worked programs per task). A row without the field is Robosuite's, as before. prepare.py's seeds are presets (--preset aspire|smoke) and its row and split writing is shared.
…out's pages The LIBERO configs granted $ASPIRE_ROOT/docs/libero-pro.md to the task sandbox, whose "Task examples" section names ASPIRE's worked programs per task; only the workspace copy was cut. launch.sh now stages the docs outside the checkout (stage_docs.py) and the Codex configs of both examples grant those (agent.params.aspire_docs). A benchmark page without its cut is refused rather than copied whole. LIBERO's demonstration datasets are hidden from the simulator's sandbox.
smoke_luna.yaml is smoke.yaml with GPT-5.6-Luna acting and coordinating on the Codex ChatGPT login. The launcher asks for FIREWORKS_API_KEY only when the config's models are Fireworks'.
examples/aspire_libero_pro/ adds the _swap and _task perturbations of LIBERO's Object, Spatial, Goal and Long suites (ten tasks a suite) as tasks of ASPIRE's harness, on the engine and the LIBERO settings of examples/aspire_libero/: run.yaml (all eighty), run_long.yaml (LIBERO-Long-Pro), smoke configs and a launcher that starts the Robosuite example's on this folder.
…t on an Object task
…ep's init mode is tested
LIBERO's grasp backend is on its fixed port 8115 (ASPIRE's SERVICE_PORTS for libero-pro); without it every generate_grasps call fails. The launcher checks it as the Robosuite launcher checks the other services.
…raspNet As the LIBERO example's launcher: LIBERO's grasp backend on its fixed port 8115 (ASPIRE's SERVICE_PORTS for libero-pro).
4 of 6 tasks
# Conflicts: # tests/selection.py
…pro-sync # Conflicts: # tests/selection.py
…ing tasks, its verifiers to the tasks and its rows to this folder; the LIBERO README points to LIBERO-Pro
…s protocol; the method page names LIBERO's docs, aspire_docs and verifier seeds
…as the pairs ASPIRE's harness requires, and the perturbations left out are listed
Contributor
Author
|
Folded into #100: LIBERO is now one benchmark, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
Adds LIBERO-Pro as a benchmark for the ASPIRE method. Stacked on #100: this branch contains its commits, and the change of this PR is
examples/aspire_libero_pro/and its registrations.examples/aspire_libero_pro/: the eighty LIBERO-Pro(suite, task)pairs ASPIRE's harness requires (libero_{object,spatial,goal,10}_{swap,task}, ten tasks a suite; LIBERO-Pro's_lan,_objectand scene-variant suites are not among them) as tasks of ASPIRE's harness (cap-harness --benchmark libero-pro), withrun.yaml(all eighty),run_long.yaml(LIBERO-Long-Pro:libero_10_swap,libero_10_task),smoke.yaml,smoke_luna.yamland alaunch.shthat starts the Robosuite example's launcher on this folder..venv-libero,--init-mode seeded, ASPIRE's LIBERO seeds 51-65 / 66-80 / 1-50, the staged docs without the section that names ASPIRE's worked programs, several of which are LIBERO-Pro tasks).Validation
python -m pytest tests/methods/aspire -qon Linux with bubblewrap: 37 passed. CI (tests.yml) passes on the head.sim.Engineon this folder's task rows (seeded): a read-only probe program on task 0 of each of the eight suites, seed 51, completes and reads the perturbed task's language. With ASPIRE's SAM3 and PyRoki services, ASPIRE's ownexamples/libero/object_swap_seed1.pyon the smoke's first task,libero_object_swap-t00, seeds 1-3 and 51-53: 6/6task_success.finaltag). Both tasks score 0/3 at the judge on seeds 1-3 and the library is unchanged. The sessions ran only seeds 51-53 and 66; their sandbox records show enforcement on, the network isolated, and no path of the other task, the judge's runs, the simulator service or ASPIRE'sexamples/anddocs/; the judged programs call only the public API. The learner was given the finished task's record, trajectory and files; a next task it named outside the split was refused.libero_object_swap; no_tasksuite has been run by a session. The budgets inrun.yamlandrun_long.yamlare scaled from Robosuite's by task count, not measured.Review
tests/methods/aspire/test_aspire.py; docs:examples/aspire_libero_pro/README.md,examples/README.md,docs/methods/aspire.md,CHANGELOG.md.sandbox/rollout-<n>.jsonare as described in Validation.0f16be1adds to LIBERO (and LIBERO-Pro) as a benchmark, with ASPIRE and prompt-only #100 against the general, sandbox and test checklists. It found the four configs,prepare.py(240 rows, unique ids,benchmarkon every row, the splits the configs name, train / val / test disjoint) andlaunch.shcorrect, no grant beyond LIBERO (and LIBERO-Pro) as a benchmark, with ASPIRE and prompt-only #100's, and nothing in this folder that shows ASPIRE's worked programs. Its finding, fixed inb549e39: the test rebuilt the ids itself and passed with the rows'benchmarkdropped, the ids made to collide, or a split renamed; it now reads whatprepare.pywrites and fails on each. ASPIRE's prompt files the task sandbox is given name no worked program and no LIBERO-Pro task's strategy. A second review, ofb549e39and the merge of LIBERO (and LIBERO-Pro) as a benchmark, with ASPIRE and prompt-only #100's latest, found nothing serious: the test fails on each regression it names, and the mergedtests/selection.pykeeps main's groups. Its finding, fixed inb65aee7: two of the config checks could not fail, so a config whose test split was not its training tasks' reruns, whose val split was not the verifiers' rows, or whose rows were another folder's passed; the test now ties each to this folder's.