8000 ASPIRE on LIBERO-Pro (Goal, Object, Spatial, Long: swap and task) by ruiling-smartbear · Pull Request #101 · Infini-AI-Lab/hermit · GitHub
[go: up one dir, main page]

Skip to content

ASPIRE on LIBERO-Pro (Goal, Object, Spatial, Long: swap and task) - #101

Closed
ruiling-smartbear wants to merge 23 commits into
Infini-AI-Lab:mainfrom
ruiling-smartbear:aspire-libero-pro
Closed

ruiling-smartbear wants to merge 23 commits into
Infini-AI-Lab:mainfrom
ruiling-smartbear:aspire-libero-pro

Conversation

@ruiling-smartbear
@ruiling-smartbear ruiling-smartbear commented Nov 11, 2026 •
Copy link
Copy Markdown
Contributor

Change

Adds LIBERO-Pro as a benchmark for the ASPIRE method. Stacked on #100: this branch contains its commits, and the change of this PR is examples/aspire_libero_pro/ and its registrations.

  • examples/aspire_libero_pro/: the eighty LIBERO-Pro (suite, task) pairs ASPIRE's harness requires (libero_{object,spatial,goal,10}_{swap,task}, ten tasks a suite; LIBERO-Pro's _lan, _object and scene-variant suites are not among them) as tasks of ASPIRE's harness (cap-harness --benchmark libero-pro), with run.yaml (all eighty), run_long.yaml (LIBERO-Long-Pro: libero_10_swap, libero_10_task), smoke.yaml, smoke_luna.yaml and a launch.sh that starts the Robosuite example's launcher on this folder.
  • No engine change: the engine and LIBERO settings are LIBERO (and LIBERO-Pro) as a benchmark, with ASPIRE and prompt-only #100's (.venv-libero, --init-mode seeded, ASPIRE's LIBERO seeds 51-65 / 66-80 / 1-50, the staged docs without the section that names ASPIRE's worked programs, several of which are LIBERO-Pro tasks).

Validation

  • python -m pytest tests/methods/aspire -q on Linux with bubblewrap: 37 passed. CI (tests.yml) passes on the head.
  • On a Linux host with ASPIRE at the pinned commit, through sim.Engine on this folder's task rows (seeded): a read-only probe program on task 0 of each of the eight suites, seed 51, completes and reads the perturbed task's language. With ASPIRE's SAM3 and PyRoki services, ASPIRE's own examples/libero/object_swap_seed1.py on the smoke's first task, libero_object_swap-t00, seeds 1-3 and 51-53: 6/6 task_success.
  • The smoke on that host with ASPIRE's SAM3, PyRoki, cuRobo and Contact-GraspNet services (< 8000 code class="notranslate">ASPIRE_CONFIG=smoke_luna.yaml ./launch.sh, 43 minutes, 10.3M tokens): the run completes (two steps, actor, judge, evolver, final tag). Both tasks score 0/3 at the judge on seeds 1-3 and the library is unchanged. The sessions ran only seeds 51-53 and 66; their sandbox records show enforcement on, the network isolated, and no path of the other task, the judge's runs, the simulator service or ASPIRE's examples/ and docs/; the judged programs call only the public API. The learner was given the finished task's record, trajectory and files; a next task it named outside the split was refused.
  • Not run yet: any training run. The smoke's two tasks are both libero_object_swap; no _task suite has been run by a session. The budgets in run.yaml and run_long.yaml are scaled from Robosuite's by task count, not measured.

Review

  • Tests and documentation cover the affected behavior, where applicable. Tests: tests/methods/aspire/test_aspire.py; docs: examples/aspire_libero_pro/README.md, examples/README.md, docs/methods/aspire.md, CHANGELOG.md.
  • Commands, defaults, and public interfaces are consistent.
  • Shared logs and examples contain no credentials or private data.
  • Documentation-only: not applicable.
  • Sandbox checklist: no new process and no change to what a sandbox shows beyond LIBERO (and LIBERO-Pro) as a benchmark, with ASPIRE and prompt-only #100. The smoke's sandbox/rollout-<n>.json are as described in Validation.
  • Independent review (post-PR audit): a separate agent reviewed what 0f16be1 adds to LIBERO (and LIBERO-Pro) as a benchmark, with ASPIRE and prompt-only #100 against the general, sandbox and test checklists. It found the four configs, prepare.py (240 rows, unique ids, benchmark on every row, the splits the configs name, train / val / test disjoint) and launch.sh correct, no grant beyond LIBERO (and LIBERO-Pro) as a benchmark, with ASPIRE and prompt-only #100's, and nothing in this folder that shows ASPIRE's worked programs. Its finding, fixed in b549e39: the test rebuilt the ids itself and passed with the rows' benchmark dropped, the ids made to collide, or a split renamed; it now reads what prepare.py writes and fails on each. ASPIRE's prompt files the task sandbox is given name no worked program and no LIBERO-Pro task's strategy. A second review, of b549e39 and the merge of LIBERO (and LIBERO-Pro) as a benchmark, with ASPIRE and prompt-only #100's latest, found nothing serious: the test fails on each regression it names, and the merged tests/selection.py keeps main's groups. Its finding, fixed in b65aee7: two of the config checks could not fail, so a config whose test split was not its training tasks' reruns, whose val split was not the verifiers' rows, or whose rows were another folder's passed; the test now ties each to this folder's.

examples/aspire_libero/ adds LIBERO's Object, Spatial, Goal, Long (libero_10)
and LIBERO-90 suites as tasks of ASPIRE's harness (cap-harness --benchmark
libero-pro): 130 task rows, run.yaml (the four ten-task suites), run_90.yaml,
smoke.yaml and a launcher that starts the Robosuite example's on this folder.

The Robosuite example's engine now takes the benchmark from the task row: the
venv that holds cap-harness, the init mode (LIBERO: seeded, so a seed is a
scene of its own), the simulator's environment and the benchmark's docs page
(LIBERO's without the section that names ASPIRE's worked programs per task).
A row without the field is Robosuite's, as before. prepare.py's seeds are
presets (--preset aspire|smoke) and its row and split writing is shared.
…out's pages

The LIBERO configs granted $ASPIRE_ROOT/docs/libero-pro.md to the task sandbox, whose
"Task examples" section names ASPIRE's worked programs per task; only the workspace
copy was cut. launch.sh now stages the docs outside the checkout (stage_docs.py) and
the Codex configs of both examples grant those (agent.params.aspire_docs). A benchmark
page without its cut is refused rather than copied whole. LIBERO's demonstration
datasets are hidden from the simulator's sandbox.
smoke_luna.yaml is smoke.yaml with GPT-5.6-Luna acting and coordinating on the Codex
ChatGPT login. The launcher asks for FIREWORKS_API_KEY only when the config's models
are Fireworks'.
examples/aspire_libero_pro/ adds the _swap and _task perturbations of LIBERO's
Object, Spatial, Goal and Long suites (ten tasks a suite) as tasks of ASPIRE's
harness, on the engine and the LIBERO settings of examples/aspire_libero/:
run.yaml (all eighty), run_long.yaml (LIBERO-Long-Pro), smoke configs and a
launcher that starts the Robosuite example's on this folder.
LIBERO's grasp backend is on its fixed port 8115 (ASPIRE's SERVICE_PORTS for libero-pro); without it every
generate_grasps call fails. The launcher checks it as the Robosuite launcher checks the other services.
…raspNet

As the LIBERO example's launcher: LIBERO's grasp backend on its fixed port 8115 (ASPIRE's SERVICE_PORTS for libero-pro).
# Conflicts:
#	tests/selection.py
…ing tasks, its verifiers to the tasks and its rows to this folder; the LIBERO README points to LIBERO-Pro
…s protocol; the method page names LIBERO's docs, aspire_docs and verifier seeds
…as the pairs ASPIRE's harness requires, and the perturbations left out are listed
@ruiling-smartbear
Copy link
Copy Markdown
Contributor Author

Folded into #100: LIBERO is now one benchmark, examples/libero/, whose tasks include LIBERO-Pro's eighty (suite, task) pairs. This PR's configs are there as aspire/run_pro.yaml, run_pro_long.yaml, smoke_pro.yaml and smoke_pro_luna.yaml, with the same settings on the libero_pro, libero_pro_long and smoke_pro splits. Closing this one in favour of #100.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0