Code for Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning.
In long-horizon agentic RL, methods such as GiGPO compare actions at shared states, but value estimates based on a single sampled continuation remain sensitive to later decisions. ConCal (Continuation Calibration) addresses this by sharing continuation evidence within a rollout batch. It matches same-task visits by their incoming state-action-state transitions, combines shared evidence with each visit’s own transition estimate, and recursively propagates calibrated values backward to refine earlier action-value estimates. This reduces dependence on individual continuations and mitigates local credit distortions without additional rollouts.
Use separate Conda environments for ALFWorld and WebShop. Run the following commands from the repository root.
Install the training environment:
conda create -n concal-alfworld python=3.12 -y
conda activate concal-alfworld
pip install vllm==0.11.0 "ray[default]==2.50.0" \
tensordict==0.10.0 transformers==4.57.3
pip install flash-attn==2.8.3 --no-build-isolation
pip install -r requirements.txt
pip install -e . --no-deps
pip install gymnasium==0.29.1 stable-baselines3==2.6.0 alfworld==0.4.2Download ALFWorld data and prepare the trainer inputs:
export ALFWORLD_DATA="$HOME/.cache/alfworld"
alfworld-download -f
mkdir -p data/verl-agent/text
python examples/data_preprocess/prepare.py --mode text \
--local_dir ./data/verl-agent \
--train_data_size 16 --val_data_size 128Install WebShop and download the 1,000-product dataset:
conda create -n concal-webshop python=3.10 -y
conda activate concal-webshop
pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pushd agent_system/environments/env_package/webshop/webshop
bash setup.sh -d small
popdInstall the training dependencies and prepare the trainer inputs:
pip install vllm==0.8.4 "ray[default]==2.58.0" \
tensordict==0.8.3 transformers==4.51.1 typer==0.9.4
pip install flash-attn==2.7.4.post1 --no-build-isolation
pip install -r requirements.txt
pip install -e . --no-deps
mkdir -p data/verl-agent/webshop/text
python examples/data_preprocess/prepare.py --mode text \
--local_dir ./data/verl-agent/webshop \
--train_data_size 16 --val_data_size 256The generated Parquet files provide text placeholders for the trainer. Actual tasks and observations come from the environments.
Download the base model from either configured environment:
python -c "from huggingface_hub import snapshot_download; snapshot_download('Qwen/Qwen2.5-1.5B-Instruct', local_dir='models/Qwen2.5-1.5B-Instruct')"The scripts default to four GPUs and seed 123. Run wandb login before training, or set WANDB_MODE=offline.
conda activate concal-alfworld
export SKIP_CONDA_ACTIVATE=1
export ALFWORLD_DATA="$HOME/.cache/alfworld"
bash run_concal_alfworld.shconda activate concal-webshop
export SKIP_CONDA_ACTIVATE=1
bash run_concal_webshop.shTo use another seed, run SEED=456 bash run_concal_alfworld.sh or SEED=456 bash run_concal_webshop.sh.
Success rates are reported as mean ± sample standard deviation across three training runs.
Success rates are reported as mean ± sample standard deviation across three training runs.
Built on verl-agent/GiGPO, with environments from ALFWorld and WebShop.