8000 Migrate RL environments to Gymnasium by GeminiLight · Pull Request #53 · GeminiLight/virne · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:

- name: Test installed package
working-directory: /tmp
run: python -c "import torch, torch_geometric, virne; assert torch.__version__.startswith('2.11.'); assert torch_geometric.__version__ == '2.8.0.post1'"
run: python -c "import importlib.util, gymnasium, torch, torch_geometric, virne; assert gymnasium.__version__ == '1.3.0'; assert torch.__version__.startswith('2.11.'); assert torch_geometric.__version__ == '2.8.0.post1'; assert importlib.util.find_spec('gym') is None"

- name: Run all tests
run: |
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Virne offers a unified and comprehensive framework for NFV-RA, with the followin

* 1️⃣ **Highly Customizable Simulations**: Simulates diverse network environments (e.g., cloud, edge, 5G), with user-defined topologies, resources, and service requirements.
* 2️⃣ **Extensive Algorithm Suite**: Registers exact, heuristic, meta-heuristic, and learning-based solvers behind a common interface.
* 3️⃣ **Reinforcement Learning Support**: Provides standardized RL pipelines and Gym-style environments for rapid development and benchmarking of RL-based solutions.
* 3️⃣ **Reinforcement Learning Support**: Provides standardized RL pipelines and Gymnasium-compatible environments for rapid development and benchmarking of RL-based solutions.
* 4️⃣ **In-depth Evaluation Aspects**: Enables insightful analysis beyond effectiveness, covering multiple practicality perspectives (e.g., solvability, generalization, and scalability).

> [!IMPORTANT]
Expand Down
2 changes: 1 addition & 1 deletion docs/source/_ext/solver_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
MOCK_IMPORTS = [
'colorama',
'colorlog',
'gym',
'gymnasium',
'matplotlib',
'networkx',
'ortools',
Expand Down
6 changes: 3 additions & 3 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Virne provides the following core capabilities:

Reinforcement Learning Support
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Provides standardized RL pipelines and Gym-style environments for rapid development and benchmarking of RL-based solutions.
Provides standardized RL pipelines and Gymnasium-compatible environments for rapid development and benchmarking of RL-based solutions.

.. grid-item-card::
:class-item: sd-font-weight-bold
Expand Down Expand Up @@ -79,11 +79,11 @@ algorithms.

.. figure:: _static/virne-rl-support.png
:width: 1000
:alt: Unified Gym-style Environment and RL Pipeline in Virne
:alt: Unified Gymnasium-compatible environment and RL pipeline in Virne
:align: center
:figclass: virne-diagram

A shared Gym-style environment connects observations, policies, rollouts,
A shared Gymnasium-compatible environment connects observations, policies, rollouts,
training, and evaluation.

Citations
Expand Down
7 changes: 7 additions & 0 deletions docs/source/intro/rl-support.rst
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,13 @@ This design keeps simulation, feasibility checking, policy architecture, and
training logic separable. Different RL solvers can therefore share the same
network scenarios and evaluation pipeline.

Instance-level RL environments follow the Gymnasium 1.3 API: ``reset()``
returns ``(observation, info)`` and ``step()`` returns ``(observation, reward,
terminated, truncated, info)``. Virne's solution-building episodes currently
end through natural termination, so they return ``truncated=False``. Training
code still keeps both flags distinct so future external time limits can
bootstrap value estimates correctly.

Run a Minimal Training Check
----------------------------

Expand Down
23 changes: 11 additions & 12 deletions docs/source/start/installation.rst
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
Installation
============

.. important::
.. note::

Virne currently depends on ``gym==0.22.0`` and is not yet compatible with
newer Gym APIs. This constraint is independent of the upgraded PyTorch and
PyG stack. NumPy therefore remains capped below 2.0 until the Gymnasium
migration is complete.
Virne's RL environments use the ``gymnasium==1.3.0`` API. NumPy remains
capped below 2.0 so that numerical compatibility can be validated in a
separate, reproducible upgrade.

Get the Source
--------------
Expand All @@ -28,10 +27,10 @@ Install with the Script
-----------------------

The script supports CPU environments on Linux and macOS, plus CUDA 12.6, 12.8,
and 13.0 on Linux. It installs PyTorch 2.11.0, PyG 2.8.0.post1, and Virne itself
in editable mode. Matching PyG acceleration wheels are selected for the active
platform and Python ABI. Run it from the repository root after activating a
Python 3.10 or 3.11 environment.
and 13.0 on Linux. It installs PyTorch 2.11.0, PyG 2.8.0.post1, Gymnasium 1.3.0,
and Virne itself in editable mode. Matching PyG acceleration wheels are selected
for the active platform and Python ABI. Run it from the repository root after
activating a Python 3.10 or 3.11 environment.

.. code-block:: bash

Expand All @@ -51,7 +50,7 @@ Keep the virtual environment active and run:

.. code-block:: bash

python -c "import torch, torch_geometric, virne; print(virne.__version__, torch.__version__, torch_geometric.__version__)"
python -c "import gymnasium, torch, torch_geometric, virne; print(virne.__version__, gymnasium.__version__, torch.__version__, torch_geometric.__version__)"

The command should print the installed Virne, PyTorch, and PyG versions. You can
then continue to the :doc:`Quickstart <running>`.
The command should print the installed Virne, Gymnasium, PyTorch, and PyG
versions. You can then continue to the :doc:`Quickstart <running>`.
2 changes: 1 addition & 1 deletion install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -91,4 +91,4 @@ fi
--find-links "https://data.pyg.org/whl/torch-${TORCH_VERSION}+${pyg_wheel_tag}.html"

"${virne_python}" -c \
"import torch, torch_geometric, virne; print(f'Virne {virne.__version__}; PyTorch {torch.__version__}; PyG {torch_geometric.__version__}')"
"import gymnasium, torch, torch_geometric, virne; print(f'Virne {virne.__version__}; Gymnasium {gymnasium.__version__}; PyTorch {torch.__version__}; PyG {torch_geometric.__version__}')"
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ classifiers = [
'Topic :: Software Development :: Libraries :: Python Frameworks',
]
dependencies = [
"gym == 0.22.0",
"gymnasium == 1.3.0",
"numpy >= 1.23.5, < 2.0",
"pandas >= 1.5.3",
"matplotlib >= 3.7.1",
Expand Down
147 changes: 140 additions & 7 deletions tests/solver/learning/test_rl_boundaries.py
< 10BC8 tr data-hunk="dbe6ddd1d503c5d812cff87e2bc1d160b3ebca0f5a3f04650a37c253795a8d8f" class="show-top-border">
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
import numpy as np
import pytest
import torch
import gymnasium as gym
from gymnasium import spaces
from gymnasium.utils.env_checker import check_env
from omegaconf import OmegaConf

from virne.core import Solution
Expand All @@ -18,6 +21,7 @@
from virne.solver.learning.utils import get_unexistent_link_pairs
from virne.solver.learning.rl_core.feature_constructor import FeatureConstructorRegistry
from virne.solver.learning.rl_core.instance_agent import InstanceAgent
from virne.solver.learning.rl_core.buffer import RolloutBuffer
from virne.solver.learning.rl_core.instance_rl_environment import (
InstanceRLEnv,
JointPRStepInstanceRLEnv,
Expand Down Expand Up @@ -136,9 +140,12 @@ def test_no_feasible_action_terminates_without_controller_mutation():
observation = env.get_observation()
assert observation['action_mask'].sum() == 1

_, _, done, info = env.step(int(np.flatnonzero(observation['action_mask'])[0]))
_, _, terminated, truncated, info = env.step(
int(np.flatnonzero(observation['action_mask'])[0])
)

assert done is True
assert terminated is True
assert truncated is False
assert env.solution.place_result is False
assert info['description'] == 'No feasible physical node'
env.controller.place_and_route.assert_not_called()
Expand All @@ -163,12 +170,104 @@ def test_first_failed_action_does_not_try_to_revoke_empty_solution():
env.num_actions = env.p_net.num_nodes + 1
env.controller.place_and_route.return_value = (False, {})

_, _, done, _ = env.step(0)
_, _, terminated, truncated, _ = env.step(0)

assert done is True
assert terminated is True
assert truncated is False
assert env.solution.revoke_times == 0


def test_joint_environment_fixed_action_trace_preserves_mapping_results():
env = make_minimal_joint_env(
candidate_nodes=[0, 1],
v_node_ids=(0, 1),
)
env.p_net_backup = env.p_net.copy()

def find_candidate_nodes(_v_net, _p_net, _v_node_id, filter):
return [node_id for node_id in (0, 1) if node_id not in filter]

def place_and_route(_v_net, _p_net, v_node_id, p_node_id, solution, **kwargs):
solution.node_slots[v_node_id] = p_node_id
return True, {}

env.controller.find_candidate_nodes.side_effect = find_candidate_nodes
env.controller.place_and_route.side_effect = place_and_route
env.counter.count_partial_solution.side_effect = (
lambda _v_net, solution: solution.to_dict()
)

observation, info = env.reset(seed=7)
assert info == {}
np.testing.assert_array_equal(observation['action_mask'], [True, True])

observation, reward, terminated, truncated, _ = env.step(1)
assert reward == 0.0
assert terminated is False
assert truncated is False
np.testing.assert_array_equal(observation['action_mask'], [True, False])

observation, reward, terminated, truncated, _ = env.step(0)
assert reward == 0.0
assert terminated is True
assert truncated is False
assert env.solution.result is True
assert env.solution.node_slots == {0: 1, 1: 0}
assert env.solution.selected_actions == [1, 0]
np.testing.assert_array_equal(observation['action_mask'], [True, False])


def test_terminal_gae_characterization_values_are_stable():
buffer = RolloutBuffer()
buffer.add({}, 0, 1.0, False, 0.0, value=0.5)
buffer.add({}, 1, 2.0, True, 0.0, value=1.0)

buffer.compute_returns_and_advantages(
last_value=0.0,
gamma=0.9,
gae_lambda=0.8,
method='gae',
)

np.testing.assert_allclose(buffer.advantages, [2.12, 1.0])
np.testing.assert_allclose(buffer.returns, [2.62, 2.0])


def test_sync_vector_environment_uses_gymnasium_reset_and_step_contracts():
def make_env():
env = make_minimal_joint_env(candidate_nodes=[])
env.p_net_backup = env.p_net.copy()
env.observation_space = spaces.Dict({
'action_mask': spaces.MultiBinary(env.num_actions),
})
return env

vector_env = gym.vector.SyncVectorEnv([make_env, make_env])
try:
observations, infos = vector_env.reset(seed=11)
np.testing.assert_array_equal(
observations['action_mask'],
[[True, False], [True, False]],
)
assert infos == {}

_, _, terminated, truncated, _ = vector_env.step(np.array([0, 0]))
np.testing.assert_array_equal(terminated, [True, True])
np.testing.assert_array_equal(truncated, [False, False])
finally:
vector_env.close()


def test_instance_environment_passes_gymnasium_contract_check():
env = make_minimal_joint_env(candidate_nodes=[])
env.p_net_backup = env.p_net.copy()
env.observation_space = spaces.Dict({
'action_mask': spaces.MultiBinary(env.num_actions),
})

check_env(env, skip_render_check=True)


def test_empty_link_index_has_pyg_compatible_shape():
link_index = ObservationHandler().get_link_index_obs(make_v_net())

Expand Down Expand Up @@ -327,11 +426,11 @@ def __init__(self, p_net, v_net, *args):
self.solution = Solution.from_v_net(v_net)
self.curr_v_node_id = list(v_net.nodes)[0]

def reset(self):
return {'value': 1.0}
def reset(self, *, seed=None, options=None):
return {'value': 1.0}, {}

def step(self, action):
return {'value': np.nan}, 0.0, True, {}
return {'value': np.nan}, 0.0, True, False, {}

agent = InstanceAgent(TerminalEnv)
agent.controller = Mock()
Expand All @@ -352,6 +451,40 @@ def step(self, action):
agent.estimate_value.assert_called_once_with({'value': 1.0})


def test_truncated_rollout_bootstraps_without_marking_transition_terminal():
class TruncatedEnv:
def __init__(self, p_net, v_net, *args):
self.solution = Solution.from_v_net(v_net)
self.curr_v_node_id = list(v_net.nodes)[0]

def reset(self, *, seed=None, options=None):
return {'value': 1.0}, {}

def step(self, action):
return {'value': 2.0}, 0.0, False, True, {}

agent = InstanceAgent(TruncatedEnv)
agent.controller = Mock()
agent.recorder = Mock()
agent.counter = Mock()
agent.logger = Mock()
agent.config = OmegaConf.create({})
agent.device = torch.device('cpu')
agent.preprocess_obs = Mock(side_effect=lambda observation, device: observation)
agent.select_action = Mock(return_value=(0, np.array(0.0)))
agent.estimate_value = Mock(
side_effect=[torch.tensor([1.0]), torch.tensor([2.0])]
)

_, buffer, last_value = agent.learn_with_instance({
'p_net': make_p_net(),
'v_net': make_v_net(),
})

assert buffer.dones == [False]
assert last_value == 2.0


def test_single_node_v_net_features_have_valid_empty_edges():
p_net = make_p_net()
v_net = make_v_net()
Expand Down
4 changes: 2 additions & 2 deletions tests/solver/learning/test_searcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def step(self, action):
self.solution.result = True
self.solution.v_net_r2c_ratio = 2.0
self.solution['node_slots'] = dict(enumerate(self.path))
return self.get_observation(), 0.0, done, {}
return self.get_observation(), 0.0, done, False, {}


class RetryEnv(TwoStepEnv):
Expand All @@ -67,7 +67,7 @@ def step(self, action):
self.solution.result = int(action) == 1
self.solution['result'] = self.solution.result
self.solution.v_net_r2c_ratio = 1.0 if self.solution.result else 0.0
return self.get_observation(), 0.0, True, {}
return self.get_observation(), 0.0, True, False, {}


class BranchPolicy(torch.nn.Module):
Expand Down
3 changes: 1 addition & 2 deletions virne/solver/learning/reinforcement_learning/att_solver.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@


import numpy as np
from gym import spaces
from gymnasium import spaces
from typing import Any, Dict, Tuple, List, Union, Optional, Type, Callable


Expand Down Expand Up @@ -75,4 +75,3 @@ def __init__(self, controller, recorder, counter, logger, config, **kwargs):
instance_env_cls = AttInstanceRLEnv
base_solver_cls = solver_info['rl_solver_cls']
make_solver_class(solver_name, instance_env_cls, base_solver_cls, policy_builder, obs_as_tensor)

2 changes: 1 addition & 1 deletion virne/solver/learning/reinforcement_learning/cnn_solver.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@


import numpy as np
from gym import spaces
from gymnasium import spaces
from typing import Any, Dict, Tuple, List, Union, Optional, Type, Callable


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@


import numpy as np
from gym import spaces
from gymnasium import spaces
from typing import Any, Dict, Tuple, List, Union, Optional, Type, Callable


Expand Down Expand Up @@ -104,4 +104,3 @@ def __init__(self, controller, recorder, counter, logger, config, **kwargs):
base_solver_cls = solver_info['rl_solver_cls']

make_solver_class(solver_name, instance_env_cls, base_solver_cls, policy_builder, obs_as_tensor)

Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@


import numpy as np
from gym import spaces
from gymnasium import spaces
from typing import Any, Dict, Tuple, List, Union, Optional, Type, Callable


Expand Down
Loading
Loading
0