CN116612636A - A collaborative control method for traffic lights based on multi-agent reinforcement learning and multi-modal signal perception - Google Patents

Abstract

The invention provides a signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal perception, which comprises the following steps: collecting data of various sensors, performing multi-mode definition, and acquiring information in real time through a data fusion technology; the signal lamp and the vehicle are cooperatively controlled by adopting a cooperative vehicle-road multi-agent reinforcement learning algorithm; preprocessing the collected data of various sensors, and fusing the data of different modes by utilizing a characteristic fusion method to construct a local state space for each intelligent body; designing action spaces for signal lamp intelligent bodies and vehicle intelligent bodies; designing a reward function for multi-agent reinforcement learning according to the traffic flow control target; designing a communication protocol suitable for a vehicle-road cooperative control scene; and training the multi-agent reinforcement learning model by using historical data or a simulation environment to find an optimal strategy. According to the invention, by introducing the vehicle as an intelligent body, more effective vehicle-road coordination is realized, and the traffic control effect is further improved.

Description

Coordinated control of signal lights based on multi-agent reinforcement learning and multi-modal signal perception method

technical field

The invention belongs to the field of vehicle-road coordination, and in particular relates to a signal light collaborative control method based on multi-agent reinforcement learning and multi-modal signal perception.

Background technique

With the increasingly busy urban traffic, the traditional signal light control method has been difficult to meet the high-efficiency traffic demand of modern cities. In order to solve this problem, researchers have begun to adopt Intelligent Transportation System (ITS) to improve the efficiency of road traffic. Among them, the signal light control system based on multi-agent reinforcement learning and multi-modal signal perception has attracted much attention.

Traditional signal light control methods are usually based on fixed signal periods or predetermined traffic flow patterns, which lack adaptability to real-time traffic conditions. Therefore, there is an urgent need for a collaborative method that breaks through the limitations of traditional traffic light control methods and improves the adaptability to real-time traffic conditions.

Contents of the invention

The purpose of the present invention is to propose a signal light collaborative control method based on multi-agent reinforcement learning and multi-modal signal perception. By introducing vehicles as intelligent agents, more effective vehicle-road coordination can be realized, and the effect of traffic control can be further improved.

In order to achieve the above object, the present invention provides a signal light collaborative control method based on multi-agent reinforcement learning and multi-modal signal perception, the method comprising:

S1. Collect data from various sensors and define multimodality, and obtain information in real time through data fusion technology;

S2. Use the cooperative vehicle-road multi-agent reinforcement learning algorithm to coordinate the control of signal lights and vehicles, and find the optimal strategy through learning to achieve efficient traffic flow control;

S3. Perform preprocessing according to the collected data of various sensors, use the feature fusion method to fuse data of different modes, and construct a local state space for each agent;

S4. Design the action space for the agent;

S5. According to the goal of traffic flow control, design a reward function for multi-agent reinforcement learning;

S6, design communication protocol;

S7. Using historical data or a simulation environment to train the multi-agent reinforcement learning model to find an optimal strategy.

Further, the multimodal definition in S1 includes a visual mode and a radar mode; the image data collected by the camera in the visual mode; the distance and speed information collected by the radar module through the radar; the information includes road conditions information, vehicle position and speed.

Further, the data fusion technology is specifically:

S1.1. Model the scene as a graph structure;

S1.2, perform feature extraction for the data of each modality;

S1.3. Feature fusion based on graph convolutional neural network;

S1.4. Output reinforcement learning status.

Further, the objective function of the collaborative vehicle-road multi-agent reinforcement learning algorithm is defined as:

The reward function is defined as R(s, a), where s represents the state and a represents the action of the agent, by adjusting the action of the agent to maximize the cumulative reward, expressed as:

J(θ)=∑_tR(s_t, a_t)

Among them, J(θ) represents the objective function; s_t represents the state of the agent at time t; a_t represents the action of the agent at time t;

Then the loss function L(θ) is expressed as:

L(θ)=0.5*E[(R(s, a)+γ*max_a'Q(s', a'; θ')-Q(s, a; θ))^2]

Among them, E[·] represents the expected value, θ represents the parameters of the current agent, θ' represents the parameters of the target agent, γ is the discount factor, a' represents the action made by the agent in state s', Q(s, a; θ) is the action value function, which is used to estimate the cumulative reward of taking action a in state s; Q(s', a'; θ') represents the network's evaluation of the action value of action a' in state s' , which is used to measure the quality of the action a' in state s'.

Further, the implementation steps of the objective function of the collaborative vehicle-road multi-agent reinforcement learning algorithm include:

S2.1. Centralized training and distributed execution strategy are used to realize the collaboration between agents through centralized training in the training phase. In the execution phase, each agent uses a distributed strategy to make decisions according to the local state ;

S2.2. Obtain a comprehensive state space containing multimodal information through the data of various sensors in the step S1, so that the agent can perceive traffic conditions more accurately;

S2.3. Using vehicles and signal lights as different agents, realize the collaborative control between vehicles and signal lights, and improve traffic flow.

Further, the data in the step S3 includes vehicle data, road condition data and signal light data;

Further, the state space is represented as follows:

s={vehicle data, road condition data, signal light data}.

Further, the step S4 specifically includes:

S4.1. Discretize the control strategy of the signal light into a series of optional actions;

S4.2. Dynamically adjust the phase setting of the signal light according to the real-time road condition data;

S4.3. Design an adaptive signal light control strategy.

Further, the communication protocol includes communication between vehicles and roadside facilities, communication between signal lights, communication between central controller and signal lights, and data fusion and processing.

Further, the step S7 specifically includes: using historical data or a simulation environment to train the multi-agent reinforcement learning model, finding an optimal strategy, and deploying the optimal strategy to the signal light control system.

Beneficial technical effect of the present invention lies in the following points at least:

(1) The multi-modal signal perception technology provides rich real-time traffic information for the signal light control system. By fusing data from various sensors such as cameras, radars, and on-board sensors, the system can more accurately perceive traffic conditions and provide more targeted decision-making basis for signal light control.

(2) Through the multimodal feature fusion method based on graph convolutional neural network, the topological relationship between vehicles and signal lights can be better captured. At the same time, through the above methods, multi-modal signal perception data can be fused into a unified state space, providing richer and more accurate information for multi-agent reinforcement learning algorithms.

(3) Through the optimal strategy of the present invention, our signal light control system will have a strong dynamic adjustment capability, be able to better adapt to changing traffic conditions in practical applications, and improve the overall signal light control effect.

Description of drawings

The present invention will be further described by using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, they can also obtain other according to the following accompanying drawings under the premise of not paying creative work. Attached picture.

Fig. 1 is a flow chart of the collaborative control method of signal lights based on multi-agent reinforcement learning and multi-modal signal perception in the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

Embodiment one

In one or more implementations, as shown in FIG. 1 , a signal lamp collaborative control method based on multi-agent reinforcement learning and multi-modal signal perception is disclosed, including the following steps:

S1. Collect data from various sensors and define multimodality, and obtain information in real time through data fusion technology;

Specifically, step S1 is responsible for collecting data from various sensors (such as cameras, radars, vehicle sensors, etc.), and obtaining road condition information, vehicle position and speed and other information in real time through data fusion technology. This information will be used to construct the state space for multi-agent reinforcement learning.

Specifically, the multimodal definition includes:

a. Visual modality: The image data collected by the camera can provide information such as vehicle position, shape, speed, etc. Cameras can be installed on the side of the road or on vehicles and transmit image data in real time.

b. Radar mode: The distance and speed information collected by radar helps to accurately detect the position, speed and distance of the vehicle. Radar can be mounted on the side of the road or on vehicles and transmit distance data in real time. These data can be transmitted to the signal light control system in real time through the vehicle communication system (such as V2X communication).

The steps of data fusion in step S1 specifically include:

S1.1. Model the scene as a graph structure. In the traffic light control scenario, there is a certain topological relationship between vehicles and signal lights. We can model the scene as a graph structure, where vehicles and traffic lights are nodes, and their interrelationships are edges;

S1.2. Feature extraction is performed on the data of each modality. For the data of each modality, feature extraction is performed first. For visual modality, convolutional neural network (CNN) can be used to extract features; for radar modality, one-dimensional convolutional neural network or recurrent neural network (RNN) can be used to extract features; for vehicle sensor modality, full connection can be used A neural network (FCNN) extracts features. Then, the extracted features are represented as node features.

S1.3. Feature fusion based on graph convolutional neural network. The extracted multimodal features are fed into a graph convolutional neural network (GCN) as node features. GCN can capture the topological relationship between nodes while maintaining the characteristics of nodes. Through the propagation and aggregation operations of GCN, we can obtain comprehensive features that contain multi-modal information and topological relationships between nodes.

S1.4. Output reinforcement learning status. The fused features of GCN are used as the state space of reinforcement learning for the subsequent multi-agent reinforcement learning algorithm.

With this graph convolutional neural network-based multimodal feature fusion method, we can better capture the topological relationship between vehicles and traffic lights. At the same time, through the above methods, multi-modal signal perception data can be fused into a unified state space, providing richer and more accurate information for multi-agent reinforcement learning algorithms.

S2. Use the collaborative vehicle-road multi-agent reinforcement learning algorithm to carry out collaborative control of signal lights and vehicles. Among them, signal lights and vehicles are regarded as multiple agents, and the optimal strategy is found through learning to achieve efficient traffic flow control.

Specifically, the collaborative vehicle-road multi-agent reinforcement learning algorithm is called CVR-MARL, and its full name is CollaborativeVehicle-Road Multi-Agent Reinforcement Learning. The objective function of CVR-MARL is to maximize the cumulative reward of the system. In this model, traffic lights and vehicles are regarded as multiple agents, which need to learn to find the optimal strategy to achieve efficient traffic flow control.

The objective function of CVR-MARL is defined as:

The reward function is defined as R(s, a), where s represents the state (feature fusion result based on multimodal perception), a represents the action of the agent, and the cumulative reward is maximized by adjusting the action of the agent, expressed as :

J(θ)=∑_tR(s_t, a_t)

Among them, J(θ) represents the objective function; s_t represents the state of the agent at time t; a_t represents the action of the agent at time t;

Then the loss function L(θ) is expressed as:

L(θ)=0.5*E[(R(s, a)+γ*max_a'Q(s', a'; θ')-Q(s, a; θ))^2]

Among them, E[·] represents the expected value, θ represents the parameters of the current agent, θ' represents the parameters of the target agent, γ is the discount factor, a' represents the action made by the agent in state s', Q(s, a; θ) is the action value function, which is used to estimate the cumulative reward of taking action a in state s; Q(s', a'; θ') represents the network's evaluation of the action value of action a' in state s' , which is used to measure the quality of the action a' in state s'.

Among them, the methods to achieve this goal are as follows:

S2.1. Multi-agent collaboration: using centralized training and distributed execution strategies, through centralized training in the training phase, the collaboration between agents is realized. In the execution phase, each agent (including vehicles and signal lights) Use distributed strategies to make decisions based on local state;

S2.2. Construction of state space: through the data of various sensors in the step S1, a comprehensive state space containing multimodal information is obtained, so that the agent can perceive traffic conditions more accurately and construct a state space;

S2.3. Vehicle-road coordination: Vehicles and signal lights are regarded as different intelligent bodies to realize the coordinated control between vehicles and signal lights and improve traffic flow.

S3. Perform preprocessing according to the collected data of various sensors, and use the feature fusion method to fuse the data of different modalities together to construct a local state space for each agent.

Specifically, according to the data collected by the multimodal signal perception module, a local state space is constructed for each agent (signal lights and vehicles). Specifically, CVR-MARL collects multimodal data from three aspects: vehicles, road conditions and signal lights. Include the following data:

Vehicle data:

a) Location information: latitude and longitude, heading angle, etc. of the vehicle.

b) Speed information: the real-time speed of the vehicle.

c) Acceleration information: the real-time acceleration of the vehicle.

d) Vehicle type: such as cars, trucks, buses, etc.

e) Vehicle communication data: communication information between vehicles, such as vehicle networking data (V2V), etc.

Traffic data:

a) Road structure: information such as the width of the road, the number of lanes, and the divider.

b) Traffic flow: the number and density of vehicles in each lane.

c) Road environment information: road conditions, weather, light and other factors.

Semaphore data:

a) Status information: the current status of the signal light (red light, green light, yellow light).

b) Remaining time: The remaining time for the status change of the signal light.

c) Signal light control strategy: such as fixed cycle control, induction control, etc.

d) Vehicle-signal light communication data: communication information between vehicles and signal lights, such as vehicle-to-infrastructure communication data (V2I), etc. Based on the collected multimodal data, the state space can be constructed as follows:

s={vehicle data, road condition data, signal light data}

When constructing the state space, the collected multimodal data needs to be preprocessed to eliminate the dimensional and scale differences between the data. For example, the data can be normalized so that they lie within the same range. Next, the data from different modalities are fused together using a feature fusion method to generate a comprehensive state representation. This comprehensive state representation can make full use of multi-modal information to provide CVR-MARL with richer environmental perception, so as to better realize vehicle-road collaborative control.

S4. Design a feasible action space for the signal light agent and the vehicle agent.

Specifically, a feasible action space is designed for the signal light agent and the vehicle agent. When designing the action space, we need to consider the control strategy of the signal light so that the action space can adapt to different road conditions. In order to realize the dynamic signal light action space, the following methods are adopted:

S4.1. Discretize the control strategy of the signal light into a series of optional actions. For example, it can be divided into several discrete actions according to parameters such as phase, duration and change rate of the signal light. This approach simplifies the representation of the action space, which facilitates exploration and optimization by reinforcement learning algorithms.

S4.2. Dynamically adjust the phase setting of the signal light according to the real-time road condition data, for example, increase the green light time on the lane with heavy traffic flow to reduce congestion. In addition, phase order and duration can be adjusted based on factors such as road structure, vehicle type, and weather to improve traffic efficiency.

S4.3. Design an adaptive signal light control strategy, for example, use induction control at intersections with small traffic, and use cooperative control at intersections with large traffic. In addition, the parameters of the control strategy can be dynamically adjusted based on real-time traffic data to adapt to changes in road conditions.

The action space is expressed as:

A = {action1, action2, ..., actionn}

Wherein, each action corresponds to a signal light control strategy or parameter setting. By dynamically adjusting the action space and control strategy, we can realize the intelligent control of signal lights, thereby improving traffic efficiency and safety.

S5. According to the goal of traffic flow control, design an appropriate reward function for multi-agent reinforcement learning.

Specifically, according to the goal of traffic flow control (such as reducing congestion, reducing emissions, etc.), an appropriate reward function is designed for multi-agent reinforcement learning. The reward function needs to balance various factors to achieve the optimal vehicle-road cooperative control effect. Specifically, the transfer reward function R(s, a, s′) is designed as the following form:

R(s,a,s')=w1*T(s,a,s')+w2*D(s,a,s')+w3*S(s,a,s')

in:

s: current state;

a: the action performed by the agent;

s′: the new state after performing the action;

w1, w2, w3: weight parameters, used to balance the importance of each indicator;

T(s, a, s′): traffic efficiency index, such as the average speed or waiting time of vehicles passing through the intersection;

D(s, a, s′): indicators of traffic congestion, such as the length of vehicles queuing at intersections or the number of vehicles waiting;

S(s, a, s′): Traffic safety indicators, such as the probability of traffic accidents or the safe distance between vehicles and pedestrians.

The design of the reward function needs to take into account many aspects such as traffic efficiency, congestion level and safety, so as to guide the agent to make decisions that are beneficial to the overall traffic situation. In practical applications, weight parameters and index functions can be adjusted according to specific scenarios and requirements to achieve better control effects.

S6. Design a communication protocol suitable for the vehicle-road collaborative control scenario.

Specifically, a communication protocol suitable for vehicle-road collaborative control scenarios is designed, so that information can be exchanged efficiently and safely between agents. Communication protocols need to consider factors such as low latency, high reliability, and security. We think that each specific device should use different communication methods before to ensure the realization of the purpose of the present invention.

The communication protocol includes communication between vehicles and roadside facilities, communication between signal lights, communication between the central controller and signal lights, and data fusion and processing, specifically:

a) Vehicle-to-roadside facility communication (vehicle-to-road communication): Dedicated short-range communication (DSRC) or vehicle-to-vehicle Internet (V2X) technology is used to realize two-way communication between vehicles and roadside facilities (such as signal lights, sensors, etc.). The vehicle can send its own status information (such as position, speed, driving direction, etc.) to roadside facilities, and at the same time receive instructions from roadside facilities (such as signal light changes, speed limit information, etc.).

b) Communication among signal lamps: Information exchange between signal lamps can be realized through wireless sensor network (WSN) or cellular network for collaborative control. Through this communication mechanism, adjacent signal lights can share local traffic information, such as traffic flow, waiting time, etc.

c) Communication between the central controller and the signal lights: The central controller communicates with each signal light through a wired or wireless network. The central controller is responsible for processing information from signal lights and vehicles, running a multi-agent reinforcement learning algorithm (CVR-MARL), and sending control strategies to corresponding signal lights.

d) Data fusion and processing: the multimodal signal perception module sends the collected data to the data processing module. The data processing module is responsible for the feature fusion of multi-modal data, constructing the state space, and passing it to the multi-agent reinforcement learning algorithm.

S7. Using historical data or a simulation environment to train the multi-agent reinforcement learning model to find an optimal strategy.

Specifically, the multi-agent reinforcement learning model is trained using historical data or a simulation environment to find the optimal strategy. In practical applications, in addition to deploying the optimal strategy to the signal light control system, we also need to consider the dynamic adjustment capability to better adapt to changing traffic conditions in practical applications. To achieve this goal, we can adopt the following strategies:

S7.1. Online learning and updating: Through online learning and updating strategies, we can update the reinforcement learning model according to the current traffic conditions in real time. This means that our system can continuously learn and adapt to the actual traffic environment, thereby improving the effectiveness of signal light control.

S7.2. Exploration-exploitation trade-off: During the implementation phase, we need to make a trade-off between exploration and exploitation. By introducing a certain degree of exploration, we can let the model continuously try new strategies in practical applications to discover better solutions. However, too much exploration may reduce the stability of the system. Therefore, we need to find a suitable balance between exploration and exploitation.

S7.3. Abnormal situation handling: In practical applications, some abnormal situations may occur, such as traffic accidents, road closures, etc. In response to these situations, we need to design an exception handling mechanism so that the system can automatically adjust when encountering these problems. For example, when a closed road is detected, the system can automatically adjust the signal light strategy to guide vehicles to detour.

S7.4. Real-time feedback and adjustment: In order to further improve the dynamic adjustment capability of the system, we can introduce a real-time feedback mechanism into the signal light control system. By collecting real-time traffic data and comparing it with model predictions, the system can adjust itself to better adapt to actual traffic conditions.

Through the above strategies, our signal light control system will have strong dynamic adjustment capabilities, be able to better adapt to changing traffic conditions in practical applications, and improve the overall signal light control effect.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions and deformations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope is defined by the claims and their equivalents.

Claims (10)

1. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal perception is characterized by comprising the following steps:

s1, collecting data of various sensors, performing multi-mode definition, and acquiring information in real time through a data fusion technology;

s2, adopting a cooperative vehicle-road multi-agent reinforcement learning algorithm to cooperatively control the signal lamp and the vehicle, and finding an optimal strategy through learning to realize efficient traffic flow control;

s3, preprocessing the collected data of various sensors, and fusing the data of different modes by utilizing a characteristic fusion method to construct a local state space for each intelligent agent;

s4, designing an action space for the intelligent body;

s5, designing a reward function for multi-agent reinforcement learning according to the traffic flow control target;

s6, designing a communication protocol;

and S7, training the multi-agent reinforcement learning model by using historical data or simulation environment to find out an optimal strategy.

2. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal perception according to claim 1, wherein the multi-mode definition in S1 includes a visual mode and a radar mode; image data collected by the visual mode through a camera; distance and speed information collected by the radar module through a radar; the information includes road condition information, vehicle position and speed.

3. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal perception according to claim 2, wherein the data fusion technology is specifically:

s1.1, modeling a scene into a graph structure;

s1.2, extracting features of data of each mode;

s1.3, feature fusion based on a graph convolution neural network;

s1.4, outputting the reinforcement learning state.

4. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal sensing according to claim 2, wherein the objective function of the cooperative vehicle-road multi-agent reinforcement learning algorithm is defined as:

the bonus function is defined as R (s, a), where s represents a state and a represents an action of an agent, expressed as:

J(θ)＝∑_tR(s_t，a_t)

wherein J (θ) represents an objective function; s_t represents the state of the agent at the moment t; a_t represents actions made by the agent at time t;

the loss function L (θ) is expressed as:

L(θ)＝0.5*E[(R(s，a)+γ*max_a′Q(s′，a′；θ′)-Q(s，a；θ))^2]

wherein E [. Cndot. ] represents an expected value, θ represents a parameter of the current agent, θ ' represents a parameter of the target agent, γ is a discount factor, a ' represents an action made by the agent in state s ', Q (s, a; θ) is an action cost function for estimating a jackpot for taking action a in state s; q (s ', a '; θ ') represents an evaluation of the action value of the network for action a ' in state s ', and is used to measure how good the action a ' is in state s '.

5. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal sensing according to claim 4, wherein the implementation step of the objective function of the cooperative vehicle-road multi-agent reinforcement learning algorithm comprises:

s2.1, utilizing a centralized training and a distributed execution strategy, realizing the cooperation among the intelligent agents by performing the centralized training in a training stage, and making a decision according to a local state by using the distributed strategy by each intelligent agent in an execution stage;

s2.2, acquiring a comprehensive state space containing multi-mode information through the data of the various sensors in the step S1, so that an intelligent agent can more accurately sense traffic conditions;

s2.3, the vehicle and the signal lamp are used as different intelligent agents, so that cooperative control between the vehicle and the signal lamp is realized, and the traffic fluency is improved.

6. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal sensing according to claim 1, wherein the data in the step S3 includes vehicle data, road condition data and signal lamp data.

7. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal sensing according to claim 6, wherein the state space is represented as follows:

s= { vehicle data, road condition data, signal lamp data }.

8. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal sensing according to claim 5, wherein the step S4 specifically comprises:

s4.1, discretizing the control strategy of the signal lamp into a series of optional actions;

s4.2, dynamically adjusting the phase setting of the signal lamp according to the real-time road condition data;

s4.3, designing a self-adaptive signal lamp control strategy.

9. The multi-agent reinforcement learning and multi-mode signal perception based signal lamp cooperative control method according to claim 1, wherein the communication protocol comprises vehicle and road side facility communication, communication between signal lamps, central controller and signal lamp communication and data fusion and processing.

10. The signal lamp cooperative control method based on multi-agent reinforcement learning and multi-mode signal sensing according to claim 1, wherein the step S7 is specifically: training the multi-agent reinforcement learning model by using historical data or a simulation environment, finding an optimal strategy, and deploying the optimal strategy to a signal lamp control system.