CN111273668A - Unmanned vehicle motion track planning system and method for structured road - Google Patents

Abstract

The invention relates to a motion trail planning system of an unmanned vehicle for a structured road, which comprises a sensing module, a positioning module, a lane change decision module, a motion planning module and a trail tracking module, wherein the sensing module is used for sensing the motion trail of the unmanned vehicle; the lane change decision module outputs decision actions according to data collected by the sensing module and the positioning module; and the motion planning module outputs an optimal track to the track tracking module according to the decision action. The invention uses deep reinforcement learning to make decisions, and modules such as trajectory planning, perception and control based on decision action dynamics are independently processed, thereby greatly improving interpretability and operability of a decision planning process compared with an end-to-end method, and being well adapted to the system architecture of the existing unmanned automobile.

Description

Motion trajectory planning system and method for unmanned vehicles for structured roads

technical field

The invention belongs to the technical field of unmanned driving, and in particular relates to a system and method for planning the motion trajectory of unmanned vehicles for structured roads.

Background technique

The goal of the decision-making and motion planning system of unmanned vehicles is to make unmanned vehicles produce safe and reasonable driving behaviors like skilled drivers, and plan a safe driving trajectory. Most of the traditional decision-making planning systems are rule-based decision-making and planning methods with clear logic and strong planning reasoning ability, but they need to anticipate various scenarios that unmanned vehicles will encounter, and the system is relatively complex. In recent years, the method of decision-making planning using end-to-end convolutional neural networks has been applied to unmanned vehicles, which greatly simplifies the decision-making planning system. The system directly inputs each frame of images obtained by the camera, and directly outputs the vehicle target steering after the neural network makes decisions. Disc corners, but they are less interpretable.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a system and method for planning the motion trajectory of an unmanned vehicle for structured roads.

To achieve the above object, the present invention adopts the following technical solutions:

An unmanned vehicle motion trajectory planning system for structured roads, comprising a perception module, a positioning module, a lane change decision module, a motion planning module and a trajectory tracking module; the lane change decision module is based on the data collected by the perception module and the positioning module. Data output decision action; the motion planning module outputs the optimal trajectory to the trajectory tracking module according to the decision action.

Further, the perception module includes lidar, millimeter-wave radar, and motion cameras.

Further, the positioning module includes a GPS satellite positioning system, an inertial navigator and a network differential module.

A method for planning the motion trajectory of an unmanned vehicle for structured roads, comprising the following steps:

Step S1: collect real-time driving data;

Step S2: transform the collected driving data into coordinates, and convert the data of the vehicle in the Frenet coordinate system into the data of the vehicle in the global coordinate system;

Step S3: Determine the state space, action space and action value return function;

Step S4: constructing a deep reinforcement learning model;

Step S5: take the surrounding environment information, decision-making action, and the state of the vehicle as input, calculate through the deep reinforcement learning model, output the Q value of the unmanned vehicle performing different actions, and select the corresponding decision-making action according to the output maximum Q value;

Step S6: according to the decision-making action given by the decision-making module, calculate the target configuration under the Frenet coordinates;

Step S7: According to the initial configuration and target configuration of the current position of the unmanned vehicle, plan a trajectory set containing a timestamp within a preset time interval;

Step S8: establish a loss function, and select an optimal trajectory from the generated trajectory set as a tracking trajectory.

纵向加速度

和横向位移1、横向速度

和横向加速度

Further, the data under the global coordinate system includes: position (x, y), velocity v _x , acceleration a _x converted into longitudinal displacement s under the Frenet coordinate system, longitudinal velocity

longitudinal acceleration

and lateral displacement 1, lateral velocity

and lateral acceleration

Further, the step S3 is specifically: defining the state space as the motion state of the unmanned vehicle and other surrounding vehicles under the Frenet coordinates based on the bird's-eye view state:

In the formula: s _ego - is the state of the unmanned vehicle, N ₀ - is the number of surrounding vehicles.

The state of the unmanned vehicle and the state of other vehicles are described by a six-tuple, and the main features are the longitudinal distance, longitudinal speed, longitudinal acceleration, lateral distance, lateral speed, and lateral acceleration of the vehicle:

The input decision-making actions are divided into six types: speed maintaining, acceleration, deceleration, lane change to the left, lane change to the right, and following the car, which are represented by a six-tuple as:

A = [KS, ACC, DEC, CL, CR, FOL] ^T

The value return function is established according to driving safety, comfort, headway, and driving speed, namely:

R(s,a)=Rs( _s ,a)+ _Rt (s,a)+ _Rc (s,a)+ _Rv (s,a)

The formula for the headway return value is as follows:

In the formula, RE is the headway index, and RE=30 in the present invention.

The formula for the return value of comfort is:

The reward value of driving speed is defined as:

In the formula, _vmax is the maximum speed limited by the current lane.

Further, the step S4 is specifically:

The information of the input layer of the DQN model is the decision-making action, the state of the vehicle itself and other vehicle states, and the output layer is the Q value corresponding to each decision-making action taken in this state. A deep neural network is used to replace the value function. When the value return is known The function Q ^* (s, a), using a deep neural network to fit the Q function, there are:

Q(s, α) ≈ Q(s, α; θ)

There is a difference between the Q function estimate obtained by solving the Bellman equation and the Q value predicted by the deep neural network, and this difference is used as a loss function to update the neural network;

The loss function is introduced to minimize the difference between the Q value estimate by the Bellman equation and the Q value estimate by the neural network, namely:

_Li (θ _i )=E[(y _i -Q(s, a, θ)) ² ]

In the formula, y _i is the Q value obtained by the Bellman equation, y _i =r+γmaxQ(s', a', θ); i is the number of iterations;

The above formula is trained by gradient descent, and the formula is:

Further, the deep neural network is composed of two neural networks with the same structure, the evaluation neural network and the target neural network. The evaluation neural network participates in training, and the parameters are continuously updated to improve its prediction ability, while the target neural network is in the evaluation neural network. After the network is trained for a certain number of times, the parameters of the estimated neural network are saved.

Further, the step S6 is specifically:

其中l_t为当前车道中心线的横向距离l，T_j为到达该目标位置的时间；纵向的目标配置为

其中

为想要保持或达到的目标车速，

是允许的车速浮动；According to the decision action of the lane change decision module, the target position information of the driverless car is determined. When the decision action is speed maintenance, acceleration and deceleration, the lateral target configuration is

where l _t is the lateral distance l of the center line of the current lane, and T _j is the time to reach the target position; the vertical target configuration is

in

For the target speed you want to maintain or achieve,

is the allowable speed fluctuation;

其中l_t为左侧或右侧车道中心线的横向距离l；如果无人驾驶汽车前方没有障碍车辆，则纵向的目标配置为

如果无人驾驶汽车前方有障碍车辆，则纵向的目标配置为

其中s_ob、

分别是前面障碍车辆的纵向距离、纵向速度、纵向加速度；When the decision-making action is to change lanes to the left or change to the right, the lateral target configuration is

where l _t is the lateral distance l of the left or right lane centerline; if there is no obstacle vehicle in front of the driverless car, the longitudinal target configuration is

If there is an obstacle vehicle in front of the driverless car, the longitudinal target configuration is

where s _ob ,

are the longitudinal distance, longitudinal velocity, and longitudinal acceleration of the preceding obstacle vehicle, respectively;

其中l_t为当前车道中心线的横向距离l；纵向的目标配置为

其中

s_lv、

分别是跟随车辆的纵向距离、纵向速度、纵向加速度。When the decision action is to follow the vehicle, the lateral target configuration is

where l _t is the lateral distance l of the current lane centerline; the longitudinal target configuration is

in

_slv ,

They are the longitudinal distance, longitudinal velocity, and longitudinal acceleration of the following vehicle, respectively.

Further, the step S8 is specifically:

Step S81: the loss function includes the following:

The lateral loss function is defined as:

C _l =k _j J _t (l(t))+k _t T+k _l l ₁ ²

Where k _j , k _t , k _l are the weight values, k _j J _t (l(t)) is the trajectory with a larger penalty Jerk; k _t T is the penalty braking time, the larger the T, the longer the braking time; k _l l ₁ ² is the penalty for deviating from the road centerline;

纵向损失函数定义为：When the portrait target is configured as

The longitudinal loss function is defined as:

C _s =k _j J _t (s(t))+k _t T+k _s [s ₁ -s _d ] ²

where k _j , k _t , and k _s are weight values, and k _s [s ₁ -s _d ] ² is the deviation between the vertical distance s ₁ and the target position s _d in the penalty target configuration;

纵向损失函数定义为：When the portrait target is configured as

The longitudinal loss function is defined as:

是惩罚目标配置中的纵向速度

与目标速度

的偏差；where k _j , _{k t} , and ks are weight values,

is the longitudinal velocity in the penalty target configuration

with the target speed

deviation;

The total loss function of the trajectory is:

C _al =k _la C _l +k _lo C _s

where k _la and k _lo are weight values;

Step S82: After calculating the value of the loss function for all trajectories, remove the trajectories that exceed the maximum speed or the maximum acceleration in the s direction, remove the trajectories that exceed the maximum curvature, remove the trajectories that may collide, and select the loss from the remaining trajectories. The trajectory with the smallest function value is taken as the optimal trajectory and sent to the trajectory tracking module for tracking.

Compared with the prior art, the present invention has the following beneficial effects:

The invention uses deep reinforcement learning to make decisions, based on the dynamic planning trajectory of the decision-making action, and the perception and control modules are processed independently. Compared with the end-to-end method, the interpretability and operability of the decision-making planning process are greatly improved, and the It is well adapted to the system architecture of previous driverless cars.

Description of drawings

Fig. 1 is the overall frame structure of the system of the present invention;

2 is a schematic diagram of the conversion between a Cartesian coordinate system and a Frenet coordinate system in an embodiment of the present invention;

3 is a structural diagram of deep reinforcement learning (DQN) in an embodiment of the present invention;

FIG. 4 is a scene diagram of a vehicle in the process of changing lanes to the left according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

Please refer to FIG. 1 , the present invention provides an unmanned vehicle motion trajectory planning system for structured roads, including a perception module, a positioning module, a lane change decision module, a motion planning module and a trajectory tracking module; the lane change decision module The decision action is output according to the data collected by the sensing module and the positioning module; the motion planning module outputs the optimal trajectory to the trajectory tracking module according to the decision action. The perception module includes lidar, millimeter-wave radar, and motion cameras. The positioning module includes a GPS satellite positioning system, an inertial navigator and a network differential module.

In this embodiment, the lane change decision module mainly includes the following contents:

纵向加速度

和横向位移l、横向速度

和横向加速度

(1) Perform coordinate transformation on the information received from the perception layer and the positioning and navigation system. Using the information of the vehicle in the Frenet coordinate system, it is necessary to convert the position (x, y) of the vehicle in the global coordinate system (Cartesian coordinate system), Velocity v _x and acceleration a _x are converted into longitudinal displacement s and longitudinal velocity in the Frenet coordinate system

longitudinal acceleration

and lateral displacement l, lateral velocity

and lateral acceleration

(2) Determine the state space, action space and action value reward function. The input of the deep neural network is the environment and state, and the output is the action, so it is necessary to determine the state space, action space and action value reward function. The state space is defined as the motion state of the unmanned vehicle and other surrounding vehicles in the Frenet coordinates based on the bird's-eye view state:

In the formula: s _ego - is the state of the unmanned vehicle, N ₀ - is the number of surrounding vehicles.

The state of the unmanned vehicle and the state of other vehicles are described by a six-tuple, and the main features are the longitudinal distance, longitudinal speed, longitudinal acceleration, lateral distance, lateral speed, and lateral acceleration of the vehicle:

In this embodiment, the input decision-making actions are divided into six types: speed maintenance, acceleration, deceleration, lane change to the left, lane change to the right, and following the car, which are represented by a six-tuple as:

A = [KS, ACC, DEC, CL, CR, FOL] ^T

The action value return function quantitatively reflects the degree to which the unmanned vehicle completes the task. The present invention mainly establishes the value return function from the aspects of driving safety, comfort, headway, and driving speed, namely:

R(s,a)=Rs( _s ,a)+ _Rt (s,a)+ _Rc (s,a)+ _Rv (s,a)

Driving safety is the basic requirement and primary task of driverless cars. When the driverless car collides with other vehicles, a large negative reward value R _s = -1000 is given to the driverless car; if the driverless car does not happen When the collision reaches the target position, a large positive reward value R _s =800 is given.

The headway is an important indicator to measure the safety and road traffic efficiency when the vehicle is following. According to experimental data, when the headway is greater than 2s, the probability of a rear-end collision is low; when the headway is greater than 2s, the probability of a rear-end collision is inversely proportional to the headway. The formula for defining the headway return value is as follows:

In the formula, RE is the headway index, and RE=30 in the present invention.

The decision-making of the driverless car should also consider the comfort of the ride, and the frequent switching of decision-making actions should be avoided as much as possible, while reducing violent acceleration actions. The reward value of the present invention for comfort is defined as:

The driving speed determines the driving efficiency of the driverless car, and the driving efficiency should be improved when the traffic rules permit. The reward value of the present invention to the driving speed is defined as:

In the formula, _vmax is the maximum speed limited by the current lane.

(3) Build a deep reinforcement learning (DQN) model. The information of the input layer of the DQN model is the decision-making action, the state of the vehicle itself and other vehicle states, and the output layer is the Q value corresponding to each decision-making action taken in this state. In the DQN model, a deep neural network is used to approximate the replacement value function. When the value return function Q ^* (s, a) is known, and the Q function is fitted by a deep neural network, there are:

Q(s, α) ≈ Q(s, α; θ)

There is a difference between the Q-function estimate obtained by solving the Bellman equation and the Q-value predicted by the deep neural network, and this difference can be used as a loss function to update the neural network. Therefore, a loss function is introduced to minimize the difference between the Q-value estimate by the Bellman equation and the Q-value estimate by the neural network, namely:

_Li (θ _i )=E[(y _i -Q(s, a, θ)) ² ]

In the formula, y _i is the Q value calculated by the Bellman equation, y _i =r+γmaxQ(s', a', θ); i is the number of iterations:

The above formula is trained by gradient descent, and the formula is:

The deep reinforcement learning model is composed of two neural networks with the same structure, the evaluation neural network and the target neural network. Only the evaluation neural network participates in the training, and the parameters are continuously updated to improve its prediction ability, while the target neural network saves the parameters of the evaluation neural network after the evaluation neural network is trained for a certain number of times. There is a difference between the two neural networks in predicting the Q value, and this difference is used to update the policy in Q-learning.

The motion planning module mainly includes the following contents:

其中l_t为当前车道中心线的横向距离l，T_j为到达该目标位置的时间；纵向的目标配置为

其中

为想要保持或达到的目标车速，

是允许的车速浮动。(1) Determine the target location information of the driverless vehicle according to the decision-making action of the lane-change decision-making module. When the decision action is speed hold, acceleration, deceleration, the lateral target configuration is

where l _t is the lateral distance l of the center line of the current lane, and T _j is the time to reach the target position; the vertical target configuration is

in

For the target speed you want to maintain or achieve,

is the allowable speed fluctuation.

其中l_t为左侧或右侧车道中心线的横向距离l；如果无人驾驶汽车前方没有障碍车辆，则纵向的目标配置为

如果无人驾驶汽车前方有障碍车辆，则纵向的目标配置为

其中s_ob、

分别是前面障碍车辆的纵向距离、纵向速度、纵向加速度。When the decision-making action is to change lanes to the left or change to the right, the lateral target configuration is

where l _t is the lateral distance l of the left or right lane centerline; if there is no obstacle vehicle in front of the driverless car, the longitudinal target configuration is

If there is an obstacle vehicle in front of the driverless car, the longitudinal target configuration is

where s _ob ,

are the longitudinal distance, longitudinal velocity, and longitudinal acceleration of the preceding obstacle vehicle, respectively.

其中l_t为当前车道中心线的横向距离l；纵向的目标配置为

其中

s_lv、

分别是跟随车辆的纵向距离、纵向速度、纵向加速度。When the decision action is to follow the vehicle, the lateral target configuration is

where l _t is the lateral distance l of the current lane centerline; the longitudinal target configuration is

in

_slv ,

They are the longitudinal distance, longitudinal velocity, and longitudinal acceleration of the following vehicle, respectively.

纵向加速度

和横向位移l、横向速度

和横向加速度

(2) Determine the information of the current driverless car in the Frenet coordinate system. Accept the information of the positioning and navigation system in real time, and convert the position (x, y), velocity v _x and acceleration a _x of the driverless car in the Cartesian coordinate system calculated by the positioning and navigation system into the longitudinal displacement s in the Frenet coordinate system , longitudinal speed

longitudinal acceleration

and lateral displacement l, lateral velocity

and lateral acceleration

引入一个控制动作的执行时间：T＝t_end-t_start，通过改变执行时间形成一个有限轨迹集。Takahashi等人已经证明：可以使用一个五次多项式来表示Jerk最优化问题在的解，形如：p(t)＝α₀+α₁t+α₂t²+α₃t³+α₄t⁴+α₅t⁵。设初始配置为

目标配置为

可求解得到对应的l方向关于时间t的五次多项式(轨迹)，使用多个不同的T_j可以得到一个l方向上关于时间t的轨迹集。同理可求得s方向上关于时间t的轨迹集。(3) In a certain time interval, plan a trajectory set from the current position of the vehicle to the target position with respect to time. Due to the introduction of the Frenet coordinate system, the trajectory planning problem is decoupled into two optimization problems in the longitudinal s and lateral l directions. The jerk Jerk of the vehicle is related to the comfort of the passengers. The Jerk accumulated by the jerk J _t in the time end t ₀ to t ₁ with respect to the configuration p is:

The execution time of a control action is introduced: T=t _end -t _start , and a finite trajectory set is formed by changing the execution time. Takahashi et al. have shown that the solution to the Jerk optimization problem can be represented by a quintic polynomial, in the form: p(t)=α ₀ +α ₁ t+α ₂ t ² +α ₃ t ³ +α ₄ t ⁴ +α ₅ t ⁵ . Set the initial configuration to

The target is configured as

The quintic polynomial (trajectory) of the corresponding l direction with respect to time t can be obtained by solving, and a set of trajectories in the l direction with respect to time t can be obtained by using multiple different T _j . Similarly, the trajectory set in the s direction with respect to time t can be obtained.

(4) Establish a loss function, and select an optimal trajectory from the generated trajectory set as the tracking trajectory of the control module. The lateral loss function is defined as:

C _l =k _j J _t (l(t))+k _t T+k _l l ₁ ²

Where k _j , k _t , k _l are the weight values, k _j J _t (l(t)) is the trajectory with a larger penalty Jerk; k _t T is the penalty braking time, the larger the T, the longer the braking time; k _l l ₁ ² is the penalty for deviating from the road centerline.

纵向损失函数定义为：When the portrait target is configured as

The longitudinal loss function is defined as:

C _s =k _j J _t (s(t))+k _t T+k _s [s ₁ -s _d ] ²

where k _j , k _t , and k _s are weight values, and k _s [s ₁ -s _d ] ² is the deviation between the longitudinal distance s ₁ and the target position s _d in the penalized target configuration.

纵向损失函数定义为：When the portrait target is configured as

The longitudinal loss function is defined as:

是惩罚目标配置中的纵向速度

与目标速度

的偏差。where k _j , _{k t} , and ks are weight values,

is the longitudinal velocity in the penalty target configuration

with the target speed

deviation.

The total loss function of the trajectory is:

C _al =k _la C _l +k _lo C _s

where k _la and k _lo are weight values.

In this embodiment, as shown in Figure 2, the coordinates of the current moment of the driverless car are:

为无人驾驶车辆在全局坐标系下的位置向量；s(t)为从参考路径的起点到点r的曲线距离；l(t)为车辆位置离参考路径上最近点r的法向距离。In the formula,

is the position vector of the unmanned vehicle in the global coordinate system; s(t) is the curve distance from the starting point of the reference path to point r; l(t) is the normal distance between the vehicle position and the closest point r on the reference path.

From the geometric relationship in the figure:

为参考路径上点R处的法向单位向量；

为点R在全局坐标系下的位置向量。In the formula,

is the normal unit vector at point R on the reference path;

is the position vector of point R in the global coordinate system.

Use θ _x to represent the heading angle of the current position of the vehicle in the global coordinate system; use θ _r to represent the heading angle of point r on the reference path in the global coordinate system; k _r is the curvature at the reference point; v _x and a _x are respectively The speed and acceleration of the vehicle at the current moment. The tangent and normal vectors at the closest point r to the vehicle on the reference path are:

Transform from (1) and (2) to get:

可得：(5) Multiply both sides by the

Available:

关于时间的一阶微分为切向量

乘以-k_r，normal vector on the reference path

The first derivative with respect to time is a tangent vector

Multiply by -k _r ,

且

的方向与

的方向垂直，因此相乘也为0，故由(7)式可得：because

and

direction with

The direction of is vertical, so the multiplication is also 0, so it can be obtained from formula (7):

的模，即：v _x is

, that is:

When a driverless car drives normally, the following two conditions are met:

l＜1/k _r (11)

It can be inferred:

和

可使用相邻两帧之间纵向和横向的速度变化率来表征，即：calculate

and

It can be characterized by the vertical and horizontal velocity change rates between two adjacent frames, namely:

In the formula, f represents the calculated frequency, generally greater than 10Hz.

Through the above formulas, the mutual conversion between the Cartesian coordinate system and the Frenet coordinate system can be completed.

In this embodiment, the information of the input layer of the DQN model is the decision action, the state of the vehicle itself and other vehicle states, and the output layer is the Q value corresponding to each decision action taken in this state.

其中l_t为左侧或右侧车道中心线的横向距离l；纵向的目标配置为

其中s_ob、

分别是前面障碍车辆的纵向距离、纵向速度、纵向加速度。由初始配置和目标配置可得方程组：In this embodiment, as shown in FIG. 4 , a scene graph of the vehicle in the process of changing lanes to the left. The decision action received by the current motion planning module is to change lanes to the left, and the lateral target configuration is

where l _t is the lateral distance l of the left or right lane centerline; the longitudinal target configuration is

where s _ob ,

are the longitudinal distance, longitudinal velocity, and longitudinal acceleration of the preceding obstacle vehicle, respectively. The system of equations can be obtained from the initial configuration and the target configuration:

Let t ₀ =0, α _l0 , α _l1 , α _l2 can be directly obtained as:

Let T=t ₁ -t ₂ , the coefficients α _l3 , α _l4 , α _l5 can be obtained by solving the following matrix equations:

By choosing different T values, a trajectory set in the l direction can be obtained; in the same way, the trajectory set in the s direction can be obtained.

Calculate the loss function value of the trajectory in the l direction using the following loss function:

C _l =k _j J _t (l(t))+k _t T+k _l l ₁ ² (18)

Calculate the loss function value of the trajectory in the s direction using the following loss function:

C _s =k _j J _t (s(t))+k _t T+k _s [s ₁ -s _d ] ² (19)

The total loss function of the trajectory is:

C _al = k _la C _l +k _lo C _s (20)

After calculating the total loss function value for all trajectories, remove the trajectories that exceed the maximum velocity or the maximum acceleration in the s direction, remove the trajectories that exceed the maximum curvature, remove the trajectories that will collide, and select the smallest loss function value from the remaining trajectories. The trajectory is taken as the optimal trajectory and sent to the trajectory tracking module for tracking.

The above descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope of the present invention.

Claims (10)

1. A system for planning the motion trail of an unmanned vehicle for a structured road is characterized by comprising a sensing module, a positioning module, a lane change decision module, a motion planning module and a trail tracking module; the lane change decision module outputs decision actions according to data collected by the sensing module and the positioning module; and the motion planning module outputs an optimal track to the track tracking module according to the decision action.

2. The unmanned automotive motion trajectory planning system for structured roads of claim 1, wherein: the sensing module comprises a laser radar, a millimeter wave radar and a motion camera.

3. The unmanned automotive motion trajectory planning system for structured roads of claim 1, wherein: the positioning module comprises a GPS satellite positioning system, an inertial navigator and a network difference module.

4. A method for planning the motion trail of an unmanned vehicle for a structured road is characterized by comprising the following steps:

step S1, collecting real-time driving data;

step S2, coordinate transformation is carried out on the collected driving data, and data of the vehicle in a Frenet coordinate system are transformed into data of the vehicle in a global coordinate system;

step S3, determining a state space, an action space and an action value return function;

step S4, constructing a deep reinforcement learning model;

step S5, taking the surrounding environment information, decision-making action and the vehicle state as input, calculating through a deep reinforcement learning model, outputting Q values of different actions executed by the unmanned vehicle, and selecting corresponding decision-making action according to the output maximum Q value;

step S6: calculating target configuration under Frenet coordinates according to decision action given by the decision module;

step S7: according to the initial configuration and the target configuration of the current position of the unmanned vehicle, a track set containing a timestamp is planned within a preset time interval;

step S8: and establishing a loss function, and selecting an optimal track from the generated track set as a tracking track.

5. The unmanned aerial vehicle motion trajectory planning method for the structured road according to claim 4, wherein the data under the global coordinate system comprises: position (x, y), velocity v_xAcceleration a_xConverting into longitudinal displacement s and longitudinal speed in Frenet coordinate system

Longitudinal acceleration

And transverse displacement l, transverse velocity

And lateral acceleration

6. The method for planning the unmanned vehicle motion trajectory for the structured road according to claim 4, wherein the step S3 specifically comprises: the state space is defined as the motion state of the unmanned vehicle and other vehicles around the unmanned vehicle in Frenet coordinates based on the bird's eye view state:

in the formula: s_egoUnmanned vehicle self-state, N₀-the number of surrounding vehicles.

The state of the unmanned vehicle and the states of other vehicles are described by a six-element group, and the unmanned vehicle is mainly characterized by comprising the following components in parts by weight:

dividing the input decision-making actions into six types of speed keeping, acceleration, deceleration, lane changing to the left, lane changing to the right and following vehicle driving, and expressing the six types of speed keeping, acceleration, deceleration, lane changing to the left, lane changing to the right and following vehicle driving by using a six-tuple:

A＝[KS，ACC，DEC，CL，CR，FOL]^T

establishing a value return function according to driving safety, comfort level, headway and driving speed, namely:

R(s，a)＝R_s(s，a)+R_t(s，a)+R_c(s，a)+R_v(s，a)

the formula of the head time interval return value is as follows:

in the formula, RE is a headway indicator, and in the present invention, RE is 30.

The return value of the comfort degree is expressed as:

the return value of the driving speed is defined as:

in the formula, v_maxThe maximum speed of the current lane constraint.

7. The method for planning the unmanned vehicle motion trajectory for the structured road according to claim 4, wherein the step S4 specifically comprises:

the information of an input layer of the DQN model is decision-making action, the state of the vehicle and other vehicle states, an output layer is a Q value corresponding to each decision-making action taken under the state, a deep neural network is used for replacing a cost function, and when the value return function Q is known^*(s, a), fitting the Q function with a deep neural network, then:

Q(s，α)≈Q(s，α；θ)

the Q function estimation obtained by solving the Bellman equation has difference with the Q value predicted by the deep neural network, and the difference is used as a loss function to update the neural network;

introducing a loss function to minimize the difference between the bellman equation to the Q value estimate and the neural network to the Q value estimate, namely:

L_i(θ_i)＝E[(y_i-Q(s，a，θ))²]

in the formula, y_iQ value, y, determined for the Bellman equation_iR + γ maxQ (s ', a', θ); i is the number of iterations;

training the formula by adopting a gradient descent method, wherein the formula is as follows:

8. the method of claim 7, wherein the deep neural network is composed of two neural networks having the same structure, the estimated neural network participates in training and continuously updates parameters to improve the prediction capability, and the target neural network stores the parameters of the estimated neural network after the estimated neural network is trained for a certain number of times.

9. The method for planning the unmanned vehicle motion trajectory for the structured road according to claim 4, wherein the step S6 specifically comprises:

determining target position information of the unmanned automobile according to the decision-making action of the lane-changing decision-making module, and when the decision-making action is speed keeping, acceleration and deceleration, configuring the transverse target into a mode of being capable of keeping, accelerating and decelerating

Wherein l_tIs the transverse distance l, T of the center line of the current lane_jIs the time to reach the target location; longitudinal target configuration

Wherein

To maintain or achieve the target vehicle speed,

is allowed vehicle speed float;

when deciding the driving as left lane change and right lane change, the target configuration of the lateral direction is

Wherein l_tThe lateral distance l is the center line of the left or right lane; if there is no obstacle vehicle in front of the unmanned vehicle, the longitudinal target is configured to

If there is obstacle in front of the unmanned automobileVehicle, longitudinal target configuration

Wherein s is_ob、

Longitudinal distance, longitudinal speed, longitudinal acceleration of the preceding obstacle vehicle, respectively;

when the decision-making action is following the car, the transverse target configuration is

Wherein l_tThe transverse distance l is the central line of the current lane; longitudinal target configuration

Wherein

s_lv、

Respectively the longitudinal distance, the longitudinal speed, the longitudinal acceleration of the following vehicle.

10. The method for planning the unmanned vehicle motion trajectory for the structured road according to claim 4, wherein the step S8 specifically comprises:

step S81: the loss function includes the following:

the lateral loss function is defined as:

C_l＝k_jJ_t(l(t))+k_tT+k_ll₁ ²

wherein k is_j、k_t、k_lIs a weight value, k_jJ_t(l (t)) penalizes trajectories with larger Jerk; k is a radical of_tT is punished braking time, and the larger T is, the longer the braking time is; k is a radical of_ll₁ ²Punishment of deviation from the center line of the road;

when the longitudinal target is configured

The longitudinal loss function is defined as:

C_s＝k_jJ_t(s(t))+k_tT+k_s[s₁-s_d]²

wherein k is_j、k_t、k_sIs a weight value, k_s[s₁-s_d]²Is the longitudinal distance s in the penalty target configuration₁And target position s_dA deviation of (a);

when the longitudinal target is configured

The longitudinal loss function is defined as:

wherein k is_j、k_t、k_sIs a weight value of the weight value,

is penalizing longitudinal velocity in target configuration

With target speed

A deviation of (a);

the total loss function of the trace is:

C_al＝k_laC_l+k_loC_s

wherein k is_la、k_loIs a weighted value;

step S82: after the values of the loss functions are calculated for all the tracks, the tracks exceeding the maximum speed or the maximum acceleration in the s direction are removed, the tracks exceeding the maximum curvature are removed, the tracks which can collide are removed, the track with the minimum loss function value is selected from the rest tracks to serve as the optimal track, and the optimal track is sent to a track tracking module to be tracked.

Images