CN116068900A - Reinforced learning behavior control method for multiple incomplete constraint mobile robots - Google Patents

Abstract

The invention provides a reinforcement learning behavior control method for a multi-incomplete constraint mobile robot, which comprises the steps of establishing a kinematic model of the multi-incomplete constraint mobile robot based on an incomplete constraint matrix, establishing a dynamic model of the multi-incomplete constraint mobile robot based on an Euler Lagrange equation, establishing basic behaviors according to the established kinematic model, and combining the designed basic behaviors into composite behaviors according to different priority orders through a zero-space projection technology; by applying the technical scheme, the use of a centralized unit in the task execution stage can be avoided, and the dynamic property and the intelligent property of behavior priority switching are improved.

Description

Reinforcement learning behavior control method for mobile robots with multiple nonholonomic constraints

Technical Field

The invention relates to the technical field of intelligent robots, and in particular to a reinforcement learning behavior control method for multiple nonholonomically constrained mobile robots.

Background Art

In recent years, nonholonomic mobile robots have been widely used in various fields. Since nonholonomic mobile robots cannot be stabilized by using arbitrary time-invariant smooth state feedback control laws, their tracking control problems have been studied preferentially. Through group collaboration, multiple nonholonomic mobile robots usually have better task performance than a single robot. However, nonholonomic constraints often affect team performance, and how to implement collaborative control under nonholonomic constraints poses a challenging control problem.

Existing cooperative control of multiple nonholonomically constrained mobile robots is usually based on centralized or distributed frameworks. Centralized methods use a centralized controller to activate team behaviors and avoid violating nonholonomic constraints. Since the controller must obtain global information, the scalability of centralized methods is not satisfactory. To this end, distributed methods avoid the use of centralized controllers by using a set of networked controllers with topological structures. Most distributed methods only solve cooperative control problems with unique tasks or control objectives. However, multi-task conflicts are common in cooperative control problems and cannot be ignored. Behavior control methods are one of the most effective solutions. The original behavior control method is a hierarchical framework, and low-level behaviors are only executed when all high-level behaviors are completed. In order to improve the efficiency of task execution, a motion mode behavior control framework is proposed by summing behavior commands with adjustable weights, but no behavior is fully executed. By combining the advantages of the above two methods, a zero-space behavior control method is proposed, which not only completes the highest priority behavior, but also executes some low-priority behaviors through zero-space projection. Although the zero-space behavior control method has been extended to different multi-agent system scenarios, it has the inherent defect of implicit centralization, that is, it relies on a centralized task supervisor to assign behavior priorities. To this end, a distributed behavioral control framework is first proposed for cluster control, but task and controller stability analysis is lacking. Next, the task error of the distributed behavioral control is shown to be asymptotically stable, but it is limited to triangular formations in an obstacle-free environment. Then, a set of nonlinear fast terminal sliding mode controllers are designed for the distributed behavioral control, achieving finite-time convergence of the tracking error. Finally, by designing a fixed-time estimator and a terminal sliding mode control law, both the task and tracking errors are fixed-time stable.

However, existing distributed behavior control methods still have the following disadvantages: 1. The priority of the behavior is fixed and pre-set, which leads to poor task dynamic performance and heavy reliance on human intelligence. 2. Lack of optimality and intelligence, which leads to excessive consumption of control resources to maintain good control performance, especially when switching behavior priorities. 3. There are no saturation constraints on the control inputs, which may cause the actuator to violate physical limitations after switching behavior priorities.

Summary of the invention

In view of this, the purpose of the present invention is to provide a reinforcement learning behavior control method for multiple non-holonomic constrained mobile robots. A reinforcement learning controller is designed based on the identifier-actor-critic algorithm to learn the unknown dynamics and optimal control strategy of the system online to ensure that the control performance and control loss are always balanced during the task execution. Input saturation constraints are also considered to avoid the actuator from violating actual physical limitations.

To achieve the above object, the present invention adopts the following technical solution: a reinforcement learning behavior control method for multiple nonholonomic constrained mobile robots, comprising the following steps:

Step S1, establishing a kinematic model of a mobile robot with multiple nonholonomic constraints based on a nonholonomic constraint matrix, establishing a dynamic model of a mobile robot with multiple nonholonomic constraints based on the Euler-Lagrange equations, and constructing basic behaviors according to the established kinematic models, and combining the designed basic behaviors into composite behaviors in different priority orders through a null space projection technique;

Step S2, modeling the behavior priority switching as a distributed partially observable Markov decision process, setting the reference speed instruction of the composite behavior as the action set of the reinforcement learning algorithm under the framework of the centralized training and distributed execution reinforcement learning algorithm, selecting the position and priority of the non-holonomic constrained robot, and the position and priority of its neighboring robots as the observation set of the reinforcement learning algorithm, designing the reward function, and thus constructing the distributed reinforcement learning task supervisor DRLMSs;

Step S3, with the goal of balancing control performance and control loss, an identifier-executor-critic reinforcement learning algorithm is introduced to online identify unknown dynamic models, implement control strategies, and evaluate control performance, thereby designing reinforcement learning controllers RLCs;

Step S4, based on the adaptive control theory, an adaptive compensator is designed to maintain the optimal control performance and offset the saturation effect in real time.

In a preferred embodiment, step S1 specifically includes the following steps:

Step S11: Kinematic modeling of mobile robots with multiple nonholonomic constraints

Consider a group of N (N＞2) nonholonomic constrained mobile robots, where each robot is driven by a differential wheel, i＝1,...,N; the generalized velocity of the i-th nonholonomic constrained mobile robot is expressed as

和

分别是线速度和角速度，

和

分别是左右轮的线速度，

是左右轮间的距离，

表示实数集合；in,

and

are the linear velocity and angular velocity, respectively.

and

are the linear speeds of the left and right wheels,

is the distance between the left and right wheels,

represents the set of real numbers;

Then, the kinematic equation of the i-th nonholonomic constrained mobile robot is expressed as

表示广义状态，

和

分别是位置和方向，

表示非完整约束矩阵；in,

represents a generalized state,

and

are position and direction,

represents a non-holonomic constraint matrix;

In addition, the kinematic equation of the i-th nonholonomic constrained mobile robot in the inertial coordinate system is:

是轮半径，

表示惯性坐标性下的非完整约束矩阵，

和

分别是左右轮的旋转速度；in,

is the wheel radius,

represents the nonholonomic constraint matrix in inertial coordinates,

and

are the rotation speeds of the left and right wheels respectively;

Step S12: Dynamics modeling of mobile robots with multiple nonholonomic constraints

By using the Euler-Lagrange equations, the dynamic model of the i-th nonholonomic constrained mobile robot is derived as

是惯性矩阵，

是科氏力和向心力矩阵，G_i(x_i)是重力矩阵，

表示未知非线性项，

是可设计的输入增益矩阵，

是控制输入，

是非完整约束力；in,

is the inertia matrix,

is the Coriolis force and centripetal force matrix, G _i ( _xi ) is the gravity matrix,

represents the unknown nonlinear term,

is a designable input gain matrix,

is the control input,

It is not completely binding;

First, the differential form of formula (3) is derived as follows

表示S_i(x_i)的微分，

是轮的角加速度；in,

represents the differential of S _i ( _xi ),

is the angular acceleration of the wheel;

得到以下方程Then, substitute formulas (3) and (5) into (4) and multiply on the left by

The following equation is obtained

in,

According to Assumption 2, formula (6) can be rewritten as

是精确项，

是非精确项；in,

is the exact term,

is an inexact term;

Assumption 1: The multi-nonholonomic constrained mobile robot system works in a static scene, and all non-robot obstacles are static and fixed;

步骤S13：多非完整约束移动机器人基本行为构建Assumption 2: The input gain matrix E _i ( _xi ) always satisfies the design

Step S13: Construction of basic behaviors of mobile robots with multiple nonholonomic constraints

进行数学建模如下Assume that each nonholonomic constrained mobile robot has M basic behaviors, where the kth basic behavior of the ith nonholonomic constrained mobile robot can use a task variable

The mathematical modeling is as follows

表示任务函数；where g _i,k (·):

Represents the task function;

Then, the differential form of the task variable σ _i,k is expressed as

是任务的雅克比矩阵；in,

is the Jacobian matrix of the task;

Finally, the reference velocity command of the kth basic behavior of the i-th nonholonomic constrained mobile robot can be expressed as

是任务的雅克比矩阵J_i,k的右伪逆，

是期望的任务函数，

是任务增益，

是任务误差；in,

is the right pseudo-inverse of the Jacobian matrix Ji _,k of the task,

is the desired task function,

is the task gain,

is the task error;

Without loss of generality, the obstacle avoidance behavior, distributed formation behavior and distributed reconstruction behavior are designed as follows: Obstacle avoidance behavior: Obstacle avoidance behavior is a local behavior that aims to ensure that the non-holonomic constrained mobile robot avoids obstacles near the path. Its corresponding task function, expected task and task Jacobian matrix are expressed as follows:

表示第i个非完整约束移动机器人与障碍物的最小距离，d_OA为安全距离，

是最小距离的相对位置，

是避障行为期望的方向，+和-分别表示障碍物在第i个非完整约束移动机器人的左边和右边；in,

represents the minimum distance between the i-th nonholonomic constrained mobile robot and the obstacle, d _OA is the safety distance,

is the relative position of the minimum distance,

is the desired direction of the obstacle avoidance behavior, + and − respectively indicate that the obstacle is to the left and right of the i-th nonholonomically constrained mobile robot;

Distributed formation behavior: Distributed formation behavior is a distributed cooperative behavior that aims to ensure that multiple nonholonomic constrained mobile robots form the desired formation by only using the states of their neighbors. The corresponding task function, expected task, and task Jacobian matrix are expressed as:

是分布式编队行为的估计状态，其通过设计具有如下更新率的自适应估计器来估计：in,

is the estimated state of the distributed formation behavior, which is estimated by designing an adaptive estimator with the following update rate:

是编队的相对位置，

表示领航者的状态，

表示第i个非完整约束移动机器人的邻居；Among them, κ _DF is a positive constant,

is the relative position of the formation,

Indicates the status of the navigator.

represents the neighbors of the i-th nonholonomic constrained mobile robot;

Distributed Reconfiguration Behavior: Distributed reconfiguration behavior is a distributed cooperative behavior designed to ensure that multiple nonholonomic constrained mobile robots reconstruct the desired formation only by using the states of their neighbors. The corresponding task function, expected task, and task Jacobian matrix are expressed as:

是分布式编队行为的估计状态，其通过设计具有如下更新率的自适应估计器来估计：in,

is the estimated state of the distributed formation behavior, which is estimated by designing an adaptive estimator with the following update rate:

是编队重构矩阵；Among them, κ _DR is a positive constant,

is the formation reconstruction matrix;

Step S14: Construction of composite behaviors of mobile robots with multiple nonholonomic constraints

为第i个非完整约束移动机器人的任务函数，其中k_m∈N_M，N_M＝{1,...,M}，m_k表示任务空间的维度，M表示任务的数量；定义与时间相关的优先级函数g_i(k_m,t):N_M×[0,∞]→N_M；同时，定义一个具有如下规则的任务层次结构：A composite task is a combination of multiple basic behaviors in a certain priority order;

is the task function of the i-th nonholonomic constrained mobile robot, where _km∈NM , _NM ={1,...,M}, _mk represents the dimension of the task space, and _M represents the number of tasks. Define a time-related priority function _gi ( _km ,t): _NM ×[0,∞]→ _NM . At the same time, define a task hierarchy with the following rules:

k_α≠k_β；1) A task k _α with priority _gi (k _α ) cannot interfere with task k _β with priority _gi (k _β ) if _gi (k _α ) _≥gi (k _β ),

k _α ≠ k _β ;

表示；2) The mapping from speed to task speed is given by the Jacobian matrix of the task

express;

因此要确保维度m_n大于所有任务的总维度；3) The dimension of the task _m with the lowest priority may be greater than

Therefore, make sure that the dimension m _n is greater than the total dimension of all tasks;

4) The value of g _i (k _m ) is assigned by the task supervisor according to the task requirements and sensor information;

By assigning a given priority to the basic tasks, the speed of the composite task at time t is expressed as

是行为优先级，

是零空间投影的增广雅克比矩阵。in,

is the behavioral priority,

is the augmented Jacobian matrix of the null space projection.

其中

是联合的位置，

是联合的优先级，

是编队标志位，S表示全局状态集合；定义b_i,t＝{v_r,i,t}∈B为局部/本地行为，其中B表示行为集合；定义

为独立的局部观测，其中s_i,t＝{x_i,Pr_i}是局部/本地状态，

表示第i个非完整约束移动机器人的邻居，O表示局部观测集合；由于局部观测，定义局部/本地的行为观测历史为z_i,t∈Z，其中Z表示行为观测历史集合；所有的分布式强化学习任务监管器贡献一个奖励信号，且奖励函数设计如下In a preferred embodiment, the step S2 is specifically as follows: defining the centralized training environment as, the global state is

in

is the joint location,

is the priority of the union,

is the formation flag, S represents the global state set; define b _i,t ={v _r,i,t }∈B as a local behavior, where B represents the behavior set; define

is an independent local observation, where s _i,t = {x _i ,Pr _i } is a local state,

represents the neighbors of the i-th nonholonomic constrained mobile robot, O represents the local observation set; due to local observation, the local behavior observation history is defined as z _i,t ∈Z, where Z represents the behavior observation history set; all distributed reinforcement learning task supervisors contribute a reward signal, and the reward function is designed as follows

r _t = r ₁ + r ₂ , (25)

分别表示无编队、重构编队和期望编队状态的标识；r₁和r₂是分别设置实现任务目标和减少行为切换的奖励信号；in,

They represent the identities of no formation, reconstructed formation, and desired formation states, respectively; _r1 and _r2 are reward signals for achieving task goals and reducing behavior switching, respectively;

然后，混合模块通过求和所有的局部的Q值

生成联合Q值

如下Multiple nonholonomic constrained mobile robots interact with the environment at t time steps, where the i-th nonholonomic constrained mobile robot observes a local observation o _i,t , obtains the previous behavior b _i,t-1 , selects a behavior b _i,t according to the ε-greedy strategy with a decay factor γ _ε , and then obtains a team reward r _t and transfers to the next local observation o _i,t+1 ; Specifically, the centralized training of the distributed reinforcement learning task supervisor is carried out through hierarchical progressive modules, including independent Q-value modules and hybrid modules; First, each nonholonomic constrained mobile robot has an independent Q-value module, which uses the recurrent Q network to input the hidden layer state h _i,t-1 of the gated recurrent neural network, the local observation o _i,t , the previous behavior b _i,t-1 , and outputs the local Q-value

The mixing module then sums up all the local Q values

Generate joint Q value

as follows

表示独立Q值网络的参数；in,

represents the parameters of the independent Q-value network;

中；特别地，将从经验池中采样最小回放次

经历以减少数据相关性和提高样本利用率；然后，训练进行到t+1时间步，且训练直至所有的回合T_total完成后停止；分布式强化学习任务监管器通过最小化以下损失进行训练：After sampling at time step t, the experience quadruple (z _t , b _t , r _t , z _t+1 ) is stored in the experience pool

In particular, the minimum number of replays will be sampled from the experience pool

Experience to reduce data correlation and improve sample utilization; then, training proceeds to t+1 time step, and training stops after all rounds T _total are completed; the distributed reinforcement learning task supervisor is trained by minimizing the following loss:

表示目标网络的参数；in,

Represents the parameters of the target network;

和参考轨迹θ_i,r。Finally, after centralized training, the multi-nonholonomic constrained mobile robot learns a set of optimal distributed behavior priority strategies. In actual scenarios, the multi-nonholonomic constrained mobile robot switches behavior priorities according to the learned strategies. Once the behavior priorities of the multi-nonholonomic constrained mobile robot are determined at each sampling moment, the reference speed v _i,r and reference trajectory x _i,r are obtained through formulas (22)-(24) and (2). According to formula (3), the reference speed in the inertial coordinate system is further calculated.

and the reference trajectory θ _i,r .

In a preferred embodiment, step S3 specifically includes: defining the position and velocity tracking errors in the inertial coordinate system as:

e _p,i =θ _i -θ _i,r , (30)

和

是左右轮的角度，

和

分别是参考位置和参考速度；in,

and

is the angle of the left and right wheels,

and

are the reference position and reference velocity respectively;

The differential forms of formulas (30) and (31) are derived as follows:

是

的微分；in,

yes

The differential of

The integrated tracking error is defined as follows

Define the value function as follows

表示代价函数，α_V,β_V∈(0,2)是可调整的代价参数，且满足α_V+β_V＝2；in,

represents the cost function, α _V , β _V ∈(0,2) are adjustable cost parameters, and satisfy α _V +β _V ＝2;

为最优的跟踪控制策略；因此，最优的值函数可以表示为definition

is the optimal tracking control strategy; therefore, the optimal value function can be expressed as

表示可容许的控制策略；in,

represents the permissible control strategies;

By combining equations (32)-(35) and (37), the Hamilton-Jacobi-Bellman (HJB) equation can be derived as

表示V_i ^*相对于e_i的梯度，

和

表示分别为V_i ^*相对于e_p,i和e_v,i的梯度；in,

represents the gradient of _Vi ^* relative to _ei ,

and

denote the gradients of _Vi ^* relative to _ep,i and ev _,i respectively;

最优的控制策略

可推导为By solving

Optimal control strategy

It can be deduced as

Furthermore, substituting formula (39) into (38) yields the following equation:

需要求解公式(40)获取

然而，由于多非完整约束移动机器人动力学模型的非线性和不精确，

的解析解难以求取；To implement

We need to solve formula (40) to obtain

However, due to the nonlinearity and inaccuracy of the dynamics model of mobile robots with multiple nonholonomic constraints,

The analytical solution of is difficult to obtain;

Therefore, the optimal value function gradient needs to be decomposed as follows:

和

是正常数，

是自适应补偿项；in,

and

is a normal number,

is the adaptive compensation term;

Substituting formula (41) into (39), we obtain the following equation:

和

对于

和

未知项f_i(x_i)和V_i ^o通过神经网络近似如下：As we all know, neural networks have powerful approximation capabilities; therefore, given a compact set

and

for

and

The unknown terms _fi ( _xi ) and _Vio ^are approximated by the neural network as follows:

和

是理想的权重矩阵，w_f和w_V是神经元数量，

和

是基函数向量，

和

是逼近误差，且有界如||δ_f,i||≤ε_f和||δ_V,i||≤ε_V，ε_f和ε_V是正常数；in,

and

is the ideal weight matrix, _wf and _wV are the number of neurons,

and

is the basis function vector,

and

is the approximation error and is bounded such that ||δ _f,i ||≤ε _f and ||δ _V,i ||≤ε _V , ε _f and ε _V are positive constants;

Then, substitute equations (43) and (44) into (41) and (42) to obtain the following equations:

和

是未知的，

无法实施；因此，使用一种辨识者-执行者-评论家强化学习算法以学习最优控制策略；However, due to

and

is unknown,

It is not feasible to implement; therefore, an Identifier-Actor-Critic reinforcement learning algorithm is used to learn the optimal control policy;

Specifically, the discriminator neural network is designed to estimate the unknown nonlinear terms as follows:

是f_i(x_i)的估计，

是辨识者神经网络的权重；辨识者神经网络的更新率可设计为：in,

is an estimate of _fi ( _xi ),

is the weight of the identifier neural network; the update rate of the identifier neural network can be designed as:

是正定矩阵，

是设计的辨识者参数；in,

is a positive definite matrix,

is the identifier parameter of the design;

Then, a critic neural network is designed to evaluate the control performance as follows:

是

的估计值，

是评论家神经网络的权重；评论家网络的更新率可设计为in,

yes

The estimated value of

is the weight of the critic neural network; the update rate of the critic network can be designed as

where γ _c,i is the critic’s learning rate;

Finally, the design of the actuator neural network facility control input is as follows:

是执行者神经网络的权重；执行者网络的更新率可设计为in,

is the weight of the actor neural network; the update rate of the actor network can be designed as

where γ _a,i is the learning rate of the executor.

受到对称执行机构饱和约束的限制如下：In a preferred embodiment: Step S4 is specifically as follows: First, consider the control input

The restrictions subject to the saturation constraints of the symmetric actuators are as follows:

Among them, τ _lim,i ＞0 is a known threshold;

Secondly, the control input can be divided into two items as follows:

τ _i =τ _0,i +τ _Δ,i , (54)

是标称项，

是补偿项，且满足如下条件：in,

is a nominal term,

is a compensation item and meets the following conditions:

且具有更新率如下Finally, the adaptive compensator is designed as

And has an update rate of

是设计的自适应补偿器参数。in,

are the designed adaptive compensator parameters.

Compared with the prior art, the present invention has the following beneficial effects: The present invention takes a multi-holonomic constrained mobile robot system as the research object, and proposes a distributed reinforcement learning behavior control method for multi-nonholonomic constrained mobile robots. First, by modeling the behavior priority switching as a distributed partially observable Markov decision process, the method proposes a set of novel distributed reinforcement learning task supervisors to learn the optimal distributed behavior priority strategy, so that during the task execution, the zero-space behavior control method can switch the behavior priority without relying on any centralized unit, thereby fundamentally solving the fatal defect of the zero-space behavior control method implicitly centralized. At the same time, the use of the learned optimal distributed behavior priority strategy to achieve the behavior priority switching not only makes up for the deficiency of fixed behavior priority under the distributed behavior control framework, improves the dynamic performance of the zero-space behavior control method, but also transfers a large number of online computing processes to the offline learning stage, reducing the dependence of the zero-space behavior control method on high-performance hardware. Secondly, the method proposes a reinforcement learning controller, which learns unknown dynamic models and optimal control strategies by using the identifier-executor-critic reinforcement learning algorithm. During the task execution, the balance between control performance and control loss is always maintained, especially when the behavior priority is switched. Compared with the existing zero-space behavior control method, the control cost will be reduced, avoiding the situation where the multi-nonholonomic constraint mobile robot consumes too much control resources in order to maintain high-performance control. Finally, compared with the existing zero-space behavior control method that does not consider input saturation constraints, in order to prevent the actuator of the multi-nonholonomic constraint mobile robot from exceeding the physical limit, the present invention implements input saturation constraints and designs a set of adaptive compensators to maintain optimal performance and offset the saturation effect in real time.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a principle block diagram of a distributed reinforcement learning behavior control method for multiple nonholonomic constrained mobile robots according to an embodiment of the present invention;

FIG2 is a schematic diagram of an i-th nonholonomic constrained mobile robot according to an embodiment of the present invention;

FIG3 is a pseudo code diagram of a distributed reinforcement learning task supervisor according to an embodiment of the present invention;

FIG4 is a schematic diagram of a network topology of multiple nonholonomically constrained mobile robots according to an embodiment of the present invention;

FIG5 is a diagram showing the selection of simulation parameter values according to an embodiment of the present invention;

6 is a task performance diagram of a distributed reinforcement learning task supervisor before learning according to an embodiment of the present invention, (a) trajectory, (b) direction, (c) distance between a nonholonomically constrained mobile robot and an obstacle, and (d) behavior priority;

7 is a task performance diagram after learning by a distributed reinforcement learning task supervisor according to an embodiment of the present invention, (a) trajectory, (b) direction, (c) distance between a nonholonomically constrained mobile robot and an obstacle, and (d) behavior priority;

FIG8 is a graph of the performance of multiple nonholonomic constrained mobile robot tasks with different task supervisors according to an embodiment of the present invention, (a) distributed reinforcement learning task supervisor, (b) distributed finite state machine task supervisor, (c) distributed model predictive control task supervisor, and (d) traditional reinforcement learning task supervisor;

FIG9 is a graph of the performance of the second nonholonomic constrained mobile robot task with different task supervisors according to an embodiment of the present invention, (a) distributed reinforcement learning task supervisor, (b) distributed finite state machine task supervisor, (c) distributed model predictive control task supervisor, and (d) traditional reinforcement learning task supervisor;

FIG10 is a performance diagram of distributed reinforcement learning control with input saturation constraints according to an embodiment of the present invention, a) trajectory, (b) control input, (c) tracking error, and (d) control cost;

FIG11 is a graph of the learning performance of distributed reinforcement learning according to an embodiment of the present invention, (a) identifier weight, (b) executor weight, (c) critic weight, and (d) adaptive compensator;

FIG12 is a graph of the control performance of the fifth nonholonomic constrained mobile robot with and without input saturation constraints according to an embodiment of the present invention, a) trajectory, (b) control input, (c) tracking error, and (d) control cost;

FIG13 is a trajectory diagram of a multi-nonholonomic constrained mobile robot with different distributed behavior controls according to an embodiment of the present invention, (a) distributed reinforcement learning behavior control, (b) finite time distributed behavior control, (c) intrinsic time distributed behavior control, and (d) traditional reinforcement learning behavior control;

14 is a control performance diagram of different distributed behavior controls according to an embodiment of the present invention, including: a) trajectory, (b) control input, (c) tracking error, and (d) control cost.

DETAILED DESCRIPTION

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and/or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and/or their combinations.

Step 1: Kinematic and dynamic models, basic behaviors and composite behaviors

a. Kinematic modeling of mobile robots with multiple nonholonomic constraints

Consider a group of N (N>2) nonholonomic constrained mobile robots, each of which is driven by a differential wheel. The schematic diagram of the i-th nonholonomic constrained mobile robot is shown in Figure 2, where i=1,...,N. The generalized velocity of the i-th nonholonomic constrained mobile robot can be expressed as

和

分别是线速度和角速度，

和

分别是左右轮的线速度，

是左右轮间的距离，

表示实数集合。in,

and

are the linear velocity and angular velocity, respectively.

and

are the linear speeds of the left and right wheels,

is the distance between the left and right wheels,

Represents the set of real numbers.

Then, the kinematic equation of the i-th nonholonomic constrained mobile robot can be expressed as

表示广义状态，

和

分别是位置和方向，

表示非完整约束矩阵。in,

represents a generalized state,

and

are position and direction,

represents a nonholonomic constraint matrix.

In addition, the kinematic equation of the i-th nonholonomic constrained mobile robot in the inertial coordinate system is:

是轮半径，

表示惯性坐标性下的非完整约束矩阵，

和

分别是左右轮的旋转速度。in,

is the wheel radius,

represents the nonholonomic constraint matrix in inertial coordinates,

and

are the rotation speeds of the left and right wheels respectively.

b. Dynamic modeling of mobile robots with multiple nonholonomic constraints

By using the Euler-Lagrange equations, the dynamic model of the i-th nonholonomic constrained mobile robot can be derived as

是惯性矩阵，

是科氏力和向心力矩阵，G_i(x_i)是重力矩阵，

表示未知非线性项，

是可设计的输入增益矩阵，

是控制输入，

是非完整约束力。in,

is the inertia matrix,

is the Coriolis force and centripetal force matrix, G _i ( _xi ) is the gravity matrix,

represents the unknown nonlinear term,

is a designable input gain matrix,

is the control input,

It is not completely binding.

First, the differential form of formula (3) can be derived as follows

表示S_i(x_i)的微分，

是轮的角加速度。in,

represents the differential of S _i ( _xi ),

is the angular acceleration of the wheel.

可以得到以下方程Then, substitute formulas (3) and (5) into (4) and multiply on the left by

The following equation can be obtained

in,

According to Assumption 2, formula (6) can be rewritten as

是精确项，

是非精确项。in,

is the exact term,

It is an inexact term.

Assumption 1: The multi-nonholonomic constrained mobile robot system works in a static scene, and all non-robot obstacles are static and fixed.

Assumption 2: The input gain matrix E _i ( _xi ) always satisfies the design

c. Construction of basic behaviors of mobile robots with multiple nonholonomic constraints

进行数学建模如下Assume that each nonholonomic constrained mobile robot has M basic behaviors, where the kth basic behavior of the ith nonholonomic constrained mobile robot can use a task variable

The mathematical modeling is as follows

σ _i,k =gi _,k ( _xi ), (8)

表示任务函数。where g _i,k (·):

Represents a task function.

Then, the differential form of the task variable σ _i,k can be expressed as

是任务的雅克比矩阵。in,

is the Jacobian matrix of the task.

Finally, the reference velocity command of the kth basic behavior of the i-th nonholonomic constrained mobile robot can be expressed as

是任务的雅克比矩阵J_i,k的右伪逆，

是期望的任务函数，

是任务增益，

是任务误差。in,

is the right pseudo-inverse of the Jacobian matrix Ji _,k of the task,

is the desired task function,

is the task gain,

It is a task error.

Without loss of generality, the obstacle avoidance behavior, distributed formation behavior and distributed reconstruction behavior are designed as follows: Obstacle Avoidance (OA): Obstacle avoidance behavior is a local behavior that aims to ensure that the non-holonomic constrained mobile robot avoids obstacles near the path. Its corresponding task function, expected task and task Jacobian matrix are expressed as follows:

表示第i个非完整约束移动机器人与障碍物的最小距离，d_OA为安全距离，

是最小距离的相对位置，

是避障行为期望的方向，+和-分别表示障碍物在第i个非完整约束移动机器人的左边和右边。in,

represents the minimum distance between the i-th nonholonomic constrained mobile robot and the obstacle, d _OA is the safety distance,

is the relative position of the minimum distance,

is the desired direction of the obstacle avoidance behavior, and + and - respectively indicate that the obstacle is on the left and right of the i-th nonholonomic constrained mobile robot.

Distributed Formation (DF): Distributed formation behavior is a distributed cooperative behavior that aims to ensure that multiple nonholonomic constrained mobile robots form the desired formation by using only the states of their neighbors. The corresponding task function, expected task, and task Jacobian matrix are expressed as:

是分布式编队行为的估计状态，其通过设计具有如下更新率的自适应估计器来估计：in,

is the estimated state of the distributed formation behavior, which is estimated by designing an adaptive estimator with the following update rate:

是编队的相对位置，

表示领航者的状态，

表示第i个非完整约束移动机器人的邻居。Among them, κ _DF is a positive constant,

is the relative position of the formation,

Indicates the status of the navigator.

denotes the neighbors of the i-th nonholonomic constrained mobile robot.

Distributed Reconstruction (DR): Distributed reconstruction behavior is a distributed cooperative behavior that aims to ensure that multiple nonholonomic constrained mobile robots reconstruct the desired formation only by using the states of their neighbors. The corresponding task function, expected task, and task Jacobian matrix are expressed as:

是分布式编队行为的估计状态，其通过设计具有如下更新率的自适应估计器来估计：in,

is the estimated state of the distributed formation behavior, which is estimated by designing an adaptive estimator with the following update rate:

是编队重构矩阵。Among them, κ _DR is a positive constant,

is the formation reconstruction matrix.

d. Construction of composite behaviors of mobile robots with multiple nonholonomic constraints

为第i个非完整约束移动机器人的任务函数，其中k_m∈N_M，N_M＝{1,...,M}，m_k表示任务空间的维度，M表示任务的数量。定义与时间相关的优先级函数g_i(k_m,t):N_M×[0,∞]→N_M。同时，定义一个具有如下规则的任务层次结构：A composite task is a combination of multiple basic behaviors in a certain priority order.

is the task function of the i-th nonholonomic constrained mobile robot, where _km ∈ _{N M} , N _M = {1, ..., M}, m _k represents the dimension of the task space, and M represents the number of tasks. Define a time-dependent priority function _gi (k _m , t): N _M × [0, ∞] → N _M . At the same time, define a task hierarchy with the following rules:

k_α≠k_β。1) A task k _α with priority _gi (k _α ) cannot interfere with task k _β with priority _gi (k _β ) if _gi (k _α ) _≥gi (k _β ),

k _α ≠k _β .

表示。2) The mapping from speed to task speed is given by the Jacobian matrix of the task

express.

因此要确保维度m_n大于所有任务的总维度。3) The dimension of the task _m with the lowest priority may be greater than

Therefore, make sure that the dimension m _n is larger than the total dimension of all tasks.

4) The value of g _i (k _m ) is assigned by the task supervisor according to the task requirements and sensor information.

By assigning a given priority to the basic tasks, the speed of the composite task at time t can be expressed as

是行为优先级，

是零空间投影的增广雅克比矩阵。in,

is the behavioral priority,

is the augmented Jacobian matrix of the null space projection.

Step 2: Design of Distributed Multi-agent Reinforcement Learning Task Supervisor

For distributed multi-nonholonomic constrained mobile robots, each agent must learn a local/local behavior priority strategy to achieve the collaborative task goal. Since multi-nonholonomic constrained mobile robots can usually be trained in a centralized offline environment, the distributed behavior priority switching problem can be innovatively modeled as a distributed partially observable Markov decision process. By using the value-decomposition networks (VDN) reinforcement learning algorithm and the centralized training distributed execution (CTDE) paradigm, a set of distributed reinforcement learning task supervisors is proposed.

其中

是联合的位置，

是联合的优先级，

是编队标志位，S表示全局状态集合。定义b_i,t＝{v_r,i,t}∈B为局部/本地行为，其中B表示行为集合。由于分布式多智能体强化学习任务监管器无法使用全局状态，因此必须使用部分可观测的状态。定义

为独立的局部观测，其中s_i,t＝{x_i,Pr_i}是局部/本地状态，

表示第i个非完整约束移动机器人的邻居，O表示局部观测集合。由于局部观测，定义局部/本地的行为观测历史为z_i,t∈Z，其中Z表示行为观测历史集合。所有的分布式强化学习任务监管器贡献一个奖励信号，且奖励函数设计如下Define the centralized training environment as, and the global state is

in

is the joint location,

is the priority of the union,

is the formation flag, S represents the global state set. Define b _i,t = {v _r,i,t }∈B as a local behavior, where B represents the behavior set. Since the distributed multi-agent reinforcement learning task supervisor cannot use the global state, it must use partially observable states. Definition

is an independent local observation, where s _i,t = {x _i ,Pr _i } is a local state,

represents the neighbors of the i-th nonholonomic constrained mobile robot, and O represents the local observation set. Due to local observation, the local behavior observation history is defined as z _i,t ∈ Z, where Z represents the behavior observation history set. All distributed reinforcement learning task supervisors contribute a reward signal, and the reward function is designed as follows

r _t = r ₁ + r ₂ , (25)

分别表示无编队、重构编队和期望编队状态的标识。r₁和r₂是分别设置实现任务目标和减少行为切换的奖励信号。in,

They represent the flags of no formation, reconstructed formation and desired formation state respectively. _r1 and _r2 are reward signals for achieving task goals and reducing behavior switching respectively.

然后，混合模块通过求和所有的局部的Q值

生成联合Q值

如下The algorithm pseudocode diagram of the distributed reinforcement learning task supervisor is shown in Figure 3. Multiple non-holonomic constrained mobile robots interact with the environment at t time steps, where the i-th non-holonomic constrained mobile robot observes a local observation o _i,t , obtains the previous behavior b _i,t-1 , selects a behavior b _i,t according to the ε-greedy strategy with a decay factor γ _ε , and then obtains a team reward r _t and transfers to the next local observation _{o i,t+1} . Specifically, the centralized training of the distributed reinforcement learning task supervisor is carried out through hierarchical progressive modules, including independent Q-value modules and hybrid modules. First, each non-holonomic constrained mobile robot has an independent Q-value module, which uses the recurrent Q network to input the hidden layer state h _i,t-1 of the gate recurrent neural network, the local observation o _i,t , the previous behavior b _i,t-1 , and outputs the local Q value

The mixing module then sums up all the local Q values

Generate joint Q value

as follows

表示独立Q值网络的参数。in,

Represents the parameters of the independent Q-value network.

中。特别地，将从经验池中采样最小回放次

经历以减少数据相关性和提高样本利用率。然后，训练进行到t+1时间步，且训练直至所有的回合T_total完成后停止。分布式强化学习任务监管器通过最小化以下损失进行训练：After sampling at time step t, the experience quadruple (z _t , b _t , r _t , z _t+1 ) is stored in the experience pool

In particular, the minimum number of replays will be sampled from the experience pool

Experience to reduce data correlation and improve sample utilization. Then, training proceeds to t+1 time step, and training stops after all rounds T _total are completed. The distributed reinforcement learning task supervisor is trained by minimizing the following loss:

表示目标网络的参数。in,

Represents the parameters of the target network.

和参考轨迹θ_i,r。Finally, the multi-nonholonomic constrained mobile robot learns a set of optimal distributed behavior priority strategies after centralized training. In actual scenarios, the multi-nonholonomic constrained mobile robot switches behavior priorities according to the learned strategies. Once the behavior priorities of the multi-nonholonomic constrained mobile robot are determined at each sampling time, the reference speed v _i,r and reference trajectory x _i,r can be obtained by formulas (22)-(24) and (2). According to formula (3), the reference speed in the inertial coordinate system can be further calculated

and the reference trajectory θ _i,r .

Step 3: Design of reinforcement learning controller

The position and velocity tracking errors in the inertial coordinate system are defined as

e _p,i =θ _i -θ _i,r , (30)

和

是左右轮的角度，

和

分别是参考位置和参考速度。in,

and

is the angle of the left and right wheels,

and

are the reference position and reference speed respectively.

The differential forms of formulas (30) and (31) can be derived as

是

的微分。in,

yes

The differential of .

The integrated tracking error is defined as follows

Define the value function as follows

表示代价函数，α_V,β_V∈(0,2)是可调整的代价参数，且满足α_V+β_V＝2。in,

Denotes the cost function, α _V , β _V ∈(0,2) are adjustable cost parameters, and satisfy α _V +β _V ＝2.

为最优的跟踪控制策略。因此，最优的值函数可以表示为definition

is the optimal tracking control strategy. Therefore, the optimal value function can be expressed as

表示可容许的控制策略。in,

Represents an admissible control strategy.

By combining equations (32)-(35) and (37), the Hamilton-Jacobi-Bellman (HJB) equation can be derived as

表示V_i ^*相对于e_i的梯度，

和

表示分别为V_i ^*相对于e_p,i和e_v,i的梯度。in,

represents the gradient of _Vi ^* relative to _ei ,

and

Denote the gradients of _Vi ^* with respect to _ep,i and ev _,i respectively.

最优的控制策略

可推导为By solving

Optimal control strategy

It can be deduced as

Furthermore, substituting formula (39) into (38) yields the following equation:

需要求解公式(40)获取

然而，由于多非完整约束移动机器人动力学模型的非线性和不精确，

的解析解难以求取。To implement

We need to solve formula (40) to obtain

However, due to the nonlinearity and inaccuracy of the dynamics model of mobile robots with multiple nonholonomic constraints,

The analytical solution of is difficult to obtain.

Therefore, the optimal value function gradient needs to be decomposed as follows:

和

是正常数，

是自适应补偿项。in,

and

is a normal number,

is the adaptive compensation term.

Substituting formula (41) into (39), we can obtain the following equation:

和

对于

和

未知项f_i(x_i)和V_i ^o可以通过神经网络近似如下：As we all know, neural networks have powerful approximation capabilities. Therefore, given a compact set

and

for

and

The unknown terms _fi ( _xi ) and _Vio can ^be approximated by a neural network as follows:

和

是理想的权重矩阵，w_f和w_V是神经元数量，

和

是基函数向量，

和

是逼近误差，且有界如||δ_f,i||≤ε_f和||δ_V,i||≤ε_V，ε_f和ε_V是正常数。in,

and

is the ideal weight matrix, _wf and _wV are the number of neurons,

and

is the basis function vector,

and

is the approximation error and is bounded such that ||δ _f,i ||≤ε _f and ||δ _V,i ||≤ε _V , where ε _f and ε _V are positive constants.

Then, substituting equations (43) and (44) into (41) and (42), we can obtain the following equations:

和

是未知的，

无法实施。因此，使用一种辨识者-执行者-评论家强化学习算法以学习最优控制策略。However, due to

and

is unknown,

Therefore, an Identifier-Actor-Critic reinforcement learning algorithm is used to learn the optimal control policy.

Specifically, the discriminator neural network is designed to estimate the unknown nonlinear terms as follows:

是f_i(x_i)的估计，

是辨识者神经网络的权重。辨识者神经网络的更新率可设计为：in,

is an estimate of _fi ( _xi ),

is the weight of the identifier neural network. The update rate of the identifier neural network can be designed as:

是正定矩阵，

是设计的辨识者参数。in,

is a positive definite matrix,

is the identifier parameter of the design.

Then, a critic neural network is designed to evaluate the control performance as follows:

是

的估计值，

是评论家神经网络的权重。评论家网络的更新率可设计为in,

yes

The estimated value of

is the weight of the critic neural network. The update rate of the critic network can be designed as

where γ _c,i is the critic’s learning rate.

Finally, the design of the actuator neural network facility control input is as follows:

是执行者神经网络的权重。执行者网络的更新率可设计为in,

is the weight of the actor neural network. The update rate of the actor network can be designed as

where γ _a,i is the learning rate of the executor.

Step 4: Adaptive compensator design

受到对称执行机构饱和约束的限制如下：First, consider the control input

The restrictions subject to the saturation constraints of the symmetric actuators are as follows:

Among them, τ _lim,i >0 is a known threshold.

Secondly, the control input can be divided into two items as follows:

τ _i =τ _0,i +τ _Δ,i , (54)

是标称项，

是补偿项，且满足如下条件：in,

is a nominal term,

is a compensation item and meets the following conditions:

且具有更新率如下Finally, the adaptive compensator is designed as

And has an update rate of

是设计的自适应补偿器参数。in,

are the designed adaptive compensator parameters.

In order to introduce the present invention in detail, a numerical simulation example is given below to demonstrate the effectiveness and superiority of the proposed distributed reinforcement learning behavior control method for multiple non-holonomic constrained mobile robots.

Simulation comparison and analysis

和

多非完整约束移动机器人的重构矩阵分别为Γ₂＝[2/5,0,0；0,0,0；0,0,0]，Γ₃＝[4/5,0,0；0,0,0；0,0,0]，Γ₄＝[3/5,0,0；0,0,0；0,0,0]和Γ₅＝[4/5,0,0；0,0,0；0,0,0]。多非完整约束移动机器人的初始辨识者权重矩阵分别为ω_f,1＝[0.46]_24×2，ω_f,2＝[0.47]_24×2，ω_f,3＝[0.48]_24×2，ω_f,4＝[0.49]_24×2和ω_f,5＝[0.5]_24×2。多非完整约束移动机器人的初始执行者权重矩阵分别为ω_a,1＝[0.93]_24×2，ω_a,2＝[0.95]_24×2，ω_a,3＝[0.96]_24×2，ω_a,4＝[0.97]_24×2和ω_a,5＝[0.99]_24×2。多非完整约束移动机器人的初始评论家权重矩阵分别为ω_c,1＝[0.92]_24×2，ω_c,2＝[0.94]_24×2，ω_c,3＝[0.96]_24×2，ω_c,4＝[0.98]_24×2和ω_c,5＝[1]_24×2。神经网络具有神经元w_f＝24和w_V＝24，中心均匀的分布在范围[-6,6]，宽度μ_i＝2。多非完整约束移动机器人的未知非线性项设置为

多非完整约束移动机器人的网络拓扑结构如图4所示。其他的仿真所使用的参数如图5所示。图6-7对比了分布式强化学习任务监管器学习前后的任务性能，其中分布式强化学习任务监管器学习前会随机选取行为优先级，导致优先级切换频繁且轨迹不光滑，同时未能完成任务目标，而分布式强化学习任务监管器学习后，优先级切换显著减少，轨迹变得光滑，且完成了预定的任务目标。图8-9对比了分布式强化学习任务监管器、分布式有限状态机任务监管器(Distributed Finite State Automata Mission Supervisors，DFSAMSs)、分布式模型预测控制任务监管器(Distributed Model Prediction Control Mission Supervisors，DMPCMSs)和传统强化学习任务监管器(Reinforcement Learning Mission Supervisor，RLMS)的任务性能，其中DFSAMSs的轨迹有强烈的震荡且有时违反了安全距离，DMPCMSs具有最大的算法迭代时间且实时性无法保证，RLMS忽略了群体智能，而DRLMSs维持了良好的任务性能，且保持了较低的迭代时间。图10-12对比了具有输入饱和约束和不具有输入饱和约束分布式行为控制方法的控制性能，其中当不具有输入饱和约束和行为优先级切换发生切换时，控制输入值和控制代价达到了高昂且无法接受的数值，而具有输入饱和约束时，控制输入值始终被维持在一个可接受的范围之内。图13-14对比了分布式强化学习行为控制、有限时间分布式行为控制(finite-time Distributed Behavioral Control，finite-timeDBC)、固有时间分布式行为控制(fixed-time Distributed Behavioral Control，fixed-time DBC)和传统强化学习行为控制(Reinforcement Learning Behavioral Control，RLBC)的控制性能，其中finite-time DBC和fixed-time DBC由于滑模控制的抖振，轨迹会存在一些震荡，RLBC忽略了群体智能导致了一些非常不理想的控制结果，而DRLBC具有光滑的轨迹，且是唯一可以满足控制饱和约束的分布式行为控制方法。所有对比都证明了所提出的分布式行为控制方法的有效性和优越性。The numerical simulation considers five networked multi-nonholonomic constrained mobile robots to form a desired formation and avoid obstacles by performing obstacle avoidance, formation and reconstruction behaviors. The first nonholonomic constrained mobile robot is set as the leader, and its desired formation task function is x _1,d = x ₀ = [t; 0]. The initial positions of the multi-nonholonomic constrained mobile robots are x _1,0 = [0; 0; 0], x _2,0 = [-7; 7; 0], x _3,0 = [-7; -7; 0], x _4,0 = [-12; 12; 0] and x _5,0 = [-12; -12; 0]. The target positions of the multi-nonholonomic constrained mobile robots are _x1,g = [80; 0; 0], _x2,g = [75; 5; 0], _x3,g = [75; -5; 0], _x4,g = [70; 10; 0] and x5 _,g = [70; -10; 0]. The relative positions of the formations of the multi-nonholonomic constrained mobile robots are

and

The reconstruction matrices of the multi-nonholonomic constrained mobile robot are Γ ₂ = [2/5, 0, 0; 0, 0, 0; 0, 0, 0], Γ ₃ = [4/5, 0, 0; 0, 0, 0; 0, 0, 0], Γ ₄ = [3/5, 0, 0; 0, 0, 0; 0, 0, 0] and Γ ₅ = [4/5, 0, 0; 0, 0, 0; 0, 0, 0]. The initial identifier weight matrices of the multi-nonholonomic constrained mobile robot are ω _{f, 1} = [0.46] _{24×2 ,} ω _{f, 2 =} [0.47] _24×2 , ω f, 3 = [0.48] 24×2 , ω _{f, 4} = [0.49] _24×2 and ω _{f, 5} = [0.5] _24×2 . The initial actor weight matrices of the multi-nonholonomic constrained mobile robot are _ωa,1 = [0.93] _24×2 , _ωa,2 = [0.95] _24×2 , _ωa,3 = [0.96] ₂₄ ×2, _ωa,4 = [0.97] _24×2 and _ωa,5 = [0.99] _24×2 . The initial critic weight matrices of the multi-nonholonomic constrained mobile robot are _ωc,1 = [0.92] _24×2 , _ωc,2 = [0.94] _24×2 , _ωc,3 = [0.96] _24×2 , _ωc,4 = [0.98] _24×2 and _ωc,5 = [1] _24×2 . The neural network has neurons w _f = 24 and w _V = 24, with centers uniformly distributed in the range [-6, 6] and width μ _i = 2. The unknown nonlinear terms of the multi-nonholonomic constrained mobile robot are set as

The network topology of multiple nonholonomic constrained mobile robots is shown in Figure 4. The other parameters used in the simulation are shown in Figure 5. Figures 6-7 compare the task performance before and after the distributed reinforcement learning task supervisor is learned. Before learning, the distributed reinforcement learning task supervisor randomly selects the behavior priority, resulting in frequent priority switching and uneven trajectory, and the task goal cannot be completed. After learning, the distributed reinforcement learning task supervisor significantly reduces the priority switching, the trajectory becomes smooth, and the predetermined task goal is completed. Figures 8-9 compare the task performance of the distributed reinforcement learning task supervisor, the distributed finite state machine task supervisor (DFSAMSs), the distributed model prediction control task supervisor (DMPCMSs) and the traditional reinforcement learning task supervisor (RLMS). The trajectory of DFSAMSs has strong oscillations and sometimes violates the safety distance. DMPCMSs has the largest algorithm iteration time and real-time performance cannot be guaranteed. RLMS ignores swarm intelligence, while DRLMSs maintains good task performance and keeps a low iteration time. Figures 10-12 compare the control performance of distributed behavior control methods with and without input saturation constraints. When there is no input saturation constraint and behavior priority switching occurs, the control input value and control cost reach high and unacceptable values, while with input saturation constraints, the control input value is always maintained within an acceptable range. Figures 13-14 compare the control performance of distributed reinforcement learning behavior control, finite-time distributed behavior control (finite-time DBC), fixed-time distributed behavior control (fixed-time DBC) and traditional reinforcement learning behavior control (RLBC). Due to the chattering of sliding mode control, the trajectory of finite-time DBC and fixed-time DBC will have some oscillations. RLBC ignores swarm intelligence and leads to some very undesirable control results. DRLBC has a smooth trajectory and is the only distributed behavior control method that can meet the control saturation constraint. All comparisons prove the effectiveness and superiority of the proposed distributed behavior control method.

Claims (5)

1. A reinforcement learning behavior control method for a mobile robot with multiple nonholonomic constraints, characterized in that it comprises the following steps: Step S1, establishing a kinematic model of a mobile robot with multiple nonholonomic constraints based on a nonholonomic constraint matrix, establishing a dynamic model of a mobile robot with multiple nonholonomic constraints based on the Euler-Lagrange equations, and constructing basic behaviors according to the established kinematic models, and combining the designed basic behaviors into composite behaviors in different priority orders through a null space projection technique; Step S2, modeling the behavior priority switching as a distributed partially observable Markov decision process, setting the reference speed instruction of the composite behavior as the action set of the reinforcement learning algorithm under the framework of the centralized training distributed execution reinforcement learning algorithm, selecting the position and priority of the non-holonomic constrained robot, and the position and priority of its neighboring robots as the observation set of the reinforcement learning algorithm, designing the reward function, and thus constructing the distributed reinforcement learning task supervisor DRLMSs; Step S3, with the goal of balancing control performance and control loss, an identifier-executor-critic reinforcement learning algorithm is introduced to online identify unknown dynamic models, implement control strategies, and evaluate control performance, thereby designing reinforcement learning controllers RLCs; Step S4, based on the adaptive control theory, an adaptive compensator is designed to maintain the optimal control performance and offset the saturation effect in real time. 2. The reinforcement learning behavior control method for multiple nonholonomic constrained mobile robots according to claim 1 is characterized in that step S1 specifically comprises the following steps: Step S11: Kinematic modeling of mobile robots with multiple nonholonomic constraints Consider a group of N (N＞2) nonholonomic constrained mobile robots, where each robot is driven by a differential wheel, i＝1,...,N; the generalized velocity of the i-th nonholonomic constrained mobile robot is expressed as

in,

and

are the linear velocity and angular velocity, respectively.

and

are the linear speeds of the left and right wheels,

is the distance between the left and right wheels,

represents the set of real numbers; Then, the kinematic equation of the i-th nonholonomic constrained mobile robot is expressed as

in,

represents a generalized state,

and

are position and direction,

represents a non-holonomic constraint matrix; In addition, the kinematic equation of the i-th nonholonomic constrained mobile robot in the inertial coordinate system is:

in,

is the wheel radius,

represents the nonholonomic constraint matrix in inertial coordinates,

and

are the rotation speeds of the left and right wheels respectively; Step S12: Dynamics modeling of mobile robots with multiple nonholonomic constraints By using the Euler-Lagrange equations, the dynamic model of the i-th nonholonomic constrained mobile robot is derived as

in,

is the inertia matrix,

is the Coriolis force and centripetal force matrix, G _i ( _xi ) is the gravity matrix,

represents the unknown nonlinear term,

is a designable input gain matrix,

is the control input,

It is not completely binding; First, the differential form of formula (3) is derived as follows

in,

represents the differential of S _i ( _xi ),

is the angular acceleration of the wheel; Then, substitute formulas (3) and (5) into (4) and multiply on the left by

The following equation is obtained

in,

According to Assumption 2, formula (6) can be rewritten as

in,

is the exact term,

is an inexact term; Assumption 1: The multi-nonholonomic constrained mobile robot system works in a static scene, and all non-robot obstacles are static and fixed; Assumption 2: The input gain matrix E _i ( _xi ) always satisfies the design

Step S13: Construction of basic behaviors of mobile robots with multiple nonholonomic constraints Assume that each nonholonomic constrained mobile robot has M basic behaviors, where the kth basic behavior of the ith nonholonomic constrained mobile robot can use a task variable

The mathematical modeling is as follows σ _i,k =gi _,k ( _xi ), (8) in,

Represents the task function; Then, the differential form of the task variable σ _i,k is expressed as

in,

is the Jacobian matrix of the task; Finally, the reference velocity command of the kth basic behavior of the i-th nonholonomic constrained mobile robot can be expressed as

in,

is the right pseudo-inverse of the Jacobian matrix Ji _,k of the task,

is the desired task function,

is the task gain,

is the task error; Without loss of generality, the obstacle avoidance behavior, distributed formation behavior, and distributed reconstruction behavior are designed as follows: Obstacle avoidance behavior: Obstacle avoidance behavior is a local behavior that aims to ensure that the non-holonomic constrained mobile robot avoids obstacles near the path. Its corresponding task function, expected task, and task Jacobian matrix are expressed as:

in,

represents the minimum distance between the i-th nonholonomic constrained mobile robot and the obstacle, d _OA is the safety distance,

is the relative position of the minimum distance,

is the desired direction of the obstacle avoidance behavior, + and − respectively indicate that the obstacle is to the left and right of the i-th nonholonomically constrained mobile robot; Distributed formation behavior: Distributed formation behavior is a distributed cooperative behavior that aims to ensure that multiple nonholonomic constrained mobile robots form the desired formation by only using the states of their neighbors. The corresponding task function, expected task, and task Jacobian matrix are expressed as:

in,

is the estimated state of the distributed formation behavior, which is estimated by designing an adaptive estimator with the following update rate:

Among them, κ _DF is a positive constant,

is the relative position of the formation,

Indicates the status of the navigator.

represents the neighbors of the i-th nonholonomic constrained mobile robot; Distributed Reconfiguration Behavior: Distributed reconfiguration behavior is a distributed cooperative behavior designed to ensure that multiple nonholonomic constrained mobile robots reconstruct the desired formation only by using the states of their neighbors. The corresponding task function, expected task, and task Jacobian matrix are expressed as:

in,

is the estimated state of the distributed formation behavior, which is estimated by designing an adaptive estimator with the following update rate:

Among them, κ _DR is a positive constant,

is the formation reconstruction matrix; Step S14: Construction of composite behaviors of mobile robots with multiple nonholonomic constraints A composite task is a combination of multiple basic behaviors in a certain priority order;

is the task function of the i-th nonholonomic constrained mobile robot, where _km∈NM , _NM ={1,...,M}, _mk represents the dimension of the task space, _and M represents the number of tasks; define a time-related priority function _gi ( _km ,t): _NM ×[0,∞]→ _NM ; and define a task hierarchy with the following rules: 1) A task k _α with priority _gi (k _α ) cannot interfere with task k _β with priority _gi (k _β ) if _gi (k _α ) _≥gi (k _β ),

k _α ≠ k _β ; 2) The mapping from speed to task speed is given by the Jacobian matrix of the task

express; 3) The dimension of the task _m with the lowest priority may be greater than

Therefore, make sure that the dimension m _n is greater than the total dimension of all tasks; 4) The value of g _i (k _m ) is assigned by the task supervisor according to the task requirements and sensor information; By assigning a given priority to the basic tasks, the speed of the composite task at time t is expressed as

in,

is the behavioral priority,

is the augmented Jacobian matrix of the null space projection. 3. The reinforcement learning behavior control method for multiple nonholonomic constrained mobile robots according to claim 1 is characterized in that: the step S2 specifically includes: defining the centralized training environment as ε, and the global state as

in

is the joint location,

is the priority of the union,

is the formation flag, S represents the global state set; define b _i,t ={v _r,i,t }∈B as a local behavior, where B represents the behavior set; define

is an independent local observation, where s _i,t = {x _i ,Pr _i } is a local state,

represents the neighbors of the i-th nonholonomic constrained mobile robot, O represents the local observation set; due to local observation, the local behavior observation history is defined as z _i,t ∈Z, where Z represents the behavior observation history set; all distributed reinforcement learning task supervisors contribute a reward signal, and the reward function is designed as follows r _t = r ₁ + r ₂ , (25)

in,

They represent the identities of no formation, reconstructed formation, and desired formation states, respectively; _r1 and _r2 are reward signals for achieving task goals and reducing behavior switching, respectively; Multiple nonholonomic constrained mobile robots interact with the environment ε at t time steps, where the i-th nonholonomic constrained mobile robot observes a local observation o _i,t , obtains the previous behavior b _i,t-1 , selects a behavior b _i,t according to the ε-greedy strategy with a decay factor γ _ε , and then obtains a team reward r _t and transfers to the next local observation o _i,t+1 ; Specifically, the centralized training of the distributed reinforcement learning task supervisor is carried out through hierarchical progressive modules, including independent Q-value modules and hybrid modules; First, each nonholonomic constrained mobile robot has an independent Q-value module, which uses the recurrent Q network input gate recurrent neural network hidden layer state h _i,t-1 , local observation o _i,t , previous behavior b _i,t-1 , and outputs a local Q-value

The mixing module then sums up all the local Q values

Generate joint Q value

as follows

in,

represents the parameters of the independent Q value network; After sampling at time step t, the experience quadruple (z _t , b _t , r _t , z _t+1 ) is stored in the experience pool

In particular, the minimum number of replays will be sampled from the experience pool

Experience to reduce data correlation and improve sample utilization; then, training proceeds to t+1 time step, and training stops after all rounds T _total are completed; the distributed reinforcement learning task supervisor is trained by minimizing the following loss:

in,

Represents the parameters of the target network; Finally, after centralized training, the multi-nonholonomic constrained mobile robot learns a set of optimal distributed behavior priority strategies; in actual scenarios, the multi-nonholonomic constrained mobile robot switches behavior priorities according to the learned strategies; once the behavior priorities of the multi-nonholonomic constrained mobile robot are determined at each sampling moment, the reference speed v _i,r and reference trajectory x _i,r are obtained through formulas (22)-(24) and (2); according to formula (3), the reference speed in the inertial coordinate system is further calculated

and the reference trajectory θ _i,r . 4. The reinforcement learning behavior control method for multiple nonholonomic constrained mobile robots according to claim 1 is characterized in that: step S3 specifically comprises: defining the position and velocity tracking errors in the inertial coordinate system as: e _p,i =θ _i -θ _i,r , (30)

in,

and

is the angle of the left and right wheels,

and

are the reference position and reference velocity respectively; The differential forms of formulas (30) and (31) are derived as follows:

in,

yes

The differential of The integrated tracking error is defined as follows

Define the value function as follows

in,

represents the cost function, α _V , β _V ∈(0,2) are adjustable cost parameters, and satisfy α _V +β _V ＝2; definition

is the optimal tracking control strategy; therefore, the optimal value function can be expressed as

in,

represents the permissible control strategies; By combining equations (32)-(35) and (37), the Hamilton-Jacobi-Bellman (HJB) equation can be derived as

in,

represents the gradient of _Vi ^* relative to _ei ,

and

denote the gradients of _Vi ^* relative to _ep,i and ev _,i respectively; By solving

Optimal control strategy

It can be deduced as

Furthermore, substituting formula (39) into (38) yields the following equation:

To implement

We need to solve formula (40) to obtain

However, due to the nonlinearity and inaccuracy of the dynamics model of mobile robots with multiple nonholonomic constraints,

The analytical solution of is difficult to obtain; Therefore, the optimal value function gradient needs to be decomposed as follows:

in,

and

is a normal number,

is the adaptive compensation term; Substituting formula (41) into (39), we obtain the following equation:

As we all know, neural networks have powerful approximation capabilities; therefore, given a compact set

and

for

and

The unknown terms _fi ( _xi ) and _Vio ^are approximated by the neural network as follows:

in,

and

is the ideal weight matrix, _wf and _wV are the number of neurons,

and

is the basis function vector,

and

is the approximation error and is bounded such that ||δ _f,i ||≤ε _f and ||δ _V,i ||≤ε _V , ε _f and ε _V are positive constants; Then, substitute equations (43) and (44) into (41) and (42) to obtain the following equations:

However, due to

and

is unknown,

It is not feasible to implement; therefore, an Identifier-Actor-Critic reinforcement learning algorithm is used to learn the optimal control policy; Specifically, the discriminator neural network is designed to estimate the unknown nonlinear terms as follows:

in,

is an estimate of _fi ( _xi ),

is the weight of the identifier neural network; the update rate of the identifier neural network can be designed as:

in,

is a positive definite matrix,

is the identifier parameter of the design; Then, a critic neural network is designed to evaluate the control performance as follows:

in,

yes

The estimated value of

is the weight of the critic neural network; the update rate of the critic network can be designed as

where γ _c,i is the critic’s learning rate; Finally, the design of the actuator neural network facility control input is as follows:

in,

is the weight of the actor neural network; the update rate of the actor network can be designed as

where γ _a,i is the learning rate of the executor. 5. The reinforcement learning behavior control method for multiple nonholonomic constrained mobile robots according to claim 1 is characterized in that: the step S4 specifically comprises: first, considering the control input

The restrictions subject to the saturation constraints of the symmetric actuators are as follows:

Among them, τ _lim,i ＞0 is a known threshold; Secondly, the control input can be divided into two items as follows: τ _i =τ _0,i +τ _Δ,i , (54) in,

is a nominal term,

is a compensation item and meets the following conditions:

Finally, the adaptive compensator is designed as

And has an update rate of

in,

are the designed adaptive compensator parameters.

Images