CN117601120A - Adaptive variable impedance control method and device, electronic equipment and storage medium - Google Patents

Abstract

The embodiment of the invention provides a self-adaptive variable impedance control method and device, electronic equipment and a storage medium. The invention adopts depth deterministic strategy gradient DDPG algorithm under the framework of reinforcement learning. The DDPG algorithm includes an Actor network and a crit ic network, and by continuously learning in actual operation, the DDPG algorithm enables the double mechanical arms to adjust their action strategies so as to maximize the jackpot. In the training stage, samples are randomly selected from an experience pool and used for training an Actor-Cr it ic network to acquire an optimal network structure, and by using the trained Actor-Cr it ic network, the double mechanical arms can make optimal action selection based on the current state data set so as to adapt to uncertain environments and realize efficient grabbing of a target object. The embodiment of the invention allows the double mechanical arms to continuously optimize the behavior of the double mechanical arms through reinforcement learning, thereby being better suitable for complex working scenes.

Description

Adaptive variable impedance control method and device, electronic equipment and storage medium

Technical field

Embodiments of the present invention relate to the field of robot control technology, and in particular, to an adaptive variable impedance control method and device, electronic equipment and storage media.

Background technique

In industrial production, traditional industrial robots use position control to achieve specific tasks in industrial scenarios. However, in applications that require interaction with the environment, traditional position control-based methods will no longer be qualified for the corresponding tasks. In fields such as welding, polishing, and shaft hole assembly, there are a large number of complex contacts with the environment. Only industrial robots move along designated paths. Once there is a position deviation between the robot and the designated path, a very huge environmental contact force will be generated. It may cause damage to the workpiece or even damage the industrial robot. With the complexity and flexibility of the production process, the existing robots working in independent stations can no longer meet the increasingly changing manufacturing needs. In order to adapt to the requirements of complex tasks, intelligent operations and system flexibility in non-structural environments, the two Robots have shown advantages in performing such tasks through mutual cooperation and cooperation. The two-arm robot maintains a certain constraint relationship between its arms during the coordination process to complete the coordination task of the two arms. The basic idea of pure position control is to first plan the trajectory of the operated target object, and obtain the trajectory of the end of the arms through the constraint relationship between the target object and the arms. However, this control method does not consider the force exerted by the arms on the target object, nor does it consider the external interference to the target object. Therefore, how to achieve efficient grabbing of target objects by dual robotic arms in uncertain and complex scenes has become an urgent technical problem to be solved.

Contents of the invention

Embodiments of the present invention provide an adaptive variable impedance control method and device, electronic equipment and storage media, which can enable the robot's dual manipulators to make optimal action selections based on the current state data set to adapt to the uncertain environment and achieve control. Efficient grasping of target objects and allowing the dual robotic arms to continuously optimize their behavior through reinforcement learning to better adapt to complex work scenarios.

In a first aspect, embodiments of the present invention provide an adaptive variable impedance control method, including:

Construct an impedance model of the robot's dual manipulator arms when grabbing the target object, decouple the internal force and external force on the target object, and perform adaptive impedance control on the internal force and the external force respectively;

Initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the experience tuples of the robot in the environment. The impedance model includes an Actor network and a Cr itic network. The Actor network is To generate continuous actions, the Cr itic network is used to evaluate the quality of the action and output the corresponding action value function;

Select an action from the state space of the robot, and after executing the selected action, store the experience tuples fed back by the environment into the experience pool, randomly sample a batch of data from the experience pool, and calculate the The loss of the Cr itic network is backpropagated, the target Q value is calculated through the Cr itic network, the parameters of the Actor network are updated to maximize the Q value, and the training is looped until the preset number of iterations is reached to obtain the trained Actor-Cr it ic network;

Using the trained Actor-Cr itic network, the actions of the dual robotic arms are executed in the actual environment to grab the target object.

In some embodiments, the method further includes:

Establish the coordinate system of the dual manipulator collaborative system. The position and attitude of the target object with respect to the reference coordinate system are solved using the following formula:

In the formula, is the transformation matrix of the target object relative to the center of mass coordinate system;/> It is the 3X3 rotation matrix of the object relative to the coordinate system at the center of mass: /> is the 3X1 position matrix of the target object relative to the coordinate system at the center of mass;

The target object is converted into a constraint between the target object and the robotic arm through the transformation between the coordinate system at the center of mass and the world coordinate system, which is expressed by the following formula:

In the formula, It is the homogeneous coordinate transformation of the coordinate system 0 at the center of mass relative to the world coordinate system W;/> Represents the homogeneous coordinate transformation of the base coordinate system of the dual robot arms relative to the world coordinate system;/> Represents the secondary transformation of the end coordinate system of the dual robotic arms relative to the base coordinates of the dual robotic arms;/> Represents the homogeneous transformation of the coordinate system of the center of mass of the target object relative to the end of the robotic arm:

The speed constraint relationship is analyzed through the following formula, so that the position and speed of the arms are consistent during the movement;

In the formula, Represents the speed of the end of the robotic arm relative to the world coordinate system;/> Represents the object’s velocity relative to the world coordinate system, angular velocity;/> Represents the position transformation matrix of the end of the manipulator arm relative to the world; P _i ^O represents the position transformation matrix of the end of the manipulator arm relative to the center of mass of the target object;/> Represents the direction rotation matrix of the center of mass of the target object relative to the world.

In some embodiments, decoupling the internal and external forces on the target object includes:

According to Newton's second law and Euler's equation, the state of the dual manipulator grabbing the target object is established, and the following dynamic equation of the target object is established:

in the formula I _O represents the inertia matrix at the center of mass of the target object; F _O ∈R ⁶ represents the vector force of the dual robotic arms acting on the target object; M _o ∈R ⁶ represents the mass inertia matrix of the target object;/> represents the linear acceleration and angular acceleration during the movement of the target object; C _O ∈R ⁶ represents the resultant force vector of the Coriolis force, gravity and centrifugal force of the target object; F _ext ∈R ⁶ represents the appropriate amount of external interference force acting on the target object Force; convert the above formula into the following formula:

In the formula, k=l, r represents the left arm and right arm of the double robotic arm, S _k ^T ∈ R ⁶ represents the grasping matrix; F _k represents the force of the robotic arm acting on the target object; decompose the grasping matrix to obtain the external force Formula F _I and the internal force formula F _E are obtained:

in the formula Yes/> The generalized inverse of a matrix.

In some embodiments, the equation of the impedance model is as follows:

In the formula, m is the inertia coefficient, b is the damping coefficient, ε is the adaptive parameter, Δf represents the error value of the force, and/> are the movement speed and movement acceleration of the machine arm respectively.

In some embodiments, a batch of data is randomly sampled from the experience pool, the loss of the Critic network is calculated and backpropagated, the target Q value is calculated through the Critic network, and the parameters of the Actor network are updated. To maximize the Q value, include:

Based on the deterministic gradient strategy, the Actor network parameters are updated according to the action value function. The deterministic behavior strategy is as follows:

μ→a _t =μ(s _t )

Among them, μ is the policy function, s is the current state, and the action of the deterministic policy is uniquely determined in state s, and its formula is as follows:

a _t =μ(s _t |θ ^μ )+N _t

Among them, θ is the policy parameter; during network training, multiple data N are randomly sampled as training data for the deterministic strategy μ. The performance of the deterministic strategy μ is measured as follows:

Among them, s _t ~ ρ ^β represents sampling a state s _t from the empirical distribution ρ ^β , and Q ^μ (s, μ (s)) represents the entire expectation of the Critic network for the given state s and action μ (s). It means taking the expected value on the sampled state and action; during training, the expected value is replaced by the sample mean;

The Actor network learns the optimal strategy by maximizing the output of the Critic network. The parameters of the Actor network are updated through the gradient ascent of the following formula:

Among them, a _t ~ π represents sampling an action a _t from the Actor policy π, Q(s, a|θ ^Q ) represents the value of the Critic network for a given state s and action a, and the gradient term Represents the gradient of the Critic network’s output action on the Actor;

The parameters θ ^Q of the current value network Q are updated according to the minimization loss function L(θ ^Q ). The minimization loss function is as follows:

Among them, N is the number of random samples, and Y is the attenuation coefficient.

In some embodiments, a batch of data is randomly sampled from the experience pool, the loss of the Critic network is calculated and backpropagated, the target Q value is calculated through the Critic network, and the parameters of the Actor network are updated. To maximize the Q value, it also includes:

Based on the deterministic gradient strategy, the gradient strategy of the behavioral strategy network Q is calculated according to the following formula, and the Critic network parameters are updated:

Among them, s _t and a _t are the states and actions sampled from the empirical distribution ρ ^β , s _t+1 is the next state sampled from the empirical distribution ε, Q (s _i , a _i |θ ^Q ) is The output of the Critic network represents the estimated value of taking action a in state s. y is the target Q value. The goal of gradient estimation is to minimize the mean square error between the output of the Critic network and the target Q value; obtained by using sampling State, action and next state, estimate the expected mean square error, and then calculate the gradient of the Critic network parameter θ ^Q to update the parameters;

y _i =r _i +γQ'(s _i+1 , μ'(s _i+1 |θ ^μ' )|θ ^Q' )

Among them, r _i is the current reward, γ is the discount factor, Q' and μ' are the target network;

The sliding average method is used to update μ′ and Q′, as shown in the following formula:

θ ^μ′ ←τθ+(1-τ)θ ^μ′

θ ^Q′ ←τθ+(1-τ)θ ^Q′

Among them, θ ^μ′ , θ ^Q′ , and τ are all parameters, and τ is the learning rate, with a value of 0.001; in the network architecture, the policy network Actor is used to update θ ^μ′ to output action a, and the value network Critic adopts the correct parameters θ ^Q′ is updated to approximate the state-behavior value function Q ^π (s, a).

In some embodiments, the state space s of the collaborative motion process of the dual manipulators is defined as follows:

s={e _f , e _x , F _a , x _a }

Among them, e _f represents the force tracking error, e _x represents the trajectory tracking error, F _a represents the actual force during the control process of the dual manipulator, and x _a represents the actual trajectory during the control process of the dual manipulator;

The goal of the DDPG algorithm is to output appropriate adaptive parameters ε based on the force tracking error e _f and the trajectory tracking error e _x ; as the output parameter of the DRL algorithm, the designed action space only contains one certain element, which represents the The action at time is the adaptive parameter ε, and the action function is as follows:

a _t ={ε}

Among them, ε is an adaptive parameter; in the process of manipulator position and force tracking, it is necessary to conduct real-time evaluation of each time step. According to the actual force and the average value of the force error/> The difference and the actual trajectory/> and the average trajectory error/> The sum of the differences at different proportions is used as part of the reward and punishment function, which is called the basic reward and punishment part; let the force tracking error e _f and the trajectory tracking error e _x always remain 0; let the force tracking error e _f and the trajectory tracking error e _x As part of the reward and punishment function, it is called the additional incentive part r _extre ; different additional rewards and punishments are given according to the interval of the force tracking error. The reward and punishment function composed of the basic reward and punishment part and the additional incentive part is as follows:

in,

Among them, k _a and k _b are scaling factors, H is the time period of the training process, is the average value of the trajectory error at time j,/> is the average force error at time j,/> is the actual force of the two robotic arms at time j,/> For the actual position of the dual manipulator arms at time j, in the additional incentive part, different additional rewards and punishments are given according to the intervals between e _f and e _x .

In a second aspect, embodiments of the present invention also provide an adaptive variable impedance control device, which includes:

A construction module used to construct an impedance model of the robot's dual manipulator arms when grabbing the target object, decouple the internal force and external force on the target object, and perform adaptive impedance control on the internal force and the external force respectively;

An initialization module, used to initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the experience tuples of the robot in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the actions and output the corresponding action value function;

A training module, used to select actions from the state space of the robot, and after executing the selected actions, store the experience tuples fed back by the environment into the experience pool, and randomly sample a batch of them from the experience pool. data, calculate the loss of the Critic network and perform backpropagation, calculate the target Q value through the Critic network, update the parameters of the Actor network to maximize the Q value, and loop training until the preset number of iterations is reached, and the training is obtained Actor-Critic network;

An execution module is used to use the trained Actor-Critic network to execute the actions of the dual robotic arms to grab the target object in the actual environment.

In a third aspect, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program The program implements the adaptive variable impedance control method as described in the first aspect.

In a fourth aspect, embodiments of the present invention also provide a computer-readable storage medium that stores computer-executable instructions, and the computer-executable instructions are used to execute the adaptive variable impedance control method as described in the first aspect.

According to the adaptive variable impedance control method and device, electronic equipment and storage medium provided by embodiments of the present invention, the adaptive variable impedance control method includes: constructing an impedance model of the robot's dual manipulator arms when grabbing the target object, and converting the target object to Decouple the internal and external forces received, and perform adaptive impedance control on internal and external forces respectively; initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the experience tuples of the robot in the environment. Among them, the impedance model includes Actor Network and Critic network, the Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the action and output the corresponding action value function; select an action from the robot's state space, and after executing the selected action, the experience element of the environment feedback The group is stored in the experience pool, and a batch of data is randomly sampled from the experience pool, the loss of the Critic network is calculated and backpropagated, the target Q value is calculated through the Critic network, the parameters of the Actor network are updated to maximize the Q value, and loop training Until the preset number of iterations is reached, the trained Actor-Critic network is obtained; the trained Actor-Critic network is used to execute the robot's actions in the actual environment to grab the target object. Based on this, when constructing the impedance model of the dual manipulator when grabbing the target object, the response of the manipulator when it contacts the environment will be considered. Therefore, the force on the target object is decomposed, the acceptance of the target object is decoupled, and the internal force and External forces are respectively controlled by adaptive impedance to optimize the overall control strategy and improve control accuracy. Under the framework of reinforcement learning, the embodiment of the present invention adopts the deep deterministic policy gradient DDPG algorithm. The DDPG algorithm includes Actor network and Critic network. Actor neural network is used to output continuous actions, and Critic neural network is used to evaluate the quality of actions. By continuously learning in actual operations, the DDPG algorithm enables the dual robotic arms to adjust their action strategies to maximize cumulative rewards. In the training phase, samples are randomly selected from the experience pool and used to train the Actor-Critic network to obtain the optimal network structure. By using the trained Actor-Critic network, the dual robot arms can make the best decision based on the current state data set. Optimal action selection to adapt to uncertain environments and achieve efficient grasping of target objects. Embodiments of the present invention allow dual robotic arms to continuously optimize their behavior through reinforcement learning, thereby better adapting to complex work scenarios.

Description of drawings

Figure 1A is a flow chart of an adaptive variable impedance control method provided by an embodiment of the present invention;

Figure 1B is a schematic diagram of the coordinate system of the two-arm collaboration system provided by an embodiment of the present invention;

Figure 2 is a force analysis diagram of a target object provided by an embodiment of the present invention;

Figure 3 is a structural diagram of a DDPG network model provided by an embodiment of the present invention;

Figure 4 is a diagram of the change process of cumulative reward R during model training provided by an embodiment of the present invention;

Figure 5 is a structural diagram of the deep reinforcement learning adaptive variable impedance control strategy provided by an embodiment of the present invention;

Figure 6 is a control block diagram of dual robotic arms provided by an embodiment of the present invention;

Figure 7 is a flow chart of dual-arm collaborative adaptive impedance control based on deep reinforcement learning provided by an embodiment of the present invention;

Figure 8 is a hardware connection structure diagram of the experimental platform provided by an embodiment of the present invention;

Figure 9A is a constant force tracking diagram under deep reinforcement learning adaptive impedance control provided by an embodiment of the present invention;

Figure 9B is a constant trajectory tracking diagram under deep reinforcement learning adaptive impedance control provided by an embodiment of the present invention;

Figure 10A is a constant force tracking diagram under deep reinforcement learning adaptive impedance control provided by an embodiment of the present invention;

Figure 10B is a variable trajectory tracking diagram under deep reinforcement learning adaptive impedance control provided by an embodiment of the present invention;

Figure 11A is a variable force tracking diagram under deep reinforcement learning impedance control provided by an embodiment of the present invention;

Figure 11B is a variable trajectory tracking diagram under deep reinforcement learning impedance control provided by an embodiment of the present invention;

Figure 12 is a schematic diagram of an adaptive variable impedance control device provided by an embodiment of the present invention;

Figure 13 is a schematic diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

It should be noted that although the functional modules are divided in the device schematic diagram and the logical sequence is shown in the flow chart, in some cases, the modules can be divided into different modules in the device or the order in the flow chart can be executed. The steps shown or described. The terms "first", "second", etc. in the description, claims, and following drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

In the embodiments of the present invention, words such as "further", "exemplarily" or "optionally" are used as examples, illustrations or illustrations, and should not be interpreted as being more preferable or better than other embodiments or designs. Advantages. The use of the words "further," "exemplarily," or "optionally" is intended to present the relevant concepts in a specific manner.

First, some terms involved in this invention are analyzed:

Dual robotic arm collaborative control: refers to two robotic arms or robots, usually working collaboratively in a system environment to achieve a common target task. This design includes synchronous motion, force and torque control, collision detection and avoidance, trajectory planning, etc. The goal of dual robotic arm collaborative control is to achieve efficient task execution, improve production efficiency, reduce human intervention, and improve task accuracy and safety.

Adaptive variable impedance control of deep reinforcement learning: is a machine learning method designed to train an agent (usually a robot or control system) to learn optimal behavioral strategies while interacting with the environment. This learning is guided by reward and punishment signals, and the agent's goal is to maximize long-term rewards. The parameters of a control system can be automatically adjusted to accommodate changes or uncertainties in the dynamic nature of the system. In adaptive control, the control system can adjust its control strategy in real time to improve performance.

In order to more conveniently describe the working principles of the embodiments of the present invention later, an introduction to relevant technical scenarios will be given below.

In industrial production, traditional industrial robots use position control to achieve specific tasks in industrial scenarios. However, in applications that require interaction with the environment, traditional position control-based methods will no longer be qualified for the corresponding tasks. In fields such as welding, polishing, and shaft hole assembly, there are a large number of complex contacts with the environment. Only industrial robots move along designated paths. Once there is a position deviation between the robot and the designated path, a very huge environmental contact force will be generated. It may cause damage to the workpiece or even damage the industrial robot. With the complexity and flexibility of the production process, the existing robots working in independent stations can no longer meet the increasingly changing manufacturing needs. In order to adapt to the requirements of complex tasks, intelligent operation and system flexibility in non-structural environments, the two Robots have shown advantages in performing such tasks through mutual cooperation and cooperation. The two-arm robot maintains a certain constraint relationship between its arms during the coordination process to complete the coordination task of the two arms. The basic idea of pure position control is to first plan the trajectory of the operated target object, and obtain the trajectory of the end of the arms through the constraint relationship between the target object and the arms. However, this control method does not take into account the force exerted by the arms on the target object, nor does it consider the external interference to the target object. Therefore, how to achieve efficient grabbing of target objects by dual robotic arms in uncertain and complex scenes has become an urgent technical problem to be solved.

Based on this, the present invention provides an adaptive variable impedance control method and device, electronic equipment and storage media. Among them, the adaptive variable impedance control method includes: constructing the impedance model of the robot's dual manipulators when grabbing the target object, decoupling the internal and external forces on the target object, and performing adaptive impedance control on the internal and external forces respectively; initializing the impedance The network parameters and experience pool of the model. The experience pool is used to store the experience tuples of the robot in the environment. The impedance model includes the Actor network and the Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to evaluate the quality of the actions. Output the corresponding action value function; select an action from the robot's state space, and after executing the selected action, store the experience tuples of environmental feedback into the experience pool, and randomly sample a batch of data from the experience pool to calculate the Critic network Loss and backpropagation, calculate the target Q value through the Critic network, update the parameters of the Actor network to maximize the Q value, and loop training until the preset number of iterations is reached to obtain the trained Actor-Critic network; use the trained Actor- Critic network performs robot actions to grab target objects in the actual environment. Based on this, when constructing the impedance model of the dual manipulator when grabbing the target object, the response of the manipulator when it contacts the environment will be considered. Therefore, the force on the target object is decomposed, the acceptance of the target object is decoupled, and the internal force and External forces are respectively controlled by adaptive impedance to optimize the overall control strategy and improve control accuracy. Under the framework of reinforcement learning, the embodiment of the present invention adopts the deep deterministic policy gradient DDPG algorithm. The DDPG algorithm includes Actor network and Critic network. Actor neural network is used to output continuous actions, and Critic neural network is used to evaluate the quality of actions. By continuously learning in actual operations, the DDPG algorithm enables the dual robotic arms to adjust their action strategies to maximize cumulative rewards. In the training phase, samples are randomly selected from the experience pool and used to train the Actor-Critic network to obtain the optimal network structure. By using the trained Actor-Critic network, the dual robot arms can make the best decision based on the current state data set. Optimal action selection to adapt to uncertain environments and achieve efficient grasping of target objects. Embodiments of the present invention allow dual robotic arms to continuously optimize their behavior through reinforcement learning, thereby better adapting to complex work scenarios.

The embodiments of the present invention will be further described below with reference to the accompanying drawings.

As shown in FIG. 1A , FIG. 1A is a flow chart of an adaptive variable impedance control method provided by an embodiment of the present invention. The adaptive variable impedance control method may include but is not limited to steps S101 to S104.

Step S101: Construct an impedance model of the robot's dual manipulator arms when grabbing the target object, decouple the internal and external forces on the target object, and perform adaptive impedance control on the internal and external forces respectively;

Step S102: Initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the experience tuples of the robot in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions, and the Critic network is used to Evaluate the quality of the action and output the corresponding action value function;

Step S103: Select an action from the robot's state space. After executing the selected action, store the experience tuples of environmental feedback into the experience pool, randomly sample a batch of data from the experience pool, calculate the loss of the Critic network and perform inverse feedback. To propagate, calculate the target Q value through the Critic network, update the parameters of the Actor network to maximize the Q value, and loop training until the preset number of iterations is reached to obtain the trained Actor-Critic network;

Step S104: Use the trained Actor-Critic network to execute the actions of the dual robotic arms to grab the target object in the actual environment.

In one embodiment, the motion trajectory planning of a single manipulator is based on the estimation of the operated target through kinematic transformation of the coordinate system. Therefore, the coordinate system of the dual-arm collaborative system established is shown in _Figure 1B. In Figure 1B, {W _X , W _Y , W _Z }, _{{ O system} ; _{ OR _{X ,} OR _Y , OR _Z }, _{{ OL} , R _Z } _{, { L} Conversion formula 1.

For the target object being operated, the position and attitude of the target object relative to the reference coordinate system are solved using Equation 1

In the formula, is the transformation matrix of the target object relative to the center of mass coordinate system;/> It is the 3X3 rotation matrix of the target object relative to the coordinate system at the center of mass;/> is the 3X1 position matrix of the target object relative to the coordinate system at the center of mass.

The target object is converted into a constraint between the target object and the robotic arm through the transformation between the coordinate system at the center of mass and the world coordinate system, which is expressed by the following equation 2:

In the formula, It is the homogeneous coordinate transformation of the coordinate system 0 at the center of mass relative to the world coordinate system W;/> Represents the homogeneous coordinate transformation of the base coordinate system of the dual robot arms relative to the world coordinate system;/> Represents the secondary transformation of the end coordinate system of the dual robotic arms relative to the base coordinates of the dual robotic arms;/> Represents the homogeneous transformation of the coordinate system of the center of mass of the target object relative to the end of the manipulator arm.

After passing the above position constraints of the arms, in order to achieve real-time collaborative operation in the true sense, in addition to the position constraints, it is also necessary to ensure the speed consistency of the arms during the movement. Therefore, it is necessary to pair the speed with Equation 3. The constraint relationship is analyzed to maintain the consistency of position and speed of the arms during movement and achieve coordinated operation.

In the formula, Represents the speed of the end of the robotic arm relative to the world coordinate system;/> Indicates the speed and angular velocity of the target object relative to the world coordinate system;/> Represents the position transformation matrix of the end of the manipulator arm relative to the world; P _i ^O represents the position transformation matrix of the end of the manipulator arm relative to the center of mass of the target object;/> Represents the direction rotation matrix of the center of mass of the target object relative to the world.

The force analysis of the target object is shown in Figure 2. In Figure 2, F _L , F _R , F _ext , τ _L , τ _R , and τ _ext are respectively expressed as the dual mechanical arms and external forces acting on the surface of the target object. According to Newton's second law and Euler's equation, the state of the dual manipulator grasping the object is established to establish the following dynamic equation of the target object:

Simplified to equation (5):

in the formula I _O represents the inertia matrix at the center of mass of the target object; F _O ∈R ⁶ represents the vector force of the dual robotic arms acting on the object; M _O ∈R ⁶ represents the mass inertia matrix of the target object;/> represents the linear acceleration and angular acceleration during the movement of the target object; C _O ∈R ⁶ represents the resultant force vector of the Coriolis force, gravity and centrifugal force of the target object; F _ext ∈R ⁶ represents the appropriate amount of external interference force acting on the target object Force transforms equation (5) into equation (6).

In the formula, k=l, r represents the left arm and right arm of the double robotic arm, S _k ^T ∈ R ⁶ represents the grasping matrix; F _k represents the force of the robotic arm acting on the target object. The six-dimensional force sensor can directly Data collection. The grasping matrix can be decomposed to obtain the external force formula (7) and the internal force formula (8):

in the formula Yes/> The generalized inverse of a matrix.

In one embodiment, traditional impedance control implements a control strategy of adjusting force based on position error, and admittance control implements a control strategy of adjusting position based on force error. Therefore, position-based impedance control is also called admittance control. The impedance/admittance control strategy relies on the second-order system of "mass-impedance-spring". The dynamics of the target object is established using the second-order system of "mass-impedance-spring". The model of the impedance control strategy is Equation (18) , the present invention decomposes the dual robot arm cooperation target object task into an inner ring and an outer ring. The inner ring mainly controls the internal force F _I to prevent the robot arm from damaging the target object, and the outer ring controls the external force F _E to ensure that the target object completes the cooperation task. From equations (7) and (8), we can know that the force acting on the target object by the double manipulator can be divided into internal force and external force, so equation (9) can be transformed into equation (13).

In the formula, x _{e and} x _a represent the expected trajectory and the actual trajectory respectively; F _e and F _a represent the expected force and the actual force respectively, where F _k = F _a ; m _d represents the inertia matrix; b _d represents the damping matrix; k _d represents the stiffness matrix; Δf represents the error value of the force.

F _a =k _e (x _e -x _a ) (10)

In the formula, k _e is K _e and is a one-dimensional quantity.

Put equation (10) into equation (9) to get equation (11):

F _e -k _e (x _e -x _a )=Δf (11)

From formula (11), x _e can be obtained as:

Substituting formula (12) into formula (9), we can get formula (13):

Using Laplace transformation on equation (13), we get:

[ms ² +bs+k _d +K _e ]ΔF(s)＝[ms ² +bs+k _d ][F _d (s)+k _e (x _e (s)-x _r (s))] ( 14)

Therefore, the force tracking steady-state error is shown in Equation (15):

It can be seen from equation (15) that there are two methods to make the force tracking steady-state error Δf _ss = 0. The first method is shown in equation (16). Since the stiffness k _e and position x _e are unknown in an uncertain environment, it is difficult to achieve Δf _ss = 0, and equation (16) is not suitable. The second method is shown in equations (15) and (17). When k = 0, Δf _ss = 0. Also because the environment is unknown, it is difficult to accurately obtain the desired trajectory x _r , so the initial operating environment x _e will be used. Replace x _r and let e=xx _e to get the improved equation, as shown in Equation (18).

According to the actual environment, appropriate inertia coefficient m and damping coefficient b should be selected to achieve stable force tracking requirements.

In an uncertain environment, considering that the operating environment trajectory x _e and the environmental stiffness k _e are unknown, all unknown parameters in equation (18) are combined to obtain (19).

Set a new adaptive parameter ε to replace the unknown parameter, as shown in Equation (20).

The adaptive impedance equation is obtained, as shown in equation (21):

In order to optimize the impedance parameters and impedance parameter sets, the force tracking control process of the manipulator is calculated as a reinforcement learning framework based on Markov decision process (MDP) suitable for different environments, which is IIC (Intelligent lmpedance Control, intelligent Impedance control) method designs appropriate state and action values and constructs a reward function, while adding a safe learning mechanism to it. The Markov decision process is the basic framework of RL (an interactive goal-oriented learning method), which includes (1) state set S; (2) action set A; (3) state transition probability (4)Reward probability/> (5) Attenuation factor γ; (6) online neural network parameters θ ^Q and θ ^μ of actor/critic.

The basic process of RL is as follows. The agent interacts with the environment at time t, observes a certain characteristic of the environment state S _t ∈ S, and outputs an action A _t ∈ A based on P. At the next time t+1, based on the result of the action, the Agent obtains an immediate reward and punishment R _t+1 ∈R, and enters a new state S _t+1 , and then the cycle continues. The above process is called the policy π of the RL method, which can be expressed by Equation (22), which refers to the distribution on the action set in state s.

π(a|s)：S→A (22)

Among them, a is the specific value of an action A _t in the action set A, and s is the specific value of an action S _t in the action set S.

Multiple learning sequences can be obtained through multiple interactions between the environment and the robotic arm. Taking the average of the rewards and punishments of a certain state (the attenuated weighted sum of all rewards starting from a certain moment) is the expected return of the state, or That is, its value; taking the average return of taking a unified action in a certain state is the expected return of the state-behavior pair, that is, the value of the state-behavior pair. Theoretically, through continuous learning, the values of all states and state-behavior pairs can be obtained, so the state value function and behavior value function can be obtained.

According to the definition, the cumulative reward and punishment (R _t ) can be recorded as equation (23).

Among them, r is the reward (reward), and the strategy π element is random, so the cumulative reward and punishment R _t also has a variety of random possible values. In order to evaluate the state s, it is necessary to define a certain value that can describe the state s. However, the cumulative reward and punishment R _t is a random value and cannot be used to describe the state s. Therefore, the expected value of the cumulative reward and punishment R _t is used as the state value function v π (s). The definition of the value function v ^π (s) is as shown in Equation (24).

v ^π (s)＝E _π [R _t |s _t =s] (24)

Correspondingly, the value of performing action a in state s under policy π is called the state-behavior value function, which is defined as Equation (25).

Q ^π (s, a) = E _π [R _t |s _t =s, a _t =a] (25)

According to equation (24), the Bellman equation of the state value function is obtained. This equation expresses the relationship between the value of the current state s and the value of the subsequent state s′, as shown in equation (26).

The strategy is the mapping from state space to action space, which is shown in Equation (27).

a=π(a|s) (27)

Therefore, by combining equations (25) and (27), we can obtain another expression of the state-behavior value function Q ^π (s, a), as shown in equation (28).

Based on the adaptive impedance control system and Markov decision-making, DRL (Deep Reinforcement Learning) is used to solve complex calculations in high-dimensional dynamic environments, and the DDPG (DeepDeterministic Policy Gradient) algorithm is used to solve the impedance control strategy of the IIC method. DDPG is one of the DRL algorithms. It is based on the Actor-Critic architecture and has the characteristics of both value function update and strategy update. It has certain advantages in solving problems in continuous action space and is suitable for force tracking control problems of robotic arms.

Based on the deterministic gradient strategy, the Actor neural network parameters are updated according to the predicted action value function value, and the deterministic behavior strategy is as shown in Equation (29).

μ→a _t =μ(s _t ) (29)

Among them, μ is the policy function, s is the current state, and the action of the deterministic policy is uniquely determined in state s, and its formula is as follows.

a _t =μ(s _t |θ ^μ )+N _t (30)

Among them, θ is the strategy parameter. During network training, multiple data N will be randomly sampled as training data for the deterministic strategy μ. The measurement of the deterministic strategy μ is shown in Equation (31).

Among them, s _t ~ ρ ^β represents sampling a state s _t from the empirical distribution ρ ^β , and Q ^μ (s, μ(s)) represents the value of the Critic network for a given state s and action μ(s). entire expectation Indicates taking the expected value on the sampled state and action. During training, this expected value is estimated by averaging a batch of samples, i.e. replacing the expected value with the sample mean.

The Actor network learns the optimal policy by maximizing the output of the Critic network. The parameters of the Actor network are updated through the gradient ascent of Equation (32):

Among them, a _t ~ π represents sampling an action a _t from the Actor policy π, Q(s, a|θ ^Q ) represents the value of the Critic network for a given state s and action a, and the gradient term Represents the gradient of the Critic network's output action on the Actor. This gradient shows how if the parameters of the Actor network are slightly changed, the corresponding action will affect the Critic network's estimate of the value function.

The parameters θ ^Q of the current value network Q are updated according to the minimization loss function L(θ ^Q ), which is shown in Equation (33):

Among them, N is the number of random samples, and Y is the attenuation coefficient.

Based on the deterministic gradient strategy, the gradient strategy of the behavioral strategy network Q is calculated according to Equation (34), and the Critic neural network parameters are updated.

Among them, s _t and a _t are the states and actions sampled from the empirical distribution ρ ^β , s _t+1 is the next state sampled from the empirical distribution ε, Q (s _i , a _i |θ ^Q ) is The output of the Critic network represents the estimated value of taking action a in state s. y is the target Q value, which is usually calculated by the target network. The goal of this gradient estimation is to minimize the mean square error between the output of the critic network and the target Q value. By using the sampled state, action and next state, the expected mean square error is estimated, and then the gradient of the Critic network parameter θ ^Q is calculated to update the parameters.

y _i =r _i +γQ'(s _i+1 , μ'(s _i+1 |θ ^μ' )|θ ^Q' ) (35)

Among them, r _i is the current reward, γ is the discount factor, Q' and μ' are the target network.

The target network is a copy of the behavioral network, and the sliding average method is used to update μ′ and Q′, as shown in Equation (36):

θ ^μ′ ←τθ+(1-τ)θ ^μ′

θ ^Q′ ←τθ+(1-τ)θ ^Q′ (36)

Among them, θ ^μ′ , θ ^Q′ and τ are all parameters, among which τ is the learning rate, which generally takes a value of 0.001. In the network architecture, the policy network Actor is used to update θ ^μ′ to output action a, and the value network Critic updates the parameter θ ^Q′ to approximate the state-behavior value function Q ^π (s, a).

Based on the DDPG algorithm, its network structure model is shown in Figure 3.

The definition of the state space s of the collaborative motion process of the double manipulator is shown in Equation (37).

s＝{e _f , e _x , F _a , x _a } (37)

Among them, e _f represents the force tracking error, e _x represents the trajectory tracking error, F _a represents the actual force during the control process of the dual manipulator, and x _a represents the actual trajectory during the control process of the dual manipulator.

The goal of the DDPG algorithm is to output appropriate adaptive parameters ε based on the force tracking error e _f and the trajectory tracking error _ex . Therefore, as the output parameter of the DRL algorithm, the designed action space only contains one certain element. At time t, it represents the action at that time, which is the adaptive parameter ε. The action function is shown in Equation (38).

a _t ={ε} (38)

Among them, ε is the adaptive parameter.

In the process of tracking the position and force of the manipulator, it is necessary to conduct real-time evaluation of each time step. According to the actual force and the average value of the force error/> The difference and the actual trajectory/> and the average trajectory error/> The sum of the differences at different proportions is used as part of the reward and punishment function, which is called the basic reward and punishment part. The goal of the adaptive impedance control system is to achieve good tracking effects, that is, to keep the force tracking error e _f and the trajectory tracking error _ex as zero as possible. Therefore, the force tracking error e _f and the trajectory tracking error e _x are regarded as part of the reward and punishment function, which is called the additional excitation part (r _extre ), as shown in Equation (40). In addition, in order to speed up the training efficiency of the DRL model, different additional rewards and penalties will be given according to the interval of the force tracking error. The reward and punishment function composed of the basic reward and punishment part and the additional incentive part is shown in Equation (39).

in,

Among them, k _a and k _b are scaling factors, H is the time period of the training process, is the average value of the trajectory error at time j,/> is the average force error at time j,/> is the actual force of the two robotic arms at time j,/> For the actual position of the dual manipulator arms at time j, in the additional incentive part, in order to speed up the training efficiency of the DRL model, different additional rewards and punishments will be given according to the intervals between e _f and e _x . The present invention uses the Adam gradient descent algorithm as the optimizer of the Actor network. The Adam optimizer is a gradient descent algorithm widely used in deep learning. It has the characteristics of adaptive learning rate and can effectively handle different gradient scales of different parameters, thereby improving training efficiency. The Adam optimizer is a robust and efficient optimizer that can usually achieve good results in many deep learning tasks. Its adaptive learning rate feature makes the network easier to converge.

The model is trained, and the training results are shown in Figure 4. For about the first 100 rounds, the algorithm is in the strategy exploration stage, and the cumulative rewards and punishments increase rapidly. After 100 rounds, the cumulative reward and penalty curve begins to converge, indicating that a better strategy has been found.

Based on the Markov decision process, basic adaptive impedance control is combined with the DDPG algorithm to form an adaptive variable impedance control for deep reinforcement learning of the reference model, as shown in Figure 5.

In one embodiment, when the two arms cooperate to hold a common target object that interacts with the ring, the target object will be acted upon by internal and external forces. Based on the above deep reinforcement learning adaptive impedance control strategy, the present invention designs a dual robotic arm deep reinforcement learning adaptive impedance control strategy. The main control block diagram is shown in Figure 6. Figure 6 shows the symmetrical control strategy of the dual robotic arms and the dual robotic arms. External impedance control and internal impedance control strategies of the robotic arm control system.

In Figure 6, F _Ea and F _Ee are represented as the actual external force and the expected external force respectively. F _Ia and F _Ie respectively represent the actual internal force and the expected internal force. σF _E and σF _I respectively represent the error of the actual external force and the expected external force and the error of the actual internal force and the expected external force. σX _E , σX _I represent the position compensation generated by the external deep reinforcement learning adaptive impedance controller and the internal force deep reinforcement learning adaptive impedance controller respectively. X _c and X _e respectively represent the actual trajectory of the target object and the expected trajectory of the target object. X _a , X _{al ,} and θ _l and θ _r are respectively expressed as the actual motion angles of the left and right robot arm joints generated by inverse kinematics through the actual motion trajectories of the end effectors of the left and right robot arms.

In one embodiment, an impedance model of a dual robotic arm grasping a target object is constructed;

When constructing the impedance model of the dual manipulator when grabbing the target object, the present invention first considers the response of the manipulator when it contacts the environment, so it decomposes the force on the target object, decouples the acceptance of the target object, and analyzes the internal and external forces. Adaptive impedance control is performed respectively to optimize the overall control strategy and improve control accuracy.

When the dual manipulator arms contact the target object or the target environment, the parameters of the manipulator arm and the control strategy will change. This includes parameters such as the stiffness and damping of the manipulator arm, which are used to describe its movement and force characteristics during the grasping process. .

In order to train the behavior of the robotic arm in an uncertain environment, the present invention first needs to collect a state data set of the robotic arm during actual operation. This includes information such as the current position, speed, and acceleration of the robotic arm, as well as sensor data such as force and torque when the robotic arm comes into contact with the environment.

Under the framework of reinforcement learning, the present invention selects the deep deterministic policy gradient (DDPG) algorithm. This algorithm consists of two key parts: the Actor neural network, which outputs continuous actions, and the Critic neural network, which evaluates the quality of the actions. By continuously learning during actual operations, the DDPG algorithm enables the robotic arm to adjust its action strategy to maximize cumulative rewards.

During the training phase, the present invention establishes an experience pool, which contains the current state of the robotic arm, the actions performed, the rewards obtained, and the next state. This invention randomly selects samples from the experience pool and uses them to train the Actor-Critic neural network to obtain the optimal network structure.

Ultimately, by using a well-trained Actor-Critic neural network, the robotic arm is able to make optimal action selections based on the current state data set to adapt to the uncertain environment and achieve efficient grasping of target objects. The invention allows the robotic arm to continuously optimize its behavior through reinforcement learning, thereby better adapting to complex work scenarios.

The present invention uses the DDPG (Deep Deterministic Policy Gradient) algorithm to solve the deep reinforcement learning algorithm for a two-arm robot to grasp target objects in a continuous action space. The entire algorithm process is shown in Figure 7:

First, in the initialization phase, the network parameters are initialized. This includes the Actor network and the Critic network, which are used to output continuous actions and evaluate the quality of actions respectively. At the same time, an experience pool is set up, which is used to store the robot's experience in the environment, including tuples of states, actions, rewards, and next states.

Secondly, clearly define the robot's state and action space to ensure that they can cope with continuous operations in the actual environment. The Actor network is designed to generate continuous actions, while the Critic network is used to evaluate the quality of actions and output the corresponding action value function.

In order to improve the stability of training, the target network is introduced, which is a replicated version of the Actor and Critic networks. The parameters of the target network are regularly updated at a certain ratio to alleviate the instability problem in training.

Then, define the optimizer and loss function. Select appropriate optimizers for the Actor and Critic networks respectively, usually using the Adam optimizer, and define the mean square error loss function of the Critic network to measure the difference between the estimated value and the target value.

The setting of hyperparameters is also an important step, including learning rate, discount factor, soft update parameters, etc. The choice of these hyperparameters directly affects the convergence and performance of the algorithm.

The training phase is the core of the algorithm. At each time step, the robot selects an action from the state space, generates the action through the Actor network and adds some exploratory noise. After performing the selected action, observe feedback from the environment, including rewards and next states, and store these experience tuples into an experience pool. Then, a batch of data is sampled from the experience pool, the loss of the Critic network is calculated and backpropagated to optimize the network parameters. The target Q value is calculated through the target Critic network, the parameters of the Actor network are updated to maximize the Q value, and finally a soft update is performed to adjust the parameters of the target network. This training loop continues to iterate, gradually optimizing the Actor and Critic networks.

After training is completed, the robot can use the trained Actor network to perform actions and enter the actual environment. In order to ensure the effectiveness of the algorithm, the performance of the robot in the actual environment can be regularly evaluated in order to adjust and optimize the hyperparameters and ensure the robustness and generalization of the algorithm. Through this process, the DDPG algorithm enables the robot to adapt to complex and continuous action spaces, thereby achieving efficient reinforcement learning.

In one embodiment, in order to verify the practicability of the adaptive control algorithm, the present invention will conduct experiments in the experimental platform as shown in Figure 8. The experimental platform includes two UR5 robotic arms, a UR5 robotic arm control cabinet, a UR5 robotic arm teaching pendant, a PC, a six-dimensional force sensor, and a switch. Two of the UR5 robotic arms communicate through their own control cabinets. In the present invention, the UR5 robotic arms communicate through the Ethernet communication protocol, and the six-dimensional force sensor communicates through the TCP communication protocol. Two UR5 robotic arms and six-dimensional force sensors are connected to the switch, and the switch communicates and transmits data with the PC through the TCP/IP communication protocol. The PC can directly control the movement of the robotic arm and read the data of the six-dimensional force sensor in real time.

In the simulation experiment, the standard force signal is set as the manipulator end force tracking target, the tracking force is set in the z-axis direction of the manipulator end operating space, and no tracking target force values are set in other axis directions. Set the target force value Fe=8 in the z-axis direction, set the simulation time to 3s, and observe the force-displacement tracking effect. As shown in Figure 9A and Figure 9B. A deep reinforcement learning module is added to the traditional admittance control, and in the simulation experiment, the present invention adds three algorithms for comparative analysis, which further highlights the advantages and disadvantages of the algorithms. In the case of constant force and constant trajectory, deep reinforcement learning is not superior to the other two algorithms in terms of tracking error of the desired force and desired trajectory, but the errors remain within 0.0005, and deep reinforcement learning is the comparison between the desired force and the desired trajectory. A fast algorithm, although the constant impedance controller is smaller than deep reinforcement learning in terms of speed and error, it has overshoot and oscillation in the early stages of simulation. Therefore, the overall performance of deep reinforcement learning is better than the other two algorithms, and the feasibility of the adaptive variable impedance control algorithm of deep reinforcement learning is verified.

In the simulation experiment, the standard force signal is set as the manipulator end force tracking target, the tracking force is set in the z-axis direction of the manipulator end operating space, and no tracking target force values are set in other axis directions. Set the target force value Fe=8 in the z-axis direction, set the simulation time to 3s, and observe the force-displacement tracking effect. As shown in Figure 10A and Figure 10B, the simulation results of the desired force and desired trajectory tracking of deep reinforcement learning adaptive impedance control under constant force and constant trajectory can be seen. In the case of constant force changing trajectory, the tracking error of deep reinforcement learning for the desired force and desired trajectory is superior to the other two algorithms, and deep reinforcement learning is faster for the desired force and desired trajectory. Constant impedance control The convergence speed of the controller is the fastest, but it has overshoot and oscillation in the early stage of simulation, and oscillation in subsequent actions. The error of the adaptive variable impedance controller is relatively large, and the convergence speed is relatively slow. Therefore, the overall performance of deep reinforcement learning is better than the other two algorithms, and the adaptive variable impedance control of deep reinforcement learning can enable it to achieve better desired force and desired trajectory.

In the simulation experiment, the standard force signal is set as the manipulator end force tracking target, the tracking force is set in the z-axis direction of the manipulator end operating space, and no tracking target force values are set in other axis directions. Set the target force value Fe=8 in the z-axis direction, set the trajectory value to Xe=0.1+0.1sin (3π), set the simulation time to 3s, and observe the force displacement tracking effect. Figures 11A and 11B show the simulation results of the desired force and desired trajectory tracking of the reference model adaptive impedance control under variable force and variable trajectories. In the case of changing force and trajectory, the tracking error of deep reinforcement learning for the desired force and desired trajectory is superior to the other two algorithms, and deep reinforcement learning is faster for the expected force and desired trajectory. Constant impedance control The convergence speed of the controller is the fastest, but it has overshoot and oscillation phenomena in the early stage of simulation. The error of the adaptive variable impedance controller is relatively large, there is overshoot phenomenon in the early stage of convergence, and the convergence speed is relatively slow. Therefore, the overall performance of deep reinforcement learning is better than the other two algorithms. At the same time, the adaptive variable impedance control of deep reinforcement learning of the reference model can stabilize the force under the dynamic trajectory, and at the same time, the expected error of the force is smaller.

Compared with the current existing technology, the deep reinforcement learning used in the present invention has learning ability and can optimize the control strategy through continuous trial and error and feedback, thereby improving the control performance of the system. Compared with traditional control algorithms, deep reinforcement learning can better adapt to unknown environments and tasks. At the same time, the control strategy can be adjusted according to real-time environment and task changes to achieve adaptive control. This adaptability enables the system to better adapt to complex and dynamic work environments. The robustness of the system is improved through training data, so that the system has better adaptability to noise, interference and uncertainty, and achieves more precise force coordination control.

In addition, as shown in Figure 12, one embodiment of the present invention also discloses an adaptive variable impedance control device, which includes:

The construction module 110 is used to construct the impedance model of the robot's dual mechanical arms when grabbing the target object, decouple the internal force and external force on the target object, and perform adaptive impedance control on the internal force and external force respectively;

Initialization module 120 is used to initialize the network parameters and experience pool of the impedance model. The experience pool is used to store the experience tuples of the robot in the environment. The impedance model includes an Actor network and a Critic network. The Actor network is used to generate continuous actions. Critic The network is used to evaluate the quality of the action and output the corresponding action value function;

The training module 130 is used to select actions from the robot's state space. After executing the selected actions, store the experience tuples of environmental feedback into the experience pool, and randomly sample a batch of data from the experience pool to calculate the loss of the Critic network. And perform back propagation, calculate the target Q value through the Critic network, update the parameters of the Actor network to maximize the Q value, and loop training until the preset number of iterations is reached to obtain the trained Actor-Critic network;

The execution module 140 is used to use the trained Actor-Critic network to execute the actions of the dual robotic arms to grab the target object in the actual environment.

The adaptive variable impedance control device in the embodiment of the present invention is used to execute the adaptive variable impedance control method in the above embodiment. The specific processing process is the same as the adaptive variable impedance control method in the above embodiment, and will not be repeated here. Repeat.

In addition, as shown in Figure 13, one embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220, used to store at least one program; when at least one program is processed by at least one When the controller 210 is executed, the adaptive variable impedance control method as in any previous embodiment is implemented.

In addition, an embodiment of the present invention also discloses a computer-readable storage medium in which computer-executable instructions are stored, and the computer-executable instructions are used to execute the adaptive variable impedance control method as in any of the previous embodiments.

The system architecture and application scenarios described in the embodiments of the present invention are to more clearly explain the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will know that with the system architecture With the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

Those of ordinary skill in the art can understand that all or some steps, systems, and functional modules/units in the devices disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof.

In hardware implementations, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may consist of several physical components. Components execute cooperatively. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. removable, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical disk storage, magnetic cassettes, tapes, disk storage or other magnetic storage devices, or may Any other medium used to store the desired information and that can be accessed by a computer. Additionally, it is known to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

The terms "component", "module", "system", etc. used in this specification are used to refer to computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process, processor, object, executable file, thread of execution, program or computer running on a processor. Through the illustrations, both applications running on the computing device and the computing device may be components. One or more components can reside in a process or thread of execution, and the component can be localized on one computer or distributed between two or more computers. Additionally, these components can execute from various computer-readable media having various data structures stored thereon. A component may, for example, be based on a signal having one or more data packets (eg, data from two components interacting with another component, such as a local system, a distributed system, or a network, such as the Internet, which interacts with other systems via signals) Communicate through local or remote processes.

Claims (10)

1. An adaptive variable impedance control method, comprising:

constructing an impedance model of a robot double mechanical arm when grabbing a target object, decoupling the internal force and the external force born by the target object, and respectively performing self-adaptive impedance control on the internal force and the external force;

Initializing network parameters and experience pools of the impedance model, wherein the experience pools are used for storing experience tuples of the robot in the environment, the impedance model comprises an Actor network and a Critic network, the Actor network is used for generating continuous actions, the Critic network is used for evaluating the quality of the actions, and corresponding action value functions are output;

selecting actions from a state space of the robot, storing the experience tuples fed back by the environment into the experience pool after the selected actions are executed, randomly sampling a batch of data from the experience pool, calculating the loss of the Critic network and carrying out back propagation, calculating a target Q value through the Critic network, updating parameters of the Actor network to maximize the Q value, and carrying out cyclic training until the preset iteration times are reached, thereby obtaining a trained Actor-Critic network;

and using the trained Actor-Critic network to execute the actions of the double mechanical arms to grab the target object in an actual environment.

2. The method according to claim 1, wherein the method further comprises:

establishing a coordinate system of a double mechanical arm cooperative system, and solving the position and the posture of a target object relative to a reference coordinate system by using the following formula:

In the method, in the process of the invention,a transformation matrix of the target object relative to a centroid coordinate system; />A 3X3 rotation matrix of the object relative to the coordinate system at the centroid; />A 3X1 matrix of positions of the target object relative to a coordinate system at the centroid;

the target object is converted into a constraint condition between the target object and the mechanical arm through the transformation between the coordinate system at the centroid and the world coordinate system, and the constraint condition is expressed by the following formula:

in the method, in the process of the invention,the method comprises the steps of performing homogeneous coordinate conversion on a coordinate system O at the centroid relative to a world coordinate system W; />Homogeneous coordinate conversion of a base coordinate system of the double mechanical arms relative to a world coordinate system is represented; />Representing the secondary transformation of the end coordinate system of the double mechanical arm relative to the base coordinate of the double mechanical arm; />Representing homogeneous conversion of a centroid coordinate system of a target object relative to the tail end of the mechanical arm;

analyzing the speed constraint relation by the following formula, so that the consistency of the position and the speed of the double arms is maintained in the movement process;

in the method, in the process of the invention,representing the velocity of the end of the robotic arm relative to the world coordinate system; />Representing the velocity of the object relative to the world coordinate system,angular velocity; />Representing a matrix of the arm tip relative to the world position; />Representing a position transformation matrix of the tail end of the mechanical arm relative to the mass center of the target object; / >Representing the rotation matrix of the centroid of the target object relative to the direction under the world.

3. The method of claim 1, wherein decoupling the internal and external forces experienced by the target object comprises:

according to Newton's second law and Euler equation, the state that the double mechanical arms grasp the target object is established, and the dynamics equation of the following target object is established:

in the middle ofI _O Representing an inertial matrix at a centroid of the target object; f (F) _O ∈R ⁶ Representing the vector force of the double mechanical arms acting on the target object; m is M _O ∈R ⁶ A mass inertia matrix representing the target object; />Representing linear acceleration and angular acceleration during the movement of the target object; c (C) _O ∈R ⁶ Resultant force vectors expressed as coriolis force, gravity force, and centrifugal force of the target object; f (F) _ext ∈R ⁶ Indicating the suitability of external disturbing forces acting on the target objectMeasuring force; converting the above formula to the following formula:

where k=l, r is denoted as left and right arms of the double mechanical arm, S _k ^T ∈R ⁶ Representing a grabbing matrix; f (F) _k Representing the force of the mechanical arm acting on the target object; decomposing the grabbing matrix to obtain an external force F _I And obtaining an internal force F _E :

In the middle ofIs->Generalized inverse of the matrix.

4. The method of claim 1, wherein the equation of the impedance model is as follows:

Wherein m is an inertia coefficient, b is a damping coefficient, ε is an adaptive parameter, Δf represents an error value of force,and->The movement speed and the movement acceleration of the robot arm are respectively.

5. The method of claim 1, wherein randomly sampling a batch of data from the experience pool, calculating a loss of the Critic network and back-propagating, calculating a target Q value through the Critic network, updating parameters of the Actor network to maximize Q value, comprises:

based on a deterministic gradient strategy, updating the Actor network parameters according to the action value function, wherein the deterministic behavior strategy has the following formula:

μ→a _t ＝μ(s _t )

wherein μ is a policy function, s is a current state, and the action of the deterministic policy is uniquely determined in state s, and the formula is as follows:

a _t ＝μ(s _t |θ ^μ )+N _t

wherein θ is a policy parameter; during network training, randomly sampling a plurality of data N as training data of a deterministic strategy mu, wherein the deterministic strategy mu is measured and expressed as the following formula:

wherein s is _t ～ρ ^β Representing the empirical distribution ρ ^β Sample a state s _t ，Q ^μ (s, μ (s)) represents the value of Critic network for a given state s and action μ(s), the whole expectationRepresenting taking expected values on the sampled state and action; during training, replacing expected values by sample mean values;

The Actor network learns the optimal strategy by maximizing the output of the Critic network, and the parameter updating of the Actor network is performed by the following gradient rising mode:

wherein a is _t Pi represents sampling an action a from the Actor policy pi _t ，Q(s,a|θ ^Q ) Gradient term representing the value of Critic network for a given state s and action aRepresenting the gradient of the Critic network to the action of the Actor output;

according to a minimization loss function L (θ ^Q ) Updating the parameter θ of the current value network Q ^Q The minimization loss function is as follows:

wherein N is the random sampling number, and gamma is the attenuation coefficient.

6. The method of claim 5, wherein the randomly sampling a batch of data from the experience pool, calculating a loss of the Critic network and back-propagating, calculating a target Q value through the Critic network, updating parameters of the Actor network to maximize Q value, further comprising:

based on a deterministic gradient strategy, calculating a gradient strategy of a behavior strategy network Q according to the following formula, and updating the Critic network parameters;

wherein s is _t And a _t Is from the empirical distribution ρ ^β State and action obtained by middle sampling s _t+1 Is the next state sampled from the empirical distribution epsilon, Q (s _i ,a _i |θ ^Q ) Is the output of the Critic network, representing the estimated value of action a taken in state s, y is the target Q value, and the goal of gradient estimation is to minimize the mean square error between the output of the Critic network and the target Q value; using the sampled state, action and next state, the expected estimate of this mean square error is made and then the parameter θ for the Critic network is calculated ^Q To perform parameter updating;

y _i ＝r _i +γQ’(s _i+1 ,μ’(s _i+1 |θ ^μ’ )|θ ^Q’ )

wherein r is _i Is the current prize, γ is the discount factor, Q 'and μ' are the target network;

updating mu 'and Q' by adopting a sliding average method, wherein the updating method is as follows:

θ ^μ′ ←τθ+(1-τ)θ ^μ′

θ ^Q′ ←τθ+(1-τ)θ ^Q′

wherein θ ^μ′ 、θ ^Q′ τ is a parameter, τ is a learning rate, and the value of τ is 0.001; in the network architecture, a policy network Actor is used to update θ ^μ′ Outputting action a, and acquiring the pair parameter theta by the value network Critic ^Q′ Update for approximating the state-behavior value function Q ^π (s,a)。

7. The method according to claim 1, wherein the state space s of the double mechanical arm cooperative motion process is defined as follows:

s＝{e _f ,e _x ,F _a ,x _a }

wherein e _f Representing force tracking error, e _x Representing track following error, F _a Representing the actual force, x, during control of a dual mechanical arm _a Representing an actual track in the control process of the double mechanical arms;

The goal of the DDPG algorithm is to track the error e based on force _f And track tracking error e _x Outputting a proper self-adaptive parameter epsilon; as an output parameter of the DRL algorithm, a designed action spaceOnly one determined element is contained, at the moment t, the action at the moment is represented, the action function is shown as the following formula, and the action function is the self-adaptive parameter epsilon:

a _t ＝{ε}

wherein epsilon is an adaptive parameter; in the mechanical arm position force tracking process, each time step needs to be evaluated in real time, and according to the actual forceMean value of force error->Difference and actual trajectory ∈ ->Mean value of track error->The sum of the differences in different ratios is used as part of a reward and punishment function, called a basic reward and punishment part; force tracking error e _f Tracking error e of track _x All the time kept at 0; tracking error e of force _f And track tracking error e _x As part of a punishment function, called the extra excitation part r _extre The method comprises the steps of carrying out a first treatment on the surface of the Different additional rewards and punishments are given according to the interval where the force tracking error is located, and the reward and punishment function formed by combining the basic reward and punishment part and the additional excitation part is shown in the following formula:

wherein,

wherein k is _a 、k _b Is a scale factor, H is the time period of the training process,is the average value of the track errors at the moment j, +.>Mean value of force errors at moment j, +. >For the actual force of the double arm at moment j +.>In order to double the actual position of the mechanical arm at time j, in the additional excitation part, according to e _f And e _x The region is subjected to different additional rewards and punishments.

8. An adaptive variable impedance control device, the device comprising:

the construction module is used for constructing an impedance model of the robot double mechanical arms when the target object is grabbed, decoupling the internal force and the external force applied to the target object, and respectively carrying out self-adaptive impedance control on the internal force and the external force;

the system comprises an initializing module, a judging module and a judging module, wherein the initializing module is used for initializing network parameters of the impedance model and an experience pool, the experience pool is used for storing experience tuples of the robot in the environment, the impedance model comprises an Actor network and a Critic network, the Actor network is used for generating continuous actions, the Critic network is used for evaluating the quality of the actions and outputting corresponding action value functions;

the training module is used for selecting actions from a state space of the robot, storing the experience tuples fed back by the environment into the experience pool after the selected actions are executed, randomly sampling a batch of data from the experience pool, calculating the loss of the Critic network and carrying out back propagation, calculating a target Q value through the Critic network, updating the parameters of the Actor network to maximize the Q value, and carrying out cyclic training until the preset iteration times are reached to obtain a trained Actor-Critic network;

And the execution module is used for executing the actions of the double mechanical arms to grab the target object in an actual environment by using the trained Actor-Critic network.

9. An electronic device, comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the adaptive variable impedance control method according to any of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions for performing the adaptive variable impedance control method of any one of claims 1 to 7.