CN116766178A - Shaft hole assembly method and related equipment - Google Patents

Abstract

The invention provides a shaft hole assembly method and related equipment, and relates to the field of automated assembly technology. The method includes: performing multiple iterative operations; each iterative operation includes: obtaining the current status of the shaft; the current status includes current force information; Input the current state into the shaft hole assembly model to obtain the current assembly action; the current assembly action includes translation and rotation; the shaft hole assembly model is trained on multiple shaft hole assembly tasks of different categories based on prior knowledge and meta-reinforcement learning. ; perform the current assembly action on the shaft; when the depth of the shaft installation hole does not meet the preset assembly requirements, perform the next iteration operation. The shaft hole assembly model of the present invention can be applied to different types of shaft hole assembly tasks, thereby improving the versatility and practicality of shaft hole assembly.

Description

Shaft hole assembly methods and related equipment

Technical field

The present invention relates to the field of automated assembly technology, and in particular to a shaft hole assembly method and related equipment.

Background technique

In the industrial field, there is an increasing demand for robots to automatically complete high-precision assembly tasks. Shaft hole assembly is the core operation of most assembly tasks. Common ones include single-axis hole assembly, dual-axis hole assembly, and three-axis hole assembly.

Existing shaft hole assembly methods are often only suitable for the same type of assembly tasks. The shaft hole size and number of shaft holes in the same type of assembly task are the same. When the shaft hole size or the number of shaft holes changes, the shaft hole assembly The method is not applicable, resulting in the shaft hole assembly method being less versatile and practical.

Therefore, it is very important to provide a shaft hole assembly method that allows robots to complete multi-category shaft hole assembly tasks.

Contents of the invention

The invention provides a shaft hole assembly method and related equipment, which are used to solve the shortcomings of low versatility and practicality of shaft hole assembly in the prior art and improve the versatility and practicality of shaft hole assembly.

The invention provides a shaft hole assembly method, which includes:

Perform multiple iteration operations; each iteration operation includes:

Obtain the current status of the axis; the current status includes current force information;

Input the current state into the shaft hole assembly model to obtain the current assembly action; the current assembly action includes the translation amount and the rotation amount; the shaft hole assembly model is based on prior knowledge and meta-reinforcement learning on multiple different categories of shafts. Trained in the hole assembly task;

Execute the current assembly action on the axis;

If the depth of the shaft installation hole does not meet the preset assembly requirements, the next iterative operation is performed.

In some embodiments, the shaft hole assembly model is trained based on the following steps:

Construct multiple different categories of shaft hole assembly tasks;

Perform multiple iterative training operations; each iterative training operation includes:

Randomly select one shaft hole assembly task from the plurality of shaft hole assembly tasks of different categories as the current training task;

Execute the current training task and update the parameters of the shaft hole assembly model;

After the current training task is completed, obtain the cumulative reward corresponding to the current training task;

According to the cumulative reward corresponding to the current training task and the cumulative reward corresponding to the obtained previous training task, determine the convergence situation of the cumulative reward corresponding to the training task;

If the cumulative reward corresponding to the training task has not converged, the next iterative training is performed.

In some embodiments, performing the current training task and updating parameters of the shaft hole assembly model includes:

The current training task is executed through multiple iterative training rounds, and the parameters of the shaft hole assembly model are updated in each iterative training round; wherein each iterative training round includes:

Obtain the similarity between the previous assembly training action and the previous demonstration action, and the current training status; the previous assembly training action was in the previous iterative training round, and the network selection network in the shaft hole assembly model was based on The training action output by the previous training state; the previous demonstration action is the demonstration action output by the assembly demonstration model according to the previous training state; the current training state includes the current force information of the training axis;

Input the similarity and the current training status into the action evaluation network in the shaft hole assembly model to obtain the current first loss function value;

Update parameters of the action evaluation network based on the current first loss function value;

Input the similarity and the current training status into the action selection network in the shaft hole assembly model to obtain the current second loss function value;

Update parameters of the action selection network based on the current second loss function value;

If the current training task is not completed, the next iterative training round is executed.

In some embodiments, inputting the similarity and the current training status into the action evaluation network in the shaft hole assembly model to obtain the current first loss function value includes:

Input the similarity and the current training status into the action evaluation network to obtain the current status value and the current reward value;

Obtain the current advantage value according to the current state value, the current reward value, and the latter state value;

According to the current advantage value, the current first loss function value is obtained.

In some embodiments, inputting the similarity and the current training status into the action selection network in the shaft hole assembly model to obtain the current second loss function value includes:

Input the similarity and the current training status into the action selection network to obtain the current assembly training action;

The current second loss function value is obtained based on the probability of the current assembly training action, the probability of the previous assembly training action, and the current advantage value.

In some embodiments, before obtaining the similarity between the previous assembly training action and the previous demonstration action, the method further includes:

Get the teaching status and teaching actions;

Model the teaching state and the teaching action to obtain an assembly demonstration model.

The invention also provides a shaft hole assembly device, which includes:

Execution module, used to perform multiple iteration operations; each iteration operation includes:

Obtain the current status of the axis; the current status includes current force information;

Input the current state into the shaft hole assembly model to obtain the current assembly action; the current assembly action includes the translation amount and the rotation amount; the shaft hole assembly model is based on prior knowledge and meta-reinforcement learning on multiple different categories of shafts. Trained in the hole assembly task;

Execute the current assembly action on the axis;

If the depth of the shaft installation hole does not meet the preset assembly requirements, the next iterative operation is performed.

The present invention also provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the assembly of the shaft hole as described above is realized. method.

The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned shaft hole assembly methods.

The present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements any of the above-mentioned shaft hole assembly methods.

The invention provides a shaft hole assembly method and related equipment, which uses meta-reinforcement learning to learn a plurality of different categories of shaft hole assembly tasks, and improves the learning efficiency of meta-reinforcement learning through prior knowledge. The obtained shaft hole assembly model can It is suitable for different types of shaft hole assembly tasks, thereby improving the versatility and practicality of shaft hole assembly.

Description of drawings

In order to explain the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are of the present invention. For some embodiments of the invention, those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is one of the flow diagrams of a shaft hole assembly method provided by an exemplary embodiment of the present invention;

Figure 2 is the second schematic flow chart of the shaft hole assembly method provided by an exemplary embodiment of the present invention;

Figure 3 is a third schematic flowchart of the shaft hole assembly method provided by an exemplary embodiment of the present invention;

Figure 4 is the fourth schematic flowchart of the shaft hole assembly method provided by an exemplary embodiment of the present invention;

Figure 5 is a schematic flow chart of a shaft hole assembly method fifth according to an exemplary embodiment of the present invention;

Figure 6 is a schematic flowchart No. 6 of a shaft hole assembly method provided by an exemplary embodiment of the present invention;

Figure 7 is a schematic structural diagram of a shaft hole assembly device provided by an exemplary embodiment of the present invention;

FIG. 8 is a schematic diagram of the physical structure of an electronic device provided by an exemplary embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention. , not all examples. It should be noted that, as long as there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present invention.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which the invention belongs. The terminology used herein in the description of the invention is for the purpose of describing specific embodiments only and is not intended to limit the invention.

It should be further noted that, as used herein, the terms "comprises," "comprises," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or device that includes a list of elements includes not only those elements , but also includes other elements not expressly listed or inherent in such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element.

In the present invention, "at least one" means one or more, and "plurality" means two or more than two. The terms "first", "second", "third", "fourth", etc. (if present) in the present invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

In the embodiments of the present invention, words such as “exemplary” or “for example” are used to represent examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "such as" in the embodiments of the invention is not to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words "exemplary" or "such as" is intended to present the concept in a concrete manner.

Please refer to FIG. 1 , which is a schematic flowchart of a shaft hole assembly method provided by an exemplary embodiment of the present invention. Embodiments of the present invention provide a shaft hole assembly method, the execution subject of which may be an intelligent agent, such as a robot. The shaft hole assembly method includes the following steps:

Step 110: Perform multiple iterative operations.

Specifically, the agent can obtain an assembly action by performing an iterative operation. Since one assembly action often cannot successfully assemble the shaft hole, the agent needs to perform multiple iterative operations to obtain multiple assembly actions. This is achieved through multiple assembly actions. The shaft hole is assembled successfully. Each iterative operation may include steps 1101 to 1104.

Step 1101, obtain the current status of the axis.

In the current iterative operation, the agent obtains the current status of the shaft through the force sensor installed on the shaft. The current status includes the current force information.

The expression for state s is as follows:

s＝[F _x ,F _y ,F _z ,M _x ,M _y ,M _z ]

In the formula, s represents the state, F _x , F _y and F _z respectively represent the forces along the x direction, y direction and z direction measured by the force sensor, M _x , M _y and M _z represent the forces along the x direction and y direction respectively. and the moment in the z direction. Among them, the x direction is the direction in which the axis moves forward and backward, the y direction is the direction in which the axis moves left and right, and the z direction is the direction in which the axis moves up and down.

In some embodiments, the number of shafts and holes is the same, with one shaft corresponding to one hole. The number of axes may be single axes, dual axes, or three axes, etc., and the number of axes is not specifically limited in the present invention.

Step 1102: Input the current status into the shaft hole assembly model to obtain the current assembly action.

Specifically, in the related technology, the Actor-Critic algorithm is used to construct the shaft hole assembly model. The Actor-Critic algorithm is an algorithm of standard deep reinforcement learning (Deep Reinforcement Learning, DRL).

Standard DRL is aimed at a specific Markov Decision Process (Markov Decision Process, MDP) and uses certain learning algorithms to solve an optimal strategy. The optimal strategy guides the agent to make the best decision under a specific task (what state to take? what action). The learning algorithm in standard DRL relies on a large number of interactions between the agent and the environment, and the training cost is high. Once the environment changes, the originally learned optimal strategy is no longer applicable. At this time, it must be retrained and learned for the new environment. Less efficient.

Meta-learning is introduced into standard DRL to form meta-reinforcement learning. Meta-reinforcement learning is a learning algorithm that can quickly adapt to different new tasks and obtain corresponding optimal strategies.

The shaft hole assembly model in the present invention is trained in multiple different categories of shaft hole assembly tasks based on prior knowledge and meta-reinforcement learning.

The agent applies meta-reinforcement learning to learn multiple different categories of shaft hole assembly tasks, and uses prior knowledge to improve the learning efficiency of meta-reinforcement learning, thereby obtaining a trained shaft hole assembly model. Therefore, the trained shaft hole assembly model can be applied to different categories of shaft hole assembly tasks. After obtaining the trained shaft hole assembly model, the agent inputs the obtained current status into the shaft hole assembly model, and the shaft hole assembly model outputs the current assembly action. The current assembly action includes translation and rotation.

The expression of assembly action a is as follows:

a＝[Δ _x ,Δ _y ,Δ _z ,α _x ,α _y ,α _z ]

In the formula, a represents the assembly action, Δ _x , Δ _y and Δ _z represent the translation amounts along the x direction, y direction and z direction respectively, α _x , α _y and α _z represent the translation along the x direction, y direction and z direction respectively. amount of rotation.

In some embodiments, the agent inputs the current state into the shaft hole assembly model, the shaft hole assembly model outputs the optimal action strategy, and selects the action with the highest probability in the optimal action strategy for denormalization to obtain the current assembly action.

Step 1103: Execute the current assembly action on the axis.

Specifically, after the agent obtains the current assembly action, it executes the current assembly action on the axis, that is, it translates and rotates the axis according to the translation amount and rotation amount in the current assembly action.

Step 1104: Check whether the depth of the shaft installation hole meets the preset assembly requirements.

Specifically, in some embodiments, the preset assembly requirements may include a preset minimum loading depth, a preset maximum loading depth, and the like.

After executing the current assembly action on the shaft, the agent detects whether the depth of the shaft installation hole meets the preset assembly requirements, that is, whether the depth of the shaft installation hole is within the preset minimum installation depth and the preset maximum installation depth. between.

When the depth of the shaft installation hole does not meet the preset assembly requirements, that is, when it is detected that the depth of the shaft installation hole is less than the preset minimum installation depth or greater than the preset maximum installation depth, the agent executes The next iteration operation, that is, the agent performs steps 1101 to 1104 again.

When the depth of the shaft installation hole meets the preset assembly requirements, that is, when it is detected that the depth of the shaft installation hole is between the preset minimum installation depth and the preset maximum installation depth, it indicates that the shaft hole The assembly is successful and the agent ends the iteration.

The shaft hole assembly method provided in this embodiment uses meta-reinforcement learning to learn a plurality of different categories of shaft hole assembly tasks, and uses prior knowledge to improve the learning efficiency of meta-reinforcement learning. The obtained shaft hole assembly model can be suitable for different categories. shaft hole assembly tasks, thereby improving the versatility and practicality of shaft hole assembly.

Please refer to FIG. 2 , which is a second schematic flowchart of a shaft hole assembly method provided by an exemplary embodiment of the present invention. This embodiment is a further improvement on the previous embodiment, and the main improvement lies in the specific process of the intelligent agent training the shaft hole assembly model. The process of this embodiment is shown in Figure 2, including the following steps:

Step 210: Construct multiple shaft hole assembly tasks of different categories.

Specifically, the automatic assembly process of an agent can be regarded as an MDP process, and each assembly task T _n can be expressed as a five-tuple <S _n ,A _n ,P _n ,R _n ,γ>. Among them, S _n is a finite set of states in the nth assembly task, and a certain state in the set is expressed as s _k ∈S _n ; A _n is a finite set of actions in the nth assembly task, and an action in the set is expressed as a _k ∈A _n , a _k is the executable action in state s _k ; p _n is the state transition equation of the nth assembly task, P _k means that after executing action a _k in state s _k , it will be P(s′ _k The probability of |s _k , a _k ) jumps to state s′ _k ; R _n is the reward function; γ is the discount coefficient, 0≤γ≤1. How the shaft hole assembly model learns the Markov decision process corresponding to each assembly task is the key to realizing automatic assembly of multi-category shaft hole assembly tasks. The training process of the shaft hole assembly model is explained in detail below.

The agent builds multiple different environments, such as physical simulation environment, mathematical simulation environment and real assembly environment. Build multiple different categories of shaft hole assembly tasks in each environment based on the size and number of shaft holes.

In some embodiments, multiple different categories of shaft hole assembly tasks may include: assembling tasks with different shaft hole sizes and numbers of shaft holes in different environments; assembling tasks with the same shaft hole size and different numbers of shaft holes in different environments. Tasks; tasks of assembling shaft holes with different sizes and the same number of shaft holes in different environments; tasks of assembling shaft holes with the same size and number of shaft holes in different environments; tasks of assembling shaft holes with the same size but different numbers of shaft holes in the same environment Tasks; tasks of assembling shaft holes with different sizes but the same number of shaft holes under the same environment, etc.

In some embodiments, the number of axial holes may be uniaxial holes, biaxial holes, triaxial holes, etc.

In the mathematical simulation environment, it is assumed that there is point contact between the shaft and the hole and the principle of elastic deformation is satisfied.

The expression for the three-dimensional force on the end of the shaft is as follows:

In the formula, F represents the three-dimensional force on the end of the shaft, K represents the number of shaft holes, δ represents the elastic deformation coefficient, p _i ¹ represents the maximum deformation vector of the i-th axis at the upper part of the hole, p _i ² represents the i-th axis at the lower part of the hole. the maximum deformation vector.

The expression for the three-dimensional moment exerted on the end of the shaft is as follows:

In the formula, M represents the three-dimensional moment experienced by the end of the shaft, K represents the number of shaft holes, δ represents the elastic deformation coefficient, p _i ¹ represents the maximum deformation vector of the i-th axis at the upper part of the hole, l _i ¹ represents the center of the i-th axis The offset vector from the point to the maximum deformation point at the upper part of the hole, p _i ² The maximum deformation vector of the i-th axis at the bottom of the hole, l _i ² represents the offset vector from the center point of the i-th axis to the maximum deformation point at the bottom of the hole .

According to the three-dimensional force and three-dimensional moment exerted on the end of the shaft, the state of the shaft can be obtained.

Step 220: Perform multiple iterative training operations.

Specifically, in order for the shaft hole assembly model to learn multiple shaft hole assembly tasks of different categories, the agent performs multiple iterative training operations. Each iterative training operation may include the following steps 2201 to 2205.

Step 2201: Randomly select one shaft hole assembly task from a plurality of shaft hole assembly tasks of different categories as the current training task.

Specifically, in each iteration of training, the agent randomly selects a shaft hole assembly task from multiple shaft hole assembly tasks of different categories, and the probability of being selected for each shaft hole assembly task is the same. For example, if there are 6 shaft hole assembly tasks of different categories, then the probability of being selected for each shaft hole assembly task is 1/6. The agent takes the selected shaft hole assembly task as the current training task.

In some embodiments, since the shaft hole assembly tasks of multiple different categories are randomly selected each time, the current training task may be the same as the training task in the previous iterative training, or may be different.

Step 2202: Execute the current training task and update the parameters of the shaft hole assembly model.

Specifically, in each iterative training round, the shaft hole assembly model outputs an assembly training action, the current training task is performed through multiple iterative training rounds, and the parameters of the shaft hole assembly model are updated in each iterative training round.

Step 2203: Check whether the current training task is completed.

Specifically, detecting whether the current training task is completed can be achieved by detecting whether the shaft hole is successfully assembled, or whether the force on the shaft exceeds the maximum radial force.

When it is detected that the shaft hole is successfully assembled, or the force on the shaft exceeds the maximum radial force, it indicates that the current training task is completed, and the agent executes step 2204.

When it is detected that the shaft hole has not been assembled successfully, and the force on the shaft does not exceed the maximum radial force, it indicates that the current training task is not completed, and the agent executes step 2203.

Step 2204: Obtain the cumulative reward corresponding to the current training task.

Specifically, each iterative training round corresponds to a reward, and multiple rewards corresponding to multiple iterative training rounds are accumulated to obtain the cumulative reward corresponding to the current training task.

Step 2205: Check whether the cumulative reward corresponding to the training task has converged.

Specifically, the training tasks include current training tasks and previous training tasks. The agent obtains the cumulative reward corresponding to the previous training task in the previous iterative training.

The agent determines the convergence of the cumulative rewards corresponding to the training tasks based on the cumulative rewards corresponding to the current training task and the cumulative rewards corresponding to the previous training tasks. The specific process may be: starting from the cumulative reward corresponding to the current training task, and detecting whether the cumulative rewards corresponding to the preset training tasks are close to each other. If the cumulative rewards corresponding to the preset training tasks are close, it indicates that the cumulative rewards corresponding to the training tasks have converged; if the cumulative rewards corresponding to the preset training tasks are greatly different, it indicates that the cumulative rewards corresponding to the training tasks have not converged.

When the cumulative reward corresponding to the training task converges, the agent ends the iterative training and obtains the trained shaft hole assembly model; when the cumulative reward corresponding to the training task does not converge, the agent performs the next iterative training, that is, Perform steps 2201 to 2205 again.

The shaft hole assembly method provided in this embodiment constructs a plurality of shaft hole assembly tasks of different categories, and randomly selects one shaft hole assembly task from them for training, so that the trained shaft hole assembly model can be adapted to different types of shaft hole assembly. The task is to improve the versatility and practicality of the shaft hole assembly model.

Please refer to FIG. 3 , which is a third schematic flowchart of a shaft hole assembly method provided by an exemplary embodiment of the present invention. This embodiment is a detailed explanation of the foregoing embodiment, and mainly describes the specific process of the agent performing the current training task and updating the parameters of the shaft hole assembly model. The process of this embodiment is shown in Figure 3, including the following steps:

Step 310: Execute the current training task through multiple iterative training rounds, and update the parameters of the shaft hole assembly model in each iterative training round.

Specifically, the shaft hole assembly model includes an action selection network and an action evaluation network. The action selection network is mainly used to train the action selection strategy to determine the optimal assembly action. The action evaluation network is used to score assembly actions and guide optimal assembly actions.

In some embodiments, the action selection network may be an Actor network, and the action evaluation network may be a Critic network.

Each iterative training round corresponds to an assembly action. Through multiple iterative training rounds, multiple assembly actions are obtained, and the current training task is executed through multiple assembly actions. The parameters of the action selection network and the action evaluation network are updated in each iterative training round.

Each iterative training round may include the following steps 3101 to 3106.

Step 3101: Obtain the similarity between the previous assembly training action and the previous demonstration action, and the current training status.

Specifically, the agent obtains the current training status through the force sensor. The current training status includes the current force information of the training axis.

The agent obtains the previous assembly training action and the previous demonstration action. The previous assembly training action is the assembly training action output by the action selection network based on the previous training state. The previous demonstration action is the demonstration action output by the assembly demonstration model based on the previous training state. The assembly demonstration model is a Gaussian model built based on prior knowledge.

The agent calculates the similarity between the previous assembly training action and the previous demonstration action to obtain the similarity.

In some embodiments, the expression of similarity is as follows:

In the formula, m _t-1 represents the assembly training action with demonstration actions/> The degree of similarity between Represents the assembly training action output by the action selection network based on the training state s _t-1 corresponding to the t-1th iteration training,/> Indicates the demonstration action output by the assembly demonstration model according to the training state s t _-1 corresponding to the t-1th iteration training round.

Step 3102: Input the similarity and current training status into the action evaluation network in the shaft hole assembly model to obtain the current first loss function value.

Specifically, the agent inputs the similarity and the current training status into the action evaluation network in the shaft hole assembly model, and obtains the data output by the action evaluation network.

In some embodiments, the data output by the action evaluation network may include data used to evaluate the value of the current state, similarity and reward values corresponding to the current training state, etc.

The agent evaluates the data output by the network based on the action and obtains the current first loss function value.

Step 3103: Update the parameters of the action evaluation network based on the current first loss function value.

Specifically, the agent sets a preset first loss function value according to the allowed accuracy of the action evaluation network, and compares the current first loss function value with the preset first loss function value. When the current first loss function value is greater than the preset first loss function value, update the parameters of the action evaluation network; when the current first loss function value is less than the preset first loss function value, update the parameters of the action evaluation network It is already the optimal parameter and does not need to be updated.

In some embodiments, the optimal parameters of the action evaluation network include weight parameters θ ^V .

Step 3104: Input the similarity and current training status into the action selection network in the shaft hole assembly model to obtain the current second loss function value.

Specifically, the agent inputs the similarity and the current training status into the action selection network in the shaft hole assembly model, and obtains the data output by the action selection network.

In some embodiments, the data output by the action selection network may include: the current action selection strategy, the current assembly training action, etc.

The agent obtains the current second loss function value based on the data output by the action selection network and the data output by the action evaluation network.

Step 3105: Update action selection network parameters based on the current second loss function value.

Specifically, the agent sets a preset second loss function value according to the allowed accuracy of the action selection network, and compares the current second loss function value with the preset second loss function value. When the current second loss function value is greater than the preset second loss function value, update the parameters of the action evaluation network; when the current second loss function value is less than the preset second loss function value, update the parameters of the action evaluation network It is already the optimal parameter and does not need to be updated.

In some embodiments, the optimal parameters of the action selection network include weight parameters θ ^A .

Step 3106: Check whether the current training task is completed.

Specifically, detecting whether the current training task is completed can be achieved by detecting whether the shaft hole is successfully assembled, or whether the force on the shaft exceeds the maximum radial force.

When it is detected that the shaft hole is successfully assembled, or the force on the shaft exceeds the maximum radial force, it indicates that the current training task is completed and the agent performs the next training task.

When it is detected that the shaft hole has not been assembled successfully, and the force on the shaft does not exceed the maximum radial force, it indicates that the current training task is not completed, and the agent executes the next iterative training round, that is, steps 3101 to 3106.

The shaft hole assembly method provided in this embodiment obtains the current first loss function value by inputting the similarity and the current training status into the action evaluation network, and updates the parameters of the action evaluation network based on the current first loss function value. Input the action selection network with the current training status, obtain the current second loss function value, and update the parameters of the action selection network based on the current second loss function value to achieve accurate acquisition of the action evaluation network and action selection network.

Please refer to FIG. 4 , which is a schematic flowchart No. 4 of a shaft hole assembly method provided by an exemplary embodiment of the present invention. This embodiment is a detailed explanation of the previous embodiment, and mainly describes the specific process of inputting the similarity and the current training status into the action evaluation network in the shaft hole assembly model to obtain the current first loss function value. The process of this embodiment is shown in Figure 4, including the following steps:

Step 410: Input the similarity and current training status into the action evaluation network to obtain the current status value and current reward value.

Specifically, the agent inputs the similarity and the current training state into the action evaluation network, and the action evaluation network outputs the current state value and the current reward value. The current state value corresponds to the similarity and the current training state, and the current reward value also corresponds to the similarity and the current training state.

In the t-th iteration training, the state value output by the action evaluation network is Among them,/> Represents the state value function, T _i represents the i-th training task, p(T) represents the probability of extracting the training task, s _t represents the training state corresponding to the t-th iterative training round, m _t-1 represents the assembly training action/> with demonstration actions The degree of similarity between Represents the assembly training action output by the action selection network according to the training state s t _-1 corresponding to the t-1th iteration training round,/> represents the demonstration action output by the assembly demonstration model according to the training state s t- ₁ corresponding to the t-1th iteration training round, and θ ^V represents the weight parameter of the action evaluation network. Status value of fused similarity Ability to more effectively represent the value of a state.

In the t-th iteration of training, the expression of the reward value output by the action evaluation network is as follows:

r _t =R (s _t ,m _t-1 )

In the formula, r _t represents the reward value corresponding to the t-th iterative training round, R() represents the reward function, s _t represents the training state corresponding to the t-th iterative training round, and m _t-1 represents the assembly training action. with demonstration actions/> The degree of similarity between Represents the assembly training action output by the action selection network according to the training state s t _-1 corresponding to the t-1th iteration training round,/> Indicates the demonstration action output by the assembly demonstration model according to the training state s t _-1 corresponding to the t-1th iteration training round.

Step 420: Obtain the current advantage value based on the current status value, the current reward value, and the latter status value.

Specifically, the agent inputs the similarity between the current assembly training action and the current demonstration action and the latter training state into the action evaluation network, and the action evaluation network outputs the value of the latter state. Among them, the current assembly training action is the assembly training action output by the action selection network according to the current training state, and the current demonstration action is the demonstration action output by the assembly demonstration model according to the current training state.

In the t-th iterative training round, the expression of the advantage value is as follows:

In the formula, A _t represents the advantage value corresponding to the t-th iterative training round, r _t represents the reward value corresponding to the t-th iterative training round, γ represents the discount coefficient, 0≤γ≤1, Represents the state value corresponding to the t+1 iteration training round, where,/> represents the state value function, T _i represents the i-th training task, p(T) represents the probability of extracting the training task, s _t+1 represents the training state corresponding to the t+1 iterative training round, m _t represents the assembly training action with demonstration actions/> The degree of similarity between Represents the assembly training action output by the action selection network according to the training state s _t corresponding to the t-th iteration training round,/> Represents the demonstration action output by the assembly demonstration model according to the training state s _t corresponding to the t-th iteration training round, θ ^V represents the weight parameter of the action evaluation network, /> represents the state value corresponding to the t-th iterative training round, s _t represents the training state corresponding to the t-th iterative training round, m _t-1 represents the assembly training action/> with demonstration actions/> degree of similarity between them.

After obtaining the value of the latter state, the agent substitutes the value of the current state, the current reward value, and the value of the latter state into the expression of the advantage value to obtain the current advantage value.

Step 430: Obtain the current first loss function value based on the current advantage value.

Specifically, the expression of the first loss function is as follows:

In the formula, L(θ ^V ) represents the first loss function, L() represents the loss function, A _t represents the advantage value corresponding to the t-th iterative training, T _i represents the i-th training task, and p(T) represents extraction training. the probability of the task, Represents the mathematical expectation with the number of iterative training rounds t as the variable when executing the T _i task selected with probability p(T).

After obtaining the current advantage value, the agent substitutes the current advantage value into the expression of the first loss function to obtain the current first loss function value.

The shaft hole assembly method provided in this embodiment obtains the current state value and the current reward value by inputting the similarity and the current training state into the action evaluation network, and obtains the current advantage based on the current state value, the current reward value, and the latter state value. value, according to the current advantage value, the current first loss function value is obtained, which achieves accurate acquisition of the current first loss function value, which is conducive to accurately updating the parameters of the action evaluation network, and is further conducive to obtaining an accurate shaft hole assembly model.

Please refer to FIG. 5 , which is a fifth schematic flowchart of a shaft hole assembly method provided by an exemplary embodiment of the present invention. This embodiment is a detailed explanation of the previous embodiment, and mainly describes the specific process of inputting the similarity and the current training status into the action selection network in the shaft hole assembly model to obtain the current second loss function value. The process of this embodiment is shown in Figure 5, including the following steps:

Step 510: Input the similarity and current training status into the current action selection network to obtain the current assembly training action.

Specifically, the agent inputs the similarity and current training status into the action selection network, and the action selection network outputs the current action selection strategy. The action selection strategy is a set of a series of assembly actions and corresponding probabilities. The action selection strategy presents a Gaussian distribution. The horizontal axis is the assembly action and the vertical axis is the probability.

In the t-th iterative training round, the action selection strategy is Among them, θ ^A represents the weight parameter of the action selection network, a _t represents the assembly training action corresponding to the t-th iterative training round, s _t represents the training state corresponding to the t-th iterative training round, m _t-1 represents the assembly training action/ > with demonstration actions/> degree of similarity between them.

The agent uses the assembly action with the highest probability in the current action selection strategy as the current assembly training action.

Step 520: Obtain the current second loss function value based on the probability of the current assembly training action, the probability of the previous assembly training action, and the current advantage value.

Specifically, the expression of the second loss function is as follows:

In the formula, L(θ ^V ) represents the second loss function, L() represents the loss function, θ ^A represents the weight parameter of the action selection network, _Ti represents the i-th training task, and p(T) represents the extraction of the current training task. probability, Represents the mathematical expectation with the number of iterative training rounds t as the variable when executing the T _i task selected with probability p(T), r _t (θ) represents the ratio of the maximum probabilities in the old and new action selection strategies, A _t represents the t-th iteration training The corresponding advantage value, ∈, is a hyperparameter, generally taking a value of 0.2.

in,

In the formula, r _t (θ) represents the ratio of the probability of assembling training actions in the new and old action selection strategies, represents the probability of assembling training actions in the new action selection strategy, θ ^A represents the weight parameter of the action selection network,/> Represents the probability of assembling training actions in the old action selection strategy, /> Indicates the unupdated weight parameters relative to θ ^A.

The probability of the current assembly training action is the probability of the assembly training action in the new action selection strategy; the probability of the previous assembly training action is the probability of the assembly training action in the old action selection strategy.

The agent brings the probability of the current assembly training action, the probability of the previous assembly training action, and the current advantage value into the second loss function to obtain the current second loss function value.

The shaft hole assembly method provided in this embodiment obtains the current assembly training action by inputting the similarity and the current training status into the current action selection network, based on the probability of the current assembly training action, the probability of the previous assembly training action, and the current advantage value. , obtaining the current second loss function value, achieving accurate acquisition of the current second loss function value, which is conducive to accurately updating the parameters of the action selection network, and is further conducive to obtaining an accurate shaft hole assembly model.

Please refer to FIG. 6 , which is a schematic flowchart No. 6 of a shaft hole assembly method provided by an exemplary embodiment of the present invention. This embodiment is a detailed description of the foregoing embodiment, and mainly describes the specific process of obtaining the assembly demonstration model. The process of this embodiment is shown in Figure 6, including the following steps:

Step 610: Obtain the teaching status and teaching action.

Specifically, the prior knowledge can be teaching data during the shaft hole assembly process, and the teaching data includes teaching status and teaching actions.

The intelligent agent obtains the teaching status and teaching actions through the demonstration learning algorithm, that is, it imitates the shaft hole assembly process to obtain the teaching status and teaching actions.

Step 620: Model the teaching status and teaching actions to obtain an assembly demonstration model.

Specifically, the expression for the initial assembly demonstration model is as follows:

In the formula, α represents the teaching action, ζ represents the teaching state, λ _k represents the weight coefficient of the k-th Gaussian model, μ _k represents the mean of the k-th Gaussian model, ∑ _k represents the covariance of the k-th Gaussian model, θ = {θ ₁ , θ ₂ ,…, θ _K }, θ _k is the parameter of the Gaussian mixture model, θ _k = {λ _k , μ _k , ∑ _k }.

The intelligent agent inputs the teaching status and teaching actions into the initial assembly demonstration model, determines the parameters of the model, and obtains the constructed assembly demonstration model.

The shaft hole assembly method provided in this embodiment constructs an assembly demonstration model through teaching states and teaching actions, which is beneficial to applying the assembly demonstration model to obtain similarity, thereby accelerating the learning efficiency of the shaft hole assembly module for multi-category skills.

The shaft hole assembly device provided by the present invention will be described below. The shaft hole assembly device described below and the shaft hole assembly method described above can be referenced correspondingly.

Figure 7 is a schematic structural diagram of a shaft hole assembly device provided by an exemplary embodiment of the present invention. As shown in Figure 7, the shaft hole assembly device includes: an execution module 710.

Execution module 710 is used to perform multiple iterative operations; each iterative operation includes:

Obtain the current status of the axis; the current status includes current force information;

Input the current state into the shaft hole assembly model to obtain the current assembly action; the current assembly action includes the translation amount and the rotation amount; the shaft hole assembly model is based on prior knowledge and meta-reinforcement learning on multiple different categories of shafts. Trained in the hole assembly task;

Execute the current assembly action on the axis;

If the depth of the shaft installation hole does not meet the preset assembly requirements, the next iterative operation is performed.

In some embodiments, the shaft hole assembly device further includes: a building module and a training module. in,

The building module is used to construct multiple different categories of shaft hole assembly tasks;

The training module is used to perform multiple iterative training operations; the training module includes: extracting sub-modules, executing sub-modules, obtaining sub-modules, and determining sub-modules, wherein:

The extraction sub-module is used to randomly select a shaft hole assembly task from the plurality of shaft hole assembly tasks of different categories as the current training task;

The execution sub-module is used to execute the current training task and update the parameters of the shaft hole assembly model;

The acquisition sub-module is used to obtain the cumulative reward corresponding to the current training task after the current training task is completed;

The determination sub-module is used to determine the convergence status of the cumulative reward corresponding to the training task based on the cumulative reward corresponding to the current training task and the obtained cumulative reward corresponding to the previous training task;

The execution sub-module is also used to execute the next iterative training when the cumulative reward corresponding to the training task has not converged.

In some embodiments, the execution sub-module is specifically configured to: execute the current training task through multiple iterative training rounds, and update the parameters of the shaft hole assembly model in each iterative training round; the execution sub-module It includes: a first acquisition unit, a second acquisition unit, a second acquisition unit, a first update unit, a third acquisition unit, a second update unit and an execution unit.

The first acquisition unit is used to obtain the similarity between the previous assembly training action and the previous demonstration action, and the current training status; the previous assembly training action was in the previous iterative training round, and the axis The network in the hole assembly model selects the training action output by the network according to the previous training state; the previous demonstration action is the demonstration action output by the assembly demonstration model according to the previous training state; the current training state includes the current training axis force information;

The second acquisition unit is used to input the similarity and the current training status into the action evaluation network in the shaft hole assembly model, and obtain the current first loss function value;

The first update unit is used to update parameters of the action evaluation network based on the current first loss function value;

The third acquisition unit inputs the similarity and the current training status into the action selection network in the shaft hole assembly model to obtain the current second loss function value;

The second update unit is used to update parameters of the action selection network based on the current second loss function value;

The execution unit is configured to execute the next iterative training round when the current training task is not completed.

In some embodiments, the second acquisition unit includes: a first acquisition sub-unit, a second acquisition sub-unit and a third acquisition sub-unit. in:

The first acquisition subunit is used to input the similarity and the current training status into the action evaluation network to obtain the current status value and the current reward value;

The second acquisition subunit is used to acquire the current advantage value according to the current state value, the current reward value, and the latter state value;

The third acquisition subunit is used to acquire the current first loss function value according to the current advantage value.

In some embodiments, the third acquisition unit includes: a fourth acquisition sub-unit and a fifth acquisition sub-unit. in:

The fourth acquisition subunit is used to input the similarity and the current training status into the action selection network to obtain the current assembly training action;

The fifth acquisition subunit is used to acquire the current second loss function value based on the probability of the current assembly training action, the probability of the previous assembly training action, and the current advantage value.

In some embodiments, the shaft hole assembly device further includes: a first acquisition module and a second acquisition module. in:

The first acquisition module is used to acquire teaching status and teaching actions;

The second acquisition module is used to model the teaching state and the teaching action and acquire an assembly demonstration model.

It should be noted here that the above-mentioned shaft hole assembly device provided by the present invention can implement all the method steps implemented by the above-mentioned method embodiments, and can achieve the same technical effect. No further explanation will be given here of the differences between this embodiment and the method embodiments. The same parts and beneficial effects will be described in detail.

Figure 8 is a schematic diagram of the physical structure of an electronic device provided by an exemplary embodiment of the present invention. As shown in Figure 8, the electronic device may include: a processor (processor) 810, a communications interface (Communications Interface) 820, and a memory (memory). 830 and communication bus 840, in which the processor 810, the communication interface 820, and the memory 830 complete communication with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the shaft hole assembly method. The method includes: performing multiple iterative operations; each iterative operation includes: obtaining the current status of the shaft; the current status includes current force information. ; Input the current state into the shaft hole assembly model to obtain the current assembly action; the current assembly action includes the translation amount and the rotation amount; the shaft hole assembly model is based on prior knowledge and meta-reinforcement learning in multiple different categories Trained in the shaft hole assembly task; perform the current assembly action on the shaft; when the depth of the shaft installation hole does not meet the preset assembly requirements, perform the next iteration operation.

In addition, the above-mentioned logical instructions in the memory 830 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code. .

On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can Executing the shaft hole assembly method provided by the above methods, the method includes: performing multiple iterative operations; each iterative operation includes: obtaining the current status of the shaft; the current status includes current force information; inputting the current status The shaft hole assembly model is used to obtain the current assembly action; the current assembly action includes the translation amount and the rotation amount; the shaft hole assembly model is trained in multiple different categories of shaft hole assembly tasks based on prior knowledge and meta-reinforcement learning. ; perform the current assembly action on the shaft; when the depth of the shaft installation hole does not meet the preset assembly requirements, perform the next iterative operation.

In another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. The computer program is implemented when executed by a processor to perform the shaft hole assembly method provided by the above methods. The method includes : Execute multiple iterative operations; each iterative operation includes: obtaining the current state of the shaft; the current state includes the current force information; inputting the current state into the shaft hole assembly model to obtain the current assembly action; the current assembly action Including translation amount and rotation amount; the shaft hole assembly model is trained in multiple different categories of shaft hole assembly tasks based on prior knowledge and meta-reinforcement learning; perform the current assembly action on the shaft; in If the depth of the shaft installation hole does not meet the preset assembly requirements, the next iteration operation is performed.

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be used Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A shaft hole assembling method, characterized by comprising:

performing a plurality of iterative operations; each iterative operation includes:

acquiring the current state of the shaft; the current state comprises current stress information;

inputting the current state into a shaft hole assembly model to obtain a current assembly action; the current assembly action includes a translation amount and a rotation amount; the shaft hole assembly model is obtained by training in a plurality of shaft hole assembly tasks of different categories based on priori knowledge and meta reinforcement learning;

performing the current assembly action on the shaft;

in the case where the depth of the shaft-fitting hole does not meet the preset fitting requirement, the next iteration operation is performed.

2. The shaft hole assembly method according to claim 1, wherein the shaft hole assembly model is trained based on the steps of:

constructing a plurality of shaft hole assembly tasks of different categories;

performing a plurality of iterative training operations; each iterative training operation includes:

randomly extracting one shaft hole assembly task from the shaft hole assembly tasks with different categories to serve as a current training task;

executing the current training task and updating parameters of the shaft hole assembly model;

after the current training task is completed, acquiring a cumulative reward corresponding to the current training task;

determining convergence condition of the accumulated rewards corresponding to the training tasks according to the accumulated rewards corresponding to the current training tasks and the acquired accumulated rewards corresponding to the preamble training tasks;

and executing the next iterative training under the condition that the accumulated rewards corresponding to the training tasks are not converged.

3. The shaft hole assembly method according to claim 2, wherein the performing the current training task and updating parameters of the shaft hole assembly model comprises:

executing the current training task through a plurality of iterative training rounds, and updating parameters of the shaft hole assembly model in each iterative training round; wherein each iterative training round comprises:

Obtaining the similarity between the previous assembly training action and the previous demonstration action and the current training state; the previous assembly training action is a training action which is output by a network selection network in the shaft hole assembly model according to the previous training state in the previous iteration training round; the previous demonstration action is a demonstration action output by the assembly demonstration model according to the previous training state; the current training state comprises current stress information of a training shaft;

inputting the similarity and the current training state into an action evaluation network in the shaft hole assembly model to obtain a current first loss function value;

updating parameters of an action evaluation network based on the current first loss function value;

inputting the similarity and the current training state into an action selection network in the shaft hole assembly model to obtain a current second loss function value;

updating parameters of the action selection network based on the current second loss function value;

and executing the next iteration training round under the condition that the current training task is not completed.

4. The shaft hole assembly method according to claim 3, wherein the inputting the similarity and the current training state into the action evaluation network in the shaft hole assembly model, obtaining a current first loss function value, comprises:

Inputting the similarity and the current training state into the action evaluation network to obtain a current state value and a current rewarding value;

acquiring a current advantage value according to the current state value, the current rewarding value and the latter state value;

and acquiring the current first loss function value according to the current dominant value.

5. The shaft hole assembly method of claim 4, wherein the act of inputting the similarity and the current training state into the shaft hole assembly model selects a network, and obtaining a current second loss function value comprises:

inputting the similarity and the current training state into the action selection network to obtain a current assembly training action;

and acquiring the current second loss function value based on the probability of the current assembly training action, the probability of the last assembly training action and the current dominance value.

6. A shaft hole assembling method according to claim 3, further comprising, before obtaining the similarity between the previous assembling training action and the previous demonstration action:

acquiring a teaching state and a teaching action;

modeling the teaching state and the teaching action to obtain an assembly demonstration model.

7. A shaft hole assembling device, characterized by comprising:

the execution module is used for executing a plurality of iterative operations; each iterative operation includes:

acquiring the current state of the shaft; the current state comprises current stress information;

inputting the current state into a shaft hole assembly model to obtain a current assembly action; the current assembly action includes a translation amount and a rotation amount; the shaft hole assembly model is obtained by training in a plurality of shaft hole assembly tasks of different categories based on priori knowledge and meta reinforcement learning;

performing the current assembly action on the shaft;

in the case where the depth of the shaft-fitting hole does not meet the preset fitting requirement, the next iteration operation is performed.

8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the shaft hole assembly method according to any one of claims 1 to 6 when executing the computer program.

9. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the shaft hole fitting method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the shaft hole assembly method according to any one of claims 1 to 6.