WO2019240330A1 - Image-based strength prediction system and method therefor - Google Patents

Abstract

An image-based strength prediction system and a method therefor are disclosed. The method for predicting strength of a robot arm of the image-based strength prediction system may comprise the steps of: acquiring real images generated by continuously photographing a motion of a robot arm interacting with an object; acquiring time-series motion information related to a change in the motion of the robot arm over time; and predicting an interaction strength of the robot arm on the basis of the real images and the time-series motion information.

Description

Image-based Force Prediction System and Method

The present invention relates to a system and method for predicting the interaction force of a robotic arm using an image.

The interaction force of the robot arm includes the force of the robot arm applied to the object and the force fed back to the robot arm by the property of the object. Therefore, when the user interacts with the object, similar. Therefore, in order to precisely control the arm of the robot, it is necessary to measure the interaction force of the robot arm.

The conventional apparatus for measuring the interaction force of the robot arm measured the interaction force of the robot arm by using a force sensor or a torque sensor attached to the robot. However, there has been a problem that the force sensor and the torque sensor are expensive devices, which contributes to the price increase of the robot arm.

In addition, depending on the structure and size of the robot arm, there is a limit that it is difficult to attach a force sensor or a torque sensor.

And, the force sensor or torque sensor measures the interaction force of the robot arm after the robot arm contacts the object, so if the strength of the object is weak, the force sensor, or the torque sensor, after the robot arm contacts the object, It was also possible that the object would break before the arm's interaction force was measured.

Therefore, there is a need for a method of measuring or predicting the interaction force of the robot arm that will interact with the object before the robot arm contacts the object without a sensor.

The present invention is to study the correlation between the real image and time series motion information, the physical properties of the object and the interaction force of the robot arm, and the real time image of the robot motion, and apply the real time image and time series motion information to the learning results Thereby, it is possible to provide a system and method for predicting the interaction force of a robotic arm without a sensor for measuring the interaction force of the robotic arm.

In addition, the present invention provides an apparatus and method for reproducing a physical sense to a user using only a camera without constructing a complex sensor network by providing feedback to a user who controls the robot arm according to the predicted interaction force of the robot arm. can do.

In addition, the present invention can predict the interaction force of the robot arm using the actual image, so if an error occurs in the sensor, the interaction between the interaction force of the robot arm measured by the sensor and the robot arm predicted using the actual image It is possible to provide an apparatus and a method capable of determining the abnormality of the sensor by comparing the action force.

According to an aspect of the present invention, there is provided a method of predicting a force of a robot arm, the method comprising: obtaining real images generated by continuously photographing an operation of a robot arm interacting with an object; Obtaining time series motion information related to a change in motion of the robot arm over time; And predicting an interaction force of the robot arm based on the actual images and the time series motion information.

Predicting the force prediction method of the robot arm according to an embodiment of the present invention, the learning results of the correlation between the actual image, the time series motion information, the physical properties of the object and the interaction force of the robot arm is stored The interaction force of the robot arm may be predicted by applying the real images, the time series motion information, and the physical properties of the object to a database.

Correlation of the force prediction method of the robot arm according to an embodiment of the present invention, the interaction time and the object of the robot arm predicted by inputting the time series motion information of the robot and the actual images to the deep learning algorithm of the neural network structure It can be learned by comparing the physical properties of the object and the interaction force of the robot arm measured using the sensor and the physical properties of the object.

Deep learning algorithm of the robot arm force prediction method according to an embodiment of the present invention, the process of extracting the region associated with the operation of the robot from the actual images using the CNN (Convolutional Neural Network) that the actual images are input ; Calculating class scores of the time series motion information of the robot using a first FC (fully-connected) layer to which the time series motion information of the robot is input; Inputting the extracted region and class scores of the time series motion information into a recurrent neural network (RNN) to learn a relationship between the area that changes over time and the time series motion information; And calculating class scores of the learning result as a second FC layer into which the learning result of the RNN is input, and predicting interaction force of the robot arm corresponding to the time series motion information and the actual image and physical property values of the object. The correlation between the actual images and time series motion information of the robot and the interaction force of the robot arm may be learned.

The deep learning algorithm of the robot arm force prediction method according to an embodiment of the present invention generates a virtual image representing a change in motion information in a graph form based on the time series motion information and generates a CNN (Convolutional Neural). Inputting to a first convolutional layer of a network; Inputting the real images accumulated over time into a second convolution layer different from the first convolution layer; Matching the virtual image output from the first convolutional layer and the actual image output from the second convolutional layer using a pooling layer; And classifying the matched information through a fully connected layer to predict the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical property of the object. The correlation between time series motion information of the robot and the interaction force of the robot arm may be learned.

Time series operation information of the force prediction method of the robot arm according to an embodiment of the present invention, the change in the power used by the motor of the robot arm to perform the operation of the robot arm changes over time, the robot arm It may include at least one of a change in the position and a change in the operation of the robot arm.

According to an aspect of the present invention, a method of predicting a force of a robot arm includes predicting haptic sensing information by using a material property of the object and an estimated interaction force of the robot arm; And providing feedback to the user of the robot arm according to the haptic sensing information.

According to an aspect of the present invention, there is provided a method of learning a force of a robot arm, the method comprising: obtaining real images generated by photographing an operation of a robot arm related to an object and time series motion information related to the operation of the robot arm; Inputting the real images and the time series motion information into a deep learning algorithm to predict the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical properties of the object; And comparing the predicted interaction force of the robot arm and the physical property of the object with the interaction force of the robot arm measured using a sensor and the physical property of the object. The method may include learning correlations between interaction forces of the robotic arms.

Predicting the interaction force of the robot arm in the force learning method of the robot arm according to an embodiment of the present invention, using the CNN to input the actual image is extracted an area related to the operation of the robot from the actual image Doing; Calculating class scores of time-series motion information of the robot using a first fully connected layer (FC) to which time-series motion information of the robot is input; Inputting the extracted region and class scores of the time series operation information into an RNN to learn a relationship between the area that changes over time and the time series operation information; And calculating class scores of the learning result as a second FC layer into which the learning result of the RNN is input, and predicting interaction force of the robot arm corresponding to the time series motion information and the actual image and physical property values of the object. It may include.

Predicting the interaction force of the robot arm in the force learning method of the robot arm according to an embodiment of the present invention, the virtual image showing a change in the motion information over time based on the time series motion information in the form of a graph Generating and inputting to the first convolutional layer of the CNN; Inputting the actual images accumulated over time to a second convolution layer different from the first convolution layer; Matching the virtual images output from the first convolutional layer with the actual images output from the second convolutional layer using a pooling layer; And classifying the matched information through a fully connected layer to predict the interaction force of the robot arm corresponding to the time series motion information and the actual image and the physical property of the object.

An apparatus for predicting a force of a robot arm according to an embodiment of the present invention includes an image acquisition unit for acquiring real images generated by continuously photographing an operation of a robot arm interacting with an object; A motion information acquisition unit for obtaining time series motion information related to a motion change of the robot arm over time; And a force estimator for predicting the interaction force of the robot arm based on the actual images and the time series motion information.

The force predicting unit of the force predicting device of the robot arm according to an embodiment of the present invention, the database storing the learning results of the correlation between the actual image, the time series motion information, the physical properties of the object and the interaction force of the robot arm The interaction force of the robot arm may be predicted by applying the actual images, the time series motion information, and the physical properties of the object.

Time series operation information of the force prediction device of the robot arm according to an embodiment of the present invention, the change in power used by the motor of the robot arm to perform the operation of the robot arm that changes over time, the robot arm It may include at least one of a change in the position and a change in the operation of the robot arm.

According to an aspect of the present invention, there is provided an apparatus for predicting force of a robot arm, comprising: a haptic sensing information predictor for predicting haptic sensing information by using the physical force of the object and the predicted interaction force of the robot arm; And a feedback unit configured to provide feedback to the user of the robot arm according to the haptic sensing information.

According to an aspect of the present invention, there is provided a force learning apparatus for a robotic arm, including: a communicator configured to obtain actual images generated by photographing an operation of a robot arm related to an object and time series motion information related to the operation of the robot arm; A force estimator for inputting the real images and the time series motion information into a deep learning algorithm to predict the interaction force of the robot arm and the physical properties of the object corresponding to the time series motion information and the real image; And comparing the predicted interaction force of the robot arm and the physical property of the object with the interaction force of the robot arm measured using a sensor and the physical property of the object. And, it may include a correlation learning unit for learning the correlation between the interaction force of the robot arm.

The force predicting unit of the force learning device of the robot arm according to an embodiment of the present invention extracts an area related to the motion of the robot from the real images by using the CNN to which the real images are input, and time-series motion information of the robot. Calculates class scores of the time series motion information of the robot by using the first FC layer inputted, inputs the extracted region and class scores of the time series motion information to RNN, and changes the area and time series Learn the relationship between the motion information, calculate the class scores of the learning result to the second FC layer into which the learning result of the RNN is input, and the interaction force of the robot arm corresponding to the time series motion information and the actual image; The physical properties of the object can be predicted.

The force predicting unit of the force learning device of the robotic arm according to an embodiment of the present invention generates a virtual image representing a change in motion information over time based on the time series motion information in a graph form, thereby generating the first cone ball of the CNN. Inputting the real image accumulated in the solution layer as time passes, into the second convolution layer different from the first convolution layer, and outputting the virtual image and the second image output from the first convolution layer The real images output from the convolutional layer are matched using the unified layer, the matched information is classified through the FC layer, and the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical property values of the object. It can be predicted.

According to an embodiment of the present invention, the correlation between the actual image and time-series motion information, the physical properties of the object and the interaction force of the robot arm, and the real-time image and time-series motion By applying the information to the learning results, it is possible to predict the interaction force of the robot arm without a sensor for measuring the interaction force of the robot arm.

And, according to an embodiment of the present invention, by providing feedback to the user controlling the robot arm in accordance with the predicted interaction force of the robot arm, to reproduce the physical senses to the user using only the camera without configuring a complex sensor network You can.

In addition, according to an embodiment of the present invention, since the interaction force of the robot arm can be predicted using the actual image, when an error occurs in the sensor, the interaction force of the robot arm measured by the sensor and the actual image are used. By comparing the interaction forces of the predicted robot arms, it is possible to determine whether the sensor is abnormal.

1 is a view showing an image-based force prediction system according to an embodiment of the present invention.

2 is a view showing the structure of the force learning device of the robot arm included in the image-based force prediction system according to an embodiment of the present invention.

3 is an example of a process of learning the correlation between the image and motion information and the interaction force of the robot arm in the force learning device of the robot arm according to an embodiment of the present invention.

Figure 4 is an example of the interaction force prediction method of the robot arm using the CNN and RNN in accordance with an embodiment of the present invention.

5 is an example of a method for predicting interaction force of a robot arm using a CNN according to an embodiment of the present invention.

6 is a diagram illustrating a structure of a force predicting device of a robot arm included in an image-based force predicting system according to an embodiment of the present invention.

Figure 7 is an example of the interaction force prediction process of the robot arm according to an embodiment of the present invention.

8 is another example of a process of predicting the interaction force of the robot arm according to an embodiment of the present invention.

9 is a flowchart illustrating a method for learning the interaction force of the robot arm according to an embodiment of the present invention.

10 is a flowchart illustrating a method for predicting interaction force of a robot arm according to an embodiment of the present invention.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, various changes may be made to the embodiments so that the scope of the patent application is not limited or limited by these embodiments. It is to be understood that all changes, equivalents, and substitutes for the embodiments are included in the scope of rights.

The terminology used herein is for the purpose of description and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described on the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

In addition, in the description with reference to the accompanying drawings, the same components regardless of reference numerals will be given the same reference numerals and duplicate description thereof will be omitted. In the following description of the embodiment, when it is determined that the detailed description of the related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

The interaction force learning method of the robot arm according to an embodiment of the present invention may be performed by the force learning apparatus of the robot arm included in the image-based force prediction system. In addition, the method of predicting the interaction force of the robot arm according to an embodiment of the present invention may be performed by the apparatus for predicting the force of the robot arm included in the image-based force prediction system.

1 is a view showing an image-based force prediction system according to an embodiment of the present invention.

As shown in FIG. 1, the image-based force prediction system may include a force learning device 110, a database 120, and a force prediction device 130 of the robot arm.

The force learning device 110 of the robot arm is configured to continuously record the actual images generated by continuously photographing the motion of the robot arm related to the object, time series motion information related to the motion of the robot arm, the physical properties of the object, and the interaction force of the robot arm. Correlation can be learned. In addition, the force learning apparatus 110 of the robot arm may store the learned correlation in the database 120. At this time, the interaction force of the robot arm is the grip force of the robot arm holding the object, the contact force of the robot arm in contact with the object, the pull force of the robot arm pulling the object, the force of the robot arm pushing the object ( Push force), and the direction of the force of the robot arm that caught or touched the object.

In this case, the physical property value of the object may include at least one of elasticity, rigidity, density, and specific gravity of the object. In addition, the time series motion information may include at least one of a change in power used by the motor of the robot arm, a change in position of the robot arm, and a change in motion of the robot arm to perform the operation of the robot arm that changes over time. have. In addition, time series operation information may be generated corresponding to each of the actual images. For example, when the actual image is taken at 0.1 second intervals, the robot arm may also generate motion information at a corresponding time point at 0.1 second intervals. In addition, the force learning apparatus 110 of the robot arm may match the motion information of the robot arm corresponding to each frame of the actual image generated at 0.1 second intervals.

Specific configurations and operations for learning the correlation by the force learning device 110 of the robot arm will be described in detail with reference to FIGS. 2 to 5.

The database 120 may store and manage correlations between actual images learned by the force learning apparatus 110 of the robot arm, time series motion information, material properties of the object, and interaction forces of the robot arm.

The force predicting device 130 of the robot arm includes at least one of actual images generated by photographing the motion of the robot arm related to the object, time series motion information related to the motion of the robot arm, and physical property values of the object in the database 120. It can be applied to the loaded correlation to predict the interaction force of the robot arm.

A detailed configuration and operation for the robot arm force predicting device 130 to predict the interaction force of the robot arm will be described in detail with reference to FIGS. 6 to 8.

The image-based force prediction system learns the correlation between the real image of the robot's motion and time series motion information, the property of the object and the interaction force of the robot arm, and the real image and the time series motion information of the robot motion. Applied to the results, the interaction force of the robot arm can be predicted without the sensor for measuring the interaction force of the robot arm. In addition, the image-based force prediction system provides feedback to the user who controls the robot arm according to the predicted interaction force of the robot arm, thereby reproducing the physical senses to the user using only the camera without constructing a complex sensor network. .

In addition, the image-based force prediction system can predict the interaction force of the robot arm using the actual image, so if an error occurs in the sensor, the robot predicts the interaction force of the robot arm measured by the sensor and the actual image. By comparing the interaction forces of the arms, it is possible to determine whether the sensor is abnormal.

In addition, since the image-based force prediction system can predict the interaction force of the robot arm without the expensive and complicated sensor network, the image-based force prediction system can be implemented at a lower cost than the interaction force measurement system of the conventional robot arm using the sensor. Thus, it is possible to increase the prevalence of a system for measuring or predicting the interaction force of the robotic arm.

2 is a view showing the structure of the force learning device of the robot arm included in the image-based force prediction system according to an embodiment of the present invention.

The force learning apparatus 110 of the robot arm may include an image acquirer 210, an operation information acquirer 220, a force predictor 230, and a correlation learner 240 as illustrated in FIG. 1. have. In this case, the force predicting unit 230 and the correlation learning unit 240 may be different modules or modules for executing programs included in one process.

The image acquirer 210 may acquire an actual image generated by photographing the operation of the robot arm 202 related to the object by the camera 201 linked to the learning robot arm 202. In this case, the image acquisition unit 210 may be a communicator for receiving the actual image wirelessly from the camera 201, or a port to which a wire for receiving the actual image from the camera 201 is coupled.

The motion information acquisition unit 220 may obtain time series motion information related to the motion of the robot arm 202. In this case, the motion information acquisition unit 220 may be a communicator that receives the motion information from the robot arm 202 wirelessly, or a port to which a wire for receiving the motion information from the robot arm 202 is coupled.

In addition, although the image acquisition unit 210 and the motion information acquisition unit 220 are described in separate configurations in FIG. 2, the force learning device 110 of the robot arm uses a single communicator to acquire the actual image and the motion information. All acquisitions may be performed.

The force predictor 230 may input real images and time series motion information to a deep learning algorithm to predict interaction forces and physical property values of the robot arm corresponding to the time series motion information and the real image. At this time, the force prediction unit 230 may predict the interaction force of the robot arm using a convolutional neural network (CNN) and a recurrent neural network (RNN), or may predict the interaction force of the robot arm using only the CNN.

A method of predicting the interaction force of the robot arm by using the CNN and the RNN will be described in detail with reference to FIG. 4. In addition, a method of predicting the interaction force of the robot arm using only the CNN will be described in detail with reference to FIG. 5.

The correlation learning unit 240 compares the interaction force of the robot arm measured by the sensor and the property of the object with the interaction force of the robot predicted by the force predictor 230 and the property of the object. And correlation between time series motion information, object properties, and interaction forces of the robot arm.

Specifically, the correlation learning unit 240 may be equal to the physical force of the robot arm and the physical force of the object measured by the sensor using the sensor and the interaction force of the robot arm predicted by the force predictor 230 or In case there is a difference below the error range, it can be learned that there is a correlation between the actual images inputted by the force predictor 230 and the time series motion information, the property values of the predicted object, and the interaction force of the robot arm. . In addition, the correlation learning unit 240 may measure the interaction force of the robot arm and the physical property values of the object predicted by the force predictor 230 and the physical force and error range of the object and the interaction force of the robot arm measured using the sensor. If there is an excessive difference, it may be determined that there is no correlation between the actual images inputted by the force predicting unit 230, time series motion information, the predicted physical properties of the object, and the interaction force of the robot arm. At this time, the force predicting unit 230 repeatedly performs a process of predicting the interaction force of the robot arm until the physical properties of the object correlated with the actual images and time series motion information and the interaction force of the robot arm are predicted. can do.

3 is an example of a process of learning the correlation between the image and motion information and the interaction force of the robot arm in the force learning device of the robot arm according to an embodiment of the present invention.

The robot arm 320 or the surgical robot forceps 240 connected to the force learning device 110 of the robot arm may include a sensor for measuring the interaction force of the robot arm. For example, the robot arm 320, or surgical robot forceps 240 may include at least one of a force sensor, a torque sensor and a pressure sensor.

The camera 310 corresponding to the robot arm 320 may transmit the actual images 311 photographed at regular intervals of the operation of the robot arm 320 to the force learning apparatus 110 of the robot arm. In addition, the robot arm 320 may transmit time series operation information 321 corresponding to each of the frames of the actual image 311 to the force learning apparatus 110 of the robot arm. In this case, the robot arm 320 may transmit the interaction force of the robot arm measured using the sensor to the force learning device 110 of the robot arm together with time series operation information 321.

In addition, the force learning device 110 of the robot arm contacts the robot arm 320 using the actual image 311 and time series motion information 321, or the interaction force of the robot arm held by the robot arm 320. Can be predicted. In addition, the force learning apparatus 110 of the robot arm may predict the physical properties of the object according to the position and position of the object using the actual image 311. Next, the force learning apparatus 110 of the robot arm compares the predicted interaction force of the robot arm with the interaction force of the robot arm received from the robot arm 320 and compares the actual images 411 with time series motion information ( 321), the correlation between the physical properties of the object and the interaction force of the robot arm can be learned.

In addition, the camera 330 corresponding to the surgical robot forceps 240 may transmit the actual image 331 of the operation of the surgical robot forceps 240 at a predetermined time interval to the force learning device 110 of the robot arm. . In addition, the surgical robot forceps 240 may transmit time series operation information 341 corresponding to each of the frames of the actual image 313 to the force learning device 110 of the robot arm. In this case, the surgical robot forceps 240 may measure the interaction force of another robot arm in contact with the object held by the surgical robot forceps 240 using a sensor. In addition, the surgical robot forceps 240 may transmit the measured interaction force of the robot arm to the force learning device 110 of the robot arm together with time series operation information 341.

In addition, the force learning device 110 of the robot arm uses the actual image 331 and time series motion information 341 to mutually interact with physical properties of other objects and other robot arms in contact with the object held by the surgical robot forceps 240. Predict the action force. In addition, the force learning apparatus 110 of the robot arm may predict the physical properties of other objects according to the positions and positions of other objects using the actual image 331. Next, the force learning apparatus 110 of the robot arm compares the predicted interaction force of the robot arm with the interaction force of the robot arm received from the surgical robot forceps 240 and the actual images 411 and time series motion information. 321, the correlation between the physical properties of other objects and the interaction force of the robot arm can be learned.

At this time, when the shooting interval between the camera 310 and the camera 330 is 0.1 second, the robot arm 320 and the surgical robot tongs 240 also generate the motion information of the corresponding time point at 0.1 second intervals, respectively, the force of the robot arm. The transmission may be transmitted to the learning device 110. In addition, the force learning apparatus 110 of the robot arm may match the motion information of the robot arm corresponding to each of the actual images generated at 0.1 second intervals.

Figure 4 is an example of the interaction force prediction method of the robot arm using the CNN and RNN in accordance with an embodiment of the present invention.

First, the force estimator 230 extracts an area related to the robot operation from the actual images using the CNN 411 to which real images acquired from each of at least one camera such as camera 1 and camera 2 are input. Can be. For example, the force predictor 230 extracts a region including the robot arm and the object from the actual image generated by photographing the robot arm as an area related to the robot's motion, and thus is related to the image to be processed by the RNN 420. The size of the information can be minimized.

In addition, when one robot arm is photographed by two cameras, the force estimator 230 displays a region related to the operation of the robot extracted by the CNN 411 corresponding to each of the cameras as illustrated in FIG. 4. Aggregated at FC layer 412 and forwarded to RNN 420. In this case, the fourth FC layer 412 may match and output regions related to the operations of the robots extracted from different images.

In addition, the force predictor 230 may process the time series motion information of the robot to the first fully-connected (FC) layer 413 to which the time series motion information of the robot is input, and transmit the processed time series motion information to the RNN 420. In this case, the first FC layer 413 may be configured of a plurality of FC layers as shown in FIG. 4. The first FC layer 413 may calculate and output time series motion information of the robot.

The force predictor 230 may generate an input network including the CNN 411 and the first FC layer 413 for each actual image. For example, when there are N actual images, the force predictor 230 may generate the input network 1 410 to the input network N as shown in FIG. 4. At this time, the input network 2 to the input network N has only the input between the actual image and the operation information corresponding to the actual image, the structure for processing the actual image and the operation information is the CNN 411 of the input network 1 (410) and It may be the same as the first FC layer 413.

Next, the force predicting unit 230 may learn the relationship between the region that changes over time and time series operation information using the RNN 420. For example, as shown in FIG. 4, the RNN 420 includes a plurality of hierarchical layers of LSTM (area) related to the operation of the robot output from the plurality of input networks and processed time series operation information. long short term memory networks) to learn the relationship between regions that change over time and time series behavior information. Although the LSTM is used in FIG. 4, the force predictor 230 may learn a relationship between a region that changes over time and time-series operation information by using another network corresponding to the RNN in addition to the LSTM.

Finally, the force predicting unit 230 processes the learning result to the second FC layer 430 into which the learning result of the RNN is input, and thus the interaction force (Task1) of the robot arm corresponding to the time series motion information and the actual image and the object. The property value (Task2) can be predicted. In this case, the second FC layer 430 may calculate class scores of respective relationships between a region that changes over time and time series operation information.

5 is an example of a method for predicting interaction force of a robot arm using a CNN according to an embodiment of the present invention.

First, the force estimator 230 generates a virtual image 521 representing a change in motion information over time based on time series motion information, and inputs the virtual image 521 to the first convolutional layer 520 of the CNN. Can be. For example, the virtual image 521 may be generated by converting a covariance matrix between the vectorized time series motion information 522 and 523 into an image. In this case, the first convolutional layer 520 may analyze correlation between time series motion information using the virtual image 521.

Next, the force predictor 230 may input the actual images 511 accumulated over time to the second convolutional layer 510 which is different from the first convolutional layer.

Next, the force predictor 230 uses a pooling layer 530 to combine the virtual image output from the first convolutional layer 520 and the actual image output from the second convolutional layer 510. Can be matched.

Finally, the force predictor 230 classifies the matched information through the FC layer 540 to predict the interaction force of the robot arm and the physical property of the object corresponding to the real time image. have.

6 is a diagram illustrating a structure of a force predicting device of a robot arm included in an image-based force predicting system according to an embodiment of the present invention.

As shown in FIG. 6, the force predicting device 130 of the robot arm includes an image acquirer 610, a motion information acquirer 620, a force predictor 630, a haptic sensing information predictor 640, and feedback. It may include a portion 640. In this case, the force predictor 630 and the haptic sensing information predictor 640 may be different modules, or modules for executing a program included in one process.

The image acquisition unit 610 may acquire an actual image generated by photographing the operation of the robot arm 602 related to the object by the camera 601 linked to the robot arm 602 to predict the force. At this time, the image acquisition unit 610 may be a communication port for receiving the actual image from the camera 601 wirelessly, or a port to which a wire for receiving the actual image from the camera 601 is coupled. In this case, the actual image may include an object deformed by the operation of the robot arm 602. In detail, the continuous real image may be included in the process of deforming the object by the operation of the robot arm 602.

The motion information acquisition unit 620 may obtain time series motion information related to the motion of the robot arm 602. In this case, the motion information acquisition unit 220 may be a communicator that receives the motion information from the robot arm 602 wirelessly, or a port to which a wire for receiving the motion information from the robot arm 602 is coupled.

In addition, although the image acquisition unit 610 and the operation information acquisition unit 620 are described in separate configurations in FIG. 6, the robot arm force predicting device 130 uses a single communicator to acquire actual image and operation information. All acquisitions may be performed.

The force predictor 630 may predict the interaction force of the robot arm based on the actual images and time series motion information. At this time, the force predictor 630 may store the actual images, time series motion information, and the object in the database 120 in which the learning result of the correlation between the actual images, time series motion information, physical properties of the object, and the interaction force of the robot arm is stored. By applying the properties, we can predict the interaction force of the robot arm.

The haptic sensing information prediction unit 640 may predict the haptic sensing information by using the material property of the object and the predicted interaction force of the robot arm.

The feedback unit 640 may provide feedback to the robot arm user by transmitting a feedback signal to the robot arm controller 603 according to the predicted haptic sensing information. In this case, the feedback unit 640 may be a communicator for transmitting a feedback signal to the robot arm controller 603 wirelessly, or a port to which a wire for transmitting a feedback signal to the robot arm controller 603 is coupled.

The robot arm force prediction apparatus 130 stores the actual images and time series motion information and objects in a database 120 in which the learning results of the correlation between the actual images, time series motion information, physical properties of the object, and interaction forces of the robot arm are stored. By applying the properties of, we can predict the interaction force of the robot arm without the sensor to measure the interaction force of the robot arm.

Figure 7 is an example of the interaction force prediction process of the robot arm according to an embodiment of the present invention.

The force predicting device 130 of the robot arm may obtain actual images 710 generated by photographing the operation of the robot arm 602 from a camera linked to the robot arm 602. At this time, the continuous real images 710 are as shown in the image 714 of the robot arm 602 is in contact with the object from the image 711 to the robot arm 602 as shown in FIG. The robot arm 602 may be an image of sequentially photographing a process of interacting with an object. In addition, the robot arm 602 may operate according to the control signal 706 sent by the robot arm controller 603 to the robot arm 602.

In addition, the force predicting device 130 of the robot arm may obtain time series motion information 720 related to the motion of the robot arm 602 from the robot arm 602.

Next, the force predicting device 130 of the robot arm may preprocess the real images 710 and time series motion information 720 using the CNN and the FC layer 701. In this case, the force predicting device 130 of the robot arm may extract an area related to the operation of the robot from the actual images 710 using the CNN and transmit the extracted area to the RNN 702. In addition, the force predicting device 130 of the robot arm may process the time series motion information of the robot to the FC layer and transmit the processed information to the RNN 702.

Subsequently, the force predicting device 130 of the robot arm may use the RNN 702 to identify a relationship between a region that changes over time and time series operation information. In addition, the force predicting device 130 of the robot arm may retrieve the interaction force of the robot arm and the physical property values of the object corresponding to the relationship between the area checked in the database 120 and the time series motion information.

Next, the force predicting device 130 of the robot arm may output the detected interaction force of the robot arm and the physical property of the object as the force prediction result 703.

Also, the force predicting device 130 of the robot arm may predict haptic sensing information to be fed back to the user according to the force prediction result 703. In addition, the force predicting device 130 of the robot arm may transmit a feedback signal 730 including the haptic sensing information to the robot arm controller 603. In this case, the robot arm controller 603 may provide feedback to a user who controls the robot arm 602 according to the received feedback signal 730.

8 is another example of a process of predicting the interaction force of the robot arm according to an embodiment of the present invention.

8 illustrates that when the robot arm 602 interacts with a breakable object (a surgical tool, a patient's organ), such as a surgical robot arm, the force predicting device 130 of the robot arm warns of a risk of damage to the object. Yes.

The force predicting device 130 of the robot arm may obtain actual images 810 generated by photographing the operation of the robot arm 602 from a camera linked to the robot arm 602. In this case, the actual images 810 may include a first image 811 to a current image 814 in which the object is deformed by the operation of the robot arm 602. In addition, the robot arm 602 may operate according to the control signal 800 transmitted by the robot arm controller 603 to the robot arm 602.

In addition, the force predicting device 130 of the robot arm may obtain time series motion information 820 related to the motion of the robot arm 602 from the robot arm 602.

Next, the force predicting device 130 of the robot arm may preprocess the real images 810 and the time series operation information 820 using the CNN and the FC layer 801. In this case, the force predicting device 130 of the robot arm may extract an area related to the operation of the robot from the actual images 810 using the CNN and transmit the extracted area to the RNN 802. In addition, the force predicting device 130 of the robot arm may process the time series motion information of the robot to the FC layer and transmit the processed information to the RNN 802.

Subsequently, the force predicting device 130 of the robot arm may use the RNN 802 to check a relationship between a region that changes over time and time series operation information. In addition, the force predicting device 130 of the robot arm may retrieve the interaction force of the robot arm and the physical property values of the object corresponding to the relationship between the area checked in the database 120 and the time series motion information. Next, the force predicting device 130 of the robot arm may output the detected interaction force of the robot arm and the physical property of the object as the force prediction result 803.

In this case, the force predicting device 130 of the robot arm may determine the possibility of breakage of the object by comparing the interaction force of the robot arm included in the force prediction result 803 and the property values of the object. In addition, when the possibility of damage of the object is greater than or equal to a threshold value, the force predicting device 130 of the robot arm may transmit a signal 840 to the robot arm controller 603 to warn of the possibility of the object being damaged. In this case, the robot arm controller 603 may be configured not to operate a warning sound or a warning lamp according to the received signal 840 or to transmit the control signal 800 in the direction in which the object is damaged to the robot arm 602. have.

9 is a flowchart illustrating a method for learning the interaction force of the robot arm according to an embodiment of the present invention.

In operation 910, the image acquirer 210 may acquire an actual image generated by photographing the operation of the robot arm 202 related to the object by the camera 201 linked to the learning robot arm 202. In this case, the motion information acquisition unit 220 may obtain time series motion information related to the motion of the robot arm 202. In addition, the force learning device 110 of the robot arm may acquire both the actual image and the time series motion information by using one communicator.

In operation 920, the force estimator 230 may input real images and time series motion information to a deep learning algorithm to predict the interaction force of the robot arm corresponding to the time series motion information and the real image and physical property values of the object. At this time, the force predicting unit 230 may predict the interaction force of the robot arm using the CNN and RNN, or may predict the interaction force of the robot arm using only the CNN.

In operation 930, the correlation learning unit 240 measures the interaction force and the object property of the robot arm measured using the sensor and the interaction force and the object property value of the robot arm predicted by the force predictor 230. By comparison, the correlation between the actual images, time series motion information, the property of the object, and the interaction force of the robot arm can be learned.

10 is a flowchart illustrating a method for predicting interaction force of a robot arm according to an embodiment of the present invention.

In operation 1010, the image acquirer 610 may acquire an actual image generated by photographing the operation of the robot arm 602 related to the object by the camera 601 linked to the robot arm 602 to predict the force. . In this case, the motion information acquisition unit 620 may obtain time series motion information related to the motion of the robot arm 602. In addition, the force predicting device 130 of the robot arm may acquire both the actual image and the time series motion information using a single communicator.

In operation 1020, the force predictor 630 may predict the interaction force of the robot arm based on the actual images and the time series motion information. In this case, the force predictor 630 may store the actual images, time series motion information, and the object in the database 120 in which the learning result of the correlation between the actual images, time series motion information, physical properties of the object, and the interaction force of the robot arm is stored. By applying the properties, we can predict the interaction force of the robot arm.

In operation 1030, the haptic sensing information prediction unit 640 may predict the haptic sensing information by using the physical properties of the object and the interaction force of the robot arm predicted in operation 1020.

In operation 1040, the feedback unit 640 may provide feedback to the robot arm controller 603 by transmitting a feedback signal to the robot arm controller 603 according to the haptic sensing information predicted in operation 1030.

The present invention is to study the correlation between the real image and time series motion information, the physical properties of the object and the interaction force of the robot arm, and the real time image of the robot motion, and apply the real time image and time series motion information to the learning results Thus, the interaction force of the robot arm can be predicted without a sensor for measuring the interaction force of the robot arm.

In addition, the present invention provides feedback to the user who controls the robot arm according to the predicted interaction force of the robot arm, thereby reproducing the physical senses to the user using only the camera without constructing a complex sensor network.

In addition, the present invention can predict the interaction force of the robot arm using the actual image, so if an error occurs in the sensor, the interaction between the interaction force of the robot arm measured by the sensor and the robot arm predicted using the actual image By comparing the working force, it is possible to determine whether the sensor is abnormal.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine code, such as produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

Although the embodiments have been described with reference to the accompanying drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than the described method, or other components. Or, even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the following claims.

Claims (20)

Obtaining an actual image generated by photographing interactions between objects; And Estimating an interaction force between the objects based on a change in position or deformation of the objects included in the real image Interactive force prediction method comprising a. The method of claim 1, Obtaining motion information of a robot arm related to a change in position of the objects or deformation of the objects over time; More, The predicting step, Interactive force prediction method for predicting the interaction force of the robot arm based on the actual image and the motion information. The method of claim 2, Acquiring the actual images, Acquire a plurality of real images generated by continuously photographing the interactions between the objects over time, Acquiring operation information of the robot arm, Acquire time series motion information of the robot arm over time, The predicting step, And predicting the interaction force of the robot arm based on the actual images and the time series motion information. The method of claim 3, The predicting step, The real images, the time series motion information, and the property values of the object are applied to a database in which a learning result of correlation between the real images, the time series motion information, the property values of the objects, and the interaction force of the robot arm is stored. Interactive force prediction method to predict the interaction force of the robot arm. The method of claim 4, wherein The correlation is Time-series motion information of the robot and the actual images are input to a deep learning algorithm of a neural network structure to predict the interaction force of the robot arm and the interaction force of the robot arm measured using the physical properties of the object and the sensor. A method of predicting interaction forces learned by comparing property values. The method of claim 3, The deep learning algorithm, Extracting a region related to the operation of the robot from the actual images by using a convolutional neural network (CNN) to which the actual images are input; Calculating class scores of the time series motion information of the robot using a first FC (fully-connected) layer to which the time series motion information of the robot is input; Inputting the extracted region and class scores of the time series motion information into a recurrent neural network (RNN) to learn a relationship between the area that changes over time and the time series motion information; And Calculating class scores of the learning result as a second FC layer in which the learning result of the RNN is input, and predicting interaction force of the robot arm corresponding to the time series motion information and the actual image and physical property values of the object; And learning a correlation between the real images and time series motion information of the robot and the interaction force of the robot arm. The method of claim 5, The deep learning algorithm, Generating a virtual image representing a change in motion information over time based on the time series motion information in a graph form and inputting the virtual image to a first convolutional layer of a convolutional neural network (CNN); Inputting the real images accumulated over time into a second convolution layer different from the first convolution layer; Matching the virtual image output from the first convolutional layer and the actual image output from the second convolutional layer using a pooling layer; And Classifying the matched information through a fully connected layer to predict the interaction force of the robot arm corresponding to the time series motion information and the actual image and the physical property of the object. And learning a correlation between the real images and time series motion information of the robot and the interaction force of the robot arm. The method of claim 3, The time series operation information, Predicting an interaction force including at least one of a change in power used by a motor of the robot arm, a change in position of the robot arm, and a change in motion of the robot arm to perform an operation of the robot arm that changes over time. Way. The method of claim 2, Predicting haptic sensing information using a material property of the object and the predicted interaction force of the robot arm; And And providing feedback to the user of the robotic arm in accordance with the haptic sensing information. Obtaining real images generated by photographing motions of the robot arm related to the object and time series motion information related to the motion of the robot arm; Inputting the real images and the time series motion information into a deep learning algorithm to predict the interaction force of the robot arm corresponding to the time series motion information and the real image and the physical properties of the object; The actual images and the time-series motion information, the physical property values of the object, and the physical force and the physical property values of the object are compared with the predicted interaction force of the robot arm and the physical property of the object measured using a sensor. Learning correlations between interaction forces of the robotic arms; And Storing the correlation in a database Robot arm power learning method comprising a. The method of claim 10, Predicting the interaction force of the robot arm, Extracting an area related to the operation of the robot from the real images using the CNN to which the real images are input; Calculating class scores of time-series motion information of the robot using a first fully connected layer (FC) to which time-series motion information of the robot is input; Inputting the extracted region and class scores of the time series operation information into an RNN to learn a relationship between the area that changes over time and the time series operation information; And Calculating class scores of the learning result with a second FC layer to which the learning result of the RNN is input, and predicting interaction time of the robot arm corresponding to the time series motion information and the actual image and physical property values of the object; Robot arm power learning method comprising a. The method of claim 10, Predicting the interaction force of the robot arm, Generating a virtual image representing a change in motion information over time based on the time series motion information in a graph form and inputting the virtual image to a first convolutional layer of the CNN; Inputting the actual images accumulated over time to a second convolution layer different from the first convolution layer; Matching the virtual images output from the first convolutional layer with the actual images output from the second convolutional layer using a pooling layer; And Classifying the matched information through a fully connected layer to predict the interaction force of the robot arm corresponding to the time series motion information and the actual image and the physical property of the object. Robot arm power learning method comprising a. An image obtaining unit obtaining an actual image generated by photographing interactions between objects; And A force estimator for predicting an interaction force between the objects based on a change in position of the objects or deformation of the objects included in the real image Interactive force prediction device comprising a. The method of claim 13, A motion information acquisition unit for acquiring motion information of a robot arm related to a change in position of the objects or deformation of the objects over time; More, The force predictor, And an interaction force prediction device predicting the interaction force of the robot arm based on the actual image and the time series motion information. The method of claim 14, The force predictor, The real images, the time series motion information, and the property values of the object are applied to a database in which a learning result of correlation between the real images, the time series motion information, the property values of the object, and the interaction force of the robot arm is stored. Predicting the Interaction Force of the Robot Arm Interactive force prediction device comprising a. The method of claim 14, The time series operation information, Predicting an interaction force including at least one of a change in power used by a motor of the robot arm, a change in position of the robot arm, and a change in motion of the robot arm to perform an operation of the robot arm that changes over time. Device. The method of claim 14, A haptic sensing information predictor for predicting haptic sensing information using the estimated physical properties of the object and the interaction force of the robot arm; And Feedback unit for providing feedback to the user of the robot arm according to the haptic sensing information Interactive force prediction device further comprising. A communicator configured to acquire actual images generated by photographing the motion of the robot arm related to the object and time series motion information related to the motion of the robot arm; A force estimator for inputting the real images and the time series motion information into a deep learning algorithm to predict the interaction force of the robot arm and the physical properties of the object corresponding to the time series motion information and the real image; And The actual images and the time-series motion information, the physical property values of the object, and the physical force and the physical property values of the object are compared with the predicted interaction force of the robot arm and the physical property of the object measured using a sensor. Correlation learning unit for learning the correlation between the interaction force of the robot arm Force learning device of the robot arm comprising a. The method of claim 18, The force predictor, Extracts an area related to the operation of the robot from the actual images by using the CNN into which the actual images are input, and uses the first FC layer to which the time-series motion information of the robot is input, and class scores of the time series operation information of the robot. Calculate a relation between the time-domain operation information and the time-domain operation information that is changed over time by inputting the extracted region and class scores of the time-series operation information to the RNN. The robot arm force learning apparatus for calculating class scores of the learning result with an FC layer to predict interaction force of the robot arm and physical properties of the object corresponding to the time series motion information and the actual image. The method of claim 18, The force predictor, Generates a virtual image representing a change in the motion information over time based on the time series motion information in a graph form, inputs it into a first convolutional layer of the CNN, and stores the actual images accumulated over time. The first convolution layer is input to a second convolution layer different from each other, and the virtual images output from the first convolutional layer and the actual images output from the second convolutional layer are matched using an integrated layer, and matched. And classifying information through an FC layer to predict the interaction force of the robot arm corresponding to the time series motion information and the actual image and the physical property of the object.

Images