WO2024201655A1 - Learning data generation device, robot system, learning data generation method, and learning data generation program - Google Patents

Abstract

The present invention achieves provision of a learning data generation device that can generate learning data without spending a high labor cost and an enormous work time. Provided is a learning data generation device that generates learning data to be used in machine learning comprises a data acquisition unit, a data processing unit, and a data storage unit. The data acquisition unit acquires an image of an existing area of a plurality of workpieces, the data processing unit estimates at least area information reflecting an area range of the whole or a part of at least one workpiece on the basis of the acquired image and generates teaching data that includes the estimated area information, and the data storage unit stores the generated teaching data and the image as the learning data.

Description

LEARNING DATA GENERATION DEVICE, ROBOT SYSTEM, LEARNING DATA GENERATION METHOD, AND LEARNING DATA GENERATION PROGRAM

The present disclosure relates to a learning data generation device, a robot system, a learning data generation method, and a learning data generation program.

In recent years, machine learning has been used and put to practical use in a variety of fields. One such machine learning method is known as "supervised learning." In "supervised learning," a large number of pairs of features (variables that represent the characteristics of the data: clues for prediction) and training data (labels, correct answer data) are prepared and given to a computer (machine learning device), and annotation (annotation teaching) is one method of generating training data.

Annotation requires the generation of teacher data (instruction data) by labeling large amounts of image data, audio data, video data, text data, etc. with relevant tags and metadata. Specifically, annotation that handles images is done by a person manually clicking on the image to teach area information about the object (work). In this way, performing annotation manually requires high labor costs and a huge amount of work time.

Various techniques have been proposed to easily generate training data for use in "supervised learning."

JP 2022-118300 A JP 2014-059729 A

As mentioned above, traditionally annotation has been done manually, resulting in high labor costs and huge amounts of work time. Furthermore, for example, when annotating the position, orientation, and shape regions of objects in a large amount of image data, it is necessary to minimize the differences in judgment between each individual (annotator) who processes the data. This requires the creation of detailed annotation rules, as well as the training and selection of each individual who processes the data, which leads to even higher labor costs and longer work hours.

Therefore, there is a demand for a learning data generation device, a robot system, a learning data generation method, and a learning data generation program that can perform annotation without incurring high labor costs and huge amounts of work time.

According to one embodiment of the present disclosure, a learning data generation device is provided that includes a data acquisition unit, a data processing unit, and a data storage unit, and generates learning data to be used in machine learning. The data acquisition unit acquires images of the areas in which multiple workpieces exist, the data processing unit estimates at least area information reflecting the area range of all or part of at least one workpiece based on the acquired images, and generates teaching data including the estimated area information, and the data storage unit stores the generated teaching data and images as learning data.

FIG. 1 is a diagram showing an example of a schematic diagram of an entire robot system for explaining an example of a training data generating device according to the present embodiment. FIG. 2 is a functional block diagram for explaining an example of a training data generating device according to this embodiment. FIG. 3 is a flowchart for explaining an example of processing in a first example of a learning data generation program according to this embodiment. FIG. 4 is a flowchart for explaining an example of processing in a second example of the learning data generation program according to this embodiment. FIG. 5 is a flowchart for explaining an example of processing in a third example of the learning data generation program according to this embodiment. FIG. 6 is a flowchart for explaining an example of processing in the fourth example of the learning data generation program according to this embodiment. FIG. 7 is a diagram for explaining an example of a process of correcting teaching data by a user in the learning data generating method according to this embodiment. FIG. 8 is a flowchart for explaining an example of processing in the fifth example of the learning data generation program according to this embodiment. FIG. 9 is a diagram showing an example of a workpiece in a robot system in which an example of a learning data generating device according to this embodiment is used. FIG. 10 is a diagram for explaining the shape region of a workpiece in one example of the learning data generation device according to this embodiment.

Below, examples of the learning data generation device, robot system, learning data generation method, and learning data generation program according to the present embodiment will be described in detail with reference to the attached drawings. In each drawing, the same or similar components are given the same or similar reference numerals. Furthermore, the embodiments described below do not limit the technical scope of the invention and the meaning of the terms described in the claims.

FIG. 1 is a schematic diagram showing an example of an entire robot system to explain an example of a learning data generation device according to this embodiment. As shown in FIG. 1, the robot system 100 includes a robot 1, a robot control device 2, a learning data generation device 3, and a camera 4. The robot 1 includes a robot mechanism 10, an arm 11, and an end effector (hand) 12.

In the robot system 100 shown in FIG. 1, a machine learning device for performing machine learning (supervised learning) is not shown because it is built into the robot control device 2. However, if the amount of calculations or data is large and it is difficult to build it into the robot control device 2, the machine learning device can be configured, for example, as a dedicated workstation installed near the robot control device 2, or a higher-level computer or general-purpose computer installed in a location remote from the robot system 100. Furthermore, when inputting a large amount of data into a learning model for learning, a general-purpose computer or processor may be used, but faster processing can be achieved by using a GPGPU (General-Purpose computing on Graphics Processing Units) or a large-scale PC cluster.

The robot 1 is configured, for example, as a multi-axis robot, and an end effector 12 is provided at the tip of an arm 11. In FIG. 1, the end effector 12 is a suction device (suction hand), but it goes without saying that this can be changed to various types depending on the work (object) and work content used by the robot system 100. The robot mechanism unit 10 is for making the robot 1 perform a specified operation based on a control command from the robot control device 2.

The robot control device 2 receives the output of the camera 4 and the output of the learning data generation device 3, and generates control commands for making the robot 1 perform a specified operation, for example based on a program or control data previously stored in an internal storage device, and outputs the generated control commands to the robot mechanism unit 10. As mentioned above, the machine learning device that performs supervised learning does not need to be built into the robot control device 2, but may be provided separately, for example, near the robot control device 2 or at a location away from the robot system 100, depending on the amount of calculation and data.

The learning data generating device 3 receives multiple images of the workpieces and generates teaching data (teacher data) including area information (information on the shape area of the workpieces) of the workpieces D1 to D9 in each image. Furthermore, the learning data generating device 3 outputs the generated teaching data and the corresponding images to the robot control device 2 (machine learning device) as learning data. Here, the learning data generating device 3 receives image data of the workpieces D1 to D9 captured by the camera 4, but the image data provided to the learning data generating device 3 is not limited to images captured by the camera 4 of the robot system 100, and may be various image data such as images of the workpieces captured in advance or images obtained by other robot systems. Furthermore, the images input to the learning data generating device 3 are not limited to two-dimensional images, and as will be described in detail later, three-dimensional data (three-dimensional images, three-dimensional point cloud data, three-dimensional measurement data) may also be input together.

The camera 4 is for acquiring a two-dimensional image of the area where multiple workpieces (e.g., multiple cardboard boxes) D1 to D9 are present, or a two-dimensional image and three-dimensional data (three-dimensional point cloud data), and includes two cameras 4a and 4b and a projector 4c. The projector 4c projects a predetermined pattern onto the area where multiple workpieces D1 to D9 are present, and the two cameras 4a and 4b capture the area where multiple workpieces are present onto which the predetermined pattern is projected by the projector 4c, and measure the three-dimensional shapes of the workpieces D1 to D9. In this way, the camera 4 may be configured to measure the three-dimensional shape of the area where multiple workpieces D1 to D9 are present and acquire three-dimensional point cloud data, but it can also be configured as a single two-dimensional camera, for example. The camera 4 in FIG. 1 is also configured to acquire a two-dimensional image of the area where multiple workpieces D1 to D9 are present, using an image captured by one of the two cameras 4a and 4b. In other words, the camera 4 only needs to be able to capture appropriate images according to, for example, the work or task being performed by the robot 1, or according to annotations in supervised learning.

2 is a functional block diagram for explaining an example of the training data generation device according to this embodiment. As shown in FIG. 2, the training data generation device 3 that generates training data to be used in machine learning includes a data acquisition unit 31, a data processing unit (arithmetic processing device) 32, a data storage unit 33, a reception unit 34, and a display unit 35. The data acquisition unit 31 acquires images (two-dimensional images, or two-dimensional images and three-dimensional point cloud data) of the areas where multiple workpieces (D1 to D9) exist and a trained model. The data processing unit 32 at least estimates area information reflecting the entire or partial area range of at least one workpiece (D0, D) based on the images acquired by the data acquisition unit 31, and generates teaching data including the estimated area information. Here, the trained model can be, for example, a training model generated by another training data generation device 3, or a training model (previous training model) generated by the training data generation device 3 itself. Note that, as will be described in detail later with reference to FIGS. 3 to 8, if a trained model is not used according to each example of this embodiment, the data acquisition unit 31 does not need to acquire the trained model. Similarly, if three-dimensional point cloud data is not used in each example according to this embodiment, the data acquisition unit 31 does not need to acquire the three-dimensional point cloud data.

The area information of the workpiece (D0) includes, for example, at least one of outer shape area information reflecting the outer shape of the workpiece, removal area information for adsorbing, suctioning or gripping the workpiece, and local area information on the workpiece. Note that the local area information of the workpiece (D0) includes, for example, at least one of flat surfaces, curved surfaces or areas with large areas on the workpiece, non-slip areas on the workpiece, and high density areas on the workpiece. This local area information of the workpiece will be described in detail later with reference to FIG. 9.

The data storage unit 33 stores the teaching data and images (e.g., two-dimensional images, or two-dimensional images and three-dimensional point cloud data) generated by the data processing unit 32 as learning data. The reception unit 34 receives, for example, correction information based on area information of at least one workpiece input by a worker (user). At this time, the data processing unit 32 corrects and outputs the teaching data based on, for example, the correction information received by the reception unit 34. The display unit 35 displays the images and teaching data, and allows, for example, the user to make further corrections to the teaching data (learning data).

Here, when the data acquisition unit 31 acquires a trained model, the data processing unit 32 estimates area information of the workpiece based on the image and the trained model, and generates teaching data. Also, when the data acquisition unit 31 acquires two-dimensional images of the areas where multiple workpieces (D0 to D9, D) exist, the data processing unit 32 estimates area information based on the two-dimensional images acquired by the data acquisition unit 31, and generates teaching data. Furthermore, when the data acquisition unit 31 acquires two-dimensional images and three-dimensional data (three-dimensional point cloud data) of the areas where multiple workpieces (D0 to D9, D) exist, the data processing unit 32 estimates area information based on the two-dimensional images and three-dimensional point cloud data acquired by the data acquisition unit 31, and generates teaching data.

The data processing unit 32 can estimate three-dimensional area information of the workpiece based on the results of at least one of the following analyses of the three-dimensional point cloud data: plane analysis, curved surface analysis, blob analysis, coordinate system analysis, attitude analysis, depth analysis, scale analysis, feature analysis, nearby point analysis, mesh analysis, and voxel analysis, and can compare the three-dimensional point cloud data with a two-dimensional image to estimate area information of the workpiece in the two-dimensional image and generate teaching data. Furthermore, the data processing unit 32 can correct and output the teaching data based on the area information of the workpiece estimated based on the analysis results of the three-dimensional point cloud data. The data processing unit 32 can also extract features by performing image processing based on the area information of at least one workpiece and the image received by the receiving unit 34, and can match the extracted features to estimate area information of the workpiece on the image and generate teaching data.

In the above, the data processing unit 32 can perform image processing based on the image, and can estimate work area information based on the image and the results of the image processing to generate teaching data. The data processing unit 32 can also estimate work area information based on the results of at least one of the following image processing methods: pattern matching, plane matching, curved surface matching, blob analysis, feature analysis, gradient analysis, edge analysis, contrast analysis, histogram analysis, and color information analysis. Furthermore, the data processing unit 32 can correct and output teaching data based on the work area information estimated based on the results of the image processing.

Before describing the first to fifth examples of the learning data generation program (learning data generation method) according to this embodiment with reference to Figures 3 to 8, we will now explain an example of the workpiece (object) D0 to be worked on with reference to Figure 9, and the term "shape area of the workpiece" in this specification with reference to Figure 10.

FIG. 9 is a diagram showing an example of a workpiece in a robot system in which an embodiment of the learning data generating device according to this embodiment is used. In the robot system according to this embodiment, the workpiece (object) D0 to be worked on is not limited to rectangular parallelepiped objects made of the same material, such as the cardboard boxes D1 to D9 shown in FIG. 1, and various objects are possible. That is, the workpiece D0 shown in FIG. 9 is an air joint, in which areas Da, Db, and Dd are made of plastic (for example, PTFE: polytetrafluoroethylene) material, and area Dc is made of metal (for example, brass or stainless steel). Here, the metal that forms area Dc has a higher specific gravity than the plastic that forms areas Da, Db, and Dd, and area Dc is designed to surround areas Db to Dd with a hexagonal plane to facilitate tightening work using a tool such as a wrench.

Incidentally, there are various local areas on a workpiece, such as flat surfaces, curved surfaces, areas with large areas, areas that are not slippery, and areas with high density (heavy). For example, when a workpiece is taken out using a suction hand (12), the success rate of taking out the workpiece changes depending on the area of the workpiece to be picked up. Therefore, it is preferable to include area information surrounding the outline of the local area on the workpiece as learning data (teaching data). Specifically, when the workpiece is the air joint D0 shown in FIG. 9, for example, it is considered that the success rate of taking out the workpiece is higher when the suction hand 12 picks up the flat area Dc rather than picking up the curved areas Da, Db, Dd. Furthermore, it is considered that the success rate of taking out the workpiece is higher when the suction hand 12 picks up the area Dc made of metal with high density (heavy) material rather than picking up the areas Da, Db, Dd made of plastic with low density (light) material, because the workpiece can be taken out at a position closer to the center of gravity of the entire workpiece D0. In other words, the suction hand 12 suctions the flat area Dc formed by the metal of the air joint D0, which is the workpiece, making it possible to stably remove the workpiece D0.

Here, as in the fourth embodiment described with reference to Figures 6 and 7, it is preferable for the user to refer to the image and teaching data displayed on the display unit 35 and correct the teaching data (learning data). Even in this case, the user only needs to perform some of the processing, which requires much less effort than the conventional method of manually clicking on an image to teach area information of an object. In this way, it is preferable for the area information of the workpiece D0 to include at least one of outer shape area information reflecting the outer shape of the workpiece D0, removal area information for adsorbing, suctioning or gripping the workpiece D0, and local area information on the workpiece D0.

Next, FIG. 10 is a diagram for explaining the shape region of a workpiece in one embodiment of the learning data generation device according to this embodiment, where FIG. 10(a) shows the shape region of the workpiece D, and FIG. 10(b) shows the region including the background of the workpiece D. In this specification, the term "shape region of the workpiece" refers to the region indicated by reference character A1 (the region of the workpiece D itself) that includes only the workpiece D and does not include the background of the workpiece D in the two-dimensional image P3 output from the camera 4, as shown in FIG. 10(a), and is not the region that includes the workpiece D and the background as shown in FIG. 10(b). In other words, the "shape region of the workpiece" in this specification is "region information that reflects the shape/outer shape of the workpiece," and this information can be used to calculate the shape/outer shape of the workpiece.

Below, first to fifth examples of the learning data generation program (learning data generation method) according to this embodiment will be described with reference to Figures 3 to 8. Note that the following description is based on the functional block diagram of the learning data generation device 3 described with reference to Figure 2, but it goes without saying that the learning data generation device 3 is not limited to the one shown in Figure 2.

FIG. 3 is a flowchart for explaining an example of processing in a first example of the learning data generation program (learning data generation method) according to this embodiment, and shows a case where the data acquisition unit 31 acquires a two-dimensional image and a trained model. As shown in FIG. 3, when an example of processing in the learning data generation program of the first example starts (START), in step ST11, the camera 4 captures a two-dimensional image of the presence area of the workpiece D. That is, as described with reference to FIG. 1, the camera 4 captures two-dimensional images of the presence areas of the workpieces (e.g., multiple cardboard boxes) D1 to D9 and outputs them to the learning data generation device 3.

Then, proceed to step ST12, where the data acquisition unit 31 acquires a two-dimensional image and a trained model and outputs them to the data processing unit 32. Here, the two-dimensional image received by the data acquisition unit 31 is a two-dimensional image of the area in which the workpiece is present, as described with reference to FIG. 2. The trained model received by the data acquisition unit 31 can be, for example, one prepared in advance by the provider that offers the robot system. Alternatively, as described above, the trained model can be, for example, a training model generated by another training data generation device 3, or a training model (previous training model) generated by the training data generation device 3 itself.

Then, the process proceeds to step ST13, where the data processing unit 32 estimates the work area based on the two-dimensional image from the data acquisition unit 31 and the trained model, and then the process proceeds to step ST14, where the data processing unit 32 generates teaching data (teacher data) based on the estimated information of the shape area of the workpiece D. That is, the data processing unit 32 estimates the shape area (area information) of the workpiece based on the two-dimensional image and the trained model, and generates teaching data. Here, the shape area of the workpiece D corresponds to, for example, the shape area A1 in FIG. 10(a), as described with reference to FIG. 10.

Then, the process proceeds to step ST15, where the data storage unit 33 stores the teaching data and the two-dimensional image as learning data in the data storage unit 33, and the process ends (END). Here, step ST16 is a process of displaying the two-dimensional image data from step ST12 and the teaching data from step ST14 on the display unit 35, and it is possible for the user to refer to the two-dimensional image and teaching data displayed on the display unit 35 and correct the teaching data, for example. It goes without saying that it is also possible to simply display the two-dimensional image and teaching data on the display unit 35 without performing any processing by the user. The process of step ST16 in this first embodiment is similar to steps ST26, ST36, ST47, and ST55 in each embodiment described below.

In this way, according to the learning data generation program (learning data generation method) of this first embodiment, learning data (teaching data) can be easily generated without incurring high labor costs and huge amounts of work time.

FIG. 4 is a flowchart for explaining an example of processing in a second example of the learning data generation program according to this embodiment, showing a case in which the data acquisition unit 31 acquires a two-dimensional image and does not acquire a trained model. As shown in FIG. 4, when an example of processing of the learning data generation program of the second example starts (START), in step ST21, the camera 4 captures a two-dimensional image of the area in which the workpiece D (D1 to D9) is present. Then, the process proceeds to step ST22, in which the data acquisition unit 31 acquires the two-dimensional image and outputs it to the data processing unit 32. Here, the two-dimensional image received by the data acquisition unit 31 is a two-dimensional image of the area in which the workpiece is present, as described with reference to FIG. 2, captured by the camera 4.

Then, the process proceeds to step ST23, where the data processing unit 32 performs image processing on the two-dimensional image from the data acquisition unit 31 to estimate the shape area of the workpiece D. The process then proceeds to step ST24, where the data processing unit 32 generates teaching data based on the estimated information on the shape area of the workpiece D. The process then proceeds to step ST25, where the data storage unit 33 stores the teaching data and the two-dimensional image as learning data and ends the process (END). Note that step ST26 corresponds to step ST16 in the first embodiment described above, and a description thereof will be omitted.

In this way, the training data generation program of this second embodiment does not use the trained model in the first embodiment described above, making it possible to simplify processing. However, since a trained model is not used, there is a risk that the accuracy of the teaching data will be somewhat inferior to that of the first embodiment. For this reason, it is preferable to determine whether to implement this second embodiment based on the shape of the target workpiece, the content of the work, or the time it takes to actually start operating the robot system, etc.

Here, as a modified example of the learning data generation program (learning data generation method) according to this embodiment, for multiple workpieces (D0 to D9, D), for example, a user (operator) teaches area information (shape area of the workpiece) of only one workpiece (D0, D) for one type of workpiece. Furthermore, image processing is performed based on the taught area information and the image (two-dimensional image) to extract features. Then, a matching process of the extracted features is performed to estimate the areas of all (multiple) workpieces on the two-dimensional image, and teaching data can be generated.

FIG. 5 is a flowchart for explaining an example of processing in a third example of the learning data generation program according to this embodiment, showing a case in which the data acquisition unit 31 acquires only a two-dimensional image and does not acquire a learned model, the reception unit 34 accepts area information (shape area) of the workpiece, and the data processing unit 32 performs image processing. As shown in FIG. 5, when an example of processing of the learning data generation program of the third example starts (START), in step ST31, the camera 4 captures a two-dimensional image of the area in which the workpiece D exists. Then, the process proceeds to step ST32, where the data acquisition unit 31 acquires the two-dimensional image and outputs it to the data processing unit 32.

Next, the process proceeds to step ST33, where the data processing unit 32 performs image processing based on the two-dimensional image from the data acquisition unit 31 and the area information of the workpiece D, and estimates the shape area of all (multiple) works D1 to D9 (D0, D) on the image. Here, the image processing performed by the data processing unit 32 is, for example, at least one of pattern matching, plane matching, curved surface matching, blob analysis, feature analysis, gradient analysis, edge analysis, contrast analysis, histogram analysis, and color information analysis.

Furthermore, the process proceeds to step ST34, where the data processing unit 32 generates teaching data based on the estimated information of the shape area of the workpiece D, and then the process proceeds to step ST35, where the data storage unit 33 stores the teaching data and the two-dimensional image as learning data and ends the process (END). The process of step ST36 corresponds to the above-mentioned steps ST16 and ST26, and the user can also refer to the two-dimensional image and teaching data displayed on the display unit 35 and correct the teaching data, for example, as in steps ST45 to ST47 of the fourth embodiment described with reference to FIG. 6. Here, the correction of the teaching data can be performed by the user based on the surface shape or condition of the workpiece D, or the material (material) forming the area, as described with reference to FIG. 9, for example. However, it is also possible to automate the determination of various areas on the workpiece D, such as flat surfaces, curved surfaces, areas with large areas, areas that are not slippery, and areas with high density (heavy), using a machine learning device, etc.

FIG. 6 is a flowchart for explaining an example of processing in the fourth example of the learning data generation program according to this embodiment, showing a case where the data acquisition unit 31 acquires a two-dimensional image and a trained model, and the user modifies the teaching data. As shown in FIG. 6, when an example of processing in the learning data generation program of the fourth example starts (START), in step ST41, the camera 4 captures a two-dimensional image of the area in which the work D exists. Then, the process proceeds to step ST42, where the data acquisition unit 31 acquires the two-dimensional image and the trained model, and outputs them to the data processing unit 32.

Next, the process proceeds to step ST43, where the data processing unit 32 estimates the shape area of the workpiece D based on the two-dimensional image from the data acquisition unit 31 and the trained model, and the process proceeds to step ST44. In step ST44, the data processing unit 32 generates teaching data based on the estimated information of the shape area of the workpiece D.

Furthermore, the process proceeds to step ST45, where the user (operator) modifies the teaching data while viewing (referring to) the display unit 35, and the process proceeds to step ST46, where the data storage unit 33 stores the teaching data and the two-dimensional image as learning data, and then the process ends (END). Here, the process of step ST47 corresponds to the above-mentioned steps ST16, ST26, and ST36. In this fourth embodiment, as shown in steps ST45 to ST47, the user can refer to the two-dimensional image and teaching data displayed on the display unit 35 and modify the teaching data. Here, the modification of the teaching data can be performed by the user based on, for example, the surface shape or state of the workpiece D, or the material (material) forming the area, as described with reference to FIG. 9, but it is also possible to automate the determination of various areas on the workpiece D, such as flat surfaces, curved surfaces, areas with large areas, areas that are not slippery, and areas with high density (heavy), using a machine learning device, etc.

FIG. 7 is a diagram for explaining an example of a user-initiated correction process for teaching data in the learning data generation method according to this embodiment. Here, the upper part of FIG. 7(a), FIG. 7(b), and FIG. 7(c) show cases where the teaching data is correct and the user does not need to correct the teaching data, while the lower part of FIG. 7(d), FIG. 7(e), and FIG. 7(f) show cases where the teaching data is incorrect and the user needs to correct the teaching data. Furthermore, FIG. 7(a) and FIG. 7(d) are two-dimensional images of multiple workpieces, FIG. 7(b) and FIG. 7(e) are images showing the results of contour extraction by image processing superimposed on the two-dimensional images, and FIG. 7(c) and FIG. 7(f) are images showing the final teaching results (final learning data).

First, a camera 4 is used to capture an image of an area where multiple workpieces (e.g., multiple cardboard boxes) are lined up, and while changing the arrangement of the workpieces, multiple two-dimensional images are captured of the area where multiple workpieces are lined up in different arrangements (Figs. 7(a) and 7(d)). Next, image processing is performed on each image, and the characteristics of the contour line of each workpiece in each image are extracted (Figs. 7(b) and 7(e)). Here, the contour line can be extracted, for example, by binarizing the two-dimensional image to generate a black and white image, and then searching for and finding the boundary line between black and white on the image. Note that, for example, when the workpiece is a cardboard box, the color within the workpiece area is almost uniform, and the greater the color difference between the workpiece area and the background area, the more accurately the contour line of the workpiece can be found.

First, we will explain the case where the contour lines extracted by image processing are correct, for example, the case where image processing is performed on the captured image of workpieces a11 to a14 as shown in FIG. 7(a), and the characteristics of the contour lines of workpieces b11 to b14 as shown in FIG. 7(b) are extracted. At this time, the user looks at (refers to) the two-dimensional image of the workpiece and the image of the teaching data as shown in FIG. 7(b) displayed on the display unit 35. Here, the user judges whether the learning data (two-dimensional image and teaching data) as shown in FIG. 7(b) displayed on the display unit 35 is correct, and if it is judged to be correct, the final learning data (teaching data c11 to c14) as shown in FIG. 7(c) is obtained without modifying the learning data (teaching data), and this learning data is saved in the data saving unit 33.

On the other hand, when the contour extracted by image processing is incorrect, for example, as a result of performing image processing on the captured image of workpieces d11 to d14 as shown in FIG. 7(d), the contour features of workpieces e11 to e14 as shown in FIG. 7(e) are extracted. That is, as shown in FIG. 7(d), for example, when the color in the area of workpiece d13 on the image is quite close to the color of the background, and workpieces d12 and d14 are stuck together and the boundary between them is unclear, as shown in FIG. 7(e), the left and lower contours of workpiece e13 cannot be recognized by image processing, and the contour between workpieces e12 and e14 cannot be recognized by image processing, and they are erroneously determined as one workpiece. In this case, the user will see the two-dimensional image of the workpiece displayed on the display unit 35 and the image of the teaching data as shown in FIG. 7(e). The user will then determine whether the learning data (two-dimensional image and teaching data) as shown in FIG. 7(e) displayed on the display unit 35 is correct.

Here, if the user judges that the learning data shown in FIG. 7(e) is incorrect, for example, if the user judges that the presence of a workpiece is not recognized in e13 in FIG. 7(e), the missing contour line in e13 is added by clicking with a mouse or the like to correct it to the area where the workpiece exists, and correction is made. Furthermore, if the user judges that e12 and e14 in FIG. 7(e) are not one workpiece but two works, the missing boundary line between e12 and e14 is added by operating a mouse or the like to correct it to two works. Furthermore, if the user judges that the contour line of the label in the area e11 in FIG. 7(e) was extracted and the label surrounded by these contour lines was erroneously recognized as one cardboard box, correction is made to delete the contour line surrounding the label. Furthermore, if the user judges that the contour line was erroneously extracted in the background area where no cardboard box exists, correction is made to delete such extra contour line. In this way, the final learning data (instruction data f11 to f14) corrected by the user as shown in FIG. 7(f) is obtained, and this learning data is stored in the data storage unit 33. Note that the correction of the instruction data by the user is preferably used, for example, as described with reference to FIG. 9, for determining areas of metal and plastic materials that can be immediately determined by the user (person) by referring to an image (e.g., a color image or a grayscale image).

In this way, various problems arise, such as, for example, if the color within the work area is not uniform, a contour line is extracted within the work area, part of the work contour line is missing if the color difference between the work area and the background area is not large, or adjacent contour lines cannot be extracted if multiple work pieces are close together. Also, if the resolution of the captured image is low, or if the lighting conditions at the time of capture are poor and noise is likely to occur, a contour line may be mistakenly extracted in a background area where no work exists. Therefore, it is preferable to superimpose such extraction results on the image and display them on the display unit 35 for the user, so that the user can visually delete/correct the incorrect contour lines and add missing contour lines, and modify the teaching data to reflect the correct work area to generate learning data.

In other words, if learning data including teaching data that includes an incorrect work area is used for machine learning, it will not be possible to learn correctly, and it will be difficult to generate a trained model that can correctly infer and calculate the work area. In response to this, it is possible to improve the accuracy (reliability) of the teaching data by presenting the generated teaching data and images (learning data) to the user and correcting the teaching data based on the user's judgment. Note that although the above-mentioned processing is performed by the user, the user does not need to perform the teaching processing from scratch, and only makes corrections to the incorrect parts, so it goes without saying that high labor costs and huge work hours are not required. Note that the correction of teaching data by the user as described above is not limited to the fourth example, and can be widely used in the examples of the learning data generation device, robot system, learning data generation method, and learning data generation program related to this embodiment.

FIG. 8 is a flowchart for explaining an example of processing in the fifth embodiment of the learning data generation program according to this embodiment. As shown in FIG. 8, when an example of processing in the learning data generation program of the fifth embodiment starts (START), in step ST51, the camera 4 captures a two-dimensional image of the area in which the workpiece D exists, and the three-dimensional sensor acquires three-dimensional point cloud data (three-dimensional data) of the area in which the workpiece D exists. Then, the process proceeds to step ST52, where the data acquisition unit 31 performs image processing on the two-dimensional image to estimate the shape area.

Next, proceed to step ST53, where the area information of work D is corrected based on the three-dimensional point cloud data. Here, correcting the area information of work D based on the three-dimensional point cloud data means, for example, comparing the three-dimensional point cloud data with the two-dimensional images for the calculation results of the shape area in each of the two-dimensional images of multiple workpieces. Furthermore, by utilizing the difference in three-dimensional position contained in the three-dimensional point cloud data, background areas, obstacle areas, areas of adjacent workpieces, etc. that are erroneously included in the calculation results are excluded, and the corrected teaching data for the work is obtained.

Then, the process proceeds to step ST54, where the data storage unit 33 stores the teaching data and the two-dimensional image as learning data, and the process ends (END). When the area information of the workpiece D from the two-dimensional image is corrected based on the three-dimensional point cloud data, it is preferable to extract, for example, grooves, gaps, steps, three-dimensional planes, and three-dimensional curved surfaces in the two-dimensional image as features. This makes it possible to improve the reliability (accuracy) of the teaching data by comparing it with the three-dimensional point cloud data, even if, for example, the workpiece and its surrounding background, or the boundaries between workpieces, cannot be identified in the two-dimensional image. Note that in this fifth embodiment, step ST55 also corresponds to, for example, step ST47 in the fourth embodiment described with reference to FIG. 6, and as described above, the processes of steps ST45 and ST46 in the fourth embodiment can be additionally performed.

The learning data generation program according to the present embodiment described above may be provided by recording it on a computer-readable non-transitory recording medium or a non-volatile semiconductor memory, or may be provided via a wired or wireless connection. Examples of computer-readable non-transitory recording media include optical discs such as CD-ROMs (Compact Disc Read Only Memory) and DVD-ROMs, or hard disk devices. Examples of non-volatile semiconductor memory include PROMs (Programmable Read Only Memory) and flash memories. Furthermore, distribution from a server device may be via a wired or wireless LAN (Local Area Network), or a WAN such as the Internet.

As described above in detail, the learning data generation device, robot system, learning data generation method, and learning data generation program according to this embodiment make it possible to perform annotation without incurring high labor costs and huge amounts of work time.

Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible to these embodiments without departing from the gist of the present disclosure, or without departing from the spirit of the present disclosure derived from the contents described in the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-mentioned embodiments, the order of each operation and the order of each process are shown as examples, and are not limited to these. The same applies when numerical values or formulas are used to explain the above-mentioned embodiments.

Regarding the above embodiment and modified examples, the following supplementary notes are further disclosed.
[Appendix 1]
A learning data generation device (3) for generating learning data for use in machine learning, comprising:
A data acquisition unit (31) that acquires images of areas where a plurality of workpieces (D0 to D9, D) are present;
A data processing unit (32) that at least estimates area information reflecting a whole or part of an area range of at least one of the workpieces (D0, D) based on the acquired image, and generates teaching data including the estimated area information;
a data storage unit (33) that stores the generated teaching data and the image as the learning data;
A training data generating device comprising:
[Appendix 2]
The learning data generation device described in Appendix 1, wherein the area information of the workpiece (D0, D) includes at least one of outer shape area information reflecting the outer shape of the workpiece (D0, D), removal area information for adsorbing, suctioning or grasping the workpiece (D0, D), and local area information on the workpiece (D0, D).
[Appendix 3]
The learning data generation device described in Appendix 2, wherein the local area information of the workpiece (D0, D) includes at least one of a flat surface, a curved surface, or a large area on the workpiece (D0, D), a non-slip area on the workpiece (D0, D), and a high-density area on the workpiece (D0, D).
[Appendix 4]
Further, a reception unit (34) is provided,
The reception unit (34) receives correction information based on area information of at least one of the works (D0, D),
The learning data generating device according to any one of claims 1 to 3, wherein the data processing unit (32) modifies and outputs the teaching data based on the teaching information accepted by the accepting unit (34).
[Appendix 5]
The data acquisition unit (31) further acquires a trained model,
The data processing unit (32) estimates area information of the work (D0, D) based on the image and the trained model, and generates the teaching data.
[Appendix 6]
The data acquisition unit (31) acquires a two-dimensional image of an area where the plurality of workpieces (D0 to D9, D) are present,
The learning data generation device according to any one of Supplementary Note 1 to Supplementary Note 5, wherein the data processing unit (32) estimates the region information based on the two-dimensional image and generates the teaching data.
[Appendix 7]
The data acquisition unit (31) acquires two-dimensional images and three-dimensional data of the areas in which the plurality of workpieces (D0 to D9, D) are present,
The learning data generation device according to any one of Supplementary Note 1 to Supplementary Note 5, wherein the data processing unit (32) estimates the region information based on the two-dimensional image and the three-dimensional data, and generates the teaching data.
[Appendix 8]
The data processing unit (32) analyzes the three-dimensional data, estimates three-dimensional area information of the workpiece based on the analysis results, compares the three-dimensional data with the two-dimensional image to estimate area information of the workpiece in the two-dimensional image, and generates the teaching data.
[Appendix 9]
The learning data generation device described in Appendix 8, wherein the data processing unit (32) estimates three-dimensional region information of the work (D0, D) based on the processing results of at least one of plane analysis, curved surface analysis, blob analysis, coordinate system analysis, posture analysis, depth analysis, scale analysis, feature analysis, nearby point analysis, mesh analysis, and voxel analysis as an analysis of the three-dimensional data.
[Appendix 10]
The learning data generation device described in Appendix 8 or Appendix 9, wherein the data processing unit (32) modifies and outputs the teaching data based on area information of the work (D0, D) estimated based on the analysis results of the three-dimensional data.
[Appendix 11]
The data processing unit (32) performs image processing based on area information of at least one of the works (D0, D) received by the receiving unit (34) and the image to extract features, performs matching processing of the extracted features to estimate area information of multiple works (D0, D) on the image, and generates the teaching data.
[Appendix 12]
The data processing unit (32) performs image processing based on the image, and estimates area information of the work (D0, D) based on a result of the image processing and the image, thereby generating the teaching data.
[Appendix 13]
The learning data generation device according to claim 11 or 12, wherein the data processing unit (32) estimates area information of the work (D0, D) based on the results of processing performed as the image processing at least one of pattern matching, plane matching, curved surface matching, blob analysis, feature analysis, gradient analysis, edge analysis, contrast analysis, histogram analysis, and color information analysis.
[Appendix 14]
The data processing unit (32) corrects and outputs the teaching data based on area information of the work (D0, D) estimated based on the processing result of the image processing.
[Appendix 15]
Further, a display unit (35) is provided,
The learning data generation device according to any one of claims 1 to 14, wherein the display unit (35) displays the image and the teaching data.
[Appendix 16]
The learning data generation device described in Appendix 15, wherein the data processing unit (32) includes a process of correcting the teaching data by referring to the image and the teaching data displayed on the display unit (35).
[Appendix 17]
A robot (1) that performs a predetermined process on a plurality of the workpieces (D0 to D9, D);
A camera (4) that captures an image of an area where the plurality of works (D0 to D9, D) are present and outputs the image;
A learning data generation device (3) according to any one of Supplementary Note 1 to Supplementary Note 16, which receives the image from the camera (4) and outputs the learning data;
a machine learning device that receives the learning data from the learning data generation device (3) and performs machine learning;
A robot system (100) comprising: a robot control device (2) that receives output from the machine learning device and controls the robot (1).
[Appendix 18]
A learning data generation method for generating learning data for use in machine learning, comprising:
A data acquisition process for acquiring images of the areas where a plurality of workpieces (D0 to D9, D) are present;
A data processing step of estimating at least area information reflecting the entire or partial area range of at least one of the workpieces (D0, D) based on the acquired image, and generating teaching data including the estimated area information;
a data storage step of storing the generated teaching data and the image as the learning data;
A learning data generation method comprising:
[Appendix 19]
The data acquisition step further includes acquiring a trained model that has been trained and generated in advance,
The learning data generation method described in Appendix 18, wherein the data processing step estimates region information of the work (D0, D) based on the image and the trained model, and generates the teaching data.
[Appendix 20]
The method further includes a receiving step, in which area information of at least one of the works (D0, D) is received,
The data processing step comprises performing image processing based on the received area information and the image for a plurality of the workpieces (D0 to D9, D) to extract features, performing a matching process for the extracted features, and estimating area information for the plurality of workpieces (D0 to D9, D) on the image to generate the teaching data.
[Appendix 21]
The data processing step performs image processing based on the image, and estimates area information of the work (D0 to D9, D) based on a processing result of the image processing and the image to generate the teaching data.
[Appendix 22]
The data acquisition step further acquires two-dimensional images and three-dimensional data of the areas where the plurality of workpieces (D0 to D9, D) are present,
22. The learning data generation method according to any one of claims 18 to 21, wherein the data processing step estimates the region information based on the two-dimensional image and the three-dimensional data, and generates the teaching data.
[Appendix 23]
The method further includes a receiving step, in which correction information based on area information of at least one of the works (D0, D) is received;
23. The learning data generating method according to any one of claims 18 to 22, wherein the data processing step corrects the teaching data based on the received correction information.
[Appendix 24]
A learning data generation program for generating learning data for use in machine learning,
A processing unit,
A data acquisition process for acquiring images of the areas where a plurality of workpieces (D0 to D9, D) are present;
A data processing step of estimating at least area information reflecting the entire or partial area range of at least one of the workpieces (D0, D) based on the acquired image, and generating teaching data including the estimated area information;
and a data storage step of storing the generated teaching data and the image as the learning data.

REFERENCE SIGNS LIST 1 Robot 2 Robot control device 3 Learning data generating device 4 Camera 10 Robot mechanism 11 Arm 12 End effector (hand)
31 Data acquisition unit 32 Data processing unit 33 Data storage unit 34 Reception unit 35 Display unit 100 Robot system A1 Shape area A2 Areas D, D1 to D9 Workpiece (object)

Claims (24)

A learning data generation device for generating learning data for use in machine learning, comprising:
A data acquisition unit that acquires images of areas where a plurality of workpieces are present;
A data processing unit that at least estimates area information reflecting an area range of all or a part of at least one of the workpieces based on the acquired image, and generates teaching data including the estimated area information;
a data storage unit that stores the generated teaching data and the image as the learning data;
A training data generating device comprising: The learning data generating device according to claim 1, wherein the area information of the workpiece includes at least one of outer shape area information reflecting the outer shape of the workpiece, removal area information for adsorbing, suctioning or gripping the workpiece, and local area information on the workpiece. The learning data generating device according to claim 2, wherein the local area information of the workpiece includes at least one of flat surfaces, curved surfaces or large areas on the workpiece, non-slip areas on the workpiece, and high-density areas on the workpiece. Further, a reception unit is provided,
The reception unit receives correction information based on area information of at least one of the works,
The training data generating device according to claim 1 , wherein the data processing unit corrects and outputs the teaching data based on the correction information received by the receiving unit. The data acquisition unit further acquires a trained model,
The learning data generation device according to claim 1 , wherein the data processing unit estimates area information of the workpiece based on the image and the trained model, and generates the teaching data. The data acquisition unit acquires two-dimensional images of areas where the plurality of workpieces are present,
The training data generating device according to claim 1 , wherein the data processing unit estimates the region information based on the two-dimensional image and generates the teaching data. The data acquisition unit acquires two-dimensional images and three-dimensional data of the areas where the workpieces are present,
The training data generating device according to claim 1 , wherein the data processing unit estimates the region information based on the two-dimensional image and the three-dimensional data, and generates the teaching data. The learning data generating device according to claim 7, wherein the data processing unit analyzes the three-dimensional data, estimates three-dimensional area information of the workpiece based on the analysis results, compares the three-dimensional data with the two-dimensional image to estimate area information of the workpiece in the two-dimensional image, and generates the teaching data. The learning data generating device according to claim 8, wherein the data processing unit estimates the three-dimensional area information of the workpiece based on the results of processing the three-dimensional data by performing at least one of plane analysis, curved surface analysis, blob analysis, coordinate system analysis, attitude analysis, depth analysis, scale analysis, feature analysis, nearby point analysis, mesh analysis, and voxel analysis. The learning data generating device according to claim 8 or claim 9, wherein the data processing unit modifies and outputs the teaching data based on area information of the workpiece estimated based on the analysis results of the three-dimensional data. The learning data generating device according to any one of claims 4 to 6, wherein the data processing unit performs image processing based on the area information of at least one of the workpieces received by the receiving unit and the image to extract features, performs matching processing of the extracted features to estimate area information of multiple workpieces on the image, and generates the teaching data. The learning data generating device according to any one of claims 1 to 11, wherein the data processing unit performs image processing based on the image, and generates the teaching data by estimating area information of the work based on the result of the image processing and the image. The learning data generating device according to claim 11 or 12, wherein the data processing unit estimates area information of the workpiece based on the results of processing performed as the image processing at least one of pattern matching, plane matching, curved surface matching, blob analysis, feature analysis, gradient analysis, edge analysis, contrast analysis, histogram analysis, and color information analysis. The learning data generating device according to any one of claims 11 to 13, wherein the data processing unit modifies and outputs the teaching data based on area information of the workpiece estimated based on the results of the image processing. Further, the device includes a display unit,
The training data generating device according to claim 1 , wherein the display unit displays the image and the teaching data. The learning data generating device according to claim 15, wherein the data processing unit includes a process for correcting the teaching data by referring to the image and the teaching data displayed on the display unit. A robot that performs a predetermined process on a plurality of the workpieces;
A camera that captures an image of an area where the workpieces are present and outputs the image;
The training data generating device according to claim 1 , which receives the image from the camera and outputs the training data;
a machine learning device that receives the learning data from the learning data generation device and performs machine learning;
A robot system comprising: a robot control device that receives output of the machine learning device and controls the robot. A learning data generation method for generating learning data for use in machine learning, comprising:
A data acquisition step of acquiring images of areas where a plurality of workpieces are present;
A data processing step of at least estimating area information reflecting an area range of all or a part of at least one of the workpieces based on the acquired image, and generating teaching data including the estimated area information;
a data storage step of storing the generated teaching data and the image as the learning data;
A learning data generation method comprising: The data acquisition step further includes acquiring a trained model that has been trained and generated in advance,
The learning data generation method according to claim 18 , wherein the data processing step estimates area information of the workpiece based on the image and the trained model, and generates the teaching data. The method further includes a receiving step, the receiving step receiving area information of at least one of the works,
The learning data generation method according to claim 18 or claim 19, wherein the data processing step performs image processing based on the received area information and the image for a plurality of the workpieces to extract features, and performs a matching process of the extracted features to estimate area information of the plurality of workpieces on the image to generate the teaching data. The learning data generation method according to any one of claims 18 to 20, wherein the data processing step performs image processing based on the image, and generates the teaching data by estimating area information of the work based on the result of the image processing and the image. The data acquisition step further includes acquiring two-dimensional images and three-dimensional data of the areas where the workpieces are present,
22. The learning data generating method according to claim 18, wherein the data processing step estimates the region information based on the two-dimensional image and the three-dimensional data, and generates the teaching data. The method further includes a receiving step, the receiving step receiving correction information based on area information of at least one of the works,
23. The method of generating training data according to claim 18, wherein the data processing step corrects the teaching data based on the received correction information. A learning data generation program for generating learning data for use in machine learning,
A processing unit includes:
A data acquisition step of acquiring images of areas where a plurality of workpieces are present;
A data processing step of at least estimating area information reflecting an area range of all or a part of at least one of the workpieces based on the acquired image, and generating teaching data including the estimated area information;
and a data storage step of storing the generated teaching data and the image as the learning data.

Images