TW202511037A - Parameter correction device - Google Patents

Abstract

The present invention provides a parameter correction device that calculates a three-dimensional estimated position of a workpiece on the basis of a sensor parameter and the position of a characteristic portion in an image of the workpiece. The parameter correction device calculates a measured position of one member out of the sensor and the workpiece on the basis of the drive state of a movement device. The parameter correction device uses the movement device to move one member to first and second positions to capture images of the workpiece. The parameter correction device corrects the sensor parameter on the basis of the amount of movement, which is based on first and second measured positions, and the distance from a first estimated position to a second estimated position.

Description

Parameter correction device

Invention Field

The present disclosure relates to a parameter correction device.

Invention Background

A robot device is known that uses an image taken by a camera to measure the position of a workpiece and performs operations based on the measured position of the workpiece. The control device of the robot device can obtain an image of a characteristic portion of the workpiece taken by a camera. The control device calculates the three-dimensional position of the workpiece from the image of the characteristic portion of the workpiece based on a pre-made reference image of the characteristic portion of the workpiece and camera parameters. Then, the control device drives the robot by correcting the action position recorded in the action program of the robot based on the three-dimensional position of the workpiece. The robot device can perform correct operations corresponding to the actual position of the workpiece. Prior Art Literature Patent Literature

Patent document 1: International publication WO2018/163450A1 Patent document 2: Japanese patent publication No. 2023-69274 Patent document 3: International publication WO2021/145280A1

Invention Summary Problem to be solved by the invention

In a robot device that corrects the robot's motion position by using an image taken by a camera, the camera parameters must be determined in advance. If the accuracy of the camera parameters is good, the three-dimensional position of the workpiece calculated by the control device will also be correct. The multiple parameters included in the camera parameters can be determined by a process called calibration. In calibration, for example, the control device uses a camera to photograph a calibration plate whose position of a point has been predetermined. Then, the control device can generate the camera parameters based on the position of the point in the image and the actual three-dimensional position of the point.

However, after the use of the robot device, the camera parameters are sometimes corrected to correctly compensate for the robot's movements. In this case, the calibration plate must be prepared again, and there is a problem of taking time to correct the camera parameter errors. In addition, sometimes the height of the calibration plate is different from the height of the feature part of the workpiece actually inspected, and in the case of a camera that takes a two-dimensional image, the three-dimensional position of the workpiece estimated from the camera image sometimes produces errors. Or, if the outline or pattern of the workpiece is incorrectly set in the reference image, the three-dimensional position of the workpiece sometimes produces errors. Such errors are difficult to find, and there is a problem that the parameter correction is difficult to perform. Means used to solve the problem

One aspect of the present disclosure is a parameter correction device that corrects sensor parameters used to calculate a three-dimensional position, wherein the three-dimensional position corresponds to a specific position in an image of a workpiece captured by the sensor. The parameter correction device includes a feature detection unit that detects a feature portion in the image of the workpiece. The parameter correction device includes an estimated position calculation unit that calculates the three-dimensional position of the workpiece, i.e., the estimated position, based on the position of the feature portion in the image of the workpiece and the sensor parameters. The parameter correction device includes a measured position calculation unit that calculates the three-dimensional position of one of the components, i.e., the measured position, based on the driving state of a moving device of at least one of the components of the moving sensor and the workpiece. The parameter correction device includes a correction unit that corrects the sensor parameters. The feature detection unit detects a feature portion in an image captured when one of the components is arranged at a first position, and a feature portion in an image captured when one of the components is arranged at a second position. The estimated position calculation unit calculates a first estimated position when one of the components is arranged at the first position, and a second estimated position when one of the components is arranged at the second position. The measured position calculation unit calculates a first measured position when one of the components is arranged at the first position, and a second measured position when one of the components is arranged at the second position. The correction unit corrects the sensor parameter based on a movement amount based on the first measured position and the second measured position, and a distance from the first estimated position to the second estimated position.

The form used to implement the invention

The parameter correction device of the embodiment is described with reference to FIGS. 1 to 18. The parameter correction device of the embodiment corrects the sensor parameters used to calculate the three-dimensional position corresponding to a specific position in the image taken by the sensor. In the embodiment, the control device of the robot device functions as the parameter correction device. In the embodiment, the sensor is exemplified by a camera that takes a two-dimensional image. In addition, as the sensor parameter, the camera parameter including the internal parameter and the external parameter is exemplified.

(Configuration of the first robot device) Fig. 1 is a schematic diagram of the first robot device having a parameter correction device of the present embodiment. Fig. 2 is a block diagram of the robot device of the present embodiment. The robot device 3 of the present embodiment detects the position of a workpiece 31 as an object and transports the workpiece.

1 and 2 , the first robot device 3 includes a hand 5 as a working tool for holding a workpiece 31 and a robot 1 for moving the hand 5. The robot device 3 includes a control device 2 as a robot control device for controlling the robot device 3. The robot device 3 also includes a stage 35 for placing the workpiece 31.

The hand 5 of this embodiment is a working tool for holding or releasing the workpiece 31. The working tool mounted on the robot 1 is not limited to this form, and any working tool corresponding to the work performed by the robot device 3 can be adopted. For example, a welding gun for welding can be adopted as the working tool.

The robot 1 of this embodiment functions as a moving device, and the moving device moves one of the components of the camera 6 as a sensor and the workpiece 31. In the first robot device 3, it corresponds to one of the components moved by the camera 6 through the moving device. In addition, the workpiece 31 corresponds to another component whose position is fixed.

The robot 1 of this embodiment is a multi-joint robot including a plurality of joints 18. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported on a rotating base 13. The rotating base 13 is supported on a base 14. The robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 for fixing the hand 5. The components of the robot 1 of this embodiment are formed to rotate around six predetermined drive axes. The robot 1 is not limited to this form, and any robot that can move a working tool can be used.

The robot 1 of this embodiment includes a robot drive device 21, and the robot drive device 21 has a drive motor for driving the upper arm 11 and other components. The hand 5 includes a hand drive device 22 for driving the hand 5. The hand drive device 22 of this embodiment drives the hand 5 by air pressure. The hand drive device 22 includes an air pump for supplying compressed air to the cylinder, etc.

The control device 2 includes a control device body 40 and a teaching operation panel 26 for an operator to operate the control device body 40. The control device body 40 includes an arithmetic processing device (computer) having a CPU (Central Processing Unit) as a processor. The arithmetic processing device has a RAM (Random Access Memory) and a ROM (Read Only Memory) etc. connected to the CPU via a bus. The robot device 3 of this embodiment automatically transports a workpiece 31 according to an action program 67. The robot drive device 21 and the hand drive device 22 are controlled by the control device 2.

The control device body 40 includes a memory unit 42, and the memory unit 42 stores arbitrary information about the robot device 3. The memory unit 42 can be configured with a non-transitory storage medium that can store information. For example, the memory unit 42 can be configured with a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium.

The control device 2 has an action program 67 input thereto, which is prepared in advance for performing the actions of the robot 1. Alternatively, the operator can operate the teaching operation panel 26 to drive the robot 1, thereby setting the teaching point of the robot 1. The control device 2 can generate the action program 67 based on the teaching point.

The action program 67 is stored in the memory unit 42. The action control unit 43 sends an action command for driving the robot 1 to the robot drive unit 44 according to the action program 67. The robot drive unit 44 includes a circuit for driving a drive motor, and supplies power to the robot drive device 21 according to the action command. In addition, the action control unit 43 sends an action command for driving the hand drive device 22 to the hand drive unit 45. The hand drive unit 45 includes a circuit for driving an air pump, etc., and supplies power to the air pump, etc. according to the action command.

The motion control unit 43 corresponds to a processor driven according to the motion program 67. The processor is configured to read information stored in the memory unit 42. The processor reads the motion program 67 and executes control determined by the motion program 67, thereby functioning as the motion control unit 43.

The robot 1 includes a state detector for detecting the driving state of the robot 1. The state detector of this embodiment detects the position and posture of the robot as the driving state of the robot 1. The state detector of this embodiment includes a position detector 23, and the position detector 23 is installed on the drive motor of each drive shaft of the robot drive device 21. The position detector 23 can be configured as an encoder, for example, and the encoder detects the rotational position of the output shaft of the drive motor. The position and posture of the robot 1 are detected by the output of the position detector 23.

The teaching operation panel 26 is connected to the control device body 40 through a communication device. The teaching operation panel 26 includes an input unit 27, and the input unit 27 inputs information about the robot 1 and the hand 5. The input unit 27 is composed of input components such as a keyboard, buttons, and a dial. The teaching operation panel 26 includes a display unit 28, and the display unit 28 displays information about the robot 1 and the hand 5. The display unit 28 can be composed of any display panel such as a liquid crystal display panel or an organic EL (Electro Luminescence) display panel. Furthermore, in the case where the teaching operation panel has a display panel of a touch panel type, the display panel functions as an input unit and a display unit.

The robot device 3 is provided with a reference coordinate system 71, which remains unchanged when the position and posture of the robot 1 change. In the example shown in FIG. 1 , the origin of the reference coordinate system 71 is arranged on the rotating base 13 of the robot 1. The reference coordinate system 71 is also called a world coordinate system. In the reference coordinate system 71, the position of the origin is fixed, and the direction of the coordinate axis is fixed. The reference coordinate system 71 has mutually orthogonal X-axis, Y-axis and Z-axis as coordinate axes.

In the robot device 3, a tool coordinate system 74 is provided, and the tool coordinate system 74 has an origin set at an arbitrary position of the working tool. The tool coordinate system 74 changes position and posture together with the working tool. In the present embodiment, the origin of the tool coordinate system 74 is set at the tool tip of the hand 5. The position of the robot 1 corresponds to the position of the tool tip in the reference coordinate system 71 (the position of the origin of the tool coordinate system). In addition, the posture of the robot 1 corresponds to the posture of the tool coordinate system 74 relative to the reference coordinate system 71.

The robot device 3 of this embodiment is equipped with a camera 6 as a sensor for photographing the workpiece 31. The camera 6 of this embodiment is a two-dimensional camera for photographing two-dimensional images. The camera 6 can photograph images in a field of view 6a.

In the first robot device 3, one of the components among the camera 6 and the workpiece 31 that is moved by the robot 1 is the camera 6. The other component among the camera 6 and the workpiece 31 is the workpiece 31. In this way, the robot 1 changes its position and posture, thereby changing the position and posture of the camera 6. The three-dimensional position of the camera 6 is changed by the robot 1. The camera 6 of this embodiment is fixed to the hand 5 installed on the robot 1, but is not limited to this form. The camera 6 can be fixed to any component of the robot 1 that moves and changes its position and posture. Or, the camera 6 can also be installed on the robot 1. For example, the camera 6 can also be fixed to the flange 16 of the robot 1.

In the robot device 3, a camera coordinate system 72 as a sensor coordinate system is set for the camera 6. The camera coordinate system 72 changes position and posture together with the camera 6. The origin of the camera coordinate system 72 is set at a predetermined position of the camera 6, such as the lens center or optical center of the camera 6. The camera coordinate system 72 has an X-axis, a Y-axis, and a Z-axis that are orthogonal to each other. The camera coordinate system 72 of this embodiment is set so that the Z-axis is parallel to the optical axis of the lens of the camera 6.

If the position and posture of the robot 1 change, the position and posture of the camera coordinate system 72 will change. The position of the camera 6 corresponds to the position of the origin of the camera coordinate system 72 in the reference coordinate system 71. Moreover, the posture of the camera 6 corresponds to the direction of the camera coordinate system 72 relative to the reference coordinate system 71. Moreover, in the present embodiment, the position and posture of the camera coordinate system 72 relative to the tool coordinate system 74 are predetermined.

The control device 2 detects the three-dimensional position of the workpiece 31 in the work space based on the image of the workpiece 31 during the operation. The control device 2 corrects the position and posture of the robot 1 based on the position of the workpiece 31.

The control device 2 functions as a parameter correction device for correcting sensor parameters. The control device 2 includes an image processing unit 51 that processes an image captured by the camera 6. The image processing unit 51 includes an image capture control unit 53 that sends an instruction to the camera 6 to capture an image.

The image processing unit 51 includes a feature detection unit 56, which detects a predetermined feature portion in the image of the workpiece 31. The image processing unit 51 includes an estimated position calculation unit 57, which calculates the three-dimensional position of the workpiece, i.e., the estimated position, using a camera coordinate system 72 based on the position of the feature portion in the image and the camera parameters as sensor parameters. The image processing unit 51 includes an action command generation unit 64, which generates action commands for the robot 1 and the hand 5 based on the result of image processing.

The image processing unit 51 includes a measured position calculation unit 52, which calculates the three-dimensional position of one of the components of the camera 6 and the workpiece 31, i.e., the measured position, based on the position and posture of the robot 1 as the driving state of the robot, using the reference coordinate system 71. The image processing unit 51 includes an evaluation unit 58 for evaluating camera parameters. The image processing unit 51 has a selection unit 59, which selects a parameter to be corrected from a plurality of parameters included in the camera parameters. The image processing unit 51 includes a correction unit 60 for correcting the camera parameters.

The image processing unit 51 includes an image estimation unit 61, which estimates an image of a characteristic portion when the camera 6 is arranged at a predetermined position. The image estimation unit 61 of the present embodiment estimates an image of a characteristic portion when one of the components is arranged at a second position based on an image of a characteristic portion photographed when one of the components is arranged at a first position, the camera parameter 68, and the amount of movement of the measured position of one of the components. The image processing unit 51 includes a score calculation unit 62, which calculates a point indicating the degree of overlap of the image of the characteristic portion estimated by the image estimation unit 61 for the image of the characteristic portion photographed when one of the components is arranged at the second position. The image processing unit 51 includes a display control unit 63 that controls the image displayed on the display unit 28 .

The image processing unit 51 and the units included in the image processing unit 51 are equivalent to a processor driven according to the action program 67. The measured position calculation unit 52, the imaging control unit 53, the feature detection unit 56, the estimated position calculation unit 57, and the action command generation unit 64 are equivalent to a processor driven according to the action program 67. Furthermore, the evaluation unit 58, the selection unit 59, the correction unit 60, the image estimation unit 61, the score calculation unit 62, and the display control unit 63 are equivalent to a processor driven according to the action program 67. The processor reads the action program 67 and implements the control determined in the action program 67, thereby functioning as each unit.

FIG3 shows a three-dimensional view of the workpiece of the present embodiment. Referring to FIG1 and FIG3, the workpiece 31 of the present embodiment has a plate-like portion 31a and a plate-like portion 31b formed on the upper side of the plate-like portion 31a. Each of the plate-like portions 31a and 31b has a circular plate shape. The workpiece 31 is arranged on the surface of the stand 35 in such a manner that the upper surface of the plate-like portion 31b on the upper side is parallel to the horizontal direction. The plate-like portion 31b has an edge 31c on the upper surface. In the present embodiment, the upper surface of the plate-like portion 31b is a characteristic portion of the workpiece 31. In the image of the workpiece 31 of the present embodiment, the characteristic portion is detected based on the image of the edge 31c whose plane shape is a circle. Furthermore, the shape of the workpiece can adopt any shape. In addition, the characteristic portion can determine any portion that can be detected by image processing.

(Transportation operation performed by the first robot device) Referring to Figures 1 to 3, the first robot device 3 places the workpiece 31 on the surface of the platform 35 by a predetermined method. The position and posture of the robot 1 change, and the hand 5 grasps the workpiece 31 placed on the upper surface of the platform 35. The robot device 3 transports the workpiece 31 to the predetermined position by changing the position and posture of the robot 1.

When the workpiece 31 is placed on the surface of the stand 35, the position of the workpiece 31 on the stand 35 may be deviated. Before the control device 2 holds the workpiece 31, the camera 6 is placed on the upper side of the workpiece 31 in the vertical direction. At this time, the distance hz from the surface of the characteristic part of the workpiece 31 to the camera 6 is predetermined.

In this example, the posture of the camera 6 is adjusted so that the optical axis of the camera 6 (the Z axis of the camera coordinate system 72) is almost perpendicular to the upper surface of the plate-shaped portion 31b, which is the characteristic portion of the workpiece 31. Furthermore, the position and posture of the robot 1 are controlled so that the position of the upper surface of the plate-shaped portion 31b becomes a predetermined value of the Z axis of the camera coordinate system 72. The position and posture of such a robot are predetermined in the motion program 67.

The imaging control unit 53 photographs the workpiece 31 with the camera 6. The feature detection unit 56 detects the feature portion of the workpiece 31 based on the reference image of the feature portion of the workpiece 31 and the image of the workpiece 31 photographed with the camera 6. The reference image of the workpiece 31 used to detect the feature portion is prepared in advance and stored in the storage unit 42. The feature detection unit 56 detects the edge 31c of the upper surface of the plate-shaped portion 31b of the workpiece 31 in the image photographed with the camera 6 by performing pattern matching.

The estimated position calculation unit 57 calculates the three-dimensional position of the workpiece 31 based on the position of the characteristic portion of the workpiece 31 in the image, the camera parameter 68 stored in the memory unit 42, and the distance hz from the characteristic portion of the workpiece 31 to the camera 6. In this embodiment, the three-dimensional position of the workpiece calculated from the image of the workpiece 31 is called the estimated position. As the estimated position, the position of a specific point 31p set on the workpiece 31 can be calculated. In this embodiment, the center of the circle of the shape of the upper surface of the plate-shaped portion 31b is set as the specific point 31p. The estimated position calculation unit 57 can calculate the estimated position of the workpiece 31 using the camera coordinate system 72. Furthermore, the estimated position calculation unit 57 can convert the coordinate values in the camera coordinate system 72 into coordinate values in the reference coordinate system 71 according to the position and posture of the robot 1.

The motion instruction generation unit 64 generates motion instructions for correcting the position and posture of the robot 1 included in the motion program 67 in a manner corresponding to the estimated position of the workpiece 31. The motion instruction generation unit 64 sends the corrected motion instructions to the motion control unit 43. The motion control unit 43 drives the robot 1 according to the corrected motion instructions. Alternatively, the motion instruction generation unit 64 may also send the position and posture of the robot 1 corresponding to the estimated position of the workpiece 31 to the motion control unit 43. Through this control, the position and posture of the robot 1 are the positions and postures that match the estimated position of the workpiece 31. The robot device 3 can hold the workpiece 31 with good precision.

In this way, the robot device 3 of this embodiment can correct the position and posture of the robot based on the image taken by the camera 6. Since the robot 1 is driven according to the estimated position of the workpiece estimated from the image of the workpiece 31, the hand 5 can be arranged at the correct position. The robot device 3 of this embodiment can perform a predetermined operation with good accuracy.

(Calculation model) Here, a calculation model for calculating a three-dimensional position in space corresponding to a specific position in an image taken by a camera is described. In the calculation model, camera parameters, which are sensor parameters including a plurality of parameters, are used for calculation. The camera parameters include internal parameters and external parameters. In addition, the camera parameters include the distance hz from the camera to the characteristic part of the workpiece. The position in the image of the camera corresponding to an arbitrary position in space is generally expressed by the following formula (1) using the pinhole camera model.

[Number 1]

The coordinate values (X, Y, Z) of the three-dimensional position are expressed by, for example, a reference coordinate system 71. The coordinate values (u, v) of the position on the image are expressed by, for example, an image coordinate system 73 set on the image. The matrix of external parameters is a conversion matrix used to convert the coordinate values (X, Y, Z) of the reference coordinate system 71 in space into the coordinate values (x, y, z) of the camera coordinate system 72. In addition, the matrix of internal parameters is a matrix used to convert the coordinate values (x, y, z) of the camera coordinate system 72 into the coordinate values (u, v) of the image coordinate system 73 in the image. Here, the coordinate value of the Z axis in the camera coordinate system 72 is predetermined corresponding to the distance hz from the camera 6 to the characteristic part of the workpiece 31.

The above equation (1) is an ideal example in which there is no deformation of the image due to lens distortion, etc. In contrast, changes in parameters due to lens distortion, etc. can be considered. First, the calculation of the three-dimensional position and the matrix of the external parameters in equation (1) can be expressed as the following equation (2).

[Number 2]

According to formula (2), the coordinate values (X, Y, Z) expressed by the reference coordinate system 71 can be converted into coordinate values (x, y, z) expressed by the camera coordinate system 72. Next, in order to consider the deformation of the image, the variables x' and y' are defined as shown in the following formulas (3) and (4). Furthermore, the variables x'' and y'' taking the deformation into consideration are calculated as shown in formulas (5) and (6). Here, the relationship between the variables x', y' and r is as shown in formula (7).

[Number 3]

In equations (5) and (6), coefficients _k1 to _k6 are coefficients for lens distortion in the radial direction, and coefficients _p1 and _p2 are coefficients for lens distortion in the circumferential direction. Using variables x'' and y'' that take lens distortion into account, the coordinate values (u, v) on the image of the image coordinate system 73 can be calculated as shown in equations (8) and (9). Equations (8) and (9) are parts corresponding to the operations performed by the matrix of the internal parameters in equation (1) above.

[Number 4]

In the above description, a method of calculating the position in the image from the three-dimensional position in space is described, but the three-dimensional position in space (X, Y, Z) can be calculated based on the above relationship and the coordinate value (u, v) of the position in the image and the distance hz from the camera 6 to the characteristic part of the workpiece 31 in the camera coordinate system 72. The coordinate value of the Z axis corresponding to the camera coordinate value of the distance hz from the camera 6 to the characteristic part of the workpiece 31 can be predetermined and stored in the memory unit 42. The estimated position calculation unit 57 calculates the three-dimensional position in space (X, Y, Z) from the coordinate value (u, v) of the specific position in the image based on the calculation model.

Here, referring to equations (1) to (9), the parameters used to convert the coordinate values of the camera coordinate system 72 into the coordinate values of the image coordinate system 73 are equivalent to the internal parameters. In the calculation model for calculating the three-dimensional position from a specific position in the image, for example, the product of the focal length and the effective size of the image f _x , f _y , the image center c _x , _cy , and the coefficients k ₁ to k ₆ , p ₁ , p ₂ related to the deformation are equivalent to the internal parameters. In addition, the parameters used to convert the coordinate values of the reference coordinate system 71 into the coordinate values of the camera coordinate system 72 are equivalent to the external parameters. For example, the parameters included in the matrix of the external parameters of equation (1) are equivalent to the external parameters.

(Parameter evaluation and correction control) The evaluation of the camera parameters and the correction of the camera parameters in this embodiment are described. When the camera parameters 68 are newly set, they can be set by any method. For example, a flat correction plate can be used to make the camera parameters. For example, a pattern of points of different sizes is drawn in a grid shape on the surface of the correction plate. The correction plate is arranged at a predetermined position on the stand 35.

The operator pre-measures the position of the calibration plate in the reference coordinate system 71. The intervals between the points drawn on the calibration plate are pre-measured. The robot 1 is driven to arrange the camera 6 at a predetermined distance in the height direction perpendicular to the calibration plate. The camera 6 is used to photograph the dot pattern of the calibration plate.

Next, the camera 6 is moved to an arbitrary position relative to the calibration plate. For example, the camera 6 is moved in the direction of the optical axis of the camera 6. The positions of the points of the calibration plate in the image coordinate system 73, the position and posture of the robot, and the positions of the points of the calibration plate in the reference coordinate system 71 measured in advance are obtained in the images taken at multiple positions of the camera 6. Then, the camera parameters are generated according to equation (1) or equation (8) and equation (9).

Fig. 4 shows a schematic side view of the camera and the workpiece of this embodiment. In Fig. 4, a correction plate 32 and a workpiece 31 are superimposed. A dot pattern is drawn on the upper surface of the correction plate 32. The upper surface of the correction plate 32 and the camera 6 are arranged at a distance hc.

The workpiece 31 of this embodiment is thicker than the correction plate 32. The distance hz between the upper surface of the plate-shaped portion 31b, which is the characteristic portion of the workpiece 31, and the camera 6 is shorter than the distance hc by the distance difference Δh. As a result, the distance hz used for calculating the three-dimensional position may be different from the preset distance hc. Therefore, it is sometimes difficult to correctly obtain the distance hz that reflects the distance difference Δh.

Alternatively, the operator may take a picture of the workpiece 31 including the feature portion to generate a reference image for pattern matching. This reference image may contain image deformation or errors in the shape of the feature portion. Furthermore, if the robot continues to operate, the values of the camera parameters used to detect the correct three-dimensional position of the workpiece may deviate over time, and the correct estimated position of the workpiece may sometimes be unable to be calculated.

When the camera parameters are newly generated and the robot device continues the operation, the parameters included in the camera parameters may deviate from the optimal value. Therefore, the control device 2 of this embodiment evaluates the camera parameters. That is, when using the current camera parameters, it is determined whether the three-dimensional position of the workpiece can be detected from the image with an error within the allowable range. Then, when the evaluation of the camera parameters is low, the camera parameters are corrected.

FIG5 shows a flow chart of the control of the evaluation of the camera parameters and the correction of the camera parameters for implementing the present embodiment. Here, the reference image for pattern matching is generated in advance. Furthermore, in order to evaluate and correct the camera parameters, the distance hz from the camera 6 to the characteristic part, i.e., the upper surface, of the workpiece 31 when taking the image is predetermined. Furthermore, the posture of the camera 6 is predetermined. In the present embodiment, the posture of the robot 1 is predetermined so that the Z direction of the camera coordinate system 72 is perpendicular to the upper surface of the workpiece 31. Furthermore, the posture of the robot 1 is predetermined so that the X axis of the camera coordinate system 72 is parallel to the X axis of the reference coordinate system 71. Furthermore, as described later, distance hz can also be corrected as one of the camera parameters.

Referring to Fig. 1, Fig. 2, and Fig. 5, in step 101, the camera 6 is moved to the first position P6a. As the first position P6a, any position at which the characteristic portion of the workpiece 31 can be photographed by the camera 6 can be selected. In the present embodiment, as shown in Fig. 1, the position and posture of the robot 1 are adjusted so that the camera 6 is arranged directly above the workpiece 31. The operator corrects the position and posture of the robot 1 using the teaching operation panel 26. Alternatively, the action command generation unit 64 can also control the position and posture of the robot 1 so that the camera 6 moves to the predetermined first position P6a.

The measured position calculation unit 52 calculates the first measured position when the camera 6 is arranged at the first position P6a based on the position and posture of the robot 1. In this embodiment, the position of one of the components calculated based on the driving state of the moving device is called the measured position. The relative position and posture of the camera coordinate system 72 with respect to the tool coordinate system 74 are predetermined. The measured position calculation unit 52 can calculate the first measured position of the camera 6, for example, using the coordinate value in the reference coordinate system 71.

In step 102, the imaging control unit 53 uses the camera 6 to photograph the workpiece 31. Specifically, the camera 6 photographs the upper surface of the plate-shaped portion 31b, which is a characteristic portion of the workpiece 31.

In step 103, the feature detection unit 56 detects the feature portion in the image taken when the camera 6 is configured at the first position P6a based on the reference image. The feature detection unit 56 detects the feature portion by pattern matching. In this embodiment, the edge 31c of the upper surface of the plate-shaped portion 31b of the workpiece 31 is detected. The feature detection unit 56 calculates the position of the specific point 31p on the upper surface of the plate-shaped portion 31b using the image coordinate system 73 as the position of the workpiece 31 in the image.

FIG6 shows an image taken by the camera when the camera is arranged at the first position. An image coordinate system 73 is set in the image 81 taken by the camera 6. The image coordinate system 73 is composed of the U axis and the V axis. An image 31g of the workpiece is captured in the image 81. In addition, in the image 81, a mark 31m is displayed indicating the position of the feature portion detected by the feature detection unit 56 by pattern matching. In this example, the mark 31m is composed of two intersecting lines. The intersection of the two lines is a specific point 31p indicating the position of the feature portion (the position of the workpiece 31).

Referring to FIG. 5 , in step 103, the estimated position calculation unit 57 detects the first estimated position of the workpiece 31 when the camera 6 is arranged at the first position. In the present embodiment, the position of the workpiece calculated based on the image captured by the camera is referred to as the estimated position. As the estimated position, the position of the specific point 31p of the characteristic portion can be calculated using the coordinate value of the camera coordinate system 72. The estimated position calculation unit 57 calculates the first estimated position of the workpiece 31 based on the position of the characteristic portion of the workpiece 31 in the image, the position and posture of the robot 1, the camera parameters, and the distance hz according to the aforementioned equation (1) or equations (2) to (9).

Next, in step 104, the position and posture of the robot 1 are controlled so that the camera 6 is arranged at the second position P6b. The second position P6b is a position different from the first position P6a. As the second position P6b, any position at which the characteristic portion of the workpiece 31 can be photographed by the camera 6 can be selected. In the present embodiment, the posture of the camera 6 is almost fixed at the first position and the second position P6b of each camera. For example, the direction of the Z axis of the camera coordinate system 72 is maintained parallel to the vertical direction. In addition, the posture of the camera 6 is maintained so that the directions of the X axis and the Y axis of the camera coordinate system 72 are constant.

FIG7 shows a schematic diagram of the first robot device when the camera is arranged at the second position. The operator operates the teaching operation panel 26 to change the position and posture of the robot, thereby moving the camera 6 in a direction parallel to the surface of the characteristic portion of the workpiece 31 as shown by the arrow 98, and arrange it at the second position P6b. The distance hz from the characteristic portion of the workpiece 31 to the camera 6 is maintained constant. In addition, the camera 6 is moved so that the interior of the field of view 6a of the camera 6 includes the characteristic portion of the workpiece 31. Alternatively, the action command generation unit 64 can also control the position and posture of the robot 1 so that the camera 6 moves to the predetermined second position P6b.

5 , in step 104, the measured position calculation unit 52 calculates the second measured position when the camera 6 is arranged at the second position P6b based on the position and posture of the robot 1. The measured position calculation unit 52 can calculate the second measured position of the camera 6 using the coordinate values of the reference coordinate system 71, for example.

In step 105, the camera 6 photographs the workpiece 31. In step 106, the same control as in step 103 is performed. The feature detection unit 56 detects the feature portion in the image photographed when the camera 6 is arranged at the second position P6b. Furthermore, the feature detection unit 56 detects the position of the feature portion in the image.

FIG8 shows an image taken by the camera when the camera is placed at the second position P6b. In the image 82, the position of the image 31g of the workpiece moves. In addition, since the image 82 is taken from the tilted direction of the workpiece 31, the side surfaces of the plate-like parts 31a and 31b of the workpiece 31 are slightly photographed. In the image 82, the mark 31m and the specific point 31p of the position of the feature part detected by the feature detection unit 56 are shown.

5, in step 106, the estimated position calculation unit 57 calculates the second estimated position when the camera 6 is arranged at the second position using the camera coordinate system 72. The estimated position calculation unit 57 calculates the second estimated position of the workpiece 31 using the position of the feature part in the image, the position and posture of the robot 1, the camera parameters, and the distance hz.

Next, in step 107, the evaluation unit 58 calculates the amount of movement between the first measured position and the second measured position. The evaluation unit 58 calculates the distance from the first estimated position to the second estimated position. Here, when the camera parameters are correct, the amount of movement between the first measured position and the second measured position and the distance from the first estimated position to the second estimated position are almost the same. In this embodiment, the amount of movement between the first measured position and the second measured position is referred to as the amount of movement of the measured position. In addition, the distance from the first estimated position to the second estimated position is referred to as the distance between the estimated positions.

In step 108, the evaluation unit 58 evaluates the camera parameters. In the present embodiment, the evaluation unit 58 evaluates the camera parameters based on the movement amount of the measured position and the distance between the estimated positions. In the present embodiment, the error obtained by subtracting the movement amount of the measured position from the distance between the estimated positions is calculated. The evaluation unit determines whether the size of the error (the absolute value of the error) is less than a predetermined judgment value. The judgment value can be set, for example, in accordance with the size of the movement amount of the measured value position. Alternatively, a certain judgment value can be set for the movement amount of all measured positions.

In step 108, when the magnitude of the error is smaller than a predetermined judgment value, it can be determined that the estimated position of the workpiece can be calculated with good accuracy using the currently set camera parameters. It can be determined that there is no need to correct the current camera parameters. In this case, the control is terminated.

On the other hand, when the error of the distance between the estimated positions relative to the movement amount of the measured position is greater than a predetermined judgment value, it can be determined that the estimated position of the workpiece calculated by the current camera parameters contains an error greater than the allowable value. In this case, control is transferred to step 109 in order to correct the camera parameters.

Furthermore, the evaluation unit 58 can evaluate the camera parameters in any method. For example, the evaluation unit 58 can also evaluate the camera parameters based on the ratio of the distance between the estimated positions to the movement amount of the measured positions. The evaluation unit 58 can determine that the closer this ratio is to 1, the better the accuracy of the camera parameters. The evaluation unit can also determine the judgment value of the ratio and evaluate the camera parameters.

In step 109, the selection unit 59 selects the parameter to be corrected from among the plurality of parameters included in the camera parameters. The operator can select the parameter to be corrected and input it using the input unit 27 of the teaching operation panel 26. Alternatively, the operator can also select all the parameters included in the camera parameters. The selection unit 59 can set the parameter to be corrected in response to the operator's operation. Alternatively, the selection unit 59 can automatically select the parameter to be corrected by a predetermined control. Alternatively, the parameter to be corrected from among the plurality of parameters included in the camera parameters can be predetermined instead of selecting the parameter using the selection unit 59.

Next, in step 110, the correction unit 60 corrects the parameters selected by the selection unit 59. The correction unit 60 corrects the camera parameters based on the amount of movement between the first measured position and the second measured position, and the first estimated position and the second estimated position. If the accuracy of the camera parameters is good, the amount of movement of the measured position and the distance between the estimated positions are almost the same. In this embodiment, the correction unit 60 calculates the error between the amount of movement of the measured position and the distance between the estimated positions. For example, the correction unit 60 calculates the error obtained by subtracting the amount of movement of the measured position from the distance between the estimated positions.

(Parameter correction method) Here, the method for correcting camera parameters by the correction unit 60 of the present embodiment is described. The control device 2 changes at least one of the first position and the second position by the robot 1 and repeats the shooting. The shooting is repeated a number of times corresponding to the number of parameters selected by the selection unit 59. The correction unit 60 can correct the camera parameters in a manner that minimizes the error based on the repeatedly shot images.

In the first robot device 3 of the present embodiment, the conversion from the coordinate value of the camera coordinate system 72 to the coordinate value of the reference coordinate system 71 can be expressed by the equation (2) including the matrix R and the variable t. Next, if the movement amount of the measured position in the camera coordinate system 72 is expressed by ( _Dxc , _Dyc , _Dzc ), the movement amount of the measured position in the camera coordinate system 72 can be calculated from the movement amount of the measured position in the reference coordinate system 71 ( _Dxw , _Dyw , _Dzw ) by the following equation (10).

[Number 5]

Next, in the image coordinate system 73, the position of the workpiece moves from the coordinate value (u1, v1) corresponding to the first position to the coordinate value (u2, v2) corresponding to the second position. The coordinate values of each image coordinate system 73 can be converted into the coordinate value (x1, y1, hz) and the coordinate value (x2, y2, hz) of the camera coordinate system 72 by using equation (1) or equation (2) to equation (9) according to the distance _hz from _the camera to the feature part and the parameters c, _f , _k , p. By transforming equation (10), equation (11) can be derived.

[Number 6]

Here, [( _x1 , _y1 , hz)-( _x2 , _y2 , hz)] on the left side of equation (11) corresponds to the distance between the estimated positions in the camera coordinate system 72. R ( _Dxw , _Dyw , _Dzw ) on the left side is the amount of movement of the measured position in the camera coordinate system. When the distance between the estimated positions calculated from the image is equal to the amount of movement of the measured position, it can be determined that the camera parameters are correct. That is, when the accuracy of the camera parameters is high, equation (11) is established. Alternatively, equation (12) which is a modification of equation (11) is established.

The left side of equation (12) is the measured value corresponding to the movement amount of the measured position. That is, it corresponds to the movement amount of the measured position of the camera calculated from the position and posture of the robot. The right side of equation (12) corresponds to the distance between the estimated positions of the workpiece obtained from the position of the feature part of the image, parameters c, f, k, p, and matrix R. In equation (12), it is expressed as a reference coordinate system 71. In practice, due to the error of the camera parameters, equation (12) does not hold. By introducing the error ΔD, equation (13) can be derived from equation (12). The (ΔDx, ΔDy, ΔDz) on the left side of equation (13) corresponds to the error ΔD obtained by subtracting the movement amount of the measured position from the distance between the estimated positions.

The correction unit 60 calculates the selected parameters by Newton's method so that (ΔDx, ΔDy, ΔDz) on the left side of equation (13) approaches (0, 0, 0). The correction unit 60 can calculate the selected parameters by calculating the function of Newton's method and repeatedly calculating by the iterative method. In the method using the error between the two points, the parameters can be calculated one by one.

Next, another example of calculating the camera parameters is described. The right side of equation (12) is a function of the parameters c, f, k, p, matrix R, and distance hz. Therefore, the displacement (Dxw, _Dyw , _{Dzw )} of the measured position in the reference coordinate system can be expressed by the following equations (14) to (16).

[Number 7]

For example, _Dxw (c, f, k, p, R, hz, u, v) on the right side of equation (14) is a function representing parameters c, f, k, p, matrix R, distance hz, and image coordinate values (u, v). Then, the least square method can be used to calculate the parameters based on the measured values of the image coordinate values (u, v) and the displacement ( _Dxw , _Dyw , _Dzw ). For example, when the Gauss-Newton method is used to calculate the parameters _fx , _fy , and distance hz, _Δfx , _Δfy , and Δhz can be obtained by using the following equation (17), and the parameters _fx , _fy , and distance hz can be corrected for the measured values. Δf _x , Δf _y , and Δhz can be calculated by calculating the inverse matrix of the matrix on the left using the singular value decomposition method. By repeating this operation, the parameters f _x , f _y , and distance hz can be made close to the true value.

[Number 8]

When data is obtained from multiple moving points, a number is attached to each point at each position and an annotation is attached to the formula to mark it, which becomes the following formula (18).

[Number 9]

When more variables are calculated at the same time, the equation (17) can be expanded. For example, if the parameters c _x and _cy are included, the equation (19) becomes as follows.

[Number 10]

(Display of graph of error) In the control device 2 of this embodiment, the display control unit 63 can display the relationship between the movement amount of the measured position and the error of the distance between the estimated positions relative to the movement amount of the measured position on the display unit 28. In the embodiment of this embodiment, the graphs shown in Figures 9 to 14 can be displayed on the display unit 28.

Fig. 9 is a graph showing the relationship between the X-direction shift amount of the measured position in the reference coordinate system and the error of the distance between the estimated positions relative to the shift amount of the measured position. Fig. 10 is a graph showing the relationship between the Y-direction shift amount of the measured position in the reference coordinate system and the error of the distance between the estimated positions relative to the shift amount of the measured position.

For example, as shown by arrow 98 in FIG. 7 , the camera 6 is moved in the direction of the X axis of the reference coordinate system 71. The image of the workpiece 31 can be captured at one first position of the camera 6 and at a plurality of second positions of the camera 6. The correction unit 60 calculates the error of the distance between the estimated positions relative to the movement amount of the measured position at each measured position. Furthermore, the display control unit 63 or the evaluation unit 58 can also calculate the error instead of the correction unit 60. The display control unit 63 displays the relationship between the movement amount of the measured position and the error on the display unit 28. The error here is independent of the X direction and the Y direction and is an error in an arbitrary direction.

The tendency of the error relative to the measured position can be known from the graphs of FIG9 and FIG10. For example, the parameter containing the error can be estimated from the tendency of the error relative to the measured position, and the parameter to be corrected can be determined. Alternatively, the position where the error is the largest can be estimated to confirm the maximum value of the error.

FIG. 11 is a graph showing the error in the X direction and the error in the Y direction when the camera is moved in the X direction of the reference coordinate system. FIG. 12 is a graph showing the error in the Y direction and the error in the X direction when the camera is moved in the Y direction of the reference coordinate system. Referring to FIG. 11 and FIG. 12, in this example, the error in the same direction as the direction in which the camera is moved increases as the amount of movement increases. In addition, the error in the direction perpendicular to the direction in which the camera is moved hardly changes. In this case, it can be determined that the distance hz from the camera to the feature portion of the workpiece is set incorrectly.

FIG. 13 and FIG. 14 are graphs showing errors measured under conditions different from those of FIG. 11 and FIG. 12. FIG. 13 is a graph showing the error in the X direction and the error in the Y direction when the camera is moved in the X direction of the reference coordinate system. FIG. 14 is a graph showing the error in the Y direction and the error in the X direction when the camera is moved in the Y direction of the reference coordinate system. Referring to FIG. 13 and FIG. 14, in this example, the error in the direction in which the camera is moved and the error in the direction perpendicular to the direction in which the camera is moved become larger. In this case, it can be determined that the camera coordinate system is set incorrectly. For example, it can be determined that the camera coordinate system has an error around the R axis. Therefore, it can be determined that the external parameters related to the setting of the camera coordinate system need to be corrected.

In this way, by displaying the relationship between the movement amount of the measured position and the error of the distance between the estimated positions with respect to the movement amount of the measured position on the display unit 28, the operator can visually understand the magnitude of the error. In other words, the operator can visually judge the accuracy of the camera parameters.

The operator or the evaluation unit 58 can determine whether to correct the camera parameters based on the graph displayed on the display unit 28. In addition, the operator or the selection unit 59 can select the parameters to be corrected based on the error in the direction in which the camera is moved and the error in the direction perpendicular to the direction in which the camera is moved. In other words, the parameters that are the reason why the estimated position cannot be calculated with good accuracy can be selected.

Furthermore, the image showing the relationship between the measured position shift amount and the distance between the estimated position and the error of the measured position shift amount is not limited to a graph, and any image may be displayed. For example, the display unit may display a table showing the measured position shift amount and the error.

(Display of estimated image of workpiece) Referring to FIG. 2, the image estimation unit 61 of the image processing unit 51 estimates the image of the characteristic portion when the camera 6 is arranged at the second position based on the image of the characteristic portion photographed when the camera 6 is arranged at the first position, the camera parameters, and the movement amount of the measured position. The display control unit 63 can display the image of the characteristic portion estimated by the image estimation unit 61 by superimposing it on the image of the characteristic portion photographed when the camera 6 is arranged at the second position.

In Fig. 15, an image including an estimated image of the workpiece estimated when the workpiece is photographed at the second position is displayed. In the image 89, an image 31g of the workpiece when the workpiece 31 is actually photographed at the second position and an estimated image 31r of the workpiece estimated by the image estimation unit 61 are displayed. The specific point 31p indicates the position of the characteristic portion of the workpiece 31 in the actual image 31g of the workpiece.

Referring to FIG. 6 and FIG. 15 , in the image 81 captured by the camera at the first position, the image 31g of the workpiece is arranged at the center of the image 81. The estimated position calculation unit 57 calculates the first estimated position of the workpiece 31 using the camera coordinate system 72. The measured position calculation unit 52 calculates the first measured position and the second measured position of the camera. The image estimation unit 61 estimates the second estimated position based on the moving direction and the moving amount of the measured position from the first measured position to the second measured position and the first estimated position. That is, the image estimation unit 61 estimates the position of the specific point when the camera is arranged at the second position using the coordinate value of the camera coordinate system 72. The image estimation unit 61 can estimate the coordinate value of the specific point 31p in the image coordinate system by using the coordinate value of the camera coordinate system 72 and the camera parameter according to the formula (1). Then, the image estimation unit 61 can create an estimated image 31r of the workpiece based on the position of the specific point 31p of the workpiece 31 in the image 89. The display control unit 63 can display the estimated image 31r of the workpiece estimated by the image estimation unit 61.

Furthermore, the image estimation unit 61 can estimate the amount of movement and the direction of movement of the specific point 31p in the image coordinate system 73 by using equation (1) based on the amount of movement and the direction of movement from the first measured position to the second measured position of the camera 6. Then, in the image, the position of the specific point 31p corresponding to the second position can be calculated from the position of the specific point 31p corresponding to the first position. The image estimation unit 61 can generate an estimated image 31r of the workpiece based on the position of the specific point 31p of the image corresponding to the second measured position.

The display control unit 63 displays the image of the characteristic portion estimated by the image estimation unit 61 superimposed on the image of the characteristic portion captured when the camera 6 is arranged at the second position. When the accuracy of the camera parameter 68 is excellent, the position of the image 31g of the workpiece in the image 89 after the camera moves to the second position is almost consistent with the estimated image 31r of the workpiece estimated by the image estimation unit 61. On the other hand, when the accuracy of the camera parameter 68 is poor, the estimated image 31r of the workpiece is displayed deviated from the actual image 31g of the workpiece.

The operator can estimate the size of the error by viewing the image 89 of the workpiece. The operator can also determine whether to correct the camera parameters 68 based on the image 89.

Here, the position of the estimated image 31r estimated by the image estimation unit 61 and the position of the image 31g of the workpiece photographed by the camera 6 move in the same direction relative to the position of the workpiece photographed at the first position, but the magnitude of the movement may be different. In this case, it can be determined that the setting of the distance hz between the camera 6 and the characteristic part of the workpiece 31 is incorrect. Or, although a part of the estimated image 31r of the workpiece is highly consistent with the image 31g of the workpiece, sometimes the other part of the estimated image 31r of the workpiece is inconsistent with the image 31g of the workpiece actually photographed. In this case, it can be determined that the reference image of the workpiece is incorrect. For example, if a plurality of surfaces of different heights are set as characteristic portions in the reference image, a portion of the estimated image of the workpiece estimated by the image estimation unit may not match the image of the workpiece taken by the camera 6. In this case, the operator may re-create the reference image.

In this way, an incorrect parameter among a plurality of parameters included in the camera parameter 68 can be identified by using the estimated image produced by the image estimation unit 61. Alternatively, it can be identified that the reference image is incorrect.

Incidentally, in the image 89 shown in FIG15 , the image 31g of the workpiece is arranged at the end of the image 89. In a two-dimensional camera, the farther away from the optical axis of the camera, the greater the deformation of the image. In this example, the image 31g of the workpiece is deformed. On the other hand, the estimated image 31r of the workpiece is an image estimated without considering the deformation of the image. That is, the image estimation unit 61 generates the estimated image 31r of the workpiece according to formula (1). Therefore, the shape of the image of the plate-like parts 31a and 31b of the workpiece is a perfect circle.

FIG. 16 shows an image including an estimated image of a workpiece in consideration of the deformation of the image. The image estimation unit 61 of the present embodiment can produce an estimated image 31s of a workpiece including the deformation of the image. Specifically, the image estimation unit 61 can estimate an estimated image of a characteristic portion of the workpiece including the deformation of the image. In the image 90, the estimated image 31s of the workpiece is superimposed on the image 31g of the workpiece actually captured when the camera 6 is arranged at the second position.

The image estimation unit 61 can calculate the outline of the workpiece 31 in the image 90 by a method that expands the method of calculating the position of a specific point in the image. For example, the image estimation unit 61 obtains the first estimated position of the workpiece 31 when the camera is arranged at the first position. Then, the image estimation unit 61 sets a plurality of points on the outline of the workpiece 31. The image estimation unit 61 calculates the coordinate values of the points on the outline in the camera coordinate system 72.

The image estimation unit 61 estimates the position of the point on the outline of the workpiece in the camera coordinate system 72 when the camera 6 is arranged at the second position based on the movement direction and the movement amount from the first measured position to the second measured position. Then, the image estimation unit 61 estimates the position of the point on the outline of the workpiece in the image coordinate system 73 from the position of the point on the outline of the workpiece in the camera coordinate system 72 based on equations (2) to (9). The image estimation unit 61 can connect the positions of the points on the outline of the workpiece in the image to generate the estimated image 31s of the workpiece.

By this control, an image including a deformation of the image is displayed in the image 90. An estimated image close to the image of the workpiece actually photographed can be displayed. By controlling to display the estimated image including the deformation of the image, it becomes easy to determine whether there is a problem in any of the camera parameters, the distance hz from the camera to the workpiece, and the reference image.

That is, when an estimated image including a deformation of an image is displayed, if the estimated image is consistent with the image taken by the camera, it can be determined that the accuracy of the camera parameters is excellent. Even if an estimated image including a deformation of an image is generated, if the position of the image is completely different, it can still be determined that the setting of the distance hz from the camera to the workpiece is incorrect. Furthermore, even if an image including a deformation of an image is generated, if a part of the image does not overlap, it can still be determined that the reference image is inappropriate. Furthermore, when the operator confirms that the reference image is appropriate, it can be determined that the parameters related to the deformation among the camera parameters are incorrect.

FIG17 shows an image of the points calculated by the score calculation unit of the present embodiment. In the present embodiment, the edge 31c of the upper surface of the plate-shaped portion 31b of the workpiece 31 is designated as a characteristic portion. Referring to FIG2 and FIG17, the score calculation unit 62 detects the overlapping portion of the characteristic portion of the estimated image 31s of the workpiece estimated by the image estimation unit 61 and the characteristic portion of the image 31g of the workpiece photographed by the camera. The overlapping portion of the present embodiment is a line segment 31n where the outline of the characteristic portion of the estimated image 31s of the workpiece and the outline of the characteristic portion of the image 31g of the workpiece photographed by the camera substantially overlap. In the present embodiment, when the distance between the contour of the characteristic portion of the estimated image 31s of the workpiece and the contour of the characteristic portion of the image 31g of the workpiece is smaller than a predetermined determination value, it is determined that the contours overlap each other.

The display control unit 63 can display an image so that the overlapping portion of the characteristic portion of the workpiece image 31g taken by the camera 6 and the characteristic portion of the estimated workpiece image 31r estimated by the image estimation unit 61 can be known. Here, the line segment 31n which is the portion where the characteristic portions overlap each other is indicated by a thick solid line.

The score calculation unit 62 calculates the points indicating the degree of overlap of the images of the characteristic parts estimated by the image estimation unit 61 for the image of the characteristic parts photographed when the camera 6 is arranged at the second position. The score calculation unit 62 of the present embodiment can calculate the points as the ratio of the length of the line segment 31n of the overlapping part to the length of the periphery of the characteristic part of the image 31g of the workpiece photographed by the camera 6. The closer this point is to 1, the better the estimated image and the image of the workpiece photographed by the camera are determined to be consistent.

Alternatively, the score calculation unit 62 may calculate a value obtained by subtracting the length of the line segment of the overlapping portion from the length of the periphery of the characteristic portion of the image 31g of the workpiece photographed by the camera 6 as a point. The closer the point is to zero, the better the estimated image and the image of the workpiece photographed by the camera are determined to be consistent. The display control unit 63 may display the point on the display unit 28.

By this control, the operator can determine that the better the score is, the more accurately the camera parameters can be set. Alternatively, the evaluation unit 58 can also determine whether to correct the camera parameters based on the score calculated by the score calculation unit 62.

(Second robot device) FIG. 18 shows a schematic diagram of the second robot device of the present embodiment. In the second robot device 4, the position of the camera 6 is fixed, and the position of the workpiece 31 is changed by the robot 1. That is, in the second robot device, one of the components between the camera 6 and the workpiece 31 is the workpiece 31. The workpiece 31 is held by the hand 5 mounted on the robot 1. The camera 6 as the other component is fixed in position by the stand 36. The robot 1 is configured to change its position and posture, thereby changing the three-dimensional position and posture of the workpiece 31.

The characteristic portion of the workpiece 31 is a portion that can be photographed with the camera 6. In the second robot device 4, the bottom surface of the workpiece 31 is set as the characteristic portion of the workpiece 31. For example, the edge of the circle of the bottom surface of the plate-shaped portion 31b is set as the characteristic portion of the workpiece 31. Then, the position of the center of the bottom surface of the plate-shaped portion 31b is set as the position of the specific point. The reference image of the bottom surface of the plate-shaped portion 31b is prepared in advance. The feature detection unit 56 detects the contour of the bottom surface of the plate-shaped portion 31b in the image of the workpiece 31.

The estimated position calculation unit 57 calculates the estimated position of the workpiece 31 using the camera coordinate system 72 based on the position of the characteristic portion in the image of the workpiece 31 and the camera parameters. The measured position calculation unit 52 calculates the three-dimensional position of the workpiece 31, i.e., the measured position, using the reference coordinate system 71 based on the position and posture of the robot 1 in the driving state as a mobile device.

The control device 2 places the workpiece 31 at the first position P31a by changing the position and posture of the robot. The camera 6 photographs the characteristic portion of the workpiece 31, i.e., the bottom surface of the workpiece 31. The control device 2 changes the position and posture of the robot, thereby moving the workpiece 31 to the second position P31b as shown by the arrow 99. The distance hz from the camera 6 to the characteristic portion of the workpiece 31 is predetermined. The robot 1 moves the workpiece 31 in parallel so that the distance hz from the camera 6 to the characteristic portion of the workpiece 31 and the posture of the workpiece 31 do not change.

The feature detection unit 56 detects a feature portion in an image captured when the workpiece 31 is arranged at the first position P31a, and a feature portion in an image captured when the workpiece 31 is arranged at the second position P31b. The estimated position calculation unit 57 calculates a first estimated position when the workpiece 31 is arranged at the first position, and a second estimated position when the workpiece 31 is arranged at the second position. The measured position calculation unit 52 calculates a first measured position when the workpiece 31 is arranged at the first position, and a second measured position when the workpiece 31 is arranged at the second position.

The evaluation unit 58 evaluates the camera parameter based on the amount of movement between the first measured position and the second measured position and the error in the distance from the first estimated position to the second estimated position. The evaluation unit 58 determines whether the camera parameter needs to be corrected.

When the camera parameters need to be corrected, the correction unit 60 corrects the camera parameters based on the amount of movement between the first measured position and the second measured position and the distance from the first estimated position to the second estimated position. Specifically, the correction unit 60 calculates the error between the amount of movement between the first measured position and the second measured position and the distance from the first estimated position to the second estimated position, and corrects the camera parameters in a manner that minimizes the error.

Thus, the evaluation and correction of the camera parameters can also be implemented in the second robot device 4. As the other structures, functions and effects are the same as those of the first robot device 3, they will not be repeated here.

In this embodiment, a robot is used as a moving device for moving a workpiece or a sensor, but the present invention is not limited to this embodiment. The moving device can be any moving device that can change the relative position of the sensor with respect to the workpiece. For example, a conveyor for transporting workpieces and an automated guided vehicle (AGV) can be used as the moving device.

In the present embodiment, one of the components of the sensor and the workpiece is moved by a moving device, and the position of the other component is fixed, but the present invention is not limited to this embodiment. It is also possible to configure the components of both the sensor and the workpiece to be moved by a moving device. For example, the moving device may also be configured to include a robot and a conveyor, the robot moves the sensor as one of the components, and the conveyor moves the workpiece as the other component. In this case, the correction unit may also correct the sensor parameters based on the movement amount based on the first measured position and the second measured position of one of the components and the distance from the first estimated position to the second estimated position.

More specifically, the measured position calculation unit calculates a first measured position when one of the components is arranged at the first position, and a second measured position when one of the components is arranged at the second position. The measured position calculation unit calculates a third measured position of the other component when one of the components is arranged at the first position, and a fourth measured position of the other component when one of the components is arranged at the second position. The correction unit can calculate a relative movement amount of one of the components with respect to the other component based on the first measured position and the second measured position of one of the components and the third measured position and the fourth measured position of the other component. The correction unit can calculate the movement amount of one of the components assuming that the other component is stationary. In other words, the correction unit can calculate the movement amount of one of the components when viewed from the other component. Then, the correction unit can correct the sensor parameters according to the relative movement of one component with respect to the other component.

In this embodiment, a camera that captures a two-dimensional image is exemplified as a sensor, but the present invention is not limited to this form. Any sensor that uses sensor parameters to calculate a three-dimensional position in space corresponding to a specific position in an image of a workpiece captured by the sensor can be used. For example, a three-dimensional sensor that detects a spatial position can be used as a sensor. When the three-dimensional sensor is fixed to a work tool, external parameters that convert the coordinate values of the sensor coordinate system into the coordinate values of the tool coordinate system can be evaluated or corrected.

In this embodiment, the operator mainly drives the robot manually or evaluates the camera parameters and images, but the present invention is not limited to this embodiment. The control device can also automatically drive the mobile device, or use the sensor to take images, or evaluate the sensor parameters and images according to a predetermined program. Furthermore, the control device can also automatically select the parameters to be corrected or correct the parameters.

In at least one embodiment as described above, a parameter correction device can be provided that can correct sensor parameters with excellent accuracy.

Although the present disclosure is described in detail, the present disclosure is not limited to the above-mentioned embodiments. These embodiments may be subject to various additions, replacements, changes, partial deletions, etc., without departing from the gist of the present disclosure, or without departing from the intent of the present disclosure derived from the contents described in the patent application and its equivalents. Furthermore, these embodiments may also be combined and implemented. For example, in the above-mentioned embodiments, the sequence of each action or the sequence of each processing is shown as an example and is not limited to such sequence. Furthermore, the same applies to the use of numerical values or formulas in the description of the above-mentioned embodiments.

The following notes are disclosed regarding the above-mentioned embodiments and modifications.

(Note 1) A parameter correction device 2 is a parameter correction device 2 for correcting sensor parameters used to calculate a three-dimensional position, wherein the three-dimensional position corresponds to a specific position in an image 31g of a workpiece taken by a sensor 6, and the parameter correction device 2 comprises: a feature detection unit 56 for detecting a feature portion 31c in the image 31g of the workpiece; an estimated position calculation unit 57 for calculating the three-dimensional position of the workpiece 31, i.e., the estimated position, based on the position of the feature portion in the image 31g of the workpiece and the sensor parameters; a measured position calculation unit 52 for calculating the three-dimensional position of one of the components, i.e., the measured position, based on the driving state of the moving device 1 of the moving sensor 6 and at least one of the components of the workpiece 31; and a correction unit 60 for correcting the sensor parameters, The feature detection unit 56 detects the feature part in the image 81 taken when one of the components is arranged at the first position P6a, P31a, and the feature part in the image 82 taken when one of the components is arranged at the second position P6b, P31b. The estimated position calculation unit 57 calculates the first estimated position when one of the components is arranged at the first position P6a, P31a, and the second estimated position when one of the components is arranged at the second position. The measured position calculation unit 52 calculates the first measured position when one of the components is arranged at the first position, and the second measured position when one of the components is arranged at the second position. The correction unit 60 corrects the sensor parameters according to the movement amount based on the first measured position and the second measured position, and the distance from the first estimated position to the second estimated position.

(Note 2) As in the parameter correction device of Note 1, the correction unit 60 calculates the error between the movement amount based on the first measured position and the second measured position and the distance from the first estimated position to the second estimated position, and corrects the sensor parameter to minimize the error.

(Supplementary Note 3) The parameter correction device of Supplementary Note 1 includes a selection unit 59, which selects a parameter to be corrected from a plurality of parameters included in the sensor parameters. The correction unit 60 corrects the parameter selected by the selection unit by changing at least one of the first position and the second position and repeatedly photographing the image a number of times corresponding to the number of parameters selected by the selection unit 59.

(Note 4) As in the parameter correction device of Note 1, the positions of the sensor and another component in the workpiece are fixed, and the correction unit corrects the sensor parameters according to the movement between the first measured position and the second measured position.

(Note 5) The parameter correction device as in any one of Notes 1 to 4 has an evaluation unit 58, which evaluates the sensor parameter. The evaluation unit 58 evaluates the sensor parameter based on the movement amount based on the first measured position and the second measured position and the error of the distance from the first estimated position to the second estimated position.

(Note 6) The parameter correction device as in any one of Notes 1 to 4 has a display control unit 63, which controls the image displayed on the display unit 28, The correction unit 60 calculates the movement amount based on the first measured position and the second measured position and the error of the distance from the first estimated position to the second estimated position, The display control unit 63 displays the relationship between the movement amount based on the first measured position and the second measured position and the error on the display unit 28.

(Supplementary Note 7) A parameter correction device as in any one of Supplementary Notes 1 to 4, comprising: a display control unit for controlling an image displayed on the display unit; and an image estimation unit 61 for estimating an image of a characteristic portion when one of the components is arranged at a second position based on an image of a characteristic portion taken when one of the components is arranged at a first position, sensor parameters, and a movement amount based on an actual position, and a display control unit 63 for displaying the image of the characteristic portion estimated by the image estimation unit superimposed on the image of the characteristic portion taken when one of the components is arranged at the second position.

(Supplementary Note 8) The parameter correction device of Supplementary Note 7, wherein the sensor parameter includes a parameter associated with the deformation of the image, and the image estimation unit 61 estimates the image including the characteristic portion of the deformation of the image.

(Supplementary Note 9) A parameter correction device as in any one of Supplementary Notes 1 to 4, comprising: an image estimation unit 61 for estimating an image of a characteristic portion when one of the components is arranged at a second position based on an image of a characteristic portion photographed when one of the components is arranged at a first position, sensor parameters, and a movement amount based on a measured position; and a score calculation unit 62 for calculating a point indicating a degree of overlap of the image of the characteristic portion estimated by the image estimation unit 61 with respect to the image of the characteristic portion photographed when one of the components is arranged at the second position.

(Note 10) As in the parameter correction device of Note 4, the moving device is a robot 1, one of the components is a sensor 6, the other component is a workpiece 31, the sensor 6 is fixed to the robot 1 or a work tool 5 mounted on the robot 1, the position and posture of the sensor 6 are changed by changing the position and posture of the robot 1, the measured position calculation unit 52 calculates the measured position of the sensor 6 based on the position and posture of the robot 1.

(Note 11) As in the parameter correction device of Note 4, the moving device is a robot 1, one of the components is a workpiece 31, the other component is a sensor 6, the workpiece 31 is held by a work tool 5 mounted on the robot 1, the position and posture of the workpiece 31 are changed by changing the position and posture of the robot 1, the measured position calculation unit 52 calculates the measured position of the workpiece 31 based on the position and posture of the robot 1.

(Note 12) A parameter correction device as in any one of Notes 1 to 4, wherein the sensor is a camera 6 for taking two-dimensional images, and the sensor parameters are camera parameters including external parameters and internal parameters.

1: Robot, moving device 2: Control device, correction device 3: Robot device, first robot device 4: Second robot device 5: Hand, work tool 6: Camera, sensor 6a: Field of view 11: Upper arm 12: Lower arm 13: Rotating base 14: Base 15: Wrist 16: Flange 18: Joint 21: Robot drive device 22: Hand drive device 23: Position detector 26: Teaching operation panel 27: Input unit 28: Display unit 31: Workpiece 31a, 31b: Plate-shaped part 31c: Edge, feature part 31g,81,82,83,84,85a,85b,86a,86b,87a,87b,88a,88b,89,90,91: Image 31m: Marker 31n: Line segment 31p: Specific point 31r,31s: Estimated image 32: Calibration plate 35,36: Stand 40: Control device body 42: Memory unit 43: Motion control unit 44: Robot drive unit 45: Hand drive unit 51: Image processing unit 52: Measured position calculation unit 53: Image control unit 56: Feature detection unit 57: Estimated position calculation unit 58: Evaluation unit 59: Selection unit 60: Correction unit 61: Image estimation unit 62: Score calculation unit 63: Display control unit 64: Action command generation unit 67: Action program 68: Camera parameters 71: Reference coordinate system 72: Camera coordinate system 73: Image coordinate system 74: Tool coordinate system 98,99: Arrow 101,102,103,104,105,106,107,108,109,110: Step hc,hz: Distance P6a,P6b,P31a,P31b: Position U,V,X,Y,Z: Axis Δh: Distance difference

FIG. 1 is a schematic diagram of the first robot device of the embodiment. FIG. 2 is a block diagram of the robot device of the embodiment. FIG. 3 is a three-dimensional diagram of the workpiece of the embodiment. FIG. 4 is a schematic diagram illustrating the distance from the camera to the characteristic portion of the workpiece. FIG. 5 is a flow chart of the control for evaluating and correcting camera parameters. FIG. 6 is an image taken when the camera is arranged at the first position. FIG. 7 is a schematic diagram of the first robot device when the camera is arranged at the second position. FIG. 8 is an image taken when the camera is arranged at the second position. FIG. 9 is a graph showing the relationship between the movement amount of the measured position in the X direction and the error. FIG. 10 is a graph showing the relationship between the amount of movement of the measured position in the Y direction and the error. FIG. 11 is a graph showing the error in the X direction and the error in the Y direction of the amount of movement of the measured position in the X direction when there is an error in the distance from the camera to the workpiece. FIG. 12 is a graph showing the error in the Y direction and the error in the X direction of the amount of movement of the measured position in the Y direction when there is an error in the distance from the camera to the workpiece. FIG. 13 is the error in the X direction and the error in the Y direction of the amount of movement of the measured position in the X direction when the camera coordinate system is incorrectly set. FIG. 14 shows the error in the Y direction and the error in the X direction of the movement amount relative to the measured position in the Y direction when the camera coordinate system is incorrectly set. FIG. 15 shows an image in which an estimated image estimated without calculating the deformation of the image is superimposed on the image of the actual workpiece and displayed. FIG. 16 shows an image in which an estimated image estimated by calculating the deformation of the image is superimposed on the image of the actual workpiece and displayed. FIG. 17 shows an image showing the overlapping portion of the image actually captured and the estimated image estimated by the image estimation unit. FIG. 18 is a schematic diagram of the second robot device of the embodiment.

1:Robots and mobile devices

2: Control device, correction device

3: Robot device, first robot device

5: Hands, work tools

6: Camera, sensor

21:Robot drive device

22: Hand drive device

23: Position detector

26: Teaching operation panel

27: Input Department

28: Display unit

40: Control device body

42: Memory Department

43: Motion control unit

44:Robot drive unit

45: Hand drive unit

51: Image processing department

52: Measured position calculation unit

53: Camera control unit

56: Feature Detection Department

57: Estimated position calculation unit

58: Evaluation Department

59:Selection Department

60: Correction Department

61: Image estimation unit

62: Score calculation part

63: Display control unit

64: Action command generation unit

67:Action Program

68: Camera parameters

Claims (12)

A parameter correction device is a parameter correction device for correcting sensor parameters used to calculate a three-dimensional position, wherein the three-dimensional position corresponds to a specific position in an image of a workpiece taken by a sensor, and the parameter correction device comprises: a feature detection unit for detecting a feature portion in the image of the workpiece; an estimated position calculation unit for calculating the three-dimensional position of the workpiece, i.e., the estimated position, based on the position of the feature portion in the image of the workpiece and the sensor parameters; a measured position calculation unit for calculating the three-dimensional position of one of the components, i.e., the measured position, based on the driving state of a moving device of a moving sensor and at least one of the components in the workpiece; and a correction unit for correcting the sensor parameters, The feature detection unit detects a feature portion in an image taken when one of the components is arranged at the first position, and a feature portion in an image taken when one of the components is arranged at the second position. The estimated position calculation unit calculates a first estimated position when one of the components is arranged at the first position, and a second estimated position when one of the components is arranged at the second position. The measured position calculation unit calculates a first measured position when one of the components is arranged at the first position, and a second measured position when one of the components is arranged at the second position. The correction unit corrects the sensor parameter based on the movement amount based on the first measured position and the second measured position, and the distance from the first estimated position to the second estimated position. A parameter correction device as claimed in claim 1, wherein the correction unit calculates the error between the movement amount based on the first measured position and the second measured position and the distance from the first estimated position to the second estimated position, and corrects the sensor parameters to minimize the error. The parameter correction device of claim 1 comprises a selection unit, wherein the selection unit selects a parameter to be corrected from a plurality of parameters included in the sensor parameter, and the correction unit corrects the parameter selected by the selection unit by changing at least one of the first position and the second position and repeatedly photographing the image a number of times corresponding to the number of parameters selected by the selection unit. In the parameter correction device of claim 1, the positions of the sensor and another component in the workpiece are fixed, and the correction unit corrects the sensor parameters according to the displacement between the first measured position and the second measured position. A parameter correction device as claimed in any one of claims 1 to 4, comprising an evaluation unit, wherein the evaluation unit evaluates sensor parameters. The evaluation unit evaluates the sensor parameters based on the movement amount based on the first measured position and the second measured position and the error of the distance from the first estimated position to the second estimated position. A parameter correction device as claimed in any one of claims 1 to 4, comprising a display control unit, the display control unit controlling an image displayed on the display unit, the correction unit calculating an error between a movement amount based on a first measured position and a second measured position and a distance from a first estimated position to a second estimated position, and the display control unit displaying the relationship between the movement amount based on a first measured position and a second measured position and the error on the display unit. A parameter correction device as claimed in any one of claims 1 to 4, comprising: a display control unit for controlling an image displayed on the display unit; and an image estimation unit for estimating an image of a characteristic portion when one of the components is arranged at a second position based on an image of a characteristic portion taken when one of the components is arranged at a first position, sensor parameters, and a movement amount based on an actual position, and the display control unit superimposes the image of the characteristic portion estimated by the image estimation unit on the image of the characteristic portion taken when one of the components is arranged at the second position for display. As in claim 7, the parameter correction device, wherein the sensor parameters include parameters associated with the deformation of the image, and the image estimation unit estimates the image including the characteristic portion of the deformation of the image. A parameter correction device as claimed in any one of claims 1 to 4, comprising: an image estimation unit for estimating an image of a characteristic portion when one of the components is arranged at a second position based on an image of a characteristic portion photographed when one of the components is arranged at a first position, sensor parameters, and a movement amount based on a measured position; and a score calculation unit for calculating a score indicating a degree of overlap of the image of the characteristic portion estimated by the image estimation unit with the image of the characteristic portion photographed when one of the components is arranged at the second position. The parameter correction device of claim 4, wherein the moving device is a robot, one of the components is a sensor, the other component is a workpiece, the sensor is fixed to the robot or a work tool mounted on the robot, the position and posture of the sensor are changed by the robot changing its position and posture, the measured position calculation unit calculates the measured position of the sensor based on the position and posture of the robot. The parameter correction device of claim 4, wherein the moving device is a robot, one of the components is a workpiece, the other component is a sensor, the workpiece is held by a work tool mounted on the robot, the position and posture of the workpiece are changed by changing the position and posture of the robot, the measured position calculation unit calculates the measured position of the workpiece based on the position and posture of the robot. A parameter correction device as claimed in any one of claim items 1 to 4, wherein the sensor is a camera for taking two-dimensional images, and the sensor parameters are camera parameters including external parameters and internal parameters.