JP7509535B2 - IMAGE PROCESSING APPARATUS, ROBOT SYSTEM, AND IMAGE PROCESSING METHOD - Google Patents

Description

The present invention relates to an image processing device, a robot system, and an image processing method.

There is a known technology for extracting a subject at a specific position from image data captured at multiple imaging positions (for example, see Patent Document 1).

Japanese Patent Application Laid-Open No. 08-210847

In the field of industrial machinery, there has been a demand for technology that can accurately detect the target part to be worked on in image data captured by a camera in a work cell containing various parts.

In one aspect of the present disclosure, the image processing device includes an image acquisition unit that acquires image data of the detection target portion captured by cameras arranged at a plurality of predetermined imaging positions, a projection image generation unit that generates projection image data by projecting the first image data captured at the first imaging position and the second image data captured at the second imaging position onto a projection surface set at the position of the detection target portion, and an overlap detection unit that detects overlapping projection image data in which the detection target portion captured in the first image data and the detection target portion captured in the second image data are projected in an overlapping manner in the projection image data generated by the projection image generation unit.

In another aspect of the present disclosure, an image processing method acquires image data of a detection target portion captured by cameras arranged at a plurality of predetermined imaging positions, generates projection image data by projecting first image data captured at the first imaging position and second image data captured at the second imaging position onto a projection surface set at the position of the detection target portion, and detects overlapping projection image data in which the detection target portion captured in the first image data and the detection target portion captured in the second image data are projected in an overlapping manner in the generated projection image data.

According to the present disclosure, even if an object other than the detection target is present in the work cell, it is possible to prevent erroneous detection of the object and reliably and automatically detect overlapping projection image data showing the detection target.

FIG. 2 is a block diagram of an image processing device, a camera, and a position measuring device according to an embodiment. A state in which multiple cameras are arranged at image capture positions is shown. 2 is a flowchart showing an example of an operation flow of the image processing apparatus shown in FIG. 1 . 13 is an example of an image of image data captured by a camera placed at a first imaging position. 13 is an example of an image of image data captured by a camera placed at a second imaging position. 1 shows an example of an image of the projection image data. 13 shows an example of an image of the second projection image data. 13 is another example of an image of image data captured by the camera placed at the first imaging position. 13 is another example of an image of image data captured by the camera placed at the second imaging position. 13 is yet another example of an image of image data captured by the camera placed at the first imaging position. 13 is yet another example of an image of image data captured by the camera placed at the second imaging position. 1 shows an example of an image of the projection image data. 13 shows overlapping projection image data detected from the projection image data shown in FIG. 12 . FIG. 11 is a block diagram of an image processing device, a camera, and a position measuring device according to another embodiment. FIG. 1 is a block diagram of a robot system according to an embodiment. FIG. 16 is a diagram of the robot system shown in FIG. 15 . FIG. 13 is a diagram of a robot system according to another embodiment.

Embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that in the various embodiments described below, similar elements will be given the same reference numerals and duplicated descriptions will be omitted. First, an image processing device 10 according to one embodiment will be described with reference to FIG. 1. The image processing device 10 is a computer having a processor 14, a memory 16, and an I/O interface 18.

The processor 14 has a CPU or a GPU, etc., and is communicatively connected to the memory 16 and the I/O interface 18 via a bus 20. The processor 14 performs arithmetic processing to execute the functions of the image processing device 10, which will be described later, while communicating with the memory 16 and the I/O interface 18. The memory 16 has a ROM or a RAM, etc., and temporarily or permanently stores various data. The I/O interface 18 has an Ethernet (registered trademark) port, a USB port, an HDMI (registered trademark) terminal, etc., and communicates data with external devices under the command of the processor 14.

In this embodiment, a plurality of cameras 22 and 24 and a position measuring device 26 are connected to the I/O interface 18 of the image processing device 10 so as to be able to communicate wirelessly or via wires. As shown in FIG. 2, the cameras 22 and 24 have the same configuration and include an optical lens 28 (such as a focus lens) and an image sensor 30 (such as a CCD or CMOS).

A camera coordinate system C 1 _C1 is set for the camera 22. The camera coordinate system C _{1 C1} is a coordinate system that defines the coordinates of each pixel of image data captured by the camera 22. The camera coordinate system C 1 _C1 is set for the camera 22 so that its xy plane is perpendicular to the line of sight of the camera 22. The camera 22 is disposed at a predetermined first imaging position P _1. The camera 22 captures an image of the workpiece W while disposed at the first imaging position P ₁ .

On the other hand, a camera coordinate system C 1 _C2 is set for the camera 24. The camera coordinate system C 1 _C2 is a coordinate system that defines the coordinates of each pixel of image data captured by the camera 24. The camera coordinate system C 1 _C2 is set for the camera 24 so that its xy plane is perpendicular to the line of sight of the camera 24.

The camera 24 is disposed at a predetermined second imaging position _P2 . The camera 24 images the workpiece W while disposed at the second imaging position _P2 . In this document, the term "imaging position" may refer to the positions of the cameras 22 and 24 in three-dimensional space and the attitudes of the cameras 22 and 24 (i.e., the line of sight).

Therefore, the first imaging position _P1 can be represented by the origin position and each axis direction of the camera coordinate system _C1 of the camera 22 arranged at the first imaging position _P1 . Similarly, the second imaging position _P2 can be represented by the origin position and each axis direction of the camera coordinate system _C2 of the camera 24 arranged at the second imaging position _P2 .

In the following description, the first imaging position _P1 may refer to the origin position and the direction of each axis of the camera coordinate system _{C_C1} of the camera 22 arranged at the first imaging position _P1 , and the second imaging position _P2 may refer to the origin position and the direction of each axis of the camera coordinate system _{C_C2} of the camera 24 arranged at the second imaging position _P2 .

The workpiece W is placed at a predetermined position in the work cell. In this embodiment, the workpiece W is cylindrical and has a surface W _S (detection target portion) at its apex. In this embodiment, the surface W _S is a plane having a circular contour (or edge) E. Meanwhile, a reference coordinate system C _R is set in the three-dimensional space of the work cell. The reference coordinate system C _R is a fixed coordinate system fixed within the three-dimensional space.

The reference coordinate system C _R is, for example, a world coordinate system that defines the three-dimensional space of the work cell, a work coordinate system that is set for the workpiece W, or a robot coordinate system that is set for the robot described below. In this embodiment, the workpiece W is fixed so that its surface W _S is approximately parallel to the xy plane of the reference coordinate system C _R. Meanwhile, a non-detection target portion NW having a shape similar to the surface W _S is present at a position separate from the workpiece W. This non-detection target portion NW is, for example, a pattern formed on an object in the work cell.

The positional relationship between the first imaging position _P1 and the reference coordinate system C _R is known by calibration. Therefore, the first imaging position _P1 (i.e., the origin position and the directions of each axis of the camera coordinate system C _C1 in FIG. 2) is expressed as coordinates and vectors (or functions) of the reference coordinate system C _R. The coordinates of the camera coordinate system C _C1 and the coordinates of the reference coordinate system C _R can be mutually converted via first coordinate conversion data. This first coordinate conversion data is, for example, a Jacobian matrix, and can be acquired by calibration between the first imaging position _P1 and the reference coordinate system C _R.

Similarly, the positional relationship between the second imaging position _P2 and the reference coordinate system C _R is known by calibration. Therefore, the coordinates of the camera coordinate system C _C2 and the coordinates of the reference coordinate system C _R can be mutually converted via second coordinate conversion data. This second coordinate conversion data is, for example, a Jacobian matrix, and can be acquired by calibration between the second imaging position _P2 and the reference coordinate system C _R.

When the camera 22 arranged at the first imaging position _P1 images the workpiece W, light (i.e., subject image) _A1 from the workpiece W is guided by the optical lens 28 of the camera 22 and imaged on the imaging sensor 30. In addition, light (subject image) _A2 from the non-detection target portion NW is also guided by the optical lens 28 of the camera 22 and imaged on the imaging sensor 30.

The image sensor 30 photoelectrically converts the received subject images _A1 and _A2 to generate image data _ID1 . In this manner, the camera 22 captures the image data _ID1 of the workpiece W. An example of an image of the image data _ID1 is shown in Fig. 4. The image data _ID1 shows the workpiece W such that the contour E of the surface _WS is visible along with the non-detection target portion NW. Each pixel of the image data _ID1 is represented as a coordinate in the camera coordinate system C _C1 .

Similarly, when the camera 24 arranged at the second imaging position _P2 images the workpiece W, light (subject image) _A3 from the workpiece W is guided by the optical lens 28 of the camera 24 and is imaged on the imaging sensor 30. In addition, light (subject image) _A4 from the non-detection target portion NW is also guided by the optical lens 28 of the camera 24 and is imaged on the imaging sensor 30.

The image sensor 30 photoelectrically converts the received subject images _A3 and _A4 to generate image data _ID2 . In this way, the camera 24 captures the image data _ID2 of the workpiece W. An example of an image of the image data _ID2 is shown in Fig. 5. In the image data _ID2 , the workpiece W is captured at a position different from that of the image data _ID1 so that the contour E of the surface _WS can be seen together with the non-detection target portion NW. Each pixel of the image data _ID2 is expressed as a coordinate in the camera coordinate system C _C2 . Note that the subject images _A1 and _A3 in Fig. 2 each indicate the optical path of the surface _WS of the workpiece W (specifically, the contour E).

The position measuring device 26 measures the position of the surface W _S. For example, the position measuring device 26 has a laser emitting unit that emits a laser beam to an object and a light receiving unit that receives the laser beam reflected by the object (both not shown), and measures the distance to the object by a triangulation method. Alternatively, the position measuring device 26 may have two cameras, and measure the distance to the object from images captured by the two cameras.

In this embodiment, the position measuring device 26 is provided integrally with the camera 22. That is, the camera 22 and the position measuring device 26 constitute a three-dimensional visual sensor that captures an image of an object and measures the distance to the object. The position measuring device 26, while being disposed at the first imaging position _P1 together with the camera 22, measures the distance _d1 from the first imaging position _P1 to the surface _WS of the workpiece W.

On the other hand, the distance _d0 between the first imaging position _P1 and the origin of the reference coordinate system C _R is known. Therefore, the position of the surface W _S in the z-axis direction of the reference coordinate system C _R can be obtained from the known distance _d0 and the distance _d1 measured by the position measuring device 26. In the example shown in Fig. 2, the surface W _S is located at the z-axis coordinate z = _z0 in the reference coordinate system C _R.

As an example, the processor 14 acquires the distance _d1 measured by the position finder 26 from the position finder 26 via the I/O interface 18, and calculates the z-axis coordinate _z0 of the reference coordinate system C _R of the surface W _S using the distance _d0 previously stored in the memory and the acquired distance _d1 . Alternatively, the position finder 26 may further have a processor for calculation, and calculates the z-axis coordinate _z0 of the surface W _S in the reference coordinate system C _R using the known distance _d0 previously acquired and the measured distance _d1 .

Next, the operation of the image processing device 10 will be described with reference to Fig. 3. The flow shown in Fig. 3 starts when the processor 14 receives a start command from an operator, a higher-level controller, or a computer program. In step S1, the processor 14 acquires image data _ID1 and _ID2 captured by the cameras 22 and 24.

Specifically, the camera 22 captures image data _ID1 while positioned at a first imaging position _P1 , and the camera 24 captures image data _ID2 while positioned at a second imaging position _P2 . The cameras 22 and 24 transmit the captured image data _ID1 and _ID2 to the I/O interface 18. The processor 14 acquires the image data _ID1 and _ID2 through the I/O interface 18 and stores them in the memory 16. Thus, in this embodiment, the processor 14 functions as an image acquisition unit 32 ( FIG. 1 ) that acquires the image data _ID1 and _ID2 .

In step S2, the processor 14 receives position information of the surface W _S as the detection target portion. As an example, the position measuring device 26 transmits the measured distance d ₁ to the I/O interface 18 as the position information of the surface W _S. The I/O interface 18 receives the distance d ₁ , and the processor 14 acquires the distance d ₁ via the I/O interface 18. Then, the processor 14 uses the distances d ₀ and d ₁ to determine the z-axis coordinate z ₀ of the surface W _S in the reference coordinate system _CR , and stores it in the memory 16.

As another example, the position measuring device 26 uses the distances _d0 and _d1 to determine the z-axis coordinate _z0 of the surface W _S in the reference coordinate system C _R , and transmits the coordinate _z0 of the surface W _S to the I/O interface 18 as information on the position of the surface W _S. The I/O interface 18 receives the coordinate _z0 , and the processor 14 acquires the coordinate _z0 via the I/O interface 18 and stores it in the memory 16. Thus, in this embodiment, the I/O interface 18 functions as an information receiving unit 34 ( FIG. 1 ) that receives information on the position of the surface W _S.

In step S3, the processor 14 generates the projection image data PD (FIG. 6). Specifically, the processor 14 sets a projection plane PP in the reference coordinate system C _R at the position of the surface W _S acquired in step S2 (i.e., the position of the coordinate _z0 ). As a result, the projection plane PP is set in the reference coordinate system C _R as shown in FIG. 2. This projection plane PP is a virtual plane parallel to the xy plane of the reference coordinate system C _R (i.e., the surface W _S of the workpiece W).

Next, the processor 14 extracts image data ID _{1_E} of the contour E of the surface W _S from the image data ID _1. For example, the processor 14 may obtain drawing data (2D or 3D CAD model) of the workpiece W and extract image data ID _{1_E} of the contour E from the image data ID ₁ by searching the image data ID ₁ for a shape similar to the model shape of the contour E of the surface _{W S.}

At this time, image data _{ID1_N} of the contour of the non-detection target portion NW having a shape similar to the contour E can also be extracted together with the image data _{ID1_E} . Similarly, the processor 14 extracts image data _{ID2_E} of the contour E from the image data _ID2 . At this time, image data _{ID2_N} of the contour of the non-detection target portion NW can also be extracted together with the image data _{ID2_E} .

Next, the processor 14 projects the extracted image data _{ID1_E} and _{ID1_N} and the image data _{ID2_E} and _{ID2_N} onto the set projection plane PP. Specifically, the processor 14 converts the coordinates of each pixel of the image data _{ID1_E} and _{ID1_N} in the camera coordinate system C1 into x-y coordinates at the position of the z-axis coordinate _z0 in the reference coordinate system _C1 , using the first coordinate conversion data indicating the positional relationship between the first imaging position _P1 and the reference coordinate _{system C2R} .

Each point converted into the xy coordinates is an intersection point between the projection plane PP and a line extending from each pixel of the image sensor 30 that receives the subject images _A1 and _A2 along the optical path of the subject images _A1 and _A2 , and the coordinates of these intersection points in the reference coordinate system _CR can be obtained by coordinate transformation using the first coordinate transformation data if the position of the projection plane PP is known. In this way, it is possible to generate contour projection image data _PE1 (FIG. 6) in which image data _{ID1_E} is projected onto the projection plane PP, and contour projection image data _PN1 in which image data _{ID1_N} is projected onto the projection plane PP.

Similarly, the processor 14 projects the image data ID2_E and ID2_N onto the projection plane PP using the second coordinate transformation data indicating the positional relationship between the second imaging position _P2 and the reference coordinate system _CR and the coordinate _z0 of the projection plane PP. As a result, it is possible to generate _contour projection image data _{PE2 by projecting the image data ID2_E onto} the projection plane _PP , and contour projection image data _PN2 by projecting the image data _{ID2_N} onto the projection plane PP.

Next, the processor 14 sets an image frame F that defines the range of the projection image data PD on the projection surface PP, and generates the projection image data PD as shown in FIG. 6. Therefore, in this embodiment, the processor 14 functions as a projection image generator 36 (FIG. 1) that generates the projection image data PD.

The processor 14 may display the projection image data PD of the generated image frame F on a display device. The position and size of the generated projection image data PD on the projection surface PP of the image frame F may be set manually by an operator. In this case, the processor 14 may receive the vertical and horizontal sizes of the image frame F as inputs of the y-axis coordinate range and the x-axis coordinate range of the reference coordinate system _CR from the operator. Alternatively, the processor 14 may automatically set the image frame F to include the contour projection image data _PE1 , _PE2 , _PN1 , and _PN2 .

Each pixel of the projection image data PD is expressed as a coordinate by the projection coordinate system _CP . The projection coordinate system _CP is a coordinate system whose origin is located at the z-axis coordinate _z0 of the reference coordinate system _CR , and whose x-axis and y-axis are parallel to the projection surface PP. The projection coordinate system _CP and the reference coordinate system _CR have a known positional relationship according to the image frame F that is set. In this embodiment, the image frame F is set so that the x-axis and y-axis of the projection coordinate system _CP are parallel to the x-axis and y-axis of the reference coordinate system _CR , respectively.

The sizes in the x-axis and y-axis directions in the projection coordinate system C _P and the sizes in the x-axis and y-axis directions in the reference coordinate system C _R correspond to each other according to the size of the set image frame F (for example, a distance of "1" in the x-axis direction in the projection coordinate system C _P corresponds to a distance of "10" in the x-axis direction in the reference coordinate system C _R ).

In this way, the projected coordinate system C _{1 P} and the reference coordinate system C 1 _R have a known positional relationship, and the origin position and the directions of each axis of the projected coordinate system C _{1 P} can be expressed as coordinates and vectors (or functions) of the reference coordinate system C 1 _R. Therefore, the coordinates of the projected coordinate system C _{1 P} and the coordinates of the reference coordinate system C 1 _R can be mutually converted via the third coordinate conversion data.

In step S4, the processor 14 detects overlapped projection image data OD in which image data ID _{1_E} reflected in image data ID ₁ and image data ID _{2_E} reflected in image data ID ₂ are projected in an overlapping manner in the projection image data PD. Here, the subject image _A1 of the surface _WS incident on the camera 22 and the subject image _A3 of the surface _WS incident on the camera 24 coincide with each other at the position of the surface _WS .

In other words, the projection position where the image data _{ID1_E} of the contour E of the surface _WS captured by the camera 22 is projected onto the projection plane PP set at the position of the surface _WS coincides with the projection position where the image data _{ID2_E} of the contour E of the surface _WS captured by the camera 24 is projected onto the projection plane PP set at the position of the surface _WS . At this coincident projection position, the image data _{ID1_E} and the image data _{ID2_E} are projected in an overlapping manner to form overlapping projection image data OD in the projection image data PD.

On the other hand, the projection position where the image data _{ID1_N} of the contour of the non-detection target portion NW captured by the camera 22 is projected onto the projection surface PP and the projection position where the image data _{ID2_N} of the contour of the non-detection target portion NW captured by the camera 24 is projected onto the projection surface PP are shifted from each other (contour projection image data _PN1 , _PN2 ). The processor 14 detects the overlapping projection image data OD from the projection image data PD.

As an example, the processor 14 counts the brightness of each pixel of the projection image data PD as "1" if it represents the contour projection image data _PE1 or _PN1 projected from the image data _ID1 , and counts it as "0" if it represents neither the contour projection image data _PE1 nor _PN1 .

In addition, the processor 14 counts the brightness of each pixel of the projection image data PD as "1" if it represents the contour projection image data _PE2 or _PN2 projected from the image data _ID2 , and counts it as "0" if it represents neither the contour projection image data _PE2 nor _PN2 .

Then, the processor 14 adds up the counted brightness for each pixel of the projection image data PD. In this case, the brightness of a pixel that includes both the contour projection image data _PE1 or _PN1 projected from the image data _ID1 and the contour projection image data _PE2 or _PN2 projected from the image data ID2 (i.e., a pixel onto which the projection image data _PE1 or _PN1 and the projection image data _PE2 or _PN2 are overlappingly projected) becomes " ₂ ".

On the other hand, the brightness of a pixel that represents either the contour projection image data _PE1 or _PN1 projected from image data _ID1 , or the contour projection image data _PE2 or _PN2 projected from image data _ID2 , is "1", and the brightness of a pixel that represents none of the contour projection image data _PE1 , _PN1 , _PE2 , or _PN2 is "0".

The processor 14 extracts pixels with a brightness of "2" in the projection image data PD, and detects the image data represented by the extracted pixels as overlapped projection image data OD. Thus, in this example, the processor 14 detects the overlapped projection image data OD based on the brightness of the pixels in the projection image data PD. Note that the brightness of pixels representing or not representing the contour projection image data _PE1 , _PN1 , _PE2 , _PN2 is not limited to the binary values of "0" and "1", and may be set to any other value.

As another example, the processor 14 extracts the shape of each of the contour projection image data _PE1 , _PN1 , _PE2 and _PN2 from the projection image data PD and identifies corresponding feature points (center points, vertices, edges, etc.) of the contour projection image data _PE1 , _PN1 , _PE2 and _PN2 . In this embodiment, the processor 14 identifies the center point _CE1 of the contour projection image data _PE1 , the center point _CE2 of the contour projection image data _PE2 , the center point _CN1 of the contour projection image data _PN1 and the center point _CN2 of the contour projection image data _PN2 as the feature points.

Then, the processor 14 acquires the coordinates of the identified feature points (center points) _CE1 , _CE2 , _CN1 and _CN2 in the projection coordinate system _CP , and calculates the distance d between the feature points of two of the plurality of contour projection image data _PE1 , _PN1 , _PE2 and _PN2 . If the distance d is smaller than a predetermined threshold _dth , the processor 14 detects one of the two contour projection image data as overlapping projection image data OD.

6, the distance dE between the feature points CE1 and CE2 is smaller than the distance _dN between the feature points _CN1 and _CN2 (specifically, _dE ≈ 0). Therefore, when the distance d is smaller than the threshold _{value dth} , it can be considered that the positions of the two contour projection image data _PE1 and _PE2 match, and _{that the image data ID1_E and} the image data _{ID2_E} are projected in an overlapping manner at that position.

In the example shown in Fig. 6, the processor 14 determines that the distance _dE between the feature points _CE1 and _CE2 is smaller than the threshold _{value dth} , and detects one of the contour projection image data _PE1 and _PE2 as overlapping projection image data OD. In the above-mentioned manner, the processor 14 can detect the overlapping projection image data OD. Therefore, in this embodiment, the processor 14 functions as the overlap detection unit 38 (Fig. 1) that detects the overlapping projection image data OD.

In step S5, the processor 14 obtains position data of the surface _WS in the reference coordinate system _CR based on the overlapping projection image data OD detected in step S4. Specifically, the processor 14 performs image processing on the projection image data PD to extract the shape of the overlapping projection image data OD. At this time, the processor 14 may execute image filtering processing to smoothly connect the contour lines of the images of the overlapping projection image data OD, and update the overlapping projection image data OD.

Then, the processor 14 acquires the position of the projection coordinate system _CP of the overlapped projection image data OD. For example, the processor 14 identifies a feature point (center point CO) of the overlapped projection image data OD by image processing as the position of the projection coordinate system _CP of the overlapped projection image data OD, and acquires the coordinates ( _xP , _{yP )} of the feature point CO in the projection coordinate system CP. Note that this feature point CO substantially coincides with the above-mentioned feature point _CE1 or _CE2 .

Then, the processor 14 converts the coordinates ( _xP , _yP ) in the projection coordinate system _CP into coordinates ( _xR , _yR , _z0 ) in the reference coordinate system _CR using the coordinate _z0 of the projection plane PP and the third coordinate conversion data. These coordinates ( _xR , _yR , _z0 ) become position data indicating the position of the feature point (center point) of the surface _WS in the reference coordinate system _CR . In this manner, in this embodiment, the processor 14 functions as a position data acquisition unit 40 (FIG. 1) that acquires position data: coordinates ( _xR , _yR , _z0 ) of the surface _WS in the reference coordinate system _CR based on the overlapping projection image data OD.

As described above, in this embodiment, the overlapping projection image data OD is automatically detected in the projection image data PD. With this configuration, even if a non-detection target portion NW having a shape similar to the surface _WS as the detection target portion is present in the work cell, the non-detection target portion NW can be prevented from being erroneously detected, and the overlapping projection image data OD showing the surface _WS as the detection target portion can be reliably and automatically detected.

In this embodiment, the position measuring device 26 measures the position of the surface W _S , and the information on this position is received by the information receiving unit 34. With this configuration, the processor 14 can automatically acquire the position of the surface W _S. In this embodiment, the processor 14 acquires position data of the surface W _S in the reference coordinate system C _R : coordinates (x _R , y _R , z ₀ ). With this configuration, position data used to control the robot that performs work on the workpiece W can be automatically acquired. The control of the robot will be described later.

In addition, in the above-mentioned step S4, when the processor 14 detects the overlapped projection image data OD, the processor 14 may function as the projection image generating unit 36 and further generate second projection image data PD2 in which only the overlapped projection image data OD is shown. An example of this second projection image data PD2 is shown in FIG. 7.

As shown in FIG. 7, the second projection image data PD2 contains the overlapping projection image data OD detected in step S4, while images other than the overlapping projection image data OD, including the contour projection image data _PN1 and _PN2 that were contained in the projection image data PD in FIG. 6, are shown with a single brightness (or are erased) so as to be visually indistinguishable.

For example, the processor 14 generates the second projection image data PD2 shown in FIG. 7 by modifying the projection image data PD so that the brightness of pixels other than the pixels that represent the detected overlapped projection image data OD is a single brightness (e.g., "0") different from the brightness of the pixels that represent the overlapped projection image data OD (e.g., "2").

Alternatively, the processor 14 may generate the second projection image data PD2 as new image data by storing the coordinates of the detected overlapped projection image data OD in the projection coordinate system C _P and plotting the coordinates on new image data in which the projection coordinate system C _P is set. In the second projection image data PD2 generated in this way, only the overlapped projection image data OD is clearly depicted.

Then, in step S5, the processor 14 performs image processing on the second projection image data PD2 to extract the shape of the overlapping projection image data OD, identifies the feature point CO, and obtains the coordinates ( _xP , _yP ) of the feature point CO in the projection coordinate system _CP . With this configuration, the processor 14 can more effectively execute image processing to extract the overlapping projection image data OD and the feature point CO.

Furthermore, according to the second projection image data PD2, even if noise or the like is present in the image data ID ₁ captured by the camera 22 or the image data ID ₂ captured by the camera 24, the overlapping projection image data OD and its feature points CO can be accurately extracted. The effect of this will be described with reference to Figs. 8 and 9.

8 and 9, noise H appears in the image data _ID1 and _ID2 captured in step S1. Such noise H may be due to, for example, the direction and illuminance of light irradiated onto the subject during image capture, electrical noise generated during photoelectric conversion by the image sensor 30, or reflected light from foreign matter such as chips.

When the processor 14 executes steps S2 to S4 using such image data _ID1 and _ID2 to generate the second projection image data PD2 in step S4, images other than the overlapped projection image data OD are visually erased in the second projection image data PD2, and therefore images of noise H are also visually erased. Therefore, even if noise H is captured in the image data _ID1 and _ID2 , the overlapped projection image data OD and its feature points CO can be accurately extracted from the second projection image data PD2.

Next, other functions of the image processing device 10 will be described with reference to Fig. 3 and Fig. 10 to Fig. 12. In this embodiment, the workpiece W has a substantially rectangular surface W _S , and the surface W _S is located at the z-axis coordinate ₀ in the reference coordinate system C _R , as in the above-mentioned embodiment. Here, as shown in Fig. 10, there are cases where a part of the surface W _S of the workpiece W is not captured in the image data ID ₁ captured by the camera 22 in step S1. This phenomenon can occur when there is an obstacle between the camera 22 and the workpiece W, or due to the above-mentioned noise, etc.

In the image data _ID1 , image data _{ID1_E1} of a portion E1 indicated by a solid line in Fig. 10 of the edge E of the surface _WS is captured, but a portion E2 indicated by a dotted line is not captured. Similarly, as shown in Fig. 11, in the image data _ID2 captured by the camera 24 in step S1, image data _{ID1_E3} of a portion E3 indicated by a solid line of the edge E of the surface _WS is captured, but a portion E4 indicated by a dotted line is not captured.

3 is executed using such image data _ID1 and _ID2 , in step S3, the processor 14 projects the image data _{ID1_E1} reflected in the image data _ID1 and the image data _{ID2_E3 reflected in the image data ID2 onto the projection surface PP to generate the projection image data PD shown in Fig. 12. In the projection image data PD shown in Fig. 12, the image data ID1_E1 is projected as contour projection image data PE-E1 and the image data ID2_E3 is projected as contour} projection image data PE _-E2 , while the image data indicated by the dotted line G which corresponds to a part of the contour E is missing.

In step S4, the processor 14 detects overlapped projection image data OD in which image data ID _{1_E1} reflected in image data ID ₁ and image data ID _{2_E3} reflected in image data ID ₂ are projected in an overlapping manner in the projection image data PD. Specifically, the processor 14 counts the brightness of each pixel in the projection image data PD as "1" if the contour projection image data _{PE_E1} is reflected, and as "0" if the contour projection image data _{PE_E1} is not reflected.

The processor 14 also counts the brightness of each pixel of the projection image data PD as "1" if the contour projection image data PE- _E3 is captured, and as "0" if the contour projection image data PE- _E3 is not captured. The processor 14 then adds up the counted brightness for each pixel of the projection image data PD.

In this case, the brightness of a pixel that represents both the contour projection image data PE _-E1 and PE- _E3 (i.e., a pixel onto which the image data _{ID1_E1} and _{ID2_E3} are overlappingly projected) is "2." On the other hand, the brightness of a pixel that represents either the contour projection image data PE _-E1 or PE- _E3 is "1," and the brightness of a pixel that represents neither the contour projection image data PE _-E1 nor PE- _E3 is "0."

Then, the processor 14 extracts pixels in the projection image data PD that have a brightness of "2" and detects the image data represented by the extracted pixels as overlapping projection image data OD. As a result, the image data OD shown in FIG. 13 is detected. In this image data OD, the image data indicated by the dotted line J, which corresponds to part of the contour E, is missing.

In step S5, the processor 14 acquires position data of the surface _WS in the reference coordinate system _CR based on the overlapping projection image data OD shown in Fig. 13 detected in step S4. Specifically, the processor 14 performs image processing on the projection image data PD shown in Fig. 13 to extract the shape of the overlapping projection image data OD.

At this time, the processor 14 may obtain a model shape of the contour E from the drawing data of the workpiece W, compare the model shape with the overlapping projection image data OD, and extract the shape of the overlapping projection image data OD corresponding to the entire circumference of the contour E, for example, by estimating and completing image data corresponding to the missing portion J.

The processor 14 then uses image processing to identify feature points (center points, vertices, etc.) of the overlapped projection image data OD as positions in the projection coordinate system _CP of the overlapped projection image data OD, and obtains the coordinates of the feature points in the projection coordinate system _CP . The processor 14 then converts the coordinates in the projection coordinate system _CP into coordinates in the reference coordinate system _CR , and obtains them as position data indicating the positions of the feature points of the surface _WS in the reference coordinate system _CR .

In this way, according to the image processing device 10, even if part of the surface _WS of the workpiece W is not captured in the image data _ID1 or _ID2 captured by the camera 22 or 24 in step S1, the overlapping projection image data OD and its feature points can be extracted, for example, based on a comparison between the drawing data of the workpiece W and the overlapping projection image data OD in which part of the image data is missing.

Next, referring to FIG. 14, further functions of the image processing device 10 will be described. In this embodiment, the above-mentioned position measuring device 26 is not connected to the I/O interface 18 of the image processing device 10, while the display device 41 and the input device 42 are connected to be able to communicate with each other via wire or wirelessly.

The display device 41 has an LCD or an organic EL display, and the processor 14 transmits image data to the display device 41 via the I/O interface 18, causing the display device 41 to display an image. The input device 42 has a keyboard, mouse, touch sensor, or the like, and transmits information input by an operator to the processor 14 via the I/O interface 18. The display device 41 and the input device 42 may be provided integrally with the image processing device 10, or may be provided separately from the image processing device 10.

Next, the operation of the image processing device 10 according to this embodiment will be described with reference to Fig. 3. The operation of this embodiment differs from the embodiment shown in Fig. 1 in step S2. Specifically, in step S2, the processor 14 receives information on the position of the surface W _S. For example, the processor 14 generates input image data for inputting the z-axis coordinate of the reference coordinate system C _R of the surface W _S , and displays it on the display device 41. At this time, the processor 14 may display known nominal dimensions or drawing data of the workpiece W on the display device 41 together with the input image.

The operator operates the input device 42 while visually checking the input image displayed on the display device 41, taking into consideration the nominal dimensions or drawing data of the workpiece W, and inputs the z-axis coordinate z = _z0 of the reference coordinate system _CR of the surface W _S as information on the position of the surface W _S. The I/O interface 18 functions as an information receiving unit 34, and receives from the input device 42 the information on the position of the surface W _S (coordinate _z0 ) input to the input device 42.

The processor 14 acquires position information (coordinate _z0 ) of the surface W _S through the I/O interface 18 and stores it in the memory 16. Thus, according to this embodiment, the operator can arbitrarily input position information (coordinate _z0 ) of the surface W _S to the input device 42 and arbitrarily set the position of the projection plane PP set in step S3 in consideration of the known nominal dimensions or drawing data of the workpiece W, etc.

Next, a robot system 50 according to one embodiment will be described with reference to Figures 15 and 16. The robot system 50 includes a control device 52, a robot 54, an image processing device 10, cameras 22 and 24, a position measuring device 26, a display device 41, and an input device 42. The control device 52 controls the operations of the robot 54, the cameras 22 and 24, and the position measuring device 26.

In this embodiment, the robot 54 is a vertical articulated robot and has a robot base 56, a rotating body 58, a robot arm 60, a wrist 62, and an end effector 64. The robot base 56 is fixed to the floor of the work cell. The rotating body 58 is mounted on the robot base 56 so as to be rotatable around a vertical axis.

The robot arm 60 has a lower arm 66 rotatably attached to the rotating body 58, and an upper arm 68 rotatably attached to the tip of the lower arm 66. The wrist 62 is connected to the tip of the upper arm 68 and rotatably supports an end effector 64. The end effector 64 is a robot hand, a cutting tool, a laser processing head, a welding torch, a paint applicator, or the like, and performs a predetermined operation (handling, cutting, laser processing, welding, coating, etc.) on the workpiece W.

Each component of the robot 54 (robot base 56, rotating body 58, robot arm 60, wrist 62) has a built-in servo motor. In response to commands from the processor 14, the servo motor operates each movable component of the robot 54 (rotating body 58, robot arm 60, wrist 62) and positions the end effector 64 at any position and orientation within three-dimensional space.

In this embodiment, the image processing device 10 is implemented in a control device 52. That is, the control device 52 is a computer having a processor 14, a memory 16, and an I/O interface 18, and the processor 14, the memory 16, and the I/O interface 18 constitute the image processing device 10.

3 and acquires position data: coordinates ( _xR , _yR , _z0 ) of the surface _WS in step S5 before the robot 54 performs work on the workpiece W. The processor 14 then controls the robot 54 based on the position data: coordinates ( _xR , _yR , _z0 ) and causes the robot 54 to perform work on the workpiece W (e.g., the surface _WS ).

As described above, in this embodiment, the position data: coordinates (xR, _yR, z0) of the surface _WS of the workpiece W is obtained from the image data _ID1 and _ID2 captured by the cameras ₂₂ and 24 at the imaging positions P1 and _P2 , and the robot 54 is controlled based on the position data: coordinates ( _xR , _{yR , z0} ), thereby enabling _work on the workpiece W to be performed accurately.

In the robot system 50, the control device 52 and the image processing device 10 can be configured as separate entities. In this case, the control device 52 is a single computer having a processor and memory, and is communicatively connected to the image processing device 10. Then, when the processor 14 of the image processing device 10 acquires the position data: coordinates ( _xR , _yR , _z0 ) in step S5, the processor 14 may transmit the position data to the control device 52.

The flow shown in FIG. 3 can also be executed using a single camera 22. Such a configuration is shown in FIG. 17. In the robot system 50' shown in FIG. 17, the above-mentioned camera 24 is not provided, and a camera 22 with an integrated position measuring device 26 is installed on the end effector 64 of the robot 54.

In this embodiment, the camera 22 is disposed in a known positional relationship with respect to the hand (or TCP) of the robot 54 by calibration. Therefore, the control device 52 is capable of recognizing the position and orientation of the camera 22 in the reference coordinate system C _R (i.e., the origin position and the directions of each axis of the camera coordinate system C _C1 ).

3 in the robot system 50', the processor 14 operates the robot 54 in step S1 to place the camera 22 at a first imaging position _P1 and capture image data _ID1 with the camera 22. The processor 14 also operates the robot 54 to place the camera 22 at a second imaging position _P2 and capture image data _ID2 with the camera 22. In this way, the flow shown in FIG. 3 can be executed using the image data _ID1 , _ID2 captured by placing one camera 22 at multiple imaging positions _P1 , _P2 .

The position data acquisition unit 40 may be omitted from the above-described image processing device 10. For example, when the overlap detection unit 38 detects overlapped projection image data OD in the projection image data PD, an operator may manually determine the position data: coordinates ( _xR , _yR , _z0 ) of the surface W _S based on the overlapped projection image data OD. Also, the memory 16 or the I/O interface 18 may be omitted from the image processing device 10.

In the above embodiment, the detection target portion is the surface W _S , but the detection target portion may have any structure, such as an edge, a convex portion, or a concave portion formed on the side of the object (workpiece W). The surface W _S is not limited to a circular shape, and may have any shape, such as an ellipse or a polygon. The surface W _S may be inclined with respect to the xy plane of the reference coordinate system C _R. In this case, the processor 14 may set the projection plane PP to be parallel to the surface W _S in the above step S3.

In addition, the first imaging position _P1 and the second imaging position _P1 are not limited to the positions shown in Fig. 2, and may be any positions. In addition, a total of n cameras (n is a positive number of 3 or more) may be arranged at the first imaging position _P1 , the second imaging position _P2 , ..., the nth imaging position _Pn , respectively, to capture image data _ID1 , _ID2 , ..., _IDn , and the flow shown in Fig. 3 may be executed using these image data _ID1 , _ID2 , ..., _IDn . In this case, in step S4, the processor 14 detects overlapped projection image data OD in which the surface _Ws (detection target portion) shown in at least two (for example, all) image data _IDn is overlapped.

Also, in the above embodiment, a case has been described in which the processor 14 sets an image frame F in step S3. However, the processor 14 can generate the projection image data PD as a collection of points of x-y coordinates at the position of the z-axis coordinate _z0 in the reference coordinate system C _R , without dividing the projection image data PD by the image frame F. Although the present disclosure has been described above through the embodiments, the above-mentioned embodiments do not limit the invention according to the claims.

REFERENCE SIGNS LIST 10 Image processing device 14 Processor 22, 24 Camera 26 Position measuring device 32 Image acquisition section 34 Information reception section 36 Projection image generation section 38 Overlap detection section 40 Position data acquisition section 50, 50' Robot system 52 Control device 54 Robot

Claims (9)

an image acquisition unit that acquires image data of a detection target portion captured by cameras arranged at a plurality of predetermined imaging positions;
a projection image generating unit that generates projection image data by projecting a first contour of the detection target portion shown in the first image data captured at the first imaging position and a second contour of the detection target portion shown in the second image data captured at the second imaging position onto a projection surface set at the position of the detection target portion;
An image processing device comprising: an overlap detection unit that detects overlapping projection image data in which the projection image data of the first contour and the projection image data of the second contour are projected in an overlapping manner in the projection image data generated by the projection image generation unit. an image acquisition unit that acquires image data of a detection target portion captured by cameras arranged at a plurality of predetermined imaging positions;
a projection image generating unit that generates projection image data by projecting the first image data captured at the first imaging position and the second image data captured at the second imaging position onto a projection surface set at the position of the detection target portion;
an overlap detection unit that detects overlapped projection image data in which the detection target portion shown in the first image data and the detection target portion shown in the second image data are projected in an overlapping manner in the projection image data generated by the projection image generation unit,
The projection image generating unit further generates second projection image data in which only the overlapped projection image data detected by the overlap detecting unit is captured. an information receiving unit that receives information about the position of the detection target portion,
The image processing device according to claim 1 , wherein the projection image generating unit generates the projection image data by setting the projection plane at the position accepted by the information accepting unit. The information receiving unit is
Receiving the information from an input device capable of inputting the information, or
The image processing apparatus according to claim 3 , wherein the information is received from a position measuring device that measures the position of the detection target portion. The detection target portion is a plane,
5. The image processing device according to claim 1, wherein the projection plane is set parallel to the plane. Each of the imaging positions is determined to have a known positional relationship with respect to a reference coordinate system;
The image processing device according to any one of claims 1 to 5, further comprising a position data acquisition unit that acquires position data of the detection target portion in the reference coordinate system based on the overlap projection image data detected by the overlap detection unit. The image processing device according to claim 6 ;
The camera;
a robot that performs an operation on a workpiece having the detection target portion;
a control device that controls the robot to perform the task based on the position data acquired by the position data acquisition unit. a first camera that captures the first image data at the first imaging position;
The robot system according to claim 7 , further comprising: a second camera configured to capture the second image data at the second imaging position. acquiring image data of the detection target portion captured by cameras arranged at a plurality of predetermined image capturing positions;
generating projection image data by projecting a first contour of the detection target part shown in the first image data captured at the first imaging position and a second contour of the detection target part shown in the second image data captured at the second imaging position onto a projection plane set at the position of the detection target part;
an image processing method for detecting overlapping projection image data in which the projection image data of the first contour and the projection image data of the second contour are projected in an overlapping manner in the generated projection image data;

Images