WO2021012681A1 - Object carrying method applied to carrying robot, and carrying robot thereof - Google Patents

Abstract

Provided are an object carrying method applied to a carrying robot, and a carrying robot thereof. A first camera (8) photographs a top-view image, and the shape of a carried object and the X-axis coordinate and the Y-axis coordinate of the carried object are acquired by means of OpenCV technology; moreover, a second camera (6) photographs a fringe pattern of the carried object, and the Z-axis coordinate of the carried object is measured by means of fringe projection measurement technology, and the X-axis coordinate, the Y-axis coordinate and the Z-axis coordinate are mutually combined to obtain the three-dimensional coordinates of the carried object; and the carrying robot controls a robot arm to grab the carried object according to the three-dimensional coordinates, so that the aim of automatically cognizing the morphological features of the carried object and grabbing the carried object according to the morphological features is achieved. The object carrying method and the carrying robot have high intelligentization, such that multiple carried objects at different positions can be carried, and diversified carrying tasks can be completed.

Description

Target handling method applied to handling robot and handling robot Technical field

The invention relates to the field of intelligent robots, in particular to a target handling method applied to a handling robot and a handling robot thereof.

Background technique

Industrial robots have been used more and more in industrial production, and industrial production is becoming more intelligent. However, most of the industrial robots currently used in the handling process run in accordance with the built-in established procedures, and can only transport a single workpiece placed in a specific fixed position. Once the position of the workpiece changes or the shape of the workpiece changes, the industrial robot will Unable to respond to changes, unable to complete handling tasks.

Summary of the invention

The purpose of the present invention is to solve at least one of the technical problems existing in the prior art, and to provide a target handling method applied to a handling robot and a handling robot thereof, which can cope with diversified handling tasks.

The technical solutions adopted by the present invention to solve its problems are:

In a first aspect of the present invention, there is provided a target handling method applied to a handling robot, the handling robot including a traveling mechanism, a first camera, a second camera, a projector, a controller, and a mechanical arm; the target handling method It includes the following steps:

The handling robot moves to the front of the handling target; the handling robot uses the first camera to photograph the handling target from above to obtain a top view image of the handling target;

The handling robot projects stripes onto the object to be transported through a projector, and uses the second camera to photograph the object from the front to obtain two fringe patterns with different wavelengths;

The handling robot uses OpenCV technology to obtain the shape of the handling target and the X-axis and Y-axis coordinates of the handling target according to the top view image;

The handling robot uses fringe projection measurement technology to measure the Z-axis coordinates of the handling target according to the fringe pattern;

The handling robot controls the robotic arm to grab the handling target according to the three-dimensional coordinates of the handling target.

The above-mentioned object handling method applied to the handling robot has at least the following beneficial effects: taking a top view image through the first camera, acquiring the shape of the handling target and its X-axis coordinates and Y-axis coordinates through the OpenCV technology, and taking pictures of the handling target through the second camera The stripe pattern is used to measure the Z-axis coordinates of the transport target through the fringe projection measurement technology, and combine with each other to obtain the three-dimensional coordinates of the transport target. At the same time, the transport robot controls the robotic arm to grasp the transport target according to the three-dimensional coordinates; this method enables the transport robot to automatically recognize The morphological characteristics of the transportation target are used to achieve the transportation purpose. It has a high degree of intelligence and can handle a variety of transportation targets in different positions to complete a variety of transportation tasks.

According to the first aspect of the present invention, the transportation robot using OpenCV technology to obtain the shape of the transportation target and the X-axis coordinates and Y-axis coordinates of the transportation target according to the top view image includes the following steps:

Binarize the bird's-eye view image so that the bird's-eye view image presents black and white areas;

Use the largest rectangular box to mark the transportation target, and calculate the center point coordinates of the largest rectangular box as the X-axis and Y-axis coordinates of the transportation target;

Collect connected areas on the white areas in the overhead image;

After processing the noise of the connected area, perform edge recognition to obtain the shape of the conveying target.

According to the first aspect of the present invention, the Z-axis coordinates of the transport target measured by the transport robot using the fringe projection measurement technique according to the fringe pattern are specifically:

The fringe image is processed by the six-step phase shift fringe analysis method to obtain the wrapped phase image φ ₁ (y) and φ ₂ (y);

Use the dual-wavelength fringe phase unwrapping algorithm based on wavelength selection to phase unwrap the wrapped phase image to obtain the phase unwrapping image;

Combine calibration parameters and phase unfolded map to reconstruct a three-dimensional point cloud;

Filter the three-dimensional point cloud to obtain the object point cloud;

Calculate the Z-axis coordinates of the transport target according to the object point cloud data.

According to the first aspect of the present invention, the phase unwrapping of the wrapped phase image using a dual-wavelength fringe phase unwrapping algorithm based on wavelength selection to obtain the phase unwrapping image includes the following steps:

Establish a lookup table for the relationship between m ₁ (y), m ₂ (y), and (m ₂ (y)λ ₂ -m ₁ (y)λ ₁ ], where λ ₁ and λ ₂ are the projections of the projector to the transport target, respectively The wavelengths of the two fringes above, m ₁ (y) and m ₂ (y) are the fringe orders of the two fringes respectively;

Calculate [λ _1φ1 (y)-λ _2φ2 (y)] and round up to get the result value, look up the value of [m ₂ (y)λ ₂ -m ₁ (y)λ ₁ ] and the result value in the relational look-up table The corresponding target values of m ₁ (y) and m ₂ (y) in the closest row;

以及m ₁(y)和m ₂(y)的目标值进行相位展开，得到相位展开图，其中，Φ ₁(y)和Φ ₂(y)是绝对相位。 According to the relationship between absolute phase and wrapped phase

And the target values of m ₁ (y) and m ₂ (y) are phase-expanded to obtain a phase expansion diagram, where Φ ₁ (y) and Φ ₂ (y) are absolute phases.

According to the first aspect of the present invention, the reconstruction of the three-dimensional point cloud by combining the calibration parameters and the phase unfolded map is specifically:

According to the phase expansion diagram, the X-axis digital matrix, the Y-axis digital matrix and the Z-axis digital matrix are established respectively;

Combined with the calibration parameters, the conversion from pixel coordinate system to world coordinate system is realized, and the X-axis digital matrix, Y-axis digital matrix and Z-axis digital matrix are respectively converted into X-axis space matrix, Y-axis space matrix and Z-axis space matrix;

According to the coordinate data of each point on the X-axis, Y-axis and Z-axis, a three-dimensional point cloud is reconstructed.

According to the first aspect of the present invention, the filtering of the three-dimensional point cloud to obtain the object point cloud specifically includes:

Design a background noise filter based on the modulated light intensity reflected by the projected fringe on the surface of the transport target;

The background noise filter is used to filter the pixels and noise points in the three-dimensional point cloud that are less affected by the stripes to obtain the object cloud points.

According to the first aspect of the present invention, the calculation of the Z-axis coordinates of the transport target based on the object point cloud data is specifically: extracting the spatial coordinates of each point from the object point cloud, and calculating the height data of the highest point and the lowest point of the transported object Calculate the difference of the height data to obtain the height of the transported object.

According to the first aspect of the present invention, the robotic arm is a six-degree-of-freedom robotic arm, including a first arm, a second arm, and a third arm; the handling robot controls the robotic arm to grasp according to the three-dimensional coordinates of the handling target The specific handling objectives are:

Set the origin O as the connection point between the robotic arm and the handling robot, and the lengths of the first arm, the second arm and the third arm are L ₁ , L ₂ and L _{3 respectively} ;

以及原点O与搬运目标的角度

Calculate the distance between the origin O and the transportation target

And the angle of origin O and the transportation target

以及原点O与第二臂杆末端的连线和原点O与搬运目标的连线所组成的角度

Calculate the bisecting angle of the angle formed by the first boom and the second boom

And the angle formed by the line between the origin O and the end of the second boom and the line between the origin O and the carrying target

第二臂杆的旋转角度

和第三臂杆的旋转角度

其中S为角度系数，取值为1或-1。 Calculate the rotation angle of the primary boom

The rotation angle of the second boom

And the rotation angle of the third boom

Where S is the angle coefficient, and the value is 1 or -1.

In a second aspect of the present invention, a transport robot is provided, including a traveling mechanism, a first camera, a second camera, a projector, a mechanical arm, a controller, and a memory; the memory is connected to the controller and stores instructions The traveling mechanism, the first camera, the second camera, the projector and the mechanical arm are respectively connected to the controller; the controller executes the instructions to control the traveling mechanism, the first camera, the second camera, The projector and the robot arm implement the object handling method applied to the handling robot as described in the first aspect of the present invention.

The above-mentioned carrying robot has at least the following beneficial effects: taking a top view image through the first camera and acquiring the shape of the carrying object and its X-axis and Y-axis coordinates through the OpenCV technology, and at the same time photographing the stripe image of the carrying object through the second camera and projecting it through the stripe The measurement technology measures the Z-axis coordinates of the transport target, and combines them to obtain the three-dimensional coordinates of the transport target. At the same time, the transport robot controls the robotic arm to grasp the transport target according to the three-dimensional coordinates; it can automatically recognize the morphological characteristics of the transport target and grasp it accordingly The purpose of the transportation target is highly intelligent, which can handle a variety of transportation targets in different positions and complete diversified transportation tasks.

Description of the drawings

The present invention will be further explained below with reference to the drawings and examples.

FIG. 1 is a flowchart of a target handling method applied to a handling robot according to an embodiment of the present invention;

2 is a schematic diagram of a handling robot according to an embodiment of the present invention;

Figure 3 is a schematic diagram of the movement of the robotic arm.

Detailed ways

This section will describe the specific embodiments of the present invention in detail. The preferred embodiments of the present invention are shown in the drawings. The function of the drawings is to supplement the description of the text part of the manual with graphics, so that people can intuitively and vividly understand the text. Each technical feature and overall technical solution of the invention cannot be understood as a limitation on the protection scope of the present invention.

In the description of the present invention, it should be understood that the orientation description involved, for example, the orientation or positional relationship indicated by up, down, front, back, left, right, etc., is based on the orientation or positional relationship shown in the drawings, and only In order to facilitate the description of the present invention and simplify the description, it does not indicate or imply that the pointed device or element must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

If it is described that the first and second are only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly specifying the number of the indicated technical features or implicitly specifying the order of the indicated technical features relationship.

In the description of the present invention, unless otherwise clearly defined, terms such as setting, installation, and connection should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meaning of the above terms in the present invention in combination with the specific content of the technical solution.

1 and 2, an embodiment of the present invention provides a method of object handling applied to a handling robot, the handling robot includes a traveling mechanism 4, a first camera 8, a second camera 6, a projector 7, Controller 2 and robotic arm 5;

The target handling method includes the following steps:

Step S100, the handling robot moves to before the handling target;

Step S200, the transport robot uses the first camera 8 to photograph the transport target from above to obtain a top view image of the transport target;

Step S300: The transport robot projects stripes onto the transport target through the projector 7, and uses the second camera 6 to photograph the transport target from the front to obtain two fringe patterns with different wavelengths;

Step S400: The handling robot uses OpenCV technology to obtain the shape of the handling target and the X-axis coordinates and Y-axis coordinates of the handling target according to the top view image;

Step S500: The transport robot uses the fringe projection measurement technology to measure the Z-axis coordinates of the transport target according to the fringe pattern;

Step S600: The handling robot controls the robotic arm 5 to grab the handling target according to the three-dimensional coordinates of the handling target.

In this embodiment, the top view image is taken through the first camera 8 and the shape of the transport target and its X-axis and Y-axis coordinates are acquired through the OpenCV technology, while the fringe pattern of the transport target is captured by the second camera 6 and measured by fringe projection The technology measures the Z-axis coordinates of the transport target and combines them to obtain the three-dimensional coordinates of the transport target. At the same time, the transport robot controls the robotic arm 5 to grab the transport target according to the three-dimensional coordinates; this method enables the transport robot to automatically recognize the morphological characteristics of the transport target. And according to this to achieve the purpose of transportation, with a high degree of intelligence, it can handle a variety of transportation targets in different locations, and complete diversified transportation tasks.

Further, step S400 specifically includes the following steps:

Step S410: Binarize the overhead image to make the overhead image present a black area and a white area, where the black area is mainly the background image, and the white area is mainly the transportation target;

Step S420: Use the largest rectangular frame to mark the transfer target, and calculate the coordinates of the center point of the largest rectangular frame as the X-axis coordinates and Y-axis coordinates of the transfer target;

Step S430: Perform connected area collection on the white area in the overhead image;

Step S440: Perform edge recognition after processing the noise of the connected area to obtain the shape of the transport target.

Through step S400, the OpenCV technology can be used to quickly measure and obtain the accurate shape of the transport target and the x-axis and y-axis coordinates of the transport target, which is conducive to the accuracy of the transport.

Further, step S500 specifically includes the following steps:

Step S510: Use a six-step phase shift fringe analysis method to process the fringe image to obtain wrapped phase images φ ₁ (y) and φ ₂ (y);

Step S520: Perform phase unwrapping on the wrapped phase map using a dual-wavelength fringe phase unwrapping algorithm based on wavelength selection to obtain a phase unwrapping map;

Step S530: Reconstruct a three-dimensional point cloud by combining the calibration parameters and the phase expansion map;

Step S540, filtering the three-dimensional point cloud to obtain the object point cloud;

Step S550: Calculate the Z-axis coordinate of the transport target according to the object point cloud data.

Further, step S520 includes the following steps:

Step S521: Establish a look-up table for the relationship between m ₁ (y), m ₂ (y) and [m ₂ (y)λ ₂ -m ₁ (y)λ ₁ ], where λ ₁ and λ ₂ are projectors 7 The wavelengths of the two fringes projected on the transport target, m ₁ (y) and m ₂ (y) are the fringe orders of the two fringes respectively;

Step S522: Calculate [λ _1φ1 (y)-λ _2φ2 (y)] and round up to get the result value, and look up the value of [m ₂ (y)λ ₂ -m ₁ (y)λ ₁ ] in the relational look-up table The corresponding target values of m ₁ (y) and m ₂ (y) in the row closest to the result value;

以及m ₁(y)和m ₂(y)的目标值进行相位展开，得到相位展开图，其中，Φ ₁(y)和Φ ₂(y)是绝对相位。 Step S523: According to the relationship between absolute phase and wrapped phase

And the target values of m ₁ (y) and m ₂ (y) are phase-expanded to obtain a phase expansion diagram, where Φ ₁ (y) and Φ ₂ (y) are absolute phases.

Further, step S530 specifically includes the following steps:

Step S531: According to the phase expansion diagram, an X-axis digital matrix, a Y-axis digital matrix, and a Z-axis digital matrix are respectively established therefrom;

Step S532: Combine the calibration parameters to realize the conversion from the pixel coordinate system to the world coordinate system, and convert the X-axis digital matrix, Y-axis digital matrix and Z-axis digital matrix into X-axis space matrix, Y-axis space matrix and Z-axis space matrix respectively ；

In step S533, a three-dimensional point cloud is reconstructed according to the coordinate data about the X axis, the Y axis and the Z axis of each point.

Further, step S540 specifically includes the following steps:

Step S541: Design a background noise filter according to the modulated light intensity reflected by the projected stripes on the surface of the conveying target;

Step S542: Use a background noise filter to filter the pixels and noise points in the three-dimensional point cloud that are less affected by the stripes to obtain the object cloud point.

Further, step S550 is specifically: extracting the spatial coordinates of each point from the object point cloud, and calculating the difference between the height data of the highest point and the height data of the lowest point of the transported object to obtain the height of the transported object, and then obtain the z-axis coordinate .

Through step S500, the fringe projection measurement technology can quickly measure and obtain the accurate z-axis coordinates of the transport target, which is conducive to the accuracy of transport.

3, further, the mechanical arm 5 is a six-degree-of-freedom mechanical arm, including a first arm 51, a second arm 52, and a third arm 53;

Step S600 specifically includes the following steps:

Step S610: Set the origin O as the connection point between the robot arm 5 and the handling robot, and the lengths of the first arm 51, the second arm 52 and the third arm 53 are respectively L ₁ , L ₂ and L ₃ ;

以及原点O与搬运目标的角度

Step S620: Calculate the distance between the origin O and the transport target

And the angle of origin O and the transportation target

以及原点O与第二臂杆52末端的连线和原点O与搬运目标的连线所组成的角度

Step S630: Calculate the bisecting angle of the angle formed by the first arm 51 and the second arm 52

And the angle formed by the line between the origin O and the end of the second arm 52 and the line between the origin O and the transport target

第二臂杆52的旋转角度

和第三臂杆53的旋转角度

其中S为角度系数，取值为1或-1。 Step S640: Calculate the rotation angle of the first arm 51

The rotation angle of the second arm 52

And the rotation angle of the third arm 53

Where S is the angle coefficient, and the value is 1 or -1.

2, another embodiment of the present invention provides a handling robot, including a main frame 1, a traveling mechanism 4, a first camera 8, a second camera 6, a projector 7, a robotic arm 5, and a controller 2. And memory 3; the projector 7 and the second camera 6 are located in front of the main frame 1, the traveling mechanism 4 is located at the bottom of the main frame 1, the first camera 8 is mounted on the main frame 1 through a telescopic rod The controller 2 and the memory 3 are arranged inside the main frame 1. The mechanical arm 5 is located at the center of the front of the main frame 1. The mechanical arm 5 is a six-degree-of-freedom mechanical arm 5 and includes a first arm 51, a second arm 52 and a third arm 53. The memory 3 is connected to the controller 2 and stores instructions; the traveling mechanism 4, the first camera 8, the second camera 6, the projector 7, and the robot arm 5 are respectively connected to the controller 2; The controller 2 executes the instructions to control the traveling mechanism 4, the first camera 8, the second camera 6, the projector 7 and the robot arm 5 to implement the object handling method applied to the handling robot as described above.

In this embodiment, the top view image is taken through the first camera 8 and the shape of the transport target and its X-axis and Y-axis coordinates are acquired through the OpenCV technology, while the fringe pattern of the transport target is captured by the second camera 6 and measured by fringe projection The technology measures the Z-axis coordinates of the transport target, and combines them to obtain the three-dimensional coordinates of the transport target. At the same time, the transport robot controls the robotic arm 5 to grab the transport target according to the three-dimensional coordinates; it can automatically recognize the morphological characteristics of the transport target and grab it accordingly The purpose of the transportation target is highly intelligent, which can handle a variety of transportation targets in different positions and complete diversified transportation tasks.

The above are only preferred embodiments of the present invention. The present invention is not limited to the above-mentioned embodiments. As long as they achieve the technical effects of the present invention by the same means, they should fall within the protection scope of the present invention.

Claims (9)

A target handling method applied to a handling robot, characterized in that the handling robot includes a traveling mechanism, a first camera, a second camera, a projector, a controller and a mechanical arm; the target handling method includes the following steps: The handling robot moves before the handling target; The handling robot shoots the handling target from above through the first camera to obtain the overhead image of the handling target; The handling robot projects stripes onto the object to be transported through a projector, and uses the second camera to photograph the object from the front to obtain two fringe patterns with different wavelengths; The handling robot uses OpenCV technology to obtain the shape of the handling target and the X-axis and Y-axis coordinates of the handling target according to the top view image; The handling robot uses fringe projection measurement technology to measure the Z-axis coordinates of the handling target according to the fringe pattern; The handling robot controls the robotic arm to grab the handling target according to the three-dimensional coordinates of the handling target. The object handling method applied to a handling robot according to claim 1, wherein the handling robot uses OpenCV technology to obtain the shape of the handling target and the X-axis and Y-axis coordinates of the handling target according to the top view image. Step: Binarize the top view image to make the top view image present black and white areas; Use the largest rectangular box to mark the transportation target, and calculate the center point coordinates of the largest rectangular box as the X-axis and Y-axis coordinates of the transportation target; Collect connected areas on the white areas in the overhead image; After processing the noise of the connected area, perform edge recognition to obtain the shape of the conveying target. The method of claim 1, wherein the Z-axis coordinate of the object to be transported is measured by the transport robot using the fringe projection measurement technique according to the fringe pattern: The two fringe patterns are processed by the six-step phase shift fringe analysis method to obtain the wrapped phase patterns φ ₁ (y) and φ ₂ (y); Use the dual-wavelength fringe phase unwrapping algorithm based on wavelength selection to phase unwrap the wrapped phase image to obtain the phase unwrapping image; Combine calibration parameters and phase unfolded map to reconstruct a three-dimensional point cloud; Filter the three-dimensional point cloud to obtain the object point cloud; Calculate the Z-axis coordinates of the transport target according to the object point cloud data. The target handling method applied to a handling robot according to claim 3, characterized in that the phase unwrapping of the package phase map using a dual-wavelength fringe phase unwrapping algorithm based on wavelength selection to obtain the phase unwrapping map comprises the following steps : Establish a lookup table for the relationship between m ₁ (y), m ₂ (y), and (m ₂ (y)λ ₂ -m ₁ (y)λ ₁ ], where λ ₁ and λ ₂ are the projections of the projector to the transport target, respectively The wavelengths of the two fringes above, m ₁ (y) and m ₂ (y) are the fringe orders of the two fringes respectively; Calculate [λ ₁ φ ₁ (y)-λ ₂ φ ₂ (y)] and round to the nearest whole number to get the result value. Look up [m ₂ (y)λ ₂ -m ₁ (y)λ ₁ ] in the relational lookup table The corresponding target values of m ₁ (y) and m ₂ (y) in the row where the value is closest to the result value; According to the relationship between absolute phase and wrapped phase

And the target values of m ₁ (y) and m ₂ (y) are phase unwrapped to obtain a phase unwrapping diagram, where Φ ₁ (y) and Φ ₂ (y) are absolute phases. The target handling method applied to a handling robot according to claim 3, wherein the reconstruction of the three-dimensional point cloud by combining the calibration parameters and the phase expansion map is specifically: According to the phase expansion diagram, the X-axis digital matrix, the Y-axis digital matrix and the Z-axis digital matrix are established respectively; Combine calibration parameters to realize the conversion from pixel coordinate system to world coordinate system, and convert X-axis digital matrix, Y-axis digital matrix and Z-axis digital matrix into X-axis space matrix, Y-axis space matrix and Z-axis space matrix respectively; According to the coordinate data of each point on the X-axis, Y-axis and Z-axis, a three-dimensional point cloud is reconstructed. The object handling method applied to a handling robot according to claim 3, wherein the filtering of the three-dimensional point cloud to obtain the object point cloud is specifically: modulated light reflected from the surface of the handling target according to the projected fringes Strong design of background noise filter; The background noise filter is used to filter the pixels and noise points in the three-dimensional point cloud that are less affected by the stripes to obtain the object cloud points. The object handling method applied to a handling robot according to claim 3, wherein the calculating the Z-axis coordinates of the handling target according to the object point cloud data is specifically: extracting the spatial coordinates of each point from the object point cloud , And calculate the difference between the height data of the highest point and the height data of the lowest point of the transported object to obtain the height of the transported object. The object handling method applied to a handling robot according to claim 1, wherein the mechanical arm is a six-degree-of-freedom mechanical arm, including a first arm, a second arm and a third arm; The handling robot controls the robotic arm to grab the handling target according to the three-dimensional coordinates of the handling target: Set the origin O as the connection point between the robotic arm and the handling robot, and the lengths of the first arm, the second arm and the third arm are L ₁ , L ₂ and L _{3 respectively} ; Calculate the distance between the origin O and the transportation target

And the angle of origin O and the transportation target

Calculate the bisecting angle of the angle formed by the first boom and the second boom

And the angle formed by the line between the origin O and the end of the second boom and the line between the origin O and the carrying target

Calculate the rotation angle of the primary boom

The rotation angle of the second boom

And the rotation angle of the third boom

Where S is the angle coefficient, and the value is 1 or -1. A handling robot, characterized in that it comprises a traveling mechanism, a first camera, a second camera, a projector, a mechanical arm, a controller, and a memory; the memory is connected to the controller and stores instructions; the traveling mechanism , The first camera, the second camera, the projector, and the robotic arm are respectively connected to the controller; the controller executes the instruction to control the traveling mechanism, the first camera, the second camera, the projector, and the robotic arm Implement the object handling method applied to a handling robot according to any one of claims 1-8.

Images