CN111325795A - An image processing method, device, storage medium and robot - Google Patents

Abstract

The embodiments of the present disclosure disclose an image processing method, a device, a storage medium and a robot, which include determining a plurality of grasping surfaces from the to-be-processed multi-dimensional image according to the image data information of the to-be-processed multi-dimensional image and matching the plurality of grasping surfaces with the multiple grasping surfaces. The corresponding multiple grasping points, wherein the multiple grasping surfaces are in one-to-one correspondence with the multiple grasping points; the multiple grasping parameters corresponding to the multiple grasping surfaces are determined; the multiple grasping parameters are used to Take the surface for evaluation, and determine the target grasping surface from multiple grasping surfaces according to the evaluation results; take the grasping point corresponding to the target grasping surface as the target grasping point; determine the target grasping point according to the target grasping point The pose of the grasping point corresponding to the point is taken, so as to grasp the target object corresponding to the target grasping point from the multi-dimensional image to be processed according to the pose of the grasping point.

Description

An image processing method, device, storage medium and robot

technical field

The present disclosure relates to the field of computer vision, and in particular, to an image processing method, device, storage medium and robot.

Background technique

In recent years, the pose calculation of objects has had very important applications in the fields of robotics, automation and machine vision, especially in the field of computer vision.

In the prior art, the image processing device determines the pose of the target object according to the height information of the target grasping surface of the target object, which reduces the accuracy of the image processing device in determining the pose of the target object.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide an image processing method, device, storage medium, and robot.

The technical solution of the present disclosure is realized as follows:

This embodiment provides an image processing method, the method includes:

According to the image data information of the multi-dimensional image to be processed, a plurality of grasping surfaces and a plurality of grasping points corresponding to the multiple grasping surfaces are determined from the to-be-processed multi-dimensional image, wherein the multiple grasping surfaces are The surfaces correspond one-to-one with the plurality of grab points;

determining multiple gripping parameters corresponding to the multiple gripping surfaces;

Evaluate the multiple gripping surfaces by using the multiple gripping parameters, and determine a target gripping surface from the multiple gripping surfaces according to the evaluation result;

Taking the grab point corresponding to the target grab surface as the target grab point;

According to the target grabbing point, the grabbing point pose corresponding to the target grabbing point is determined, so as to grab the target grabbing point from the multi-dimensional image to be processed according to the grabbing point pose the corresponding target object.

The image processing device evaluates multiple grasping surfaces according to multiple grasping parameter values, thereby determining the target grasping surface, instead of determining the target grasping surface according to a single height parameter value, which improves the image processing device to determine the target. With regard to the accuracy of the grasping surface, the grasping point pose of the target object is determined according to the target grasping surface with high accuracy, which improves the accuracy of the image processing device when determining the pose of the target object.

In the above solution, evaluating the multiple grasping surfaces by using the multiple grasping parameters, and determining a target grasping surface from the multiple grasping surfaces according to the evaluation result, includes:

Using the plurality of grasping parameters to evaluate each of the plurality of grasping surfaces, respectively, to obtain a plurality of grasping surface evaluation values corresponding to the plurality of grasping surfaces;

A first grasping surface evaluation value with the highest evaluation value is determined from the plurality of grasping surface evaluation values, and the grasping surface corresponding to the first grasping surface evaluation value is used as the target grasping surface.

The image processing device evaluates each grasping surface respectively according to a plurality of parameter values, and determines the target grasping surface from the plurality of grasping surfaces according to the evaluation value, which improves the accuracy of the image processing device in determining the target grasping surface.

In the above solution, the multiple grab parameters include at least one of the following:

The area parameter of the grasping surface, the height parameter of the grasping surface, the flatness parameter of the grasping surface, and the inclination parameter of the grasping surface.

It is specifically stated that the plurality of grasping parameters are the area parameter of the grasping surface, the height parameter of the grasping surface, the flatness parameter of the grasping surface or the inclination parameter of the grasping surface, and the image processing device can use these grasping parameters to Each gripping surface is evaluated to improve the accuracy of determining the target gripping surface.

In the above solution, according to the image data information of the multi-dimensional image to be processed, the plurality of grasping surfaces and the multiple grasping points corresponding to the multiple grasping surfaces are determined from the multi-dimensional image to be processed, including: :

Input the image data information of the multi-dimensional image to be processed into a deep learning network model to obtain a plurality of pixel points and a plurality of center points corresponding to the plurality of pixel points, and the deep learning network model is a sample using the sample multi-dimensional image. The image data information is a model obtained by training the initial deep learning network model, and the multiple pixel points correspond to the multiple center points one-to-one;

dividing the plurality of center points into multiple groups of center points;

According to any group of center points in the multiple groups of center points, determine a grab point corresponding to the any group of center points, until the multiple grab points are determined from the multiple groups of center points;

According to any group of pixel points corresponding to any group of center points in the multiple groups of center points, a grasping surface corresponding to any group of center points is determined, until the multiple groups of center points corresponding to the multiple groups of center points are determined. The plurality of grasping surfaces are determined from the pixel points, and the center point of any group corresponds to the pixel point of any group one-to-one.

The image processing device will use the deep learning network model to determine multiple pixels and multiple center points corresponding to the multiple pixel points from the image data information of the multi-dimensional image to be processed, and according to the multiple pixel points and multiple center points The multiple grasping surfaces and multiple grasping points are determined, so that the image processing device can infer multiple grasping points corresponding to the multiple grasping surfaces without labeling the collected data and retraining.

In the above solution, a grab point corresponding to any group of center points is determined according to any group of center points in the multiple groups of center points, until the center point is determined from the multiple groups of center points Describe multiple grab points, including:

A group of position data of any group of center points is averaged to obtain an average position data, until a plurality of average position data are obtained from the multiple groups of position data information of the multiple groups of center points;

A plurality of points corresponding to the plurality of average position data are used as a plurality of grab points, wherein the plurality of average position data are in one-to-one correspondence with the plurality of grab points.

The image processing device calculates an average value according to a group of position data of any group of center points to determine the positions of the plurality of grab points, which improves the accuracy of the position information of the plurality of grab points.

In the above solution, before determining multiple grasping surfaces and multiple grasping points corresponding to the multiple grasping surfaces from the to-be-processed multi-dimensional image according to the image data information of the to-be-processed multi-dimensional image, The method also includes:

obtaining the original image data information of the multi-dimensional image to be processed;

The original image data information is preprocessed to obtain the image data information.

Unify the data points of the original image data to improve the accuracy of matching the sample image data with the image data.

In the above solution, the preprocessing of the original image data information to obtain the image data information includes:

In the case that the quantity of raw data information in the raw image data information does not meet the preset quantity value, adjusting the quantity of the raw data information to a preset quantity;

The image data information is obtained by dividing the data of the original data information adjusted to a preset amount by a preset value respectively.

The original data information is divided by the preset value to improve the convergence degree of the original data information calculation and improve the accuracy of the calculation result.

In the above solution, the image data information includes color channel data information and depth data information.

The present disclosure determines the pose of the grasping point of the target object according to the RGB information and depth information of the multi-dimensional image to be processed, so as to improve the accuracy of calculating the pose of the grasping point of the target object.

An embodiment of the present disclosure provides an image processing apparatus, and the apparatus includes:

The determining unit is configured to determine, according to the image data information of the multi-dimensional image to be processed, a plurality of grasping surfaces and a plurality of grasping points corresponding to the multiple grasping surfaces from the multi-dimensional image to be processed, wherein the The multiple gripping surfaces are in one-to-one correspondence with the multiple gripping points; multiple gripping parameters corresponding to the multiple gripping surfaces are determined; the gripping points corresponding to the target gripping surfaces are used as the target gripping point; according to the target grab point, determine the grab point pose corresponding to the target grab point, so as to grab the target grab point from the RGBD image according to the grab point pose the corresponding target object;

The evaluating unit is configured to evaluate the multiple gripping surfaces by using the multiple gripping parameters, and determine the target gripping surface from the multiple gripping surfaces according to the evaluation result.

In the above solution, the evaluation unit is specifically configured to use multiple parameters of the multiple gripping parameters to evaluate each of the multiple gripping surfaces respectively, and obtain the multiple gripping surfaces. Take the evaluation values of multiple grab surfaces corresponding to the surface;

The determining unit is specifically configured to determine the first grasping surface evaluation value with the highest evaluation value from the plurality of grasping surface evaluation values, and use the grasping surface corresponding to the first grasping surface evaluation value as the first grasping surface evaluation value. the target gripping surface.

In the above solution, the multiple grab parameters include at least one of the following:

The area parameter of the grasping surface, the height parameter of the grasping surface, the flatness parameter of the grasping surface, and the inclination parameter of the grasping surface.

In the above solution, the determining unit is specifically configured to input the image data information of the multi-dimensional image to be processed into the deep learning network model, and obtain a plurality of pixel points and a plurality of center points corresponding to the plurality of pixel points, The deep learning network model is a model obtained by using the sample image data information of the sample multidimensional images to train the initial deep learning network model, and the plurality of pixel points correspond to the plurality of center points one-to-one; The center point is divided into multiple groups of center points; according to any group of center points in the multiple groups of center points, a grab point corresponding to the any group of center points is determined, until it is determined from the multiple groups of center points According to any group of pixel points corresponding to any group of center points in the multiple groups of center points, determine a grab surface corresponding to the any group of center points, until the The multiple grasping surfaces are determined from the multiple sets of pixel points corresponding to the multiple sets of center points, and the center points of any group are in one-to-one correspondence with the pixel points of any group.

In the above solution, the determining unit is specifically configured to average a set of position data of any set of center points to obtain an average position data, until multiple sets of position data information from the multiple sets of center points , obtain a plurality of average position data; take a plurality of points corresponding to the plurality of average position data as a plurality of grab points, wherein the plurality of average position data are in one-to-one correspondence with the plurality of grab points.

In the above solution, the device further includes an acquisition unit and a preprocessing unit;

the acquisition unit, configured to acquire the original image data information of the multi-dimensional image to be processed;

The preprocessing unit is configured to preprocess the original image data information to obtain the image data information.

In the above solution, the preprocessing unit is specifically configured to adjust the quantity of the raw data information to a preset value when the quantity of the raw data information in the raw image data information does not meet the preset quantity value Quantity; divide the data of the original data information adjusted to the preset quantity by the preset value to obtain the image data information.

In the above solution, the image data information includes color channel data information and depth data information.

An embodiment of the present disclosure provides an image processing apparatus, and the apparatus includes:

A memory and a graphics processor, where the memory stores an image processing program executable by the graphics processor, and when the image processing program is executed, the method described above is executed by the graphics processor.

An embodiment of the present disclosure provides a storage medium on which a computer program is stored, which is applied to an image processing apparatus, and is characterized in that, when the computer program is executed by a graphics processor, the above-mentioned method is implemented.

An embodiment of the present disclosure provides a robot, including a robotic arm and an image processing device, where the image processing device is configured to execute the above method, and the robotic arm is configured to determine a grasping point of a target object on the image processing device In the case of pose, grab the target object at the grab point pose.

Embodiments of the present disclosure provide an image processing method, a device, a storage medium, and a robot. The image processing method includes: according to image data information of the multi-dimensional image to be processed, determining a plurality of grasping surfaces from the multi-dimensional image to be processed, and determining a plurality of grasping surfaces from the multi-dimensional image to be processed. Multiple grasping points corresponding to grasping surfaces, wherein multiple grasping surfaces correspond to multiple grasping points one-to-one; determine multiple grasping parameters corresponding to multiple grasping surfaces; use multiple grasping parameters Evaluate multiple grasping surfaces, and determine the target grasping surface from the multiple grasping surfaces according to the evaluation results; take the grasping point corresponding to the target grasping surface as the target grasping point; according to the target grasping point, The grasping point pose corresponding to the target grasping point is determined, so as to grasp the target object corresponding to the target grasping point from the multi-dimensional image to be processed according to the grasping point pose. By adopting the above method implementation scheme, the image processing device first determines multiple grasping parameter values corresponding to multiple grasping surfaces, and then evaluates multiple grasping surfaces according to the multiple grasping parameter values, so as to obtain multiple grasping parameters from multiple grasping surfaces. The target grasping surface is determined in the surface taking, and the target grasping surface is not determined according to a single height parameter value, which improves the accuracy of the image processing device in determining the target grasping surface. The target grasping surface determines the grasping point pose of the target object, which improves the accuracy of the image processing device when determining the target object's pose.

Description of drawings

FIG. 1 is a flowchart 1 of an image processing method provided in this embodiment;

2 is a flowchart of an image processing method provided in this embodiment;

FIG. 3 is a schematic diagram 1 of the composition structure of an image processing apparatus provided in this embodiment;

FIG. 4 is a second schematic diagram of the composition and structure of an image processing apparatus provided in this embodiment.

Detailed ways

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure.

An embodiment of the present disclosure provides an image processing method. FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure. As shown in FIG. 1 , the image processing method may include:

S101. According to the image data information of the multi-dimensional image to be processed, determine a plurality of grasping surfaces and a plurality of grasping points corresponding to the multiple grasping surfaces from the multi-dimensional image to be processed, wherein the multiple grasping surfaces and the Each grab point corresponds to each other.

An image processing method provided by an embodiment of the present disclosure is suitable for a scene in which the image data information of a multi-dimensional image to be processed is processed to determine the pose of a grasping point of a target object.

In the embodiment of the present disclosure, the image processing method is applied to an image processing device, and the image processing device can be integrated in a robot, so that the robot can grasp the target object according to the grasping point pose of the target object. In some possible implementations, the image processing method may be implemented by the processor calling computer-readable instructions stored in the memory.

It should be noted that the multi-dimensional image to be processed may be a three-dimensional image to be processed, a four-dimensional image to be processed, or a two-dimensional image to be processed. limited.

In the embodiment of the present disclosure, when the image processing device obtains the image data information of the multi-dimensional image to be processed, the image processing device can determine, from the image to be processed, a plurality of grasping surfaces and a plurality of grasping surfaces according to the image data information. Grab the multiple grab points corresponding to the surface.

It should be noted that, the way that the image processing device obtains the image data information of the multi-dimensional image to be processed may be that the image processing device obtains the multi-dimensional image to be processed through an image acquisition device such as a camera, and reads out the multi-dimensional image from the to-be-processed image. The image data information of the multi-dimensional image to be processed can also be directly obtained by the image processing device from other devices. The specific method for the image processing device to obtain the image data information of the multi-dimensional image to be processed can be determined according to the actual situation. The embodiment does not limit this.

It should also be noted that, if the image processing device obtains the multi-dimensional image to be processed through an image acquisition device such as a camera, and obtains the image processing data by means of the image data information read out from the multi-dimensional image to be processed, Then, the image processing apparatus can control the camera through a central processing unit (Central Processing Unit, CPU) to acquire the multi-dimensional image to be processed, and acquire image data information according to the multi-dimensional image to be processed. When the image processing device obtains the image data information, the image processing device transmits the image data information to a graphics processor (Graphics Processing Unit, GPU), and uses the graphics processor to process the image data information to determine the grasp of the target object. Take a point pose.

In the embodiment of the present disclosure, the image processing apparatus uses the GPU to perform parallel processing on the image data information of the multi-dimensional image to be processed, that is, the GPU simultaneously processes the image data information at all pixels in the multi-dimensional image to be processed, which improves the image processing apparatus. The speed when processing the image data information, that is, the speed when the image processing device uses the image data information in the multi-dimensional image to be processed to determine the pose of the grasping point of the target object is increased.

It should be noted that the multiple grasping surfaces correspond to the multiple grasping points one-to-one.

The multiple grasping surfaces are multiple surfaces of the target object to be grasped in the multi-dimensional image to be processed, and the multiple grasping points are the center points of the multiple grasping surfaces, wherein one grasping surface corresponds to one grasping point .

Exemplarily, the target object in the multi-dimensional image to be processed may be two hexahedrons, and the multiple grasping surfaces may be two front faces of the two hexahedrons, that is, the number of the multiple graspable surfaces is 2, and the first graspable surface can be grasped. The face is the front of the first hexahedron, the second grippable face is the front of the second hexahedron, and the multiple grippable points are the center points of the two front faces, that is, the number of multiple grippable points is 2. The first graspable point is the center point of the front face of the first hexahedron, and the second graspable point is the center point of the front face of the second hexahedron.

Exemplarily, the target object in the multi-dimensional image to be processed may be a tetrahedron, and the multiple grasping surfaces may be two sides of the tetrahedron, that is, the number of multiple graspable surfaces is 2, and the first graspable surface can be grasped. The surface is the first side of the tetrahedron, the second graspable surface is the side adjacent to the first side, that is, the second side, and the multiple graspable points are the center points of the two sides, namely The number of multiple grab points is 2, the first grab point is the center point of the first side, and the second grab point is the center point of the second side.

In the embodiment of the present disclosure, the image data information includes color channel data information and depth data information.

Exemplarily, the image data information may be RGBD data information of the multi-dimensional image to be processed, wherein the color channel data information may be RGB data information of the multi-dimensional image to be processed, and the depth data information may be depth information of the multi-dimensional image to be processed. When the image data information is the RGBD data of the multi-dimensional image to be processed, the image processing device can determine the pose of the grasping point of the target object according to the RGB data of the multi-dimensional image to be processed and the depth image data of the multi-dimensional image to be processed, thereby improving the image quality. The accuracy of the processing device when calculating the grasping point pose of the target object.

In the embodiment of the present disclosure, the image processing apparatus determines the multiple grasping surfaces and the multiple grasping points corresponding to the multiple grasping surfaces from the to-be-processed image according to the image data information of the to-be-processed multi-dimensional image may be as follows: : The image processing device inputs the image data information of the multi-dimensional image to be processed into the deep learning network model, and obtains multiple pixel points and multiple center points corresponding to the multiple pixel points.

In the embodiment of the present disclosure, the image processing device includes a deep learning network model. When the image processing device obtains the image data information of the multi-dimensional image to be processed, the image processing device inputs the image data information into the deep learning network model. The learning network model determines a plurality of pixel points and a plurality of center points from the image data information.

In the embodiment of the present disclosure, the image processing apparatus uses the sample image data information to train the initial deep learning network model, and adjusts the initial parameters of the initial deep learning network model, so that the initial deep learning network model after adjusting the initial parameters can The image data information is classified and detected, the surface information of the sample object in the sample multi-dimensional image is learned, the sample image point and the sample center point are determined, and the deep learning network model is obtained. When the deep learning network model obtains the image data information of the multi-dimensional image to be processed, the deep learning network model can classify and detect the image data information, and determine multiple pixel points and multiple center points from the image data information.

It should be noted that a plurality of pixel points are in one-to-one correspondence with a plurality of center points, wherein one pixel point corresponds to one center point.

It should be noted that the sample image data information is the data information for training the initial deep learning network model. The sample image data information can be multiple surface information of a polyhedron, or the surface information of a sphere, or multiple surface information of a toy, or multiple surface information of other objects. The specific sample image data information can be based on The actual situation is determined, which is not limited in this embodiment of the present disclosure.

In this embodiment of the present disclosure, the information of the multiple surfaces may be the position information of the multiple surfaces, the size information of the multiple surfaces, or the information of multiple sample image points in the multiple surfaces. The specific information may be based on the actual situation. The situation is determined, which is not limited in this embodiment of the present disclosure.

In the embodiment of the present disclosure, before the image processing apparatus trains the initial deep learning network model, the image processing apparatus needs to obtain a lot of sample image data information, such as: multiple face information of a polyhedron, multiple face information of a toy, and face of a sphere information, the face information of the water cup, etc. When the image processing device acquires the sample image data information, the image processing device uses the sample image data information to train the initial deep learning network model, and adjusts the initial parameters of the initial deep learning network model, so that the initial parameters after adjusting the initial parameters are adjusted. The deep learning network model can classify and detect the sample image data information, learn the surface information of the sample object in the sample multi-dimensional image, determine the sample image point and the sample center point, and then obtain the deep learning network model. Therefore, when the image processing device inputs the image data information of the multi-dimensional image to be processed into the deep learning network model, the deep learning network model can determine a plurality of pixel points on the grasping surface from the image data information of the multi-dimensional image to be processed.

It should be noted that the sample image point is a point on the sample image, and the sample center point is the center point of the sample image.

It can be understood that the model parameters in the deep learning network model are parameters obtained by learning the surface information of the objects in the multi-dimensional image of the sample. When the deep learning network model obtains the image data information of the unknown objects in the multi-dimensional image to be processed, The deep learning network model can directly determine the surface information of the unknown object from the image data information according to the model parameters, so as to determine multiple pixel points and multiple center points corresponding to the surface information, without the need for the unknown object. The image data information is marked and retrained, which improves the generalization and practicability of the image processing device when processing the image data information.

In the embodiment of the present disclosure, after the image processing apparatus determines from the image data information of the multi-dimensional image to be processed by using the deep learning network model, multiple pixel points and multiple center points corresponding to the multiple pixel points are determined, the image processing apparatus will Multiple center points can be divided into groups of center points.

In the embodiment of the present disclosure, the image processing apparatus divides multiple center points to obtain multiple sets of center points, which can be used for the image processing apparatus to cluster the multiple center points, so as to determine the correspondence between multiple grasping surfaces multiple sets of center points. The image processing device can also first use the deep learning network model to divide multiple center points, and determine multiple groups of grasping points. The specific image processing device divides multiple center points into multiple groups of center points. The method can be carried out according to the actual situation. It is determined that this embodiment of the present disclosure does not limit this.

It should be noted that, the method for clustering the plurality of center points by the image processing device may be a mean shift clustering algorithm, a hierarchical clustering algorithm, or a density clustering algorithm, and the specific clustering algorithm may be based on The actual situation is determined, which is not limited in this embodiment of the present disclosure.

It should be noted that a plurality of grasping surfaces correspond to a plurality of sets of center points one-to-one, wherein one grasping surface corresponds to a group of grasping points.

In the embodiment of the present disclosure, after the image processing apparatus divides the plurality of center points into multiple groups of center points, the image processing apparatus may determine any group of center points according to any group of center points in the multiple groups of center points A corresponding grab point, until multiple grab points are determined from multiple sets of center points.

In the embodiment of the present disclosure, the manner in which the image processing apparatus determines a grab point corresponding to any group of center points according to any group of center points in the multiple groups of center points may be that the image processing apparatus selects one of the center points from any group of center points. Randomly determine a point and use this point as a grab point, or randomly select a part of the center point from any group of center points for the image processing device, and determine a grab point according to the position information of the part of the center point, image processing The device can also determine a grab point according to the position information of all points of any group of center points, and the specific image processing device determines a grab point according to a group of center points in multiple groups of center points. It is determined, which is not limited in this embodiment of the present disclosure.

In the embodiment of the present disclosure, when the image processing apparatus determines a grab point corresponding to any set of center points from any set of center points in the multiple sets of center points according to a certain method, the image processing apparatus may also use this method The remaining grab points are determined from other groups of center points in the plurality of groups of center points, thereby obtaining a plurality of grab points.

In the embodiment of the present disclosure, the image processing device determines a grab point corresponding to any group of center points according to any group of center points, until a plurality of grab points are determined from the multiple groups of center points The process can be as follows: the image processing device averages a set of position data of any set of center points to obtain an average position data, until the image processing device obtains multiple averages from multiple sets of position data information of multiple sets of center points location data.

It should be noted that one average position data corresponds to a group of position data.

It should be noted that a set of position data can be a set of three-dimensional coordinate data of center points, a set of three-dimensional pose data of a set of center points, or other position data of a set of center points. The situation is determined, which is not limited in this embodiment of the present disclosure.

In the embodiment of the present disclosure, after the image processing apparatus obtains the plurality of average position data, the image processing apparatus may use the plurality of points corresponding to the plurality of average position data as the plurality of grasping points.

It should be noted that a plurality of average position data correspond to a plurality of grab points one-to-one, wherein one average position data corresponds to one grab point.

In the embodiment of the present disclosure, after the image processing apparatus determines a plurality of grasping points among the multiple sets of center points, the image processing apparatus may determine, according to any set of pixel points corresponding to any set of center points in the multiple sets of center points, to determine A grasping surface corresponding to any set of center points is obtained until multiple grasping surfaces are determined from multiple sets of pixel points corresponding to multiple sets of center points.

It should be noted that any group of center points corresponds to any group of pixel points one-to-one, wherein a group of center points corresponds to a group of pixel points.

It should be noted that multiple groups of pixel points correspond to multiple grasping surfaces one-to-one, wherein one group of pixel points corresponds to one grasping surface.

In the embodiment of the present disclosure, before the image processing device determines a plurality of grasping surfaces and grasping points corresponding to the multiple grasping surfaces from the multi-dimensional image to be processed according to the image data information of the multi-dimensional image to be processed, the image processing device The original image data information of the multi-dimensional image to be processed needs to be obtained first.

It should be noted that, the way for the image processing device to obtain the original image data information of the multi-dimensional image to be processed may be that the image processing device obtains the multi-dimensional image to be processed through an image acquisition device such as a camera, and reads the multi-dimensional image to be processed from the multi-dimensional image to be processed. The extracted original image data information can also be the original image data information of the multi-dimensional image to be processed directly obtained by the image processing device from other devices. The situation is determined, which is not limited in this embodiment of the present disclosure.

It should be noted that the original image data information may be original RGBD data information.

In the embodiment of the present disclosure, when the image processing apparatus acquires the original image data information of the multi-dimensional image to be processed, the image processing apparatus preprocesses the original image data information to obtain the image data information.

It can be understood that, by preprocessing the original image data information, the image processing device improves the accuracy of matching between the preprocessed original image data information and the sample image data information, and improves the image data of the multi-dimensional image to be processed by the image processing device. Accuracy in Information Processing.

It should be noted that, the way of preprocessing the original image data by the image processing device may be to remove noise in the original image data information, or to amplify the original image data information, or to change the original image data information. The manner of quantity may also be other manners for processing the original data information, and the specific manner may be determined according to the actual situation, which is not limited in this embodiment of the present disclosure.

In the embodiment of the present disclosure, the image processing apparatus preprocesses the original image data information, and the process of obtaining the image data information may be: when the quantity of the original data information in the original image data information does not meet the preset quantity value, the image processing The device adjusts the amount of raw data information to a preset amount.

It can be understood that the preset amount is the amount of data information preset when the image processing device processes the image data information, and the image processing device adjusts the data amount of the original data information to the preset amount, so that the adjusted original data information and The accuracy of the sample image data matching improves the accuracy of the image processing device when processing the image data information of the multi-dimensional image to be processed.

In the embodiment of the present disclosure, when the image processing apparatus obtains the original image data information of the multi-dimensional image to be processed, the image processing apparatus compares the quantity of the original image data information with the preset quantity value. When the quantity of the original data information does not meet the preset quantity value, the image processing apparatus adjusts the quantity of the original data information to the preset quantity.

It should be noted that, if the quantity of raw data information does not meet the preset quantity value, it may be that the quantity of raw data information is greater than or less than the preset quantity value, and the quantity of raw data information meets the preset quantity value, which may be the quantity of raw data information. equal to the preset amount value.

It should be noted that the preset number value is the number value of the original image data information preset in the image processing device, for example, the preset number value may be 65536, and the corresponding number of points of the abscissa of the multi-dimensional image to be processed may be 256 points, and the number of points corresponding to the ordinate of the multi-dimensional image to be processed may be 256 points.

Exemplarily, the abscissa of the multi-dimensional image to be processed obtained by the image processing device has 1024 points, and the ordinate of the multi-dimensional image to be processed has 1024 points, then the image processing device will use the original image data information of the multi-dimensional image to be processed. Compared with the preset number of data, that is, multiplying 1024 points by 1024 points, a total of 1,048,576 points, is compared with the preset number of 65,536 points. Since the data volume of the original image data information is 1,048,576 points, If it is greater than the preset quantity value of 65536 points, the quantity of the original data information does not meet the preset quantity value.

In this embodiment of the present disclosure, the image processing apparatus may adjust the quantity of the original data information to a preset quantity, as follows: when the image processing apparatus determines that the quantity of the original data information is less than the preset quantity, increase the quantity of the original data information , when the image processing apparatus determines that the quantity of the original data information is greater than the preset quantity, the original data information is reduced.

It should be noted that, the way of adding the original data information by the image processing apparatus may be to add one or more pieces of data information between two adjacent pieces of the original data information, and the value of the added one or more pieces of data information can be obtained. The value can be determined according to the two adjacent original data information, and the way of adding the original data information by the image processing apparatus can be to add one or more data information between two adjacent original data information of the original data information, the one Or the values of multiple data information can be determined according to all the original data information. The image processing device can also add original data information in other ways. The specific method of adding original data information to the image processing device can be determined according to the actual situation. , which is not limited in this embodiment of the present disclosure.

It should be noted that, the way of reducing the original data information by the image processing device may be that the image processing device deletes the data at the odd-numbered point position of the original data information, or the image processing device may delete the data at the even-numbered point position of the original data information. Deleting the data can also be other ways of reducing the original data information, which can be determined according to the actual situation, which is not limited in this embodiment of the present disclosure.

In the embodiment of the present disclosure, after the image processing apparatus adjusts the quantity of the original data information to a preset quantity, the image processing apparatus divides the data of the original data information adjusted to the preset quantity by a preset value to obtain the image data information .

It can be understood that the image processing device divides the original data information by the preset value, so that the original data information can be quickly converged, which improves the degree of convergence in the calculation of the original data information, and the image processing device processes the image data information of the multi-dimensional image to be processed. time accuracy.

It should be noted that the preset value is a preset value in the image processing device.

In the embodiment of the present disclosure, there may be multiple preset values in the image processing device. When the value ranges of the original image data information are different, they may correspond to different preset values. The specific preset values may be selected according to The actual situation is determined, which is not limited in this embodiment of the present disclosure.

S102: Determine multiple gripping parameters corresponding to multiple gripping surfaces.

In the embodiment of the present disclosure, after the image processing device determines a plurality of grasping surfaces from the multi-dimensional image to be processed, the image processing device can respectively determine the number of grasping surfaces corresponding to each of the multiple grasping surfaces. fetch parameters.

It should be noted that the plurality of grasping parameters include at least one of the following: area parameters of the grasping surface, height parameters of the grasping surface, flatness parameters of the grasping surface, and inclination parameters of the grasping surface.

It should also be noted that the multiple grasping parameters may be determined by the image processing apparatus according to the multiple grasping surfaces.

In the embodiment of the present disclosure, the image processing device assigns values to multiple grasping surfaces corresponding to a single grasping parameter, and determines the multiple grasping parameter values corresponding to the single grasping parameter for the multiple grasping surfaces, until the image The processing device assigns multiple grasping surfaces corresponding to the multiple grasping parameters, and determines multiple grasping parameters corresponding to the multiple grasping surfaces.

In the embodiment of the present disclosure, the image processing device is provided with a first set of grasping parameter values corresponding to different area parameters of the grasping surface, a second set of grasping parameter values corresponding to different height parameters of the grasping surface, and the grasping surface is different and flat. The third group of grasping parameter values corresponding to the degree parameter and the fourth group of grasping parameter values corresponding to the different inclination parameters of the grasping surface, when the image processing device obtains one grasping surface among the plurality of grasping surfaces, the image processing device According to the grabbing area parameter of the grabbing surface, the first grabbing parameter value corresponding to the grabbing area parameter is determined from the first group of preset parameter values; The second grabbing parameter value corresponding to the height parameter of the grabbing surface is determined from the grabbing parameter values; the flatness of the grabbing surface is determined from the third group of grabbing parameter values according to the flatness parameter of the grabbing surface According to the inclination parameter of the grabbing surface, the fourth grabbing parameter value corresponding to the inclination parameter of the grabbing surface is determined from the fourth group of grabbing parameter values, until The image processing device determines that each of the multiple grasping surfaces corresponds to four grasping parameter values.

S103: Evaluate multiple grasping surfaces by using multiple grasping parameters, and determine a target grasping surface from the multiple grasping surfaces according to the evaluation result.

In the embodiment of the present disclosure, after the image processing device determines multiple grasping parameters corresponding to the multiple grasping surfaces, the image processing device can use the multiple grasping parameters to evaluate the multiple grasping surfaces, from multiple grasping surfaces. The target grasping surface is determined from the grasping surfaces.

In the embodiment of the present disclosure, the image processing device uses multiple grasping parameters to evaluate multiple grasping surfaces, and the process of determining the target grasping surface from the multiple grasping surfaces may be as follows: the image processing device uses multiple grasping surfaces. The parameters are taken to evaluate each of the multiple grasping surfaces respectively, and multiple grasping surface evaluation values corresponding to the multiple grasping surfaces are obtained.

In the embodiment of the present disclosure, the image processing device may, according to the area parameter of the grasping surface, the height parameter of the grasping surface, the flatness parameter of the grasping surface, and the inclination parameter of the grasping surface corresponding to each grasping surface, Each parameter value corresponding to each grasping surface is obtained until multiple first parameter values corresponding to multiple grasping surfaces are obtained, according to multiple first parameter values and multiple grasping probabilities of multiple grasping surfaces value to get multiple grab surface evaluation values.

It should also be noted that the multiple grasping probability values are the probability values of using multiple grasping surfaces as target grasping surfaces.

Exemplarily, the image processing device may determine a plurality of grasping surface evaluation values through formula (1).

If a set of grasping parameters corresponding to a grasping surface is the area parameter of the grasping surface and the height parameter of the grasping surface, then P(c _i |x) is the number of grasping surfaces where each grasping surface is at The grasping parameter is the evaluation value under the area parameter of the grasping surface and the height parameter of the grasping surface, where c _i is one grasping surface among the multiple grasping surfaces, and x is one grasping surface among the multiple grasping surfaces. Take the area parameter of the surface and the height parameter of one of the multiple grasping surfaces, P(x|c _i ) is the total grasping parameter value corresponding to each grasping surface, and P(c _i ) can be Without considering the grasping parameters, the probability of each grasping surface being grasped, P(x) is the probability when the grasping parameters are the area parameter of the grasping surface and the height parameter of the grasping surface, that is, P (x) has the value 1.

In the embodiment of the present disclosure, the total grasping parameter value corresponding to each grasping surface may be the product of a set of grasping parameters corresponding to each grasping surface, and the total grasping parameter corresponding to each grasping surface The value may also be the sum of a set of grasping parameters corresponding to each grasping surface, and the specific value may be determined according to the actual situation, which is not limited in the embodiments of the present disclosure.

If the total grasping parameter value corresponding to each grasping surface is the product of a set of grasping parameters corresponding to each grasping surface, and the grasping parameter is the area parameter of the grasping surface and the height parameter of the grasping surface, The total grasping parameter value corresponding to each grasping surface is the product of the area parameter of the grasping surface corresponding to the grasping surface and the height parameter of the grasping surface.

Exemplarily, the image processing device determines that there are only two grasping surfaces, and the grasping parameters are the area parameter of the grasping surface and the height parameter of the grasping surface, wherein the area of the grasping surface corresponding to the first grasping surface is The parameter value is 0.5, the area parameter value of the grasping surface corresponding to the second grasping surface is 0.2, the height parameter value of the grasping surface corresponding to the first grasping surface is 0.4, and the second grasping surface The area parameter value of the corresponding grasping surface is 0.6, then the total grasping parameter value corresponding to the first grasping surface is 0.5, and the area parameter value of the first grasping surface is 0.5 and the height parameter of the first grasping surface The value of the product of 0.4 is 0.2; then the total grabbing parameter value corresponding to the second grabbing surface is 0.2, and the area parameter value of the second grabbing surface is 0.2 and the height parameter value of the second grabbing surface is 0.6 The product of , is 0.12.

In the embodiment of the present disclosure, the image processing apparatus may determine the probability of each grasping surface being grasped according to the number of the multiple grasping surfaces.

Exemplarily, if the image processing device determines that there are 5 grasping surfaces, the probability of each grasping surface being grasped, that is, the value of P(c _i ) is 0.2; if the image processing device determines that there are 2 grasping surfaces If the image processing device determines that there are 3 grasping surfaces, then the probability of each grasping surface being grasped, that is, the value of P(c _i ) is 0.5; , that is, the value of P( _ci ) is 1/3.

In the embodiment of the present disclosure, if a set of grasping parameters corresponding to a grasping surface is the area parameter of the grasping surface, the height parameter of the grasping surface, and the flatness parameter of the grasping surface, then the formula (1) x is the area parameter of one of the multiple gripping surfaces, the height parameter of one of the multiple gripping surfaces, and the flatness parameter of one of the multiple gripping surfaces, P( c _i |x) is the evaluation value of each grasping surface among the multiple grasping surfaces when the grasping parameters are the area parameter of the grasping surface, the height parameter of the grasping surface and the flatness parameter of the grasping surface, P (x) is the probability when the grasping parameters are the area parameter of the grasping surface, the height parameter of the grasping surface and the flatness parameter of the grasping surface, that is, the value of P(x) is 1, and P(x|c _i ) is the total grab parameter value corresponding to each grab surface.

If the grasping surface parameter is the product of a set of grasping parameters corresponding to each grasping surface, the total grasping parameter value is the product of a set of grasping parameters corresponding to each grasping surface, and the grasping parameter is the area parameter of the grasping surface, the grasping surface When the height parameter and the flatness parameter of the grasping surface are set, the total grasping parameter value corresponding to each grasping surface is the area parameter of the grasping surface corresponding to the grasping surface, the height parameter of the grasping surface and the grasping surface. Take the product of the flatness parameters of the face.

In the embodiment of the present disclosure, if a set of grasping parameters corresponding to a grasping surface is the area parameter of the grasping surface, the height parameter of the grasping surface, the flatness parameter of the grasping surface, and the inclination parameter of the grasping surface , then x in formula (1) is the area parameter of one of the multiple gripping surfaces, the height parameter of one of the multiple gripping surfaces, and the one gripping surface of the multiple gripping surfaces The flatness parameter and the inclination parameter of one of the multiple grasping surfaces, P(c _i |x) is each grasping surface in the multiple grasping surfaces, and the grasping parameter is the grasping surface The area parameter, the height parameter of the grasping surface, the flatness parameter of the grasping surface and the evaluation value of the inclination parameter of the grasping surface, P(x) is the grasping parameter, which is the area parameter of the grasping surface, the The probability of the height parameter, the flatness parameter of the grasping surface, and the smoothness parameter of the grasping surface, that is, the value of P(x) is 1, and P(x|c _i ) is the total corresponding to each grasping surface. Grab the parameter value.

If the grasping surface parameter is the product of a set of grasping parameters corresponding to each grasping surface, the total grasping parameter value is the product of a set of grasping parameters corresponding to each grasping surface, and the grasping parameter is the area parameter of the grasping surface, the grasping surface When the height parameter, the flatness parameter of the grasping surface, and the inclination parameter of the grasping surface are obtained, the total grasping parameter value corresponding to each grasping surface is the area parameter of the grasping surface corresponding to the grasping surface, The product of the height parameter of the grasping surface, the flatness parameter of the grasping surface, and the inclination parameter of the grasping surface.

In the embodiment of the present disclosure, after the image processing device obtains multiple grasping surface evaluation values corresponding to the multiple grasping surfaces, the image processing device can determine the first evaluation value with the highest evaluation value from the multiple grasping surface evaluation values. A grasping surface evaluation value, and the grasping surface corresponding to the first grasping surface evaluation value is used as the target grasping surface.

In this embodiment of the present disclosure, the manner in which the image processing device determines the first grasping surface evaluation value with the highest evaluation value from the plurality of grasping surface evaluation values may be that the image processing device first selects the first grasping surface evaluation value from the plurality of grasping surface evaluation values. Randomly determine a grasping surface evaluation value in the system, and compare the grasping surface evaluation value with other grasping surface evaluation values respectively, so as to determine the first grasping surface evaluation value with the highest grasping surface evaluation value, and the image processing device The grasping surface corresponding to the first grasping surface evaluation value can be used as the target grasping surface; the image processing device can also sort the multiple grasping surface evaluation values in descending order, and sort the first grasping surface evaluation value. The grasping surface corresponding to the grasping surface evaluation value at one position is used as the target grasping surface; the image processing device can also sort the multiple grasping surface evaluation values according to the order from small to large, and will sort at the last position The grasping surface corresponding to the grasping surface evaluation value at is regarded as the target grasping surface, and the specific method for determining the target grasping surface may be determined according to the actual situation, which is not limited in this embodiment of the present disclosure.

S104, take the grab point corresponding to the target grab surface as the target grab point.

In the embodiment of the present disclosure, after the image processing device evaluates multiple grasping surfaces by using multiple grasping parameter values, and determines the target grasping surface from the multiple grasping surfaces, the image processing device grasps the target grasping surface. The grab point corresponding to the surface is taken as the target grab point.

The target grasping point may be the center point of the target grasping surface. Of course, in other embodiments, the target grasping point may not be the center point of the target grasping surface, for example, it may be a neighborhood point of the center point of the target grasping surface.

S105 , according to the target grasping point, determine the grasping point pose corresponding to the target grasping point, so as to grasp the target object corresponding to the target grasping point from the multi-dimensional image to be processed according to the grasping point pose.

In the embodiment of the present disclosure, after the image processing device determines the target grasping point, the image processing device may determine the grasping point pose corresponding to the target grasping point according to the target grasping point, so as to determine the corresponding grasping point pose according to the grasping point. Pose, grab the target object corresponding to the target grab point from the multi-dimensional image to be processed.

It should be noted that the pose of the grasping point corresponding to the target grasping point may be the six-dimensional pose of the target grasping point, the five-dimensional pose of the target grasping point, or the other The pose of the dimension can be specifically determined according to the actual situation, which is not limited in this embodiment of the present disclosure.

It should be noted that the information of the target grasping point may be three-dimensional coordinate point information.

When the grasping point pose corresponding to the target grasping point is the six-dimensional pose of the target grasping point, the image processing device may determine the three-dimensional coordinate point information of the target grasping point and the rotation degree information of the target grasping point Get the grab point pose corresponding to the target grab point.

In the embodiment of the present disclosure, the way for the image processing device to determine the rotation degree information of the target grasping point may be that the image processing device performs plane fitting according to a plurality of target pixel points on the target grasping surface to obtain a fitted target For the grasping plane, the image processing device determines the tangent of the target grasping point on the fitting plane, and the image processing device uses the vertical vector of the tangent as the rotation degree information, thereby obtaining the rotation degree information of the target grasping point.

The embodiment of the present disclosure provides an exemplary flow chart of an image processing method. As shown in FIG. 2 , when the CPU in the intelligent robot controls the camera to collect the original image data information of the multi-dimensional image to be processed, that is, the image processing device collects the original image data information. When the original RGBD image data information of the multi-dimensional image to be processed is reached, the CPU controls the transmission of the original RGBD image data to the GPU, and the GPU preprocesses the original RGBD image data to obtain the RGBD image data information, that is, obtains the image data information. , the GPU uses the deep learning network model to determine multiple grasping surfaces and multiple grasping points corresponding to the multiple grasping surfaces from the RGBD image data information, and the GPU uses multiple grasping parameters to determine the Each grasping surface is evaluated. According to the evaluation results of each grasping surface, the target grasping surface is determined from the multiple grasping surfaces, and the grasping point corresponding to the target grasping surface is used as the target grasping point. The grasping point pose of the target object is determined according to the target grasping point. After the image processing device determines the grasping point pose of the target object, the intelligent robot can grasp the target object at the grasping point pose.

It can be understood that the image processing device first determines multiple grasping parameter values corresponding to the multiple grasping surfaces, and then evaluates the multiple grasping surfaces according to the multiple grasping parameter values, so as to obtain multiple grasping parameters from the multiple grasping surfaces. The target grasping surface is determined in the surface, and the target grasping surface is not determined according to a single height parameter value, which improves the accuracy of the image processing device in determining the target grasping surface. The grasping surface determines the grasping point pose of the target object, which improves the accuracy of the image processing device when determining the pose of the target object.

When the image processing device is applied in the logistics transmission process, the image processing device can determine the position and posture of the grabbing point of the target express delivery according to the above-mentioned implementation manner, so as to realize the sorting and palletizing of the express delivery.

When the image processing device is applied in the process of medicine and beauty, the image processing device can determine the grasping point pose of the target drug according to the above implementation method, or the image processing device can determine the grasping point of the target beauty product according to the above implementation method. Point poses, so as to realize the classification and packaging of medicines and beauty products.

When the image processing device is applied to the heavy industry, the image processing device can determine the grasping point pose of the target industrial product according to the above implementation manner, so as to realize the handling of the target industrial product.

When the image processing device is applied in the process of garbage disposal, the image processing device can determine the pose of the grabbing point of the target garbage according to the above implementation manner, so as to realize the classification and processing of garbage.

Based on the same inventive concept, an embodiment of the present disclosure provides an image processing apparatus 1, which corresponds to an image processing method; FIG. 3 is a schematic structural diagram 1 of an image processing apparatus provided by an embodiment of the present disclosure. The image processing apparatus 1 can include:

The determining unit 11 is configured to determine a plurality of grasping surfaces and a plurality of grasping points corresponding to the multiple grasping surfaces from the to-be-processed multi-dimensional image according to the image data information of the to-be-processed multi-dimensional image, wherein, The plurality of grasping surfaces are in one-to-one correspondence with the multiple grasping points; multiple grasping parameters corresponding to the multiple grasping surfaces are determined; the grasping points corresponding to the target grasping surfaces are used as the target grasping taking a point; according to the target grabbing point, determine the grabbing point pose corresponding to the target grabbing point, so as to grab the target grabbing from the RGBD image according to the grabbing point pose The target object corresponding to the point;

The evaluating unit 12 is configured to evaluate the multiple gripping surfaces by using the multiple gripping parameters, and determine the target gripping surface from the multiple gripping surfaces according to the evaluation result.

In some embodiments of the present disclosure, the evaluating unit 12 is specifically configured to use multiple parameters of the multiple gripping parameters to evaluate each of the multiple gripping surfaces, respectively, to obtain multiple gripping surface evaluation values corresponding to the multiple gripping surfaces;

The determining unit 11 is specifically configured to determine the first grasping surface evaluation value with the highest evaluation value from the plurality of grasping surface evaluation values, and determine the grasping surface corresponding to the first grasping surface evaluation value. as the target gripping surface.

In some embodiments of the present disclosure, the plurality of grasping parameters include at least one of the following:

The area parameter of the grasping surface, the height parameter of the grasping surface, the flatness parameter of the grasping surface, and the inclination parameter of the grasping surface.

In some embodiments of the present disclosure, the determining unit 11 is specifically configured to input the image data information of the multi-dimensional image to be processed into a deep learning network model, and obtain a plurality of pixels and the corresponding data of the plurality of pixels. A plurality of center points, the deep learning network model is a model obtained by using the sample image data information of the sample multi-dimensional image to train the initial deep learning network model, and the plurality of pixel points correspond to the plurality of center points one-to-one; The multiple center points are divided into multiple sets of center points; according to any set of center points in the multiple sets of center points, a grab point corresponding to the any set of center points is determined, until the point from the multiple sets of center points is determined. Determine the plurality of grab points from the group center points; determine a grab point corresponding to any group of center points according to any group of pixel points corresponding to any group of center points in the multiple groups of center points surface, until the multiple grasping surfaces are determined from the multiple sets of pixel points corresponding to the multiple sets of center points, and the center points of any group correspond to the pixel points of any group one-to-one.

In some embodiments of the present disclosure, the determining unit 11 is specifically configured to average a set of position data of any set of center points to obtain an average position data, until the data from the multiple sets of center points is obtained. In multiple sets of position data information, multiple average position data are obtained; multiple points corresponding to the multiple average position data are used as multiple grab points, wherein the multiple average position data and the multiple grab points are obtained. One-to-one correspondence.

In some embodiments of the present disclosure, the apparatus further includes an acquisition unit 13 and a preprocessing unit 14;

The obtaining unit 13 is configured to obtain the original image data information of the multi-dimensional image to be processed;

The preprocessing unit 14 is configured to preprocess the original image data information to obtain the image data information.

In some embodiments of the present disclosure, the preprocessing unit 14 is specifically configured to, if the quantity of the raw data information in the raw image data information does not meet a preset quantity value, The number is adjusted to a preset number; the data of the original data information adjusted to the preset number is divided by a preset value to obtain the image data information.

In some embodiments of the present disclosure, the image data information includes color channel data information and depth data information.

It should be noted that, in practical applications, the above determination unit 11 , evaluation unit 12 , acquisition unit 13 and preprocessing unit 14 may be implemented by the processor 15 on the motion quality evaluation device 1 , specifically a GPU (Graphics Processing Unit, graphics processor). ), MPU (Microprocessor Unit, microprocessor), DSP (Digital Signal Processing, digital signal processor) or Field Programmable Gate Array (FPGA, Field Programmable Gate Array), etc.; the above data storage can be achieved by the memory on the motion quality evaluation device 1 16 realized.

An embodiment of the present disclosure further provides an image processing apparatus 1. As shown in FIG. 4 , the image processing apparatus 1 includes: a processor 15 and a memory 16, where the memory 16 stores image processing executable by the processor 14 A program, when the program is executed, executes the image processing method as described above by the processor 15 .

In practical applications, the above-mentioned memory 16 may be a volatile memory (volatile memory), such as a random-access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), flash memory (flash memory), hard disk (Hard DiskDrive, HDD) or solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, and provide the processor 15 with instructions and data.

An embodiment of the present disclosure provides a computer-readable storage medium having a computer program thereon, and when the program is executed by the processor 15, the image processing method as described above is implemented.

This embodiment provides a robot, including a robotic arm and an image processing device, the image processing device is used to execute the above method, and the robotic arm is used to determine the grasping point of a target object on the image processing device In the case of the grasping point pose, grab the target object at the grasping point pose.

Specifically, after obtaining the grasping point pose of the target object determined by the image processing device, the robotic arm can calculate the grasping pose of the robotic arm to grasp the target object at the grasping point pose according to the grasping point pose, so as to obtain the grasping pose of the robotic arm. Plan the motion path of the robotic arm for object grasping.

It can be understood that the image processing device first determines multiple grasping parameter values corresponding to the multiple grasping surfaces, and then evaluates the multiple grasping surfaces according to the multiple grasping parameter values, so as to obtain multiple grasping parameters from the multiple grasping surfaces. The target grasping surface is determined in the surface, and the target grasping surface is not determined according to a single height parameter value, which improves the accuracy of the image processing device in determining the target grasping surface. The grasping surface determines the grasping point pose of the target object, which improves the accuracy of the image processing device when determining the pose of the target object.

When the image processing device is applied in the logistics transmission process, the image processing device can determine the position and posture of the grabbing point of the target express delivery according to the above-mentioned implementation manner, so as to realize the sorting and palletizing of the express delivery.

When the image processing device is applied in the process of medicine and beauty, the image processing device can determine the grasping point pose of the target drug according to the above implementation method, or the image processing device can determine the grasping point of the target beauty product according to the above implementation method. Point poses, so as to realize the classification and packaging of medicines and beauty products.

When the image processing device is applied to the heavy industry, the image processing device can determine the grasping point pose of the target industrial product according to the above implementation manner, so as to realize the handling of the target industrial product.

When the image processing device is applied in the process of garbage disposal, the image processing device can determine the pose of the grabbing point of the target garbage according to the above implementation manner, so as to realize the classification and processing of garbage.

As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above descriptions are merely preferred embodiments of the present disclosure, and are not intended to limit the protection scope of the present disclosure.

Claims (12)

1. an image processing method, is characterized in that, described method comprises: According to the image data information of the multi-dimensional image to be processed, a plurality of grasping surfaces and a plurality of grasping points corresponding to the multiple grasping surfaces are determined from the to-be-processed multi-dimensional image, wherein the multiple grasping surfaces are The surfaces correspond one-to-one with the plurality of grab points; determining multiple gripping parameters corresponding to the multiple gripping surfaces; Evaluate the multiple gripping surfaces by using the multiple gripping parameters, and determine a target gripping surface from the multiple gripping surfaces according to the evaluation result; Taking the grab point corresponding to the target grab surface as the target grab point; According to the target grabbing point, the grabbing point pose corresponding to the target grabbing point is determined, so as to grab the target grabbing point from the multi-dimensional image to be processed according to the grabbing point pose the corresponding target object. 2 . The method according to claim 1 , wherein the plurality of grasping surfaces are evaluated by using the plurality of grasping parameters, and are determined from the plurality of grasping surfaces according to the evaluation results. 3 . Out of the target grasping surface, including: Using the plurality of grasping parameters to evaluate each of the plurality of grasping surfaces, respectively, to obtain a plurality of grasping surface evaluation values corresponding to the plurality of grasping surfaces; A first grasping surface evaluation value with the highest evaluation value is determined from the plurality of grasping surface evaluation values, and the grasping surface corresponding to the first grasping surface evaluation value is used as the target grasping surface. 3. The method according to claim 1 or 2, wherein the plurality of grasping parameters comprise at least one of the following: The area parameter of the grasping surface, the height parameter of the grasping surface, the flatness parameter of the grasping surface, and the inclination parameter of the grasping surface. 4. The method according to any one of claims 1-3, wherein, according to the image data information of the multi-dimensional image to be processed, a plurality of grasping surfaces are determined from the multi-dimensional image to be processed and the Multiple grab points corresponding to multiple grab surfaces, including: Input the image data information of the multi-dimensional image to be processed into a deep learning network model to obtain a plurality of pixel points and a plurality of center points corresponding to the plurality of pixel points, and the deep learning network model is a sample using the sample multi-dimensional image. The image data information is a model obtained by training the initial deep learning network model, and the multiple pixel points correspond to the multiple center points one-to-one; dividing the plurality of center points into multiple groups of center points; According to any group of center points in the multiple groups of center points, determine a grab point corresponding to the any group of center points, until the multiple grab points are determined from the multiple groups of center points; According to any group of pixel points corresponding to any group of center points in the multiple groups of center points, a grasping surface corresponding to any group of center points is determined, until the multiple groups of center points corresponding to the multiple groups of center points are determined. The plurality of grasping surfaces are determined from the pixel points, and the center point of any group corresponds to the pixel point of any group one-to-one. 5 . The method according to claim 4 , wherein, according to any one group of center points in the multiple groups of center points, a grab point corresponding to the any group of center points is determined, until from The multiple grab points are determined from the multiple sets of center points, including: A group of position data of any group of center points is averaged to obtain an average position data, until a plurality of average position data are obtained from the multiple groups of position data information of the multiple groups of center points; A plurality of points corresponding to the plurality of average position data are used as a plurality of grab points, wherein the plurality of average position data are in one-to-one correspondence with the plurality of grab points. 6. The method according to any one of claims 1-5, wherein, according to the image data information of the multi-dimensional image to be processed, a plurality of grasping surfaces are determined from the multi-dimensional image to be processed and the Before the multiple gripping points corresponding to the multiple gripping surfaces, the method further includes: obtaining the original image data information of the multi-dimensional image to be processed; The original image data information is preprocessed to obtain the image data information. 7. The method according to claim 6, wherein the preprocessing of the original image data information to obtain the image data information comprises: In the case that the quantity of raw data information in the raw image data information does not meet the preset quantity value, adjusting the quantity of the raw data information to a preset quantity; The image data information is obtained by dividing the data of the original data information adjusted to a preset amount by a preset value respectively. 8. The method according to any one of claims 1-7, wherein the image data information includes color channel data information and depth data information. 9. An image processing device, wherein the device comprises: The determining unit is configured to determine, according to the image data information of the multi-dimensional image to be processed, a plurality of grasping surfaces and a plurality of grasping points corresponding to the multiple grasping surfaces from the multi-dimensional image to be processed, wherein the The multiple gripping surfaces are in one-to-one correspondence with the multiple gripping points; multiple gripping parameters corresponding to the multiple gripping surfaces are determined; the gripping points corresponding to the target gripping surfaces are used as the target gripping point; according to the target grab point, determine the grab point pose corresponding to the target grab point, so as to grab the target grab point from the RGBD image according to the grab point pose the corresponding target object; The evaluating unit is configured to evaluate the multiple gripping surfaces by using the multiple gripping parameters, and determine the target gripping surface from the multiple gripping surfaces according to the evaluation result. 10. An image processing device, wherein the device comprises: A memory and a graphics processor, the memory stores an image processing program executable by the graphics processor, and when the image processing program is executed, the graphics processor executes any one of claims 1 to 8 Methods. 11 . A storage medium having a computer program stored thereon and applied to an image processing apparatus, wherein the computer program implements the method according to any one of claims 1 to 8 when the computer program is executed by a graphics processor. 12. A robot, characterized by comprising: a robotic arm and an image processing device, wherein the image processing device is used to execute the method according to any one of claims 1-8, and the robotic arm is used to determine when the image processing device is used. When the grasping point pose of the target object is obtained, the target object is grasped at the grasping point pose.

Images