CN116704111A - Image processing method and apparatus - Google Patents

Abstract

The present application provides an image processing method and device, which is beneficial to improving the accuracy of three-dimensional reconstruction results. The method includes: obtaining multiple sets of matching feature points based on multiple original images, and determining the three-dimensional coordinates of the multiple sets of matching feature points and the relative camera position corresponding to each original image based on the pixel coordinates of the multiple sets of matching feature points in the original image Pose, based on the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image, determine the depth information of the corresponding matching feature points on each original image, and combine multiple original images to determine the depth information of each original image The corresponding environmental light map, based on the environmental light map and the relative camera pose corresponding to each original image, determine the three-dimensional coordinates of the virtual light source; based on the three-dimensional coordinates of the virtual light source and multiple environmental light maps, multiple light compensation images are obtained; based on multiple The three-dimensional coordinates of the illumination compensation image and multiple sets of matching feature points are used to determine the three-dimensional model of the target object.

Description

Image processing method and device

technical field

The present application relates to the technical field of image processing, and in particular to an image processing method and device.

Background technique

3D reconstruction technology refers to the use of computer technology to reconstruct real world scenes or objects into data models that can be expressed and processed by computers. Image-based 3D reconstruction technology is more and more widely used due to its low requirements for image acquisition equipment and low cost of reconstruction process.

At present, image-based 3D reconstruction technologies mainly include: extracting feature points in multi-view images, completing feature point matching, performing sparse point cloud reconstruction based on motion restoration structure technology, and dense reconstruction based on multi-view stereo technology. Usually, after the sparse point cloud is reconstructed, the mean value of the color of the matching feature point in the image under different viewing angles is taken as the corresponding 3D point color, and the sparse point cloud is colored. Based on the sparse point cloud after coloring, dense reconstruction is performed through multi-view stereo technology to improve the accuracy of 3D reconstruction results.

However, due to the surface material of the target object, the position or color of the ambient light when the image of the target object is collected, the images under different viewing angles may have the phenomenon that the texture of the target object is not clear, the color is not real, and the texture is inconsistent under multiple viewing angles. The mean value of the colors of the points in the images under different viewing angles does not represent the true color of the target object, therefore, the accuracy of the 3D reconstruction results of the above methods is poor.

Contents of the invention

The present application provides an image processing method and device, which can take into account the influence of ambient light on a target object, and help improve the accuracy of 3D reconstruction results.

In the first aspect, an image processing method is provided, including: acquiring a plurality of original images of a target object, and the plurality of original images are respectively obtained by shooting the target object under different viewing angles; Group matching feature points, and based on the pixel coordinates of multiple sets of matching feature points in the corresponding original image, determine the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image; based on multiple sets of matching feature points The three-dimensional coordinates and the relative camera pose corresponding to each original image determine the depth information of the corresponding matching feature points on each original image; based on the depth information of the corresponding matching feature points on each original image and multiple original images, determine The environmental light map corresponding to each original image; based on the environmental light map corresponding to each original image and the camera relative pose corresponding to each original image, determine the three-dimensional coordinates of the virtual light source; based on the three-dimensional coordinates of the virtual light source and the corresponding position of each original image In the environmental light map, light compensation is performed on each original image to obtain multiple light compensation images; based on the multiple light compensation images and the 3D coordinates of multiple sets of matching feature points, the 3D model of the target object is determined.

In the image processing method of the embodiment of the present application, by considering the impact of ambient light on the target object, the environment light map is introduced during 3D reconstruction, and the three-dimensional coordinates of the virtual light source are determined by using the environment light map. Based on the three-dimensional coordinates of the virtual light source and the environment light map Light compensation is performed on the original image to obtain multiple light compensation images. Since the light compensation image contains the influence of ambient light on the target object, using multiple light compensation images to perform 3D reconstruction of the target object can make the 3D model obtained after 3D reconstruction The color is closer to the real color of the target object in the actual ambient lighting scene, which is conducive to improving the accuracy of the 3D reconstruction results, thereby improving the user experience.

It should be understood that the depth information is the vertical distance from each group of matching feature points to the imaging plane of the corresponding image acquisition device (such as a camera).

It should also be understood that the ambient light map is an image used to represent the brightness information of each pixel in the original image under the influence of the ambient light source. Each of the above multiple original images has a corresponding ambient light map, that is, multiple original light maps can be obtained in this step.

It should also be understood that the number of virtual light sources can be one or more

With reference to the first aspect, in some implementations of the first aspect, the above method further includes: based on the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source, and the three-dimensional coordinates of multiple sets of matching feature points, determine to update the virtual The three-dimensional coordinates of the light source and the updated three-dimensional coordinates of multiple sets of matching feature points; based on the three-dimensional coordinates of the virtual light source and the environmental light map corresponding to each original image, perform illumination compensation for each original image, including: based on the updated three-dimensional coordinates of the virtual light source and each The ambient light map corresponding to the original image performs light compensation on each original image; based on multiple light compensation images and the 3D coordinates of multiple matching feature points, the 3D model of the target object is determined, including: based on multiple light compensation images and multiple The updated 3D coordinates of the group matching feature points are used to determine the 3D model of the target object.

The image processing method of the embodiment of the present application performs illumination compensation on each original image by determining and updating the three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of multiple sets of matching feature points, and based on the image after illumination compensation and the multiple sets of matching feature points Updating the three-dimensional coordinates, determining the three-dimensional model of the target object, the updated three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of the multiple sets of matching feature points make the error of the updated three-dimensional coordinates of the three-dimensional coordinates of the virtual light source and the multiple sets of matching feature points smaller than that of the non-updated three-dimensional coordinates of the virtual light source, It is beneficial to improve the accuracy of 3D reconstruction results.

In combination with the first aspect, in some implementations of the first aspect, light compensation is performed on each original image to obtain multiple light compensation images, including: determining the position of the virtual light source in each original image based on updating the three-dimensional coordinates of the virtual light source The updated pixel coordinates of the virtual light source; based on the difference between the updated pixel coordinates of the virtual light source in each original image and the corresponding pixel coordinates of the virtual light source in each original image, the pixels in the ambient light map corresponding to each original image Points are displaced to obtain the ambient light compensation map corresponding to each original image; based on the ambient light compensation map corresponding to each original image, light compensation is performed on each original image to obtain multiple light compensation images.

Exemplarily, the image processing device may map the ambient illumination compensation map corresponding to each original image to the corresponding original image, perform illumination compensation on each original image through a back-projection operation, and obtain multiple illumination compensation images.

In combination with the first aspect, in some implementations of the first aspect, the 3D model of the target object is determined based on multiple illumination compensation images and updated 3D coordinates of multiple sets of matching feature points, including: updating multiple sets of matching feature points The three-dimensional coordinates are determined as a colorless sparse three-dimensional point cloud; obtain the pixel coordinates of multiple sets of matching feature points in the corresponding multiple illumination compensation images; use the pixel coordinates of multiple sets of matching feature points in the corresponding multiple illumination compensation images to The colorless sparse three-dimensional point cloud is colored to obtain the colored sparse three-dimensional point cloud; based on the colored sparse three-dimensional point cloud and multiple illumination compensation images, the three-dimensional model of the target object is determined.

Exemplarily, the image processing device may color the colorless sparse 3D point cloud by using the mean value of the pixel coordinates of multiple sets of matching feature points in the corresponding multiple illumination compensation images to obtain the colored sparse 3D point cloud.

In combination with the first aspect, in some implementations of the first aspect, multiple sets of matching feature points are obtained based on multiple original images, including: extracting feature points of each original image in multiple original images; Feature point matching to obtain multiple sets of matching feature points.

It should be understood that the image processing device may extract the feature points of each original image in the multiple original images through a feature extraction algorithm. Exemplarily, the feature extraction algorithm may be a scale-invariant feature transform algorithm (scale-invariant feature transform, SIFT), an accelerated robust feature algorithm (speeded up robust features, SURF), a corner detection algorithm (features from accelerated segment test, FAST) Or a feature point extraction algorithm such as a fast feature point extraction and description algorithm (oriented fast and rotated brief, ORB) algorithm extracts the feature points of each original image in multiple original images, which is not limited in this embodiment of the present application.

It should also be understood that the image processing device may perform feature point matching on multiple feature points extracted from multiple original images through a feature matching strategy to obtain multiple sets of matching feature points. Exemplarily, the feature matching strategy may be a brute force matching strategy, a K-nearest neighbor (KNN) matching strategy, etc., which are not limited in this embodiment of the present application.

Through appropriate feature extraction algorithms and feature matching strategies, image processing equipment can more accurately extract feature points, and can obtain more accurate feature matching results, that is, the above-mentioned multiple sets of matching feature points, which is conducive to improving the subsequent 3D reconstruction results accuracy.

In combination with the first aspect, in some implementations of the first aspect, determining the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image includes: based on multiple sets of matching feature points in the corresponding original image The pixel coordinates in the corresponding original images and the camera internal references corresponding to multiple original images, use the triangulation method to determine the relative camera pose corresponding to each original image; based on the pixel coordinates of multiple sets of matching feature points in the corresponding original images and each original The relative pose of the camera corresponding to the image, using the triangulation method, determines the three-dimensional coordinates of multiple sets of matching feature points.

In combination with the first aspect, in some implementations of the first aspect, determining the depth information of the corresponding matching feature points on each original image includes: combining the three-dimensional coordinates of multiple sets of matching feature points with the camera corresponding to each original image The relative pose is input to the depth estimation network model to obtain the depth information of the corresponding matching feature points on each original image.

Exemplarily, the depth estimation network model may be a convolutional neural network (convolutional neural networks, CNN).

In combination with the first aspect, in some implementations of the first aspect, determining the ambient light map corresponding to each original image includes: inputting depth information of corresponding matching feature points on each original image and multiple original images to The illumination estimation network model obtains the environmental illumination map corresponding to each original image.

Exemplarily, the illumination estimation network model may be Gardner's illumination estimation network model.

With reference to the first aspect, in some implementations of the first aspect, determining the three-dimensional coordinates of the virtual light source includes: determining the pixel coordinate with the smallest pixel amplitude in the ambient light map corresponding to each original image as the virtual light source in each original image. The corresponding pixel coordinates in the image; based on the pixel coordinates corresponding to the virtual light source in each original image and the relative camera pose corresponding to each original image, determine the three-dimensional coordinates of the virtual light source.

In a second aspect, an image processing device is provided, including: an acquisition module, configured to acquire multiple original images of a target object, where the multiple original images are respectively obtained by shooting the target object under different viewing angles; a processing module , used to obtain multiple sets of matching feature points based on multiple original images, and based on the pixel coordinates of multiple sets of matching feature points in the corresponding original images, determine the three-dimensional coordinates of multiple sets of matching feature points relative to the camera corresponding to each original image Pose; based on the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image, determine the depth information of the corresponding matching feature points on each original image; based on the corresponding matching feature points on each original image Based on the depth information of each original image and multiple original images, determine the ambient light map corresponding to each original image; determine the three-dimensional coordinates of the virtual light source based on the environmental light map corresponding to each original image and the relative camera pose corresponding to each original image; The three-dimensional coordinates of the light source and the ambient light map corresponding to each original image, perform illumination compensation on each original image, and obtain multiple illumination compensation images; and, based on multiple illumination compensation images and the three-dimensional coordinates of multiple sets of matching feature points, determine the target 3D model of the object.

In combination with the second aspect, in some implementations of the second aspect, the processing module is further configured to: based on the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source, and the three-dimensional coordinates of multiple sets of matching feature points, determine the updated The three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of multiple sets of matching feature points; based on the updated three-dimensional coordinates of the virtual light source and the environmental light map corresponding to each original image, light compensation is performed on each original image; and, based on multiple light compensation images and The updated three-dimensional coordinates of multiple groups of matching feature points determine the three-dimensional model of the target object.

With reference to the second aspect, in some implementations of the second aspect, the processing module is further configured to: determine the updated pixel coordinates of the virtual light source in each original image based on the updated three-dimensional coordinates of the virtual light source; The difference between the updated pixel coordinates in the image and the pixel coordinates corresponding to the virtual light source in each original image, and the pixels in the environment light map corresponding to each original image are displaced to obtain the environment corresponding to each original image Illumination compensation map: Based on the ambient light compensation map corresponding to each original image, light compensation is performed on each original image to obtain multiple light compensation images.

In combination with the second aspect, in some implementations of the second aspect, the processing module is also used to: determine the updated 3D coordinates of multiple sets of matching feature points as a colorless sparse 3D point cloud; the acquisition module is also used to: acquire multiple sets of Match the pixel coordinates of the feature points in the corresponding multiple illumination compensation images; the processing module is also used to: use the pixel coordinates of multiple sets of matching feature points in the corresponding multiple illumination compensation images to color the colorless and sparse 3D point cloud , to obtain the colored sparse 3D point cloud; based on the colored sparse 3D point cloud and multiple illumination compensation images, the 3D model of the target object is determined.

In combination with the second aspect, in some implementations of the second aspect, the processing module is also used to: extract feature points of each original image in multiple original images; and perform feature point matching on multiple original images to obtain multiple Group matching feature points.

In conjunction with the second aspect, in some implementations of the second aspect, the processing module is further configured to: use triangulation based on the pixel coordinates of multiple sets of matching feature points in the corresponding original image and the internal camera parameters corresponding to multiple original images method to determine the relative pose of the camera corresponding to each original image; Set the 3D coordinates of the matching feature points.

In combination with the second aspect, in some implementations of the second aspect, the processing module is further configured to: input the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image into the depth estimation network model, and obtain The depth information of the corresponding matching feature points on each original image.

With reference to the second aspect, in some implementations of the second aspect, the processing module is further configured to: input the depth information of the corresponding matching feature points on each original image and multiple original images to the illumination estimation network model, and obtain each The ambient light map corresponding to the original image.

With reference to the second aspect, in some implementations of the second aspect, the processing module is further configured to: determine the pixel coordinate with the smallest pixel amplitude in the environment light map corresponding to each original image as the virtual light source in each original image The corresponding pixel coordinates; and, based on the pixel coordinates corresponding to the virtual light source in each original image and the relative camera pose corresponding to each original image, determine the three-dimensional coordinates of the virtual light source.

In a third aspect, another image processing device is provided, including a processor and a memory. The processor is used to read instructions stored in the memory, and may receive signals through the receiver and transmit signals through the transmitter, so as to execute the method in any possible implementation manner of the first aspect above.

Optionally, there are one or more processors, and one or more memories.

Optionally, the memory may be integrated with the processor, or the memory may be separated from the processor.

In a specific implementation process, the memory may be a non-transitory (non-transitory) memory, such as a read-only memory (read only memory, ROM), which may be integrated with the processor on the same chip, or may be respectively arranged in different On the chip, the embodiment of the present application does not limit the type of the memory and the configuration of the memory and the processor.

The image processing device in the third aspect above can be a chip, and the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; When implemented, the processor may be a general-purpose processor, which is realized by reading software codes stored in a memory, and the memory may be integrated in the processor, or may be located outside the processor and exist independently.

In a fourth aspect, a computer-readable storage medium is provided, the computer-readable storage medium stores a computer program (also referred to as code, or an instruction), and when it runs on a computer, it causes the computer to perform the above-mentioned first aspect. A method in any of the possible implementations.

In a fifth aspect, a computer program product is provided, and the computer program product includes: a computer program (also referred to as code, or an instruction), when the computer program is executed, it causes the computer to perform any one of the possibilities in the first aspect above. method in the implementation.

Description of drawings

FIG. 1 is a schematic diagram of an application scenario of an embodiment of the present application;

FIG. 2 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

Fig. 3 is a schematic diagram of the position of the virtual light source provided by the embodiment of the present application;

Fig. 4 is a schematic flowchart of another image processing method provided by the embodiment of the present application;

Fig. 5 is a schematic block diagram of an image processing device provided by an embodiment of the present application;

Fig. 6 is a schematic block diagram of another image processing apparatus provided by an embodiment of the present application.

Detailed ways

The technical solution in this application will be described below with reference to the accompanying drawings.

In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with basically the same function and effect. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

It should be noted that, in this application, words such as "exemplarily" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a concrete manner.

In addition, "at least one" means one or more, and "plurality" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (one) of a, b and c may represent: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, wherein a, b, c can be single or multiple.

3D reconstruction technology refers to the use of computer technology to reconstruct real world scenes or objects into data models that can be expressed and processed by computers. Image-based 3D reconstruction is more and more widely used due to its low requirements for acquisition equipment and low cost of the reconstruction process.

In order to facilitate the understanding of the present application, the application scenario involved in the embodiment of the present application will be introduced below with reference to FIG. 1 .

FIG. 1 shows a schematic diagram of an application scenario 100 of an embodiment of the present application. The application scenario includes an image acquisition device 101 and an image processing device 102 . The image acquisition device 101 may acquire images under different viewing angles of the three-dimensionally reconstructed target object or a scene where the target object is located, and send the images under different viewing angles to the image processing device 102 . Correspondingly, after the image processing device 102 receives the above-mentioned images under different viewing angles, it may use the images under different viewing angles to perform three-dimensional reconstruction of the target object or the scene where the target object is located.

Optionally, the above-mentioned image acquisition device 101 can use the camera to take pictures of the target object, and directly collect multiple images of different viewing angles; it can also use the camera to perform video recording, and collect the video that includes the pictures of the target object under different viewing angles, and then display the images in the video. Images under different viewing angles including the target object are clipped; network pictures of the same target object under different viewing angles may also be obtained from the Internet, which is not limited in this embodiment of the present application.

It should be understood that the above-mentioned image acquisition device 101 may specifically be a device including a camera, and the above-mentioned image processing device 102 may specifically be a device having a data processing capability. It should also be understood that, in the foregoing application scenario, there may be one or more image acquisition devices, which is not limited in this embodiment of the present application.

The above Figure 1 only shows one possible scenario. In other possible scenarios, the image acquisition device and the image processing device can be combined into one physical device. For the convenience of description, this embodiment of the application refers to it as an image processing device. equipment. In other words, the image processing device in the embodiment of the present application may have the function of image collection, collect images from different perspectives of the target object by itself, or receive images collected from other devices, which is not limited in the embodiment of the present application.

At present, the process of image processing equipment for 3D reconstruction based on images from different perspectives may include: extracting feature points in multi-view images, completing feature point matching, performing sparse point cloud reconstruction based on motion restoration structure technology, and performing multi-view stereo technology. Dense reconstruction. Usually, after the sparse point cloud is reconstructed, the mean value of the color of the same feature point in the image under different viewing angles is taken as the corresponding 3D point color, and the sparse point cloud is colored. Based on the sparse point cloud after coloring, dense reconstruction is performed through multi-view stereo technology to improve the accuracy of 3D reconstruction results.

However, due to the surface material of the target object, the position or color of the ambient light when the image of the target object is collected, the images under different viewing angles may have the phenomenon that the texture of the target object is not clear, the color is not real, and the texture is inconsistent under multiple viewing angles. The mean value of the colors of the points in the images under different viewing angles does not represent the true color of the target object, therefore, the accuracy of the 3D reconstruction results of the above methods is poor.

In view of this, the embodiment of the present application proposes an image processing method and device. By considering the influence of ambient light on the target object, the environment light map is introduced during 3D reconstruction, and the three-dimensional coordinates of the virtual light source are determined by using the environment light map. Based on The 3D coordinates of the virtual light source and the ambient light map perform light compensation on the original image to obtain multiple light compensation images. Since the light compensation image contains the influence of the environmental light on the target object, the three-dimensional reconstruction of the target object is performed using multiple light compensation images, improving Accuracy of 3D reconstruction results.

The image processing method provided by the embodiment of the present application will be described below with reference to FIG. 2 to FIG. 3 .

FIG. 2 is a schematic flow chart of an image processing method 200 provided by an embodiment of the present application. The method 200 can be executed by the image processing device 102 shown in FIG. 1 or by other similar devices. The embodiment of the present application is specific to This is not limited. For ease of description, they are collectively referred to as an image processing device in this embodiment of the application. The method 200 includes the following steps:

S201. Acquire multiple original images of a target object, where the multiple original images are respectively obtained by shooting the target object under different viewing angles.

It should be understood that the above multiple original images may be captured by the image processing device itself (if the image processing device has an image acquisition function), or may be captured by the image acquisition device. In addition, the above multiple original images may be panoramic images captured by a panoramic camera, or wide-angle images, or images captured by a common camera, which is not limited in this application.

S202. Obtain multiple sets of matching feature points based on multiple original images, and determine the three-dimensional coordinates of the multiple sets of matching feature points and the relative camera position corresponding to each original image based on the pixel coordinates of the multiple sets of matching feature points in the corresponding original images. posture.

It should be understood that the image processing device can extract multiple feature points on multiple original images through a feature extraction algorithm, and use a feature matching strategy to match multiple feature points of each original image on multiple original images to obtain multiple Group matching feature points.

The above matching feature points can be matching feature points in two images, for example, there are feature points _1-1 and _1-2 in the first original image, and feature points _2-1 and _2-2 in the second original image. _2. The feature point _1-1 and the feature point _2-1 are imaging points of the same physical space point in the above two original images, therefore, the feature point _1-1 and the feature point _2-1 are a group of matching feature points.

The above-mentioned matching feature points can also be matching feature points in more than two images, for example, there are feature points _1-1 and feature points _1-2 in the first original image, and feature points _2-1 and feature points in the second original image _2-2 , the third original image has feature point _3-1 and feature point _3-2 , feature point _1-1 , feature point _2-1 and feature point _3-1 are the same physical space point in the above three original images Therefore, feature point _1-1 , feature point _2-1 and feature point _3-1 are a group of matching feature points.

In the above example, the image processing device can determine the pixel coordinates of feature point _1-1 in the first original image, the pixel coordinates of feature point _2-1 in the second original image, and the pixel coordinates of feature point _3-1 in the third original image. Pixel coordinates in the image. Since feature point _1-1 , feature point _2-1 and feature point _3-1 are the imaging points of the same physical space point in the above three original images, the three-dimensional coordinates of this group of matching feature points are the corresponding The three-dimensional coordinates of the same physical space point.

The above example is only described by taking a set of matching feature points as an example. In actual operation, multiple original images may have one set of matching feature points or multiple sets of matching feature points, which is not limited in this embodiment of the present application.

S203. Based on the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image, determine the depth information of the corresponding matching feature points on each original image.

It should be understood that the depth information is the vertical distance from each group of matching feature points to the imaging plane of the corresponding image acquisition device (such as a camera). Exemplarily, there is a feature point _1-1 in the first original image, and the feature point _1-1 and the feature point _2-1 in the second original image are a set of matching feature points, and the feature point _1-1 in the first original image ₁ The vertical distance from the camera imaging plane is the depth information of the feature point.

S204. Based on the depth information of the corresponding matching feature points on each original image and the multiple original images, determine an ambient light map corresponding to each original image.

It should be understood that the ambient light map is an image used to represent the brightness information of each pixel in the original image under the influence of the ambient light source. Each of the above multiple original images has a corresponding ambient light map, that is, multiple original light maps can be obtained in this step.

S205. Determine the three-dimensional coordinates of the virtual light source based on the ambient light map corresponding to each original image and the relative camera pose corresponding to each original image.

It should be understood that the number of virtual light sources may be one or multiple, and correspondingly, the number of three-dimensional coordinates of virtual light sources may be one or multiple, which is not limited in this embodiment of the present application.

Optionally, if the above-mentioned multiple original images are collected under outdoor natural light scenes, the number of virtual light sources can be one; if the above-mentioned multiple original images are collected under outdoor natural light or indoor scenes , the number of virtual light sources can be one or more.

S206. Based on the three-dimensional coordinates of the virtual light source and the ambient light map corresponding to each original image, perform light compensation on each original image to obtain multiple light compensation images.

S207. Determine a 3D model of the target object based on the multiple illumination compensation images and the 3D coordinates of multiple sets of matching feature points.

In the image processing method of the embodiment of the present application, by considering the impact of ambient light on the target object, the environment light map is introduced during 3D reconstruction, and the three-dimensional coordinates of the virtual light source are determined by using the environment light map. Based on the three-dimensional coordinates of the virtual light source and the environment light map Light compensation is performed on the original image to obtain multiple light compensation images. Since the light compensation image contains the influence of ambient light on the target object, using multiple light compensation images to perform 3D reconstruction of the target object can make the 3D model obtained after 3D reconstruction The color is closer to the real color of the target object in the actual ambient lighting scene, which is conducive to improving the accuracy of the 3D reconstruction results, thereby improving the user experience.

As an optional embodiment, the above method also includes:

Based on the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source and the three-dimensional coordinates of multiple sets of matching feature points, determine and update the three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of the multiple sets of matching feature points;

S206. Based on the three-dimensional coordinates of the virtual light source and the ambient light map corresponding to each original image, perform illumination compensation for each original image, including:

Based on updating the three-dimensional coordinates of the virtual light source and the ambient light map corresponding to each original image, perform illumination compensation for each original image;

S207. Determine the 3D model of the target object based on the multiple illumination compensation images and the 3D coordinates of the multiple sets of matching feature points, including:

The three-dimensional model of the target object is determined based on the updated three-dimensional coordinates of the multiple illumination compensation images and the multiple sets of matching feature points.

The image processing method of the embodiment of the present application performs illumination compensation on each original image by determining and updating the three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of multiple sets of matching feature points, and based on the image after illumination compensation and the multiple sets of matching feature points Updating the three-dimensional coordinates, determining the three-dimensional model of the target object, the updated three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of the multiple sets of matching feature points make the error of the updated three-dimensional coordinates of the three-dimensional coordinates of the virtual light source and the multiple sets of matching feature points smaller than that of the non-updated three-dimensional coordinates of the virtual light source, It is beneficial to improve the accuracy of 3D reconstruction results.

It should be understood that updating the three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of the multiple sets of matching feature points may also be referred to as the precise three-dimensional coordinates of the virtual light source and the precise three-dimensional coordinates of the multiple sets of matching feature points.

It should be understood that the image processing device has errors in the process of calculating the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source, and the three-dimensional coordinates of multiple sets of matching feature points, which affects the accuracy of the three-dimensional model of the target object. Since the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source, and the three-dimensional coordinates of multiple sets of matching feature points affect each other, whether the relative pose of the camera corresponding to each original image is updated will affect the three-dimensional coordinates of the virtual light source and the multi-dimensional coordinates of the virtual light source. The image processing equipment can adjust the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source and the three-dimensional coordinates of multiple matching feature points, and obtain the precise camera corresponding to each original image. The relative pose, the precise 3D coordinates of the virtual light source and the precise 3D coordinates of multiple matching feature points ultimately improve the accuracy of the 3D model of the target object.

As an optional embodiment, the above step S206 is performing illumination compensation on each original image to obtain multiple illumination compensation images, including: determining the updated pixel coordinates of the virtual light source in each original image based on the updated three-dimensional coordinates of the virtual light source; Based on the difference between the updated pixel coordinates of the virtual light source in each original image and the corresponding pixel coordinates of the virtual light source in each original image, the pixels in the ambient light map corresponding to each original image are displaced to obtain The ambient light compensation map corresponding to each original image; based on the ambient light compensation map corresponding to each original image, light compensation is performed on each original image to obtain multiple light compensation images.

Exemplarily, the image processing device may map the ambient illumination compensation map corresponding to each original image to the corresponding original image, perform illumination compensation on each original image through a back-projection operation, and obtain multiple illumination compensation images.

As an optional embodiment, in the above S207, determining the 3D model of the target object based on multiple illumination compensation images and updated 3D coordinates of multiple sets of matching feature points includes: determining the updated 3D coordinates of multiple sets of matching feature points as None color sparse 3D point cloud; obtain the pixel coordinates of multiple sets of matching feature points in corresponding multiple illumination compensation images; use the pixel coordinates of multiple sets of matching feature points in corresponding multiple illumination compensation images to perform The cloud is colored to obtain a colored sparse 3D point cloud; based on the colored sparse 3D point cloud and multiple illumination compensation images, the 3D model of the target object is determined.

Exemplarily, the image processing device may color the colorless sparse 3D point cloud by using the mean value of the pixel coordinates of multiple sets of matching feature points in the corresponding multiple illumination compensation images to obtain the colored sparse 3D point cloud.

As an optional embodiment, the above S202, obtaining multiple sets of matching feature points based on multiple original images, includes: extracting feature points of each original image in multiple original images; performing feature point matching on multiple original images to obtain Multiple sets of matching feature points.

It should be understood that the image processing device may extract the feature points of each original image in the multiple original images through a feature extraction algorithm. Exemplarily, the feature extraction algorithm may be a scale-invariant feature transform algorithm (scale-invariant feature transform, SIFT), an accelerated robust feature algorithm (speeded up robust features, SURF), a corner detection algorithm (features from accelerated segment test, FAST) Or a feature point extraction algorithm such as a fast feature point extraction and description algorithm (oriented fast and rotated brief, ORB) algorithm extracts the feature points of each original image in multiple original images, which is not limited in this embodiment of the present application.

It should be understood that the image processing device may use a feature matching strategy to perform feature point matching on multiple feature points extracted from multiple original images to obtain multiple sets of matching feature points. Exemplarily, the feature matching strategy may be a brute force matching strategy, a K-nearest neighbor (KNN) matching strategy, etc., which are not limited in this embodiment of the present application.

Through appropriate feature extraction algorithms and feature matching strategies, image processing equipment can more accurately extract feature points, and can obtain more accurate feature matching results, that is, the above-mentioned multiple sets of matching feature points, which is conducive to improving the subsequent 3D reconstruction results accuracy.

As an optional embodiment, in the above S202, determining the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image includes: based on the pixel coordinates of multiple sets of matching feature points in the corresponding original image and The camera internal parameters corresponding to multiple original images, use the triangulation method to determine the relative camera pose corresponding to each original image; based on the pixel coordinates of multiple sets of matching feature points in the corresponding original image and the camera relative to each original image Pose, using the triangulation method to determine the three-dimensional coordinates of multiple sets of matching feature points.

Exemplarily, there is a feature point _1-1 in the first original image, and a feature point _2-1 in the second original image, and the feature point _1-1 and the feature point _2-1 are a set of matching feature points. Based on the pixel coordinates of feature point _1-1 in the first original image, the pixel coordinates of feature point _2-1 in the second original image, the camera relative pose of the first original image (such as the preset unit matrix) and the camera The internal reference, through the triangulation method, can obtain the relative pose of the camera corresponding to the second original image. Based on the relative pose of the camera corresponding to the second original image, the pixel coordinates of feature point _1-1 in the first original image and the pixel coordinates of feature point _2-1 in the second original image, the matching can be determined by triangulation method The three-dimensional coordinates of the feature points.

Exemplarily, there are feature points _1-1 in the first original image, feature points _2-1 in the second original image, feature points _3-1 in the third original image, feature points _1-1 , feature points _{2- 1} and feature point _3-1 are a set of matching feature points. Based on the pixel coordinates of feature point _1-1 in the first original image, the pixel coordinates of feature point _2-1 in the second original image, the camera relative pose of the first original image (such as the preset unit matrix) and the camera As for the internal reference, the relative pose of the camera corresponding to the second original image and the three-dimensional coordinates P1' of the matching feature points can be determined through the above-mentioned triangulation method. Based on the pixel coordinates of feature point _1-1 in the first original image, the pixel coordinates of feature point _3-1 in the third original image, the camera relative pose of the first original image (such as the preset unit matrix) and the camera The internal reference, through the above triangulation method, can determine the relative pose of the camera corresponding to the third original image and the three-dimensional coordinate P1" of the matching feature point. Based on the three-dimensional coordinates P1' and P1", through the joint equation, determine P1 as the corresponding matching feature The 3D coordinates of the point.

As an optional embodiment, in the above S203, determining the depth information of the corresponding matching feature points on each original image includes: inputting the three-dimensional coordinates of multiple sets of matching feature points and the relative camera poses corresponding to each original image to The depth estimation network model obtains the depth information of the corresponding matching feature points on each original image.

Exemplarily, the depth estimation network model may be a convolutional neural network (convolutional neural networks, CNN).

As an optional embodiment, in the above S204, determining the ambient light map corresponding to each original image includes: inputting the depth information of the corresponding matching feature points on each original image and multiple original images to the illumination estimation network model, Obtain the ambient light map corresponding to each original image.

Exemplarily, the illumination estimation network model may be Gardner's illumination estimation network model.

As an optional embodiment, the above S205, determining the three-dimensional coordinates of the virtual light source includes: determining the pixel coordinate with the smallest pixel amplitude in the environment light map corresponding to each original image as the pixel corresponding to the virtual light source in each original image Coordinates: Determine the three-dimensional coordinates of the virtual light source based on the pixel coordinates corresponding to each original image of the virtual light source and the relative pose of the camera corresponding to each original image.

It should be understood that the pixel magnitude indicates the brightness of a single pixel point. Exemplarily, the range of the pixel amplitude is 0-256, the pixel amplitude value of 0 can be used to indicate white, and the pixel amplitude value of 256 can be used to indicate black.

Exemplarily, Fig. 3a is an environment light map, in which there is a virtual light source P, and Fig. 3b is the position of the virtual light source P in the camera coordinate system. Point P in Figure 3a is the point whose pixel amplitude is closest to 0, and point P is determined to be the position of the virtual light source in the environment light map in Figure 3a. The pixel coordinates of the virtual light source in the environment light map in Figure 3a can be transformed from the pixel coordinate system to the image coordinate system to the camera coordinate system to obtain the virtual light source P coordinates in the camera coordinate system as shown in Figure 3b.

Optionally, based on the pixel coordinates of the virtual light source corresponding to each original image and the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source are determined through a triangulation method.

Hereinafter, with reference to FIG. 4 , the embodiment of the present application will be described in detail by taking 10 original images of different viewing angles collected under outdoor natural light as an example.

Fig. 4 is a schematic flowchart of another image processing method 400 provided by the embodiment of the present application. The method 400 can be executed by the image processing device 102 shown in Fig. 1, or by other similar devices. The embodiment of the present application There is no limit to this. The method 400 includes the following steps:

S401. The image processing device acquires 10 original images of a target object, where the 10 original images are obtained by shooting the target object under different viewing angles.

S402. The image processing device extracts a plurality of feature points from each of the above 10 original images, and obtains a plurality of feature point information of each original image, and the plurality of feature point information includes feature points in the original image. The pixel coordinates of .

Exemplarily, the image processing device may extract a plurality of feature points on each of the above ten original images by using a SIFT algorithm, a SURF algorithm, a FAST algorithm or an ORB algorithm.

S403. The image processing device performs feature point matching on the 10 original images based on the multiple feature point information of the above 10 original images to obtain multiple sets of matching feature points. Matching feature points can be understood as imaging points of the same physical space point of the target object in multiple original images.

Exemplarily, the above image processing device may use a brute-force matcher (BFM) strategy to match multiple feature points of the above ten original images to obtain multiple sets of matching feature points.

S404. Based on the pixel coordinates of the multiple sets of matching feature points in the corresponding original image and the camera internal parameters corresponding to the original image, the image processing device obtains the three-dimensional coordinates of the multiple sets of matching feature points and the relative camera pose corresponding to each original image.

Optionally, the image processing device may obtain the relative pose of the camera through a triangulation method based on the pixel coordinates of the matching feature points in the corresponding original images in each two original images and the internal camera parameters. The image processing device can obtain the three-dimensional coordinates of the matching feature points in each two original images based on the obtained relative pose of the camera and the pixel coordinates of the matching feature points in the corresponding original images through a triangulation method. coordinate.

S405. Based on the three-dimensional coordinates of the matching feature points in each of the above two original images and the relative camera pose corresponding to each of the above original images, the image processing device obtains the matching feature points in the corresponding original image through a depth estimation network model. in-depth information.

Exemplarily, the depth estimation network model may be CNN.

S406. Based on the above 10 original images and the depth information of the corresponding matching feature points on the 10 original images, the image processing device uses the illumination estimation network model to obtain the environment illumination map corresponding to the 10 original images.

Exemplarily, the aforementioned illumination estimation network model may be Gardner's illumination estimation network model.

S407. The image processing device determines, based on the environmental light maps corresponding to the 10 original images, the pixel coordinate with the smallest pixel amplitude in the environmental light map as the pixel coordinate corresponding to the virtual light source in each of the original images.

Optionally, the image processing device may divide each environment light map into connected regions according to the pixel amplitude of the pixel points in the environment light map, and find the point with the smallest pixel amplitude in the area less than or equal to the preset threshold value, And the position of this point is used as the position of the virtual point light source. The range of pixel amplitude: 0-256, when the pixel amplitude is 0, it is white, and when the pixel amplitude is 256, it is black.

S408. The image processing device calculates the three-dimensional coordinates of the virtual light source based on the relative camera poses corresponding to the 10 original images and the pixel coordinates corresponding to the virtual light source in each original image.

Optionally, the image processing device may calculate the three-dimensional coordinates of the virtual light source by triangulation based on the relative camera poses corresponding to the 10 original images and the pixel coordinates corresponding to the virtual light source in each original image.

S409. The image processing device calculates and updates the three-dimensional coordinates of the virtual light source and the updates of the multiple sets of matching feature points based on the relative poses of the cameras corresponding to the 10 original images, the three-dimensional coordinates of the virtual light source, and the three-dimensional coordinates of multiple sets of matching feature points, using a bundle optimization equation. 3D coordinates.

S410. The image processing device performs light compensation on the corresponding 10 original images based on the updated three-dimensional coordinates of the virtual light source and the environmental light map corresponding to the above 10 original images, and obtains 10 images after light compensation.

S411. The image processing device determines the updated three-dimensional coordinates of the multiple sets of matching feature points as a colorless sparse three-dimensional point cloud, obtains the pixel coordinates of the multiple sets of matching feature points in the corresponding 10 illumination compensation images, and calculates the coordinates of the multiple sets of matching feature points in The average value of the pixel coordinates in the corresponding multiple illumination compensation images is used to color the colorless sparse 3D point cloud to obtain the colored sparse 3D point cloud.

S412. The image processing device obtains a 3D model of the target object based on the 10 light-compensated images and the colored sparse 3D point cloud.

Optionally, the image processing device obtains the 3D model of the target object through a dense reconstruction algorithm based on the 10 light-compensated images and the colored sparse 3D point cloud. The dense reconstruction algorithm may be a multi-view stereo vision (multi-view stereo, MVS) algorithm.

The image processing method of the embodiment of the present application performs illumination compensation on each original image by determining and updating the three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of multiple sets of matching feature points, and based on the image after illumination compensation and the multiple sets of matching feature points Updating the three-dimensional coordinates, determining the three-dimensional model of the target object, the updated three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of the multiple sets of matching feature points make the error of the updated three-dimensional coordinates of the three-dimensional coordinates of the virtual light source and the multiple sets of matching feature points smaller than that of the non-updated three-dimensional coordinates of the virtual light source, It is beneficial to improve the accuracy of the 3D reconstruction results.

The image processing method of the embodiment of the present application is described in detail above with reference to FIG. 2 to FIG. 4 , and the image processing device of the embodiment of the present application will be described in detail below in conjunction with FIG. 5 and FIG. 6 .

FIG. 5 shows an image processing device 500 provided by an embodiment of the present application, and the image processing device 500 includes: an acquisition module 501 and a processing module 502 .

The obtaining module 501 is used to obtain multiple original images of the target object, and the multiple original images are respectively obtained by shooting the target object under different viewing angles; the processing module 502 is used to obtain multiple matching groups based on the multiple original images feature points, and based on the pixel coordinates of multiple sets of matching feature points in the corresponding original image, determine the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image; based on the three-dimensional coordinates of multiple sets of matching feature points Based on the relative pose of the camera corresponding to each original image, determine the depth information of the corresponding matching feature points on each original image; based on the depth information of the corresponding matching feature points on each original image and multiple original images, determine each The environmental light map corresponding to the original image; based on the environmental light map corresponding to each original image and the relative camera pose corresponding to each original image, determine the three-dimensional coordinates of the virtual light source; based on the three-dimensional coordinates of the virtual light source and the environmental light corresponding to each original image In the figure, light compensation is performed on each original image to obtain multiple light compensation images; and, based on the multiple light compensation images and the three-dimensional coordinates of multiple sets of matching feature points, the three-dimensional model of the target object is determined.

Optionally, the processing module 502 is further configured to: determine and update the three-dimensional coordinates of the virtual light source and the three-dimensional coordinates of the multiple sets of matching feature points based on the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source, and the multiple sets of matching feature points Update the three-dimensional coordinates; based on the updated three-dimensional coordinates of the virtual light source and the environmental light map corresponding to each original image, perform illumination compensation on each original image; and, based on the updated three-dimensional coordinates of multiple illumination compensation images and multiple sets of matching feature points, determine A three-dimensional model of the target object.

Optionally, the processing module 502 is further configured to: determine the updated pixel coordinates of the virtual light source in each original image based on the updated three-dimensional coordinates of the virtual light source; The difference between the corresponding pixel coordinates in each original image, and the pixel points in the environmental light map corresponding to each original image are displaced to obtain the environmental light compensation map corresponding to each original image; based on the corresponding to each original image Ambient light compensation map, light compensation is performed on each original image separately, and multiple light compensation images are obtained.

Optionally, the processing module 502 is further configured to: determine the updated 3D coordinates of multiple groups of matching feature points as a colorless sparse 3D point cloud; the obtaining module 501 is also configured to: obtain multiple groups of matching The pixel coordinates in the image; the processing module is also used to: use the pixel coordinates of multiple sets of matching feature points in the corresponding multiple illumination compensation images to color the colorless sparse 3D point cloud to obtain the colored sparse 3D point cloud; based on coloring Sparse 3D point cloud and multiple light compensation images to determine the 3D model of the target object.

Optionally, the processing module 502 is further configured to: extract feature points of each of the multiple original images; and perform feature point matching on the multiple original images to obtain multiple sets of matching feature points.

Optionally, the processing module 502 is also configured to: determine the camera relative to each original image based on the pixel coordinates of multiple groups of matching feature points in the corresponding original image and the internal camera parameters corresponding to the multiple original images, using the triangulation method pose; and, based on the pixel coordinates of the multiple sets of matching feature points in the corresponding original images and the relative camera poses corresponding to each original image, using a triangulation method to determine the three-dimensional coordinates of the multiple sets of matching feature points.

Optionally, the processing module 502 is also configured to: input the three-dimensional coordinates of multiple sets of matching feature points and the relative camera pose corresponding to each original image into the depth estimation network model, and obtain the corresponding matching feature points on each original image. depth information.

Optionally, the processing module 502 is further configured to: input the depth information of corresponding matching feature points on each original image and multiple original images to the illumination estimation network model to obtain an environment illumination map corresponding to each original image.

Optionally, the processing module 502 is further configured to: determine the pixel coordinate with the smallest pixel amplitude in the ambient light map corresponding to each original image as the pixel coordinate corresponding to the virtual light source in each original image; and, based on the virtual light source in The pixel coordinates corresponding to each original image and the relative pose of the camera corresponding to each original image determine the three-dimensional coordinates of the virtual light source.

It should be understood that the device 500 here is embodied in the form of functional modules. The term "module" herein may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processor, etc.) and memory, incorporated logic, and/or other suitable components to support the described functionality. In an optional example, those skilled in the art can understand that the device 500 may specifically be the image processing device in the above-mentioned embodiment, and the device 500 may be used to execute each process corresponding to the image processing device in the above-mentioned method embodiment and/or The steps are not repeated here to avoid repetition.

The above device 500 has the function of implementing the corresponding steps performed by the image processing device in the above method; the above function can be realized by hardware, or by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions.

In the embodiment of the present application, the device 500 in FIG. 5 may also be a chip or a chip system, such as a system on chip (system on chip, SoC).

FIG. 6 shows another image processing device 600 provided by an embodiment of the present application. The image processing device 600 includes a processor 601 , a transceiver 602 and a memory 603 . Wherein, the processor 601, the transceiver 602 and the memory 603 communicate with each other through an internal connection path, the memory 603 is used to store commands, and the processor 601 is used to execute the instructions stored in the memory 603 to control the transceiver 602 to send signals and/or or receive signals.

It should be understood that the image processing device 600 may specifically be the image processing device in the foregoing embodiments, and may be configured to execute various steps and/or processes corresponding to the image processing device in the foregoing method embodiments. Optionally, the memory 603 may include a read-only memory and a random access memory, and provides instructions and data to the processor 601 . A portion of memory 603 may also include non-volatile random access memory. For example, the memory 603 may also store information of device types. The processor 601 may be used to execute the instructions stored in the memory, and when the processor 501 executes the instructions stored in the memory, the processor 601 is used to execute the steps of the above-mentioned method embodiments corresponding to the image processing device and/or or process. The transceiver 602 may include a transmitter and a receiver, the transmitter may be used to implement various steps and/or processes corresponding to the above transceiver for performing sending actions, and the receiver may be used to implement the application corresponding to the above transceiver Various steps and/or processes for performing receiving actions.

It should be understood that, in the embodiment of the present application, the processor 601 may be a central processing unit (central processing unit, CPU), and the processor 601 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits ( ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the methods disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory, and the processor executes the instructions in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, no detailed description is given here.

The present application also provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, and the computer program is used to implement the method corresponding to the image processing device in the foregoing embodiments.

The present application also provides a computer program product, which includes a computer program (also referred to as code, or instruction). When the computer program is run on a computer, the computer can execute the above-mentioned embodiment and image processing. The method corresponding to the device.

Those skilled in the art can appreciate that the modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device, and module can refer to the corresponding process in the foregoing method embodiment, and details are not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, each module may exist separately physically, or two or more modules may be integrated into one module.

If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other various media that can store program codes. .

The above is only the specific implementation of the application, but the protection scope of the embodiment of the application is not limited thereto, and any skilled person familiar with the technical field can easily think of changes within the technical scope disclosed in the embodiment of the application Or replacement, should be covered within the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims (21)

1. An image processing method, characterized in that, comprising: Acquiring a plurality of original images of the target object, the plurality of original images are respectively obtained by shooting the target object under different viewing angles; Multiple sets of matching feature points are obtained based on the multiple original images, and based on the pixel coordinates of the multiple sets of matching feature points in the corresponding original images, determine the three-dimensional coordinates of the multiple sets of matching feature points and each of the original The relative pose of the camera corresponding to the image; Based on the three-dimensional coordinates of the plurality of sets of matching feature points and the relative camera pose corresponding to each original image, determine the depth information of the corresponding matching feature points on each original image; Based on the depth information of the corresponding matching feature points on each original image and the plurality of original images, determine the ambient light map corresponding to each original image; Determine the three-dimensional coordinates of the virtual light source based on the ambient light map corresponding to each original image and the camera relative pose corresponding to each original image; performing illumination compensation on each original image based on the three-dimensional coordinates of the virtual light source and the environmental illumination map corresponding to each original image, to obtain multiple illumination compensation images; A three-dimensional model of the target object is determined based on the plurality of illumination compensation images and the three-dimensional coordinates of the plurality of matching feature points. 2. The method according to claim 1, characterized in that the method further comprises: Based on the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source and the three-dimensional coordinates of the multiple sets of matching feature points, determine and update the three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of the multiple sets of matching feature points ; The performing illumination compensation on each original image based on the three-dimensional coordinates of the virtual light source and the ambient light map corresponding to each original image includes: performing illumination compensation on each original image based on the three-dimensional coordinates of the updated virtual light source and the ambient light map corresponding to each original image; The determining the 3D model of the target object based on the multiple illumination compensation images and the 3D coordinates of the multiple sets of matching feature points includes: A three-dimensional model of the target object is determined based on the multiple illumination compensation images and the updated three-dimensional coordinates of the multiple sets of matching feature points. 3. The method according to claim 2, wherein said performing illumination compensation on said each original image to obtain multiple illumination compensation images comprises: Determine updated pixel coordinates of the virtual light source in each original image based on the updated three-dimensional coordinates of the virtual light source; Based on the difference between the updated pixel coordinates of the virtual light source in each original image and the corresponding pixel coordinates of the virtual light source in each original image, the environment corresponding to each original image The pixels in the illumination map are displaced to obtain the ambient illumination compensation map corresponding to each original image; Based on the ambient light compensation map corresponding to each original image, light compensation is performed on each original image to obtain the plurality of light compensation images. 4. The method according to claim 2 or 3, wherein the determination of the three-dimensional model of the target object based on the updated three-dimensional coordinates of the plurality of illumination compensation images and the plurality of sets of matching feature points includes : Determining the updated three-dimensional coordinates of the multiple groups of matching feature points as a colorless sparse three-dimensional point cloud; Obtain the pixel coordinates of the plurality of matching feature points in the corresponding plurality of illumination compensation images; Coloring the colorless sparse 3D point cloud by using the pixel coordinates of the multiple groups of matching feature points in the corresponding multiple illumination compensation images to obtain a colored sparse 3D point cloud; A three-dimensional model of the target object is determined based on the colored sparse three-dimensional point cloud and the plurality of illumination compensation images. 5. The method according to any one of claims 1 to 4, wherein said obtaining multiple sets of matching feature points based on said multiple original images comprises: Extracting feature points of each original image in the plurality of original images; Perform feature point matching on the multiple original images to obtain the multiple sets of matching feature points. 6. The method according to any one of claims 1 to 5, wherein the determining the three-dimensional coordinates of the plurality of matching feature points and the relative camera pose corresponding to each original image comprises: Based on the pixel coordinates of the plurality of sets of matching feature points in the corresponding original images and the camera internal parameters corresponding to the plurality of original images, using a triangulation method to determine the relative camera pose corresponding to each original image; Based on the pixel coordinates of the multiple sets of matching feature points in the corresponding original images and the relative camera pose corresponding to each original image, the three-dimensional coordinates of the multiple sets of matching feature points are determined by using a triangulation method. 7. The method according to any one of claims 1 to 6, wherein the determining the depth information of the corresponding matching feature points on each original image comprises: The three-dimensional coordinates of the multiple sets of matching feature points and the relative camera poses corresponding to each original image are input to the depth estimation network model to obtain the depth information of the corresponding matching feature points on each original image. 8. The method according to any one of claims 1 to 7, wherein the determining the ambient light map corresponding to each original image comprises: Inputting the depth information of the corresponding matching feature points on each original image and the plurality of original images to an illumination estimation network model to obtain an environment illumination map corresponding to each original image. 9. The method according to any one of claims 1 to 8, wherein said determining the three-dimensional coordinates of the virtual light source comprises: Determining the pixel coordinate with the smallest pixel amplitude in the ambient light map corresponding to each original image as the pixel coordinate corresponding to the virtual light source in each original image; The three-dimensional coordinates of the virtual light source are determined based on the pixel coordinates of the virtual light source corresponding to each original image and the camera relative pose corresponding to each original image. 10. An image processing device, comprising: An acquisition module, configured to acquire multiple original images of the target object, where the multiple original images are respectively obtained by shooting the target object under different viewing angles; A processing module, configured to obtain multiple sets of matching feature points based on the multiple original images, and determine the three-dimensional coordinates and sum of the multiple sets of matching feature points based on the pixel coordinates of the multiple sets of matching feature points in the corresponding original images The relative camera pose corresponding to each original image; based on the three-dimensional coordinates of the multiple sets of matching feature points and the relative camera pose corresponding to each original image, determine the corresponding matching feature on each original image Point depth information; based on the depth information of the corresponding matching feature points on each original image and the plurality of original images, determine the corresponding environmental light map of each original image; based on the corresponding The environment light map corresponding to each original image and the camera relative pose, determine the three-dimensional coordinates of the virtual light source; based on the three-dimensional coordinates of the virtual light source and the environment light map corresponding to each original image, each original Perform illumination compensation on the image to obtain multiple illumination compensation images; and determine a three-dimensional model of the target object based on the multiple illumination compensation images and the three-dimensional coordinates of the multiple sets of matching feature points. 11. The device according to claim 10, wherein the processing module is also used for: Based on the relative pose of the camera corresponding to each original image, the three-dimensional coordinates of the virtual light source and the three-dimensional coordinates of the multiple sets of matching feature points, determine and update the three-dimensional coordinates of the virtual light source and the updated three-dimensional coordinates of the multiple sets of matching feature points ; performing illumination compensation on each original image based on the updated three-dimensional coordinates of the virtual light source and the environmental illumination map corresponding to each original image; and, based on the multiple illumination compensation images and the multiple sets of matching features The updated 3D coordinates of the points determine the 3D model of the target object. 12. The device according to claim 11, wherein the processing module is further used for: Based on the updated three-dimensional coordinates of the virtual light source, determine the updated pixel coordinates of the virtual light source in each of the original images; The difference between the corresponding pixel coordinates in each of the original images is used to displace the pixels in the ambient illumination map corresponding to each of the original images to obtain the ambient illumination compensation map corresponding to each of the original images; Based on the ambient light compensation map corresponding to each original image, light compensation is performed on each original image to obtain the plurality of light compensation images. 13. The device according to claim 11 or 12, wherein the processing module is further used for: Determining the updated three-dimensional coordinates of the multiple groups of matching feature points as a colorless sparse three-dimensional point cloud; The acquisition module is also used for: Obtain the pixel coordinates of the plurality of matching feature points in the corresponding plurality of illumination compensation images; The processing module is also used to: Using the pixel coordinates of the multiple groups of matching feature points in the corresponding multiple illumination compensation images, color the colorless sparse 3D point cloud to obtain a colored sparse 3D point cloud; based on the colorized sparse 3D point cloud and the obtained The multiple illumination compensation images are used to determine the three-dimensional model of the target object. 14. The device according to any one of claims 10 to 13, wherein the processing module is further used for: Extracting feature points of each original image in the multiple original images; performing feature point matching on the multiple original images to obtain the multiple sets of matching feature points. 15. The device according to any one of claims 10 to 14, wherein the processing module is further used for: Based on the pixel coordinates of the multiple groups of matching feature points in the corresponding original images and the internal camera parameters corresponding to the multiple original images, using a triangulation method to determine the relative camera pose corresponding to each original image; and, Based on the pixel coordinates of the multiple sets of matching feature points in the corresponding original images and the relative camera pose corresponding to each original image, the three-dimensional coordinates of the multiple sets of matching feature points are determined by using a triangulation method. 16. The device according to any one of claims 10 to 15, wherein the processing module is further used for: The three-dimensional coordinates of the multiple sets of matching feature points and the relative camera poses corresponding to each original image are input to the depth estimation network model to obtain the depth information of the corresponding matching feature points on each original image. 17. The device according to any one of claims 10 to 16, wherein the processing module is further used for: Inputting the depth information of the corresponding matching feature points on each original image and the plurality of original images to an illumination estimation network model to obtain an environment illumination map corresponding to each original image. 18. The device according to any one of claims 10 to 17, wherein the processing module is further used for: Determining the pixel coordinate with the smallest pixel amplitude in the ambient light map corresponding to each original image as the pixel coordinate corresponding to the virtual light source in each original image; and, based on the virtual light source in each original image The pixel coordinates corresponding to the images and the camera relative poses corresponding to each original image determine the three-dimensional coordinates of the virtual light source. 19. An image processing device, characterized in that it comprises: a processor, the processor is coupled to a memory, the memory is used to store a computer program, and when the processor invokes the computer program, the image is A processing device performs a method as claimed in any one of claims 1 to 9. 20. A computer-readable storage medium, characterized by being used for storing a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 9. 21. A computer program product, characterized in that the computer program product includes computer program code, and when the computer program code is run on a computer, the computer implements any one of claims 1 to 9. the method described.