RU2693327C1 - Method of reconstructing 3d model of static object and device for its implementation - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: invention relates to the computer equipment. Method of reconstructing 3D model of a static object includes: stereoscopic shot a static object at different angles of filtering polarized light; obtaining normal polarization maps and a sparse surface depth map; determining augmented polarization depth map for determining 3D shape of surface; obtaining 3D model of a static object, wherein obtaining intensity-corrected images, depth stereo maps and a polarization depth map based on polarization normal maps, merging depth stereo maps and polarization depth map into polarization depth stereo map and obtaining surface map of normals based on augmented depth map, repeating survey, wherein for each view on the supplemented polarization depth map, a polarization cloud of points is obtained, which are combined into a multi-view point cloud of static object 3D surface, 3D shape of the surface is obtained from the formed multi-view point cloud, and the initial polarized images of the static object are corrected by color intensity maps.

EFFECT: high accuracy of 3D reconstruction of a static object.

4 cl, 7 dwg

Description

The invention relates to the field of reconstruction of the electronic 3D model of a static object, in particular, to the reconstruction of the three-dimensional shape of the surface and comparison of the original images of the object being reconstructed and can be used in various industries, including flaw detection, construction, monitoring of structural forms, as well as medicine, archeology, gaming and film industries.

The known method [1] for reconstructing the surface shape of a static object from its images taken with a polarization filter taken with several filtering angles, in which restoring the surface shape includes the step of obtaining a normal map and a 3-dimensional surface shape of an object having a reflective metal surface. However, the known method is inaccurate when reproducing the surface shape of a 3D model of a static object due to the fact that when a reconstructed object is reproduced, the orientation of its surface is determined ambiguously taking into account existing problems Pi of the normalization calculation of normals, inaccuracy of normal calculation for front-parallel surfaces, the possibility of obtaining only relative 3 -dimensional shape of the surface, and does not allow to obtain multi-view 3D reconstruction. Also this method has a limitation in its application, since it is designed only for the analysis of static objects with a reflecting surface.

The known method [2] for the reconstruction of the 3D surface using images, which are analyzed part of the surface to be restored. In the known method, the shape of the surface of a static object is reconstructed from polarization images and shadow images that appear when it is illuminated at different angles when using the image surface reconstruction by polarization, where the measured values can be combined with each other in the context of the minimized error function. However, this method does not reproduce the exact shape of the surface of a 3D model of a static object due to the fact that when a reconstructed object is reproduced, the orientation of its surface is determined ambiguously taking into account the existing problems Pi of calculating normals as well as the refractive inaccuracy of calculating the zenith angle due to an unknown surface refraction index, inaccurate calculation normals for front-parallel surfaces, gaps in the depth map, which makes it impossible to obtain a multi-view 3D camera struction, and this narrows as the scope and quality of the reconstruction of 3D models. In addition, the known method limits the scope of its application by using the active reconstruction method, which requires shooting only using controlled lighting, which is provided by a more complex structural device due to the need for controlled lighting depending on the 3-dimensional shape of the reconstructed surface of a static object.

The known method [3] reconstruction of the 3D form of an object using images of the object on which coded light patterns are projected. However, the known method provides a low resolution reconstruction of the 3-dimensional surface shape, since it depends on the resolution of the used projection device with a lower resolution relative to the camera of this device. Also, the known method is complex and expensive to implement, since it requires the availability of additional equipment for projecting the light pattern onto the object. In addition, this method has limited application, as it requires special lighting with a projection device.

The closest technical solution to the claimed invention is a method [4], adopted as a prototype, which consists in reconstructing the three-dimensional shape of the object's surface, which consists in obtaining an updated and augmented depth map of a static object and its subsequent 3-dimensional shape using a depth map and polarizing images of the object.

The disadvantage of this method, adopted as a prototype, is the relatively low accuracy of the reconstruction of a static object due to the fact that the depth map obtained from the stereo camera has limited information about the 3-dimensional shape of the surface of the static object, as well as the inevitable errors in measuring the filtering angles of a polarized Sveta. In addition, the known method has less information due to the limited one-view shooting and, as a consequence, obtaining less qualitative visualization of the desired 3D model of a static object.

The closest device [4] for implementing the method according to the combination of features with the claimed group of the invention (method and device) is a device adopted as a prototype, using a depth sensor to create an initial depth map of an object, cameras for obtaining polarization images and a computing device.

A disadvantage of this device is the limited receiving of the depth map through the stereo camera due to the constructive arrangement of the cameras located in it, which entails informational errors in obtaining the depth map using triangulation in two polarization cameras installed at a non-zero angle to each other, since with this arrangement Polarization filters significantly change the visible image of a static object in a different way for each of the cameras. In addition, it complicates the device and increases the error of having an independent polarization filter in each of the chambers.

The technical result of the claimed invention is to improve the accuracy of 3D reconstruction of a static object and the information content of the reconstructed 3D model, as well as simplifying the device for implementing the method.

This technical result is achieved by the fact that in a known method for reconstructing a 3D model of a static object consisting in polarizing stereo shooting of a static object at different filtering angles of polarized light when changing them sequentially and using two generated sets of polarized images, two polarization normal maps and a sparse surface depth map are obtained , and determining by them an augmented polarization map of the depth, which determines the three-dimensional shape of the surface and a static object, and obtaining a 3D model of a static object in accordance with the claimed invention from this 3-dimensional form of the surface, in each of the two formed sets of at least 4 polarized images of a static object change the polarization angles, which are chosen uniformly in the range from 0 to 180 degrees. In this case, each of the formed sets is taken as a base for determining a polarized normal map, after which two intensity corrections are obtained for each of the two sets of polarized images image, which receive stereographic depth maps, then get a polarization depth map using two polarization normal maps, then merge the stereo depth maps and a polarization depth map into a polarization stereo depth map and get a surface normal maps based on the augmented depth map, and then repeat the polarization stereo shooting not less than two times from different angles, intersecting in pairs in the observation area of a static object, and for each angle along augmented polarization A polarization cloud of dots is obtained on the depth map, which unite the 3D surface of a static object into a multi-angle cloud of points, a 3-dimensional surface shape is obtained from the formed multi-angle cloud of points, and the original polarized images of the static object are corrected using color intensity maps from each set of polarized images then corrected source polarized images of a static object are compared with a three-dimensional surface of a static object, and of the results obtained to compare them determine a 3D model of a static object.

In addition, this technical result is achieved by the fact that the number of angles of filtration is chosen according to the number of polarized images in one set of images.

In addition, this technical result is achieved by the fact that the angles of the polarization survey are chosen uniformly around a static object.

This technical result is also achieved by the fact that in a known device containing a system of 2 imaging cameras fixed relative to each other, one of which has a rotary polarization filter located in front of its lens, and a computing device in accordance with the claimed invention of paragraph 2, the second imaging camera is equipped with a rotary polarization filter located in front of its lens, while the polarization filters of both cameras have the same angles of rotation a and are connected by a rotating device, the optical axes of the imaging cameras are parallel to each other, the housing of each of the image cameras has two connectors, one of which is connected by cable to the computing device and the other to the second imaging camera, the turntable is connected to the computing device for circular rotation of a static object.

The essence of the claimed invention is explained by the numerous experimental results of the reconstruction of a 3D model of a static object, tested in laboratory conditions on the equipment of the company IT Center SPbGU and explained below by theoretical and applied research, which gave real confirmation of the increase in the 3D object that was stated in the present invention. informativeness of the reconstructed 3D model, as well as simplifying the device for implementing the claimed method.

As is known, the method of reconstruction of normals from polarized images is based on measuring the intensity of light for different filtering angles of polarized light, which is used to calculate the azimuth angles ϕ and zenith of θ normals to the surface in a spherical coordinate system.

In accordance with the Fresnel equations, the measured luminance at one point of the scene is expressed as [6]:

where ϕ is the azimuth angle,

I _max and I _min are the maximum and minimum of light intensity.

Since the harmonic has three unknowns, with three different measurements of ϕ _pol , ϕ, I _max and I _min can be calculated.

When calculating the azimuth angle ϕ, two main problems arise.

First, since the cosine argument in equation 1 is multiplied by 2, then in the range from 0 to 2π, the angle ϕ can have 2 values shifted by π radians, and determine the correct solution from these 2 solutions based only on the analysis of polarized images without additional no assumptions are possible. This first problem is called azimuthal ambiguity and is present in all methods for the reconstruction of normals from polarized images.

Secondly, if the surface type is unknown a priori - diffuse or specular, then the displacement of the azimuth angle ϕ by π / 2 radians is not determined. This second ambiguity, due to the reflection of the surface, is called the mismatch of the azimuthal model, and is critical of the reconstruction of the normals from polarized images.

As is known, the calculation of the zenith angle depends on the degree of polarization, and has the form:

Substituting the Fresnel formula into this equation, we obtain the following formula by which we can calculate the zenith angle for diffuse surfaces:

where n - denotes the refractive index, and θ - zenith angle.

And the zenith angle for the mirror surfaces:

Based on the stated formulas (1-4) in the claimed method, normal maps are obtained using 4 or more images with different filtering angles of polarized light. The system of equations consists of equations by the number of images with 3 unknowns. When calculating the normal map, the problem is solved using the method of least squares. The solution of this problem leads to the solution of a system of such equations:

Thus, due to the larger amount of data obtained at different angles of filtration of polarized light, the accuracy of determining the phase of a sinusoidally varying intensity increases when there are various interferences in the system related to the inaccuracy of measuring the intensities of individual pixels of the camera, which appear due to disturbances acting on the system from the outside and if there are inaccuracies in the measurement of the filtration angle of polarized light.

After solving this system of equations for each pixel of the image obtained from the cameras, get the minimum and maximum light intensity, the azimuth angles ϕ and zenith θ normal to the surface of the object, the light from which falls into this pixel, which together form a polarization normal map of the surface of the static object, a map of the minimum light intensity and a map of the maximum light intensity. The map of the minimum light intensity and the map of the maximum light intensity receive a map of the average light intensity. Such maps are obtained for each of the two sets of images of the static object being reconstructed.

Since the images were obtained from cameras whose optical axes are parallel, the planes of the image sensors are parallel and the filtering directions of polarized light are also parallel, the pixels in two maps of average light intensity corresponding to the same surface areas will have the same values. Thus, based on the calibration data of the stereo cameras, by correlating the images along the epipolar lines, a stereo depth map is obtained by triangulation. The obtained normalization polarization maps for two sets of images will also have the same values for the same surface areas. Thus, the normal maps along the epipolar lines are correlated, and a polarization depth map is obtained by triangulation. The resulting stereo depth map and polarization depth map are combined into a polarization stereo depth map.

The connection of depth maps is as follows: in the absence of data in one of the depth maps, they take data from the other, in other cases they take the arithmetic average between the values of the depth maps, obtaining a polarizing stereo depth map. Then, from the polarization normal map and polarization stereo depth map, the augmented depth map is calculated.

Using a polarization normal map, a supplemented depth map, and internal camera parameters, a 3D cloud of points is obtained, in which each point has 3D coordinates and a polarization normal. For each point in the 3D point cloud, a normal to the surface is obtained.

According to the method above, a multi-angle survey of a static object is carried out, repeating the previous steps, after which a 3D cloud of points is obtained for each angle, which are combined into a single 3D point cloud by comparing the points of the surface of the static object observed from several angles. Comparison results in minimization of the comparison error using the formula for the standard deviation of the coordinates of points in the clouds, surface normal maps and polarization normal maps for those points that were compared as points belonging to the same surface point of a static object. Two points from two different 3D point clouds are considered to be mapped if the distance between them does not exceed the specified threshold value, which is determined by the parameters of the cameras and the size of the object. In the aggregate, this method allows for a more accurate reconstruction of the object, since the correlation takes into account the refractive index of the material.

Then, over a point cloud of a 3D surface of a static object, a 3-dimensional shape of the surface of a static object is obtained according to the nearest neighbor algorithm.

After that, a 3D texture of the surface of the static object is obtained. To do this, the original polarized image must be corrected in intensity to reduce the polarization effect, leading to a change in the intensity of the color of each pixel of the image depending on the angle between the corresponding point on the object's surface and the filtering plane of polarized light. On the basis of solving the system of equations when calculating the polarization normal map, determine the values of the minimum and maximum light intensities for each analyzed pixel of the original image. As the intensity of the color of the image in the corrected polarized image, choose the arithmetic average of the values of the minimum and maximum light intensities.

Thus, for each angle shooting, two corrected polarized images are obtained, which are compared with the 3-dimensional shape of the surface of a static object. If more than one image corresponds to one flat segment of a 3D surface of a static object, then the image that provides the largest number of pixels for this segment is selected for matching. Mapped fragments of corrected images together determine the texture of a 3-dimensional surface of a static object.

After that, the 3D model of the reconstructed static object is determined by the 3-dimensional shape of the surface of the static object and the matched images of the object (textures).

The essence of the claimed invention is illustrated FIG. 1 - FIG. 7

FIG. 1 shows a device for implementing a method for reconstructing a 3D model of a static object with a polarization survey, consisting of two cameras 1 and 2, fixed relative to each other, for example, a common stand 3, forming a stereo pair, connecting to synchronize the gates of cameras with cable 10, polarizing filters 4 and 6 , with a turning mechanism for polarizing filters 7, a computing device 5, connecting cameras to a computing device with cables 8 and 9, turning mechanism 12 and connecting a turning mechanism to the calculator Nome device cable 11.

The claimed device for implementing the method according to claim 1 of the invention is intended for shooting with two cameras (1-2) having stereo camera properties, two cameras mounted on a stand (3) and connected to each other by a cable (10), two synchronized polarizing filters are installed in front of the cameras (4 and 6), the polarization filters are connected by a rotating mechanism (7) to build a more accurate depth map by minimizing the difference between the intensities of reflection of light from a static object, the camera is connected to the computing device (5) using Abel (8 and 9), the turntable (12) is connected to the computing device with a cable (11).

FIG. 2 shows a set of polarized images obtained from the camera (1) shown in FIG. 1, with the indicated angles of rotation of the polarization filter in degrees 0, 45, 90, 135.

FIG. 3 shows a set of polarized images obtained from the camera (2) shown in FIG. 1, with the indicated angles of rotation of the polarization filter in degrees 0, 45, 90, 135.

FIG. 4 shows a depth map obtained using two images from the first and second set of images obtained from the cameras (1-2) shown in FIG. 1, obtained from one perspective.

FIG. 5 shows a depth map obtained using two normal maps from the first and second set of images obtained from the cameras (1-2) shown in FIG. 1, obtained from one perspective.

FIG. 6 shows a point cloud obtained using a multi-angle survey of a static object formed on the depth map, which is corrected and supplemented with the help of a surface normal map.

FIG. Figure 7 shows the 3D reconstruction of a static object.

The claimed invention, as mentioned above, was experimentally tested on equipment and in the laboratory of the company "IT Center SPbSU"

Examples of specific implementations are given below.

Example 1

As an example of obtaining a concrete expected result with a higher accuracy of reconstruction of a 3D model of a static object, it was located on the turntable (12) shown in FIG. 1. FIG. 2 and FIG. 3 shows two sets of polarized images taken from the same angle. For each set of images are built I _max , I _min and the average map I _avg . Two I- _avg cards are used to construct a stereo depth map using triangulation, which is shown in FIG. 4. For two sets of images, the polarization normal maps were calculated, on the basis of which the polarization depth map shown in FIG. 2 was calculated using triangulation. 5. This example experimentally confirms that the claimed method, based on the construction of a depth map using polarization normal maps, complements the stereo depth map, which provides additional more objective information about the depth of points, which means that for the surface areas where the depth map was added, additionally resolve π ambiguity and refine the refractive inaccuracy of the calculation of the zenith angle, thereby increasing both the accuracy and the quality of the reconstructed 3D model.

Example 2

Creating a full 3D model of a static object requires multi-angle shooting and texturing of the object. When the polarization and stereo depth maps obtained using polarization depth maps and stereo images were combined, the polarization stereo depth map was corrected using a polarization normal map. After repeatedly shooting at different angles and using the depth map calculations in Example 1, point clouds were computed, which were summarized into one cloud, which is shown in FIG. 6. Based on a cloud of points, a 3-dimensional surface of a static object was formed. After that, the averaged image of the static object I _avg was obtained using the I _max and I _min maps calculated by the above Fresnel equations (1-4), which made it possible to obtain a more accurate surface texture of the static object. After that, the obtained averaged image I _{avg was} compared with the 3-dimensional surface of a static object, which made it possible to obtain a textured 3D model shown in FIG. 7, which experimentally confirms the achievement of the high informativeness of a 3D model of a static object by the claimed invention.

Example 3

FIG. 1 shows a diagram of a specific device consisting of two PointGrey CM3-U3-31S4C-CS color cameras with a resolution of 3 megapixels, the cameras are equipped with Kowa LM8JC10M lenses with a focal length of 8 mm, and 57 mm Goyo CPL polarization filters, each with drive to one stepping motor. The cameras are installed so that their optical axes are parallel, and the sides of the image sensors inside them are parallel to each other. The distance between the centers of the chambers is 76 mm. At a distance of 2 meters from the cameras there is a turntable. An experimental sample of the claimed device allows polarization imaging of static objects of up to 1 × 1 × 1 m ³ in size. Due to the location of filters in the same plane parallel to each other, the installation of drives to a single stepping motor is simplified, and significantly reduces the measurement errors of the angles of rotation of the filters. In addition, the claimed device, implemented in the form of a specific experimental setup, allows images to be obtained by two cameras, where the same point on the surface of a static object has the same maximum and minimum intensity, and the azimuth and zenith angles of the normal have the same value , which significantly increases the accuracy of triangulation.

The claimed invention, as shown by the results of the tests, can be used for the reconstruction of a 3D model of a static object with high accuracy and information.

The claimed invention has a wide range of applications, in particular, it is most effective for solving, for example, medical tasks in monitoring the process of treating and recovering patients, as well as in archeology for reconstructing physically obsolete monuments, sculptures and various types of minerals; Effectively also the use of reconstructed 3D models of objects in the film industry, the gaming industry and many other industries and production areas where high quality, accuracy and information content of the objects being reconstructed is required.

References

[1] PCT / FR2006 / 002550 WO 2007057578 A1 “Method and system for reconstructing surfaces using polarization imagery”, IPC G01B 11/24

[2] Patent DE102004062461A1 "Image supported surface reconstruction reconstruction methods of property methods from 2008.01.31, IPC G01B 11/24

[3] PCT / IL2007 / 001432 WO2008062407A2 "3d geometric modeling and 3d video content creation", IPC G06T 15/205.

[4] US Patent №20160261844 "Methods and Apparatus for Enhancing Depth Maps with Polarization Cues" dated 12/27/2015; IPC H04N 13/00, H04N 13/02 (prototype)

[5] Daisuke Miyazaki, Masataka Kagesaway and Katsushi Ikeuchi, "Polarization-based Transparent Surface Modeling from Two Views" IEEE International Conference on Computer Vision, 2003

[6] Matthew Howard. (2018) CV online, [Shape from Polarisation] // (https://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV0405/HOWARD/prac2.htm 1)

Claims (4)

1. A method of reconstructing a 3D model of a static object, consisting in polarizing stereo shooting of a static object at various filtering angles of polarized light when they are successively changing, two polarized normal maps and a rarefied surface depth map are obtained from the two formed sets of polarized images and the augmented polarization map is determined by them depth, which determine the three-dimensional shape of the surface of a static object, and on this three-dimensional shape of the surface, There is a 3D model of a static object, characterized in that in each of two formed sets of at least 4 polarized images of a static object, the polarization angles are changed, which are chosen uniformly in the range from 0 to 180 degrees. Each of the formed sets is taken as a base to determine the polarized normal map, for each of the two sets of polarized images, two intensity-corrected images are obtained, for which stereographic depth maps are obtained, a polarization depth map is obtained by two m polarization normal maps, then combine the stereo depth maps and the polarization depth map into a polarization stereo depth map and get a surface normal map based on the augmented depth map, then repeat the polarization stereo shooting at least two times from different angles, pairwise intersecting areas of the static object, then for of each angle, according to the augmented polarization depth map, a polarization point cloud is obtained, which merge into a multi-angle 3D point cloud. the static object and the three-dimensional shape of the surface are obtained from the formed multi-angle point cloud, and the original polarized images of the static object are corrected using the color intensity maps obtained from each set of polarized images, after which the corrected original polarized images of the static object are compared and The results of their comparison are determined by the 3D model of the static object. 2. The method according to p. 1, characterized in that the number of angles of filtration is chosen according to the number of polarized images in one set of images. 3. The method according to p. 1, characterized in that the angles of polarization shooting is chosen uniformly around a static object. 4. A device for reconstruction of a 3D model of a static object according to claim 1, containing a system of 2 imaging cameras fixed relative to each other, one of which has a rotary polarizing filter located in front of its lens, and a computing device, characterized in that the second camera image acquisition is equipped with a rotary polarization filter located in front of its lens, while the polarization filters of both cameras have the same angles of rotation and are connected by a rotating mechanism, optically The axes of the image acquisition chambers are parallel to each other, the housing of each of the image cameras has two connectors, one of which is connected by cable to the computing device and the other to the second camera of image acquisition, a turntable for circular rotation of the static object is connected to the computing device. .

Images