RU2806009C2 - Method for constructing a depth map from a pair of images - Google Patents

Abstract

FIELD: image data processing.

SUBSTANCE: invention is aimed to increase the accuracy of the depth map values. It is achieved by the fact that the claimed solution provides for stages in which first and second images are obtained from the first and second cameras; the rectification procedure of the first and second images is performed; the first and second image tensors are generated, where each element of the tensor represents the brightness value of the corresponding pixel; the tensors are normalized; the tensors are combined using an encoder; by means of a transformer, vector representations of at least one object contained in the tensor obtained at the previous stage are compared line by line to form a tensor containing information about the values of shifts of image pixels relative to each other; by means of a decoder, the first and the second shift maps are generated for the first and second images, containing said shift values, based on the tensor obtained at the previous stage; based on the values of the shift maps generated at the previous stage, an image depth map is formed.

EFFECT: increase in the accuracy of the depth map values.

8 cl, 6 dwg

Description

TECHNICAL FIELD

[0001] The presented technical solution relates generally to the field of image data processing, and in particular to a method and device for constructing a depth map from a pair of images obtained, for example, by means of stereo cameras using a TUDE (Transformer-Unet for Depth Estimation) device ,Transformer-Unet for Depth Estimation).

BACKGROUND OF THE ART

[0002] Existing analogs that are closest to the presented solution can be divided into two families:

1. Classic - using line-by-line comparison of windows with pixel brightness values from frames of two cameras to build a depth map;

2. Neural network - using features identified by a neural network for comparison.

[0003] Despite the relatively high speed of operation, solutions implemented based on classical approaches have a number of disadvantages:

- do not work well in areas with a uniform texture (for example, an image of a road);

- give many omissions in predictions and are often mistaken;

- the accuracy of predictions strongly depends on the size of objects.

[0004] Most solutions available in the open source are based on methods from the OpenCV library. OpenCV (Open Source Computer Vision Library) is an open source library of computer vision, image processing and general-purpose numerical algorithms.

[0005] Neural network approaches such as AANet [see AANet: Adaptive Aggregation Network for Efficient Stereo Matching], LEAStereo [see. LEAStereo: Learning Effective Architecture Stereo] are significantly superior in quality to classic ones, but they work slower. This is critical for applications such as autonomous driving, as it directly affects how quickly the autopilot reacts to obstacles. Also, these solutions do not have a mechanism for cutting off neural network predictions based on a confidence threshold, which leads to artifacts at the edges of objects and in areas of objects that are visible on one camera and not visible on another. These inaccuracies can negatively affect other algorithms that work with the depth map, which impairs the performance of the autopilot algorithm.

[0006] Safe driving of an unmanned vehicle requires rapid response to events in a traffic scene, such as a vehicle suddenly leaving the oncoming lane. At high vehicle speeds, quick decision making by the autopilot becomes critical. This prevents the use of algorithms that work with data from lidars, the frequency of which is lower than the frequency of the camera. In addition, the point cloud obtained from the lidar is very sparse, as a result of which it is not always possible to draw a correct conclusion about the presence and nature of the object.

[0007] The use of two cameras combined into a stereo pair allows you to quickly and relatively accurately obtain a dense depth map (each pixel from the camera is associated with a distance to an object), which gives a dense coverage of 3D points of small objects [see. article Smolyanskiy, Nikolai and Kamenev, Alexey and Birchfield, Stan “On the Importance of Stereo for Accurate Depth Estimation: An Efficient Semi-Supervised Deep Neural Network Approach”, 2018.].

[0008] The proposed technical solution works up to five times faster than similar neural network approaches AANet and LEAStereo, which also work with a pair of images, with comparable accuracy. The high speed of the algorithm is achieved through the use of lightweight layers operating in a relatively low spatial resolution. Also, the proposed solution allows you to filter areas with a high error in determining the depth map.

DISCLOSURE OF INVENTION

[0009] The technical problem or task posed in this technical solution is to create a new efficient, simple and reliable method for constructing a depth map from a pair of images.

[0010] The technical result is to increase the accuracy of the depth map values.

[0011] The specified technical result is achieved by implementing a method for constructing a depth map from a pair of images, performed by at least one computing device, containing the steps of:

- obtaining from the first and second cameras first and second images containing an image of at least one object;

- perform the procedure of rectification of the first and second images by projecting them into one plane;

- determine for each pixel of the first and second images a shift value indicating the number of pixels by which the most similar pixel of the other image is shifted;

- generating first and second shift maps for the first and second images, containing said shift values;

- based on the values of the shift maps generated at the previous stage, an image depth map is formed.

[0012] In one of the particular examples of the method, the step of determining the shift value for each pixel of the first and second images contains steps in which:

- determine the brightness value (the amount of illumination) of each pixel of the first and second images;

- the brightness values of the pixels of the first image are compared with the brightness values of the pixels of the second image to determine the shift value for each pixel of the first and second images, and the brightness values of neighboring pixels are also taken into account during the comparison.

[0013] In another particular example of the method, the steps of checking the consistency of the pixel shift values of the left and right images are additionally performed.

[0014] In another particular example of the method, the step of generating an image depth map based on the values of the shift maps contains the steps of:

- based on the pixel shift values contained in the shift maps, the distance from the line connecting the centers of the cameras to each pixel of at least one object is determined;

- based on the obtained distance values from the line connecting the centers of the cameras to each pixel of at least one object, an image depth map is formed.

[0015] In another particular example of the method, said distance is determined by the formula: distance=B×f/D,

where B is the size of the base (distance between cameras),

f - focal length in pixels,

D - shift value.

[0016] In another particular example of the method, the computing device is additionally equipped with an encoder, transformer and decoder, and the step of determining for each pixel of the first and second images a shift value indicating the number of pixels by which the most similar pixel of the other image is shifted, contains the steps of: of which:

- form the first and second image tensors containing vector representations (feature vectors) of at least one object, each element of the tensor representing the brightness value of the corresponding pixel;

- normalize the tensors obtained at the previous stage;

- by means of an encoder, the mentioned two tensors are combined into one tensor;

- by means of a transformer, vector representations of at least one object contained in the tensor obtained at the previous stage are compared line by line to form a tensor containing information about the values of shifts of image pixels relative to each other;

in this case, the stage of forming the first and second shift maps for the first and second images is performed by the decoder based on the tensor obtained at the previous stage.

[0017] In another particular example of the method, the encoder, transformer and decoder are implemented on the basis of neural networks pre-trained on a training data set.

[0018] In another particular example of the method, the steps are additionally performed in which:

- based on the depth map, a point cloud is formed in three-dimensional space;

- use a point cloud to plan the trajectory of an autonomous unmanned vehicle

[0019] In another preferred embodiment of the claimed solution, a device for constructing a depth map from a pair of images is presented, containing at least one computing device and at least one memory device containing machine-readable instructions that, when executed by at least one computing device, perform the above way.

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The features and advantages of the present technical solution will become apparent from the following detailed description of the technical solution and the accompanying drawings, in which:

[0021] In FIG. 1 - an example of the implementation of an image processing system is presented.

[0022] in FIG. 2 - examples of images obtained from the left and right cameras are presented.

[0023] in FIG. 3 - examples of images with obscured areas are presented.

[0024] in FIG. 4 - shows an example of images with filtering on a depth map.

[0025] in FIG. 5 - shows an example of a neural network architecture diagram.

[0026] in FIG. 6 shows an example of a general view of a computing device.

IMPLEMENTATION OF THE INVENTION

[0027] The concepts and terms necessary to understand this technical solution will be described below.

[0028] In this technical solution, a system means, including a computer system, a computer (electronic computer), CNC (computer numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given task. , a clearly defined sequence of operations (actions, instructions).

[0029] By command processing device is meant an electronic unit, a computing device, or an integrated circuit (microprocessor) that executes machine instructions (programs).

[0030] A command processing device reads and executes machine instructions (programs) from one or more storage devices. Storage devices can include, but are not limited to, hard drives (HDD), flash memory, ROM (read-only memory), solid-state drives (SSD), and optical drives.

[0031] A computing device is a computing device that automatically performs one mathematical operation or a sequence of them in order to solve one problem or a class of similar problems (Great Soviet Encyclopedia. - M.: Soviet Encyclopedia. 1969-1978.).

[0032] Program - a sequence of instructions intended for execution by a computer control device or command processing device.

[0033] Database (DB) - a collection of data organized in accordance with a conceptual structure that describes the characteristics of that data and the relationships between them, and such a collection of data that supports one or more areas of application (ISO/IEC 2382: 2015, 2121423 " database").

[0034] A signal is a material embodiment of a message for use in the transmission, processing and storage of information.

[0035] Attention mechanism (attention model) is a technique used in recurrent neural networks (RNN) and convolutional neural networks (CNN) to find relationships between different parts of input and output data.

[0036] In accordance with the diagram presented in Fig. 1, the image processing system includes: a first camera 1, a second camera 2, and a device 10 for constructing a depth map from a pair of images.

[0037] Any cameras widely known from the prior art can be used as cameras, located, for example, on a vehicle, at a given distance from each other in any direction, for example, 10-100 cm, and directed in one direction towards at least one object. The objects that the cameras are aimed at can be, for example, pedestrians, road barriers, other vehicles and road users, building walls, curbs, the sky, trees, animals, roads, sidewalks, etc. For convenience, we will further assume that the cameras are distant from each other horizontally. Cameras can be used that operate both in the visible spectral range (see, for example, “Visible radiation”), and in the UV (see “Ultraviolet radiation”) and IR (see “Infrared radiation”) ranges. The cameras can use CMOS (complementary metal-oxide-semiconductor, complementary logic on metal-oxide-semiconductor transistors, CMOS) matrices with active sensing elements (Active Pixel Sensor) and CCD (charge-coupled device, device with charge feedback) .

[0038] For example, the camera model “LI-IMX390-GW5200-GMSL2-120H” can be used as a camera (see, for example, https://www.leopardimaqing.com/product/autonomous-camera/maxim-gmsl2-cameras /li-imx390-gw5200-gmsl2/li-imx390-gw5200-gmsl2-120h/) or camera model “acA2040-25gc - Basler ace” (see, for example, https://www.baslerweb.com/en/products /cameras/area-scan-cameras/ace/aca2040-25gc/).

[0039] The cameras are calibrated by methods known from the prior art, and information about the position and angles of inclination of the cameras relative to each other is entered by the developer into the device 10 for constructing a depth map from a pair of images. Lens distortions in images obtained from cameras can also be corrected using known methods (see, for example, the camera calibration procedure disclosed at the Internet link: https://docs.opencv.org/4.5.2/dc/dbb/tutorial_р_calibration.html ).

[0040] The device 10 for constructing a depth map from a pair of images can be implemented on the basis of at least one computing device and contain: a data acquisition module 11, an image rectification module 12, a shift map generation module 20, and a depth map generation module 30. The mentioned modules can be implemented on the basis of software and hardware of a computing device, in particular on the basis of its processor or microcontroller, and are equipped with appropriate communication interfaces, logic elements, ADC and DAC for exchanging signals for the purpose of transmitting data, including image information, disclosed below.

[0041] Accordingly, the first and second images containing at least one image of the object, for example, a left image and a right image, from cameras 1 and 2 are supplied to the data acquisition module 11, which may be equipped, for example, with a buffer device providing synchronous acquisition of data from two cameras. The resulting images are sent by the mentioned module 11 to the image rectification module 12, which projects the images into one plane. The rectification procedure for a pair of images can be carried out using methods known from the prior art, for example, those disclosed in the book Zisserman R. N. A. Multiple view geometry in computer vision, publ. in 2004, posted on the Internet at: https://www.r-5.org/files/books/computers/algo-list/image-processing/vision/Richard_Hartley_Andrew_Zisserman-Multiple_View_Geometry_in_Computer_Vision-EN.pdf, and allows you to get two frames with line-by-line correspondence of the location of objects on them (epipolar lines are projected horizontally), as a result of which the accuracy of the depth map values will increase.

[0042] Next, the first and second images that have undergone the rectification procedure are supplied to the shift map generation module 20, which performs a comparison of image pixels. When matching pixels in an image, the brightness value of the pixel itself is compared with the brightness value of the pixels adjacent to it. The result of comparing image pixels is a shift map, in which for each compared image pixel there is a shift value indicating the number of pixels by which the most similar image pixel from another camera is shifted. Each camera gets its own shift map. Thus, first and second shift maps are obtained, in particular for the first and second images, which are sent to the depth map generating unit 30.

[0043] To generate a depth map from the shift maps, said module 30, based on the pixel shift values, determines the distance from the line connecting the centers of the cameras to each pixel of at least one object in the image. This distance is inversely proportional to the shift value and is calculated, for example, by the formula:

distance=B×f/D;

where B is the size of the base (distance between cameras);

f is the focal length in pixels (in particular, the same cameras and images with corrected lens distortions are used);

D - shift value.

The focal length f and base size B are specified by the developer in the memory of the mentioned module 30, with which it can be additionally equipped.

[0044] Accordingly, for the first and second shift maps, module 30 generates first and second matrices, the values of which characterize the distance from the points of at least one object in the image to the line connecting the centers of the cameras, after which module 30 assigns the first or second matrix as depth maps, depending on the software algorithm specified by the developer.

[0045] Additionally, the depth map generating module 30 can be configured to filter the shift maps by checking the consistency of the pixel shift values of the left and right images, as a result of which the module 30 will generate a matrix containing information about the coordinates of the pixels and values indicating that the values the shifts of the pixels contained in the first and second shift maps are consistent or inconsistent. The algorithm for checking the consistency of pixel shift values will be described in more detail later in the text of the application. When generating a depth map using the method described above, inconsistent pixel shift values are not taken into account when generating a depth map.

[0046] Next, based on the data of the obtained depth map, the mentioned module 30 uses known methods to form a cloud of points in three-dimensional space, performing a back projection from the camera into 3D space (see, for example, Bostanci, Gazi Erkan & Kanwal, Nadia & Clark, Adrian. (2015 Augmented reality applications for cultural heritage using Kinect. Human-centric Computing and Information Sciences. 5. 1-18. 10.1186/s13673-015-0040-3.). A point cloud (a collection of points in three-dimensional space) can be used to plan the trajectory of an autonomous unmanned vehicle using well-known methods.

[0047] In an alternative embodiment of the claimed solution, the shift map can be generated by using a neural network that consists of several other neural networks. In this embodiment, the shift map generation module 20 is additionally equipped with a tensor normalization module 21, an encoder 22, a transformer 23 and a decoder 24.

[0048] The tensor normalization module 21 can be used at least one processor or microcontroller configured in firmware to perform the functions assigned to the module 21 below.

[0049] At least one neural network can be used as encoder 22, which performs feature extraction in the image, obtaining vector representations of objects in reduced spatial resolution. For example, the neural network can be implemented as a standard convolutional neural network ResNet18 (see, for example, the article by He, Kaiming et al, “Deep Residual Learning for Image Recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)).

[0050] At least one neural network can be used as transformer 23, performing line-by-line comparison of features identified by the previous network. Said neural network may be equipped with an attention mechanism that evaluates how similar the vector features of a line of the image captured by the first camera are to the vector features of the corresponding line of the image obtained from the second camera. The output tensors with the activation map have the same dimension as the input data. However, tensor vectors carry information not about local features of one frame, but information about the shift of pixels of one camera relative to another. During training, transformer 23 predicts a tensor from which a shift map can be obtained in a lower spatial resolution than that of the input image. Transformer 23 can be implemented as a neural network of the BERT family (Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv: 1810.04805 (2018)) with two layers and eight heads.

[0051] At least one neural network can be used as decoder 24, decoding feature tensors output from encoder 22 and transformer 23 to the original spatial resolution. Several feature tensors of different spatial resolutions are taken from the encoder. The decoder 24 may be part of the Unet, complemented by an attention mechanism. The output of decoder 24 is a shift map normalized from 0 to 1.

[0052] Images in the digital environment are represented as tensors (or matrices) containing values that characterize the colors of the pixels or their brightness. Accordingly, the first (for example, left) and second (for example, right) images that have passed the rectification procedure are supplied in the form of tensors to the tensor normalization module 21. The tensors of the first and second images, T1 and T2, respectively, can be represented, for example, in the form of tensors with dimensions C×H×W, where C is the number of channels, H is the height of the image, W is the width of the image. For example, for an image with a resolution of 512 by 960 pixels - H=512, W=960 pixels. Also, images can be either color or black and white. Color images contain 3 channels (C=3, one channel for each of the three colors: red, green and blue), and black and white images contain one channel (C=1). The values of dimensions C, H and W can be specified by the developer.

[0053] Next, the algorithm for constructing a depth map will be described using color images as an example. In this embodiment of the technical solution, each element of the image tensor contains an integer with values from 0 to 255 inclusive, which shows the amount of illumination - the brightness of the corresponding pixel of the camera matrix. Accordingly, the generated first and second image tensors are input to the tensor normalization module 21.

[0054] Module 21 normalizes the input tensors as follows. From the brightness value (the amount of illumination) of each pixel, the k1 value is subtracted and then divided by the k2 value. Coefficients k1 and k2 are set by the developer. Then the mentioned tensors are combined into one tensor N×C×H×W (N=2, C=3, H=512, W=960) for simultaneous processing of the first and second image tensors using the encoder 22. The packet size N is determined by the mentioned module 21 based on the number of images (tensors) received at the input of module 21 simultaneously or sequentially in a time interval specified by the developer. Since the input of module 21 received two images, the value of the packet size N will be determined as 2. Encoder 22, as well as all other modules, can be implemented using the PyTorch library (see https://pytorch.org/). Other libraries can be used for implementation, for example, TensorFlow (https://www.tensorflow.org/), MxNet (https://mxnet.apache.org). Simultaneous processing of data from the first and second tensors by combining them into one tensor makes it possible to speed up the algorithm compared to sequential processing of the two original tensors. For example, two tensors T1 and T2 of dimension 3×512×960 are combined into one tensor O by adding a new dimension (increasing the rank of the tensor). The first dimension of the tensor O actually numbers the combined tensors: O = (T1, T2).

According to the first dimension of the tensor O, index 0 contains the tensor T1, and index 1 contains T2:

O[0]=T1

0[1]=T2

[0055] An example of merging for the case of two 2×2 matrices.

[0056] The output of the encoder 22 when processing one pair of image tensors combined into one tensor is a tensor containing vector representations (feature vectors) of at least one object in a reduced spatial resolution, for example, by a factor of 16, i.e. having a dimension of 2×256×32×60 (N=2, C=256, H=32, W=60), where the first dimension N corresponds to the size of the processing package, the second C - the dimensions of the feature vector of each element, and the remaining two dimensions H and W - reduced spatial size by 16 times. A description of the procedure for generating a vector representation of at least one object from an input image tensor is disclosed, for example, in the article by He, Kaiming et al, “Deep Residual Learning for Image Recognitions, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).” As a result of processing the input image tensor by the neural network, a new tensor is obtained, which can be presented to the user in the form of a feature map containing information about at least one object, including information about the boundaries of the object. This information is encoded as numeric values. Also, the encoder 22 generates additional tensors in at least one spatial scale (resolution) specified by the developer, for example, three tensors with dimensions 2×128×64×120, 2×64×128×240, 2×64×256×480, which will be used further by decoder 24. The procedure for obtaining tensors (feature maps) at different spatial scales from encoder 22 is described in the article by He, Kaiming et al, “Deep Residual Learning for Image Recognition”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[0057] Next, the depth map device 10 proceeds to the stage of line-by-line comparison of vector representations (feature vectors) of at least one object to obtain information about the shift of pixels relative to each other. Vector representations (feature vectors) contain information about at least one object in the images and are extracted from the tensors obtained in the previous step, and, if necessary, can be displayed to the user using the well-known index inversion operation. To compare line-by-line vector representations (feature vectors) of an object on the left and right images, it is necessary to combine features from the two images line-by-line using the encoder 22. To do this, encoder 22 strings together elements of feature maps that correspond to the same string of input images. For each packet (tensor) along dimension N, encoder 22 combines lines along dimension W, corresponding to the width of the image. Index number 0 contains the data of the first packet, index 1 contains the data of the second packet. For example, let the resulting feature map P have dimensions N×C×W (2×2×3, N=2, C=2, W=3). For simplicity, the dimension H is omitted:

By combining rows along the dimension W, we obtain a tensor of dimension C×2W (2×6):

[0058] Thus, from the tensor N×C×H×W (2×256×32×60, N=2, C=256, H=32, W=60), the encoder 22 generates the tensor C×H×2W ( 256×32×120), after which, for batch processing of all lines, the encoder 22 rearranges the dimensions of the tensor (the numbers contained in the tensor remain unchanged), for example, the tensor is reduced to the form H×2W×C (32×120×256). The algorithm for permuting dimensions can be specified by the developer using methods known from the prior art. The mentioned tensor is then transmitted by the encoder 22 to the input of the transformer 23, which returns a tensor of the same size 32×120×256. Transformer 23 extracts from the input tensor the object vector representations defined for the first image, the object vector representations defined for the second image, and then compares said vector representations to determine the values of the pixel shifts of the images relative to each other (see Vaswani, Ashish, et al. " Attention is all you need." Advances in neural information processing systems 30 (2017)), which can also be referred to as pixel shift prediction values. After processing, transformer 23 forms a tensor, which includes the received information about the values of shifts of image pixels relative to each other. This information is encoded as numeric values. Next, the resulting tensor is converted back into a tensor with dimensions N×C×H×W (2×256×32×60) by dividing the original tensor into row dimensions and further permuting the dimensions. The algorithm for permuting dimensions was specified by the developer.

[0059] Accordingly, the decoder 24 receives as input from the module 21 a tensor with normalized images (2×3×512×960), a tensor from the output of the transformer 23 (2×256×32×60), containing information about the values of the pixel shifts of each image relative to each other, as well as additional reinforcement in the form of feature maps (tensors) from encoder 22 at spatial scales specified by the developer (2×128×64×120, 2×64×128×240, 2×64×256×480). A description of the operation of decoder 24 is given in the article (Roy, Abhijit Guha, Nassir Navab, and Christian Wachinger. "Recalibrating fully convolutional networks with spatial and channel "squeeze and excitation" blocks." IEEE transactions on medical imaging 38.2 (2018): 540-549 ). Based on the received data, decoder 24 generates two tensors containing pixel shift values, which can be presented to the user in the form of shift maps. The tensors correspond to the first (for example, left) and second (for example, right) images and have a dimension specified by the developer, for example, 512x960. In what follows, we will call the resulting two tensors shift maps. The displacement maps are then sent to the depth map generating module 30.

[0060] Obtaining two shift maps at once is necessary for the post-processing algorithm performed by module 30, which checks the consistency of the pixel shift values for the left and right images. The algorithm for matching pixel shift values is as follows. We consider sequentially all the pixels in the first (left) image. For each pixel, using the shift map generated by the decoder 24 for the first (left) image, module 30 finds the pixel shift value. Based on the shift value, module 30 determines which pixel in the second (right) image corresponds to the pixel in the first (left) image. Similarly, for a found pixel on the right image, using the shift map generated by the decoder 24, module 30 finds the corresponding pixel on the left image.

[0061] Specifically, for each pixel of the first (left) shift map having coordinates x_left, y_left (horizontally and vertically of the matrix, respectively) and a value d_left (shift amount), module 30 finds a corresponding pixel on the second (right) shift map with coordinates x_right, y_right, where x_right=x_left+d_left, y_right=y_left. Next, module 30, based on the determined coordinates of the found pixel, extracts from the second shift map the shift value defined for this pixel: d_right. The shift value d_right determined for the pixel of the second image is compared by module 30 with the pixel shift value d_left determined for the pixel of the first image, and if the difference in magnitude between d_left and d_right is less than a given threshold value (for example, 2), then module 30 determines that the shift values are consistent, whereupon module 30 generates a pixel shift value consistency matrix, the size of which corresponds to the size of the original shift map, and assigns a value to the pixel (for example, indicates a True value in the cell at coordinates x_left, y_left) indicating that the shift values , defined for a given pixel, are consistent. If the magnitude difference between d_left and d_right is greater than a predetermined threshold value, then module 30 assigns a value (eg, False) to the pixel indicating that the offset values are inconsistent. The threshold value can be set by the developer of module 30. Accordingly, if a pixel returns to the same location with an error of less than 2 pixels, then the pixels are considered consistent. If the error is more than 2 pixels, then the shift values are not consistent.

[0062] This is necessary to increase the accuracy of the obtained offset values by filtering out areas where the offset values are inconsistent. As a rule, these are areas that are visible on one camera and not visible on another. Examples are shown in FIGS. 2 and 3. In Fig. Figure 2 shows the original images for the left and right cameras. In FIG. Figure 3 shows areas that are visible on one camera but not on another due to the parallax effect (https://en.wikipedia.org/wiki/Parallax). For such zones the shift value is not defined. Our approach can predict shifts in such areas, but the accuracy of the predictions here is much lower than in the case of an object being visible on two cameras simultaneously. In FIG. Figure 4 shows the original depth map and the depth map after filtering. As can be seen, the clarity of object boundaries has increased and areas with a high error in displacement predictions have been detected, which will have a positive effect on obstacle analysis algorithms that will use the resulting depth maps.

[0063] Accordingly, based on the result of checking the consistency of the pixel shift values, the module 30 generates a matrix containing information about the coordinates of the pixels and values indicating that the pixel shift values contained in the first and second shift maps are consistent or inconsistent. Next, module 30 proceeds to the stage of generating, using known methods, for example, those disclosed earlier, a depth map based on the values of the shift maps, and inconsistent pixel shift values are not taken into account when generating the depth map.

[0064] Thus, due to the fact that the images received from the cameras undergo a rectification procedure, and the depth map is built based on the values of the pixel shift maps determined for each acquired image, the accuracy of the depth map values increases.

[0065] A common approach is used to train a neural network [see Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016]. A dataset with images and known depth maps is taken. For example, you can use SceneFlow (https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html) and KITTI (http://www.cvlibs.net/datasets/kitti) as datasets /eval_stereo_flow.php?benchmark=stereo, http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo).

[0066] A neural network is trained on a training set of images to predict a labeled depth map.

Due to problems with training transformer 23, training of the entire neural network is carried out in three stages:

1. Parts of the neural network are trained, in particular the modules: encoder 22 and transformer 23 as follows. Images from two cameras in the form of tensors are supplied to normalization module 21. The resulting normalized images are further processed by encoder 22 and then by transformer 23. A tensor of dimension 2×256×32×60 from the output of transformer 23 is reduced to a shift map using averaging over the channel dimension C= 256. From a tensor of dimension 2×256×32×60, a tensor of dimension 2×32×60 is obtained, which corresponds to two shift maps for the left and right images (the first dimension of the tensor numbers the shift maps). The resulting maps are passed to the loss function along with the known shift map, reduced by a factor of 16. This stage is needed to pretrain transformer 23. Without this stage, transformer 23 will learn slowly, because Encoder 22 and decoder 24 will predominantly be trained, and the model will remain in a local minimum.

2. All parts of the neural network are trained, modules encoder 22, transformer 23 and decoder 24. But the loss function for the shift map obtained from transformer 23 is not calculated. This mode trains primarily a decoder that predicts shift maps at full resolution (the same as the input image).

3. All parts of the neural network are trained, modules encoder 22, transformer 23 and decoder 24. The optimized loss function looks like a weighted sum of the values of loss functions for two outputs of the neural network: from transformer 23 and from decoder 24. Transformer 23 predicts shift maps in reduced resolution, decoder 24 - in full resolution. The weights are 0.1 and 0.001, respectively. This allows for more precise adjustment of the weights of the entire neural network. The loss function for predictions of transformer 23 plays the role of regularization, which consists in the fact that the features from transformer 23 must contain information sufficient to create a shift map.

[0067] During training, the known shift map is normalized using a selected maximum shift value of 200 pixels. If the value is greater than this threshold, then it is cut off to 200. Thus, all values are included in the segment [0;1], which allows using the sigmoid at the output of the neural network as an activation function and training using 2D binary cross-entropy (ALL - Binary Cross Entropy, see links https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a

https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html).

[0068] An example of calculating the ALL function. Let p _i,gt and p _i,pred be the known and predicted normalized shift map for pixel number i, respectively. Then the value of the loss function L _bce is equal to:

where N is the number of pixels, the summation is over all pixels, the logarithm is natural.

[0069] Adabelief was selected as the optimizer [see Adabelief]. The implementation of the optimizer algorithm is described in more detail at the link: https://github.com/jettify/pytorch-optimizer.

[0070] In general (see Fig. 6), a computing device (200) contains one or more processors (201), memory devices such as RAM (202) and ROM (203), and input/ output (204), input/output devices (205), and network communication device (206).

[0071] The processor (201) (or multiple processors, multi-core processor, etc.) may be selected from a variety of devices commonly used today, for example, from manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos ™, MediaTEK™, Qualcomm Snapdragon™, etc. The processor or one of the processors used in the system (200) must also include a graphics processor, for example an NVIDIA GPU with a CUDA-compatible programming model or Graphcore, the type of which is also suitable for carrying out the method in whole or in part, and can also be used for training and application of machine learning models in various information systems.

[0072] RAM (202) is a random access memory and is designed to store machine-readable instructions executed by the processor (201) to perform the necessary logical data processing operations. The RAM (202) typically contains executable operating system instructions and associated software components (applications, program modules, etc.). In this case, the available memory capacity of the graphics card or graphics processor can act as RAM (202).

[0073] The ROM (203) is one or more permanent storage devices, such as a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[0074] To organize the operation of system components (200) and organize the operation of external connected devices, various types of I/O interfaces (204) are used. The choice of appropriate interfaces depends on the specific design of the computing device, which can be, but is not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0075] To ensure user interaction with the computing device (200), various means (205) of I/O information are used, for example, a keyboard, a display (monitor), a touch display, a touch pad, a joystick, a mouse, a light pen, a stylus, touch panel, trackball, speakers, microphone, augmented reality tools, optical sensors, tablet, light indicators, projector, camera, biometric identification tools (retina scanner, fingerprint scanner, voice recognition module), etc.

[0076] The network communication means (206) provides data transmission via an internal or external computer network, for example, an Intranet, the Internet, a LAN, etc. One or more means (206) may be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and/or BLE module, Wi-Fi module and etc.

[0077] Additionally, satellite navigation tools can also be used as part of the device (200), for example, GPS, GLONASS, BeiDou, Galileo.

[0078] The specific selection of device elements (200) for implementing various software and hardware architectural solutions may vary while maintaining the required functionality provided.

[0079] Modifications and improvements to the above-described embodiments of the present technical solution will be apparent to those skilled in the art. The foregoing description is provided by way of example only and is not intended to be limiting. Thus, the scope of the present technical solution is limited only by the scope of the attached claims.

Claims (25)

1. A method for constructing a depth map from a pair of images, performed by at least one computing device, comprising the steps of: - obtaining from the first and second cameras first and second images containing an image of at least one object; - perform the procedure of rectification of the first and second images by projecting them into one plane; - form the first and second image tensors containing vector representations (feature vectors) of at least one object, each element of the tensor representing the brightness value of the corresponding pixel; - normalize the tensors obtained at the previous stage; - by means of an encoder, the mentioned two tensors are combined into one tensor; - by means of a transformer, vector representations of at least one object contained in the tensor obtained at the previous stage are compared line by line to form a tensor containing information about the values of shifts of image pixels relative to each other; - generating, by means of a decoder, the first and second shift maps for the first and second images, containing the mentioned shift values, based on the tensor obtained at the previous stage; - based on the values of the shift maps generated at the previous stage, an image depth map is formed. 2. The method according to claim 1, characterized in that the step of determining the shift value for each pixel of the first and second images is additionally performed, containing the steps of: - determine the brightness value (the amount of illumination) of each pixel of the first and second images; - the brightness values of the pixels of the first image are compared with the brightness values of the pixels of the second image to determine the shift value for each pixel of the first and second images, and the brightness values of neighboring pixels are also taken into account during the comparison. 3. The method according to claim 1, characterized in that it additionally performs the steps of checking the consistency of the pixel shift values of the left and right images. 4. The method according to claim 1, characterized in that the stage of generating an image depth map based on the values of the shift maps contains stages in which: - based on the pixel shift values contained in the shift maps, the distance from the line connecting the centers of the cameras to each pixel of at least one object is determined; - based on the obtained distance values from the line connecting the centers of the cameras to each pixel of at least one object, an image depth map is formed. 5. The method according to claim 4, characterized in that said distance is determined by the formula distance=B×f/D, where B is the size of the base (distance between cameras), f - focal length in pixels, D - shift value. 6. The method according to claim 1, characterized in that the encoder, transformer and decoder are implemented on the basis of neural networks pre-trained on the training data set. 7. The method according to claim 1, characterized in that additional steps are performed in which: - based on the depth map, a point cloud is formed in three-dimensional space; - use a point cloud to plan the trajectory of an autonomous unmanned vehicle. 8. A device for constructing a depth map from a pair of images, comprising at least one computing device and at least one memory device containing machine-readable instructions, which, when executed by at least one computing device, perform the method according to any one of claims. 1-7.

Images