RU2423018C2 - Method and system to convert stereo content - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: initial chart of disparity/depth is calculated for a stereo image from 3D video; the depth chart is smoothened; parameters of depth perception are varied in compliance with estimation of eye fatigue; a new stereo image is generated in compliance with parameters of depth perception. The system of stereo content conversion for the purpose to reduce eye fatigue when viewing 3D video comprises a unit of depth chart calculation and smoothening, a unit of depth control, a unit of visualisation, at the same time the first outlet of the calculation unit and the depth chart smoothening is connected to the first inlet of the visualisation unit, the second outlet of the depth chart calculation and smoothening is connected to the inlet of depth control unit, the outlet of which is connected to the second inlet of the visualisation unit.

EFFECT: development of a device and a method to convert stereo content for reduction of eye fatigue when viewing 3D video and provision of stable control of depth perception during demonstration at a 3D television receiver.

22 cl, 22 dwg, 1 tbl

Description

The claimed invention relates to methods and systems for processing stereo images and video information and, in particular, to methods and devices for converting stereo content in order to reduce eye fatigue when watching three-dimensional video.

It is expected that three-dimensional (surround) television will replace modern television technology, showing the viewer not only the video sequence of two-dimensional images, but also images of three-dimensional scenes. One of the requirements for the functionality of a three-dimensional television device is to provide the ability to change depth in accordance with the wishes of the viewer in relation to increased viewing comfort. To provide the ability to control the image depth, it is necessary to solve the problem of the synthesis of new species (images). New virtual views are synthesized using information obtained from the disparity / depth map, which is calculated based on incoming stereo pairs of images. The correct calculation of the disparity is a very difficult task, because the quality of the synthesized stereo pair with a changed depth largely depends on the quality of the depth map. Thus, it is required to apply a certain method of combining stereo pairs to generate an unprocessed (initial) disparity / depth map with subsequent processing in order to be able to use this method to synthesize virtual views in the process of demonstrating three-dimensional content.

The calculation of disparity or the procedure of combining a stereo pair is reduced to the task of detecting pixel-by-pixel (point-to-point) correspondence in stereo types. Two or more images from a plurality of cameras enter the input, and a communications map (disparity map) is received at the output, which displays the correspondence of each point of one image to a similar point on another image. The resulting disparity will be large for nearby objects and will be expressed as a small value for remote objects. Thus, the disparity map can be considered as an inversion of the scene depth.

Stereopair matching algorithms can be divided into local algorithms (working with the neighborhood of the current pixel) and global (working with the whole image) [4]. Local algorithms are based on the assumption that the calculated disparity function is smooth in the support window. The results of such algorithms are usually not very accurate, but acceptable for real-time use. On the other hand, global algorithms use an explicitly defined smoothness function and then solve the optimization problem. This usually requires the use of complex computational methods, such as dynamic programming or graph section algorithms.

The recently developed disparity assessment method described in US Patent Application No. 2006/0120594 [1] consists of two main steps: local alignment and global optimization. Local alignment reveals pixels with a high degree of compatibility. During this step, pixels with low compatibility are reset to zero. After that, global optimization is used to estimate the final value of the depth map. Global optimization is performed using the two-pass method of dynamic programming, with the first pass intended for the horizontal direction, and the second for the vertical direction. The disadvantage of this method is that it requires high computing resources. It is required to calculate as many levels of disparity as they are contained in the closest object in a given scene. For cameras with high resolution and large focal lengths, the number of disparities to be calculated can range from 100 to 300. In the process of local alignment, a number of directional filters are used to determine the compatibility weight. And finally, the global optimization process places high demands on computing resources, even when it is applied independently (separately).

Another solution regarding the sequential alignment of stereo pairs can be found in US patent No. 7106899 [2]. The invention uses iterative technology, which includes the principle of limiting the disparity gradient and the strategy of least commitment. The idea is to switch to a highly reliable pixel disparity map with unique alignment through multiple iterations. This method is sequential in the sense of sequentially reducing the number of pixels that will have to be analyzed at each iteration. Unambiguous pixel alignments are found using the new correlation technology and based on the correlation mark associated with pixel alignment. The disadvantage of this method is the high requirements for computing resources.

Published international patent application WO 2008/041167 [3] provides a method for calculating disparity based on iterative filtering of the initial disparity map, which is represented by a noise image, that is, an image whose pixels are randomly set in the range from minimum to maximum values. At each iteration, the current disparity score is refined by filtering in accordance with the reference stereo images. The advantage of the method is the relatively rapid achievement of an acceptable quality of the depth map, in which the boundaries of the objects are well developed based on the color of the objects. The disadvantage of this method is the inadequate assignment of depth values by color inside objects. Brighter objects or parts of objects in the color image appear closer, and darker objects will have a deeper depth. In addition, in order to achieve good results, it is required to apply a filter with a large kernel size, which necessitates significant computational resources.

In Russian patent application No. 2008144840 [5], the idea from [3] was further developed. To reduce the tendency to assign depth values only in accordance with the image colors, the raw disparity estimate was calculated in a standard way based on window-wise matching of stereo pairs. Then a filtering scheme was applied based on color information from a stereo pair. To reduce the number of incorrect depth values, the gradient limitation principle was applied [6]. To reduce the computational load, we studied the adaptation of the filter radius depending on the iteration number. For a large number of iterations, for example, over six, the algorithm runs approximately 40% faster with improved quality characteristics.

Russian patent application No. 2008140111 [7] describes a method for rapidly improving an unprocessed disparity card that results from pixel-by-pixel matching in the support windows. The idea of the method is to detect “bad pixels” in the raw disparity map, that is, pixels containing erroneous depth data. These pixels are usually located in the occlusal and low-texture areas of the image. After identifying such areas, the proposed method involves distributing the correct depth values to these areas by filtering by image color. This method uses only one color image, which can give an excellent result if the number of bad pixels in the raw disparity card does not exceed 30%.

Patent application RU 2009110511 [8] describes a system for capturing three-dimensional images and reproducing them on an autostereoscopic display. The main components of the system are the image capture unit, which receives images from stereo or multicamera; a disparity assessment unit that calculates disparity between adjacent views; a view synthesis unit that generates several views in accordance with the characteristics of the three-dimensional display of the three-dimensional reproduction unit. Appropriate methods for assessing the depth and synthesis of species are applied in such a way as to enable implementation on high-performance computing devices such as GPUs or FPGAs.

The recently published US patent application No. 2009/0129667 [9] describes a device and method for evaluating a depth map, generating an intermediate image, and encoding multiple video images. The disparity assessment is carried out in two stages. First, an untreated disparity map is computed, and then the Belief Propagation (abbreviated BP) method is used to improve the depth map. The use of BP methods provides the best results in solving the disparity assessment problem. The disadvantages of such methods are expressed in very high computational complexity and increased requirements for memory resources. Therefore, they are usually implemented as a program for a personal computer with multi-view data processing in off-line mode. For the generation of intermediate images, the “depth image based rendering” (DIBR) technology, known from [10], is used. In order to encode multiview images, a method similar to the MPEG standard is used for video compression based on a discrete cosine transformation or DCT followed by encoding of entropy.

The closest to the claimed invention features the technical solution described in published application for US patent No. 2008/0240549 [11], which proposes a method of controlling the depth of the three-dimensional effect when playing stereo images or multiview images by evaluating the disparity of the corresponding stereo images with setting the depth of the three-dimensional effect based on the histogram of disparity, and rearrangement of stereo images. The degree of depth control is determined as the result of convolution of the disparity histogram with the characteristic function. Two types of characteristic functions are described: one of them is intended only for scenes containing only a background, and the other is for video with a pronounced foreground object and background. The degree of rearrangement of the image stereo-image depends on the sum of the convolution of the characteristic function with the disparity histogram. Said patent application is selected as a prototype of the claimed invention.

The disadvantage of the prototype is as follows. In the prototype method there is no explanation as to how to classify the input video in the "background" mode or in the "foreground + background" mode. In addition, it is not clear how disparity is assessed. Calculation of the disparity card is a very difficult task, therefore, the method of calculating the disparity card must be clearly defined so that it can be used in hardware (HW), while ensuring quality not lower than in known solutions. In addition, it is very important to use as few memory resources as possible for the hardware implementation to reduce the cost of the device. Thus, there is a need to develop a depth control method based on high-quality depth generation with low requirements for memory size and the number of calculations.

The problem that is solved by the claimed invention is to create a device and method for converting stereo content to reduce eye fatigue when watching a three-dimensional video, in other words, to provide reliable control of depth perception during a demonstration on a three-dimensional television receiver. This is achieved by generating high-quality stereo images based on the calculated disparity / depth map.

The technical result is achieved by developing a method and depth control device for reproducing stereo content in three-dimensional television, the method provides for the following operations:

- perform the calculation of the original map disparity / depth for stereo images from three-dimensional video;

- perform smoothing of the depth map;

- change the parameters of depth perception in accordance with the assessment of eye fatigue;

- generate a new stereo image in accordance with the parameters of the perception of depth.

In terms of practical implementation of the proposed method, a stereo content conversion system has been developed to reduce eye fatigue when watching a three-dimensional video, including a depth map calculation and smoothing unit, a depth control unit and a visualization unit, while the first output of the depth map calculation and smoothing unit is connected to the first input of the visualization unit, the second output of the depth calculation and smoothing unit is connected to the input of the depth control unit, the output of which is connected to the second input of the unit in zualizatsii.

Calculating depth based on stereo content is a difficult task, especially for surfaces with homogeneous areas (non-textured areas), areas with a heterogeneous depth, in occlusal areas and in areas with a repeating pattern (pattern), which leads to an ambiguous solution. This means that only a few of the assigned depth values are reliable and unambiguous. Some depth values, for example, for occlusal (i.e. enclosed) areas, cannot be calculated by combining, since these areas are visible in only one image. Moreover, the synthesis procedure for a high-quality virtual view requires 1) a dense depth map; 2) the exact boundaries of the depth, which strictly coincide with the boundaries of the object; 3) aligned depths within an object.

In this regard, it is necessary to identify and correct ambiguous depth values in such a way that the synthesis procedure of the virtual form does not generate visible artifacts and ensures maximum approximation to the real depth. The prior art solutions mainly resorted to the use of various optimization strategies, such as dynamic programming, graph cross-section, stereo pairing by segmentation, etc. However, such solutions require very high computational resources and do not allow generating a smooth depth map suitable for the synthesis of species free of artifacts.

In the proposed method, efforts are aimed at rapidly improving the initial depth map in the local window, and not at using the global optimization method for calculating the disparity. The initial depth map can be obtained by standard methods of local stereo-alignment. Usually this kind of depth is very noisy, especially in areas with a weak texture and in the area of occlusion. The main idea of this method is to use a weighted average filter to smooth and improve the initial depth map based on reference color images and reliable depth pixels. In this case, it is assumed that for pixels with a similar color in some neighborhoods, the depth values are also similar. Therefore, reliable depth values can be assigned to undefined pixels based on color similarity and proximity in the reference color images. In addition, such filtering refines the pixels with reliable depth and forms a dense and smooth depth map that meets the above requirements for depth maps.

The proposed method uses several technologies for determining whether the current pixel is abnormal (unreliable) or not. Unreliable pixels are marked with some mask values to exclude them from the filtering process. Various technologies may be used to evaluate pixel reliability. In the proposed method, cross-checking the depth values on the left and on the right is used. In other words, if the difference between the depth values on the left and on the right for the corresponding points is less than the threshold value, then the depth values are considered reliable. Otherwise, they are marked as abnormal and removed from the smoothing process. However, for successful processing of anomalous pixels, filters with a large core are required in cases of occlusion of an object or noisy depth map. The proposed method is based on a recursive implementation to reduce the size of the filter core. A recursive implementation means that the filter result is written to the source buffer. This also leads to faster convergence of the algorithm with fewer iterations.

In the proposed method, the idea of detecting abnormal pixels in a depth map was analyzed. To eliminate the noisiness of the raw depth map, histogram analysis was applied. The noise figures of the depth map are represented by waves at the low and high borders of the histogram (Fig. 7, view 7.3). To remove the anomalous pixels, trimming the histogram at its borders was used. The method of trimming a histogram is suitable for implementation in hardware, since it uses local histograms based on information stored on memory lines, and therefore there is no need to work with the entire image.

To effectively eliminate the noise of the original depth map in areas with a weak texture, the depth smoothing algorithm processes such areas with stronger settings of the smoothing filter. For this purpose, a binary mask of textured and weakly textured areas of the corresponding color image is formed using a special gradient filter. The filter is suitable for implementation on hardware, as it calculates four types of gradients in the local window.

The method described in this invention allows to generate a high-quality depth map, providing a synthesis of the view with the specified depth perception parameters. 13 shows some results of applying the proposed method. These results were obtained using seven memory lines at three iterations of the algorithm. From the above examples it can be seen that the proposed method has the ability to improve even a very noisy depth map in just a few iterations when working in a local window.

Further, the essence of the claimed invention is illustrated with the use of graphic materials.

Figure 1. A block diagram of a system for converting stereo content in order to reduce eye fatigue when watching a three-dimensional video according to the invention.

Figure 2. A block diagram of a phased implementation of the method of converting stereo content in order to reduce eye fatigue when watching a three-dimensional video according to the invention.

Figure 3. A block diagram of a system for calculating a depth map and a smoothing block according to the invention.

Figure 4. A flowchart of a phased implementation of a method for smoothing a depth map based on recursive filtering according to the invention.

Figure 5. Examples of different stereo frame orientations.

6. An illustration of a depth histogram where five percent of the darkest and five percent of the brightest areas of the histogram are black.

7. Example of cropping a depth histogram: view 7.1 - color image, view 7.2 - corresponding depth map, view 7.3 - depth histogram with cropping threshold values for six percent of the darkest and three percent of the brightest pixels.

Fig. 8. Flow chart of cross depth checking method.

Fig.9. Example of cross-checking depth: view 9.1 - left image with depth, view 9.2 - right image with depth, view 9.3 - depth of the left side with anomalous pixels marked in black.

Figure 10. An example of binary image segmentation into areas with a pronounced texture and areas with a weakly expressed texture: view 10.1 - color image, view 10.2 - corresponding depth map, 10.3 - binary segmentation mask (areas with a pronounced texture are marked in black, white means areas with a weak texture).

11. Flowchart of a method for filtering depth.

Fig. 12. Illustration of the principle of depth filtering: view 12.1 - reference color image, view 12.2 - compatible color image, view 12.3 - reference depth map.

Fig.13. The results of smoothing the depth according to the invention: top view 13.1: from top to bottom: original depth map, corresponding color image, depth map smoothed by the claimed method, view 13.2 from top to bottom: original depth map, corresponding color image, depth map smoothed by the claimed method.

Hereinafter, with reference to the attached drawings, a preferred embodiment of the claimed invention is described in detail. However, the scope of protection of the claimed invention is not limited to the preferred embodiment and covers the implementation in various embodiments. The preferred implementation described in the description is only an example presented in order to disclose the essence of the invention and to help specialists fully understand the claimed invention.

Figure 1 is a block diagram illustrating a stereo content conversion system in order to reduce eye fatigue when watching a three-dimensional video according to the claimed invention. The system shown in FIG. 1 includes the following blocks connected to one another: a depth map calculation and smoothing unit 102, a depth control unit 103, and a visualization unit 104. The depth map calculation and smoothing unit 102 is designed to calculate the depth map based on the stereo image 101. In essence, a depth map is required to generate a new stereo image 105 by the visualization unit 104 in accordance with the depth sensing parameters specified by the depth control unit 103. The depth map calculation and smoothing unit 102 is described in more detail in the following paragraphs.

Figure 2 presents the process (method) of converting stereo content to reduce eye fatigue when watching a three-dimensional video according to the claimed invention. Consider a phased implementation of the method of converting stereo content to reduce eye fatigue when watching a three-dimensional video. The first step (Step 201) is the calculation of the original depth map. The calculation of the initial depth map is performed using standard methods of local stereo-alignment, known from the prior art. After the raw depth map has been calculated in the step of calculating the original depth map, the depth control method proceeds to the step of smoothing the depth map (Step 202). The purpose of this step is to remove the abnormal pixels from the raw depth map and make it suitable for representing the view in accordance with the depth perception parameters. The method of smoothing the depth map will be discussed in detail in the next subsection. The next step in the depth control method is to adjust the depth perception of the observed three-dimensional television content (Step 203). This is done by repositioning the images for the left and right eye. In the example implementation of the claimed invention, this is ensured by the fact that the perception of depth is controlled by the parameter D, which varies from 0 to 1. The parameter D corresponds to the position of the right kind. The value 1 corresponds to the configuration of the input stereo view, while the value 0 describes the case of the monocular representation when the images for the left eye and for the right eye coincide in space. An acceptable value when setting this parameter lies in the range from 0.1 to 1. In the next step (Step 204) of the depth control method, a new view for the right eye is formed based on the value of the parameter D. A new view for the right eye can be synthesized by interpolation based on on the disparity map [12], since the depth map computed in Step 203 shows the correspondence of pixels between the source images for the left and right eyes. The original image for the left eye, together with the new image for the right eye, forms a modified stereo image that has a reduced parallax compared to the original stereo image. The generated stereo image with reduced parallax reduces eye fatigue during long-term viewing of three-dimensional television.

Figure 3 is a block diagram illustrating the structure of a system for smoothing a depth map based on recursive filtering according to the claimed invention. The system for smoothing a depth map shown in FIG. 3 includes the following blocks connected to each other, a preprocessing unit 320, an initial depth map calculating unit 330, a depth map smoothing unit 340, and a time filtering unit 350. A stereo image 310 is used as input to the smoothing system. The stereo image can be presented as a separate image or composed on the basis of video frames received from a stereo camera. In the case of shooting with multiple cameras, a pair of images from the selected cameras can form a stereo image. After performing the necessary calculations, the smoothing system of the depth map generates a dense depth map 307 for the selected species.

The pre-processing unit 320 is intended for preparing a stereo image 310 for the purpose of subsequent efficient processing in the initial depth map calculation unit 330 and in the depth map smoothing unit 340. The preprocessing unit 320 includes a stereo image preprocessing unit 321 and a reference image segmentation unit 322. The stereo imaging unit 321 has two main functions. Firstly, he selects from the original stereo image individual images corresponding to each type. Images are divided into reference and combined. The reference image 360 is an image formed from a stereo pair for which the depth map will be smoothed. And the combined image 370 is another image of a stereo pair. Accordingly, the reference depth map is a depth map related to the reference image, while the overlapping depth map is a compatible image map.

The input video stream can be encoded in various formats. The most common formats are based on the use of left-right orientation, top-bottom orientation, checkerboard format and left-right orientation with the division of frames in the time domain. Examples of left-right orientation (501) and top-bottom orientation (502) are shown in FIG. 5. To better calculate the depth map, the original color images should be processed with a spatial filter that eliminates noise. For this purpose, you can use the Gaussian filter. However, any other filter is applicable for this task. This functionality is implemented in block 321. The segmentation block 322 of the reference image 322 generates a reference binary mask 302. This binary mask corresponds to image segmentation in areas with a pronounced and weakly pronounced texture. The pixels of the binary mask are indexed as a unit if the area is considered as a region with a weak texture. Otherwise, the pixels of the mask are indexed as zero if the area is considered to have a pronounced structure. To establish the severity of the texture, use a gradient filter in the local window.

The function of the block 330 calculating the original depth map is to make an approximate calculation of the depth map using standard methods of local alignment [4]. This is implemented in block 331 for calculating a reference depth map and in block 332 for calculating a compatible depth map. Other functionality of the source depth map calculation unit relates to detecting abnormal pixels on an approximate depth map. Trimming the histogram of the depth map is performed in the histogram analysis section of the reference depth map, while the cross-checking of the depth map is performed by the checking block 334. At the output of block 330, the reference and compatible depth maps with marked anomalous pixels 305 are formed.

The depth map smoothing unit 340 is designed to improve the depth map by recursively filtering the raw depth maps 305 (compatible and reference). The depth map is filtered in block 342 depth filtering. The number of iterations required is set by the iteration control block 341. During each iteration, block 341 calculates convergence criteria for the filtering process. In a preferred embodiment of the invention, two embodiments of the convergence criteria are described. The first version of the convergence criterion calculates the afterimage between adjacent calculations of the disparity map. The sum of residual pixels must not exceed the convergence threshold T _{dec1 of the disparity} map calculation. The convergence criterion can be formulated in another way, as the number of iterations of filtering the depth map. If the number of iterations exceeds the threshold T _{dec2 of the} convergence of the disparity map computations, then the filtering process is stopped.

The post-processing block 343 is intended for the final refinement of the calculated depth maps. In a preferred embodiment of the claimed invention, the post-processing unit 343 consists of a median filter unit. Median filtering is known in the art, so this block is not described here. Other types of filters to improve image quality can also be applied in block 343, but this does not change the purpose of the block. In the end, you need to get smoothed depth maps 306 (reference and compatible).

Block 350 time filter is designed to filter the depth map by time. The temporal filter block 350 consists of a frame buffer 351 that stores a number of depth frames with corresponding color images, and a temporal filter block 352 that performs interframe filtering of the depth map using information from the corresponding color images.

Figure 4 illustrates in detail the inventive method of smoothing a depth map based on recursive filtering. Consider a phased implementation of the method of smoothing depth, based on recursive filtering. The first step (Step 401) consists of preprocessing color images. Preliminary processing consists in filtering color images with a Gaussian filter in a small window, for example, 5 × 5 pixels. The result of filtering is noise suppression of color images. This significantly affects the quality of smoothing the depth map, since when smoothing the depth map, weighted averages of neighboring pixels are used, and the weights are calculated based on color images. The next stage of the depth smoothing method is the analysis and cropping of the histogram of the depth reference map (Step 402). Trimming the histogram is performed to suppress the noise of the depth map. A raw depth map can contain a significant number of abnormal pixels. Noise may occur due to improper combination in the occlusal areas and in areas with a weak texture. The inventive method involves the use of two threshold values: threshold B in the lower part of the histogram and threshold T in the upper part of the histogram. These thresholds are calculated automatically from the given numbers α and β% for abnormal pixels, where α corresponds to the ratio of image pixels that lie below the histogram edge to all pixels of the image. And β corresponds to the ratio of image pixels that lie above the top of the histogram to all pixels in the image. In this case, the thresholds B and T are calculated as follows

where H (s) is the value of the histogram;

M - maximum pixel level, for a single-byte representation, this value is 255;

N _x is the image width;

N _y - image height;

An example of thresholds corresponding to α = β = 5% of the image pixels is shown in FIG. 6. In this case, B is 48, and T is 224. Another example of trimming the depth map histogram is shown in FIG. 7. The histogram of FIG. 7 is plotted using all image data. However, for hardware implementation, a local histogram can be calculated based on the information currently contained in the memory lines.

The next step in the depth smoothing method is to check the consistency (uniformity) of the depth map (Step 403). The procedure identifies "consistent" pixels, that is, pixels for which the depth map is calculated correctly. The proposed method for smoothing the depth map is based on crosschecking, which allows to identify abnormal pixels.

According to Fig.8, the procedure is implemented as follows:

1. The reference disparity vector (RDV) vector is calculated based on the values of the depth reference map;

2. Retrieve the value of the combined depth map displayed through the RDV;

3. The matching disparity vector (MDV) vector is calculated based on the values of the combined depth map;

4. The difference of disparity difference (DD) maps of the absolute values of RDV and MDV are calculated;

5. If DD exceeds the threshold value, then the pixel of the depth reference map is marked as abnormal.

An example of a depth map with marked "noise" pixels according to the procedure for cross-checking depth is shown in Fig. 9, view 9.3. To obtain the result in Fig. 9, view 9.3, the threshold value for cross-checking the disparity map is set to two, and the "noise" pixels are marked as zero.

The next step of the depth smoothing method is to perform binary segmentation of the reference color image into areas with a pronounced and weakly pronounced texture (Step 404). For this purpose, gradients are calculated in four directions. These directions are horizontal, vertical and diagonal. Gradients are calculated as the sum of the absolute differences of neighboring pixels in the respective directions. If the values of all the gradients are lower than the predetermined threshold value, then such pixels are considered as having a weakly pronounced texture, otherwise they are considered as having a pronounced texture. This can be formulated as follows

where BS is a segmentation binary mask for a pixel with coordinates (x, y), a value of 255 corresponds to a pixel of an image with a weak texture, while a value of 0 corresponds to a pixel with a pronounced texture.

10 illustrates an example of a color image with a binary segmentation mask. Figure 10 shows that the applied method of segmentation successfully segments the image in areas with a weak texture and areas with a pronounced texture. At the same time, the method reveals the boundaries of the subject, for example, between the ramp and the seabed (marked with dark ovals). This is an important feature, since the boundaries of the subject must be carefully processed on the depth map to avoid artifacts.

After binary segmentation of the left color image in areas with a weakly pronounced texture and with a pronounced texture, the smoothing method proceeds to the main filtering cycle (Steps 405-408). At the beginning of the processing, the iteration index is set to zero. Then it is increased after each iteration of smoothing. When the index becomes equal to the number of iterations, the filter cycle opens.

Before this happens, the next step in the depth smoothing method reveals the type of pixel in accordance with the binary segmentation mask (Step 406). If a pixel is recognized as having a pronounced texture, then a depth map anti-aliasing filter with default settings is applied (Step 408). Otherwise, the pixel is recognized as having a weak texture, and a depth map anti-aliasing filter is applied with settings for stronger anti-aliasing that provide better noise reduction (Step 407).

The operation algorithm of the smoothing filter of the depth map is presented in Fig.11. It explains how it can be effectively applied in hardware. The algorithm works with memory buffers, which write corrections to local images, and not the entire image. The table shows the memory buffers used in the description of the algorithm.

Table Memory buffers Memory buffer index Description of stored (stored) data Buffer size one Local area from the reference color image Kernel size * Number of lines * Number of color channels 2 Local area from the reference depth map Kernel Size * Number of Rows 3 Pixels from a color matching image displayed by a disparity reference map vector Kernel size * Number of lines * Number of color channels

Consider a phased implementation of a filtering method to smooth a depth map. At the input of the algorithm, you need to have a stereo pair of color images (left and right) and an raw depth map computed for one of the color images. According to our above classification, an image from a stereo pair for which a depth map is smoothed is called a reference color image (RCI), while another image is called a matching color image (MCI). Accordingly, a smoothed depth map is called a reference depth map (RD). We will consider the left raw depth map as a reference depth map, and the processing for the right raw depth map is similar. 11 shows one smoothing iteration. If the number of iterations is more than one, then you should process the entire image of the depth map, write the result into RD memory, and then use the same memory buffer with updated data at the input.

According to FIG. 11, in the first step of the algorithm (Step 1101), it is necessary to copy the pixel region from the reference color image (RCI) to memory 1 for further processing. The height of the window should be equal to the number of available lines. In the next step (Step 1102), the algorithm copies the pixels from the reference depth map (RD) to memory 2. The next step is to check whether the pixel from the raw depth map is abnormal or not (Step 1103). For this purpose, threshold values B and T are used, calculated by analyzing the histogram.

The equation for checking the range of the depth map is as follows:

where d (x, y) is a pixel of the raw depth map with coordinates (x + x1, y + y1);

(x, y) are the image coordinates of the current pixel of the depth map for which filtering is performed, while x1, y1 are the pixel indices of the reference depth map recorded in memory 2.

If inequality (1) is not satisfied, then the corresponding pixel of the map of depth d (x + x1, y + y1) is not taken into account for filtering the pixel d (x, y) (Step 1104), and the next pixel from memory 2 is checked for the subject of abnormality, performing such an operation until all the pixels from memory 2 are checked. If all pixels are considered abnormal, the current pixel of the depth map is left untouched. Typically, a raw depth map contains a large number of erroneous pixels. To facilitate effective filtering of such areas by a filter with a small window, we use a recursive filter, that is, the result of filtering the current pixel is written to the original depth map. This allows you to extend the correct depth map values to erroneous areas.

The next step of the algorithm is to calculate the disparity map values based on the pixels of the depth map recorded in memory 2 (Step 1105). After calculating the disparity map vectors from the values of the depth map, the corresponding disparity maps are used as coordinates for color pixels in a matching color image (MCI). After that, the pixels from the MCI represented by the disparity card are copied to memory 3 (Step 1106).

According to Fig. 12, the main idea of the inventive depth map smoothing device is to refine the raw reference depth map by applying weighted averaging of pixels of the depth map located in the filter window (1201). Filter weights are calculated using information obtained from color images. 12, the current pixel with respect to which filtering has been performed is marked in black (1202, 1203, 1204). In all images (12.1-RCI, 12.2-MCI, 12.3-RD) the spatial coordinates of this pixel are the same. To calculate the weight, the smoothing filter must calculate two color distances. The following is a description of the procedure for calculating such distances.

The first color distance between the current color pixel IC (1202) and the reference pixel Ir (1205) is calculated in Step 1107 of the depth map smoothing algorithm. Both pixels are stored in memory 1. The current metric is the Euclidean distance and is calculated as follows

where the quadratic difference of each color channel is summed, after which the square root is extracted from it. 12, an arrow shows which pixels the 1st color distance is calculated between.

The next step of the algorithm is to calculate the second color distance between the reference pixel Ir (1205) and the final pixel It (1206) (Step 1108). A final pixel is a pixel in a composite image that is displayed by the Ir pixel disparity map vector. Since this disparity map is one-dimensional, that is, a horizontal disparity map, Ir and It will lie on the lines with the same indices (1207) (Fig. 12).

Equation (2) can be used to calculate the color distance. 12, an arrow indicates between which pixels the second color distance is calculated.

After the calculation of the two color distances is completed, the pixel weight of the depth reference map is calculated as follows (Step 1109)

where C () means the function used to compare the color of the pixels,

e is the Euler number with a numerical value of 2.718 ...,

σ _r is the smoothing parameter of the depth map for the reference pixel in the reference image,

σ _t is the smoothing parameter of the depth map for the final pixel in the combined image,

(x _r , y _r ) - coordinates of the reference pixel,

(x _t , y _t ) - coordinates of the final pixel. For a one-dimensional disparity map, y _t = y _r .

Upon completion of the weight calculations for each pixel of the depth reference map, the algorithm proceeds to the calculation of the weighted averaging (Step 910). The weighted averaging value is calculated as follows

where d _out (x _c , y _c ) means the result of smoothing the depth map for the current pixel with coordinates (x _c , y _c ),

d _in (x _r , y _r ) means the raw depth map for the reference pixel with coordinates (x _r = x _{c + p} , y _r = y _{c + s} ),

w _r denotes the pixel weight of the depth reference map,

до

в направлении X,index p varies from

before

in the direction of X,

до

в направлении Y,index s varies from

before

in the direction of Y

.the normalization factor is calculated as

.

And finally, the filter result d _out (x _c , y _c ) is stored in the RD memory (Step 1111).

After performing a certain number of iterations on smoothing filtering of the depth map, the algorithm for smoothing the depth map proceeds to the post-processing of the reference depth map (Step 909). In the inventive method, a median filter is used for this, which makes it possible to efficiently remove impulse noise of the disparity card. After the reference depth map is smoothed out during post-processing, it is entered into the RD memory (Step 1110).

To eliminate the effect of flicker (bounce) while watching a three-dimensional video, a time filter in the form of a moving average can be applied to the depth map. The filter uses several smoothed depth maps, which are stored in the frame buffer 351, and at the output gives a filtered depth map frame for the current time stamp.

Fig.13 shows some results of the application of the proposed method. The results were obtained using seven memory lines with three iterations of the algorithm. It can be seen from these examples that the proposed method makes it possible to improve even a very noisy depth map in just a few iterations, working in a local window.

The main area of industrial application is the content processing unit in three-dimensional television devices. Currently, one of the main tasks in three-dimensional television is to remove user complaints related to eye fatigue when watching television. The main reason for the occurrence of such a problem is that people have individual differences and preferences in the perception of stereoscopic images. Their gender, age, race can influence their preferences in stereoscopy, since each individual is unique in their unique characteristics of the visualization system. Inappropriate content when transmitting stereo sequences leads to eye fatigue. This is a complex phenomenon that depends on many parameters, such as high parallax values, crosstalk, conflict between depth signals, and so on.

The depth control function to reduce eye fatigue can be implemented in two different scenarios. First, manual tuning is performed when the user has some means of control and can switch parameters according to his own personal preferences, so that viewing is more convenient for the eyes. The second scenario is the use of a function that reduces eye fatigue, allows you to control depth and increase comfort when watching three-dimensional television broadcasts. The depth enhancement function is used after calculating the depth map to pre-process the depth parameters before changing the depth map or showing new frames.

The invention may also find direct application in stereo cameras to create a high-quality and reliable map of disparity / depth. In addition, the claimed invention can find application in multi-camera systems or in special devices in which two separate video streams form a stereoscopic stream, and stereo alignment is required.

References

[1] US patent application No. 2006/0120594, J. C. Kim et al. "Apparatus and Method for Determining Stereo Disparity based on Two-path Dynamic Programming and GGCP."

[2] US patent No. 7106899, Z. Zhang, Y. Shan "System and Method for Progressive Stereo Matching of Digital Images".

[3] WO patent application No. 2008/041167, F. Boughorbel, "Method and Filter for Recovery of Disparities in a Video Stream".

[4] D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense two-frame stereo correspondence Algorithms." In IJCV, volume 47 (1), pages 7.42, 2002.

[5] RU patent application No. 2008144840 "Method and Apparatus for Disparity Estimation and Filtering from Stereo Content" from November 14, 2008 by A. Ignatov, V. Bucha.

[6] S. P. Pollard et al. "PMF: a stereo correspondence algorithm using a disparity gradient constraint". Perception, 14: 449-470, 1985.

[7] RU patent application No. 2008140111 "Method of a disparity map refinement and apparatus for implementing the same method" from October 10, 2008 by V. Bucha, A. Ignatov.

[8] RU patent application No. 2009110511 "System and method for the three-dimensional video acquisition and reproduction" from March 24, 2009 by A. Ignatov, V. Bucha, M. Rychagov.

[9] US patent application No. 2009/0129667, Y. S. Ho et al. "Device and Method for Estimating Depth Map, and Method for Generating Intermediate Image and Method for Encoding Multi-view Video Using the Same."

[10] L. Zhang et al. "Stereoscopic Image Generation Based on Depth Images for 3D TV", IEEE Trans. on Broadcasting, 2005, vol. 51, pp. 191-199.

[11] US patent application No. 2008/0240549, J. P. Koo et al. "Method and Apparatus for Controlling Dynamic Depth of Stereo-view or Multi-view Sequence Images."

[12] US patent application No. 2007/0047040, T. N. On "Apparatus and Method for Controlling Depth of Three-dimensional Image".

Claims (20)

1. The method of converting stereo content to reduce eye fatigue when watching a three-dimensional video, including the following operations:
- calculate the original disparity / depth map for the stereo image from three-dimensional video, pre-processing the original color images with a spatial filter configured to eliminate noise;
- perform smoothing of the depth map by performing the following operations:
- analyze and crop the histogram of the reference depth map;
- check the consistency of the depth map;
- form a binary mask of the reference color image according to areas with a pronounced texture and areas with a weakly expressed texture;
- smoothing of reference and combined depth maps by successive iterations of filtering depth maps;
- perform filtering of the reference depth map in accordance with the binary mask of the reference image in areas with a pronounced texture and areas with a weakly expressed texture;
- carry out post-processing of reference and compatible depth maps;
- perform temporary filtering of reference and combined depth maps;
- change the parameters of depth perception in accordance with the assessment of eye fatigue, while the depth perception parameter is represented by the parameter D, which varies from 0 to 1, and the parameter D corresponds to the position of the view for the right eye, and the value 1 corresponds to the configuration of the original stereo image, while value 0 describes the case of a monocular view, when the images for the left and right eyes coincide in space, while the corresponding settings of this parameter are in the range from 0.1 to 1;
- generate a new stereo image in accordance with the parameters of the perception of depth. 2. The method according to claim 1, characterized in that the depth perception parameters are changed in accordance with the wishes of the user. 3. The method according to claim 1, characterized in that the view for the right eye is synthesized using interpolation based on a disparity map, while the position of the view for the right eye is described by parameter D. 4. The method according to claim 3, characterized in that the synthesized view for the right eye is used together with the original image for the left eye to form a modified stereo image having reduced parallax compared to the image of the original stereo view. 5. The method according to claim 1, characterized in that the histogram of the depth map is cut off by threshold values B and T, which are calculated as follows:

,
where H (s) is the value of the histogram; M is the maximum pixel level, which for a single-byte representation is 255; N _x is the width of the plot; N _y is the height of the plot; α is a value characterizing the ratio of image pixels that lie below the edge of the histogram to all pixels of the image; β is a value characterizing the ratio of image pixels that lie above the top of the histogram to all pixels of the image. 6. The method according to claim 1, characterized in that the consistency check of the depth map is performed using a cross-check of the depth map. 7. The method according to claim 1, characterized in that the binary mask of the reference color image is defined as

,
where BS is a binary segmentation mask for a pixel with coordinates (x, y), a value of 255 corresponds to a pixel of an image with a weak texture, and a value of 0 corresponds to a pixel of an image with a pronounced texture, gradients (x, y) are a function for evaluating horizontal gradients , verticals and diagonals, while the gradients are calculated as the sum of the absolute differences of neighboring pixels in the corresponding directions, and the values of the gradients must lie within GradTh to recognize the site as a site with a weakly pronounced texture, in contrast Mr event site acknowledge plot with a pronounced texture. 8. The method according to claim 1, characterized in that the filtering of the disparity map at the k-th iteration is expressed as

,
where d _k (x _c , y _c ) means the depth map at the kth iteration for the current pixel with coordinates (x _s , y _s ), d _k-1 (x _r , y _r ) means the depth map at (k-1 ) th iteration for the reference pixel with coordinates (x _r = x _{c + p} , y _r = y _{c + s} ), w _r (x _r , y _r ) means the weight of the reference pixel, the index p changes from

before

in the X direction, the index s changes from

before

in the Y direction, the normalization factor is calculated

. 9. The method according to claim 8, characterized in that the weight of the depth map filter is calculated as follows:

,
where C () means the function used to compare pixels, e is the Euler number, σ _r is the parameter for controlling the weight of the reference pixel in the reference image, σ _t is the parameter for controlling the weight of the target (final) pixel in the combined image, (x _r , y _r ) - coordinates of the reference pixel, (x _t , y _t ) - coordinates of the target (final) pixel. 10. The method according to claim 9, characterized in that the function used to compare the pixel is defined as

,
where I _T (x _s , y _s ) means the intensity of the current pixel in the corresponding color channel, I _T (x _r , y _r ) indicates the intensity of the reference pixel in the corresponding color channel. 11. The method according to claim 9, characterized in that the filter weight w _r (x _r , y _r ) is zeroed if the corresponding pixel of the depth map is abnormal, using the following ratio:
if ((d (x _r , y _r ) <B) OR (d (x _r , y _r )> T)
w _r (x _r , y _r ) = 0, where d (x _r , y _r ) is the pixel of the depth reference map, w _r (x _r , y _r ) is the weight of the depth depth map, B and T are the threshold values obtained when processing a histogram. 12. The method according to claim 9, characterized in that for the filter parameters σ _r and σ _t apply different settings in accordance with the binary image segmentation in areas with a pronounced texture and with a weakly pronounced texture. 13. The method according to claim 1, characterized in that the post-processing of the depth map is performed using a median filter. 14. The method according to claim 1, characterized in that the temporary filtering is performed using a moving average filter. 15. A system for converting stereo content in order to reduce eye fatigue when watching a three-dimensional video, including
- a unit for calculating and smoothing the depth map, configured to analyze and crop the histogram of the reference depth map, check the consistency of the depth map, generate a binary mask of the reference color image according to areas with a pronounced texture and areas with a weakly expressed texture, smoothing the reference and compatible depth maps by successive iterations of filtering depth maps, filtering the reference depth map in accordance with the binary mask of the reference image in areas with a pronounced Sture and portions with weak structure, post-processing and the reference mating depth maps, temporal filtering the reference and mating depth map;
- depth control unit, configured to change the depth perception parameters in accordance with the assessment of eye fatigue, while the depth perception parameter is represented by the parameter D, which varies from 0 to 1, and the parameter D corresponds to the view position for the right eye, and the value 1 corresponds to the configuration the initial stereo image, while the value 0 describes the case of a monocular view, when the images for the left and right eyes coincide in space, while the corresponding settings of this pair meter lying in the range from 0.1 to 1;
- a visualization unit configured to display the resulting image;
the first output of the unit for calculating and smoothing the depth map is connected to the first input of the visualization unit, the second output of the unit for calculating and smoothing the depth map is connected to the input of the depth control unit, the output of which is connected to the second input of the visualization unit. 16. The system of clause 15, wherein the depth map calculation and smoothing unit includes a preprocessing unit, an initial depth map calculating unit, a depth map smoothing unit, and a time filter unit, wherein the first output of the preliminary processing unit is connected to an input block for calculating the initial depth map, the output of which is connected to the first input of the smoothing block of the depth map, the second output of the preliminary processing unit is connected to the second input of the smoothing block of the depth map, the output of which is It is connected to the input of the temporary filter unit. 17. The system according to clause 16, wherein the pre-processing unit includes a stereo image pre-processing unit and a reference image segmentation unit, wherein the input of the pre-processing unit coincides with the input of the stereo-image pre-processing unit, the output of the stereo-image pre-processing unit is connected to the input block segmentation of the reference image, and the output of the block segmentation of the reference image coincides with the output of the pre-processing unit. 18. The system of clause 16, wherein the initial depth map calculation unit includes a depth reference map calculation unit, a compatible depth map calculation unit, a histogram analysis map of the depth reference map, and a depth map consistency check unit, wherein the first input of the unit the calculation of the initial depth map coincides with the input of the reference depth map calculation unit, the output of the reference depth map calculation unit is connected to the histogram analysis unit of the reference map, the second input of the initial depth map calculation unit coincides with the input of the compatible depth calculation unit, the output of the compatible depth calculation unit is connected to the first input of the depth map consistency check unit, the output of the depth chart reference histogram analysis unit is connected to the second input of the depth map consistency check unit, and the output of the depth map consistency check unit coincides with the output block calculating the original depth map. 19. The system according to clause 16, wherein the smoothing unit of the depth map includes an iteration control unit, a depth filtering unit and a post-processing unit, while the input of the depth map smoothing unit coincides with the input of the iteration control unit, the output of the iteration control unit connected to the input of the depth filtering unit, the output of which is connected to the input of the post-processing unit, the output of which coincides with the output of the smoothing unit of the depth map. 20. The system according to clause 16, wherein the temporary filter block includes a frame buffer and a temporary depth filter block, while the input of the frame buffer coincides with the input of the temporary filter block, the output of the frame buffer is connected to the input of the temporary depth filter block, output which matches the output of the time filter block.

Images