TWI748426B - Method, system and computer program product for generating depth maps of monocular video frames - Google Patents

Abstract

The proposed invention targets on an efficient 2D-to-3D conversion system, which can help stereo video content providers convert 2D content to 3D content with a small amount of labor cost. With AI-based keyframe selection network and AI-based depth map generation network for all 2D image frames, the proposed system can generate their corresponding depth maps to achieve a high-efficiency, high-quality and low-cost 2D-to-3D video conversion. Furthermore, the proposed system can be applied to all 2D videos, including animation or nature types of scene and all kinds of frame resolution and size.

Description

Single-view image depth map sequence generation method, system and computer program product

The present invention relates to a technology for converting two dimensional (2D) images into three dimensional (3D) images.

With the vigorous development of 3D display technology in recent years, the generation of rich 3D image content has become an important issue. Currently we have a wealth of 2D movies, but the resources of 3D images are still very scarce in today's situation. It is because the production of 3D images requires multiple cameras to shoot, which not only requires complex corrections, but also difficult and tiring shooting, and the cost of multiple cameras. It is also very expensive. Therefore, converting existing 2D images to 3D images should be the best solution. The existing 2D 3D image generation technology is operated by humans, using the color image RGB pixels before and after the large change, select the key image, and then use the post-production tool of cutting the image to generate the depth map of the key image frame, and then follow the clues before and after. , Based on the block movement direction and the block matching method, all the depth maps between the front and rear frames are obtained by interpolation. However, the key picture selected based on the pixel difference, using block motion or matching prediction, will be limited by the large amount of movement of the front and rear pictures Impact, when the amount of exercise is too large, it will cause a large number of key pictures to be selected and serious prediction errors. Therefore, in order to avoid this problem in the prior art, the front and rear frames selected during post-production cannot be too long, and the number of manual markings is about 30% to 50% of the overall picture (that is, assuming 30 frames per second) Picture, 9~15 pictures must be manually marked every second), which is a huge labor cost and consumption. In addition, the pixel matching and motion prediction methods have no surrounding information, and the effect is not good. The block prediction method is prone to errors in areas where the screen has gradient changes and object deformations.

The embodiment of the present invention proposes a method for generating a single-view image depth map sequence, which includes: obtaining multiple pictures, and converting each picture into a feature map according to a feature converter; The feature maps are divided into multiple groups, and the images corresponding to the group's feature maps are obtained as multiple key images; a user interface is provided to obtain multiple depth maps of the key images; initialize according to the feature converter A depth map generation network, and train the depth map generation network according to the key pictures and the depth map; and input the other pictures in the picture except the key pictures to the depth map generation network to calculate the corresponding depth map.

In some embodiments, the single-view image depth map sequence generation method further includes: training an autoencoder according to the picture. The autoencoder includes a feature converter and a feature inverse converter. The feature converter and the feature inverse converter respectively include corresponding Neural network.

In some embodiments, the single-view image depth map sequence generation method It also includes: setting the number of multiple candidate groups, for each number of candidate groups, performing an unsupervised clustering algorithm according to the feature map to calculate the contour coefficient; and setting the number of candidate groups corresponding to the maximum contour coefficient to the unsupervised grouping calculation The number of groups of the law, the number of groups is the same as the number of key pictures.

In some embodiments, the aforementioned unsupervised clustering algorithm is a k-means algorithm.

In some embodiments, the depth map generation network includes a time-space similarity operation, and the time-space similarity operation is used to receive the main feature map and multiple auxiliary feature maps adjacent to the main feature map in time sequence. For the main feature points in the main feature map, the space-time similarity operation obtains multiple auxiliary feature points in a preset range in the auxiliary feature map, and calculates the similarity between the main feature points and the auxiliary feature points to compensate the main feature points.

From another perspective, the embodiment of the present invention also provides a computer program product, which is loaded and executed by a computer system to complete the above-mentioned single-view image depth map sequence generation method.

From another perspective, the embodiment of the present invention also provides a single-view image sequence depth generation system, including a memory and a processor. The memory is used to store a plurality of instructions, and the processor is used to execute these instructions to complete the above-mentioned single-view image depth sequence generation method.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

100: Single-view image depth map sequence generation system

110: processor

120: memory

200: Image sequence

201, 202: Select screen

310: Key Picture Extraction System

320,330: steps

311: Key Picture

312: Screen

321: Depth Map

340: Depth Map Generation Network

341: Depth Map

410: feature converter

420: feature inverse converter

421: Image Sequence

510: feature map

511: Feature Map Group Heart

520: step

610: feature converter

620: Depth Generator

701~703: Input screen

710: Feature Extraction Convolutional Neural Network

711~713: feature map

714: Feature Points

715: preset range

720: Time and Space Similarity Operations

730: Convolutional Neural Network

[Fig. 1] is a schematic diagram showing a single-view image depth map sequence generating system according to an embodiment.

[Fig. 2] is a schematic diagram showing multiple frames in an image sequence according to an embodiment.

[Fig. 3] is a schematic flowchart of a method for generating a single-view image depth map sequence according to an embodiment.

[Fig. 4] is a schematic diagram of a self-encoder according to an embodiment.

[Fig. 5] is a schematic diagram showing an unsupervised grouping algorithm according to an embodiment.

[Fig. 6] is a schematic diagram showing a depth map generation network according to an embodiment.

[Fig. 7] is a schematic diagram illustrating a space-time similarity operation according to an embodiment.

[Fig. 8] is a schematic diagram showing the result of generating a depth map according to an embodiment.

FIG. 1 is a schematic diagram of a single-view image depth map sequence generating system according to an embodiment. Please refer to FIG. 1, the single-view image depth map sequence generation system 100 can be a smart phone, a tablet computer, a personal computer, a notebook computer, a server, an industrial computer, or various electronic devices with computing capabilities, etc. This disclosure is not here. limit. The single-view image depth map sequence generating system 100 includes a processor 110 and a memory 120, and the processor 110 is electrically connected to the memory 120. The processor 110 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, a digital signal processor, an image processing chip, a special application integrated circuit, etc., and the memory 120 may be volatile Memory or non-volatile memory. A plurality of instructions are stored in the memory 120, and the processor 110 executes these instructions to complete a single-view image depth map sequence generation method. This method is used to generate an image The image sequence corresponds to the depth map sequence of the picture. For example, please refer to FIG. 2, which is a schematic diagram showing multiple frames in an image sequence according to an embodiment. In this embodiment, the content of the image sequence 200 is an animation, but in other embodiments, natural scenes can also be used, and the present disclosure does not limit the content of the image sequence 200. In this embodiment, each picture is colored, that is, it includes channels such as red, green, and blue, but in other embodiments, each picture may also be grayscale.

FIG. 3 is a schematic flowchart of a method for generating a single-view image depth map sequence according to an embodiment. 3, the key picture extraction system 310 is a software module used to obtain a few key pictures 311 from the image sequence 200. These key pictures 311 are the most characteristic pictures in the image sequence 200. The key picture extraction system 310 How to obtain the key picture 311 will be explained in the subsequent paragraphs. The user can use step 320 to mark the corresponding depth values of these key frames 311 according to the provided user interface, thereby obtaining the depth map 321 of each key frame. In some embodiments, the user interface is used to allow the user to determine which objects are first and which objects are behind, or to allow the user to determine the depth value. However, the present disclosure does not limit the content of the user interface. In step 330, a depth map generation network 340 is trained based on the key image 311 and its depth map 321. Then, the images 312 other than the key images 311 in the image sequence 200 can be input to the depth map generation network 340 to generate the corresponding depth map 341. The key picture extraction system 310 will be described below first.

”便表示特徵圖的高度為

，寬度為

，通道數為32。經過這些卷積層與池化層的計算以後可得到高度為H/128、寬度為/128，通道數為512的特徵圖510以輸出至特徵逆轉換器420。特徵逆轉換器420是要根據這些特徵圖510重建出高度為H，寬度為W的重建影像序列421，這些影像序列421中的畫面應該要近似於影像序列200中原本的畫面，因此計算對應兩張畫面之間的損失(loss)可以更新自編碼器中特徵轉換器410與特徵逆轉換器420的權重，當影像序列421中的畫面非常接近或相同於影像序列200中的畫面時便表示訓練完畢。然而，本領域具有通常知識者當可理解自編碼器，在此不再詳細贅述。值得注意的是，在此是根據影像序列200中的畫面來訓練自編碼器，但這個訓練是非監督式的，並不需要使用者介入。此外，圖4中關於7層卷積神經網路的架構僅是 範例，在其他實施例中可以採用其他層數及其他架構的卷積神經網路。 4, the key picture extraction system 310 includes a feature converter 410 and its corresponding feature inverse converter 420. The feature converter 410 compresses each frame in the image sequence 200 into a lower-dimensional feature map 510, and the feature map 510 can be decompressed by the feature inverse converter 420 to reconstruct the image sequence 421. Both the feature converter 410 and the feature inverse converter 420 include their own neural networks, and the two can also be collectively referred to as an auto encoder. In this embodiment, the height of each frame in the image sequence 200 is H and the width is W, where H and W are positive integers, and each frame includes three channels of red, green, and blue. The feature converter 410 includes a 7-layer convolutional layer and a 7-layer pooling layer. For example, "Conv.Block_1" in FIG. 4 represents a convolutional layer and a pooling layer, and so on, as shown in FIG. 4 Shows the size of each feature map, such as "

"Means that the height of the feature map is

, The width is

, The number of channels is 32. After calculation of these convolutional layers and pooling layers, a feature map 510 with a height of H/128, a width of /128, and a channel number of 512 can be obtained to output to the feature inverse converter 420. The feature inverse converter 420 is to reconstruct the reconstructed image sequence 421 with height H and width W based on these feature maps 510. The images in these image sequences 421 should be similar to the original images in the image sequence 200, so the calculation corresponds to two The loss between images can update the weights of the feature converter 410 and the feature inverse converter 420 in the encoder. When the images in the image sequence 421 are very close to or the same as the images in the image sequence 200, it means training complete. However, those with ordinary knowledge in the art should understand the autoencoder, and it will not be described in detail here. It is worth noting that here, the autoencoder is trained based on the images in the image sequence 200, but this training is unsupervised and does not require user intervention. In addition, the architecture of the 7-layer convolutional neural network in FIG. 4 is only an example. In other embodiments, a convolutional neural network with other layers and other structures may be used.

Fig. 5 is a schematic diagram illustrating an unsupervised grouping algorithm according to an embodiment. 4 and 5, after the self-encoder is trained, a feature converter 410 can be obtained from the self-encoder, and then each picture in the image sequence 200 is converted into a corresponding feature according to the feature converter 410 Figure 510. Next, in step 520, an unsupervised grouping algorithm is performed on the feature map 510 to obtain the feature map cluster center 511, and the corresponding picture is selected as the key picture 311. For example, the above-mentioned unsupervised clustering algorithm is the k-means algorithm. This algorithm can divide the feature map 510 into multiple groups. In Figure 5, each particle corresponds to one For the feature maps of the screen, each group has a feature map cluster center 511, and the screens corresponding to these feature map cluster centers 511 can be regarded as the key screen 311.

In some embodiments, the number of groups in the above-mentioned unsupervised grouping algorithm (that is, the positive integer k in the k-means algorithm) can also be automatically calculated. Specifically, the number of multiple candidate groups can be set first, for example, from 2 to N, where N is any suitable positive integer. The number of each candidate group can be regarded as the above-mentioned positive integer k to perform the k-means algorithm, and a silhouette coefficient (Silhouette Coefficient) can be calculated according to the grouping result. The calculation of the contour coefficient is as shown in the following mathematical formulas 1~3.

[Math 2]

Among them, i and j are positive integers. d ( i , j ) represents the distance between the i-th feature map and the j-th feature map, for example, the Euler distance. C _i represents the group to which the i-th feature map belongs, and C _k represents the set of all the feature points in groups different from i. Mathematical formula 2 shows the similarity within clusters, that is, the average distance between the feature points in the cluster; Mathematical formula 3 shows the similarity between clusters, that is the average distance between the feature points of different clusters, and performs the above-mentioned mathematics for each feature map Formula 1~3. Then calculate the average of all s ( i ) as the contour coefficient. The larger the contour coefficient, the better the grouping effect. Therefore, the optimal number of candidate groups corresponding to the largest contour coefficient can be set to the value of k, which is also called It is the number of groups of the unsupervised grouping algorithm, and the number of groups is the same as the number of key pictures 311.

Please refer to Figure 2, where the selected pictures 201 and 202 are the key pictures. It can be seen that the image sequence 200 mainly contains two characters. The selected picture 201 is a scene of one character, and the selected picture 202 is where both characters are present. For the scene, selecting the screens 201 and 202 is sufficient to describe the depth of field relationship in the scene.

Referring back to FIG. 3, after obtaining the key picture 311, the depth map 321 of the key picture can be obtained through the user interface. The depth map generation network 340 will be described in detail below. Fig. 6 is a schematic diagram illustrating a depth map generation network according to an embodiment. Please refer to FIG. 6, the depth map generation network 340 includes a feature converter 610 and a depth generator 620. Worth noting The feature converter 610 is a part of the feature converter 410 of FIG. 4, and the width, height, and number of channels of the first five feature maps of the feature converter 410 and the five feature maps of the feature converter 610 are the same. Therefore, in some embodiments, the network parameters of the feature converter 610 can be initialized according to the network parameters of the feature converter 410, that is, the same network parameters are initially set. After initialization, the depth map generation network 340 can be trained based on the key picture 311 and its depth map 321.

It is worth noting that in Figure 6, "Conv_Block_1" and so on represent the convolutional layer and the pooling layer, but "Conv_Block_3 Spatial-Temporal Similarity Block" means that in addition to the convolutional layer and the pooling layer, a temporal and spatial similarity operation is also used. The spatio-temporal similarity operation is used to compensate the current frame based on the similarity of adjacent frames in time series. Therefore, a total of T frames will be input to the depth map generation network 340, where T is a positive integer, such as 3. These T pictures will go through the operation of the convolutional layer and the pooling layer to obtain T feature maps. In spatio-temporal similarity calculations, when a main feature point of a feature map is to be processed, a preset range of auxiliary feature points in the main feature map and the adjacent feature map will be obtained to compensate the current feature points. This can be considered Before and after timing information, this will make the final depth map more consistent in timing, which can increase accuracy and eliminate screen jitter. Specifically, please refer to FIG. 7, which is a schematic diagram illustrating a space-time similarity operation according to an embodiment. No matter in the training phase or the inference phase, T pictures 701~703 are adjacent pictures in time series, the middle picture 702 is the main picture currently to be trained (or predicted), and the pictures 701 and 703 are adjacent auxiliary pictures . These screens 701~703 will undergo feature extraction The convolutional neural network 710, such as "Conv._Block_1" and "Conv._Block_2" in Figure 6, will then get the corresponding feature maps 711~713, where the feature map 712 is the main feature map to be processed currently. Figures 711 and 713 are called auxiliary feature maps. The spatiotemporal similarity operation 720 will receive feature maps 711 to 713, and for a feature point 714 in the main feature map 712, it will obtain the main feature map 712 and multiple auxiliary feature points within the preset range 715 in the auxiliary feature maps 711 and 713, and The similarity between the feature points 714 and these feature points is calculated to compensate the feature points 714. The present disclosure does not limit the size of the preset range 715. The above-mentioned compensation calculation can be expressed as the following mathematical formulas 4-7.

[Math 4] z _i = h ( y _i )+ x _i

Where x _i represents the feature point 714, z _i is the result of increasing the compensated characteristic of FIG., Y _i is the weighted feature similar to FIG. h ( y _i ) and g ( x _i ) can be regarded as the non-linear transformation of the neural network for dimensionality reduction and dimensionality enhancement of similarly weighted feature maps y _i and x _i , respectively. They can also be general linear transformations of feature maps, such as h ( y _i ) = W _z y _i and g ( x _j ) = W _g x _j , weight matrix W _g and W _z weight matrix, the present invention is not limited to non-linear neural network or linear. f ( x _i , x _j ) is the weighted value of the similarity of the feature map, and C ( x ) is the normalized value after weighting. The above mathematical formula 7 is to calculate the similarity between two feature points, but the mathematical formula 7 is only an example, and other similarities can also be used in other embodiments.

A represents the set formed by all the feature points in the preset range 715 in the feature maps 711, 712, 713, and x _j represents the feature points within the preset range 715. Among them, h ( y _i ) and g ( x _i ) are non-linear or linear dimensionality reduction and dimensionality increase transformations, which can be used to reduce the complexity of calculating the similarity of feature points. In some embodiments, this may be omitted here. For continuous image sequences, due to the similarity of the scene background, in this embodiment, only the similarity of the feature points in the preset range 715 is calculated, which can effectively reduce the redundant feature interference of the scene background. Compared with the conventional technology, except The amount of calculation can be reduced, and the prediction accuracy and timing consistency can also be improved.

After each feature point in the main feature map 712 is compensated, the compensated feature point will be combined with the original uncompensated feature and sent to the subsequent convolutional neural network 730, such as the volume "Conv._Block_5" in Figure 6 Multilayer and pooling layer, here is no limit to the feature combination method, any combination of features such as addition, series, multiplication, etc. can be used. The temporal-spatial similarity operation can be added to any convolutional layer in the feature converter 610. In the embodiment of FIG. 6, there are two convolutional layers that use the temporal-spatial similarity operation, but in other embodiments, more or less convolutional layers are used. Spatio-temporal similar operations are used in the convolutional layer of.

In order to speed up the above method, the feature map of the auxiliary screen can be released after the space-time similarity calculation. This operation is marked as "Discard" in Figure 6 Auxiliary Frames". Therefore, the convolutional layer of "Conv.Block_5" and the subsequent convolutional layers no longer need the feature map of the auxiliary picture. Then the feature map after compensation by the space-time similarity calculation will be converted by the depth generator 620 to generate the main The depth map of the screen.

According to the above-mentioned method, a depth map generation network 340 can be trained and used to generate depth maps of other pictures except the key picture. These results are shown in FIG. 8. In this embodiment, the user only needs to mark the depth map of the key picture. After actual testing, only about 1% to 3% of the picture needs to be marked. Compared with the conventional technology, the labor cost can be greatly reduced, and the quality is better than the known one. Other methods. These generated depth maps can be used to synthesize stereoscopic or multi-view 3D movies, and general depth image based rendering (DIBR) algorithms can be used. This disclosure does not limit how to synthesize stereo or multi-view 3D movies.

One of the contributions of this disclosure is to use convolutional neural networks to search for key images, thereby obtaining the most representative images from the image sequence that can represent all the presented objects, and other images can be regarded as slight distortions based on the key images. In order to improve the robustness of clustering, this technology first trains the feature converter in advance to project the image to a lower spatial dimension and a larger number of channel features, so that key image selection can be achieved more efficiently.

From another perspective, the present invention also proposes a computer program product, which can be written in any programming language and/or platform. When the computer program product is loaded into the computer system and executed, it can execute the above Single-view image depth map sequence generation method.

Although the present invention has been disclosed as above in the embodiments, it is not intended to limit Anyone with ordinary knowledge in the technical field of the present invention can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be defined by the scope of the attached patent application. Prevail.

200: Image sequence

310: Key Picture Extraction System

320,330: steps

311: Key Picture

312: Screen

321,341: Depth Map

340: Depth Map Generation Network

Claims (6)

A method for generating a single-view image depth map sequence includes: obtaining multiple images, and converting each of the images into a feature map according to a feature converter; training an autoencoder according to the images, wherein the autoencoder It includes the feature converter and a feature inverse converter, the feature converter and the feature inverse converter each include a corresponding neural network; an unsupervised grouping algorithm is performed on the feature maps to divide the feature maps For multiple groups, and obtain the images corresponding to the feature maps of the groups as multiple key images; provide a user interface to obtain multiple depth maps of the key images; according to the feature The converter initializes a depth map generation network, and trains the depth map generation network according to the key pictures and the depth maps; and inputs the pictures other than the key pictures to the depth map Generate a network to calculate the corresponding depth map. The single-view image depth map sequence generation method according to claim 1, further comprising: setting a plurality of candidate group numbers, and for each of the candidate group numbers, the unsupervised grouping algorithm is executed according to the feature maps to calculate a Contour factor; and The number of candidate groups corresponding to a largest one of the contour coefficients is set as the number of groups of the unsupervised grouping algorithm, and the number of groups is the same as the number of the key pictures. The method for generating a single-view image depth map sequence according to claim 2, wherein the unsupervised grouping algorithm is a k-means algorithm. The method for generating a single-view image depth map sequence according to claim 1, wherein the depth map generation network includes a spatio-temporal similarity operation, and the spatio-temporal similarity operation is used to receive a main feature map and a sequence adjacent to the main feature map A plurality of auxiliary feature maps in the main feature map, where for a main feature point in the main feature map, the space-time similarity operation obtains a plurality of auxiliary feature points within a preset range of the auxiliary feature maps, and calculates the main feature point and The similarity between each of the auxiliary feature points compensates for the main feature point. A computer program product loaded and executed by a computer system to complete the single-view image depth map sequence generation method described in claim 1. A single-view image depth map sequence generation system, including: a memory for storing multiple commands; A processor for executing the instructions to complete multiple steps: obtaining multiple pictures, and converting each of the pictures into a feature map according to a feature converter; training an autoencoder according to the pictures, wherein The autoencoder includes the feature converter and a feature inverse converter, the feature converter and the feature inverse converter each include a corresponding neural network; an unsupervised clustering algorithm is performed on the feature maps to The feature maps are divided into multiple groups, and the frames corresponding to the feature maps of the groups are obtained as multiple key frames; a user interface is provided to obtain multiple depth maps of the key frames ; According to the feature converter to initialize a depth map generation network, and train the depth map generation network according to the key pictures and the depth maps; and input other pictures of the pictures except the key pictures Go to the depth map generation network to calculate the corresponding depth map.

Images