CN107301662B - Depth image compression recovery method, device, device and storage medium - Google Patents

Abstract

The present invention is applicable to the field of computer technology, and provides a depth image compression and restoration method, device, mobile terminal and storage medium. The method includes: receiving an input restoration request of a depth image to be restored, and the depth image to be restored is associated with a corresponding texture image, preprocess the texture image and the depth image to be restored, obtain the high-frequency information of the texture image and the depth image to be restored, and input the high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored respectively. To the Y branch model, D branch model and M branch model of the preset depth image restoration model, input the feature images recovered by the Y branch model and D branch model into the M branch model, and restore the depth image to be restored through the M branch model Corresponding depth images, thereby improving the compression and restoration quality of the depth images, thereby improving the user experience.

Description

Depth image compression recovery method, device, device and storage medium

technical field

The invention belongs to the technical field of computers, and in particular relates to a method, device, device and storage medium for compression and restoration of depth images.

Background technique

In 3D computer graphics, depth images contain information about the distance between the viewpoint and the surface of objects in the scene. The traditional machine vision is to project a three-dimensional object into a two-dimensional image, and restore the three-dimensional scene through the relationship between the characteristics of the object, the image data and the imaging process. Depth information plays a key role in 3D reconstruction. Sending stereoscopic video (left and right views) can provide a 3D experience, but has significant limitations. To reduce the number of transmitted views, the texture-plus-depth format has become widely accepted. This method transmits the color information and depth information of several viewpoints together, and then uses the depth-image-based rendering (DIBR) technique to synthesize the virtual view. Reconstructing a 3D scene requires obtaining texture information and depth information, and high-resolution depth images take up a lot of space during storage and transmission. Therefore, depth images need to be compressed to improve space utilization and transmission efficiency. However, the compressed depth image will have distortions such as blurring and blockiness, which will further lead to the distortion of 3D scene rendering.

Lossy compression can compress images with larger compression ratios, but lossy compressed images are irreversible. Therefore, image compression and recovery is also a research direction that people have been studying. In the early years, many researchers designed various smoothing filters to remove blockiness from the spatial or transform domain. Luo et al. proposed to adaptively remove block effects in the spatial domain and discrete cosine transform (DCT); the model proposed by Singh et al. can achieve smooth and non-smooth regions with different filters. Shape Adaptive Discrete Cosine Transform (SA-DCT for short) is probably the most popular method to get rid of blockiness. Computationally, it can transform the shape and size of the filter to reconstruct the sharp edges of the image. However, these filters may over-smooth the image, resulting in blurred edges.

In recent years, Convolutional Neural Network (CNN) learns feature detection through training data, and its special structure of local weight sharing has unique advantages in speech recognition and image processing. CNN also shows excellent performance in the image restoration neighborhood. The Super-Resolution Convolutional Neural Network (SRCNN) model proposed by Dong et al. illustrates the potential of an end-to-end Data Communication Network (DCN) to solve image super-resolution. Deeper SRCNN makes image restoration better by increasing the number of layers. However, their proposed algorithm mainly performs super-resolution analysis on images, and the effect of deblocking is poor. On the basis of SRCNN, Artifacts Reduction Convolutional Neural Network (AR-CNN) is used to restore compressed images such as JPEG, JPEG2000, and Twitter. Although this is a more general model, their model is used to recover texture images, which are significantly different from depth images.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method, device, equipment and storage medium for compression and restoration of depth images, aiming to solve the problem that the quality of compression and restoration of depth images is not good due to the inability to provide an effective method for compression and restoration of depth images in the prior art. Good, bad user experience problem.

In one aspect, the present invention provides a method for compressing and restoring a depth image, the method comprising the following steps:

receiving an input restoration request for a depth image to be restored, where the depth image to be restored is associated with a corresponding texture image;

Preprocessing the texture image and the depth image to be restored to obtain high-frequency information of the texture image and the depth image to be restored;

Input the high frequency information of the texture image, the high frequency information of the depth image to be restored and the depth image to be restored into the Y branch model, the D branch model and the M branch model of the preset depth image restoration model respectively;

The feature images recovered by the Y-branch model and the D-branch model are input into the M-branch model, and a depth image corresponding to the depth image to be restored is recovered through the M-branch model.

In another aspect, the present invention provides an apparatus for compressing and restoring a depth image, the apparatus comprising:

a request receiving unit, configured to receive an input restoration request of a depth image to be restored, where the depth image to be restored is associated with a corresponding texture image;

an image preprocessing unit, configured to preprocess the texture image and the depth image to be restored, and obtain high-frequency information of the texture image and the depth image to be restored;

The corresponding input unit is used to input the high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored to the Y branch model and D branch of the preset depth image restoration model respectively. models and M-branch models; and

The image restoration unit is configured to input the feature images recovered by the Y branch model and the D branch model into the M branch model, and restore the depth image corresponding to the depth image to be restored through the M branch model.

In another aspect, the present invention also provides a depth image compression and restoration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the The computer program implements the steps of the method for compressing and restoring the depth image.

On the other hand, the present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements the steps of the method for compressing and restoring the depth image .

The present invention receives an input restoration request of the depth image to be restored, the depth image to be restored is associated with a corresponding texture image, preprocesses the texture image and the depth image to be restored, obtains the high frequency information of the texture image and the depth image to be restored, The high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored are respectively input into the Y branch model, the D branch model and the M branch model of the preset depth image restoration model. The feature image obtained by the model restoration is input into the M-branch model, and the depth image corresponding to the depth image to be restored is restored through the M-branch model, thereby improving the compression and restoration quality of the depth image, thereby improving the user experience.

Description of drawings

Fig. 1 is the realization flow chart of the method for compressing and restoring a depth image provided by Embodiment 1 of the present invention;

2 is a schematic structural diagram of an apparatus for compressing and restoring a depth image according to Embodiment 2 of the present invention;

3 is a schematic diagram of a preferred structure of an apparatus for compressing and restoring a depth image according to Embodiment 2 of the present invention; and

FIG. 4 is a schematic structural diagram of a device for compressing and restoring a depth image according to Embodiment 3 of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

The specific implementation of the present invention is described in detail below in conjunction with specific embodiments:

Example 1:

FIG. 1 shows the implementation process of the method for compressing and restoring a depth image provided by the first embodiment of the present invention. For the convenience of description, only the part related to the embodiment of the present invention is shown, and the details are as follows:

In step S101, an input restoration request of a depth image to be restored is received, and the depth image to be restored is associated with a corresponding texture image.

The embodiment of the present invention is applicable to a restoration system of a compressed depth image, so as to facilitate restoration of the compressed depth image. In this embodiment of the present invention, a restoration request for a depth image to be restored inputted by a user is first received, wherein the depth image to be restored is associated with a corresponding texture image to assist in restoring the depth image to be restored.

然后构建该深度图像恢复模型的D分支模型，构建的D分支模型包含2个卷积层：

最后构建该深度图像恢复模型的M分支模型，构建的M分支模型包含5个卷积层：

Preferably, before receiving an input restoration request for a depth image to be restored, a depth image restoration model is first constructed. Specifically, when building a depth image restoration model, firstly build a Y branch model of the depth image restoration model, and the constructed Y branch model contains 2 convolution layers:

Then a D-branch model of the deep image restoration model is constructed, and the constructed D-branch model contains 2 convolutional layers:

Finally, the M-branch model of the deep image restoration model is constructed, and the constructed M-branch model contains 5 convolutional layers:

“*”表示卷积操作，

为滤波器，

为偏置向量。in,

"*" indicates convolution operation,

is the filter,

is the bias vector.

Further preferably, after the step of constructing the depth image restoration model, before the step of receiving the input restoration request of the depth image to be restored, the input training set is first received, and the training set includes uncompressed texture images, uncompressed depth images and The corresponding compressed depth image, and then preprocess the uncompressed texture image, uncompressed depth image and corresponding compressed depth image in the training set to obtain the high-frequency information of the uncompressed texture image and the compressed depth image , and then input the acquired high-frequency information of the uncompressed texture image, the high-frequency information of the compressed depth image and the compressed depth image into the Y branch model, D branch model and M branch model of the pre-built depth image restoration model respectively , train the Y-branch model, D-branch model and M-branch model of the depth image restoration model, calculate the loss function for training according to the uncompressed depth images in the training set and the restored images obtained from training, and update the filters of the depth image restoration model and Bias vector, when the preset number of iterations is not reached, repeat the training and update steps until the preset number of iterations is reached, set the trained depth image restoration model to the preset depth image restoration model, thereby improving Recovery accuracy of compressed depth images.

计算训练的损失函数，其中，N为训练集中训练目标的数量，

恢复得到的深度图像，

为

对应的纹理图像，i表示每一次训练，θ为待优化参数，包括滤波器和偏置向量参数，从而提高训练的准确性和速率。Preferably, when calculating the loss function for training, use the formula

Calculate the training loss function, where N is the number of training targets in the training set,

restore the resulting depth image,

for

For the corresponding texture image, i represents each training, and θ is the parameter to be optimized, including filter and bias vector parameters, so as to improve the accuracy and speed of training.

In step S102, the texture image and the depth image to be restored are preprocessed to obtain high frequency information of the texture image and the depth image to be restored.

In the embodiment of the present invention, after the restoration request of the depth image to be restored, the corresponding texture image and the depth image to be restored are first obtained, and then the texture image and the depth image to be restored are preprocessed respectively, and the texture image and the depth image to be restored are extracted. The high-frequency information of the depth image to be restored is obtained, thereby obtaining the high-frequency information of the texture image and the depth image to be restored.

对纹理图像进行预处理，获取纹理图像的高频信息，根据公式

对待恢复深度图像进行预处理，获取待恢复深度图像的高频信息，其中，参数Y为纹理图像，参数D为待恢复深度图像，h(Y)表示对纹理图像进行均值池化处理，h(D)表示对待恢复深度图像进行均值池化处理，abs()为取绝对值函数，即首先对需要预处理的纹理图像或者待恢复深度图像做均值池化(9x9)，然后用均值池化前的像素值减去均值池化后的像素值，得到均值池化前后的像素值差值，再对该差值取绝对值，这样就得到了对应图像的边缘图像(即图像的高频信息)。均值池化可以使图像变得模糊，即除去了图像的高频信息部分，剩下图像的低频信息，然后用原图像减去只有低频信息的图像，高频信息就可以被保留下来。Preferably, when preprocessing the texture image and the depth image to be restored, according to the formula

Preprocess the texture image to obtain the high-frequency information of the texture image, according to the formula

Preprocess the depth image to be restored to obtain high-frequency information of the depth image to be restored, where the parameter Y is the texture image, the parameter D is the depth image to be restored, h(Y) represents mean pooling of the texture image, h( D) means that the depth image to be restored is subjected to mean pooling, and abs() is the absolute value function, that is, the texture image that needs to be preprocessed or the depth image to be restored is mean pooled (9x9), and then the mean pooling is used before Subtract the pixel value after mean pooling from the pixel value of the mean pooling, get the pixel value difference before and after mean pooling, and then take the absolute value of the difference, so as to get the edge image of the corresponding image (that is, the high-frequency information of the image) . Mean pooling can blur the image, that is, remove the high-frequency information part of the image, leaving the low-frequency information of the image, and then subtract the image with only low-frequency information from the original image, and the high-frequency information can be retained.

In step S103, the high frequency information of the texture image, the high frequency information of the depth image to be restored and the depth image to be restored are respectively input into the Y branch model, the D branch model and the M branch model of the preset depth image restoration model.

In the embodiment of the present invention, after obtaining the high-frequency information of the texture image and the depth image to be restored, the high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored are respectively input into a preset Y-branch model, D-branch model and M-branch model of depth image restoration model.

In step S104, the feature images recovered by the Y branch model and the D branch model are input into the M branch model, and the depth image corresponding to the depth image to be restored is recovered through the M branch model.

和

将Y分支模型、D分支模型恢复得到的特征图像输入到M分支模型，M分支模型根据公式

恢复出待恢复深度图像对应的深度图像。In this embodiment of the present invention, according to the formula

and

The feature images recovered by the Y branch model and the D branch model are input into the M branch model, and the M branch model is based on the formula

The depth image corresponding to the depth image to be restored is restored.

In the embodiment of the present invention, an input restoration request of the depth image to be restored is received, the depth image to be restored is associated with a corresponding texture image, the texture image and the depth image to be restored are preprocessed, and the difference between the texture image and the depth image to be restored is obtained. High-frequency information, the high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored are respectively input into the Y branch model, the D branch model and the M branch model of the preset depth image restoration model. The feature images recovered by the branch model and the D-branch model are input to the M-branch model, and the depth image corresponding to the depth image to be restored is restored through the M-branch model, thereby improving the compression and restoration quality of the depth image, thereby improving the user experience.

Embodiment 2:

FIG. 2 shows the structure of an apparatus for compressing and restoring a depth image provided by the second embodiment of the present invention. For the convenience of description, only the parts related to the embodiment of the present invention are shown, including:

The request receiving unit 21 is configured to receive an input restoration request of a depth image to be restored, where the depth image to be restored is associated with a corresponding texture image.

In the embodiment of the present invention, the request receiving unit 21 first receives a restoration request of the depth image to be restored input by the user, wherein the depth image to be restored is associated with a corresponding texture image to assist in restoring the depth image to be restored.

然后构建该深度图像恢复模型的D分支模型，构建的D分支模型包含2个卷积层：

最后构建该深度图像恢复模型的M分支模型，构建的M分支模型包含5个卷积层：

Preferably, before receiving an input restoration request for a depth image to be restored, a depth image restoration model is first constructed. Specifically, when building a depth image restoration model, firstly build a Y branch model of the depth image restoration model, and the constructed Y branch model contains 2 convolution layers:

Then a D-branch model of the deep image restoration model is constructed, and the constructed D-branch model contains 2 convolutional layers:

Finally, the M-branch model of the deep image restoration model is constructed, and the constructed M-branch model contains 5 convolutional layers:

“*”表示卷积操作，

为滤波器，

为偏置向量。in,

"*" indicates convolution operation,

is the filter,

is the bias vector.

Further preferably, after the step of constructing the depth image restoration model, before the step of receiving the input restoration request of the depth image to be restored, the input training set is first received, and the training set includes uncompressed texture images, uncompressed depth images and The corresponding compressed depth image, and then preprocess the uncompressed texture image, uncompressed depth image and corresponding compressed depth image in the training set to obtain the high-frequency information of the uncompressed texture image and the compressed depth image , and then input the acquired high-frequency information of the uncompressed texture image, the high-frequency information of the compressed depth image and the compressed depth image into the Y branch model, D branch model and M branch model of the pre-built depth image restoration model respectively , train the Y-branch model, D-branch model and M-branch model of the depth image restoration model, calculate the loss function for training according to the uncompressed depth images in the training set and the restored images obtained from training, and update the filters of the depth image restoration model and Bias vector, when the preset number of iterations is not reached, repeat the training and update steps until the preset number of iterations is reached, set the trained depth image restoration model to the preset depth image restoration model, thereby improving Recovery accuracy of compressed depth images.

计算训练的损失函数，其中，N为训练集中训练目标的数量，

恢复得到的深度图像，

为

对应的纹理图像，i表示每一次训练，θ为待优化参数，包括滤波器和偏置向量参数，从而提高训练的准确性和速率。Preferably, when calculating the loss function for training, use the formula

Calculate the training loss function, where N is the number of training targets in the training set,

restore the resulting depth image,

for

For the corresponding texture image, i represents each training, and θ is the parameter to be optimized, including filter and bias vector parameters, so as to improve the accuracy and speed of training.

The image preprocessing unit 22 is configured to preprocess the texture image and the depth image to be restored, and obtain high-frequency information of the texture image and the depth image to be restored.

In the embodiment of the present invention, after the restoration request of the depth image to be restored, the image preprocessing unit 22 first obtains the corresponding texture image and the depth image to be restored, and then preprocesses the texture image and the depth image to be restored respectively, and extracts the The high frequency information of the texture image and the depth image to be restored is obtained, thereby obtaining the high frequency information of the texture image and the depth image to be restored.

对纹理图像进行预处理，获取纹理图像的高频信息，根据公式

对待恢复深度图像进行预处理，获取待恢复深度图像的高频信息，其中，参数Y为纹理图像，参数D为待恢复深度图像，h(Y)表示对纹理图像进行均值池化处理，h(D)表示对待恢复深度图像进行均值池化处理，abs()为取绝对值函数，即首先对需要预处理的纹理图像或者待恢复深度图像做均值池化(9x9)，然后用均值池化前的像素值减去均值池化后的像素值，得到均值池化前后的像素值差值，再对该差值取绝对值，这样就得到了对应图像的边缘图像。均值池化可以使图像变得模糊，即除去了图像的高频信息部分，剩下图像的低频信息，然后用原图像减去只有低频信息的图像，高频信息就可以被保留下来。Preferably, when preprocessing the texture image and the depth image to be restored, according to the formula

Preprocess the texture image to obtain the high-frequency information of the texture image, according to the formula

The depth image to be restored is preprocessed to obtain high-frequency information of the depth image to be restored, where the parameter Y is the texture image, the parameter D is the depth image to be restored, h(Y) represents mean pooling of the texture image, h( D) means that the depth image to be restored is subjected to mean pooling, and abs() is the absolute value function, that is, first mean pooling (9x9) is performed on the texture image that needs to be preprocessed or the depth image to be restored (9x9), and then use the mean pooling before The pixel value of , subtracts the pixel value after mean pooling to obtain the pixel value difference before and after mean pooling, and then takes the absolute value of the difference to obtain the edge image of the corresponding image. Mean pooling can blur the image, that is, remove the high-frequency information part of the image, leaving the low-frequency information of the image, and then subtract the image with only low-frequency information from the original image, and the high-frequency information can be preserved.

The corresponding input unit 23 is used to input the high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored into the Y branch model, the D branch model and the M branch model of the preset depth image restoration model respectively. .

In the embodiment of the present invention, after obtaining the high frequency information of the texture image and the depth image to be restored, the corresponding input unit 23 inputs the high frequency information of the texture image, the high frequency information of the depth image to be restored, and the depth image to be restored respectively as input to the Y branch model, D branch model and M branch model of the preset depth image restoration model.

The image restoration unit 24 is configured to input the feature images restored by the Y branch model and the D branch model into the M branch model, and restore the depth image corresponding to the depth image to be restored through the M branch model.

和

将Y分支模型、D分支模型恢复得到的特征图像输入到M分支模型，M分支模型根据公式

恢复出待恢复深度图像对应的深度图像。In the embodiment of the present invention, the image restoration unit 24 according to the formula

and

The feature images recovered by the Y branch model and the D branch model are input into the M branch model, and the M branch model is based on the formula

The depth image corresponding to the depth image to be restored is restored.

Therefore, preferably, as shown in Figure 3, the device further comprises:

A model construction unit 30, for constructing a depth image restoration model;

Preferably, the model building unit 30 includes:

The first construction unit 301 is used to construct the Y branch model of the depth image restoration model, and the convolution layer of the Y branch model is

The second construction unit 302 is used to construct a D-branch model of the depth image restoration model, and the convolutional layer of the D-branch model is

The third construction unit 303 is configured to construct an M-branch model of the depth image restoration model, and the convolutional layer of the M-branch model is F ₁ ^M =max(0, W ₁ ^M *D ^q +B ₁ ^M ),

“*”表示卷积操作，

为滤波器，

为偏置向量；in,

"*" indicates convolution operation,

is the filter,

is the bias vector;

Preferably, the image preprocessing unit 22 includes:

对纹理图像进行预处理，获取纹理图像的高频信息，其中，参数Y为纹理图像，h(Y)表示对纹理图像进行均值池化处理，abs()为取绝对值函数；以及The first processing unit 321 is used for formulating

Preprocessing the texture image to obtain high-frequency information of the texture image, wherein the parameter Y is the texture image, h(Y) represents mean pooling processing of the texture image, and abs() is the absolute value function; and

对待恢复深度图像进行预处理，获取待恢复深度图像的高频信息，其中，参数D为待恢复深度图像，h(D)表示对待恢复深度图像进行均值池化处理。The second processing unit 322 is used for formulating according to the formula

The depth image to be restored is preprocessed to obtain high-frequency information of the depth image to be restored, wherein the parameter D is the depth image to be restored, and h(D) represents mean pooling of the depth image to be restored.

In the embodiment of the present invention, the request receiving unit receives an input restoration request of the depth image to be restored, the depth image to be restored is associated with a corresponding texture image, and the image preprocessing unit preprocesses the texture image and the depth image to be restored, and obtains the texture The high-frequency information of the image and the depth image to be restored, the corresponding input unit inputs the high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored into the Y branch model of the preset depth image restoration model, The D-branch model and the M-branch model, the image restoration unit inputs the feature images recovered by the Y-branch model and the D-branch model into the M-branch model, and restores the depth image corresponding to the depth image to be restored through the M-branch model, thereby improving the depth image. Compression recovery quality, which in turn improves the user experience.

In this embodiment of the present invention, each unit of the apparatus for compressing and restoring a depth image may be implemented by a corresponding hardware or software unit, and each unit may be an independent software or hardware unit, or may be integrated into a software and hardware unit. Limit the invention.

Embodiment three:

FIG. 4 shows the structure of the device for compressing and restoring a depth image provided by Embodiment 4 of the present invention. For convenience of description, only parts related to the embodiment of the present invention are shown.

The apparatus 4 for compressing and restoring a depth image according to the embodiment of the present invention includes a processor 40 , a memory 41 , and a computer program 42 stored in the memory 41 and running on the processor 40 . When the processor 40 executes the computer program 42 , the steps in the above-mentioned embodiment of the method for compressing and restoring a depth image are implemented, for example, steps S101 to S104 shown in FIG. 1 . Alternatively, when the processor 40 executes the computer program 42, the functions of the units in the above-mentioned apparatus embodiments, such as the functions of the units 21 to 24 shown in FIG. 2, are implemented.

In this embodiment of the present invention, when the processor 40 executes the computer program 42 to implement the steps in each of the above-mentioned control method embodiments for screen wake-up, it receives an input restoration request for the depth image to be restored, and the depth image to be restored is associated with a corresponding depth image. Texture image: Preprocess the texture image and the depth image to be restored, obtain high-frequency information of the texture image and the depth image to be restored, and separate the high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored. Input the Y-branch model, D-branch model and M-branch model of the preset depth image restoration model, input the feature images recovered by the Y-branch model and D-branch model into the M-branch model, and restore the depth to be restored through the M-branch model The depth image corresponding to the image, thereby improving the compression and restoration quality of the depth image.

For details of the steps implemented by the processor 40 in the depth image compression and restoration device 4 when the computer program 42 is executed, reference may be made to the description of the method in the first embodiment, which will not be repeated here.

Embodiment 4:

In an embodiment of the present invention, a computer-readable storage medium is provided, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the above-mentioned embodiments of the method for compressing and restoring a depth image are implemented, For example, steps S101 to S104 shown in FIG. 1 . Alternatively, when the computer program is executed by the processor, the functions of each unit in the above-mentioned apparatus embodiment, for example, the functions of units 21 to 24 shown in FIG. 2 , are implemented.

In the embodiment of the present invention, an input restoration request of the depth image to be restored is received, the depth image to be restored is associated with a corresponding texture image, the texture image and the depth image to be restored are preprocessed, and the difference between the texture image and the depth image to be restored is obtained. High-frequency information, the high-frequency information of the texture image, the high-frequency information of the depth image to be restored, and the depth image to be restored are respectively input into the Y branch model, the D branch model and the M branch model of the preset depth image restoration model. The feature images recovered by the branch model and the D-branch model are input to the M-branch model, and the depth image corresponding to the depth image to be restored is restored through the M-branch model, thereby improving the compression and restoration quality of the depth image. The method for compressing and restoring a depth image when the computer program is executed by the processor may further refer to the description of the steps in the foregoing method embodiments, which will not be repeated here.

The computer-readable storage medium of the embodiments of the present invention may include any entity or device capable of carrying computer program codes, recording medium, for example, memory such as ROM/RAM, magnetic disk, optical disk, flash memory, and the like.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (8)

1. A method for compression restoration of a depth image, the method comprising the steps of:

receiving an input restoration request of a depth image to be restored, wherein the depth image to be restored is associated with a corresponding texture image;

preprocessing the texture image and the depth image to be restored to obtain high-frequency information of the texture image and the depth image to be restored;

respectively inputting the high-frequency information of the texture image, the high-frequency information of the depth image to be restored and the depth image to be restored into a Y-branch model, a D-branch model and an M-branch model of a preset depth image restoration model;

inputting the characteristic images obtained by recovering the Y branch model and the D branch model into the M branch model, and recovering the depth image corresponding to the depth image to be recovered through the M branch model;

constructing the depth image recovery model;

the step of constructing the depth image restoration model includes:

the Y-branch model of the depth image restoration model is constructed, and the convolution layer of the Y-branch model is

Constructing the D-branch model of the depth image restoration model, the convolution layer of the D-branch model being

Constructing the M-branch model of the depth image restoration model, the convolution layer of the M-branch model being

The above-mentioned

The above-mentioned

Said ". mark" denotes a convolution operation, said W_j ^Y、W_j ^D、

Being a filter, said parameter D^qFor the depth image to be restored, the

Is a bias vector.

2. The method of claim 1, wherein after the step of constructing the depth image restoration model and before the step of receiving an input restoration request for the depth image to be restored, the method further comprises:

receiving an input training set, wherein the training set comprises an uncompressed texture image, an uncompressed depth image and a compressed depth image;

preprocessing the uncompressed texture image, the uncompressed depth image and the compressed depth image in the training set to obtain high-frequency information of the uncompressed texture image and the compressed depth image;

inputting the high-frequency information of the uncompressed texture image, the high-frequency information of the compressed depth image and the compressed depth image into the Y-branch model, the D-branch model and the M-branch model of the depth image recovery model which are constructed in advance respectively, training the Y-branch model, the D-branch model and the M-branch model of the depth image recovery model, calculating a loss function of the training, and updating the filter and the offset vector of the depth image recovery model;

and when the preset iteration times are not reached, repeating the training and updating steps until the iteration times are reached, and setting the trained depth image recovery model as the preset depth image recovery model.

3. The method of claim 2, wherein the step of calculating the trained loss function comprises:

using the formula

Calculating a loss function of the training, wherein N is the number of training targets in the training set, and

to restore the resulting depth image, the

Is composed of

And corresponding texture images, wherein i represents each training, and theta is a parameter to be optimized and comprises the filter and the bias vector.

4. The method of claim 1, wherein the step of preprocessing the texture image and the depth image to be restored to obtain high-frequency information of the texture image and the depth image to be restored comprises:

according to the formula

Preprocessing the texture image to obtain high-frequency information of the texture image, wherein the parameter Y is the texture image, the h (Y) represents that the texture image is subjected to mean pooling, and abs () is an absolute value taking function;

according to the formula

Preprocessing the depth image to be recovered to obtain high-frequency information of the depth image to be recovered, wherein h (D)^q) And representing that the depth image to be restored is subjected to mean pooling.

5. An apparatus for compression restoration of a depth image, the apparatus comprising:

the device comprises a request receiving unit, a processing unit and a processing unit, wherein the request receiving unit is used for receiving an input recovery request of a depth image to be recovered, and the depth image to be recovered is associated with a corresponding texture image;

the image preprocessing unit is used for preprocessing the texture image and the depth image to be restored to obtain high-frequency information of the texture image and the depth image to be restored;

the corresponding input unit is used for respectively inputting the high-frequency information of the texture image, the high-frequency information of the depth image to be restored and the depth image to be restored into a Y-branch model, a D-branch model and an M-branch model of a preset depth image restoration model; and

the image recovery unit is used for inputting the characteristic images obtained by recovering the Y branch model and the D branch model into the M branch model and recovering the depth image corresponding to the depth image to be recovered through the M branch model;

the device further comprises:

a model construction unit for constructing the depth image restoration model;

the model building unit includes:

a first constructing unit for constructing the Y-branch model of the depth image restoration model, wherein the convolution layer of the Y-branch model is

A second construction unit for constructing the D-branch model of the depth image restoration model, the convolution layer of the D-branch model being

And

a third constructing unit, configured to construct the M-branch model of the depth image restoration model, where a convolution layer of the M-branch model is

The above-mentioned

The above-mentioned

Said ". mark" denotes a convolution operation, said W_j ^Y、W_j ^D、

Being a filter, said parameter D^qFor the depth image to be restored, the

Is a bias vector.

6. The apparatus of claim 5, wherein the image pre-processing unit comprises:

a first processing unit for processing the data according to a formula

Preprocessing the texture image to obtain high-frequency information of the texture image, wherein the parameter Y is the texture image, the h (Y) represents that the texture image is subjected to mean pooling, and abs () is an absolute value taking function; and

a second processing unit for processing the data according to the formula

Preprocessing the depth image to be recovered to obtain high-frequency information of the depth image to be recovered, wherein h (D)^q) And representing that the depth image to be restored is subjected to mean pooling.

7. An apparatus for compression restoration of a depth image, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 4 when executing the computer program.

8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 4.

Images