CN117036539A - Virtual viewpoint drawing cavity filling method and device based on deep learning - Google Patents

Abstract

The invention discloses a virtual viewpoint drawing cavity filling method and device based on deep learning, which are characterized in that a graph to be repaired is obtained based on an original graph and a mask graph thereof, the graph to be repaired and the mask graph are input into a progressive iterative network, the progressive iterative network carries out local cavity recognition on the graph to be repaired through partial convolution, cavity filling is carried out based on a knowledge consistency attention mechanism, and a loss function is constructed to promote semantic consistency of a background cavity area and a known area; constructing a context characteristic propagation loss, merging the loss function constructed in the step S1, outputting a progressive iterative network, performing similarity coding to obtain the similarity of the image blocks and the image blocks in the non-cavity area, and generating a filling block with semantic consistency by the background cavity based on the similarity; and carrying out weighted combination on the output of the progressive iterative network to obtain a final repair graph.

Description

A method and device for filling holes in virtual viewpoint rendering based on deep learning

Technical field

The invention belongs to the technical fields of deep learning and virtual viewpoint rendering, and specifically relates to a method and device for filling holes in virtual viewpoint rendering based on deep learning.

Background technique

Emerging three-dimensional (3D) multimedia visual services such as free-viewpoint video, stereoscopic TV, and virtual reality can bring immersive and interactive visual experiences to users, and are increasingly attracting people's attention and love. However, characterizing interactive 3D videos requires a large amount of viewpoint information. Due to collection cost and bandwidth limitations, only a limited number of viewpoint scenes can be collected and transmitted in practical applications. Currently, the multi-view plus depth (MVD) encoding format is the mainstream format for compressing 3D videos and free-viewpoint videos. It uses depthimage based rendering (DIBR) technology on the decoding end to convert MVD videos from The required virtual viewpoints are drawn to make up for the lack of number of viewpoints. However, when drawing virtual viewpoints, problems such as front and rear occlusion between different viewpoints will cause holes, cracks and other missing areas in the drawn image. Therefore, the missing areas of the drawn image need to be repaired.

Traditional virtual viewpoint rendering hole filling technologies are mainly based on spatial domain consistency and temporal consistency. Technologies based on spatial domain consistency mainly include the use of filters and patch-based methods; technologies based on temporal consistency propose to use background modules to construct the hole region of the background. For example, the patent document No. CN201310017391.3 proposes a new virtual viewpoint synthesis method based on depth map rendering technology. The virtual viewpoint depth image is obtained through 3D image transformation, and the depth image is optimized; then a reverse 3D image transformation is performed based on the depth map obtained by the optimization processing to obtain a virtual viewpoint color image; finally, the hole is filled through an image repair algorithm based on depth information. . Through reverse 3D image transformation, cracks on the virtual viewpoint color image can be avoided and the quality of the virtual viewpoint image can be improved; in addition, the non-occlusion hole area in the virtual viewpoint image is filled with image repair methods to ensure that the rendered image Produce the best display effect. The above-mentioned patent still uses a single-view patch method for filling based on spatial consistency, which easily gives the foreground and background the same weight, resulting in serious artifacts.

In the existing technology, filters based on spatial consistency technology can only repair cracks and small baseline holes that appear during the drawing process. Patch-based methods will search and match similar patch blocks. Due to the accuracy of the search algorithm, The degree is not high, and it is easy to give the same weight to the foreground and background; in the method based on temporal consistency, the reconstructed background image is used to fill the virtual view, and there are many steps, and the scene contains moving objects, which easily causes the foreground to be modeled as the background , leading to aliasing of foreground and background pixels and artifacts.

Contents of the invention

In order to solve the deficiencies of the existing technology, the present invention adopts end-to-end deep learning technology to achieve the purpose of filling holes in virtual viewpoint rendering. The present invention adopts the following technical solutions:

A method for filling holes in virtual viewpoint rendering based on deep learning, including the following steps:

Step S1: Based on the original image and its mask image, obtain the image to be repaired, and input the image to be repaired and the mask image into the progressive iterative network. The progressive iterative network uses partial convolution to identify local holes in the image to be repaired, and based on knowledge The consistent attention mechanism performs hole filling and constructs a loss function to improve the semantic consistency between background hole areas and known areas;

Step S2: Construct the context feature propagation loss and integrate it into the loss function constructed in step S1 to improve the robustness of feature matching. Perform similarity coding on the output of the progressive iterative network to obtain the image blocks and non-hole area image blocks. Similarity, based on similarity, enables background holes to generate filling blocks with semantic consistency;

Step S3: Perform weighted merging of the outputs of the progressive iterative network to obtain the final repair map. When the number of progressive iterations reaches the set threshold, the filling of the hole area is completed. However, if the feature map output at this time is used directly, there will be problems of gradient disappearance and intermediate generated feature loss. If average merging and adaptive merging are used, then The missing areas in the reconstructed images output early will affect the quality of the final output image. In order to solve the above problem, the feature maps generated by each progressive iteration are fused through weighted merging.

Furthermore, in step S1, some convolutions only use valid pixels in the hole area for operation, and the updated mask is retained throughout the iteration process until it is reduced and updated in the next iteration, which is conducive to effective shallow layers. Feature extraction.

Further, the knowledge-consistent attention mechanism in step S1 first obtains the attention score of each patch by normalizing the inner product between the known feature patch vector and the generated feature patch vector; then, based on the previous Whether the iteratively generated mask pixel values are valid, the final attention score is weighted; finally, the attention score is used to update the features of the missing patch, and the convolutional layer is used to improve the structural consistency of the input features and reconstructed features. The attention mechanism can extract feature maps with reasonable semantics and more accurate pixel information, while the advantage of the knowledge-consistent attention module is that the attention score is measured by the weighted sum of the current attention score and the score of the previous iteration, so that the preceding and following frame patches There is correlation between them, thereby solving the problem of the same foreground and background weights based on patch filling in traditional methods.

Further, in step S1, the loss function combines L ₁ loss, perceptual loss, style loss and smoothing loss; the L ₁ loss includes the loss of the hole area L _valid and the loss of the hole area L _hole :

L ₁ =λ ₁ L _valid +λ ₂ L _hole

Among them, λ ₁ and λ ₂ represent the weights of L _valid and L _hole respectively, M is a binary image, divided into 0 to represent the hole area, 1 to represent the effective area, ⊙ represents the dot product, and I _gt represents the original image. Represents the prediction map output by the t-th progressive iteration network, C, H, and W are the number of channels, the height and width of the feature patch respectively;

The perceptual loss is used to enhance the similarity of high-level feature structures between filled images and real images, defined as:

Among them, I _o is the output result graph, represented by It consists of hole-generated pixels in and non-hole pixels in I _gt ; ψ _m (I _gt ) and ψ _m (I _o ) respectively represent the m-th pooling of I _gt and I _o through the pre-trained feature extraction network VGG-16 Features output after the layer; H _m , W _m and C _m represent the height, width and number of channels of the extracted m-th feature map; N ₂ is the number of pooling layers;

The style loss is used to compensate for the problem that perceptual loss cannot effectively maintain the style consistency between the filled area and the surrounding area, and is defined as:

in, The similarity of the features of each layer in the network is obtained by calculating the Gram matrix Gram of the feature map, that is/>

The smoothing loss is used to maintain the smoothness of the filled image and is defined as:

Among them, P _i,j represents a pixel in I _o , P _i,j+1 and P _i+1,j respectively represent the adjacent pixels of P _i,j in the vertical and horizontal directions, and R is the pixel in the hole area. The expansion area with a pixel of 1, N _c is the total number of image patches of I _o .

Further, in step S2, the similarity between the image block and the non-hole area image block is calculated through similarity coding convolution, where the convolution kernel acquisition operation is to process the background feature map by extracting its own patch, and the size is set to A 4×4 convolution kernel with a stride of 2 to avoid the checkerboard effect.

Further, in the step S2, the image to be repaired is auxiliary encoded, and the encoding result is combined with the similarity for auxiliary decoding to obtain an auxiliary image. Based on the auxiliary image guidance, the background hole is generated into a filling block with semantic consistency.

Further, in step S2, the context feature propagation loss is defined as the hole filling loss of the auxiliary image, which can be regarded as the sum of the local losses of the auxiliary image block, that is,

Among them, i represents the image block index with the highest similarity S in the iterative output I _out , u _i represents the auxiliary image block with the highest similarity S, and l _i (·) is the local hole loss of the i-th auxiliary image block.

Further, in step S3, adaptive merging generates weights by analyzing masks without a learning process, including the following steps:

First divide the feature map:

Among them, t is the number of iterations, and/> They are the feature maps updated after the t-th and t-1th mask iterations respectively, and are the output images generated for each progressive iteration/> Construct a weight map/> The formula is as follows:

Among them, j is the number of image blocks to be repaired this time, N ₁ is the total number of image blocks to be repaired, Represents the t-th iteration output image/> The weight mapping of the corresponding j-th repair area, n is the total number of image blocks generated during this repair process;

Then, use the normalization function to obtain the weight mapping value w _j , and the final output map I _o is the feature map output by the t-th iteration. The weighted sum at position (x, y) consists of:

in, H and W respectively represent the t-th iteration output image/> The height and width of , δ represents the activation function, /> Represents the t-th iteration output image/> eigenvalues.

Further, in step S3, weighted merging is an adaptive map obtained through a learning process after splicing the output feature maps of soft weight mapping and adaptive merging. The splicing operation can be performed as a whole without destroying the original feature map. Preserving feature information, the soft weight map is obtained by connecting the iterative output I _out and the input feature map I _in :

W _s =σ(C _onv ([I _in ,I _out ]))·(1-M)+M

Among them, σ is the activation function, M is the binary map, which is divided into 0 to represent the hole area and 1 to represent the effective area. The value I _s of the feature map obtained by soft weight is expressed as follows:

A device for filling holes in virtual viewpoint rendering based on deep learning, including a memory and one or more processors, with executable code stored in the memory. When the one or more processors execute the executable code, To implement the deep learning-based virtual viewpoint rendering hole filling method.

The advantages and beneficial effects of the present invention are:

The present invention uses a deep learning-based virtual viewpoint rendering hole filling method and device. Through end-to-end deep learning technology, labor costs are shortened compared with traditional technologies; the knowledge consistent attention module and context propagation feature loss module are added to improve efficiency. The accuracy of feature matching alleviates the problems of foreground and background aliasing and artifacts; adding a weighted merging module facilitates the fusion of iteratively generated feature maps without losing detailed information.

Description of the drawings

Figure 1 is a flow chart of a method in an embodiment of the present invention.

Figure 2 is a further detailed flow chart of the method in the embodiment of the present invention.

Figure 3 is a schematic structural diagram of the context feature propagation loss module in the embodiment of the present invention.

Figure 4 is a schematic structural diagram of a weighted merging module in an embodiment of the present invention.

Figure 5a is a flow chart of an adaptive merging method in an embodiment of the present invention.

Figure 5b is a flow chart of the soft weighting method in the embodiment of the present invention.

Figure 6 is a schematic structural diagram of the device in the embodiment of the present invention.

Detailed ways

Specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described here are only used to illustrate and explain the present invention, and are not intended to limit the present invention.

As shown in Figure 1, a deep learning-based virtual viewpoint rendering hole filling method includes the following steps:

Step S1: Input the original image I _gt and the mask image M to obtain the image to be repaired I _in , and input the image to be repaired I _in and the mask image M into the progressive iteration network (PINet);

Specifically, the original image I _gt and the mask image M are input to obtain the image to be repaired I _in , and the image to be repaired I _in and the mask image M are input into PINet. The PINet model is a convolutional neural network based on the U-Net network, including partial convolution, knowledge consistent attention module, and context feature propagation loss module, as shown in Figure 2. In the initial stage of the network, four 7×7 partial convolutions are introduced for local hole identification, and batch normalization and activation functions are used to accelerate model convergence. Partial convolution only uses effective pixels in the hole area to perform operations. The updated mask is retained throughout the iteration process until it is reduced and updated in the next iteration, which is beneficial to the extraction of shallow effective features.

The attention mechanism can extract feature maps with reasonable semantics and more accurate pixel information. In order to solve the problem of the same foreground and background weights in traditional methods based on patch filling, this paper combines the knowledge consistent attention module to perform hole filling. The advantage of the knowledge-consistent attention module is that the attention score is measured by the weighted sum of the current attention score and the score of the previous iteration, so that there is correlation between the preceding and following frame patches. It first obtains the attention score of each patch by normalizing the inner product between the known feature patch vector and the generated feature patch vector; then, based on whether the mask pixel value generated in the previous iteration is 1 (indicating a valid pixel) The final attention score is weighted; finally, the attention score is used to update the features of the missing patch, and the convolutional layer is used to improve the structural consistency of the input features and reconstructed features.

In the loss function design, this article combines L ₁ loss, perceptual loss, style loss and smoothing loss, as well as context feature propagation loss to improve the semantic consistency of background hole areas and known areas.

L ₁ loss is composed of the loss L _valid in the non-hole area and the loss L _hole in the hole area, that is

L ₁ ＝λ ₁ L _valid +λ ₂ L _hole (1)

In the formula, λ ₁ and λ ₂ represent the weights of L _valid and L _hole respectively, and

In the formula, M is a binary image, 0 represents the hole area, 1 represents the effective area, and ⊙ represents the dot product. I _gt represents the original reference view, Represents the prediction map output by the t-th progressive iteration network. C, H, and W are the number of channels and the height and width of the feature patch. They are C=256, H=32, and W=32 respectively. The weight parameter is set to λ _1. =1, λ ₂ =6.

Perceptual loss is used to enhance the similarity of high-level feature structures between filled images and real images, which is defined as

In the formula, I _o is the output result graph, represented by It consists of hole-generated pixels in and non-hole pixels in I _gt ; ψ _m (I _gt ) and ψ _m (I _o ) respectively represent the output of I _gt and I _o after the mth pooling layer of the pre-trained network VGG-16 features (m=1, 2, 3); H _m , W _m and C _m represent the height, width and number of channels of the extracted m-th feature map; N ₂ is the number of pooling layers.

Style loss is used to compensate for the problem that perceptual loss cannot effectively maintain the style consistency between the filled area and the surrounding area, and is defined as:

In the formula, The similarity of the features of each layer in the network is obtained by calculating the Gram matrix (Gram) of the feature map, that is/>

Smoothing loss is used to maintain the smoothness of the filled image, defined as:

In the formula, Pi _,j represents a pixel in _Io , and Pi _,j+1 and _Pi+1,j represent adjacent pixels in the vertical and horizontal directions of Pi _,j respectively. R is the expansion area with a pixel of 1 in the hole area, and _Nc is the total number of image blocks of _Io .

To sum up, combined with the context feature propagation loss L _CFP , the total loss function can be expressed as

In the formula, λ _per , λ _style , λ _tv , and λ _CFP are the weight parameters of perceptual loss, style loss, smoothing loss and context feature propagation loss respectively. Through experiments, λ _per =0.05, λ _style =120, and λ _tv =0.1 are obtained. , λ _CFP =0.5.

Step S2: Add a context feature propagation loss module in the deep stage of the network to improve the robustness of feature matching;

A context feature propagation loss module is added to the deep stage of the network to improve the robustness of feature matching. The process is shown in Figure 3. The similarity encoder takes the output image I _out generated by the progressive iteration as input, and the auxiliary encoder takes the input feature I _in as input. Before the features of the auxiliary encoder pass through the decoder, image patches and are generated through similarity encoding convolution calculations. The similarity S of the image block in the non-hole area, where the convolution kernel acquisition operation is to process the background feature map by extracting its own patch, and the size is set to a 4×4 convolution kernel with a step size of 2 to avoid causing a checkerboard effect. Then the similarity S is sent to the auxiliary decoder, and the auxiliary image is obtained through deconvolution and reconstruction of the auxiliary decoder. Under the guidance of auxiliary images, background holes can be generated with semantically consistent filling blocks.

Different from other loss terms used in this article, the context feature propagation loss is defined as the hole filling loss of the auxiliary image, which can be regarded as the sum of the local losses of the auxiliary image block, that is

In the formula, i represents the image block index with the highest similarity S in the iterative output I _out , u _i represents the auxiliary image block with the highest similarity S, and l _i (·) is the local hole loss of the i-th auxiliary image block.

Step S3: Input the feature map group I _out iteratively generated by the network into the weighted merging module for fusion;

The feature map group I _out generated iteratively by the network is input into the weighted merging module for fusion. When the number of progressive iterations reaches the set threshold, the filling of the hole area is completed. However, if the feature map output at this time is used directly, there will be problems of gradient disappearance and intermediate generated feature loss. If average merging and adaptive merging are adopted, the missing areas in the reconstructed images output early will affect the quality of the final output image. The overall process is shown in Figure 4.

In order to solve the above problems, this paper proposes a weighted merging method to fuse the feature maps generated by each progressive iteration. The adaptive merging method generates weights by analyzing masks without a learning process, as shown in Figure 5a, first dividing the feature map:

In the formula, t is the number of iterations, and/> They are the feature maps after the t-th and t-1th mask iterations, respectively. This article is the output image generated for each progressive iteration/> Construct a weight map/> The formula is as follows:

In the formula, j is the number of image blocks to be repaired this time, N ₁ is the total number of image blocks to be repaired, Represents the t-th iteration output image/> The corresponding weight mapping of the jth repair area, n is the total number of image blocks generated during this repair process.

Then, use the softmax function to normalize to obtain the weight mapping value w _j . I _o outputs the feature map from the t-th iteration The weighted sum at position (x, y) consists of:

In the formula, H and W respectively represent the t-th iteration output image/> height and width. δ represents the Leaky-ReLU activation function,/> Represents the t-th iteration output image/> eigenvalues.

Weighted merging is an adaptive map obtained through a learning process after concatenating (concat) the output feature maps of soft weight mapping and adaptive merging. The concatenation operation can retain the feature information as a whole without destroying the original feature map. The details of the soft weight method are shown in Figure 5b. This paper connects I _out and the input feature map I _in to obtain a soft weight map:

W _s =σ(Conv([I _in ,I _out ]))·(1-M)+M (12)

In the formula, σ is the sigmoid activation function, M is the binary image, 0 represents the hole area, and 1 represents the effective area. The value I _s of the feature map obtained by soft weighting is expressed as follows:

Corresponding to the foregoing embodiment of a deep learning-based virtual viewpoint rendering hole filling method, the present invention also provides an embodiment of a deep learning-based virtual viewpoint rendering hole filling device.

Referring to Figure 6, an embodiment of the present invention provides a virtual viewpoint rendering hole filling device based on deep learning, including a memory and one or more processors. The memory stores executable code, and the one or more processors execute The executable code is used to implement a deep learning-based virtual viewpoint rendering hole filling method in the above embodiment.

The embodiment of the present invention's virtual viewpoint rendering hole filling device based on deep learning can be applied to any device with data processing capabilities, and any device with data processing capabilities can be a device or device such as a computer. The device embodiments may be implemented by software, or may be implemented by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running them through the processor of any device with data processing capabilities. From the hardware level, as shown in Figure 6, it is a hardware structure diagram of any device with data processing capabilities where the virtual viewpoint drawing hole filling device based on deep learning of the present invention is located. In addition to the processor shown in Figure 6 , memory, network interface, and non-volatile memory, any device with data processing capabilities where the device in the embodiment is located may also include other hardware based on the actual functions of any device with data processing capabilities. In this regard No longer.

For details on the implementation process of the functions and effects of each unit in the above device, please refer to the implementation process of the corresponding steps in the above method, and will not be described again here.

As for the device embodiment, since it basically corresponds to the method embodiment, please refer to the partial description of the method embodiment for relevant details. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Embodiments of the present invention also provide a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the virtual viewpoint rendering hole filling method based on deep learning in the above embodiments is implemented.

The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart memory card (SMC), an SD card, or a flash memory card equipped on the device. (Flash Card) etc. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments. Modify the technical solution, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solution to depart from the scope of the technical solution of the embodiments of the present invention.

Claims (10)

1. A virtual viewpoint drawing cavity filling method based on deep learning is characterized by comprising the following steps:

step S1: based on the original image and a mask image thereof, obtaining a to-be-repaired image, inputting the to-be-repaired image and the mask image into a progressive iterative network, carrying out local hole identification on the to-be-repaired image by the progressive iterative network through partial convolution, carrying out hole filling based on a knowledge consistency attention mechanism, and constructing a loss function to promote semantic consistency of a background hole area and a known area;

step S2: constructing a context characteristic propagation loss, merging the loss function constructed in the step S1, outputting a progressive iterative network, performing similarity coding to obtain the similarity of the image blocks and the image blocks in the non-cavity area, and generating a filling block with semantic consistency by the background cavity based on the similarity;

step S3: and carrying out weighted combination on the output of the progressive iterative network to obtain a final repair graph.

2. The virtual viewpoint drawing cavity filling method based on deep learning according to claim 1, wherein: in step S1, the partial convolution uses only the effective pixels in the hole area to perform the operation, and the updated mask is preserved in the whole iteration process until the next iteration is reduced and updated.

3. The virtual viewpoint drawing cavity filling method based on deep learning according to claim 1, wherein: the knowledge consistent attention mechanism in step S1 is that first, the attention score of each patch is obtained by normalizing the inner product between the known feature patch vectors and the generated feature patch vectors; then, weighting the final attention score according to whether the mask pixel value generated in the last iteration is valid or not; finally, the attention score is used for updating the missing patch characteristics, and the structural consistency of the input characteristics and the reconstruction characteristics is improved through a convolution layer.

4. The virtual viewpoint drawing cavity filling method based on deep learning according to claim 1, wherein: in the step S1, the loss function merges L ₁ Loss, perceptual loss, style loss, and smoothing loss;

the L is ₁ Loss includes loss L of the cavity region _valid And loss L of cavity area _hole ：

L ₁ ＝λ ₁ L _valid +λ ₂ L _hole

Wherein lambda is ₁ And lambda (lambda) ₂ Respectively represent L _valid And L _hole M is a binarization map divided into a hole area and an effective area, and "" indicates dot product, I _gt The original image is represented by the original image,the C, H, W is the number of channels, the height and the width of the characteristic patch respectively;

the perceived loss is defined as:

wherein Io is the output result graph, composed ofHole-generating pixel and I in (1) _gt Non-hole pixels in (a); psi phi type _m (I _gt ) Sum phi _m (I _o ) Respectively represent I _gt And I _o Extracting the features output after the m pooling layer of the network through the pre-training features; h _m 、W _m And C _m Representing the height, width and channel number of the extracted mth feature map; n (N) ₂ Is the number of pooling layers;

the style loss is defined as:

wherein,the similarity of each layer of characteristics in the network is obtained by calculating the Gram matrix Gram of the characteristic diagram, namely

The smoothing loss is defined as:

wherein P is _i,j Representation I _o One pixel point of P _i,j+1 And P _i+1,j Respectively represent P _i,j Adjacent pixels in the vertical and horizontal directions, R is an expansion region in the hollow region, N _c Is I _o Is not included in the image block total number.

5. The virtual viewpoint drawing cavity filling method based on deep learning according to claim 1, wherein: in the step S2, the similarity between the image block and the non-hole area image block is generated by similarity encoding convolution calculation, wherein the convolution kernel obtaining operation is to process the background feature map by extracting the self patch.

6. The virtual viewpoint drawing cavity filling method based on deep learning according to claim 1, wherein: in the step S2, the image to be repaired is subjected to auxiliary encoding, and the encoding result is combined with the similarity to perform auxiliary decoding, so that an auxiliary image is obtained, and a filling block with semantic consistency is generated in the background cavity based on the auxiliary image guidance.

7. The virtual viewpoint drawing hole filling method based on deep learning according to claim 6, wherein: in the step S2, the context feature propagation loss is defined as the hole filling loss of the auxiliary image, i.e

Wherein I represents I of iterative output _out Image block index with highest similarity S, u _i Auxiliary image block with highest similarity S, l _i (·) is the local hole loss of the i-th auxiliary image block.

8. The virtual viewpoint drawing cavity filling method based on deep learning according to claim 1, wherein: in the step S3, the adaptive combining generates weights by analyzing the mask, and the method includes the following steps:

firstly, dividing a feature map:

where t is the number of iterations,and->The feature images after the iteration update of the t-th mask and the t-1 th mask are respectively the output images generated for each progressive iteration>Constructing a weight map +.>The formula is as follows:

wherein j is the number of image blocks to be repaired at this time, N ₁ Is the total number of image blocks to be repaired,representing the t-th iteration output image +.>The weight mapping of the corresponding j-th repair area, wherein n is the total number of image blocks generated in the current repair process;

then, a weight mapping value w is obtained by using a normalization function _j Most, at bestFinal output figure I _o Outputting the characteristic diagram by the t-th iterationWeighted sum composition at position (x, y):

wherein,h and W respectively represent the t-th iteration output image +.>Delta represents the activation function,/-the height and width of->Representing the t-th iteration output image +.>Is a characteristic value of (a).

9. The virtual viewpoint drawing cavity filling method based on deep learning according to claim 1, wherein: in the step S3, the weighted combination is an adaptive graph obtained through a learning process after the soft weight mapping and the adaptive combination of the output feature graphs are spliced, and the soft weight mapping is to iterate the output I _out And input feature map I _in And (3) connecting to obtain:

W _s ＝σ(Conv([I _in ,I _out ]))·(1-M)+M

wherein sigma is an activation function, M is a binarization map, and the map is divided into a cavity area, an effective area and a value I of a feature map obtained by soft weight _s The expression is as follows:

10. a deep learning-based virtual viewpoint drawing hole filling device, comprising a memory and one or more processors, wherein executable code is stored in the memory, and the one or more processors are used for realizing the deep learning-based virtual viewpoint drawing hole filling method according to any one of claims 1-9 when executing the executable code.