CN111275638A - Face restoration method for generating confrontation network based on multi-channel attention selection - Google Patents

Abstract

The present invention provides a face restoration method based on multi-channel attention selection and generating confrontation network, which comprises the following steps: S1, collecting face data and performing preprocessing; S2, establishing a face restoration model and a loss function; S3, the first step In the first stage, learn the image generation sub-network _{G i} and initially repair the image; S4, in the second stage, generate the intermediate output map IG and learn the multi-channel attention map _IA ; S5, construct the multi-channel attention selection model and output the final composite map ; S6, perform face repair. The face inpainting model includes a generator network G _i , a parameter sharing discriminator D and a multi-channel attention selection network _Ga , and the loss function includes an uncertainty pixel loss function and an adversarial loss function. The face repairing method provided by the present invention effectively learns the uncertainty map to guide pixel loss, thereby achieving more powerful optimization and providing a better face repairing method.

Description

Face inpainting method based on multi-channel attention selection generative adversarial network

【Technical field】

The invention relates to the fields of deep learning and image processing, in particular to a face inpainting method based on multi-channel attention selection and generating confrontation network.

【Background technique】

In the field of image inpainting, especially for intraocular painting, although DNNs (deep neural networks) can produce semantically plausible and realistic-looking results, most deep learning techniques fail to preserve the identities of people in photos. For example, a DNN can learn to open a pair of closed eyes, but the DNN itself does not guarantee that the new eyes will correspond to the specific eye structure of the original person.

GAN (Generative adversarial networks) is a specific type of deep network that includes a learnable adversarial loss function represented by a discriminator network. GANs have been successfully used to generate faces from scratch, or to draw missing areas on faces, suitable for general facial manipulation.

A GAN variant, Conditional GAN (cGAN), can constrain the generator with additional information. By adding the reference information of the unified identity, the GAN does not have to phantom texture or structure from scratch, but will still retain the semantics of the original image to produce high-quality personalized inpainting results. However, GANs still fail in some cases, such as when a person's eyes are partially obscured by a strand of hair, or sometimes cannot be colored correctly, resulting in strange artifacts.

The generative adversarial network three-channel generative space may be insufficient for learning good mappings, and it becomes a feasible attempt to expand the generative space and learn automatic selection mechanisms to synthesize more fine-grained generative results. It is possible to use the multi-channel attention selection GAN framework (SelectionGAN) for image inpainting tasks.

Therefore, the present invention provides a face inpainting system based on multi-channel attention selection generative adversarial network.

[Content of the invention]

In order to solve the problems of image occlusion, incorrect coloring and repairing and strange artificial repairing traces in the face inpainting technology under individual conditions, the present invention provides a face inpainting method based on multi-channel attention selection generative confrontation network.

A face inpainting method based on multi-channel attention selection generative adversarial network, which includes the following steps:

S1. Collect face data and perform preprocessing: acquire face image pairs of the same person, including images with eyes open and eyes closed, and perform preprocessing on the collected images;

S2, establish a face restoration model and a loss function: design and build a face restoration model and a loss function, the face restoration model is based on a conditional confrontation generation network, and the face restoration model includes a generator network G _i , a parameter sharing identification A device D and _a multi-channel attention selection network Ga, the loss function includes an uncertainty pixel loss function and an adversarial loss function;

S3, the first stage, learning the image generation sub-network G _i and preliminarily repairing the image: learning the image generation sub-network G _i , the image generation sub-network G _i receives the image composed of the labeled input image I _a and the reference image R _g pair, and initially repair the image pair to generate a repaired image I' _g =G _i (I _a , R _g );

S4, the second stage, generate the intermediate output map IG and learn the channel attention map _IA : the rough repaired image _I'g , the _ground -truth image _Ig from the image generation sub-network _Gi , and the generator network from the The deep feature map F _i of the last layer of G _i is taken as a new feature F _c =concat(I' _g , F _i , I _g ), where concat( ) is a function of cascade operation by channel; the new feature _Fc is input into the multi-channel attention selection module _Ga , generates multiple intermediate output graphs _IG , and simultaneously learns a set of multi-channel attention graphs _IA with the same number of intermediate generated graphs to guide multiple optimization losses;

S5, construct a multi-channel attention selection model and output the final synthetic graph: use the multi _- channel attention _graph IA to perform channel selection from the intermediate output graph IG, and obtain the final synthetic graph 1"_g;

S6. Perform face restoration: input the test image into the trained face restoration model to obtain a high-quality face restoration image.

Preferably, the face restoration model in step S2 adopts a cascade strategy, and outputs a rough restoration image through the generator network G _i , so as to generate blurred eye details and high pixel-level dissimilarity of the target image, and then pass The multi-channel attention selection network Ga utilizes the coarse inpainted image to produce _a fine-grained final output.

Preferably, in step S4, the input of the new feature F _c into the multi-channel attention selection module _Ga specifically includes: selecting each merged feature by element-wise multiplication with the input feature, and selecting the feature with the same The resolution _rescales the pooled features, and feeds the feature Fc to the convolutional layer to generate a new multi-scale feature _Fc ' for use in the multi-channel attention selection module Ga, applying _a set of M spatial scales {S _i } (i=1～M) are used to generate merged elements with different spatial resolutions, and the pooling process is shown as:

表示为逐元素乘法。Among them, concat( ) is a function of cascade operation by channel, F _c is a new feature, pl_ups( ) is expressed as pooling with scale s,

Represented as element-wise multiplication.

之后进行tanh(·)非线性激活操作得到，所述多通道注意力图I_A通过N个卷积滤波器

之后进行标准化的基于通道的softmax函数操作后得到，所述中间输出图I_G和所述多通道注意力图I_A的计算分别为：Preferably, in step S4, the intermediate output _{graph IG} uses N convolution filters

After that, the tanh(·) nonlinear activation operation is performed to obtain the multi-channel attention map I _A through N convolution filters

After performing a standardized channel-based softmax function operation, the intermediate output _{graph IG} and the multi-channel attention graph _IA are calculated as:

Preferably, in step S5, the calculation formula of described final composite graph 1 " _g is:

表示逐元素加法，

表示为逐元素乘法。where I” _g represents the final synthesized generative map selected from multiple different results, I _A is the multi-channel attention map, I _G is the intermediate output map, and the symbol

represents element-wise addition,

Represented as element-wise multiplication.

Preferably, in the first stage, the parameter sharing discriminator D takes the roughly repaired image I'g of the image generation sub-network _{G i} and the ground truth image _Ig as input, and discriminates whether the two are related to each other; The parameter-sharing discriminator D takes the final synthetic image _I'g and the ground-truth image _Ig as input in the second stage, encouraging the parameter-sharing discriminator D to distinguish the diversity of image structures and capture local perceptual information .

Preferably, the uncertainty pixel loss function is:

where L ⁱ _p represents the pixel-level loss map, U _i represents the ith uncertainty map, and σ( ) is the sigmoid function for pixel-level normalization.

Preferably, the adversarial loss function in the first stage is to distinguish the pair [I _a , I' _g ] from the real image pair [I _a , I _g ], and in the second stage, the adversarial loss of D is formulated is: distinguish the synthetic image pair [I _a , I " _g ] from the real image pair [I _a , I _g ], and the formulas are as follows:

The adversarial loss function formula is as follows: L _cGAN =L _cGAN (I _a , I' _g )+ _{λL cGAN} (I _a , I " _g ),

The total optimization loss is:

where L ⁱ _p uses L1 reconstruction to calculate the pixel loss between the generated images I'g, I" _g and the corresponding _ground -truth images respectively, and _LtV is the total variation regularization of the final synthetic image I" _g (TV)regularization):

where λ _i and λ _tv are trade-off parameters to control the relative importance of different objectives.

Compared with the prior art, the present invention applies the generative adversarial network based on multi-channel attention selection to face restoration, and expands the generation space through the generator network G _i , the parameter sharing discriminator D and the multi-channel attention selection network _Ga . The automatic learning automatic selection mechanism synthesizes more fine-grained generation results, and the multi-channel attention selection network _Ga concentrates on selecting the intermediate generated graphs of interest, and can significantly improve the quality of the final output. The multi-channel attention module can also effectively learn the uncertainty map to guide the pixel loss for more powerful optimization, providing a better face inpainting method.

【Description of drawings】

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, under the premise of no creative work, other drawings can also be obtained from these drawings, wherein:

1 is a flowchart of a face repair method based on multi-channel attention selection generative adversarial network provided by the present invention;

2 is a schematic diagram of a face restoration model provided by the present invention;

FIG. 3 is a network structure diagram of a multi-channel attention selection module provided by the present invention.

【Detailed ways】

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to FIG. 1 to FIG. 3 , the present invention provides a face restoration method based on multi-channel attention selection generative adversarial network. The steps of the face restoration method are as follows:

S1. Collect face data and perform preprocessing: acquire face image pairs of the same person, including images with eyes open and eyes closed, and perform preprocessing on the collected images. Collect a large number of images as a dataset, use such as openCV to perform face recognition on the image, and extract the information of the face, especially the eyes. Crop the collected images into face training images of a set size so that the eyes and mouth can be centered.

S2, establish a face restoration model and a loss function: design and build a face restoration model and a loss function, the face restoration model is based on a conditional confrontation generation network, and the face restoration model includes a generator network G _i , a parameter sharing identification D and multi-channel attention selection network _Ga , the loss functions include uncertainty pixel loss function and adversarial loss function.

The face inpainting model adopts a cascade strategy, and outputs a rough inpainting image through the generator network G _i , thereby producing blurred eye details and high pixel-level dissimilarity of the target image. The first stage is from coarse to fine. Generate policies to improve synthetic performance based on rough predictions. The coarse inpainted image is then used in the second stage to generate _a fine-grained final output through the multi-channel attention selection network Ga.

S3, the first stage, learning the image generation sub-network G _i and preliminarily repairing the image: learning the image generation sub-network G _i , the image generation sub-network G _i receives the image composed of the labeled input image I _a and the reference image R _g pair, and preliminarily inpaint the image pair to generate an inpainted image I' _g =G _i (I _a , R _g ). The reference image _Rg can provide stronger supervision ability. This generation adds more robust supervision between the input image I _a , the reference image R _g , and the ground-truth image I _g , thereby facilitating the optimization of the network.

In the first stage, the parameter sharing discriminator D is used to take the roughly inpainted image I'g of the image generation sub-network _{G i} and the ground-truth image _Ig as input to discriminate whether the two are related to each other.

S4, the second stage, generate intermediate output and learn multi-channel attention map: the rough repaired image I' _g from the image generation sub-network G _i , the ground-truth image I _g and the final image from the generator network G _i The deep feature map F _i of the layer is taken as a new feature F _c =concat(I' _g , F _i , I _g ), where concat( ) is a function of cascade operation by channel; the new feature F _c is input to In the multi-channel attention selection module _Ga , multiple intermediate output graphs _IG are generated, and at the same time a set of multi-channel attention graphs _IA with the same number as the intermediate generated graphs are learned to guide multiple optimization losses.

Single-scale features may not capture all the necessary details for fine-grained generation, so the present invention proposes a multi-scale spatial pooling scheme that uses a set of different kernel sizes and strides forward, performing on the same input features Global average pooling. In this way, multi-scale features with different receptive fields can be obtained to perceive different details. The input of the new feature F _c into the multi-channel attention selection module _Ga specifically includes: selecting each merged feature by element-wise multiplication with the input feature, and re-adjusting and pooling the feature at the same resolution The features of F _c are fed to the convolutional layers to generate new multi-scale features F' _c for use in the multi-channel attention selection module Ga, _a set of M spatial scales {S _i are applied in the merging } (i=1～M) is used to generate merged elements with different spatial resolutions, and the pooling process is expressed as:

表示为逐元素乘法。Among them, concat( ) is a function of cascade operation by channel, F _c is a new feature, pl_ups( ) is expressed as pooling with scale s,

Represented as element-wise multiplication.

之后进行tanh(·)非线性激活操作得到，所述多通道注意力图I_A通过N个卷积滤波器

之后进行标准化的基于通道的softmax函数操作后得到，所述中间输出图I_G和所述多通道注意力图I_A的计算分别为：The multi-channel attention selection module _Ga can automatically make spatial and temporal selections from the generation to synthesize fine-grained final outputs. Given a multi-scale feature quantity F _c '∈R (above h×w×c), where h and w are the width and height of the feature, and c is the number of channels. The intermediate output _{graph IG} is obtained by using N convolutional filters

After that, the tanh(·) nonlinear activation operation is performed to obtain the multi-channel attention map I _A through N convolution filters

After performing a standardized channel-based softmax function operation, the intermediate output _{graph IG} and the multi-channel attention graph _IA are calculated as:

In the second stage, the parameter sharing discriminator D takes the final synthetic image _I'g ' and the ground-truth image _Ig as input, and encourages the parameter sharing discriminator D to distinguish the diversity of image structure and capture Local perception information.

S5. Build a multi-channel attention selection model and output a final synthetic graph: the multi _- channel attention graph IA is used to perform channel selection from the intermediate output graph _{IG, and a final synthetic graph I"g is} obtained.

The calculation formula of the final synthetic graph I" _g is:

表示逐元素加法，

表示为逐元素乘法。where I” _g represents the final synthesized generative map selected from multiple different results, I _A is the multi-channel attention map, I _G is the intermediate output map, and the symbol

represents element-wise addition,

Represented as element-wise multiplication.

S6. Perform face restoration: input the test image into the trained face restoration model to obtain a high-quality face restoration image.

的卷积层，以生成一组K个不确定性图。所述不确定性像素损失函数为：It should be noted that the inpainting map initially obtained from the pre-trained model is inaccurate for all pixels, which leads to wrong guidance during training. To solve this problem, the present invention proposes a generated multi-channel attention map _IA to learn the uncertainty map to control the optimization loss. Suppose we have K different loss maps to guide, first concatenate multiple generated multi-channel attention maps I _A and pass them to filters with K filters

to generate a set of K uncertainty maps. The uncertainty pixel loss function is:

where L ⁱ _p represents the pixel-level loss map, U _i represents the ith uncertainty map, and σ( ) is the sigmoid function for pixel-level normalization.

The adversarial loss function in the first stage is to distinguish the pair [I _a , I′ _g ] from the real image pair [I _a , I _g ]. In the second stage, the adversarial loss of D is formulated as: The synthetic image pair [I _a , I″ _g ] is distinguished from the real image pair [I _a , I _g ], and the formulas are as follows:

Both losses aim to preserve local structural information and produce visually pleasing synthetic images. Therefore, the adversarial loss of the proposed SelectionGAN is the sum of the equations of (5) and (6). The adversarial loss function formula is as follows:

L _cGAN = L _cGAN (I _a , I' _g )+ _{λL cGAN} (I _a , I " _g ) (7)

The total optimization loss is the weighted sum of the above losses, the generator network G _i , the parameter sharing discriminator D and the multi-channel attention selection network _Ga are trained in an end-to-end manner, optimizing the following min-max function The total optimization loss is:

where L ⁱ _p uses L1 reconstruction to calculate the pixel loss between the generated images I'g, I" _g and the corresponding _ground -truth images respectively, and _LtV is the total variation regularization of the final synthetic image I" _g (TV)regularization):

where λ _i and λ _tv are trade-off parameters to control the relative importance of different objectives.

Compared with the prior art provided by the present invention, the present invention applies the generative confrontation network based on multi-channel attention selection to face restoration, through the generator network G _i , the parameter sharing discriminator D and the multi-channel attention selection network _Ga Enlarging the generation space and automatically learning the automatic selection mechanism to synthesize more fine-grained generation results, the multi-channel attention selection network _Ga concentrates on selecting the intermediate generated graphs of interest, and can significantly improve the quality of the final output. The multi-channel attention selection network _Ga can also effectively learn the uncertainty map to guide the pixel loss, so as to achieve more powerful optimization and provide a better face inpainting method.

The above are only the embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, improvements can be made without departing from the inventive concept of the present invention, but these belong to the present invention. scope of protection.

Claims (8)

1. a face repair method based on multi-channel attention selection generation confrontation network, is characterized in that, comprises the following steps: S1. Collect face data and perform preprocessing: acquire face image pairs of the same person, including images with eyes open and eyes closed, and perform preprocessing on the collected images; S2, establish a face restoration model and a loss function: design and build a face restoration model and a loss function, the face restoration model is based on a conditional confrontation generation network, and the face restoration model includes a generator network G _i , a parameter sharing identification A device D and _a multi-channel attention selection network Ga, the loss function includes an uncertainty pixel loss function and an adversarial loss function; S3, the first stage, learning the image generation sub-network G _i and preliminarily repairing the image: learning the image generation sub-network G _i , the image generation sub-network G _i receives the image composed of the labeled input image I _a and the reference image R _g pair, and initially repair the image pair to generate a repaired image I' _g =G _i (I _a , R _g ); S4, the second stage, generate an intermediate output _{graph IG} and learn a multi-channel attention graph IA: combine the roughly _inpainted image _I'g , the ground-truth image _Ig from the image generation sub-network _Gi , and the images from the generator The deep feature map F _i of the last layer of the network G _i is taken as a new feature F _c =concat(I' _g , F _i , I _g ), where concat( ) is a function of cascade operation by channel; The feature _Fc is input into the multi-channel attention selection module _Ga , which generates multiple intermediate output graphs _IG , while learning a set of multi-channel attention graphs _IA with the same number of intermediate generated graphs to guide multiple optimization losses ; S5, construct a multi-channel attention selection model and output the final synthetic graph: use the multi _- channel attention _graph IA to perform channel selection from the intermediate output graph IG, and obtain the final synthetic graph 1"_g; S6. Perform face restoration: input the test image into the trained face restoration model to obtain a high-quality face restoration image. 2. the face repairing method based on multi-channel attention selection generating adversarial network according to claim 1, is characterized in that, described in step S2, the face repairing model adopts cascade strategy, through described generator network G _i A coarse inpainted image is output, resulting in blurred eye details and high pixel-level dissimilarity of the target image, and then the coarse inpainted image is used by the multi-channel attention selection network Ga to produce _a fine-grained final output. 3. The face restoration method based on multi-channel attention selection generative adversarial network according to claim 1, is characterized in that, in step S4, described new feature F _c is input to described multi-channel attention selection module _Ga Specifically including: selecting each merged feature by element-wise multiplication with the input feature, re-adjusting the pooled feature with the same resolution, feeding the feature _Fc to the convolutional layer to generate a new multi-scale Feature F _c ' for use in the multi-channel attention selection module Ga, a set of M spatial scales {S _i } (i= _1∼M ) are applied in the binning for generating bins with different spatial resolutions elements, and the pooling process is as follows:

Among them, concat( ) is a function of cascade operation by channel, F _c is a new feature, pl_ups( ) is expressed as pooling with scale s,

Represented as element-wise multiplication. 4. the face repairing method based on multi-channel attention selection to generate adversarial network according to claim 1, is characterized in that, in step S4, described intermediate output _{graph IG} by using N convolution filters

After that, the tanh(·) nonlinear activation operation is performed to obtain the multi-channel attention map I _A through N convolution filters

After performing a standardized channel-based softmax function operation, the intermediate output _{graph IG} and the multi-channel attention graph _IA are calculated as:

5. the face repairing method based on multi-channel attention selection generating adversarial network according to claim 1, is characterized in that, in step S5, the calculation formula of described final synthetic graph 1 " _g is:

where I” _g represents the final synthesized generative map selected from multiple different results, I _A is the multi-channel attention map, I _G is the intermediate output map, and the symbol

represents element-wise addition,

Represented as element-wise multiplication. 6. The face inpainting method based on multi-channel attention selection generative adversarial network according to claim 1, characterized in that, in the first stage, the parameter sharing discriminator D generates the image into the sub-network G _i . Roughly repaired image _I'g and the ground truth image _Ig are used as input to discriminate whether the two are related to each other; the parameter sharing discriminator D combines the final composite image _I'g and the ground truth in the second stage The image _Ig as input, encourages the parameter sharing discriminator D to distinguish the diversity of image structure and capture local perceptual information. 7. The face restoration method based on multi-channel attention selection generative adversarial network according to claim 1, is characterized in that, described uncertainty pixel loss function is:

in

represents the pixel-level loss map, U _i represents the ith uncertainty map, and σ( ) is the sigmoid function for pixel-level normalization. 8. The face restoration method based on multi-channel attention selection generative adversarial network according to claim 7, characterized in that, the adversarial loss function in the first stage is a pair of [I _a , I′ _g ] and real Image _pairs [I _a , I _g ] are _{discriminated} , and in the second stage, the adversarial loss of D is formulated _{as :} To distinguish, the formulas are as follows:

The adversarial loss function formula is as follows: L _cGAN =L _cGAN (I _a , I' _g )+ _{λL cGAN} (I _a , I " _g ), and the total optimization loss is:

in

Calculate the pixel loss between the generated images I'g, I" _g and the corresponding _ground -truth image using L1 reconstruction respectively, _LtV is the total variation (TV) regularization of the final synthetic image I" _g ):

where λ _i and λ _tv are trade-off parameters to control the relative importance of different objectives.

Images