CN111814566A - Image editing method, device, electronic device, and storage medium - Google Patents

Abstract

The embodiments of the present application disclose an image editing method, device, electronic device, and storage medium. The method includes: acquiring a face image to be edited and a mask map corresponding to a target attribute region in the face image to be edited, and acquiring The reference image; according to the mask map, the face image to be edited is processed as a missing image missing the target attribute area; the missing image is encoded by the first encoder to obtain the missing feature corresponding to the missing image; Perform image coding on the reference image to obtain the reference feature corresponding to the reference image; according to the mask map, the missing feature and the reference feature are fused by the attention model to obtain the fusion feature; the image fusion feature is processed by the decoder. Decoding is performed to obtain the target image corresponding to the face image to be edited and the reference image. The embodiment of the present application improves the diversity of face attribute editing, and avoids the influence on irrelevant areas outside the target attribute area.

Description

Image editing method, device, electronic device, and storage medium

technical field

The embodiments of the present application relate to the technical field of image processing, and in particular, to an image editing method, apparatus, electronic device, and storage medium.

Background technique

Face editing has attracted great attention in the computer vision community due to its potential applications in film and television production, photo manipulation, and interactive entertainment. Face editing has made great progress in recent years with the development of generative adversarial networks. The current mainstream face editing methods are roughly divided into three categories: methods based on label conditions, methods based on geometric guidance, and methods based on reference guidance. Label-condition-based methods edit face attributes conditioned on predefined attribute labels. This method is conditioned on binary attribute labels and is only suitable for editing notable attributes with changes in appearance and texture (such as hair color, ageing, and beard removal, etc.) , it is difficult to achieve abstract shape changes (such as hooked nose, phoenix eyes, etc.), and it lacks the flexibility to control the shape of high-level semantic facial components (such as eyes, nose, and mouth, etc.). In order to flexibly edit the shape of face attributes, geometric guidance-based methods propose to use accurate intermediate representations (such as keypoints, segmentation maps and contour sketches, etc.) to achieve face attribute editing with obvious topological deformation, however, this method is time-consuming It is laborious and requires drawing skills. Different from the method based on geometric guidance, the method based on reference guidance directly learns the corresponding face information from the reference image for face editing, does not require accurate auxiliary representation, and avoids the need for accurate contour sketches, color maps and face editing. Dependency of the segmentation map.

In the prior art, there are two methods based on reference guidance: ExGANs and ELEGANT.

ExGANs is a method for face completion based on reference images. ExGANs are an extension of Conditional Generative Adversarial Networks conditioned on a reference image or perceptual encoding with target completion content. The training process of ExGANs can be summarized as: label the eyes of the input image, that is, remove the eye area; complete the image with the reference image or perceptual coding with the target completion content as a guide; pass the content between the input image and the completed image Reconstruction loss, calculate the gradient of the generator parameters; calculate the gradient of the discriminator parameters through the completed image, original image and reference image or perceptual code; backpropagate the error of the discriminator through the generator.

ELEGANT is a method for editing face attributes based on reference images. ELEGANT takes as input two images A and B with opposite properties, the two images do not require the same identity. All attributes of a picture are encoded in a decoupled manner in the latent space, that is, all attributes are assumed to be independent of each other and can be represented separately. Attributes are transferred from one image to another by exchanging latent encodings of attributes of the same type of two images. Combining the original attribute encoding and the exchanged attribute encoding of the two pictures, four encoding results can be obtained. To alleviate the impact on the target attribute-independent region, the edited attribute region is represented in the form of a residual map. Finally, the residual map is added to the corresponding original image to obtain four generation results: the reconstruction of the A picture, the A picture after exchanging attributes, the reconstruction of B picture, and the B picture after exchanging attributes.

Since ExGANs requires that the reference image must be of the same identity, and the facial features of the same identity will not change, the model can only be applied to the eye-opening-eye-closed eye completion task, and cannot be extended to other face attribute editing . However, ELEGANT is limited to editing the properties of appearance and texture changes, and cannot edit the properties of abstract semantic shapes, and because ELEGANT encodes all properties in a decoupled manner, however, the properties of the attribute annotation set are related to each other (for example: hair color and age), it is impossible to edit the properties of abstract semantic shapes. Encodings are completely separate and independent, therefore, the premise of decoupling does not correspond to reality, resulting in significant effects on regions unrelated to the target attribute.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an image editing method, apparatus, electronic device, and storage medium, which help to improve the diversity of face attribute editing and avoid influence on irrelevant areas.

In order to solve the above problems, in the first aspect, an embodiment of the present application provides an image editing method, including:

Obtain the face image to be edited and the mask map corresponding to the target attribute area in the face image to be edited, and obtain the reference image;

According to the mask map, the face image to be edited is processed as a missing image missing the target attribute area;

Image encoding is performed on the missing image by the first encoder to obtain the missing feature corresponding to the missing image;

Image coding is performed on the reference image by the second encoder to obtain a reference feature corresponding to the reference image;

According to the mask map, the missing feature and the reference feature are fused through an attention model to obtain a fused feature;

The decoder performs image decoding on the fusion feature to obtain the target image corresponding to the face image to be edited and the reference image.

In a second aspect, an embodiment of the present application provides an image editing apparatus, including:

an image acquisition module, used for acquiring the face image to be edited and the mask map corresponding to the target attribute region in the face image to be edited, and acquiring the reference image;

an image missing processing module, configured to process the to-be-edited face image as a missing image missing the target attribute area according to the mask map;

a first encoding module, configured to perform image encoding on the missing image by the first encoder to obtain the missing feature corresponding to the missing image;

a second encoding module, configured to perform image encoding on the reference image through a second encoder to obtain a reference feature corresponding to the reference image;

a feature fusion module, configured to fuse the missing feature and the reference feature through an attention model according to the mask map to obtain a fused feature;

The decoding module is used for performing image decoding on the fusion feature through a decoder to obtain the target image corresponding to the face image to be edited and the reference image.

In a third aspect, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor, and the processor implements the computer program when the processor executes the computer program. The image editing method described in the embodiments of the present application.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the image editing method disclosed in the embodiment of the present application are performed.

The image editing method, device, electronic device, and storage medium provided by the embodiments of the present application process the face image to be edited as a missing target attribute area according to the mask map corresponding to the target attribute area in the face image to be edited Image, the missing image is encoded by the first encoder to obtain the missing feature corresponding to the missing image, the image encoding is performed on the reference image by the second encoder to obtain the reference feature corresponding to the reference image, and the missing feature and the reference feature are combined by the attention model. The fusion features are fused into fusion features, and the fused features are decoded by the decoder to obtain the target image. Since only the mask map corresponding to the face image to be edited and the target attribute area needs to be obtained, the target attribute area in the reference image can be converted into the target attribute area. The corresponding features are migrated to the target attribute area of the face image to be edited, so that the features of each area can be migrated, which solves the defect that ExGANs in the prior art can only edit the eye area, and the reference image can be the same as the face to be edited. Image images with different identities, thus improving the diversity of face attribute editing, and there is no need to decouple the target attribute area from other areas, avoiding the impact on irrelevant areas outside the target attribute area.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some of the drawings in the present application. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

1 is a flowchart of an image editing method according to Embodiment 1 of the present application;

2 is a computational structure diagram of fusion of missing features and reference features through an attention model in an embodiment of the present application;

3 is a schematic structural diagram of an image editing apparatus according to Embodiment 2 of the present application;

FIG. 4 is a schematic structural diagram of an electronic device according to Embodiment 3 of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

Example 1

An image editing method provided by this embodiment, as shown in FIG. 1 , includes steps 110 to 140 .

Step 110: Obtain a face image to be edited and a mask map corresponding to a target attribute region in the face image to be edited, and obtain a reference image.

The target attribute area includes at least one of an eye area, a nose area, a mouth area, or other areas (such as hair, beard, etc.). The face image to be edited is an image to be edited with face attributes, and the reference image is used to provide reference features corresponding to the target attribute area.

The face image to be edited can be obtained according to the storage path of the face image to be edited specified by the user, and the target attribute region to be edited specified by the user can be obtained, and the reference image can be obtained according to the storage path of the reference image specified by the user; The face image to be edited and the reference image uploaded by the user can be obtained, and the target attribute area to be edited specified by the user can be obtained. Obtaining the target attribute area to be edited specified by the user may be obtaining the name of the target attribute area specified by the user, and performing face recognition based on the name to determine the target attribute area; or, on the face image to be edited for the user to operate and select Target property area.

After the target attribute region in the face image to be edited, a mask map corresponding to the target attribute region in the face image to be edited is generated, thereby obtaining a mask map corresponding to the target attribute region in the face image to be edited. In the mask map, the value of the position corresponding to the target attribute area is 0, while other areas have values, and the value may be 1.

Step 120: According to the mask map, the face image to be edited is processed as a missing image missing the target attribute area.

According to the mask map corresponding to the target attribute area in the face image to be edited, the target attribute area in the face image to be edited is cut out, so that the face image to be edited is processed into a missing image missing the target attribute area.

In an embodiment of the present application, processing the face image to be edited into a missing image missing the target attribute area according to the mask map includes: eroding and dilating the mask map processing to obtain a processed mask map; and according to the processed mask map, remove the target attribute area in the face image to be edited to obtain a missing image.

Its value, expansion is the operation of finding the local maximum, that is, convolving the image with the kernel, calculating the maximum value of the pixels in the area covered by the kernel, and assigning this maximum value to the element specified by the reference point. Dilation causes the highlighted areas in the image to grow gradually. Erosion and dilation are opposite operations. Erosion is an operation to find a local minimum. Erosion will gradually reduce the highlight area in the image.

First, the mask image corresponding to the target attribute area in the face image to be edited is subjected to corrosion processing, and then the corrosion result is expanded to obtain the processed mask image. Noise in the mask image can be removed by eroding and dilating the original mask image. Use the processed mask image to perform a matting operation on the face image to be edited, so as to process the face image to be edited as a missing image of the missing target attribute area. Since the processed mask image eliminates noise, the obtained Missing images are more accurate and can improve the effect of subsequent image editing.

Step 130: Perform image encoding on the missing image by the first encoder to obtain the missing feature corresponding to the missing image.

The first encoder is an encoder used for images missing a certain area. The first encoder may be an image encoder based on a neural network. After training, the network parameters have been determined, and the image with a missing area can be directly encoded.

Image coding is performed on the missing image by the first encoder to compress the missing image, and the information contained in the missing image is represented by a smaller number of bits on the premise of satisfying a certain quality, so as to obtain the missing feature corresponding to the missing image.

Step 140: Perform image coding on the reference image by using the second encoder to obtain a reference feature corresponding to the reference image.

Wherein, the second encoder is the encoder used for the complete image. The second encoder can be an image encoder based on a neural network. After training, the network parameters have been determined, and the complete image can be directly encoded.

Image coding is performed on the reference image by the second encoder to compress the reference image to obtain the reference feature corresponding to the reference image.

Step 150: According to the mask map, the missing feature and the reference feature are fused through an attention model to obtain a fused feature.

Through the attention model, first self-attention processing is performed on the reference features to obtain self-attention features, and the reference is strengthened according to the mask map corresponding to the target attribute area in the face image to be edited or the mask map after erosion and expansion processing. In the feature, the feature of the area corresponding to the target attribute area is weakened, and then the processed missing feature and the reference feature are fused to transfer the feature of the reference feature corresponding to the target attribute area to the missing feature. Among the features corresponding to the target attribute region of , the fusion features are obtained.

In an embodiment of the present application, according to the mask map, the missing feature and the reference feature are fused through an attention model to obtain a fused feature, including: generating an attention map corresponding to the missing feature; fusing The missing feature and the attention map are used to obtain the self-attention feature corresponding to the missing feature; according to the mask map, the attention map and the reference feature, an instance guiding feature of the reference feature to the missing feature is generated; fusion The self-attention feature and the instance guidance feature are used to obtain a fusion feature of the missing feature and the reference feature.

FIG. 2 is a computational structure diagram of fusing missing features and reference features through an attention model in an embodiment of the present application. As shown in Figure 2, the missing feature is first subjected to 1×1 convolution processing to obtain the query feature corresponding to the missing feature, then the transpose of the query feature is multiplied by the query feature, and the matrix product is softmaxed Calculate, get the attention map corresponding to the missing feature, fuse the missing feature and the attention map, get the self-attention feature corresponding to the missing feature, and get the self-attention feature through the missing feature, which is expressed in the following formula:

F_q＝Conv(F_s)，表示询问特征，Conv表示1×1卷积。where F _a represents the self-attention feature, F _s represents the original feature input to the attention model, that is, the missing feature, and F _m represents the attention map,

F _q = Conv(F _s ), representing the query feature, and Conv representing a 1×1 convolution.

The reference features are processed according to the mask map and attention map to strengthen the features of the regions corresponding to the target attribute region in the reference features, and weaken the features of other regions, so as to generate the instance-guided features of the missing features from the reference features. The self-attention feature and instance-guided feature are fused to obtain the fusion feature of the missing feature and the reference feature, that is, the feature of the region corresponding to the target attribute area in the reference image is transferred to the target attribute area in the missing image, and the obtained fusion feature .

In an embodiment of the present application, generating an instance guidance feature of the reference feature for the missing feature according to the mask map, the attention map and the reference feature includes: fusing the reference feature and the self- attention map to obtain the reference attention feature; multiply the mask map and the reference attention feature to obtain the instance guidance feature corresponding to the area outside the target attribute area, and the area outside the target attribute area corresponds to The instance guiding feature of is as the first instance guiding feature; calculate the difference between the 1 matrix and the mask map, obtain the inverse mask map of the mask map, and multiply the inverse mask map and the reference feature operation to obtain the instance guidance feature corresponding to the target attribute area, and use the instance guidance feature corresponding to the target attribute area as the second instance guidance feature; compare the first instance guidance feature with the second instance guidance feature An addition operation is performed to obtain an instance guiding feature of the reference feature to the missing feature.

The instance-guided feature of the reference feature to the missing feature can be obtained by the following formula:

是待编辑人脸图像中目标属性区域对应的掩码图，或者进行腐蚀和膨胀处理后的掩码图，F_r表示参考特征，

表示对参考特征F_r和注意力图F_m进行融合得到的参考注意特征。上述公式中的前一部分，即

表示第一实例指导特征，上述公式中的后一部分即

表示第二实例指导特征，计算这两部分之和，得到参考特征对缺失特征的实例指导特征。in,

is the mask map corresponding to the target attribute area in the face image to be edited, or the mask map after corrosion and expansion processing, F _r represents the reference feature,

Represents the reference attention feature obtained by fusing the reference feature _Fr and the attention map _Fm . The first part of the above formula, i.e.

represents the guiding feature of the first instance, the latter part of the above formula is

Represents the second instance guidance feature, calculates the sum of these two parts, and obtains the instance guidance feature of the reference feature to the missing feature.

Step 160: Perform image decoding on the fusion feature through a decoder to obtain a target image corresponding to the face image to be edited and the reference image.

The decoder can be an image decoder based on a neural network. After training, the network parameters have been determined, and the image features can be directly decoded into a complete image.

The decoder performs image decoding on the fusion features to obtain the target image, that is, the image in which the features of the region corresponding to the target attribute region in the reference image are migrated to the missing image.

In the image editing method provided by the embodiment of the present application, the face image to be edited is processed into a missing image missing the target attribute area according to the mask map corresponding to the target attribute area in the face image to be edited, and the The missing image is encoded by the image to obtain the missing feature corresponding to the missing image, and the reference image is encoded by the second encoder to obtain the reference feature corresponding to the reference image. The image decoder decodes the fusion features to obtain the target image. Since only the mask image corresponding to the face image to be edited and the target attribute area needs to be obtained, the features corresponding to the target attribute area in the reference image can be migrated to the person to be edited. In the target attribute area of the face image, the features of each area can be migrated, which solves the defect that ExGANs in the prior art can only edit the eye area, and the reference image can be an image with a different identity from the face image to be edited, thereby improving The diversity of face attribute editing is improved, and the target attribute area does not need to be decoupled from other areas, avoiding the impact on irrelevant areas outside the target attribute area.

Based on the above technical solution, the method further includes: when training the first encoder, the second encoder, the attention model and the decoder, respectively determining the face image to be edited and the target image The value of the perceptual loss function and the value of the style loss function, and determine the value of the contextual loss function of the target image and the reference image; according to the value of the perceptual loss function, the value of the style loss function and the value of the contextual loss function value, the network parameters of the first encoder, the second encoder, the attention model and the decoder are adjusted so that the values of the perceptual loss function, the style loss function and the contextual loss function converge.

The perceptual loss function measures the similarity of high-dimensional features (such as overall spatial structure) between two images, while the style loss function measures the similarity of style features (such as color) between two images. In the process of training the first encoder, the second encoder, the attention model and the decoder, continuously reducing the value of the perceptual loss function and the value of the style loss function can make the target image have the same characteristics as the face image to be edited. Consistent style features that look natural and authentic.

The contextual loss function is used to measure the similarity between asymmetric images. In this scheme, the loss ensures that the target attribute in the generated target image is consistent with the target attribute shape of the reference image.

In an embodiment of the present application, the perceptual loss function and the style loss function are respectively expressed as follows:

表示待编辑人脸图像中目标属性区域对应的掩码图，C_l、H_l和W_l分别表示通道数、高度和宽度，‖·‖₁表示1范数，

表示预设网络第l层的特征图，G_l(·)＝Φ_l(·)^TΦ_l(·)表示Gram矩阵。Among them, l _perc is the perceptual loss function, l _style is the style loss function, I _g represents the target image, I _s represents the face image to be edited, I _c represents the missing image,

represents the mask map corresponding to the target attribute area in the face image to be edited, C _l , H _l and W _l represent the number of channels, height and width respectively, ‖·‖ ₁ represents the 1 norm,

Represents the feature map of the first layer of the preset network, G _l (·)=Φ _l (·) ^T Φ _l (·) represents the Gram matrix.

表示待编辑人脸图像中目标属性区域对应的掩码图，或者是对待编辑人脸图像中目标属性区域对应的掩码图进行腐蚀和膨胀处理后的掩码图。在进行训练时，待编辑人脸图像为训练样本。预设网络可以是VGG-19网络，当然，也可以是其他网络。VGG-19网络是在ImageNet上预训练的卷积神经网络。

Indicates the mask map corresponding to the target attribute region in the face image to be edited, or the mask map corresponding to the target attribute region in the face image to be edited after etching and dilation processing. During training, the face images to be edited are training samples. The preset network can be the VGG-19 network, of course, can also be other networks. The VGG-19 network is a convolutional neural network pretrained on ImageNet.

When calculating the value of the perceptual loss function according to the above formula, the target image I _g generated for each sample is input into the VGG-19 network, the feature map Φ _l (I _g ) of the lth layer is obtained, and the original image of the sample is to be Edit the face image I _s and input it into the VGG-19 network to obtain the feature map Φ _l (I _s ) of the lth layer, and then calculate the feature map Φ _l (I _g ) of the target image and the feature map Φ of the face image to be edited. _l (I _s ) is the 1 norm of the difference, divide the calculation result by the product of the number of channels, height and width to obtain the value of the perceptual loss function corresponding to the feature map of the lth layer, which corresponds to the feature map of each layer of the VGG-19 network. The value of the perceptual loss function is accumulated to obtain the value of the perceptual loss function corresponding to one training.

的乘积输入VGG-19网络，获取第一层的特征图

并计算该特征图的Gram矩阵

将缺失图像I_c输入VGG-19网络，获取第一层的特征图Φ_l(I_c)，计算该特征图的Gram矩阵G_l(I_c)，计算两个Gram矩阵之差即

除以通道数、高度和宽度的乘积的结果，求取该结果的1范数，并除以通道数与通道数的乘积，得到第l层特征图对应的风格损失函数的值，对VGG-19网络每层特征图对应的风格损失函数的值进行累加，得到一次训练对应的风格损失函数的值。When calculating the value of the perceptual loss function according to the above formula, the target image I _g generated by the sample and the mask map corresponding to the target attribute area in the face image to be edited

The product of is input into the VGG-19 network to obtain the feature map of the first layer

and calculate the Gram matrix of this feature map

Input the missing image I _c into the VGG-19 network, obtain the feature map Φ _l (I _c ) of the first layer, calculate the Gram matrix G _l (I _c ) of the feature map, and calculate the difference between the two Gram matrices namely

Divide by the result of the product of the number of channels, height and width, obtain the 1 norm of the result, and divide by the product of the number of channels and the number of channels to obtain the value of the style loss function corresponding to the feature map of the lth layer. For VGG- 19 The value of the style loss function corresponding to the feature map of each layer of the network is accumulated to obtain the value of the style loss function corresponding to one training.

The above perceptual loss function can make the target image obtained by the trained network and the face image to be edited have high-dimensional similarity, and the above style loss function can make the target image obtained by the trained network and the face image to be edited. Having similarity between style features means that the target image can retain style features such as pose or skin color in the face image to be edited.

In an embodiment of the present application, the contextual loss function is expressed as follows:

表示待编辑人脸图像中目标属性区域对应的掩码图，I_r表示参考图像，

表示参考图像中与所述目标属性区域对应区域的掩码图，

表示预设网络第l层的特征图，CX(x,y)表示输入图像x和结果图像y之间的Contextual相似度，计算公式如下：Among them, l _cx represents the contextual loss function, I _g represents the target image,

represents the mask map corresponding to the target attribute area in the face image to be edited, I _r represents the reference image,

represents the mask map of the region corresponding to the target attribute region in the reference image,

Represents the feature map of the first layer of the preset network, CX(x, y) represents the contextual similarity between the input image x and the result image y, and the calculation formula is as follows:

和

N表示特征的个数，max_i CX_ij表示对于每个特征y_j在集合X中找到与y_j相似度最高的特征x_i并计算y_j与x_i之间的相似度。The input image x and the result image y can be represented as collections of features, respectively:

and

N represents the number of features, and max _i CX _ij represents that for each feature y _{j, find the feature xi with the highest similarity with y j in the} set X and calculate the similarity between y _j and _xi .

Wherein, the preset network may be the VGG-19 network, and of course, may also be other networks.

输入VGG-19网络，获取第l层的特征图

并将参考图像与参考图像中与目标属性区域对应区域的掩码图的乘积

输入VGG-19网络，获取第l层的特征图

计算特征图

与

之间的Contextual相似度，在上述计算Contextual相似度的公式中，

并对

与

之间的Contextual相似度求对数，得到Contextual损失函数的值。通过Contextual损失函数可以使得训练完成的网络得到的目标图像中的目标属性与参考图像中的目标属性的形状一致，可以保证将参考图像中的目标属性较好的迁移到待编辑人脸图像的目标属性区域中。When calculating the value of the contextual loss function, the product of the target image and the mask map corresponding to the target attribute area in the face image to be edited

Enter the VGG-19 network to obtain the feature map of the lth layer

and multiply the reference image with the mask map of the region corresponding to the target attribute region in the reference image

Enter the VGG-19 network to obtain the feature map of the lth layer

Calculate the feature map

and

The Contextual similarity between , in the above formula for calculating the Contextual similarity,

and to

and

Take the logarithm of the Contextual similarity between them to get the value of the Contextual loss function. Through the contextual loss function, the shape of the target attribute in the target image obtained by the trained network can be consistent with the shape of the target attribute in the reference image, which can ensure that the target attribute in the reference image is better migrated to the target of the face image to be edited. in the properties area.

Embodiment 2

An image editing apparatus provided by this embodiment, as shown in FIG. 3 , the image editing apparatus 300 includes:

An image acquisition module 310, configured to acquire a face image to be edited and a mask map corresponding to a target attribute region in the face image to be edited, and acquire a reference image;

An image missing processing module 320, configured to process the to-be-edited face image as a missing image missing the target attribute area according to the mask map;

A first encoding module 330, configured to perform image encoding on the missing image by the first encoder to obtain the missing feature corresponding to the missing image;

A second encoding module 340, configured to perform image encoding on the reference image through a second encoder to obtain a reference feature corresponding to the reference image;

A feature fusion module 350, configured to fuse the missing feature and the reference feature through an attention model according to the mask map to obtain a fused feature;

The decoding module 360 is configured to perform image decoding on the fusion feature through a decoder to obtain a target image corresponding to the face image to be edited and the reference image.

Optionally, the feature fusion module includes:

an attention map generation unit, used to generate an attention map corresponding to the missing feature;

a first feature fusion unit, configured to fuse the missing feature and the attention map to obtain a self-attention feature corresponding to the missing feature;

a feature generation unit, configured to generate an instance guidance feature of the reference feature to the missing feature according to the mask map, the attention map and the reference feature;

A second feature fusion unit, configured to fuse the self-attention feature and the instance guidance feature to obtain a fusion feature of the missing feature and the reference feature.

Optionally, the feature generation unit is specifically used for:

fusing the reference feature and the attention map to obtain the reference attention feature;

Multiply the mask map and the reference attention feature to obtain the instance guidance feature corresponding to the area outside the target attribute area, and use the instance guidance feature corresponding to the area outside the target attribute area as the first instance guiding features;

Calculate the difference between the 1 matrix and the mask map, obtain the inverse mask map of the mask map, and multiply the inverse mask map and the reference feature to obtain the instance corresponding to the target attribute area Guidance feature, taking the instance guidance feature corresponding to the target attribute region as the second instance guidance feature;

The first instance guidance feature and the second instance guidance feature are added to obtain an instance guidance feature of the reference feature for the missing feature.

Optionally, the device further includes:

The loss value determination module is used for respectively determining the value of the perceptual loss function of the face image to be edited and the target image and the the value of the style loss function, and determine the value of the contextual loss function of the target image and the reference image;

A network parameter adjustment module, configured to adjust the network parameters of the first encoder, the second encoder, the attention model and the decoder according to the value of the perceptual loss function, the value of the style loss function and the value of the contextual loss function Adjustments are made such that the values of the perceptual loss function, the style loss function, and the contextual loss function converge.

Optionally, the perceptual loss function and the style loss function are respectively expressed as follows:

表示待编辑人脸图像中目标属性区域对应的掩码图，C_l、H_l和W_l分别表示通道数、高度和宽度，‖·‖₁表示1范数，

表示预设网络第l层的特征图，G_l(·)＝Φ_l(·)^TΦ_l(·)表示Gram矩阵。Among them, l _perc is the perceptual loss function, l _style is the style loss function, I _g represents the target image, I _s represents the face image to be edited, I _c represents the missing image,

represents the mask map corresponding to the target attribute area in the face image to be edited, C _l , H _l and W _l represent the number of channels, height and width respectively, ‖·‖ ₁ represents the 1 norm,

Represents the feature map of the first layer of the preset network, G _l (·)=Φ _l (·) ^T Φ _l (·) represents the Gram matrix.

Optionally, the contextual loss function is expressed as follows:

表示待编辑人脸图像中目标属性区域对应的掩码图，I_r表示参考图像，

表示参考图像中与所述目标属性区域对应区域的掩码图，

表示预设网络第l层的特征图，CX(x,y)表示输入图像x和结果图像y之间的Contextual相似度，计算公式如下：Among them, l _cx represents the contextual loss function, I _g represents the target image,

represents the mask map corresponding to the target attribute area in the face image to be edited, I _r represents the reference image,

represents the mask map of the region corresponding to the target attribute region in the reference image,

Represents the feature map of the first layer of the preset network, CX(x, y) represents the contextual similarity between the input image x and the result image y, and the calculation formula is as follows:

和

N表示特征的个数，max_i CX_ij表示对于每个特征y_j在集合X中找到与y_j相似度最高的特征x_i并计算y_j与x_i之间的相似度。The input image x and the result image y can be represented as collections of features, respectively:

and

N represents the number of features, and max _i CX _ij represents that for each feature y _{j, find the feature xi with the highest similarity with y j in the} set X and calculate the similarity between y _j and _xi .

The image editing apparatus provided in this embodiment of the present application is used to implement the steps of the image editing method described in the first embodiment of the present application. For the specific implementation of each module of the apparatus, refer to the corresponding steps, which will not be repeated here.

In the image editing device provided by the embodiment of the present application, the image acquisition module acquires the mask image corresponding to the face image to be edited and the target attribute region in the face image to be edited, and acquires the reference image, and the image matting module obtains the mask image according to the mask image. The face image to be edited is processed as a missing image missing the target attribute area, the first encoding module performs image encoding on the missing image through the first encoder to obtain the missing feature corresponding to the missing image, and the second encoding module uses the second encoder to refer to the reference. The image is encoded by the image to obtain the reference feature corresponding to the reference image. The feature fusion module fuses the missing feature and the reference feature into the fusion feature through the attention model according to the mask map. The decoding module decodes the fusion feature through the decoder to obtain the target image. , because only the mask map corresponding to the face image to be edited and the target attribute area needs to be obtained, the features corresponding to the target attribute area in the reference image can be migrated to the target attribute area of the face image to be edited, so that the migration can be performed. The features of each area solve the defect that ExGANs in the prior art can only edit the eye area, and the reference image can be an image with a different identity from the face image to be edited, thereby improving the diversity of face attribute editing, and it is not The target attribute area needs to be decoupled from other areas to avoid the influence on irrelevant areas outside the target attribute area.

Embodiment 3

This embodiment of the present application further provides an electronic device. As shown in FIG. 4 , the electronic device 400 may include one or more processors 410 and one or more memories 420 connected to the processors 410 . The electronic device 400 may also include an input interface 430 and an output interface 440 for communicating with another device or system. Program code executed by processor 410 may be stored in memory 420 .

The processor 410 in the electronic device 400 invokes the program code stored in the memory 420 to execute the image editing method in the above-mentioned embodiment.

The above-mentioned elements in the above-mentioned electronic device may be connected to each other by a bus such as one or any combination of a data bus, an address bus, a control bus, an expansion bus and a local bus.

The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, implements the steps of the image editing method described in the first embodiment of the present application.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments may be referred to each other. As for the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for related parts.

The image editing method, device, electronic device, and storage medium provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described with specific examples. The descriptions of the above embodiments are only used for In order to help understand the method of the present application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope. In summary, this specification The content should not be construed as a limitation on this application.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Claims (10)

1. An image editing method, comprising:

acquiring a face image to be edited and a mask image corresponding to a target attribute region in the face image to be edited, and acquiring a reference image;

processing the face image to be edited into a missing image missing the target attribute region according to the mask image;

carrying out image coding on the missing image through a first encoder to obtain the missing characteristics corresponding to the missing image;

carrying out image coding on the reference image through a second encoder to obtain reference characteristics corresponding to the reference image;

fusing the missing features and the reference features through an attention model according to the mask image to obtain fused features;

and decoding the image of the fusion characteristic through a decoder to obtain the target image corresponding to the face image to be edited and the reference image.

2. The method according to claim 1, wherein the fusing the missing feature and the reference feature by the attention model according to the mask map to obtain a fused feature comprises:

generating an attention map corresponding to the missing feature;

fusing the missing feature and the attention map to obtain a self-attention feature corresponding to the missing feature;

generating an example guide feature of the reference feature to the missing feature according to the mask map, the attention map and the reference feature;

and fusing the self-attention feature and the example guidance feature to obtain a fused feature of the missing feature and the reference feature.

3. The method of claim 2, wherein the generating an instance guideline feature of the reference feature for the missing feature from the mask map, an attention map, and a reference feature comprises:

fusing the reference feature and the attention map to obtain a reference attention feature;

multiplying the mask image and the reference attention feature to obtain an example guidance feature corresponding to a region outside the target attribute region, and taking the example guidance feature corresponding to the region outside the target attribute region as a first example guidance feature;

calculating the difference between the matrix 1 and the mask map to obtain a mask map of the mask map, multiplying the mask map and the reference feature to obtain an example guidance feature corresponding to the target attribute region, and taking the example guidance feature corresponding to the target attribute region as a second example guidance feature;

and adding the first example guide characteristic and the second example guide characteristic to obtain the example guide characteristic of the reference characteristic to the missing characteristic.

4. The method according to any one of claims 1-3, further comprising:

when the first encoder, the second encoder, the attention model and the decoder are trained, respectively determining a value of a perception loss function and a value of a style loss function of the face image to be edited and the target image, and determining a value of a context loss function of the target image and the reference image;

and adjusting the network parameters of the first encoder, the second encoder, the attention model and the decoder according to the value of the perceptual loss function, the value of the style loss function and the value of the context loss function, so that the value of the perceptual loss function, the value of the style loss function and the value of the context loss function are converged.

5. The method of claim 4, wherein the perceptual and stylistic loss functions are each expressed as follows:

wherein l_percAs a function of perceptual loss,/_styleAs a style loss function, I_gRepresenting the target image, I_sRepresenting a face image to be edited, I_cA missing image is represented by a number of images,

representing a mask map corresponding to the target attribute region in the face image to be edited, C_l、H_lAnd W_lRepresenting the number, height and width of the channels, |₁The expression is given in the 1 norm,

characteristic diagram, G, representing the l-th layer of the default network_l(·)＝Φ_l(·)^TΦ_l(. cndot.) denotes a Gram matrix.

6. The method of claim 4, wherein the context loss function is represented as follows:

wherein l_cxRepresenting a Contextual loss function, I_gA representation of the target image is shown,

representing a mask map corresponding to the target attribute region in the face image to be edited, I_rWhich represents a reference image, is shown,

a mask map representing a region in the reference image corresponding to the target attribute region,

the characteristic diagram of the l-th layer of the preset network is represented, CX (x, y) represents the context similarity between the input image x and the result image y, and the calculation formula is as follows:

the input image x and the resulting image y may be represented as a set of features, respectively:

and

n denotes the number of features, max_iCX_ijRepresentation for each feature y_jFind y in the set X_jFeature x with highest similarity_iAnd calculate y_jAnd x_iThe similarity between them.

7. The method according to claim 1, wherein the processing the face image to be edited into a missing image missing the target attribute region according to the mask map comprises:

carrying out corrosion and expansion treatment on the mask image to obtain a treated mask image;

and removing the target attribute region in the face image to be edited according to the processed mask image to obtain a missing image.

8. An image editing apparatus characterized by comprising:

the image acquisition module is used for acquiring a face image to be edited and a mask image corresponding to a target attribute region in the face image to be edited and acquiring a reference image;

the image missing processing module is used for processing the face image to be edited into a missing image missing the target attribute region according to the mask image;

the first coding module is used for carrying out image coding on the missing image through a first coder to obtain the missing characteristic corresponding to the missing image;

the second coding module is used for carrying out image coding on the reference image through a second coder to obtain the reference characteristics corresponding to the reference image;

the feature fusion module is used for fusing the missing features and the reference features through an attention model according to the mask image to obtain fusion features;

and the decoding module is used for carrying out image decoding on the fusion characteristics through a decoder to obtain the face image to be edited and a target image corresponding to the reference image.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the image editing method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the steps of the image editing method as claimed in any one of claims 1 to 7.

Images