CN117036185A - Image processing method, device, electronic equipment and storage medium - Google Patents

Abstract

The application discloses an image processing method, an image processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring an image to be processed with unbalanced illumination; encoding an image to be processed based on a preset encoder neural network to obtain a plurality of groups of feature latent codes belonging to different scales, wherein the preset encoder neural network adopts a feature pyramid structure based on a channel attention mechanism and a space attention mechanism, and each scale corresponds to different levels in the feature pyramid structure; inputting each group of characteristic latent codes into a pre-trained StyleGAN generator, acquiring a target image with balanced illumination according to the output result of the StyleGAN generator, and coding the image to be processed into a plurality of groups of characteristic latent codes conforming to a characteristic pyramid structure by introducing a channel attention mechanism and a space attention mechanism, so that the StyleGAN generator directly completes image reconstruction according to the characteristic latent codes, thereby improving the efficiency and accuracy of carrying out illumination equalization processing on the image.

Description

An image processing method, device, electronic equipment and storage medium

Technical field

The present application relates to the field of computer technology, and more specifically, to an image processing method, device, electronic device and storage medium.

Background technique

When there is uneven illumination in the image, the image will have unbalanced illumination such as shadows and highlights, causing the image to display abnormally (such as a "yin-yang face", etc.). At present, when solving the problem of uneven illumination in images, the main solutions used are:

Manual processing based on tools such as photoshop. This method can effectively process different images, but it requires manual operation and is inefficient.

Based on traditional image processing solutions, for example, background modeling based on filtering algorithms is first performed, and then operations at the pixel gray level are performed to achieve illumination equalization of the image. Traditional technology cannot adjust the brightness distribution, contrast and saturation of the image in a targeted manner. It is difficult to effectively improve the dark areas in the image, and it is difficult to prevent the originally bright areas from being overexposed, which ultimately leads to the loss of information in the processed image and makes it unnatural. .

Therefore, how to perform illumination equalization processing on images efficiently and accurately is a technical problem that needs to be solved.

It should be noted that the information disclosed in the above background section is only used to enhance understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to those of ordinary skill in the art.

Contents of the invention

The embodiment of this application proposes an image processing method, device, electronic device and storage medium, which introduces a channel attention mechanism and a spatial attention mechanism to encode the image to be processed into multiple sets of feature latent codes that conform to the feature pyramid structure, making StyleGAN The generator directly completes image reconstruction based on the feature latent code to achieve efficient and accurate illumination equalization of the image.

In a first aspect, an image processing method is provided. The method includes: acquiring an image to be processed with unbalanced illumination; encoding the image to be processed based on a preset encoder neural network to obtain multiple sets of feature latent features belonging to different scales. code, wherein the preset encoder neural network adopts a feature pyramid structure based on a channel attention mechanism and a spatial attention mechanism, and each scale corresponds to a different level in the feature pyramid structure; each group of features The latent code is input into the pre-trained StyleGAN generator, and the illumination-balanced target image is obtained according to the output result of the StyleGAN generator.

In a second aspect, an image processing device is provided. The device includes: an acquisition module for acquiring an image to be processed with unbalanced illumination; and an encoding module for encoding the image to be processed based on a preset encoder neural network. , obtain multiple sets of feature latent codes belonging to different scales, wherein the preset encoder neural network adopts a feature pyramid structure based on a channel attention mechanism and a spatial attention mechanism, and each of the scales corresponds to the feature pyramid structure. Different levels; a generation module, used to input each group of feature latent codes into a pre-trained StyleGAN generator, and obtain a target image with balanced lighting according to the output result of the StyleGAN generator.

In a third aspect, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the first aspect via executing the executable instructions. The image processing method described.

A fourth aspect provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the image processing method described in the first aspect is implemented.

By applying the above technical solution, the image to be processed with unbalanced illumination is first obtained, and then the image to be processed is encoded based on the preset encoder neural network to obtain multiple sets of feature latent codes belonging to different scales, and finally each set of feature latent codes is input into the pre-processed image. The trained StyleGAN generator obtains a well-illuminated target image based on the output of the StyleGAN generator. The preset encoder neural network adopts a feature pyramid structure based on the channel attention mechanism and the spatial attention mechanism. Each scale corresponds to the feature pyramid structure. At different levels, by introducing the channel attention mechanism and the spatial attention mechanism, the image to be processed is encoded into multiple sets of feature latent codes that conform to the feature pyramid structure, so that the StyleGAN generator can directly complete image reconstruction based on the feature latent codes, thereby improving It improves the efficiency and accuracy of illumination equalization processing of images.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 shows a schematic flow chart of an image processing method proposed by an embodiment of the present invention;

Figure 2 shows a schematic flow chart of training and generating a preset encoder neural network in an embodiment of the present invention;

Figure 3 shows a schematic flow chart of encoding an image to be processed in an embodiment of the present invention;

Figure 4 shows a schematic diagram of the principle of an image processing method proposed by another embodiment of the present invention;

Figure 5 shows a schematic structural diagram of an image processing device proposed by an embodiment of the present invention;

FIG. 6 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

It should be noted that those skilled in the art will easily come up with other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary technical means in the technical field that are not disclosed in this application. . It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the claims.

It is to be understood that the present application is not limited to the precise structures described below and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

The present application may be used in numerous general purpose or special purpose computing device environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.

The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

An embodiment of the present application provides an image processing method, as shown in Figure 1. The method includes the following steps:

Step S101: Obtain an image to be processed with unbalanced illumination.

Illumination balance means that the difference in brightness of each part of the image is within a preset range. Parameters can be set according to the actual application scenario to determine whether the image is illumination balanced, such as the average or standard deviation of each pixel in the image. Correspondingly, the image to be processed with unbalanced illumination is an image in which the brightness difference of each part of the image exceeds the preset range. The image to be processed can be an image captured in real time, an image obtained from a local storage path, or an image obtained from cloud storage or other servers. The display objects in the image to be processed can be human faces, animals, landscapes, avatars, buildings, etc., and the format of the image to be processed can also be any image format, which is not limited in the embodiments of the present application.

It can be understood that in some embodiments, when the image to be processed with uneven illumination is an image uploaded or selected by the user, even if the image does not have the problem of uneven illumination, after the image is obtained, the image is still used as the illumination The unbalanced image to be processed is processed, and the embodiment of the present application does not judge whether the image to be processed actually has illumination imbalance.

In some embodiments of the present application, the image to be processed is a face image, and obtaining the image to be processed with unbalanced illumination includes:

Obtain raw images including faces and uneven lighting;

Perform face detection on the original image to obtain the face image to be processed;

Perform face alignment processing on the face image to be processed and normalize it to a preset size to obtain the image to be processed.

In this embodiment, the image to be processed is a face image with unbalanced illumination, which is obtained by preprocessing the original image. Specifically, the original image including the face and unbalanced illumination is first obtained by photographing the face or from the storage path specified by the user, and then performs face detection on the original image based on the preset face detection algorithm to obtain the person to be processed. face image, then perform face alignment processing on the face image to be processed, and finally normalize the aligned face image to a preset size. For example, the preset size can be 256*256 to obtain the image to be processed.

Among them, the preset face detection algorithm can be any one of the algorithms including Haar cascade+opencv, HOG+Dlib, CNN+Dlib, SSD, MTCNN, etc. Face alignment processing includes alignment based on feature points and alignment based on template matching. Among them, alignment based on feature points extracts feature points of the face in different images to determine the position of the face in the image. Angle, size and other parameters, and then align. The alignment method based on template matching extracts face templates from different images to determine the position, angle, size and other parameters of the face in the image, and then performs alignment.

After performing face detection on the original image to obtain the face image to be processed, face alignment and normalization processing can make the preset encoder neural network more accurate and efficient in encoding the image to be processed, and then perform lighting more efficiently. Equalization processing.

Step S102: Encode the image to be processed based on a preset encoder neural network to obtain multiple sets of feature latent codes belonging to different scales. The preset encoder neural network uses a channel-based attention mechanism and spatial attention. The feature pyramid structure of the mechanism, each of the scales corresponds to different levels in the feature pyramid structure.

StyleGAN is a style-based generative adversarial network. In this embodiment, the preset encoder neural network and StyleGAN generator are used to form a pixel2style2pixel (pSp for short) framework for image processing. The first pixel refers to the input image to be processed, and style is refers to the feature latent code, and the second pixel refers to the output target image, that is, the image to be processed is first converted into a feature latent code, and then the feature latent code is generated into the target image. Since the StyleGAN generator needs to be used later to generate the target image, and the input of the StyleGAN generator needs to be the feature latent code of the image, the image to be processed is first converted into the feature latent code based on the preset encoder neural network.

The feature latent code is a kind of feature vector, which can also be called a feature map. Specifically, it can be a multi-dimensional vector, and each value in the vector is in the range of [-1,1]. For example, it can be 18*512 A vector where each value in the vector is in the range [-1,1]. The feature latent code can also be understood as the characteristics of the image extracted from the image based on the neural network. The feature latent code can represent the image. When the feature latent code is determined, the image generated based on the feature latent code is also determined, and in another case From one perspective, the feature latent code can also be understood as the vector output after the image passes through the convolutional layer in the neural network.

The default encoder neural network adopts a feature pyramid structure, which includes different levels. During the encoding process, channel attention mechanisms and spatial attention mechanisms are introduced between different levels. The channel attention mechanism can display the correlation between different channel feature maps when modeling neural networks, automatically obtain the importance of each feature channel in network learning, and finally assign different weight coefficients to each channel. The spatial attention mechanism transforms the spatial information in the image into another space through the spatial transformation module and retains the key information. It generates a weight mask for each image area and then weights the output, thereby enhancing the specific target area of interest while weakening the Irrelevant background areas.

Input the image to be processed into the preset encoder neural network. The preset encoder neural network encodes the image to be processed based on the channel attention mechanism and spatial attention mechanism, and maps it into multiple sets of feature latent codes belonging to different scales. Each scale is respectively Corresponding to different levels in the feature pyramid structure, different scales can represent the details of the image at different granularities. For example, if the image to be processed is a face image, three scales of large, medium and small can be output, corresponding to facial contours, postures and more. Detailed hair color, etc. Correspondingly, the feature pyramid structure also includes three levels.

A set of feature latent codes belonging to the same scale can include multiple layers of feature latent codes. For example, if it includes 18 layers of 512-dimensional feature latent codes, the small scale can correspond to layers 0-2, and the medium scale can correspond to layers 3-6. , large scale can correspond to layers 7-18.

Step S103: Input each set of feature latent codes into a pre-trained StyleGAN generator, and obtain a target image with balanced illumination based on the output result of the StyleGAN generator.

The StyleGAN generator is pre-trained. After obtaining multiple sets of feature latent codes, each set of feature latent codes is input into the StyleGAN generator. The StyleGAN generator directly completes image reconstruction based on the feature latent codes and outputs a target image with balanced lighting.

The image processing method provided by the embodiment of the present application first obtains the image to be processed with unbalanced illumination, and then encodes the image to be processed based on the preset encoder neural network to obtain multiple groups of feature latent codes belonging to different scales. Finally, each group of features is The latent code is input to the pre-trained StyleGAN generator, and the illumination-balanced target image is obtained according to the output result of the StyleGAN generator. The preset encoder neural network adopts a feature pyramid structure based on the channel attention mechanism and the spatial attention mechanism, and each scale corresponds to At different levels in the feature pyramid structure, by introducing the channel attention mechanism and the spatial attention mechanism, the image to be processed is encoded into multiple sets of feature latent codes that conform to the feature pyramid structure, allowing the StyleGAN generator to directly complete image reconstruction based on the feature latent codes. This improves the efficiency and accuracy of illumination equalization processing of images.

Based on any embodiment of the present application, before encoding the image to be processed based on the preset encoder neural network to obtain multiple sets of feature latent codes belonging to different scales, as shown in Figure 2, the method also includes Following steps:

Step S21: Acquire multiple groups of sample images, each group of sample images including a first image of a sample object under unbalanced illumination and a second image under balanced illumination.

In this embodiment, the preset initial encoder neural network is trained. After the training is completed, the preset encoder neural network is obtained. Specifically, multiple groups of sample images are first obtained. Those skilled in the art can use different numbers of sample images according to actual needs, for example, 10,000 groups. Each group of sample images corresponds to a sample object, including a first image of a sample object under unbalanced illumination and a second image under balanced illumination, where the first image is used as a preset initial encoder neural network As input, the second image is used to compare with the output of the preset initial encoder neural network to determine the training effect.

In some embodiments of this application, the sample object is a human face, and the acquisition of multiple groups of sample images includes:

Get multiple original sample images:

Perform face detection on the original sample image to obtain a sample face image;

Face alignment processing is performed on each of the sample face images and normalized to a preset size to obtain multiple groups of the sample images.

In this embodiment, the sample object is a human face. By processing the original sample image, multiple groups of sample images are obtained. Specifically, face detection is first performed on the original sample image based on a preset face detection algorithm to obtain the sample face image. Then face alignment processing is performed on each sample face image and normalized to a preset size. For example, the preset size is 256*256 size, and multiple sets of sample images are obtained.

Among them, the preset face detection algorithm can be any one of the algorithms including Haar cascade+opencv, HOG+Dlib, CNN+Dlib, SSD, MTCNN, etc. Face alignment processing includes alignment based on feature points and alignment based on template matching. Among them, alignment based on feature points extracts feature points of the face in different images to determine the position of the face in the image. Angle, size and other parameters, and then align. The alignment method based on template matching extracts face templates from different images to determine the position, angle, size and other parameters of the face in the image, and then performs alignment.

After face detection is performed on the original sample image to obtain the sample face image, face alignment and normalization processing can be performed to train the preset initial encoder neural network more efficiently and improve the obtained preset encoder neural network. Network accuracy.

It can be understood that, in order to ensure accuracy, if the sample object is a human face, the image to be processed is also the face image after face detection, face alignment processing and normalization.

Step S22: Train the preset initial encoder neural network using the feature pyramid structure based on the sample image, and determine the loss function during the training process.

The preset initial encoder neural network also uses a feature pyramid structure based on the channel attention mechanism and the spatial attention mechanism. The preset initial encoder neural network is trained based on multiple sets of sample images, and the loss function during the training process is determined. Specifically , the loss function can be determined by the difference between the output result of the preset initial encoder neural network and the second image in the sample image.

In some embodiments of the present application, training a preset initial encoder neural network using the feature pyramid structure based on the sample image, and determining the loss function during the training process includes:

Input the first image into the preset initial encoder neural network, encode the first image based on the preset initial encoder neural network, and obtain multiple sets of sample feature latent codes corresponding to each of the scales;

Input the sample feature latent code of each group into the StyleGAN generator, and obtain the third image according to the output result of the StyleGAN generator;

The loss function is determined based on the difference between the third image and the second image.

In this embodiment, the preset initial encoder neural network is connected to the pre-trained StyleGAN generator in advance, and the first image with unbalanced illumination is first input into the preset initial encoder neural network, and the preset initial encoder neural network performs the processing. Encoding process, obtain multiple sets of sample feature latent codes corresponding to each scale, and then input the multiple sets of sample feature latent codes into the StyleGAN generator, and the StyleGAN generator generates a third image corresponding to the sample feature latent code, and finally The third image is compared with the second image, and the loss function is determined based on the difference between the two, so that the loss function more truly reflects the training effect of the preset initial encoder neural network and improves the accuracy of the loss function.

In some embodiments of the present application, determining the loss function based on the difference between the third image and the second image includes:

Determine pixel point loss, perceptual loss, identity information loss, saturation loss, contrast loss and brightness loss based on the difference;

The loss function is generated by performing a weighted sum of the pixel point loss, the perceptual loss, the identity information loss, the saturation loss, the contrast loss and the brightness loss according to a preset weight coefficient combination.

In this embodiment, the loss function is a composite loss function determined based on multiple losses. Specifically, pixel loss, perceptual loss, identity information loss, saturation loss, Contrast loss and brightness loss, where the pixel loss represents the loss at the pixel level, which can be specifically the mean square error between the pixel vector of the third image and the pixel vector of the second image. Perceptual loss is a loss consistent with the human perception process. The identity information loss specifically uses a pre-trained face recognition model to calculate the cosine similarity between the third image and the second image on the identity information vector. The saturation loss, contrast loss and brightness loss may be the mean square error on the saturation vector, contrast vector and brightness vector respectively between the third image and the second image.

The weight coefficients corresponding to each loss are preset to form a preset weight coefficient combination, and each loss is weighted and summed according to the preset weight coefficient combination to obtain the loss function. In this way, multiple losses are combined during the training process to further ensure the accuracy of the preset encoder neural network obtained by training, and due to the consideration of saturation loss, contrast loss and brightness loss, the focus on the target illumination equalization effect is consistent. With a unique inspection dimension, it can converge to the target illumination balance effect faster during model training.

It should be noted that the solution in the above embodiment is only a specific implementation solution proposed by this application. Those skilled in the art can flexibly use other types of losses to construct the loss function, which does not affect the protection scope of this application.

In the specific application scenario of this application, if the pixel loss is L ₂ (x, x), the pixel vector of the second image is x′, and the pixel vector of the third image is pSp(x), then L ₂ ( x, x)=||x-pSp(x)|| ² , where pSp represents the aforementioned pSp framework.

If the perceptual loss is L _LPIPS (x, x), the perceptual vector of the second image is F(x′), and the perceptual vector of the third image is F(pSp(x)), then L _LPIPS (x, x′)= ||F(x′)-F(pSp(x))|| ² .

If the identity information loss is L _ID (x, x′), the identity information vector of the second image is R(x′), and the identity information vector of the third image is R(pSp(x)), then L _ID (x, x)=1-(R(x), R(pSp(x))).

If the saturation loss is L _SAT (x, x′), the saturation vector of the second image is S(x′), and the saturation vector of the third image is S(pSp(x)), then L _SAT (x, x)=||S(x)-S(pSp(x))|| ² .

If the contrast loss is L _CON (x, x′), the contrast vector of the second image is C(x′), and the contrast vector of the third image is C(pSp(x)), then L _CON (x, x′) =||C(x′)-C(pSp(x))|| ² .

If the brightness loss is L _LUM (x, x′), the brightness vector of the second image is L(x′), and the brightness vector of the third image is L(pSp(x)), then L _LUM (x, x′) =||L(x′)-L(pSp(x))|| ² .

If the loss function is L(x, x′), the weight coefficients corresponding to pixel loss, perceptual loss, identity information loss, saturation loss, contrast loss and brightness loss are λ ₁ , λ ₂ , λ ₃ , λ ₄ respectively. , λ ₅ , λ ₆ , then:

L(x,x′)＝λ ₁ L ₂ (x,x′)+λ ₂ L _LPIPS (x,x′)+λ ₃ L _ID (x,x′)+λ ₄ L _SAT (x,x′ )+λ ₅ L _CON (x, x′)+λ ₆ L _LUM (x, x′).

Optional, λ ₁ =1, λ ₂ =0.85, λ ₃ =0.15, λ ₄ =1, λ ₅ =1, λ ₆ =1.

Step S23: Adjust the parameters of the preset initial encoder neural network according to the loss function, and generate the preset encoder neural network when the loss function meets the preset conditions.

During the training process, the loss function is fed back to the preset initial encoder neural network, and the parameters of the preset initial encoder neural network are adjusted through the loss function, so that the weights of the network are updated and adjusted. When the loss function satisfies the preset When conditions are set, a preset encoder neural network is generated, where the preset condition can be that the loss value converges to an allowed range or reaches a preset number of iterations.

The preset initial encoder neural network is trained based on the sample image, and the parameters of the preset initial encoder neural network are updated based on the loss function, which further improves the accuracy of the preset encoder neural network.

Based on any embodiment of the present application, the preset encoder neural network includes a preset residual network, a plurality of first fully connected layers and a plurality of second fully connected layers corresponding to each of the scales, Each of the first fully connected layers corresponds to each of the second fully connected layers, and the image to be processed is encoded based on the preset encoder neural network to obtain multiple sets of feature latent codes belonging to different scales, As shown in Figure 3, it includes the following steps:

Step S301: Input the image to be processed into the preset residual network, and perform feature extraction on the image to be processed based on the channel attention mechanism and the spatial attention mechanism, and obtain a plurality of features corresponding to each location. The first eigenvector corresponding to the above scale.

In this embodiment, the preset encoder neural network includes a preset residual network, a plurality of first fully connected layers and a plurality of second fully connected layers corresponding to each scale, each first fully connected layer and each second fully connected layer. The fully connected layers have a one-to-one correspondence. Each group of first fully connected layers can include multiple first fully connected layers, and each second fully connected layer outputs a layer of latent feature codes. The image to be processed is input into the preset residual network. The preset residual network performs feature extraction based on the channel attention mechanism and the spatial attention mechanism, and extracts multiple first feature vectors corresponding to each scale. Each scale can correspond to multiple first feature vectors.

Step S302: Input the first feature vector into the first fully connected layer according to the scale to which the first feature vector belongs, and obtain a plurality of second feature vectors.

Each first feature vector is input to the first fully connected layer at the same scale, and the first fully connected layer outputs a second feature vector to obtain multiple second feature vectors.

Step S303: Input each of the second feature vectors into the second fully connected layer to obtain multiple sets of feature latent codes.

The second feature vectors are respectively input into the second fully connected layer, and the second fully connected layer maps each second feature vector into multiple sets of feature latent codes.

By sequentially processing the image to be processed through the preset residual network, each first fully connected layer, and each second fully connected layer, the encoded feature latent code is obtained, which further improves the accuracy of the feature latent code.

It should be noted that the solution in the above embodiment is only a specific implementation solution proposed by this application. Those skilled in the art can flexibly use encoder neural networks with other structures to perform encoding processing, which does not affect the protection scope of this application. .

In order to further elaborate on the technical idea of the present invention, the technical solution of the present invention will now be described in conjunction with specific application scenarios.

The embodiment of the present application provides an image processing method, as shown in Figure 4, applied in the pSp framework composed of a preset encoder neural network and a StyleGAN generator. The preset encoder neural network adopts a channel-based attention mechanism and spatial The feature pyramid structure of the attention mechanism includes three levels, corresponding to three scales, and includes the following steps:

Step S1: Obtain 10,000 sets of sample images. Each set of sample images includes a first image of a face under unbalanced illumination and a second image under balanced illumination.

Each set of sample images has been processed by face detection and face alignment, and normalized to 256*256 size.

Step S2: Enter any first image as an input image into the preset encoder neural network to obtain sample feature latent codes of three scales.

Specifically, the three-layer vector is first extracted through the preset residual network, in which the preset residual network introduces a convolutional block attention module that adopts the channel attention mechanism and the spatial attention mechanism. Each first feature vector in each layer of vector is converted into a second feature vector after passing through a map2style first fully connected layer. Each second feature vector then passes through a second fully connected layer A to obtain the sample feature latent of three scales. code. It includes 18 layers of 512-dimensional sample feature latent codes. The small scale corresponds to layers 0-2, the medium scale corresponds to layers 3-6, and the large scale corresponds to layers 7-18.

Step S3: Input the sample feature latent codes of three scales into the pre-trained StyleGAN generator to obtain the third image as the output image.

Step S4: Determine the loss function based on the difference between the third image and the second image.

Specifically, the pixel loss, perceptual loss, identity information loss, saturation loss, contrast loss and brightness loss are first determined based on the difference, and then the pixel loss, perceptual loss, identity information loss and saturation loss are combined according to the preset weight coefficients. , contrast loss and brightness loss are weighted and summed to generate a loss function.

Step S5, feedback the loss function and adjust parameters.

Feed the loss function back to the network, adjust the weights of the network, and return to step S2 until the loss value of the loss function converges to the allowed range. The training ends and the preset encoder neural network is obtained.

Step S6: Obtain the original image including the face and with unbalanced illumination, perform face detection, face alignment processing and normalize to 256*256 size to obtain the image to be processed.

Step S7: The image to be processed is input into the preset encoder neural network as the input image. After the preset encoder neural network performs encoding, the corresponding feature latent codes of the three scales are input into the StyleGAN generator, and the output of the StyleGAN generator is used as the output. The target image of the image.

By applying the above technical solution, the channel attention mechanism and the spatial attention mechanism are introduced to encode the image to be processed into multiple sets of feature latent codes that conform to the feature pyramid structure, allowing the StyleGAN generator to directly complete image reconstruction based on the feature latent codes, thereby improving The efficiency and accuracy of illumination equalization processing of images, and the combination of multiple losses during the training process, further ensure the accuracy of the preset encoder neural network obtained by training, and due to the consideration of saturation loss, contrast loss and brightness loss, adding an inspection dimension that focuses on the consistency of the target illumination equalization effect, and can converge to the target illumination equalization effect faster during model training.

The embodiment of the present application also proposes an image processing device, as shown in Figure 5. The device includes:

The acquisition module 501 is used to acquire the image to be processed with unbalanced illumination; the encoding module 502 is used to encode the image to be processed based on a preset encoder neural network and obtain multiple sets of feature latent codes belonging to different scales, wherein, The preset encoder neural network adopts a feature pyramid structure based on a channel attention mechanism and a spatial attention mechanism, and each scale corresponds to a different level in the feature pyramid structure; the generation module 503 is used to combine the features of each group into The feature latent code is input into the pre-trained StyleGAN generator, and the illumination-balanced target image is obtained according to the output result of the StyleGAN generator.

In a specific application scenario, the device also includes a training module for: acquiring multiple groups of sample images, each group of sample images including a first image of a sample object under unbalanced illumination and a first image under balanced illumination. The second image; based on the sample image, train the preset initial encoder neural network using the feature pyramid structure, and determine the loss function during the training process; train the preset initial encoder neural network according to the loss function The parameters are adjusted, and when the loss function meets the preset conditions, the preset encoder neural network is generated.

In a specific application scenario, the training module is specifically used to: input the first image into the preset initial encoder neural network, encode the first image based on the preset initial encoder neural network, and obtain the respective Multiple sets of sample feature latent codes corresponding to each of the scales; input each set of sample feature latent codes into the StyleGAN generator, and obtain a third image according to the output result of the StyleGAN generator; according to the third image and The difference between the second images determines the loss function.

In specific application scenarios, the training module is also specifically used to: determine pixel loss, perceptual loss, identity information loss, saturation loss, contrast loss and brightness loss based on the difference; and adjust the pixels based on the preset weight coefficient combination. The point loss, the perceptual loss, the identity information loss, the saturation loss, the contrast loss and the brightness loss are weighted and summed to generate the loss function.

In a specific application scenario, the sample object is a human face, and the training module is also specifically used to: obtain multiple original sample images: perform face detection on the original sample images to obtain sample face images; The face image is subjected to face alignment processing and normalized to a preset size to obtain multiple sets of sample images.

In a specific application scenario, the preset encoder neural network includes a preset residual network, a plurality of first fully connected layers and a plurality of second fully connected layers corresponding to each of the scales. There is a one-to-one correspondence between a fully connected layer and each of the second fully connected layers. The encoding module 502 is specifically configured to: input the image to be processed into the preset residual network, and based on the channel attention mechanism and the The spatial attention mechanism performs feature extraction on the image to be processed to obtain a plurality of first feature vectors corresponding to each of the scales; input the first feature vector into the corresponding scale according to the scale to which the first feature vector belongs. The first fully connected layer is used to obtain a plurality of second feature vectors; each of the second feature vectors is input into the second fully connected layer to obtain multiple sets of feature latent codes.

In a specific application scenario, the image to be processed is a human face image. The acquisition module 501 is specifically used to: acquire an original image including a human face and with unbalanced illumination; perform face detection on the original image to obtain the face image to be processed. Face image; perform face alignment processing on the face image to be processed and normalize it to a preset size to obtain the image to be processed.

The image processing device in the embodiment of the present application includes: an acquisition module, used to acquire an image to be processed with unbalanced illumination; an encoding module, used to encode the image to be processed based on a preset encoder neural network, and acquire multiple groups of images belonging to different scales. Feature latent code, in which the preset encoder neural network adopts a feature pyramid structure based on the channel attention mechanism and the spatial attention mechanism. Each scale corresponds to different levels in the feature pyramid structure; the generation module is used to convert each group of feature latent codes into The code is input to the pre-trained StyleGAN generator, and the illumination-balanced target image is obtained according to the output result of the StyleGAN generator. By introducing the channel attention mechanism and the spatial attention mechanism, the image to be processed is encoded into multiple sets of latent features that conform to the feature pyramid structure. code, allowing the StyleGAN generator to directly complete image reconstruction based on the feature latent code, thereby improving the efficiency and accuracy of illumination equalization processing of images.

An embodiment of the present invention also provides an electronic device, as shown in Figure 6, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604. The processor 601, the communication interface 602, and the memory 603 communicate through the communication bus 604. complete mutual communication,

Memory 603, used to store executable instructions of the processor;

Processor 601 is configured to, via execution of the executable instructions:

Obtain the image to be processed with unbalanced illumination; encode the image to be processed based on a preset encoder neural network to obtain multiple sets of feature latent codes belonging to different scales, wherein the preset encoder neural network uses channel attention-based The feature pyramid structure of the force mechanism and the spatial attention mechanism, each of the scales corresponds to different levels in the feature pyramid structure; each set of feature latent codes is input into the pre-trained StyleGAN generator, and according to the StyleGAN generator The output result obtains a well-illuminated target image.

The above communication bus may be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect Standard) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above terminal and other devices.

The memory may include RAM (Random Access Memory), or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far away from the aforementioned processor.

The above-mentioned processor can be a general-purpose processor, including CPU (Central Processing Unit, central processing unit), NP (Network Processor, network processor), etc.; it can also be DSP (Digital Signal Processing, digital signal processor), ASIC ( Application Specific Integrated Circuit, FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In yet another embodiment of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the image processing as described above is implemented. method.

In yet another embodiment of the present invention, a computer program product containing instructions is also provided, which when run on a computer causes the computer to execute the image processing method as described above.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available media may be magnetic media (eg, floppy disk, hard disk, tape), optical media (eg, DVD), or semiconductor media (eg, solid state drive), etc.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations are mutually exclusive. any such actual relationship or sequence exists between them. Furthermore, the terms "comprises," "comprises," or any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes the stated element.

Each embodiment in this specification is described in a related manner. The same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on its differences from other embodiments.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims (10)

1. An image processing method, characterized in that the method includes: Obtain the image to be processed with uneven illumination; The image to be processed is encoded based on a preset encoder neural network to obtain multiple sets of feature latent codes belonging to different scales. The preset encoder neural network uses features based on channel attention mechanism and spatial attention mechanism. Pyramid structure, each of the scales corresponds to different levels in the characteristic pyramid structure; Each set of feature latent codes is input into a pre-trained StyleGAN generator, and a target image with balanced illumination is obtained based on the output result of the StyleGAN generator. 2. The method of claim 1, characterized in that, before encoding the image to be processed based on a preset encoder neural network to obtain multiple groups of feature latent codes belonging to different scales, the method further includes: Acquire multiple sets of sample images, each set of sample images including a first image of a sample object under unbalanced illumination and a second image under balanced illumination; Train a preset initial encoder neural network using the feature pyramid structure based on the sample image, and determine the loss function during the training process; The parameters of the preset initial encoder neural network are adjusted according to the loss function, and when the loss function meets the preset conditions, the preset encoder neural network is generated. 3. The method of claim 2, wherein training a preset initial encoder neural network using the feature pyramid structure based on the sample image and determining a loss function during the training process includes: Input the first image into the preset initial encoder neural network, encode the first image based on the preset initial encoder neural network, and obtain multiple sets of sample feature latent codes corresponding to each of the scales; Input the sample feature latent code of each group into the StyleGAN generator, and obtain the third image according to the output result of the StyleGAN generator; The loss function is determined based on the difference between the third image and the second image. 4. The method of claim 3, wherein determining the loss function based on the difference between the third image and the second image includes: Determine pixel point loss, perceptual loss, identity information loss, saturation loss, contrast loss and brightness loss based on the difference; The loss function is generated by performing a weighted sum of the pixel point loss, the perceptual loss, the identity information loss, the saturation loss, the contrast loss and the brightness loss according to a preset weight coefficient combination. 5. The method of claim 2, wherein the sample object is a human face, and obtaining multiple groups of sample images includes: Get multiple original sample images: Perform face detection on the original sample image to obtain a sample face image; Face alignment processing is performed on each of the sample face images and normalized to a preset size to obtain multiple groups of the sample images. 6. The method of claim 1, wherein the preset encoder neural network includes a preset residual network, a plurality of first fully connected layers corresponding to each of the scales, and a plurality of second fully connected layers. Fully connected layer, each of the first fully connected layers corresponds to each of the second fully connected layers, the image to be processed is encoded based on the preset encoder neural network, and multiple groups of images belonging to different scales are obtained. Feature latent codes include: The image to be processed is input into the preset residual network, and feature extraction is performed on the image to be processed based on the channel attention mechanism and the spatial attention mechanism, and a plurality of images corresponding to each of the scales are obtained. The first eigenvector of; Input the first feature vector into the first fully connected layer according to the scale to which the first feature vector belongs, and obtain a plurality of second feature vectors; Each of the second feature vectors is input into the second fully connected layer to obtain multiple sets of feature latent codes. 7. The method of claim 1, wherein the image to be processed is a face image, and obtaining the image to be processed with unbalanced illumination includes: Obtain raw images including faces and uneven lighting; Perform face detection on the original image to obtain the face image to be processed; Perform face alignment processing on the face image to be processed and normalize it to a preset size to obtain the image to be processed. 8. An image processing device, characterized in that the device includes: The acquisition module is used to acquire images to be processed with uneven illumination; An encoding module, used to encode the image to be processed based on a preset encoder neural network and obtain multiple sets of feature latent codes belonging to different scales, wherein the preset encoder neural network adopts a channel attention mechanism and space-based The feature pyramid structure of the attention mechanism, each of the scales corresponds to different levels in the feature pyramid structure; A generation module, configured to input each set of feature latent codes into a pre-trained StyleGAN generator, and obtain a target image with balanced illumination according to the output result of the StyleGAN generator. 9. An electronic device, characterized in that it includes: processor; and memory for storing executable instructions for the processor; Wherein, the processor is configured to execute the image processing method according to any one of claims 1 to 7 by executing the executable instructions. 10. A computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.