CN111126568B - Image processing method and device, electronic equipment and computer readable storage medium - Google Patents

Abstract

The application discloses an image processing method, an image processing device, an electronic apparatus, and a nonvolatile computer-readable storage medium. The image processing method comprises the following steps: acquiring exposure time of an image to be repaired in shooting; determining an exposure time interval in which the exposure time is located; determining a repair model according to the exposure time interval, wherein different exposure time intervals correspond to different repair models; and carrying out restoration processing on the portrait area of the image to be restored according to the determined restoration model. According to the image processing method, the image processing device, the electronic equipment and the nonvolatile computer readable storage medium, different repair models are selected according to different exposure time lengths of the image to be repaired, so that a repair model suitable for the current exposure time length can be selected to repair the image to be repaired, and the repair effect of the repaired image can be improved.

Description

Image processing method and device, electronic device, and computer-readable storage medium

technical field

The present application relates to the technical field of image processing, and in particular to an image processing method, an image processing device, electronic equipment, and a non-volatile computer-readable storage medium.

Background technique

Ultra-clear portrait technology refers to a technology that uses image processing algorithms to process portraits in images to make the details of the portraits richer and clearer. However, when using the ultra-clear portrait algorithm to restore portraits, one of the selected algorithms is often used to process portraits, which may lead to poor image restoration effects.

Contents of the invention

Embodiments of the present application provide an image processing method, an image processing device, electronic equipment, and a non-volatile computer-readable storage medium.

The image processing method according to the embodiment of the present application is used in an electronic device. The image processing method includes: acquiring the exposure time of the image to be repaired when it is taken; determining the exposure time interval in which the exposure time is located; determining the repair model according to the exposure time interval, and different exposure time intervals correspond to different The inpainting model: performing inpainting processing on the portrait region of the image to be inpainted according to the determined inpainting model.

The image processing device according to the embodiment of the present application is used in electronic equipment. The image processing device includes an acquisition module, a first determination module, a second determination module, and a restoration module. The acquiring module is used to acquire the exposure time of the image to be repaired when it is taken. The first determining module is used to determine the exposure duration interval in which the exposure duration is located. The second determining module is configured to determine a repair model according to the exposure time interval, and different exposure time intervals correspond to different repair models. The inpainting module is used for inpainting the portrait area of the image to be inpainted according to the determined inpainting model.

The electronic device in the embodiment of the present application includes a casing and a processor. The processor is mounted on the housing. The processor is used to: obtain the exposure duration of the image to be repaired when shooting; determine the exposure duration interval of the exposure duration; determine the restoration model according to the exposure duration interval, and the different exposure duration intervals correspond to different The inpainting model; performing inpainting processing on the portrait area of the image to be inpainted according to the determined inpainting model.

A non-volatile computer-readable storage medium containing computer-readable instructions according to an embodiment of the present application. When the computer-readable instructions are executed by a processor, the processor is made to perform the following steps: acquire the image of the image to be restored when it is taken Exposure duration; determine the exposure duration interval in which the exposure duration is located; determine the repair model according to the exposure duration interval, and the different exposure duration intervals correspond to different repair models; The portrait area of the image to be repaired is repaired.

The image processing method, image processing device, electronic equipment, and non-volatile computer-readable storage medium in the embodiments of the present application select different repair models according to the exposure time of the image to be repaired, so that the current exposure time can be selected The inpainting model is used to inpaint the image to be inpainted, which can improve the inpainting effect of the inpainted image.

Additional aspects and advantages of embodiments of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, wherein:

FIG. 1 is a schematic flow diagram of an image processing method in some embodiments of the present application;

FIG. 2 is a schematic diagram of an image processing device in some embodiments of the present application;

FIG. 3 is a schematic diagram of an electronic device according to some embodiments of the present application;

4 is a schematic flow diagram of an image processing method in some embodiments of the present application;

5 is a schematic diagram of an image processing device in some embodiments of the present application;

6 is a schematic flow diagram of an image processing method in some embodiments of the present application;

FIG. 7 is a schematic flowchart of an image processing method in some embodiments of the present application;

Fig. 8 is a schematic diagram of a restoration module in an image processing device according to some embodiments of the present application;

Fig. 9 is a schematic diagram of a second processing unit in an image processing device according to some embodiments of the present application;

Fig. 10 is a schematic diagram of a face detection model in some embodiments of the present application;

Fig. 11 is a schematic diagram of a human face restoration model in some embodiments of the present application;

FIG. 12 is a schematic flowchart of an image processing method in some embodiments of the present application;

Fig. 13 is a schematic diagram of an image processing device in some embodiments of the present application;

FIG. 14 is a schematic flowchart of an image processing method in some embodiments of the present application;

Fig. 15 is a schematic diagram of a third acquisition module in an image processing device in some embodiments of the present application;

Fig. 16 is a schematic flowchart of an image processing method in some embodiments of the present application;

Fig. 17 is a schematic diagram of a processing module in an image processing device in some embodiments of the present application;

Fig. 18 is a schematic flowchart of an image processing method in some embodiments of the present application;

Fig. 19 is a schematic diagram of interaction between a non-volatile readable storage medium and a processor in some embodiments of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary, are only for explaining the embodiments of the present application, and should not be construed as limiting the embodiments of the present application.

Referring to Fig. 1 and Fig. 3, the present application provides an image processing method. Image processing methods include:

01: Obtain the exposure time of the image to be repaired when shooting;

02: Determine the exposure duration interval of the exposure duration;

03: Determine the repair model according to the exposure time interval, and different exposure time intervals correspond to different restoration models; and

04: Repair the portrait area of the image to be repaired according to the determined repair model.

Referring to FIG. 2 and FIG. 3 , the present application also provides an image processing device 10 . The image processing method in the embodiment of the present application can be implemented by the image processing device 10 in the embodiment of the present application. The image processing device 10 includes a first acquisition module 11 , a first determination module 12 , a second determination module 13 , and a restoration module 14 . Step 01 can be implemented by the first acquisition module 11 . Step 02 can be implemented by the first determination module 12 . Step 03 can be implemented by the second determining module 13 . Step 04 can be implemented by the repair module 14 . That is to say, the first acquiring module 11 can be used to acquire the exposure time of the image to be repaired when it is taken. The detection module 12 may be used to determine the exposure duration interval in which the exposure duration is located. The restoration module 13 can be used to determine the restoration model according to the exposure time interval, and different exposure time intervals correspond to different restoration models.

Referring to FIG. 3 , the present application also provides an electronic device 20 . The image processing method in the embodiment of the present application may also be implemented by the electronic device 20 in the embodiment of the present application. The electronic device 20 can be a mobile phone, a tablet computer, a notebook computer, a smart wearable device (such as a smart watch, a smart bracelet, a smart helmet, smart glasses, etc.), a smart mirror, a drone, an unmanned vehicle, an unmanned ship, etc. This is not limited. The electronic device 20 includes a casing 22 , a processor 21 and a camera 23 . Both the processor 21 and the camera 23 are mounted on the casing 22 . Step 01 , step 02 , step 03 , and step 04 can all be implemented by the processor 21 . That is to say, the processor 21 may be configured to obtain the exposure time of the image to be repaired when it is taken and determine the exposure time interval in which the exposure time is located. The processor 21 may also be configured to determine a restoration model according to the exposure time interval, and perform restoration processing on the portrait area of the image to be repaired according to the determined restoration model. Among them, different exposure time intervals correspond to different restoration models.

Among them, the inpainting model is a pre-trained model, and one inpainting model can be regarded as an ultra-clear portrait algorithm, and different inpainting models belong to different ultra-clear portrait algorithms. Figure 11 is an example of a repair model. As shown in Figure 11, the portrait area in the image to be repaired is used as the input of the repair model, and the repair model will perform multiple convolutions, pooling, upsampling, and deconvolution on the portrait area. The inpainting models corresponding to different exposure times can have different numbers of convolutional layers, different numbers of pooling layers, different numbers of upsampling layers, different numbers of deconvolutional layers, and different weights. In one embodiment of the present application, different inpainting models have the same number of convolutional layers, the same number of pooling layers, the same number of upsampling layers, and the same number of deconvolutional layers, but different inpainting models have different weights. Different weights make the inpainting model have different inpainted image outputs for the same input image to be inpainted, in other words, inpainted images with different inpainting effects can be obtained. When the exposure time of the input image to be repaired is within the exposure time interval corresponding to the repair model, using the repair model to perform repair processing on the image to be repaired can output a repaired image with the best repair effect.

Specifically, it is assumed that there are three exposure time intervals, namely the exposure time interval [T1, T2), the exposure time interval [T2, T3), the exposure time interval [T3, T4), and the repair corresponding to the exposure time interval [T1, T2] The model is P1, the repair model corresponding to the exposure time interval [T2, T3) is P2, the repair model corresponding to the exposure time interval [T3, T4] is P2, the exposure time of the image to be repaired is t, and T2<t<T3, Then, the processor 21 should select the inpainting model P2 to perform inpainting on the image to be inpainted. It should be noted that the number of exposure time intervals is 3 only as an example, and in other examples, the number of exposure time intervals may also be two, four, five, eight, ten, or fifteen , twenty, thirty, etc., are not limited here. The number of inpainted models is equal to the number of exposure time intervals.

It can be understood that the clarity of the image to be repaired acquired by the camera 23 is usually different when the exposure time is different. Generally, the longer the exposure time is, the more likely the electronic device 20 is shaken, and the resolution of the image to be repaired acquired by the camera 23 is lower. The clarity of the image to be repaired is different, and the richness of the details of the portrait area in the image to be repaired is also different. Then, for the images to be repaired obtained under different exposure time lengths, the restoration model corresponding to the exposure time length can be selected, which can make the use of the model The repaired image output after repairing has the best restoration effect.

In related technologies, each time an electronic device acquires an image containing a portrait, it usually uses the same ultra-clear portrait algorithm to repair the portrait area in the image. However, in the actual application process, different images usually have different sharpness. If the same ultra-clear portrait algorithm is always used to repair the portrait area in the image, it may lead to the problem of poor image restoration effect.

The image processing method, the image processing device 10 and the electronic device 20 in the embodiment of the present application will select a repair model corresponding to the exposure time according to the exposure time of the image to be repaired, so that the best repair model can be used to perform repair processing on the image to be repaired , which is beneficial to improve the restoration effect of the repaired image.

Referring to Fig. 4, in some embodiments, the image processing method also includes:

051: Establish an initial model;

052: Obtain multiple frames of training images whose exposure durations are in different exposure duration intervals; and

053: Use multiple frames of training images whose exposure time is within the same exposure time interval to train the initial model to obtain a restoration model corresponding to the exposure time interval.

Referring to FIG. 5 , in some implementations, the image processing device 10 further includes a building module 151 , a second acquiring module 152 , and a training module 153 . Step 051 can be implemented by the establishment module 151 . Step 052 may be performed by the second acquiring module 152 . Step 053 can be implemented by the training module 153 . That is to say, the establishment module 151 can be used to establish an initial model. The second acquiring module 152 may be used to acquire multiple frames of training images whose exposure durations are in different exposure duration intervals. The training module 153 may be used to train an initial model by using multiple frames of training images whose exposure durations are within the same exposure duration interval, so as to obtain a restoration model corresponding to the exposure duration interval.

Please refer to FIG. 3 again. In some embodiments, step 051 , step 052 , and step 053 can all be implemented by the processor 21 . That is to say, the processor 21 can be used to establish an initial model and acquire multiple frames of training images whose exposure durations are in different exposure duration intervals. The processor 21 may also be configured to train an initial model by using multiple frames of training images whose exposure durations are within the same exposure duration interval, so as to obtain a restoration model corresponding to the exposure duration interval.

Specifically, it is assumed that three exposure time intervals are set, namely the exposure time interval [T1, T2), the exposure time interval [T2, T3), and the exposure time interval [T3, T4]. Processor 21 first builds an initial model. Exemplarily, the initial model has the same model architecture as the restoration model shown in FIG. 11 , but the weights in the initial model have not been trained yet. The processor 21 needs to input multiple training images into the initial model to train weights in the initial model. Specifically, the processor 21 needs to acquire a large number of training images of the first type whose exposure time is within the exposure time interval [T1, T2), a large number of training images of the second type whose exposure time is within the exposure time interval [T2, T3), And a large number of training images of the third type whose exposure time is within the exposure time interval [T3, T4]. Subsequently, the processor 21 inputs a large number of training images of the first type into the initial model for training, so as to obtain a restoration model P1 corresponding to the exposure time interval [T1, T2), and the restoration model P1 has a weight of the first type. Processor 21 inputs a large number of training images of the second type into the initial model for training to obtain a restoration model P2 corresponding to the exposure time interval [T2, T3), the restoration model P2 has weights of the second type, wherein the first The class weights are different from the second class weights. The processor 21 inputs a large number of training images of the third type into the initial model for training, so as to obtain a repair model P3 corresponding to the exposure time interval [T3, T4], and the repair model P3 has a weight of the third type, wherein, the third The class weights are different from the first class weights and different from the second class weights. Thus, the processor 21 can train a plurality of restoration models. The processor 21 can store the restoration models in the memory, and the processor 21 can directly call these restoration models in the subsequent portrait restoration process.

Please refer to FIG. 6 and FIG. 7, in some implementations, step 04 performs repair processing on the portrait area of the image to be repaired according to the determined repair model, including:

041: Detect the portrait area in the image to be repaired;

042: Perform multiple convolutions on the portrait area to obtain multiple feature images;

043: Perform multiple upsampling and at least one deconvolution on the feature image output by the last convolution to obtain a residual image; and

044: Fusion residual image and portrait area to obtain repaired image.

Wherein, step 043 performs multiple upsampling and at least one deconvolution on the feature image output by the last convolution to obtain a residual image, including:

0431: During the first upsampling process, perform upsampling and deconvolution on the feature image output by the last convolution;

0432: In the second or more upsampling process, fuse the image obtained by the previous upsampling, the feature image corresponding to the size of the image obtained by the previous upsampling, and the previous N times of deconvolution to obtain , and perform upsampling or upsampling and deconvolution on the fused image;

0433: Fuse the image obtained by the last upsampling and the image obtained by the first N times of deconvolution to obtain a residual image, where N≥1, and N∈N+.

Referring to FIG. 8 and FIG. 9 , in some implementations, the restoration module 14 includes a first detection unit 141 , a first processing unit 142 , a second processing unit 143 , and a fusion unit 144 . The second processing unit 143 includes a first processing subunit 1431 , a second processing subunit 1432 , and a fusion subunit 1433 . Step 031 can be implemented by the first detection unit 141 . Step 042 may be implemented by the first processing unit 142 . Step 043 may be implemented by the second processing unit 143 . Step 044 can be implemented by the fusion unit 144 . Step 0431 may be implemented by the first processing subunit 1431. Step 0432 may be implemented by the second processing subunit 1432 . Step 0433 can be implemented by the fusion subunit 1433 . That is to say, the first detection unit 141 can be used to detect the portrait area in the image to be repaired. The first processing unit 142 may be configured to perform multiple convolutions on the portrait area to obtain multiple feature images. The second processing unit 143 may be configured to perform multiple upsampling and at least one deconvolution on the feature image output by the last convolution to obtain a residual image. The fusion unit 144 can be used to fuse the residual image and the portrait area to obtain a repaired image. The first processing subunit 1431 may be configured to perform upsampling and deconvolution on the feature image output by the last convolution during the first upsampling process. The second processing subunit 1432 may be configured to fuse the image obtained by the previous upsampling, the feature image corresponding to the size of the image obtained by the previous upsampling, and The image obtained by the first N times of deconvolution, and perform upsampling or perform upsampling and deconvolution on the fused image. The fusion subunit 1433 can be used to fuse the image obtained by the last upsampling and the image obtained by the previous N times of deconvolution to obtain a residual image, where N≥1, and N∈N+.

Please refer to FIG. 3 again. In some embodiments, step 041 , step 042 , step 043 , step 0431 , step 0432 , step 0433 , step 0434 , and step 044 can all be implemented by the processor 21 . That is to say, the processor 21 can also be used to detect the portrait area in the image to be repaired and perform multiple convolutions on the portrait area to obtain multiple feature images. The processor 21 may also be configured to perform multiple upsampling and at least one deconvolution on the feature image output by the last convolution to obtain a residual image. The processor 21 can also be used to fuse the residual image and the portrait area to obtain a repaired image. When the processor 21 is used to perform multiple upsampling and at least one deconvolution on the feature image output by the last convolution to obtain a residual image, it is specifically used for: during the first upsampling process, the last convolution After the output feature image performs upsampling and deconvolution; in the second and more than the second upsampling process, the image obtained by the previous upsampling is fused, and the image corresponding to the size of the image obtained by the previous upsampling The feature image, and the image obtained by the first N times of deconvolution, and perform upsampling or upsampling and deconvolution on the fused image; fuse the image obtained by the last upsampling and the image obtained by the first N times of deconvolution To get the residual image, where N≥1, and N∈N+.

Specifically, referring to FIG. 3 , FIG. 10 and FIG. 11 , the processor 21 first detects the portrait area in the image to be repaired. For example, the processor may detect the portrait area in the image to be repaired according to the face detection model shown in FIG. 10 . Specifically, the specific detection process of the face detection model shown in Fig. 10. is: the convolutional layer and the pooling layer (Convolution and Pooling) perform feature extraction on the image to be repaired to obtain multiple feature images; the last layer of convolutional layer ( Final ConvFeature Map) performs the last convolution on the feature image output by the convolution layer and the pooling layer, and outputs the feature image obtained by the last convolution to the fully-connected layer (Fully-connected Layers). The fully connected layer classifies the feature image output by the last convolutional layer, and outputs the classification result to the coordinate output branch (Coordinate). The coordinate output branch outputs the position coordinates of the face in the image to be repaired. So far, the detection of the portrait area in the image to be repaired is completed. Subsequently, the processor 21 inputs the portrait region into a repair model corresponding to the exposure time of the image to be repaired for repair. In an example, the restoration model can be the face restoration model shown in Figure 11, wherein different restoration models can adopt the model architecture shown in Figure 11, but different restoration models have different weights (not shown in Figure 11 Shows). As shown in FIG. 11 , after the portrait area is input into the face restoration model, the processor 21 first performs the first convolution on the portrait area, and then performs the first pooling on the feature image after the first convolution. Subsequently, the processor 21 performs a second convolution on the feature image after the first pooling, and then performs a second pooling on the feature image after the second convolution. Subsequently, the processor 21 performs a third convolution on the feature image after the second pooling, and then performs a third pooling on the feature image after the third convolution. Subsequently, the processor 21 performs a fourth convolution on the feature image after the third pooling, and then performs a fourth pooling on the feature image after the fourth convolution. Subsequently, the processor 21 performs a fifth convolution on the feature image after the fourth pooling. Subsequently, the processor 21 performs the first upsampling and the first deconvolution on the feature image after the fifth convolution, wherein the first deconvolution actually performs two deconvolution actions, one An action of deconvolution outputs an image of a first size, and another action of deconvolution outputs an image of a second size, the second size being larger than the first size. Subsequently, processor 21 fuses the image obtained by upsampling for the first time and the feature image corresponding to the size of the image obtained by upsampling for the first time (i.e. the feature image after the fourth convolution) (i.e., as shown in FIG. 11 The link between the convolutional layer corresponding to the fourth convolution and the upsampling layer corresponding to the second upsampling shown), and performs the second upsampling and the second deconvolution on the fused image, where the first The second deconvolution actually performs two deconvolution actions, one deconvolution action outputs an image of the second size, and the other deconvolution action outputs an image of the third size, and the third size is larger than the second size. Subsequently, the processor 21 combines the image obtained by the second upsampling and the feature image corresponding to the size of the image obtained by the second upsampling (i.e. the feature image after the third convolution, as shown in FIG. 11 , the first The convolutional layer corresponding to the third convolution is linked to the upsampling layer corresponding to the third upsampling), and the image obtained by the second deconvolution (that is, the image of the first size obtained by the first deconvolution) Fusion is performed and a third upsampling is performed on the fused image. Subsequently, the processor 21 fuses the image obtained by the third upsampling, and the feature image corresponding to the size of the image obtained by the third upsampling (that is, the feature image after the second convolution, as shown in FIG. 11 , the first The convolutional layer corresponding to the second convolution is linked to the upsampling layer corresponding to the fourth upsampling), the image obtained by the second deconvolution before (that is, the image of the second size obtained by the first deconvolution), and the image obtained by the previous deconvolution (that is, the image of the second size obtained by the second deconvolution), and perform the fourth upsampling on the fused image. Subsequently, the processor 21 fuses the image obtained by the fourth upsampling and the image obtained by the previous deconvolution (that is, the image of the third size obtained by the second deconvolution) to obtain a residual image. Finally, the processor 21 fuses the residual image with the portrait area input into the face restoration model to obtain the restoration image. The details of the face in the repaired image are richer and the definition is higher.

The image processing method, the image processing device 10, and the electronic device 20 in the embodiment of the present application extract the features of the portrait area through multiple convolutions and multiple pooling processes, and then amplify the extracted features through multiple upsampling, and In the process of partial upsampling, the image fused with deconvolution is processed to realize the transfer of features and the expansion of image size. In addition, partial upsampling is to process the image that combines the feature image corresponding to the size of the image obtained by the previous upsampling, that is, the feature extraction layer of the corresponding level is connected, so that the high-level semantic features can be more fully when upsampling. The details of portrait restoration will be more obvious, and the restoration of portrait details will be more delicate. In addition, in the face inpainting model shown in Figure 11, the image obtained by the first deconvolution is not directly passed to the second upsampling process to be fused with the image to be performed the second upsampling, but is passed In the third upsampling process and the fourth upsampling process, similarly, the image obtained by the second deconvolution is not directly passed to the third upsampling process, but to the fourth upsampling process In the sampling process, this method can combine high-level features with low-level features, making the features richer and the details restored more accurately. Furthermore, in the face inpainting model shown in Figure 11, deconvolution is no longer performed on the images obtained by the second and above upsampling. This method can avoid deconvolution of low-level features. And there is the problem of block effect.

Referring to Fig. 12, in some embodiments, the image processing method also includes:

06: Determine whether the exposure time is within the predetermined time range;

When the exposure duration is within the predetermined duration range, performing the step of determining the exposure duration interval in which the exposure duration is located;

07: When the exposure time is outside the predetermined time range, determine whether the exposure time is greater than the maximum value in the predetermined time range;

08: When the exposure time is longer than the maximum value in the predetermined time range, obtain a reference image, and the resolution of the reference image is higher than the predetermined resolution; and

09: Perform portrait super-resolution algorithm processing on the image to be repaired according to the reference image to obtain the repaired image.

Please refer to FIG. 13 , in some implementations, the image processing device 10 further includes a first judging module 16 , a second judging module 17 , a third acquiring module 18 , and a processing module 19 . Step 06 can be implemented by the first judging module 16 . Step 07 can be realized by the second judging module 17 . Step 08 can be implemented by the third acquiring module 18 . Step 09 can be implemented by the processing module 19 . That is to say, the judging module 16 can be used to judge whether the exposure time is within the predetermined time range. When the exposure duration is within the predetermined duration range, the first determining module 12 may be configured to determine an exposure duration interval in which the exposure duration is located. When the exposure time is outside the predetermined time range, the second judging module 17 may be used to judge whether the exposure time is greater than the maximum value in the predetermined time range. When the exposure time is longer than the maximum value in the predetermined time range, the second acquisition module 18 may be used to acquire a reference image whose resolution is higher than the predetermined resolution. The processing module 19 can be used to perform portrait super-resolution algorithm processing on the image to be repaired according to the reference image, so as to obtain the repaired image.

Please refer to FIG. 3 again. In some embodiments, step 06 , step 07 , step 08 , and step 09 can all be implemented by the processor 21 . That is to say, the processor 21 can be used to determine whether the exposure time is within the predetermined time range. When the exposure time is within the predetermined time range, the processor 21 may be configured to determine an exposure time interval in which the exposure time is located. When the exposure time is outside the predetermined time range, the processor 21 may be configured to determine whether the exposure time is greater than a maximum value in the predetermined time range. When the exposure time is greater than the maximum value in the predetermined time range, the processor 21 can be used to acquire a reference image and perform portrait super-resolution algorithm processing on the image to be repaired according to the reference image to obtain a repaired image. Wherein, the resolution of the reference image is higher than the predetermined resolution.

Specifically, the plurality of exposure duration intervals are all within a predetermined duration range. For example, assuming that there are three exposure time intervals, namely the exposure time interval [T1, T2), the exposure time interval [T2, T3), and the exposure time interval [T3, T4], the predetermined time length range may be [T1, T4] . When the exposure time of the image to be repaired is within the predetermined time range [T1, T4], the processor 21 may use a repair model corresponding to the exposure time to repair the image to be repaired. When the exposure time of the image to be repaired is outside the predetermined time range [T1, T4], the blur of the image to be repaired may be high, and the details of the portrait are less obvious. At this time, it may not be possible to use the repair model to repair the image to be repaired achieve better restoration results. Therefore, the processor 21 can use the reference The image to be repaired is repaired to obtain a repaired image with better repair effect. Wherein, the reference image may include a preset user portrait or a preset standard portrait. Taking the electronic device 20 as a mobile phone as an example, the preset user portrait can be a portrait taken by the user in the electronic device 20 in advance. Tall images with portraits. When there is no preset user portrait in the electronic device 20, a preset standard portrait can be obtained, and any high-definition portrait in the same region as the user can be downloaded from the network, such as a high-definition poster. Both the preset user portrait and the preset standard portrait have a resolution higher than the predetermined resolution, and the predetermined resolution can be set in advance, and only images with a higher than the predetermined resolution can be used as reference images (preset user portrait or preset standard portrait) , in order to achieve better image processing effect.

Please refer to Fig. 14, in some embodiments, step 08 acquires a reference image, including:

081: Perform face detection on the portrait area of the image to be repaired and the preset user portrait;

082: Determine whether the similarity between the face of the image to be repaired and the preset user's face is greater than or equal to the first preset similarity;

083: When the similarity between the face of the image to be repaired and the face of the preset user is greater than or equal to the first preset similarity, use the preset user portrait as a reference image;

084: When the similarity between the face of the image to be repaired and the preset user's face is less than the first preset similarity, acquire a preset standard portrait as a reference image.

Please refer to FIG. 15 , in some implementations, the third acquisition module 18 includes a second detection unit 181 , a judgment unit 182 , a determination unit 183 , and a first acquisition unit 184 . Step 081 can be implemented by the second detection unit 181 . Step 082 can be implemented by the judging unit 182 . Step 083 can be implemented by the determination unit 183 . Step 084 may be implemented by the first acquiring unit 184 . That is to say, the second detection unit 181 may be used to perform face detection on the portrait area of the image to be repaired and the preset user portrait. The judging unit 182 may be configured to judge whether the similarity between the face of the image to be repaired and the preset user's face is greater than or equal to the first preset similarity. The determining unit 183 may be configured to use the preset user portrait as the reference image when the similarity between the face of the image to be repaired and the preset user's face is greater than or equal to the first preset similarity. The first acquiring unit 184 may be configured to acquire a preset standard portrait as a reference image when the similarity between the face of the image to be repaired and the preset user's face is less than the first preset similarity.

Please refer to FIG. 3 again. In some embodiments, step 081 , step 082 , step 083 and step 084 can all be implemented by the processor 21 . That is to say, the processor 21 can be used to perform face detection on the portrait area of the image to be repaired and the portrait of the preset user and determine whether the similarity between the face of the image to be repaired and the face of the preset user is greater than or equal to the first Default similarity. The processor 21 may also be configured to use the preset user portrait as a reference image when the similarity between the face of the image to be repaired and the preset user's face is greater than or equal to the first preset similarity. The processor 21 may also be configured to obtain a preset standard portrait as a reference image when the similarity between the face of the image to be repaired and the preset user's face is less than a first preset similarity.

Specifically, the processor 21 may detect the face of the image to be repaired and the portrait of the preset user. For example, the face detection model shown in FIG. 10 may be used to detect the face of the image to be repaired and the portrait of the preset user. Subsequently, the processor 21 may further detect the facial feature points in the image to be repaired and the facial feature points in the preset user portrait. Subsequently, the processor 21 compares the facial feature points of the two images. If the similarity of the face feature points of the two images is greater than the first preset similarity, it means that the portrait area of the image to be repaired and the preset user portrait are the same person (that is, the actual user corresponding to the portrait in the image to be repaired is the same as the preset user). The user is the same person), at this time, the processor 21 may perform portrait super-resolution algorithm (ie ultra-clear portrait algorithm) processing on the portrait area of the image to be repaired according to the preset user portrait to obtain the repaired image. Using two images of the same person for processing, the portrait in the repaired image obtained is more similar and more natural to the actual user, and the user experience will be better. If the similarity of the face feature points of the two images is lower than the first preset similarity, it means that the portrait area of the image to be repaired is not the same person as the preset user portrait (that is, the actual user is not the same person as the preset user), At this time, the standard portrait is used as the reference image for super-resolution algorithm processing, and the effect obtained will be better. Therefore, the processor 21 may perform portrait super-resolution algorithm processing on the portrait area of the image to be repaired according to the preset standard portrait to obtain the repaired image.

Please refer to FIG. 16. In some embodiments, step 09 performs portrait super-resolution algorithm processing on the image to be repaired according to the reference image to obtain the repaired image, including:

091: Obtain the first feature map of the upsampled image to be repaired;

092: Obtain the second feature map of the reference image after upsampling and downsampling;

093: Obtain the third feature map of the reference image without upsampling and downsampling;

094: Acquiring features in the second feature map whose similarity to the first feature map exceeds the second preset similarity as reference features;

095: Obtain features whose similarity with the reference feature in the second feature map exceeds the third preset similarity to obtain an exchange feature map;

096: Merge the exchange feature map and the first feature map to obtain the fourth feature map;

097: Enlarge the fourth feature map by a predetermined factor to obtain the fifth feature map; and

098: Use the fifth feature map as the image to be repaired and execute the above steps in a loop until the obtained fifth feature map is the target magnification, then the fifth feature map with the target magnification is the repair image.

Referring to FIG. 17 , in some implementations, the processing module 19 includes a second acquiring unit 191, a third acquiring unit 192, a fourth acquiring unit 193, a fifth acquiring unit 194, a sixth acquiring unit 195, a combining unit 196, an amplification unit 197 , and a third processing unit 198 . Step 091 may be implemented by the second acquiring unit 191 . Step 092 may be implemented by the third obtaining unit 192 . Step 093 may be implemented by the fourth acquiring unit 193 . Step 094 may be implemented by the fifth acquiring unit 194 . Step 095 can be implemented by the sixth acquiring unit 195 . Step 096 can be implemented by the merging unit 196 . Step 097 can be implemented by the amplification unit 197 . Step 098 may be implemented by the third processing unit 198 . That is to say, the second acquiring unit 181 may be configured to acquire the upsampled first feature map of the image to be repaired. The third acquiring unit 192 may be configured to acquire the second feature map of the reference image after upsampling and downsampling. The fourth acquiring unit 193 may be configured to acquire a third feature map of the reference image that has not been up-sampled or down-sampled. The fifth acquiring unit 194 may be configured to acquire features in the second feature map whose similarity to the first feature map exceeds a second preset similarity as reference features. The sixth obtaining unit 195 may be configured to obtain features in the second feature map whose similarity to the reference feature exceeds a third preset similarity, so as to obtain the exchange feature map. The merging unit 196 can be used for merging the exchanged feature map and the first feature map to obtain a fourth feature map. The enlarging unit 197 may be used to amplify the fourth feature map by a predetermined factor to obtain a fifth feature map. The third processing unit 198 may be configured to use the fifth feature map as the image to be repaired and execute the above steps in a loop until the obtained fifth feature map is the target magnification, then the fifth feature map with the target magnification is the repair image.

Please refer to FIG. 3 again. In some embodiments, step 091 , step 092 , step 093 , step 094 , step 095 , step 096 , step 097 and step 098 can all be implemented by the processor 21 . That is to say, the processor 21 can be used to obtain the first feature map of the image to be repaired after being up-sampled, obtain the second feature map of the reference image after up-sampling and down-sampling, and obtain the reference image without up-sampling and Downsampled third feature map. The processor 21 can also be used to obtain features in the second feature map whose similarity to the first feature map exceeds a second preset similarity as reference features, and obtain features in the second feature map whose similarity to the reference feature exceeds a third preset Similarity features to get the exchange feature map. The processor 21 can also be used for merging the swapped feature map and the first feature map to obtain a fourth feature map and amplifying the fourth feature map by a predetermined factor to obtain a fifth feature map. The processor 21 may also be configured to use the fifth feature map as the image to be repaired and execute the above steps in a loop until the obtained fifth feature map is the target magnification, then the fifth feature map with the target magnification is the repair image.

Specifically, upsampling can be understood as enlarging the image to be repaired or a reference image, and downsampling can be understood as reducing the reference image.

More specifically, please refer to FIG. 19. In some implementations, step 091 acquires the upsampled first feature map of the image to be repaired, including:

0911, upsampling the image to be repaired;

0912, input the upsampled image to be repaired into the convolutional neural network for feature extraction, and obtain the first feature map;

Step 092 obtains the second feature map of the reference image after upsampling and downsampling, including:

0921. Downsample the reference image;

0922. Upsample the downsampled reference image;

0923, input the up-sampled reference image to the convolutional neural network for feature extraction, and obtain the second feature map;

Step 093 obtains the third feature map of the reference image without upsampling and downsampling, including:

0931. Input the reference image to the convolutional neural network for feature extraction to obtain a third feature map.

The processor 21 may perform upsampling (enlargement) processing on the image to be repaired, and then input the upsampled image to be repaired into a convolutional neural network for feature extraction to obtain a first feature map. The first feature map can be understood as an enlarged image of the portrait area in the image to be repaired. The first feature map includes various features of the portrait, such as facial features, skin texture, hair, outline, and so on. Since the first feature map is obtained by processing the enlarged image to be repaired, the definition of the first feature map is relatively low, while the definition of the reference image is relatively high, so the processor 21 also needs to perform the following steps on the reference image. Sampling (shrinking), and then upsampling the downsampled image to achieve blurring of the reference image and improve the similarity between the second feature map and the first feature map. Features such as facial features, skin texture, hair, and contour may also be included in the second feature map. In addition, the processor 21 also needs to input the reference image to the convolutional neural network for feature extraction to obtain a third feature map. It should be noted that the convolutional neural network is a deep learning network that can extract features of the input image with high accuracy.

Subsequently, the processor 21 compares the features in the second feature map with the features in the first feature map, judges the similarity between them, and compares the similarity with a second preset similarity. If the similarity is greater than or equal to the second preset similarity, it means that the feature in the second feature map is very similar to the corresponding feature in the first feature map, and the processor 21 can use the feature in the second feature map as a reference feature. The processor 21 then compares the third feature map with the reference feature, judges the similarity between the two, and compares the similarity with the third preset similarity, if the similarity is greater than or equal to the third preset similarity, then Get the corresponding exchange feature map. Subsequently, the processor 21 merges the exchange feature map with the first feature map to obtain a fourth feature map, and then enlarges the fourth feature map by a predetermined factor to obtain a fifth feature map. Subsequently, the processor 21 judges the magnification factor of the fifth characteristic map. If the magnification is equal to the target magnification, the processor 21 uses the fifth feature map as the inpainted image. It should be noted that the second preset similarity and the third preset similarity may be the same as the first preset similarity above.

In some implementations, when the exposure duration is greater than the maximum value in the predetermined duration range and less than or equal to the predetermined value, the processor 21 may perform portrait super-resolution processing on the image to be repaired according to the reference image to obtain the repaired image. When the exposure time is longer than the predetermined value, it means that the blur degree of the image to be repaired is very high. At this time, neither the repair model shown in Figure 11 is used to repair the image to be repaired nor the reference image is used to repair the image with repair. processing effect. Therefore, at this time, the processor 21 does not perform restoration processing on the image to be repaired. When the exposure time is less than the minimum value in the predetermined time range, it means that the image to be repaired has less blur and higher definition, and at this time, the processor 21 does not perform repair processing on the image to be repaired. In this way, the processor 21 processes the image to be repaired only when the exposure time is greater than or equal to the minimum value in the predetermined time range and less than or equal to the predetermined value, which can avoid power consumption loss caused by unnecessary repair processing, and is conducive to improving electronic equipment. 20 endurance.

Referring to FIG. 19 , the present application also provides a non-volatile computer-readable storage medium 30 . The non-transitory computer readable storage medium 30 contains computer readable instructions. When the computer-readable instructions are executed by the processor 21, the processor 21 is made to execute the image processing method described in any one of the above-mentioned implementation manners.

For example, please refer to FIG. 1 and FIG. 19 , when the computer readable instructions are executed by the processor 21, the processor 21 is made to perform the following steps of the image processing method:

01: Obtain the exposure time of the image to be repaired when shooting;

02: Determine the exposure duration interval of the exposure duration;

03: Determine the repair model according to the exposure time interval, and different exposure time intervals correspond to different restoration models; and

04: Repair the portrait area of the image to be repaired according to the determined repair model.

For another example, please refer to FIG. 6 and FIG. 19 , when the computer readable instructions are executed by the processor 21, the processor 21 is made to perform the following steps of the image processing method:

041: Detect the portrait area in the image to be repaired;

042: Perform multiple convolutions on the portrait area to obtain multiple feature images;

043: Perform multiple upsampling and at least one deconvolution on the feature image output by the last convolution to obtain a residual image; and

044: Fusion residual image and portrait area to obtain repaired image.

The non-volatile computer-readable storage medium 30 can be set in the image processing apparatus 10 (shown in FIG. 2 ) or the electronic device 20 (shown in FIG. 3 ), and can also be set in the cloud server. When the non-volatile computer-readable storage medium 30 is set in the cloud server, the image processing apparatus 10 or the electronic device 20 can communicate with the cloud server to obtain corresponding computer-readable instructions.

It will be appreciated that computer readable instructions comprise computer program code. The computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The non-volatile computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory ), random access memory (RAM, Random Access Memory), and software distribution media, etc.

The processor 21 may refer to a driver board. The driver board can be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gates Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

In the description of this specification, reference to the terms "one embodiment", "certain embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" etc. The specific features, structures, materials, or characteristics described in the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which shall It should be understood by those skilled in the art to which the embodiments of the present application belong.

Although the implementation of the present application has been shown and described above, it can be understood that the above-mentioned implementation is exemplary and should not be construed as limiting the application, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (9)

1. an image processing method, is characterized in that, described image processing method comprises: Obtain the exposure time of the image to be repaired when it is taken; determining the exposure duration interval in which the exposure duration is located; determining a restoration model according to the exposure time interval, and different exposure duration intervals correspond to different restoration models; and performing repair processing on the portrait area of the image to be repaired according to the determined repair model; The image processing method also includes: Create an initial model; acquiring multiple frames of training images whose exposure durations are located in different exposure duration intervals; and The initial model is trained by using multiple frames of the training images whose exposure durations are within the same exposure duration interval, so as to obtain the restoration model corresponding to the exposure duration interval. 2. The image processing method according to claim 1, wherein said repairing the portrait region of the image to be repaired according to the determined repair model includes: Detecting the portrait area in the image to be repaired; Perform multiple convolutions on the portrait area to obtain multiple feature images; performing multiple upsampling and at least one deconvolution on the feature image output by the last convolution to obtain a residual image; and The residual image and the portrait area are fused to obtain a repaired image. 3. The image processing method according to claim 2, wherein said performing multiple upsampling and at least one deconvolution on the output feature image after the last convolution to obtain a residual image, comprising: In the first upsampling process, perform upsampling and deconvolution on the feature image output by the last convolution; In the second or more upsampling process, fuse the image obtained by the previous upsampling, the feature image corresponding to the size of the image obtained by the previous upsampling, and the previous N times of deconvolution The image obtained by the multiplication, and perform upsampling or perform upsampling and deconvolution on the fused image; The image obtained by the last upsampling and the image obtained by the previous N times of deconvolution are fused to obtain the residual image, where N≥1, and N∈N+. 4. image processing method according to claim 1, is characterized in that, described image processing method also comprises: judging whether the exposure duration is within a predetermined duration range; When the exposure duration is within the predetermined duration range, performing the step of determining the exposure duration interval in which the exposure duration is located; When the exposure duration is longer than the maximum value in the predetermined duration range, acquiring a reference image, the resolution of the reference image is higher than the predetermined resolution; and Perform portrait super-resolution algorithm processing on the image to be repaired according to the reference image to obtain a repaired image. 5. The image processing method according to claim 4, wherein said obtaining a reference image comprises: Perform face detection on the portrait area of the image to be repaired and the preset user portrait; When the similarity between the face of the image to be repaired and the face of the preset user is greater than or equal to a first preset similarity, using the preset user portrait as the reference image; When the similarity between the face of the image to be repaired and the preset user's face is smaller than the first preset similarity, a preset standard portrait is acquired as the reference image. 6. The image processing method according to claim 4, wherein said performing portrait super-resolution algorithm processing on said image to be repaired according to said reference image to obtain a repaired image comprises: Obtaining the first feature map of the upsampled image to be repaired; Obtaining a second feature map of the reference image after upsampling and downsampling; Obtaining a third feature map of the reference image; Obtaining a feature in the second feature map whose similarity to the first feature map exceeds a second preset similarity as a reference feature; Obtaining features in the second feature map whose similarity to the reference feature exceeds a third preset similarity to obtain an exchange feature map; merging the exchanged feature map with the first feature map to obtain a fourth feature map; amplifying the fourth characteristic map by a predetermined factor to obtain a fifth characteristic map; and Using the fifth feature map as the image to be repaired and performing the above steps in a loop until the fifth feature map obtained is the target magnification, then the fifth feature map with the target magnification is the Fix the image. 7. An image processing device for electronic equipment, characterized in that it comprises: An acquisition module, configured to acquire the exposure time of the image to be repaired when it is taken; A first determining module, configured to determine the exposure duration interval in which the exposure duration is located; The second determining module is configured to determine a repair model according to the exposure time interval, and different exposure time intervals correspond to different repair models; and An inpainting module, configured to inpaint the portrait area of the image to be inpainted according to the determined inpainting model; Build modules for building initial models; The second acquisition module is used to acquire multiple frames of training images whose exposure durations are located in different exposure duration intervals; A training module, configured to use multiple frames of the training images whose exposure durations are within the same exposure duration interval to train the initial model, so as to obtain the restoration model corresponding to the exposure duration interval. 8. An electronic device, characterized in that it comprises: casing; and A processor, the processor is installed on the casing, and the processor is used to implement the image processing method described in any one of claims 1-6. 9. A non-volatile computer-readable storage medium containing computer-readable instructions, characterized in that, when the computer-readable instructions are executed by a processor, the processor performs any one of claims 1-6 The image processing method described.