CN116567397A - Image processing method and device - Google Patents

Abstract

The application discloses an image processing method and device. The method comprises the steps of obtaining a first image, wherein the first image is an image processed by an image signal processor ISP, performing inverse processing corresponding to a target image processing process on the first image by utilizing an image inverse processing model to obtain a second image, wherein the target image processing process comprises other image processing processes except a high dynamic range imaging HDR processing process in the ISP processing process, and performing fuzzy processing on a local area in the second image by utilizing the HDR inverse processing model to obtain a third image. According to the image processing method, the simulation process of the ISP can be simplified, and the simulation effect is improved.

Description

Image processing method and device

technical field

The present application belongs to the technical field of image processing, and in particular relates to an image processing method and device.

Background technique

Image sensors are an important part of cameras. The image sensor can collect an optical image and convert the optical image into an image signal usable by a back-end image signal processor (Image Signal Processor, ISP). ISP can perform high dynamic range imaging (High Dynamic Range, HDR), automatic white balance (Auto white balance, AWB), color correction (Color correctionMatrix, CCM) and Gamma correction on the image signal output by the front-end image sensor (that is, the input image signal) And so on, in order to better restore the scene details of the camera under different optical conditions, and obtain the target image.

In the prior art, in order to simulate the input image of the ISP, the inverse processing corresponding to the above-mentioned ISP processing is usually performed on the target image, that is, inverse Gamma transformation, inverse CCM color correction, inverse white balance and other processing are performed on the target image , to get the input image before inputting to the ISP. In addition, inverse processing corresponding to HDR cannot be realized at present.

However, the current inverse processing corresponding to the ISP processing is complicated, and the accuracy of the obtained input image is low. That is to say, the inverse processing simulation process of the ISP in the prior art is complicated, and the inverse processing simulation effect is poor.

Contents of the invention

Embodiments of the present application provide an image processing method, device, device, computer-readable storage medium, and computer program product, which can simplify the ISP inverse processing simulation process and improve the simulation effect.

In the first aspect, the embodiment of the present application provides an image processing method, the method comprising:

acquiring a first image, where the first image is an image processed by an image signal processor ISP,

Using an image inverse processing model to perform inverse processing on the first image corresponding to the target image processing process to obtain a second image, wherein the target image processing process includes a high dynamic range imaging HDR processing process in the ISP processing process In addition to other image processing processes,

The local area in the second image is blurred by using the HDR inverse processing model to obtain a third image.

In a possible implementation manner, the inverse processing corresponding to the target image processing process is performed on the first image by using the image inverse processing model, and before obtaining the second image, the method further includes:

Acquiring a plurality of first image samples after ISP processing and corresponding second image samples before ISP processing,

performing feature extraction on the first image sample and the second image sample to obtain a first image feature corresponding to each of the first image samples, and a first image feature corresponding to each of the second image samples Two image features,

performing inverse processing corresponding to the target image processing process on the first image features by using an initial image inverse processing model to obtain first predicted image features respectively corresponding to each of the first image features,

determining a first loss function value according to the similarities between the plurality of second image features and their respective corresponding first predicted image features,

Adjusting model parameters of the initial image inverse processing model according to the first loss function value, and training to obtain the image inverse processing model.

In a possible implementation manner, the determining the first loss function value according to the similarities between the multiple second image features and their respective corresponding first predicted image features includes:

calculating the mean square error between a plurality of said first predicted image features and a plurality of said second image features,

The mean square error is determined as the first loss function value.

In a possible implementation manner, before using the HDR inverse processing model to perform blurring processing on the local area in the second image, and obtaining the third image, the method further includes:

Acquiring a plurality of third image samples after HDR processing and corresponding fourth image samples before HDR processing,

Using the initial HDR inverse processing model to perform blurring processing on the local area of each of the third image samples to obtain a second predicted image corresponding to each of the third image samples,

determining a second loss function value according to the similarities between the plurality of fourth image samples and their respective corresponding second prediction images,

Adjusting model parameters of the initial HDR inverse processing model according to the second loss function value, and training to obtain the HDR inverse processing model.

In a possible implementation manner, the acquiring a plurality of HDR-processed third image samples includes:

acquire a plurality of temporally consecutive fourth images,

Using the image inverse processing model to respectively perform inverse processing corresponding to the target image processing process on a plurality of the fourth images to obtain a plurality of fifth images,

Acquiring three consecutive frames of third image samples in the plurality of fifth images.

In a possible implementation, the initial HDR inverse processing model is used to perform blurring processing on the local area of each of the third image samples to obtain a second prediction corresponding to each of the third image samples. images, including:

Using the initial HDR inverse processing model to extract the associated features between the three consecutive frames of the third image sample,

According to the associated features, the partial area of the target image sample is blurred to obtain a second predicted image corresponding to the target image sample, and the target image sample is located in the middle of the three consecutive frames of the third image sample. image sample.

In a possible implementation manner, the blurring the local area of the target image sample according to the associated feature to obtain the second predicted image corresponding to the target image sample includes:

determining the associated feature as a target image feature corresponding to the target image sample,

Using a random algorithm to process the features of the target image to obtain a second predicted image feature,

Determining an image corresponding to the feature of the second predicted image as the second predicted image.

In a possible implementation manner, the image inverse processing model is a four-layer convolutional neural network model, and the HDR inverse processing model is a bidirectional long-short-term memory recurrent neural network model.

In a possible implementation manner, the blurring processing includes any one of underexposure processing and overexposure processing.

In a second aspect, an embodiment of the present application provides an image processing device, which includes:

A first acquiring module, configured to acquire a first image, where the first image is an image processed by an image signal processor ISP,

The first processing module is configured to use an image inverse processing model to perform inverse processing on the first image corresponding to the target image processing process to obtain a second image, wherein the target image processing process includes removing Other image processing processes other than high dynamic range imaging HDR processing,

The second processing module is configured to use the HDR inverse processing model to perform blurring processing on the local area in the second image to obtain a third image.

In a third aspect, an embodiment of the present application provides an electronic device, the device includes: a processor and a memory storing computer program instructions,

When the processor executes the computer program instructions, the method in any possible implementation method of the first aspect above is implemented.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, any one of the possible implementations in the above-mentioned first aspect can be realized. Implement the method in the method.

In the fifth aspect, the embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device executes any one of the possible implementation methods in the first aspect above. Methods.

The image processing method, device, equipment, computer-readable storage medium, and computer program product of the embodiments of the present application, by using the image inverse processing model to perform inverse processing on the first image corresponding to the target image processing process, can obtain Processed second image. By using the HDR inverse processing model to perform blurring processing on the local area in the second image, inverse processing corresponding to the HDR processing process can be performed to obtain the third image. Since the first image is the image processed by ISP, by using the image inverse processing model and the HDR inverse processing model to process the image after ISP processing, the image before ISP processing can be obtained, which can simplify the simulation process of ISP and improve the simulation effect .

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the accompanying drawings that need to be used in the embodiments of the present application. Additional figures can be derived from these figures.

FIG. 1 is a schematic diagram of an ISP inverse processing principle provided in an embodiment of the present application.

Fig. 2 is a schematic flow chart of an image processing method provided by an embodiment of the present application,

Fig. 3 is a schematic diagram of an ISP inverse processing neural network model provided by an embodiment of the present application,

Fig. 4 is a schematic structural diagram of an image processing device provided by an embodiment of the present application,

Fig. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The characteristics and exemplary embodiments of various aspects of the application will be described in detail below. In order to make the purpose, technical solution and advantages of the application clearer, the application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only intended to explain the present application rather than limit the present application. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the statement "comprising..." does not exclude the presence of additional same elements in the process, method, article or device comprising said element.

As mentioned in the background section, the inverse processing corresponding to the ISP processing is complicated, and the accuracy of the obtained input image is low.

Specifically, as the requirements for image quality become higher and higher, the progress of HDR technology makes the display field develop to a new height. Most of the current ISPs have the function of HDR. If the inverse transformation process is still performed sequentially according to the ISP processing (ISP Pipeline) process, there are many defects. First of all, if the ISP contains the HDR function, HDR uses multi-frame fusion technology, which is to fuse multiple pictures with different exposure levels to obtain a high-dynamic and expressive picture. It is not easy to directly use inverse transformation Restore the image before fusion. Secondly, under different color temperature conditions, ISP will dynamically adjust the correction strength and white balance strength of the image, which means that the correction strength of white balance is not fixed, which increases the difficulty of inverse transformation. , Finally, multiple image processing modules are involved in the ISP processing flow, so that there is a series of complex linear and nonlinear superposition relationships between the input image and the output image, and the complex relationship is difficult to express through mathematics or a fixed model .

Therefore, for the inverse processing simulation of ISP, there is currently a lack of an effective simulation method that can take into account its dynamic changes. At the same time, for the ISP image processing unit with HDR function, there is also a lack of an inverse processing simulation method that can take into account the multi-frame fusion technology. As a result, the simulation process of the ISP is complicated and the simulation effect is poor.

Thus, in order to solve the problems in the prior art, the embodiments of the present application provide an image processing method, device, equipment, computer-readable storage medium, and computer program product.

Here, the image processing method may be a simulation method that uses a neural network to realize the inverse processing of the ISP, that is, the inverse processing process of each module in the ISP is replaced by the neural network. The simulation method may be to input an image acceptable to human eyes without noise or blemish into the ISP inverse processing neural network model. The ISP inverse processing neural network model can perform target image inverse processing (image inverse processing in ISP inverse processing except HDR inverse processing) and HDR inverse processing on the input image data. That is, the ISP inverse processing neural network model may include a neural network structure for performing target image inverse processing and a neural network structure for performing HDR inverse processing. After the ISP inverse processing neural network model can output the image before input to ISP, so as to achieve the purpose of simulating the input image of ISP. Based on this, a schematic diagram of an ISP inverse processing principle provided in the embodiment of the present application may be shown in FIG. 1 .

The image processing method provided by the embodiment of the present application is firstly introduced below.

FIG. 2 shows a schematic flowchart of an image processing method provided by an embodiment of the present application. As shown in FIG. 2 , the image processing method provided by the embodiment of the present application includes steps S210 to S230.

S210. Acquire a first image, where the first image is an image processed by an image signal processor (ISP).

Here, the first image may be an image without defects such as noise, underexposure, or overexposure. Specifically, the first image may be an image processed by the ISP. Among them, ISP processing can include HDR, AWB, CCM color correction and Gamma correction and other processing. Of course, the ISP processing may also include black battery correction (Black level correction, BLC), lens shading correction (Lens shading correction, LSC) and other processing, which is not limited here.

S220. Use the image inverse processing model to perform inverse processing on the first image corresponding to the target image processing process to obtain the second image, wherein the target image processing process includes other processes in the ISP processing process except the high dynamic range imaging HDR processing process Image processing process.

Here, the image inverse processing model may have a function of inverse processing the target image. Wherein, the target image reverse processing is the reverse processing corresponding to the target image processing process. The target image processing process may include other image processing processes in the ISP processing process except the HDR processing process. That is, the target image processing process may include processing processes such as AWB, CCM color correction, BLC, LSC, and Gamma correction. In this way, the second image may be an image with defects such as noise, underexposure, overexposure, and the like. In addition, the image inverse processing model can be a separate neural network model, or can be used as the first layer of neural network structure in the ISP inverse processing neural network model.

As an example, the image inverse processing model may be a four-layer convolutional neural network (ConvolutionalNeural Networks, CNN) model. On this basis, the first layer of neural network structure in the ISP inverse processing neural network model can be a four-layer convolutional neural network.

Based on this, in order to enable the image inverse processing model to have the function of performing target image inverse processing, in some embodiments, before S220, it may also include:

Acquiring a plurality of first image samples after ISP processing and corresponding second image samples before ISP processing,

performing feature extraction on the first image sample and the second image sample to obtain a first image feature corresponding to each first image sample and a second image feature corresponding to each second image sample,

Using the initial image inverse processing model to perform inverse processing corresponding to the target image processing process on the first image features to obtain first predicted image features corresponding to each first image feature,

determining a first loss function value according to the similarities between the plurality of second image features and their respective corresponding first predicted image features,

The model parameters of the initial image inverse processing model are adjusted according to the value of the first loss function, and the image inverse processing model is obtained through training.

Here, the first image sample processed by the ISP may be an image sample without defects such as noise, underexposure, and overexposure. The second image sample before ISP processing may be an image sample with defects such as noise, underexposure, and overexposure. Wherein, for the entire image, defects such as noise, underexposure, and overexposure may belong to a local image feature. In addition, the plurality of first image samples and their respective corresponding second image samples can be divided into a training group and a test group at a ratio of 7:3 for training the initial image inverse processing model to obtain an image inverse processing model. The second image samples may be training sample labels.

As an example, when the image inverse processing model is a four-layer convolutional neural network model, the convolutional neural network can be used to learn inverse processing features of other modules in the ISP except the HDR module. The convolutional neural network can use a 3*3 convolution kernel to perform convolution traversal on the first image sample and the second image sample to extract the image feature information corresponding to the first image sample and the second image sample respectively, and obtain the first image feature and the second image features. In addition, the Softmax activation function can be set between the neural networks of each layer. The activation function can map the input of the upper layer network to the output terminal as the data of the next layer network.

As an example, model parameters may include weight values, biases, and learning rates. In the initial case of model training, model parameters can be set randomly. In the process of convolution iteration, according to the first loss function value between the predicted value (the first predicted image feature) and the real value (the second image feature), the weight value and bias in the initial image inverse processing model The model parameters such as learning rate and learning rate are dynamically adjusted until the model converges, and the training process of the initial image inverse processing model can be completed, and the trained image inverse processing model can be obtained.

Based on this, in some embodiments, the determination of the first loss function value based on the similarity between the plurality of second image features and their respective corresponding first predicted image features may include:

calculating the mean square error between a plurality of first predicted image features and a plurality of second image features,

The mean square error is determined as the first loss function value.

That is, according to the mean square error between the predicted value (the first predicted image feature) and the real value (the second image feature), the model parameters such as weight value, bias and learning rate in the initial image inverse processing model can be dynamically adjusted . In this way, after multiple convolution iterations, the model convergence can be determined when the mean square error is the smallest, and the trained image inverse processing model can be obtained. At the same time, the model parameters such as weight, bias, and learning rate when the mean square error is the smallest can also be obtained.

In this way, the image inverse processing model is obtained through training based on a plurality of first image samples after ISP processing and their respective corresponding second image samples before ISP processing, so that the image inverse processing model can have the function of inverse processing of the target image.

S230. Use the HDR inverse processing model to perform blurring processing on the local area in the second image to obtain a third image.

Here, the HDR inverse processing model may have a function of performing HDR inverse processing. Wherein, the HDR inverse processing is inverse processing corresponding to the HDR processing process. Since the multi-frame fusion technology of HDR is to fuse pictures under different exposure levels, brighten the underexposed part and lower the overexposed part brightness, therefore, the blurring process of the local area in the second image can be specifically Underexplode or overexposure. Wherein, performing underexposure processing on the local area in the second image may be to reduce the brightness of the local area in the second image, and performing overexposure processing on the local area in the second image may be to increase the brightness of the local area in the second image. brightness. In this way, overexposure and underexposure of the image before HDR processing can be simulated. That is, the third image may be an image with defects such as noise, underexposure, and overexposure. In addition, the HDR inverse processing model can be a separate neural network model, or it can be used as a second-layer neural network structure in the ISP inverse processing neural network model.

As an example, the HDR inverse processing model may be a bidirectional long short term memory recurrent neural network (Bidirectional long short term memory, Bi-LSTM) model. On this basis, the second-layer neural network structure in the ISP reverse processing neural network model can be a bidirectional long-short-term memory recurrent neural network.

Based on this, in order to enable the HDR inverse processing model to have the function of performing HDR inverse processing, in some embodiments, before S230, it may also include:

Acquiring a plurality of third image samples after HDR processing and corresponding fourth image samples before HDR processing,

Using the initial HDR inverse processing model to perform blurring processing on the local area of each third image sample to obtain a second predicted image corresponding to each third image sample,

determining a second loss function value according to the similarities between the plurality of fourth image samples and their respective corresponding second prediction images,

The model parameters of the initial HDR inverse processing model are adjusted according to the value of the second loss function, and the HDR inverse processing model is obtained through training.

Here, both the third image sample after HDR processing and the fourth image sample before HDR processing may be image samples with defects such as noise, underexposure, and overexposure. The third image sample is an image sample obtained after HDR processing is performed on the corresponding fourth image sample. Wherein, the plurality of third image samples and their respective corresponding fourth image samples can be divided into a training group and a test group at a ratio of 7:3, so as to be used for training an initial HDR inverse processing model to obtain an HDR inverse processing model. The fourth image sample may be a training sample label. In addition, since the HDR module may be the first processing module in the ISP processing process, the fourth image sample before HDR processing and the second image sample before ISP processing may be the same image sample. Of course, the fourth image sample and the second image sample may also be different image samples, which is not limited here. In addition, the third image sample can be an image sample obtained after HDR processing of an image with defects such as noise, underexposure, or overexposure, or an image without defects such as noise, underexposure, or overexposure after an image inverse processing model The image samples obtained after processing are not limited here.

Based on this, in some embodiments, the acquisition of a plurality of HDR-processed third image samples may include:

acquire a plurality of temporally consecutive fourth images,

Using the image inverse processing model to respectively perform inverse processing corresponding to the target image processing process on the plurality of fourth images to obtain a plurality of fifth images,

Three consecutive frames of third image samples are acquired in the plurality of fifth images.

Here, the multiple temporally continuous fourth images may be multiple images in one video stream, or may be multiple images continuously captured for the same target, which is not limited here. In addition, the fourth image may be an image without defects such as noise, underexposure, or overexposure. The image inverse processing model may be an image inverse processing model that has been trained. After performing inverse processing corresponding to the target image processing process on the fourth image by using the image inverse processing model, a fifth image with defects such as noise, underexposure, and overexposure can be obtained. After the fifth image is obtained, each fifth image may be determined as a third image sample, or three consecutive frames of third image samples may be obtained from multiple fifth images.

In addition, as an example, the HDR inverse processing model can be used to simulate the HDR function in the ISP. In the initial case of model training, model parameters can be set randomly. During the model training process, the model parameters in the initial HDR inverse processing model are dynamically adjusted according to the second loss function value between the predicted value (the second predicted image) and the real value (the fourth image sample) until the model converges , the training process of the initial HDR inverse processing model can be completed, and the trained HDR inverse processing model can be obtained. Wherein, the predicted value may specifically be a second predicted image feature corresponding to the second predicted image, and the real value may specifically be a fourth image feature corresponding to the fourth image sample.

In this way, the HDR inverse processing model is obtained through training based on a plurality of third image samples after HDR processing and their respective corresponding fourth image samples before HDR processing, so that the HDR inverse processing model can have the function of performing HDR inverse processing.

Based on this, in order to simulate the HDR function in the ISP, in some embodiments, the local area of each third image sample is respectively blurred by using the initial HDR inverse processing model to obtain the corresponding The second predicted image may include:

Using the initial HDR inverse processing model to extract the associated features between the three consecutive frames of the third image sample,

Blurring is performed on the local area of the target image sample according to the associated feature to obtain a second prediction image corresponding to the target image sample, and the target image sample is an image sample in the middle of three consecutive frames of third image samples.

Here, the initial HDR inverse processing model may be a bidirectional long-short-term memory recurrent neural network, and the bidirectional long-short-term memory recurrent neural network may have a function of extracting correlation features between multiple images. If what is extracted is the correlation feature between the third image samples of three consecutive frames, then the correlation feature may include the feature information corresponding to the current frame image, the feature information corresponding to the previous frame image of the current frame image, and the feature information corresponding to the current frame image The feature information corresponding to the next frame of the image. Here, the current frame image may be a target image sample at an intermediate position among three consecutive frames of third image samples. Since the target image features include the feature information corresponding to the previous frame image sample of the target image sample and the feature information corresponding to the next frame image sample of the target image sample, therefore, by blurring the local area of the target image sample, Ability to emulate the HDR function in the ISP.

Based on this, in order to simulate the overexposure and underexposure of the image before HDR processing, in some embodiments, the local area of the target image sample is blurred according to the above-mentioned associated features to obtain the second predicted image corresponding to the target image sample, which can be include:

determining the associated feature as the target image feature corresponding to the target image sample,

Using a random algorithm to process the target image features to obtain the second predicted image features,

An image corresponding to the feature of the second predicted image is determined as the second predicted image.

Here, using the random algorithm to process the target image feature may be to multiply the weight matrix with the random seed set in the Bi-LSTM and the feature matrix corresponding to the target image feature. After the target image feature is processed by the random algorithm, the local RGB value of the target image feature can be reduced overall, or the local RGB value of the target image feature can be increased overall to obtain the second predicted image feature. Among them, reducing the RGB value of the local range of the target image feature as a whole can be to perform underexplosive processing on the local area of the target image sample, and improving the RGB value of the local range of the target image feature as a whole can be to perform on the local area of the target image sample. Overexposure.

In this way, by using a random algorithm to process the characteristics of the target image, it is possible to simulate the overexposure and underexposure of the image before HDR processing.

In the image processing method of the embodiment of the present application, by using the image inverse processing model to perform inverse processing corresponding to the target image processing process on the first image, a second image that has not undergone HDR inverse processing can be obtained. By using the HDR inverse processing model to perform blurring processing on the local area in the second image, inverse processing corresponding to the HDR processing process can be performed to obtain the third image. Since the first image is the image processed by ISP, by using the image inverse processing model and the HDR inverse processing model to process the image after ISP processing, the image before ISP processing can be obtained, which can simplify the simulation process of ISP and improve the simulation effect .

Based on this, in order to better describe the whole solution, a specific example is given based on the foregoing embodiments.

For example, as shown in FIG. 3 , the ISP inverse processing neural network model may have a two-layer neural network structure. Wherein, the neural network structure of the first layer may be CNN, and the neural network structure of the second layer may be Bi-LSTM.

In this way, by inputting images without noise, underexposure, overexposure, etc. into the ISP inverse processing neural network model, images with noise, underexposure, overexposure, etc. for the purpose of simulation. Since the ISP inverse processing neural network model can automate the ISP inverse processing simulation process, it can reduce complexity and improve simulation accuracy compared to sequential inverse transformation according to the ISP Piepline. At the same time, the combination of CNN and Bi-LSTM network can make the simulation process compatible with the HDR inverse processing process. In this way, the simulation process of the ISP can be simplified and the simulation effect can be improved.

Based on the image processing method provided by the foregoing embodiments, correspondingly, the present application also provides a specific implementation manner of an image processing apparatus. See the examples below.

As shown in FIG. 4, the image processing device 400 provided in the embodiment of the present application includes the following modules:

The first acquiring module 410 is configured to acquire a first image, where the first image is an image processed by an image signal processor ISP,

The first processing module 420 is configured to use the image inverse processing model to perform inverse processing on the first image corresponding to the target image processing process to obtain the second image, wherein the target image processing process includes removing high dynamic range imaging HDR during the ISP processing process Image processing other than processing,

The second processing module 430 is configured to use the HDR inverse processing model to perform blurring processing on the local area in the second image to obtain a third image.

The above-mentioned image processing device 400 will be described in detail below, specifically as follows:

In some of these embodiments, the image processing device 400 may further include:

The second acquisition module is used to acquire a plurality of ISP-processed first image samples and their respective corresponding ones before performing inverse processing corresponding to the target image processing process on the first image by using the image inverse processing model to obtain the second image The second image sample before ISP processing,

A feature extraction module, configured to perform feature extraction on the first image sample and the second image sample, to obtain a first image feature corresponding to each first image sample, and a second image corresponding to each second image sample feature,

The third processing module is used to use the initial image inverse processing model to perform inverse processing on the first image features corresponding to the target image processing process, to obtain the first predicted image features respectively corresponding to each first image feature,

The first determination module is configured to determine a first loss function value according to the similarity between a plurality of second image features and their respective corresponding first predicted image features,

The first training module is configured to adjust the model parameters of the initial image inverse processing model according to the value of the first loss function, and obtain the image inverse processing model through training.

In some of these embodiments, the above-mentioned first determination module may include:

Calculation sub-module, used to calculate the mean square error between a plurality of first predicted image features and a plurality of second image features,

A determination submodule is used to determine the mean square error as the first loss function value.

In some of these embodiments, the image processing device 400 may further include:

The third acquisition module is used to obtain a plurality of HDR-processed third image samples and their respective corresponding pre-HDR processing samples before using the HDR inverse processing model to perform blurring processing on the local area in the second image to obtain the third image The fourth image sample of ,

The fourth processing module is used to use the initial HDR inverse processing model to perform blurring processing on the local area of each third image sample to obtain a second predicted image respectively corresponding to each third image sample,

The second determination module is configured to determine a second loss function value according to the similarity between a plurality of fourth image samples and their respective corresponding second prediction images,

The second training module is configured to adjust the model parameters of the initial HDR inverse processing model according to the second loss function value, and obtain the HDR inverse processing model through training.

In some of these embodiments, the above-mentioned third acquisition module may include:

The first acquisition submodule is used to acquire a plurality of temporally continuous fourth images,

The first processing sub-module is used to use the image inverse processing model to respectively perform inverse processing corresponding to the target image processing process on a plurality of fourth images to obtain a plurality of fifth images,

The second acquisition sub-module is used to acquire three consecutive frames of third image samples in a plurality of fifth images.

In some of these embodiments, the fourth processing module may include:

The extraction sub-module is used to extract the associated features between the three consecutive frames of the third image sample by using the initial HDR inverse processing model,

The second processing sub-module is used to blur the local area of the target image sample according to the associated features to obtain a second predicted image corresponding to the target image sample, and the target image sample is in the middle of three consecutive frames of the third image sample image sample.

In some of these embodiments, the second processing submodule may include:

a first determining unit, configured to determine the associated feature as a target image feature corresponding to the target image sample,

A processing unit, configured to use a random algorithm to process the target image features to obtain the second predicted image features,

The second determining unit is configured to determine the image corresponding to the feature of the second predicted image as the second predicted image.

In some of the embodiments, the image inverse processing model is a four-layer convolutional neural network model, and the HDR inverse processing model is a bidirectional long-short-term memory recurrent neural network model.

In some of the embodiments, the blur processing includes any one of under-exposure processing and over-exposure processing.

The image processing apparatus according to the embodiment of the present application can obtain a second image that has not undergone HDR inverse processing by using an image inverse processing model to perform inverse processing corresponding to a target image processing process on the first image. By using the HDR inverse processing model to perform blurring processing on the local area in the second image, inverse processing corresponding to the HDR processing process can be performed to obtain the third image. Since the first image is the image processed by ISP, by using the image inverse processing model and the HDR inverse processing model to process the image after ISP processing, the image before ISP processing can be obtained, which can simplify the simulation process of ISP and improve the simulation effect .

Based on the image processing method provided in the foregoing embodiments, the embodiments of the present application also provide specific implementation manners of electronic devices. FIG. 5 shows a schematic diagram of an electronic device 500 provided by an embodiment of the present application.

The electronic device 500 may include a processor 510 and a memory 520 storing computer program instructions.

Specifically, the above-mentioned processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured to implement one or more integrated circuits in the embodiments of the present application.

Memory 520 may include mass storage for data or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (Hard Disk Drive, HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (Universal Serial Bus, USB) drive or two or more Combinations of multiple of the above. Storage 520 may include removable or non-removable (or fixed) media, where appropriate. Under appropriate circumstances, the storage 520 may be inside or outside the comprehensive gateway disaster recovery device. In a particular embodiment, memory 520 is a non-volatile solid-state memory.

Memory may include read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions, and when the software is executed (e.g., by one or a plurality of processors) operable to perform the operations described with reference to the method according to the first aspect of the present application.

The processor 510 reads and executes the computer program instructions stored in the memory 520 to implement any image processing method in the above-mentioned embodiments.

In one example, the electronic device 500 may further include a communication interface 530 and a bus 540 . Wherein, as shown in FIG. 5 , the processor 510 , the memory 520 , and the communication interface 530 are connected through a bus 540 to complete mutual communication.

The communication interface 530 is mainly used to implement communication between modules, devices, units and/or devices in the embodiments of the present application.

Bus 540 includes hardware, software, or both, and couples the components of the electronic device to each other. By way of example and not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), HyperTransport (HT) interconnect, Industry Standard Architecture (ISA) Bus, Infiniband Interconnect, Low Pin Count (LPC) Bus, Memory Bus, Micro Channel Architecture (MCA) Bus, Peripheral Component Interconnect (PCI) Bus, PCI-Express (PCI-X) Bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or a combination of two or more of these. Bus 540 may comprise one or more buses, where appropriate. Although the embodiments of this application describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

Exemplarily, the electronic device 500 may be a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a vehicle electronic device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA) wait.

The electronic device can execute the image processing method in the embodiment of the present application, so as to realize the image processing method and apparatus described in conjunction with FIG. 1 to FIG. 4 .

In addition, in combination with the image processing method in the foregoing embodiments, the embodiments of the present application may provide a computer-readable storage medium for implementation. Computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any one of the image processing methods in the above-mentioned embodiments is implemented.

It is to be understood that the application is not limited to the specific configurations and processes described above and shown in the figures. For conciseness, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art may make various changes, modifications and additions, or change the order of the steps after understanding the spirit of the present application.

The functional blocks shown in the structural block diagrams described above may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the present application are the programs or code segments employed to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted over transmission media or communication links by data signals carried in carrier waves. "Machine-readable medium" may include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. Code segments may be downloaded via a computer network such as the Internet, an Intranet, or the like.

It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps may be performed in the order mentioned in the embodiment, or may be different from the order in the embodiment, or several steps may be performed simultaneously.

Aspects of the present application are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each block of the flowchart and/or block diagrams, and combinations of blocks in the flowchart and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that execution of these instructions via the processor of the computer or other programmable data processing apparatus enables Implementation of the functions/actions specified in one or more blocks of the flowchart and/or block diagrams. Such processors may be, but are not limited to, general purpose processors, special purpose processors, application specific processors, or field programmable logic circuits. It can also be understood that each block in the block diagrams and/or flowcharts and combinations of blocks in the block diagrams and/or flowcharts can also be realized by dedicated hardware for performing specified functions or actions, or can be implemented by dedicated hardware and Combination of computer instructions to achieve.

The above is only a specific implementation of the present application, and those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described systems, modules and units can refer to the foregoing method embodiments The corresponding process in , will not be repeated here. It should be understood that the protection scope of the present application is not limited thereto, and any person familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the application, and these modifications or replacements should cover all Within the protection scope of this application.

Claims (10)

1. An image processing method, characterized in that, comprising: acquiring a first image, where the first image is an image processed by an image signal processor ISP, Using an image inverse processing model to perform inverse processing on the first image corresponding to the target image processing process to obtain a second image, wherein the target image processing process includes a high dynamic range imaging HDR processing process in the ISP processing process In addition to other image processing processes, The local area in the second image is blurred by using the HDR inverse processing model to obtain a third image. 2. The method according to claim 1, wherein the first image is subjected to inverse processing corresponding to the target image processing process using the image inverse processing model, and before the second image is obtained, the method further comprises : Acquiring a plurality of first image samples after ISP processing and corresponding second image samples before ISP processing, performing feature extraction on the first image sample and the second image sample to obtain a first image feature corresponding to each of the first image samples, and a first image feature corresponding to each of the second image samples Two image features, performing inverse processing corresponding to the target image processing process on the first image features by using an initial image inverse processing model to obtain first predicted image features respectively corresponding to each of the first image features, determining a first loss function value according to the similarities between the plurality of second image features and their respective corresponding first predicted image features, Adjusting model parameters of the initial image inverse processing model according to the first loss function value, and training to obtain the image inverse processing model. 3. The method according to claim 2, wherein the determining the first loss function value according to the similarity between a plurality of the second image features and their respective corresponding first predicted image features comprises: calculating the mean square error between a plurality of said first predicted image features and a plurality of said second image features, The mean square error is determined as the first loss function value. 4. The method according to claim 1, wherein the local area in the second image is blurred using the HDR inverse processing model, and before obtaining the third image, the method further comprises: Acquiring a plurality of third image samples after HDR processing and corresponding fourth image samples before HDR processing, Using the initial HDR inverse processing model to perform blurring processing on the local area of each of the third image samples to obtain a second predicted image corresponding to each of the third image samples, determining a second loss function value according to the similarities between the plurality of fourth image samples and their respective corresponding second prediction images, Adjusting model parameters of the initial HDR inverse processing model according to the second loss function value, and training to obtain the HDR inverse processing model. 5. The method according to claim 4, wherein said acquiring a plurality of HDR-processed third image samples comprises: acquire a plurality of temporally consecutive fourth images, Using the image inverse processing model to respectively perform inverse processing corresponding to the target image processing process on a plurality of the fourth images to obtain a plurality of fifth images, Acquiring three consecutive frames of third image samples in the plurality of fifth images. 6. The method according to claim 5, wherein the local area of each of the third image samples is respectively blurred by using the initial HDR inverse processing model to obtain an image corresponding to each of the third image samples The corresponding second prediction images respectively include: Using the initial HDR inverse processing model to extract the associated features between the three consecutive frames of the third image sample, According to the associated features, the partial area of the target image sample is blurred to obtain a second predicted image corresponding to the target image sample, and the target image sample is located in the middle of the three consecutive frames of the third image sample. image sample. 7. The method according to claim 6, wherein said blurring the local area of the target image sample according to the associated feature to obtain a second predicted image corresponding to the target image sample comprises: determining the associated feature as a target image feature corresponding to the target image sample, A random algorithm is used to process the features of the target image to obtain features of the second predicted image, and an image corresponding to the features of the second predicted image is determined as the second predicted image. 8. The method according to claim 1, wherein the image inverse processing model is a four-layer convolutional neural network model, and the HDR inverse processing model is a bidirectional long-short-term memory recurrent neural network model. 9. The method according to claim 1, wherein the blurring process includes any one of under-exposure processing and over-exposure processing. 10. An image processing device, characterized in that the device comprises: A first acquiring module, configured to acquire a first image, where the first image is an image processed by an image signal processor ISP, The first processing module is configured to use an image inverse processing model to perform inverse processing on the first image corresponding to the target image processing process to obtain a second image, wherein the target image processing process includes removing Other image processing processes other than high dynamic range imaging HDR processing, The second processing module is configured to use the HDR inverse processing model to perform blurring processing on the local area in the second image to obtain a third image.