CN112446835B - Image restoration method, image restoration network training method, device and storage medium - Google Patents

Abstract

The application provides an image restoration method, an image restoration network training device and a storage medium. Relates to the field of artificial intelligence, in particular to the field of computer vision. The method comprises the following steps: acquiring a plurality of training image pairs, wherein each training image pair comprises a real high-quality image and an approximate real low-quality image, and the approximate real low-quality image is obtained by synthesizing and realising the real high-quality image; then, training the image restoration network according to the plurality of training image pairs until an image restoration network with the image restoration performance meeting the requirement is obtained. Because each of the plurality of training image pairs includes an approximately true low quality image, an image restoration network with better image restoration performance can be trained from the plurality of training image pairs.

Description

Image restoration method, image restoration network training method, device and storage medium

Technical Field

The present application relates to the field of computer vision, and more specifically, to an image restoration method, an image restoration network training method, a device and a storage medium.

Background technique

Computer vision is an integral part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a discipline about how to use cameras/camcorders and computers to obtain the data and information we need about the objects being photographed. Figuratively speaking, it is to equip computers with eyes (cameras/camcorders) and brains (algorithms) to replace human eyes in identifying, tracking, and measuring targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

In the field of computer vision, for some degraded low-quality images, it is generally necessary to first restore these low-quality images to obtain high-quality images, and then process the restored high-quality images (for example, image classification, image recognition).

The traditional image restoration scheme mainly includes the following processes: first, obtain some high-quality images, and add blur or noise to these high-quality images to obtain synthetic low-quality images paired with the original high-quality images; second, train the neural network model based on these paired high-quality images and synthetic low-quality images to obtain a trained neural network model; finally, use the trained neural network model to restore the input low-quality image to obtain the corresponding high-quality image.

However, in real low-quality images, degradation factors such as low resolution, noise, and blur are usually coupled together. The synthetic low-quality images obtained by traditional solutions through artificial synthesis are still very different from real low-quality images. As a result, the neural network model trained by pairing high-quality images and synthetic low-quality images in traditional solutions cannot perform image restoration well, making the image restoration effect of traditional solutions mediocre.

Summary of the invention

The present application provides an image restoration network training method, an image restoration method, an apparatus and a storage medium to train an image restoration network with better image restoration performance.

In a first aspect, a method for training an image restoration network is provided, the method comprising: acquiring a plurality of training image pairs; and training the image restoration network according to the plurality of training image pairs until the image restoration performance of the image restoration network meets preset requirements.

Among them, each of the above-mentioned multiple training image pairs includes a real high-quality image and an approximate real low-quality image, the approximate real low-quality image in each training image pair is obtained by processing the real high-quality image in each training image pair, and the difference in image clarity between the approximate real low-quality image in each training image pair and the image clarity of the existing real low-quality image is within a preset range.

In addition, the above-mentioned real low-quality images and the images to be restored are acquired by using the same device, and the images to be restored are images processed by the trained image restoration network. In other words, the above-mentioned real low-quality images and the images to be restored processed by the trained image restoration network during subsequent image restoration are acquired by using the same device.

The above-mentioned image restoration network may also be called an image restoration model, and the image restoration network may be a neural network.

When the image restoration network is trained according to the above-mentioned multiple training image pairs, the image restoration network can be trained by using the training image pairs batch by batch. Each batch can use one or more training image pairs.

The same type of device mentioned above may refer to devices of completely the same device model, for example, cameras of the same model, terminal devices of the same model, cameras of the same model, and so on.

In the present application, since the approximate real low-quality images included in the training image pairs are obtained by synthesizing and realizing the real high-quality images, that is, the approximate real low-quality images included in the training image pairs are relatively close to the real low-quality images, therefore, according to the present application, the image restoration network is trained according to multiple training image pairs, and an image restoration network with better image restoration effect for the real low-quality images can be obtained. This enables the subsequent image restoration based on the trained image restoration network to have a better image restoration effect.

In combination with the first aspect, in certain implementations of the first aspect, the training of the image restoration network according to the plurality of training image pairs until the image restoration performance of the image restoration network meets preset requirements includes:

Step 1: Initialize the network parameters of the image restoration network to obtain the initial values of the network parameters of the image restoration network;

Step 2: inputting the approximate real low-quality image in at least one training image pair among the multiple training image pairs into the image restoration network for processing to obtain at least one restored high-quality image;

Step 3: determining a function value of a loss function according to a difference between at least one restored high-quality image and at least one true high-quality image in a training image pair;

Step 4: Update the network parameters of the image restoration network according to the function value of the loss function;

Repeat steps 2 to 4 above until the image restoration network meets the preset requirements.

In the above step 2, the network parameters of the image restoration network can be randomly initialized so that the network parameters of the image restoration network randomly take some values. In addition, in the above step 2, the image restoration network can also be preliminarily trained by using synthetic low-quality images and real high-quality images, and the network parameter values of the image restoration network obtained by the preliminary training are used as the initial values of the network parameters of the image restoration network.

The loss function in the above step 3 may also be referred to as an image loss function. When the difference between at least one restored high-quality image and the real high-quality image in at least one training image pair is greater, the function value of the image loss function is also greater, and when the difference between at least one restored high-quality image and the real high-quality image in at least one training image pair is smaller, the function value of the image loss function is also smaller.

In the above training process, the network parameters of the image restoration network can be changed in a direction that reduces the function value of the loss function.

Optionally, the image restoration network meets preset requirements, including: the image restoration network meets at least one of the following conditions:

(1) The image restoration performance of the image restoration network meets the preset performance requirements;

(2) The number of updates of the network parameters of the image restoration network is greater than or equal to a preset number;

(3) The function value of the loss function is less than or equal to the preset value.

Specifically, when the image restoration network satisfies at least one of the above conditions (1) to (3), the image restoration network meets the preset requirements and the training process of the image restoration network ends; and when the image restoration network does not meet any of the above conditions (1) to (3), it means that the image restoration network has not yet met the preset requirements and needs to continue to train the image restoration network, that is, it is necessary to repeat the above steps 2 to 4 until the image restoration network meets the preset requirements.

In combination with the first aspect, in certain implementations of the first aspect, the loss function includes a mean square error between at least one restored high-quality image and a true high-quality image in at least one training image pair.

The mean square error between two images refers to the mean of the sum of squares of the differences in pixel values at corresponding positions between the two images.

Optionally, the above-mentioned loss function is the average value of at least one first loss function value, and each first loss function in the at least one first loss function is the mean square error between each restored high-quality image in the at least one restored high-quality image and the corresponding true high-quality image.

In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned loss function also includes perceptual loss and adversarial loss of at least one restored high-quality image relative to a real high-quality image in at least one training image pair.

The perceptual loss between two images may refer to the mean of the sum of the squared norms of the differences between corresponding positions of the feature maps of the two images.

Adversarial loss is used to determine whether one image distribution is similar to another image distribution. Adversarial loss can be specifically described by a discriminative neural network.

Specifically, when the image distributions of two images are relatively similar, the two images are input into the discriminant neural network, and the result output by the discriminant neural network is that the image distributions of the two images are the same. When the image distributions of the two images are quite different, the two images are input into the discriminant neural network, and the result output by the discriminant neural network is that the image distributions of the two images are different.

When determining whether the image distributions of two images are the same or similar, a discriminative neural network can be used to make a judgment. Specifically, the discriminative neural network can perform feature extraction and feature transformation on the two images respectively, map them to a feature space, and then use a binary classifier to give the probability that an image is synthetic and the probability that it is real. When the classifier cannot distinguish whether an image is synthetic or real, that is, when the probability of an image being synthetic and the probability of being real output by the classifier are less than a certain threshold, it is considered that the two cannot be distinguished. At this time, the image distributions of the two images are the same.

When the above loss function includes multiple losses such as mean square error, perceptual loss and adversarial loss, the loss function reflects more comprehensive information. Therefore, using this loss function during training can train an image loss network with better image performance.

In combination with the first aspect, in certain implementations of the first aspect, the image clarity includes at least one of image blur degree, image noise distribution, and image resolution.

In combination with the first aspect, in certain implementations of the first aspect, the approximate real low-quality image in each of the above-mentioned training image pairs is obtained by processing the real high-quality image in each training image pair, including: the approximate real low-quality image in each training image pair is obtained by processing the real high-quality image in each training image pair using at least one of blurring processing, noise adding processing and downsampling processing.

In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned existing real low-quality image and the image to be restored are acquired by using the same device, including: the existing real low-quality image is acquired by using a first device, and the image to be restored is acquired by using a second device, the first device and the second device are of the same device type, and the image acquisition parameters of the first device are the same as the image acquisition parameters of the second device, and the image acquisition parameters include at least one of focal length, exposure, and shutter time.

When the existing real low-quality image and the image to be restored are acquired using the same type of equipment and the same image sampling parameters, the existing real low-quality image and the image to be restored are closer, so that the image restoration network trained with reference to the existing real low-quality image in this application has a better image restoration effect when processing the image to be restored.

In combination with the first aspect, in certain implementations of the first aspect, the approximate real low-quality image in each of the above-mentioned training image pairs is obtained by adjusting the real high-quality image in each of the above-mentioned training image pairs, and the above-mentioned adjustment processing is used to adjust the image clarity of the real high-quality image in each of the above-mentioned training image pairs, so that the image clarity of the approximate real low-quality image in each of the above-mentioned training image pairs is as similar as possible to the image clarity of the existing real low-quality image.

It should be understood that the image clarity of the approximate real low-quality image in each of the above-mentioned training image pairs is as similar as possible to the image clarity of the existing real low-quality image. Specifically, it may mean that the difference in image clarity between the approximate real low-quality image in each training image pair and the image clarity of the existing real low-quality image is within a preset range, and the preset range can be flexibly set according to actual needs.

It should also be understood that when obtaining an approximate true low-quality image in each training image pair, the image clarity of the true high-quality image in each of the above training image pairs can be adjusted so that the difference between the image clarity of the adjusted image and the clarity of the existing true low-quality image is minimized, and the adjusted image is the approximate true low-quality image in each of the above training images.

The above-mentioned adjustment processing of the real high-quality image in each training image pair can be directly blurring, adding noise and downsampling the real high-quality image in each training image pair, or using an image generation network to process the real high-quality image in each training image pair.

In combination with the first aspect, in certain implementations of the first aspect, the approximate real low-quality image in each of the above-mentioned training image pairs is obtained by processing the real high-quality image in each of the training image pairs using a pre-trained image generation network.

The above-mentioned image generation network can be used to convert a real high-quality image into an approximate real low-quality image. The image generation network can be obtained by training based on multiple real high-quality images and multiple real low-quality images. Specifically, during the training process, multiple real high-quality images can be input into the image generation network so that the difference between the image clarity of the output image and the image clarity of any one of the multiple real low-quality images is as small as possible. The training is terminated until the trained image generation network meets the preset requirements. The training termination condition can be flexibly set according to the actual situation, for example, when the image processing performance of the trained image generation network meets the preset requirements.

In addition, in the above-mentioned image generation network training process, multiple real high-quality images can also be synthesized first to obtain multiple synthetic low-quality images, and then the image generation network can be trained using the multiple synthetic low-quality images and the multiple real low-quality images.

In combination with the first aspect, in certain implementations of the first aspect, the difference in clarity between the image clarity of the approximate real low-quality image in each of the above-mentioned training image pairs and the image clarity of the existing real low-quality image is within a preset range, including: the distance between the feature vector of the approximate real low-quality image in each of the above-mentioned training image pairs and the feature vector of the existing real low-quality image is less than the preset distance.

The above preset distance can be flexibly set according to actual conditions.

The feature vector of the approximate real low-quality image in each of the above-mentioned training image pairs can be obtained by extracting features of the approximate real low-quality image in each of the above-mentioned training image pairs using a discriminant neural network (a type of neural network), and the feature vector of the above-mentioned existing real low-quality image can also be obtained by extracting features of the existing real low-quality image using the discriminant neural network.

When the discriminant neural network is used to extract the feature vectors of the two images, the same convolution parameters may be used for extraction, and the specific parameter values of the convolution parameters may be obtained by training the discriminant neural network.

Optionally, the obtaining of multiple training image pairs includes: determining the multiple training image pairs from an initial training image set, wherein the difference between the image clarity of the approximate real low-quality image in each training image pair in the multiple training image pairs and the image clarity of the existing real low-quality image is less than a preset threshold.

The above preset threshold can be flexibly set according to actual conditions. If the difference between image clarity during training is required to be strict, a smaller preset threshold can be set. If the difference between image clarity during training is required to be less strict, a larger preset threshold can be set.

By selecting a plurality of training images whose image clarity differences meet the requirements from the initial training image set for training, the training effect can be improved so that the image restoration network obtained after training has better image restoration performance.

In a second aspect, a method for training an image restoration network is provided, the method comprising the following steps:

Step A: Initializing the network parameters of the image generation network and the network parameters of the image restoration network to obtain initial values of the network parameters of the image generation network and the initial values of the network parameters of the image restoration network;

Repeat the following steps B to H until the image restoration performance of the image restoration network meets the preset requirements;

Step B: inputting at least one of the multiple synthetic low-quality images into a first generation network in the image generation network for processing to obtain at least one approximate real low-quality image;

Step C: inputting at least one real low-quality image among the multiple real low-quality images into the second generation network in the image generation network for processing to obtain at least one approximate synthetic low-quality image;

Step D: inputting at least one approximate real low-quality image into a second generation network for processing to obtain at least one reconstructed synthetic low-quality image;

Step E: inputting at least one approximately synthesized low-quality image into a first generation network for processing to obtain at least one reconstructed real low-quality image;

Step F: inputting at least one approximate real low-quality image into an image restoration network for processing to obtain at least one restored high-quality image;

Step G: Determine a loss function, where the loss function includes a first loss term, a second loss term, and a third loss term;

Step H: According to the function value of the loss function, the network parameters of the image generation network and the image restoration network are updated.

The multiple synthesized low-quality images in the above step B are obtained by synthesizing multiple real high-quality images respectively, and the synthesis processing includes at least one of blurring processing, noise adding processing and downsampling processing.

The multiple real low-quality images in the above step C and the image to be restored are acquired by using the same device, and the image to be restored is an image processed by the trained image restoration network.

The same type of device mentioned above may refer to devices of completely the same device model, for example, cameras of the same model, terminal devices of the same model, cameras of the same model, and so on.

The above-mentioned existing real low-quality image and the image to be restored are acquired by using the same device, specifically including: the above-mentioned existing real low-quality image is acquired by using a first device, and the image to be restored is acquired by using a second device, the first device and the second device are of the same device type, and the image acquisition parameters of the first device are the same as the image acquisition parameters of the second device, and the image acquisition parameters include at least one of focal length, exposure, and shutter time.

When the existing real low-quality image and the image to be restored are acquired using the same type of equipment and the same image sampling parameters, the existing real low-quality image and the image to be restored are closer, so that the image restoration network trained with reference to the existing real low-quality image in this application has a better image restoration effect when processing the image to be restored.

The first loss item, the second loss item and the third loss item are respectively:

The first loss function term:

The first loss function term includes an adversarial loss of at least one approximate real low-quality image relative to any one of the multiple real low-quality images, and an adversarial loss of at least one approximate synthetic low-quality image relative to any one of the multiple synthetic low-quality images;

The second loss function term:

The second loss function term includes a difference between a pixel value of at least one reconstructed synthetic low-quality image and a pixel value of at least one synthetic low-quality image, and a difference between a pixel value of at least one reconstructed real low-quality image and a pixel value of at least one real low-quality image;

The third loss function term:

The third loss function term includes a mean square error between at least one restored high-quality image and at least one real high-quality image among a plurality of real high-quality images, wherein at least one synthetic low-quality image is obtained by synthesizing at least one real high-quality image.

In the present application, by jointly training the image generation network and the image restoration network, the approximate real low-quality image generated by the image generation network can be closer to the real low-quality image, so that the image restoration network finally trained has better image restoration performance.

In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned loss function also includes a fourth loss function term, and the fourth loss function term includes perceptual loss and adversarial loss between at least one restored high-quality image and at least one real high-quality image.

When the above loss function also includes the fourth loss function term, the information reflected by the loss function is more comprehensive. Therefore, using this loss function during training can train an image loss network with better image restoration performance.

In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned loss function also includes a fifth loss function term, and the above-mentioned method also includes: inputting at least one synthetic low-quality image into the second generation network for processing to obtain at least one converted synthetic low-quality image; inputting at least one real low-quality image into the first generation network for processing to obtain at least one converted real low-quality image; wherein the fifth loss function term includes the difference between the pixel values of at least one converted real low-quality image and the pixel values of at least one real low-quality image, and the difference between the pixel values of at least one converted synthetic low-quality image and the pixel values of at least one synthetic low-quality image.

When the above loss function also includes the fifth loss function term, the information reflected by the loss function is more comprehensive. Therefore, using this loss function during training can train an image loss network with better image restoration performance.

In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned existing real low-quality image and the image to be restored are acquired by using the same device, including: the existing real low-quality image is acquired by using a first device, and the image to be restored is acquired by using a second device, the first device and the second device are of the same device type, and the image acquisition parameters of the first device are the same as the image acquisition parameters of the second device, and the image acquisition parameters include at least one of focal length, exposure, and shutter time.

When the existing real low-quality image and the image to be restored are acquired using the same type of equipment and the same image sampling parameters, the existing real low-quality image and the image to be restored are closer, so that the image restoration network trained with reference to the existing real low-quality image in this application has a better image restoration effect when processing the image to be restored.

In a third aspect, an image restoration method is provided, the method comprising: obtaining an image to be restored; using an image restoration network to perform restoration processing on the image to be restored to obtain a restored high-quality image, wherein the image clarity of the restored high-quality image is higher than the image clarity of the image to be restored.

Among them, the above-mentioned image restoration network is obtained by training based on multiple training image pairs, each of the multiple training image pairs includes a real high-quality image and an approximate real low-quality image, the approximate real low-quality image in each training image pair is obtained by processing the real high-quality image in each training image pair, the difference in image clarity between the approximate real low-quality image and the existing real low-quality image is within a preset range, the image clarity includes at least one of the image blur degree, image noise distribution and image resolution, and the existing real low-quality image and the image to be restored are collected using the same device.

That is to say, the image restoration method of the third aspect may adopt the image restoration network trained by the training method of the first aspect.

Since the approximate real low-quality images included in the training image pairs are obtained by synthesizing and realizing the real high-quality images, that is, the approximate real low-quality images included in the training image pairs are relatively close to the real low-quality images, therefore, according to the training of the image restoration network according to multiple training image pairs proposed in the present application, an image restoration network with better image restoration effect for the real low-quality images can be obtained. Therefore, the image restoration method of the present application can have a better image restoration effect when performing image restoration using the trained image restoration network.

In combination with the third aspect, in certain implementations of the third aspect, image clarity includes at least one of image blur degree, image noise distribution, and image resolution.

In combination with the third aspect, in certain implementations of the third aspect, the above-mentioned existing real low-quality image and the image to be restored are acquired by using the same device, including: the existing real low-quality image is acquired by using a first device, and the image to be restored is acquired by using a second device, the first device and the second device are of the same device type, and the image acquisition parameters of the first device are the same as the image acquisition parameters of the second device, and the image acquisition parameters include at least one of focal length, exposure, and shutter time.

When the existing real low-quality image and the image to be restored are acquired using the same type of equipment and the same image sampling parameters, the existing real low-quality image and the image to be restored are closer, so that the image restoration network trained with reference to the existing real low-quality image in this application has a better image restoration effect when processing the image to be restored.

In a fourth aspect, an image restoration method is provided, the method comprising: obtaining an image to be restored; using an image restoration network to perform restoration processing on the image to be restored to obtain a restored high-quality image, wherein the image clarity of the restored high-quality image is higher than the image clarity of the image to be restored.

Among them, the image restoration network in the image restoration method of the fourth aspect is trained according to the training method of the second aspect.

By jointly training the image generation network and the image restoration network, the approximate real low-quality image generated by the image generation network can be closer to the real low-quality image, so that the image restoration network obtained by the joint training method in the present application has better image restoration performance when performing image restoration processing.

In a fifth aspect, a training device for an image restoration network is provided, wherein the training device for an image restoration network includes modules for executing the methods in the first aspect or the second aspect.

In a sixth aspect, an image restoration device is provided, which includes modules for executing the methods in the third aspect or the fourth aspect.

In the seventh aspect, a training device for an image restoration network is provided, the device comprising: a memory for storing programs; a processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is used to execute the method in the first aspect or the second aspect above.

In an eighth aspect, an image restoration device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is used to execute the method in the third aspect or the fourth aspect above.

In a ninth aspect, a computer device is provided, which includes the training device of the image restoration network in the fourth aspect.

In the ninth aspect above, the computer device may specifically be a server or a cloud device, etc.

In a tenth aspect, an electronic device is provided, which includes the image restoration device of the sixth aspect.

In the above-mentioned tenth aspect, the electronic device can specifically be a mobile terminal (for example, a smart phone), a tablet computer, a laptop computer, an augmented reality/virtual reality device, a vehicle-mounted terminal device, and the like.

In an eleventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program code, wherein the program code includes instructions for executing the steps in any one of the methods in the first aspect, the second aspect, the third aspect, and the fourth aspect.

In a twelfth aspect, a computer program product comprising instructions is provided. When the computer program product is run on a computer, the computer is enabled to execute any one of the methods in the first, second, third and fourth aspects above.

In the thirteenth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute any one of the methods in the first, second, third and fourth aspects above.

Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute any one of the methods in the above-mentioned first aspect, second aspect, third aspect and fourth aspect.

The above chip can specifically be a field programmable gate array FPGA or an application specific integrated circuit ASIC.

It should be understood that in the present application, the method of the first aspect may specifically refer to the method in the first aspect and any one of the various implementations of the first aspect, the method of the second aspect may specifically refer to the method in the second aspect and any one of the various implementations of the second aspect, and the method of the third aspect may specifically refer to the method in the third aspect and any one of the various implementations of the third aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the structure of the system architecture provided in an embodiment of the present application;

FIG2 is a schematic diagram of target detection using a convolutional neural network model provided in an embodiment of the present application;

FIG3 is a schematic diagram of a chip hardware structure provided in an embodiment of the present application;

FIG4 is a schematic diagram of a system architecture provided in an embodiment of the present application;

FIG5 is a schematic diagram of a training method for an image restoration network according to an embodiment of the present application;

FIG6 is a schematic diagram of an image restoration method according to an embodiment of the present application;

FIG7 is a schematic flow chart of a training method for an image restoration network according to an embodiment of the present application;

FIG8 shows a process of processing a real high-quality image to obtain an approximate real low-quality image;

FIG9 is a schematic diagram of using an image generation network to process multiple near-real low-quality images to obtain multiple synthetic low-quality images;

FIG10 is a schematic diagram of training an image restoration network according to training image pairs in an embodiment of the present application;

FIG11 is a schematic diagram of a training method for an image restoration network according to an embodiment of the present application;

FIG12 is a schematic diagram of jointly training an image restoration network and an image generation network;

FIG13 is a schematic diagram of the process of determining the loss function of the first stage;

FIG14 is a schematic diagram of the process of determining the loss function of the second stage;

FIG15 is a schematic diagram of the process of determining the loss function of the first stage;

FIG16 is a schematic diagram of the process of determining the loss function of the first stage;

FIG17 is a schematic diagram of the process of determining the loss function of the second stage;

FIG18 is a schematic flow chart of an image restoration network method according to an embodiment of the present application;

FIG19 is a schematic block diagram of a training device for an image restoration network according to an embodiment of the present application;

FIG20 is a schematic diagram of the hardware structure of the training device of the image restoration network according to an embodiment of the present application;

FIG21 is a schematic block diagram of an image device according to an embodiment of the present application;

FIG. 22 is a schematic diagram of the hardware structure of the image restoration device according to an embodiment of the present application.

Detailed ways

The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The solution of the present application can be applied to computer vision fields such as assisted driving, automatic driving, safe cities, and smart terminals that require image processing (e.g., image classification, image recognition). (Specifically, before image classification or image recognition is performed on low-quality images, image restoration can be performed on low-quality images to obtain high-quality images, and then image classification or image recognition can be performed on the high-quality images).

Specifically, when taking photos with a smart terminal (e.g., a mobile phone) or displaying images with a smart terminal, a television, or other displays, it is necessary to eliminate noise in the image as much as possible, reduce the blurriness of the image, and improve the resolution of the image so that users can view high-definition and high-resolution images.

In addition, in the security field, the image quality captured by surveillance cameras is generally low, which will affect the accuracy of people or recognition algorithms in identifying targets and judging events. Therefore, it is necessary to improve the resolution and clarity of these pictures, that is, the images need to be restored so that accurate judgments can be made based on the restored images.

In the present application, a neural network (model) may be used for image restoration. To better understand the present application, the relevant terms and concepts of the neural network are first introduced below.

(1) Neural Network

The neural network may be composed of neural units, and the neural unit may refer to an operation unit with x _s and intercept 1 as input, and the output of the operation unit may be shown as formula (1):

Wherein, s=1, 2, ...n, n is a natural number greater than 1, _Ws is the weight of _xs , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to perform nonlinear transformation on the features in the neural network, thereby converting the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

(2) Deep Neural Networks

Deep neural network (DNN), also known as multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: Among them,/> is the input vector, /> is the output vector, /> is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector/> After such a simple operation, the output vector is obtained/> Since DNN has many layers, coefficient W and offset vector/> The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient W as an example, assuming that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as/> The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

(3) Convolutional Neural Network

Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

(4) Residual Network

The residual network is a deep convolutional network proposed in 2015. Compared with traditional convolutional neural networks, the residual network is easier to optimize and can improve accuracy by increasing the depth. The core of the residual network is to solve the side effects (degradation problem) brought by increasing the depth, so that the network performance can be improved by simply increasing the network depth. The residual network generally contains many sub-modules with the same structure. Usually, a residual network (ResNet) is used to connect a number to indicate the number of times the sub-module is repeated. For example, ResNet50 means that there are 50 sub-modules in the residual network.

(6) Classifier

Many neural network structures have a classifier at the end to classify objects in the image. The classifier is generally composed of a fully connected layer and a softmax function (which can be called a normalized exponential function), which can output the probability of different categories based on the input.

(7) Loss function

In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

(8) Back propagation algorithm

Neural networks can use the error back propagation (BP) algorithm to correct the values of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

The above is a brief introduction to some basic contents of neural networks. The following is an introduction to some specific neural networks that may be used in image data processing.

The system architecture of the embodiment of the present application is described in detail below in conjunction with FIG. 1 .

FIG1 is a schematic diagram of a system architecture of an embodiment of the present application. As shown in FIG1 , the system architecture 100 includes an execution device 110 , a training device 120 , a database 130 , a client device 140 , a data storage system 150 , and a data acquisition system 160 .

In addition, the execution device 110 includes a calculation module 111, an I/O interface 112, a preprocessing module 113 and a preprocessing module 114. The calculation module 111 may include the target model/rule 101, and the preprocessing module 113 and the preprocessing module 114 are optional.

The data acquisition device 160 is used to collect training data. For the training method of the image restoration network of the embodiment of the present application, the training data may include a real high-quality image and an approximate real low-quality image corresponding to the real high-quality image. After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 obtains the target model/rule 101 based on the training data maintained in the database 130.

The following describes how the training device 120 obtains the target model/rule 101 based on the training data. The training device 120 performs image restoration on the input approximate real high-quality image to obtain a restored high-quality image. Next, the restored high-quality image is compared with the real high-quality image, and the image restoration network is updated according to the difference between the restored high-quality image and the real high-quality image until the image restoration network meets the preset requirements, thereby completing the training of the target model/rule 101. The target model/rule 101 here is equivalent to the image restoration network.

The above-mentioned target model/rule 101 can be used to implement the image restoration method of the embodiment of the present application, that is, the image to be restored (the image to be restored may be an input low-quality image that needs to be restored) is input into the target model/rule 101, and a high-quality image restored after image restoration processing of the image to be restored can be obtained. The target model/rule 101 in the embodiment of the present application may specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not necessarily train the target model/rule 101 entirely based on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

The target model/rule 101 obtained by training with the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG1 . The execution device 110 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR)/virtual reality (VR), a vehicle terminal, etc., and can also be a server or a cloud. In FIG1 , the execution device 110 is configured with an input/output (I/O) interface 112 for data interaction with an external device. The user can input data to the I/O interface 112 through the client device 140. The input data in the embodiment of the present application may include: the image to be restored input by the client device. The client device 140 here can specifically be a terminal device.

The preprocessing module 113 and the preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be restored) received by the I/O interface 112. In the embodiment of the present application, there may be no preprocessing module 113 and the preprocessing module 114 or only one preprocessing module. When there is no preprocessing module 113 and the preprocessing module 114, the computing module 111 may be directly used to process the input data.

When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

Finally, the I/O interface 112 presents the processing result (specifically, a high-quality image obtained by image restoration), such as a high-quality restored image obtained by performing image restoration processing on the image to be restored by the target model/rule 101, to the client device 140, thereby providing it to the user.

Specifically, the high-quality image obtained by image restoration through the target model/rule 101 in the computing module 111 can be processed by the preprocessing module 113 (and may also be processed by the preprocessing module 114) (for example, image rendering processing) and the processing result can be sent to the I/O interface, and then the I/O interface can send the processing result to the client device 140 for display.

It should be understood that when the preprocessing module 113 and the preprocessing module 114 do not exist in the above-mentioned system architecture 100, the computing module 111 can also transmit the high-quality image obtained by the image restoration processing to the I/O interface, and then the I/O interface sends the processing result to the client device 140 for display.

It is worth noting that the training device 120 can target different goals or different tasks (for example, the training device can be trained on real high-quality images and approximate low-quality images in different scenarios), and generate corresponding target models/rules 101 based on different training data. The corresponding target models/rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

In FIG. 1 , the user can manually give input data (the input data can be an image to be restored), and the manual giving can be operated through the interface provided by the I/O interface 112. In another case, the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data input to the I/O interface 112 and the output results output from the I/O interface 112 as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I/O interface 112 directly stores the input data input to the I/O interface 112 and the output results output from the I/O interface 112 as new sample data in the database 130.

It is worth noting that Figure 1 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

As shown in FIG1 , the target model/rule 101 obtained by training with the training device 120 may be a neural network in an embodiment of the present application. Specifically, the neural network provided in the embodiment of the present application may be a CNN and a deep convolutional neural network (DCNN), and the like.

Since CNN is a very common neural network, the following focuses on the detailed introduction of the structure of CNN in conjunction with Figure 2. As mentioned in the basic concept introduction above, convolutional neural network is a deep neural network with a convolution structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.

As shown in Fig. 2, a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer/pooling layer 220 (wherein the pooling layer is optional), and a fully connected layer 230. The relevant contents of these layers are described in detail below.

Convolutional layer/pooling layer 220:

Convolutional Layer:

As shown in FIG2 , the convolution layer/pooling layer 220 may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layers 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is usually processed one pixel after another (or two pixels after two pixels... depending on the value of the stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to blur unnecessary noise points in the image, etc. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple convolution feature maps of the same size are then merged to form the output of the convolution operation.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. In each layer 221-226 as shown in 220 in FIG. 2, a convolution layer may be followed by a pooling layer, or multiple convolution layers may be followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

Fully connected layer 230:

After being processed by the convolution layer/pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. Because as mentioned above, the convolution layer/pooling layer 220 will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the fully connected layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (231, 232 to 23n as shown in Figure 2) and an output layer 240. The parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

After the multiple hidden layers in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (the propagation from 210 to 240 in FIG. 2 is the forward propagation) is completed, the back propagation (the propagation from 240 to 210 in FIG. 2 is the back propagation) will begin to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

It should be noted that the convolutional neural network 200 shown in FIG. 2 is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

It should be understood that the convolutional neural network (CNN) 200 shown in Figure 2 can be used to perform the image restoration method of the embodiment of the present application. As shown in Figure 2, the image to be restored can be restored to a high-quality image after being processed by the input layer 210, the convolution layer/pooling layer 220 and the fully connected layer 230.

FIG3 is a chip hardware structure provided in an embodiment of the present application, and the chip includes a neural network processor 50. The chip can be set in the execution device 110 as shown in FIG1 to complete the calculation work of the calculation module 111. The chip can also be set in the training device 120 as shown in FIG1 to complete the training work of the training device 120 and output the target model/rule 101. The algorithms of each layer in the convolutional neural network shown in FIG2 can be implemented in the chip shown in FIG3.

The neural-network processing unit (NPU) 50 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations.

In some implementations, the operation circuit 503 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 503 takes the corresponding data of the matrix B from the weight memory 502 and caches it on each PE in the operation circuit 503. The operation circuit 503 takes the matrix A data from the input memory 501 and performs a matrix operation with the matrix B, and the partial result or the final result of the matrix is stored in the accumulator 508.

The vector calculation unit 507 can further process the output of the operation circuit 503, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional/non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

In some implementations, the vector calculation unit can 507 store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, such as for use in a subsequent layer in a neural network.

The unified memory 506 is used to store input data and output data.

The weight data is directly transferred from the external memory to the input memory 501 and/or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

In addition, in the present application, the operations of each layer in the convolutional neural network shown in Figure 2 can be performed by the operation circuit 503 or the vector calculation unit 507.

As shown in Fig. 4, the embodiment of the present application provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 210 and a data storage system 250, wherein the local device 301 and the local device 302 are connected to the execution device 210 via a communication network.

The execution device 210 can be implemented by one or more servers. Optionally, the execution device 210 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 210 can be arranged at one physical site, or distributed at multiple physical sites. The execution device 210 can use the data in the data storage system 250, or call the program code in the data storage system 250 to implement the image restoration method of the embodiment of the present application.

Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism/communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

In one implementation, the local device 301 and the local device 302 obtain network parameters of the image restoration network from the execution device 210, deploy the image restoration network on the local device 301 and the local device 302, and use the image restoration network to perform image restoration.

In another implementation, the image restoration network can be directly deployed on the execution device 210. The execution device 210 obtains the image to be restored from the local device 301 and the local device 302 (the local device 301 and the local device 302 can upload the image to be restored to the execution device 210), and restores the image to be restored according to the image restoration network, and sends the high-quality image obtained by image restoration to the local device 301 and the local device 302.

The execution device 210 may also be referred to as a cloud device. In this case, the execution device 210 is generally deployed in the cloud.

The embodiments of the present application are described in detail below with reference to the accompanying drawings.

FIG5 is a schematic diagram of a training method for an image restoration network according to an embodiment of the present application.

As shown in Figure 5, after obtaining multiple training image pairs, the image restoration network can be trained according to the multiple training image pairs to obtain a trained image restoration network. The trained image restoration network can be used for image restoration, thereby converting the input low-quality image into a high-quality image and improving the image display effect.

In this application, the image restoration network may also be referred to as an image restoration model, and the image restoration network may be a neural network.

FIG. 6 is a schematic diagram of an image restoration method according to an embodiment of the present application.

As shown in Figure 6, the image restoration network can restore the image to be restored (generally a low-quality image) to obtain a restored high-quality image. The image restoration network in Figure 6 can be trained using the training method of the image restoration network shown in Figure 5.

The following is a detailed introduction to the training method of the image restoration network of the embodiment of the present application in conjunction with FIG. 7 .

FIG7 is a schematic flow chart of a training method for an image restoration network according to an embodiment of the present application. The method shown in FIG7 can be performed by a training device for an image restoration network according to an embodiment of the present application. The method shown in FIG7 includes steps 1001 and 1002. Steps 1001 and 1002 are described in detail below.

1001. Obtain multiple training image pairs.

Each of the plurality of training image pairs includes a real high-quality image and an approximately real low-quality image.

For each of the above-mentioned training image pairs, the approximate real low-quality image therein is obtained by processing the real high-quality image in the training image pair, and the difference in image clarity between the approximate real low-quality image in each training image pair and the image clarity of the existing real low-quality image is within a preset range.

The above-mentioned image clarity includes at least one of the image blur degree, image noise distribution and image resolution.

Optionally, the approximate real low-quality image in each of the above-mentioned training image pairs is obtained by adjusting the real high-quality image in each of the above-mentioned training image pairs, and the above-mentioned adjustment processing is used to adjust the image clarity of the real high-quality image in each of the above-mentioned training image pairs, so that the image clarity of the approximate real low-quality image in each of the above-mentioned training image pairs is as similar as possible to the image clarity of the existing real low-quality image.

It should be understood that the image clarity of the approximate real low-quality image in each of the above-mentioned training image pairs is as similar as possible to the image clarity of the existing real low-quality image. Specifically, it may mean that the difference in image clarity between the approximate real low-quality image in each training image pair and the image clarity of the existing real low-quality image is within a preset range, and the preset range can be flexibly set according to actual needs.

Optionally, there are two ways to obtain an approximately true low-quality image in each training image pair, and the two ways are introduced below.

The first method is to obtain a low-quality image that approximates the real image through blurring, noise addition, and downsampling.

Specifically, in the first mode, the adjustment process for the real high-quality image in each training image pair can be directly blurring, adding noise, downsampling, etc. the real high-quality image in each training image pair.

Specifically, the approximate real low-quality image in each of the training image pairs is obtained by performing synthesis processing and authenticity processing on the real high-quality image in the training image pair.

FIG8 shows a process of processing a real high-quality image to obtain a near-real low-quality image.

Specifically, as shown in FIG8 , for each training image pair, the real high-quality image in the image pair can be first synthesized to obtain a synthesized low-quality image, and then the synthesized low-quality image can be realistically processed to obtain an approximate real low-quality image in the image pair.

Among them, the above-mentioned synthesis processing may include at least one of blurring processing, noise adding processing and downsampling processing, and the above-mentioned reality processing is used to adjust the image clarity so that the image clarity of the adjusted image is as similar as possible to the image clarity of the existing real low-quality image.

In addition, the above-mentioned existing real low-quality image and the image to be restored processed by the trained image restoration network during subsequent image restoration are acquired using the same device. In other words, the above-mentioned existing real low-quality image and the image to be restored are acquired using the same device, and the image to be restored is the image processed by the trained image restoration network.

The second method is to use an image generation network to obtain a low-quality image that approximates the real image.

In the second mode, the approximate real low-quality image in each of the above training image pairs is obtained by processing the real high-quality image in each of the training image pairs using a pre-trained image generation network.

The above-mentioned image generation network can be used to convert a real high-quality image into an approximate real low-quality image. The image generation network can be obtained by training based on multiple real high-quality images and multiple real low-quality images. Specifically, during the training process, multiple real high-quality images can be input into the image generation network so that the difference between the image clarity of the output image and the image clarity of any one of the multiple real low-quality images is as small as possible. The training is terminated until the trained image generation network meets the preset requirements. The training termination condition can be flexibly set according to the actual situation, for example, when the image processing performance of the trained image generation network meets the preset requirements.

In addition, in the above-mentioned image generation network training process, multiple real high-quality images can also be synthesized first to obtain multiple synthetic low-quality images, and then the image generation network can be trained using the multiple synthetic low-quality images and the multiple real low-quality images.

Optionally, the difference in clarity between the image clarity of the approximate real low-quality image in each of the above-mentioned training image pairs and the image clarity of the existing real low-quality image is within a preset range, including: the distance between the feature vector of the approximate real low-quality image in each of the above-mentioned training image pairs and the feature vector of the existing real low-quality image is less than a preset distance.

The above preset distance can be flexibly set according to actual conditions.

The feature vector of the approximate real low-quality image in each of the above-mentioned training image pairs can be obtained by extracting features of the approximate real low-quality image in each of the above-mentioned training image pairs using a discriminant neural network (also referred to as a discriminator), and the feature vector of the above-mentioned existing real low-quality image can also be obtained by extracting features of the existing real low-quality image based on the discriminant neural network.

When the discriminant neural network is used to extract the feature vectors of the two images, the same convolution parameters may be used for extraction, and the specific parameter values of the convolution parameters may be obtained by training the discriminant neural network.

Optionally, the obtaining of multiple training image pairs includes: determining the multiple training image pairs from an initial training image set, wherein the difference between the image clarity of the approximate real low-quality image in each training image pair in the multiple training image pairs and the image clarity of the existing real low-quality image is less than a preset threshold.

The above preset threshold can be flexibly set according to actual conditions. If the difference between image clarity during training is required to be strict, a smaller preset threshold can be set. If the difference between image clarity during training is required to be less strict, a larger preset threshold can be set.

For example, the initial training image set includes 100 training image pairs, then 40 training image pairs can be selected therefrom, and the difference between the image clarity of the approximate real low-quality image in each of the 40 training image pairs and the image clarity of the existing real low-quality image is less than a preset threshold.

By selecting a plurality of training images whose image clarity differences meet the requirements from the initial training image set for training, the training effect can be improved so that the image restoration network obtained after training has better image restoration performance.

FIG9 is a schematic diagram of using an image generation network to process multiple approximately real low-quality images to obtain multiple synthetic low-quality images.

As shown in Fig. 9, the multiple approximate real low-quality images in the multiple training image pairs can be obtained by processing multiple synthetic low-quality images through an image generation network. Specifically, the image generation network shown in Fig. 9 includes a first generation network and a second generation network, and the first generation network can be used to convert multiple synthetic low-quality images into multiple approximate real low-quality images.

The image generation network shown in Figure 9 can specifically be a cycle-consistent adversarial network (CycleGAN), an unsupervised image-to-image training network (UNIT), a unified generative adversarial network for multi-domain image-to-image translation (StarGAN), and a diverse image-to-image translation (DRIT) network.

1002. Train an image restoration network according to a plurality of training image pairs until image restoration performance of the image restoration network meets preset requirements.

In the present application, since the approximate real low-quality images contained in the training image pairs are obtained by synthesizing and realizing the real high-quality images, that is, the approximate real low-quality images contained in the training image pairs are relatively close to the real low-quality images, therefore, the present application proposes to train the image restoration network according to multiple training image pairs, which can obtain an image restoration network with better image restoration effect for real low-quality images.

The process of training the image restoration network according to the multiple training image pairs in the above step 1002 is described in detail below in conjunction with FIG. 10 .

Fig. 10 is a schematic diagram of training an image restoration network according to a training image pair in an embodiment of the present application. The method shown in Fig. 10 is equivalent to step 1002 in the method shown in Fig. 7. The training process shown in Fig. 10 includes steps 2001 to 2007, which are described in detail below.

2001. Start.

Step 2001 indicates starting to train the image restoration network using multiple training image pairs, and the training process of the image restoration network begins.

2002. Initialize the network parameters of the image restoration network to obtain the initial values of the network parameters of the image restoration network.

In the above step 2002, the network parameters of the image restoration network may be randomly initialized so that the network parameters of the image restoration network take some random values.

In addition, in the above step 2002, the image restoration network can also be preliminarily trained by using synthetic low-quality images and real high-quality images, and the network parameter values of the image restoration network obtained by the preliminary training can be used as the initial values of the network parameters of the image restoration network, and then formal training can be performed in subsequent steps.

2003. Inputting at least one approximate true low-quality image in the training image pair into an image restoration network for processing to obtain at least one restored high-quality image.

It should be understood that in the above step 2003, at least one training image pair can be used for each training, that is, during the training process, one approximate real low-quality image in a training image pair can be input into the image restoration network for processing to obtain a restored high-quality image, or multiple approximate real low-quality images in multiple training image pairs can be input into the image restoration network for processing to obtain multiple restored high-quality images. The present application does not limit the number of images input into the image restoration network for processing each time.

2004. Determine a function value of a loss function according to a difference between at least one restored high-quality image and at least one real high-quality image in a training image pair.

Specifically, in step 2004, when there is only one restored high-quality image and one true high-quality image, the function value of the loss function can be determined according to the difference between the restored high-quality image and the true high-quality image, and the function value of the loss function is positively correlated with the difference between the restored high-quality image and the true high-quality image.

Specifically, when the difference between at least one restored high-quality image and the true high-quality image in at least one training image pair is greater, the function value of the image loss function is also greater, and when the difference between at least one restored high-quality image and the true high-quality image in at least one training image pair is smaller, the function value of the image loss function is also smaller.

In the above step 2004, when there are multiple restored high-quality images and multiple true high-quality images, the function values of the corresponding loss functions can be determined according to the differences between the multiple restored high-quality images and the corresponding true high-quality images, and then the function values of the corresponding loss functions are summed or averaged to obtain the final function value of the loss function in step 2004.

Optionally, the function value of the above loss function includes a mean squared error (MSE) between at least one restored high-quality image and a real high-quality image in at least one training image pair.

The mean square error between two images refers to the mean of the sum of squares of the differences in pixel values at corresponding positions between the two images.

That is, the function value of the above loss function includes the mean square error loss (MSE loss) between at least one restored high-quality image and the real high-quality image in at least one training image pair.

Optionally, the above-mentioned loss function is the average value of at least one first loss function value, and each first loss function in the at least one first loss function is the mean square error between each restored high-quality image in the at least one restored high-quality image and the corresponding true high-quality image.

For example, the at least one restored high-quality image only includes the restored high-quality image A, and the real high-quality images in the at least one training image pair only include the real high-quality image A’, then the function value of the loss function can be the square of the difference between the pixel value of the restored high-quality image A and the pixel value of the real high-quality image A’.

For another example, the at least one restored high-quality image includes a restored high-quality image A and a restored high-quality image B, and the real high-quality images in the at least one training image pair include a real high-quality image A’ and a real high-quality image B’. Then, the function value of the loss function may be the sum or average of the square of the first difference and the square of the second difference, wherein the first difference is the difference between the pixel value of the restored high-quality image A and the pixel value of the real high-quality image A’, and the second difference is the difference between the pixel value of the restored high-quality image B and the pixel value of the real high-quality image B’.

Optionally, the above loss function also includes perceptual loss and adversarial loss of at least one restored high-quality image relative to a real high-quality image in at least one training image pair.

The perceptual loss between two images may refer to the mean of the sum of the squared norms of the differences between corresponding positions of the feature maps of the two images.

Adversarial loss is generally used to determine whether one image distribution is similar to another image distribution. Adversarial loss can be specifically described by a discriminative neural network.

Specifically, when the image distributions of two images are relatively similar, the two images are input into the discriminant neural network, and the result output by the discriminant neural network is that the image distributions of the two images are the same. When the image distributions of the two images are quite different, the two images are input into the discriminant neural network, and the result output by the discriminant neural network is that the image distributions of the two images are different.

2005. Update the network parameters of the image restoration network according to the function value of the loss function.

Specifically, in step 2005, the network parameters of the image restoration network may be changed according to the function value of the loss function, so that the function value of the loss function obtained by subsequent calculation is as small as possible.

2006. Determine whether the image restoration network meets the preset requirements.

Optionally, the image restoration network meets preset requirements, including: the image restoration network meets at least one of the following conditions:

(1) The image restoration performance of the image restoration network meets the preset performance requirements;

(2) The number of updates of the network parameters of the image restoration network is greater than or equal to a preset number;

(3) The function value of the loss function is less than or equal to the preset value.

In step 2006, when the image restoration network satisfies at least one of the above conditions (1) to (3), it is determined that the image restoration network meets the preset requirements, and step 2007 is executed to end the training process of the image restoration network. When the image restoration network does not meet any of the above conditions (1) to (3), it means that the image restoration network has not yet met the preset requirements and it is necessary to continue to train the image restoration network, that is, re-execute steps 2002 to 2006 until an image restoration network that meets the preset requirements is obtained.

2007. Training ends.

Step 2007 indicates that the image restoration network has met the preset requirements, and the training process of the image restoration network is completed.

In the present application, in addition to training the image restoration network separately according to the acquired training images, the image restoration network can also be jointly trained with the generation network of the approximate real low-quality images. Through the joint training, the generation quality of the approximate real low-quality images obtained in the first stage can be constrained so that they are as similar as possible to the real low-quality images, thereby ultimately obtaining an image restoration network with better image restoration effect.

The following is a detailed description of the method of jointly training the image restoration network and the image generation network in conjunction with Figure 11.

Fig. 11 is a schematic diagram of training an image restoration network according to a training image pair in an embodiment of the present application. The training process shown in Fig. 11 includes steps 3001 to 3010, which are described in detail below.

3001. Start.

Step 3001 indicates starting to train the image restoration network and the image generation network using the training images. The image restoration network includes a first generation network and a second generation network, the first generation network is used to process the synthetic low-quality image to obtain an approximate real low-quality image, and the second generation network is used to process the real low-quality image to obtain an approximate synthetic low-quality image.

3002. Initialize the network parameters of the image generation network and the network parameters of the image restoration network to obtain initial values of the network parameters of the image generation network and the initial values of the network parameters of the image restoration network.

It should be understood that in step 3002, the network parameters of the image generation network and the network parameters of the image restoration network can be initialized separately, wherein the image generation network and the image restoration network can be initialized successively or simultaneously, and the present application does not limit the order of initialization of the image generation network and the image restoration network.

In the above step 3002, the image restoration network can be preliminarily trained using synthetic low-quality images and real high-quality images, and the network parameter values of the image restoration network obtained by the preliminary training are used as the initial values of the network parameters of the image restoration network.

3003. Input at least one of the multiple synthetic low-quality images into the first generation network in the image generation network for processing to obtain at least one approximate real low-quality image.

The above-mentioned multiple synthetic low-quality images are obtained by synthesizing multiple real high-quality images respectively, and the synthesis processing includes at least one of blurring processing, noise adding processing and downsampling processing.

3004. Input at least one real low-quality image among the multiple real low-quality images into the second generation network in the image generation network for processing to obtain at least one approximate synthetic low-quality image.

The multiple real low-quality images in the above step 3004 and the image to be restored are acquired by using the same device, and the image to be restored is an image processed by the trained image restoration network.

The same type of device mentioned above may refer to devices of completely the same device model, for example, cameras of the same model, terminal devices of the same model, cameras of the same model, and so on.

The above-mentioned existing real low-quality image and the image to be restored are acquired by using the same device, specifically including: the above-mentioned existing real low-quality image is acquired by using a first device, and the image to be restored is acquired by using a second device, the first device and the second device are of the same device type, and the image acquisition parameters of the first device are the same as the image acquisition parameters of the second device, and the image acquisition parameters include at least one of focal length, exposure, and shutter time.

When the existing real low-quality image and the image to be restored are acquired using the same type of equipment and the same image sampling parameters, the existing real low-quality image and the image to be restored are closer, so that the image restoration network trained with reference to the existing real low-quality image in this application has a better image restoration effect when processing the image to be restored.

3005. Input at least one approximate real low-quality image into a second generation network for processing to obtain at least one reconstructed synthetic low-quality image.

In the above steps 3003 and 3005, a loop processing process is implemented in the image generation network, that is, the synthetic low-quality image is processed through steps 3003 and 3005 respectively to obtain an approximate real low-quality image, and then the approximate real low-quality image is processed again to obtain a reconstructed synthetic low-quality image. Next, the difference between the reconstructed synthetic low-quality image and the synthetic low-quality image can be compared, and the network parameters of the image generation network can be adjusted during the training process to gradually reduce the difference.

3006. Input at least one approximately synthesized low-quality image into a first generation network for processing to obtain at least one reconstructed real low-quality image.

In the above steps 3004 and 3006, a loop processing process is implemented in the image generation network, that is, the real low-quality image is processed through steps 3004 and 3006 respectively to obtain an approximate synthetic low-quality image, and then the approximate synthetic low-quality image is processed again to obtain a reconstructed real low-quality image. Next, the difference between the reconstructed real low-quality image and the real low-quality image can be compared, and the network parameters of the image generation network can be adjusted during the training process to gradually reduce the difference.

3007. Input at least one approximately true low-quality image into an image restoration network for processing to obtain at least one restored high-quality image.

3008. Determine the loss function.

The loss function in step 3008 includes a first loss term, a second loss term and a third loss term. The first loss function term and the second loss function term reflect the image loss of the image generation network, and the third loss function term reflects the image loss of the image restoration network. The specific meanings of these three loss function terms are as follows:

The first loss function term:

The first loss function term includes an adversarial loss of at least one approximate real low-quality image relative to any one of the multiple real low-quality images, and an adversarial loss of at least one approximate synthetic low-quality image relative to any one of the multiple synthetic low-quality images;

The second loss function term:

The second loss function term includes a difference between a pixel value of at least one reconstructed synthetic low-quality image and a pixel value of at least one synthetic low-quality image, and a difference between a pixel value of at least one reconstructed real low-quality image and a pixel value of at least one real low-quality image;

The third loss function term:

The third loss function term includes a mean square error between at least one restored high-quality image and at least one real high-quality image among a plurality of real high-quality images, wherein at least one synthetic low-quality image is obtained by synthesizing at least one real high-quality image.

Optionally, in addition to the above three loss function terms, the loss function in the above step 3008 may also include a fourth loss function term, which includes the perceptual loss and adversarial loss between the at least one restored high-quality image and the at least one real high-quality image.

When the above loss function also includes the fourth loss function term, the information reflected by the loss function is more comprehensive. Therefore, using this loss function during training can train an image loss network with better image restoration performance.

3009. Update the network parameters of the image generation network and the image restoration network according to the function value of the loss function.

Specifically, in step 3009, the network parameters of the image restoration network and the network parameters of the image generation network can be changed according to the function value of the loss function, so that the function value of the loss function obtained by subsequent calculation is as small as possible.

3010. Determine whether the image restoration network meets preset requirements.

Optionally, the image restoration network meets preset requirements, including: the image restoration network meets at least one of the following conditions:

(1) The image restoration performance of the image restoration network meets the preset performance requirements;

(2) The number of updates of the network parameters of the image restoration network is greater than or equal to a preset number;

(3) The function value of the loss function is less than or equal to the preset value.

In step 3010, when the image restoration network satisfies at least one of the above conditions (1) to (3), it is determined that the image restoration network meets the preset requirements, and step 3011 is executed, and the training process of the image restoration network ends. When the image restoration network does not meet any of the above conditions (1) to (3), it means that the image restoration network has not yet met the preset requirements, and it is necessary to continue to train the image restoration network, that is, re-execute steps 3003 to 3010 until an image restoration network that meets the preset requirements is obtained.

3011. Training is over.

Step 3011 indicates that the image restoration network has met the preset requirements and the training process of the image restoration network is completed.

In the present application, by jointly training the image generation network and the image restoration network, the approximate real low-quality image generated by the image generation network can be closer to the real low-quality image, so that the image restoration network finally trained has better image restoration performance.

Optionally, the method shown in FIG. 11 may further include:

3012. Input at least one synthesized low-quality image into the second generation network for processing to obtain at least one converted synthesized low-quality image.

3013. Input at least one real low-quality image into the first generation network for processing to obtain at least one transformed real low-quality image.

The above steps 3012 and 3013 may occur before step 3008. When the method shown in FIG. 11 includes steps 3012 and 3013, the above loss function also includes a fifth loss function term, which includes the difference between the pixel value of at least one converted real low-quality image and the pixel value of at least one real low-quality image, and the difference between the pixel value of at least one converted synthetic low-quality image and the pixel value of the at least one synthetic low-quality image.

When the above loss function also includes the fifth loss function term, the information reflected by the loss function is more comprehensive. Therefore, using this loss function during training can train an image loss network with better image restoration performance.

In the training process shown in FIG. 11 above, the image generation network and the image restoration network are trained jointly, that is, the training process shown in FIG. 11 can be divided into two stages of training, the first stage is the training of the image generation network, and the second stage is the training of the image restoration network.

As shown in FIG12 , when training the image generation network, multiple synthetic low-quality images can be input (the multiple synthetic low-quality images can be obtained by synthesizing multiple real high-quality images). After being processed by the first generation network, an approximate real low-quality image is obtained. The approximate real low-quality image can be sent to the image restoration network for processing to obtain a restored high-quality image.

Next, we can calculate the loss functions of the first stage and the second stage respectively, and then add the loss functions of the first stage and the second stage to get the total loss function during the training of the image restoration network and the image generation network, and then continue to adjust the network parameters of the image restoration network and the image generation network according to the total loss function.

In order to better understand the above-mentioned first stage and second stage training processes, the first stage and second stage training processes are described in detail below with reference to the accompanying drawings.

FIG. 13 is a schematic diagram of the process of determining the loss function of the first stage.

The process shown in FIG. 13 includes steps 4001 to 4011 , and steps 4001 to 4011 are respectively described in detail below.

4001. Start.

Step 4001 indicates the start of the first phase of training.

4002. Obtain real high-quality images.

4003. Obtain real low-quality images.

Specifically, in step 4002 and step 4003, multiple real high-quality images and multiple real low-quality images can be obtained respectively, wherein the multiple real high-quality images are used for subsequent training of the image restoration network, and the multiple real low-quality images are used for training the image generation network.

4004. Down-sample the real high-quality image to obtain a synthetic low-quality image.

In step 4004, the synthesized low-quality image can also be obtained by blurring or adding noise to the real high-quality image. Therefore, in step 4004, the real high-quality image can also be processed in a variety of ways to reduce the image quality to obtain a synthesized low-quality image.

4005. Using the first generation network to process the synthesized low-quality image, a low-quality image approximating the real image is obtained.

The process of generating a low-quality image that approximates the real image in step 4005 is similar to the process in step 3003 and will not be described in detail here.

4006. Using a second generative network to process the approximate real low-quality image, a reconstructed synthetic low-quality image is obtained.

The process of generating the reconstructed synthetic low-quality image in step 4006 is similar to the process in step 3005 and will not be described in detail here.

4007. Using the second generation network to process the synthesized low-quality image, a transformed synthesized low-quality image is obtained.

The process of generating the converted synthetic low-quality image in step 4007 is similar to the process in step 3012 and will not be described in detail here.

4008. Using the second generation network to process the real low-quality image, an approximate synthetic low-quality image is obtained.

The process of generating an approximate synthetic low-quality image in step 4008 is similar to the process in step 3004 and will not be described in detail here.

4009. Using the first generation network to process the approximate real low-quality image, obtain a reconstructed real low-quality image.

The process of generating and reconstructing the real low-quality image in step 4009 is similar to the process in step 3006 and will not be described in detail here.

4010. Using the first generation network to process the real low-quality image, obtain a transformed real low-quality image.

The process of generating the converted real low-quality image in step 4010 is similar to the process in step 3013 and will not be described in detail here.

4011. Determine the loss function of the first stage.

The loss function of the first stage may include the first loss function and the second loss function term. Furthermore, the loss function of the first stage may also include a fifth loss function term.

After the loss function of the first stage is determined in step 4011, the process shown in Figure 13 can be continued to obtain the loss function of the second stage. It should be understood that during the training process, determining the loss function of the first stage and determining the loss function of the second stage can be carried out simultaneously or one after the other. This application does not limit the order of the processes of determining the loss function of the first stage and determining the loss function of the second stage.

FIG. 14 is a schematic diagram of the process of determining the loss function of the second stage.

The process shown in FIG. 14 includes steps 5001 to 5004 , and steps 5001 to 5004 are respectively described in detail below.

5001. Start.

Step 5001 indicates the start of the first phase of training.

5002. Obtain an approximately true low-quality image.

Acquiring the approximately true low-quality image in the above step 5002 can be specifically achieved by acquiring the approximately true low-quality image generated in step 4005 in the first stage.

5003. An image restoration network is used to process a low-quality image that approximates the true image to obtain a restored high-quality image.

5004. Determine the loss function of the second stage.

The loss function of the second stage may include the third loss function term. Furthermore, the loss function of the second stage may also include a fourth loss function term.

When the loss function of the second stage includes the third loss function term and the fourth loss function term, the information reflected by the loss function of the second stage is more comprehensive.

The process of determining the loss function of the first stage is explained below in conjunction with Figures 15 and 16.

The process of determining the loss function of the first stage may include a process of the first stage and a process of the first stage, and the a process of the first stage and the b process of the first stage may occur simultaneously.

As shown in FIG15 , by downsampling the input high-quality image, a synthetic low-quality image can be obtained. Next, the synthetic low-quality image is processed by the first generation network to obtain an approximately real low-quality image. Next, the approximately real low-quality image is processed by the second generation network to obtain a reconstructed synthetic low-quality image. Then, the loss function of the first stage is determined based on the reconstructed synthetic low-quality image and the synthetic low-quality image, and the input real low-quality image and the output approximately real low-quality image are input into the first discriminant network to obtain the discriminant loss, thereby obtaining a partial loss function of the first stage.

As shown in FIG16 , by downsampling the input high-quality image, a synthetic low-quality image can be obtained, and the input real low-quality image can also be processed by the second generation network to obtain an output approximate real low-quality image. Next, the output approximate real low-quality image is processed by the first generation network to obtain a reconstructed real low-quality image, and then the loss function of the first stage is determined based on the reconstructed real low-quality image and the real low-quality image. In addition, the synthetic low-quality image and the output approximate real low-quality image can be input into the second discriminant network to obtain the discriminant loss, and then the partial loss function of the first stage is obtained.

It should be understood that the a process of the first stage shown in FIG. 15 and the b process of the first stage shown in FIG. 16 may occur simultaneously. FIG. 15 or FIG. 16 only determines part of the loss function of the first stage. The sum of the loss function determined by the a process of the first stage shown in FIG. 15 and the loss function determined by the b process of the first stage shown in FIG. 16 is the final loss function of the first stage.

After executing the first stage, the second stage can be executed to calculate the loss function of the second stage, and then the network parameters of the image restoration network and the image generation network are updated according to the sum of the loss functions of the first stage and the second stage to obtain the final image restoration network.

As shown in FIG17 , after the first stage, a low-quality image close to the real image and the loss function of the first stage can be obtained. For the low-quality image close to the real image, it can be processed by the image generation network to obtain a restored high-quality image, and then the restored high-quality image and the input high-quality image can be input into the image restoration discriminant network to judge the discriminant loss of the restored high-quality image relative to the high-quality image, and the image loss of the restored high-quality image relative to the high-quality image can also be calculated, and then the loss function of the second stage can be calculated. After obtaining the loss function of the first stage and the loss function of the second stage, the gradients of the loss functions of the two stages and the network parameters of the image restoration network and the image generation network can be calculated, and it can be determined whether the loss functions of the two stages converge. If the loss functions of the two stages converge, it is determined that the training process is over and the image restoration network and the image generation network are obtained; if the loss functions of the two stages do not converge, it is necessary to continue to update the network parameters of the image restoration network and the image generation network, and re-execute the process of the first stage and the second stage until the loss functions of the two stages converge.

The above text, in conjunction with the accompanying drawings, has provided a detailed description of the training method of the image restoration network of the embodiment of the present application. The image restoration network method of the embodiment of the present application is described below in conjunction with Figure 18. It should be understood that the image restoration network used in the method shown in Figure 18 can be trained using the training method of the image restoration network of the embodiment of the present application.

Fig. 18 is a schematic flow chart of the image restoration network method according to an embodiment of the present application. The method shown in Fig. 18 includes step 6001 and step 6002, which are described below.

6001. Obtain the image to be restored.

The image to be restored may be an image that needs to be restored.

6002. Use an image restoration network to restore the image to be restored to obtain a restored high-quality image.

Generally speaking, the definition of the image to be restored is generally low. By performing restoration processing on the image to be restored, a high-quality restored image with higher definition can be obtained.

The image restoration network in the above step 6002 can be trained according to the training method of the image restoration network in the embodiment of the present application.

Specifically, the image restoration network in the above step 6002 can be obtained by training the image restoration network separately according to multiple training image pairs, or by jointly training the image restoration network and the image generation network according to the training image pairs.

For example, the image restoration network in the above step 6002 can be trained by the method shown in Figure 7, the method shown in Figure 10, and the method shown in Figure 11.

In the present application, since the approximate real low-quality images included in the training image pairs are obtained by synthesizing and realizing the real high-quality images, that is, the approximate real low-quality images included in the training image pairs are relatively close to the real low-quality images, therefore, according to the present application, the image restoration network is trained according to multiple training image pairs, and an image restoration network with better image restoration effect for the real low-quality images can be obtained. Therefore, the image restoration method of the present application can have a better image restoration effect when performing image restoration using the trained image restoration network.

Optionally, the above-mentioned image clarity includes at least one of image blur degree, image noise distribution and image resolution.

Specifically, the image restoration network in the above step 6002 can be trained using the training method shown in FIG. 7 .

That is to say, the above-mentioned image restoration network is obtained by training based on multiple training image pairs, each of the multiple training image pairs includes a real high-quality image and an approximate real low-quality image, the approximate real low-quality image in each training image pair is obtained by processing the real high-quality image in each training image pair, and the difference in image clarity between the approximate real low-quality image and the existing real low-quality image is within a preset range, wherein the image clarity includes at least one of the image blur degree, image noise distribution and image resolution, and the existing real low-quality image and the image to be restored are collected using the same device.

Among them, the above-mentioned existing real low-quality image and the image to be restored are acquired by using the same device, including: the existing real low-quality image is acquired by using a first device, and the image to be restored is acquired by using a second device, the first device and the second device are of the same device type, and the image acquisition parameters of the first device are the same as the image acquisition parameters of the second device, and the image acquisition parameters include at least one of focal length, exposure, and shutter time.

In the present application, when the existing real low-quality image and the image to be restored are acquired using the same type of equipment and the same image sampling parameters, the existing real low-quality image and the image to be restored are closer, so that the image restoration network trained with reference to the existing real low-quality image in the present application has a better image restoration effect when processing the image to be restored.

In order to evaluate the effect of the training method of the image restoration network of the embodiment of the present application, the image restoration performance of the image restoration network trained by the training method of the image restoration network of the embodiment of the present application is tested using a test set.

Table 1 shows the performance of the image restoration network obtained by the existing solution and the solution of the present application on the test sets NTIRE 17 and NTIRE 17. The data in the table represent the peak signal-noise ratio (PSNR) and structured similarity (SSIM), respectively.

Table 1

Among them, the existing solution 1 represents directly using the bicubic upsampling method to restore the image to be restored; the existing solution 2 represents a solution for training the image restoration network using synthetic approximate real low-quality images and real high-quality images.

Solution 1 (Cycle+SR) of the present application is to first perform the first stage of training, and after obtaining a plurality of low-quality images that are close to the real ones, then form a plurality of training image pairs for the second stage of training to obtain an image restoration network;

Solution 2 (CycleSR) of the present application is to jointly train the image restoration network and the image generation network, and only the mean square error loss function is used when training the image restoration network;

Solution 3 (CycleSRGAN) of the present application is to jointly train the image restoration network and the image generation network, and adopt the mean square error loss function, the perceptual loss function and the adversarial loss function when training the image restoration network.

It can be seen from Table 1 that on different test sets, the test results of the three schemes of the present application are better than those of the existing schemes. Specifically, the PSNR and SSIM of the three schemes of the present application are improved compared with the existing two schemes.

In addition, in order to verify the image restoration performance of the image restoration network obtained by the image restoration network training method of the embodiment of the present application, the embodiment of the present application attempts to use the image restoration network obtained by the image restoration network training method of the embodiment of the present application to perform image restoration on old video images with low image quality.

Specifically, the video images in the new version of The Legend of the Condor Heroes are obtained as high-quality images, and the video images in the old version of The Legend of the Condor Heroes are obtained as real low-quality images. Compared with the high-quality images, the degradation mode of the video images in the old version of The Legend of the Condor Heroes is more complicated, including problems caused by limited lighting during shooting, and problems such as poor imaging quality and low resolution caused by insufficient physical components of the machine during shooting. In this example, the real low-quality images in the old version of The Legend of the Condor Heroes do not have corresponding high-quality images, so the method of the present invention shows its superiority at this time.

By training the image restoration network with high-quality images from the new version of The Legend of the Condor Heroes and real low-quality images from the old version of The Legend of the Condor Heroes, and using the obtained image restoration network to process video images from the old version of The Legend of the Condor Heroes, better image restoration effects can be obtained than existing solutions.

Fig. 19 is a schematic block diagram of a training device for an image restoration network according to an embodiment of the present application. The training device 8000 for an image restoration network shown in Fig. 19 comprises an acquisition unit 8001 and a training unit 8002.

The acquisition unit 8001 and the training unit 8002 can be used to execute the training method of the image restoration network of the embodiment of the present application.

Specifically, the acquisition unit 8001 can execute the above step 1001, and the training unit 8002 can execute the above step 1002.

In addition, the above-mentioned training unit 8002 can also be used to execute the various processes shown in Figures 10 and 11.

The acquisition unit 8001 in the device 8000 shown in Figure 19 above may be equivalent to the communication interface 9003 in the device 9000 shown in Figure 20, through which the corresponding training image may be obtained. Alternatively, the acquisition unit 8001 may also be equivalent to the processor 9002, in which case the training image may be acquired from the memory 9001 through the processor 9002, or the training image may be acquired from the outside through the communication interface 9003.

FIG20 is a schematic diagram of the hardware structure of the training device of the image restoration network of the embodiment of the present application. The training device 9000 of the image restoration network shown in FIG20 (the device 9000 may be a computer device) includes a memory 9001, a processor 9002, a communication interface 9003 and a bus 9004. The memory 9001, the processor 9002, and the communication interface 9003 are connected to each other through the bus 9004.

The memory 9001 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 9001 may store a program. When the program stored in the memory 9001 is executed by the processor 9002, the processor 9002 is used to execute each step of the training method of the image restoration network of the embodiment of the present application.

Processor 9002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the training method of the image restoration network of the method embodiment of the present application.

The processor 9002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the training method of the image restoration network of the present application may be completed by hardware integrated logic circuits in the processor 9002 or software instructions.

The processor 9002 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 9001, and the processor 9002 reads the information in the memory 9001, and completes the functions required to be performed by the units included in the training device of the image restoration network in combination with its hardware, or executes the training method of the image restoration network of the method embodiment of the present application.

The communication interface 9003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 9000 and other devices or a communication network. For example, the image to be restored can be obtained through the communication interface 9003.

The bus 9004 may include a path for transmitting information between various components of the device 9000 (eg, the memory 9001 , the processor 9002 , and the communication interface 9003 ).

Fig. 21 is a schematic block diagram of an image device according to an embodiment of the present application. The image restoration device 10000 shown in Fig. 21 comprises an acquisition unit 10001 and an image restoration unit 10002.

The acquisition unit 10001 and the image training unit 10002 can be used to execute the image restoration method of the embodiment of the present application.

Specifically, the acquisition unit 10001 may execute the above step 6001, and the image restoration unit 10002 may execute the above step 6002.

The acquisition unit 10001 in the device 10000 shown in Figure 21 above can be equivalent to the communication interface 11003 in the device 11000 shown in Figure 22, and the image to be restored can be obtained through the communication interface 11003. Alternatively, the acquisition unit 10001 can also be equivalent to the processor 11002. At this time, the image to be restored can be obtained from the memory 11001 through the processor 11002, or the image to be restored can be obtained from the outside through the communication interface 11003.

FIG22 is a schematic diagram of the hardware structure of the image restoration device of the embodiment of the present application. Similar to the above-mentioned device 10000, the image restoration device 11000 shown in FIG22 includes a memory 11001, a processor 11002, a communication interface 11003 and a bus 11004. The memory 11001, the processor 11002, and the communication interface 11003 are connected to each other through the bus 11004.

The memory 11001 may be a ROM, a static storage device, or a RAM. The memory 11001 may store a program. When the program stored in the memory 11001 is executed by the processor 11002, the processor 11002 and the communication interface 11003 are used to execute each step of the image restoration method of the embodiment of the present application.

Processor 11002 can be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits, which are used to execute relevant programs to implement the functions required to be performed by the units in the image processing device of the embodiment of the present application, or to execute the image restoration method of the method embodiment of the present application.

The processor 11002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image restoration method of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 11002 or by instructions in the form of software.

The processor 11002 may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 11001, and the processor 11002 reads the information in the memory 11001, and completes the functions required to be performed by the units included in the image processing device of the embodiment of the present application in combination with its hardware, or executes the image recovery method of the method embodiment of the present application.

The communication interface 11003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 11000 and other devices or a communication network. For example, the image to be restored can be obtained through the communication interface 11003.

The bus 11004 may include a path for transmitting information between various components of the device 11000 (eg, the memory 11001 , the processor 11002 , and the communication interface 11003 ).

It should be noted that although the above-mentioned apparatus 9000 and apparatus 11000 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 9000 and apparatus 11000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the apparatus 9000 and apparatus 11000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the apparatus 9000 and apparatus 11000 may also only include the devices necessary for implementing the embodiments of the present application, and do not necessarily include all the devices shown in Figures 20 and 22.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (10)

1. A training method for an image restoration network, comprising: Step A: Initializing the network parameters of the image generation network and the network parameters of the image restoration network to obtain initial values of the network parameters of the image generation network and the initial values of the network parameters of the image restoration network; Step B: inputting at least one of the multiple synthesized low-quality images into a first generation network in the image generation network for processing to obtain at least one approximate real low-quality image, wherein the multiple synthesized low-quality images are obtained by synthesizing multiple real high-quality images respectively, and the synthesis processing includes at least one of blurring processing, noise adding processing and downsampling processing; Step C: inputting at least one real low-quality image from a plurality of real low-quality images into a second generation network in the image generation network for processing to obtain at least one approximate synthetic low-quality image, wherein the plurality of real low-quality images and the image to be restored are acquired using the same device, and the image to be restored is an image processed by the image restoration network after training; Step D: inputting the at least one approximate real low-quality image into the second generation network for processing to obtain at least one reconstructed synthetic low-quality image; Step E: inputting the at least one approximately synthesized low-quality image into the first generation network for processing to obtain at least one reconstructed real low-quality image; Step F: inputting the at least one approximate real low-quality image into the image restoration network for processing to obtain at least one restored high-quality image; Step G: Determine a loss function, where the loss function includes a first loss function term, a second loss function term, and a third loss function term; The first loss function term includes an adversarial loss of the at least one approximate real low-quality image relative to any one of the multiple real low-quality images, and an adversarial loss of the at least one approximate synthetic low-quality image relative to any one of the multiple synthetic low-quality images; The second loss function term includes a difference between a pixel value of the at least one reconstructed synthetic low-quality image and a pixel value of the at least one synthetic low-quality image, and a difference between a pixel value of the at least one reconstructed real low-quality image and a pixel value of the at least one real low-quality image; The third loss function term includes a mean square error between the at least one restored high-quality image and at least one real high-quality image among the plurality of real high-quality images, wherein the at least one synthesized low-quality image is obtained by performing the synthesizing process on the at least one real high-quality image; Step H: updating the network parameters of the image generation network and the image restoration network according to the function value of the loss function; Repeat the above steps B to H until the image restoration performance of the image restoration network meets the preset requirements. 2. The training method as described in claim 1 is characterized in that the loss function also includes a fourth loss function term, and the fourth loss function term includes a perceptual loss and an adversarial loss between the at least one restored high-quality image and the at least one real high-quality image. 3. The training method according to claim 1 or 2, characterized in that the method further comprises: Inputting the at least one synthesized low-quality image into the second generation network for processing to obtain at least one transformed synthesized low-quality image; Inputting the at least one real low-quality image into the first generation network for processing to obtain at least one transformed real low-quality image; The loss function also includes a fifth loss function term, which includes the difference between the pixel values of the at least one converted real low-quality image and the pixel values of the at least one real low-quality image, and the difference between the pixel values of the at least one converted synthetic low-quality image and the pixel values of the at least one synthetic low-quality image. 4. The training method according to any one of claims 1 to 2, characterized in that the real low-quality image and the image to be restored are acquired by using the same device, comprising: The real low-quality image is acquired by using a first device, and the image to be restored is acquired by using a second device. The first device and the second device are of the same device type, and image acquisition parameters of the first device and the second device are the same, and the image acquisition parameters include at least one of focal length, exposure, and shutter time. 5. A training device for an image restoration network, comprising: An initialization unit, used to execute step A; Step A: Initializing the network parameters of the image generation network and the network parameters of the image restoration network to obtain the initial values of the network parameters of the image generation network and the initial values of the network parameters of the image restoration network; A training unit, used to repeat steps B to H until the image restoration performance of the image restoration network meets preset requirements; Step B: inputting at least one of the multiple synthesized low-quality images into a first generation network in the image generation network for processing to obtain at least one approximate real low-quality image, wherein the multiple synthesized low-quality images are obtained by synthesizing multiple real high-quality images respectively, and the synthesis processing includes at least one of blurring processing, noise adding processing and downsampling processing; Step C: inputting at least one real low-quality image from a plurality of real low-quality images into a second generation network in the image generation network for processing to obtain at least one approximate synthetic low-quality image, wherein the plurality of real low-quality images and the image to be restored are acquired using the same device, and the image to be restored is an image processed by the image restoration network after training; Step D: inputting the at least one approximate real low-quality image into the second generation network for processing to obtain at least one reconstructed synthetic low-quality image; Step E: inputting the at least one approximately synthesized low-quality image into the first generation network for processing to obtain at least one reconstructed real low-quality image; Step F: inputting the at least one approximate real low-quality image into the image restoration network for processing to obtain at least one restored high-quality image; Step G: Determine a loss function, where the loss function includes a first loss function term, a second loss function term, and a third loss function term; The first loss function term includes an adversarial loss of the at least one approximate real low-quality image relative to any one of the multiple real low-quality images, and an adversarial loss of the at least one approximate synthetic low-quality image relative to any one of the multiple synthetic low-quality images; The second loss function term includes a difference between a pixel value of the at least one reconstructed synthetic low-quality image and a pixel value of the at least one synthetic low-quality image, and a difference between a pixel value of the at least one reconstructed real low-quality image and a pixel value of the at least one real low-quality image; The third loss function term includes a mean square error between the at least one restored high-quality image and at least one real high-quality image among the plurality of real high-quality images, wherein the at least one synthesized low-quality image is obtained by performing the synthesizing process on the at least one real high-quality image; Step H: According to the function value of the loss function, the network parameters of the image generation network and the image restoration network are updated. 6. The training device as described in claim 5 is characterized in that the loss function also includes a fourth loss function term, and the fourth loss function term includes the perceptual loss and adversarial loss between the at least one restored high-quality image and the at least one real high-quality image. 7. The training device according to claim 5 or 6, characterized in that the loss function includes a fifth loss function term, and the training unit is further used to repeatedly perform step I and step J: Step I: inputting the at least one synthesized low-quality image into the second generation network for processing to obtain at least one converted synthesized low-quality image; Step J: inputting the at least one real low-quality image into the first generation network for processing to obtain at least one transformed real low-quality image; Wherein, the fifth loss function term includes the difference between the pixel values of the at least one converted real low-quality image and the pixel values of the at least one real low-quality image, and the difference between the pixel values of the at least one converted synthetic low-quality image and the pixel values of the at least one synthetic low-quality image. 8. The training device according to any one of claims 5 to 6, wherein the real low-quality image and the image to be restored are acquired by using the same device, comprising: The real low-quality image is acquired by using a first device, and the image to be restored is acquired by using a second device. The first device and the second device are of the same device type, and image acquisition parameters of the first device and the second device are the same, and the image acquisition parameters include at least one of focal length, exposure, and shutter time. 9. A computer-readable storage medium, characterized in that the computer-readable medium stores program codes for execution by a device, the program codes comprising code for executing the method according to any one of claims 1 to 4. 10. A chip, characterized in that the chip comprises a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1 to 4.