CN105976318A - Image super-resolution reconstruction method - Google Patents

Abstract

The invention discloses an image super-resolution reconstruction method, belonging to the technical field of image processing; the method comprises the following steps: extracting a low-resolution image Y in a training image database; performing bicubic interpolation and enlarging the obtained low-resolution image, and enlarging to the required size; design a convolutional neural network with a dynamic convolutional layer; input the low-resolution image Y into the pre-trained convolutional neural network B to obtain filters SH1 and SV1; combine the low-resolution image Y with the filter SV1, SH1 input pre-trained convolutional neural network containing dynamic convolutional layers; reconstruction of high-resolution image X; experimental results show that the algorithm proposed by the present invention not only achieves visual effects but also objective evaluation criteria The effect is more significant than the other three excellent algorithms, showing excellent super-resolution reconstruction performance.

Description

A method for image super-resolution reconstruction

technical field

The invention relates to the technical field of image processing, in particular to an image super-resolution reconstruction method.

Background technique

Image super-resolution reconstruction refers to the conversion of existing low-resolution images into high-resolution images through software algorithms. It is widely used in image printing, video surveillance, medical image processing, satellite imaging, criminal investigation analysis and other fields, and a large number of excellent algorithms have emerged. These algorithms can be roughly divided into three categories: image super-resolution algorithms based on interpolation, weight-based Constructed image super-resolution reconstruction algorithm, image super-resolution reconstruction algorithm based on learning.

In recent years, deep learning theory has developed rapidly. Unlike traditional feature extraction algorithms that rely on prior knowledge, deep neural networks can adaptively construct feature descriptions driven by training data, which has higher flexibility and universality. As an important technology for deep learning, convolutional neural network has a history of decades of development. Recently, deep convolutional neural network has become an explosive research hotspot due to its excellent performance in image classification, and has been successfully applied to other fields. field of computer vision. Therefore, convolutional neural networks can also be used to perform super-resolution reconstruction of images, directly using the original image as input, and under the drive of training data to obtain feature descriptions through autonomous learning, simplifying feature models and improving computing efficiency. Image super-resolution reconstruction using convolutional neural networks directly learns an end-to-end mapping between low-resolution and high-resolution images.

Contents of the invention

The object of the present invention is to propose a kind of image super-resolution reconstruction method, and this method comprises the following steps:

S1 blurs and down-samples the images in the high-resolution training image set according to the image degradation model to obtain the corresponding low-resolution training image set, and records the low-resolution image as Y;

S2 enlarges the obtained low-resolution image by bicubic interpolation and enlarges it to the required size;

S3 designs a convolutional neural network with dynamic convolutional layers;

S4 Input the low-resolution image Y into the pre-trained convolutional neural network B to obtain filters SH1 and SV1;

S5 inputs the low-resolution image Y and filters SV1, SH1 into a pre-trained convolutional neural network containing a dynamic convolutional layer;

S6 reconstructs the high-resolution image X;

The S3, a convolutional neural network with a dynamic convolutional layer consists of three parts:

S3.1 Extract image patches from the low-resolution image Y, each image patch is represented as a high-dimensional vector. These vectors include a set of feature maps equal in number to the dimensionality of the vectors:

F1(Y)＝max(0,W1*Y+B1)

W1 is the weight, B1 is the bias, the symbol * is convolution, W1 corresponds to n1 filters, that is, n1 convolutions are applied to the image, the size of the convolution kernel is c1×f1×f1, and the filter comes from an independent convolution Network B.

S3.2 Each high-dimensional vector is non-linearly mapped to another high-dimensional vector. Conceptually, each mapping vector is a representative of a high-resolution image patch, and these vectors include another set of feature maps;

F2(Y)＝max(0,W2*F1(Y)+B2)

W2 is weight, B2 is bias, symbol * is convolution, W2 corresponds to n2 filters, that is, n2 convolutions are applied to the image, and the size of each convolution kernel is n1×f2×f2, and the filter comes from an independent Convolutional Network B.

S3.3 aggregate the above-mentioned high-resolution image blocks to generate a final high-resolution image;

F(Y)＝W3*F2(Y)+B3

W3 is the weight, B3 is the deviation, the symbol * is convolution, W3 corresponds to n3 filters, that is, n3 convolutions are applied to the image, and the size of the convolution kernel is n2×f3×f3.

S5 includes the following

S5.1 Forward Propagation

Assume is the i-th feature map input based on sample t, is the jth feature map output based on sample t, is the convolution kernel, and its calculation formula is

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

Unlike traditional convolutional layers, convolutional neural networks with dynamic convolutional layers have different convolutional kernels for each convolutional layer.

S5.2 Backpropagation

The gradient loss function L relative to

$\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; L</span>}}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; L</span>}}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; )</span>\underset{<span\; class="notranslate">\; \&OverBar;</span>}{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{{\mathrm{<span\; class="notranslate">\; i</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>$

The symbol * means zero-padded convolution.

The gradient loss function L relative to

$\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; L</span>}}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; L</span>}}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

for transpose.

S4 includes the following

Unlike traditional convolutional layers, dynamic convolutional layers accept two inputs. The first input is the feature map of the previous layer, and the second input is the filter. The feature maps come from ConvNet A, and the filters come from a separate ConvNet B.

Convolutional network B structure:

1) Convolutional layer C1, input the same low-resolution training data as convolutional network A, and output n1 feature maps through n1 filters of size c1×f1×f1

2) Maximum value-downsampling layer M1, n1 feature maps generated by C1 layer, through a window with a step size of 2 and a size of 2×2

3) Convolutional layer C2, input n1 feature maps, pass through n2 size n1×f1×f1 filters, output n2 feature maps

4) Maximum value-downsampling layer M2, n2 feature maps generated by C2 layer, through a window with a step size of 2 and a size of 2×2;

5) Convolutional layer C3, input n2 feature maps, and output n3 feature maps through n3 large n2×f1×f1 filters;

6) Maximum value-downsampling layer M3, n3 feature maps generated by C3 layer, through a window with a step size of 2 and a size of 2×2;

7) The output of M3 is converted into a one-dimensional row vector H1 through a fully connected layer: 1×h1;

8) The output of M3 is converted into a one-dimensional column vector V1 through a fully connected layer: v1×1;

9) Apply the Softmax function to H1 and V1 to obtain vectors SH1 and SV1;

10) Filter SV1 is applied to the dynamic convolutional layer;

11) Filter SH1 is applied to the dynamic convolutional layer.

Description of drawings

Fig. 1 is the framework of the convolutional neural network image super-resolution reconstruction algorithm that contains the dynamic convolutional layer of the present invention;

Fig. 2 is that the present invention obtains the convolutional neural network B frame of dynamic convolution layer filter SV1 and SH1;

Fig. 3 is a comparison between the reconstructed results of the image enlarged by 2 times and processed by the convolutional neural network containing the dynamic convolution layer and other three algorithms using the present invention; wherein, a is the original image, b is bicubic interpolation, and c is improved The anchor point nearest neighbor regression algorithm, d is the super-resolution reconstruction algorithm based on the convolutional neural network, and e is the present invention.

Fig. 4 is a comparison between the reconstruction result of the image enlarged by 2 times and the convolutional neural network processing with the dynamic convolution layer and other three algorithms using the present invention; wherein, a is the original image, b is the bicubic interpolation, and c is the improvement The anchor point nearest neighbor regression algorithm, d is the super-resolution reconstruction algorithm based on the convolutional neural network, and e is the present invention

detailed description

With reference to Fig. 1, framework of the present invention is

Step 1, input low-resolution image Y;

Step 2, utilize the imresize function in the Matlab software to carry out 2 times of bicubic interpolation amplification to the low-resolution image to obtain the low-resolution image Y;

Step 3, the low-resolution image Y is input into the pre-trained convolutional network neural B to obtain filters SH1 and SV1;

Step 4, input the low-resolution image Y and the filter SV1 into the first dynamic convolutional layer of the pre-trained convolutional neural network containing the dynamic convolutional layer;

Step 5, the output in the previous step and the filter SH1 input the second dynamic convolutional layer of the convolutional neural network containing the dynamic convolutional layer;

Step 6, the output in the previous step is input into the third layer of the convolutional neural network containing the dynamic convolution layer to obtain a high-resolution image;

Step 7, reconstruct the high-resolution image.

In order to verify the effectiveness of the algorithm, it is compared with other three excellent algorithms on the test library set5 and test library 14 respectively. The four images in Fig. 3 are original images, Bicubic bicubic interpolation algorithm, A+ is an improved anchor point neighbor regression algorithm, SRCNN is a super-resolution reconstruction algorithm based on a convolutional neural network, and the image super-resolution reconstruction algorithm of the present invention. The four images in Fig. 4 are original images, Bicubic bicubic interpolation algorithm, A+ is an improved anchor point neighbor regression algorithm, SRCNN is a super-resolution reconstruction algorithm based on a convolutional neural network, and the image super-resolution reconstruction algorithm of the present invention.

Table 1 compares the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) of the reconstruction results in Fig. 3.

Table 1

Table 2 compares the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) of the reconstruction results in Fig. 4.

Table 2

Experimental results show that the algorithm proposed by the present invention achieves better results than the other three excellent algorithms not only in terms of visual effects but also in terms of objective evaluation criteria, and exhibits excellent super-resolution reconstruction performance.

Claims (4)

1. A method for image super-resolution reconstruction, characterized in that: the method comprises the following steps, S1 extracts the low-resolution image Y in the training image library; S2 enlarges the obtained low-resolution image by bicubic interpolation and enlarges it to the required size; S3 designs a convolutional neural network with dynamic convolutional layers; S4 Input the low-resolution image Y into the pre-trained convolutional neural network B to obtain filters SH1 and SV1; S5 inputs the low-resolution image Y and filters SV1, SH1 into a pre-trained convolutional neural network containing a dynamic convolutional layer; S6 reconstructs the high-resolution image X. 2. A kind of image super-resolution reconstruction method according to claim 1, is characterized in that: described S3, the convolutional neural network that contains dynamic convolutional layer comprises three parts: S3.1 Extract image blocks from the low-resolution image Y, each image block is represented as a high-dimensional vector; these vectors include a set of feature maps, the number of which is equal to the dimension of the vector: F1(Y)＝max(0,W1*Y+B1) W1 is weight, B1 is bias, symbol * is convolution, W1 corresponds to n1 filters, that is, n1 convolutions are applied to the image, and the size of each convolution kernel is c1×f1×f1, and n1 filters come from Independent convolutional network B; S3.2 Each high-dimensional vector is non-linearly mapped to another high-dimensional vector; conceptually, each mapped vector is a representative of a high-resolution image patch, and these vectors include another set of feature maps; F2(Y)＝max(0,W2*F1(Y)+B2) W2 is weight, B2 is bias, symbol * is convolution, W2 corresponds to n2 filters, that is, n2 convolutions are applied to the image, and the size of each convolution kernel is n1×f2×f2, and n2 filters come from Independent convolutional network B; S3.3 aggregate the above-mentioned high-resolution image blocks to generate a final high-resolution image; F(Y)＝W3*F2(Y)+B3 W3 is the weight, B3 is the deviation, the symbol * is convolution, W3 corresponds to n3 filters, that is, n3 convolutions are applied to the image, and the size of each convolution kernel is n2×f3×f3. 3. A kind of image super-resolution reconstruction method according to claim 1, is characterized in that: S5 comprises as follows, S5.1 Forward Propagation Assume is the i-th feature map input based on sample t, is the jth feature map output based on sample t, is the convolution kernel, and its calculation formula is ${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$ Different from the traditional convolutional layer, the convolutional kernel of each convolutional layer in the convolutional neural network with dynamic convolutional layer is different; S5.2 Backpropagation The gradient loss function L relative to $\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; L</span>}}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; L</span>}}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; )</span>\underset{<span\; class="notranslate">\; \&OverBar;</span>}{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{{\mathrm{<span\; class="notranslate">\; i</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>$ The symbol * means zero padding convolution; The gradient loss function L relative to $\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; L</span>}}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; \&part;</span>\mathrm{<span\; class="notranslate">\; L</span>}}{<span\; class="notranslate">\; \&part;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; *</span>\stackrel{<span\; class="notranslate">\; ~</span>}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}$ for transpose. 4. A kind of image super-resolution reconstruction method according to claim 1, is characterized in that: S4 comprises as follows, is different with traditional convolution layer, and dynamic convolution layer accepts two inputs; The first input is last The feature map of the layer, the second input is the filter; the feature map comes from the convolutional network A, and the filter comes from the independent convolutional network B; Convolutional network B structure: 1) Convolutional layer C1, input the same low-resolution training data as convolutional network A and convolutional network A, output n1 feature maps through n1 filters of size c1×f1×f1 2) Maximum value - downsampling layer M1, n1 feature map generated by C1 layer, through a window with a step size of 2 and a size of 2×2 3) Convolutional layer C2, input n1 feature map, pass n2 filters of size c2×f2×f2, and output n2 feature maps 4) Maximum value-downsampling layer M2, the n2 feature map generated by the C2 layer, through a window with a step size of 2 and a size of 2×2; 5) Convolutional layer C3, input n2 feature maps, and output n3 feature maps through n3 filters of size c3×f3×f3; 6) Maximum value-downsampling layer M3, the n3 feature map generated by the C3 layer, through a window with a step size of 2 and a size of 2×2; 7) The output of M3 is converted into a one-dimensional row vector H1 through a fully connected layer: 1×h1; 8) The output of M3 is converted into a one-dimensional column vector V1 through a fully connected layer: v1×1; 9) Apply the Softmax function to H1 and V1 to obtain vectors SH1 and SV1; 10) Filter SV1 is applied to the dynamic convolutional layer; 11) Filter SH1 is applied to the dynamic convolutional layer.