CN116563144A - Dynamic attention-based intensive LSTM residual network denoising method - Google Patents

Abstract

A dense LSTM residual network denoising method based on dynamic attention comprises constructing a training data set, and preprocessing the training data set; constructing a denoising model by using a dynamic attention mechanism and a dense LSTM residual error network; setting super parameters and loss functions of a network denoising model, adding different levels of noise to a training data set, and training to obtain a trained network model; and (3) carrying out image denoising test by adopting the trained network model, and evaluating denoising effect by using the structural similarity and the peak signal-to-noise ratio. The method has the beneficial effects of improving the noise reduction performance and the imaging quality, effectively improving the image quality, and being beneficial to understanding and analyzing tasks for high-level images such as subsequent image classification, segmentation, identification and tracking.

Description

A Dense LSTM Residual Network Denoising Method Based on Dynamic Attention

technical field

The invention belongs to the field of computer vision and image processing, and in particular relates to a dynamic attention-based dense LSTM residual network denoising method.

Background technique

In recent years, with the continuous development of the information society, image information has become a main way for people to receive information. In the process of acquisition and transmission, digital images are affected by environmental, equipment, human factors, etc., which inevitably introduce noise, thereby reducing image quality and affecting image readability. Image denoising is one of the very important methods to improve image quality. Image denoising has been used in a variety of fields over the years. For example, in the medical field, noise-containing medical images are denoised to improve the clarity of the image, thereby helping medical staff to better judge the patient's condition; in aerospace and other fields, images are often damaged during the imaging process. The interference of electronic equipment causes the image to contain Gaussian white noise, salt and pepper noise, etc., which need to be improved through image denoising. Hand-held mobile devices or vehicle-mounted recorders often produce noise due to factors such as light and environment when shooting at night, which affects image quality and brings a lot of inconvenience to subsequent image analysis and processing. At the same time, denoising is convenient for subsequent image classification, segmentation and recognition tasks.

Image denoising is a classic problem in the field of image processing, and it is also an important preprocessing step in computer vision. Traditional image denoising algorithms rely on strong prior model assumptions and manual parameter adjustment, which involve complex optimization problems and require a lot of time and cost. With the continuous improvement of deep learning theory in recent years, relevant experts and scholars have successfully applied it to the field of image denoising and achieved excellent results. Deep neural network effectively extracts feature information from images through operations such as convolution and activation, and contributes a lot to image denoising research. At present, the attention mechanism is often used in the construction of convolutional neural networks, and is widely used in the fields of natural language understanding, image denoising and recognition. Although the attention mechanism can improve the denoising effect of the convolutional neural network, it is necessary to manually set the position of the attention module in the convolutional neural network, and the strength of the effect is different at different positions, which leads to the effective improvement of the convolutional neural network through the attention mechanism. Network performance requires multiple experiments on different network structures, which greatly increases labor costs.

Contents of the invention

In order to solve the above technical problems, the present invention provides a dynamic attention-based dense LSTM residual network denoising method, by considering the spatial attention weight and the inter-channel attention weight, so as to better guide the network to denoise, And ensure the denoising quality of the image. By designing the LSTM structure, taking into account the dependence of global information and local information, this method has significantly improved the noise reduction performance and imaging quality.

A kind of dense type LSTM residual network denoising method based on dynamic attention, it is characterized in that: comprise the steps:

Step S1, constructing a training data set, and performing a preprocessing operation on the data set;

Step S2, establishing a denoising network model composed of a dynamic attention mechanism and a dense LSTM residual network;

Step S3, setting hyperparameters and loss functions of the denoising network model;

Step S4, adding different levels of noise to the training data set, and training the network to obtain a trained denoising network model;

Step S5, input the test set into the trained denoising network model, obtain the denoised image, and use structural similarity and peak signal-to-noise ratio to evaluate the noise image (the image to be evaluated is the denoised image output by the network model and the original image Substitute into the calculation formula for evaluation); put the denoising network model that has passed the evaluation into use, input the image to be processed, and output the denoised image.

Further, in step S1, the preprocessing operation on the training data set includes the following steps:

S1.1), selecting training samples from the training data set as the original training set, wherein the images in the training samples are all noise-free images of the same size;

S1.2), the images in the training data set are respectively scaled by 1 times, 0.9 times, 0.8 times, and 0.7 times, and each image after scaling is segmented using a sliding fixed-size segmentation window;

S1.3), performing an augmentation operation on the segmented image, the method of the augmentation operation includes: flipping the image up and down, rotating by 90°, rotating by 180°, rotating by 270°, rotating by 90° after flipping up and down, 180° rotation after flipping up and down, 270° rotation after flipping up and down;

Further, in step S2, the denoising network model is composed of convolution, LReLU and 8 dynamic attention modules, where the dynamic attention module includes front-end and back-end residual units, non-attention branches, attention branches and weight distribution branch, and adopts the LSTM structure inside the dynamic attention module, and the convolution kernel size used inside the network model is 3×3 and 5×5.

Further, the residual unit of the front end and the back end is a residual module, which is composed of two convolutional layers with a convolution kernel size of 3×3 and an LReLU activation function. Let Y be the input, and the mathematical expression is as follows:

X ₀ ＝ψ(W ₂ *ψ(W ₁ *Y+b ₁ )+b ₂ )

In the above formula, W ₁ and W ₂ represent a 3×3 convolutional layer with 64 output channels, b ₁ and b ₂ are biases, ψ represents the LReLU activation function, and X ₀ represents the extracted shallow features;

Define Xi _-1 to represent the input of the dynamic attention module; the front-end feature extraction part consists of a residual module and an LReLU activation function, and the corresponding mathematical expression is:

In the formula, it represents the E residual module; then, the Input the attention branch and non-attention branch respectively.

Further, the non-attention branch consists of 2 residual units and 1 LSTM module to form a backbone network, where the residual unit contains 1 3×3 convolutional layer and 1 5×5 convolutional layer, and the LSTM module consists of Recurrent network structure composed of residual module, LSTM unit, convolution layer and attention mask layer; the mathematical expression of LSTM unit is as follows

i _t ＝σ(W _xi *X _t +W _hi *H _t-1 +W _ci ⊙C _t-1 +b _i )

f _t ＝σ(W _xf *X _t +W _hf *H _t-1 +W _cf ⊙C _t-1 +b _f )

C _t ＝f _t ⊙C _t-1 +i _t ⊙tanh(W _xc *X _t +W _hc *H _t-1 +b _c )

o _t ＝σ(W _xo *X _t +W _ho *H _t-1 +W _co ⊙C _t +b _o )

H _t ＝o _t ⊙tanh(C _t )

Among them, LSTM is composed of input gate _it , forgetting gate f _t , output gate o _t , long-term memory state C _t and short-term memory state _{H t} ; Convolution operation from j to k, ⊙ represents matrix point multiplication, b _j represents the activation value of unit j; X _t represents the feature map obtained by the residual module, C _t encodes the features sent to the next LSTM, H _t As the output of the current LSTM unit, and input to the LSTM unit.

Further, the attention branch consists of a backbone network B _i and a mask part M _i , expressed as:

In the above formula, F ^att represents the attention branch, i represents the i-th dynamic attention module, the backbone network _Bi contains 2 residual modules, and the mask part _Mi contains dilated convolution, channel attention module, downsampling operation, Upsampling operation and Sigmoid function.

Further, the weight assignment branch is composed of a spatial attention module and a channel attention module, wherein the spatial attention module includes a pair of maximum pooling operations and average pooling operations, and the feature maps are spliced after two pooling operations into a large feature map, and then the spatial weight is obtained by the Sigmoid activation function, and input to the channel attention module; the channel attention module is composed of an average pooling layer, two linear layers and a Softmax activation function, and will eventually produce a dynamic The attention weight distribution is used for non-attention branch and attention branch.

Further, in step S3, the hyperparameters of the denoising network model include batch size, initial learning rate, number of iterations, and learning rate decay strategy; the loss function is the mean square error, and its mathematical expression is:

In the formula, Θ is the dense LSTM residual network parameter of the dynamic attention mechanism, R(y _i ; Θ) is the residual image learned by the network, y _i is the noise image, _xi represents the clean image, and N is the training sample number; the Train400 data set is used as the training data set, and 40 image blocks with a size of 128×128 are randomly extracted in each round as samples, and the Adam optimizer is used for training, and the initial learning rate is set to 1×10 ^-4 . The learning rate is reduced by 0.2 times in 10 rounds, and the network is trained for 100 rounds in total.

Further, in step S4, the training method of the denoising network model is:

S4.1), adding Gaussian white noise with noise levels of 15, 25, and 50 to the images in the original data training set;

S4.2), inputting the noise-added training image into the network model for training, so as to obtain a trained denoising network model, and save it. The specific training steps are realized by running PYcharm.

Further, in step S5, the calculation formula of structural similarity is:

where x and y are two images, is the mean value; /> is the variance; σ _xy is the covariance of x and y; d ₁ =(k ₁ L) ² , d ₂ =(k ₂ L) ² , L is the dynamic range of the pixel; k ₁ =0.01, k ₂ =0.03;

The calculation formula of peak signal-to-noise ratio is:

In the formula: x(i,j) and y(i,j) respectively represent the pixel values of the corresponding positions in the initial image x(i,j) and the denoised image y(i,j), and Q represents the image The maximum gray value in the medium.

The beneficial effects that the present invention reaches are:

(1) By considering the spatial attention weight and the inter-channel attention weight, it can better guide the network to denoise and ensure the denoising quality of the image.

(2) By designing the LSTM structure, taking into account the dependence of global information and local information, this method has significantly improved the noise reduction performance and imaging quality.

(3) It is beneficial for subsequent high-level image understanding and analysis tasks such as image classification, segmentation, recognition and tracking.

Description of drawings

Fig. 1 is a flow chart of the method in the embodiment of the present invention.

Fig. 2 is a schematic diagram of a denoising network model in an embodiment of the present invention.

Fig. 3 is a schematic diagram of the dynamic attention module in the network in the embodiment of the present invention.

Fig. 4 is a table of PSNR values of denoising results of various algorithms in the embodiment of the present invention.

Fig. 5 is a table of SSIM values of denoising results of various algorithms in the embodiment of the present invention.

Detailed ways

The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, a dense LSTM residual network denoising algorithm based on dynamic attention includes the following steps:

Step S1: Construct a training data set, and perform preprocessing operations on the data set.

In step S1, the preprocessing operation includes the following steps:

S1.1), selecting training samples from the training data set as the original training set, wherein the images in the training samples are all noise-free images of the same size; in this embodiment, the capacity of the training samples is 400 images with a height of 180 pixels and a width of 180 pixels.

S1.2), the images in the training data set are respectively scaled by 1 times, 0.9 times, 0.8 times, and 0.7 times, and each image after scaling is segmented using a sliding fixed-size segmentation window.

S1.3), performing an augmentation operation on the segmented image, the method of the augmentation operation includes: flipping the image up and down, rotating by 90°, rotating by 180°, rotating by 270°, rotating by 90° after flipping up and down, 180° rotation after flipping up and down, 270° rotation after flipping up and down.

S1.4), adding Gaussian white noise with noise levels of 15, 25, and 50 to the images in the data set.

Step S2: Use a denoising algorithm composed of a dynamic attention mechanism and a dense LSTM residual network.

As shown in Figure 2, the network denoising model consists of convolution operation, LReLU and 8 dynamic attention modules, where the dynamic attention module includes spatial attention, channel attention, residual module, splicing layer, etc. And the LSTM structure is used inside the dynamic attention module, and the convolution kernel sizes used in the network model are both 3×3 and 5×5.

As shown in FIG. 3 , the components of the dynamic attention module include: a front-end and a back-end residual unit, a non-attention branch, an attention branch and a weight distribution branch. The residual unit of the front end and the back end is a residual module, which is composed of two convolutional layers with a convolution kernel size of 3×3 and an LReLU activation function. Let Y be the input, and the mathematical expression is as follows:

X ₀ ＝ψ(W ₂ *ψ(W ₁ *Y+b ₁ )+b ₂ )

In the above formula, W ₁ and W ₂ represent a 3×3 convolutional layer with 64 output channels, b ₁ and b ₂ are biases, ψ represents the LReLU activation function, and X ₀ represents the extracted shallow features.

Let Xi _-1 denote the input of the dynamic attention module. The front-end feature extraction part consists of a residual module and an LReLU activation function, and the corresponding mathematical expression is:

where represents the E residual module. Next, put Input the attention branch and non-attention branch respectively.

The non-attention branch consists of 2 residual units and 1 LSTM module to form the backbone network, where the residual unit contains two layers of 3×3 and 5×5 convolutional layers, and the LSTM module consists of a residual module, LSTM unit, convolutional layer and the attention mask layer to form a recurrent network structure. The mathematical expression of the LSTM unit is as follows:

i _t ＝σ(W _xi *X _t +W _hi *H _t-1 +W _ci ⊙C _t-1 +b _i )

f _t ＝σ(W _xf *X _t +W _hf *H _t-1 +W _cf ⊙C _t-1 +b _f )

C _t ＝f _t ⊙C _t-1 +i _t ⊙tanh(W _xc *X _t +W _hc *H _t-1 +b _c )

o _t ＝σ(W _xo *X _t +W _ho *H _t-1 +W _co ⊙C _t +b _o )

H _t ＝o _t ⊙tanh(C _t )

Among them, LSTM is composed of input gate _it , forget gate _ft , output gate _ot , long-term memory state _Ct and short-term memory state _Ht . σ and tanh represent the Sigmoid activation function and Tanh activation function respectively, W _jk represents the convolution operation from j to k, ⊙ represents the point multiplication between matrices, and b _j represents the activation value of the jth unit. X _t represents the feature map obtained by the residual module, C _t encodes the features sent to the next LSTM, H _t is output as the current LSTM unit, and is input to the LSTM unit.

The attention branch consists of a backbone network _Bi and a mask part _Mi , which can be expressed as:

In the above formula, the backbone network B _i contains 2 residual modules, and the mask part M _i includes hole convolution, channel attention module, downsampling operation, upsampling operation and Sigmoid function.

The weight distribution branch consists of a spatial attention module and a channel attention module. The spatial attention module includes a pair of maximum pooling operations and average pooling operations. The feature maps are spliced into a large feature map after two pooling operations. , and then the spatial weight is obtained by the Sigmoid activation function, and input to the channel attention module. The channel attention module consists of an average pooling layer, two linear layers and a Softmax activation function, which will eventually produce a dynamic attention weight distribution for the non-attention branch and the attention branch.

Step S3: setting hyperparameters and loss functions of the denoising network model.

The hyperparameters of the network model include batch size, initial learning rate, number of iterations, and learning rate decay strategy.

The loss function is the mean square error, and its mathematical expression is:

In the formula, Θ is the dense LSTM residual network parameter of the dynamic attention mechanism, R(y _i ; Θ) is the residual image learned by the network, y _i is the noise image, _xi represents the clean image, and N is the training sample number. The Train400 data set is used as the training data set, and 40 image blocks with a size of 128×128 are randomly extracted in each round as samples, and the Adam optimizer is used for training. The initial learning rate is set to 1×10 ^-4 , and the learning rate is Dropped by 0.2 times, the network has been trained for a total of 100 rounds.

Step S4: Add different levels of noise to the training data set, and train the network to obtain a trained network model.

In step S4, the training method of the denoising network model is:

S4.1): adding Gaussian white noise with noise levels of 15, 25, and 50 to the images in the original data training set;

S4.2): Input the noise-added training image into the network model for training, so as to obtain a trained denoising network model, and save it.

Step S5: Input the test set to the network to obtain the denoised image, and evaluate the noisy image with structural similarity and peak signal-to-noise ratio.

The test set includes Set12, BSD68 grayscale data set, and the test set is input into the trained network model for image denoising.

The formula for calculating structural similarity is:

where x and y are two images, is the mean value; /> is the variance; σ _xy is the covariance of x and y; d ₁ =(k ₁ L) ² , d ₂ =(k ₂ L) ² , L is the dynamic range of the pixel; k ₁ =0.01, k ₂ =0.03.

The calculation formula of peak signal-to-noise ratio is:

In the formula, x(i,j) and y(i,j) respectively represent the pixel values of the corresponding positions in the initial image x(i,j) and the denoised image y(i,j), and Q represents the image The maximum gray value in the medium.

Figure 4 and Figure 5 show the PSNR and SSIM index results of various algorithms on different data sets. This method achieves good PSNR results under three different noise conditions. On the Set12 test data set, when the noise intensity is 15 and 50, the average PSNR indicators of this method reach 33.05dB and 27.59dB respectively, which is 0.20dB higher than the FFDNet algorithm. In the case of noise intensity 25, the PSNR of the algorithm of the present invention on Set12 is slightly lower than that of FDnCNN. In the case of high noise intensity 15 and 25, the SSIM index results of our method are better than other algorithms.

The above descriptions are only preferred embodiments of the present invention, and the scope of protection of the present invention is not limited to the above embodiments, but all equivalent modifications or changes made by those of ordinary skill in the art according to the disclosure of the present invention should be included within the scope of protection described in the claims.

Claims (10)

1. A dynamic attention-based intensive LSTM residual network denoising method is characterized in that: the method comprises the following steps:

s1, constructing a training data set, and preprocessing the data set;

step S2, a denoising network model formed by a dynamic attention mechanism and a dense LSTM residual network is established;

step S3, setting super parameters and loss functions of the denoising network model;

step S4, adding different levels of noise to the training data set, and training the network to obtain a trained denoising network model;

s5, inputting a test set to the trained denoising network model to obtain a denoised image, and evaluating the noise image by using the structural similarity and the peak signal-to-noise ratio; and putting the denoising network model which passes the evaluation into use, inputting an image to be processed, and outputting a denoised image.

2. The method for denoising the intensive LSTM residual network based on dynamic attention according to claim 1, wherein the method comprises the following steps: in step S1, the preprocessing operation for the training data set includes the following steps:

s1.1), selecting a training sample from the training data set as an original training set, wherein images in the training sample are noise-free images with the same size;

s1.2), scaling the images in the training data set according to 1 times, 0.9 times, 0.8 times and 0.7 times respectively, sliding a segmentation window with a fixed size, and extracting image blocks from each scaled image;

s1.3), performing an augmentation operation on the segmented image, wherein the augmentation operation method comprises the following steps: the image is turned up and down, rotated by 90 degrees, rotated by 180 degrees, rotated by 270 degrees, rotated by 90 degrees after being turned up and down, rotated by 180 degrees after being turned up and down, and rotated by 270 degrees after being turned up and down.

3. The method for denoising the intensive LSTM residual network based on dynamic attention according to claim 2, wherein the method comprises the following steps: in step S2, the denoising network model is composed of convolution, lrehu and 8 dynamic attention modules, wherein the dynamic attention modules include residual units of front end and back end, non-attention branches, attention branches and weight distribution branches, and an LSTM structure is adopted inside the dynamic attention modules, and the convolution kernel size used inside the network model is 3×3 and 5×5.

4. A method for denoising an intensive LSTM residual network based on dynamic attention according to claim 3, wherein: the residual units of the front end and the rear end are residual modules, the modules are composed of 2 convolution layers with the convolution kernel size of 3 multiplied by 3 and an LReLU activation function, Y is taken as an input, and the mathematical expression is as follows:

X ₀ ＝ψ(W ₂ *ψ(W ₁ *Y+b ₁ )+b ₂ )

w in the above ₁ And W is ₂ 3 x 3 convolutional layer representing output channel 64, b ₁ And b ₂ For bias, ψ represents the LReLU activation function, X ₀ Representing the extracted shallow features;

definition X _i-1 An input representing a dynamic attention module; the front-end feature extraction part consists of 1 residual error module and LReLU activation function, and the corresponding mathematical expression is:

wherein the formula represents an E residual error module; next, the process willThe attention branches and the non-attention branches are input respectively.

5. A method for denoising an intensive LSTM residual network based on dynamic attention according to claim 3, wherein: the non-attention branch consists of a main network composed of 2 residual units and 1 LSTM module, wherein the residual units comprise 1 3×3 convolution layer and 15×5 convolution layer, and the LSTM module is a cyclic network structure composed of the residual units, the LSTM units, the convolution layer and the attention mask layer; the LSTM unit mathematical expression is as follows

i _t ＝σ(W _xi *X _t +W _hi *H _t-1 +W _ci ⊙C _t-1 +b _i )

f _t ＝σ(W _xf *X _t +W _hf *H _t-1 +W _cf ⊙C _t-1 +b _f )

C _t ＝f _t ⊙C _t-1 +i _t ⊙tanh(W _xc *X _t +W _hc *H _t-1 +b _c )

o _t ＝σ(W _xo *X _t +W _ho *H _t-1 +W _co ⊙C _t +b _o )

H _t ＝o _t ⊙tanh(C _t )

Wherein LSTM is defined by input gate i _t Forgetting door f _t Output door o _t State C of long-term memory _t And a short-term memory state H _t Composition; sigma and Tanh represent the Sigmoid activation function and the Tanh activation function, respectively, W _jk Indicating the convolution operation from j to k, +. _j An activation value representing a j-th cell; x is X _t Representing the feature map obtained by the residual block, C _t Encoding features to the next LSTM, H _t As the current LSTM cell output, and input a next LSTM cell.

6. A method for denoising an intensive LSTM residual network based on dynamic attention according to claim 3, wherein: the attention branch is formed by a backbone network B _i And a mask portion M _i The composition is expressed as:

f in the above ^att Representing the attention branch, i representing the ith dynamic attention module, backbone network B _i Comprising 2 residual modules, mask portion M _i Comprising hollow rollsProduct, channel attention module, downsampling operation, upsampling operation, and Sigmoid function.

7. A method for denoising an intensive LSTM residual network based on dynamic attention according to claim 3, wherein: the weight distribution branch consists of a space attention module and a channel attention module, wherein the space attention module comprises a pair of maximum pooling operation and average pooling operation, the feature images are spliced into a large feature image after the two pooling operations respectively, and then the space weight is obtained by a Sigmoid activation function and is input to the channel attention module; the channel attention module consists of an average pooling layer, two linear layers and a Softmax activation function, and finally, the dynamic attention weight distribution of output is used for non-attention branches and attention branches.

8. A method for denoising an intensive LSTM residual network based on dynamic attention according to claim 3, wherein: in step S3, the super parameters of the denoising network model comprise batch size, initial learning rate, iteration times and learning rate attenuation strategies; the loss function is a mean square error, and the mathematical expression is:

where Θ is the dense LSTM residual network parameter for dynamic attention mechanism, R (y _i The method comprises the steps of carrying out a first treatment on the surface of the Θ) is a network learned residual image, y _i Is a noise image, x _i Representing a clean image, wherein N is the number of training samples; adopting Train400 data set as training data set, setting network super parameter, randomly extracting 40 image blocks with 128×128 size as samples in each round, training by Adam optimizer, and setting initial learning rate to 1×10 ^-4 Every 10 rounds of learning rate is reduced by 0.2 times, and the total training of the network is 100 rounds.

9. The method for denoising the intensive LSTM residual network based on dynamic attention according to claim 8, wherein the method comprises the following steps: in step S4, the training method of the denoising network model is as follows:

s4.1), respectively adding Gaussian white noise with noise levels of 15, 25 and 50 to the images in the original data training set;

s4.2), inputting the training image added with noise into the network model for training, thereby obtaining a trained denoising network model, and storing the trained denoising network model.

10. The method for denoising the intensive LSTM residual network based on dynamic attention according to claim 9, wherein the method comprises the following steps: in step S5, the calculation formula of the structural similarity is:

where x and y are two images,is the mean value; />Is the variance; sigma (sigma) _xy Is the covariance of x and y; d, d ₁ ＝(k ₁ L) ² ,d ₂ ＝(k ₂ L) ² L is the dynamic range of the pixel; k (k) ₁ ＝0.01,k ₂ ＝0.03；

The peak signal-to-noise ratio is calculated as:

where x (i, j) and y (i, j) represent pixel values at corresponding positions in the original image x (i, j) and the denoised image y (i, j), respectively, and Q represents a maximum gray value in the image.