CN110598600A - Remote sensing image cloud detection method based on UNET neural network - Google Patents

Abstract

The invention discloses a remote sensing image cloud detection method based on UNET neural network, comprising the following steps: establishing a cloud detection network with 5 down-sampling layers and 5 up-sampling layers, wherein the first four down-sampling layers of the cloud detection network are Sampling layer, each layer is followed by a convolutional layer and a pooling layer, and each of the last four upsampling layers is followed by a deconvolution layer; cloud labeling, manual re-inspection, and data enhancement processing are performed on the original remote sensing image set, and the processing The final remote sensing image set is divided into a training set, an evaluation set and a test set; the cloud detection network is continuously optimized by using the training set and the test set; the cloud detection network is used to detect the remote sensing image, and the cloud is output A picture of the test result. The invention solves the problem of unsatisfactory detection results due to insufficient extraction of cloud features, improves the detection accuracy, and strengthens the universality of the algorithm.

Description

A Cloud Detection Method of Remote Sensing Image Based on UNET Neural Network

technical field

The invention relates to the field of remote sensing image detection, and more specifically, relates to a remote sensing image cloud detection method based on a UNET neural network.

Background technique

With the rapid development of satellite remote sensing technology, remote sensing images have been widely used in the fields of environment, agriculture, and meteorology. However, satellite remote sensing images of optical imaging are often easily affected by the weather, resulting in cloud occlusion in the image, which affects the further application and analysis of the image; images can only be taken when the weather conditions permit and there is no cloud at all, which not only affects timeliness, but also increases cost of operating the satellite.

In the prior art, the processing of remote sensing images needs to extract various parameters from the pictures, or transform and map the pictures, and then perform threshold judgment. This process is very complicated and requires relatively large image data, and the fault tolerance is not high. Not conducive to promotion. In the prior art, the detection of images with different backgrounds has different success rates, and there are certain requirements for the accuracy of the spectrum and the level of preprocessing, otherwise the error will be large and the adaptability is not strong.

Contents of the invention

In view of this, the main purpose of the present invention is to provide a remote sensing image cloud detection method based on the UNET neural network, which solves the problem of unsatisfactory detection results due to insufficient cloud feature extraction, improves detection accuracy, and strengthens the algorithm. universality.

The remote sensing image cloud detection method based on UNET neural network of the present invention, specific technical scheme comprises:

Set up a cloud detection network with 5 downsampling layers, 5 upsampling layers, and 4 skip connection chains, wherein the first four downsampling layers of the cloud detection network are each followed by a pooling layer, and the last four upsampling layers are each followed by a pooling layer. Each layer of the sampling layer is followed by a deconvolution layer;

Perform cloud labeling, manual re-inspection, and data enhancement processing on the original remote sensing image set, and divide the processed remote sensing image set into a training set, an evaluation set, and a test set;

Using the images in the training set to train the cloud detection network, and constantly update the parameters of the cloud detection network;

Using the images in the test set to test the accuracy of the trained cloud detection network, and perform parameter adjustment and retraining of the cloud detection network;

The cloud detection network is used to perform cloud detection on the remote sensing image, and the result picture of the cloud detection is output.

Further, the method includes: the fifth down-sampling layer of the cloud detection network is connected to the first up-sampling layer.

Further, it includes: the last upsampling layer of the cloud detection network is connected to the fully connected layer.

Further, performing cloud labeling on the original remote sensing image set includes: using ENVI software to perform cloud labeling on the remote sensing image set.

Further, performing data enhancement on the original remote sensing image set includes: performing rotation translation, elastic deformation, and contrast enhancement on the remote sensing image set.

Further, using the images in the test set to test the accuracy of the trained cloud detection network, performing parameter adjustment and retraining of the cloud detection network includes: according to the performance of the loss function in training, the training The number of times and the batch size of training are adjusted.

Further, dividing the processed remote sensing image set into a training set, an evaluation set and a test set further includes: randomly selecting 70% of the images from the remote sensing image set as a training set, and 20% of the images as an evaluation set , and the remaining 10% is used as the test set.

Further, it includes: the activation function of the cloud detection network is a ReLU function.

Further, the cloud detection network is trained using the images in the training set, and the parameters of the cloud detection network are continuously updated, including: the parameters of the cloud detection network can be convolution weights and convolution biases value.

Further, use the images in the test set to test the accuracy of the trained cloud detection network, and perform parameter adjustment and retraining of the cloud detection network, including: setting the learning rate to 0.001.

The present invention utilizes a deep learning model UNET network to down-sample an optical remote sensing image to extract image features, and then combine some skip connections with up-sampling to generate a target detection image. It solves the problem of unsatisfactory detection results due to insufficient cloud feature extraction, improves the detection accuracy, strengthens the universality of the algorithm, and is more conducive to the development of scientific research and economic activities, with high intelligence and convenience.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

Fig. 1 is a schematic flow sheet of the present invention;

Fig. 2 is a cloud detection network structural diagram of the present invention;

Fig. 3 is a schematic diagram of the dotted line in the cloud detection network structure diagram of the present invention;

Fig. 4 is the training set sample diagram of the embodiment of the present invention, wherein the left half is the original image of the remote sensing image, and the right half is the corresponding labeled diagram;

Fig. 5 is the original picture of the remote sensing image detected by the embodiment of the present invention;

Fig. 6 is an annotated diagram after annotating the original image of the remote sensing image according to the embodiment of the present invention;

Fig. 7 is a network detection result diagram of the original image of the remote sensing image according to the embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

The main task of the UNET neural network-based remote sensing image cloud detection method proposed by the present invention is to build and train a UNET network, which uses remote sensing images as input and outputs the remote sensing image cloud mask. Use the deep learning platform to realize the construction of the network, and the training of the network includes the production of data sets and the process of training and tuning parameters. The production of the data set in the embodiment of the present invention does not require pairing of pictures and labels like traditional supervised learning. It only needs to collect a large number of satellite original remote sensing images, use ENVI software to perform preliminary cloud annotation on the original remote sensing images, and then manually check and modify them. The cloud detection mask map corresponding to the original remote sensing image is obtained, and the cloud detection mask map is used as the training set.

A kind of remote sensing image cloud detection method based on UNET neural network that the embodiment of the present invention provides, specifically comprises the following steps:

1. Establish a UNET network model:

UNET adopts a network structure including down-sampling and up-sampling. The UNET network can be simply regarded as down-sampling first, and after different degrees of convolution, deep-level features are learned. After up-sampling, it is restored to the size of the original image. Sampling is implemented using deconvolution. Downsampling is used to gradually reveal environmental information, and the process of upsampling is to combine downsampled layers of information and upsampled input information to restore detailed information and gradually restore image accuracy.

The up-sampling stage and the down-sampling stage of the UNET network use the same number of convolution operations, and use the skip connection chain structure to connect the down-sampling layer to the up-sampling layer, so that the features extracted by the down-sampling layer can be directly transferred to the up-sampling layer. layer, which makes the pixel positioning of the UNET network more accurate and the segmentation accuracy higher.

As shown in Figure 2, the cloud detection network established by the embodiment of the present invention is based on the UNET neural network, including 5 down-sampling layers and 5 up-sampling layers, wherein the first four down-sampling layers are each followed by a pooling layer , each of the last four upsampling layers is followed by a deconvolution layer, and has four skip connection chains, the fifth downsampling layer is connected to the first upsampling layer, and the last upsampling layer is connected to the fully connected layer output , the activation function is the ReLU function.

The role of the downsampling layer is to extract high-dimensional feature maps from the matrix corresponding to the remote sensing image through a large number of convolution kernels, and multiple feature maps can be extracted. Each feature map is a local perception extracted from the picture, and these features are integrated Graphs can extract interesting parts of a picture. In the embodiment of the present invention, a convolution kernel with a size of 3*3 and a step size of 1 is used to convert the image in the 3*3 window of the original remote sensing image into one pixel, and perform two convolutions on each convolution layer, each The number of convolution kernels in each layer is doubled after each downsampling, and the number of convolution kernels in the next downsampling layer is twice the number of convolution kernels in the previous downsampling layer, which can effectively extract the feature information of the image and realize the Multiscale Feature Recognition of Image Features.

In a multi-layer neural network, the function of the activation function is to perform nonlinear mapping on the eigenvector map obtained by convolution, because after multi-layer convolution, the value of the eigenvector map does not change much during training, which will cause the gradient to disappear , which in turn makes the convolutional network model untrainable. After each convolution, the ReLU activation function can be used to nonlinearly map the results of the linear calculation of the convolutional layer, so that the feature map changes before and after training will not be too small, and then the convolutional network model can be used for training.

For the first four convolutional layers, each convolutional layer is followed by a pooling layer. Its function is to reduce the feature dimension, compress the number of data and parameters, reduce training overfitting, and improve the fault tolerance of the cloud detection network. The embodiment of the present invention uses a pooling kernel with a size of 2*2 and a step size of 2, and the length and width of the feature map after pooling will be reduced by half.

The role of the upsampling layer is to restore the picture from the feature vector map of the picture. The feature map of the upsampling layer of each layer and the feature map of the downsampling layer corresponding to the number of layers are skipped and connected, that is, the splicing shown in Figure 2, and then convolution is performed. The size of the upsampling layer is also 3*3, and the step size The convolution kernel is 2. The purpose of this is to allow the image to refer to lower-dimensional features during the restoration process, and the restored image is closer to the original image. The number of convolution kernels of each upsampling layer corresponds to the number of convolution kernels of the downsampling layer of the same layer.

The role of the deconvolution layer can be seen as the reaction of the pooling layer. The interpolation method is used to expand the feature map, and the deconvolution kernel with a size of 2*2 and a step size of 2 is used.

Fully connected layer: The last layer of the upsampling layer connects two fully connected layers. The fully connected layer uses a convolution kernel with a size of 1*1 to obtain the foreground and background of the segmentation map respectively, and then combines the output mask.

As shown in Figure 2, the cloud detection network of the embodiment of the present application has 5 down-sampling layers and 5 up-sampling layers, each of the first 4 down-sampling layers is followed by a convolutional layer and a pooling layer, and the fifth down-sampling layer The sampling layer is connected to the first upsampling layer, and each of the last 4 upsampling layers is followed by a deconvolution layer. There is a skip connection chain between the upsampling layer and the downsampling layer of the corresponding layers in the first 4 layers. Input a remote sensing image of any size into the cloud detection network, and use the downsampling layer to perform feature extraction. The embodiment of the present invention uses a convolution kernel with a size of 3*3 and a step size of 1 to extract high-dimensional images from the matrix of remote sensing images. For the feature map, two convolution operations are performed on each layer, and after each convolution, the activation function is used to perform nonlinear mapping on the feature vector map obtained by convolution. The first downsampling layer uses 32 convolution kernels. After performing two convolution operations, the pooling layer is used to reduce the amount of data and parameters to be processed in the next downsampling layer. The pooling layer in the embodiment of the present invention uses a size For a filter of 2*2 and a step size of 2, the length and width of the feature map after pooling are 1/2 of the original feature map, that is, the amount of data after pooling is reduced to 1/4 of that before pooling, This prevents the network from overfitting. The downsampling layer is used as the feature extraction part, and each pooling layer has a scale. The embodiment of this application includes the scale of the original image. There are 5 scales in total. The 4 pooling layers realize the multi-scale feature recognition of image features by the network.

The number of convolution kernels in the latter downsampling layer is twice the number of convolution kernels in the previous downsampling layer. As shown in Figure 2, the number of convolution kernels in the second downsampling layer is 64, and the number of convolution kernels in the third downsampling layer is 64. The number of convolution kernels in the downsampling layer is 128, the number of convolution kernels in the fourth downsampling layer is 256, and the number of convolution kernels in the last downsampling layer is 512. connected to the sampling. In the upsampling part, every time upsampling is performed, the feature map of the downsampling layer of the corresponding level is spliced, and then the feature map of the previous upsampling layer is spliced with the feature map of the downsampling layer of the corresponding level, and then the size is 3*3 , the convolution kernel with a step size of 2 performs two convolution operations. After the two convolution operations of each upsampling layer are completed, the deconvolution operation is performed again, and the feature map is expanded by interpolation. The embodiment of the present invention uses a deconvolution kernel with a size of 2*2 and a step size of 2. The deconvolution kernel is the transposition matrix of the original convolution kernel, which uses deconvolution to fill the image content, making the image content richer. The last upsampling layer connects two fully connected layers. The fully connected layer uses a convolution kernel with a size of 1*1 to obtain the foreground and background of the segmented image respectively, and then combines the output results to obtain the mask image of the original remote sensing image.

As shown in Figure 3, the upsampling part will fuse the output of the feature extraction part, which actually fuses multi-scale features together. Taking the second upsampling layer as an example, its features come from the fourth The characteristics of the output of the downsampling layer, that is, the dotted rectangle on the right side of the dotted arrow in Figure 3, also has the output from the first upsampling layer. Such splicing runs through the entire cloud detection network. In the network of the embodiment of the present application, there are Four splicing process.

The feature map obtained by each convolutional layer of the UNET network in the embodiment of the present invention will be connected to the corresponding upsampling layer, so that the feature map of each layer can be effectively used in subsequent calculations, avoiding the direct calculation of high-level feature maps Supervision and loss calculation are carried out in the middle, but the features in the low-level feature map are combined, so that the final feature map can not only contain high-level features, but also protect many low-level features, and realize different levels of features. The fusion can improve the accuracy of the model results.

2. The production process of the data set:

In the embodiment of the present application, the Landsat satellite image is used as a data set, and the background is roughly divided into five types of image sets: snow, grassland, desert, ocean, and town;

The embodiment of the present application utilizes the multi-spectral picture set and sensor data of the corresponding pictures in the Landsat data set, and uses the cloud layer identification tool in the ENVI5.4 software to adjust the outward expansion of the cloud area (Kernel Size) and the cloud-cloudless land and water areas. The cloud probability threshold (Cloud Probability Threshold) parameter can initially identify the cloud in the picture, but there is still the possibility of misjudgment or omission. Among them, the greater the possibility threshold of cloud-cloudless land and water area, the smaller the cloud area that may be detected.

Manually re-inspect the pictures after preliminary automatic processing. For the mask pictures with cloud misjudgments and omissions, use picture editing tools to make up for them. As training labels, each original remote sensing image has a manual The marked cloud detection mask image corresponds to it, and the original remote sensing image and its corresponding cloud detection mask image form a training data set, as shown in the training set sample shown in Figure 3. The left side is the original remote sensing image, and the right side is the corresponding annotation picture.

In the embodiment of the present application, the 460 pictures obtained after manual labeling are processed by using python's openCV library to rotate, deform, and enhance the contrast of each picture, and the original picture and the mask picture can be rotated at random angles, such as rotating 90 ° or rotate 180°, use the getAffineTransform function to deform the picture, and enhance the contrast of the picture by enhancing the maximum and minimum values of the contrast. Binarize some pictures, because the binarization effect of some pictures is meaningless, such as clouds and water are connected together, so all pictures are not binarized.

A certain number of image sets are obtained through data expansion and enhancement, that is, a certain degree of rotation and translation, elastic deformation, and gray value changes, etc., to enrich the diversity of data sets and enhance the robustness of network models.

If there is a new data set to be added, the cloud detection result map can be manually verified after rough detection with ENVi software.

3. The process of using the data set for network training is as follows:

Use the dropout method to train the network on the remote sensing image data set, and repeat the training of the above-built network. After the number of training times reaches the preset threshold or the accuracy of the test reaches the target value, it indicates that the built UNET network model has met the requirements.

The loss function of the UNET network model of the embodiment of the present application is:

Among them, E is the cross-entropy calculation function, x is the pixel point of the input image, p _l(x) is the approximate maximum function, w _c (x) is the category frequency weight, and d ₁ (x) indicates that the pixel point x is closest to it d ₂ (x) represents the distance from the pixel point x to the second closest cloud boundary, the parameter w ₀ and the standard deviation σ are adjustable.

The loss function is used to calculate the error loss, which is used to evaluate the gap between two vectors. The object of the approximate maximum function p _l(x) in the loss function is the manually labeled image and the image output by the cloud detection network. In the embodiment of the present application, the loss function is used to evaluate the gap between the mask generated by the cloud detection network and the real labeled image, and it is fed back to the cloud detection network for parameter update during the training process. In order to make the result of the loss function closer to the needs of practical applications, the cross-entropy function is weighted to compensate for the different frequencies of each type of pixel in the training data set.

When the embodiment of the present application uses the training data set to train the cloud detection network, each time an original remote sensing picture is input into the current network, the forward pass calculation is used to obtain the output picture of the current network, and the loss function is used to calculate the output picture and its corresponding The error between the manually marked mask images is backpropagated to the network by using the chain rule. In the process of backpropagation, the parameters in the network such as the weight of convolution and A round of updating is performed on the offset of the product, and a learning is completed. In the embodiment of the present invention, 1050 original pictures are used as training samples, and the size of each picture is 572*572. During training, 1050 pictures are trained once as a cycle, and a total of 50 cycles are trained in the embodiment of the present application.

4. Adjust parameters and retrain the network model:

Input the images in the test data set and evaluation data set into the above cloud detection network to test the accuracy, then train the cloud detection network, adjust the convolution weight, convolution bias and other parameters, and repeat training process until the desired effect is achieved.

In deep learning, the weight initialization method of the neural network has a crucial impact on the convergence speed and performance of the model. The neural network iteratively updates the weight parameters in order to achieve better performance. The parameter learning in the training process of the neural network is optimized based on the gradient descent method. The gradient descent method needs to assign an initial value to each parameter at the beginning of training. In practical applications, parameters obeying Gaussian distribution or uniform distribution are more effective initialization methods. In a deep neural network, as the number of layers increases, gradient disappearance or gradient explosion is prone to occur during gradient descent. The weight can be initialized in the form of all 0s, all 1s, or a fixed value. In the embodiment of the present invention, the weight is initialized with a fixed value, and the adopted fixed value is 0.1.

According to the performance of the training loss function, the number of training times and the batch size of training can be adjusted, and the training speed should be considered when adjusting. Adjust the learning rate. If the rate is too high, the network will enter local optimization prematurely, and the effect is not ideal. The learning rate in the embodiment of the present invention is finally set to 0.001.

The present invention uses a deep learning network for detection, which saves the design and implementation of complex algorithms, and can achieve the goal more conveniently and directly. At the same time, when dealing with this type of problem with the existing artificial intelligence methods, the network structure is usually too simple, or is limited by the structure, and the increase in the number of layers cannot achieve higher accuracy, which is prone to misjudgment and misjudgment. Compared with the traditional multi-layer convolutional fully connected network, the upsampling of UNET still has a large number of channels and downsampling connections, which makes the network propagate the context information to a higher layer resolution. As a result, the expansion path and the contraction path are symmetrical, forming A U-shaped shape effectively preserves the underlying details for higher-precision detection.

The remote sensing image cloud detection method based on the UNET neural network proposed by the present invention is a method directly from the image input to the image mask output, and the up-sampling of the UNET neural network still has a large number of channels connected to the down-sampling, which makes the network the context The information propagates to higher-level resolution, and the expansion path is symmetrical to the contraction path, forming a U-shaped shape, effectively retaining the details of the bottom layer, and performing higher-precision detection.

Based on the combined use of deep learning convolutional neural network, the present invention performs cloud detection on remote sensing pictures, and then takes time-sharing photographs of the same area, replaces the cloudy area with its cloudless area, and integrates it into a cloudless image , the key points of this method include: direct processing of remote sensing images, not used as pre-processing of previous images or pre-extraction of parameters; direct output of mask images without post-generation.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions recorded in each embodiment are modified, or some of the technical features are replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (10)

1. A remote sensing image cloud detection method based on a UNET neural network is characterized by comprising the following steps:

establishing a cloud detection network with 5 down-sampling layers and 5 up-sampling layers, wherein the front four down-sampling layers of the cloud detection network are connected with a convolution layer and a pooling layer after each layer, and the back four up-sampling layers are connected with an anti-convolution layer after each layer;

carrying out cloud labeling, manual rechecking and data enhancement processing on an original remote sensing image set, and dividing the processed remote sensing image set into a training set, an evaluation set and a test set;

training the cloud detection network by using the images in the training set, and continuously updating the parameters of the cloud detection network;

carrying out accuracy test on the trained cloud detection network by using the images in the test set, and carrying out parameter adjustment and retraining on the cloud detection network;

and carrying out cloud detection on the remote sensing image by utilizing the cloud detection network, and outputting a mask image of the cloud detection.

2. The UNET neural network-based remote sensing image cloud detection method according to claim 1, comprising the following steps: and a fifth down-sampling layer of the cloud detection network is connected with the first up-sampling layer.

3. The UNET neural network-based remote sensing image cloud detection method according to claim 1, comprising the following steps: and the last upsampling layer of the cloud detection network is connected with the output of the full connection layer.

4. The remote sensing image cloud detection method based on the UNET neural network as claimed in claim 1, wherein cloud labeling of the original remote sensing image set comprises: and carrying out cloud annotation on the remote sensing image set by utilizing ENVI software.

5. The UNET neural network-based remote sensing image cloud detection method according to claim 1, wherein the data enhancement of the original remote sensing image set comprises: and carrying out rotation translation, elastic deformation and contrast enhancement on the remote sensing image set.

6. The UNET neural network-based remote sensing image cloud detection method according to claim 1, wherein the accuracy test is performed on the trained cloud detection network by using the image in the test set, and the parameter adjustment and retraining of the cloud detection network comprises: and adjusting the training times and the training batch size according to the performance of the loss function in the training.

7. The UNET neural network-based remote sensing image cloud detection method according to claim 1, wherein the dividing the processed remote sensing image set into a training set, an evaluation set, and a test set further comprises: and randomly selecting 70% of images from the remote sensing image set as a training set, 20% of images as an evaluation set, and the rest 10% of images as a test set.

8. The UNET neural network-based remote sensing image cloud detection method according to claim 1, comprising: the activation function of the cloud detection network is a ReLU function.

9. The UNET neural network-based remote sensing image cloud detection method according to claim 1, wherein the cloud detection network is trained by using the images in the training set, and parameters of the cloud detection network are continuously updated, and the method includes: the parameters of the cloud detection network may be weights of convolution and bias values of convolution.

10. The UNET neural network-based remote sensing image cloud detection method according to claim 1, wherein the accuracy test is performed on the trained cloud detection network by using the image in the test set, and parameter adjustment and retraining of the cloud detection network are performed, and the method includes: the learning rate is set to 0.001.