CN116630700A - Remote sensing image classification method based on introduction channel-space attention mechanism - Google Patents

Abstract

The application relates to the field of hyperspectral remote sensing image classification, in particular to a remote sensing image classification method based on an introduction channel-space attention mechanism, which comprises the following steps: s1, obtaining characteristic information of a hyperspectral pixel module as a hyperspectral remote sensing image; s2, inputting the hyperspectral pixel module into a model multichannel convolutional neural network to extract deep features so as to obtain a spatial spectrum fusion feature map; s3, introducing a channel-space attention module during feature extraction to obtain a full spatial spectrum feature map; s4, inputting the feature map subjected to feature extraction through the model into a full-connection layer, integrating the abstract features extracted by the convolution layer, and mapping the input vector elements to a (0, 1) interval by using a Softmax function at an output layer to obtain an output classification result map. The method is embedded into a hyperspectral remote sensing image classification frame of the multichannel convolutional neural network, and the spatial characteristics and the spectral characteristics are fully considered, so that the classification accuracy of hyperspectral images is improved.

Description

Remote Sensing Image Classification Method Based on Introducing Channel-Spatial Attention Mechanism

technical field

The invention relates to the field of hyperspectral remote sensing image classification, in particular to a remote sensing image classification method based on introducing a channel-spatial attention mechanism, wherein a multi-channel convolutional neural network is used.

Background technique

Remote sensing is a technique for observing and researching the earth's surface through remote sensing detection of the earth's surface to obtain relevant information such as its spectrum and radiation. The hyperspectral sensor acquires hyperspectral images by carrying a spectral remote sensing platform. Compared with traditional remote sensing images, hyperspectral images have richer spatial and spectral information. Therefore, hyperspectral images are widely used in agriculture, environment, military affairs, and grasslands. The hyperspectral image assigns a unique label to each pixel, and the computer automatically distinguishes by identifying the label attributes, and finally automatically classifies the pixels of the hyperspectral image into predefined categories to complete the classification task of the hyperspectral image. Hyperspectral image classification is the basis of hyperspectral image applications.

In recent years, deep learning (DL) hyperspectral image classification methods have gradually become a research hotspot due to their unique advantages such as parameter sharing. However, Convolutional Neural Networks (CNNs) come out on top in deep learning. There are currently three types of common convolution kernels: 1D-CNN, 2D-CNN, and 3D-CNN. Spectral information is a one-dimensional vector, which is usually classified using 1D-CNN; 2D-CNN can be used to extract local spatial information around target pixels in hyperspectral remote sensing images, and the spectral structure may be destroyed in the process; 3D-CNN is widely used, and it can Considering spatial information and spectral information, feature extraction is more comprehensive. In terms of network structure, CNN-based hyperspectral image classification methods are mainly divided into two categories: one is based on traditional CNN structures, such as LeNet, AlexNet, VGG, etc.; the other is based on deep residual network (ResNet), Inception and other deeper network structures. In terms of feature extraction, some researchers have proposed some new convolution kernel design methods, such as Spectral-Spatial Convolution, Multi-Scale Convolution, Multi-Channel Convolution Convolution) and so on, these methods can extract more abundant and meaningful features, thereby improving the accuracy of hyperspectral image classification.

At present, the feature fusion method of multi-channel and multi-scale feature extraction has been widely used. For example, in the CNN network with dual-channel spectrum enhancement, the spectral feature extraction channel is designed to enhance the spectral feature representation of specific pixels. In the spatial spectral channel, a small convolution kernel is used to extract the spatial spectral features of HSI. By adjusting the fusion ratio of the two channels, it is better. for HSI classification.

In order to improve the performance of the model for feature extraction, researchers try to combine other techniques, such as using the attention mechanism (Attention Mechanism) to improve classification accuracy. The essence of the attention mechanism is to assign contribution weights in space, spectrum, and channel, to decide which features to focus on at a specific moment, and to ignore useless information, thereby improving the performance of the model. Woo et al. added the CBAM module to the basic CNN model. The CBAM module is a combination of the channel attention module and the spatial attention module. In the channel attention module, the feature layer is compressed in the spatial domain. The spatial attention module is Compresses the channel domain. Weighting is performed on spectrum and space to effectively capture image features.

The deep learning method is aimed at classifying hyperspectral images. Due to the simple network structure and limited feature extraction, the classification accuracy needs to be improved. The main contribution of this application is to introduce a channel-spatial attention mechanism module, which is embedded in the hyperspectral remote sensing image classification framework of multi-channel convolutional neural network, pay more attention to useful feature information, and ignore useless information, thereby improving hyperspectral image classification accuracy.

Contents of the invention

In view of the above problems, the present invention provides a remote sensing image classification method based on introducing a channel-spatial attention mechanism. Embed it into the hyperspectral remote sensing image classification framework of multi-channel convolutional neural network, fully consider the spatial characteristics and spectral characteristics, so as to improve the classification accuracy of hyperspectral images.

The present invention provides the following technical solutions: a remote sensing image classification method based on introducing a channel-spatial attention mechanism, comprising the following steps:

S1. Obtain a hyperspectral remote sensing image, and preprocess the hyperspectral remote sensing image data to obtain the characteristic information that the hyperspectral pixel module is a hyperspectral remote sensing image;

S2. Input the hyperspectral pixel module into the model multi-channel convolutional neural network AMC-CNN for deep feature extraction to obtain the spatial spectral fusion feature map. Each channel in the network uses three-dimensional convolution as a feature extractor to perform spatial spectral fusion The feature map is used to extract spectral key information and spatial key information to obtain a feature map R ₄ ;

S3. In feature extraction, in order to improve the classification performance of the model, a channel-spatial attention module is introduced to extract spectral key information and spatial key information from the spectral fusion feature map, and obtain a sufficient spatial spectral feature map R ₅ ;

S4. Input the feature map _R5 after feature extraction through the model to the fully connected layer FC, integrate the abstract features extracted by the convolution layer, and use the Softmax function in the output layer to map the input vector elements to the (0, 1) interval, Finally, the probabilities of different categories are output and the output classification result graph R ₆ is obtained.

In step S1, the hyperspectral data R ₁ is divided into two modules of different scales during preprocessing, namely: 3×3 pixel module R ₂ and 5×5 pixel module R ₃ , to obtain the characteristic information of the hyperspectral pixel module;

The step S1 specifically includes:

S101: The original hyperspectral remote sensing image is three-dimensional data, which contains rich spatial information and rich spectral information;

S102: The original hyperspectral data R ₁ is centered on a single pixel, and the adjacent pixel modules are acquired according to the multi-scale 3×3 and 5×5 set size values;

S103: Input the acquired 3×3 pixel module R ₂ and 5×5 pixel module R ₃ into channels of different sizes respectively, the input sizes are 1×3×3×B and 1×5×5×B, where B Indicates the number of bands of the hyperspectral image;

S104: Divide the preprocessed pixel module into a data set, and divide it into a training sample set and a testing sample set.

Described step S2 specifically comprises:

S201. Each channel in the network uses three-dimensional convolution as a feature extractor to extract spectral key information and spatial key information from the spatial spectral fusion feature map to obtain the feature map R ₄ ;

S202. In AMC-CNN, pass through two convolutional layers, a pooling layer (AvgPool), a channel attention module, a convolutional layer, a spatial attention module, a pooling layer, and a convolutional layer in sequence. , fully connected layer (FC layer), and finally obtain the classification result through the Softmax layer;

S203. The convolutional layer includes several convolutional kernels, which are used for feature extraction and are also the most important component of the convolutional neural network. Let the input image be X, and the output Q of the convolutional layer be formula (1)

Among them, w represents the weight and b represents the bias;

S204. During the convolution operation of the hyperspectral remote sensing pixel modules R ₂ and R ₃ using the convolutional neural network, n convolution kernels are used in different convolution layers. For 3×3 pixels of different scales The module and the 5×5 pixel module use different convolution kernel sizes;

S205. The pooling layer is used to reduce the size of the feature space after the convolution operation, thereby reducing the network parameters, speeding up the calculation depth, reducing the number of parameters to the fully connected layer, and preventing overfitting. The pooling operation includes a maximum pooling layer and the average pooling layer;

S206, using the cross-entropy loss function in the network model, the expression is as formula (2)

Among them, M is the number of categories, y _ic refers to the sign function, the value is 0 or 1, if the real category of sample i is equal to c, it takes 1, otherwise it takes 0, p _ic is the predicted probability that the observed sample i belongs to category c.

Described step S3 comprises:

S301: Perform two consecutive convolution layers on the input pixel module, and then perform zero padding operation on it, so that the multi-scale pixel module vector and the channel-spatial attention module vector are the same;

S302: The multi-scale pixel module weights the channels in the channel attention mechanism module, and obtains two The feature strip, the length of the feature strip is the number of channels of the input feature layer, and the features output while sharing the MLP network parameters are added and summed, and mapped to [0,1] through the Sigmoid activation operation to obtain channel attention at the same time M _c , the number of neurons is multiplied by the number of channels of the input feature layer in one-to-one correspondence, and finally the input features required by the spatial attention module are output to complete the process of the channel attention mechanism;

S303: The feature layer output by the channel attention mechanism module is used as the input of the feature layer of the spatial attention module, and the space is weighted in the spatial attention mechanism module, mainly focusing on the more important features of the spatial shape, and weakening the unimportant positions The characteristics of the spatial attention mechanism are completed.

In step S303, when completing the process of the spatial attention mechanism, the feature map output by the channel attention is first used as the input of the spatial attention module, and a channel-based maximum pooling and average pooling are performed on the input feature layer, and the obtained As a result, the number of channels increases, and a convolution size of (7, 7, 1) is used for dimensionality reduction. The proportion of each feature point is obtained through the Sigmoid function, and the spatial attention feature map M _s is generated. Finally, the generated spatial attention feature The graph M _s is multiplied by the input feature layer to obtain the final output feature map, completing the process of the spatial attention mechanism.

Described step S4 comprises:

The feature map R ₅ after feature extraction through the model is input to the fully connected layer FC, and the fully connected layer uses the Softmax function to output the probability of different categories, where the Softmax function normalizes the feature map output by the fully connected layer and then inputs it to the output Layer, the Softmax function uses the formula (3)

Among them, Y _i represents the i-th element of the vector Y, i is a positive integer, Y _i is also the output of the hyperspectral pixel after passing through the fully connected layer, and P _i represents the output probability of the object category to which it belongs.

From the above description, it can be seen that the remote sensing image classification method based on the introduction of the channel-spatial attention mechanism in this scheme, and the multi-channel convolutional neural network that introduces the channel-spatial attention mechanism. The use of multi-scale pixel modules is more suitable for the size of feature extraction, which is conducive to the speed of network operations. The model adopts a multi-channel convolutional neural network structure. The multi-channel structure can more fully and effectively extract deep-level spatial domain features and spectral domain features. The channel-spatial attention module is added to focus on the important features of spectral information and ignore useless information, which improves the efficiency, improves the classification accuracy and makes the classification more accurate.

Description of drawings

Fig. 1 is a flowchart of hyperspectral image classification according to a specific embodiment of the present invention.

Figure 2 is a network diagram of hyperspectral remote sensing image classification with pixel module convolutional neural network.

Fig. 3 is a schematic diagram of a comparison of hyperspectral remote sensing image classification results according to an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the specific embodiments of the present invention in conjunction with the accompanying drawings in the specific embodiments of the present invention. Obviously, the described specific embodiment is only a specific embodiment of the present invention, not all specific implementation. Based on the specific implementation modes in the present invention, all other specific implementation modes obtained by persons of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

As can be seen from the accompanying drawings, the remote sensing image classification method based on the introduction of channel-spatial attention mechanism of the present invention, the specific steps are as follows:

S1. Obtain the hyperspectral remote sensing image, and preprocess the hyperspectral remote sensing image data. In the hyperspectral remote sensing image data preprocessing stage, divide the hyperspectral data _R1 into two modules of different scales, namely: 3×3 pixel modules R ₂ and 5×5 pixel module R ₃ obtain the characteristic information of the hyperspectral pixel module as a hyperspectral pixel module, specifically including:

S101: The original hyperspectral remote sensing image is three-dimensional data, which contains rich spatial information and rich spectral information;

S102: The original hyperspectral data R ₁ is centered on a single pixel, and the size value is set according to the scale 3×3 and 5×5 to obtain adjacent pixel modules;

S103: Input the acquired 3×3 pixel module R ₂ and 5×5 pixel module R ₃ into channels of different sizes respectively, the input sizes are 1×3×3×B and 1×5×5×B, where B Indicates the number of bands of the hyperspectral image;

S104: Divide the preprocessed pixel module into a data set, and divide it into a training sample set and a testing sample set.

S2. Input the hyperspectral 3×3 pixel module R ₂ and 5×5 pixel module R ₃ into the model multi-channel convolutional neural network (AMC-CNN) for deep feature extraction, and each channel in the network uses three-dimensional convolution As a feature extractor, extract spectral key information and spatial key information from the spatial-spectral fusion feature map to obtain a feature map R ₄ , specifically including: S201. Each channel in the network uses a three-dimensional convolution as a feature extractor, and the spatial The spectral fusion feature map is used to extract spectral key information and spatial key information to obtain a feature map R ₄ ;

S202. In AMC-CNN, sequentially pass through two convolutional layers, a pooling layer, a channel attention module, a convolutional layer, a spatial attention module, a pooling layer, a convolutional layer, and a full connection layer, and finally the classification result is obtained through the Softmax layer;

S203, the convolution layer includes several convolution kernels for feature extraction, the input image is set to X, and the output Q of the convolution layer is formula (1)

Among them, w represents the weight and b represents the bias;

S204. During the convolution operation of the hyperspectral remote sensing pixel modules R ₂ and R ₃ using the convolutional neural network, n convolution kernels are used in different convolution layers. For 3×3 pixels of different scales The module and the 5×5 pixel module use different convolution kernel sizes;

S205. The pooling layer is used to reduce the size of the feature space after the convolution operation, thereby reducing the network parameters, speeding up the calculation depth, reducing the number of parameters to the fully connected layer, and preventing overfitting. The pooling operation includes a maximum pooling layer and the average pooling layer;

S206, using the cross-entropy loss function in the network model, the expression is as formula (2)

Among them, M is the number of categories, y _ic refers to the sign function, the value is 0 or 1, if the real category of sample i is equal to c, it takes 1, otherwise it takes 0, p _ic is the predicted probability that the observed sample i belongs to category c.

S3. In feature extraction, in order to improve the classification performance of the model, the channel-spatial attention mechanism module is introduced to pay more attention to the useful spatial spectral information, so that the spectral key information and spatial key information can be better extracted from the spectral fusion feature map. , to obtain a more sufficient spatial spectral feature map R ₅ , specifically including: S301: first perform two consecutive convolutional layers on the input pixel module, and then perform zero padding on it, so that the multi-scale pixel module vector and channel-space The attention module vectors are the same;

S302: The multi-scale pixel module weights the channels in the channel attention mechanism module, and obtains two The feature strip, the length of the feature strip is the number of channels of the input feature layer, and the features output while sharing the MLP network parameters are added and summed, and mapped to [0,1] through the Sigmoid activation operation to obtain channel attention at the same time M _c , the number of neurons is multiplied by the number of channels of the input feature layer in one-to-one correspondence, and finally the input features required by the spatial attention module are output to complete the process of the channel attention mechanism;

S303: The feature layer output by the channel attention mechanism module is used as the input of the feature layer of the spatial attention module, and the space is weighted in the spatial attention mechanism module, mainly focusing on the more important features of the spatial shape, and weakening the unimportant positions Characteristics. First, the feature map output by the channel attention is used as the input of the spatial attention module, and a channel-based maximum pooling and flat pooling is performed on the input feature layer, and the number of resulting channels is increased, and a convolution size of (7, 7.1) Perform dimensionality reduction, obtain the proportion of each feature point through the Sigmoid function, generate a spatial attention feature map M _s , and finally multiply the generated spatial attention feature map M _s with the input feature layer to obtain the final output feature Figure, complete the process of spatial attention mechanism.

S4. Input the feature map _R5 after feature extraction through the model to the fully connected layer FC, integrate the abstract features extracted by the convolution layer, and use the Softmax function in the output layer to map the input vector elements to the (0, 1) interval, Finally, the probabilities of different categories are output and the output classification result graph R ₆ is obtained.

Hyperspectral images also have technical indicators for comparative evaluation of classification results. It accurately obtains classification accuracy through calculation formulas. Common evaluation indicators include: overall classification accuracy (OA), average classification accuracy (AA), and Kappa coefficient.

In order to verify the model method described in step 2, test is carried out on the Pavia Center data set, and table one is the present invention on the Pavia Center data set with traditional method SVM, 3D-CNN based on deep learning, dual-channel 3D-CNN model and Multi-channel model classification results comparison results. details as follows:

The classification result diagram of the present invention is compared with the traditional method SVM, 3D-CNN based on deep learning, dual-channel 3D-CNN and multi-channel MC-CNN model, and the comparison result diagram is shown in Figure 3, (a) three-channel RGB image, (b) real object label, (c) SVM classification result map, (d) 3D-CNN classification result map, (e) dual-channel 3D-CNN, (f) MC-CNN, (g) classification result of the present invention (MC-CNN). Through comparison, it can be seen that various algorithms classify 9 kinds of ground objects, but the image results show many blurred spots. As shown in Figure 3, the hyperspectral image classification model described in step 2 is relatively excellent. From the rectangular frame area, it can be seen that the classification spots are gradually reduced. It can also be seen that the deep learning method is superior to the traditional method, but the method described in step 2 The classification method is the best among deep learning methods, with the highest classification accuracy and the best classification effect.

Although specific embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, Alternatives and modifications, the scope of the present invention is defined by the appended claims and their equivalents.

Claims (7)

1. based on introducing the remote sensing image classification method of channel-spatial attention mechanism, it is characterized in that comprising the steps: S1. Obtain a hyperspectral remote sensing image, and preprocess the hyperspectral remote sensing image data to obtain the characteristic information that the hyperspectral pixel module is a hyperspectral remote sensing image; S2. Input the hyperspectral pixel module into the model multi-channel convolutional neural network AMC-CNN for deep feature extraction to obtain the spatial spectral fusion feature map. Each channel in the network uses three-dimensional convolution as a feature extractor to perform spatial spectral fusion The feature map is used to extract spectral key information and spatial key information to obtain a feature map R ₄ ; S3. In feature extraction, in order to improve the classification performance of the model, a channel-spatial attention module is introduced to extract spectral key information and spatial key information from the spectral fusion feature map, and obtain a sufficient spatial spectral feature map R ₅ ; S4. Input the feature map _R5 after feature extraction through the model to the fully connected layer FC, integrate the abstract features extracted by the convolution layer, and use the Softmax function in the output layer to map the input vector elements to the (0, 1) interval, Finally, the probabilities of different categories are output and the output classification result graph R ₆ is obtained. 2. the remote sensing image classification method based on introducing channel-spatial attention mechanism according to claim 1, is characterized in that: In step S1, during preprocessing, the hyperspectral data R ₁ is divided into two modules of different scales, namely: 3×3 pixel module R ₂ and 5×5 pixel module R ₃ , to obtain the feature information of the hyperspectral pixel module. 3. the remote sensing image classification method based on introducing channel-spatial attention mechanism according to claim 2, is characterized in that, described step S1 comprises: S101: The original hyperspectral remote sensing image is three-dimensional data, which contains rich spatial information and rich spectral information; S102: The original hyperspectral data R ₁ is centered on a single pixel, and the size value is set according to the scale 3×3 and 5×5 to obtain adjacent pixel modules; S103: Input the acquired 3×3 pixel module R ₂ and 5×5 pixel module R ₃ into channels of different sizes respectively, the input sizes are 1×3×3×B and 1×5×5×B, where B Indicates the number of bands of the hyperspectral image; S104: Divide the preprocessed pixel module into a data set, and divide it into a training sample set and a testing sample set. 4. the remote sensing image classification method based on introducing channel-spatial attention mechanism according to claim 2, is characterized in that, described step S2 comprises: S201. Each channel in the network uses three-dimensional convolution as a feature extractor to extract spectral key information and spatial key information from the spatial spectral fusion feature map to obtain the feature map R ₄ ; S202. In AMC-CNN, sequentially pass through two convolutional layers, a pooling layer, a channel attention module, a convolutional layer, a spatial attention module, a pooling layer, a convolutional layer, and a full connection layer, and finally the classification result is obtained through the Softmax layer; S203, the convolution layer includes several convolution kernels for feature extraction, the input image is set to X, and the output Q of the convolution layer is formula (1) Among them, w represents the weight and b represents the bias; S204. During the convolution operation of the hyperspectral remote sensing pixel modules R ₂ and R ₃ using the convolutional neural network, n convolution kernels are used in different convolution layers. For 3×3 pixels of different scales The module and the 5×5 pixel module use different convolution kernel sizes; S205. The pooling layer is used to reduce the size of the feature space after the convolution operation, thereby reducing the network parameters, speeding up the calculation depth, reducing the number of parameters to the fully connected layer, and preventing overfitting. The pooling operation includes a maximum pooling layer and the average pooling layer; S206, using the cross-entropy loss function in the network model, the expression is as formula (2) Among them, M is the number of categories, y _ic refers to the sign function, the value is 0 or 1, if the real category of sample i is equal to c, it takes 1, otherwise it takes 0, p _ic is the predicted probability that the observed sample i belongs to category c. 5. the remote sensing image classification method based on introducing channel-spatial attention mechanism according to claim 4, is characterized in that, described step S3 comprises: S301: Perform two consecutive convolution layers on the input pixel module, and then perform zero padding operation on it, so that the multi-scale pixel module vector and the channel-spatial attention module vector are the same; S302: The multi-scale pixel module weights the channels in the channel attention mechanism module, and obtains two The feature strip, the length of the feature strip is the number of channels of the input feature layer, and the features output while sharing the MLP network parameters are added and summed, and mapped to [0,1] through the Sigmoid activation operation to obtain channel attention at the same time M _c , the number of neurons is multiplied by the number of channels of the input feature layer in one-to-one correspondence, and finally the input features required by the spatial attention module are output to complete the process of the channel attention mechanism; S303: The feature layer output by the channel attention mechanism module is used as the input of the feature layer of the spatial attention module, and the space is weighted in the spatial attention mechanism module to complete the process of the spatial attention mechanism. 6. The remote sensing image classification method based on the introduction of channel-spatial attention mechanism according to claim 5, characterized in that, in step S303, when completing the process of the spatial attention mechanism, at first the feature map of the channel attention output is used as The input of the spatial attention module performs a channel-based maximum pooling and average pooling on the input feature layer, increases the number of resulting channels, and uses a convolution size of (7, 7, 1) for dimensionality reduction. After The Sigmoid function obtains the proportion of each feature point, generates a spatial attention feature map M _s , and finally multiplies the generated spatial attention feature map M _s with the input feature layer to obtain the final output feature map, and completes the process of the spatial attention mechanism. 7. the remote sensing image classification method based on introducing channel-spatial attention mechanism according to claim 5, is characterized in that, described step S4 comprises: The feature map R ₅ after feature extraction through the model is input to the fully connected layer FC, and the fully connected layer uses the Softmax function to output the probability of different categories, where the Softmax function normalizes the feature map output by the fully connected layer and then inputs it to the output Layer, the Softmax function uses the formula (3) Among them, Y _i represents the i-th element of the vector Y, i is a positive integer, Y _i is also the output of the hyperspectral pixel after passing through the fully connected layer, and P _i represents the output probability of the object category to which it belongs.