CN117058367A - Semantic segmentation method and device for high-resolution remote sensing image building - Google Patents

Abstract

The invention provides a semantic segmentation method and device for a high-resolution remote sensing image building, wherein the method comprises the following steps: preprocessing a remote sensing image to obtain a target image; and inputting the target image into the segmentation model to obtain a building segmentation result output by the segmentation model. According to the high-resolution remote sensing image building semantic segmentation method and device, the first feature extraction layer with the pyramid structure is used for extracting the refined feature images, the middle-layer feature images of various scale features are fused, convolution operations with different scales are executed in parallel through the refined feature images with reduced sizes, features unified into the same scale are fused into the deep feature images, and images obtained by fusing the feature images of three semantic levels are restored into building segmentation results. The convolution receptive field can be increased layer by layer under the condition of no information loss, so that the output of each convolution contains semantic information in a larger range, the definition of image semantic segmentation can be improved, and a building can be accurately identified.

Description

High-resolution remote sensing image building semantic segmentation method and device

Technical field

The invention relates to the technical field of remote sensing processing, and in particular to a method and device for semantic segmentation of buildings in high-resolution remote sensing images.

Background technique

As an essential place in production and life, buildings are one of the basic geographical information elements. Accurately obtaining building information has important application value in many fields such as urban planning, map production and updating, environmental monitoring, and the construction of "digital cities" and "smart cities". Therefore, building extraction plays a very important role in the application of high-resolution remote sensing to earth observation, and has always been one of the hot topics in the field of remote sensing information processing. Compared with the background of natural scene images, high-resolution remote sensing images often have complex backgrounds and various types of land species due to their large scale and different imaging methods, making it difficult to extract buildings from remote sensing images. At the same time, compared with natural features, buildings, as typical representatives of artificial features, have more complex and diverse features, and are interfered by problems such as shadows and occlusions. The results obtained by using low-level feature extraction or image segmentation methods are often unsatisfactory. satisfy. Therefore, it is very important to study a high-precision and automated building extraction method in high-resolution remote sensing images.

Methods for extracting building information based on remote sensing images continue to emerge, which can be divided into two categories according to feature construction rules: traditional methods and methods based on deep learning. Traditional methods extract buildings by manually designing corresponding features based on the spectrum, edge, shape, and shadow characteristics of the objects in the image. Traditional automatic building extraction methods perform well on remote sensing images of lower resolution and complexity, but on high-resolution remote sensing images they are exposed to poor robustness, low accuracy, vulnerability to human interference, and difficulty in precise positioning on a pixel-by-pixel basis. And other issues. In recent years, with the development of artificial intelligence, machine learning and other technologies, automatic extraction methods of buildings from high-scoring remote sensing images based on deep learning have received widespread attention and research. Compared with traditional methods, methods based on deep learning have the advantages of strong extraction ability, less human intervention, and high algorithm robustness. However, the continuous multiple down-sampling operations in existing deep learning-based processing methods cause the loss of spatial information, making it impossible to obtain global information in the feature maps generated by the model, and the accuracy of extracting buildings is limited. Although it is also proposed to use residuals to connect deep features and shallow features to enhance the expressive ability of features, skip connections may introduce redundant information, which will cause a reduction in extraction accuracy. It can be seen that neither traditional methods nor methods based on deep learning can take into account both the accuracy and efficiency of building segmentation extraction.

Contents of the invention

The present invention provides a method and device for semantic segmentation of buildings in high-resolution remote sensing images to solve the defect in the existing technology that cannot simultaneously take into account the accuracy and efficiency of building segmentation and extraction.

The present invention provides a method for semantic segmentation of buildings in high-resolution remote sensing images, including:

Preprocess the remote sensing image to obtain the target image; wherein the observation content of the remote sensing image at least includes buildings;

Input the target image to the segmentation model and obtain the building segmentation result output by the segmentation model;

Wherein, the segmentation model is trained based on sample remote sensing images and building area labels corresponding to the sample remote sensing images; the segmentation model includes:

The first feature extraction layer is used to perform step-by-step downsampling of the refined feature map obtained by the target image, and then step-by-step upsample and fuse the refined feature map to obtain a mid-level feature map;

The second feature extraction layer is used to perform convolution operations on the refined feature map at different scales to obtain a deep feature map that is fused from features of the same scale;

A feature fusion layer, used to fuse the low-level feature map, the middle-level feature map and the high-level feature map to obtain the building segmentation result segmented from the target image;

Wherein, the first feature extraction layer includes a plurality of down-sampling sub-layers cascaded from bottom to top, and a plurality of levels of up-sampling sub-layers cascaded correspondingly from top to bottom to each of the down-sampling sub-layers; The low-level feature map is obtained by extracting features of the target image in the first down-sampling sub-layer from bottom to top.

According to a method for semantic segmentation of buildings in high-resolution remote sensing images provided by the present invention, after step-by-step downsampling of the target image is performed to obtain a refined feature map, the refined feature map is progressively up-sampled. Sampling and fusion to obtain mid-level feature maps, including:

The attention mechanism is used to sequentially perform convolution operations and pooling operations of corresponding scales in each of the downsampling sub-layers to obtain the target feature map;

The target feature map is gradually upsampled to the original size corresponding to each downsampling sub-layer and then feature fusion is performed to obtain the mid-level feature map;

Perform dilated convolution operations and pooling operations on the target feature map to obtain the refined feature map;

Wherein, the two-dimensional dimensions of the refined feature map and the target feature map are the same, and the number of channels of the refined feature map is greater than the number of channels of the feature map.

According to the semantic segmentation method of high-resolution remote sensing images of buildings provided by the present invention, each downsampling sub-layer is a Bottleneck structure.

According to a method for semantic segmentation of buildings in high-resolution remote sensing images provided by the present invention, the convolution operation of different scales is performed on the refined feature map to obtain a deep feature map that is fused with features of the same scale, including:

Perform a 1*1 convolution operation on the refined feature maps, perform convolution operations on n convolution kernels with equal increasing hole rates, and perform global average pooling operations to obtain n+2 two-dimensional sizes and channel numbers. The same feature map;

Fusion of n+2 feature maps with the same two-dimensional size and channel number into the deep feature map.

According to a method for semantic segmentation of high-resolution remote sensing images of buildings provided by the present invention, any one of the n convolution kernels with equally increasing void rates is composed of a depth convolution kernel and a standard convolution kernel. Merge cores.

According to a method for semantic segmentation of buildings in high-resolution remote sensing images provided by the present invention, after fusing the low-level feature map, the mid-level feature map and the high-level feature map, all the features segmented from the target image are obtained. The building segmentation results include:

The high-level feature map is upsampled and then spliced and fused with the mid-level feature map to obtain a first feature fusion map;

Splice and fuse the first feature fusion map whose two-dimensional size and channel number are consistent with the low-level feature map and the low-level feature map to obtain the second feature fusion map;

The two-dimensional size and channel number of the second feature fusion map are restored to be equal to the target image, and the building segmentation result is obtained.

According to a method for semantic segmentation of buildings in high-resolution remote sensing images provided by the present invention, preprocessing the remote sensing images to obtain the target image includes:

The remote sensing image is cut into sub-images of the same size, and preprocessing operations such as random rotation, perspective transformation, brightness transformation, and color transformation are performed to obtain the target image.

The invention also provides a device for semantic segmentation of high-resolution remote sensing images of buildings, including:

A preprocessing module, used to preprocess remote sensing images and obtain target images; wherein the observation content of the remote sensing images at least includes buildings;

A segmentation module, used to input the target image to a segmentation model and obtain the building segmentation result output by the segmentation model;

Wherein, the segmentation model is trained based on sample remote sensing images and building area labels corresponding to the sample remote sensing images; the segmentation model includes:

The first feature extraction layer is used to perform step-by-step downsampling of the refined feature map obtained by the target image, and then step-by-step upsample and fuse the refined feature map to obtain a mid-level feature map;

The second feature extraction layer is used to perform convolution operations on the refined feature map at different scales to obtain a deep feature map that is fused from features of the same scale;

A feature fusion layer, used to fuse the low-level feature map, the middle-level feature map and the high-level feature map to obtain the building segmentation result segmented from the target image;

Wherein, the first feature extraction layer includes a plurality of down-sampling sub-layers cascaded from bottom to top, and a plurality of levels of up-sampling sub-layers cascaded correspondingly from top to bottom to each of the down-sampling sub-layers; The low-level feature map is obtained by extracting features of the target image in the first down-sampling sub-layer from bottom to top.

The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it achieves any of the above high resolutions. Semantic segmentation method of buildings in remote sensing images.

The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any one of the above high-resolution remote sensing image building semantic segmentation methods.

The present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements any one of the above high-resolution remote sensing image building semantic segmentation methods.

The method and device for semantic segmentation of high-resolution remote sensing images of buildings provided by the present invention are based on multi-channel remote sensing images to obtain target images, extract refined feature maps through the first feature extraction layer with a pyramid structure, and fuse multiple The mid-level feature map of three-scale features performs convolution operations of different scales in parallel through the reduced-size refined feature map, fuses the features unified into the same scale into a deep feature map, and then fuses the feature maps of the three semantic levels. The image restores the building segmentation results. It can increase the receptive field of convolution layer by layer without losing information, so that the output of each convolution contains a larger range of semantic information, which can improve the sophistication of image semantic segmentation, thereby improving the accuracy of building recognition. accuracy.

Description of the drawings

In order to explain the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are of the present invention. For some embodiments of the invention, those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a schematic flow chart of the semantic segmentation method of high-resolution remote sensing images of buildings provided by the present invention;

Figure 2 is one of the substructure schematic diagrams of the segmentation model provided by the present invention;

Figure 3 is the second schematic diagram of the substructure of the segmentation model provided by the present invention;

Figure 4 is the third schematic diagram of the substructure of the segmentation model provided by the present invention;

Figure 5 is the fourth schematic diagram of the substructure of the segmentation model provided by the present invention;

Figure 6 is a schematic diagram of the overall structure of the segmentation model provided by the present invention;

Figure 7 is one of the schematic diagrams of the simulation results of the semantic segmentation method of high-resolution remote sensing images of buildings provided by the present invention;

Figure 8 is the second schematic diagram of the simulation results of the semantic segmentation method for high-resolution remote sensing images of buildings provided by the present invention;

Figure 9 is the third schematic diagram of the simulation results of the semantic segmentation method of high-resolution remote sensing images of buildings provided by the present invention;

Figure 10 is a schematic structural diagram of a semantic segmentation device for high-resolution remote sensing images of buildings provided by the present invention;

Figure 11 is a schematic structural diagram of the electronic device provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention. , not all examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present invention.

The terms "first", "second", etc. in this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It is to be understood that the figures so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in orders other than those illustrated or described herein, and that "first," "second," etc. are distinguished Objects are usually of one type, and the number of objects is not limited. For example, the first object can be one or multiple.

It should be understood that the terminology used in the description of the present invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this invention, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise.

The terms "comprises" and "comprising" indicate the presence of described features, integers, steps, operations, elements and/or components but do not exclude the presence of one or more other features, integers, steps, operations, elements, components and/or The existence or addition to its collection.

Figure 1 is a schematic flow chart of the semantic segmentation method of high-resolution remote sensing images of buildings provided by the present invention. As shown in Figure 1, the method for semantic segmentation of buildings in high-resolution remote sensing images provided by embodiments of the present invention includes: step 101, preprocessing the remote sensing images to obtain target images.

Wherein, the observation content of the remote sensing image at least includes buildings.

It should be noted that the execution subject of the high-resolution remote sensing image building semantic segmentation method provided by the embodiment of the present invention is a high-resolution remote sensing image building semantic segmentation device. The detection object of the high-resolution remote sensing image building semantic segmentation device is remote sensing images.

Remote sensing images refer to obtaining electromagnetic spectrum information of ground objects in different bands through remote sensing means. Remote sensing images can be segmented according to different bands to generate two-dimensional arrays of different channels to represent the feature information of different bands. The pixel values in the two-dimensional array are called brightness values (or gray values, DN values).

Specifically, in step 101, the high-resolution remote sensing image building semantic segmentation device performs corresponding preprocessing on the remote sensing image according to the set requirements to obtain the target image.

The target image refers to an image that is as close to the real image as possible in terms of geometry and radiation compared with the original remote sensing image. The target image has at least three channels, and the grayscale images corresponding to different channels have different resolutions and different width and height dimensions. The target image is used as input to the segmentation model to obtain the corresponding building segmentation results.

Preprocessing refers to the processing performed before applying the segmentation model to the input original remote sensing image. Preprocessing is used to eliminate distorted information in remote sensing images, so that the generated target images contain useful real information, enhance the detectability of relevant information and simplify the data to the greatest extent. The embodiment of the present invention does not specifically limit the preprocessing.

For example, the preprocessing can be radiation correction. The high-resolution remote sensing image building semantic segmentation device respectively corrects the radiation in the remote sensing image caused by the characteristics of the remote sensing sensor itself, the illumination conditions of the ground objects (the influence of terrain and the influence of the sun's altitude angle), and the effects of the atmosphere. Three types of radiation distortion are used to perform remote sensor calibration, atmospheric correction (or solar height correction) and terrain correction to obtain target images.

For example, the preprocessing can also be geometric correction. The high-resolution remote sensing image building semantic segmentation device needs to select control point pairs in the remote sensing image to be corrected and the standard image with coordinate information, perform pixel coordinate transformation, and perform image coordinate transformation. Resample the brightness value of the element to obtain the target image.

Step 102: Input the target image to the segmentation model, and obtain the building segmentation result output by the segmentation model.

Wherein, the segmentation model is trained based on sample remote sensing images and building area labels corresponding to the sample remote sensing images. The segmentation model includes:

The first feature extraction layer is used to perform step-by-step downsampling of the refined feature map obtained by the target image, and then step-by-step upsample and fuse the refined feature map to obtain a mid-level feature map.

The second feature extraction layer is used to perform convolution operations of different scales on the refined feature map to obtain a deep feature map that is fused with features of the same scale.

The feature fusion layer is used to fuse the low-level feature map, the middle-level feature map and the high-level feature map to obtain the building segmentation result segmented from the target image.

Wherein, the first feature extraction layer includes a plurality of down-sampling sub-layers cascaded from bottom to top, and a plurality of levels of up-sampling sub-layers cascaded from top to bottom corresponding to each of the down-sampling sub-layers. The low-level feature map is obtained by extracting features of the target image in the first down-sampling sub-layer from bottom to top.

It should be noted that the segmentation model is obtained after training based on remote sensing image sample data and predetermined remote sensing image type labels.

The segmentation model may be an artificial intelligence model, and the embodiment of the present invention does not specifically limit the model type.

For example, the segmentation model can be a convolutional neural network model. As a data-driven algorithm, the convolutional neural network relies on its stacked sampling layers and convolutional layers to extract hierarchical features of the data, and uses reverse The propagation algorithm iteratively optimizes model parameters. The structure and parameters of the convolutional neural network model include but are not limited to the number of layers of the model and the weight parameters of each layer.

It should be noted that the sample data includes the remote sensing images corresponding to the sample data, as well as the labels and type labels for pixels belonging to the building area in the remote sensing images. The remote sensing image sample data is divided into a training set and a test set. The embodiment of the present invention does not specifically limit the sample ratio of the training set and the test set.

For example, the remote sensing image sample data can be randomly divided into a training set and a test set according to a ratio of 8:2.

Among them, the training set is used to train the network model and adjust model parameters. The test set is used to evaluate model performance. At the same time, data enhancement strategies such as random flipping and random rotation are used to enhance the number and complexity of training samples.

It should be noted that during the segmentation model training process, the high-resolution remote sensing image building semantic segmentation device integrates the input target images in batches, and the image format is batchsize*channel*height*width, where channel is The number of channels of the target image, height and width are respectively the height and width in the two-dimensional dimensions of the target image. This allows a batchsize number of training samples to be selected and input into the model during one training session, and then their gradients are calculated for back propagation to improve memory utilization.

Specifically, in step 102, the high-resolution remote sensing image building semantic segmentation device sets the segmentation model according to the trained model parameters, and then uses the model to perform segmentation of the building area on any target image as in step 101. Through identification and segmentation, the building segmentation results corresponding to the target image can be obtained.

The embodiment of the present invention does not specifically limit the form of the building segmentation results.

For example, the building segmentation result can be a two-digit array with the same width and height as the target image. The value range of any value in the array is [0, 1], where 0 means that the pixel does not belong to the building area, and 1 The pixel belongs to the building area, and the position information of the pixel in the two-dimensional array can be used to know which areas in the image are buildings.

For example, the building segmentation result can be a statistical value. The target image can be described as "with buildings" by counting all pixels, and the number of buildings contained in the target image can be given, or it can be "without buildings". "object" to indicate that there are no complete or partial buildings in the target image.

The embodiment of the present invention does not specifically limit the segmentation model.

For example, the device for semantic segmentation of buildings in high-resolution remote sensing images uses a segmentation model to identify building areas in remote sensing images. The model is composed of at least an input layer, a hidden layer and an output layer.

The input layer is at the forefront of the entire network and directly receives the target image generated in step 101.

The hidden layer contains at least four layers, namely the first feature extraction layer, the second feature extraction layer and the feature fusion layer. The function of the hidden layer is to downsample the target image layer by layer to obtain feature maps of different levels of remote sensing semantic information. By paralleling dilated convolution kernels of different scales, the extraction of features and advanced semantic information of buildings of different scales can be enhanced. . By fusing feature maps of different sizes, broader and deeper features can be obtained. Hidden layers of a segmentation model can include:

The function of the first feature extraction layer is to extract high-dimensional feature maps from the matrix corresponding to the target image through a large number of convolution kernels. Multiple feature maps can be extracted. Each feature map is a local perception extracted from the image. Comprehensive These feature maps can extract interesting parts of the image. Then perform multi-level feature fusion according to the channel dimension, upsample the fused feature map, and enlarge the image width and height to twice the original size to restore the spatial information of the image (i.e., channel dimension), and obtain the fusion of different channel dimensions. mid-level feature map.

The four downsampling sub-layers used in the embodiment of the present invention use pooling operations to perform downsampling, and the pooling window size is 2. In each layer, convolution is performed first, and then pooling is performed, so that after each layer of image processing, the height and width of the feature map will be shortened by half, and the number of channels will be doubled. It can effectively extract feature information of images, reduce feature dimensions, compress the number of data and parameters, and reduce training overfitting.

The function of the second feature extraction layer is to extract multi-scale features from remote sensing images of buildings of different sizes. A convolution kernel with a larger hole rate is only beneficial to extracting information about large-sized buildings, and a convolution kernel with a smaller hole rate is beneficial to extracting information about large-sized buildings. It is conducive to extracting small-sized building information. By fusing features extracted at multiple convolution scales, a deep feature map that takes into account detailed information and spatial information is obtained.

The function of the feature fusion layer is to upsample the low-level feature map, mid-level feature map and high-level feature map feature map respectively, restore the width and height to the same width and height dimensions as the target image, and then map the fused features to the building Segmentation results.

The output layer is the last layer. The function of the output layer is to use a convolution kernel with a size of 1x1 and a number of 1 to perform fully connected dimensionality reduction to obtain a feature map with the same width and height as the original image. The channel dimension of this feature map is The grayscale image of 1 is the final building segmentation result.

In a multi-layer neural network, the function of the excitation function ReLU is to perform nonlinear mapping on the feature map obtained by convolution in the output layer, because after multi-layer convolution, the value of the feature vector map does not change much during training. This causes the gradient to disappear and the convolutional network model to be untrainable. By using the ReLU excitation function after each convolution, the results of the linear calculation of the convolution layer can be nonlinearly mapped, so that the feature map changes before and after training will not be too small, and the model can be trained.

The embodiment of the present invention acquires a target image based on multi-channel remote sensing images, and extracts a refined feature map through the first feature extraction layer with a pyramid structure, as well as a mid-level feature map that integrates multiple scale features. The feature map performs convolution operations of different scales in parallel, fuses the features unified into the same scale into a deep feature map, and then fuses the three semantic level feature maps to restore the building segmentation result. It can increase the receptive field of convolution layer by layer without losing information, so that the output of each convolution contains a larger range of semantic information, which can improve the sophistication of image semantic segmentation, thereby improving the accuracy of building recognition. accuracy.

Based on any of the above embodiments, after stepwise downsampling of the target image to obtain the refined feature map, the refined feature map is stepwise upsampled and fused to obtain a mid-level feature map, including : The attention mechanism is used to sequentially perform convolution operations and pooling operations of corresponding scales in each of the downsampling sub-layers to obtain the target feature map.

Specifically, the high-resolution remote sensing image building semantic segmentation device performs convolution operations and pooling operations of corresponding scales on the input feature maps at each downsampling sub-layer, performs corresponding spatial feature compression, and uses the attention mechanism to compress The final feature map is subjected to channel feature learning to learn the importance between different channels until the target feature map is output in the last downsampling sub-layer.

The target feature map is gradually upsampled to the original size corresponding to each downsampled sub-layer and then feature fusion is performed to obtain the mid-level feature map.

Specifically, the high-resolution remote sensing image building semantic segmentation device fuses feature maps of different sizes and different levels generated by each downsampling sub-layer in the backbone network from bottom to top, effectively combining low-level and high-level feature information to obtain Mid-level feature map.

Perform dilated convolution operations and pooling operations on the target feature map to obtain the refined feature map.

Wherein, the two-dimensional dimensions of the refined feature map and the target feature map are the same, and the number of channels of the refined feature map is greater than the number of channels of the feature map.

Specifically, the building segmentation device based on remote sensing images does not perform downsampling in this step, and uses convolution with a hole rate of 2 instead of convolution with a stride of 2 to perform hole convolution operations and pooling operations on the target feature map. Reduce the loss of information in the target feature map, while ensuring the receptive field, extracting features with stronger and deeper semantics to obtain the final refined feature map. The embodiment of the present invention is based on the channel attention mechanism and improves it from two aspects: avoiding dimensionality reduction and cross-channel information interaction, effectively capturing cross-channel interaction information, enhancing the expression of key features, suppressing noise and unimportant features, and further enhancing model extraction. accuracy.

Based on any of the above embodiments, each downsampling sub-layer is a bottleneck structure.

Specifically, the large-size convolutions used in each downsampling sub-layer of the high-resolution remote sensing image building semantic segmentation device are replaced by multiple small-size convolutions, that is, the data is dimensionally reduced first, and then conventional The size convolution kernel is convolved, and finally the dimension is raised to form a bottleneck structure similar to an hourglass shape.

Illustratively, with ECA channel attention to extract features with higher semantics, the embodiment of the present invention takes setting up three layers of downsampling sub-layers to build a backbone network as an example.

A good backbone network plays a vital role in improving segmentation efficiency and accuracy, so ResNet50 can be used as the backbone network layer by layer. The residual structure in the ResNet50 network simplifies the learning process and enhances gradient propagation. The Resnet50 network is mainly composed of 4 ECA_Blocks, namely ECA_Block_1, ECA_Block_1, ECA_Block_3 and ECA_Block_4, among which:

The first layer structure of ECA_Block_1 is relatively simple and can be regarded as preprocessing of the input. Figure 2 is one of the substructure schematic diagrams of the segmentation model provided by the present invention. As shown in Figure 2, the processing process of ECA_Block_1 includes four sequential operations: convolution, BN layer, ReLU activation function, and maximum pooling layer, which are directly downsampled to 1/4 of the original image, as shown in Figure 2. The second layer of ECA_Block_1 contains 3 Bottleneck modules for shallow feature extraction. For ECA_Block_1, the input target image size is (3,512,512), and the final output size is (64,128,128).

ECA_Block_2 can be composed of 4 Bottleneck modules, the input size is (64,128,128), and the output size is (128,64,64).

ECA_Block_3 can be composed of 6 Bottleneck modules, the input size is (128,64,64), and the output size is (256,32,32).

ECA_Block_4 does not perform downsampling and uses convolution with a hole rate of 2 instead of convolution with a stride of 2 to reduce the loss of information and extract features with stronger semantics and deeper depth while ensuring the receptive field. The final output shape of ECA_Block_4 is ( 512,32,32).

Figure 3 is the second schematic diagram of the substructure of the segmentation model provided by the present invention. As shown in Figure 3, in the Bottleneck module with ECA channel attention in the above steps, it is necessary to first enter the ECA channel attention module through convolutions with sizes of 1, 3, and 1. The specific process of this module is: first, the input The feature map is subjected to a global average pooling operation; secondly, a 1-dimensional convolution operation with a convolution kernel size of K is performed, and the weight ω of each channel is obtained through the Sigmoid activation function; the weight is multiplied by the corresponding element of the original input feature map to obtain the final Output feature map. The ECA attention mechanism utilizes one-dimensional convolution with shared weights for feature learning after global average pooling without dimensionality reduction, and captures cross-channel information by considering the correlation of each channel with its neighboring K channels, so that it can be selected Express features more accurately to improve the extraction accuracy of the network model.

Figure 4 is the third schematic diagram of the substructure of the segmentation model provided by the present invention. As shown in Figure 4, feature pyramid FPN can be used to connect large-sized low-level features with small-sized high-level features to enhance the feature information of feature maps at different scales. The specific implementation process is: on the one hand, the feature map of 1/16 size and 1024 channels generated by ECA_Block_3 in the backbone network is first subjected to 1×1 convolution and dimensionality reduction, so that the number of channels becomes 256. Then it is upsampled by 2 times to obtain a branch in FPN.

On the other hand, the feature map of 1/8 size and 512 channels generated by ECA_Block_2 in the backbone network is subjected to 1×1 convolution and dimensionality reduction, so that the number of channels becomes 256, and another branch in FPN is obtained.

Finally, the feature maps of the two branches are fused in an additive manner to obtain the mid-level feature map.

The embodiment of the present invention uses the first 1x1 convolution in the Bottleneck structure to reduce the number of channels, so that the number of channels in the intermediate convolution is reduced to 1/4, and then uses the ordinary convolution in the middle to complete the convolution. The number of output channels is equal to the input The number of channels is then used to restore the number of channels by using the last 1x1 convolution, so that the number of output channels of bottleneck is equal to the number of input channels of bottleneck, which can effectively reduce the number of convolution parameters and the amount of calculation.

On the basis of any of the above embodiments, performing convolution operations of different scales on the thinned feature maps to obtain deep feature maps fused with features of the same scale includes: performing 1* on the thinned feature maps respectively. 1 convolution operation, n convolution kernels with equal increasing hole rates perform convolution operations and global average pooling operations to obtain n+2 feature maps with the same two-dimensional size and number of channels.

Specifically, the high-resolution remote sensing image building semantic segmentation device performs 1×1 convolution on the refined feature maps of the layer-by-layer compressed spatial features of the first feature extraction layer, with an initial hole rate of 4, and a difference n convolution kernels with a value of 4 and arithmetic increments are convolved at the same time. Finally, and global average pooling, n+2 feature maps with the same size and number of channels can be obtained.

The convolution kernel here can be any convolution kernel variant, for example, it can be a depthwise over-parameterized convolutional layer (DO-Conv). Do-Conv combines ordinary convolution and depth convolution to improve the feature expression ability of the model by adding linear layers.

Then, the hole Do-Conv convolution kernel with a larger hole rate is used to segment buildings with a larger area, and the hole Do-Conv kernel with a smaller hole rate is used to segment buildings with a smaller area.

Fusion of n+2 feature maps with the same two-dimensional size and channel number into the deep feature map.

Specifically, the high-resolution remote sensing image building semantic segmentation device adds and fuses n+2 feature maps with the same size and channel number to obtain a deep feature map.

Exemplarily, FIG. 5 is the fourth schematic diagram of the substructure of the segmentation model provided by the present invention. As shown in Figure 5, the embodiment of the present invention provides a specific implementation process for extracting high-level features by the Do-ASPP module structure formed when the number n of the hole Do-Conv convolution kernel is set to 4:

The Do-ASPP module is used for multi-scale feature enhancement, and the input refined feature map is passed through a 1*1 ordinary convolution kernel, a depth-wise over-parameterized convolution kernel with hole rates of 4, 8, 12, and 16, and Pooling learns building features at six scales, and finally the feature maps with different void rates are spliced and fused to obtain a final high-level feature map with multi-scale features.

Embodiments of the present invention use dilated convolution kernels of different scales to perform convolution in parallel, which can enhance the extraction of features and high-level semantic information of buildings of different scales, and effectively reduce the occurrence of hole phenomena.

Based on any of the above embodiments, any one of the n convolution kernels with equally increasing hole rates is composed of a depth convolution kernel and a standard convolution kernel.

Specifically, the high-resolution remote sensing image building semantic segmentation device will set up convolution kernels with different hole rate parameters to form a Do-Conv convolution kernel by combining ordinary convolution and depth convolution, and improve the performance by adding linear layers. The feature expression ability of the model, while continuous linear layers can be represented by a single linear layer, thus not significantly increasing the complexity of the model.

Among them, the combination of ordinary convolution and deep convolution has two forms, namely feature composition and kernel composition. Feature composition first performs depth convolution on the input feature map, and then performs standard convolution. The kernel composition method first merges the depth convolution kernel and the standard convolution kernel, and then uses the obtained intermediate results to perform standard convolution. The two methods are mathematically equivalent, but the kernel composition method has higher training efficiency, because the merger of the two convolution kernels does not increase the computational overhead during inference and does not affect the calculation speed. Therefore, Do-Conv chooses to use kernel composition for training, and its calculation formula is as follows:

Among them, D and W represent the depth convolution kernel and the standard convolution kernel respectively, O and P represent the output feature map and input feature map respectively, and the operator , * represent depth convolution and standard convolution respectively.

The embodiment of the present invention uses a deep convolution kernel and a standard convolution kernel to merge, and then uses the obtained intermediate results to perform standard convolution to train convolution kernel variants with different hole rate parameters. Compared with traditional convolution, network reasoning is not increased. Under the premise of reducing the amount of calculation, the convolution kernel variant Do-Conv obtained by using the kernel composition method uses more parameters to participate in training, which not only converges faster, but also improves network performance.

Based on any of the above embodiments, after fusing the low-level feature map, the middle-level feature map and the high-level feature map, the building segmentation result segmented from the target image is obtained, including: The high-level feature map is upsampled and then spliced and fused with the mid-level feature map to obtain a first feature fusion map.

Specifically, the high-resolution remote sensing image building semantic segmentation device upsamples the high-level feature map to a two-dimensional size consistent with the mid-level feature map, and then splices and fuses the two to obtain the first feature fusion map that stores all high-level semantic information. .

The first feature fusion map whose two-dimensional size and channel number are consistent with the low-level feature map is spliced and fused with the low-level feature map to obtain a second feature fusion map.

Specifically, the high-resolution remote sensing image building semantic segmentation device upsamples the first feature fusion map to a two-dimensional size equal to the low-level feature map, and also uses a 1*1 convolution kernel to reduce the number of channels of the low-level feature map. Consistent with the first feature fusion map, the first feature fusion map with the same two-dimensional size and channel number is then spliced and fused with the low-level feature map to obtain the second feature fusion that stores all low-level semantic information and high-level semantic information. picture.

The two-dimensional size and channel number of the second feature fusion map are restored to be equal to the target image, and the building segmentation result is obtained.

Specifically, the high-resolution remote sensing image building semantic segmentation device first uses a 1*1 convolution to restore the channel number to be equal to the target image, and then performs an upsampling operation until the two-dimensional size leaf is restored to be equal to the target image. The final scaled feature map is decoded, and each feature vector is mapped into a binary pixel value, and the building segmentation result is obtained.

Exemplarily, FIG. 6 is a schematic diagram of the overall structure of the segmentation model provided by the present invention. The embodiment of the present invention provides a specific implementation method for semantic segmentation of buildings in high-resolution remote sensing images:

(1) Data sample set construction:

Use data sets with rich texture, shape, and color features as training data to lay the foundation for the accuracy of the network. Due to the limited processing capabilities of existing hardware, this invention uniformly cuts the building remote sensing images into sub-images with a size of 512*512, and randomly divides the sub-images into training sets and test sets according to a ratio of 8:2. The training set is used to train the network model and adjust model parameters, and the test set is used to evaluate model performance.

(2) Segmentation model construction:

The segmentation model adopts an encoder and decoder structure. The encoder uses ResNet50 with ECA channel attention as the backbone network. After extracting refined feature maps from the input three-channel high-resolution remote sensing images, a DO-ASPP module is added to enhance the classification. The multi-scale feature extraction of the feature map is refined, and feature pyramid FPN is used to perform feature fusion in the coding layer. FPN is a way to fuse different levels of feature maps, which can connect large-sized low-level features with small-sized high-level features from bottom to top, and enhance the mid-level feature maps of different scale feature information.

Then, the layer-by-layer upsampling method is used for fusion in the decoding layer. The feature map output by the Do-ASPP module is first upsampled by 2 times and then fused with the feature map output by the FPN module. The fusion result is then upsampled by 2 times for the second time. Sampling is equivalent to using two 2x upsampling to achieve 4x upsampling, making the pixel values in the building extraction results more continuous and closer to the pixel values in the original image, thus effectively reducing the hole phenomenon in the results and improving accuracy. The specific implementation details are: the high-level feature map obtained by the ASPP module is upsampled 2 times and then spliced and fused with the mid-level feature map obtained by FPN to obtain a feature map with a size of 1/8 and a channel number of 512, using 1×1 convolution. After changing the number of channels to 256, continue to perform 2 times upsampling, and splice and fuse it with the low-level feature map obtained in the first stage of the backbone network.

(3) Segmentation model training:

The data sample set is used for model parameter training. The embodiment of the present invention adopts a stochastic gradient descent optimizer for model training. The initial learning rate of the encoding layer is 0.001, and the initial learning rate of the decoding layer is 0.01. In order to prevent over-fitting, the weight attenuation rate is set to is 0.0005, the size of the input target image is 512*512, the batch size (i.e. batchsize) of the image input into the network each time is 16, and 1 Epoch means that all data are sent into the network to complete one forward calculation and back propagation. process, the total Epoch number is set to 33.

(4) Optical remote sensing image building detection:

The remote sensing image to be detected is divided into sub-images (i.e. target images) with a size of 512*512, and then the target image is put into the trained segmentation model to obtain the building extraction results.

In order to verify the feasibility and effectiveness of the present invention, the embodiment of the present invention uses the urban aerial remote sensing building data set released by the French National Institute of Information and Automation (Institut national de re-cherche en informatique et en automatique, Inria) in 2017 ( Inria Aerial Image Labeling Dataset) data set was used for training and testing, and it was also evaluated qualitatively and quantitatively with the current classic deep learning network. Quantitative evaluation uses Pixel Accuracy (PA), Mean Pixel Accuracy (MPA), Mean Intersection over Union (MIoU), and Frequency Weighted Intersection over Union (FWIoU) as the The main criteria for evaluating the accuracy of building extraction, the index calculation formula used is as follows:

Among them, k is the number of categories of image pixels, p _ij represents the number of pixels with actual category i and predicted category j, and p _ii represents the number of pixels with both actual category and predicted category i.

Exemplarily, FIG. 7 is one of the simulation results of the semantic segmentation method for high-resolution remote sensing images of buildings provided by the present invention. Figure 8 is a second schematic diagram of the simulation results of the semantic segmentation method for high-resolution remote sensing images of buildings provided by the present invention. Figure 9 is the third schematic diagram of the simulation results of the semantic segmentation method of high-resolution remote sensing images of buildings provided by the present invention. As shown in Figure 7-9, a set of remote sensing images and corresponding building segmentation results are provided.

Among them, sub-figure (a) in any diagram is the true color image of the remote sensing image, sub-figure (b) is the building segmentation result of the remote sensing image in sub-figure (a) in the UNet network, and sub-figure (c) is the building segmentation result of the remote sensing image in the DeepLabv3 network in subfigure (a), subfigure (d) is the building segmentation result of the remote sensing image in subfigure (a) in the DeepLabv3+ network, and subfigure (e) is the subfigure Figure (a) shows the building segmentation result of the remote sensing image in the DeepResUnet network. Subfigure (f) shows the building segmentation result of the remote sensing image in subfigure (a) using the segmentation model of the embodiment of the present invention.

Using the above solution of this embodiment, it performs best among the building cloud segmentation results of all networks. It not only has high accuracy, but also maintains a low missed detection rate and a high intersection and union ratio.

In the segmentation result image, the building is white (that is, the pixel value is 1), and the background is black (that is, the pixel value is 0). This solution not only significantly improves the hole phenomenon in the segmentation results, but also improves the phenomenon of misclassification and missing classification.

See Figures 7 and 8. When faced with buildings with relatively complex shapes, some classic networks mistakenly identify roads with similar colors as buildings. This solution can better avoid this situation and obtain accurate building segmentation results. .

Referring to Figure 9, the embodiment of the present invention has a certain improvement in segmentation results for densely distributed buildings, and the phenomenon of missing small targets is also improved. And compare according to the relevant indicators of the target image in different networks, as shown in Table 1.

Table 1 One of the indicator comparison tables

In order to prove the effectiveness of the attention mechanism, depth-by-depth over-parameterized convolution, and feature fusion module, the present invention conducted an ablation experiment in the embodiment. As shown in Table 2, the three modules are added one after another, and each module can improve the building detection accuracy. Among them, the most obvious effect is the attention mechanism, and the least obvious one is Do_Conv. The speculated reasons are as follows: Do_Conv is mainly aimed at model training to accelerate the convergence of the model, so the improvement in MIoU is small.

Table 2 Indicator Comparison Table 2

The solution in the embodiment of the present invention is based on the red, green and blue bands commonly used in optical sensors for building detection, and has good universality; the Do-ASPP module can enhance the detection of different sizes by paralleling dilated convolution kernels of different scales. The features of scale buildings and the extraction of advanced semantic information can effectively reduce the occurrence of holes.

The feature fusion module can obtain broader and deeper features by fusing feature maps of different sizes, effectively improving the accuracy of extracting buildings of different sizes.

The attention mechanism module effectively captures cross-channel interactive information, enhances the expression of key features, suppresses noise and unimportant features, and further enhances the accuracy of model extraction, which has great practical application prospects.

By fusing feature maps with different semantic depths, embodiments of the present invention can obtain broader and deeper features, effectively improving the accuracy of extracting buildings of different sizes. It can effectively solve the loss of spatial information and detailed information caused by excessive upsampling multiples.

On the basis of any of the above embodiments, preprocessing the remote sensing image to obtain the target image includes: cropping the remote sensing image into sub-images of the same size, and performing random rotation, perspective transformation, brightness transformation and color transformation. Wait for preprocessing operations to obtain the target image.

Specifically, in step 101, the high-resolution remote sensing image building semantic segmentation device will crop the three-band remote sensing image into sub-images of the same size, and randomly rotate, perspective transform, brightness transform and each sub-image. Data enhancement operations such as color transformation are performed to obtain multiple target images.

By fusing feature maps with different semantic depths, embodiments of the present invention can obtain broader and deeper features, effectively improving the accuracy of extracting buildings of different sizes. It can effectively solve the loss of spatial information and detailed information caused by excessive upsampling multiples.

Figure 10 is a schematic structural diagram of a semantic segmentation device for high-resolution remote sensing images of buildings provided by the present invention. Based on any of the above embodiments, as shown in Figure 10, the device includes a preprocessing module 100 and a segmentation module 200, wherein:

The preprocessing module 100 is used to preprocess remote sensing images and obtain target images. Wherein, the observation content of the remote sensing image at least includes buildings.

The segmentation module 200 is configured to input the target image to a segmentation model and obtain a building segmentation result output by the segmentation model.

Wherein, the segmentation model is trained based on sample remote sensing images and building area labels corresponding to the sample remote sensing images. The segmentation model includes:

The first feature extraction layer is used to perform step-by-step downsampling of the refined feature map obtained by the target image, and then step-by-step upsample and fuse the refined feature map to obtain a mid-level feature map.

The second feature extraction layer is used to perform convolution operations of different scales on the refined feature map to obtain a deep feature map that is fused with features of the same scale.

The feature fusion layer is used to fuse the low-level feature map, the middle-level feature map and the high-level feature map to obtain the building segmentation result segmented from the target image.

Wherein, the first feature extraction layer includes a plurality of down-sampling sub-layers cascaded from bottom to top, and a plurality of levels of up-sampling sub-layers cascaded from top to bottom corresponding to each of the down-sampling sub-layers. The low-level feature map is obtained by extracting features of the target image in the first down-sampling sub-layer from bottom to top.

Specifically, the preprocessing module 100 and the segmentation module 200 are electrically connected in sequence.

The preprocessing module 100 performs corresponding preprocessing on the remote sensing image according to the set requirements to obtain the target image.

After the segmentation module 200 sets the segmentation model according to the trained model parameters, the model is used to identify and segment the building area of any target image in the preprocessing module 100, and the building corresponding to the target image can be obtained. object segmentation results.

Optionally, the first feature extraction unit in the segmentation module 200 includes a hierarchical downsampling subunit, a mid-level feature fusion unit and a feature refinement subunit, where:

The hierarchical downsampling subunit is used to sequentially perform convolution operations and pooling operations of corresponding scales in each downsampling sublayer through the attention mechanism to obtain the target feature map.

A middle-level feature fusion unit is used to gradually upsample the target feature map to the original size corresponding to each downsampling sub-layer and then perform feature fusion to obtain the middle-level feature map.

The feature refinement subunit is used to perform dilated convolution operations and pooling operations on the target feature map to obtain the refined feature map.

Wherein, the two-dimensional dimensions of the refined feature map and the target feature map are the same, and the number of channels of the refined feature map is greater than the number of channels of the feature map.

Optionally, each downsampling sub-layer is a bottleneck structure.

Optionally, the second feature extraction unit in the segmentation module 200 includes a parallel convolution subunit and a deep feature map fusion subunit, where:

The parallel convolution subunit is used to perform 1*1 convolution operations on the refined feature maps, perform convolution operations on n convolution kernels with equal increasing hole rates, and perform global average pooling operations to obtain n+ 2 feature maps with the same two-dimensional size and number of channels.

The deep feature map fusion subunit is used to fuse n+2 feature maps with the same two-dimensional size and channel number into the deep feature map.

Optionally, any one of the n convolution kernels with equally increasing hole rates is composed of a depth convolution kernel and a standard convolution kernel.

Optionally, the feature fusion unit in the segmentation module 200 includes a first fusion sub-unit, a second fusion sub-unit and a decoding sub-unit, where:

The first fusion subunit is used to upsample the high-level feature map and then splice and fuse it with the middle-level feature map to obtain a first feature fusion map.

The second fusion subunit is used to splice and fuse the first feature fusion map whose two-dimensional size and channel number are consistent with the low-level feature map and the low-level feature map to obtain a second feature fusion map.

The decoding subunit is used to restore the two-dimensional size and channel number of the second feature fusion map to be equal to the target image to obtain the building segmentation result.

Optionally, the preprocessing module 100 is specifically configured to crop the remote sensing image into sub-images of the same size, and perform preprocessing operations such as random rotation, perspective transformation, brightness transformation, and color transformation to obtain the target image.

The high-resolution remote sensing image building semantic segmentation device provided by the embodiment of the present invention is used to perform the above-mentioned high-resolution remote sensing image building semantic segmentation method of the present invention. Its implementation is the same as the high-resolution remote sensing image building semantic segmentation provided by the present invention. The implementation of the segmentation method is consistent and can achieve the same beneficial effects, and will not be described again here.

The embodiment of the present invention acquires a target image based on multi-channel remote sensing images, and extracts a refined feature map through the first feature extraction layer with a pyramid structure, as well as a mid-level feature map that integrates multiple scale features. The feature map performs convolution operations of different scales in parallel, fuses the features unified into the same scale into a deep feature map, and then fuses the three semantic level feature maps to restore the building segmentation result. It can increase the receptive field of convolution layer by layer without losing information, so that the output of each convolution contains a larger range of semantic information, which can improve the sophistication of image semantic segmentation, thereby improving the accuracy of building recognition. accuracy.

Figure 11 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 11, the electronic device may include: a processor (processor) 1110, a communications interface (Communications Interface) 1120, a memory (memory) 1130 and a communication bus 1140. Among them, the processor 1110, the communication interface 1120, and the memory 1130 complete communication with each other through the communication bus 1140. The processor 1110 can call the logic instructions in the memory 1130 to execute a method for semantic segmentation of buildings in high-resolution remote sensing images. The method includes: preprocessing the remote sensing images to obtain the target image; wherein the observation content of the remote sensing images is at least Including buildings; input the target image into the segmentation model to obtain the building segmentation result output by the segmentation model; wherein the segmentation model is based on the sample remote sensing image, and the corresponding labeled building area of the sample remote sensing image Obtained by label training; the segmentation model includes: a first feature extraction layer, which is used to perform step-by-step upsampling on the refined feature map obtained by step-by-step downsampling of the target image. and fuse to obtain a mid-level feature map; the second feature extraction layer is used to perform convolution operations on the refined feature map at different scales to obtain a deep feature map fused by features of the same scale; the feature fusion layer is used to After the low-level feature map, the middle-level feature map and the high-level feature map are fused, the building segmentation result segmented from the target image is obtained; wherein the first feature extraction layer includes a bottom-up cascade Multiple down-sampling sub-layers, and multiple levels of up-sampling sub-layers cascaded from top to bottom corresponding to each down-sampling sub-layer; the low-level feature map is a bottom-up view of the target image It is obtained by performing feature extraction on the first downsampling sub-layer.

In addition, the above-mentioned logical instructions in the memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code. .

On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can Execute the high-resolution remote sensing image building semantic segmentation method provided by each of the above methods. The method includes: preprocessing the remote sensing image to obtain the target image; wherein the observation content of the remote sensing image at least includes buildings; The target image is input to the segmentation model, and the building segmentation result output by the segmentation model is obtained; wherein the segmentation model is trained based on the sample remote sensing image and the corresponding labeled building area label of the sample remote sensing image; The segmentation model includes: a first feature extraction layer, which is used to perform step-by-step down-sampling of the refined feature map obtained by the target image, and then step-by-step upsample and fuse the refined feature map to obtain a middle-level feature map. ; The second feature extraction layer is used to perform convolution operations on the refined feature map at different scales to obtain deep feature maps that are fused with the same scale features; the feature fusion layer is used to combine the low-level feature map and the middle-level feature map After the feature map and the high-level feature map are fused, the building segmentation result segmented from the target image is obtained; wherein the first feature extraction layer includes multiple down-sampling sub-layers cascaded from bottom to top. , and multiple levels of upsampling sub-layers cascaded from top to bottom corresponding to each of the down-sampling sub-layers; the low-level feature map is the first bottom-up down-sampling sub-layer of the target image. It is obtained by feature extraction on layers.

In another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. The computer program is implemented when executed by the processor to execute the high-resolution remote sensing image building semantics provided by the above methods. Segmentation method, the method includes: preprocessing the remote sensing image to obtain a target image; wherein the observation content of the remote sensing image at least includes buildings; inputting the target image into a segmentation model to obtain the buildings output by the segmentation model Object segmentation results; wherein, the segmentation model is trained based on sample remote sensing images and building area labels corresponding to the sample remote sensing images; the segmentation model includes: a first feature extraction layer, used to After the target image is subjected to step-by-step downsampling to obtain the refined feature map, the refined feature map is gradually upsampled and fused to obtain a mid-level feature map; the second feature extraction layer is used to extract the refined feature map The graph is subjected to convolution operations at different scales to obtain deep feature maps that are fused from features of the same scale; the feature fusion layer is used to fuse the low-level feature map, the mid-level feature map and the high-level feature map to obtain the result from all the features. The building segmentation result obtained by segmenting the target image; wherein, the first feature extraction layer includes a plurality of down-sampling sub-layers cascaded from bottom to top, and corresponds to each of the down-sampling sub-layers from top to bottom. Multiple levels of cascaded upsampling sub-layers; the low-level feature map is obtained by extracting features of the target image in the first down-sampling sub-layer from bottom to top.

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be used Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A method for semantic segmentation of buildings in high-resolution remote sensing images, which is characterized by: Preprocess the remote sensing image to obtain the target image; wherein the observation content of the remote sensing image at least includes buildings; Input the target image to the segmentation model and obtain the building segmentation result output by the segmentation model; Wherein, the segmentation model is trained based on sample remote sensing images and building area labels corresponding to the sample remote sensing images; the segmentation model includes: The first feature extraction layer is used to perform step-by-step downsampling of the refined feature map obtained by the target image, and then step-by-step upsample and fuse the refined feature map to obtain a mid-level feature map; The second feature extraction layer is used to perform convolution operations on the refined feature map at different scales to obtain a deep feature map that is fused from features of the same scale; A feature fusion layer, used to fuse the low-level feature map, the middle-level feature map and the high-level feature map to obtain the building segmentation result segmented from the target image; Wherein, the first feature extraction layer includes a plurality of down-sampling sub-layers cascaded from bottom to top, and a plurality of levels of up-sampling sub-layers cascaded correspondingly from top to bottom to each of the down-sampling sub-layers; The low-level feature map is obtained by extracting features of the target image in the first down-sampling sub-layer from bottom to top. 2. The method for semantic segmentation of buildings in high-resolution remote sensing images according to claim 1, characterized in that, after the refined feature map obtained by stepwise downsampling of the target image, the refined feature map is Feature maps are upsampled and fused step by step to obtain mid-level feature maps, including: The attention mechanism is used to sequentially perform convolution operations and pooling operations of corresponding scales in each of the downsampling sub-layers to obtain the target feature map; The target feature map is gradually upsampled to the original size corresponding to each downsampling sub-layer and then feature fusion is performed to obtain the mid-level feature map; Perform dilated convolution operations and pooling operations on the target feature map to obtain the refined feature map; Wherein, the two-dimensional dimensions of the refined feature map and the target feature map are the same, and the number of channels of the refined feature map is greater than the number of channels of the feature map. 3. The method for semantic segmentation of buildings in high-resolution remote sensing images according to claim 2, characterized in that each downsampling sub-layer is a bottleneck structure. 4. The method for semantic segmentation of buildings in high-resolution remote sensing images according to claim 1, characterized in that the convolution operation of different scales is performed on the thinned feature map to obtain the deep layer after fusion of features of the same scale. Feature maps, including: Perform a 1*1 convolution operation on the refined feature maps, perform convolution operations on n convolution kernels with equal increasing hole rates, and perform global average pooling operations to obtain n+2 two-dimensional sizes and channel numbers. The same feature map; Fusion of n+2 feature maps with the same two-dimensional size and channel number into the deep feature map. 5. The method for semantic segmentation of buildings in high-resolution remote sensing images according to claim 4, characterized in that any one of the n convolution kernels with asymmetrically increasing void rates is composed of a depth The convolution kernel is merged with the standard convolution kernel. 6. The method for semantic segmentation of buildings in high-resolution remote sensing images according to claim 1, characterized in that, after fusing the low-level feature map, the mid-level feature map and the high-level feature map, the obtained result is obtained from the high-level feature map. The building segmentation results obtained by segmenting the target image include: The high-level feature map is upsampled and then spliced and fused with the mid-level feature map to obtain a first feature fusion map; Splice and fuse the first feature fusion map whose two-dimensional size and channel number are consistent with the low-level feature map and the low-level feature map to obtain the second feature fusion map; The two-dimensional size and channel number of the second feature fusion map are restored to be equal to the target image, and the building segmentation result is obtained. 7. The method for semantic segmentation of buildings in high-resolution remote sensing images according to claim 1, characterized in that preprocessing the remote sensing images to obtain the target image includes: The remote sensing image is cut into sub-images of the same size, and preprocessing operations such as random rotation, perspective transformation, brightness transformation, and color transformation are performed to obtain the target image. 8. A device for semantic segmentation of buildings in high-resolution remote sensing images, which is characterized by including: A preprocessing module, used to preprocess remote sensing images and obtain target images; wherein the observation content of the remote sensing images at least includes buildings; A segmentation module, used to input the target image to a segmentation model and obtain the building segmentation result output by the segmentation model; Wherein, the segmentation model is trained based on sample remote sensing images and building area labels corresponding to the sample remote sensing images; the segmentation model includes: The first feature extraction layer is used to perform step-by-step downsampling of the refined feature map obtained by the target image, and then step-by-step upsample and fuse the refined feature map to obtain a mid-level feature map; The second feature extraction layer is used to perform convolution operations on the refined feature map at different scales to obtain a deep feature map that is fused from features of the same scale; A feature fusion layer, used to fuse the low-level feature map, the middle-level feature map and the high-level feature map to obtain the building segmentation result segmented from the target image; Wherein, the first feature extraction layer includes a plurality of down-sampling sub-layers cascaded from bottom to top, and a plurality of levels of up-sampling sub-layers cascaded correspondingly from top to bottom to each of the down-sampling sub-layers; The low-level feature map is obtained by extracting features of the target image in the first down-sampling sub-layer from bottom to top. 9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that when the processor executes the program, it implements claim 1 The semantic segmentation method of buildings in high-resolution remote sensing images described in any one of to 7. 10. A non-transitory computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the high-resolution remote sensing image as described in any one of claims 1 to 7 is realized. A method for semantic segmentation of buildings.