CN116188881A - Method, device, electronic equipment and storage medium for extracting feature elements from remote sensing images - Google Patents

Abstract

The disclosure provides a method, a device, electronic equipment and a storage medium for extracting ground feature elements of a remote sensing image, which can be applied to the technical field of remote sensing image analysis. The method comprises the following steps: inputting the remote sensing image into a coding network for self-adaptive boundary extraction, extracting image features of the remote sensing image by utilizing different convolution groups, and respectively outputting a first feature image and a second feature image, wherein the convolution times in the first convolution group used for outputting the first feature image are smaller than the convolution times in the second convolution group used for outputting the second feature image, and the remote sensing image comprises boundary information of ground feature elements; pyramid pooling is carried out on the second feature map to obtain a pooled feature map; utilizing a coding network to code and enhance the boundary information of the first feature map to obtain an enhanced feature map; capturing a long-distance dependency relationship of the enhanced feature map to obtain an extracted feature map; and carrying out feature fusion on the extracted feature map and the pooled feature map to generate a target feature map.

Description

Method, device, electronic equipment and storage medium for extracting feature elements from remote sensing images

technical field

The present disclosure relates to the technical field of remote sensing image analysis, and more specifically, to a method, device, electronic equipment, storage medium, and program product for extracting feature elements from remote sensing images.

Background technique

With the rapid development of remote sensing technology, the ground object information that can be obtained from remote sensing images is becoming more and more abundant, so the extraction of ground object elements from remote sensing images has a wide range of applications in many fields. The purpose of extracting ground features is to accurately outline the boundaries of ground features in remote sensing images. In related technologies, the deep learning method converts low-level features into high-level and abstract features through cascading nonlinear mapping, which improves the performance of remote sensing image feature extraction.

In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related technologies. In the process of extracting ground features from remote sensing images using existing deep learning methods, the remote sensing images themselves have large size, complex scenes, small There are many targets, and there are shadows, occlusions, etc. at the boundaries of objects, resulting in low prediction accuracy of ground object elements.

Contents of the invention

In view of this, the embodiments of the present disclosure provide a method, device, electronic equipment, storage medium, and program product for extracting feature elements from remote sensing images.

One aspect of the embodiments of the present disclosure provides a method for extracting feature elements from remote sensing images, including: inputting remote sensing images into an encoding network for adaptive boundary extraction, using different convolution groups to extract the image features of the above remote sensing images, and outputting The first feature map and the second feature map, wherein the number of convolutions in the first convolution group used to output the first feature map is smaller than the number of convolutions in the second convolution group used to output the second feature map, The above-mentioned remote sensing image includes boundary information of feature elements; pyramid pooling is performed on the above-mentioned second feature map to obtain a pooled feature map; the above-mentioned encoding network is used to encode and enhance the boundary information of the above-mentioned first feature map to obtain an enhanced feature map ; Capture the long-distance dependency of the above-mentioned enhanced feature map to obtain the extracted feature map; perform feature fusion on the above-mentioned extracted feature map and the above-mentioned pooled feature map to generate a target feature map.

According to an embodiment of the present disclosure, the remote sensing image is input into the encoding network for adaptive boundary extraction, and different convolution groups are used to extract the image features of the remote sensing image, and the first feature map and the second feature map are respectively output, including: using The first convolutional group of the encoding network downsamples the remote sensing image to obtain the first feature map; the second convolutional group of the encoding network downsamples the first feature map to obtain the second feature map , the number of convolutions in the second convolution group is greater than the number of convolutions in the first convolution group.

According to an embodiment of the present disclosure, using the first convolutional group of the encoding network to down-sample the remote sensing image to obtain the first feature map includes: using the first dynamic hybrid gradient volume in the first convolutional group The above remote sensing image is down-sampled to obtain an initial feature map; the above-mentioned initial feature map is down-sampled by using the second dynamic hybrid gradient convolution in the above-mentioned first convolution group to obtain the above-mentioned first feature map.

According to an embodiment of the present disclosure, the aforementioned remote sensing image is down-sampled using the first dynamic hybrid gradient convolution in the first convolution group to obtain an initial feature map, including: determining the first dynamic hybrid gradient based on the remote sensing image A plurality of convolution weight coefficients for convolution, the above-mentioned first dynamic hybrid gradient convolution includes ordinary convolution kernels, first gradient convolution kernels and second gradient convolution kernels; based on the convolution weight coefficients corresponding to the above-mentioned ordinary convolution kernels , using the above-mentioned ordinary convolution kernel to perform feature extraction on the above-mentioned remote sensing image to obtain ordinary features; A gradient feature; based on the convolution weight coefficient corresponding to the second gradient convolution kernel, use the second gradient convolution kernel to perform feature extraction on the remote sensing image to obtain a second gradient feature; the above-mentioned common feature, the above-mentioned first gradient feature Perform feature fusion with the above-mentioned second gradient feature to generate the above-mentioned initial feature map.

According to an embodiment of the present disclosure, using the second convolution group of the encoding network to down-sample the first feature map to obtain the second feature map includes: using the first dynamic mixing in the second convolution group Gradient convolution downsamples the above first feature map to obtain the first convolution map; uses the second dynamic hybrid gradient convolution in the above second convolution group to encode and enhance the above first convolution map to obtain the second Convolution map: using the third dynamic hybrid gradient convolution in the second convolution group to encode and enhance the second convolution map to obtain the second feature map.

According to an embodiment of the present disclosure, the method further includes: using the encoding network to encode and enhance the boundary information of the target feature map to obtain the target enhanced feature map; capturing the long-distance dependency of the above target enhanced feature map to obtain the target extraction feature map; output the above target extraction feature map.

Another aspect of the embodiments of the present disclosure provides a device for extracting feature elements from remote sensing images, including: an encoding module, configured to input remote sensing images into an encoding network for adaptive boundary extraction, and use different convolution groups to extract the above remote sensing images The image features of , respectively output the first feature map and the second feature map, wherein the number of convolutions in the first convolution group used to output the first feature map is smaller than the second convolution group used to output the second feature map The number of convolutions in the above-mentioned remote sensing image includes the boundary information of the feature elements; the pooling module is used to perform pyramid pooling on the above-mentioned second feature map to obtain a pooled feature map; the enhancement module is used to use the above-mentioned encoding network Encoding and enhancing the boundary information of the above-mentioned first feature map to obtain an enhanced feature map; the capture module is used to capture the long-distance dependency of the above-mentioned enhanced feature map to obtain an extracted feature map; the fusion module is used to combine the above-mentioned extracted feature map with The above pooled feature map performs feature fusion to generate a target feature map.

Another aspect of the embodiments of the present disclosure provides an electronic device, including: one or more processors; memory for storing one or more programs, wherein, when the above one or more programs are used by the above one or more When the processor executes, the above-mentioned one or more processors are made to implement the above-mentioned method.

Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, and the above-mentioned instructions are used to implement the above-mentioned method when executed.

Another aspect of the embodiments of the present disclosure provides a computer program product, the above computer program product includes computer executable instructions, and the above instructions are used to implement the above method when executed.

According to the embodiments of the present disclosure, the remote sensing image is input into the encoding network for adaptive boundary extraction, and different convolution groups are used to extract the image features of the remote sensing image, so as to extract and enhance the boundary information in the remote sensing image. And pyramid pooling is performed on the second feature map to obtain richer context information. The long-distance dependencies of the enhanced feature maps are captured, and the spatial details of the boundary information in remote sensing images are extracted. Therefore, it at least partially overcomes the technical problem of low prediction accuracy of ground features due to complex scenes of remote sensing images, and improves the accuracy of remote sensing image boundary prediction.

Description of drawings

The above and other objects, features and advantages of the present disclosure will be more clearly described through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

Fig. 1 schematically shows an exemplary system architecture in which a method and device for extracting feature elements from remote sensing images can be applied according to an embodiment of the present disclosure;

FIG. 2 schematically shows a flow chart of a method for extracting feature elements from remote sensing images according to an embodiment of the present disclosure;

Fig. 3 schematically shows a schematic diagram of downsampling a remote sensing image in a method for extracting feature elements from a remote sensing image according to an embodiment of the present disclosure;

Fig. 4 schematically shows a flow chart of extracting remote sensing image features in a method for extracting feature elements from remote sensing images according to an embodiment of the present disclosure;

Fig. 5 schematically shows a block diagram of an apparatus for extracting feature elements from remote sensing images according to an embodiment of the present disclosure;

Fig. 6 schematically shows a block diagram of an electronic device adapted to implement a method for extracting feature elements from remote sensing images according to an embodiment of the present disclosure.

Detailed ways

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. It should be understood, however, that these descriptions are exemplary only, and are not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Also, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting of the present disclosure. The terms "comprising", "comprising", etc. used herein indicate the presence of stated features, steps, operations and/or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

All terms (including technical and scientific terms) used herein have the meaning commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted to have a meaning consistent with the context of this specification, and not be interpreted in an idealized or overly rigid manner.

Where expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted as those skilled in the art would normally understand the expression (for example, "having A, B, and C A system of at least one of "shall include, but not be limited to, systems with A alone, B alone, C alone, A and B, A and C, B and C, and/or A, B, C, etc. ). Where expressions such as "at least one of A, B, or C, etc." are used, they should generally be interpreted as those skilled in the art would normally understand the expression (for example, "having A, B, or C A system of at least one of "shall include, but not be limited to, systems with A alone, B alone, C alone, A and B, A and C, B and C, and/or A, B, C, etc. ).

In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of the data involved (including but not limited to user personal information) are all in compliance with relevant laws and regulations. Necessary confidentiality measures have been taken, and it does not violate public order and good customs.

With the rapid development of remote sensing technology, the acquisition of optical remote sensing images has become easier, and the ground object information that can be obtained from remote sensing images is becoming more and more abundant. The demand for fine interpretation of remote sensing images is also becoming more and more urgent. The purpose of surface feature extraction is to assign a feature category label to each pixel of the remote sensing image, which can not only identify the target in the image, but also accurately outline the boundary of the target. Therefore, the extraction of ground object elements from optical remote sensing images has a wide range of applications in many fields. However, the commonly used method of feature extraction uses manual features. This method of feature extraction requires rich prior knowledge and experience, and the representation ability of features is very limited. It is difficult to achieve impressive results in complex scenes such as remote sensing images. satisfactory effect.

The emergence of deep learning has brought a series of revolutionary progress to the field of image segmentation. Deep learning methods convert low-level features into high-level and abstract features through cascading non-linear mapping, and such high-level features are not easily obtained in previous manual feature extraction methods. The powerful feature learning ability of deep learning improves the performance of feature extraction from optical remote sensing images.

However, the prediction results obtained by existing deep learning methods still have the disadvantage of poor boundary prediction performance. On the one hand, the remote sensing image itself has large size, complex scene, many small targets, shadows and occlusions at the object boundary, which makes the boundary extraction of remote sensing image more difficult than that of natural scene pictures. On the other hand, the existing common convolution kernel does not have explicit gradient encoding restrictions during initialization, making it difficult to focus on the extraction of image gradient information during training, thus affecting the accuracy of boundary prediction.

In related technologies, in order to solve the problem of boundary ambiguity, a variety of strategies have been proposed, which can be roughly divided into the following three categories: skip connection, multi-scale context and additional boundary branch. In the first method, low-level features rich in boundary information are embedded into high-level features via skip connections. The second method is based on the multi-scale context method to expand the receptive field to generate semantic feature maps with strong contextual relations under the premise of maintaining high resolution, such as Pyramid Pooling Module (PPM) and Porous Spatial Pyramid Pooling. The third method involves an additional boundary branch, and multiple weighted edge supervisions are designed to emphasize useful boundary information.

However, the above methods still have shortcomings, and the boundary information is difficult to be explicitly encoded. How to adaptively model the difference between semantic information and boundary information, and how to adaptively enhance the extraction of boundary information while ensuring the extraction of semantic information is also a challenge.

In view of this, the embodiments of the present disclosure provide a method for extracting feature elements from remote sensing images, an apparatus for extracting feature elements from remote sensing images, an electronic device, a readable storage medium, and a computer program product. Wherein, the method for extracting feature elements from remote sensing images includes: inputting remote sensing images into an encoding network for adaptive boundary extraction, using different convolution groups to extract image features of remote sensing images, and outputting the first feature map and the second feature map respectively. Feature map, wherein the number of convolutions in the first convolution group used to output the first feature map is less than the number of convolutions in the second convolution group used to output the second feature map, and the remote sensing image includes the boundary of the feature element information; perform pyramid pooling on the second feature map to obtain the pooled feature map; use the coding network to encode and enhance the boundary information of the first feature map to obtain the enhanced feature map; capture the long-distance dependency of the enhanced feature map to obtain the extracted Feature map: feature fusion of the extracted feature map and the pooled feature map to generate the target feature map.

Fig. 1 schematically shows an exemplary system architecture in which the method and device for extracting feature elements from remote sensing images can be applied according to an embodiment of the present disclosure. It should be noted that, what is shown in FIG. 1 is only an example of the system architecture to which the embodiments of the present disclosure can be applied, so as to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other device, system, environment or scenario.

As shown in FIG. 1 , a system architecture 100 according to this embodiment may include terminal devices 101 , 102 , 103 , a network 104 and a server 105 . The network 104 is used as a medium for providing communication links between the terminal devices 101 , 102 , 103 and the server 105 . Network 104 may include various connection types, such as wired and/or wireless communication links, and the like.

Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages and the like. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and/or social platform software, etc. (only for example ).

The terminal devices 101, 102, 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers.

The server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the terminal devices 101 , 102 , 103 (just an example). The background management server can analyze and process received data such as user requests, and feed back processing results (such as webpages, information, or data obtained or generated according to user requests) to the terminal device.

It should be noted that the method for extracting feature elements from remote sensing images provided by the embodiments of the present disclosure can generally be executed by the server 105 . Correspondingly, the apparatus for extracting feature elements from remote sensing images provided in the embodiments of the present disclosure may generally be set in the server 105 . The method for extracting feature elements from remote sensing images provided in the embodiments of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101 , 102 , 103 and/or the server 105 . Correspondingly, the apparatus for extracting feature elements from remote sensing images provided in the embodiments of the present disclosure may also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101 , 102 , 103 and/or the server 105 . Alternatively, the method for extracting feature elements from remote sensing images provided by the embodiments of the present disclosure may also be executed by the terminal device 101 , 102 , or 103 , or may also be executed by other terminal devices different from the terminal device 101 , 102 , or 103 . Correspondingly, the apparatus for extracting feature elements from remote sensing images provided by the embodiments of the present disclosure may also be set in the terminal device 101 , 102 , or 103 , or in other terminal devices different from the terminal device 101 , 102 , or 103 .

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Fig. 2 schematically shows a flow chart of a method for extracting feature elements from remote sensing images according to an embodiment of the present disclosure

As shown in FIG. 2, the method includes operations S201-S205.

In operation S201, the remote sensing image is input into the encoding network for adaptive boundary extraction, the image features of the remote sensing image are extracted using different convolution groups, and the first feature map and the second feature map are respectively output, wherein the first feature map is output using The number of convolutions in the first convolution group is smaller than the number of convolutions in the second convolution group used to output the second feature map, and the remote sensing image includes boundary information of feature elements.

In operation S202, pyramid pooling is performed on the second feature map to obtain a pooled feature map.

In operation S203, the encoding network is used to encode and enhance the boundary information of the first feature map to obtain an enhanced feature map.

In operation S204, the long-distance dependency of the enhanced feature map is captured to obtain the extracted feature map.

In operation S205, feature fusion is performed on the extracted feature map and the pooled feature map to generate a target feature map.

According to an embodiment of the present disclosure, a training set composed of multiple frames of optical remote sensing images is obtained, wherein each frame of remote sensing images includes boundary information of feature elements, and different semantic categories in the remote sensing images correspond to corresponding dense labels. Semantic categories include seawater, agricultural land, green forest land, houses, and farms. Select a frame of remote sensing image from the training set and input it into the encoding network of adaptive boundary extraction. The encoding network can include the first convolution group and the second convolution group, and use different convolution groups to extract the image features of the remote sensing image, and output them respectively The first feature map and the second feature map.

According to an embodiment of the present disclosure, different convolution groups can be used in the encoding network to convolve the remote sensing image multiple times to extract boundary information in the enhanced remote sensing image, and the boundary information clearly corresponds to the drastic gradient change in the remote sensing image. Wherein, the number of convolutions in the first convolution group and the number of convolutions in the second convolution group can be adjusted according to training requirements. For example: in the encoding network, the first convolution group and the second convolution group can be used to convolute the remote sensing image five times. Wherein, the number of convolutions in the first convolution group used by the first feature map may be two times, and the number of convolutions in the second convolution group used by the second feature map may be three times.

According to an embodiment of the present disclosure, pyramid pooling is performed on the second feature map, and after pooling, it is introduced into a decoder for upsampling, so that the pooled feature map has rich context information and enhances its feature expression. The first feature map is input into the encoding network of adaptive boundary extraction, and the boundary information in the first feature map is encoded and enhanced to obtain an enhanced feature map. The enhanced feature maps are fed into the CSA module (coordinate attention module) to capture their long-distance dependencies, resulting in extracted feature maps. The feature fusion of the extracted feature map and the pooled feature map is performed, and then the features in the extracted feature map are introduced into the features of the pooled feature map. It also introduces the feature map downsampled by the encoding network into the feature map upsampled by the decoder, so as to more accurately predict the boundary of category pixels. Among them, during the fusion process, the pixel registration of the two images can be performed according to the image features of the extracted feature map and the pooled feature map, and the pixel values of the extracted feature map can be screened and corrected according to the pixel intensity values provided by the pooled feature map. And use the overall image features provided by the pooling feature map to expand the extracted feature map, so as to generate the target feature map.

According to the embodiments of the present disclosure, the remote sensing image is input into the encoding network for adaptive boundary extraction, and different convolution groups are used to extract the image features of the remote sensing image, so as to extract and enhance the boundary information in the remote sensing image. And pyramid pooling is performed on the second feature map to obtain richer context information. The long-distance dependencies of the enhanced feature maps are captured, and the spatial details of the boundary information in remote sensing images are extracted. Therefore, it at least partially overcomes the technical problem of low prediction accuracy of ground features due to complex scenes of remote sensing images, and improves the accuracy of remote sensing image boundary prediction.

According to an embodiment of the present disclosure, inputting a remote sensing image into an adaptive boundary extraction coding network, using different convolution groups to extract image features of the remote sensing image, and outputting a first feature map and a second feature map respectively may include the following operations.

Use the first convolution group of the encoding network to down-sample the remote sensing image to obtain the first feature map; use the second convolution group of the encoding network to down-sample the first feature map to obtain the second feature map, and the second convolution The number of convolutions in the group is greater than the number of convolutions in the first convolution group.

According to an embodiment of the present disclosure, assuming that the resolution of the remote sensing image is 16×16, the remote sensing image is down-sampled using the first convolution group of the encoding network to generate a first feature map with a resolution of 4×4. Wherein, the number of convolutions in the first convolution group is twice. The first feature map is down-sampled using the second convolutional group of the encoding network to generate a second feature map with a resolution of 2×2. Wherein, the number of convolutions in the first convolution group is three times. The coding network is used to down-sample the remote sensing image, and different convolution groups are used to extract the image features of the remote sensing image, which effectively extracts and enhances the boundary information in the remote sensing image.

Fig. 3 schematically shows a schematic diagram of down-sampling a remote sensing image in a method for extracting feature elements from a remote sensing image according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, using the first convolution group of the encoding network to down-sample the remote sensing image to obtain the first feature map may include the following operations.

The remote sensing image is down-sampled by using the first dynamic hybrid gradient convolution in the first convolution group to obtain an initial feature map; the initial feature map is down-sampled by using the second dynamic hybrid gradient convolution in the first convolution group, Get the first feature map.

According to the embodiment of the present disclosure, the main body of the encoding network for adaptive boundary extraction is a ResNet network (residual network). Replace all ordinary convolutions in ResNet with dynamic hybrid gradient convolutions. As shown in FIG. 3 , the first convolution group includes a first dynamic mixed gradient convolution and a second dynamic mixed gradient convolution. The remote sensing image with a resolution of 16×16 is down-sampled for the first time by using the first dynamic hybrid gradient convolution in the first convolution group to generate an initial feature map with a resolution of 8×8. The initial feature map with a resolution of 8×8 is downsampled a second time using the second dynamic hybrid gradient convolution in the first convolution group to generate the first feature map with a resolution of 4×4. By replacing all ordinary convolutions with dynamic hybrid gradient convolutions, the feature extraction of remote sensing images by the encoding network is more accurate.

Fig. 4 schematically shows a flow chart of extracting features of remote sensing images in the method for extracting feature elements from remote sensing images according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, using the first dynamic hybrid gradient convolution in the first convolution group to down-sample the remote sensing image to obtain an initial feature map may include the following operations.

Determine a plurality of convolution weight coefficients of the first dynamic mixed gradient convolution based on the remote sensing image, the first dynamic mixed gradient convolution includes a common convolution kernel, a first gradient convolution kernel and a second gradient convolution kernel; based on the common convolution The convolution weight coefficient corresponding to the kernel, using the ordinary convolution kernel to extract the feature of the remote sensing image, and obtaining the common feature; based on the convolution weight coefficient corresponding to the first gradient convolution kernel, using the first gradient convolution kernel to extract the feature of the remote sensing image, Obtain the first gradient feature; based on the convolution weight coefficient corresponding to the second gradient convolution kernel, use the second gradient convolution kernel to perform feature extraction on the remote sensing image to obtain the second gradient feature; combine the common feature, the first gradient feature and the second The gradient feature is used for feature fusion to generate an initial feature map.

According to an embodiment of the present disclosure, as shown in FIG. 4 , the dynamic hybrid gradient convolution consists of three parts: a coefficient prediction module 410 , a hybrid gradient convolution module 420 and a kernel aggregation module 430 . The dynamic hybrid gradient convolution consists of a common convolution kernel and two gradient convolution kernels. Among them, one gradient convolution kernel is obtained by rotating another gradient convolution kernel by 90 degrees. The ordinary convolution kernel has no constraints on the parameters during the convolution process, but the gradient convolution kernel has constraints on the parameters during the convolution process. During the convolution process, the gradient convolution kernel responds to 0 when it acts on the inside of the target, and the response is not 0 when it acts on the boundary of the target. Aiming at the adaptive joint modeling of semantic information and boundary information in remote sensing images, inspired by the traditional hybrid space enhancement method, the ordinary convolution kernel and gradient convolution kernel are combined into a dynamic hybrid gradient convolution, which is beneficial to a large number of remote sensing image boundaries extraction of information.

According to the embodiments of the present disclosure, since different input remote sensing images have different boundary information, the hybrid gradient convolution should be able to dynamically adjust the weight ratio of common convolution kernels and gradients according to remote sensing image content. The coefficient prediction module 410 adaptively generates convolution weight coefficients for the three convolution kernels of the hybrid gradient convolution according to the content of the remote sensing image, and the sum of the three convolution weight coefficients is 1. Aggregating ordinary convolution kernels into a new high-efficiency operator enables dynamic hybrid gradient convolution to adaptively enhance boundary information in semantic segmentation information flow, which is beneficial to the extraction of remote sensing image boundaries.

According to an embodiment of the present disclosure, in the hybrid gradient convolution module 420, a common convolution kernel is used to perform feature extraction on remote sensing images to obtain common features. The features of the remote sensing image are extracted by using the first gradient convolution kernel to obtain the first gradient features. Using the second gradient convolution kernel to perform feature extraction on the remote sensing image to obtain the second gradient feature. The kernel aggregation module 430 performs feature fusion of common features, first gradient features and second gradient features to generate an initial feature map. The kernel aggregation module 430 adopts the method of convolution kernel aggregation to equivalently replace the method of feature map aggregation, so as to accelerate the network reasoning process.

According to an embodiment of the present disclosure, using the second convolution group of the encoding network to down-sample the first feature map to obtain the second feature map may include the following operations.

Use the first dynamic hybrid gradient convolution in the second convolution group to downsample the first feature map to obtain the first convolution map; use the second dynamic hybrid gradient convolution in the second convolution group to the first volume Coding enhancement is performed on the product map to obtain a second convolution map; the second convolution map is coded and enhanced by using the third dynamic hybrid gradient convolution in the second convolution group to obtain a second feature map.

According to an embodiment of the present disclosure, the first feature map with a resolution of 4×4 is down-sampled using the first dynamic hybrid gradient convolution in the second convolution group to obtain the first convolution with a resolution of 2×2 picture. The second dynamic hybrid gradient convolution in the second convolution group is used to encode and enhance the first convolutional image to obtain a second convolutional image with a boundary gradient information enhanced resolution of 2×2. The second convolution map is coded and enhanced by using the third dynamic hybrid gradient convolution in the second convolution group to obtain a second feature map with a boundary gradient information enhanced resolution of 2×2. By using the dynamic mixed gradient convolution to extract and enhance the boundary information of the remote sensing image, the prediction of the pixel boundary of the ground object is more accurate.

According to an embodiment of the present disclosure, the method may further include the following operations.

Use the encoding network to encode and enhance the boundary information of the target feature map to obtain the target enhanced feature map; capture the long-distance dependency of the target enhanced feature map to obtain the target extraction feature map; output the target extraction feature map.

According to an embodiment of the present disclosure, a coding network is used to further encode and enhance the fused target feature map to obtain a target enhanced feature map. The object-enhanced feature map is fed into the CSA module to capture its long-distance dependencies, and the object-extracted feature map is obtained. Among them, the target feature map is the prediction result of the pixel boundary of the feature, and the target feature map is output and used for training the model, so that the prediction model can predict the pixel boundary of the feature more accurately.

Fig. 5 schematically shows a block diagram of an apparatus for extracting feature elements from remote sensing images according to an embodiment of the present disclosure.

As shown in FIG. 5 , the device 500 for extracting feature elements from remote sensing images includes an encoding module 510 , a pooling module 520 , an enhancement module 530 , a capture module 540 and a fusion module 550 .

The encoding module 510 is used to input the remote sensing image into the encoding network of adaptive boundary extraction, use different convolution groups to extract the image features of the remote sensing image, and output the first feature map and the second feature map respectively, wherein the output of the first feature The number of convolutions in the first convolution group used by the map is smaller than the number of convolutions in the second convolution group used to output the second feature map, and the remote sensing image includes boundary information of feature elements;

A pooling module 520, configured to perform pyramid pooling on the second feature map to obtain a pooling feature map;

The enhancement module 530 is used to encode and enhance the boundary information of the first feature map by using the coding network to obtain the enhanced feature map;

Capture module 540, used to capture the long-distance dependency of the enhanced feature map to obtain the extracted feature map;

The fusion module 550 is configured to perform feature fusion of the extracted feature map and the pooled feature map to generate a target feature map.

According to the embodiments of the present disclosure, the remote sensing image is input into the encoding network for adaptive boundary extraction, and different convolution groups are used to extract the image features of the remote sensing image, so as to extract and enhance the boundary information in the remote sensing image. And pyramid pooling is performed on the second feature map to obtain richer context information. The long-distance dependencies of the enhanced feature maps are captured, and the spatial details of the boundary information in remote sensing images are extracted. Therefore, it at least partially overcomes the technical problem of low prediction accuracy of ground features due to complex scenes of remote sensing images, and improves the accuracy of remote sensing image boundary prediction.

According to an embodiment of the present disclosure, the encoding module 510 includes a first down-sampling sub-module and a second down-sampling sub-module.

The first down-sampling sub-module is configured to use the first convolution group of the encoding network to down-sample the remote sensing image to obtain a first feature map.

The second downsampling sub-module is used to downsample the first feature map using the second convolution group of the encoding network to obtain a second feature map, and the number of convolutions in the second convolution group is greater than that in the first convolution group the number of convolutions.

According to an embodiment of the present disclosure, the first downsampling sub-module includes a first downsampling unit and a second downsampling unit.

The first down-sampling unit is configured to use the first dynamic mixed gradient convolution in the first convolution group to down-sample the remote sensing image to obtain an initial feature map.

The second down-sampling unit is configured to use the second dynamic hybrid gradient convolution in the first convolution group to down-sample the initial feature map to obtain the first feature map.

According to an embodiment of the present disclosure, the first downsampling unit includes a weight determination subunit, a first feature extraction subunit, a second feature extraction subunit, a third feature extraction subunit, and a feature fusion subunit.

The weight determination subunit is used to determine a plurality of convolution weight coefficients of the first dynamic mixed gradient convolution based on the remote sensing image, and the first dynamic mixed gradient convolution includes a common convolution kernel, a first gradient convolution kernel and a second gradient convolution Accumulation.

The first feature extraction subunit is configured to extract features of the remote sensing image by using the common convolution kernel based on the convolution weight coefficients corresponding to the common convolution kernel to obtain common features.

The second feature extraction subunit is configured to use the first gradient convolution kernel to perform feature extraction on the remote sensing image based on the convolution weight coefficient corresponding to the first gradient convolution kernel to obtain the first gradient feature.

The third feature extraction subunit is configured to use the second gradient convolution kernel to perform feature extraction on the remote sensing image based on the convolution weight coefficient corresponding to the second gradient convolution kernel to obtain a second gradient feature.

The feature fusion subunit is used to perform feature fusion of ordinary features, first gradient features and second gradient features to generate an initial feature map.

According to an embodiment of the present disclosure, the second downsampling sub-module includes a third downsampling unit, a first enhancement unit and a second enhancement unit.

The third down-sampling unit is configured to use the first dynamic hybrid gradient convolution in the second convolution group to down-sample the first feature map to obtain the first convolution map.

The first enhancement unit is configured to use the second dynamic hybrid gradient convolution in the second convolution group to perform coding enhancement on the first convolutional image to obtain a second convolutional image.

The second enhancement unit is configured to use the third dynamic hybrid gradient convolution in the second convolution group to perform coding enhancement on the second convolution map to obtain a second feature map.

Modules, sub-modules, units, any multiple of sub-units according to the embodiments of the present disclosure, or at least part of the functions of any multiple of them may be implemented in one module. Any one or more of modules, submodules, units, and subunits according to the embodiments of the present disclosure may be implemented by being divided into multiple modules. Any one or more of modules, submodules, units, and subunits according to embodiments of the present disclosure may be at least partially implemented as hardware circuits, such as Field Programmable Gate Array (Field Programmable Gate Array, FPGA), programmable logic Arrays (Programmable LogicArrays, PLA), system-on-chip, system-on-substrate, system-on-package, application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or any other reasonable means of hardware or firmware, or any one of software, hardware, and firmware, or an appropriate combination of any of them. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of the present disclosure may be at least partially implemented as computer program modules, and when the computer program modules are executed, corresponding functions may be performed.

For example, any number of encoding module 510, pooling module 520, enhancement module 530, capture module 540, and fusion module 550 can be implemented in one module/unit/subunit, or any one of the modules/units/subunits Units can be split into multiple modules/units/subunits. Alternatively, at least part of the functions of one or more modules/units/subunits of these modules/units/subunits can be combined with at least part of the functions of other modules/units/subunits, and combined in one module/unit/subunit realized in. According to an embodiment of the present disclosure, at least one of the encoding module 510, pooling module 520, enhancement module 530, capture module 540, and fusion module 550 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA) , programmable logic array (PLA), system-on-chip, system-on-substrate, system-on-package, application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuits, such as hardware or firmware , or implemented by any one of software, hardware and firmware, or by an appropriate combination of any of them. Alternatively, at least one of the encoding module 510, the pooling module 520, the enhancement module 530, the capture module 540, and the fusion module 550 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding Function.

It should be noted that the remote sensing image feature element extraction device part in the embodiment of the present disclosure corresponds to the remote sensing image feature element extraction method part in the embodiment of the present disclosure, and the description of the remote sensing image feature element extraction device part is specific Refer to the part of extracting method of feature elements from remote sensing images, and will not go into details here.

Fig. 6 schematically shows a block diagram of an electronic device adapted to implement a method for extracting feature elements from remote sensing images according to an embodiment of the present disclosure. The electronic device shown in FIG. 6 is only an example, and should not limit the functions and application scope of the embodiments of the present disclosure.

As shown in FIG. 6, an electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can be loaded into a random access memory according to a program stored in a read-only memory (Read-Only Memory, ROM) 602 or from a storage part 608. (Random Access Memory, RAM) 603 to perform various appropriate actions and processing. Processor 601 may include, for example, a general-purpose microprocessor (eg, a CPU), an instruction set processor and/or related chipsets, and/or a special-purpose microprocessor (eg, an application-specific integrated circuit (ASIC)), and the like. Processor 601 may also include on-board memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for executing different actions of the method flow according to the embodiments of the present disclosure.

In the RAM 603, various programs and data necessary for the operation of the electronic device 600 are stored. The processor 601 , ROM 602 and RAM 603 are connected to each other through a bus 604 . The processor 601 executes various operations according to the method flow of the embodiment of the present disclosure by executing programs in the ROM 602 and/or RAM 603 . It should be noted that the program may also be stored in one or more memories other than ROM 602 and RAM 603 . The processor 601 may also perform various operations according to the method flow of the embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, the electronic device 600 may further include an input/output (I/O) interface 605 which is also connected to the bus 604 . The system 600 may also include one or more of the following components connected to the I/O interface 605: an input section 606 including a keyboard, a mouse, etc.; ) and the like and an output section 607 of speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, and the like. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as necessary so that a computer program read therefrom is installed into the storage section 608 as necessary.

According to the embodiments of the present disclosure, the method flow according to the embodiments of the present disclosure can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable storage medium, where the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication portion 609 and/or installed from removable media 611 . When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to the embodiments of the present disclosure, the above-described systems, devices, devices, modules, units, etc. may be implemented by computer program modules.

The present disclosure also provides a computer-readable storage medium. The computer-readable storage medium may be included in the device/apparatus/system described in the above embodiments; it may also exist independently without being assembled into the device/system device/system. The above-mentioned computer-readable storage medium carries one or more programs, and when the above-mentioned one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but not limited to: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM (Erasable Programmable Read Only Memory, EPROM) or flash memory), Portable compact disk read-only memory (Computer Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include one or more memories other than the above-described ROM 602 and/or RAM 603 and/or ROM 602 and RAM 603 .

Embodiments of the present disclosure also include a computer program product, which includes a computer program, and the computer program includes program codes for executing the method provided by the embodiments of the present disclosure. When the computer program product is run on an electronic device, the program The code is used to enable the electronic device to implement the method for extracting feature elements from remote sensing images provided by the embodiments of the present disclosure.

When the computer program is executed by the processor 601, the above-mentioned functions defined in the system/device of the embodiment of the present disclosure are executed. According to the embodiments of the present disclosure, the above-described systems, devices, modules, units, etc. may be implemented by computer program modules.

In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program can also be transmitted and distributed in the form of a signal on a network medium, downloaded and installed through the communication part 609, and/or installed from the removable medium 611. The program code contained in the computer program can be transmitted by any appropriate network medium, including but not limited to: wireless, wired, etc., or any appropriate combination of the above.

According to the embodiments of the present disclosure, the program codes for executing the computer programs provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, specifically, high-level procedural and/or object-oriented programming language, and/or assembly/machine language to implement these computing programs. Programming languages include, but are not limited to, programming languages such as Java, C++, python, "C" or similar programming languages. The program code can execute entirely on the user computing device, partly on the user device, partly on the remote computing device, or entirely on the remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (for example, using an Internet service provider). business to connect via the Internet).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that includes one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block in the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a A combination of dedicated hardware and computer instructions. Those skilled in the art can understand that various combinations and/or combinations can be made in the various embodiments of the present disclosure and/or the features described in the claims, even if such combinations or combinations are not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the various embodiments of the present disclosure and/or the features described in the claims can be combined and/or combined in various ways. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the various embodiments have been described separately above, this does not mean that the measures in the various embodiments cannot be advantageously used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims (10)

1. A method for extracting feature elements from remote sensing images, comprising: Input the remote sensing image into the encoding network of adaptive boundary extraction, use different convolution groups to extract the image features of the remote sensing image, and output the first feature map and the second feature map respectively, wherein the first feature map is output using The number of convolutions in the first convolution group is less than the number of convolutions in the second convolution group used to output the second feature map, and the remote sensing image includes boundary information of feature elements; performing pyramid pooling on the second feature map to obtain a pooled feature map; Encoding and enhancing the boundary information of the first feature map by using the encoding network to obtain an enhanced feature map; capturing the long-distance dependency of the enhanced feature map to obtain the extracted feature map; performing feature fusion on the extracted feature map and the pooled feature map to generate a target feature map. 2. The method according to claim 1, wherein the remote sensing image is input into the encoding network of adaptive boundary extraction, and different convolution groups are used to extract the image features of the remote sensing image, and the first feature map and the second feature map are respectively output. Feature maps, including: using the first convolution group of the encoding network to down-sample the remote sensing image to obtain the first feature map; Using the second convolution group of the encoding network to down-sample the first feature map to obtain the second feature map, the number of convolutions in the second convolution group is greater than that in the first convolution group number of convolutions. 3. The method according to claim 2, wherein said utilizing the first convolution group of said encoding network to down-sample said remote sensing image to obtain said first feature map, comprising: Downsampling the remote sensing image by using the first dynamic hybrid gradient convolution in the first convolution group to obtain an initial feature map; The second dynamic mixed gradient convolution in the first convolution group is used to downsample the initial feature map to obtain the first feature map. 4. The method according to claim 3, wherein said utilizing the first dynamic hybrid gradient convolution in said first convolution group to downsample said remote sensing image to obtain an initial feature map, comprising: Determine a plurality of convolution weight coefficients of the first dynamic hybrid gradient convolution based on the remote sensing image, the first dynamic hybrid gradient convolution includes a common convolution kernel, a first gradient convolution kernel, and a second gradient convolution nuclear; Based on the convolution weight coefficient corresponding to the ordinary convolution kernel, using the ordinary convolution kernel to perform feature extraction on the remote sensing image to obtain ordinary features; Based on the convolution weight coefficient corresponding to the first gradient convolution kernel, using the first gradient convolution kernel to perform feature extraction on the remote sensing image to obtain a first gradient feature; Based on the convolution weight coefficient corresponding to the second gradient convolution kernel, using the second gradient convolution kernel to perform feature extraction on the remote sensing image to obtain a second gradient feature; performing feature fusion on the common feature, the first gradient feature, and the second gradient feature to generate the initial feature map. 5. The method according to claim 2, said utilizing the second convolution group of said encoding network to downsample said first feature map to obtain said second feature map, comprising: downsampling the first feature map by using the first dynamic hybrid gradient convolution in the second convolution group to obtain a first convolution map; Encoding and enhancing the first convolutional image by using the second dynamic hybrid gradient convolution in the second convolutional group to obtain a second convolutional image; Encoding and enhancing the second convolution map by using the third dynamic hybrid gradient convolution in the second convolution group to obtain the second feature map. 6. The method of claim 1, further comprising: Encoding and enhancing the boundary information of the target feature map by using the coding network to obtain a target enhanced feature map; capturing the long-distance dependency of the target enhancement feature map to obtain the target extraction feature map; Output the target extraction feature map. 7. A remote sensing image feature extraction device, comprising: The encoding module is used to input the remote sensing image into the encoding network of adaptive boundary extraction, use different convolution groups to extract the image features of the remote sensing image, and output the first feature map and the second feature map respectively, wherein the output The number of convolutions in the first convolution group used by the first feature map is less than the number of convolutions in the second convolution group used to output the second feature map, and the remote sensing image includes boundary information of feature elements; A pooling module, configured to perform pyramid pooling on the second feature map to obtain a pooling feature map; An enhancement module, configured to use the encoding network to encode and enhance the boundary information of the first feature map to obtain an enhanced feature map; A capturing module, configured to capture the long-distance dependency of the enhanced feature map to obtain the extracted feature map; A fusion module, configured to perform feature fusion on the extracted feature map and the pooled feature map to generate a target feature map. 8. An electronic device comprising: one or more processors; memory for storing one or more programs, Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method according to any one of claims 1-6. 9. A computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor can implement the method according to any one of claims 1-6. 10. A computer program product, the computer program product comprising a computer program, the computer program being used to implement the method according to any one of claims 1-6 when executed by a processor.

Images