CN118799179A - Dual-domain network reconstruction method for hyperspectral image super-resolution based on progressive hybrid convolution - Google Patents

Abstract

The present invention discloses a method for super-resolution dual-domain network reconstruction of hyperspectral images based on progressive hybrid convolution, which performs downsampling preprocessing on hyperspectral images to form a data set of high-resolution and low-resolution image pairs; constructs a super-resolution network based on progressive hybrid convolution and trains it; and inputs the low-resolution hyperspectral image into the trained network based on progressive hybrid convolution to obtain a high-resolution hyperspectral image. The present invention utilizes a dual-domain strategy of spatial domain and frequency domain to enhance image feature information, and forces the model to adaptively restore high-frequency information by converting the reconstructed hyperspectral image and the original high-resolution image into frequency domain representation and adding additional modules. This invention achieves high fidelity of spatial super-resolution and spectral dimensions through an end-to-end neural network, effectively improving the utilization of spatial-spectral feature information, thereby achieving a high-spatial-resolution hyperspectral image reconstruction effect.

Description

Hyperspectral image super-resolution dual-domain network reconstruction based on progressive hybrid convolution Method

Technical Field

The invention relates to a hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution, and belongs to the technical field of computer vision enhancement.

Background Art

Hyperspectral images (HSI) contain large-scale geographic information, but due to the limitation of sensor resolution, the resolution is not enough to capture details, which affects the continuity of the edges and texture structures of objects. Super-resolution image reconstruction refers to the process of recovering a high-resolution image from a series of noisy, blurred and undersampled low-resolution image sequences. Super-resolution (SR) reconstruction can improve image quality and visualization, and is crucial in fields such as change detection, map making and environmental monitoring.

For hyperspectral images, the problem that needs to be solved is how to improve their spatial resolution while retaining spectral information. Remote sensing super-resolution can be divided into two main categories: fusion-based methods and single super-resolution methods. Fusion-based methods aim to find and extract missing high-frequency detail information from high-resolution panchromatic images (PAN) or multispectral images (MSI). However, in practical application scenarios, these auxiliary registration images are often difficult to obtain. Therefore, single hyperspectral image super-resolution methods are increasingly valued in practical scenarios. Single hyperspectral image super-resolution methods use limited low-resolution image information to constrain the uncertainty of the super-resolution solution space by exploring and learning image prior knowledge in a large number of additional data sets. Early studies mainly used sparse representation for processing and analysis.

In recent years, with the rapid development of deep learning technology, especially the successful application of convolutional neural networks (CNNs) in the field of image processing, new breakthroughs have been made in the image super-resolution problem. By introducing deep learning models, complex mapping relationships can be effectively learned from a large amount of training data, and high-resolution images can be predicted based on this. However, compared with RGB image super-resolution, hyperspectral image super-resolution is more challenging. First, hyperspectral images usually cover a wide area and have limited spatial resolution compared to the size of the detected objects. Second, due to the relatively small number of hyperspectral image samples, it is challenging to establish a high-precision data-driven model. Although existing deep learning methods have achieved certain success, they still face many challenges and problems, such as how to design a more effective network structure to improve super-resolution performance, how to deal with the problem of insufficient training samples, and how to balance reconstruction accuracy and computational complexity. Therefore, hyperspectral image super-resolution is still an issue worthy of further study.

Existing CNN-based hyperspectral image super-resolution (HSI SR) methods have the following problems: (1) CNNs are limited in capturing extended range correlation and spectral self-similarity. Some methods try to incorporate attention mechanisms, but often sacrifice detail features with smaller weights to improve the computational efficiency of attention maps. (2) Existing methods focus on reconstructing hyperspectral images in the spatial domain, that is, using spatial domain loss functions to optimize the network, so that the image is not well modeled in the frequency domain.

Therefore, a hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution is needed to solve the above problems.

Summary of the invention

Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution.

A hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution includes the following steps:

Step 1: Degrade the high-resolution hyperspectral image to obtain a corresponding low-resolution hyperspectral image, and form an image pair data set with the low-resolution hyperspectral image and the high-resolution hyperspectral image;

Step 2: construct a hyperspectral image super-resolution network, which includes a shallow feature extraction module, a deep feature extraction module, a global upsampling module and a reconstruction module.

The shallow feature extraction module includes a 3×3 convolution and a residual block, and the residual block includes 2 convolution layers and a ReLu nonlinear function;

The deep feature extraction module is a hybrid convolutional network, and the hybrid convolutional network includes a 2D branch network and a 3D branch network;

Using the data set obtained in step 1 to train the hyperspectral image super-resolution network, a trained hyperspectral image super-resolution network is obtained;

Step 3: Optimizing the hyperspectral image super-resolution network using a dual-domain network with spatial domain and frequency domain losses to obtain a hyperspectral image super-resolution dual-domain network based on progressive hybrid convolution;

Step 4: Repeat steps 2 and 3 to update the convolution and weights until convergence, and obtain a trained hyperspectral image super-resolution dual-domain network based on progressive hybrid convolution;

Step 5: Use the trained hyperspectral image super-resolution dual-domain network based on progressive mixed convolution to reconstruct the low-resolution hyperspectral image to obtain a hyperspectral image with high spatial resolution.

Furthermore, the shallow feature extraction module in step 2 is represented by the following formula:

Where _x0 is a shallow feature, is the first residual block, is a low-resolution hyperspectral image, and F _Conv (·) is a 3×3 convolution.

Furthermore, the deep feature extraction module in step 2 is represented by the following formula:

In the formula, _xt is the deep feature, _x0 is the shallow feature, is the second residual block, ^y2D represents the output of the 2D branch network, and ^y3D represents the output of the 3D branch network;

y ^2D is represented by the following formula:

In the formula, represents the output features of the self-attention module, represents a 2D local upsampling operation, σ represents the ReLU activation function;

y ^3D is expressed by the following formula:

Where f _Squeese (·) represents the dimension of the feature map obtained after dimensionality reduction. represents a 3D local upsampling operation, Represents the cascade operation of each layer output of the 3D unit, represents a 3D convolution with a convolution kernel of 1×1×1. and They represent the layered outputs of three 3D convolutions respectively.

Furthermore, the upsampling module in step 2 is expressed by the following formula:

x _up = F _{G_up} (x _t , r')

Where _xt is the deep feature, _{FG_up} (·) represents the global upsampling operation, and r' is the scaling factor remaining after the 2D local upsampling operation and the 3D local upsampling operation.

Furthermore, the reconstruction module in step 2 is represented by the following formula:

x _rec = F _conv ( x _up )

Among them, x _rec represents the reconstruction of the convolutional layer, x _up represents the feature output of the upsampling module, and F _UP (I _LR ↑, r) means that the input low-resolution hyperspectral image is upsampled by a scale factor of r through bicubic interpolation.

Furthermore, in step 3, a dual-domain network with spatial domain and frequency domain losses is used to optimize the hyperspectral image super-resolution network, and the loss function is expressed by the following formula:

L _total = L ₁ + βL _HFL

In the formula, _L1 is the pixel loss in the spatial domain, L _HFL is the loss in the frequency domain, and β is the weight of the loss function;

Among them, the _L1 loss function is expressed by the following formula:

Where I _LR represents the low-resolution hyperspectral image, and denote the nth original high-resolution image and the reconstructed hyperspectral image, respectively, H _Net (·) denotes the hyperspectral image super-resolution network, N denotes the number of hyperspectral images in the training batch, and Θ denotes the learnable network parameters;

The L _HFL loss function is expressed as follows:

Where (u, v) represents the coordinates of the spatial frequency on the spectrum, F(u, v) is the complex frequency value, which is the function and traversal of each image pixel in the spatial domain; H and W represent the spatial size of the image; F _GT represents the frequency domain value of the original high-resolution image after frequency domain transformation; F _SR represents the frequency domain value of the reconstructed hyperspectral image after frequency domain transformation; is the Euclidean distance of the kth frequency, there are S frequencies in total, and d represents the Euclidean distance; is the weight of the spatial frequency at (u,v).

Furthermore, β is 0.1:

Beneficial effects: The hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution of the present invention designs a hybrid convolution model with progressive upsampling, which includes two-dimensional convolution (2D) and three-dimensional convolution (3D) modules, wherein the 2D module is used to improve the spatial resolution, and the 3D module is used to extract spectral information. A dual-domain network is also provided, which can better utilize spectral priors to enhance the consistency between spatial information and spectral information. In the spatial domain, a self-attention mechanism (HSL) with a pyramid structure is designed to globally model the features. In the frequency domain, the frequency domain loss (HFL) is used to optimize the generated model to improve the image quality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for super-resolution dual-domain network reconstruction of hyperspectral images based on progressive hybrid convolution;

FIG2 is a flowchart of constructing and training a dual-domain network for super-resolution hyperspectral images based on progressive hybrid convolution;

FIG3 is a framework diagram of a dual-domain network reconstruction method for hyperspectral image super-resolution based on progressive hybrid convolution, which includes shallow feature extraction, deep feature extraction, upsampling and reconstruction modules, and a dual-domain loss transformation network;

FIG4 is a network architecture diagram of heterogeneous modules provided by the present invention;

FIG5 is a model diagram of the spectral attention mechanism provided by the present invention.

DETAILED DESCRIPTION

The preferred embodiments of the present invention will be described below in conjunction with the accompanying drawings to more clearly and completely illustrate the technical solutions of the present invention.

In recent years, neural network algorithms have been widely used to enhance the resolution of hyperspectral images, but there are two main problems: (1) neural networks are limited in capturing long-range dependencies and inter-spectral correlations; (2) existing methods mainly focus on spatial domain reconstruction, but insufficiently model the frequency domain information. To address these problems, this paper proposes an innovative method that uses a hybrid convolutional model with a parallel structure with progressive upsampling to fuse 2D and 3D modules to achieve progressive reconstruction of hyperspectral images; in addition, this paper proposes a super-resolution dual-domain network to better focus on high-frequency information. Specifically, in the spatial-spectral domain, a heterogeneous group block (IGM) is designed to enhance the relationship between channels, and a spatial-spectral attention mechanism (HSL) is used to capture correlation and self-similarity; in the frequency domain, a hyperspectral frequency loss (HFL) is used to optimize the modeling.

The following will further introduce the specific implementation method, as well as the technical difficulties and inventive points of this invention in combination with this design example.

In conjunction with FIG. 1 to FIG. 5 , the present invention discloses a hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution, comprising the following steps:

Step 1: Generate a training sample set.

In order to further implement the above technical solution, in step 1, the high-resolution hyperspectral image is degraded to obtain the corresponding low-resolution hyperspectral image; then the high-resolution and low-resolution hyperspectral images are divided into blocks, and each pair of high-resolution and low-resolution hyperspectral image blocks is a training sample. In this embodiment, the following method is adopted:

Step 1-1: Take a hyperspectral data set, create a low-resolution image and a high-resolution image pair on the data set, use the original image as the high-resolution image, perform Gaussian kernel blur downsampling and interpolation processing on the original image to obtain a low-resolution image;

Step 1-2: Set the image block size and slider step size, obtain image blocks through the slider on the high-low resolution image pair and save the corresponding high- and low-resolution image block pairs.

Step 2: Construct a dual-domain network model for hyperspectral image super-resolution based on progressive hybrid convolution: The network constructed by the present invention is divided into three main parts: shallow feature extraction, deep feature extraction, and upsampling and reconstruction modules. The hyperspectral image super-resolution reconstruction network of the present invention is shown in Figure 3. First, 3×3 convolution and residual blocks are used for shallow feature extraction, where the residual block contains 2 convolution layers and a ReLu nonlinear function. Secondly, a hybrid convolution network with a parallel structure with progressive upsampling is used for deep feature extraction, where the hybrid convolution contains 2D/3D units. Finally, upsample and reconstruct a hyperspectral high-resolution image of the target size.

Step 3: Using the training sample set, a hyperspectral image super-resolution reconstruction model based on spectral grouping and hybrid convolution is constructed to learn the mapping relationship from low-resolution images to high-resolution images:

The training process in this embodiment is as follows:

Step 3-1: First, represent the high-low resolution image pairs in the training sample set as (x ₁ ,y ₁ ),(x ₂ ,y ₂ ),…,(x _i ,y _i ),…,(x _n , _yn ), where _yi is the high-resolution image corresponding to _xi ; i = 1, 2,…,n, as input.

Step 3-2: The goal of the present invention is to reconstruct the input low-resolution hyperspectral image I _LR through a super-resolution network to generate a high-resolution hyperspectral image I _SR . represents a low-resolution hyperspectral image, Represents the reconstruction result, represents the original high-resolution hyperspectral image, W and H are the width and height of HSI, L is the number of channels of HSI, and r is the SR scale factor generated by LR. The specific expression is as follows:

I _SR = H _Net (I _LR ) (1)

where H _Net (·) represents the function of the proposed hyperspectral image super-resolution reconstruction model based on spectral grouping and hybrid convolution.

Step 3-3: Use 3×3 convolution and the first residual block Perform shallow feature extraction:

Step 3-4: Use a parallel structure 2D/3D hybrid module (PAM) and a second residual block Extract deep features. To alleviate the vanishing gradient of HSI SR and effectively improve the performance of the algorithm, a residual connection is added to the network to combine the shallow feature _x0 with the deep feature _xt to further improve the model stability and information flow. The corresponding feature _xt is defined as:

Wherein, F _PAM (·) represents the parallel structure 2D/3D hybrid module operation. In order to reduce the burden of the final super-resolution reconstruction, the present invention adopts a local upsampling strategy at the end of both the 2D and 3D branch networks to decompose the complex task into multiple simple subtasks. The details are as follows:

In the 2D branch network, heterogeneous modules and attention mechanisms are included. The heterogeneous module consists of a symmetric group convolution block and a complementary convolution block, which uses a parallel approach to enhance the internal and external relationships of different channels to obtain more representative structural information of different types, as shown in Figure 4. The symmetric group convolution block contains two 3-layer branch sub-networks, and each layer of each branch sub-network is Conv+ReLU, which is used to convert feature information into nonlinear features. The complementary convolution block has only one 3-layer branch sub-network, which takes into account the overall characteristics of all channels to enhance their external correlation. The hyperspectral attention mechanism consists of two aspects, spatial and spectral, to explore the long-range dependencies between pixel positions and the correlations between spectral dimensions, as shown in Figure 5.

in represents the output features of the HSL module, and f _L-up (·) represents the local upsampling operation.

In the 3D branch network: 3D convolution is used to extract the spatial-spectral information of the hyperspectral image, so that the spectral information can be kept high-fidelity while enhancing the spatial resolution. The 3D unit contains 3 separable convolution layers and a ReLU nonlinear function:

In the formula, Indicates the cascade operation of each layer output of 3D-Unit; Indicates a 3D convolution with a convolution kernel of 1×1×1; Represent the layered outputs of three 3D convolutions respectively; Represents a local upsampling operation. Finally, the obtained features Reshape from four dimensions (1×C×W×H) to three dimensions (C×W×H) image size to get the final output of the 3DUnit branch:

Where f _Squeese (·) represents the dimension of the feature map obtained after dimensionality reduction.

In the PAM network, the outputs of the 3D unit branch network and the 2DUnit branch network are fused from the spatial pixels:

Wherein, ^y2D represents the output of the 2D branch network, and ^y3D represents the output of the 3D branch network. Note that formula (8) is equivalent to the above formula (3).

Step 3-5: In order to upgrade the obtained features to the target size, an upsampling module is used to generate the spatial spectral feature map of the target. It is particularly noted that in order to reduce the burden of the final super-resolution reconstruction, progressive upsampling is used in this paper, which is divided into local upsampling and global upsampling.

x _up = F _{G_up} (x _t , r) (9)

where F _{G_up} (·) represents the global upsampling operation, where a transposed 2D convolutional layer is used to upsample the feature map to the required scale by a scaling factor r, and the corresponding feature output is x _up .

Step 3-6: After upsampling, the convolution layer reconstruction operation is used to reconstruct the SR hyperspectral image I _SR :

x _rec =F _conv ( x _up ) (10)

Where x _rec represents the reconstruction of the convolutional layer, and F _UP (I _LR ↑) represents the upsampling operation of the input high-spectral low-resolution image with a scale factor of r through bicubic interpolation.

Step 4: Use the dual-domain loss transformation network to optimize modeling from the spatial domain and frequency domain, and gradually refine the generated features to improve image quality, as follows:

Step 4-1, the present invention uses _L1 loss to calculate the loss of pixels in the spatial domain, effectively penalizing small errors:

Step 4-2: Since the inherent bias of CNNs makes the network tend to avoid frequencies that are difficult to synthesize, and the spatial domain loss makes it difficult for the network to find these frequencies, the present invention converts the HSI data into a frequency domain representation and adds an additional module to force the model to adaptively restore high-frequency information:

Where (u, v) represents the coordinates of the spatial frequency on the spectrum, F(u, v) is the complex frequency value, and is the function sum of each image pixel traversed in the spatial domain; is the Euclidean distance of a single frequency (taking the kth frequency as an example); is the weight of the spatial frequency at (u, v); L _HFL is the frequency domain loss.

Step 4-3, the total loss used in the present invention is as follows: _L1 loss is used to calculate the pixel loss in the spatial domain, and L _HFL is used to calculate the loss in the spectral domain, combining the dual-domain losses:

L _total = L ₁ + βL _HFL (15)

Wherein, β is used to balance the two losses, and in the present invention, β is set to 0.1.

Step 5: Repeat steps 3 to 4 until the model converges by updating the convolution and weights. Once the training is completed, a trained hyperspectral image super-resolution network model based on spectral grouping and hybrid convolution will be obtained.

Step 6: Use the trained hyperspectral image super-resolution network model based on spectral grouping and mixed convolution to process the low-resolution hyperspectral image to be reconstructed, so as to reconstruct the super-resolution hyperspectral image.

Claims (7)

1. A method for super-resolution dual-domain network reconstruction of hyperspectral images based on progressive hybrid convolution, characterized in that it comprises the following steps: Step 1: Degrade the high-resolution hyperspectral image to obtain a corresponding low-resolution hyperspectral image, and form an image pair data set with the low-resolution hyperspectral image and the high-resolution hyperspectral image; Step 2: construct a hyperspectral image super-resolution network, which includes a shallow feature extraction module, a deep feature extraction module, a global upsampling module and a reconstruction module. The shallow feature extraction module includes a 3×3 convolution and a residual block, and the residual block includes 2 convolution layers and a ReLu nonlinear function; The deep feature extraction module is a hybrid convolutional network, and the hybrid convolutional network includes a 2D branch network and a 3D branch network; Using the data set obtained in step 1 to train the hyperspectral image super-resolution network, a trained hyperspectral image super-resolution network is obtained; Step 3: Optimizing the hyperspectral image super-resolution network using a dual-domain network with spatial domain and frequency domain losses to obtain a hyperspectral image super-resolution dual-domain network based on progressive hybrid convolution; Step 4: Repeat steps 2 and 3 to update the convolution and weights until convergence, and obtain a trained hyperspectral image super-resolution dual-domain network based on progressive hybrid convolution; Step 5: Use the trained hyperspectral image super-resolution dual-domain network based on progressive mixed convolution to reconstruct the low-resolution hyperspectral image to obtain a hyperspectral image with high spatial resolution. 2. The hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution as claimed in claim 1, characterized in that the shallow feature extraction module in step 2 is represented by the following formula: Where _x0 is a shallow feature, is the first residual block, is a low-resolution hyperspectral image, and F _Conv (·) is a 3×3 convolution. 3. The hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution as claimed in claim 1, characterized in that the deep feature extraction module in step 2 is represented by the following formula: In the formula, _xt is the deep feature, _x0 is the shallow feature, is the second residual block, ^y2D represents the output of the 2D branch network, and ^y3D represents the output of the 3D branch network; y ^2D is represented by the following formula: In the formula, represents the output features of the self-attention module, represents a 2D local upsampling operation, σ represents the ReLU activation function; y ^3D is expressed by the following formula: Where f _S queese(·) represents the dimension of the feature map obtained after dimensionality reduction. represents a 3D local upsampling operation, Represents the cascade operation of each layer output of the 3D unit, represents a 3D convolution with a convolution kernel of 1×1×1. and They represent the layered outputs of three 3D convolutions respectively. 4. The hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution as claimed in claim 1, characterized in that the global upsampling module in step 2 is represented by the following formula: x _up = F _{G_up} (x _t , r') Where _xt is the deep feature, _{FG_up} (·) represents the global upsampling operation, and r' is the scaling factor remaining after the 2D local upsampling operation and the 3D local upsampling operation. 5. The hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution as claimed in claim 1, characterized in that the reconstruction module in step 2 is represented by the following formula: x _rec = F _conv ( x _up ) Among them, x _rec represents the reconstruction of the convolutional layer, x _up represents the feature output of the upsampling module, and F _UP (I _LR ↑, r) means that the input low-resolution hyperspectral image is upsampled by a scale factor of r through bicubic interpolation. 6. The method for reconstructing hyperspectral image super-resolution based on dual-domain network with progressive hybrid convolution as claimed in claim 1, characterized in that in step 3, a dual-domain network with spatial domain and frequency domain losses is used to optimize the hyperspectral image super-resolution network, and the loss function is expressed by the following formula: L _total = L ₁ + βL _HFL In the formula, _L1 is the pixel loss in the spatial domain, L _HFL is the loss in the frequency domain, and β is the weight of the loss function; Among them, the _L1 loss function is expressed by the following formula: Where I _LR represents the low-resolution hyperspectral image, and denote the nth original high-resolution image and the reconstructed hyperspectral image, respectively, H _Net (·) denotes the hyperspectral image super-resolution network, N denotes the number of hyperspectral images in the training batch, and Θ denotes the learnable network parameters; The L _HFL loss function is expressed as follows: Where (u, v) represents the coordinates of the spatial frequency on the spectrum, F(u, v) is the complex frequency value, which is the function and traversal of each image pixel in the spatial domain; H and W represent the spatial size of the image; F _GT represents the frequency domain value of the original high-resolution image after frequency domain transformation; F _SR represents the frequency domain value of the reconstructed hyperspectral image after frequency domain transformation; is the Euclidean distance of the kth frequency, there are S frequencies in total, and d represents the Euclidean distance; is the weight of the spatial frequency at (u,v). 7. The hyperspectral image super-resolution dual-domain network reconstruction method based on progressive hybrid convolution as described in claim 6 is characterized in that β is 0.1.