WO2025107271A1 - Unsupervised super-pixel segmentation method and system assisted by collaboration between atrous pyramid and attention mechanism - Google Patents

Abstract

The present invention pertains to the technical field of digital image processing. Disclosed are an unsupervised super-pixel segmentation method and system assisted by collaboration between an atrous pyramid and an attention mechanism. The method comprises: combining image RGB channel information with position information of a pixel point; constructing a channel attention module by using an attention mechanism; processing the result of the attention mechanism by using atrous spatial pyramid pooling; constructing a loss function, constructing a clustering loss term, and, by using a spatial smoothing loss term, constructing a reconstruction loss term; updating model parameters by using an Adam optimizer, and using for super-pixel generation an effective depth feature obtained from the last separation; and obtaining the maximum value of a channel dimension by using an argmax function, converting a processing result of the argmax function into a two-dimensional array, and, on the basis of a defining condition, completing adaptive super-pixel segmentation in a CPU. According to the present invention, the complexity is low, the adaptive and generalization capabilities are high, and effective support is provided for improving image processing efficiency and accuracy.

Description

A hollow pyramid collaborative attention mechanism to assist unsupervised superpixel segmentation method and system Technical Field

The present invention belongs to the technical field of digital image processing, and in particular relates to a method and system for unsupervised superpixel segmentation assisted by a hollow pyramid collaborative attention mechanism.

Background Art

Superpixel segmentation is an effective means to improve the efficiency and accuracy of image processing. Superpixel segmentation methods based on deep learning rely on a large amount of labeled data, and there are problems such as data bias leading to inaccurate segmentation results and insufficient generalization of segmentation models. Traditional superpixel segmentation methods based on energy optimization, watershed, graph and clustering rely on appropriate parameter selection, and cannot adaptively determine the number of superpixels according to the characteristics of the image itself. They are sensitive to noise, and large-size images bring high computational complexity. Unsupervised superpixel segmentation methods are not limited by labeled data, do not need to manually adjust model parameters and structures, have better generalization ability, avoid noise interference, and will not bring high computational complexity due to changes in image size. Therefore, they are an important alternative to supervised and traditional superpixel segmentation methods. However, the use of unsupervised superpixel segmentation requires solving two key problems, namely effective deep feature extraction and superpixel generation.

Commonly used feature extraction methods are mainly divided into manually designed features and deep features extracted by convolutional neural networks. The former usually converts the RGB channel of the image into LAB channel representation, and then combines the position information of the pixels in the image to generate five-dimensional features for subsequent superpixel generation. The latter automatically learns deep features through the constructed convolutional model. Both methods can extract useful features from images. Manually designed features rely on the researchers' domain knowledge and experience of the problem, while convolutional neural networks extract deep features relying on the construction of convolutional models and the selection of feature extraction dimensions. However, manually designed features have the problem of insufficient features for superpixel segmentation of complex images, and convolutional neural networks extract deep features relying on the constructed convolutional models. Simple convolutional models cannot extract useful features, and complex convolutional models have the problem of high training difficulty.

Superpixel generation mainly refers to using the effective information of feature extraction to divide the original image into continuous, compact and small areas with similar features. Common methods include watershed, graph and clustering. In watershed technology, dark areas are usually regarded as valleys and brighter areas as ridges. The height value of the ridge is defined by the gray value or gradient size of a specific pixel, thereby realizing superpixel generation, but there is a problem of untrainability. In graph technology, pixels are regarded as nodes and edge weights are defined by the similarity of adjacent pixels, but it is difficult to process large-size images and the superpixel segmentation performance is highly dependent on the selection of parameters such as merging rules, similarity metrics and the number of superpixels. In clustering technology, superpixel generation can be quickly realized without additional labels. At the same time, trainable clustering algorithms have been developed (typically differentiable K-means clustering), but multiple iterations are required to achieve good results, which increases the computational cost and time overhead of the overall method. Other clustering technologies generally have the problem of high computational complexity.

Technical issues

Through the above analysis, the problems and defects of the existing technology are: the existing superpixel segmentation method has insufficient generalization ability, high complexity, difficulty in obtaining effective feature information and lack of adaptability.

Technical Solutions

In view of the problems existing in the prior art, the present invention provides a hollow pyramid collaborative attention mechanism to assist unsupervised superpixel segmentation method and system.

The present invention is implemented in the following way: first, the image is preprocessed to introduce the spatial relationship between pixels, and the attention mechanism is applied to the superpixel segmentation task for the first time to make its model pay more attention to important feature channels and suppress the response to irrelevant channels; the hollow space pyramid pooling is used to reduce the parameters while expanding the receptive field, and the optimized function is used to update the parameters and extract the final effective depth features in combination with the constructed loss function. The argmax function is applied to the extracted effective depth features to convert the superpixel segmentation task into a classification problem, and the final adaptive superpixel generation is achieved by adding size limiting conditions. In this way, the use of clustering algorithms can be avoided and effective depth features can be extracted with a small number of parameters, thereby greatly reducing the complexity of the superpixel segmentation algorithm. The proposed superpixel segmentation method is unsupervised and therefore has strong transferability.

Furthermore, the attention mechanism cooperates with the dilated space pyramid pooling to promote the unsupervised superpixel segmentation method, which includes the following steps:

Step 1: Combine the image RGB channel information with the pixel position information to convert the three-dimensional features into five-dimensional features. First, assign the image to be segmented into superpixels to the variables in image preprocessing, and change the variables expressed in array form to tensor form. Use the permute function to rearrange the dimensions into And change the data type to floating point type, and add a batch dimension externally through the None operation to get the variable image, whose shape is ; Use the torch.arange function to generate height and width sequences, combine the torch.meshgrid function to convert the two sequences into two coordinate grids and stack them using the torch.stack function, and compare the results to the tensor The final image preprocessing result is obtained by concatenating and normalizing in the channel dimension.

Step 2: Use the attention mechanism to build a channel attention module; apply the image preprocessing results to the point-by-point convolution layer to obtain a shape of The tensor is then processed using channel global average pooling to obtain aggregate features. The result is automatically calculated with a kernel size of The fast one-dimensional convolution and sigmoid function are processed, and the tensor obtained by point-by-point convolution is multiplied by elements to obtain the processing result of the attention mechanism. The Kaiming initialization method is used to initialize the weights of the point-by-point convolution and fast one-dimensional convolution involved, and the bias is initialized with a constant value of 0; for the instance normalization layer, the normalized weights are initialized with a constant value of 1 to ensure that there are appropriate initial parameter values during training.

Step 3: Use dilated spatial pyramid pooling to process the results of the attention mechanism and extract deep features suitable for superpixel segmentation. The tensors processed by the channel attention module are respectively processed by convolution with a size of , fill size is 0, sampling rate is 1 Convolutional layer; the convolution kernel size is , padding size is 2, sampling rate is 2 Convolutional layer; the convolution kernel size is , padding size is 4, sampling rate is 4 Convolutional layer; the convolution kernel size is , padding size is 6, sampling rate is 6 Convolutional layer; resizes the input tensor to , and then perform a step of 1 Convolution and applying the ReLU activation function Operation; set the output channels of all operations to 16, and use instance normalization to normalize each output channel, and concatenate the results of each output channel in the channel dimension to obtain a shape of The concatenated tensor uses a convolution kernel size of The convolution layer, instance normalization layer and ReLU function are processed to obtain the final shape The deep feature tensor of is obtained by , and the weights and biases of the convolutional layers and the instance normalization layers in the dilated spatial pyramid pooling are initialized using the same initialization method as the attention mechanism.

Step 4: Construct the loss function. First, construct the clustering loss term. At the same time, use the spatial smoothing loss term to quantify the difference between adjacent pixels. Then construct the reconstruction loss term. Separate the deep features obtained by the hollow space pyramid pooling into a shape of A tensor of is used to calculate the clustering loss term and the spatial smoothing loss term, and another shape is The tensor is used to calculate the reconstruction loss term; the softmax function is used to calculate the tensor The channel dimension of is processed to convert it into the corresponding category probability. The negative log-likelihood of each pixel is calculated and the average of all pixel loss values is taken to obtain the final loss value and the average probability estimate of each sample for each category is used to construct the clustering loss term. The softmax function is used again to transform the tensor The channel dimension of is processed to obtain the corresponding category probability and regard it as a probability map. The difference between each element of the probability map and the variable image in the W dimension and its adjacent element to the right and the difference between each element of the H dimension and its adjacent element below are calculated to obtain the gradient of the probability map and the variable image in the horizontal and vertical directions. The four calculated gradients are used to construct the spatial smoothing loss term. The tensor and the variable image are used to calculate the mean square error function in PyTorch to measure the difference between the two tensors, thereby determining the reconstruction loss term in the loss function.

Step 5: Update the model parameters by setting the learning rate and number of iterations of the Adam optimizer to find the model parameters that minimize the loss function; iterate through the defined optimize function and use the Adam optimizer to update the model parameters, set the number of iterations to 500, and the learning rate to The coefficient defined in the clustering loss term in the loss function is set to a constant value of 2, and the weight of the spatial smoothing loss term and the weight of the reconstruction loss term are set to constant values of 2 and 10 respectively, completing the construction of the overall loss function. In this way, the model can automatically find appropriate parameters within the set number of iterations to minimize the overall loss function.

Step 6: Use the argmax function to obtain the maximum value of the channel dimension, and convert the final effective depth feature into an optimal superpixel label index corresponding to each pixel point; convert the processing result of the argmax function into a two-dimensional array and complete the adaptive superpixel segmentation in the CPU according to the limited conditions.

Furthermore, in step 1, the image is preprocessed using the following formula:

In the formula , , Indicates the color channel value of the k-th pixel in the image converted from integer to floating point, where , Indicates the row and column number of the k-th pixel in the image. Represents the five-dimensional features of the k-th pixel in the image and applies them to subsequent processing.

Further, the construction method of step 2 specifically includes:

Step 21, applying the preprocessed five-dimensional features to a point-by-point convolutional layer, and linearly combining and transforming the five-dimensional features to achieve eight-dimensional feature output while keeping the height and width unchanged;

Step 22, calculate the channel global average pooling pair The aggregated features are obtained by

Step 23, calculate the aggregated features through a fast one-dimensional convolution with a kernel size of L:

In the formula, is the sigmoid function, is a fast one-dimensional convolution with kernel size L, is the learned channel weight; the kernel size L is automatically calculated based on the number of channels:

In the formula , , Defined as the odd integer closest to t.

Step 24, the aggregated features are processed by fast one-dimensional convolution with kernel size L and sigmoid function, and the result is expressed as , and Perform element-wise multiplication to obtain the attention mechanism processing result expressed as , the shape is consistent with the eight-dimensional features of the input attention mechanism;

.

Furthermore, the eight-dimensional feature output is expressed as And directly apply channel global average pooling, where H represents the image height, W represents the image width, and C represents the number of channels of the image. In this case, C=8.

Furthermore, constructing the channel attention module in step 3 specifically includes:

Step 31, the depth feature obtained after the hollow space pyramid pooling process is expressed as , where H and W remain unchanged and still correspond to the height and width of the image, ; The calculation process of the intermediate convolutional layer is as follows:

In the formula, Indicates that the convolution kernel size is , convolutional layer with padding size 0 and sampling rate 1; Indicates that the convolution kernel size is , convolutional layer with padding size 2 and sampling rate 2; Indicates that the convolution kernel size is , convolutional layer with padding size 4 and sampling rate 4; Indicates that the convolution kernel size is , convolutional layers with padding size 6 and sampling rate 6; Indicates that the size of the input tensor is adjusted to ; Perform a step of 1 Convolution is performed with 8 input channels and 16 output channels. Each output channel is normalized using instance normalization, and the ReLU activation function is applied to introduce nonlinearity. , , , as well as is the intermediate tensor;

Step 32, deep features The calculation is as follows:

In the formula, Indicates that , , , and Concatenate in the channel dimension to form a larger tensor, , , , and The output channels are all 16, and the number of channels of the spliced tensor is 80; Indicates that the concatenated tensor is Convolution, the output channel is set to 128 suitable for superpixel segmentation requirements; instance normalization is used to normalize each output channel; finally, the ReLU activation function is applied to introduce nonlinearity.

Furthermore, step 4 constructs the loss function, which specifically includes:

Step 41, the overall loss function consists of three parts: clustering loss term, spatial smoothing loss term and reconstruction loss term:

In the formula, Represents the overall loss function; represents the clustering loss term; represents the spatial smoothing loss term; represents the reconstruction loss term; and are constant coefficients and , .

Step 42, the clustering loss term is calculated as follows:

In the formula, ; Represents the average value of the class probability vector over all pixels; Represents the category probability vector of the pixel located at row i and column j, which is calculated as follows:

Step 43: The first three channels in the channel dimension in are separated for subsequent reconstruction loss calculation, and the features of the remaining 125 channels are used To indicate, confirm Round off, Indicates that the softmax function is used to The channel dimension is converted into the category probability of the corresponding pixel;

Step 44, the calculation of the spatial smoothing loss term is divided into two parts: Direction and The smoothness loss in the direction is to rearrange the original input image to obtain the image ; By calculating the absolute value of the probability difference and the image The exponential function of the squared difference of the gradient is defined, and the average spatial smoothness loss of all pixels is calculated as follows:

In the formula, Indicates that the channel and clustering losses are consistent; Expressed as Pixel probability difference in direction; Expressed as The pixel intensity difference in direction; Expressed as Pixel probability difference in direction; Expressed as The pixel intensity difference in the direction; the specific calculation method is as follows:

Step 45, the reconstruction loss term is calculated as follows:

When calculating the clustering loss and spatial smoothness loss, the first three channels separated from the deep features are used for image reconstruction and expressed as ; Indicates the use of 2-norm.

Furthermore, in step 44, the original input image dimension is ,image The dimension is .

Furthermore, in step five, a depth feature is extracted under the minimized model parameters, and the depth feature is an effective depth feature. The effective depth feature obtained by the last separation is used for superpixel generation.

Furthermore, the size limit condition for superpixel generation in step 6 is calculated as follows:

In the formula, represents the average size of an ideal superpixel; Represents the total number of pixels in the image; and are the minimum and maximum thresholds for limiting superpixels, respectively, and are used to filter superpixels that are too small or too large to make the size of the final generated superpixels as uniform as possible; the hyperparameters in the formula .

Another object of the present invention is to provide an attention mechanism and a hollow space pyramid pooling method to promote unsupervised superpixel segmentation system, the system comprising:

Image preprocessing module, used to combine the image RGB channel information with the pixel position information to convert the three-dimensional features into five-dimensional features;

Attention mechanism module, used to build channel attention module using attention mechanism;

The dilated spatial pyramid pooling module is used to process the results of the attention mechanism using dilated spatial pyramid pooling to extract deep features suitable for superpixel segmentation;

The loss function construction module is used to construct the loss function. First, the clustering loss term is constructed; at the same time, the spatial smoothing loss term is used to quantify the difference between adjacent pixels; and then the reconstruction loss term is constructed;

The parameter update module is used to update the model parameters by setting the learning rate and number of iterations of the Adam optimizer;

The superpixel segmentation module is used to convert the processing result of the argmax function into a two-dimensional array and complete the adaptive superpixel segmentation in the CPU according to the limited conditions.

Beneficial Effects

In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:

The present invention provides an attention mechanism and hollow space pyramid pooling to promote unsupervised superpixel segmentation method, which realizes the extraction of effective deep features through image preprocessing, attention mechanism and hollow space pyramid pooling, loss function and Adam optimizer parameter update, and converts superpixel segmentation into a classification problem to realize adaptive superpixel generation. The existing superpixel segmentation method solves the problems of insufficient generalization ability, high complexity, difficulty in obtaining effective feature information and lack of adaptive ability.

Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

Image preprocessing: Using the coordinates of the pixel positions in the image as additional features helps improve the performance of the superpixel segmentation method, allowing the model to better capture the relationship between the spatial structure and pixels in the image. The row and column numbers corresponding to each pixel in the image can generate coordinate information, and the generated coordinate information can be attached to each pixel value of the original image to create a new image with five channels. By inputting the new image with five-dimensional features into the deep feature extraction network, we can better understand the image from these features and extract deep features that are more suitable for superpixel segmentation.

Attention mechanism: In this superpixel segmentation method, the coverage of local cross-channel interactions is determined by adaptively selecting the size of the one-dimensional convolution kernel. Efficient channel attention is achieved with only a small number of parameters, which can improve the superpixel segmentation performance while weighing the complexity it brings.

Atrous spatial pyramid pooling: In superpixel segmentation tasks, each pixel needs to be classified to determine which superpixel category it belongs to. Traditional convolutional layers can only capture information within a limited range, so it is necessary to model information at different scales and contexts. Atrous spatial pyramid pooling expands the receptive field by introducing a dilation rate on the input image, which can capture information in a wider range without increasing the number of parameters. The constructed atrous spatial pyramid pooling can extract deep features suitable for superpixel segmentation tasks from images of different complexities.

Loss function: To ensure that the extracted deep features come from the image itself, a reconstruction loss term is introduced into the loss function so that the deep feature extraction is forced to generate outputs that match the original image, thereby restoring the original input image. The constructed clustering loss term is an entropy-based clustering cost, similar to the mutual information term of maximizing regularized information. Clustering is achieved by maximizing the mutual information of pixels under the deep feature representation, and the complexity of the superpixel segmentation method is controlled by the regularization term to avoid overfitting. The constructed spatial smoothness loss term is the main prior for image processing tasks, which can quantify the differences between adjacent pixels. While ensuring that the superpixel segmentation method generates smooth outputs, it also improves the generalization ability of the superpixel segmentation method and prevents overfitting.

Parameter update and superpixel generation: To achieve model parameter update, the Adam optimizer is used for gradient descent optimization. By calculating the first-order moment estimate and second-order moment estimate of the gradient, independent adaptive learning rates are designed for different parameters. In addition, the parameter update is not affected by the gradient scaling transformation, which can overcome the large noise problem of the gradient, making the model parameter update simpler and more efficient. The final effective depth feature is converted into a superpixel label index corresponding to each pixel through the argmax function. This process can convert the superpixel segmentation problem into a classification problem. Combined with the limited conditions, the adaptive superpixel segmentation can be completed quickly, avoiding the use of complex clustering algorithms.

Third, the expected benefits and commercial value of the technical solution of the present invention after transformation are: it can be applied to remote sensing image processing, greatly accelerating its processing speed.

The technical solution of the present invention overcomes technical prejudice: traditional superpixel segmentation methods are mostly implemented using clustering algorithms, while the present invention converts superpixel segmentation into a classification task, thereby avoiding the use of clustering algorithms, making the superpixel segmentation method highly adaptable to changes in image size.

Fourth, the significant technical progress brought about by the unsupervised superpixel segmentation method provided by the present invention is mainly reflected in the following aspects:

1) Enhanced feature extraction capabilities:

By combining the attention mechanism and dilated spatial pyramid pooling, this method can more effectively extract deep features of images. The introduction of the attention mechanism enables the model to pay more attention to important feature channels, thereby improving the accuracy of feature extraction.

2) Improved superpixel segmentation quality:

Traditional superpixel segmentation methods ignore some detail information, while this method can better preserve the details and structure of the image by combining spatial relationships and depth features, thereby generating more accurate and detailed superpixel segmentation results.

3) Improved flexibility and adaptability:

This method can flexibly adapt to images of different types and qualities by adaptively generating superpixels, which is especially important when dealing with diverse image datasets.

4) Parameter optimization and computational efficiency:

Using the Adam optimizer in combination with a specific loss function can more effectively optimize model parameters, reducing computational cost and time. In addition, the atrous spatial pyramid pooling reduces the number of parameters while expanding the receptive field, which further improves computational efficiency.

5) Wide application potential:

This method is not only applicable to standard image processing tasks, but can also be extended to other fields such as medical image analysis, machine vision, image recognition, etc., showing a wide range of application potential.

In general, the method provided by the present invention has achieved significant technological progress in the field of superpixel segmentation, improved segmentation quality, expanded the scope of application, and improved processing efficiency through its innovative technical combination and optimization strategy.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a flow chart of a method for promoting unsupervised superpixel segmentation by using an attention mechanism in collaboration with a dilated spatial pyramid pooling according to an embodiment of the present invention;

FIG2 is a structural diagram of a system for promoting unsupervised superpixel segmentation by using an attention mechanism in collaboration with a dilated spatial pyramid pooling according to an embodiment of the present invention;

FIG3 is a diagram of an attention mechanism constructed according to an embodiment of the present invention;

FIG4 is a hollow space pyramid pooling diagram constructed according to an embodiment of the present invention;

FIG5 is a diagram showing the effect of superpixel segmentation provided by an embodiment of the present invention;

FIG6 is a comparison diagram of the effects of the present invention; A, true label; B, SLIC; C, algorithm 2; D, superpixel segmentation result of the present invention.

Embodiments of the present invention

In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

The present invention provides an unsupervised superpixel segmentation method based on attention mechanism and void space pyramid pooling. The method mainly includes six steps, specifically:

1) Image preprocessing and feature conversion:

The RGB channel information of the image is combined with the position information of the pixel points to convert the three-dimensional features into five-dimensional features.

2) Construct channel attention module:

Through the attention mechanism, the model's attention to important feature channels is enhanced, and responses to irrelevant channels are suppressed.

3) Apply atrous spatial pyramid pooling:

Atrous spatial pyramid pooling is applied to the results of attention mechanism processing to extract deep features suitable for superpixel segmentation.

4) Construct and apply loss function:

Construct a clustering loss term, a spatial smoothness loss term (quantifying the differences between adjacent pixels), and a reconstruction loss term.

5) Model parameter update:

The model parameters are updated by setting the learning rate and number of iterations of the Adam optimizer to find the model parameters that minimize the loss function.

6) Superpixel label generation and adaptive superpixel segmentation:

Use the argmax function to obtain the maximum value of the channel dimension and convert the effective depth feature into the superpixel label index of each pixel. Convert the result of argmax into a two-dimensional array and complete the adaptive superpixel segmentation in the CPU according to the limited conditions.

The following are two specific embodiments and specific implementations of the present invention. These embodiments generally include specific applications of the method, such as specific types of image datasets (such as natural scene images, medical images, etc.) or specific application scenarios (such as image segmentation, target tracking, etc.). However, since your description does not provide enough details to identify these embodiments, I will provide two embodiments:

Application Example 1: Superpixel Segmentation of Natural Scene Images

- In this embodiment, the method can be applied to natural scene images. By analyzing features such as color and texture in the image, the model can effectively segment the image into superpixel groups with similar features. This is very useful in image editing and enhancement applications.

Application Example 2: Medical Image Analysis

- In medical images, such as MRI or CT scans, superpixel segmentation can be used to identify and differentiate different tissue types or lesions. This approach can help doctors diagnose and plan treatment more accurately.

The specific implementation will involve selecting an appropriate image dataset, adjusting model parameters (such as learning rate, number of iterations, configuration of pooling layers, etc.), and post-processing steps such as merging or segmenting superpixel groups to improve segmentation accuracy and efficiency.

In view of the problems existing in the prior art, the present invention provides a hollow pyramid collaborative attention mechanism to assist unsupervised superpixel segmentation method and system.

As shown in FIG1 , an embodiment of the present invention provides an attention mechanism and a hollow space pyramid pooling method to promote unsupervised superpixel segmentation, including the following steps:

Step 1: Image preprocessing

In order to ensure the extraction of effective depth features, the image needs to be preprocessed. The image RGB channel information is directly combined with the pixel position information to convert the three-dimensional features into five-dimensional features, ensuring the efficiency of the overall superpixel segmentation method.

In the above formula , , Indicates the color channel value of the k-th pixel in the image converted from integer to floating point type, where the type conversion is mainly to ensure the consistency of subsequent data types and the accuracy of the overall method. , Indicates the row and column number of the kth pixel in the image. Represents the five-dimensional features of the k-th pixel in the image and applies them to subsequent processing.

Step 2: Attention Mechanism

In order to enable the overall superpixel segmentation method to dynamically adjust pixel attention according to different input images, a channel attention module is constructed that does not require local cross-channel interaction of dimensionality reduction and can adaptively determine the size of one-dimensional convolution kernel and is efficient.

First, the preprocessed five-dimensional features are applied to a point-by-point convolution layer. While keeping the height and width unchanged, the five-dimensional features are linearly combined and transformed to achieve eight-dimensional feature output. The eight-dimensional feature output is represented as And directly apply channel global average pooling, where H represents the image height, W represents the image width, and C represents the number of channels of the image. In this case, C=8.

Calculate the channel global average pooling pair The aggregated features are obtained by

Calculate the aggregated features through a fast one-dimensional convolution with a kernel size of L to achieve channel attention learning for local cross-channel interaction

In the formula, is the sigmoid function, is a fast one-dimensional convolution with kernel size L, is the learned channel weight.

Automatically calculate kernel size L based on the number of channels

In the formula, , , Defined as the odd integer closest to t.

The aggregated features are processed by fast one-dimensional convolution with kernel size L and sigmoid function, and the result is expressed as , and The result of the attention mechanism processing is expressed as Its shape is consistent with the eight-dimensional feature input to the attention mechanism.

Step 3: Atrous Space Pyramid Pooling

The results of the attention mechanism are processed using atrous spatial pyramid pooling to extract deep features suitable for superpixel segmentation.

The deep features obtained after the void space pyramid pooling process are expressed as , where H and W remain unchanged and still correspond to the height and width of the image. The calculation process of the intermediate convolutional layer is as follows:

Indicates that the convolution kernel size is , convolutional layer with padding size 0 and sampling rate 1; Indicates that the convolution kernel size is , convolutional layer with padding size 2 and sampling rate 2; Indicates that the convolution kernel size is , convolutional layer with padding size 4 and sampling rate 4; Indicates that the convolution kernel size is , convolutional layers with padding size 6 and sampling rate 6; Indicates that the size of the input tensor is adjusted to , and then perform a step of 1 Convolution is performed with 8 input channels and 16 output channels. Each output channel is normalized using instance normalization, and the ReLU activation function is applied to introduce nonlinearity. , , and The input channels are all set to 8, the output channels are all set to 16, and each convolution layer uses instance normalization to normalize each output channel. , , , as well as is the intermediate tensor.

Deep Features The calculation is as follows:

In the formula, Indicates that , , , and Concatenate in the channel dimension to form a larger tensor, , , , and The output channels are all 16, so the number of channels of the concatenated tensor is 80; Indicates that the concatenated tensor is Convolution, the output channel is set to 128 which is suitable for superpixel segmentation requirements; Similarly, instance normalization is used to normalize each output channel; finally, the ReLU activation function is applied to introduce nonlinearity.

Step 4: Construct a loss function

In order to encourage deterministic superpixel allocation and make the size of each superpixel as uniform as possible, a clustering loss term is constructed; at the same time, a spatial smoothing loss term is introduced to quantify the differences between adjacent pixels; by constructing a reconstruction loss term, it can be ensured that the extracted deep features come from the image itself.

The overall loss function consists of three parts: clustering loss, spatial smoothing loss and reconstruction loss.

In the formula, Represents the overall loss function; represents the clustering loss term; represents the spatial smoothing loss term; represents the reconstruction loss term; and are constant coefficients and , .

The clustering loss term is calculated as follows:

in, ; Represents the average value of the class probability vector over all pixels; Represents the category probability vector of the pixel located at row i and column j, which is calculated as follows:

First, The first three channels in the channel dimension in are separated for subsequent reconstruction loss calculation, and the features of the remaining 125 channels are used To indicate that Round off, Indicates that the softmax function is used to The channel dimension is converted into the category probability of the corresponding pixel.

The computation of spatial smoothness loss is divided into two parts: Direction and The smoothness loss in the direction is to rearrange the original input image to obtain the image (The original input image dimension is ,image The dimension is ). By calculating the absolute value of the probability difference and the image The exponential function of the squared difference of the gradient is defined, and the average spatial smoothness loss of all pixels is calculated. The specific calculation method is as follows:

In the formula, Indicates that the channel and clustering losses are consistent; Expressed as Pixel probability difference in direction; Expressed as The pixel intensity difference in direction; Expressed as Pixel probability difference in direction; Expressed as The pixel intensity difference in the direction. The specific calculation method is as follows:

The reconstruction loss term is calculated as follows:

In the formula, when calculating the clustering loss and spatial smoothing loss, the first three channels separated from the deep features are used for image reconstruction and expressed as ; Indicates the use of 2-norm.

Step 5: Parameter update and superpixel generation

The model parameters are updated by setting the learning rate and number of iterations of the Adam optimizer, that is, finding the model parameters that minimize the loss function. The deep features extracted under this parameter are regarded as effective deep features. Since different deep features are obtained each time, the deep features obtained in the last separation are The effective depth features are used for superpixel generation. The argmax function is used to obtain the maximum value of the channel dimension, so that each pixel corresponds to the most accurate superpixel label index. The result of the argmax function is converted into a two-dimensional array and the adaptive superpixel segmentation is completed in the CPU according to the limiting conditions. The size limiting conditions for superpixel generation are calculated as follows:

In the formula, represents the average size of an ideal superpixel; Represents the total number of pixels in the image; and are the minimum and maximum thresholds for limiting superpixels, respectively, and are used to filter superpixels that are too small or too large to make the size of the final generated superpixels as uniform as possible; the hyperparameters in the formula .

As shown in FIG2 , an embodiment of the present invention provides an attention mechanism and a hollow space pyramid pooling method for promoting unsupervised superpixel segmentation. The attention mechanism and the hollow space pyramid pooling promote unsupervised superpixel segmentation system, the system comprising:

Image preprocessing module, used to combine the image RGB channel information with the pixel position information to convert the three-dimensional features into five-dimensional features;

Attention mechanism module, used to build channel attention module using attention mechanism;

The dilated spatial pyramid pooling module is used to process the results of the attention mechanism using dilated spatial pyramid pooling to extract deep features suitable for superpixel segmentation;

The loss function construction module is used to construct the loss function. First, the clustering loss term is constructed; at the same time, the spatial smoothing loss term is used to quantify the difference between adjacent pixels; and then the reconstruction loss term is constructed;

The parameter update module is used to update the model parameters by setting the learning rate and number of iterations of the Adam optimizer;

The superpixel segmentation module is used to convert the processing result of the argmax function into a two-dimensional array and complete the adaptive superpixel segmentation in the CPU according to the limited conditions.

Example:

The deep learning framework used in this embodiment is PyTorch, and the programming language is Python.

Step 1: Assign the acquired 0.8m resolution color image of GF-2 to the variable img, and check whether cuda is available. If it is available, run the subsequent code in GPU; if it is not available, run it in CPU. The acquired image is applied to the written image preprocessing function. By rearranging the dimensions, converting the data type to floating point, and adding a batch dimension externally with None operation, the shape is Tensor of ;

Read the last two dimensions of the tensor to get the height and width values of the image, use the torch.arange function to generate the height and width sequence, use the torch.meshgrid function to convert the two sequences into two coordinate grids and stack them with the torch.stack function; connect the image and the coordinate grid and perform normalization operations to get the tensor shape result.

Step 2: Apply the image preprocessing result to the point-by-point convolution layer to obtain a shape of The tensor is then processed using channel global average pooling to obtain aggregate features. The result is automatically calculated with a kernel size of The fast one-dimensional convolution and sigmoid function are processed, and the tensor obtained by point-by-point convolution is element-wise multiplied to obtain the attention mechanism (as shown in Figure 3). The shape of the processing result tensor is also ;

The Kaiming initialization method is used to initialize the weights of point-by-point convolution and fast one-dimensional convolution, and the bias is initialized with a constant value of 0. For the instance normalization layer, the normalized weights are initialized with a constant value of 1 to ensure appropriate initial parameter values during training.

Step 3: Apply the tensor obtained by the attention mechanism to the dilated space pyramid pooling (as shown in Figure 4), so that it passes through the convolution size of , fill size is 0, sampling rate is 1 Convolutional layer; the convolution kernel size is , padding size is 2, sampling rate is 2 Convolutional layer; the convolution kernel size is , padding size is 4, sampling rate is 4 Convolutional layer; the convolution kernel size is , padding size is 6, sampling rate is 6 Convolutional layer; the input tensor is resized to , and then perform a step of 1 Convolution, using the ReLU activation function to introduce nonlinearity operate;

Set the output channels of the above operations to 16, so that the shape after splicing is The concatenated tensor uses a convolution kernel size of The convolutional layer, instance normalization layer, and ReLU are processed to obtain the final shape The deep feature tensor of is obtained; each output channel is normalized using instance normalization, and finally the ReLU activation function is applied to introduce nonlinearity; the weights and biases of the convolutional layer and the instance normalization layer in the atrous spatial pyramid pooling are initialized using the same initialization method as the attention mechanism.

Step 4: Separate the extracted deep features into a shape A tensor of is used to calculate the reconstruction loss term, and another one of shape The tensor is used to calculate the clustering loss term and the spatial smoothing loss term; it is iterated through the defined optimize function, and the Adam optimizer is used to update the model parameters to minimize the loss function; the number of iterations is set to 500 times, and the learning rate is set to ; When the number of iterations is reached, the argmax function is used to obtain the maximum value of the channel dimension, so that each pixel corresponds to the most appropriate superpixel label index;

After obtaining the label index, the channel dimension is removed through the squeeze function. At the same time, the gradient is no longer calculated, and the result is transferred from the GPU to the CPU for processing; the PyTorch tensor is converted into a Numpy array for subsequent superpixel segmentation, and finally the _enforce_label_connectivity_cython function is constructed to adaptively implement superpixel segmentation according to the limited superpixel size conditions.

This embodiment processes a facial image obtained from the scipy library, and FIG5 is a superpixel segmentation effect diagram of the embodiment.

It can be seen from the above embodiments that the present invention realizes unsupervised superpixel segmentation, and the segmentation accuracy test result reaches 94.7%. The method provided by the present invention flexibly and efficiently realizes effective depth feature extraction and superpixel generation adaptively according to the characteristics of the image itself without providing the real label and the number of superpixels. Under the conditions of different image sizes and image complexities, unsupervised superpixel segmentation can also be realized quickly and accurately. It has the advantages of low complexity, adaptability and strong generalization ability, and provides effective support for improving image processing efficiency and accuracy.

When applied to classification tasks, most researchers choose graph convolution to extract features of irregularly distributed objects because traditional convolution cannot effectively extract features of irregularly distributed objects. However, it is difficult to obtain the appropriate graph structure required for graph convolution. This superpixel segmentation method can be used to generate superpixels for intermediate features during network training to adaptively generate homogeneous regions, obtain graph structures, and further generate spatial descriptors as graph nodes. The adjacency matrix obtained by considering the relationship between descriptors can meet the necessary conditions for graph convolution.

When applied to segmentation tasks, if the size of the processed image is too large, the computational overhead will increase and the segmentation algorithm will be easily disturbed by noise. The traditional pixel-level segmentation method is also prone to over-segmentation problems. These problems can be solved well by the provided superpixel algorithm, because superpixels can reduce the number of pixels in the image. Compared with segmenting each pixel, segmenting superpixels can significantly reduce the computational cost of segmentation while maintaining the image structure; because superpixels tend to merge similar pixels together, they can provide spatial consistency in a larger range, making the segmentation results more continuous while reducing the occurrence of over-segmentation problems; by aggregating local similarities, the influence of a single pixel is reduced, so various noises in the image can be suppressed to a certain extent.

During the experiment, the embodiment of the present invention was compared with the public SLIC algorithm and the superpixel segmentation algorithm using a convolutional neural network with maximum regular information (hereinafter referred to as Algorithm 2), and the relevant parameters involved were consistent with the parameters used by the algorithm developers. The following are the superpixel segmentation results generated by the real label and the segmentation algorithm respectively. As shown in Figure 6, A, real label; B, SLIC; C, Algorithm 2; D, superpixel segmentation result of the present invention.

From the above results, it can be seen that the SLIC algorithm and Algorithm 2 have poor adaptability to superpixel segmentation of large-size images. The SLIC algorithm has certain under-segmentation problems and poor boundary adhesion, but its algorithm operation efficiency is higher than that of the segmentation method of the present invention; Algorithm 2 has an over-segmentation problem and cannot generate good superpixels for the image boundaries. Although it is unsupervised like the segmentation method of the present invention, its algorithm time complexity is about 13 times that of the segmentation method of the present invention, and the segmentation accuracy generated by the two superpixel segmentation algorithms used for comparison is lower than that of the superpixel segmentation method of the present invention.

The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered within the protection scope of the present invention.

Claims (10)

A method for promoting unsupervised superpixel segmentation by using an attention mechanism and dilute spatial pyramid pooling is proposed. The method is characterized in that the image is first preprocessed to introduce the spatial relationship between pixels, and the attention mechanism is applied to the superpixel segmentation task for the first time to enable the model to strengthen the focus on important feature channels and suppress the response to irrelevant channels; dilute spatial pyramid pooling is used to reduce parameters while expanding the receptive field, and an optimizer is used in combination with a constructed loss function to achieve parameter update and extraction of the final effective deep features. The argmax function is applied to the extracted effective deep features to transform the superpixel segmentation task into a classification problem, and the final adaptive superpixel generation is achieved by adding size restriction conditions. The method for promoting unsupervised superpixel segmentation by using an attention mechanism in collaboration with atrous space pyramid pooling as claimed in claim 1 is characterized in that the method for promoting unsupervised superpixel segmentation by using an attention mechanism in collaboration with atrous space pyramid pooling comprises the following steps: Step 1: Combine the image RGB channel information with the pixel position information to convert the three-dimensional features into five-dimensional features; Step 2: Use the attention mechanism to build a channel attention module; Step 3: Use dilated spatial pyramid pooling to process the results of the attention mechanism and extract deep features suitable for superpixel segmentation; Step 4: Construct a loss function. First, construct a clustering loss term. At the same time, use a spatial smoothing loss term to quantify the difference between adjacent pixels. Then construct a reconstruction loss term. Step 5: Update the model parameters by setting the learning rate and number of iterations of the Adam optimizer to find the model parameters that minimize the loss function. Step 6: Use the argmax function to obtain the maximum value of the channel dimension, and convert the final effective depth feature into an optimal superpixel label index corresponding to each pixel point; convert the processing result of the argmax function into a two-dimensional array and complete the adaptive superpixel segmentation in the CPU according to the limited conditions. The attention mechanism and hollow space pyramid pooling to promote unsupervised superpixel segmentation method as claimed in claim 2 is characterized in that the image is preprocessed using the following formula in step 1: In the formula , , Indicates the color channel value of the k-th pixel in the image converted from integer to floating point, where , Indicates the row and column number of the k-th pixel in the image. Represents the five-dimensional features of the k-th pixel in the image and applies them to subsequent processing. The method for promoting unsupervised superpixel segmentation by using the attention mechanism in collaboration with the dilated space pyramid pooling as claimed in claim 2 is characterized in that the construction method of step 2 specifically includes: Step 21, applying the preprocessed five-dimensional features to a point-by-point convolutional layer, and linearly combining and transforming the five-dimensional features to achieve eight-dimensional feature output while keeping the height and width unchanged; Step 22, calculate the channel global average pooling pair The aggregated features are obtained by Step 23, calculate the aggregated features through a fast one-dimensional convolution with a kernel size of L: In the formula, is the sigmoid function, is a fast one-dimensional convolution with kernel size L, is the learned channel weight; the kernel size L is automatically calculated based on the number of channels: In the formula , , Defined as the odd integer closest to t. Step 24, the aggregated features are processed by fast one-dimensional convolution with kernel size L and sigmoid function, and the result is expressed as , and Perform element-wise multiplication to obtain the attention mechanism processing result expressed as , the shape is consistent with the eight-dimensional features of the input attention mechanism; . The attention mechanism and hollow space pyramid pooling method for promoting unsupervised superpixel segmentation as claimed in claim 4 is characterized in that the eight-dimensional feature output in step 21 is represented as And directly apply channel global average pooling, where H represents the image height, W represents the image width, and C represents the number of channels of the image. In this case, C=8. The method for promoting unsupervised superpixel segmentation by using the attention mechanism in collaboration with the atrous space pyramid pooling as claimed in claim 2 is characterized in that the construction of the channel attention module in step 3 specifically comprises: Step 31, the depth feature obtained after the hollow space pyramid pooling process is expressed as , where H and W remain unchanged and still correspond to the height and width of the image, ; The calculation process of the intermediate convolutional layer is as follows: In the formula, Indicates that the convolution kernel size is , convolutional layer with padding size 0 and sampling rate 1; Indicates that the convolution kernel size is , convolutional layer with padding size 2 and sampling rate 2; Indicates that the convolution kernel size is , convolutional layer with padding size 4 and sampling rate 4; Indicates that the convolution kernel size is , convolutional layers with padding size 6 and sampling rate 6; Indicates that the size of the input tensor is adjusted to ; Perform a step of 1 Convolution is performed with 8 input channels and 16 output channels. Each output channel is normalized using instance normalization, and the ReLU activation function is applied to introduce nonlinearity. , , , as well as is the intermediate tensor; Step 32, deep features The calculation is as follows: In the formula, Indicates that , , , and Concatenate in the channel dimension to form a larger tensor, , , , and The output channels are all 16, and the number of channels of the spliced tensor is 80; Indicates that the concatenated tensor is Convolution, the output channel is set to 128 suitable for superpixel segmentation requirements; instance normalization is used to normalize each output channel; finally, the ReLU activation function is applied to introduce nonlinearity. The attention mechanism and hollow space pyramid pooling to promote unsupervised superpixel segmentation method as claimed in claim 2 is characterized in that step 4 constructing a loss function specifically includes: Step 41, the overall loss function consists of three parts: clustering loss term, spatial smoothing loss term and reconstruction loss term: In the formula, Represents the overall loss function; represents the clustering loss term; represents the spatial smoothing loss term; represents the reconstruction loss term; and are constant coefficients and , . Step 42, the clustering loss term is calculated as follows: In the formula, ; Represents the average value of the class probability vector over all pixels; Represents the category probability vector of the pixel located at row i and column j, which is calculated as follows: Step 43: The first three channels in the channel dimension in are separated for subsequent reconstruction loss calculation, and the features of the remaining 125 channels are used To indicate, confirm Rounding, Indicates that the softmax function is used to The channel dimension is converted into the category probability of the corresponding pixel; Step 44, the calculation of the spatial smoothing loss term is divided into two parts: Direction and The smoothness loss in the direction is to rearrange the original input image to obtain the image ; By calculating the absolute value of the probability difference and the image The exponential function of the squared difference of the gradient is defined, and the average spatial smoothness loss of all pixels is calculated as follows: In the formula, Indicates that the channel and clustering losses are consistent; Expressed as Pixel probability difference in direction; Expressed as The pixel intensity difference in direction; Expressed as Pixel probability difference in direction; Expressed as The pixel intensity difference in the direction; the specific calculation method is as follows: Step 45, the reconstruction loss term is calculated as follows: When calculating the clustering loss and spatial smoothness loss, the first three channels separated from the deep features are used for image reconstruction and expressed as ; Indicates the use of 2-norm; Step 44: The original input image dimension is ,image The dimension is . The attention mechanism and hollow space pyramid pooling as described in claim 2 promote unsupervised superpixel segmentation method, characterized in that in step five, deep features are extracted under minimized model parameters, and the deep features are effective deep features, and the effective depth features obtained by the last separation are used for superpixel generation. The method for promoting unsupervised superpixel segmentation by using an attention mechanism in collaboration with atrous space pyramid pooling as claimed in claim 2 is characterized in that the size limiting condition for superpixel generation in step 6 is calculated as follows: In the formula, represents the average size of an ideal superpixel; Represents the total number of pixels in the image; and are the minimum and maximum thresholds for limiting superpixels, respectively, and are used to filter superpixels that are too small or too large to make the size of the final generated superpixels as uniform as possible; the hyperparameters in the formula . The attention mechanism and atrous space pyramid pooling method for promoting unsupervised superpixel segmentation according to any one of claims 1 to 9, wherein the attention mechanism and atrous space pyramid pooling system for promoting unsupervised superpixel segmentation comprises: Image preprocessing module, used to combine the image RGB channel information with the pixel position information to convert the three-dimensional features into five-dimensional features; Attention mechanism module, used to build channel attention module using attention mechanism; The dilated spatial pyramid pooling module is used to process the results of the attention mechanism using dilated spatial pyramid pooling to extract deep features suitable for superpixel segmentation; The loss function construction module is used to construct the loss function. First, the clustering loss term is constructed; at the same time, the spatial smoothing loss term is used to quantify the difference between adjacent pixels; and then the reconstruction loss term is constructed; The parameter update module is used to update the model parameters by setting the learning rate and number of iterations of the Adam optimizer; The superpixel segmentation module is used to convert the processing result of the argmax function into a two-dimensional array and complete the adaptive superpixel segmentation in the CPU according to the limited conditions.