WO2025060272A1 - Weakly supervised semantic segmentation method and apparatus based on attention mask - Google Patents

Abstract

The present disclosure relates to a weakly supervised semantic segmentation method and apparatus based on an attention mask. The method comprises: inputting a sample image into an attention encoder, so as to obtain a global class token feature, an image classification result and two semantic segmentation results; then inputting the sample image into the attention encoder again, so as to generate an attention matrix and to randomly generate a target mask matrix; compensating for the attention matrix by means of the target mask matrix, so as to obtain a compensated attention matrix; on the basis of the compensated attention matrix, generating each local class token feature; and distinguishing positive and negative characteristics of the local class token features. Model losses not only include losses in terms of image classification and image semantic segmentation, but also comprises losses caused by distinguishing positive and negative characteristics of local class token features and performing comparative learning together with global class token features, and semantic segmentation of a model is supervised by means of introducing a plurality of losses, thereby improving the accuracy of semantic segmentation.

Description

A weakly supervised semantic segmentation method and device based on attention mask Technical Field

The present disclosure relates to the technical field of semantic segmentation, and in particular to a weakly supervised semantic segmentation method and device based on attention mask.

Background Art

The current image semantic segmentation method uses a fully supervised training method with pixel-level annotations, which requires densely annotating the label information of all pixels in the image, thus greatly increasing the manpower and time costs. The weakly supervised semantic segmentation method only requires training data with the object category labels that appear in the image. Therefore, the advantages in manpower and time costs have made weakly supervised semantic segmentation gain widespread attention.

Weakly supervised semantic segmentation (WSSS) refers to predicting the category label of each pixel in an image using bounding box annotation, scribble annotation, point annotation or image-level category annotation. This paper mainly studies a method of training a neural network based on image-level labels (such as the category of objects appearing in the image) to implement a model for semantic segmentation of images.

Most of the weakly supervised semantic segmentation methods based on image-level classification annotation are based on the Class Activation Map (CAM) method. Class Activation Map is a technique based on deep classification networks that is used to generate feature maps with the same number of channels as the total number of categories, showing the approximate location of objects in each category.

Currently, how to improve the accuracy of weakly supervised semantic segmentation is an urgent problem to be solved.

Summary of the invention

The present invention provides a method and device for weakly supervised semantic segmentation based on attention mask.

According to a first aspect of an embodiment of the present disclosure, a weakly supervised semantic segmentation method based on attention mask is provided, comprising:

Obtaining a sample image and an image classification label corresponding to the sample image;

Input the sample image into the attention encoder;

Obtaining patch token features and global category token features through the attention encoder, generating an image classification result and a class activation map for the sample image through the patch token features, obtaining a first semantic segmentation result for the sample image through the class activation map, and decoding the patch token features to obtain a second semantic segmentation result;

The attention encoder generates an attention matrix, randomly generates a target mask matrix, and compensates the attention matrix with the target mask matrix to obtain a compensated attention matrix. Attention matrix, generating local category token features;

According to the auxiliary class activation map and the target mask matrix, determine the average activation value of some activation values in the class activation map that are not affected by the mask in the target mask matrix, if the average activation value is greater than a preset threshold, use the local class token feature as a positive local class token feature, if the average activation value is less than or equal to the preset threshold, use the local class token feature as a negative local class token feature;

The attention encoder is trained with minimizing the overall loss as the optimization goal, and the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

Optionally, inputting the sample image into an attention encoder specifically includes:

Segmenting the sample image into a plurality of sub-images to determine a plurality of initial patch token features corresponding to the plurality of sub-images, and determining an initial category token feature corresponding to the sample image;

The multiple initial patch token features are concatenated with the initial category token features and input into the attention encoder.

Optionally, the randomly generating a target mask matrix specifically includes:

determining a size of the attention matrix;

Downsampling the size by a preset multiple to obtain a sampling size, and randomly generating an initial mask matrix having a size equal to the sampling size;

The initial mask matrix is upsampled by the preset multiple to obtain the target mask matrix.

Optionally, the attention matrix includes: a query matrix, a key matrix and a value matrix;

The compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix specifically includes:

The compensated attention matrix is determined by the following formula:


Z ^(l) = A′ ^(l) V ^(l)

Among them, A' is the compensated attention matrix, M is the target mask matrix, Q is the query matrix, K is the key matrix, V is the value matrix, Z is the output matrix, and l is the lth self-attention encoding layer of the attention encoder.

Optionally, the attention encoder includes a plurality of self-attention encoding layers; and the method further includes:

Obtain an output result of a target self-attention coding layer among the multiple self-attention coding layers.

Optionally, the output result of the target self-attention encoding layer includes intermediate patch token features, and the method further includes:

Determining auxiliary classification results through the intermediate patch token features;

The attention encoder is trained with minimizing the overall loss as an optimization goal, wherein the overall loss also includes: the difference between the auxiliary classification result and the image classification label.

Optionally, the output result of the target self-attention encoding layer includes the auxiliary class activation map, and the method further includes:

Determine a first affinity matrix based on the foreground-background segmentation result determined by the auxiliary class activation map;

Determining a second affinity matrix according to the patch token characteristics;

The attention encoder is trained with minimizing the overall loss as an optimization goal, wherein the overall loss also includes: a difference between the first affinity matrix and the second affinity matrix.

Optionally, generating an attention matrix by the attention encoder, randomly generating a target mask matrix, and compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix, and generating a local category token feature according to the compensated attention matrix, specifically includes:

A plurality of target mask matrices are randomly generated. For each target mask matrix, the sample image is input into the attention encoder to generate an attention matrix corresponding to the sample image. The attention matrix generated this time is compensated by the target mask matrix to obtain a corresponding compensated attention matrix. According to the corresponding compensated attention matrix, a local category token feature corresponding to the target mask matrix is generated; and a plurality of local category token features are used as the local category token features.

Optionally, the method further comprises:

Use the trained attention encoder to perform image semantic segmentation on the image to be recognized.

According to a second aspect of an embodiment of the present disclosure, a weakly supervised semantic segmentation device based on attention mask is provided, comprising:

An acquisition module, used to acquire a sample image and an image classification label corresponding to the sample image;

An input module is used to input the sample image into an attention encoder, obtain patch token features and global category token features through the attention encoder, generate an image classification result and a class activation map for the sample image through the patch token features, obtain a first semantic segmentation result for the sample image through the class activation map, and decode the patch token features to obtain a second semantic segmentation result;

The mask compensation module is used to generate an attention matrix through the attention encoder and randomly generate a target mask matrix. matrix, and compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix, and generating local category token features according to the compensated attention matrix;

A judgment module, used for determining, according to the auxiliary class activation map and the target mask matrix, an average activation value of some activation values in the class activation map that are not affected by the mask in the target mask matrix, if the average activation value is greater than a preset threshold, using the local class token feature as a positive local class token feature, if the average activation value is less than or equal to the preset threshold, using the local class token feature as a negative local class token feature;

A training module is used to train the attention encoder with minimizing the overall loss as the optimization goal, wherein the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned weakly supervised semantic segmentation method based on attention mask is implemented.

According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned weakly supervised semantic segmentation method based on attention mask when executing the computer program.

At least one of the above technical solutions adopted in the present disclosure can achieve the following beneficial effects:

It can be seen from the above-mentioned weakly supervised semantic segmentation method based on attention mask that the method can obtain a sample image and an image classification label corresponding to the sample image, input the sample image into the attention encoder, obtain the patch token feature and the global category token feature through the attention encoder, and generate the image classification result and the class activation map of the sample image through the patch token feature, obtain the first semantic segmentation result of the sample image through the class activation map, and decode the patch token feature to obtain the second semantic segmentation result. Generate an attention matrix through the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix through the target mask matrix to obtain a compensated attention matrix, and generate a local category token feature based on the compensated attention matrix. According to the auxiliary class activation map and the target mask matrix, determine the average activation value of the part of the activation value in the class activation map that is not affected by the mask in the target mask matrix, if the average activation value is greater than the preset threshold, the local category token feature is used as a positive local category token feature, if the average activation value is less than or equal to the preset threshold, the local category token feature is used as a negative local category token feature. The attention encoder is trained with the optimization goal of minimizing the overall loss, which includes the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature. The difference between , and the difference between the negative local category token features and the global category token features, are used to perform image semantic segmentation on the image to be recognized through the trained attention encoder.

From the above content, it can be seen that this method can be divided into two extraction methods when extracting features through the attention encoder. The first method does not use a mask mechanism and directly obtains the image classification result, as well as two semantic segmentation results obtained through the class activation map and patch token features respectively. The second method uses a mask mechanism to supplement the attention mechanism through a mask matrix to obtain local category token features. Then, the idea of contrastive learning can be combined to distinguish the positive and negative local category token features, so as to participate in model training. Therefore, model training not only includes classification loss and segmentation loss, but also includes contrastive learning loss related to local category token features. By introducing multiple losses, the loss of image semantic segmentation can be supervised more accurately, thereby improving the accuracy of semantic segmentation.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

FIG1 is a flow chart of a weakly supervised semantic segmentation method based on attention mask provided by an embodiment of the present disclosure;

FIG2 is a schematic diagram of a masking strategy provided by an embodiment of the present disclosure;

FIG3 is a schematic diagram of generating a target mask matrix provided by an embodiment of the present disclosure;

FIG4 is a schematic diagram of a complete process of model training provided by an embodiment of the present disclosure;

FIG5 is a schematic diagram of a weakly supervised semantic segmentation device based on attention mask provided by an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of an electronic device corresponding to FIG. 1 provided in an embodiment of the present disclosure.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in combination with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

The present disclosure provides a training method for a weakly supervised semantic segmentation model based on an attention mask, and the execution process of the training method of the model can be executed by an electronic device such as a server. The present disclosure takes the server executing the training method of the model as an example for explanation.

FIG1 is a flowchart of a weakly supervised semantic segmentation method based on attention mask provided by an embodiment of the present disclosure. The figure specifically includes the following steps.

S100: Obtain a sample image and an image classification label corresponding to the sample image.

S102: Input the sample image into the attention encoder.

S104: obtaining patch token features and global category token features through the attention encoder, and generating an image classification result and a class activation map for the sample image through the patch token features, obtaining a first semantic segmentation result for the sample image through the class activation map, and decoding the patch token features to obtain a second semantic segmentation result.

S106: Generate an attention matrix through the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix through the target mask matrix to obtain a compensated attention matrix, and generate each local category token feature according to the compensated attention matrix.

In weakly supervised semantic segmentation, the label corresponding to the training sample refers to the category of the target object contained in the image, rather than marking the target object category of each pixel in the image as in conventional semantic segmentation.

Based on this, the server can obtain a sample image and an image classification label corresponding to the sample image, where the image classification label is used to indicate a category corresponding to a target object in the image.

The sample image can be input into the attention encoder, through which the patch token feature and the global class token feature are obtained, and the image classification result and the class activation map of the sample image are generated through the patch token feature, and the first semantic segmentation result of the sample image is obtained through the class activation map, and the patch token feature is decoded to obtain the second semantic segmentation result. The attention encoder can be an attention encoder in a Transformer model, and the attention encoder in the Transformer model uses a self-attention mechanism, including multiple self-attention layers.

In addition, the sample image can be input into the attention encoder to generate an attention matrix and a randomly generated target mask matrix, and the attention matrix is compensated by the target mask matrix to obtain a compensated attention matrix, and local category token features are generated according to the compensated attention matrix.

The difference between the above two processes is that one is to input the sample image into the attention encoder without using the target mask matrix (which can be called the first process), and the other is to input the sample image into the attention encoder and use the target mask matrix (which can be called the second process). The role of the target mask matrix is to use the mask in the attention mechanism to focus on extracting local information in the image. This is because the attention mechanism in the Transformer model has the sparse characteristics. The attention encoder will extract the associations between each feature, but this extraction may extract unnecessary associations. Therefore, this method can cover some of the associations through the mask (target mask matrix) and obtain the final category token features without considering these associations.

There are different ways to input the sample image into the attention encoder. For example, it can be input in batches. For example, the sample image is first input into the attention encoder that does not use the target mask matrix, and then the sample image is input into the attention encoder that uses the target mask matrix. For another example, different branches of the attention encoder can be set, one branch is the first sub-encoder that does not include the target mask matrix, and the other branch is the second sub-encoder that includes the target mask matrix, and the sample image can be input into the first sub-encoder and the second sub-encoder at the same time.

It should be noted that in some examples, a target mask matrix can be generated once, and the sample image can be input into the attention encoder. The attention matrix is compensated by the target mask matrix generated this time, thereby obtaining a local category token feature.

In some examples, the target mask matrix can be randomly generated multiple times, each time the generated target mask matrix is different, the sample image can be input into the attention encoder multiple times, each time the attention matrix is compensated by a target mask matrix, and multiple local category token features can be obtained. For example, multiple target mask matrices can be randomly generated, the sample image can be input into the attention encoder to obtain the attention matrix, and for each target mask matrix, the attention matrix is compensated by the target mask matrix to obtain the corresponding compensated attention matrix, and the local category token feature corresponding to the target mask matrix is generated according to the compensated attention matrix. The local category token features finally extracted have multiple local category tokens.

The above two processes are described in detail below.

It should be noted that the sample image can be divided into several sub-images to determine multiple initial patch token features corresponding to the several sub-images, and to determine the initial category token features corresponding to the sample image (the initial category token features can be randomly initialized), and the multiple initial patch token features and the initial category token features are input into the attention encoder (the above two processes are the same when the sample image is input into the attention encoder). In some examples, the multiple initial patch token features and the initial category token features are input into the attention encoder, which can be a concatenation of multiple initial patch token features of dimension N*D and initial category token features of dimension 1*D along the first dimension, thereby obtaining a token of dimension (N+1)*D, and inputting it into the attention encoder, where D represents the dimension of token embedding.

The attention encoder can perform attention weighting on the initial patch token features and the initial category token features through the attention mechanism, so as to obtain weighted patch token features and category token features. The patch token features and global category token features obtained in the first process of the above two processes are weighted. In the second process, there is no need to finally obtain the patch token features (patch token features exist in the intermediate process). Therefore, only the features of each local category token are obtained, and the features of each local category token are weighted.

For example, For the input image, the visual Transformer encoder can split I into N = W′×H′ non-overlapping patches, which can be represented as (i.e., sub-image), where P is the patch size, H is the height of the input image, and W is the width of the input image. Then the block (patch) I' is tiled and projected to obtain N patch tokens (i.e., Patch Token), which can be expressed as Where D represents the dimension of token embedding. Token embeddings are concatenated with the associated learnable class token (i.e., Class Token) and input into the standard Transformer encoder. In some examples, multiple patch tokens are concatenated with one associated learnable class token. In addition, positional embeddings (i.e., Positional Embedding) need to be added on top of the patch token. The patch tokens mentioned here are the initial patch token features, and the learnable class tokens are the initial class token features.

The attention encoder can contain multiple self-attention encoding layers, i.e., Transformer layers.

Specifically, in the Transformer layer l (in the Transformer model, layer l can be a self-attention encoding layer), a linear learnable transformation is first applied to map the token sequence to the query matrix Key Matrix Sum Matrix Here, the token sequence refers to the sequence mentioned above after adding the position embedding on top of the patch token and concatenating the patch token with the category token. In some examples, this token sequence can be mapped through a simple linear layer.

Here, _Dk represents the dimensions of Q and K, and _Dv is the dimension of V. This is done by computing the product of the query matrix and the key matrix and then dividing by To implement the self-attention mechanism. This process obtains a continuous attention matrix Contains the global relationship between pairs in each layer. The output matrix Z ^(l) is generated as a weighted mixture of V ^(l) , with the weights coming from the attention matrix after the softmax operation (normalization operation) The above relationship can be expressed by the following formulas (1) and (2):

Z ^(l) = A′ ^(l) V ^(l) (2)

Assume that the patch token output sequence of each Transformer layer is reshaped into a feature map Where D is the feature dimension and H′×W′ is the spatial dimension. Then applying pooling and convolution layers can generate image classification results (classification logits). At the same time, through the feature map and the parameters of the corresponding classifier The matrix multiplication between them can generate the class activation map of the lth layer It can be expressed by the following formula (3):
F ^(l) = W ^(l)` Z ^(l) (3)

It should be noted that the above process is the attention weighting process of one layer in the attention encoder.

The class activation map obtained from the last layer of the Transformer encoder can be passed through the pixel refinement module to generate a pseudo segmentation label (the first semantic segmentation result). In some examples, if the Transformer encoder has a total of l layers, then the class activation map obtained here is the class activation map output by the lth layer of the Transformer encoder. Pixel Refinement The module can improve the class activation map based on the pixel information in the input image and the spatial information of the neighborhood, thereby ensuring that adjacent pixels of similar image appearance have the same semantics. The patch tokens output by the last layer in the Transformer encoder are reshaped into feature maps, which can be passed into the decoder to generate predicted segmentation labels (second semantic segmentation results). The decoder can be composed of two 3×3 convolutional layers (with an expansion rate of 5) and a 1×1 linear layer.

The above process is a process in which a mask is not added in the attention mechanism. The process of adding a mask in the attention mechanism is described below, as shown in Figures 2 and 3. Figure 2 is a schematic diagram of a mask strategy provided by an embodiment of the present disclosure. Figure 3 is a schematic diagram of generating a target mask matrix provided by an embodiment of the present disclosure.

In Transformer layer l, an attention mask M ^(l) ∈ {0,1} ^N×N (target mask matrix for layer l) is introduced as the switch of the attention matrix, thus obtaining a regularized attention matrix The output matrix Z ^(l) is generated as a weighted mixture of V ^(l) , with the weights coming from the regularized attention matrix after the softmax operation (normalization operation). The output matrix Z ^(l) now includes the local category tokens. The above association can be expressed by the following formulas (4) and (5):

The details of the masking strategy are shown in Figures 2 and 3. Specifically, masking is achieved by randomly dropping columns in the attention matrix. Dropping columns in the attention matrix can also be said to be dropping keys, where the key refers to the key matrix Note that the keys to be masked are independently extracted from the Bernoulli distribution, and the mask ratio p∈(0,1), that is, the sampling of the elements in the key follows the Bernoulli distribution. The Bernoulli distribution here is the binomial distribution in mathematics. The mask can be obtained by independently extracting the keys to be masked at different downsampling resolutions and then upsampling by the same multiple. Downsampling is also called decimation. For a sample value sequence, a few sample values are sampled once, so that the new sequence obtained is the downsampling of the original sequence. Upsampling is the inverse process of downsampling, also known as upsampling or interpolation. In this embodiment, the size of the attention matrix can be determined first, and the preset multiples of downsampling are performed according to the size to obtain the sampling size, and the initial mask matrix of the sampling size is randomly generated, and the initial mask matrix is upsampled by a preset multiple to obtain the target mask matrix. This strategy can form a continuous area for the keys not covered by the mask at different resolutions. At this time, the mask generated is in the token dimension, and then expanded to the shape of the attention matrix to form M∈{0,1} ^N×N .

S108: According to the auxiliary class activation map and the target mask matrix, determine the average activation value of some activation values in the class activation map that are not affected by the mask in the target mask matrix, if the average activation value is greater than a preset threshold, use the local class token feature as a positive local class token feature, if the average activation value is less than or equal to the preset threshold Set a threshold and use the local category token feature as a negative local category token feature.

S110: The attention encoder is trained with minimizing the overall loss as the optimization goal, and the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

In some examples, the overall loss is the sum of the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local class token feature and the global class token feature, and the difference between the negative local class token feature and the global class token feature.

In some examples, the trained attention encoder can further perform image semantic segmentation on the image to be recognized.

After the local category token features are determined, they can be trained with the local category token features and the global category token features in a manner similar to contrastive learning, i.e., the loss for training the attention encoder includes the loss associated with the local category token features.

Specifically, the average activation value of some activation values in the class activation map that are not affected by the mask in the target mask matrix can be determined based on the auxiliary class activation map and the target mask matrix. If the average activation value is greater than a preset threshold, the local class token feature can be used as a positive local class token feature, otherwise, the local class token feature is used as a negative local class token feature. In some examples, the mask generated by the auxiliary class activation map and the mask generation strategy is used to calculate the average activation value of the area of the obtained class activation map that is not masked by the mask, and determine the positive/negative label. The purpose of the attention mask is to cut off the connection between the areas generated by the tokens masked by the mask when performing the attention operation.

Based on this, the loss associated with the local category token feature can be: the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature. Among them, the smaller the difference between the positive local category token feature and the global category token feature, the better, and the larger the difference between the negative local category token feature and the global category token feature, the better.

The following first introduces other losses of model training, and then explains how to determine positive local category token features and negative local category token features. The complete process of model training provided by this method can be shown in Figure 4. Figure 4 is a schematic diagram of a complete process of model training provided by an embodiment of the present disclosure.

As can be seen from Figure 4, the losses included in model training also include classification loss and segmentation loss. Furthermore, it can also include auxiliary classification loss and affinity loss.

The classification loss is the difference between the image classification result and the image classification label, which can be expressed as follows: (6) is calculated. The patch token features can be input into the pooling and classification layers to calculate the class probability vector p _cls , and then the multi-label soft margin loss is used as the classification function. The calculation method of the classification loss L _cls is expressed as follows:

Where C is the total number of categories and y is the true label at the image level. Similarly, the auxiliary classification loss can be calculated using the above formula.

The auxiliary classification loss can be determined by the output result of the intermediate layer of the model. Specifically, the output result of the target self-attention coding layer in the self-attention coding layer of the attention encoder can be obtained. For example, the intermediate patch token features output by a certain intermediate self-attention coding layer (e.g., when the attention encoder has 12 self-attention coding layers, the target self-attention coding layer can be the 10th layer) can be obtained. Then, the auxiliary classification result is determined by the output result, and then the attention encoder is trained with the optimization goal of minimizing the difference between the auxiliary classification result and the image classification label. That is, the auxiliary classification loss is the difference between the auxiliary classification result and the image classification label.

The segmentation loss is the difference between the first semantic segmentation result and the second semantic segmentation result, which can be calculated by cross entropy.

When determining the affinity loss, the output result of the target self-attention encoding layer can be obtained, and the first affinity matrix can be determined according to the foreground background segmentation result determined by the output result, and the second affinity matrix can be determined according to the patch token features. Finally, the attention encoder is trained with the optimization goal of minimizing the difference between the first affinity matrix and the second affinity matrix.

That is, the first affinity matrix is obtained through the middle self-attention encoding layer, and the auxiliary class activation map can be determined through the middle self-attention encoding layer, and the foreground background segmentation result can be obtained through the auxiliary class activation map, and then the first affinity matrix is determined. The second affinity matrix is obtained directly through the attention encoder, and the patch token features obtained by the last layer of the self-attention encoding layer can be used as the second affinity matrix. The auxiliary class activation map can be processed into a pseudo affinity label as supervision for affinity learning.

It is difficult to accurately distinguish the foreground and background from the activation values of the class activation map, because pixels with medium confidence in the class activation map are not suitable for marking as annotated objects or background. Therefore, in order to generate reliable affinity labels, two thresholds _βfg and _βbg are introduced to satisfy 0＜ _βbg ＜ _βfg ＜1, and the auxiliary class activation map is divided into foreground, background and uncertain areas.

Reliable segmentation labels As follows, it can be expressed by formula (7):

The background class is marked as 0 and the uncertain region is marked as 255. Then the first affinity matrix is constructed based on this segmentation label. Specifically, any two pixels in the reliable segmentation label Y’ are selected to form a pixel pair. If the pixel pairs sampled from the segmentation label have the same semantics (for example, the pixels in the pixel pair are all from the foreground or all from the background), the affinity is determined to be positive, otherwise, their affinity is considered negative. When the pixel is sampled from the uncertain region, the affinity will be ignored.

The first affinity matrix determined above is then used as supervision to facilitate the representation of patch tokens from the last layer of the Transformer encoder. In addition, the cosine similarity can be used to measure the predicted affinity between two final patch tokens. Therefore, the affinity loss L _aff can be calculated by the following formula (8):

where T ^(L) = Γ ^{D×H′×W′→D×N} (Z ^(L) ), Y = Γ ^{H′×W′→N} (Y′). Γ(·) is the reshape operator, cos(·,·) represents the cosine function, and N ⁺ /N ^- counts the number of positive/negative samples. This objective function directly encourages the final patch token features with positive relationships to be more similar, otherwise more discriminative. According to the chain rule, it also helps to learn token representations from early Transformer layers.

After introducing the above segmentation label Y′, it is explained how to determine the positive local category token features and the negative local category token features. The method of determining Y′ _t is similar to that of the segmentation label Y ^′ , but not exactly the same.

Specifically, let M _t = Γ ^{N → H′×W′} (M _i: ) be the key mask derived from the corresponding attention mask, where i is an arbitrary row index and Γ(·) represents the reshape operator. Here we use 1 to represent the masked tokens and 0 to represent the unmasked tokens. Intuitively, if the average activation value of the remaining tokens is high, they may belong to the semantic object. Therefore, we distinguish between positivity and negativity by the following formula (9).

Where II(·) is the indicator function, ⊙ represents the Hadamard product, Y _t ′ is the discretized token-level label derived from the activation value of the auxiliary class activation map, μ is associated with the threshold for judging positivity and negativity, and the following formula (10) gives the calculation method of Y′ _t , given two thresholds β _bg and β _fg .

The InfoNCE loss function can be used for contrastive learning (i.e., the InfoNCE loss function can be used to calculate The global and local category token features are linearly projected into a representation space suitable for contrastive learning by global and local projectors, respectively. The projector consists of three linear layers and an L2 normalization layer. Let q be the projected global category token feature, and k ⁺ /k ^- be the projected positive/negative local category token feature. The training goal is to minimize/maximize the distance between the global category token feature and the positive/negative local category token feature. The contrast loss L _mcc can be expressed by the following formula (11):

Where N ⁺ is the number of k ⁺ samples, τ is the temperature factor, and ∈ is a small value to ensure numerical stability. The parameters of the global projector are updated using a moving average strategy, i.e., _θg ← _mθg +(1-m) _θl , where m is the momentum factor, _θg and _θl are the parameters of the global projector and the local projector, respectively. This slowly evolving update of the parameters of the two projectors ensures training stability and enforces representation consistency. The global projector and the local projector can be two fully connected layers.

It should be noted that the above explanation is basically for the model training process. After the training is completed, the attention encoder can be used to perform semantic segmentation on images with unknown semantic segmentation results. Specifically, the image to be identified can be input into the trained attention encoder to obtain the patch token features of the image to be identified, and then the semantic segmentation result can be obtained through the decoder.

From the above, it can be seen that the overall loss can be the sum of classification loss, segmentation loss, loss associated with local category token features, auxiliary classification loss and affinity loss. Taking minimizing the overall loss as the optimization goal, the attention encoder can be trained with the minimum sum of classification loss, segmentation loss, loss associated with local category token features, auxiliary classification loss and affinity loss.

From the above content, it can be seen that this method can be divided into two extraction methods when extracting features through the attention encoder. The first method does not use a mask mechanism and directly obtains the image classification result, as well as two semantic segmentation results obtained through the class activation map and the patch token feature respectively. The second method uses a mask mechanism to supplement the attention mechanism through a mask matrix to obtain local category token features. Then, the local category token features can be distinguished between positive and negative based on the idea of contrastive learning, so as to participate in model training. Therefore, model training not only includes classification loss and segmentation loss, but also includes contrastive learning loss related to local category token features. Furthermore, model training can also include auxiliary classification loss and affinity loss. By introducing multiple losses (introducing losses obtained through intermediate outputs of multiple attention encoders or various types of outputs), the loss of image semantic segmentation can be supervised more accurately, thereby improving the accuracy of semantic segmentation.

It should be noted that, for the sake of ease of description, the execution subject of this method is described as a server. The execution subject can be a desktop computer, a server, a large service platform, etc., which is not limited here.

The above provides a weakly supervised semantic segmentation method based on attention mask for one or more embodiments of the present disclosure. Based on the same idea, the present disclosure also provides a weakly supervised semantic segmentation device based on attention mask, as shown in FIG5 .

FIG5 is a schematic diagram of a weakly supervised semantic segmentation device based on attention mask provided by an embodiment of the present disclosure, including:

An acquisition module 501 is used to acquire a sample image and an image classification label corresponding to the sample image;

An input module 502 is used to input the sample image into an attention encoder, obtain patch token features and global category token features through the attention encoder, generate an image classification result and a class activation map for the sample image through the patch token features, obtain a first semantic segmentation result for the sample image through the class activation map, and decode the patch token features to obtain a second semantic segmentation result;

A mask compensation module 503 is used to generate an attention matrix through the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix through the target mask matrix to obtain a compensated attention matrix, and generate each local category token feature according to the compensated attention matrix;

A judgment module 504 is used to determine, for each local class token feature, an average activation value of some activation values in the class activation map that are not affected by the mask in the target mask matrix according to the auxiliary class activation map and the target mask matrix, and if the average activation value is greater than a preset threshold, the local class token feature is used as a positive local class token feature; if the average activation value is less than or equal to the preset threshold, the local class token feature is used as a negative local class token feature;

A training module 505 is used to train the attention encoder with minimizing the overall loss as the optimization goal, and the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature.

Optionally, the input module 502 is specifically used to divide the sample image into several sub-images to determine the initial patch token features corresponding to the several sub-images and multiple initial category token features corresponding to the sample image; splice the multiple initial patch token features with the initial category token features, and input them into the attention encoder.

Optionally, the mask compensation module 503 is specifically used to determine the size of the attention matrix; downsample the size by a preset multiple to obtain a sampling size, and randomly generate an initial mask matrix of the sampling size; upsample the initial mask matrix by the preset multiple to obtain the target mask matrix.

Optionally, the attention matrix includes: a query matrix, a key matrix, and a value matrix;

The mask compensation module 503 is specifically used to determine the compensated attention matrix by the following formula:


Z ^(l) = A′ ^(l) V ^(l)

Among them, A' is the compensated attention matrix, M is the target mask matrix, Q is the query matrix, K is the key matrix, V is the value matrix, Z is the output matrix, and l is the lth self-attention encoding layer of the attention encoder.

Optionally, the attention encoder comprises a plurality of self-attention encoding layers;

The training module 505 is also used to obtain the output result of the target self-attention coding layer among the multiple self-attention coding layers.

Optionally, the output result of the target self-attention encoding layer includes intermediate patch token features, and the training module 505 is also used to determine the auxiliary classification result through the intermediate patch token features; the attention encoder is trained with minimizing the overall loss as the optimization goal, wherein the overall loss also includes: the difference between the auxiliary classification result and the image classification label.

Optionally, the attention encoder comprises a plurality of self-attention encoding layers;

Optionally, the output result of the target self-attention encoding layer includes the auxiliary class activation map, and the training module 505 is also used to determine a first affinity matrix based on the foreground-background segmentation result determined by the auxiliary class activation map; determine a second affinity matrix based on the patch token features; and train the attention encoder with minimizing the overall loss as the optimization goal, wherein the overall loss also includes: the difference between the first affinity matrix and the second affinity matrix.

Optionally, the mask compensation module 503 is specifically used to randomly generate multiple target mask matrices; for each target mask matrix, the sample image is input into the attention encoder to generate an attention matrix corresponding to the sample image, the attention matrix generated this time is compensated by the target mask matrix to obtain a corresponding compensated attention matrix, and local category token features corresponding to the target mask matrix are generated according to the corresponding compensated attention matrix; and multiple local category token features are used as the local category token features.

Optionally, the training module 505 is further used to perform image semantic segmentation on the image to be recognized using the trained attention encoder.

The present disclosure also provides a computer-readable storage medium storing a computer program, which can be used to execute the above-mentioned weakly supervised semantic segmentation method based on attention mask.

The present disclosure also provides a schematic structural diagram of an electronic device as shown in FIG6. As shown in FIG6, at the hardware level, The electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement a weakly supervised semantic segmentation method based on an attention mask.

Of course, in addition to software implementation, the present disclosure does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many of the improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and produce dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing programs. The original code before compilation must also be written in a specific programming language, which is called Hardware Description Language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that realizes the logic method flow.

The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code, it is entirely possible to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present disclosure, the functions of each unit can be implemented in the same or multiple software and/or hardware.

Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

The present disclosure is described with reference to the flowchart and/or block diagram of the method, device (system), and computer program product according to the embodiment of the present disclosure. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the process and/or box in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions can also be loaded into a computer or other programmable data processing device so that A series of operational steps are executed on a computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of the flowchart and/or one or more blocks of the block diagram.

In a typical configuration, a computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

It will be appreciated by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

The present disclosure may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are connected through a communications network. In distributed computing environments, the present disclosure may be implemented in a variety of ways, such as by a computer program or a computer program that is not directly connected to the computer program. In a typical environment, program modules may be located in local and remote computer storage media including memory storage devices.

Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims (12)

A weakly supervised semantic segmentation method based on attention mask, characterized by comprising: Obtaining a sample image and an image classification label corresponding to the sample image; Input the sample image into the attention encoder; Obtaining patch token features and global category token features through the attention encoder, and generating an image classification result and a class activation map for the sample image through the patch token features, obtaining a first semantic segmentation result for the sample image through the class activation map, and decoding the patch token features to obtain a second semantic segmentation result; Generate an attention matrix through the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix through the target mask matrix to obtain a compensated attention matrix, and generate a local category token feature according to the compensated attention matrix; According to the auxiliary class activation map and the target mask matrix, determine the average activation value of some activation values in the class activation map that are not affected by the mask in the target mask matrix, if the average activation value is greater than a preset threshold, use the local class token feature as a positive local class token feature, if the average activation value is less than or equal to the preset threshold, use the local class token feature as a negative local class token feature; The attention encoder is trained with minimizing the overall loss as the optimization goal, and the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature. The method according to claim 1, characterized in that the step of inputting the sample image into an attention encoder specifically comprises: Segmenting the sample image into a plurality of sub-images to determine a plurality of initial patch token features corresponding to the plurality of sub-images, and determining an initial category token feature corresponding to the sample image; The multiple initial patch token features are concatenated with the initial category token features and input into the attention encoder. The method according to claim 1 or 2, characterized in that the randomly generating a target mask matrix specifically comprises: determining a size of the attention matrix; Downsampling the size by a preset multiple to obtain a sampling size, and randomly generating an initial mask matrix having a size equal to the sampling size; The initial mask matrix is upsampled by the preset multiple to obtain the target mask matrix. The method according to any one of claims 1 to 3, characterized in that the attention matrix comprises: a query matrix, a key matrix and a value matrix; The compensating the attention matrix by the target mask matrix to obtain a compensated attention matrix specifically includes: The compensated attention matrix is determined by the following formula:


Z ^(l) = A′ ^(l) V ^(l) Among them, A' is the compensated attention matrix, M is the target mask matrix, Q is the query matrix, K is the key matrix, V is the value matrix, Z is the output matrix, and l is the lth self-attention encoding layer of the attention encoder. The method according to any one of claims 1 to 4, characterized in that the attention encoder comprises a plurality of self-attention encoding layers; the method further comprising: Obtain an output result of a target self-attention coding layer among the multiple self-attention coding layers. The method of claim 5, wherein the output of the target self-attention encoding layer comprises intermediate patch token features, and the method further comprises: Determining auxiliary classification results through the intermediate patch token features; The attention encoder is trained with minimizing the overall loss as an optimization goal, wherein the overall loss also includes: the difference between the auxiliary classification result and the image classification label. The method according to claim 5 or 6, characterized in that the output result of the target self-attention encoding layer includes the auxiliary class activation map, and the method further comprises: Determine a first affinity matrix based on the foreground-background segmentation result determined by the auxiliary class activation map; Determining a second affinity matrix according to the patch token characteristics; The attention encoder is trained with minimizing the overall loss as an optimization goal, wherein the overall loss also includes: a difference between the first affinity matrix and the second affinity matrix. The method according to any one of claims 1 to 7, characterized in that the attention matrix is generated by the attention encoder, a target mask matrix is randomly generated, and the attention matrix is compensated by the target mask matrix to obtain a compensated attention matrix, and a local category token feature is generated according to the compensated attention matrix, specifically comprising: Randomly generate multiple target mask matrices; For each target mask matrix, Input the sample image into the attention encoder to generate an attention matrix corresponding to the sample image, The attention matrix generated this time is compensated by the target mask matrix to obtain the corresponding compensated attention matrix. Generate local category token features corresponding to the target mask matrix according to the corresponding compensated attention matrix; A plurality of local category token features are used as the local category token features. The method according to any one of claims 1 to 8, characterized in that the method further comprises: Use the trained attention encoder to perform image semantic segmentation on the image to be recognized. A weakly supervised semantic segmentation device based on attention mask, characterized by comprising: An acquisition module, used to acquire a sample image and an image classification label corresponding to the sample image; An input module is used to input the sample image into an attention encoder, obtain patch token features and global category token features through the attention encoder, generate an image classification result and a class activation map for the sample image through the patch token features, obtain a first semantic segmentation result for the sample image through the class activation map, and decode the patch token features to obtain a second semantic segmentation result; A mask compensation module, used to generate an attention matrix through the attention encoder, randomly generate a target mask matrix, and compensate the attention matrix through the target mask matrix to obtain a compensated attention matrix, and generate each local category token feature according to the compensated attention matrix; A judgment module, used for determining, according to the auxiliary class activation map and the target mask matrix, an average activation value of some activation values in the class activation map that are not affected by the mask in the target mask matrix, if the average activation value is greater than a preset threshold, using the local class token feature as a positive local class token feature, if the average activation value is less than or equal to the preset threshold, using the local class token feature as a negative local class token feature; A training module is used to train the attention encoder with minimizing the overall loss as the optimization goal, wherein the overall loss includes: the difference between the image classification result and the image classification label, the difference between the first semantic segmentation result and the second semantic segmentation result, the difference between the positive local category token feature and the global category token feature, and the difference between the negative local category token feature and the global category token feature. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented. Law. An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 9 when executing the computer program.