CN114399639A - Semantic segmentation model training method, electronic equipment and storage medium - Google Patents

Abstract

The disclosure relates to a semantic segmentation model training method, an electronic device and a storage medium. The semantic segmentation model training method comprises the following steps: inputting the training image into a semantic segmentation model to obtain semantic segmentation features and a semantic segmentation prediction result; constructing a pseudo label of the training image based on a tree relation matrix, wherein the tree relation matrix comprises a first tree relation matrix determined based on the semantic segmentation features and a second tree relation matrix determined based on the training image; determining a first target loss based on the pseudo-label and the semantic segmentation prediction result; training the semantic segmentation model based on the first target loss until the semantic segmentation model converges. The method and the device for training the semantic segmentation model are efficient and fast.

Description

A semantic segmentation model training method, electronic device and storage medium

technical field

The present disclosure relates to the technical field of artificial intelligence, and in particular, to a method for training a semantic segmentation model, an electronic device and a storage medium.

Background technique

Semantic segmentation task, which aims to train a segmentation network to generate dense class labels for images.

In the related art, for semantic segmentation of images, a large amount of finely labeled training data is required, and there is a problem of high labeling cost.

SUMMARY OF THE INVENTION

In order to overcome the problems existing in the related art, the present disclosure provides a semantic segmentation model training method, an electronic device and a storage medium.

According to a first aspect of the embodiments of the present disclosure, a method for training a semantic segmentation model is provided, including:

Input the training image into the semantic segmentation model to obtain the semantic segmentation feature and the semantic segmentation prediction result; construct the pseudo-label of the training image based on the tree relationship matrix, and the tree relationship matrix includes the first tree determined based on the semantic segmentation feature relationship matrix and a second tree relationship matrix determined based on the training image; based on the pseudo-label and the semantic segmentation prediction result, determine a first target loss; based on the first target loss, perform the semantic segmentation model Train until the semantic segmentation model converges.

In one embodiment, the tree relationship matrix is determined in the following manner: constructing a first plane grid graph based on the semantic segmentation feature, constructing a second plane grid graph based on the training image, and the first plane grid graph The graph includes nodes corresponding to each pixel in the semantic segmentation feature, and the second plane grid graph includes nodes corresponding to each pixel in the training image; wherein, the Nodes included in the first plane grid graph are connected by undirected edges, and nodes included in the second plane grid graph are connected by undirected edges; based on each node in the first plane grid graph The similarity between the first planar grid graph is determined, and the first minimum spanning tree of the first planar grid graph is determined, and the second planar grid graph is determined based on the similarity between the nodes in the second planar grid graph. The second minimum spanning tree, the minimum spanning tree includes nodes without loops and undirected edges; based on the nodes and undirected edges included in the first minimum spanning tree, determine the corresponding semantic segmentation feature. distance matrix, and based on the nodes and undirected edges included in the second minimum spanning tree, determine the distance matrix corresponding to the training image, the distance between the nodes in the distance matrix is each undirected edge on the shortest path between the nodes The corresponding cumulative value of node similarity; perform non-negative mapping on the distance matrix corresponding to the semantic segmentation feature to obtain the first tree relationship matrix, and perform non-negative mapping on the distance matrix corresponding to the training image to obtain the The second tree relationship matrix.

In an embodiment, the constructing the pseudo-label of the training image based on the tree relationship matrix includes: using the first tree relationship matrix and the second tree relationship matrix as filter kernel functions respectively, Semantic segmentation prediction results are filtered to obtain pseudo-labels for the training images.

In one embodiment, the filtering of the semantic segmentation prediction result by using the first tree relationship matrix and the second tree relationship matrix as filter kernel functions respectively includes: filtering the semantic segmentation prediction result. , using the second tree relationship matrix as a filter kernel function to perform initial filtering; the semantic segmentation prediction result after the initial filtering is filtered again with the first tree relationship matrix as a filter kernel function.

In one embodiment, the method further includes: adjusting the size of the semantic segmentation feature, so that the resolution corresponding to the adjusted semantic segmentation feature is consistent with the resolution corresponding to the semantic segmentation prediction result; The training image is resized so that the resolution of the adjusted training image is consistent with the resolution corresponding to the semantic segmentation prediction result.

In one embodiment, the first target loss is determined by the following formula: wherein, is the first target loss function, the described is the prediction result of the pixel whose position index is in the semantic segmentation prediction result, and is the position index in the pseudo tag is The label value of , represents the set of unlabeled pixels in the pseudo-label, and is the absolute value of the difference between the semantic segmentation prediction result of the unlabeled pixels and the pseudo-label.

In one embodiment, the method further includes: acquiring the sparse label of the training image, and according to the sparse label, using a cross-entropy loss function to calculate the second target loss of the semantic segmentation prediction result; The two-target loss trains the semantic segmentation model until the semantic segmentation model converges.

According to a second aspect of the embodiments of the present disclosure, an image segmentation method is provided, including:

Input the image to be segmented into the semantic segmentation model, and the semantic segmentation model is pre-trained using any one of the above semantic segmentation model training methods; based on the output result of the semantic segmentation model, the segmentation result of the to-be-segmented image is determined.

According to a third aspect of the embodiments of the present disclosure, there is provided an apparatus for training a semantic segmentation model, including:

The processing unit is used to input the training image into the semantic segmentation model to obtain the semantic segmentation feature and the semantic segmentation prediction result; it is also used to construct the pseudo-label of the training image based on the tree relationship matrix, and the tree relationship matrix includes a first tree relationship matrix determined by the semantic segmentation feature and a second tree relationship matrix determined based on the training image; a determining unit for determining a first target loss based on the pseudo-label and the semantic segmentation prediction result; the The processing unit is further configured to train the semantic segmentation model based on the first target loss until the semantic segmentation model converges.

In one embodiment, the processing unit determines the tree relationship matrix in the following manner: constructing a first plane grid graph based on the semantic segmentation feature, constructing a second plane grid graph based on the training image, and the first plane grid graph. The grid graph includes nodes that correspond to each pixel in the semantic segmentation feature, and the second plane grid graph includes nodes that correspond to each pixel in the training image; wherein, For the nodes included in the first plane grid graph or the nodes included in the second plane grid graph, the nodes are connected by undirected edges; based on each node in the first plane grid graph The similarity between the first planar grid graph is determined, and the first minimum spanning tree of the first planar grid graph is determined, and the second planar grid graph is determined based on the similarity between the nodes in the second planar grid graph. The second minimum spanning tree, the minimum spanning tree includes nodes without loops and undirected edges; based on the nodes and undirected edges included in the first minimum spanning tree, determine the corresponding semantic segmentation feature. distance matrix, and based on the nodes and undirected edges included in the second minimum spanning tree, determine the distance matrix corresponding to the training image, the distance between the nodes in the distance matrix is each undirected edge on the shortest path between the nodes The corresponding cumulative value of node similarity; perform non-negative mapping on the distance matrix corresponding to the semantic segmentation feature to obtain the first tree relationship matrix, and perform non-negative mapping on the distance matrix corresponding to the training image to obtain the The second tree relationship matrix.

In one embodiment, the processing unit constructs the pseudo-label of the training image based on the tree relationship matrix in the following manner: the first tree relationship matrix and the second tree relationship matrix are used as filter kernel functions, respectively, The semantic segmentation prediction result is filtered to obtain the pseudo-label of the training image.

In one embodiment, the processing unit uses the first tree relationship matrix and the second tree relationship matrix as filter kernel functions, respectively, to filter the semantic segmentation prediction result in the following manner: For the segmentation prediction result, initial filtering is performed using the second tree relationship matrix as a filter kernel function; the semantic segmentation prediction result after the initial filtering is filtered again using the first tree relationship matrix as a filter kernel function.

In one embodiment, the processing unit is further configured to: adjust the size of the semantic segmentation feature, so that the resolution corresponding to the adjusted semantic segmentation feature is consistent with the resolution corresponding to the semantic segmentation prediction result; The training image is resized so that the resolution of the adjusted training image is consistent with the resolution corresponding to the semantic segmentation prediction result.

In one embodiment, the first target loss is determined by the following formula: wherein, is the first target loss function, the described is the prediction result of the pixel whose position index is in the semantic segmentation prediction result, and is the position index in the pseudo tag is The label value of , represents the set of unlabeled pixels in the pseudo-label, and is the absolute value of the difference between the semantic segmentation prediction result of the unlabeled pixels and the pseudo-label.

In one embodiment, the processing unit is further configured to: acquire the sparse label of the training image, and use a cross-entropy loss function to calculate the second target loss of the semantic segmentation prediction result according to the sparse label; The second target loss is used to train the semantic segmentation model until the semantic segmentation model converges.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an image segmentation apparatus, including:

The input unit is used to input the image to be segmented into the semantic segmentation model, and the semantic segmentation model is pre-trained by using any one of the above-mentioned semantic segmentation model training methods; the determination unit is used to determine the output result of the semantic segmentation model based on the The segmentation result of the image to be segmented.

According to a fifth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

processor; memory for storing processor-executable instructions;

The processor is configured to: execute the semantic segmentation model training method described in the first aspect or any embodiment of the first aspect, or execute the second aspect or any embodiment of the second aspect image segmentation method.

According to a sixth aspect of the embodiments of the present disclosure, a storage medium is provided, where instructions are stored in the storage medium, and when the instructions in the storage medium are executed by a processor, the processor can execute the first aspect or the first aspect The semantic segmentation model training method described in any one of the embodiments, or the image segmentation method described in the second aspect or any one of the embodiments of the second aspect.

According to a seventh aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, the computer program being configured to implement the semantics described in the first aspect or any one of the implementation manners of the first aspect when executed by a processor A segmentation model training method, or the image segmentation method described in the second aspect or any one of the implementation manners of the second aspect.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: tree relationship matrices can be generated for training images and semantic segmentation features corresponding to the training images, respectively, and then pseudo-labels for training a semantic segmentation model can be constructed through the tree relationship matrix. . Further, the target loss can be calculated through the pseudo-label and the semantic segmentation prediction result, and the semantic segmentation model can be trained with the calculated target loss. This method can realize the rapid training of the semantic segmentation model by constructing the pseudo-label.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

Fig. 1 is a flowchart of a method for training a semantic segmentation model according to an exemplary embodiment.

Fig. 2 is a flowchart of a method for determining a tree relationship matrix according to an exemplary embodiment.

FIG. 3 is a schematic flowchart of constructing a tree relationship matrix according to an exemplary embodiment.

Fig. 4 is a flowchart of a method for constructing pseudo-labels of training images through a tree relationship matrix according to an exemplary embodiment.

Fig. 5 is a flowchart showing another method for training a semantic segmentation model according to an exemplary embodiment.

Fig. 6 is a flowchart of a method for constructing pseudo-labels of training images through a tree relationship matrix according to an exemplary embodiment.

Fig. 7 is a flowchart of a method for constructing pseudo-labels of training images through a tree relationship matrix according to an exemplary embodiment.

Fig. 8 is a schematic flowchart of training a semantic segmentation model according to an exemplary embodiment.

Fig. 9 is a flowchart of an image segmentation method according to an exemplary embodiment.

Fig. 10 is a block diagram of an apparatus for training a semantic segmentation model according to an exemplary embodiment.

Fig. 11 is a block diagram of an image segmentation apparatus according to an exemplary embodiment.

Fig. 12 is a block diagram of an electronic device for image processing according to an exemplary embodiment.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

Throughout the drawings, the same or similar reference numbers refer to the same or similar elements or elements having the same or similar functions. The described embodiments are some, but not all, of the embodiments of the present disclosure. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present disclosure and should not be construed as a limitation of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

The semantic segmentation model training method provided by the embodiments of the present disclosure can be applied to a semantic segmentation scene for images. For example, it can be applied to sparse label-based semantic segmentation scenarios.

Semantic segmentation task, which aims to train a segmentation network to generate dense class labels for images. In the related art, for semantic segmentation of images, a large amount of finely labeled training data is required, and there is a problem of high labeling cost.

In order to reduce the cost of labeling and maintain good segmentation quality, related technologies start to study how to use sparse labels (such as pixel point, brush or pixel block labels) to train segmentation networks. In the related art, semantic segmentation methods based on sparse labels mainly adopt the following methods: (1) construct auxiliary tasks, add additional task branches to detect semantic boundaries, and use the detected information to assist in semantic segmentation; (2) based on multi-stage Iterative training to obtain pseudo-labels, and design algorithms to complement the original sparse labels, thereby increasing the number of supervised samples; (3) Regularize the loss function, using the low-level visual information of the image (such as pixel color, position coordinates) to design a loss function , to constrain the segmentation results.

In the related art, the semantic segmentation method based on sparse labels has the following problems: (1) The prediction error of the auxiliary task branch usually affects the segmentation quality, and the final segmentation result is poor; (2) The pseudo-label generation usually requires multi-stage iterative training. , the time cost required for training is relatively high; (3) The semantic segmentation features are high-level semantic information, and the loss function is designed through low-level visual information. The difference between low-level visual information and high-level semantic information will lead to Semantic segmentation results are poor.

In view of this, the present disclosure provides an efficient and fast semantic segmentation model training method, which can generate a tree relationship matrix for training images and semantic segmentation features corresponding to the training images respectively, and then construct a tree relationship matrix for training a semantic segmentation model. pseudo-label. Since the semantic segmentation model training method provided by the embodiments of the present disclosure does not need to construct and reference auxiliary tasks for the training process of the semantic segmentation model, the converged semantic segmentation model can obtain better semantic segmentation results. Moreover, since the semantic segmentation features are high-level structured information, and the color information in the training images is low-level structured information, the pseudo-labels obtained from the training images and semantic segmentation features are less affected by the difference in information levels. In addition, since the pseudo-labels are generated by the tree relationship matrix, the pseudo-labels have high semantic segmentation accuracy and can provide better reference values for the semantic segmentation model during the training process of the semantic segmentation model. Further, training the semantic segmentation model with the first target loss determined by the pseudo-label and the semantic segmentation prediction result can make the semantic segmentation model have higher semantic segmentation accuracy after convergence.

Hereinafter, for the convenience of description, the tree relationship matrix determined by the semantic segmentation feature is referred to as the first tree relationship matrix, the tree relationship matrix determined by the training image is referred to as the second tree relationship matrix, and the pseudo-label and semantic segmentation prediction will be The resulting loss value is called the first objective loss.

Fig. 1 is a flowchart of a method for training a semantic segmentation model according to an exemplary embodiment. As shown in Fig. 1 , the method includes the following steps.

In step S11, the training image is input to the semantic segmentation model, and the semantic segmentation feature and the semantic segmentation prediction result are obtained.

The semantic segmentation prediction result may be obtained, for example, by classifying semantic segmentation features.

In step S12, a pseudo-label of the training image is constructed based on the tree relationship matrix.

In the embodiment of the present disclosure, the tree relationship matrix includes a first tree relationship matrix determined by semantic segmentation features, and a second tree relationship matrix determined by training images.

Exemplarily, determining the second tree relationship matrix by using the training image can be understood as determining the second tree relationship matrix by using the color information contained in the training image. The color information may be, for example, three primary colors (Red Green Blue, RGB) color space values corresponding to each pixel position in the training image.

In the embodiment of the present disclosure, the first tree relationship matrix may represent the semantic segmentation feature relationship between pixel positions, and the second tree relationship matrix may represent the color information relationship between pixel positions in the training image. In other words, the first tree relationship matrix and the second tree relationship matrix include low-level color information and high-level semantic segmentation features. The pseudo-label of the training image is constructed by the first tree relationship matrix and the second tree relationship matrix, so that the pseudo-label can have higher semantic segmentation accuracy.

In step S13, a first target loss is determined based on the pseudo-label and the semantic segmentation prediction result.

In step S14, based on the first target loss, the semantic segmentation model is trained until the semantic segmentation model converges.

Through the semantic segmentation model training method provided by the embodiments of the present disclosure, a semantic segmentation model with higher segmentation accuracy can be obtained, and then the semantic segmentation task for images can be performed through the converged semantic segmentation model to obtain a semantic segmentation result that meets actual needs.

Exemplarily, the first tree relationship matrix or the second tree relationship matrix may be determined in the following manner. In the present disclosure, for the convenience of description, the plane grid graph constructed by the semantic segmentation feature is referred to as the first plane grid graph, and the plane grid graph constructed from the training image is referred to as the second plane grid graph. The minimum spanning tree obtained from the graph is called the first minimum spanning tree, and the minimum spanning tree obtained from the second plane grid graph is called the second minimum spanning tree.

Fig. 2 is a flowchart of a method for determining a tree relationship matrix according to an exemplary embodiment. As shown in Fig. 2 , the method includes the following steps S21 to S24. In addition, FIG. 3 is a schematic flowchart of constructing a tree relationship matrix according to an exemplary embodiment. For ease of understanding, each step in FIG. 2 will be explained below with reference to FIG. 3 .

In step S21, a first plane grid map is constructed based on the semantic segmentation feature, and a second plane grid map is constructed based on the training image.

In the embodiment of the present disclosure, the first plane grid graph includes nodes that correspond to each pixel in the semantic segmentation feature, and the second plane grid graph includes nodes that correspond to each pixel in the training image. . Illustratively, the plane grid diagram is shown in ① in FIG. 3 (in an example, the plane grid diagram involved in ① is a four-connected plane grid diagram, in addition, the plane grid diagram can also be, for example, an eight-connected plane grid diagram. and other types of plane grid graphs, the present disclosure does not limit the specific types of plane grid graphs), the nodes are shown as n1 to n16 included in ①, and the nodes in the plane grid graph are connected by undirected edges . For example, the nodes included in the first plane grid graph are connected by undirected edges, and the nodes included in the second plane grid graph are connected by undirected edges. Exemplarily, a line segment connected between two nodes (for example, a line segment connecting with n15 and n16) is the undirected edge mentioned above.

In step S22, a first minimum spanning tree of the first plane grid graph is determined based on the similarity between the nodes in the first plane grid graph, and based on the similarity between the nodes in the second plane grid graph degree to determine the second minimum spanning tree of the second planar grid graph.

In the embodiment of the present disclosure, the similarity between nodes can be determined for any two nodes in the plane grid graph, and the undirected edge of the two nodes with the largest connected similarity can be deleted in turn, until the plane grid There are no loops in the graph, and a minimum spanning tree is obtained. Illustratively, the minimum spanning tree determined by the planar grid graph is shown in ② in Figure 3. Among them, there are no loops in the plane grid graph, which can be understood as the fact that there is no quadrilateral composed of four undirected edges in the plane grid graph. For example, for the first plane grid graph, an edge deletion operation may be performed on the first plane grid graph according to the similarity between the nodes in the first plane grid graph, until there is no A quadrilateral consisting of four undirected edges is the first minimum spanning tree. In addition, the manner of determining the second minimum spanning tree is similar to the manner of determining the first minimum spanning tree mentioned above, and details are not described herein.

In step S23, the distance matrix corresponding to the semantic segmentation feature is determined based on the nodes and undirected edges included in the first minimum spanning tree, and based on the nodes and undirected edges included in the second minimum spanning tree, the corresponding distance matrix of the training image is determined distance matrix.

In the embodiment of the present disclosure, the distance between nodes in the distance matrix is the accumulated value of the node similarity corresponding to each undirected edge on the shortest path between the nodes. As an example, as shown in Figure 3, taking the distance between n3 and n7 as an example, the cumulative value of the node similarity corresponding to the multiple undirected edges indicated by the dotted line in ② is the distance between n3 and n7. For each node in the minimum spanning tree, a pair of extraction methods are used as a group to determine a variety of different node combinations corresponding to the extraction methods, and then the distance is determined for each node combination to obtain the final distance matrix.

In step S24, non-negative mapping is performed on the distance matrix corresponding to the semantic segmentation feature to obtain a first tree relationship matrix, and non-negative mapping is performed on the distance matrix corresponding to the training image to obtain a second tree relationship matrix.

Illustratively, for the semantic segmentation feature distance matrix corresponding to the semantic segmentation feature (an example is represented by D ^high ∈ R ^HW*HW ), the first tree relationship can be obtained in the manner of A ^high =exp(-D ^high )∈R ^HW*HW Matrix (examples are denoted by A ^high ). For the distance matrix corresponding to the training image (the example is represented by D ^low ∈ R ^HW*HW ), the second tree relationship matrix can be obtained by A ^low =exp(-D ^low /σ)∈R ^HW*HW (the example is represented by A ^low indicates). Among them, ∈R ^HW*HW represents the dimension of the matrix, and σ is a preset constant. Exemplarily, for the minimum spanning tree shown in ② in FIG. 3 , the corresponding tree relationship matrix is shown in ③ in FIG. 3 .

The semantic segmentation model training method provided by the embodiments of the present disclosure can construct a minimum spanning tree by constructing a plane grid graph, and then calculate the distance between each node through the minimum spanning tree to obtain a final distance matrix. Further, by performing non-negative mapping on the obtained distance matrix, a minimum spanning tree for calculating pseudo-labels can be obtained.

As an example, the first tree relationship matrix (the example is represented by A ^high ) and the second tree relationship matrix (the example is represented by A ^low ) can be used as filter kernel functions respectively, and the semantic segmentation prediction result can be filtered to obtain the training image. pseudo-label.

FIG. 4 is a flowchart of a method for constructing pseudo-labels of training images by using a tree relationship matrix according to an exemplary embodiment. As shown in FIG. 4 , steps S31 , S33 and S34 in the embodiment of the present disclosure are the same as The implementation processes of step S11 , step S13 and step S14 in FIG. 1 are similar, and will not be repeated here.

In step S32, the first tree relationship matrix and the second tree relationship matrix are used as filter kernel functions respectively, and the semantic segmentation prediction result is filtered to obtain the pseudo-label of the training image.

The semantic segmentation model training method provided by the embodiments of the present disclosure can implement filtering of the semantic segmentation prediction result by using the first tree relationship matrix and the second tree relationship matrix as filtering kernel functions respectively. Based on this, pseudo-labels for training the semantic segmentation model can be obtained.

In one embodiment, for the first tree relationship matrix and the second tree relationship matrix, the first tree relationship matrix and the second tree relationship matrix may be used as filter kernel functions respectively, and the semantic segmentation prediction results are cascaded and filtered in sequence. , for example, the following methods 1 and 2 may be included.

Method 1: First, use the first tree relationship matrix as a filter kernel function to perform initial filtering on the semantic segmentation prediction result, and then use the second tree relationship matrix as a filter kernel function to filter the semantic segmentation prediction result after the initial filtering again.

Method 2: First, use the second tree relationship matrix as a filter kernel function to perform initial filtering on the semantic segmentation prediction result, and then use the first tree relationship matrix as a filter kernel function to filter the semantic segmentation prediction result after the initial filtering again.

The present disclosure takes the second mode as an example to describe the process of obtaining a pseudo tag by means of cascaded filtering.

Fig. 5 is a flowchart showing another method for training a semantic segmentation model according to an exemplary embodiment. As shown in Fig. 5 , step S41, step S44 and step S45 in the embodiment of the present disclosure and step S31 in Fig. 4 . The implementation processes of step S33 and step S34 are similar, and will not be repeated here.

In step S42, initial filtering is performed on the semantic segmentation prediction result using the second tree relationship matrix as a filtering kernel function.

表示。其中，

表示初始滤波所使用的滤波函数。Illustratively, for the semantic segmentation prediction result (the example is represented by P) and the second tree relationship matrix (the example is represented by A ^low ), the semantic segmentation prediction result after the initial filtering can be obtained by

express. in,

Indicates the filter function used for the initial filtering.

In step S43, the semantic segmentation prediction result after the initial filtering is filtered again by using the first tree relation matrix as a filtering kernel function to obtain a pseudo-label of the training image.

表示)，以及第一树关系矩阵(示例以Ahigh表示)，再次滤波后得到的结果值可以通过

表示。其中，可以理解的是初始滤波与再次滤波使用了相同的滤波函数，滤波函数可通过

表示。此外，由于再次滤波后得到的结果值即为训练图像的伪标签，因此，可以通过

的方式，得到训练图像的伪标签(示例以

表示)。Illustratively, for the initial filtered semantic segmentation prediction results (examples start with

represented), and the first tree relationship matrix (the example is represented by Ahigh), the result value obtained after filtering again can be passed through

express. Among them, it can be understood that the same filter function is used for the initial filtering and the second filtering, and the filtering function can be passed through

express. In addition, since the result value obtained after filtering again is the pseudo-label of the training image, it can be obtained by

way to get the pseudo-labels of the training images (examples start with

express).

。其中，Ω表示待处理图像中的全体像素集，z_i表示正则化项。示例的，针对上述滤波操作，正则化项z_i可表示为

此外，*表示树关系矩阵的矩阵类型，针对第一树关系矩阵以及第二树关系矩阵，*的范围可以是*∈{low，high}。换言之，所采用的树关系矩阵A^*，包括第一树关系矩阵A^low以及第二树关系矩阵A^high。Illustratively, for the above-mentioned initial filtering or re-filtering, the filtering function can be

. Among them, Ω represents the entire set of pixels in the image to be processed, and _zi represents the regularization term. Illustratively, for the above filtering operation, the regularization term _zi can be expressed as

In addition, * represents the matrix type of the tree relationship matrix, and for the first tree relationship matrix and the second tree relationship matrix, the range of * may be *∈{low,high}. In other words, the adopted tree relationship matrix A ^* includes the first tree relationship matrix A ^low and the second tree relationship matrix A ^high .

的方式，进行滤波结果值的计算，以此得到初始滤波后的语义分割预测结果。此外，再次滤波的计算过程与上述初始滤波的计算过程相似，在此不做赘述。example, via

In this way, the filtering result value is calculated, so as to obtain the semantic segmentation prediction result after the initial filtering. In addition, the calculation process of the re-filtering is similar to the calculation process of the above-mentioned initial filtering, which is not repeated here.

Since the second tree relationship matrix contains low-level structural information (color information), and the first tree relationship matrix contains high-level structural information (semantic segmentation features), the semantic segmentation prediction results are classified in the above manner. The results of the semantic segmentation prediction can be gradually optimized through the joint filtering. Further, the result value after the cascade filtering is used as the pseudo-label of the training image, and then the semantic segmentation model is supervised and trained through the pseudo-label.

In one embodiment, the semantic segmentation features and training images may be resized before constructing the planar grid map.

FIG. 6 is a flowchart of a method for constructing pseudo-labels of training images by using a tree relationship matrix according to an exemplary embodiment. As shown in FIG. 6 , steps S52 , S53 , S54 and S54 in the embodiment of the present disclosure Step S55 is similar to the implementation process of step S21 , step S22 , step S23 and step S24 in FIG. 2 , and is not repeated here.

In step S51, the size of the semantic segmentation feature is adjusted so that the resolution corresponding to the adjusted semantic segmentation feature is consistent with the resolution corresponding to the semantic segmentation prediction result, and the size of the training image is adjusted, so that the adjusted training image The resolution of the image is consistent with the resolution corresponding to the semantic segmentation prediction result.

Exemplarily, the size adjustment method may adopt a conventional method in the related art. For example, size adjustment for semantic segmentation features or training images can be achieved by calculating bilinear differences.

The semantic segmentation model training method provided by the embodiments of the present disclosure can achieve pixel alignment by adjusting the resolution among the semantic segmentation features, training images, and semantic segmentation prediction results to be consistent, thereby reducing the complexity of the subsequent operation process .

的方式确定第一目标损失。其中，

为第一目标损失函数，P_l为语义分割预测结果中位置索引为l的像素的预测结果，

为伪标签中位置索引为l的标签值，Ω_U代表伪标签中未标注像素集，

为未标注像素的语义分割预测结果与伪标签之间的差值绝对值。其中，第一目标损失函数用于计算得到第一目标损失。并且示例的，与伪标签相匹配的，为第一目标损失函数为树能量损失函数(Tree Energy Loss，TEL)。In this embodiment of the present disclosure, the

way to determine the first target loss. in,

is the first objective loss function, P _l is the prediction result of the pixel whose position index is l in the semantic segmentation prediction result,

is the label value with position index l in the pseudo-label, Ω _U represents the unlabeled pixel set in the pseudo-label,

The absolute value of the difference between the semantic segmentation prediction for unlabeled pixels and the pseudo-label. Wherein, the first target loss function is used to calculate the first target loss. And for example, what matches the pseudo-label is that the first objective loss function is a tree energy loss function (Tree Energy Loss, TEL).

In one embodiment, the unlabeled pixel set in the pseudo-label, the semantic segmentation prediction result, and the pseudo-label of the training image may be input into the first target loss function, and the result value corresponding to the output of the first target loss function is the first target loss.

In the embodiment of the present disclosure, while the semantic segmentation model is supervised and trained through pseudo labels, the semantic segmentation model can also be supervised and trained through sparse labels. The following describes the training process of the semantic segmentation model by taking the supervised training of the semantic segmentation model through pseudo-labels and sparse labels as an example. Hereinafter, for the convenience of description in the present disclosure, the loss value obtained by calculating the semantic segmentation prediction result through the cross-entropy loss function is referred to as the second target loss.

FIG. 7 is a flowchart of a method for constructing pseudo-labels of training images by using a tree relationship matrix according to an exemplary embodiment. As shown in FIG. 7 , step S61 in the embodiment of the present disclosure and step S11 in FIG. 1 The implementation process is similar and will not be repeated here.

In step S62a, a pseudo-label of the training image is constructed based on the tree relationship matrix, and a first target loss is determined based on the pseudo-label and the semantic segmentation prediction result.

In step S62b, the sparse labels of the training images are obtained, and according to the sparse labels, a cross-entropy loss function is used to calculate the second target loss of the semantic segmentation prediction result.

In step S63, the semantic segmentation model is trained based on the first target loss, and the semantic segmentation model is trained based on the second target loss until the semantic segmentation model converges.

In the semantic segmentation model training method provided by the embodiments of the present disclosure, the first target loss is calculated by using pseudo labels, and the second loss is calculated by using sparse labels. Further, the semantic segmentation model can be trained through the first target loss and the second target loss, so that the semantic segmentation model converges quickly.

Fig. 8 is a schematic flowchart of training a semantic segmentation model according to an exemplary embodiment. As shown in FIG. 8 , the features of the training images can be extracted by the semantic segmentation model to obtain the semantic segmentation features. Further, for the extracted semantic segmentation features, a first tree relationship matrix can be constructed by constructing a minimum spanning tree. Similarly, for the color information contained in the training image, a second tree relationship matrix can also be constructed by constructing a minimum spanning tree. Further, the second tree relationship matrix may be used as a filter kernel function to perform initial filtering on the semantic segmentation prediction result to obtain a filtered segmentation result. and using the first tree relationship matrix as a filtering kernel function, and filtering the filtered segmentation result again to obtain a pseudo-label of the training image.

Based on this, the first target loss can be calculated by using the tree energy loss function through the obtained pseudo-label and semantic segmentation prediction results. And through the pseudo-label and semantic segmentation prediction results, use the standard cross-entropy loss function to calculate the second target loss. Further, the semantic segmentation model is supervised and trained through the first target loss and the second target loss, so that the semantic segmentation model converges quickly.

Based on the same concept, the embodiments of the present disclosure also provide an image segmentation method.

Fig. 9 is a flowchart of an image segmentation method according to an exemplary embodiment. As shown in Fig. 9 , the method includes the following steps.

In step S71, the image to be segmented is input into the semantic segmentation model.

Wherein, the used semantic segmentation model may be obtained by the semantic segmentation model training method involved in any of the foregoing embodiments. For the training process of the semantic segmentation model, please refer to any of the above embodiments for details, and the present disclosure will not repeat them here.

In step S72, the segmentation result of the image to be segmented is determined based on the output result of the semantic segmentation model.

Due to the semantic segmentation model training method provided by the embodiments of the present disclosure, a semantic segmentation model with higher precision can be obtained. Therefore, by using the semantic segmentation model obtained by the semantic segmentation model training method provided by the embodiments of the present disclosure to perform image segmentation on the image to be segmented, a relatively accurate image segmentation result can be obtained, which meets the actual use requirements for image segmentation.

Based on the same concept, an embodiment of the present disclosure also provides a semantic segmentation model training apparatus.

It can be understood that, in order to implement the above-mentioned functions, the apparatus for training a semantic segmentation model provided by the embodiments of the present disclosure includes hardware structures and/or software modules corresponding to each function. Combining with the units and algorithm steps of each example disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present disclosure.

Fig. 10 is a block diagram of an apparatus 100 for training a semantic segmentation model according to an exemplary embodiment. Referring to FIG. 10 , the apparatus 100 includes a processing unit 101 and a determination unit 102 .

The processing unit 101 is configured to input the training image into the semantic segmentation model to obtain the semantic segmentation feature and the semantic segmentation prediction result. It is also used for constructing pseudo-labels of training images based on the tree relationship matrix, where the tree relationship matrix includes a first tree relationship matrix determined based on the semantic segmentation feature and a second tree relationship matrix determined based on the training image. The determining unit 102 is configured to determine the first target loss based on the pseudo-label and the semantic segmentation prediction result. The processing unit 101 is further configured to train the semantic segmentation model based on the first target loss until the semantic segmentation model converges.

In one embodiment, the processing unit 101 determines the tree relationship matrix in the following manner: constructing a first plane grid graph based on the semantic segmentation feature, constructing a second plane grid graph based on the training image, and the first plane grid graph includes a Nodes corresponding to each pixel in the semantic segmentation feature one-to-one, and the second plane grid graph includes nodes corresponding to each pixel in the training image one-to-one. Wherein, the nodes included in the first plane grid graph are connected by undirected edges, and the nodes included in the second plane grid graph are connected by undirected edges. Determine the first minimum spanning tree of the first plane grid graph based on the similarity between the nodes in the first plane grid graph, and determine the second minimum spanning tree based on the similarity between the nodes in the second plane grid graph The second minimum spanning tree of a flat grid graph, the minimum spanning tree includes nodes without loops and undirected edges. Based on the nodes and undirected edges included in the first minimum spanning tree, determine the distance matrix corresponding to the semantic segmentation feature, and based on the nodes and undirected edges included in the second minimum spanning tree, determine the distance matrix corresponding to the training image, the distance matrix The distance between the middle nodes is the cumulative value of the node similarity corresponding to each undirected edge on the shortest path between the nodes. Perform non-negative mapping on the distance matrix corresponding to the semantic segmentation feature to obtain the first tree relationship matrix, and perform non-negative mapping on the distance matrix corresponding to the training image to obtain the second tree relationship matrix.

In one embodiment, the processing unit 101 constructs the pseudo-label of the training image based on the tree relationship matrix in the following manner: the first tree relationship matrix and the second tree relationship matrix are respectively used as filter kernel functions to filter the semantic segmentation prediction result. , to get the pseudo-labels of the training images.

In one embodiment, the processing unit 101 uses the first tree relationship matrix and the second tree relationship matrix as filter kernel functions, respectively, to filter the semantic segmentation prediction result in the following manner: The matrix is the initial filter for the filter kernel function. The semantic segmentation prediction result after the initial filtering is filtered again with the first tree relation matrix as the filtering kernel function.

In one embodiment, the processing unit 101 is further configured to: adjust the size of the semantic segmentation feature, so that the resolution corresponding to the adjusted semantic segmentation feature is consistent with the resolution corresponding to the semantic segmentation prediction result. The training image is resized so that the resolution of the resized training image is consistent with the resolution corresponding to the semantic segmentation prediction.

In one embodiment, the first target loss is determined by the following formula: where is the first target loss function, is the prediction result of the pixel whose position index is in the semantic segmentation prediction result, is the label value whose position index is in the pseudo-label, Represents the set of unlabeled pixels in the pseudo-label, and is the absolute value of the difference between the semantic segmentation prediction result of the unlabeled pixels and the pseudo-label.

In one embodiment, the processing unit 101 is further configured to: acquire the sparse labels of the training images, and use the cross-entropy loss function to calculate the second target loss of the semantic segmentation prediction result according to the sparse labels. The semantic segmentation model is trained based on the second target loss until the semantic segmentation model converges.

FIG. 11 is a block diagram of an image segmentation apparatus 200 according to an exemplary embodiment. Referring to FIG. 11 , the apparatus 200 includes an input unit 201 and a determination unit 202 .

The input unit is used for inputting the image to be segmented into the semantic segmentation model, and the semantic segmentation model is pre-trained by any one of the above-mentioned semantic segmentation model training methods.

The determining unit is configured to determine the segmentation result of the image to be segmented based on the output result of the semantic segmentation model.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

FIG. 12 is a block diagram of an electronic device 300 for image processing according to an exemplary embodiment.

As shown in FIG. 12 , an embodiment of the present disclosure provides an electronic device 300 . The electronic device 300 includes a memory 301 , a processor 302 , and an input/output (I/O) interface 303 . Among them, the memory 301 is used for storing instructions. The processor 302 is configured to invoke the instructions stored in the memory 301 to execute the method for training a semantic segmentation model according to the embodiment of the present disclosure. Wherein, the processor 302 is respectively connected with the memory 301 and the I/O interface 303, for example, it can be connected through a bus system and/or other forms of connection mechanisms (not shown). The memory 301 can be used to store programs and data, including the programs of the semantic segmentation model training method involved in the embodiments of the present disclosure. The processor 302 executes various functional applications and data processing of the electronic device 300 by running the programs stored in the memory 301 .

In this embodiment of the present disclosure, the processor 302 may use at least one of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). It is implemented in a hardware form, and the processor 302 may be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capability and/or instruction execution capability.

The memory 301 in the embodiments of the present disclosure may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, a random access memory (Random Access Memory, RAM) and/or a cache memory (cache). The non-volatile memory may include, for example, a read only memory (Read Only Memory, ROM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, HDD), or a solid state drive (Solid State Drive, SSD), and the like.

In the embodiment of the present disclosure, the I/O interface 303 can be used to receive input instructions (such as numeric or character information, and generate key signal input related to user settings and function control of the electronic device 300 , etc.), and can also output various external information (for example, images or sounds, etc.). In this embodiment of the present disclosure, the I/O interface 303 may include one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel, etc. indivual.

In some embodiments, the present disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, perform any of the methods described above .

In some embodiments, the present disclosure provides a computer program product comprising a computer program that, when executed by a processor, performs any of the methods described above.

Although operations are depicted in the figures in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown, or in a serial order, or that all operations shown be performed to obtain desirable results . In certain circumstances, multitasking and parallel processing may be advantageous.

The methods and apparatus of the present disclosure can be accomplished using standard programming techniques, using rule-based logic or other logic to implement the various method steps. It should also be noted that the terms "means" and "module" as used herein and in the claims are intended to include implementations using one or more lines of software code and/or hardware implementations and/or means for receiving input.

Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented using a computer program product comprising a computer readable medium containing computer program code executable by a computer processor for performing any or all of the described steps, operations or procedures.

The foregoing descriptions of implementations of the present disclosure have been presented for the purposes of illustration and description. The foregoing description is not intended to be exhaustive nor to limit the disclosure to the precise forms disclosed, and various variations and modifications are possible in light of the above teachings or may be obtained from practice of the disclosure. The embodiments were chosen and described in order to explain the principles of the disclosure and its practical application, to enable others skilled in the art to utilize the disclosure in various embodiments and with various modifications as are suited to the particular use contemplated.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

It should be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship. The singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise.

It is further understood that the terms "first", "second", etc. are used to describe various information, but the information should not be limited to these terms. These terms are only used to distinguish the same type of information from one another, and do not imply a particular order or level of importance. In fact, the expressions "first", "second" etc. are used completely interchangeably. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure.

It can be further understood that, unless otherwise specified, "connection" includes a direct connection between the two without other components, and also includes an indirect connection between the two with other elements.

It is further to be understood that although the operations in the embodiments of the present disclosure are described in a specific order in the drawings, it should not be construed as requiring that the operations be performed in the specific order shown or the serial order, or requiring Perform all operations shown to obtain the desired result. In certain circumstances, multitasking and parallel processing may be advantageous.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended rights.

Claims (11)

1. A semantic segmentation model training method is characterized by comprising the following steps:

inputting the training image into a semantic segmentation model to obtain semantic segmentation features and a semantic segmentation prediction result;

constructing a pseudo label of the training image based on a tree relation matrix, wherein the tree relation matrix comprises a first tree relation matrix determined based on the semantic segmentation features and a second tree relation matrix determined based on the training image;

determining a first target loss based on the pseudo-label and the semantic segmentation prediction result;

training the semantic segmentation model based on the first target loss until the semantic segmentation model converges.

2. The method of claim 1, wherein the tree relationship matrix is determined as follows:

constructing a first plane grid graph based on the semantic segmentation features, and constructing a second plane grid graph based on the training image, wherein the first plane grid graph comprises nodes which are in one-to-one correspondence with all pixel points in the semantic segmentation features, and the second plane grid graph comprises nodes which are in one-to-one correspondence with all pixel points in the training image; the nodes in the first planar grid graph are connected through a non-directional edge, and the nodes in the second planar grid graph are connected through a non-directional edge;

determining a first minimum spanning tree of the first planar grid graph based on the similarity among the nodes in the first planar grid graph, and determining a second minimum spanning tree of the second planar grid graph based on the similarity among the nodes in the second planar grid graph, wherein the minimum spanning tree comprises nodes without loops and non-directional edges;

determining a distance matrix corresponding to the semantic segmentation features based on the nodes and the non-directional edges included in the first minimum spanning tree, and determining a distance matrix corresponding to the training image based on the nodes and the non-directional edges included in the second minimum spanning tree, wherein the distance between the nodes in the distance matrix is a node similarity accumulated value corresponding to each non-directional edge on the shortest path between the nodes;

and carrying out non-negative mapping on the distance matrix corresponding to the semantic segmentation characteristics to obtain the first tree relation matrix, and carrying out non-negative mapping on the distance matrix corresponding to the training image to obtain the second tree relation matrix.

3. The method of claim 1 or 2, wherein constructing the pseudo label of the training image based on the tree relationship matrix comprises:

and respectively taking the first tree relation matrix and the second tree relation matrix as filtering kernel functions, and filtering the semantic segmentation prediction result to obtain the pseudo label of the training image.

4. The method according to claim 3, wherein the filtering the semantic segmentation prediction result by using the first tree relationship matrix and the second tree relationship matrix as filtering kernel functions respectively comprises:

performing initial filtering on the semantic segmentation prediction result by taking the second tree relation matrix as a filtering kernel function;

and filtering the initial filtered semantic segmentation prediction result again by taking the first tree relation matrix as a filtering kernel function.

5. The method of claim 2, further comprising:

performing size adjustment on the semantic segmentation features to enable the resolution corresponding to the adjusted semantic segmentation features to be consistent with the resolution corresponding to the semantic segmentation prediction result;

and adjusting the size of the training image to enable the resolution of the adjusted training image to be consistent with the resolution corresponding to the semantic segmentation prediction result.

6. The method of claim 1, wherein the first target loss is determined by the formula:

wherein,

is a first target loss function, said P_lFor a predictor of a pixel with position index l in the semantic segmentation predictors,

for a tag value, Ω, indexed by position l in the pseudo tag_URepresenting an unmarked set of pixels in the pseudo label,

and (4) segmenting the absolute value of the difference between the prediction result and the pseudo label for the semanteme of the unmarked pixel.

7. The method of claim 1, further comprising:

acquiring a sparse label of the training image, and calculating a second target loss of the semantic segmentation prediction result by using a cross entropy loss function according to the sparse label;

training the semantic segmentation model based on the second target loss until the semantic segmentation model converges.

8. A method of image segmentation, the method comprising:

inputting an image to be segmented into a semantic segmentation model, wherein the semantic segmentation model is obtained by adopting the method of any one of claims 1 to 7 and training in advance;

and determining the segmentation result of the image to be segmented based on the output result of the semantic segmentation model.

9. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: performing the semantic segmentation model training method of any one of claims 1 to 7, or performing the image segmentation method of claim 8.

10. A storage medium having stored therein instructions that, when executed by a processor, enable the processor to perform the semantic segmentation model training method of any one of claims 1 to 7 or the image segmentation method of claim 8.

11. A computer program product comprising a computer program for implementing the semantic segmentation model training method of any one of claims 1 to 7, or for implementing the image segmentation method of claim 8, when executed by a processor.

Images