WO2022141969A1 - Image segmentation method and apparatus, electronic device, storage medium, and program - Google Patents

Abstract

The present disclosure relates to an image segmentation method and apparatus, an electronic device, a computer storage medium, and a computer program. The method comprises: performing feature extraction on an image to be segmented to obtain a first feature of the image to be segmented, the image to be segmented comprising N pixel categories, and N being an integer greater than 1; fusing the first feature with M second features to obtain M first target features, the M second features and the M first target features both having one-to-one correspondence to M pixel categories, the M second features being determined and obtained on the basis of a first sample data set, the first sample data set comprising at least one sample image corresponding to each pixel category among the M pixel categories and annotation information of each sample image, M being greater than or equal to N, and the N pixel categories being a subset of the M pixel categories; and according to the M first target features, performing image segmentation on the image to be segmented to obtain a target segmentation result of the image to be segmented.

Description

Image segmentation method and device, electronic device, storage medium and program

CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure claims a priority titled "image segmentation method and device, electronic device and storage medium" based on a Chinese patent application with application number 202011595659.8 and an application date of December 29, 2020, the entire content of which is hereby incorporated by reference This disclosure is incorporated by reference.

technical field

The present disclosure relates to the field of computer technology, and relates to, but is not limited to, an image segmentation method and apparatus, an electronic device, a computer storage medium, and a computer program.

Background technique

Image semantic segmentation is one of the basic tasks in computer vision, and it has important applications in many fields, such as autonomous driving, scene understanding, etc. The training process of a neural network for image semantic segmentation requires pixel-level labeled samples. It is very difficult to obtain such finely labeled samples, and it is very difficult to obtain samples of many categories. In addition, an image to be segmented may contain multiple pixel categories. Therefore, in practical applications, how to use few samples to achieve multi-class image segmentation is an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present disclosure provides technical solutions for an image segmentation method and apparatus, electronic equipment, computer storage medium and computer program.

An embodiment of the present disclosure provides an image segmentation method, including: performing feature extraction on an image to be segmented to obtain a first feature of the image to be segmented, where the image to be segmented includes N pixel categories, where N is an integer greater than 1 ; The first feature and M second features are fused to obtain M first target features, and the M second features and the M first target features are all in one-to-one correspondence with M pixel categories, The M second features are determined based on a first sample data set, and the first sample data set includes at least one sample image corresponding to each pixel category in the M pixel categories and a sample image corresponding to each sample image. Labeling information, M is greater than or equal to N, and the N pixel categories are subsets of the M pixel categories; perform image segmentation on the to-be-segmented image according to the M first target features to obtain the to-be-segmented image The target segmentation result of the segmented image.

In some embodiments of the present disclosure, the merging the first feature and M second features to obtain M first target features includes: for the i th in the M second features Two features, perform feature multiplication on the first feature and the i-th second feature to obtain the i-th third feature, 1≤i≤M; perform the feature multiplication on the first feature and the i-th second feature Perform feature subtraction on the feature to obtain the i-th fourth feature; perform feature connection on the first feature, the i-th third feature, and the i-th fourth feature to obtain the M first target features The i-th first target feature in the i-th second feature, the i-th third feature, the i-th fourth feature, and the i-th first target feature are all the same The feature corresponding to the i-th pixel category in the M pixel categories.

In some embodiments of the present disclosure, performing image segmentation on the to-be-segmented image according to the M first target features to obtain a target segmentation result of the to-be-segmented image includes: according to the M first target features a target feature, perform category-by-category prediction on the image to be segmented, and determine M segmentation sub-results corresponding to the to-be-segmented image, and the M segmentation sub-results are in one-to-one correspondence with the M pixel categories; according to the There are M segmentation sub-results, and the target segmentation result is determined.

In some embodiments of the present disclosure, performing category-by-category prediction on the to-be-segmented image according to the M first target features, and determining M segmentation sub-results corresponding to the to-be-segmented image, includes: The M first target features are input into a cosine classifier, and based on the cosine classifier and the M first target features, category-by-category prediction is performed on the to-be-segmented image, and the M segmentation sub-results are determined.

In some embodiments of the present disclosure, performing category-by-category prediction on the to-be-segmented image according to the M first target features, and determining M segmentation sub-results corresponding to the to-be-segmented image, includes: The i-th first target feature in the M first target features, according to the i-th first target feature, determine the i-th segmentation sub-result in the M segmentation sub-results corresponding to the image to be segmented As a result, the i-th sub-result of segmentation includes pixels whose pixel category in the to-be-segmented image is the i-th pixel category among the M pixel categories.

In some embodiments of the present disclosure, the image segmentation method is implemented by an image segmentation neural network.

In some embodiments of the present disclosure, the training samples of the image segmentation neural network include a first sample image to be segmented, segmentation annotation information of the first sample image to be segmented, and the first sample data set, where The first sample image to be segmented includes at least two pixel classes among the M pixel classes; the method further includes: performing feature extraction on the first sample image to be segmented through the image segmentation neural network, to obtain The fifth feature of the first to-be-segmented sample image, and the feature extraction of the target sample image corresponding to each pixel category in the M pixel categories through the image segmentation neural network, to obtain M sixth features, the said The M sixth features are in one-to-one correspondence with the M pixel categories, and the target sample image corresponding to each pixel category is any one of the at least one sample image corresponding to each pixel category; according to the M sixth features feature and the annotation information of the target sample image corresponding to each pixel category in the M pixel categories, determine M seventh features, and fuse the fifth feature with the M seventh features to obtain M th Two target features, the M seventh features and the M second target features are in one-to-one correspondence with the M pixel categories; according to the M second target features, the first sample to be segmented is Perform image segmentation on the image to obtain the segmentation result of the first sample image to be segmented; determine the segmentation loss according to the segmentation result of the first sample image to be segmented and the segmentation annotation information; determine the segmentation loss according to the segmentation loss; The image segmentation neural network is trained, and the trained image segmentation neural network is obtained.

In some embodiments of the present disclosure, the labeling information of the target sample image corresponding to each pixel category in the M pixel categories is a mask; Labeling information of the target sample image corresponding to the pixel category, and determining M seventh features, including: for the i-th sixth feature in the M sixth features, according to the i-th sixth feature and the M sixth features The mask of the target sample image corresponding to the ith pixel category in the pixel categories, perform the mask average pooling operation, and obtain the ith seventh feature among the M seventh features, and the ith seventh feature is obtained. Both the six features and the i-th seventh feature are features corresponding to the i-th pixel category in the M pixel categories.

In some embodiments of the present disclosure, the image segmentation neural network is performed on the image segmentation neural network according to the first sample image to be segmented, segmentation annotation information of the first sample image to be segmented, and the first sample data set. Before training, the method further includes: pre-training the image segmentation neural network according to the second sample image to be segmented, the segmentation annotation information of the second sample image to be segmented, and the second sample data set, the The second sample data set includes a plurality of sample images corresponding to each of the P pixel categories and labeling information corresponding to each sample image, the M pixel categories are new pixel categories other than the P pixel categories, and the M pixel categories are new pixel categories. The second sample image to be segmented includes at least two pixel categories among the P pixel categories.

In some embodiments of the present disclosure, the method further includes: determining the M second features according to the first sample data set and the trained image segmentation neural network.

In some embodiments of the present disclosure, the labeling information of the sample images corresponding to each pixel category in the M pixel categories is a mask; the segmentation neural network according to the first sample data set and the trained image network, determining the M second features, comprising: performing feature extraction on the sample images corresponding to each pixel category in the M pixel categories through the trained image segmentation neural network, to obtain M eighth features; The i-th eighth feature in the M eighth features, according to the i-th eighth feature and the mask of the sample image corresponding to the i-th pixel category in the M pixel categories, perform mask average pooling operation to obtain the i-th second feature in the M second features; the i-th eighth feature and the i-th second feature are both the i-th pixel in the M pixel categories feature corresponding to the category.

An embodiment of the present disclosure further provides an image segmentation apparatus, including: a feature extraction module configured to perform feature extraction on an image to be segmented to obtain a first feature of the image to be segmented, where the image to be segmented includes N pixel categories , N is an integer greater than 1; the feature fusion module is configured to fuse the first feature with M second features to obtain M first target features, the M second features and the M second features A target feature is in one-to-one correspondence with M pixel categories, the M second features are determined based on a first sample data set, and the first sample data set includes each pixel in the M pixel categories At least one sample image corresponding to the category and label information corresponding to each sample image, M is greater than or equal to N, and the N pixel categories are subsets of the M pixel categories; the image segmentation module is configured to The first target feature is to perform image segmentation on the to-be-segmented image to obtain a target segmentation result of the to-be-segmented image.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented.

Embodiments of the present disclosure further provide a computer program, including computer-readable codes, when the computer-readable codes are executed in an electronic device, the processor in the electronic device executes to implement the above method.

In the embodiment of the present disclosure, the first feature of the to-be-segmented image is obtained by performing feature extraction on the to-be-segmented image including N pixel categories, and the first feature and the first sample data set based on the M pixel categories are determined to obtain the first feature. The M second features of the , according to the M first target features obtained by merging the first features of the image to be segmented and the M second features, the image to be segmented can be segmented by image segmentation, which can quickly and accurately segment to obtain the target segmentation result of the image to be segmented, thereby realizing the target segmentation result of the image to be segmented. Fast segmentation of multiple pixel classes in images.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the present disclosure.

FIG. 1a is a schematic diagram of an application scenario of an embodiment of the present disclosure;

FIG. 1b is a flowchart of an image segmentation method according to an embodiment of the disclosure;

2 is a schematic diagram of an image segmentation neural network according to an embodiment of the disclosure;

3 is a schematic diagram of a category-by-category prediction according to an embodiment of the present disclosure;

4 is a schematic diagram of training an image segmentation neural network according to an embodiment of the present disclosure;

5 is a schematic diagram of a two-stage training image segmentation neural network according to an embodiment of the present disclosure;

6 is a block diagram of an image segmentation apparatus according to an embodiment of the disclosure;

7 is a block diagram of an electronic device according to an embodiment of the disclosure;

FIG. 8 is a block diagram of another electronic device according to an embodiment of the disclosure.

Detailed ways

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently. B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are set forth in the following detailed description. It will be understood by those skilled in the art that the present disclosure may be practiced without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.

Image semantic segmentation is an important research content in the field of computer vision. Its goal is to segment an image into regions with different semantic information, and label the corresponding semantic labels of each region, for example, by performing image semantic segmentation on an image. Semantic labels can be added to objects in images (eg, cats, tables, chairs, walls, etc.), which can be applied to fields such as unmanned driving, scene understanding, etc. At present, the mainstream method for image semantic segmentation is deep convolutional neural networks (CNNs). With different semantic information, for example, pixel categories include cats, tables, chairs, walls, etc.) for learning, learning the semantic feature representation of each pixel category in the sample image, and then using the learned CNNs to segment any size of input to be segmented Image output pixel-level image segmentation results. However, in practical applications, it is very difficult to obtain sample images with pixel-level annotation information, and it is very difficult to obtain sample images of many pixel categories, that is, in practical applications, most images are segmented into few-sample semantic segmentation scenes. In addition, an image to be segmented may contain multiple pixel categories with different semantic information that need to be segmented. The image segmentation method according to the embodiment of the present disclosure can be applied to a few-sample semantic segmentation scene, and can realize multi-category segmentation of an image to be segmented containing multiple pixel categories with different semantic information.

The present disclosure provides technical solutions for an image segmentation method and apparatus, electronic equipment, computer storage medium and computer program.

The application scenarios of the present disclosure will be described below with reference to the accompanying drawings. FIG. 1a is a schematic diagram of an application scenario of an embodiment of the present disclosure. As shown in FIG. 1a, the image 1 to be divided may be a road condition image, and the image 1 to be divided may be input into the above-mentioned image segmentation device 60, and the image 1 to be divided may be divided After processing, the target segmentation result of the segmented image can be obtained; for example, in the field of autonomous driving, the road condition image corresponding to the road ahead captured by the autonomous vehicle is obtained. The road condition image may include multiple pixel categories such as roads, other vehicles, and pedestrians. The image segmentation method of the embodiment of the present disclosure is used to perform image segmentation on the road condition image to obtain segmentation results of multiple pixel categories, so that the current road conditions can be analyzed according to the segmentation results, so that driving decisions can be made.

The image segmentation method according to the embodiment of the present disclosure will be described in detail below.

FIG. 1b is a flowchart of an image segmentation method according to an embodiment of the present disclosure. The image segmentation method can be performed by electronic equipment such as terminal equipment or server, and the terminal equipment can be user equipment (User Equipment, UE), mobile equipment, user terminal, terminal, cellular phone, cordless phone, Personal Digital Assistant (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the image segmentation method can be implemented by the processor calling the computer-readable instructions stored in the memory. Alternatively, the image segmentation method may be performed by a server. As shown in Figure 1b, the image segmentation method may include:

In step S11, feature extraction is performed on the to-be-segmented image to obtain a first feature of the to-be-segmented image. The to-be-segmented image includes N pixel categories, where N is an integer greater than 1.

In step S12, the first feature and M second features are fused to obtain M first target features, M second features and M first target features are in one-to-one correspondence with M pixel categories, and M The second feature is determined based on the first sample data set. The first sample data set includes at least one sample image corresponding to each pixel category in the M pixel categories and the label information of each sample image, where M is greater than or equal to N, The N pixel classes are a subset of the M pixel classes.

In step S13, image segmentation is performed on the image to be segmented according to the M first target features, and a target segmentation result of the image to be segmented is obtained.

In the embodiment of the present disclosure, the first feature of the to-be-segmented image is obtained by performing feature extraction on the to-be-segmented image including N pixel categories, and the first feature and the first sample data set based on the M pixel categories are determined to obtain the first feature. The M second features of the , according to the M first target features obtained by merging the first features of the image to be segmented and the M second features, the image to be segmented can be segmented by image segmentation, which can quickly and accurately segment to obtain the target segmentation result of the image to be segmented, thereby realizing the target segmentation result of the image to be segmented. Fast segmentation of multiple pixel classes in images.

In some embodiments of the present disclosure, the image segmentation method is implemented by an image segmentation neural network.

Using image segmentation neural network, it is possible to achieve fast segmentation of multiple pixel categories in the image to be segmented.

FIG. 2 is a schematic diagram of an image segmentation neural network according to an embodiment of the present disclosure. As shown in Figure 2, the image segmentation neural network includes a feature extractor, a segmentation module, a class-sensitive reshaping module and a cosine classifier.

As shown in Figure 2, the feature extractor in the image segmentation neural network is used to extract the feature of the image to be segmented, and the ninth feature of the image to be segmented is obtained; after the ninth feature of the image to be segmented passes through the segmentation module in the image segmentation neural network, Obtain the first feature of the image to be segmented. Exemplarily, the ResNet-50 network structure with atrous convolution can be adopted as the feature extractor.

In some embodiments of the present disclosure, the first feature and M second features are fused to obtain M first target features, including: for the i-th second feature in the M second features, for the first Perform feature multiplication with the i-th second feature to obtain the i-th third feature, 1≤i≤M; perform feature subtraction on the first feature and the i-th second feature to obtain the i-th fourth feature; The first feature, the ith third feature, and the ith fourth feature are feature-connected to obtain the ith first target feature among the M first target features; the ith second feature, the ith third feature The feature, the ith fourth feature, and the ith first target feature are all features corresponding to the ith pixel category in the M pixel categories.

In the process of image segmentation of the image to be segmented, M second features corresponding to the M pixel categories are introduced. In order to avoid the noise effect caused by the M second features, a multi-feature aggregation method is adopted to obtain the first feature of the image to be segmented. A feature, the ith third feature obtained by performing feature multiplication between the first feature of the image to be segmented and the ith second feature, and the ith third feature obtained by performing feature subtraction on the first feature of the image to be segmented and the ith second feature The i fourth features are feature-connected, thereby obtaining the i-th first target feature corresponding to the i-th pixel category among the M first target features that can improve the accuracy of segmentation prediction.

Still taking the above Figure 2 as an example, as shown in Figure 2, the category-sensitive reshaping module in the image segmentation neural network performs the above-mentioned multi-feature fusion method, and performs feature multiplication respectively for the first feature of the image to be segmented and the M second features, Obtain M third features, perform feature subtraction on the first features of the image to be segmented and M second features respectively, obtain M fourth features, and perform features on the first features, M third features and M fourth features connected to obtain M first target features.

For example, if the first feature of the image to be segmented is F, and the ith second feature among the M second features is ω _i , the ith feature among the M first target features can be determined by the following formula (1) The first target feature F _i :

In a possible implementation manner, performing image segmentation on the image to be segmented according to the M first target features to obtain a target segmentation result of the image to be segmented, including: performing category-by-category prediction on the image to be segmented according to the M first target features , determine the M segmentation sub-results corresponding to the image to be segmented, and the M segmentation sub-results are in one-to-one correspondence with the M pixel categories; determine the target segmentation result according to the M segmentation sub-results.

Since the M first target features combine the category features of the M pixel categories and the first features of the image to be segmented, the category-by-category prediction of the image to be segmented according to the M first target features can obtain the same number as the M pixel categories. One corresponding M segmentation sub-results, so by combining the M segmentation sub-results, the target segmentation result of the image to be segmented can be obtained.

For example, in the field of autonomous driving, a road condition image corresponding to the road ahead captured by an autonomous vehicle is obtained. The road condition image may include multiple pixel categories such as roads, other vehicles, pedestrians, etc. The pixel category included in the road condition image is M pixels A subset of categories. Using the image segmentation method of the embodiment of the present disclosure, feature extraction is performed on a road condition image to obtain a first feature, and then the first feature and M second features are fused to obtain M first target features, and M first target features are obtained according to the M first target features. The road condition image is predicted by category, and M segmentation sub-results can be obtained (segmentation sub-results corresponding to road pixel categories, segmentation sub-results corresponding to other vehicle pixel categories, segmentation sub-results corresponding to pedestrian pixel categories, etc.), by combining M segmentation sub-results Sub-results are obtained to obtain the segmentation results of the road condition images, and then the current road conditions can be analyzed according to the segmentation results, so as to make driving decisions.

In some embodiments of the present disclosure, performing category-by-category prediction on the image to be segmented according to the M first target features, and determining M segmentation sub-results corresponding to the to-be-segmented image, includes: inputting the M first target features into a cosine classifier , based on the cosine classifier and the M first target features, perform category-by-category prediction on the image to be segmented, and determine M segmentation sub-results.

Since the cosine classifier can reduce the differences within the category and achieve better classification performance, using the cosine classifier and the M first target features to predict the image to be segmented by category can effectively obtain M segmentations corresponding to the image to be segmented Sub-results are obtained, and then the target segmentation results of the images to be segmented are obtained.

Still taking the above FIG. 2 as an example, as shown in FIG. 2 , the M first target features are input into the same cosine classifier in the image segmentation neural network, and then the cosine classifier can output the target segmentation result of the image to be segmented.

In some embodiments of the present disclosure, performing category-by-category prediction on the image to be segmented according to the M first target features, and determining M segmentation sub-results corresponding to the to-be-segmented image, including: for the ith in the M first target features a first target feature, according to the i-th first target feature, determine the i-th segmentation sub-result among the M segmentation sub-results corresponding to the image to be segmented, and the i-th segmentation sub-result includes the pixel category in the image to be segmented. The pixel point of the ith pixel class among the M pixel classes.

Since the ith first target feature is mainly used for image segmentation of the ith pixel category among the M pixel categories, according to the ith first target feature, the pixel category in the image to be segmented can be effectively the ith pixel category Image segmentation is performed on the pixel points of the pixel category to obtain the i-th segmentation sub-result among the M segmentation sub-results corresponding to the image to be segmented.

FIG. 3 is a schematic diagram of a category-by-category prediction according to an embodiment of the present disclosure. As shown in Figure 3, for the ith first target feature in the M first target features, after the ith first target feature passes through the cosine classifier in the image neural network, the cosine classifier can output the ith first target feature. M segmentation results corresponding to a target feature, and each segmentation result includes pixel points whose pixel categories in the image to be segmented are each of the M pixel categories.

For example, when M=3, i=1, there are 3 pixel categories and 3 first target features corresponding to one of them. For the first first target feature (corresponding to 3 first target features) The first pixel category in the pixel category), after the first first target feature passes through the cosine classifier in the image segmentation neural network, the cosine classifier outputs three segmentation results corresponding to the first first target feature: the first The first segmentation result includes pixels whose pixel category of the image to be segmented is the first pixel category; the second segmentation result includes pixels whose pixel category is the second pixel category in the image to be segmented; the third segmentation result includes The pixel category in the image to be segmented is the pixel of the third pixel category. Since the first first target feature is mainly used for image segmentation of the first pixel category, only the first segmentation result among the three segmentation results corresponding to the first first target feature is extracted to determine as The first segmentation sub-result corresponding to the image to be segmented.

For example, when M=3, there are 3 pixel categories and 3 first target features corresponding to one of them, which can be determined according to the first first target feature (corresponding to the first pixel category in the 3 pixel categories) Include the first segmentation sub-result of pixels whose pixel class is the first pixel class in the image to be segmented, and determine the pixel to be segmented according to the second first target feature (corresponding to the second pixel class in the three pixel classes). The pixel category in the image is the second segmentation sub-result of the pixel point of the second pixel category, and according to the third first target feature (corresponding to the third pixel category in the three pixel categories), it is determined to be included in the image to be segmented. The pixel category is the third segmentation sub-result of the pixels of the third pixel category. Finally, according to the first segmentation sub-result, the second segmentation sub-result and the third segmentation sub-result, the target segmentation of the sample image to be segmented is obtained. result.

Before using the image segmentation neural network to quickly segment multiple pixel categories in the image to be segmented, the image segmentation neural network also needs to be trained. The image segmentation neural network is trained, that is, the feature extractor, segmentation module, class-sensitive reshaping module and cosine classifier in the image segmentation neural network are trained.

The training process of the image segmentation neural network is described in detail below.

In some embodiments of the present disclosure, the training samples of the image segmentation neural network include a first sample image to be segmented, segmentation annotation information of the first sample image to be segmented, and a first sample data set, in which the first sample image to be segmented Including at least two pixel categories in the M pixel categories; the image segmentation method further includes: performing feature extraction on the first sample image to be segmented through an image segmentation neural network to obtain the fifth feature of the first sample image to be segmented, and by The image segmentation neural network performs feature extraction on the target sample images corresponding to each pixel category in the M pixel categories, and obtains M sixth features, the M sixth features are in one-to-one correspondence with the M pixel categories, and the target samples corresponding to each pixel category are obtained. The image is any one of at least one sample image corresponding to each pixel category; according to the M sixth features and the label information of the target sample image corresponding to each pixel category in the M pixel categories, determine the M seventh features, and assign the first The five features and the M seventh features are fused to obtain M second target features, and the M seventh features and the M second target features are in one-to-one correspondence with the M pixel categories; Perform image segmentation on the first sample image to be segmented to obtain the segmentation result of the first sample image to be segmented; determine the segmentation loss according to the segmentation result of the first sample image to be segmented and the segmentation annotation information; and perform the image segmentation neural network according to the segmentation loss. Train to get the trained image segmentation neural network.

Utilize the first sample data set including at least one sample image corresponding to each pixel class among the M pixel classes and the labeling information of each sample image, and the first sample image to be segmented including at least two pixel classes among the M pixel classes and the segmentation annotation information of the first sample image to be segmented, the image segmentation neural network is trained to rapidly segment at least two pixel categories, so that the trained image segmentation neural network can quickly and accurately segment to obtain at least two pixel categories including M pixel categories. The target segmentation result of the image to be segmented of the pixel category, so as to realize the fast segmentation of at least two pixel categories.

FIG. 4 is a schematic diagram of training an image segmentation neural network according to an embodiment of the present disclosure. As shown in FIG. 4 , the first sample image to be segmented, the target sample image corresponding to each pixel category in the M pixel categories, and the labeling information of each target sample image are input into the image segmentation neural network. The feature extractor in the image segmentation neural network is shared between the first sample image to be segmented and the target sample image corresponding to each of the M pixel classes.

Use the shared feature extractor to perform feature extraction on the first sample image to be segmented and the target sample image corresponding to each pixel category in the M pixel categories to obtain the tenth feature of the first sample image to be segmented and M sixth features . After the tenth feature of the first sample image to be segmented passes through the segmentation module in the image segmentation neural network, the fifth feature of the first sample image to be segmented is obtained.

In some embodiments of the present disclosure, the labeling information of the target sample image corresponding to each pixel category in the M pixel categories is a mask; according to the M sixth features and the target sample image corresponding to each pixel category in the M pixel categories Labeling information, determining M seventh features, including: for the i-th sixth feature in the M sixth features, according to the i-th sixth feature and the target sample image corresponding to the i-th pixel category in the M pixel categories mask, perform the mask average pooling operation, and obtain the i-th seventh feature in the M seventh features, the i-th sixth feature and the i-th seventh feature are the same as the M pixel categories. Features corresponding to i pixel categories.

Since the M sixth features extracted by the feature extractor are directly fused with the fifth feature of the first sample image to be segmented, the amount of calculation is large, therefore, mask averaging is performed on the M sixth features by using the corresponding mask. The pooling operation obtains M seventh features, so that when the M seventh features are subsequently fused with the fifth feature of the first sample image to be segmented, the amount of computation can be reduced, and M second target features can be obtained quickly.

Still taking the above Figure 4 as an example, as shown in Figure 4, for the i-th sixth feature in the M sixth features, the category-sensitive reshaping module in the image segmentation neural network, according to the i-th sixth feature and M For the mask of the target sample image corresponding to the ith pixel category in the pixel categories, the mask average pooling operation is performed to obtain the ith seventh feature corresponding to the ith pixel category among the M seventh features.

For example, if the target image corresponding to the i-th pixel category is S, the i-th sixth feature is F _S,i , and the mask of the target image corresponding to the i-th pixel category is M _i , the following formula (2 ) determine the i-th seventh feature ω _i corresponding to the i-th pixel category among the M seventh features:

成立时，

的值为1；当

不成立时，

的值为0。 Among them, (x,y) is the pixel position in the feature image corresponding to the ith sixth feature F _S,i ; 1[ ] is an indicator function, when

When established,

is 1; when

When not established,

value of 0.

In some embodiments of the present disclosure, the fifth feature and the M seventh features are fused to obtain M second target features, including: for the i-th seventh feature in the M seventh features, for the fifth Perform feature multiplication with the i-th seventh feature to obtain the i-th eleventh feature, 1≤i≤M; perform feature subtraction on the fifth feature and the i-th seventh feature to obtain the i-th twelfth feature ; Carry out feature connection to the fifth feature, the i-th eleventh feature and the i-th twelfth feature to obtain the i-th second target feature in the M second target features; the i-th seventh feature, the i-th second target feature The i eleventh feature, the i twelfth feature, and the i second target feature are all features corresponding to the i pixel class among the M pixel classes.

Since in a training process, the target sample images corresponding to each pixel category in the M pixel categories are randomly selected, in order to avoid the effect of noise caused by random selection, a multi-feature aggregation method is adopted. Still taking the above Figure 4 as an example, as shown in Figure 4, the category-sensitive reshaping module in the image segmentation neural network performs the above-mentioned multi-feature fusion method, and performs the fifth feature and M seventh features of the first sample image to be segmented. feature multiplication to obtain M eleventh features, and perform feature subtraction on the fifth feature and M seventh features of the first sample image to be segmented to obtain M twelfth features, and perform feature subtraction on the fifth feature and M tenth features. A feature and the M twelfth features are feature-connected to obtain M second target features. The specific manner of the feature connection may be similar to the above formula (1), which will not be repeated here.

Still taking the above Figure 4 as an example, as shown in Figure 4, the M second target features are input into the same cosine classifier in the image segmentation neural network, and the cosine classifier performs category-by-category prediction on the sample images to be segmented, and then outputs the first Segmentation result of segmenting the sample image. The specific segmentation process is similar to the foregoing segmentation process of the image to be segmented, and details are not described here.

Since the training sample includes the segmentation annotation information of the first sample image to be segmented, the segmentation loss of the image segmentation neural network can be determined according to the segmentation result of the first sample image to be segmented and the segmentation annotation information of the first sample image to be segmented, Then, according to the segmentation loss, adjust the network parameters of the image segmentation neural network (adjust the network parameters of the feature extractor, segmentation module, class-sensitive reshaping module, and cosine classifier) to complete the training of the image segmentation neural network. By performing multiple iterations of training, a trained image segmentation neural network that meets the preset requirements is obtained.

In some embodiments of the present disclosure, a cross-entropy loss function may be used to determine the segmentation loss, or other loss functions may be used to determine the segmentation loss, which is not specifically limited in this disclosure.

In some embodiments of the present disclosure, before the image segmentation neural network is trained according to the first sample image to be segmented, the segmentation annotation information of the first sample image to be segmented, and the first sample data set, the image segmentation method further Including: pre-training the image segmentation neural network according to the second sample image to be divided, the segmentation annotation information of the second sample image to be divided, and the second sample data set, and the second sample data set includes each pixel in the P pixel categories The multiple sample images corresponding to the categories and the labeling information of each sample image, the M pixel categories are new pixel categories other than the P pixel categories, and the second sample image to be segmented includes at least two pixel categories in the P pixel categories.

The P pixel categories are the base categories, that is, each pixel category in the P pixel categories corresponds to multiple sample images, and the M pixel categories are new pixel categories other than the P pixel categories, that is, each pixel category in the M pixel categories The corresponding sample images are few, for example, each pixel category corresponds to only 1 sample image (1-shot), or each pixel category corresponds to only 5 sample images (5-shot). In the embodiment of the present disclosure, the sample images corresponding to each pixel category in the M pixel categories may also be extended to 10-shot, or the number of more shots, which is not specifically limited in the present disclosure.

First, the image segmentation neural network is trained in the first stage by using the second image data set corresponding to the P pixel categories, the second sample image to be divided, and the segmentation annotation information of the second sample image to be divided. The image segmentation neural network has the ability to quickly segment multiple pixel categories, and then uses the first image data set corresponding to the M pixel categories, the first sample image to be segmented, and the segmentation annotation information of the first sample image to be segmented to segment the image. The neural network is trained in the second stage, so that the image segmentation neural network trained in the second stage has the ability to quickly segment multiple pixel categories in the M pixel categories as new categories.

The specific training process of the first-stage training of the image segmentation neural network using the second image data set corresponding to the P pixel categories, the second sample image to be segmented, and the segmentation annotation information of the second sample image to be segmented is the same as the above-mentioned use of M The specific training process of the second-stage training of the image segmentation neural network for the first image data set corresponding to each pixel category, the first sample image to be segmented, and the segmentation annotation information of the first sample image to be segmented is similar, and will not be repeated here.

FIG. 5 is a schematic diagram of a two-stage training image segmentation neural network according to an embodiment of the present disclosure. As shown in FIG. 5 , firstly, using the second sample data set corresponding to the P pixel categories as the base category, the second to-be-segmented sample image and the segmentation annotation information of the second to-be-segmented sample image, the image segmentation neural network performs the first stage training; and then use the first sample data set, the first sample image to be segmented and the segmentation annotation information of the first sample image to be segmented corresponding to the M pixel categories as new categories to segment the neural network of the images trained in the first stage. The network is trained in the second stage to obtain the final trained image segmentation neural network.

In some embodiments of the present disclosure, the image segmentation method further includes: determining M second features according to the first sample data set and the trained image segmentation neural network.

The image segmentation neural network after the above two-stage training can determine M second features for subsequent segmentation of the M pixel categories as new categories.

In some embodiments of the present disclosure, determining the M second features according to the first sample data set and the trained image segmentation neural network includes: using the trained image segmentation neural network to segment each pixel in the M pixel categories The sample images corresponding to the categories are subjected to feature extraction to obtain M eighth features; for the i-th eighth feature in the M eighth features, according to the i-th eighth feature and the i-th pixel category in the M pixel categories corresponding to The mask of the sample image, perform the mask average pooling operation, and obtain the ith second feature among the M second features; the ith eighth feature and the ith second feature are both in the M pixel categories The feature corresponding to the ith pixel category of .

For the ith pixel category in the M pixel categories, the feature extractor in the image segmentation neural network after two-stage training is used to extract the feature of the sample image corresponding to the ith pixel category, and the ith pixel category is obtained. feature (corresponding to the i-th pixel category in the M pixel categories), and then use the category-sensitive reshaping module in the image segmentation neural network after the second-stage training, according to the i-th eighth feature and the i-th pixel category corresponds to The mask of the sample image is performed, and the mask average pooling operation is performed to obtain the ith second feature (corresponding to the ith pixel category in the M pixel categories). The specific processing process of the feature extractor and the category-sensitive module is similar to the above-mentioned training process, and will not be repeated here.

In some embodiments of the present disclosure, when each pixel category in the M pixel categories corresponds to only one sample image (1-shot), the last feature extraction and mask average pooling operations are performed only once to obtain the M second features for image segmentation for M pixel categories. When each of the M pixel categories corresponds to multiple sample images, the previous feature extraction and mask average pooling operations are repeatedly performed for multiple times to obtain M second features for image segmentation of the M pixel categories.

For example, in the 5-shot scenario, repeat the last feature extraction and mask average pooling operations 5 times. For the same pixel category, each time the selected sample images are different, and each pixel category in the M pixel categories is obtained for 5 times. The average corresponding to the second features is obtained to obtain the final M second features.

Using the image segmentation neural network after the two-stage training and the first sample data set corresponding to the M pixel categories, after determining the M second features used to reflect the category features of the M pixel categories, in the subsequent actual image segmentation process In the method, it is not necessary to input the first image data set into the image segmentation neural network, but only need to input the image to be segmented and M second features into the image segmentation neural network, so as to realize the fast segmentation of multiple pixel categories in the image to be segmented.

It can be understood that the above-mentioned method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides image segmentation devices, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any image segmentation method provided by the present disclosure. For the corresponding technical solutions and descriptions, refer to the corresponding records in the Methods section. ,No longer.

FIG. 6 is a block diagram of an image segmentation apparatus according to an embodiment of the disclosure. As shown in FIG. 6, the image segmentation device 60 includes:

The feature extraction module 61 is configured to perform feature extraction on the to-be-segmented image to obtain the first feature of the to-be-segmented image, where the to-be-segmented image includes N pixel categories, where N is an integer greater than 1;

The feature fusion module 62 is configured to fuse the first feature with the M second features to obtain M first target features, and the M second features and the M first target features are in one-to-one correspondence with the M pixel categories, The M second features are determined based on the first sample data set, and the first sample data set includes at least one sample image corresponding to each pixel category in the M pixel categories and label information corresponding to each sample image, where M is greater than or equal to N, where N pixel categories are subsets of M pixel categories;

The image segmentation module 63 is configured to perform image segmentation on the image to be segmented according to the M first target features to obtain a target segmentation result of the image to be segmented.

In a possible implementation, the feature fusion module 62 includes:

The feature multiplication sub-module is configured to perform feature multiplication on the first feature and the ith second feature for the ith second feature among the M second features to obtain the ith third feature, 1≤i≤M ;

a feature subtraction submodule, configured to perform feature subtraction on the first feature and the i-th second feature to obtain the i-th fourth feature;

A feature connection submodule, configured to perform feature connection on the first feature, the i-th third feature and the i-th fourth feature to obtain the i-th first target feature in the M first target features;

The ith second feature, the ith third feature, the ith fourth feature, and the ith first target feature are all features corresponding to the ith pixel category in the M pixel categories.

In a possible implementation, the image segmentation module 63 includes:

The category-by-category prediction submodule is configured to perform category-by-category prediction on the image to be segmented according to the M first target features, and determine M segmentation sub-results corresponding to the to-be-segmented image, and the M segmentation sub-results correspond one-to-one with the M pixel categories;

The determining sub-module is configured to determine the target segmentation result according to the M segmentation sub-results.

In a possible implementation, the sub-module is predicted by category, and the specific configuration is:

Input the M first target features into the cosine classifier, and based on the cosine classifier and the M first target features, perform category-by-category prediction on the image to be segmented, and determine M segmentation sub-results.

In a possible implementation, the sub-module is predicted by category, and the specific configuration is:

For the i-th first target feature in the M first target features, according to the i-th first target feature, determine the i-th segmentation sub-result among the M segmentation sub-results corresponding to the image to be segmented, and determine the i-th segmentation sub-result. The sub-result includes pixels whose pixel class in the image to be segmented is the i-th pixel class among the M pixel classes.

In a possible implementation manner, the image segmentation method performed by the image segmentation device 60 is implemented by an image segmentation neural network.

In a possible implementation manner, the training samples of the image segmentation neural network include a first sample image to be segmented, segmentation annotation information of the first sample image to be segmented, and a first sample data set, in which the first sample image to be segmented including at least two pixel classes out of the M pixel classes;

The feature extraction module 61 is further configured to perform feature extraction on the first sample image to be segmented through an image segmentation neural network to obtain a fifth feature of the first sample image to be segmented, and to perform feature extraction on each pixel in the M pixel categories through an image segmentation neural network. The target sample images corresponding to the categories are subjected to feature extraction to obtain M sixth features, the M sixth features are in one-to-one correspondence with the M pixel categories, and the target sample images corresponding to each pixel category are at least one sample image corresponding to each pixel category. any one of ;

The feature fusion module 62 is further configured to determine the M seventh features according to the M sixth features and the annotation information of the target sample image corresponding to each pixel category in the M pixel categories, and combine the fifth feature and the M seventh features. Perform fusion to obtain M second target features, and M seventh features and M second target features are in one-to-one correspondence with M pixel categories;

The image segmentation module 63 is further configured to perform image segmentation on the first sample image to be divided according to the M second target features, to obtain a segmentation result of the first sample image to be divided;

The image segmentation device 60 also includes:

a segmentation loss determination module, configured to determine the segmentation loss according to the segmentation result of the first sample image to be segmented and segmentation annotation information;

The training module is configured to train the image segmentation neural network according to the segmentation loss, and obtain the trained image segmentation neural network.

In a possible implementation manner, the labeling information of the target sample image corresponding to each pixel category in the M pixel categories is a mask;

Feature fusion module 62, including:

The mask average pooling sub-module is configured for the ith sixth feature in the M sixth features, according to the ith sixth feature and the target sample image corresponding to the ith pixel category in the M pixel categories. mask, perform the mask average pooling operation, and obtain the i-th seventh feature in the M seventh features, the i-th sixth feature and the i-th seventh feature are the same as the i-th feature in the M pixel categories features corresponding to each pixel category.

In a possible implementation manner, the image segmentation device 60 further includes

The pre-training module is configured to, before training the image segmentation neural network according to the first sample image to be divided, the segmentation annotation information of the first sample image to be divided, and the first sample data set, according to the second sample image to be divided, The segmentation annotation information of the second sample image to be segmented, and the second sample data set for pre-training the image segmentation neural network, and the second sample data set includes multiple sample images corresponding to each pixel category in the P pixel categories and each sample The labeling information of the image, the M pixel categories are new pixel categories other than the P pixel categories, and the second sample image to be segmented includes at least two pixel categories in the P pixel categories.

In a possible implementation manner, the image segmentation apparatus 60 further includes:

The determining module is configured to segment the neural network according to the first sample data set and the trained image to determine M second features.

In a possible implementation manner, the labeling information of the sample image corresponding to each pixel category in the M pixel categories is a mask;

Determine the module, the specific configuration is:

Perform feature extraction on the sample images corresponding to each pixel category in the M pixel categories through the trained image segmentation neural network to obtain M eighth features;

For the i-th eighth feature in the M eighth features, according to the i-th eighth feature and the mask of the sample image corresponding to the i-th pixel category in the M pixel categories, the mask average pooling operation is performed to obtain the i-th second feature in the M second features;

The ith eighth feature and the ith second feature are both features corresponding to the ith pixel category in the M pixel categories.

In some embodiments, the functions or modules included in the apparatuses provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the descriptions of the above method embodiments. For brevity, here No longer.

Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

Embodiments of the present disclosure also provide a computer program product, including computer-readable codes. When the computer-readable codes are run on a device, a processor in the device executes a method for implementing the image segmentation method provided by any of the above embodiments. instruction.

Embodiments of the present disclosure further provide another computer program product for storing computer-readable instructions, which, when executed, cause the computer to perform the operations of the image segmentation method provided by any of the foregoing embodiments.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 7 is a structural block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. terminal.

7, an electronic device 800 may include one or more of the following components: a first processing component 802, a first memory 804, a first power supply component 806, a multimedia component 808, an audio component 810, a first input/output (Input Output, I/O) interface 812 , sensor component 814 , and communication component 816 .

The first processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The first processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Additionally, the first processing component 802 may include one or more modules to facilitate interaction between the first processing component 802 and other components. For example, the first processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the first processing component 802.

The first memory 804 is configured to store various types of data to support operations at the electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The first memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read Only Memory, EEPROM), Erasable Programmable Read Only Memory (Electrical Programmable Read Only Memory, EPROM), Programmable Read Only Memory (Programmable Read-Only Memory, PROM), Read Only Memory (Read- Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The first power supply component 806 provides power to various components of the electronic device 800 . The first power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a touch panel (Touch Pad, TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes at least one of the following: a front-facing camera, a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, at least one of the front camera and the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component 810 is configured to perform at least one of: outputting an audio signal, inputting an audio signal. For example, audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when electronic device 800 is in operating modes, such as calling mode, recording mode, and voice recognition mode. The received audio signal may be further stored in the first memory 804 or transmitted via the communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The first input/output interface 812 provides an interface between the first processing component 802 and a peripheral interface module, and the above-mentioned peripheral interface module may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 . For example, the sensor assembly 814 can detect the on/off state of the electronic device 800, the relative positioning of the components, such as the display and the keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or one of the electronic device 800 Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as wireless network (WiFi), second generation mobile communication technology (2G) or third generation mobile communication technology (3G), or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (Bluetooth, BT) technology and other technology to achieve.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing (Digital Signal Process, DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing any of the above a way.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a first memory 804 including computer program instructions that can be executed by the processor 820 of the electronic device 800 to accomplish any of the above a way.

FIG. 8 is a schematic structural diagram of another electronic device according to an embodiment of the disclosure. As shown in FIG. 8 , the electronic device 1900 may be provided as a server. 8, the electronic device 1900 includes a second processing component 1922, which further includes one or more processors, and a memory resource represented by a second memory 1932 for storing instructions executable by the second processing component 1922, such as applications. The application program stored in the second memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the second processing component 1922 is configured to execute instructions to perform any of the methods described above.

The electronic device 1900 may also include a second power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and a second input/output interface 1958. The electronic device 1900 can operate based on an operating system stored in the second memory 1932, such as a Microsoft server operating system (Windows ServerTM), a graphical user interface based operating system (Mac OS XTM) introduced by Apple, a multi-user and multi-process computer operating system. (UnixTM), Free and Open Source Unix-like Operating System (LinuxTM), Open Source Unix-like Operating System (FreeBSDTM) or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a second memory 1932 comprising computer program instructions executable by the second processing component 1922 of the electronic device 1900 to complete any of the above methods.

The present disclosure may be a system, method or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), ROM, erasable programmable read only memory (EPROM or flash memory), SRAM , portable compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanically encoded devices, such as punched cards or raised structures in grooves on which instructions are stored, and the above any suitable combination. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the present disclosure may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or, can be connected to an external computer (e.g. use an internet service provider to connect via the internet). In some embodiments, electronic circuits, such as programmable logic circuits, FPGAs, or Programmable Logic Arrays (PLAs), that can execute computer-readable Program instructions are read to implement various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Industrial Applicability

Embodiments of the present disclosure provide an image segmentation method and device, electronic equipment, computer, storage medium, and computer program. The method includes: performing feature extraction on an image to be segmented to obtain a first feature of the image to be segmented, and the image to be segmented in the image to be segmented. Including N pixel categories, N is an integer greater than 1; the first feature and M second features are fused to obtain M first target features, M second features and M first target features are combined with M The pixel categories are in one-to-one correspondence, and the M second features are determined based on the first sample data set. The first sample data set includes at least one sample image corresponding to each pixel category in the M pixel categories and the annotation of each sample image. information, M is greater than or equal to N, and the N pixel categories are subsets of the M pixel categories; according to the M first target features, image segmentation is performed on the image to be segmented, and the target segmentation result of the image to be segmented is obtained. The embodiments of the present disclosure can realize fast segmentation of multiple pixel categories in the image to be segmented.

Claims (25)

An image segmentation method, applied in an electronic device, the method comprising: Perform feature extraction on the to-be-segmented image to obtain the first feature of the to-be-segmented image, where the to-be-segmented image includes N pixel categories, where N is an integer greater than 1; The first features and M second features are fused to obtain M first target features, and the M second features and the M first target features are in one-to-one correspondence with the M pixel categories, so The M second features are determined based on the first sample data set, and the first sample data set includes at least one sample image corresponding to each pixel category in the M pixel categories and the annotation information of each sample image. , M is greater than or equal to N, and the N pixel categories are subsets of the M pixel categories; Perform image segmentation on the to-be-segmented image according to the M first target features to obtain a target segmentation result of the to-be-segmented image. The method according to claim 1, wherein the first feature and M second features are fused to obtain M first target features, comprising: For the i-th second feature in the M second features, feature multiplication is performed on the first feature and the i-th second feature to obtain the i-th third feature, 1≤i≤M; performing feature subtraction on the first feature and the i-th second feature to obtain the i-th fourth feature; Feature connection is performed on the first feature, the i-th third feature and the i-th fourth feature to obtain the i-th first target feature in the M first target features; The i-th second feature, the i-th third feature, the i-th fourth feature, and the i-th first target feature are all related to the i-th in the M pixel categories Features corresponding to pixel classes. The method according to claim 1 or 2, wherein, performing image segmentation on the to-be-segmented image according to the M first target features to obtain a target segmentation result of the to-be-segmented image, comprising: According to the M first target features, perform category-by-category prediction on the to-be-segmented image, and determine M segmentation sub-results corresponding to the to-be-segmented image, where the M segmentation sub-results are the same as the M pixel categories one correspondence; The target segmentation result is determined according to the M segmentation sub-results. The method according to claim 3, wherein the performing category-by-category prediction on the to-be-segmented image according to the M first target features, and determining the M segmentation sub-results corresponding to the to-be-segmented image, comprises: The M first target features are input into a cosine classifier, and based on the cosine classifier and the M first target features, category-by-category prediction is performed on the to-be-segmented image, and the M segmentation sub-results are determined. The method according to claim 3, wherein the performing category-by-category prediction on the to-be-segmented image according to the M first target features, and determining the M segmentation sub-results corresponding to the to-be-segmented image, comprises: For the i-th first target feature among the M first target features, determine the i-th one of the M segmentation sub-results corresponding to the to-be-segmented image according to the i-th first target feature Segmentation sub-results, the i-th segmentation sub-result includes pixels whose pixel types in the to-be-segmented image are the i-th pixel types among the M pixel types. The method according to any one of claims 1 to 5, wherein the image segmentation method is implemented by an image segmentation neural network. The method according to claim 6, wherein the training samples of the image segmentation neural network include a first sample image to be segmented, segmentation label information of the first sample image to be segmented, and the first sample data set , the first sample image to be segmented includes at least two pixel categories in the M pixel categories; The method also includes: Perform feature extraction on the first sample image to be segmented through the image segmentation neural network to obtain a fifth feature of the first sample image to be segmented, and perform feature extraction on the M pixel categories through the image segmentation neural network The target sample images corresponding to each pixel category are subjected to feature extraction to obtain M sixth features, the M sixth features are in one-to-one correspondence with the M pixel categories, and the target sample images corresponding to each pixel category are corresponding to each pixel category. Any one of at least one sample image of ; According to the M sixth features and the labeling information of the target sample image corresponding to each pixel category in the M pixel categories, M seventh features are determined, and the fifth features and the M seventh features are combined Perform fusion to obtain M second target features, and the M seventh features and the M second target features are in one-to-one correspondence with the M pixel categories; performing image segmentation on the first sample image to be segmented according to the M second target features, to obtain a segmentation result of the first sample image to be segmented; determining the segmentation loss according to the segmentation result of the first sample image to be segmented and the segmentation annotation information; According to the segmentation loss, the image segmentation neural network is trained to obtain a trained image segmentation neural network. The method according to claim 7, wherein the labeling information of the target sample image corresponding to each pixel category in the M pixel categories is a mask; The M seventh features are determined according to the M sixth features and the annotation information of the target sample image corresponding to each pixel category in the M pixel categories, including: For the i-th sixth feature in the M sixth features, according to the i-th sixth feature and the mask of the target sample image corresponding to the i-th pixel category in the M pixel categories, execute Mask average pooling operation to obtain the i-th seventh feature in the M seventh features, the i-th sixth feature and the i-th seventh feature are both related to the M pixel categories The feature corresponding to the i-th pixel category in . The method according to any one of claims 7 to 8, wherein, according to the first sample image to be divided, the segmentation annotation information of the first sample image to be divided, and the first sample data set Before training the image segmentation neural network, the method further includes: The image segmentation neural network is pre-trained according to the second sample image to be segmented, the segmentation annotation information of the second sample image to be segmented, and a second sample data set, where the second sample data set includes P pixels Multiple sample images corresponding to each pixel category in the category and label information of each sample image, the M pixel categories are new pixel categories other than the P pixel categories, and the second sample image to be segmented includes the At least two pixel classes of the P pixel classes. The method according to any one of claims 7 to 9, wherein the method further comprises: The M second features are determined according to the first sample data set and the trained image segmentation neural network. The method according to claim 10, wherein the labeling information of the sample image corresponding to each pixel category in the M pixel categories is a mask; Determining the M second features according to the first sample data set and the trained image segmentation neural network, including: Perform feature extraction on the sample images corresponding to each pixel category in the M pixel categories through the trained image segmentation neural network to obtain M eighth features; For the i-th eighth feature in the M eighth features, perform mask average pooling according to the i-th eighth feature and the mask of the sample image corresponding to the i-th pixel category in the M pixel categories transformation operation to obtain the i-th second feature in the M second features; The i-th eighth feature and the i-th second feature are both features corresponding to the i-th pixel category in the M pixel categories. An image segmentation device, comprising: A feature extraction module, configured to perform feature extraction on the image to be segmented, to obtain the first feature of the image to be segmented, and N pixel categories are included in the image to be segmented, and N is an integer greater than 1; A feature fusion module, configured to fuse the first features and M second features to obtain M first target features, and the M second features and the M first target features are both associated with M pixels The categories are in one-to-one correspondence, and the M second features are determined based on the first sample data set, and the first sample data set includes at least one sample image corresponding to each pixel category in the M pixel categories and Labeling information corresponding to each sample image, M is greater than or equal to N, and the N pixel categories are subsets of the M pixel categories; The image segmentation module is configured to perform image segmentation on the to-be-segmented image according to the M first target features to obtain a target segmentation result of the to-be-segmented image. The apparatus according to claim 12, wherein the feature fusion module comprises: The feature multiplication sub-module is configured to perform feature multiplication on the first feature and the ith second feature for the ith second feature among the M second features to obtain the ith third feature, 1≤i≤M ; a feature subtraction submodule, configured to perform feature subtraction on the first feature and the i-th second feature to obtain the i-th fourth feature; A feature connection submodule, configured to perform feature connection on the first feature, the i-th third feature and the i-th fourth feature to obtain the i-th first target feature in the M first target features; The ith second feature, the ith third feature, the ith fourth feature, and the ith first target feature are all features corresponding to the ith pixel category in the M pixel categories. The apparatus according to claim 12 or 13, wherein the image segmentation module 63 comprises: The category-by-category prediction submodule is configured to perform category-by-category prediction on the image to be segmented according to the M first target features, and determine M segmentation sub-results corresponding to the to-be-segmented image, and the M segmentation sub-results correspond one-to-one with the M pixel categories; The determining sub-module is configured to determine the target segmentation result according to the M segmentation sub-results. The apparatus according to claim 14, wherein the category-by-category prediction submodule is specifically configured as: Input the M first target features into the cosine classifier, and based on the cosine classifier and the M first target features, perform category-by-category prediction on the image to be segmented, and determine M segmentation sub-results. The apparatus according to claim 14, wherein the category-by-category prediction submodule is specifically configured as: For the i-th first target feature in the M first target features, according to the i-th first target feature, determine the i-th segmentation sub-result among the M segmentation sub-results corresponding to the image to be segmented, and the i-th segmentation sub-result is determined. The sub-result includes pixels whose pixel class in the image to be segmented is the i-th pixel class among the M pixel classes. The apparatus according to any one of claims 12 to 16, wherein the image segmentation method performed by the apparatus is implemented by an image segmentation neural network. The device according to claim 17, wherein the training samples of the image segmentation neural network include a first sample image to be segmented, segmentation label information of the first sample image to be segmented, and a first sample data set, the first sample image to be segmented. The segmentation sample image includes at least two pixel categories in the M pixel categories; The feature extraction module is further configured to perform feature extraction on the first sample image to be segmented through an image segmentation neural network to obtain a fifth feature of the first sample image to be segmented, and to perform feature extraction on each of the M pixel categories through an image segmentation neural network. The target sample images corresponding to the pixel categories are subjected to feature extraction to obtain M sixth features, the M sixth features are in one-to-one correspondence with the M pixel categories, and the target sample images corresponding to each pixel category are at least one sample image corresponding to each pixel category. any one of; The feature fusion module is further configured to determine the M seventh features according to the M sixth features and the annotation information of the target sample image corresponding to each pixel category in the M pixel categories, and combine the fifth feature with the M seventh features. The features are fused to obtain M second target features, and the M seventh features and the M second target features are in one-to-one correspondence with the M pixel categories; The image segmentation module is further configured to perform image segmentation on the first sample image to be segmented according to the M second target features, to obtain a segmentation result of the first sample image to be segmented; The device also includes: a segmentation loss determination module, configured to determine the segmentation loss according to the segmentation result of the first sample image to be segmented and segmentation annotation information; The training module is configured to train the image segmentation neural network according to the segmentation loss, and obtain the trained image segmentation neural network. The device according to claim 18, wherein the labeling information of the target sample image corresponding to each pixel category in the M pixel categories is a mask; The feature fusion module includes: The mask average pooling submodule is configured for the ith sixth feature in the M sixth features, according to the ith sixth feature and the target sample image corresponding to the ith pixel category in the M pixel categories. mask, perform the mask average pooling operation, and obtain the i-th seventh feature among the M seventh features, the i-th sixth feature and the i-th seventh feature are the same as the i-th feature in the M pixel categories features corresponding to each pixel class. The apparatus according to any one of claims 18 to 19, wherein the apparatus further comprises The pre-training module is configured to, before training the image segmentation neural network according to the first sample image to be divided, the segmentation annotation information of the first sample image to be divided, and the first sample data set, according to the second sample image to be divided, The segmentation annotation information of the second sample image to be segmented, and the second sample data set, which pre-trains the image segmentation neural network, and the second sample data set includes multiple sample images corresponding to each pixel category in the P pixel categories and each sample Labeling information of the image, the M pixel categories are new pixel categories other than the P pixel categories, and the second sample image to be segmented includes at least two pixel categories in the P pixel categories. The device according to any one of claims 18 to 20, wherein the device further comprises: The determining module is configured to segment the neural network according to the first sample data set and the trained image to determine M second features. The device according to claim 21, wherein the labeling information of the sample image corresponding to each pixel category in the M pixel categories is a mask; The determining module is specifically configured as: Perform feature extraction on the sample images corresponding to each pixel category in the M pixel categories through the trained image segmentation neural network to obtain M eighth features; For the i-th eighth feature in the M eighth features, according to the i-th eighth feature and the mask of the sample image corresponding to the i-th pixel category in the M pixel categories, the mask average pooling operation is performed to obtain the i-th second feature in the M second features; The ith eighth feature and the ith second feature are both features corresponding to the ith pixel category in the M pixel categories. An electronic device comprising: processor; memory configured to store processor executable instructions; wherein the processor is configured to invoke instructions stored in the memory to perform the method of any one of claims 1-11. A computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the method of any one of claims 1 to 11 when executed by a processor. A computer program comprising computer readable code, when the computer readable code is executed in an electronic device, a processor in the electronic device executes the method for implementing any one of claims 1 to 11.

Images