CN116433681A

CN116433681A - Image segmentation method, device, computer equipment, storage medium and program product

Info

Publication number: CN116433681A
Application number: CN202111645392.3A
Authority: CN
Inventors: 魏东; 卢东焕; 周联昱; 李悦翔; 马锴; 王连生; 郑冶枫
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-12-30
Filing date: 2021-12-30
Publication date: 2023-07-14

Abstract

The application discloses an image segmentation method, device, computer equipment, storage medium and program product, and relates to the technical field of image processing. The method includes: determining an under-segmentation mask and an over-segmentation mask corresponding to the sample medical image based on the sample diameter label corresponding to the sample segmentation region in the sample medical image; and training a first image segmentation network based on the sample medical image and the under-segmentation mask , the first image segmentation network is used to predict the under-segmented region of the sample medical image; based on the sample medical image and the over-segmentation mask, the second image segmentation network is trained, and the second image segmentation network is used to perform over-segmented regions on the sample medical image predict. Using sample diameter labeling to generate weak supervision information can reduce manual labeling operations, so that large-scale training sample data can be obtained through less manual labeling, which improves the efficiency of generating corresponding supervisory information for training image segmentation networks, and further improves Model training efficiency.

Description

Image segmentation method, image segmentation device, computer device, storage medium and program product

Technical Field

The embodiment of the application relates to the technical field of image processing, in particular to an image segmentation method, an image segmentation device, computer equipment, a storage medium and a program product.

Background

The medical image segmentation is used for segmenting parts with certain special meanings in the medical image and extracting relevant characteristics, thereby providing reliable basis for clinical medicine and pathology research.

The medical image may contain various types of images to be segmented, such as organs, tissues, etc. in the human body, and the computer device segments the medical image by invoking a related image segmentation network. In order to realize the image segmentation function of the image segmentation network, training is required by medical images marked with segmentation areas; in the related art, the contours of the segmented regions are manually marked by medical professionals to obtain a sample image with marked regions.

Obviously, by adopting manually marked sample images, special medical staff is required to mark, the sample image acquisition difficulty is high, and therefore, the training efficiency of the image segmentation network is low.

Disclosure of Invention

The embodiment of the application provides an image segmentation method, an image segmentation device, computer equipment, a storage medium and a program product. The technical scheme comprises the following aspects.

In one aspect, there is provided an image segmentation method, the method comprising:

carrying out under-segmentation region prediction on a target segmentation region in a target medical image to obtain a first target probability image, wherein the first target probability image comprises probability values of all pixel points in the target medical image belonging to the under-segmentation region, and the under-segmentation region is smaller than the target segmentation region;

the target segmentation area in the target medical image is subjected to excessive segmentation area prediction to obtain a second target probability image, wherein the second target probability image comprises the probability that each pixel point in the target medical image belongs to an excessive segmentation area, and the excessive segmentation area is larger than the target segmentation area;

and determining a target segmentation mask corresponding to the target medical image based on the first target probability image and the second target probability image.

In another aspect, there is provided an image segmentation method, the method comprising:

determining an under-segmentation mask and an over-segmentation mask corresponding to a sample medical image based on sample diameter labels corresponding to sample segmentation areas in the sample medical image, wherein the under-segmentation areas corresponding to the under-segmentation mask are smaller than the sample segmentation areas, and the over-segmentation areas corresponding to the over-segmentation mask are larger than the sample segmentation areas;

Training a first image segmentation network based on the sample medical image and the under-segmentation mask, the first image segmentation network being used for under-segmentation region prediction of the sample medical image;

a second image segmentation network is trained based on the sample medical image and the over-segmentation mask, the second image segmentation network being used to over-segment the sample medical image.

In another aspect, there is provided an image segmentation apparatus, the apparatus including:

the first segmentation prediction module is used for carrying out under-segmentation region prediction on a target segmentation region in a target medical image to obtain a first target probability image, wherein the first target probability image comprises probability values of all pixel points in the target medical image belonging to the under-segmentation region, and the under-segmentation region is smaller than the target segmentation region;

the second segmentation prediction module is used for predicting the target segmentation area in the target medical image to obtain a second target probability image, wherein the second target probability image comprises the probability that each pixel point in the target medical image belongs to the segmentation area, and the segmentation area is larger than the target segmentation area;

And the first determining module is used for determining a target segmentation mask corresponding to the target medical image based on the first target probability image and the second target probability image.

the second determining module is used for determining an under-segmentation mask and an over-segmentation mask corresponding to the sample medical image based on sample diameter labels corresponding to sample segmentation areas in the sample medical image, wherein the under-segmentation areas corresponding to the under-segmentation mask are smaller than the sample segmentation areas, and the over-segmentation areas corresponding to the over-segmentation mask are larger than the sample segmentation areas;

the first training module is used for training a first image segmentation network based on the sample medical image and the under-segmentation mask, and the first image segmentation network is used for carrying out under-segmentation region prediction on the sample medical image;

and the second training module is used for training a second image segmentation network based on the sample medical image and the over-segmentation mask, wherein the second image segmentation network is used for performing over-segmentation region prediction on the sample medical image.

In another aspect, there is provided a computer device comprising a processor and a memory, the memory having stored therein at least one program that is loaded and executed by the processor to implement the image segmentation method as described in the above aspect.

In another aspect, a computer readable storage medium having stored therein at least one instruction, at least one program, code set, or instruction set loaded and executed by a processor to implement the image segmentation method as described in the above aspect is provided.

According to another aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image segmentation method provided in the alternative implementation described above.

The beneficial effects that technical scheme that this application embodiment provided include at least:

the method comprises the steps of constructing supervision information for training a first image segmentation network and a second image segmentation network by using sample diameter labels corresponding to sample segmentation areas in a sample medical image: the under-segmentation mask and the over-segmentation mask enable the first image segmentation network to have an under-segmentation region prediction function, and the second image segmentation network to have an over-segmentation region prediction function, so that the trained first image segmentation network and second image segmentation network can be used for segmentation prediction of a segmentation region in a medical image; compared with the prior art that sample segmentation contour labeling of a sample segmentation area is required to be used as supervision information, the method and the device for generating the weak supervision information by using sample diameter labeling can reduce manual labeling operation, enable large-scale training sample data to be obtained through fewer manual labeling, improve efficiency of generating the supervision information corresponding to a training image segmentation network, and further improve model training efficiency.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 illustrates a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

FIG. 2 illustrates a flowchart of an image segmentation method provided by an exemplary embodiment of the present application;

FIG. 3 illustrates a flowchart of an image segmentation method provided in another exemplary embodiment of the present application;

FIG. 4 illustrates a schematic diagram of the generation of an under-split mask and an over-split mask according to an exemplary embodiment of the present application;

FIG. 5 illustrates a training process diagram of an image segmentation model, as shown in an exemplary embodiment of the present application;

FIG. 6 illustrates a flowchart of an image segmentation method provided by another exemplary embodiment of the present application;

FIG. 7 illustrates a training process diagram of an image segmentation model shown in another exemplary embodiment of the present application;

FIG. 8 illustrates a training process diagram of an image segmentation model shown in another exemplary embodiment of the present application;

FIG. 9 illustrates a flowchart of an image segmentation method provided by an exemplary embodiment of the present application;

FIG. 10 illustrates a flowchart of an image segmentation method provided by another exemplary embodiment of the present application;

FIG. 11 illustrates a schematic diagram of an image segmentation process shown in an exemplary embodiment of the present application;

FIG. 12 is a schematic diagram of the results of a test for the effect of lambda value on the data set KiTS19 on the segmentation results;

FIG. 13 is a schematic diagram of the results of a test for the effect of lambda value on the split results on dataset LiTS 17;

fig. 14 is a block diagram of an image segmentation apparatus provided in an exemplary embodiment of the present application;

fig. 15 is a block diagram of an image segmentation apparatus provided in another exemplary embodiment of the present application;

fig. 16 shows a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.

The artificial intelligence technology is a comprehensive subject, and relates to the technology with wide fields, namely the technology with a hardware level and the technology with a software level. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

Computer Vision (CV) is a science of how to "look" at a machine, and more specifically, to use a camera and a Computer to identify and measure a target instead of human eyes, and further perform graphic processing to make the Computer process an image more suitable for human eyes to observe or transmit to an instrument for detection. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (Optical Character Recognition, OCR), video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D techniques, virtual reality, augmented reality, synchronous positioning, map construction, etc., as well as common biometric recognition techniques such as face recognition, fingerprint recognition, etc.

The method disclosed by the embodiment of the application is to perform image segmentation processing on a specific area in a medical image by utilizing a computer vision technology. For example, by using the scheme provided by the embodiment of the application, the lesion region in the liver can be segmented based on the computed tomography (Computed Tomography, CT) image of the liver region.

FIG. 1 illustrates a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment includes a terminal 110 and a server 120. The data communication between the terminal 110 and the server 120 is performed through a communication network, alternatively, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network, a metropolitan area network, and a wide area network.

The terminal 110 is an electronic device with medical image segmentation requirements, which may be a smart phone, a tablet computer, a personal computer, or the like, and the embodiment is not limited thereto. In fig. 1, a computer used by a medical care provider is illustrated as a terminal 110.

In some embodiments, the terminal 110 has an application program installed therein that has a medical image segmentation function. When the scanned medical image is required to be subjected to image segmentation, a user inputs the medical image to be segmented into an application program, so that the medical image is uploaded to the server 120, the server 120 performs image segmentation on the medical image, and an image segmentation result is fed back.

The server 120 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), basic cloud computing services such as big data and an artificial intelligence platform.

In some embodiments, the server 120 is used to provide medical image segmentation services for applications installed in the terminal 110. Optionally, an image segmentation model is set in the server 120, the image segmentation network is of a twin network structure, and comprises a first image segmentation network 121 and a second image segmentation network 122, and the first image segmentation network 121 performs under-segmentation region prediction on the medical image to obtain an under-segmentation region prediction result; meanwhile, the second image segmentation network 122 performs over-segmentation region prediction on the medical image to obtain an over-segmentation region prediction result, and then generates an image segmentation result corresponding to the medical image based on the under-segmentation region prediction result and the over-segmentation region prediction result.

Of course, in other possible embodiments, the image segmentation model may be deployed on the terminal 110 side, and the terminal 110 may locally implement image segmentation, without using the server 120, which is not limited in this embodiment.

Optionally, in other possible application scenarios, the embodiment of the present application further provides a training method for an image segmentation model, where the corresponding terminal 110 may upload training sample data to the server 120, and the server 120 performs training for the image segmentation model based on the training sample data and feeds back the trained image segmentation model to the terminal 110.

For convenience of description, the following embodiments are described as examples of the medical image segmentation method performed by a computer device.

Referring to fig. 2, a flowchart of an image segmentation method according to an exemplary embodiment of the present application is shown. The embodiment is exemplified by taking the computer device as an execution subject of the method, and the method includes:

step 201, determining an under-segmentation mask and an over-segmentation mask corresponding to a sample medical image based on a sample diameter label corresponding to a sample segmentation region in the sample medical image.

In the related art, when training an image segmentation network, training samples are adopted as follows: sample medical images and sample region contour labels corresponding to sample segmentation regions in the sample medical images are used as supervision information in the image segmentation network training process, the sample region contour labels often need to be manually drawn into regions, the sample segmentation regions in the medical images are often convex polygon contours, if more training samples are to be obtained, the obvious manual labeling amount is larger, and the obtaining efficiency of the training samples is lower.

Aiming at the problem that the acquisition difficulty of a training sample in the image segmentation network training process is high in the related technology, the embodiment of the application provides a weak supervision training mode, wherein a sample medical image, an under-segmentation mask and an over-segmentation mask are used as a training data set, and the under-segmentation mask and the over-segmentation mask are used as weak supervision information in the image segmentation network training process; because both the under-segmentation mask and the over-segmentation mask can be determined by the sample diameter labeling corresponding to the sample segmentation region in the sample medical image, the sample diameter labeling only needs to manually label the diameter position of the sample segmentation region, and compared with the whole region outline information of the labeled sample segmentation region, the embodiment can obtain more available training sample sets through fewer manual labeling as much as possible.

Taking a sample medical image as a CT image as an example, the diameter label of the sample is RECIST label, and in clinical medical treatment, a solid tumor curative effect evaluation standard (Response Evaluation Criteria in Solid Tumors, RECIST) is used for CT to measure tumors or lymph nodes; a plurality of RECIST diameter measurements and corresponding CT images are stored in the picture archiving and communication systems of the corresponding hospitals, wherein the RECIST diameter measurements generally comprise a long diameter and a short diameter, which are a common way for radiologists to highlight a target area; thus, in one possible implementation, only the CT images and their corresponding RECIST diameter measurements need to be acquired, so that the under-segmented mask and over-segmented mask corresponding to each CT image can be generated without requiring other manual labeling processes.

In one illustrative example, the relationship between the under-segmentation mask, the over-segmentation mask, and the sample segmentation region is:

wherein,,

under-segmented region representing a sample segmented region in a sample medical image,/->

Representing a sample segmentation area in a sample medical image, < >>

An over-segmented region representing a sample segmented region in a sample medical image. As can be seen from the formula (1), the under-divided region corresponding to the under-divided mask is smaller than the sample divided region, and the over-divided region corresponding to the over-divided mask is larger than the sample divided region.

In one possible implementation, a sample medical image and a sample diameter label corresponding to the sample medical image are obtained, and an under-segmentation area and an over-segmentation area corresponding to a sample segmentation area in the sample medical image are determined through the sample diameter label, so that a corresponding under-segmentation mask and over-segmentation mask are generated.

Step 202, training a first image segmentation network based on a sample medical image and an under-segmentation mask, the first image segmentation network being used for under-segmentation region prediction of the sample medical image.

Because the under-segmentation mask and the over-segmentation mask are adopted as weak supervision information, twin networks are correspondingly required to be arranged and are respectively used for under-segmentation region prediction and over-segmentation prediction, in one possible implementation mode, a first image segmentation network is arranged, under-segmentation region prediction is carried out on a sample medical image through the first image segmentation network, an under-segmentation region prediction result is obtained, and in order to enable the under-segmentation region prediction result to learn towards the under-segmentation mask, the first image segmentation network is required to be trained by using the under-segmentation mask and the under-segmentation prediction result; that is, the training data set corresponding to the first image segmentation network is: the method comprises the steps of training a first image segmentation network through a sample medical image and an under-segmentation mask, so that the first image segmentation network realizes a prediction function which can be used for under-segmentation areas of the sample medical image.

Optionally, the specific network structure corresponding to the first image segmentation network may use any network structure having an image segmentation function, for example, a feedback neural network (Hopfield Neural Network, HNN), U-Net, ARUNet, swin Transformer, etc., and the specific network structure adopted by the first image segmentation network in the embodiment of the present application is not limited.

Step 203, training a second image segmentation network based on the sample medical image and the over-segmentation mask, the second image segmentation network being used for over-segmentation region prediction of the sample medical image.

Corresponding to the first image segmentation network provided for the under-segmented region, in one possible embodiment, a second image segmentation network is also provided for over-segmented region prediction of the sample medical image by means of which over-segmented region prediction results are obtained, in order to enable the over-segmented prediction results to be learned towards the over-segmentation mask, the second image segmentation network being trained using the over-segmentation mask and the over-segmentation prediction results; that is, the training data set for the second image segmentation network is: the sample medical image and the over-segmentation mask to train the second image segmentation network through the sample medical image and the over-segmentation mask such that the second image segmentation network implements a prediction function that is operable to over-segment the sample medical image.

Optionally, the specific network structure corresponding to the second image segmentation network may use any network structure having an image segmentation function, for example, HNN, U-Net, ARUNet, swin Transformer, etc., and the specific network structure adopted by the second image segmentation network in the embodiment of the present application is not limited.

Optionally, after the first image segmentation network has an under-segmentation region prediction function of a region to be segmented in the medical image, the second image segmentation network has an over-segmentation region prediction function of the region to be segmented in the medical image, and in a model application process, a segmentation result corresponding to the medical image can be obtained through model integration according to a probability map predicted by the first image segmentation network and a probability map predicted by the second image segmentation network; for specific model application procedures, reference may be made to the following embodiments, which are not described herein.

It should be noted that, the first image segmentation network and the second image segmentation network are of twin network structure, so that the first image segmentation network and the second image segmentation network may have the same neural network structure, for example, both the first image segmentation network and the second image segmentation network may adopt U-net networks; the first image segmentation network may also have a different neural network structure than the second image segmentation network, for example, the first image segmentation network may employ a U-net network and the second image segmentation network may employ HNN.

In this embodiment of the present application, the first image segmentation network and the second image segmentation network train simultaneously, that is, in each training process, the same sample medical image is used as a training sample.

In summary, in the embodiment of the present application, a weakly supervised image segmentation method is provided, and supervision information for training a first image segmentation network and a second image segmentation network is constructed by using sample diameter labels corresponding to sample segmentation areas in a sample medical image: the under-segmentation mask and the over-segmentation mask enable the first image segmentation network to have an under-segmentation region prediction function, and the second image segmentation network to have an over-segmentation region prediction function, so that the trained first image segmentation network and second image segmentation network can be used for segmentation prediction of a segmentation region in a medical image; compared with the prior art that sample segmentation contour labeling of a sample segmentation area is required to be used as supervision information, the method and the device for generating the weak supervision information by using sample diameter labeling can reduce manual labeling operation, enable large-scale training sample data to be obtained through fewer manual labeling, improve efficiency of generating the supervision information corresponding to a training image segmentation network, and further improve model training efficiency.

In the process of generating the underspection mask and the overspection mask of the weak supervision information, the underspection region and the overspection region are firstly determined according to the sample diameter mark, and then the corresponding mask (mask) image is obtained through binarization processing.

Referring to fig. 3, a flowchart of an image segmentation method according to another exemplary embodiment of the present application is shown. The embodiment is exemplified by taking the computer device as an execution subject of the method, and the method includes:

step 301, determining a long diameter label and a short diameter label corresponding to a sample segmentation area in a sample medical image based on the sample diameter label.

In one possible embodiment, the sample diameter labels include a long diameter label and a short diameter label, the long diameter label being the label of the sample segmentation region corresponding to the longest diameter and the short diameter label being the label of the sample segmentation region corresponding to the shortest diameter.

As shown in fig. 4, a schematic diagram of the generation process of the under-split mask and the over-split mask according to an exemplary embodiment of the present application is shown. The sample medical image 400 includes a sample segmentation region 401, the longest diameter of the sample segmentation region 401 is shown by a dashed line 402, and the shortest diameter of the sample segmentation region 401 is shown by a dashed line 403.

Step 302, determining an under-segmentation mask corresponding to the sample medical image based on the long-diameter label and the short-diameter label.

In one possible implementation, the under-divided region indicated by the under-divided mask is a quadrilateral region, and four vertices corresponding to the quadrilateral region are determined by a long-diameter label and a short-diameter label, that is, the diagonal lines of the quadrilateral region are respectively the longest diameter indicated by the long-diameter label and the shortest diameter indicated by the short-diameter label.

In one illustrative example, step 302 may include step 302A and step 302B.

Step 302A, an under-segmented region in a sample medical image is determined based on the long diameter annotation and the short diameter annotation.

In one possible implementation manner, if a corresponding under-segmentation mask needs to be generated, firstly, determining an under-segmentation region corresponding to the sample medical image according to the longest diameter indicated by the long-diameter label and the shortest diameter indicated by the short-diameter label, and further performing binarization processing on the sample medical image based on the under-segmentation region to obtain the under-segmentation mask corresponding to the sample medical image.

In step 302B, the pixel values of the pixels belonging to the under-segmented region in the medical sample image are set as the first pixel values, and the pixel values of the pixels not belonging to the under-segmented region in the medical sample image are set as the second pixel values, so as to generate an under-segmented mask corresponding to the medical sample image.

Because the under-segmentation mask can clearly indicate the under-segmentation region and other regions in the sample medical image, in one possible implementation manner, the pixel values of the pixel points belonging to the under-segmentation region are set as first pixel values, and the pixel values of the pixel points not belonging to the under-segmentation region are set as second pixel values, so that the under-segmentation region in the sample medical image can be determined directly according to the difference of the pixel values.

Taking the first pixel value as 1 and the second pixel value as 0 as an example, the pixel values of the pixel points located in the under-segmented region in the sample medical image are all 1, and the pixel values of the pixel points located outside the under-segmented region in the sample medical image are all 0.

As shown in fig. 4, the under-segmented region 404 is determined by using the longest diameter 402 and the shortest diameter 403 as diagonal lines, and the sample medical image 400 is binarized based on the under-segmented region 404 to obtain an under-segmented mask 405.

Step 303, determining an over-division mask corresponding to the sample medical image based on the long-diameter label.

In one possible implementation manner, the over-dividing region indicated by the over-dividing mask is a circular region, and if the circular region is larger than the sample dividing region, the longest diameter indicated by the long diameter label is taken as the diameter, so that the over-dividing region in the sample medical image is determined, and the corresponding over-dividing mask is generated.

In the actual execution process, the execution sequence of step 302 and step 303 is not limited, and step 302 may be executed first, and then step 303 may be executed; or, step 303 is executed first, and then step 302 is executed; or step 302 and step 303 may be performed simultaneously.

In one illustrative example, step 303 may include step 303A and step 303B.

In step 303A, an over-segmented region in the sample medical image is determined based on the long diameter labeling.

In one possible implementation manner, after determining the position of the longest diameter indicated by the long diameter label in the sample medical image, the over-segmentation area corresponding to the sample medical image may be determined by taking the longest diameter as the diameter, and then binarization processing is performed on the sample medical image based on the over-segmentation area, so as to generate an over-segmentation mask corresponding to the sample medical image.

In step 303B, the pixel values of the pixels belonging to the over-segmentation area in the medical sample image are set as the first pixel values, and the pixel values of the pixels not belonging to the over-segmentation area in the medical sample image are set as the second pixel values, so as to generate an over-segmentation mask corresponding to the sample medical image.

Since the over-segmentation mask can clearly indicate the over-segmentation region and other regions in the sample medical image, in one possible implementation, the pixel values of the pixel points belonging to the over-segmentation region are set to be first pixel values, and the pixel values of the pixel points not belonging to the over-segmentation region are set to be second pixel values, so that the over-segmentation region in the sample medical image can be determined directly according to the difference of the pixel values.

Taking the first pixel value as 1 and the second pixel value as 0 as an example, the pixel values of the pixels belonging to the over-segmentation area in the sample medical image are all 1, and the pixel values of the pixels not belonging to the under-segmentation area in the sample medical image are all 0.

As shown in fig. 4, the over-divided region 406 is determined with the longest diameter 402 as a diameter, and the sample medical image 400 is binarized based on the over-divided region 406 to obtain an over-divided mask 407.

Step 304, inputting the sample medical image into a first image segmentation network to obtain a first sample probability image output by the first image segmentation network, wherein the first sample probability image comprises probability values of each pixel point in the sample medical image belonging to an undersegmented region.

The image segmentation model in this embodiment includes a first image segmentation network for under-segmentation region prediction of the sample medical image and a second image segmentation network for over-segmentation region prediction of the sample medical image.

Based on the above-mentioned under-segmentation region prediction function of the first image segmentation network, in one possible implementation manner, the sample medical image is input into the first image segmentation network, under-segmentation region prediction is performed on the sample medical image by the first image segmentation network, and whether each pixel point in the sample medical image belongs to an under-segmentation region is determined, so that a prediction result first sample probability image is output, wherein a pixel value of each pixel point in the first sample probability image indicates a probability that the pixel point belongs to the under-segmentation region, that is, a pixel value in the first sample probability image is a probability value that the pixel point belongs to the under-segmentation region.

For the first sample probability image, if the pixel value (probability value) corresponding to the pixel point is larger, the probability that the pixel point belongs to the under-segmented region is higher; conversely, if the pixel value (probability value) corresponding to the pixel is smaller, the probability that the pixel belongs to the under-divided region is lower.

Step 305, training a first image segmentation network based on the first sample probability image and the under-segmentation mask.

In order to enable the first image segmentation network to accurately predict the under-segmented region in the sample medical image, an under-segmented mask corresponding to the sample medical image is required to serve as weak supervision information, and supervision training is performed on the first image segmentation network, so that in a possible implementation manner, segmentation loss in the under-segmented region prediction process can be determined according to the first sample probability image and the under-segmented mask, and the first image segmentation network is trained based on the segmentation loss.

In one illustrative example, the segmentation penalty of the first image segmentation network may be expressed as:

wherein,,

represents a segmentation loss of the first image segmentation network, < >>

Representing a first sample probability image, Q representing an under-segmentation mask, >

Representing the first sample probability image and the under-segmentation mask in the whole image domain>

Soft Dice loss on (soft Dice).

Step 306, inputting the sample medical image into a second image segmentation network to obtain a second sample probability image output by the second image segmentation network, wherein the second sample probability image comprises probability values of each pixel point in the sample medical image belonging to an excessive region.

Based on the above-mentioned over-segmentation region prediction function of the second image segmentation network, in one possible implementation manner, the sample medical image is input into the second image segmentation network, the over-segmentation region prediction is performed on the sample medical image by the second image segmentation network, and whether each pixel point in the sample medical image belongs to the over-segmentation region is determined, so as to output a prediction result second sample probability image, wherein the pixel value of each pixel point in the second sample probability image is a probability that the pixel point belongs to the over-segmentation region, that is, the pixel value in the second sample probability image is a probability value that the pixel point belongs to the over-segmentation region.

For the second sample probability image, if the pixel value (probability value) corresponding to the pixel point is larger, the probability that the pixel point belongs to the excessive region is higher; conversely, if the pixel value (probability value) corresponding to the pixel point is smaller, the probability that the pixel point belongs to the excessive region is lower.

Step 307, training a second image segmentation network based on the second sample probability image and the segmentation mask.

In order to make the second image segmentation network accurately predict the over-segmentation region in the sample medical image, the over-segmentation mask corresponding to the sample medical image needs to be used as weak supervision information, and supervision training is performed on the second image segmentation network, so in one possible implementation, the segmentation loss in the over-segmentation region prediction process can be determined according to the second sample probability image and the over-segmentation mask, and the second image segmentation network is trained based on the segmentation loss.

In one illustrative example, the segmentation penalty of the second image segmentation network may be expressed as:

wherein,,

representing segmentation loss of the second image segmentation network, < >>

Representing a second sample probability image, C representing an over-division mask,>

representing the second sample probability image and the over-division mask in the whole image domain>

Soft Dice loss on (soft Dice).

As shown in fig. 5, a schematic diagram of a training process of an image segmentation model according to an exemplary embodiment of the present application is shown. The image segmentation model 500 comprises a first image segmentation network 501 and a second image segmentation network 502, and a sample medical image 503 is respectively input into the first image segmentation network 501 and the second image segmentation network 502 to obtain a first sample probability image 504 output by the first image segmentation network 501 and a second sample probability image 505 output by the second image segmentation network 502; based on the under-segmentation mask 506 corresponding to the first sample probability image 504 and the sample medical image 503, determining an under-segmentation loss 507, and training the first image segmentation network 501 according to the under-segmentation loss 507; based on the second sample probability image 505 and the corresponding over-segmentation mask 508 of the sample medical image 503, an over-segmentation penalty 509 is determined, and the second image segmentation network 502 is trained on the over-segmentation penalty 509.

In the embodiment, the under-segmentation area is determined through long diameter labeling and short diameter labeling in the sample diameter labeling so as to generate a corresponding under-segmentation mask; determining an over-dividing region through long diameter marks in the sample diameter marks to generate a corresponding over-dividing mask, so that the under-dividing region, the sample dividing region and the over-dividing region meet a specific relation; in addition, the under-segmentation area prediction is carried out on the sample medical image through the first image segmentation network, a first sample probability image is obtained, and the first image segmentation network is trained through the first sample probability image and the under-segmentation mask, so that the first image segmentation network has the under-segmentation area prediction function; and meanwhile, performing excessive region prediction on the sample medical image through a second image segmentation network to obtain a second sample probability image, and training the second image segmentation network through the second sample probability image and an excessive mask so that the second image segmentation network has a prediction function of an excessive region.

In the above embodiment, when the first image segmentation network and the second image segmentation network are trained by using different losses, the training processes of the two image segmentation networks are independent of each other and cannot learn each other, so in one possible implementation, in order to connect the first image segmentation network and the second image segmentation network, a consistency loss is added in the training process.

Referring to fig. 6, a flowchart of an image segmentation method according to another exemplary embodiment of the present application is shown. The embodiment is exemplified by taking the computer device as an execution subject of the method, and the method includes:

step 601, determining a long diameter label and a short diameter label corresponding to a sample segmentation area in a sample medical image based on the sample diameter label.

Step 602, determining an under-segmentation mask corresponding to the sample medical image based on the long-diameter label and the short-diameter label.

Step 603, determining an over-division mask corresponding to the sample medical image based on the long-diameter label.

Step 604, inputting the sample medical image into a first image segmentation network to obtain a first sample probability image output by the first image segmentation network, wherein the first sample probability image comprises probability values of each pixel point in the sample medical image belonging to an undersegmented region.

The implementation manners of steps 601 to 604 may refer to the above embodiments, and the description of this embodiment is omitted here.

Step 605 determines a first segmentation loss based on the first and second sample probability images.

The second sample probability image is obtained by carrying out over-segmentation region prediction on the sample medical image through a second image segmentation network, and the second sample probability image comprises probability values of over-segmentation regions of all pixel points in the sample medical image.

In addition to using the under-segmentation prediction loss between the first sample probability image and the under-segmentation mask when training the first image segmentation network, the present embodiment introduces the prediction loss between the first sample probability image and the second sample probability image in order to allow the first image segmentation network and the second image segmentation network to learn each other.

The prediction functions of the first image segmentation network and the second image segmentation network are different, but the prediction results of the prediction functions of the first image segmentation network and the second image segmentation network are basically related to the sample segmentation areas, so in one possible implementation, after the first sample probability image output by the first image segmentation network and the second sample probability image output by the second image segmentation network are acquired, the first segmentation loss can be determined according to the first sample probability image and the second sample probability image and used for training the first image segmentation network and the second image segmentation network so that the first image segmentation network and the second image segmentation network learn each other.

In an illustrative example, the calculation formula of the first segmentation loss may be expressed as:

wherein,,

representing the first segmentation loss,/- >

Representing a first sample probability image, < >>

Representing a second sample probability image,/for>

Representing the first and second sample probability images in the entire image field +.>

Soft Dice loss on (soft Dice).

Alternatively, the first segmentation loss may use cross entropy loss in addition to soft position loss, and the specific calculation formula of the first segmentation loss in the embodiment of the present application is not limited.

As can be seen from equation (4), the segmentation loss between the first sample probability image and the second sample probability image is calculated over the whole image domain, but for the under-segmented region prediction and the over-segmented region prediction, the under-segmented region and the over-segmented region should be focused in the loss determination process, and the consistency loss is applied to the regions other than the under-segmented region and the over-segmented region, which may instead introduce unnecessary factors, thereby affecting the prediction accuracy of the segmentation network, and therefore, in order to further improve the prediction accuracy of the segmentation network, in one possible embodiment, the consistency loss is applied only to a specific region.

In an illustrative example, step 605 may also include step 605A and step 605B.

Step 605A, determining a first region image of the target location region in the first sample probability image and a second region image of the target location region in the second sample probability image.

In one possible embodiment, in determining the first segmentation loss, a consistency loss is applied only for the region images at the target position region (specific region) in the first sample probability image and the second sample probability image, corresponding to the first region image of the target position region in the first sample probability image and the second region image of the target position region in the second sample probability image that need to be determined.

The target position region is a region which is irrelevant to the under-divided region prediction and the over-divided region prediction.

In one illustrative example, the process of determining the first region image and the second region image may include the following steps 1 to 3.

1. And determining the over-segmentation area outside the under-segmentation area as a target position area.

In the present embodiment, the specific region (target position region) is set as a portion outside the sub-divided region, i.e., a portion outside the quadrangular region in the circular region, in the sub-divided region.

Since the under-segmented region is smaller than the sample-segmented region and smaller than the over-segmented region:

Then, for any pixel, if the pixel belongs to the under-divided region, the pixel value of the pixel in the under-divided mask, the over-divided mask and the sample divided mask (the mask for indicating the sample divided region) is 1:

Q _p ＝M _p ＝C _p =1; if any pixel does not belong to the over-segmentation region, the pixel value of the pixel in the under-segmentation mask, the over-segmentation mask and the sample segmentation mask is 0:

Q _p ＝M _p ＝C _p =0; thus, it can be concluded that: if any pixel belongs to the image domain except for the under-segmentation region or the over-segmentation region, the pixel values of the pixel in the under-segmentation mask, the over-segmentation mask and the sample segmentation mask are the same:

Q _p ＝M _p ＝C _p That is, the under-split mask, the over-split mask, and the sample split mask are uniform outside the circular region (over-split region) and inside the quadrangular region (under-split region).

Since the under-split mask, over-split mask, and sample split mask are consistent outside of the over-split region and inside of the under-split region, in order to avoid the consistency region adding unnecessary loss in the loss calculation process, in one possible embodiment, it is only necessary to apply a consistency loss in the region that does not have consistency, i.e., the target location region is an over-split region outside of the under-split region.

2. A first region image in the first sample probability image is determined based on the target location region.

In one possible implementation manner, after determining the target position area according to the under-segmented area and the over-segmented area corresponding to the sample medical image, the pixel point located in the target position area in the first sample probability image may be determined as the first area image.

3. A second region image in the second sample probability image is determined based on the target location region.

In one possible implementation manner, after determining the target position area according to the under-segmented area and the over-segmented area corresponding to the sample medical image, the pixel point located in the target position area in the second sample probability image may be determined as the first area image.

Step 605B, determining a first segmentation loss based on the first region image and the second region image.

In one possible embodiment, after the first region image and the second region image are determined, the first segmentation loss may be determined based on the first region image and the second region image.

In one illustrative example, a calculation formula for applying a consistency loss to a target location area may be expressed as:

Wherein,,

representing the first segmentation loss,/->

Representing a first sample probability image, < >>

Representing a second sample probability image,/for>

Representing the first and second sample probability images in the image domain +.>

Soft Dice loss on (soft Dice), image domain +.>

I.e. representing the target location area.

Step 606, determining a second segmentation penalty based on the first sample probability image and the under-segmentation mask.

As can be seen from equation (2), in order to provide the first image segmentation network with an under-segmentation region prediction function, in one possible implementation, a second segmentation penalty is determined from the first sample probability image and the under-segmentation mask to train the first image segmentation network.

Step 607 trains the first image segmentation network based on the first segmentation loss and the second segmentation loss.

In one illustrative example, after introducing a consistency loss in training the first image segmentation network, the total loss of the first image segmentation network may be expressed as:

wherein,,

representing the total loss of the first image segmentation network, < >>

Representing the loss between the under-segmentation mask and the first sample probability image,

Representing the first sample probability image and the second sample probability image in the image domain

Loss on lambda is the super parameterThe value range can be [0.2,0.8]。

The consistency loss in equation (6) is over the entire image domain

As calculated above, in order to avoid the influence of the consistency region on the loss, the consistency loss is applied only to the target position region, and the loss of the corresponding reconstruction formula (6) is:

wherein,,

representing a loss of the first image segmentation network, +.>

Representing the first sample probability image and the second sample probability image in the image domain +.>

Loss on (target location area), lambda being a superparameter, can take the value range of [0.2,0.8]。

From equations (6) and (7), in one possible implementation, the first image segmentation network may be trained jointly from the first segmentation loss and the second segmentation loss; the first segmentation loss can be selected from the formula (4) and the formula (5) according to actual requirements.

Step 608, inputting the sample medical image into a second image segmentation network to obtain a second sample probability image output by the second image segmentation network, wherein the second sample probability image comprises probability values of each pixel point in the sample medical image belonging to an excessive region.

The implementation of step 608 may refer to the above embodiments, which are not described herein.

Step 609, determining a first segmentation loss based on the first sample probability image and the second sample probability image.

The first sample probability image is obtained by carrying out under-segmentation region prediction on the sample medical image through a first image segmentation network, and the first sample probability image comprises probability values of the under-segmentation region of each pixel point in the sample medical image.

Similar to the training process of the first image segmentation network, in the process of training the second image segmentation network, a first sample probability image and a first segmentation loss between the sample probability images can be introduced, so that the first image segmentation network and the second image segmentation network can learn each other; in one possible implementation, the first segmentation loss may be determined from the first sample probability image and the second sample probability image.

The determining process of the first segmentation loss may refer to the above embodiment, and this embodiment is not described herein.

A third segmentation penalty is determined based on the second sample probability image and the segmentation mask, step 610.

As can be seen from equation (3), in order to provide the second image segmentation network with an over-segmentation region prediction function, in one possible implementation, a third segmentation penalty is determined from the second sample probability image and the over-segmentation mask to train the second image segmentation network.

Step 611, training a second image segmentation network based on the first segmentation loss and the third segmentation loss.

In one illustrative example, after introducing a consistency loss in training the second image segmentation network, the total loss of the second image segmentation network may be expressed as:

wherein,,

representing the total loss of the first image segmentation network, < >>

Representing the loss between the separation mask and the second sample probability image,/for the second sample probability image>

The loss of the component lambda is a super parameter, and the value range of the component lambda can be 0.2,0.8]。

As shown in fig. 7, a schematic diagram of a training process of an image segmentation model according to another exemplary embodiment of the present application is shown. The image segmentation model 700 comprises a first image segmentation network 701 and a second image segmentation network 702, and the sample medical image 703 is respectively input into the first image segmentation network 701 and the second image segmentation network 702 to obtain a first sample probability image 704 output by the first image segmentation network 701 and a second sample probability image 705 output by the second image segmentation network 702; determining a second segmentation penalty 707 based on the under-segmentation mask 706 corresponding to the first sample probability image 704 and the sample medical image 703, and determining a first segmentation penalty 710 based on the first sample probability image 704 and the second sample probability image 705, thereby training the first image segmentation network 701 based on the first segmentation penalty 710 and the second segmentation penalty 707; similarly, a third segmentation loss 709 is determined based on the second sample probability image 705 and the corresponding segmentation mask 708 of the sample medical image 703, and the second image segmentation network 702 is trained according to the third segmentation loss 709 and the first segmentation loss 710; wherein the first segmentation penalty 710 is the segmentation penalty of the calculated first sample probability image 704 and the second sample probability image 705 over the full image domain.

The consistency loss in equation (8) is over the entire image domain

As calculated above, in order to avoid the influence of the consistency region on the loss, the consistency loss is applied only to the target position region, and the loss corresponding to the reconstruction formula (8) is:

wherein,,

representing the total loss of the first image segmentation network, < >>

As shown in fig. 8, a schematic diagram of a training process of an image segmentation model according to another exemplary embodiment of the present application is shown. The image segmentation model 800 includes a first image segmentation network 801 and a second image segmentation network 802, and a sample medical image 803 is input into the first image segmentation network 801 and the second image segmentation network 802, respectively, to obtain a first sample probability image 804 output by the first image segmentation network 801 and a second sample probability image 805 output by the second image segmentation network 802; determining a second segmentation loss 807 based on the under-segmentation mask 806 corresponding to the first sample probability image 804 and the sample medical image 803; in determining the first segmentation loss 812, first a first region image 810 of the target location region in the first sample probability image 804 and a second region image 811 of the target location region in the second sample probability image 805 are determined; and determining a first segmentation loss 812 from the first region image 810 and the second region image 811; further training the first image segmentation network 801 according to the first 812 and second 807 segmentation losses; similarly, a third segmentation loss 809 is determined based on the second sample probability image 805 and the corresponding segmentation mask 808 of the sample medical image 803, and the second image segmentation network 802 is trained according to the third segmentation loss 809 and the first segmentation loss 812; wherein the first segmentation loss 812 is the computed segmentation loss of the first sample probability image 804 and the second sample probability image 805 over the target location area.

From equations (8) and (9), in one possible implementation, the second image segmentation network may be trained jointly from the first segmentation loss and the third segmentation loss; the first segmentation loss can be selected from the formula (4) and the formula (5) according to actual requirements.

In the embodiment, by increasing the consistency loss in the process of training the first image segmentation network and the second image segmentation network, the first image segmentation network and the second image segmentation network can learn each other while independently predicting, and the prediction accuracy of the first image segmentation network and the second image segmentation network is improved; in addition, the consistency loss is only applied to the target position area, so that the additional influence of the consistency area can be avoided, and the prediction accuracy of the image segmentation network can be further improved.

The above embodiments mainly describe the training process of the image segmentation network, when the model training is performed, the first image segmentation network with the under-segmentation region prediction function and the second image segmentation network with the over-segmentation prediction function can be obtained, and then the first image segmentation network and the second image segmentation network can be deployed in a isomorphic manner, so as to perform image segmentation on the medical image, and determine the segmentation region corresponding to the medical image.

Referring to fig. 9, a flowchart of an image segmentation method according to an exemplary embodiment of the present application is shown. The embodiment is exemplified by taking the computer device as an execution subject of the method, and the method includes:

step 901, performing under-segmentation region prediction on a target segmentation region in a target medical image to obtain a first target probability image, wherein the first target probability image comprises probability values of each pixel point belonging to the under-segmentation region in the target medical image, and the under-segmentation region is smaller than the target segmentation region.

The image segmentation model in the embodiment of the application comprises two sub-networks: the image segmentation system comprises a first image segmentation network and a second image segmentation network, wherein the first image segmentation network is used for carrying out undersegmentation on a segmentation area in a medical image, the second image segmentation network is used for carrying out oversegmentation on the segmentation area in the medical image, and then a segmentation result of the segmentation area is determined through model integration.

Similar to the model training process, in the model application process, the first image segmentation network and the second image segmentation network are also required to respectively predict the segmentation areas of the target medical image; in one possible implementation manner, a target medical image is input into a first image segmentation network, under-segmentation region prediction is performed on a target segmentation region in the target medical image by the first image segmentation network, and the probability that each pixel point in the target medical image belongs to the under-segmentation region is determined, so that an under-segmentation region prediction result is obtained: a first target probability image.

Step 902, performing over-segmentation region prediction on a target segmentation region in the target medical image to obtain a second target probability image, wherein the second target probability image comprises the probability that each pixel point in the target medical image belongs to the over-segmentation region, and the over-segmentation region is larger than the target segmentation region.

In one possible implementation manner, the target medical image is input into a second image segmentation network, the second image segmentation network performs over-segmentation region prediction on a target segmentation region in the target medical image, and the probability that each pixel point in the target medical image belongs to the over-segmentation region is determined, so that an over-segmentation region prediction result is obtained: a second target probability image.

Similar to the first image segmentation network training process, the relationship between the under-segmented region, the target segmented region, and the over-segmented region is: the under-divided region is included in the target divided region, and the target divided region is included in the over-divided region.

It should be noted that, step 903 of step 902 may be performed simultaneously, or step 902 may be performed first, and then step 903 may be performed. Or step 903 is performed before step 902 is performed.

Step 903, determining a target segmentation mask corresponding to the target medical image based on the first target probability image and the second target probability image.

Since the first target probability image is an under-segmented region prediction result of a target segmented region in the target medical image, and the second target probability image is an over-segmented region prediction result of the target segmented region in the target medical image, and the under-segmented region and the over-segmented region are different from the target segmented region, it is obvious that the single prediction result is not an accurate segmentation result of the target medical image, so that in order to obtain an accurate segmentation result of the target medical image, in a possible implementation, a target segmentation mask corresponding to the target medical image is determined based on the first target probability image and the second target probability image through model integration.

The target segmentation mask may explicitly indicate a target segmentation region in the target medical image.

In the embodiment of the application, in the model training process, by using sample diameter labels corresponding to sample segmentation areas in sample medical images, monitoring information for training a first image segmentation network and a second image segmentation network is constructed: the under-segmentation mask and the over-segmentation mask enable the first image segmentation network to have an under-segmentation region prediction function, and the second image segmentation network to have an over-segmentation region prediction function, so that the trained first image segmentation network and second image segmentation network can be used for segmentation prediction of a segmentation region in a medical image; compared with the prior art that sample segmentation contour labeling of a sample segmentation area is required to be used as supervision information, the method and the device for generating the weak supervision information by using sample diameter labeling can reduce manual labeling operation, enable large-scale training sample data to be obtained through fewer manual labeling, improve efficiency of generating the supervision information corresponding to a training image segmentation network, and further improve model training efficiency.

Since the first target probability image is an under-segmented region prediction result of a target segmented region in the target medical image and the second target probability image is an over-segmented region prediction result of the target segmented region in the target medical image, based on a relationship among the under-segmented region, the over-segmented region and the target segmented region, in a possible implementation manner, an accurate segmentation result of the target medical image may be obtained by performing a pixel value averaging operation on the first target probability image and the second target probability image.

Referring to fig. 10, a flowchart of an image segmentation method according to another exemplary embodiment of the present application is shown. The embodiment is exemplified by taking the computer device as an execution subject of the method, and the method includes:

step 1001, performing under-segmentation region prediction on a target segmentation region in a target medical image to obtain a first target probability image, where the first target probability image includes probability values of each pixel point in the target medical image belonging to the under-segmentation region, and the under-segmentation region is smaller than the target segmentation region.

Step 1002, performing over-segmentation region prediction on a target segmentation region in the target medical image to obtain a second target probability image, where the second target probability image includes probabilities that each pixel point in the target medical image belongs to the over-segmentation region, and the over-segmentation region is larger than the target segmentation region.

The implementation of step 1001 and step 1002 may refer to the above embodiments, and this embodiment is not described herein.

In step 1003, a target segmentation probability image is generated based on the first target probability image and the second target probability image.

In one possible implementation, the result is predicted by an integrated model: the first target probability image and the second target probability image can obtain a target segmentation probability image corresponding to the target segmentation area, and then a corresponding target segmentation mask is generated according to the target segmentation probability image.

In one illustrative example, the relationship between the first target probability image, the second target probability image, and the target segmentation probability image may be expressed as:

wherein,,

representing a target segmentation probability image (prediction result of target segmentation region),>

representing a first target probability image,

Representing a second target probability image.

As can be seen from the formula (10), in one possible implementation manner, the target segmentation probability image corresponding to the target segmentation region may be obtained by performing an averaging process on the pixel values corresponding to the same pixel point in the first target probability image and the second target probability image.

Step 1004, determining a target segmentation mask corresponding to the target medical image based on the target segmentation probability image.

The pixel value of each pixel point in the target segmentation probability image may represent the probability that the pixel point belongs to the target segmentation region, and in order to better distinguish the target segmentation region from other regions, in one possible implementation manner, the target segmentation probability image may be subjected to binarization processing to obtain a target segmentation mask containing two pixel values.

In an illustrative example, the process of binarizing the target segmentation probability image may include steps of 1004A to 1004C.

In step 1004A, in a case where the pixel value of the pixel point in the target segmentation probability image is higher than the probability threshold value, the pixel value of the pixel point is determined as the first pixel value.

Since the pixel value of each pixel point in the target segmentation probability image may represent the probability that the pixel point belongs to the target segmentation area, and the higher the pixel value (probability value), the more likely the pixel point belongs to the target segmentation area, whereas the lower the pixel value (probability value) is, the lower the probability that the pixel point belongs to the target segmentation area is, in one possible implementation manner, a probability threshold is set, the target segmentation probability image is subjected to binarization processing through the probability threshold, and if the pixel value of the pixel point in the target segmentation probability image is higher than the probability threshold, the pixel value of the pixel point belongs to the target segmentation area, and the pixel value of the pixel point may be set as the first pixel value.

Illustratively, the probability threshold may be 0.5 and the first pixel value may be 1.

In step 1004B, in the case that the pixel value of the pixel point in the target segmentation probability image is lower than the probability threshold value, the pixel value of the pixel point is determined as the second pixel value.

Otherwise, if the pixel value of the pixel point in the target segmentation probability image is lower than the probability threshold value, which indicates that the pixel point does not belong to the target segmentation region, the pixel value of the pixel point can be set to be a second pixel value.

Illustratively, the second pixel value may be 0.

In step 1004C, a target segmentation mask is generated based on the first pixel value and the second pixel value.

In one possible implementation manner, after the target segmentation probability image is subjected to binarization processing, the pixel values of each pixel point are divided into a first pixel value and a second pixel value, an image formed by the first pixel value and the second pixel value is a target segmentation mask, and the target segmentation mask shows that a set of pixel points with the pixel values being the first pixel value is a target segmentation region of the target medical image.

As shown in fig. 11, which illustrates a schematic diagram of an image segmentation process illustrated in an exemplary embodiment of the present application. The target medical image 1103 is respectively subjected to a first image segmentation network 1101 and a second image segmentation network 1102 in the image segmentation model 1100, so as to obtain a first target probability image 1104 output by the first image segmentation network 1101 and a second target probability image 1105 output by the second image segmentation network 1102; and a target segmentation probability image 1106 is generated according to the first target probability image 1104 and the second target probability image 1105, and the target segmentation mask 1107 corresponding to the target medical image 1103 can be obtained by performing binarization processing on the target segmentation probability image 1106.

In step 1005, a target segmentation image corresponding to the target segmentation region is generated based on the target segmentation mask and the target medical image.

In one possible implementation manner, if the target segmented image needs to be segmented from the target medical image, the target medical image may be multiplied based on the target segmented mask, so as to obtain the target segmented image corresponding to the target segmented region.

In this embodiment, the result is predicted by the integrated model: the first target probability image and the second target probability image can obtain a target segmentation probability image corresponding to the target segmentation area, and then a corresponding target segmentation mask is generated according to the target segmentation probability image, so that the image segmentation purpose of the target segmentation area in the target medical image is achieved.

As shown in table one, it shows the comparison result of the segmentation accuracy of the related technical solution and the image segmentation method provided in the present application on two data sets of the kit 19 and the LiTS17 (the higher the numerical value, the higher the segmentation accuracy).

List one

As can be seen from the table, the segmentation performance of the image segmentation method provided by the embodiment of the application on the two public data sets exceeds the comparison scheme, and when the U-Net and Swin-transducer are adopted in the image segmentation network, the segmentation performance of the embodiment of the application can be close to the full supervision upper limit.

As shown in table two, the sensitivity (%) under 0.5 pseudo-yang/CT images (FPs/scan) was used as a detection evaluation index, and the comparison results (the higher the numerical value, the more accurate the detection result) of the image segmentation methods provided in the related art and the present application were obtained.

Watch II

Method	Sensitivity
		3DCE，27Slices	62.48
Deformable Faster-RCNN+VA	69.1
		3DCE+CS_Att，21slices	71.4
Anchor-Free RPN	68.73
		FPN+MSB(weights sharing)	67.0
Improved RetinaNet	72.15
		MVP-Net，9slices	73.83
MULAN	76.12
		MR-CNN	75.92
RECIST-Net	76.14
		Oours(with Swin Transformer)	76.43

It can be seen from table two that, although the image segmentation method provided in the embodiment of the present application is not specifically designed for lesion detection, the detection result converted from the segmentation result obtained by using the image segmentation method provided in the embodiment of the present application still exceeds each comparison scheme.

In order to further verify the effectiveness of the image segmentation method provided in the application embodiment, table three is the result of an ablation experiment on the LiTS17 dataset.

Watch III

Table four is the results of the ablation experiments on the kit 19 dataset.

Table four

As can be seen from tables three and four, the experiments on both experimental data sets illustrate the addition of

Significantly promoteSplit representation of the network, while region-restricted +.>

The segmentation performance of the network is further improved. In addition, the segmentation performance of the technical scheme on the KiTS19 data set approaches the full supervision upper limit.

Furthermore, the sensitivity of the present solution to the hyper-parameter λ was examined by testing the influence of different λ values on the segmentation result in the range [0.2,0.8 ]. FIG. 12 is a graph of the results of a test of the effect of lambda values on the segmentation results on the data set KiTS19, wherein the horizontal axis represents lambda values, the vertical axis represents the Score (Score coefficient) of the measurement segmentation results, and the broken line 1210 represents the corresponding segmentation results at different lambda values when the image segmentation network is Swin transducer; fold line 1220 is the corresponding segmentation result at different lambda values when the image segmentation network is U-Net; FIG. 13 is a graph showing the results of a test of the effect of lambda values on the segmented results on the dataset LiTS17, wherein the horizontal axis represents lambda values, the vertical axis represents the Dice coefficient for measuring the segmented results, and the broken line 1310 is the segmented result corresponding to the different lambda values when the image segmentation network is Swin transducer; the broken line 1320 is a segmentation result corresponding to different lambda values when the image segmentation network is U-Net; as can be seen from fig. 12 and fig. 13, the image segmentation method provided in the embodiment of the present application is insensitive to the value of the super parameter λ within a reasonable range.

The following are device embodiments of the present application, reference being made to the above-described method embodiments for details of the device embodiments that are not described in detail.

Fig. 14 is a block diagram of an image segmentation apparatus according to an exemplary embodiment of the present application. The apparatus may include:

a first segmentation prediction module 1401, configured to perform under-segmentation region prediction on a target segmentation region in a target medical image, so as to obtain a first target probability image, where the first target probability image includes probability values of each pixel point in the target medical image belonging to the under-segmentation region, and the under-segmentation region is smaller than the target segmentation region;

a second segmentation prediction module 1402, configured to perform an over-segmentation region prediction on the target segmentation region in the target medical image, to obtain a second target probability image, where the second target probability image includes a probability that each pixel point in the target medical image belongs to an over-segmentation region, and the over-segmentation region is greater than the target segmentation region;

a first determining module 1403 is configured to determine a target segmentation mask corresponding to the target medical image based on the first target probability image and the second target probability image.

Optionally, the first determining module 1403 includes:

a generation unit configured to generate a target segmentation probability image based on the first target probability image and the second target probability image;

and the first determining unit is used for determining a target segmentation mask corresponding to the target medical image based on the target segmentation probability image.

Optionally, the first determining unit is further configured to:

determining the pixel value of the pixel point as a first pixel value under the condition that the pixel value of the pixel point in the target segmentation probability image is higher than a probability threshold value;

determining the pixel value of the pixel point as a second pixel value under the condition that the pixel value of the pixel point in the target segmentation probability image is lower than the probability threshold value;

the target segmentation mask is generated based on the first pixel value and the second pixel value.

Optionally, the first generating unit is further configured to:

and carrying out average processing on pixel values corresponding to the same pixel point in the first target probability image and the second target probability image to obtain the target segmentation probability image.

Optionally, the apparatus further includes:

and the generation module is used for generating a target segmentation image corresponding to the target segmentation area based on the target segmentation mask and the target medical image.

Fig. 15 is a block diagram of an image segmentation apparatus according to another exemplary embodiment of the present application. The device comprises:

A second determining module 1501, configured to determine, based on a sample diameter label corresponding to a sample segmentation region in a sample medical image, an under-segmentation mask and an over-segmentation mask corresponding to the sample medical image, where the under-segmentation region corresponding to the under-segmentation mask is smaller than the sample segmentation region, and the over-segmentation region corresponding to the over-segmentation mask is larger than the sample segmentation region;

a first training module 1502 for training a first image segmentation network based on the sample medical image and the under-segmentation mask, the first image segmentation network for under-segmentation region prediction of the sample medical image;

a second training module 1503 is configured to train a second image segmentation network based on the sample medical image and the over-segmentation mask, the second image segmentation network being configured to perform over-segmentation region prediction on the sample medical image.

Optionally, the second determining module 1501 includes:

the second determining unit is used for determining a long-diameter label and a short-diameter label corresponding to the sample segmentation area in the sample medical image based on the sample diameter label, wherein the long-diameter label is a label of the longest diameter corresponding to the sample segmentation area, and the short-diameter label is a label of the shortest diameter corresponding to the sample segmentation area;

The third determining unit is used for determining the under-segmentation mask corresponding to the sample medical image based on the long-diameter label and the short-diameter label;

and a fourth determining unit, configured to determine the over-division mask corresponding to the sample medical image based on the long diameter label.

Optionally, the third determining unit is further configured to:

determining the under-segmented region in the sample medical image based on the long diameter annotation and the short diameter annotation;

setting the pixel value of the pixel point which belongs to the undersegmented area in the medical sample image as a first pixel value, and setting the pixel value of the pixel point which does not belong to the undersegmented area in the medical sample image as a second pixel value, so as to generate the undersegmented mask corresponding to the sample medical image.

Optionally, the fourth determining unit is further configured to:

determining the over-segmentation region in the sample medical image based on the long diameter annotation;

setting the pixel value of the pixel point which belongs to the over-segmentation area in the medical sample image as a first pixel value, and setting the pixel value of the pixel point which does not belong to the over-segmentation area in the medical sample image as a second pixel value, so as to generate the over-segmentation mask corresponding to the sample medical image.

Optionally, the first training module 1502 includes:

the first segmentation prediction unit is used for inputting the sample medical image into the first image segmentation network to obtain a first sample probability image output by the first image segmentation network, wherein the first sample probability image comprises probability values of each pixel point in the sample medical image belonging to the underspected region;

and a first training unit, configured to train the first image segmentation network based on the first sample probability image and the under-segmentation mask.

Optionally, the apparatus further includes:

a third determining module, configured to determine a first segmentation loss based on the first sample probability image and a second sample probability image, where the second sample probability image is obtained by performing over-segmentation region prediction on the sample medical image by using the second image segmentation network, and the second sample probability image includes probability values that each pixel point in the sample medical image belongs to the over-segmentation region;

the first training unit is further configured to:

determining a second segmentation loss based on the first sample probability image and the under-segmentation mask;

the first image segmentation network is trained based on the first segmentation loss and the second segmentation loss.

Optionally, the second training module 1503 includes:

the second segmentation prediction unit is used for inputting the sample medical image into the second image segmentation network to obtain a second sample probability image output by the second image segmentation network, wherein the second sample probability image comprises probability values of each pixel point in the sample medical image belonging to the over-segmentation region;

and a second training unit for training the second image segmentation network based on the second sample probability image and the segmentation mask.

Optionally, the apparatus further includes:

a fourth determining module, configured to determine a first segmentation loss based on a first sample probability image and the second sample probability image, where the first sample probability image is obtained by performing under-segmentation region prediction on the sample medical image by using the first image segmentation network, and the first sample probability image includes probability values that each pixel point in the sample medical image belongs to the under-segmentation region;

the second training unit is further configured to:

determining a third segmentation loss based on the second sample probability image and the over segmentation mask;

the second image segmentation network is trained based on the first segmentation loss and the third segmentation loss.

Optionally, the fourth determining module includes:

a fifth determining unit configured to determine a first area image of a target position area in the first sample probability image and a second area image of the target position area in the second sample probability image;

a sixth determination unit configured to determine the first segmentation loss based on the first region image and the second region image.

Optionally, the fifth determining unit is further configured to:

determining the over-segmented region outside the under-segmented region as the target position region;

determining the first region image in the first sample probability image based on the target location region;

the second region image in the second sample probability image is determined based on the target location region.

Referring to fig. 16, a schematic structural diagram of a computer device according to an embodiment of the present application is shown, where the computer device may be used to implement the image segmentation method performed by the computer device according to the embodiment. The computer apparatus 1600 includes a central processing unit (CPU, central Processing Unit) 1601, a system Memory 1604 including a random access Memory (RAM, random Access Memory) 1602 and a Read-Only Memory (ROM) 1603, and a system bus 1605 connecting the system Memory 1604 and the central processing unit 1601. The computer device 1600 also includes a basic Input/Output system (I/O) 1606 to facilitate transfer of information between the various devices within the computer, and a mass storage device 1607 for storing an operating system 1613, application programs 1614, and other program modules 1615.

The basic input/output system 1606 includes a display 1608 for displaying information and an input device 1609, such as a mouse, keyboard, etc., for user input of information. Wherein the display 1608 and the input device 1609 are connected to the central processing unit 1601 by way of an input/output controller 1610 connected to the system bus 1605. The basic input/output system 1606 may also include an input/output controller 1610 for receiving and processing input from a keyboard, mouse, or electronic stylus, among a plurality of other devices. Similarly, the input/output controller 1610 also provides output to a display screen, printer, or other type of output device.

The mass storage device 1607 is connected to the central processing unit 1601 by a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1607 and its associated computer-readable media provide non-volatile storage for the computer device 1600. That is, the mass storage device 1607 may include a computer readable medium (not shown) such as a hard disk or CD-ROM (Compact Disc Read-Only Memory) drive.

The computer readable medium may include computer storage media and communication media without loss of generality. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash Memory or other solid state Memory technology, CD-ROM, DVD (Digital Video Disc, high density digital video disc) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the one described above. The system memory 1604 and mass storage 1607 described above may be collectively referred to as memory.

According to various embodiments of the present application, the computer device 1600 may also operate through a network, such as the Internet, to remote computers connected to the network. That is, the computer device 1600 may be connected to the network 1612 through a network interface unit 1611 coupled to the system bus 1605, or alternatively, the network interface unit 1611 may be used to connect to other types of networks or remote computer systems (not shown).

The memory also includes one or more programs stored in the memory and configured to be executed by the one or more central processing units 1601.

The present application also provides a computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions loaded and executed by a processor to implement the image segmentation method provided by any of the above-described exemplary embodiments.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image segmentation method provided in the alternative implementation described above.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The foregoing description of the preferred embodiments is merely exemplary in nature and is in no way intended to limit the invention, since it is intended that all modifications, equivalents, improvements, etc. that fall within the spirit and scope of the invention.

Claims

1. An image segmentation method, the method comprising:

2. The method of claim 1, wherein the determining a target segmentation mask corresponding to the target medical image based on the first target probability image and the second target probability image comprises:

generating a target segmentation probability image based on the first target probability image and the second target probability image;

and determining a target segmentation mask corresponding to the target medical image based on the target segmentation probability image.

3. The method of claim 2, wherein the determining a target segmentation mask corresponding to the target medical image based on the target segmentation probability image comprises:

4. The method of claim 2, wherein the generating a target segmentation probability image based on the first target probability image and the second target probability image comprises:

5. The method according to any one of claims 1 to 4, further comprising:

and generating a target segmentation image corresponding to the target segmentation region based on the target segmentation mask and the target medical image.

6. An image segmentation method, the method comprising:

7. The method of claim 6, wherein determining the under-segmentation mask and over-segmentation mask corresponding to the sample medical image based on the sample diameter labels corresponding to the sample segmentation regions in the sample medical image comprises:

determining a long-diameter label and a short-diameter label corresponding to the sample segmentation region in the sample medical image based on the sample diameter label, wherein the long-diameter label is a label of the longest diameter corresponding to the sample segmentation region, and the short-diameter label is a label of the shortest diameter corresponding to the sample segmentation region;

determining the under-segmentation mask corresponding to the sample medical image based on the long-diameter label and the short-diameter label;

and determining the over-dividing mask corresponding to the sample medical image based on the long diameter label.

8. The method of claim 7, wherein the determining the under-segmentation mask corresponding to the specimen medical image based on the long-diameter annotation and the short-diameter annotation comprises:

9. The method of claim 7, wherein the determining the over-segmentation mask corresponding to the specimen medical image based on the long diameter labeling comprises:

10. The method according to any one of claims 6 to 9, wherein the training a first image segmentation network based on the sample medical image and the under-segmentation mask comprises:

Inputting the sample medical image into the first image segmentation network to obtain a first sample probability image output by the first image segmentation network, wherein the first sample probability image comprises probability values of each pixel point in the sample medical image belonging to the underspected region;

the first image segmentation network is trained based on the first sample probability image and the under-segmentation mask.

11. The method according to claim 10, wherein the method further comprises:

determining a first segmentation loss based on the first sample probability image and a second sample probability image, wherein the second sample probability image is obtained by carrying out excessive segmentation region prediction on the sample medical image by the second image segmentation network, and the second sample probability image comprises probability values of all pixel points in the sample medical image belonging to the excessive segmentation region;

the training the first image segmentation network based on the first sample probability image and the under-segmentation mask comprises:

12. The method according to any one of claims 6 to 9, wherein the training a second image segmentation network based on the sample medical image and the over-segmentation mask comprises:

inputting the sample medical image into the second image segmentation network to obtain a second sample probability image output by the second image segmentation network, wherein the second sample probability image comprises probability values of each pixel point in the sample medical image belonging to the excessive region;

the second image segmentation network is trained based on the second sample probability image and the over-segmentation mask.

13. The method according to claim 12, wherein the method further comprises:

determining a first segmentation loss based on a first sample probability image and a second sample probability image, wherein the first sample probability image is obtained by carrying out under-segmentation region prediction on the sample medical image by the first image segmentation network, and the first sample probability image comprises probability values of each pixel point in the sample medical image belonging to the under-segmentation region;

the training the second image segmentation network based on the second sample probability image and the over-segmentation mask, comprising:

14. The method of claim 13, wherein the determining a first segmentation loss based on the first sample probability image and the second sample probability image comprises:

determining a first area image of a target position area in the first sample probability image and a second area image of the target position area in the second sample probability image;

the first segmentation loss is determined based on the first region image and the second region image.

15. The method of claim 14, wherein said determining a first region image of a target location region in said first sample probability image and a second region image of said target location region in said second sample probability image comprises:

16. An image segmentation apparatus, the apparatus comprising:

17. An image segmentation apparatus, the apparatus comprising:

18. A computer device comprising a processor and a memory, wherein the memory has stored therein at least one program that is loaded and executed by the processor to implement the image segmentation method according to any one of claims 1 to 5 or to implement the image segmentation method according to any one of claims 6 to 15.

19. A computer-readable storage medium, wherein at least one program is stored in the readable storage medium, and the at least one program is loaded and executed by a processor to implement the image segmentation method according to any one of claims 1 to 5, or to implement the image segmentation method according to any one of claims 6 to 15.

20. A computer program product, the computer program product comprising computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executing the computer instructions, causing the computer device to perform the image segmentation method according to any one of claims 1 to 5 or to implement the image segmentation method according to any one of claims 6 to 15.