CN116630299A - A medical image processing method based on semi-supervised neural network - Google Patents

Abstract

The invention discloses a medical image processing method based on a semi-supervised neural network, which comprises the following steps of S1, acquiring a picture data set; s2, constructing and training a DFCPS model, wherein the DFCPS model comprises a data enhancement strategy and a neural network, and the neural network comprises a main network, an ASPP module, an up-sampling module and a Softmax function; the backbone network adopts ResNet-50, and introduces residual connection to construct a deep network, the ASPP module comprises four convolution layers, and the cavity convolution rates of the four convolution layers are respectively 1, 12, 24 and 36; s3, performing image segmentation by using the trained neural network. The method trains the model and obtains ideal results by reasonably utilizing unlabeled data and a small amount of labeled data and a data enhancement technology of strong enhancement and weak enhancement.

Description

A medical image processing method based on semi-supervised neural network

technical field

The invention relates to the technical fields of data enhancement and semantic segmentation, in particular to a medical image processing method based on a semi-supervised neural network.

Background technique

In medicine, in order to conduct a comprehensive diagnosis of the patient's disease stage to determine whether there is a disease in the organ and propose a corresponding treatment plan, the methods that are often used on the patient are: DR scan, CT scan or MRI (nuclear magnetic resonance) scan. The image is then handed over to the doctor to determine the pathological changes of the patient's internal organs through naked eye observation. Before the rise of deep learning, this process was often directly observed by experienced doctors.

Although doctors have a low misdiagnosis rate and high judgment accuracy, it takes a lot of time and money to train doctors who meet the above requirements, and doctors, as human beings, will be affected by factors such as mood swings and lack of energy for long hours of work, resulting in The instability of judgment accuracy. Therefore, in order to reduce the misdiagnosis rate, medical image segmentation to assist doctors in diagnosis came into being.

As an important tool in semi-supervised learning, neural networks have achieved remarkable results in medical image segmentation. Neural networks can learn complex feature representations from large-scale image data and optimize them through an end-to-end training process. However, in medical image segmentation, due to the limitation of labeled data and the complexity of the model, it is difficult to achieve satisfactory performance for neural networks that only rely on limited labeled data for supervised training. Therefore, the need for neural networks to deal with the above problems is imminent .

Self-training for medical images is a semi-supervised learning method, which can use unlabeled data to improve the performance of the model through continuous iterative training, and achieve the purpose of reducing the amount of labeled data. Self-training will have the characteristics of high pseudo-label noise, and the performance of self-training methods depends on the quality and reliability of pseudo-labels. Therefore, confidence and noise issues need to be carefully considered when selecting pseudo-labels. Due to the similarity of medical anatomical features, labeled images can convey strong reference points for matching and information transfer to unlabeled images.

The core idea of consistent learning is to perturb the original sample to different degrees, such as Gaussian noise, color change, random rotation, and the model still has a similar output to the original sample. Through consistent learning, the model can obtain additional training signals from unlabeled data and learn more robust and generalizable feature representations. However, if the model is not well optimized and provides wrong supervision information, the risk of soft and true labels colliding becomes high.

If the above two problems can be solved, it may be an effective breakthrough to improve the accuracy of medical image segmentation and recognition.

Contents of the invention

Aiming at the deficiencies of existing technologies, a medical image processing method based on semi-supervised neural network is proposed. By rationally using unlabeled data and a small amount of labeled data, as well as strong and weak enhanced data enhancement techniques, the model is trained and obtained ideal result.

In order to solve the problems of the technologies described above, the technical solution of the present invention is:

A medical image processing method based on a semi-supervised neural network, comprising the steps of:

S1. Acquiring a picture data set;

S2, build DFCPS model and train, described DFCPS model comprises data enhancement strategy and neural network, and described neural network comprises backbone network, ASPP module, upsampling module and Softmax function;

The backbone network adopts ResNet-50, and introduces residual connections to build a deep network. The ASPP module includes four convolutional layers, and the atrous convolution rates of the four convolutional layers are 1, 12, 24, and 36, respectively. ;

The training method of the DFCPS model is as follows: firstly, the original sample X is subjected to two different degrees of strong and weak data enhancement through the data enhancement strategy, and two sets of enhanced samples are generated, one group is a strong enhanced sample, and the other is a strong enhanced sample. Weakly enhanced samples, strong and weakly enhanced samples are grouped and combined, and the combined strong and weak enhanced samples are entered into four neural networks for training, and the loss value is minimized by continuously optimizing the parameters of the model;

During the training process, the model minimizes the loss value by continuously optimizing the parameters of the model to prevent over-fitting and under-fitting phenomena, so as to improve the performance and accuracy of the model. The loss design of the present invention involves two key loss functions in the whole neural network training process: supervision loss L _s and cross pseudo-supervision loss L _cps .

To sum up, the loss function in the whole training process mainly includes supervised loss and cross-pseudo-supervised loss. The supervised loss guides the learning of the network through the comparison between pseudo-labels of strongly augmented samples and weakly augmented samples. The cross-pseudo-supervised loss encourages the network to learn more consistent segmentation results as a whole by comparing the pseudo-segmentation maps generated between different groups. Such a training strategy helps to improve the performance of the model and enables the network to better adapt to different sets of data and labels. This design draws on the idea of Fixmatch in the method of strong and weak enhancement, and expands it accordingly to apply to the task of medical image label segmentation. However, special attention needs to be paid to the reliability, accuracy, and cleanliness of the pseudo-labels, since the quality of the pseudo-labels directly affects the performance and generalization ability of the model.

In order to ensure that the quality of the pseudo-label is high enough, the present invention introduces the concept of a confidence threshold, which is defined as μ in the present invention. By setting a confidence threshold, the model can filter out high-quality pseudo-labels, thereby avoiding negative impact on model training. Specifically, I compare the confidence of the pseudo-label against a confidence threshold. If the confidence of the pseudo-label is close to the confidence threshold, that is, around 0.5, it can be ignored, because such a pseudo-label may not be reliable enough, and the part of the pseudo-segmentation map greater than the confidence threshold μ is involved in the loss calculation. For strongly amplified samples, cross-entropy loss is also performed on the output prediction results and the corresponding pseudo-segmentation maps obtained from target weakly labeled samples that exceed the confidence threshold.

S3, use the trained neural network for image segmentation

S3-1. Using the preprocessed data set as input, obtain the feature map through the backbone network;

S3-2. Using the acquired feature map as input, the multi-scale comprehensive feature extraction is realized through multiple parallel branches through the ASPP module;

S3-3. Represent the obtained comprehensive features through upsampling technology to increase data details, map the output of the model to the probability of each category through the Softmax function, and then select the most likely category as the prediction result according to the probability to generate credibility High predictive samples.

Preferably, in the step S1, the picture data is pre-processed through a data enhancement strategy.

Preferably, the data enhancement strategy includes random rotation, random brightness, contrast adjustment, and random translation.

Preferably, in the step S2, during training, each group is used for strong and weak enhanced neural network sharing parameters and weights, and in each group, the pseudo-label generated by the prediction result output by the neural network after weak enhancement is used as each sample The target of strong augmented sample prediction.

Preferably, in the step S2, in the training process, the supervisory loss is introduced as L _s , and the information of the strong enhancement sample and the weak enhancement sample is used to guide the learning process of the DFCPS model, specifically, the strong enhancement sample is generated by the neural network The difference between the predicted results of the predicted results and the pseudo-labels of the corresponding weakly enhanced samples is used to calculate the supervised loss, and then iterate the model parameters.

In the above technical solution, the supervised loss L _s is generated by targeting the pseudo labels generated by the weakly enhanced samples as the targets of the strongly enhanced samples. Specifically, the supervised loss is computed using the difference between the predictions generated by the neural network for strongly augmented samples and the pseudo-labels of the corresponding weakly augmented samples. The purpose of this is to let the network learn the mapping relationship from weakly enhanced samples to strongly enhanced samples, so as to improve the prediction accuracy of the model on strongly enhanced samples. By minimizing the supervised loss, the network is encouraged to learn the correct segmentation target.

Preferably, in the step S2, the training process design introduces a cross pseudo-supervision loss L _cps so that the pseudo-labels generated by samples of different groups of weakly supervised versions will be mutually constrained.

In the above technical solution, the cross-pseudo-supervised loss _Lcps is used to constrain the difference between the pseudo-segmentation maps generated between different groups. This design considers the pseudo-labels generated by each set of weakly supervised version samples as the goal of training other sets of weakly supervised version samples, and uses the cross-entropy loss function to calculate the loss between the pseudo-labels. Through this cross-pseudo-supervision, the pseudo-labels between different groups are encouraged to influence and correct each other to improve the overall segmentation consistency.

Preferably, the supervised loss L _s is determined by a cross-entropy loss function, and normal supervised learning is performed on marked samples, and the expression is as follows:

Among them, ^Du : represents the collection of unlabeled samples, D ^l represents the collection of samples with labels, S is defined as the area of the input image, calculated by height*width, p _i is the corresponding confidence vector, y _j is the ground truth, l _ce is the cross entropy loss function.

Preferably, the cross-pseudo-supervised loss is defined as and /> where /> Represents the supervised loss of labeled data, /> Represents the supervised loss of unlabeled data, the expression is as follows:

So the cross-pseudo-supervised loss can be defined as

The total loss can be correspondingly defined as, where ω is the weight

Loss=L _s +ωL _cps (5).

In the above technical solution, an initial model is first trained using labeled samples with real labels. Second, use this initial model to make predictions on unlabeled samples, and add the predictions as pseudo-labels to these samples. Then unlabeled samples with pseudo-labels and labeled samples with real labels are combined into an expanded training set. Finally retrain the model using the expanded training set. Through repeated iterations, the model can gradually use the information in the unlabeled data to improve performance.

Preferably, the specific method of the step S3-1 is: using the trained DFCPS model, in the backbone network ResNet-50, using the preprocessed data set as input, using the pooling layer and the fully connected layer for feature extraction and classification. Build a deep network by introducing residual connections. The residual connection allows information to skip some layers directly, making it easier for the network to learn the identity mapping, avoiding the problem of deep network degradation. The network structure of ResNet-50 is relatively deep, including 50 convolutional layers, including multiple residual blocks (Residual Blocks). Each residual block consists of two convolutional layers, one of which is used to reduce the size of the feature map and the other convolutional layer is used to keep the size of the feature map unchanged.

Preferably, the specific method of the step S3-2 is: after obtaining the corresponding output, the sample will enter the ASPP module, use multiple parallel branches to realize multi-scale feature extraction, and increase the receptive field by using dilated convolution, The ASPP module can make full use of the context information of the input feature map, which can help the model better understand the semantic associations of different positions in the image, and can be flexibly embedded in different network architectures. Each branch adopts a different sampling rate (also known as hole rate or dilation rate), which determines the receptive field size of the branch on the input feature map. The sampling rate is positively correlated with the receptive field. By using strength-enhanced samples and an ASPP module, the neural network designed in the present invention can obtain global and local context information from different scales to improve the performance of image segmentation tasks.

Preferably, the specific method of step S3-3 is as follows: the final classification or prediction task provides a more discriminative feature representation, after convolution, batch normalization and ReLU function activation, all dilated convolution branches and The results of the spatial pyramid pooling branches are concatenated in parallel to form a comprehensive feature representation. Then apply a 1x1 convolutional layer to perform dimensionality reduction operations on the input features, thereby reducing the number of channels. This helps to reduce the computational burden. The number of channels of input features is reduced to smaller dimensions to reduce computational complexity and extract more abstract features. Next, the acquired comprehensive feature representation is upsampled to increase data details, thereby improving image quality. Upsampling is a processing technique used to upscale low-resolution or low-frequency data to a higher resolution or frequency. The comprehensive features after upsampling are connected with the underlying features after convolution processing to form a more comprehensive feature representation. After upsampling and final convolution, the input features are nonlinearly transformed and characterized by the operation of the convolution layer. extraction to produce higher-level feature representations. These operations help to capture more complex patterns and structures in the image, and provide a more discriminative feature representation for the final classification or prediction task. After the output of the model is mapped to the probability of each category through the Softmax function, then it can be According to the probability, the most probable class is selected as the prediction result to generate a prediction sample with high reliability.

The present invention has following characteristics and beneficial effect:

The invention trains the model and obtains ideal results by rationally utilizing unmarked data and a small amount of marked data, as well as data enhancement techniques of strong enhancement and weak enhancement. First, the original sample X is enhanced twice with different degrees of strong and weak data, and two sets of enhanced samples are generated: one is strong enhanced samples, the other is weakly enhanced samples, and the strong and weak enhanced samples are grouped and combined, where the combined The strong and weak enhancement samples enter into four different neural networks F(θ _n ) for training, and use the information of strong enhancement samples and weak enhancement samples to guide the learning process of the model and improve the performance of the model.

The backbone network used in the present invention is ResNet-50, which mainly solves the problems of gradient disappearance and gradient explosion in the deep neural network training process, and a deep network is constructed by introducing a residual connection (Residual Connection). The residual connection allows information to skip some layers directly, making it easier for the network to learn the identity mapping, avoiding the problem of deep network degradation. The network structure of ResNet-50 is relatively deep, including 50 convolutional layers, including multiple residual blocks (Residual Blocks). Each residual block consists of two convolutional layers, one of which is used to reduce the size of the feature map and the other convolutional layer is used to keep the size of the feature map unchanged. In addition, ResNet-50 also uses pooling layers and fully connected layers for feature extraction and classification. The network structure of ResNet-50 is very suitable for processing large-scale image classification tasks, and has strong expressive ability and generalization ability. It has been widely used in various tasks in the field of computer vision, such as image classification, object detection and semantic segmentation, etc., and has achieved remarkable results in multiple computer vision tasks

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 is a network architecture diagram of a DFCPS model in an embodiment of the present invention.

Figure 2 is a diagram of the neural network structure in the DFCPS model.

Figure 3 is a design diagram of the designed loss function.

Fig. 4 is a Loss trend diagram of an embodiment of the present invention.

Fig. 5 is a mIOU curve diagram of an embodiment of the present invention.

FIG. 6 is a comparison diagram of segmentation structures according to an embodiment of the present invention.

Fig. 7 is a mIOU comparison curve of the embodiment of the present invention.

Detailed ways

It should be noted that, in the case of no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

In describing the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", " The orientations or positional relationships indicated by "vertical", "horizontal", "top", "bottom", "inner" and "outer" are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and Simplified descriptions, rather than indicating or implying that the device or element referred to must have a particular orientation, be constructed and operate in a particular orientation, and thus should not be construed as limiting the invention. In addition, the terms "first", "second", etc. are used for descriptive purposes only, and should not be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, a feature defined as "first", "second", etc. may expressly or implicitly include one or more of that feature. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection. Connected, or integrally connected; it may be mechanically connected or electrically connected; it may be directly connected or indirectly connected through an intermediary, and it may be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention based on specific situations.

The invention provides a medical image processing method based on a semi-supervised neural network, comprising the steps of:

S1. Acquiring a picture data set;

Two data sets are provided in this embodiment, namely the PASCAL VOC 2012 data set and the Kvasir-SEG data set.

Among them, PASCAL VOC 2012 dataset: PASCAL VOC 2012 is a standard object-centric semantic segmentation dataset, which consists of more than 13,000 images, including 20 object classes and 1 background class. The standard training set, validation set and test set consist of 1464, 1449 and 1456 images respectively ^[3] . Covers a wide range of real-world scenes and object categories, such as people, vehicles, animals, furniture, etc. The images in the dataset are derived from a variety of sources, including Internet images, annotated images, and contributions from professional photographers. Each image has corresponding pixel-level annotations, which are used to mark the location and category of objects in the image. For object detection tasks, each object is labeled with a bounding box; for semantic segmentation tasks, each pixel is labeled as which category it belongs to. This design will use this dataset as a pre-training set to obtain pre-training weights.

The Kvasir-SEG dataset is based on the previous Kvasir dataset, the first multi-class dataset for gastrointestinal (GI) disease detection and classification, which contains a series of endoscopic images, including gastroscope and colon mirror image. These images cover a variety of common gastrointestinal disorders such as polyps, ulcers and cancer. Each image is equipped with pixel-level segmentation labels annotating the boundaries of different lesion regions in the image. The original Kvasir ^[6] dataset contains 8,000 GI tract images from 8 categories, each category consists of 1000 images. While the Kvasir-SEG ^[6] segmented polyp dataset replaces 13 images of polyps with new images to improve the quality of the dataset. This experiment uses a data set of 1,000 images of intestinal polyps, and the model will be used to identify specific lesion areas of intestinal polyps.

According to the division protocol of the CPC model, the Kvasir-SEG dataset was randomly selected into two groups: one group contained 1/2, 1/4, 1/8 and 1/16 label data, while the other group The remaining unlabeled data is called the unlabeled group, which is used to simulate the scarcity of labels. During the evaluation, the baseline model and the design model will be trained and tested using the same backbone network to check the performance of the model in the absence of labeled data. Such a setting can simulate as much as possible the performance difference of the evaluation model in the field of medical images in the absence of annotations.

S2. Construct and train a DFCPS model, as shown in FIG. 1 , including a data enhancement strategy and a neural network.

Further, as shown in Figure 2, the neural network includes a backbone network, an ASPP module, an upsampling module and a Softmax function;

The backbone network adopts ResNet-50, and introduces residual connections to build a deep network. The ASPP module includes four convolutional layers, and the atrous convolution rates of the four convolutional layers are 1, 12, 24, and 36, respectively. ;

The training method of the DFCPS model is as follows: firstly, the original sample X is subjected to two different degrees of strong and weak data enhancement through the data enhancement strategy, and two sets of enhanced samples are generated, one group is a strong enhanced sample, and the other is a strong enhanced sample. Weakly enhanced samples, strong and weakly enhanced samples are grouped and combined, and the combined strong and weak enhanced samples are entered into four neural networks for training, and the loss value is minimized by continuously optimizing the parameters of the model;

During the training process, the model minimizes the loss value by continuously optimizing the parameters of the model to prevent over-fitting and under-fitting phenomena, so as to improve the performance and accuracy of the model. The loss design of the present invention involves two key loss functions in the whole neural network training process: supervision loss L _s and cross pseudo-supervision loss L _cps .

Specifically, in this embodiment, firstly, pre-training will be completed in the PASCAL VOC 2012 data set. By performing pre-training on this rich data set, the corresponding pre-training weights will be obtained, and the neural network can learn rich visual features. Thereby improving its performance on various image segmentation tasks. The purpose of pre-training is to enable the network to better understand and represent the features of the image, and provide a good initial state for subsequent fine-tuning. Then fine-tune according to the specific task of the Kvasir-SEG dataset, and optimize for this specific task, so that the network can better adapt to the characteristics and segmentation requirements of medical images. Fully consider data characteristics, task requirements, and model constraints, as well as the handling and validation of pseudo-labels. As data augmentation, random cropping, rotation, Gaussian noise, and random dithering with additional color flipping will be used to accomplish the design goals. Specifically, the data enhancement parameters are shown in Table 1:

Table 1: Data Augmentation Methods

With the basic learning rate set to 0.01, PASCAL VOC 2012 was trained for 60 cycles, and then the pre-trained model weights were transferred to the Kvasir-SEG dataset for 100 epoch training. Determine the current batch_size and adjust the adaptive learning rate. The batch_size is set to 12 by default, the maximum learning rate is set to 1e-4, and the minimum learning rate is set to 0.01 times the maximum learning rate.

Further, the training is divided into two stages: freezing stage and unfreezing stage. In the freezing stage, some or all parameters of the pre-trained model are fixed, and only the newly added layers are trained. In the unfreezing phase, after a certain training period, the restrictions on the parameters of the pre-trained model are lifted, so that it can participate in the training of the entire model.

The main advantage of freezing training is that it speeds up training. The following are the specific steps to freeze training:

1. Create a basic model: First, create a basic model, the backbone network, on which to perform frozen training. This base model can be an already trained model or a new, untrained model.

2. Freeze the layers of the model: determine which layers will be frozen. Usually, the bottom layer (i.e., the layers around the input layer) and the top layer (the layer around the output layer) of the model are preserved, because these layers contain the core functions of the model, while the middle layers are frozen.

3. Adding new layers: In order to adapt to specific problems, it is necessary to add new layers on the frozen model or replace some intermediate layers to increase the depth or breadth of the model.

4. Freeze layer weights: Set the weights of frozen layers to be non-trainable and lock their weights. This means that during training, these layers will not be updated.

5. Compile the model: Set parameters such as the loss function, optimizer, and evaluation indicators of the model, and combine them with the model. This is a critical step in the training process.

6. Training model: After setting the loss function, optimizer and evaluation indicators, start training the model. Usually, a certain batch of data is used as input to train the model. Only the new layers that are added are trained, while the weights of frozen layers remain unchanged.

7. Unlock frozen layers: If the model does not produce satisfactory results, fine-tuning can be done by unlocking previously frozen layers. This allows the weights of these layers to be further trained and tuned to further improve the performance of the model.

8. Evaluate the model: After the training, the final evaluation of the model is carried out. Use an independent test dataset to evaluate model performance and accuracy.

Generally speaking, the first few layers of the model learn low-level features, while the later layers learn higher-level features. However, in some cases, the parameters of the first few layers have captured the characteristics of the input data well enough that no further updates are needed, while the later layers require more training to improve performance. By only training the newly added layers, the amount of training computation can be greatly reduced, thus improving the training efficiency. Additionally, freezing training helps prevent overfitting problems. In the early stage of training, the parameters of the model may overfit the training data, resulting in a decrease in generalization performance. By freezing some parameters, the learning ability of the model can be reduced, thereby effectively reducing the risk of overfitting and improving the generalization ability of the model. Since the pre-trained model has been trained on a large-scale data set, it has better generalization ability. By freezing its parameters, it can avoid overfitting on small sample data sets and improve the generalization performance of the model. In addition, freezing training can also protect the weights of pre-trained models. In some cases, the parameters of the pre-trained model may have high accuracy and quality, and by freezing the parameters of the pre-trained model, it can be avoided from being destroyed or overwritten. And freeze the training, you can keep training when the performance of the machine is not enough. However, as the training progresses, the model gradually learns higher-level features, and the fixed parameters may become a bottleneck that limits the performance of the model. In order to give full play to the expressive ability of the model, unfreezing training is to gradually release the fixed parameters, allowing the parameters of the entire model to be updated.

The main advantage of unfreezing training lies in fine-tuning the pre-trained model. The following are the specific steps to unfreeze training:

1. Freeze part of the model layer: After completing the training of the backbone model, freeze part of the model layer layer by layer from the output layer forward, usually select the layer farther away from the output layer for freezing.

2. Training part of the model layer: train the remaining unfrozen model layers, generally using a smaller learning rate.

3. Unfreeze more model layers: As the model is continuously optimized during the training process, continue to unfreeze more model layers, gradually making the model more complex.

4. Repeat steps 2 and 3: Repeat steps 2 and 3 until all model layers are trained.

5. Overall fine-tuning: Finally, the entire model is fine-tuned to further improve performance and accuracy.

The performance of the model can be further optimized by removing restrictions on the parameters of the pre-trained model and incorporating them into the training of the entire model. The pre-trained model already has a good feature extraction ability, and it can be better adapted to the target task through unfreezing and fine-tuning. Unfreezing training can improve the performance of the model on the target task and further improve its generalization ability. To sum up, the combination of freezing training and unfreezing training takes advantage of the pre-trained model, which not only speeds up the training speed, but also improves the generalization ability of the model.

After the model training is completed, an appropriate evaluation index will be selected to evaluate and compare the results of different models on the training set or validation set. By calculating the value of each indicator, select the model with the best performance as the final choice, apply it to the segmentation task of the data set, and observe the segmentation results, which involve the clarity, accuracy, consistency and consistency of the segmentation boundary. The degree of conformity between expertise and clinical application, through these evaluations, finally determines the completion of the model in the dataset segmentation task, and comprehensively evaluates its performance.

Further, as shown in Figure 3, in summary, the loss function in the whole training process mainly includes supervised loss and cross-pseudo-supervised loss. The supervised loss guides the learning of the network through the comparison between pseudo-labels of strongly augmented samples and weakly augmented samples. The cross-pseudo-supervised loss encourages the network to learn more consistent segmentation results as a whole by comparing the pseudo-segmentation maps generated between different groups. Such a training strategy helps to improve the performance of the model and enables the network to better adapt to different sets of data and labels. This design draws on the idea of Fixmatch in the method of strong and weak enhancement, and expands it accordingly to apply to the task of medical image label segmentation. However, special attention needs to be paid to the reliability, accuracy, and cleanliness of the pseudo-labels, since the quality of the pseudo-labels directly affects the performance and generalization ability of the model.

The supervised loss L _s is determined by the cross-entropy loss function, and normal supervised learning is performed on marked samples, and the expression is as follows:

Among them, ^Du : represents the collection of unlabeled samples, D ^l represents the collection of samples with labels, S is defined as the area of the input image, calculated by height*width, p _i is the corresponding confidence vector, y _j is the ground truth, l _ce is the cross entropy loss function.

The cross-pseudo-supervised loss is defined as and /> where /> Represents the supervised loss of labeled data, /> Represents the supervised loss of unlabeled data, the expression is as follows:

So the cross-pseudo-supervised loss can be defined as

The total loss can be correspondingly defined as, where ω is the weight

Loss=L _s +ωL _cps .

In order to ensure that the quality of the pseudo-label is high enough, the present invention introduces the concept of a confidence threshold, which is defined as μ in the present invention. By setting a confidence threshold, the model can filter out high-quality pseudo-labels, thereby avoiding negative impact on model training. Specifically, I compare the confidence of the pseudo-label against a confidence threshold. If the confidence of the pseudo-label is close to the confidence threshold, that is, around 0.5, it can be ignored, because such a pseudo-label may not be reliable enough, and the part of the pseudo-segmentation map greater than the confidence threshold μ is involved in the loss calculation. For strongly amplified samples, cross-entropy loss is also performed on the output prediction results and the corresponding pseudo-segmentation maps obtained from target weakly labeled samples that exceed the confidence threshold.

After the training is completed, in this embodiment, further, using mIOU (Mean Intersection over Union, average intersection and union ratio) as an index for evaluating model performance, by calculating the intersection and union ratio of the segmentation results predicted by the model and each category of the real label species And taking the average, we can get a comprehensive performance evaluation index, which measures the degree of overlap between the segmentation results predicted by the model and the ground truth labels. The higher the mIOU, the higher the segmentation accuracy of the model for different categories. And the model is evaluated every 10 iterations.

However, simply observing the mIOU value is not as high as possible, and the higher the mIOU value can reflect the higher the generalization performance of the model. At the same time, it is also necessary to observe the size of the Loss value to judge whether the model is converged, but more importantly, to observe its trend, especially the trend of the verification set Loss. When the validation loss keeps decreasing, the model can be considered to be converging. If the verification set Loss is basically unchanged, then the model has basically converged. When the model converges, the Loss value usually shows a downward trend, especially the Loss on the validation set. This means that the model is continuously optimized during the training process and gradually approaches the optimal solution. If the verification set Loss remains basically unchanged, it means that the model has stabilized, and further optimization may bring small benefits.

To evaluate the performance of the model, the model is evaluated every 10 iterations during the training process. In order to monitor the performance of the model in time, and adjust and optimize according to the evaluation results. During evaluation, image samples from the test set are used for inference and compared with the corresponding ground truth labels to calculate the mIOU metric.

During the evaluation process, the model that performed best on the training set will be selected and applied to the test set to verify the generalization ability of the model on unseen data. By applying the best model to the test set, a more accurate and reliable evaluation result can be obtained, further verifying the performance of the model. Such a design can ensure the stability and generalization ability of the model on different data sets.

With CPC, CPS model as baseline baseline, compare with the model of design, in order to evaluate model performance, by comparing and evaluating with CPC and CPS model, hope can fully understand the neural network model of the present invention design in semi-supervised learning task performance. This comparative analysis helps to verify the advantages and innovations of the designed model, and provides guidance for further improvement and optimization.

When the constructed DFCPS model meets the mIOU index, the segmentation model trained by DFCPS is used for simulation experiments:

1. The preprocessed data set is used as input, and the feature map is obtained through the backbone network;

2. Take the obtained feature map as input, and realize multi-scale comprehensive feature extraction through ASPP module through multiple parallel branches;

3. The obtained comprehensive features are represented by upsampling technology to increase data details, and the output of the model is mapped to the probability of each category through the Softmax function, and then the most likely category is selected as the prediction result according to the probability to generate a highly reliable Forecast samples.

4. Use the mIOU index to compare the performance of the model prediction results between different experimental settings.

In this embodiment, in view of the fact that the data enhancement strategy has a significant influence on the model in FixMatch, the present invention plans to design two sets of ablation experiments to verify the impact of different levels of data enhancement strategies on the DFCPS model. The data enhancement strategy originally set in this model is to perform different strong enhancement and weak enhancement on the same group of samples, and the strong and weak enhancements are combined into one group before entering the network.

In the current ablation experiment setting, this design sets two kinds of data enhancement strategies customized by the experiment. One data enhancement strategy is set to perform two different degrees of strong enhancement on the same group of image samples, and it remains unchanged in other cases. Based on the combination, it enters the network and obtains the results to observe the performance of the model. Another data enhancement strategy is set to perform two weak enhancements of different degrees on the original image samples of the same group. On the basis of keeping other conditions unchanged, the combination is then entered into the network, and at the same time, the performance of the model is observed. . In addition, raw samples (not processed by any data augmentation strategy) will be directly used to feed the network for training and evaluate the performance of the model. By comparing the above control data enhancement strategy with the data enhancement strategy adopted by DFCPS, and analyzing the impact of different levels of data enhancement strategies on the DFCPS model, it will help to better understand the role of data enhancement in the improvement of model performance, and provide Provide guidance for further optimization of the model.

In this embodiment, the experimental server is configured with six Nvidia GTX 2080Ti graphics cards with a memory size of 64GB and runs on the Ubuntu 20.04.1 system. The version of opencv_python is 4.1.2.30

After training for 100 epochs, we performed a visual analysis of the loss and mIOU. By observing the loss trend and mIOU curve, we can see that the loss gradually stabilized, and the mIOU value reached more than 80%, which is quite ideal.

In the visualization of the Loss trend, as shown in Figure 4, it is observed that the Loss gradually decreases and tends to be stable during the training process, which indicates that the training of the model has converged. In the visualization of the mIOU curve, as shown in Figure 5, the mIOU value gradually increases with the training, and finally exceeds 80%, and it is more accurate in the later training stage. This shows that the performance of the model for the segmentation task is good, the model can more accurately capture the target boundary in the image and perform segmentation, and has good performance and generalization ability. Therefore, the overall design proposed by the present invention has reached the expected performance level, and these results further verify the effectiveness and feasibility of the method proposed by the present invention.

Through this neural network, the segmentation image of intestinal polyps can be obtained, as shown in Figure 6, in addition to numerical indicators, visual evaluation is also one of the important methods to evaluate the results of medical image segmentation. By comparing the segmentation results with the ground truth labels, we can visually check their features such as boundary clarity, contour consistency, etc. When visually observing the segmentation results, the following observations can be drawn:

First, the segmentation boundary is clear, and the edge contour of intestinal polyps can be clearly distinguished without blurring or blurring. Second, the segmentation results are basically consistent with the contours of the ground truth labels, i.e. the segmented regions match the actual intestinal polyp locations. In addition, the segmentation results did not miss or mis-segment, and did not miss intestinal polyp regions or falsely label normal regions as intestinal polyps. In addition to boundary clarity and contour consistency, the segmentation results also match medical expertise and clinical application requirements. This means that our model can accurately label the lesion area, and the segmented area contains the anatomical structure of interest, which can basically complete the design task. The advantages of the proposed model in performance are further verified by comparative evaluation with the baseline model. And compared to the baseline model, the proposed model performs better in terms of contour segmentation and clarity. Its pixel accuracy is slightly higher than the baseline model, indicating that the proposed model is capable of more accurate pixel-level segmentation.

In conclusion, from the visual evaluation we can conclude that our model performs well on the segmentation task of intestinal polyps. The segmentation results have clear boundaries, are basically consistent with the contours of the real labels, and there is no missing segmentation or mis-segmentation. In addition, the segmentation results meet the requirements of medical expertise and clinical application.

In Table 2, under the RESNET-50 model, the mIOU values of different methods under different label data set ratios are shown. Compared with the CPC and CPS baseline groups, the method of the present invention performs better. Especially in the case of label coverage of 1/2, DFCPS achieves the best performance, compared to the CPC ^[2] and CPS ^[4] models, the DFCPS method I designed improves the mIOU value by 2.21% and 1.65%, respectively . These results show that the method of the present invention has certain advantages in the medical image segmentation task under the condition of lack of annotation, and shows ideal performance.

Table 2 mIOU values

By comparing the line graphs, as shown in Figure 7, it can be clearly seen that the method of the present invention has achieved better performance under different ratios of label data sets. Compared with the baseline group, the DFCPS model achieved higher mIOU values under the conditions of 1/2, 1/4, 1/8 and 1/16 label coverage, and achieved obvious advantages, further proving its The stability and robustness under different data set division conditions show that this model is more competent for medical image labeling tasks in the case of sparse labeling. It also shows that the method of the present invention can more accurately label medical images and improve the model's ability to detect and locate targets. It can also be seen from the visualization figure that the semi-supervised learning neural network designed by the present invention has shown better performance than the baseline group in each division of data sets, and these experimental results further support the semi-supervised learning designed by the present invention Feasibility of neural networks in solving medical image annotation problems. Compared with the baseline group, the method of the present invention makes full use of limited labeled data, and uses unlabeled data for model training and optimization through the idea of semi-supervised learning. This method not only improves the training efficiency, but also significantly improves the model performance, providing an effective solution to the challenge of data scarcity in medical image annotation.

In ablation experiments, the impact of different augmentation strategies on model performance can be observed. The results show (as shown in Table 3) that the combination of weak-weak enhancement is better than the combination of strong-strong enhancement, and the effect of directly using the original sample without any data enhancement strategy is the worst. In addition, the DFCPS model designed by the present invention performs best when using the strong-weak augmentation combination data augmentation strategy, which shows that the data augmentation strategy adopted in this work is reasonable.

Table 3: mIOU values under different enhancement strategies

In the six-card 2080ti experimental environment, the training time is compared with the baseline model (Table 4). Compared with the CPS ^[4] and CPC ^[2] models, DFCPS has a longer training time. In DFCPS, the training time is The increase mainly comes from two aspects. First, DFCPS needs to utilize additional neural network models to extract features and calculate the consistency loss of features. This involves training and optimizing multiple models, thus increasing the overall training time. Second, DFCPS also needs to update the policy network and feature extraction network through backpropagation. This process usually requires multiple iterations and training cycles in the hope of achieving good performance. In contrast, the CPS ^[4] and CPC ^[2] models may omit this step of feature consistency loss and backpropagation, so their training time may be relatively short. After DFCPS completes the training, the time consumption of using the learned strategy for reasoning is usually similar. Since DFCPS performs better in terms of performance, I think such an exchange is worthwhile.

Table 4: Time comparison

The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, without departing from the principle and spirit of the present invention, various changes, modifications, replacements and modifications to these implementations, including components, still fall within the protection scope of the present invention.

Claims (10)

1. A medical image processing method based on a semi-supervised neural network is characterized by comprising the following steps:

s1, acquiring a picture data set;

s2, constructing and training a DFCPS model, wherein the DFCPS model comprises a data enhancement strategy and a neural network, and the neural network comprises a main network, an ASPP module, an up-sampling module and a Softmax function;

the backbone network adopts ResNet-50, and introduces residual connection to construct a deep network, the ASPP module comprises four convolution layers, and the cavity convolution rates of the four convolution layers are respectively 1, 12, 24 and 36;

The training method of the DFCPS model comprises the following steps: firstly, carrying out two times of strong and weak data enhancement on an original sample X through a data enhancement strategy to generate two groups of enhancement samples, wherein one group is a strong enhancement sample, the other group is a weak enhancement sample, the strong and weak enhancement samples are combined in groups, the combined strong and weak enhancement samples enter four neural networks for training, and the loss value is minimized by continuously optimizing parameters of a model;

s3, performing image segmentation by using the trained neural network

S3-1, taking the preprocessed data set as input, and acquiring a feature map through a backbone network;

s3-2, taking the obtained feature map as input, extracting features through an ASPP module and fusing to obtain comprehensive features;

s3-3, increasing data details of the acquired comprehensive characteristic representation through an up-sampling technology, mapping the output of the model into the probability of each category through a Softmax function, and then selecting the most probable category as a prediction result according to the probability to generate a prediction sample with high reliability.

2. The method for processing medical images based on semi-supervised neural network as set forth in claim 1, wherein the data enhancement strategy is used for preprocessing the image data.

3. The method of claim 2, wherein the data enhancement strategy comprises random rotation, random brightness, contrast adjustment, random translation.

4. The method according to claim 1, wherein in the step S2, each group of the neural networks for strong and weak enhancement shares parameters and weights during training, and a pseudo tag generated from a prediction result outputted from the neural network after weak enhancement in each group is used as a target for strong enhancement sample prediction for each sample.

5. The method for processing medical images based on semi-supervised neural network as set forth in claim 4, wherein in the step S2, a supervision loss is introduced as L during the training process _s The learning process of the DFCPS model is guided by the information of the strong enhanced samples and the weak enhanced samples.

6. The method for medical image processing based on semi-supervised neural network as set forth in claim 5, wherein in the step S2, the training process design introduces a cross pseudo-supervision loss L _cps So that the pseudo tags generated by the samples of different groups of weakly supervised versions are constrained by each other.

7. The method for processing medical images based on semi-supervised neural network as set forth in claim 6, wherein the supervision loss L _s The normal supervised learning is performed on the marked samples, determined by the cross entropy loss function, expressed as follows:

wherein D is ^u Representing a set of unlabeled exemplars, D ^l A sample set representing existing labels, S being defined as the area of the input image, p calculated by height x width _i For the corresponding confidence vector, y _j For group trunk, l _ce Is a cross entropy loss function.

8. The method for processing medical images based on semi-supervised neural network as set forth in claim 7, wherein the cross-pseudo-supervision loss is defined asAnd->Wherein->Indicated is the supervised loss of tagged data,representing the supervision loss of unlabeled dataThe expression is as follows:

so the cross pseudo-supervision loss can be defined as

The total loss can be defined accordingly as, where ω is the weight

Loss＝L _s +ωL _cps (5)。

9. The medical image processing method based on the semi-supervised neural network according to claim 1, wherein the specific method of the step S3-1 is as follows: in the backbone network ResNet-50, the preprocessed data set is taken as input, the pooling layer and the full-connection layer in the ResNet-50 are used for feature extraction and classification, and information is allowed to directly skip some layers through residual connection, so that a feature map is output.

10. The medical image processing method based on the semi-supervised neural network as set forth in claim 9, wherein the specific method of step S3-2 is as follows: and taking the output of the neural network as the input of an ASPP module, wherein the ASPP module uses a plurality of parallel branches to perform multi-scale feature extraction, enhances a receptive field through hole convolution, and finally performs semantic association at different positions by utilizing the context information of the input feature map so as to output the extracted features.