CN111666972A - Liver case image classification method and system based on deep neural network - Google Patents

Abstract

The invention provides a method and system for classifying liver case images based on a deep neural network, including: obtaining multiple case images belonging to the same user-set time period; resampling the multiple case images to form a single three-dimensional case image The above-mentioned single three-dimensional case image is input into the deep neural network model to extract image features, and the image features include image color channels and position information; the attention mechanism is used to give weights to the above-mentioned image features, and the weight of the image features related to the liver is greater than that of other images. The weight of the image features; the weighted image features are input into the classifier to obtain the classification probability of a single three-dimensional case image, and the classification includes lesions and normal. The above method and system do not need to label the position information in the case image.

Description

Image classification method and system of liver cases based on deep neural network

technical field

The invention relates to the technical field of image analysis, and more particularly, to a method and system for classifying images of liver cases based on a deep neural network.

Background technique

In the case image recognition method in the prior art, classification and recognition are usually performed based on the tumor image of the labeled position. In order to ensure the correctness of the classification, manual labeling is usually used, and the workload of labeling is large.

SUMMARY OF THE INVENTION

The invention provides a method and system for classifying images of liver cases based on a deep neural network without labeling position information.

According to one aspect of the present invention, a method for classifying images of liver cases based on a deep neural network is provided, comprising:

Obtain multiple case images belonging to the same user-set time period;

Resampling multiple case images to form a single 3D case image;

The above-mentioned single three-dimensional case image is input into the deep neural network model to extract image features, and the image features include the correlation between image color channels, position information, texture and image pixels;

The attention mechanism is used to give weight to the above image features, and the weight of the image features related to the liver is greater than the weight of other image features;

The weighted image features are input into the classifier, and the classification probability of a single three-dimensional case image is obtained, and the classification probability represents the normal probability and the abnormal probability.

The method for classifying images of liver cases based on a deep neural network, further comprising:

The resampled single 3D case image is cropped to obtain multiple 3D case image blocks of uniform size, and the multiple 3D image blocks are input into the deep neural network model.

The described deep neural network-based liver case image classification method, wherein, the training steps of the deep neural network model and the classifier include:

Obtain case images of multiple users of known classification, each user has multiple liver case images;

Resampling multiple case images of each user to form a single 3D case image for each user;

Crop a single 3D case image resampled by each user to obtain multiple 3D case image blocks of uniform size for each user, forming the first training set of the deep neural network;

Use a deep neural network to extract the image features of each user in the first training set;

The attention mechanism is used to give weights to the image features of each user to form the second training set of the classifier;

Input the second training set into the classifier to obtain the classification of each user's case image in the second training set;

Deep neural networks and classifiers are iteratively trained using a cross-entropy loss function.

The method for classifying liver case images based on a deep neural network, wherein the step of resampling multiple case images includes:

stacking two-dimensional images of multiple case images belonging to the same user into a three-dimensional case image from the case image containing the heart to the case image containing the abdomen;

The three-dimensional case images are resampled, and the stacked multiple case images are linearly interpolated to the set thickness in the thickness direction.

The method for classifying images of liver cases based on a deep neural network, wherein the step of resampling multiple case images further includes:

Threshold mapping is performed on the resampled 3D case images.

The method for classifying liver case images based on a deep neural network, wherein the plurality of case images are weakly labeled case images, and the weak labeling indicates whether there is a lesion in the case image, but does not label the location of the lesion. case image.

The method for classifying images of liver cases based on a deep neural network, wherein the deep neural network is a Bninception network, which includes multiple convolutional layers, pooling layers, data flattening layers and fusion layers. The image features output by the last three convolutional layers of the Bninception network are screened, and the image features related to the liver are screened out. feature fusion.

The method for classifying images of liver cases based on a deep neural network, wherein the step of using an attention mechanism to assign weights to the above image features includes:

The compression operation is used to compress the image features according to the spatial dimension, and the two-dimensional feature channels are changed into 1 number;

Using the excitation operation, each feature channel is assigned a weight, and the weight reflects the correlation between the feature channels;

Using a repeated value operation, the output of the excitation operation is weighted to the original image features.

The method for classifying images of liver cases based on a deep neural network, wherein the deep neural network includes multiple neural network models, and the average value of multiple classification probabilities output by the multiple neural network models through the classifier is used as the classification probability. the final result.

According to another aspect of the present invention, a deep neural network-based liver case image classification system is provided, comprising:

The acquisition module collects multiple case images belonging to the same user-set time period;

Resampling module, resampling multiple case images to form a single 3D case image;

Image feature extraction module, input the above-mentioned single three-dimensional case image into the deep neural network model to extract image features, and the image features include image color channel, position information, texture and correlation between image pixels;

The weighting module uses the attention mechanism to give weights to the above image features, and the weights of the image features related to the liver are greater than the weights of other image features;

The classification module inputs the weighted image features into the classifier, and obtains the classification probability of a single three-dimensional case image, where the classification probability represents the normal probability and the abnormal probability.

The above-mentioned deep neural network-based liver case image classification method and system use the attention mechanism to help the deep neural network model to automatically find location information, without the need to label location information, greatly reducing the workload of labeling, and passing the method with less labeling workload. Deep deep neural network, attention mechanism and classification model get better recognition effect.

Description of drawings

Fig. 1 is the flow chart of the liver case image classification method based on the deep neural network of the present invention;

Fig. 2 is the schematic diagram of the resampling of the present invention;

3 is a schematic diagram of a method for assigning weights to the above-mentioned image features using an attention mechanism according to the present invention;

4 is a block diagram of the structure of the deep neural network-based liver case image classification system according to the present invention;

FIG. 5 is a block diagram of a preferred embodiment of the deep neural network-based liver case image classification system according to the present invention.

Detailed ways

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It will be apparent, however, that the embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more embodiments.

Various embodiments according to the present invention will be described in detail below with reference to the accompanying drawings.

Fig. 1 is the flow chart of the liver case image classification method based on the deep neural network according to the present invention, as shown in Fig. 1, the described case image classification method includes:

Step S1, obtaining multiple case images belonging to the same user-set time period;

Step S2, resampling multiple case images to form a single three-dimensional case image;

Step S4, the above-mentioned single three-dimensional case image is input into the deep neural network model to extract image features, and the image features include image color channel, position information, texture and correlation between image pixels;

Step S5, using the attention mechanism to give weights to the above-mentioned image features, and the weights of the image features related to the liver are greater than the weights of other image features;

In step S6, the weighted image features are input into the classifier to obtain the classification probability of a single three-dimensional case image, where the classification probability represents normal probability and abnormal probability.

In one embodiment, the training steps of the deep neural network model and the classifier include:

Obtain case images of multiple users of known classification, each user has multiple liver case images;

Resampling multiple case images of each user to form a single 3D case image for each user;

Crop a single 3D case image resampled by each user to obtain multiple 3D case image blocks of uniform size for each user, forming the first training set of the deep neural network;

Use a deep neural network to extract the image features of each user in the first training set;

The attention mechanism is used to give weights to the image features of each user to form the second training set of the classifier;

Input the second training set into the classifier to obtain the classification of each user's case image in the second training set;

Iterative training of deep neural networks and classifiers using cross-entropy loss function iteration

是分类器的输出分类概率大的分类标签、y是真实的分类标签。in

is the classification label with a large classification probability of the output of the classifier, and y is the real classification label.

In one embodiment, in step S2, the step of resampling multiple case images includes:

stacking two-dimensional images of multiple case images belonging to the same user into a three-dimensional case image from the case image containing the heart to the case image containing the abdomen;

The three-dimensional case images are resampled, and the stacked multiple case images are linearly interpolated to the set thickness in the thickness direction.

Preferably, the step of resampling the multiple case images further includes:

Threshold mapping is performed on the resampled three-dimensional case image, preferably, HU mapping is used,

HU=pixel_value*rescale_slope+rescale_intercept

Among them, pixel_value is the pixel value, rescale_slope is the scaling range, rescale_intercept is the scaling intercept, and HU is the mapping value.

Preferably, before step S4, it also includes step S3:

The resampled single 3D case image is cropped to obtain multiple 3D case image blocks of uniform size, and the multiple 3D image blocks are input into the deep neural network model.

In a specific embodiment, as shown in FIG. 2 , the CT images captured by different patients at one time are multiple 2D images of different numbers. We first stack all the 2D images of each patient into a 3D image from top to bottom, and then Resampling is performed, that is, all samples are linearly interpolated in the Z dimension (height dimension) to a uniform thickness (1mm x 1mm x 1mm), and then mapped through HU, and finally cropped to the same size of (160, 256, 256).

In a preferred embodiment, the deep neural network is a Bninception network, including a plurality of convolutional layers, pooling layers, data flattening layers and fusion layers. The image features are screened to screen out the image features related to the liver, and the screened image features are input into the pooling layer, the data flattening layer and the fusion layer to perform feature fusion of the image features related to the liver.

In step S4, as shown in Figure 3, the step of using the attention mechanism to assign weights to the above image features includes:

The compression operation is used to compress the image features according to the spatial dimension, and the two-dimensional feature channel is changed into a number. For example, the deep neural network input c1 (B, C, W, H undergoes a 1x1 convolution operation Ftr to change the feature channel. The number is C2, B is the number of images, C is the feature channel, W is the width of the image, H is the height of the image, W, H can take the cumulative sum of each pixel and then average;

Using the excitation operation, each feature channel is equipped with a weight, and the weight reflects the correlation between the feature channels. For example, the number of feature channels C2 is subjected to the global pooling operation of Fsq, and the number of Ftr feature channels C2 is operated by 1x1 convolution;

Using repeated value operation, the output of the excitation operation is weighted to the original image feature, for example, the weight is multiplied by the original input to operate Fscale, and the final output is the weighted image feature x~.

In one embodiment, the deep neural network includes multiple neural network models, and the average value of multiple classification probabilities output by the multiple neural network models through the classifier is used as the final result of the classification probability.

The classification method of liver pathological images based on the deep neural network of the present invention First, considering the characteristics of liver CT images as multiple 2D image slices, the CT images are combined into a single 3D image; secondly, considering the tissue slices Due to the characteristics of different thicknesses and different number of slices for different patients, the 3D images are resampled to a uniform scale, HU threshold conversion and cropped to the same size; then, because there are many interfering organs displayed in the liver CT, in the network An attention mechanism is added to allow the model network to focus more on the classification of liver lesions. In the end, we got better results from the neural network, which is to directly classify all CT slices of a patient to output the lesion or normal category.

FIG. 4 is a block diagram showing the structure of the liver case image classification system based on the deep neural network of the present invention. As shown in FIG. 4 , the liver case image classification system includes:

The acquisition module 1 collects multiple case images belonging to the same user-set time period;

Resampling module 2, resampling multiple case images to form a single three-dimensional case image;

Image feature extraction module 3, the above-mentioned single three-dimensional case image is input into the deep neural network model to extract image features, and the image features include image color channels, position information, texture and correlation between image pixels;

The weighting module 4 uses the attention mechanism to give weights to the above-mentioned image features, and the weights of the image features related to the liver are greater than the weights of other image features;

The classification module 5 inputs the weighted image features into the classifier to obtain the classification probability of a single three-dimensional case image, where the classification probability represents normal probability and abnormal probability.

Preferably, a cropping module 6 is also included to crop the resampled single 3D case image to obtain multiple 3D case image blocks of uniform size, and input the multiple 3D image blocks into the deep neural network model.

Preferably, a training module is also included to train the deep neural network model and the classifier, including:

The first training set construction unit obtains case images of multiple users of known classification, each user has multiple liver case images; resamples multiple case images of each user to form a single image of each user 3D case image; crop a single 3D case image resampled by each user to obtain multiple 3D case image blocks of uniform size for each user, forming the first training set of the deep neural network;

The second training set construction unit adopts the attention mechanism to assign weights to the image features of each user to form the second training set of the classifier;

The training unit adopts a deep neural network to extract the image features of each user in the first training set; the second training set is input into the classifier to obtain the classification of the case images of each user in the second training set; the cross entropy loss function is used to iteratively analyze the depth Neural networks and classifiers are iteratively trained.

In one embodiment, the resampling module 2 includes:

a stacking unit, which stacks the two-dimensional images of the multiple case images belonging to the same user into a three-dimensional case image from the case image containing the heart to the case image containing the abdomen;

The interpolation unit resamples the three-dimensional case images, and linearly interpolates the stacked multiple case images to the set thickness in the thickness direction.

Preferably, the resampling module 2 further includes a threshold mapping unit, which performs threshold mapping on the resampled three-dimensional case image.

In a preferred embodiment of the present invention, as shown in FIG. 5 , the deep neural network is a Bninception network, including multiple convolution layers, pooling layers, data flattening layers and fusion layers, and the weighting module 4 Including a plurality of SE modules, the classification module 5 includes a classifier and a convergence module, wherein,

The SE module includes a compression unit, an excitation unit and a reassignment unit. The compression unit uses a compression operation (Squeeze operation) to compress the image features according to the spatial dimension, and changes the two-dimensional feature channel into a number, for example, input To global average pooling, the image color channel and position information are directly reduced to a number; the excitation unit adopts an excitation operation (Excitation operation) to assign a weight to each feature channel, and the weight reflects the correlation between feature channels. , for example, successively through 1x1 convolution, sigmoid function, and 1x1 convolution to obtain the weight value; the reassignment unit adopts the repeated value operation (Reweight operation) to weight the output of the excitation operation to the original image feature, that is, The output weight of the excitation unit is multiplied by the feature before the network input to the compression unit, which is the output of the SE module;

Among them, the SE module uses the attention mechanism to screen the image features output by the last three convolutional layers of the Bninception network, screen out the image features related to the liver, and input the screened image features into the pooling layer and the data flattening layer. and fusion layer to perform feature fusion of liver-related image features.

The CNN output of different depths of BNinception network, each SE module is the same, the purpose is to filter the importance of different CNN output values, increase the weight of important features, reduce the rest of the weight values, CNN features are input to the global average pooling -> 1x1 convolution ->sigmoid->1x1 convolution and then multiplied with the original input, specifically: the Squeeze operation is to compress the features according to the spatial dimension, and change the two-dimensional feature channel into 1 number (global average pooling), which is in To some extent, it represents the global receptive field. The excitation operation is a gate mechanism in the neural network (1x1 convolution, with a weight w for each channel to reflect the correlation between channels. The Reweight operation weights the output of the excitation operation to the original features to obtain the final result.

Bninception includes multiple convolutional layers. The fusion of different layer features is beneficial to obtain high-level information, but the smaller the stage, the lower-level features are not conducive to classification. In the experiment, 3 layers are the best, and the fusion is directly spliced together. Pooling and max pooling, splicing together, and outputting the final probability value of canceration through the fully connected layer.

In one embodiment, the deep neural network of the image feature extraction module 3 includes multiple neural network models, and the classification module 5 uses the average value of multiple classification probabilities output by the multiple neural network models through the classifier as the final result of the classification probability, For example, the deep neural network can also include resnet50, and also add the SE module. The SE module only affects the internal multiple CNN output values of resnet\bninception, which in turn affects the final output probability value.

In each of the above embodiments, the multiple case images are weakly labeled case images, and the weakly labeled case images are case images that indicate whether there is a lesion in the case image, but do not label the location of the lesion. Because the data is weakly labeled, the location of the patient's lesion is not marked, but simply gives the information about whether there is any lesion in all CTs of the patient (1 means there is a lesion, 0 means no lesion), because there are many human organs, there is no labelling. The model is more difficult to classify in the case of the lesion location. Therefore, the attention mechanism is integrated into the BNInception network. One is to change the 2D network to 3D, and the other is to use the SEModu1e attention mechanism to screen features related to the liver, so that the model can focus on The third is to fuse the output features of the last three stages (convolutional layers).

Although the foregoing disclosure shows exemplary embodiments of the present invention, it should be noted that various changes and modifications can be made without departing from the scope as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the inventive embodiments described herein need not be performed in any particular order. Furthermore, although elements of the invention may be described or claimed in an individual form, it is also contemplated that there are multiple elements unless explicitly limited to a single element.

Claims (10)

1. a liver case image classification method based on deep neural network, is characterized in that, comprises: Obtain multiple case images belonging to the same user-set time period; Resampling multiple case images to form a single 3D case image; The above-mentioned single three-dimensional case image is input into the deep neural network model to extract image features, and the image features include the correlation between image color channels, position information, texture and image pixels; The attention mechanism is used to give weight to the above image features, and the weight of the image features related to the liver is greater than the weight of other image features; The weighted image features are input into the classifier, and the classification probability of a single three-dimensional case image is obtained, and the classification probability represents the normal probability and the abnormal probability. 2. the liver case image classification method based on deep neural network according to claim 1, is characterized in that, also comprises: The resampled single 3D case image is cropped to obtain multiple 3D case image blocks of uniform size, and the multiple 3D image blocks are input into the deep neural network model. 3. the liver case image classification method based on deep neural network according to claim 2, is characterized in that, the training step of described deep neural network model and classifier comprises: Obtain case images of multiple users of known classification, each user has multiple liver case images; Resampling multiple case images of each user to form a single 3D case image for each user; Crop a single 3D case image resampled by each user to obtain multiple 3D case image blocks of uniform size for each user, forming the first training set of the deep neural network; Use a deep neural network to extract the image features of each user in the first training set; The attention mechanism is used to give weights to the image features of each user to form the second training set of the classifier; Input the second training set into the classifier to obtain the classification of each user's case image in the second training set; Deep neural networks and classifiers are iteratively trained using a cross-entropy loss function. 4. The liver case image classification method based on deep neural network according to claim 1, wherein the step of resampling the multiple case images comprises: stacking two-dimensional images of multiple case images belonging to the same user into a three-dimensional case image from the case image containing the heart to the case image containing the abdomen; The three-dimensional case images are resampled, and the stacked multiple case images are linearly interpolated to the set thickness in the thickness direction. 5. The liver case image classification method based on deep neural network according to claim 4, wherein the step of resampling the multiple case images further comprises: Threshold mapping is performed on the resampled 3D case images. 6. The liver case image classification method based on a deep neural network according to claim 1, wherein the multiple case images are weakly labeled case images, and the weak labeling is to mark whether there are any in the case images. Lesions, but without annotated case images for the location of the lesions. 7. The liver case image classification method based on a deep neural network according to claim 1, wherein the deep neural network is a Bninception network, comprising a plurality of convolutional layers, pooling layers, data flattening layers and The fusion layer uses the attention mechanism to screen the image features output by the last three convolutional layers of the Bninception network, screen out the image features related to the liver, and input the screened image features into the pooling layer, data flattening layer and fusion layer. , perform feature fusion of liver-related image features. 8. The liver case image classification method based on a deep neural network according to claim 1, wherein the step of using an attention mechanism to give weights to the above-mentioned image features comprises: The compression operation is used to compress the image features according to the spatial dimension, and the two-dimensional feature channels are changed into 1 number; Using the excitation operation, each feature channel is assigned a weight, and the weight reflects the correlation between the feature channels; Using a repeated value operation, the output of the excitation operation is weighted to the original image features. 9. The liver case image classification method based on a deep neural network according to claim 1, wherein the deep neural network comprises a plurality of neural network models, and the plurality of neural network models are output by a classifier. The average of the classification probabilities is taken as the final result of the classification probabilities. 10. A liver case image classification system based on a deep neural network, comprising: The acquisition module collects multiple case images belonging to the same user-set time period; Resampling module, resampling multiple case images to form a single 3D case image; Image feature extraction module, input the above-mentioned single three-dimensional case image into the deep neural network model to extract image features, and the image features include image color channel, position information, texture and correlation between image pixels; The weighting module uses the attention mechanism to give weights to the above image features, and the weights of the image features related to the liver are greater than the weights of other image features; The classification module inputs the weighted image features into the classifier, and obtains the classification probability of a single three-dimensional case image, where the classification probability represents the normal probability and the abnormal probability.

Images