CN111861924B - A cardiac magnetic resonance image data enhancement method based on evolutionary GAN - Google Patents

Abstract

The invention relates to a cardiac magnetic resonance image data enhancement method based on evolutionary GAN. When training a generator, the method mutates the generator to generate multiple descendant generators, and evaluates the adaptation of the multiple generators through an adaptability score function. sex score, and select the optimal offspring generator as the parent generator for the next iteration based on the score. At the same time, in the discriminator training stage, new training samples are synthesized using linear interpolation of feature vectors and related linear interpolation labels are generated. It not only expands the distribution of the entire training set, but also continuousizes the discrete sample space and improves the smoothness between fields, so that the model can be better trained. The image enhancement method of the present invention can generate high-quality and diverse samples to expand the training set, and ultimately improves various indicators of classification results.

Description

A cardiac magnetic resonance image data enhancement method based on evolutionary GAN

Technical field

The invention relates to the field of image processing, and in particular to a cardiac magnetic resonance image data enhancement method based on evolved GAN.

Background technique

Cardiac magnetic resonance is known as the gold standard for evaluating cardiac function. Conventional cardiac magnetic resonance scanning technology is relatively mature and plays a vital role in disease diagnosis. At present, many cardiac magnetic resonance image-assisted diagnosis tasks based on deep learning have achieved good results. However, cardiac magnetic resonance images not only require expensive medical equipment to obtain, but also require a large amount of manual data annotation by experienced radiologists. , which is undoubtedly extremely time-consuming and labor-intensive. In addition, patient privacy issues in the field of medical images have always been quite sensitive, so obtaining a large data set with balanced positive and negative samples requires a very high cost.

A big challenge in the field of medical imaging based on deep learning is how to deal with small-scale data sets and limited amounts of annotated data. Especially when using complex deep learning models, insufficient data sets or unbalanced data set samples will cause Deep convolutional neural networks with huge parameters suffer from overfitting. In the field of computer vision, scholars have proposed many effective methods to solve the problem of over-fitting, such as batch regularization, dropout, early stopping method, weight sharing, weight attenuation, etc. The above method is to adjust the network structure. In addition, data enhancement is an effective method to operate on the data itself. It alleviates the problem of overfitting to a certain extent in image analysis and classification. Phenomenon. Classic data enhancement techniques mainly include affine transformation methods such as translation, rotation, scaling, flipping and shearing, and mixing original samples with new samples as training sets and inputting them into the convolutional neural network; adjusting the sample color space is also a As a data enhancement method, Wang et al. used the method of changing the brightness value to expand the sample size; although these methods have improved, they only operate on the original samples and do not produce new features, and the diversity of the original samples has not been substantially improved. Improvement, the improvement effect is weak when processing small-scale data.

Generative Adversarial Network (GAN) is a generative model proposed by Ian Good fellow and others. It consists of a generator G and a discriminator D. The generator G uses noise sampled from a uniform distribution or a normal distribution. z is used as input to synthesize the image G(z). The discriminator D tries to judge the synthesized image G(z) as false as possible and the real image x as true. It adjusts the parameters of each model through successive adversarial training, and finally generates The device obtains the distribution model of real samples and obtains generation performance close to real images.

Generative adversarial networks generate new samples by fitting the original sample distribution. The new samples are generated from the distribution learned by the generative model, which gives them new characteristics that are different from the original samples. This feature makes it possible to use the samples generated by the generative network as new training samples to achieve data expansion. Although GAN has achieved good results in many computer vision fields, there are many problems in practical applications of GAN. On the one hand, GAN is very difficult to train. Once the data distribution and the distribution fitted by the generating network do not substantially overlap at the beginning of training, the gradient of the generating network will easily point in a random direction, thus causing the problem of gradient disappearance. On the other hand, the generator will try to generate a single sample that is safer but lacks diversity in order to let the discriminator give a high score, which will lead to the problem of mode collapse.

Contents of the invention

Aiming at the shortcomings of the existing technology, a cardiac magnetic resonance image data enhancement method based on evolutionary GAN is proposed. The specific steps of the data enhancement method include:

Step 1: Collect a cardiac magnetic resonance image data set, the data set includes benign cardiac magnetic resonance images and malignant cardiac magnetic resonance images;

Step 2: Preprocess the data set and divide the data set into a training set and a test set;

Step 3: Perform affine transformation on the preprocessed training set to obtain a data enhancement data set;

Step 4: Input the data augmentation data set into the constructed evolutionary GAN model for training, which specifically includes:

Step 41: Collect the noise z in the mixed Gaussian distribution as the initial input of the generator, and the generator synthesizes the input noise into an image;

Step 42: In the generator training stage, the parameters of the discriminator are fixed, and the generator is trained through three stages: mutation, evaluation and selection;

Step 43: In the discriminator training stage, the parameters of the generator are fixed, and the image synthesized by the generator and the image x in the data enhancement data set are synthesized into an image through linear interpolation method as the input of the discriminator;

Step 44: The generator and the discriminator are trained against each other in stages, and steps 42 to 43 are repeated until the number of training times is reached, and the training ends;

Step 5: Use the evolved GAN model after training to synthesize new images, and add the synthesized images to the training set to obtain a second data enhancement data set;

Step 6: Use the second data enhancement data set to train a classifier to verify the effect of data enhancement, wherein the synthetic image is used to train the second classifier and obtain a second classification result, and the training set is used for training the first classifier and obtain the first classification result;

Step 7: Use the test set to test the first classifier and the second classifier.

According to a preferred implementation, the generator training in step 42 also includes:

Step 421: Mutation. In the generator training phase, the discriminator parameters are fixed, and three mutation operations are performed on the current parent generator to obtain multiple child generators;

Step 422: Evaluate, use the fitness function to calculate the fitness score of each child generator under the current parent discriminator, and use the fitness function to evaluate the generation performance of the child generator under the current parent discriminator. and quantified as the corresponding fitness score:

F＝ _Fq + _γFd

Among them, F _q is used to measure the quality of the generated samples, F _d is used to measure the diversity of the generated samples, F represents the fitness score, and γ represents the hyperparameter;

Step 423: Selection, select the offspring generator with the highest fitness score through sorting as the parent generator for the next iteration.

The beneficial effects of the present invention are:

1. The data enhancement method of the present invention selects the current relatively optimal generator among multiple generator mutations to take into account the quality and diversity of generated images. It can generate high-quality and diverse samples to expand the training set, and ultimately improves classification. indicators of the results.

2. Combined with linear interpolation of feature vectors to synthesize new training samples and generate related linear interpolation labels, it not only expands the distribution of the entire training set, but also continuousizes the discrete sample space and improves the smoothness between fields, thus making the model Be better trained.

Description of the drawings

Figure 1 is a flow chart of the enhancement method of the present invention;

Figure 2 is a schematic diagram of the residual block structure;

Figure 3(a) is a real diseased image;

Figure 3(b) is a synthetic diseased image;

Figure 3(c) is a real non-diseased image; and

Figure 3(d) is a synthetic non-diseased image.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the specific embodiments and the accompanying drawings. It should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. Furthermore, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily confusing the concepts of the present invention.

A detailed description will be given below with reference to the accompanying drawings.

In view of the over-fitting problem that occurs easily when training deep convolutional neural networks with small-scale data sets, the present invention proposes a cardiac magnetic resonance image data enhancement method based on evolutionary generative adversarial networks. The method of the present invention selects the current relatively optimal generator among multiple generator mutations to take into account the quality and diversity of the generated pictures. At the same time, it also combines linear interpolation of feature vectors to synthesize new training samples and generate relevant linear interpolation labels. It not only expands It not only improves the distribution of the entire training set, but also continuousizes the discrete sample space and improves the smoothness between fields.

Figure 1 is a flow chart of the method of the present invention. As shown in Figure 1, the specific steps of the generative medical image data enhancement method based on evolutionary generative adversarial networks include:

Step 1: Collect a cardiac magnetic resonance image data set, the data set includes benign cardiac magnetic resonance images and malignant cardiac magnetic resonance images;

Step 2: Preprocess the data set and randomly divide the data set into a training set and a test set. Preprocessing methods include resampling, region of interest selection, normalization and final region of interest selection. The training set and the test set can be allocated according to actual needs. Generally, they are dynamically allocated in a ratio of 4:1, and the training set or the test set is randomly selected.

Step 3: Perform affine transformation on the preprocessed training set to obtain a data enhancement data set. Affine transformation operations include horizontal flip, vertical flip, 0°-20° random enlargement and rotation, 90°, 180°, 270° rotation, and 0-2% random enlargement and translation on the vertical and horizontal axes.

Step 4: Input the data augmentation data set into the constructed evolutionary GAN model for training. The method of this application integrates the ideas of evolutionary algorithm and linear interpolation during the training process, specifically including:

Step 41: Collect the noise z in the mixed Gaussian distribution as the initial input of the generator, and the generator synthesizes the input noise into an image;

Generally, generative adversarial networks will use noise z that obeys multivariate uniform distribution or multivariate normal distribution as the input of the model. The present invention uses multimodal distribution as input, which can better adapt to the inherent multimodality of real training data distribution. The invention uses multi-modality as input method to improve the quality and diversity of generated images.

Step 42: In the generator training stage, the parameters of the discriminator are fixed, and the generator is trained through three stages: mutation, evaluation and selection;

Step 421: Mutation. In the generator training phase, the discriminator parameters are fixed, and three mutation operations are performed on the current generator to obtain the offspring generator. The three mutation operations are maximum and minimum mutation, heuristic mutation, and least squares mutation.

Maximum and minimum value mutation: This mutation has small changes to the original objective function. This mutation can provide effective gradients and alleviate the phenomenon of gradient disappearance. The maximum and minimum values can be written as follows:

Heuristic mutation: Unlike minimax mutation, which minimizes the log probability that the discriminator is correct, heuristic mutation aims to maximize the log probability that the discriminator is wrong. When the discriminator judges the generated sample as false, heuristic mutation does not will be saturated and can still provide effective gradients, allowing the generator to continue training. The heuristic mutation can be written as follows:

Least squares mutation: Inspired by LSGAN, least squares mutation can also avoid vanishing gradients. At the same time, compared with heuristic mutation, although least squares mutation does not use a very high cost to generate false samples, it does not use a very low cost to avoid penalties, which avoids model collapse to a certain extent. The least squares mutation can be written as follows:

Step: 422: Evaluation, calculate the fitness score of each child generator under the current parent discriminator through the fitness function. That is, under the current parent discriminator, use the fitness function to evaluate the generation performance of the offspring generator, and quantify it into the corresponding fitness score:

F＝ _Fq + _γFd

Among them, F represents the adaptability score, and F _q measures the quality of the generated samples, that is, whether the descendant generator can fool the discriminator. Its expression is as follows:

F _q =E _z [D(G(z))]

F _d measures the diversity of generated samples. It measures the gradient size generated when the discriminator parameters are updated again according to the descendant generator. If the samples generated by the descendant generator are relatively concentrated, that is, they lack diversity, then the corresponding When updating the discriminator parameters, it is more likely to cause large gradient fluctuations. The expression is as follows:

γ (≥0) is a hyperparameter used to adjust the weight of generation quality and diversity, and can be adjusted freely in experiments.

Step 423: Selection, select the offspring generator with the highest fitness score through sorting as the parent generator for the next iteration.

The training method of improving the generative adversarial network is to train multiple descendant generators based on the mutation of the parent generator. After the evaluation of the adaptability score function, the optimal generator or generators are selected as the parent in the next discriminant environment. Builder.

Step 43: In the discriminator training stage, the parameters of the generator are fixed, and the image synthesized by the generator and the image x in the data enhancement data set are synthesized into an image through linear interpolation method as the input of the discriminator. Obtain the interpolated new sample and new label, and the discriminator loss is:

Update the discriminator parameters by calculating the average loss of the discriminator.

The method of the present invention constructs virtual training samples from original samples, combines linear interpolation of feature vectors to synthesize new training samples, and generates related linear fork labels to expand the distribution of the entire training set. The specific disclosure is as follows:

where x _i , x _j are the original input vectors, y _i , y _j are the label codes, (x _i , y _i ), (x _j , y _j ) are two samples randomly sampled from the original samples, λ∈Beta [α, α] is the weight vector, and α∈(0,+∞) is the hyperparameter that controls the interpolation strength between feature-target vectors. This linear interpolation method enables the model to behave linearly when processing the original sample and the area between the samples, so as to reduce the incompatibility when predicting test samples other than the training samples, enhance the generalization ability, and at the same time, it can be applied to the discrete sample space. Continuity is performed to improve smoothness between domains.

Step 44: The generator and the discriminator are trained against each other in stages, and steps 42 to 43 are repeated until the set training times are reached, and the training ends;

Step 5: Use the evolved generative adversarial model after training to synthesize new images, and add the synthesized images to the training set to obtain a second data enhancement data set;

Step 6: Use the second data augmentation data set to train a classifier, wherein the synthetic image is used to train the second classifier and obtain the second classification result, and the training set is used to train the first classifier and obtain the second classification result. One classification result.

Step 7: Use the test set to test the first classifier and the second classifier.

The effectiveness of the method was verified through classification experiments, and the impact of the diversity of generated samples and the number of generated samples on the classification results of cardiac magnetic resonance images was explored through comparative experiments.

The method of the present invention uses the TTUR (Two-Timescale Update Rule) method in the training process, specifically: the low-speed update rule is used in the generation network, the high-speed update rule is used in the discriminant network, and the learning rate of the generation network is set to 0.0001. The learning rate of the discriminant network is 0.0004, and 1:1 update can be achieved in the experiment.

The cardiac magnetic resonance image data enhancement method of the present invention uses an improved residual block structure and a self-attention module to train the generator and discriminator in the model. The residual block structure can alleviate the gradient disappearance problem and speed up the model convergence speed. Thus, high-performance generators can be trained more quickly in the same training time. The residual block structure is shown in Figure 2.

The generator is trained based on the residual block structure and self-attention module. An example is as follows:

Step a1: Perform fully connected mapping on the noise z and resize it. The output size is 1024×4×4;

Step a2: Input the output of the previous layer into the improved residual structure. The input will be divided into two channels, one of which is the residual part. This channel consists of 5 sub-operations: a convolution operation with a step size of 1, Batch normalization processing, LeakyReLU activation function, convolution operation with stride 1 and batch normalization. The other channel is a direct connection channel, which is combined with the output of the residual channel into a unified output. The output size is 1024×4× 4;

Step a3: After passing through the residual structure, use a transposed convolution layer with a size of 3×3 and a stride of 2 to process the output of the previous layer. After this, it also needs to undergo normalization processing and ReLU activation function to output Size 512×8×8;

Step a4: After repeating the operations of steps a2 to a3 three times, an output with a size of 128×32×32 is obtained. The output is input into the self-attention module, and the output size obtained is still 128×32×32;

Step a5: Repeat step a2 once, using a transposed convolution layer with a size of 3×3 and a stride of 1, to obtain an output with an output size of 64×32×32, after transposed convolution with a stride of 2 Operate with the tanh function to obtain a 3×64×64 image;

The discriminator is trained based on the residual structure and self-attention module. Examples are as follows:

Step b1: Enter a picture with size 3×64×64;

Step b2: Use a convolutional layer with a size of 3×3 and a stride of 1 to obtain an output with an output size of 64×64×64;

Step b3: Input the output of the previous layer into the improved residual structure. The input will be divided into two channels, one of which is the residual part. This channel consists of 5 sub-operations: a convolution operation with a step size of 1, Batch normalization processing, LeakyReLU activation function, convolution operation with stride 1 and batch normalization. The other channel is a direct connection channel, which is synthesized with the output of the residual channel into a unified output.

Step b4: After repeating step b3 once, pass the output through a convolution layer with a size of 3×3 and a stride of 2 to obtain an output with a size of 128×32×32;

Step b5: Input the output into the self-attention module, and the resulting output size is still 128×32×32;

Step b6: After repeating the operation of step b3 three times, an output with a size of 1024×4×4 is obtained;

Step b7: Use fully connected mapping to map the output of the previous layer to an output of size 1 as the final output.

Figure 3 is a comparison between the synthetic image and the real image obtained by the method of the present invention. Compared with the real image, the clarity of the synthetic image is slightly worse, and the outline is not as sharp as the real image, but it can be effectively used in data enhancement tasks.

In order to further illustrate the data enhancement effect of the method of the present invention, the accuracy, specificity and sensitivity of the method of the present invention are compared with existing methods. The specific results are as follows:

It can be seen from the table that the accuracy, specificity and sensitivity of the method proposed in the present invention are higher than those of the existing technology.

It should be noted that the above specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the scope of the present invention. within the scope of protection of the invention. Those skilled in the art should understand that the description of the present invention and the accompanying drawings are illustrative and do not constitute limitations on the claims. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims (1)

1. A cardiac magnetic resonance image data enhancement method based on evolved GAN, characterized in that the specific steps of the data enhancement method include: Step 1: Collect a cardiac magnetic resonance image data set, the data set includes benign cardiac magnetic resonance images and malignant cardiac magnetic resonance images; Step 2: Preprocess the data set and divide the data set into a training set and a test set; Step 3: Perform affine transformation on the preprocessed training set to obtain a data enhancement data set; Step 4: Input the data augmentation data set into the constructed evolutionary GAN model for training. During the training process, the ideas of evolutionary algorithm and linear interpolation are integrated, including: Step 41: Collect the noise z in the mixed Gaussian distribution as the initial input of the generator, and the generator synthesizes the input noise into an image; Step 42: In the generator training phase, the parameters of the discriminator are fixed, and the generator is trained through three stages of mutation, evaluation and selection. The training steps specifically include: Step 421: Mutation. In the generator training phase, the discriminator parameters are fixed, and three mutation operations are performed on the current parent generator to obtain multiple child generators; Step 422: Evaluate, use the fitness function to calculate the fitness score of each child generator under the current parent discriminator, and use the fitness function to evaluate the generation performance of the child generator under the current parent discriminator. and quantified as the corresponding fitness score: F＝ _Fq + _γFd Among them, F _q is used to measure the quality of the generated samples, F _d is used to measure the diversity of the generated samples, F represents the fitness score, and γ represents the hyperparameter; Step 423: Select, select the offspring generator with the highest fitness score through sorting as the parent generator for the next iteration; Step 43: In the discriminator training stage, the parameters of the generator are fixed, and the image synthesized by the generator and the image For new labels, the discriminator loss is: By calculating the average loss of the discriminator, the discriminator parameters are updated; virtual training samples are constructed from the original samples, new training samples are synthesized by linear interpolation of feature vectors, and related linear interpolation labels are generated to expand the distribution of the entire training set, where , λ∈Beta[α,α] is the weight vector, α∈(0,+∞) is the hyperparameter that controls the interpolation strength between feature-target vectors; Step 44: The generator and the discriminator are trained against each other in stages. At the same time, use the improved residual block structure and self-attention module to train the generator and the discriminator. Repeat steps 42 to 43 until the number of training times is reached, and the training is completed. ; Step 5: Use the evolved GAN model after training to synthesize new images, and add the synthesized images to the training set to obtain a second data enhancement data set; Step 6: Use the second data enhancement data set to train a classifier to verify the effect of data enhancement, wherein the synthetic image is used to train the second classifier and obtain a second classification result, and the training set is used for training the first classifier and obtain the first classification result; Step 7: Use the test set to test the first classifier and the second classifier.