JP7728002B2 - Medical image observation support device - Google Patents

Description

The present invention relates to a medical image analysis device that analyzes medical images using machine learning, and in particular to a device that analyzes images obtained by a cystoscope.

In the medical field, computer-aided diagnosis (CAD) using medical images is widely used. Medical images taken with various imaging devices are analyzed using computers, and the results of this analysis are used to support observation.

Japanese Patent Application Laid-Open No. 2017-070609

Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y., “Generative Adversarial Nets,” Advances in Neural Information Processing Systems 27 (NIPS) (2014)

One example is shown in Patent Document 1. Patent Document 1 discloses an image processing device that performs texture analysis on the overall shading of images captured using an endoscope or the like, including blood vessels in the mucosal epithelium, and calculates vascular features, and uses these to output a diagnosis of either non-tumor, adenoma, cancer, or SSA/P.

On the other hand, when it comes to diagnosis using medical images, there are some tests where it is extremely difficult to determine the presence or absence of a specific disease from the images. Diagnosis of Hunner-type interstitial cystitis (HIC) is one such example. Specifically, images taken with a cystoscope are used to diagnose whether a subject has Hunner-type interstitial cystitis, but there are few doctors who can correctly diagnose Hunner-type interstitial cystitis through cystoscopy. Furthermore, there is currently no mechanical device that can determine the presence or absence of Hunner-type interstitial cystitis from images.

The present invention has been made from this perspective, and aims to provide a medical image observation support device that can use machine learning methods to (1) differentiate (distinguish between benign and malignant diseases) between Hannah type interstitial cystitis (a designated intractable disease that is a benign disease that significantly impairs quality of life) and bladder intraepithelial carcinoma (malignant tumor), which have similar cystoscopic images, and further (2) differentiate between Hannah type interstitial cystitis and bladder pain syndrome (no obvious abnormalities found in the bladder), which have similar clinical symptoms, in other words, (3) differentiate between the three groups of diseases: Hannah type interstitial cystitis, bladder pain syndrome, and bladder cancer.

The gist of the first invention of this application is (a) a medical image observation support device that classifies medical images, (b) having a classification unit with a trained network, (c) the medical images are endoscopic images obtained using a cystoscope, and (d) the classification unit classifies the input endoscopic images into three categories: Hannah-type interstitial cystitis, bladder pain syndrome, and bladder cancer.

According to the medical image observation support device of the first invention, images obtained by a cystoscope are classified into three categories: Hanna-type interstitial cystitis, bladder pain syndrome, and bladder cancer, by a classification unit having a trained network.

Preferably, the medical image observation support device according to the second aspect of the present invention is characterized in that, in the medical image observation support device according to the first aspect of the present invention, (e) it performs two classifications, one into Hanna type interstitial cystitis and the other, based on the results of the three classifications. In this way, more favorable classification results can be obtained than when the classification unit directly performs two classifications, one into Hanna type interstitial cystitis and the other.

Preferably, the medical image observation support device according to the third aspect of the present invention is characterized by the medical image observation support device according to the first or second aspect of the present invention, further comprising: (f) a learning unit for training the network; (g) a memory unit for storing training data by the learning unit; and (h) an additional data generation unit for generating additional training data based on the training data stored in the memory unit, and (i) the learning unit trains the network using the additional training data in addition to training data previously stored in the memory unit. In this way, the additional training data is generated by the additional data generation unit based on the training data stored in the memory unit, and the network is trained by the learning unit using the additional training data in addition to the training data, allowing for more learning to be performed even when only training data is provided.

Preferably, the medical image observation support device according to the fourth aspect of the present invention is characterized in that, in the medical image observation support device according to any one of the first to third aspects, (j) the classification unit outputs the likelihood of the classification result. In this way, information about the likelihood can be obtained in addition to the classification result, making it possible to refer to the classification result taking the likelihood into consideration.

Preferably, the medical image observation support device according to the fifth invention is characterized in that it is the medical image observation support device according to any one of the first to fourth inventions, further comprising: (k) a plurality of the classifiers; (l) a judgment integration unit that determines classification results based on the outputs of the plurality of classifiers; and (m) the plurality of classifiers have a corresponding plurality of the networks. In this way, classification results can be determined based on the outputs of a plurality of classifiers, each having a plurality of trained networks, thereby reducing statistical recognition errors that occur due to training.

Preferably, the medical image observation support system according to the sixth aspect of the present invention is characterized in that it includes an endoscope device and any one of the first to fifth aspects of the medical image observation support device, and when a release switch provided on the endoscope device is operated, an image is input from the endoscope device to the medical image observation support device. In this way, the process from capturing images using the endoscope device to classifying images using the medical image observation support device can be easily performed as a series of operations.

1 is a diagram illustrating the configuration of a medical image observation support device according to an embodiment of the present invention; FIG. 2 is a functional block diagram illustrating the main control functions of the assistance device. FIG. 1 illustrates an example of a network. FIG. 4 is a diagram illustrating dilated convolution in the network of FIG. 3. 10 is a table summarizing the output results of the three classifications by the output unit. Of the actual input images, the correct answer was Hanna type interstitial cystitis, and the images that the support device of the present invention judged to be Hanna type interstitial cystitis are shown. Among the actual input images, the correct answer was Hanna type interstitial cystitis, and the support device of the present invention judged the image to be bladder pain syndrome. Of the actual input images, the correct answer was Hannah type interstitial cystitis, and the support device of the present invention judged the image to be bladder cancer. Of the actual input images, the correct answer was bladder pain syndrome, and the images that the support device of the present invention judged to be bladder pain syndrome are shown. Among the actual input images, the correct answer was bladder pain syndrome, and the support device of the present invention judged the image to be Hanna-type interstitial cystitis. Of the actual input images, the correct answer was bladder pain syndrome, and the support device of the present invention determined that the image was bladder cancer. Of the actual input images, the correct answer was bladder cancer, and the image was determined to be bladder cancer by the support device of the present invention. Among the actual input images, the correct answer was bladder cancer, and the support device of the present invention judged the image to be Hanna-type interstitial cystitis. Among the actual input images, the correct answer was bladder cancer, and the support device of the present invention judged the image to be bladder pain syndrome. 10 is a table summarizing the two classification output results by the output unit. 10A and 10B are diagrams showing the receiver operating characteristic (ROC) curve and AUC of the assistance device 10 of the present embodiment. FIG. 10 is a diagram illustrating an outline of a medical image observation support system according to another embodiment of the present invention. FIG. 10 is a functional block diagram illustrating the main parts of the control functions provided in the assistance device according to another embodiment of the present invention.

One embodiment of the present invention will be described in detail below with reference to the drawings.

Figure 1 is a diagram illustrating the configuration of a medical image observation support device 10 (hereinafter simply referred to as support device 10) according to one embodiment of the present invention. As shown in Figure 1, support device 10 of this embodiment comprises a central processing unit (CPU) 12, a graphics processing unit (GPU) 13 that is a processing device specialized for image processing, a read-only memory (ROM) 14, a random access memory (RAM) 16, and a storage device 18. The CPU 12 and GPU 13 are so-called microcomputers that process and control electronic information based on a predetermined medical image analysis program pre-stored in ROM 14, while utilizing the temporary storage function of RAM 16. The medical image analysis program is supplied, for example, from a medium such as a CD-ROM or via a network interface 30, which will be described later.

The storage device 18 is preferably a known storage device (storage medium) such as a hard disk drive or SSD (Solid State Drive) capable of storing information. It may also be a removable storage medium such as a small memory card or optical disk. The output interface 26 is for connecting a display device or the like as needed, and the input interface 28 is for connecting known input devices such as a keyboard or mouse, or for connecting a cable for capturing a specified image signal. If an image signal is input to the input interface 28, a converter or encoder may also be provided as needed. The network interface 30 is for connecting the support device 10 to a network, enabling the support device 10 to communicate with other computers via the network. A network cable or an antenna for a wireless network may be connected. Note that the output interface 26, input interface 28, and network interface 30 may be provided as needed and are not necessarily required.

Figure 2 is a functional block diagram explaining the main control functions of the support device 10. As shown in Figure 2, the support device 10 is functionally configured to include an image reading unit 32, an image processing unit 34, an output unit 38, and a learning unit 40.

The image reading unit 32 inputs the image to be analyzed by the assistance device 10 to the image processing unit 34, which will be described later. Specifically, the image to be analyzed by the assistance device 10 is stored in advance in the storage device 18, and in response to an input operation by the operator, for example, the image to be analyzed is read from the storage device 18 and passed to the image processing unit 34. In this embodiment, an image of the inside of the bladder, taken in advance by a cystoscope, is input.

The image processing unit 34 identifies and classifies regions of interest in an image based on the image input from the image reading unit 32. The image processing unit 34 has an internal trained network 36, which processes the input image and generates an output image.

The trained network 36 (hereinafter referred to as network 36) has the configuration of a convolutional neural network (CNN). The trained network 36 calculates the features of the input image using so-called deep learning.

The output unit 38 converts the output results from the image processing unit 34 into a specified format and outputs them to a file or another program, outputs them to a device such as a display device connected via the output interface 26, or transmits them to another computer via the network interface 30.

Figure 3 is a diagram showing an example of network 36. As described above, network 36 is a network with a CNN-type structure. In Figure 3, rectangular layers 50a to 50s represent feature maps or images. In the following description, when individual layers are not distinguished, they are referred to as layers 50. In Figure 3, a block is composed of three to five layers 50 connected horizontally. That is, layers 50a to 50c form block 51a, layers 50d to 50g form block 51b, layers 50h to 50k form block 51c, layers 50l to 50o form block 51d, and layers 50p to 50t form block 51e. Note that the number written below the leftmost layer 50 in each block 51 indicates the number of kernels in layer 50 in that block.

Of the thick lines connecting blocks, thick solid line 52 indicates 3x3 convolution. Here, 3x3 indicates the kernel size. Thick double line 54 indicates dilated convolution. Thick triple line 58 indicates global average pooling. This global average pooling 58 calculates and outputs the average of all pixel values that make up the feature map in the immediately preceding layer 50s for each feature map. Note that for the leftmost layers 50a, 50d, 50h, 50l, and 50p of each block 51, which have multiple inputs, these inputs are integrated.

As shown in Figure 3, in this embodiment, block 51a in network 36 performs two convolutions, block 51b performs three convolutions, and blocks 51c and 51d perform two convolutions and one dilated convolution. In addition, block 51e performs three convolutions and one full-level average pooling.

In addition, a residual path 56 is provided in each block 51 in the network 36, and dense pooling (64-70) is performed between each block 51, with input and output also being performed between blocks 51 other than adjacent blocks 51.

Of these, the residual path 56 passes the contents of the layer 50 preceding a given layer 50, combined with the output of that layer 50, to the next layer 50. Specifically, for example, looking at layer 50b in block 51a, the contents of layer 50a are convolved 52 and input to layer 50b. Meanwhile, the contents of layer 50b are convolved 52 and output from layer 50b to layer 50c. Here, the residual path 56 provided in block 51a combines the contents of layer 50a, the layer preceding layer 50b, with the output of layer 50b to form layer 50c. By providing this residual path 56, gradient loss in each layer is prevented.

Next, we will explain dense pooling. Dense pooling is a type of pooling that reduces the amount of data while retaining important elements from features (feature maps) obtained by convolution. Blocks 51a to 51c of network 36, feature maps or images that have undergone convolution 52 or dilated convolution, are used in subsequent convolutions using dense pooling.

In this embodiment, dense pooling is performed using mixed pooling. Specifically, the feature maps output from the two convolutions undergo pooling with a kernel size of 2x2 (hereinafter referred to as 2x2 pooling, and the same applies to other convolutions) as shown by the thin solid line 64, and are used as input for the next two convolutions (i.e., the first-stage output is used as input for the second stage, the second-stage output is used as input for the third stage, the third-stage output is used as input for the fourth stage, and the fourth-stage output is used as input for the fifth stage). As shown by the thin dotted line 66, 4x4 pooling is performed and the next two convolutions (i.e., the first-stage output is used as input for the third stage, the second-stage output is used as input for the fourth stage, and the third-stage output is used as input for the fifth stage). As shown by the thin dashed line 68, 8x8 pooling is performed and the next two convolutions (i.e., the first-stage output is used as input for the fourth stage, and the second-stage output is used as input for the fifth stage). Furthermore, as shown by the thin dashed dotted line 70, 16x16 pooling is performed and the result is used as the input for the next two convolutions (i.e., the fifth stage for the output of the first stage). This makes it possible to repeat the convolution while obtaining information that would otherwise be lost, unlike when the output of two convolutions is max-pooled and then used as the input for the next two convolutions.

Figure 4 illustrates the dilated convolution represented by the thick double line 54 in Figure 3. This dilated convolution is performed following the two convolutions 52 in blocks 51c and 51d in network 36 in Figure 3. Dilated convolution enables convolution processing that takes into account features that are spread out in the feature map.

In Figure 4, the thin solid line 72, thin dotted line 74, thin dashed line 76, and thin dashed-dotted line 78 all represent 3x3 convolutions, but each has a different dilation rate (DR): thin solid line 72 corresponds to DR=1, thin dotted line 74 corresponds to DR=2, thin dashed line 76 corresponds to DR=4, and thin dashed-dotted line 78 corresponds to DR=8. The dilation rate DR indicates the gap in the kernel used in the convolution, with DR=1 indicating the use of the kernel as is. Furthermore, DR=n indicates the use of a kernel with (n-1) gaps between each element of the original kernel. In other words, the dilated convolution 54 of this embodiment simultaneously performs dilated convolutions with multiple different dilation rates. Dilated convolutions enable convolution of large feature maps without using pooling, and, unlike pooling, have the advantage of not reducing the size of the output feature map.

That is, as shown in FIG. 4, the dilated convolution 54 integrates and outputs the output obtained by performing 1x1 convolution 58a on the input, the output obtained by performing 3x3 convolution 72 with DR=1 and 1x1 convolution 58b on that output, the output obtained by performing 3x3 convolution 74 with DR=2 and 1x1 convolution 58c on that output, the output obtained by performing 3x3 convolution 76 with DR=4 and 1x1 convolution 58d on that output, and the output obtained by performing 3x3 convolution 78 with DR=8 and 1x1 convolution 58e on that output.

Returning to Figure 2, the learning unit 40 trains the network 36. Specifically, it performs so-called supervised learning using training images and corresponding ground truths. The learning unit 40 may train the network 36 in advance before using the assistance device 10, or it may perform further training during use.

The learning unit 40 (1) divides endoscopic images captured in advance and stored in the memory unit 18 into training images and evaluation images for evaluating the network after training at a fixed rate. Then, (2) it uses the training images to train the network 36, and (3) it uses the trained network 36 to classify the evaluation images. The learning of the network 36 is completed by repeating this series of steps (1) to (3) a predetermined number of times. In this way, the training images are replaced with each repetition of the steps, allowing the network 36 to be trained optimally. Furthermore, because multiple evaluations can be performed, the evaluation results are less susceptible to the selection of the training images. Furthermore, because the above series of steps are independent of each other, data contamination, in which the same data is mixed in the training data and the evaluation data, does not occur.

The additional data generation unit 42 also performs a so-called data augmentation process, which creates additional data by performing a predetermined process on the endoscopic images previously stored in the storage unit 18 that are to be used as learning images by the learning unit 40 as described above. The additional data created in this way is used for training the network 36 in the learning unit 40, along with the aforementioned learning images. In this embodiment, a preset number of additional data are generated by the additional data generation unit 42.

Here, the predetermined processing performed by the additional data generation unit 42 includes, for example, the following processes: (1) Rotating the endoscopic image stored in the memory unit 18 (hereinafter referred to as the "original image") by an angle of ±30 degrees or less. (2) translating the original image in at least one of the vertical, horizontal, and vertical directions by ±22 pixels or less (in other words, within 10% of the image width). (3) Enlarging or reducing the original image by a factor of 0.8 to 1.2. (4) Flip the original image vertically or horizontally. (5) Changing at least one of the hue, saturation, and brightness in the HSV color space. The additional data generation unit 42 generates a new image by performing at least one of the processes (1) to (5) above on the original image. At this time, which process is applied is determined randomly. Note that if the processes (1) to (5) above result in pixels in the new image that are outside the original image, those pixels are blacked out.

Alternatively, a generative learning model called a GAN (Generative Adversarial Model), as described in the aforementioned Non-Patent Document 1, can be used to train cystoscope images and generate training images.

When the learning unit 40 trains the network 36, the mini-batch size, which is the number of data items in one learning session, and the number of learning epochs, which is the repetition index for the learning process for all data, are determined appropriately. The additional data generation unit 42 creates additional data for each epoch.

(Experimental Example)
The following experiment was carried out to verify the effectiveness of this embodiment.
In this experimental example, the support device 10 classifies the cases corresponding to the cystoscope images into three categories: Hanna type interstitial cystitis, bladder pain syndrome, or bladder cancer, and then, based on the results of these three categories, classifies the cases into two categories: Hanna type interstitial cystitis and others.

First, the storage device 18 stores cystoscopic images obtained from 681 patients. The cystoscopic images are taken using the cystoscope, and each image is 224 x 224 pixels in size. These images are stored in association with classification information previously prepared by a specialist, i.e., information on whether the image is Hanna-type interstitial cystitis, bladder pain syndrome, or bladder cancer.

Prior to the experiment, the network 36 is trained. This training is performed by the training unit 40 based on the cystoscope images of the 681 subjects previously stored in the storage device 18 and the additional data created by the additional data generation unit 42.

First, the learning unit 40 divides the cystoscope images of 681 people previously stored in the storage device 18 into learning images and evaluation images at a predetermined ratio, in this experimental example, a ratio of 80:20.

The additional data generation unit 42 then generates additional data from the separated training images. Specifically, as described above, (1) the training image (original image) is rotated by an angle of ±30 degrees or less. (2) the original image is translated by ±22 pixels or less in at least one of the up, down, left, and right directions (in other words, within 10% of the image width). (3) the original image is enlarged or reduced by a factor of 0.8 to 1.2. (4) the original image is flipped vertically or horizontally. (5) at least one of the hue, saturation, and brightness is changed in the HSV color space. The additional data generation unit 42 generates a new image by performing at least one of the processes (1) to (5) above on the original image. At this time, the correct answer data for the generated additional data is that of the original data. In addition, in this experimental example, the additional data generated by the additional data generation unit 42 is performed so that the sum of the learning image and the additional data generated by the additional data generation unit 42 is 50 times the size of the learning image.

The learning unit 40 uses the learning images and additional data to train the network 36. The learning images and additional data (hereinafter referred to as "input images") are input to the network 36 shown in Figure 3. At this time, the input images are three images for each color element in the image, i.e., three images for the R element, G element, and B element, and are input to the network 36 as density values for each color element.

In addition, in this experimental example, the learning of the network 36 by the learning unit 40 was performed with a mini-batch size of 15, which is the number of data items in one learning session, and a learning epoch count, which is the repetition index for the learning process for all data, of 200. These values can be changed as appropriate, as long as they are values at which the weight parameters can be considered to have converged experimentally. In addition, the learning rate, which is the amount by which the weight parameters are changed when learning, was initially set to 0.005, and gradually decreased after 100 epochs. This is because the initial value is relatively large to prevent the weights from reaching a local minimum solution, but the value is reduced to avoid divergence as learning is repeated.

An image to be classified by the assistance device 10 of this embodiment is input to the network 36 trained in this manner. As mentioned above, in this experimental example, the input image is 224 x 224 pixels in size and is input as an image for each of the R, G, and B color components. The input image input to the first layer 50a passes sequentially through each layer 50 of the network 36, undergoing the above-mentioned processing. When it reaches the final layer 50t of the network 36, global average pooling 58 is performed as mentioned above. At this time, each image in layer 50s of block 51e is digitized, and the input image is evaluated for one of three classifications: Hanna-type interstitial cystitis, bladder pain syndrome, or bladder cancer. Note that if one side of the input image is less than 224 pixels, that portion may be blacked out or enlarged to 224 pixels. Furthermore, if one side is greater than 224 pixels, it may be trimmed to 224 pixels or reduced to 224 pixels.

At this time, the network 36 is configured to output the likelihood of each of the three classifications for the input image. In other words, it outputs the likelihood that each of the three classifications, Hanna-type interstitial cystitis, bladder pain syndrome, and bladder cancer, is likely to be the case.

Based on the output of the network 36, the output unit 38 outputs which of the three categories the input image falls into. Specifically, for example, it outputs that the input image falls into the category with the highest likelihood of each of the three categories output.

Figure 5 is a table summarizing the output results from the output unit 38 when 646 of the 681 cystoscope images stored in the storage device 18 were input to the trained network 36. Figure 5 shows the relationship between the correct answer and each of the three classifications entered by the output unit 38. According to this, the accuracy rate for images where the correct answer was Hannah-type interstitial cystitis was 82.9%, the accuracy rate for images where the correct answer was bladder pain syndrome was 93.1%, and the accuracy rate for images where the correct answer was bladder cancer was 86.5%.

Figures 6 to 14 show actual input images for each classification result. Of these, Figures 6 to 8 show images for which the correct answer is Hanna-type interstitial cystitis, and of these, Figure 6 is an image that the support device 10 of this embodiment also classified as Hanna-type interstitial cystitis. Figure 7 is an image that the support device 10 of this embodiment classified as bladder pain syndrome. Also, Figure 8 is an image that the support device 10 of this embodiment classified as bladder cancer. Note that Figures 6 to 8, as well as Figures 9 to 11 and Figures 12 to 14 described below, show the likelihood that served as the basis for the judgment together with the image. The output unit 38 can display the image, the judgment result, and the likelihood for that judgment result together.

Figures 9 to 11 show images for which the correct answer is bladder pain syndrome. Of these, Figure 9 is an image that the support device 10 of this embodiment also classified as bladder pain syndrome. Figure 10 is an image that the support device 10 of this embodiment classified as Hannah type interstitial cystitis. Furthermore, Figure 11 is an image that the support device 10 of this embodiment classified as bladder cancer.

Figures 12 to 14 show images for which the correct answer is bladder cancer. Of these, Figure 12 is an image that the support device 10 of this embodiment also classified as bladder cancer. Figure 13 is an image that the support device 10 of this embodiment classified as Hannah-type interstitial cystitis. Furthermore, Figure 14 is an image that the support device 10 of this embodiment classified as bladder pain syndrome.

Furthermore, in this experimental example, the output unit 38 can classify the results of the three classifications, i.e., Hanna-type interstitial cystitis, bladder pain syndrome, and bladder cancer, that are determined to be bladder pain syndrome or bladder cancer as not being Hanna-type interstitial cystitis. In other words, the output unit can output two classifications, either Hanna-type interstitial cystitis or not Hanna-type interstitial cystitis, based on the results of the three classifications.

Figure 15 is a table summarizing the output results of the two-category classification by the output unit 38. Figure 15 shows the relationship between each of the two categories classified by the output unit 38 and the correct answer. According to this, the sensitivity of the support device 10 is the ratio of the number of images determined to be Hanna-type interstitial cystitis to the total number of images for which the correct answer is Hanna-type interstitial cystitis, that is, 165/(165 + 34), or 82.9%. Furthermore, the specificity is the number of images determined not to be Hanna-type interstitial cystitis to the total number of images for which the correct answer is not Hanna-type interstitial cystitis, that is, 416/(416 + 31), or 93.1%. According to this, the accuracy rate for the two-category classification is 89.9%, which is higher than when three-category classification is performed.

As such, it can be seen that both the three-category and two-category methods provide good judgment as to whether the cystoscope images are of Hanna-type interstitial cystitis.

Figure 16 shows the receiver operating characteristic (ROC) curves of the support device 10 of this embodiment. The solid line represents Hanna-type interstitial cystitis (truth) versus the other condition, i.e., bladder pain syndrome or bladder cancer (false). The dashed line represents bladder pain syndrome (truth) versus the other condition, i.e., bladder cancer or Hanna-type interstitial cystitis (false). The dashed line represents bladder cancer (truth) versus the other condition, i.e., Hanna-type interstitial cystitis or bladder pain syndrome (false). As shown in Figure 16, all ROC curves demonstrate good classification by the support device 10 of this embodiment. Furthermore, the AUC (the ratio of the area under each ROC curve) was 0.96 for the solid line, 0.98 for the dashed line, and 0.97 for the dashed line, respectively. In all cases, it can be seen that the classifier (classification algorithm) of the support device 10 of this embodiment performed well.

The support device 10 of this embodiment has an image processing unit 34 (classification unit) with a trained network 36, and the image processing unit 34 can classify endoscopic images obtained by a cystoscope and read by the image reading unit 32 into three categories: Hanna-type interstitial cystitis, bladder pain syndrome, and bladder cancer.

Furthermore, according to the support device 10 of this embodiment, the output unit 38 performs two classifications, either Hanna-type interstitial cystitis or others, based on the results of the three classifications, thereby obtaining more favorable classification results than when directly performing two classifications, either Hanna-type interstitial cystitis or others.

Furthermore, the assistance device 10 of this embodiment includes a learning unit 40 for training the network, a memory unit 18 for storing training data from the learning unit 40, and an additional data generation unit 42 for generating additional training data based on the training data stored in the memory unit 18.The learning unit 40 trains the network 36 using the additional training data in addition to the training data previously stored in the memory unit 18, so that even when only training data is provided, more learning can be performed.

Furthermore, according to the support device 10 of this embodiment, the image processing unit 34 outputs the likelihood of the classification result, so that it is possible to refer to the classification result taking the likelihood into consideration.

Next, another embodiment of the present invention will be described. In the following description, parts that are common to the previous embodiment will be assigned the same reference numerals and will not be described again.

Figure 17 is a diagram illustrating an overview of a medical image observation support system (hereinafter referred to as the "support system") 100 including the support device 10 of the present invention, and corresponds to Figure 2 in the previously described embodiment.

The support system 100 of this embodiment includes an endoscopic device 102 in addition to the support device 10 of Embodiment 1. While in the previously described Embodiment 1, an input image previously stored in the memory unit 18 was input to the image reader 32, this embodiment differs in that the input is made directly from the endoscopic device 102. Specifically, the endoscopic device 102, i.e., the cystoscope 102, is provided with a release switch 104. When the practitioner presses the release switch 104 while capturing the affected area of the patient with the cystoscope 102, the cystoscope image at that time is input to the image reader 32.

When an input image is input to the image reading unit 32 from the cystoscope 102, the image reading unit 32 causes the image processing unit 34 to read the image in the same manner as in Example 1 described above, and the image processing unit 34 then uses the trained network 36 to classify the input image into one of three categories. The output unit 38 also outputs the results of the three-category classification into Hanna-type interstitial cystitis, bladder pain syndrome, or bladder cancer, and, if necessary, outputs the likelihood calculated by the image processing unit 34, or outputs a two-category classification of whether or not the input image is Hanna-type interstitial cystitis based on the results of the three-category classification.

The support system 100 of this embodiment includes an endoscope device 102 and a support device 10. When the release switch 104 provided on the endoscope device 102 is operated, an image is input from the endoscope device 102 to the support device 10. This allows the process from capturing images using the endoscope device 102 to classifying the images using the support device 10 to be easily performed as a series of operations.

This embodiment is an alternative embodiment corresponding to the image processing unit 34 and output unit 38 in the previous embodiment.

Figure 18 is a diagram illustrating yet another embodiment of the present invention, and is a block diagram outlining the control functions of the image processing unit 134, output unit 138, and decision integration unit 139, which are used in place of the image processing unit 34 and output unit 38 in Figure 2 in the previously described embodiment.

As shown in Figure 18, the image processing unit 134 of this embodiment is configured to include multiple (n in Figure 18) trained networks 36a, 36b, ..., 36n. Each of the trained networks 36a, 36b, ..., 36n corresponds to the trained network 36 of the previously described embodiment, but the training data used for each network may be different or the same.

In addition, the output unit 138 in this embodiment is configured to include output units 38a, 38b, ..., 38n corresponding to the trained networks 36a, 36b, ..., 36n, respectively. The output units 38a, 38b, ..., 38n correspond to the output unit 38 in the above-mentioned embodiment. Therefore, the output units 38a, 38b, ..., 38n perform classification based on the features calculated by the corresponding trained networks 36a, 36b, ..., 36n. Here, since the trained networks 36a, 36b, ..., 36n are trained using different training data, the individual output units 38a, 38b, ..., 38n may output different classifications for a common judgment image. Note that there may be multiple trained networks 36a, 36b, ..., 36n and output units 38a, 38b, ..., 38n (i.e., the number is not limited as long as n≧2).

The decision integration unit 139 determines the final decision result by integrating the results output from the multiple output units 38a, 38b, ..., 38n. The decision integration unit 139 may determine the final decision result by taking a majority vote of the individual output results, or by performing majority voting processing that takes into account the likelihood calculated by each of the trained networks 36a, 36b, ..., 36n.

(Experimental Example)
The inventors of the present application conducted an experiment to determine images of Hannah interstitial cystitis from multiple cystoscope images using the support device 10 of this embodiment and the support device 10 of Example 1. In this experiment, 84 cystoscope images were prepared. The breakdown of the images was 37 images of Hannah interstitial cystitis, 36 images of bladder cancer, and 11 images of bladder pain syndrome. In the support device of this embodiment, 10 networks were used as the multiple networks 36a, 36b, ..., 36n. Furthermore, the determination by the integrated determination unit 139 was based on a majority vote of the output results from the 10 output units 38a, 38b, ..., i.e., the determination that was most frequently output as the output result was adopted.

When using the support device 10 of Example 1, the accuracy rate (correct diagnosis rate) for images determined to be Hannah type interstitial cystitis was 84.5%. On the other hand, when using the support device 10 of this example, the accuracy rate was 89.2%. Furthermore, the accuracy rate for all three types of images was 75.3% when using the support device 10 of Example 1, and 79.8% when using the support device 10 of this example. These experiments demonstrated that the accuracy rate of the support device 10 of this example, i.e., the judgment by the judgment integration unit 139 based on the judgment results of multiple networks 36a, 36b, ..., 36n, was 4% or more higher than that of the support device 10 of Example 1, i.e., the judgment based on a single network 38.

The assistance device 10 of this embodiment can determine the image for determination using likelihoods calculated by multiple trained networks 36a, 36b, ..., 36n, each trained using different training data, thereby reducing statistical recognition errors that occur during training.

One embodiment of the present invention has been described in detail above with reference to the drawings, but the present invention is not limited to this embodiment and may be implemented in other forms.

For example, the network 36 described above is composed of five blocks 51, but the number of blocks 51 is not limited to this. Furthermore, the number of layers 50 that make up each block 51 is not limited to that shown in the example.

Furthermore, in the above-described embodiment, the likelihood is displayed by the output unit 38, but this is not essential.

In addition, in the above-described embodiment, when the image processing unit 34 and the output unit 38 perform three classifications, the classification result is the one with the highest likelihood, but this is not limited to this. For example, if the difference in likelihood between each classification is within a certain threshold, a classification result indicating that classification is impossible may be output, or a definitive classification result may not be output, but rather the names of classifications that fall within the predetermined range and the likelihood of those classifications may be displayed.

In addition, the parameters used when the learning unit 40 learns the network 36, i.e., the number of mini-batches and the number of epochs, can be changed as appropriate. In addition, the amount of additional data generated by the additional data generation unit 42 can also be changed as appropriate.

In addition, in the above-described embodiment, a convolutional network (CNN) was used as the network 36, but this is not limited to this. As long as the network functions as a decision-maker capable of three-way classification using a network and is capable of learning global and local features, it may be a network with a different configuration, such as YOLO.

Furthermore, the above-described embodiment is not limited to a configuration in which the support device 10 and the endoscopic device 102 are directly connected by a dedicated cable. The support device 10 and the endoscopic device 102 may be capable of communicating with each other via a network, and images captured by the endoscopic device 102 may be transmitted to the support device 10 via the network, so that the support device 10 and the endoscopic device 102 are installed in separate locations. Furthermore, in such a configuration, the support system 100 may be configured so that multiple endoscopic devices 102 share a single support device 10.

In addition, the determination by the assistance device 10 of this embodiment may be made in real time during the examination using the cystoscope 102, or it may be possible to only take images during the cystoscopic examination and then send the captured images to the assistance device 10 for determination. Alternatively, the images may be sent to a server via a communication line (information network), where a determination is made and the results are sent to the endoscope device. Alternatively, the endoscopic images sent from the endoscope device may be stored on a server, and the diagnostic results may be presented offline for the stored images.

Furthermore, in the network 36 shown in Figure 3, a 3x3 convolution is performed, for example, at the location indicated by the thick solid line 52. In addition to this convolution, batch normalization, i.e., normalizing the same channel of all data within one mini-batch so that their mean is 0 and their variance is 1, may also be performed.

In addition, in the above-described embodiment, cystoscopic images were classified into three categories and images of Hannah type interstitial cystitis were identified, but cystoscopic images may be classified into four or more categories and images of Hannah type interstitial cystitis may be identified.

The above is merely one embodiment, and although other examples will not be provided, the present invention can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art, without departing from the spirit of the invention.

10: Medical image observation support device 34, 134: Image processing unit (classification unit)
36, 36a, 36b, ..., 36n: Networks 38, 138: Output section (classification section)
40: Learning unit 42: Additional data generation unit 100: Medical image observation support system 102: Endoscope device (bladder endoscope)
104: Release switch 139: Judgment integration unit

Claims (6)

A medical image observation support device that classifies medical images,
a classifier having a trained network;
the medical image is an endoscopic image obtained by a cystoscope,
the classification unit classifies the input endoscopic image into three categories: Hannah type interstitial cystitis, bladder pain syndrome, and bladder cancer;
A medical image observation support device characterized by: Based on the results of the three classifications, the patient is classified into two categories: Hanna type interstitial cystitis and others.
2. The medical image observation support device according to claim 1, wherein: a learning unit for learning the network;
a storage unit that stores data for learning by the learning unit;
an additional data generating unit that generates additional learning data based on the learning data stored in the storage unit,
the learning unit causes the network to learn using the additional learning data in addition to the learning data stored in advance in a storage unit;
3. The medical image observation support device according to claim 1, wherein: the classification unit outputs a likelihood of the classification result;
4. The medical image observation support device according to claim 1, wherein: A plurality of the classifiers;
a decision integration unit that determines a classification result based on outputs of the plurality of classification units;
the plurality of classifiers have a corresponding plurality of the networks;
5. The medical image observation support device according to claim 1, wherein: A medical image observation support system including an endoscope apparatus and the medical image observation support apparatus according to any one of claims 1 to 5,
When a release switch provided on the endoscope device is operated, an image is input from the endoscope device to the medical image observation support device;
A medical image observation support system characterized by the above.