WO2021054477A2 - Disease diagnostic support method using endoscopic image of digestive system, diagnostic support system, diagnostic support program, and computer-readable recording medium having said diagnostic support program stored therein - Google Patents

Abstract

The present invention provides a diagnostic support method and a diagnostic support system with which it is possible to accurately identify various small intestinal diseases using a convolutional neural network (CNN) system based on small intestine endoscopic images or moving images from a wireless capsule endoscope (WCE). A diagnostic support system according to an aspect of the present invention uses first endoscopic images of small intestines from a WCE and definitive diagnostic results of diseases respectively corresponding to the first endoscopic images to train a CNN system. The trained CNN system detects a small intestinal disease on the basis of a second endoscopic image of a small intestine from a WCE, and outputs at least one of a probability score and a region corresponding to the disease positivity. Said diagnostic support system is characterized in that the first endoscopic images are WCE endoscopic static images of small intestines, and the second endoscopic image is a WCE endoscopic moving image of a small intestine.

Description

A computer-readable recording medium that stores a disease diagnosis support method, a diagnosis support system, a diagnosis support program, and this diagnosis support program using endoscopic images of the digestive organs.

The present invention relates to a method for supporting diagnosis of a disease by endoscopic images of the digestive organs using a neural network, a diagnosis support system, a diagnosis support program, and a computer-readable recording medium that stores this diagnosis support program.

Many endoscopy is performed on the digestive organs, such as the larynx, pharynx, esophagus, stomach, duodenum, biliary tract, pancreatic duct, small intestine, and large intestine. Endoscopy of the upper digestive organs is for screening for gastric cancer, esophageal cancer, peptic ulcer, reflux gastritis, etc., and endoscopy of the large intestine is for colon cancer, colorectal polyps, ulcerative colitis. It is often done for screening such as. In particular, endoscopy of the upper digestive organs is commonly incorporated into detailed examinations of various upper abdominal symptoms, detailed examinations based on positive barium tests for gastric diseases, and regular medical examinations in Japan. It is also useful for close examination of abnormal serum pepsinogen levels. In recent years, gastric cancer screening has been shifting from conventional barium examination to gastroscopy.

Gastric cancer is one of the most common malignant tumors, and it is estimated that about 1 million cases have occurred worldwide several years ago. In particular, 50% to 70% of gastric cancers in East Asian countries are detected as early gastric cancer. Endoscopic submucosal dissection (ESD) is a minimally invasive treatment for early gastric cancer, and according to the Japanese Gastric Cancer Treatment Guidelines, there is almost no risk of lymph node relocation. Therefore, early detection and treatment of gastric cancer is the best way to reduce mortality, and improving the accuracy of endoscopic diagnosis of early gastric cancer is very helpful in reducing gastric cancer incidence and mortality.

Among the root causes of gastric cancer development, Helicobacter pylori (hereinafter sometimes referred to as "H. pylori") infection induces atrophic gastritis and intestinal metaplasia, and eventually leads to the development of gastric cancer. Connect. 98% of non-cardiac gastric cancers worldwide are H. Helicobacter pylori is believed to have contributed. H. Patients infected with Helicobacter pylori are at increased risk of gastric cancer, and H. pylori. Considering that the incidence of gastric cancer after Helicobacter pylori eradication has decreased, the International Agency for Research on Cancer has announced that H. pylori has been eradicated. Helicobacter pylori is classified as a clear carcinogen. From this result, in order to reduce the risk of developing gastric cancer, H. Eradication of Helicobacter pylori is useful, and H. pylori with antibacterial agents. Helicobacter pylori eradication has become an insurance practice in Japan, and will continue to be a treatment method that is strongly encouraged in terms of health and hygiene. In fact, the Ministry of Health, Labor and Welfare of Japan announced in February 2013 that H. Approved health insurance coverage for the eradication treatment of gastritis patients due to Helicobacter pylori infection.

H. Gastroscopy provides extremely useful information for the differential diagnosis of the presence of Helicobacter pylori infection. If the capillaries look clean (RAC (regular arrangement of collecting venules)) or fundic gland polyps are H. Characteristic of Helicobacter pylori-negative gastric mucosa, atrophy, redness, mucosal swelling, and wrinkled wall hypertrophy are associated with H. pylori. This is a typical finding of Helicobacter pylori-infected gastritis. In addition, patchy red spots are described in H. It is a characteristic of the gastric mucosa that has been sterilized by Helicobacter pylori. H. Accurate endoscopic diagnosis of Helicobacter pylori infection is based on anti-H. pylori in blood or urine. Patients who are confirmed by various tests such as Helicobacter pylori IgG level measurement, fecal antigen measurement, urea breath test, or rapid urease test and have a positive test result are H. pylori. You can proceed to Helicobacter pylori eradication. Endoscopy is widely used to examine gastric lesions, but when confirming gastric lesions without clinical specimen analysis, H. If even Helicobacter pylori infection can be identified, the burden on patients will be greatly reduced without performing uniform blood tests and urine tests, and it can be expected to contribute to the medical economy.

In this way, endoscopy of the upper digestive organs and large intestine has become widespread, but endoscopy of the small intestine makes it difficult to insert a general endoscope into the inside of the small intestine. Therefore, it is not often done. A general endoscope is about 2 m in length, and to insert the endoscope into the small intestine, it is taken orally via the stomach and duodenum, or transanally via the large intestine to the small intestine. This is because it is necessary to insert the small intestine, and since the small intestine itself is a long organ as long as 6-7 m, it is difficult to insert and observe the entire small intestine with a general endoscope. Therefore, for endoscopy of the small intestine, double-balloon endoscopy (see Patent Document 1) or wireless capsule endoscopy (hereinafter sometimes referred to simply as "WCE") (see Patent Document 2). Is used.

In a double-balloon endoscope, a balloon provided on the tip side of the endoscope and a balloon provided on the tip side of an overtube covering the endoscope are inflated or deflated alternately or at the same time. It is a method of performing an examination while shortening and straightening the long small intestine by pulling it together, but it is difficult to perform an examination over the entire length of the small intestine at one time because the length of the small intestine is long. Therefore, the examination of the small intestine by double-balloon endoscopy is usually performed in two parts, an oral endoscopy and a transanal endoscopy.

In addition, endoscopy by WCE swallows an orally ingestible capsule with a built-in camera, flash, battery, transmitter, etc., and wirelessly transmits the image taken while the capsule is moving in the digestive tract to the outside. The examination is performed by receiving and recording this externally, and it is possible to take an image of the entire small intestine at one time.

The most common symptoms in the small intestine found by WCE are mucosal disorders such as erosions and ulcers, which are mainly caused by nonsteroidal anti-inflammatory drugs (NSAIDs) and sometimes Crohn's disease or malignant small intestine. Early diagnosis and early treatment are needed because it is caused by the tumor. According to various previous reports, the part where the mucous membrane is destroyed due to erosion or ulcer of the small intestine has a small color difference from the surrounding normal mucosa, so vasodilation is detected for automatic detection by software. It was inferior to the case (see Non-Patent Document 1).

In addition, pharyngeal cancer is often detected at an advanced stage, and the prognosis is poor. In addition, patients with advanced laryngeal cancer require surgical resection and chemoradiotherapy, have both cosmetic problems and dysphagia and speech loss, resulting in a significant reduction in quality of life. .. Previously, in esophagogastroduodenal endoscopy (EGD) examination, it was considered important for the endoscope to pass through the pharynx quickly in order to reduce the discomfort of the patient, and observation of the pharynx was not established. .. Endoscopes cannot use iodine staining in the pharynx because of the risk of aspiration into the airways, unlike in the esophagus. Therefore, superficial pharyngeal cancer was rarely detected.

However, in recent years, with the development of image-enhanced endoscopy such as narrow-band imaging (NBI) and the increase in the awareness of endoscopists, the detection of pharyngeal cancer during esophagogastric duodenal endoscopy has increased. There is. With increasing detection of superficial pharyngeal cancer (SPC), endoscopic resection (ER) and endoscopic submucosal dissection (ESD) have been established as local resections of superficial gastrointestinal cancer. , Or endoscopic mucosal resection (EMR) provides an opportunity to treat superficial laryngeal cancer. In addition, short-term and long-term favorable treatment results by endoscopic submucosal dissection for superficial pharyngeal cancer, which is an ideal minimally invasive treatment for maintaining the function and quality of life of patients with superficial pharyngeal cancer. Has also been reported.

In addition, white photoacoustic imaging (WLI) in esophagogastric duodenoscopy is the most sensitive method for early detection of gastric cancer. However, accurate diagnosis of early gastric cancer can be difficult with WLI alone, especially for small lesions. Among endoscopy by narrow band imaging, NBI combined magnifying endoscopy (ME-NBI (magnifying endoscopy with narrow-band imaging)) is a recently developed image-enhanced endoscopy technique. In the diagnosis of early gastric cancer by ME-NBI, VSCS (vessel plus surface classification system: a diagnostic system that makes a differential diagnosis of cancer and non-cancer using anatomical structures and indicators visualized by narrow band imaging) It has been clarified that it is very useful for distinguishing between early gastric cancer and non-cancer, and that each cancer can be diagnosed with a smaller number of biopsies (see Non-Patent Document 19).

Recently, a simple diagnostic algorithm for early gastric cancer (Magnifying Endoscopy Simple Diagnostic Algorithm for Early Gastric Cancer) has been proposed as a unified system for the diagnosis of early gastric cancer by ME-NBI, and has become widely known. (See Non-Patent Document 20). VSCS is an important diagnostic algorithm and serves as the basis for the MESDA-G algorithm. ME-NBI is believed to make a significant contribution to clinical practice, but these benefits have been reported primarily based on results obtained by endoscopists (experts) and ME-using VSCS. Acquiring NBI diagnostic skills requires considerable expertise and experience.

Japanese Unexamined Patent Publication No. 2002-301019 Japanese Unexamined Patent Publication No. 2006-095304 Japanese Unexamined Patent Publication No. 2017-045341 Japanese Unexamined Patent Publication No. 2017-067489 International Publication WO2018 / 225448

Iakovidis DK, Koulaouzidis A. Automatic lesion detection based on capsule endoscopy based on color saliency: closer to an essential adjunct for reviewing software. Gastrointestinal endoscopy 2014 Nov; 80 (5): 877-83. Bibault JE, Giraud P, Burgun A. Big Data and machine learning in radiation oncology: State of the art and future prospects. Cancer Lett. 2016; 382 (1): 110-117. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542 (7639): 115-118. Gulshan V, Peng L, Coram M, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographys. JAMA. 2016; 316 (22): 2402-2410. Byrne MF, Chapados N, Soudan F, et al. Real-time differentiation of adenomatous and hyperplastic diminutive colorectal polyps during analysis of unaltered videos of standard colonoscopy using a deep learning model. Chen PJ, Lin MC, Lai MJ, Lin JC, Lu HH, Tseng VS. Accurate Classification of Diminutive Colorectal Polyps Using Computer-Aided Analysis. Gastroenterology. 2018; 154 (3): 568-575. Misawa M, Kudo SE, Mori Y, et al. Artificial Intelligence-Assisted Polyp Detection for Colonoscopy: Initial Experience. Gastroenterology. 2018. Takiyama H, Ozawa T, Ishihara S, et al. Automatic anatomical classification of esophagogastroduodenoscopy images using deep convolutional neural networks. Sci Rep. 2018; 8 (1): 7497. Hirasawa T, Aoyama K, Tanimoto T, et al. Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images. Gastric Cancer. 2018. Shichijo S, Nomura S, Aoyama K, et al. Application of Convolutional Neural Networks in the Diagnosis of Helicobacter pylori Infection Based on Endoscopic Images. EBioMedicine. 2017; 25: 106-111. Aoki T, Yamada A, Aoyama K, Saito H, Tsuboi A, Nakada A, Niikura R, Fujishiro M, Oka S, Ishihara S et al: Automatic detection of erosions . Gastrointest Endosc 2019, 89 (2): 357-363 e352. Ding Z, Shi H, Zhang H, Meng L, Fan M, Han C, Zhang K, Ming F, Xie X, Liu H et al: Gastroenterologist-Level Identification of Small-Bowel Diseases and Normal Variants -Learning Model. Gastroenterology 2019, 157 (4): 1044-1054 e1045. Tsuboi A, Oka S, Aoyama K, Saito H, Aoki T, Yamada A, Matsuda T, Fujishiro M, Ishihara S, Nakahori M et al: Artificial intelligence using a convolutional neural network for images. Dig Endosc 2019 (1443-1661 (Electronic)). Korman LY, Delvaux M, Hagenmuller F, Keuchel M, Friedman S, Weinstein M, Shetzline M, Cave D, de Franchis R: Capsule endoscopy structured terminology (CEST): proposal 2005, 37 (10): 951-959. Boal Carvalho P, Magalhaes J, Dias DE Castro F, et al. Suspected blood indicator in capsule endoscopy: a valuable tool for gastrointestinal bleeding diagnosis. Arq Gastroenterol. 2017; 54 (1): 16-20 Hosoe N, Rey JF, Imaeda H, et al. Evaluations of capsule endoscopy software in reducing the reading time and the rate of false negatives by inexperienced endoscopists. Clin.Res.Hepatol. .. Aoki T, Yamada A, Hirata Y, et al. Etiology and long-term rebleeding of endoscopic ulcerative lesions in the small bowel in patients with obscure gastrointestinal bleeding: A multicenter cohort study. : 1327-1334. Yano T, Yamamoto H, Sunada K, et al. Endoscopic classification of vascular lesions of the small intestine (with videos). Gastrointest. Endosc. 2008; 67 (1): 169-172. Yao K, Anagnostopoulos GK, Ragunath K. Magnifying endoscopy for diagnosticing and delining early gastric cancer. Endoscopy 2009; 41: 462-467 Muto M, Yao K, Kaise M et al. Magnifying endoscopy simple diagnostic algorithm for early gastric cancer (MESDA-G). Dig Endosc 2016; 28: 379-393 Kaiming He, Xiangyu Zhang, Shaoqing Ren at al. Deep Residual Learning for Image Recognition. ArXiv: 1512.03385. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, et al. Grad-CAM: Visual Expansions from Deep Networks via Gradient-based Localization. ArXiv: 1610.02391.

In such endoscopy of the digestive organs, many endoscopic images are collected, but double-checking of the endoscopic images by an endoscopist is obligatory for quality control. With tens of thousands of endoscopic examinations a year, the number of images read by an endoscopist in secondary interpretation is enormous, about 2,800 per person per hour, which is a heavy burden on the site. It has become.

Especially in the examination by WCE of the small intestine, the movement of WCE is not due to the movement of WCE itself, but due to the peristalsis of the intestine, so the movement cannot be regulated from the outside. Therefore, in order to prevent oversight, a large number of images are taken in one examination, and since the WCE takes about 8 hours to move in the small intestine, the number of images taken in one examination is very large. For example, WCE wirelessly sends about 60,000 images per person, so endoscopists will have to fast-forward and check, but unusual findings may only appear in one or two frames. Therefore, the average WCE image analysis by this requires 30-120 minutes of strict attention and concentration. Moreover, in recent years, moving images have been taken by WCE, and the burden on endoscopists for checking becomes extremely large.

Moreover, these endoscopic image-based diagnoses are not only time-consuming to train endoscopists and check stored images, but are also subjective, with various false-positive and false-negative judgments. May occur. In addition, the diagnosis by an endoscopist may be less accurate due to fatigue. Such a large burden on the site and a decrease in accuracy may lead to a limit on the number of examinees, and there is a concern that medical services according to demand will not be sufficiently provided.

In addition, recently, H. The incidence of gastric cancer uninfected with H. pylori or after eradication is increasing. Endoscopic diagnosis is considered to be particularly difficult due to factors such as the fact that these gastric cancers are tumors with a very low degree of atypia and the surface layer is covered with non-tumor mucosa. Since the number of cases of early gastric cancer (EGC), which is difficult to diagnose endoscopically, is expected to continue to increase in the future, more accurate endoscopic diagnosis of early gastric cancer using ME-NBI as well as WLI. Development of technology is required. Moreover, in early gastric cancer (EGC), the progress of endoscopic submucosal dissection (ESD) has made it possible to perform batch resection of tumors regardless of size or the presence or absence of ulcers. Preoperative diagnosis is required.

It is expected that AI (artificial intelligence) will be used to improve the labor load and accuracy deterioration of the above endoscopy. If AI, whose image recognition ability exceeds that of humans in recent years, can be used as an assist for endoscopists, it is expected to improve the accuracy and speed of secondary interpretation work.

In recent years, AI using deep learning has attracted attention in various medical fields, such as radiation oncology, skin cancer classification, diabetic retinopathy (see Non-Patent Document 2-4), and gastrointestinal endoscopy. It has been reported that medical images can be screened on behalf of specialists not only in the field of mirroring, especially in the field including colonoscopy (see Non-Patent Document 5-7), but also in various medical fields. In addition, there are patent documents (see Patent Documents 3 and 4) in which medical image diagnosis is performed using various AIs. However, it has not been sufficiently verified whether AI's endoscopic diagnostic imaging ability can satisfy the accuracy (accuracy) and performance (speed) that are useful in actual medical practice, and various endoscopes using AI have not been fully verified. Image-based diagnosis has not yet been put to practical use.

Deep learning can learn higher-order features from input data using a neural network configured by stacking multiple layers. Deep learning also uses a backpropagation algorithm to update the internal parameters used to calculate the representation of each layer from the representation of the previous layer by showing how the device should change. can do.

In associating medical images, deep learning can be a powerful machine learning technique that can be trained with previously accumulated medical images and can directly derive the patient's clinical features from the medical images. .. Neural networks are mathematical models that express the characteristics of neural circuits in the brain by computer simulations, and the algorithmic approach that supports deep learning is neural networks. Convolutional Neural Networks (CNN), developed by Szegedy et al., Is the most common network architecture for deep learning of images.

The inventors have already constructed a CNN system that can classify images of the esophagus, stomach, and duodenum according to anatomical sites and can reliably find gastric cancer in endoscopic images (non-patent documents). See 8 and 9). Furthermore, the inventors, etc., have described H.I. We report the role of CNN in the diagnosis of Helicobacter pylori gastritis, and show that the ability of CNN is comparable to that of an experienced endoscopist, and the diagnosis time is considerably shortened (see Patent Document 5 and Non-Patent Document 10).

In addition, CNN is expected to be applied to WCE because it can process a large amount of images automatically and quickly, and it is possible to achieve good results in some WCE fields. Is also shown (see Non-Patent Documents 11-13). However, there are still many unclear points about applying CNN to these WCE images of the small intestine to diagnose various diseases of the small intestine and bleeding or elevated lesions, and further improvements can be made. It has been demanded. In particular, there is a great demand to reduce the burden on endoscopists by applying CNN to the moving image of WCE of the small intestine and automatically detecting various types of the small intestine.

The present invention has been made to solve the above-mentioned problems of the prior art. That is, an object of the present invention is to accurately detect the presence or absence of mucosal damage (erosion / ulcer), vasodilation, elevated lesion or bleeding of the small intestine using the CNN system based on the endoscopic image or moving image of the small intestine by WCE. It is an object of the present invention to provide a method for supporting diagnosis of a disease of the small intestine that can be identified, a diagnosis support system, a diagnosis support program, and a computer-readable recording medium that stores the diagnosis support program.

Another object of the present invention is a diagnostic support method, a diagnostic support system, a diagnostic support program, and a diagnostic support program for early gastric cancer that can accurately diagnose early gastric cancer using a CNN system based on ME-NBI images. It is an object of the present invention to provide a computer-readable recording medium in which a device is stored.

Furthermore, another object of the present invention is that when diagnosing early gastric cancer using the CNN system, the depth of invasion of the lesion can be diagnosed more accurately, and the applicability of ESD can be accurately diagnosed. An object of the present invention is to provide a diagnostic support method, a diagnostic support system, a diagnostic support program, and a computer-readable recording medium that stores the diagnostic support program.

The method for supporting the diagnosis of a disease by the endoscopic image of the digestive organ using the CNN of the first aspect of the present invention corresponds to the first endoscopic image of the digestive organ and the first endoscopic image. The CNN system was trained using at least one definitive diagnosis result of information corresponding to the positive or negative of the digestive organ disease, and the trained CNN system was input from the endoscopic image input unit. Digestion using the CNN system, which detects the disease in the digestive organ and outputs at least one of the regions corresponding to the positive of the disease and the probability score based on the second endoscopic image of the digestive organ. A method for supporting the diagnosis of a disease using an endoscopic image of an organ, wherein the endoscopic image is a WCE image of the small intestine, and the trained CNN system is a second input from the endoscopic image input unit. It is characterized by outputting the region of the elevated lesion as the disease detected in the WCE image of the above.

According to the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the first aspect of the present invention, WCE images of a plurality of small intestines for which the CNN system is obtained in advance for each of a plurality of subjects. Since it is trained based on the first endoscopic image consisting of the above and the positive or negative definitive diagnosis result of the disease obtained in advance for each of the plurality of subjects, it is substantially internal in a short time. With accuracy comparable to that of an endoscopist, it is possible to obtain a region or probability score corresponding to the positive of the digestive organ disease of the subject, and it is possible to select the subject who must make a definitive diagnosis in a short time. Become.

Moreover, according to the method for supporting the diagnosis of a disease by endoscopic images of digestive organs using the CNN system of the first aspect of the present invention, it is shorter than the endoscopic images of the small intestine by WCE for a large number of subjects. In time, it is possible to obtain the presence or absence of an elevated lesion of the small intestine and its region if it exists, and the probability score of that region with an accuracy substantially comparable to that of an endoscopist, and a definitive diagnosis is made separately. It will be possible to select the subjects who must be selected in a short time, and it will be easy for the endoscopist to check / correct. The elevated lesions of such an embodiment include not only polyps but also nodules, epithelial tumors, submucosal tumors, vascular structures and the like.

The method for supporting the diagnosis of a disease by an endoscopic image of the digestive organ using the CNN system of the second aspect of the present invention is a method of supporting the diagnosis of a disease by an endoscopic image of the digestive organ using the CNN system of the first aspect. In the method, the trained convolutional neural network system is characterized by further displaying the probability score of the disease in the second endoscopic image.

According to the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the second aspect of the present invention, a definitive diagnosis result by an endoscopic specialist can be obtained in the second endoscopic image. Accurate contrast between the areas identified and the areas positive for the disease detected by the trained CNN system can result in better sensitivity and specificity of the CNN.

Further, the method for supporting the diagnosis of a disease by an endoscopic image of a digestive organ using the CNN system of the third aspect of the present invention is a method of supporting a diagnosis of a disease by an endoscopic image of the digestive organ using the CNN system of the second aspect. In the diagnostic support method, the trained convolutional neural network system is associated with the area of the elevated lesion based on the positive or negative definitive diagnosis of the disease in the small intestine displayed in the second endoscopic image. , The area of the elevated lesion of the second endoscopic image detected by the trained convolutional neural network system is displayed, and the above-mentioned based on the definitive diagnosis result displayed in the second endoscopic image. It is characterized in that the correctness of the diagnosis result of the trained convolutional neural network system is determined by the overlap between the region of the raised lesion and the detected region of the raised lesion.

According to the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the third aspect of the present invention, a definitive diagnosis result by an endoscopist can be obtained in the second endoscopic image. Areas that have been identified and areas that are positive for the disease detected by the trained CNN system are displayed so that the overlap of those areas can be immediately compared to the diagnostic results of the trained CNN. become.

Further, the method for supporting the diagnosis of a disease by the endoscopic image of the digestive organ using the CNN system of the fourth aspect of the present invention is the method of supporting the diagnosis of the disease by the endoscopic image of the digestive organ using the CNN system of the third aspect. In the diagnostic support method, the overlap is
(1) When it is 80% or more of the area of the elevated lesion based on the definitive diagnosis result, or
(2) When there are a plurality of positive regions of the disease detected by the trained convolutional neural network system, and any one region overlaps with the region of the elevated lesion based on the definitive diagnosis result. ,
The diagnosis of the trained convolutional neural network system is characterized by determining that it is correct.

According to the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the fourth aspect of the present invention, the correctness of the diagnosis of the CNN system can be easily determined and trained. The accuracy of the diagnosis of the CNN system will be improved, and it will be possible to clarify the direction in which improvement is necessary.

Further, the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the fifth aspect of the present invention is the endoscopic examination of the digestive organs using the CNN system of any one of the first to fourth aspects. In the method of assisting diagnosis of a disease by endoscopy, the trained convolutional neural network system determines that the detected elevated lesion is one of a polyp, a nodule, an epithelial tumor, a submucosal tumor, and a vascular structure. It is characterized by doing.

According to the method for supporting the diagnosis of a disease by endoscopic images of digestive organs using the CNN system of the fifth aspect of the present invention, the CNN system itself is raised even with an endoscopic image of the small intestine by a large amount of WCE. Since the specific type of sexual lesion is determined, the endoscopist can select the subjects who must make a definitive diagnosis in a short time.

The method for supporting the diagnosis of a disease by an endoscopic image of a digestive organ using the CNN system of the sixth aspect of the present invention corresponds to the first endoscopic image of the digestive organ and the first endoscopic image. A convolutional neural network system is trained using at least one definitive diagnosis result of information corresponding to positive or negative of the digestive organ disease, and the trained convolutional neural network system inputs an endoscopic image. Based on the second endoscopic image of the digestive organ input from the unit, the convolution that detects the disease of the digestive organ and outputs at least one of the region corresponding to the positive of the disease and the probability score. A method for supporting disease diagnosis using an endoscopic image of a digestive organ using a neural network system, wherein the endoscopic image is a WCE image of the small intestine, and the trained convolutional neural network system is the endoscope. It is characterized in that the probability score of bleeding as the disease detected in the second WCE image input from the image input unit is output.

According to the method for supporting the diagnosis of a disease by endoscopic images of digestive organs using the CNN system of the sixth aspect, an image containing blood components of the small intestine and a normal mucosal image can be distinguished accurately and at high speed. Easy to check / correct by an endoscopist.

The method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the seventh aspect of the present invention corresponds to the first endoscopic image of the digestive organs and the first endoscopic image. The convolutional neural network system is trained using the definitive diagnosis result of the information corresponding to the positive or negative of the digestive organ disease, and the trained convolutional neural network system is input from the endoscopic image input unit. A convolutional neural network system that detects the disease in the digestive organ and outputs at least one of a region corresponding to the positive of the disease and a probability score based on the second endoscopic image of the digestive organ. The first endoscopic image is a still image of the WCE of the small intestine, and the second endoscopic image is the WCE of the small intestine. The trained convolutional neural network system displays the region of the disease detected in the second WCE image input from the endoscopic image input unit.

According to the method for assisting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the seventh aspect of the present invention, since the CNN system is accurately trained by the still image by WCE of the small intestine, the small intestine Since virtually all diseases can be picked up without overlooking the disease in the moving image by WCE, the endoscopist can select the subjects who must make a definitive diagnosis in a short time. In the method for supporting the diagnosis of diseases by endoscopic images of the digestive organs using the CNN system of the above aspect, not only continuous moving images captured by a so-called video camera but also still images are used as moving images by WCE of the small intestine. It also includes images taken so that the imaging interval is shortened so that the image can be regarded as a moving image.

The method for supporting the diagnosis of a disease by the endoscopic image of the digestive organ using the CNN system of the eighth aspect of the present invention is the diagnosis support of the disease by the endoscopic image of the digestive organ using the CNN system of the seventh aspect. The method is characterized in that the area of the disease is at least one of mucosal disorders, vasodilation, elevated lesions, and bleeding.

According to the method for supporting the diagnosis of diseases by endoscopic images of the digestive organs using the CNN system of the eighth aspect of the present invention, at least one of mucosal disorders, vasodilatation, elevated lesions, and bleeding even with one CNN system. Since the two diseases can be detected individually or simultaneously, the endoscopist can quickly select the subjects for whom a separate definitive diagnosis must be made.

The method for supporting the diagnosis of a disease by an endoscopic image of a digestive organ using the CNN system of the ninth aspect of the present invention is a method of supporting the diagnosis of a disease by an endoscopic image of a digestive organ using the CNN system of the eighth aspect. In the method, the area of mucosal damage is at least one of erosions and ulcers, the area of vasodilation is at least one of vasodilator 1a and vasodilator 1b, and the area of elevated lesions. Is characterized by at least one of polyps, nodules, submucosal tumors, vascular structures and epithelial tumors.

According to the method for supporting the diagnosis of diseases by endoscopic images of digestive organs using the CNNCNN system of the ninth aspect of the present invention, at least one of mucosal disorders, vasodilatation, elevated lesions and bleeding even with one CNN system. Not only can one disease be detected individually or simultaneously, but mucosal disorders, vasodilators and elevated lesions can be further subdivided and detected individually or simultaneously. The endoscopist will be able to quickly and in detail select subjects for whom a separate definitive diagnosis must be made.

The method for supporting the diagnosis of a disease by an endoscopic image of a digestive organ using the CNN system of the tenth aspect of the present invention is a method of supporting the diagnosis of a disease by an endoscopic image of the digestive organ using the CNN system of the seventh-9th aspect. In the diagnostic support method, the trained CNN system has a first CNN system portion in which the definitive diagnosis result is trained by a still image of mucosal damage and a second CNN system whose definitive diagnosis result is trained by a still image of vasodilatory disease. CNN system part, a third CNN system part whose definitive diagnosis results were trained by still images of elevated lesions, and a fourth CNN system part whose definitive diagnosis results were trained by still images of bleeding. It is characterized by using a complex CNN system.

Training the CNN system with definitive diagnostic results including still images of mucosal disorders and vasodilators reduces the detectability of mucosal disorders and nodules. According to the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the tenth aspect of the present invention, the CNN system using a still image of mucosal damage and a still image of vasodilator as a definitive diagnosis result. However, because it is separated from the CNN system, which uses still images of definitive diagnostic results of other diseases, it can accurately detect at least one of four types: mucosal damage, vasodilation, elevated lesions, and bleeding. become able to.

The method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the eleventh aspect of the present invention corresponds to the first endoscopic image of the digestive organs and the first endoscopic image. The convolutional neural network system is trained using at least one definitive diagnosis result of the information corresponding to the positive or negative of the digestive organ disease, and the trained convolutional neural network system is used for endoscopic image input. Convolution that detects the disease in the digestive organs and outputs at least one of the regions corresponding to the positives of the disease and the probability score based on the second endoscopic image of the digestive organs input from the unit. It is a method for assisting diagnosis of a disease by endoscopic images of the digestive organs using a neural network system, in which the endoscopic images are endoscopic images by water invasion narrow band imaging of the stomach, and the trained convolution. The neural network system is characterized in that the region of early gastric cancer as the disease is output from the endoscopic image input from the endoscopic image input unit.

According to the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the eleventh aspect of the present invention, the endoscopic images by the inundation method narrow band imaging of the stomach have less halation and are endoscopic. Since it is possible to generate a well-focused and clear image with uniform quality suitable for endoscopic diagnosis, H. Even when it is difficult to diagnose with conventional endoscopic images such as gastric cancer after Helicobacter pylori eradication and uninfected gastric cancer, early gastric cancer can be diagnosed with high diagnostic accuracy.

The method for supporting the diagnosis of a disease by the endoscopic image of the digestive organ using the CNN system of the twelfth aspect of the present invention is the diagnosis support of the disease by the endoscopic image of the digestive organ using the CNN system of the eleventh aspect. In the method, the trained convolutional neural network network system is characterized by having a function of displaying an area of early gastric cancer as the disease on a heat map.

According to the method for supporting the diagnosis of diseases by endoscopic images of the digestive organs using the CNN system of the twelfth aspect of the present invention, the heat map is a color corresponding to the concept for the point of interest, here the probability value of gastric cancer. For example, the higher the probability value of gastric cancer, the darker the red color can be displayed. Therefore, the part of the second endoscopic image where there is a possibility of gastric cancer and the magnitude of the probability of gastric cancer at that part should be confirmed at a glance. Will be able to.

The method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the thirteenth aspect of the present invention corresponds to the first endoscopic image of the digestive organs and the first endoscopic image. The convolutional neural network system is trained using at least one definitive diagnosis result of the information corresponding to the positive or negative of the digestive organ disease, and the trained convolutional neural network system is used for endoscopic image input. Convolution that detects the disease in the digestive organs and outputs at least one of the regions corresponding to the positives of the disease and the probability score based on the second endoscopic image of the digestive organs input from the unit. It is a method for supporting the diagnosis of diseases by endoscopic images of the digestive organs using a neural network system, and the endoscopic images are selected from a white light image of the stomach, a non-magnifying narrow band light image, and an indigocarmine dye spray image. The trained convolutional neural network system is characterized in that it outputs the depth of invasion of the disease as the disease of the endoscopic image input from the endoscopic image input unit. To do.

According to the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the thirteenth aspect of the present invention, the invasion depth (infiltration depth) of the disease can be accurately known, and thus early gastric cancer. You will be able to accurately diagnose whether you have advanced gastric cancer.

The method for supporting the diagnosis of a disease by the endoscopic image of the digestive organ using the CNN system of the 14th aspect of the present invention is the diagnostic support of the disease by the endoscopic image of the digestive organ using the CNN system of the 13th aspect. The method is characterized in that the depth of invasion is output as to whether the submucosal invasion is less than 500 μm or more than 500 μm.

If the submucosal infiltration of the disease is less than 500 μm, endoscopic submucosal dissection (ESD) may achieve radical resection. According to the method for supporting the diagnosis of diseases by endoscopic images of digestive organs using the CNN system of the 14th aspect of the present invention, the depth of invasion under the mucosa can be accurately diagnosed. It will be possible to accurately distinguish whether the case is applicable to ESD or requires surgery.

The method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system according to the fifteenth aspect of the present invention is the convolutional neural network in the method for supporting the diagnosis of any one of the specific aspects of the first to fourteenth aspects. Is further characterized in that it is combined with three-dimensional information from an X-ray computed tomography apparatus, an ultrasonic computed tomography apparatus or a magnetic resonance imaging apparatus.

Since the X-ray computed tomography apparatus, the ultrasonic computed tomography apparatus, or the magnetic resonance imaging apparatus can represent the structure of each digestive organ three-dimensionally, the CNN system according to any one of the first to fourteenth aspects can be used. When combined with the output, it becomes possible to more accurately grasp the part where the endoscopic image was taken.

The method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system according to the sixteenth aspect of the present invention is the method for supporting the diagnosis of a disease according to any one of the first to fourteenth aspects. The image may be an image being taken with an endoscope, an image transmitted via a communication network, an image provided by a remote control system or a cloud-based system, an image recorded on a computer-readable recording medium, or an image. , At least one of the moving images.

According to the method for supporting the diagnosis of diseases by endoscopic images of the digestive organs using the CNN system of the 16th aspect, the probabilities of positive and negative diseases of the digestive organs with respect to the input second endoscopic image are different. Since the severity can be output in a short time, it can be used regardless of the input format of the second endoscopic image, for example, an image transmitted from a remote place or a moving image. As the communication network, the well-known Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication network, etc. are used. It is possible. In addition, the transmission media constituting the communication network are well-known IEEE1394 serial bus, USB, power line carrier, cable TV line, telephone line, ADSL line and other wired, infrared, Bluetooth (registered trademark), IEEE802.11 and other wireless. Wireless networks such as mobile phone networks, satellite lines, and terrestrial digital networks can be used. As a result, it can be used as a form of so-called cloud service or remote support service.

Computer-readable recording media include well-known tape systems such as magnetic tapes and cassette tapes, floppy (registered trademark) disks, magnetic disks such as hard disks, and compact disks-ROM / MO / MD / digital video discs / compact. A disk system including an optical disk such as a disk-R, a card system such as an IC card, a memory card, and an optical card, or a semiconductor memory system such as a mask ROM / EPROM / EPROM / flash ROM can be used. As a result, it is possible to provide a form in which the system can be easily transplanted or installed in a so-called medical institution or a medical examination institution.

Further, the disease diagnosis support system using the endoscopic image of the digestive organ using the CNN system of the 17th aspect of the present invention includes an endoscopic image input unit, an output unit, a computer incorporating the CNN, and a computer. It is a disease diagnosis support system by an endoscopic image of a digestive organ using a CNN system having the above, and the computer is a first storage area for storing a first endoscopic image of the digestive organ and the first storage area. A second storage area for storing definitive diagnosis results of information corresponding to the positive and negative of the disease in the digestive organ, which corresponds to the endoscopic image of 1, and a third storage area for storing the CNN program. The CNN program is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area. Corresponds to the positive or negative of the disease of the digestive organ with respect to the second endoscopic image based on the second endoscopic image of the digestive organ input from the endoscopic image input unit. Information is supposed to be output to the output unit, the endoscopic image is a WCE endoscopic image of the small intestine, and the trained CNN program is a WCE input from the endoscopic image input unit. It is characterized by outputting an area of an elevated lesion as the disease in an endoscopic image.

Further, the disease diagnosis support system using the endoscopic image of the digestive organ using the CNN system of the eighteenth aspect of the present invention is an endoscopic image of the digestive organ using the CNN system of the seventeenth aspect of the present invention. In the disease diagnosis support system according to the above, the trained CNN program is characterized in that the probability score of the disease is further displayed in the second endoscopic image.

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system of the 19th aspect of the present invention is an endoscopic image of the digestive organ using the CNN system of the 17th aspect of the present invention. In the disease diagnosis support system according to the above, the CNN program includes the area of the elevated lesion displayed in the second endoscopic image based on the positive or negative definitive diagnosis result of the disease in the small intestine. The area of the elevated lesion detected by the trained CNN system is displayed, and the area of the elevated lesion based on the definitive diagnosis result displayed in the second endoscopic image and the area of the elevated lesion are displayed. It is characterized in that the correctness of the diagnosis result is determined by the overlap with the detected area of the elevated lesion.

Further, the disease diagnosis support system using the endoscopic image of the digestive organ using the CNN system of the twentieth aspect of the present invention is an endoscopic image of the digestive organ using the CNN system of the seventeenth aspect of the present invention. In the disease diagnosis support system based on
(1) When it is 80% or more of the area of the elevated lesion based on the definitive diagnosis result, or
(2) When there are a plurality of positive regions of the disease detected by the trained CNN system, and any one region overlaps with the region of the elevated lesion based on the definitive diagnosis result.
The diagnosis of the trained CNN system is characterized by determining that it is correct.

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system of the 21st aspect of the present invention is an endoscopic image of the digestive organ using the CNN system of the 17th aspect of the present invention. In the disease diagnosis support system according to the above, the trained CNN program indicates in the second image that the elevated lesion is either a polyp, a nodule, an epithelial tumor, a submucosal tumor or a vascular structure. It is characterized by doing.

Further, the disease diagnosis support system using the endoscopic image of the digestive organ using the CNN system of the 22nd aspect of the present invention includes an endoscopic image input unit, an output unit, a computer incorporating a CNN, and a computer. It is a disease diagnosis support system by an endoscopic image of a digestive organ using a CNN system having the above, and the computer is a first storage area for storing a first endoscopic image of the digestive organ and the first storage area. A second storage area for storing definitive diagnosis results of information corresponding to the positive and negative of the disease in the digestive organ, which corresponds to the endoscopic image of 1, and a third storage area for storing the CNN program. The CNN program is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area. Corresponds to the positive or negative of the disease of the digestive organ with respect to the second endoscopic image based on the second endoscopic image of the digestive organ input from the endoscopic image input unit. The information is supposed to be output to the output unit, the endoscopic image is a WCE image of the small intestine, and the trained CNN program displays the probability score of bleeding as the disease in the second image. It is characterized by displaying.

Further, the disease diagnosis support system using the endoscopic image of the digestive organ using the CNN system of the 23rd aspect of the present invention includes an endoscopic image input unit, an output unit, a computer incorporating the CNN, and a computer. It is a disease diagnosis support system by an endoscopic image of a digestive organ using a CNN system having the above, and the computer is a first storage area for storing a first endoscopic image of the digestive organ and the first storage area. A second storage area for storing definitive diagnosis results of information corresponding to the positive and negative of the disease in the digestive organ, which corresponds to the endoscopic image of 1, and a third storage area for storing the CNN program. The CNN program is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area. Corresponds to the positive or negative of the disease of the digestive organ with respect to the second endoscopic image based on the second endoscopic image of the digestive organ input from the endoscopic image input unit. Information is supposed to be output to the output unit, the first endoscopic image is a still image of the WCE of the small intestine, and the second endoscopic image is a moving image of the WCE of the small intestine. The trained CNN program is characterized in displaying the area of the disease detected in a second WCE image input from the endoscopic image input unit.

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system of the 24th aspect of the present invention is an endoscopic image of the digestive organ using the CNN system of the 23rd aspect of the present invention. In the disease diagnosis support system according to the above, the area of the disease is at least one of mucosal disorders, vasodilators, elevated lesions, and bleeding.

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system of the 25th aspect of the present invention is an endoscopic image of the digestive organ using the CNN system of the 24th aspect of the present invention. In the disease diagnosis support system according to the above, the area of mucosal damage is at least one of erosion and ulcer, and the area of vasodilation is at least one of vasodilator type 1a and vasodilator type 1b. The area of the elevated lesion is characterized by at least one of polyps, nodules, submucosal tumors, vascular structures and epithelial tumors.

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system according to the 26th aspect of the present invention is the digestive organ using the CNN system according to any 23-25 aspect of the present invention. In the disease diagnosis support system using endoscopic images, the trained CNN system has a definitive diagnosis result of a first CNN system portion trained by a still image of mucosal damage and a definitive diagnosis result of vasodilatory disease. A second CNN system part trained by still images,
From a composite CNN system of a third CNN system part where the definitive diagnosis was trained with a still image of an elevated lesion and a fourth CNN system part where the definitive diagnosis was trained with a still image of bleeding. It is characterized by becoming.

Further, the disease diagnosis support system using the endoscopic image of the digestive organ using the CNN system according to the 27th aspect of the present invention is a computer incorporating an endoscopic image input unit, an output unit, and a convolutional neural network. It is a disease diagnosis support system by endoscopic images of the digestive organs using a convolutional neural network system having
The computer
A first storage area for storing the first endoscopic image of the digestive organs,
A second storage area corresponding to the first endoscopic image and storing a definitive diagnosis result of information corresponding to the positive and negative of the disease in the digestive organs.
A third storage area for storing the convolutional neural network program,
With
The convolutional neural network program
It is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area.
Based on the second endoscopic image of the digestive organs input from the endoscopic image input unit, the information corresponding to the positive or negative of the disease of the digestive organs with respect to the second endoscopic image is described. It is supposed to be output to the output section,
The endoscopic image is an endoscopic image obtained by water invasion narrow band imaging of the stomach, and the trained convolutional neural network program is a disease of the endoscopic image input from the endoscopic image input unit. It is characterized by outputting the area of early gastric cancer.

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system of the 28th aspect of the present invention is an endoscopic image of the digestive organ using the CNN system of the 27th aspect of the present invention. In the disease diagnosis support system according to the above, the trained convolutional neural network program is characterized by having a function of displaying an area of early gastric cancer as the disease on a heat map.

Further, the disease diagnosis support system using the endoscopic image of the digestive organ using the CNN system according to the 29th aspect of the present invention is a computer incorporating an endoscopic image input unit, an output unit, and a convolutional neural network. A disease diagnosis support system using an endoscopic image of a digestive organ using a convolutional neural network system, wherein the computer stores a first endoscopic image of the digestive organ. A second storage area for storing definitive diagnosis results of information corresponding to the positive and negative of the disease in the digestive organ corresponding to the first endoscopic image, and the convolutional neural network program are stored. The convolutional neural network program includes a third storage area, the first endoscopic image stored in the first storage unit, and a determination stored in the second storage area. The above-mentioned digestive organs with respect to the second endoscopic image based on the second endoscopic image of the digestive organs input from the endoscopic image input unit, which is trained based on the diagnosis result. Information corresponding to the positive or negative of the disease is to be output to the output unit, and the endoscopic image is selected from a white light image of the stomach, a non-magnifying narrow band light image, and an indigocarmine dye spray image. At least one, the trained convolutional neural network program is characterized in that it outputs the invasion depth of the disease as the disease of the endoscopic image input from the endoscopic image input unit. ..

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system of the thirtieth aspect of the present invention is an endoscopic image of the digestive organ using the CNN system of the 29th aspect of the present invention. In the disease diagnosis support system according to the above, the trained convolutional neural network program has a function of outputting whether the submucosal invasion is less than 500 μm or more than 500 μm as the invasion depth of the disease. It is characterized by.

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system according to the 31st aspect of the present invention is the digestive organ using the CNN system according to any 17-30 aspect of the present invention. In the disease diagnosis support system using endoscopic images, the CNN program is further combined with three-dimensional information from an X-ray computed tomography apparatus, an ultrasonic computed tomography apparatus, or a magnetic resonance imaging apparatus. It is a feature.

Further, the disease diagnosis support system based on the endoscopic image of the digestive organ using the CNN system according to the 32nd aspect of the present invention is the digestive organ using the CNN system according to any 17-31 of the present invention. In the disease diagnosis support system using the endoscopic image, the second endoscopic image is an image being taken by the endoscope, an image transmitted via a communication network, a remote control system, or a cloud type. It is characterized by being at least one of an image provided by the system, an image recorded on a computer-readable recording medium, or a moving image.

According to the disease diagnosis support system based on the endoscopic image of the digestive organ according to any aspect 17-32 of the present invention, the digestive organ using CNN according to any aspect 1-16 of the present invention, respectively. It can have the same effect as the disease diagnosis support method using endoscopic images.

Further, the diagnostic support program based on the endoscopic image of the digestive organ using the CNN system according to the 33rd aspect of the present invention is the disease based on the endoscopic image of the digestive organ according to any one of 17-32 of the present invention. It is characterized in that it is for operating a computer as each means in the diagnostic support system of.

According to the diagnostic support program based on the endoscopic image of the digestive organ according to the 33rd aspect of the present invention, as each means in the disease diagnosis support system based on the endoscopic image of the digestive organ according to any 17-32 aspect. It is possible to provide a diagnostic support program using endoscopic images of the digestive organs for operating a computer.

Furthermore, the computer-readable recording medium of the 34th aspect of the present invention is characterized by recording a diagnostic support program by endoscopic images of the digestive organs using the CNN system of the 33rd aspect of the present invention. ..

According to the computer-readable recording medium using the endoscopic image of the digestive organ of the 34th aspect of the present invention, the computer-readable recording recording the diagnostic support program by the endoscopic image of the digestive organ of the 33rd aspect. A medium can be provided.

As described above, according to the present invention, a WCE image of the small intestine or various endoscopic images of the stomach obtained in advance for each of a plurality of subjects by a program incorporating CNN, and each of the plurality of subjects in advance. Since it is trained based on the positive or negative definitive diagnosis of the obtained disease, it responds to the positive of the gastrointestinal disease of the subject in a short time and with substantially the same accuracy as the endoscopist. Areas or probability scores can be obtained, and subjects who must make a definitive diagnosis can be selected in a short time.

It is the schematic of the flowchart for the CNN system construction of Embodiment 1. It is a schematic conceptual diagram which shows the operation of backpropagation. It is a figure which shows an example of the ROC curve by CNN of Embodiment 1. FIG. 4A-4D are diagrams showing a representative enteroscopy image correctly diagnosed by the CNN of Embodiment 1 and a probability score of a specific site recognized by the CNN. 5A-5E are examples of images diagnosed as false positives by the CNN of Embodiment 1, respectively, based on darkness, laterality, bubbles, debris, and vasodilation, with FIGS. 5F-5H being true. This is an example of an image that is eroded but diagnosed as a false positive. It is the schematic of the flowchart for the CNN system construction of Embodiment 2. It is a figure which shows an example of the ROC curve by CNN of Embodiment 2. 8A-8E show representative regions correctly detected and classified by the CNN of Embodiment 2, respectively, into five categories: polyps, nodules, epithelial tumors, submucosal tumors and vascular structures. 9A-9C are examples of images of one patient that could not be detected correctly by the CNN of Embodiment 2. Among the false positive images of the second embodiment, this is an example of an image diagnosed by an endoscopist as a true elevated lesion. It is the schematic of the flowchart for the CNN system construction of Embodiment 3. It is a figure which shows an example of the ROC curve by CNN of Embodiment 3. FIG. 13A is an image containing representative blood correctly classified by the CNN system of Embodiment 3, and FIG. 13B is also an image showing a normal mucosal image. FIG. 14A is an image correctly classified as containing a blood component by the red region estimation display function (SBI), and FIG. 14B is an image erroneously classified as a normal mucosa by SBI. It is an example of representative images of various anomalies correctly detected by the CNN system of Embodiment 4. FIG. 16A is a graph showing the difference in the detection rate of the whole disease, and FIG. 16B is a graph showing the difference in the detection rate of each of the four types of diseases such as mucosal disorder, vasodilation, elevated lesion and blood component. 16C is a graph showing the difference in the detection rate for each disease when the diseases shown in FIG. 16B are further subdivided. 17A-17C are graphs corresponding to FIGS. 16A-16C in the case of strict criteria considering single or multiple lesions, respectively. Four false-negative images that could not be detected by CNN in a patient-by-patient analysis based on strict criteria, FIG. 18A is erosion (n = 1), and FIGS. 18B and 18C are vasodilator type 1a (n = 2). And FIG. 18D shows an example of a submucosal tumor (n = 1). This is an example of an endoscopic image for training according to the fifth embodiment. This is an example of a training and test image set of the fifth embodiment. It is a receiver operating characteristic curve of the test data set of Embodiment 5. This is an example of an endoscopic image misdiagnosed by the CNN system of the fifth embodiment. This is an example of a heat map image obtained by occlusion analysis in the CNN system of the fifth embodiment. (A) is an example of an endoscopic image of gastric cancer invading deep submucosa, and (b) is an example of an endoscopic image of gastric cancer perched in the mucosa. FIG. 25A is a diagram showing an example of a gastric cancer ROC curve using a white light image in the CNN system of the sixth embodiment, and FIG. 25B is a diagram showing an example of a gastric cancer ROC curve using a non-enlarged narrow band optical image. FIG. 25C is a diagram showing an example of the ROC curve of gastric cancer using the same indigo carmine pigment spraying image. It is a block diagram of the disease diagnosis support method by the endoscopic image of the digestive organ using the neural network of Embodiment 7. It is a block diagram of the disease diagnosis support system by the endoscopic image of the digestive organ of Embodiment 8.

Hereinafter, a wireless capsule endoscope (WCE) will be described with respect to a disease diagnosis support method, a diagnosis support system, a diagnosis support program, and a computer-readable recording medium storing the diagnosis support program according to an endoscopic image of the digestive organs according to the present invention. ) Will be described in detail by taking as an example. However, the embodiments shown below show examples for embodying the technical idea of the present invention, and are not intended to specify the present invention in these cases. That is, the present invention can be equally applied to those of other embodiments included in the claims.

[Embodiment 1]
In the first embodiment, the method for diagnosing a disease by the endoscopic image of the present invention, the diagnosing support system, the diagnosing support program, and the computer-readable recording medium in which the diagnosing support program is stored are subjected to small intestinal spread using WCE. An example applied in the case of an ulcer will be described. In the first embodiment, it was difficult to distinguish between erosion and ulcer, so both are collectively referred to as "erosion / ulcer". That is, the term "erosion / ulcer" as used herein means not only "erosion", "erosion", "erosion and ulcer", but also "either erosion or ulcer, but at least in normal mucosa". It is used to mean that there is no such thing.

[About dataset]
At the clinic to which one of the inventors belongs, 5360 images of small intestinal erosion / ulcer were collected as a training data set from 115 patients who underwent WCE between October 2009 and December 2014. In addition, for verification of the CNN system of Embodiment 1, 10,440 independent images from 65 patients were prepared from January 2015 to January 2018 and used as a verification data set. .. Of these validation datasets, 440 images of 45 patients had erosions / ulcers of the small intestine, and 10,000 images of 20 patients had normal mucosa of the small intestine. Diagnosed by a mirror specialist. The WCE was performed using a Pillcam® SB2 or SB3 WCE device (Given Imaging, Yokne'am, Israel).

In order to train / verify the CNN system of Embodiment 1, all patient information attached to the image was anonymized prior to the development of the algorithm. None of the endoscopists involved in the CNN system of Embodiment 1 had access to identifiable patient information. Since the training / validation of this CNN system was a retrospective study using anonymized data, we adopted an opt-out approach to patient consent. This study was approved by the University of Tokyo Ethics Committee (No. 11931) and the Japan Medical Association (ID: JMA-IIA00283). The outline of the flowchart of the CNN system of Embodiment 1 is shown in FIG.

The main indications for WCE are gastrointestinal bleeding (OGIB: Obscure Gastrointestinal Bleeding) of unknown cause, and other cases where abnormal small intestinal images were observed using other medical equipment, abdominal pain, and follow-up of past small intestinal cases Referrals from primary care physicians regarding ups, diarrhea, and screening. The main causes were non-steroidal anti-inflammatory drug, followed by inflammatory bowel disease, malignant tumor of the small intestine, and anastomotic ulcer, but the etiology could not be determined in many cases. Table 1 shows the patient characteristics of the data debt used for training and validation of the CNN system.

[Training / Verification / Algorithm]
As shown in FIG. 2, the CNN system used in the first embodiment is trained by using backpropagation (backpropagation method). To build the CNN system of Embodiment 1, a deep neural network architecture called Single Shot MultiBox Detector (SSD, https://arxiv.org/abs/1512.02325) was used without changing the algorithm. First, two endoscopists manually annotated all areas of erosion / ulcer in the images of the training dataset with a rectangular border box. These images were incorporated into the SSD architecture through the Caffe framework, which was first developed at the Berkeley Vision and Learning Center. The Caffe framework is one of the first developed, most popular and widely used frameworks.

The CNN system of Embodiment 1 was trained that the area inside the boundary box was the erosion / ulcer area and the other area was the background. The CNN system of Embodiment 1 itself then extracted specific features of the border box region and learned erosion / ulcer features via training datasets. All layers of the CNN are probabilistically optimized with a global learning rate of 0.0001. Each image was resized to 300 x 300 pixels. The size of the bounding box was changed accordingly. These values were set by trial and error to ensure that all data is compatible with SSDs. Intel's Core i7-7700K was used as the CPU, and NVIDIA's GeForce GTX 1070 was used as the GPU for the graphics processing device.

[Measurement and statistics of results]
First, all erosions / ulcers in the image of the validation dataset were manually marked with a rectangular border box (hereinafter referred to as the "true box") with thick lines. In addition, the trained CNN system of the first embodiment imparts a rectangular boundary box (hereinafter referred to as "CNN box") to the detected erosion / ulcer region in the image of the verification data set with a thin line. , Erosion / ulcer probability score (range 0-1) was output. The higher the probability score, the higher the probability that the trained CNN system of Embodiment 1 contains erosions / ulcers in the area.

The inventors evaluated the ability of the CNN system of Embodiment 1 to determine whether each image contained erosions / ulcers. To perform this evaluation, we used the following definitions:
1) When the CNN box overlaps the true box by 80% or more, the answer is correct.
2) It was concluded that if multiple CNN boxes were present in one image and even one of those boxes correctly detected erosions / ulcers, the images were correctly identified.
The WCE endoscopy image determined to be the correct answer in this way can be used as a diagnostic aid at the site of double-checking the image taken by adding that information to the image, or a video during WCE endoscopy. Information is displayed in real time and used as a diagnostic aid.

Also, by varying the cutoff value of the probability score, the receiver operating characteristic (ROC) curve is plotted and the area under the curve (AUC) for evaluation of erosion / ulcer discrimination by the trained CNN system of Embodiment 1. Was calculated. Various cutoff values for probability scores, including scores according to the Youden index, were used to calculate the ability of the CNN system of Embodiment 1 to detect erosions / ulcers, sensitivity, specificity and accuracy. The Youden index is one of the standard methods for determining the optimum cutoff value calculated by sensitivity and specificity, and the numerical value of "sensitivity + specificity -1" is maximized. The cutoff value is calculated. Here, the data were statistically analyzed using Stata software (version 13; Stata Corp, College Station, TX, USA).

The validation dataset consisted of 10,440 images from 65 patients (male = 62%, mean age = 57 years, standard deviation (SD) = 19 years). The trained CNN system of Embodiment 1 took 233 seconds to evaluate these images. This is equal to the speed of 44.8 images per second. The AUC of the trained CNN system of Embodiment 1 in which erosions / ulcers were detected was 0.960 (95% confidence interval [CI], 0.950-0.969; see FIG. 3).

According to the Youden index, the optimum cutoff value of the probability score was 0.481, and the region with the probability score of 0.481 was recognized as erosion / ulcer by CNN. At that cutoff value, the sensitivity, specificity and accuracy of CNN were 88.2% (95% CI (confidence interval), 84.8-91.0%), 90.9% (95% CI, 90.). It was 3-91.4%) and 90.8% (95% CI, 90.2-91.3%) (see Table 2). Table 2 shows the sensitivity, specificity, and accuracy of each calculated by increasing the cutoff value of the probability score by 0.1 from 0.2 to 0.9.

In this way, the relationship between the erosion / ulcer classification result and the erosion / ulcer classification result by the endoscopist classified by the trained CNN system of Embodiment 1 with the cutoff value of the probability score = 0.481. Are summarized in Table 3.

In addition, FIGS. 4A-4D show typical regions correctly detected by the CNN system, and FIGS. 5A-5H show typical regions misclassified by the CNN system, respectively. False negative images, as shown in Table 4, have unclear boundaries (see FIG. 5A), a color similar to the surrounding normal mucosa, too small, and totally unobservable (lateral (visible because the affected area is on the side). It was classified into four causes (difficult) or partial (only partially visible)) (see FIG. 5B).

On the other hand, as shown in Table 5, there are five false positive images: normal mucosa, foam (Fig. 5C), debris (Fig. 5D), vasodilation (Fig. 5E), and true erosion (Fig. 5F-5H). It was classified as a cause.

As mentioned above, according to the trained CNN system of Embodiment 1, a CNN-based program for automatic detection of erosions and ulcers in the small intestinal image of WCE was constructed with a high accuracy of 90.8% (AUC). , 0.960) revealed that erosions / ulcers could be detected in independent test images.

[Embodiment 2]
In the second embodiment, a method for diagnosing a disease related to an elevated lesion of the small intestine using a wireless capsule endoscopy (WCE) image, a diagnostic support system, a diagnostic support program, and a computer-readable recording medium storing the diagnostic support program will be described. To do. The morphological characteristics of elevated lesions vary from polyps, nodules, masses / tumors to vascular structures, and the etiology of these lesions includes neuroendocrine tumors, adenocarcinomas, and familial adenocarcinoma polyposis. Includes Peutz-Jeghers syndrome, follicular lymphoma and gastrointestinal stromal tumors. Since these lesions require early diagnosis and treatment, it is necessary to avoid overlooking the WCE test.

[About dataset]
From October 2009 to May 2018, 30,584 images of elevated lesions were collected from 292 patients who underwent WCE at multiple clinics to which the inventor belongs as a training data set. Also, a total of 17 from 93 patients, including 10,000 images without elevated lesions and 7,507 images with elevated lesions, regardless of the images used for CNN training. , 507 images were used for verification. Elevated lesions were morphologically classified into five categories: polyps, nodules, epithelial tumors, submucosal tumors, and vascular structure, based on the definition of CEST classification (see Non-Patent Document 14). However, mass / tumor lesions in the definition of CEST classification were classified into epithelial tumors and submucosal tumors.

[Training / Verification / Algorithm]
The outline of the flowchart of the CNN system of the second embodiment is shown in FIG. The CNN system of the second embodiment used the same SSD deep neural network architecture and Caffe framework as in the first embodiment. First, six endoscopists manually annotated all areas of the elevated lesion in the image of the training dataset with a rectangular bounding box. Annotations were performed individually by each endoscopist and a consensus was later determined. These images were incorporated into the SSD architecture through the Caffe framework.

The CNN system of Embodiment 2 was trained to be a raised lesion inside the border box, with other areas as the background. The CNN system of Embodiment 2 itself then extracted specific features of the border box region and "learned" the features of the elevated lesions via the training dataset. All layers of the CNN are probabilistically optimized with a global learning rate of 0.0001. Each image was resized to 300 x 300 pixels. The size of the bounding box was changed accordingly. These values were set by trial and error to ensure that all data is compatible with SSDs. Intel's Core i7-7700K was used as the CPU, and NVIDIA's GeForce GTX 1070 was used as the GPU for the graphics processing device. As the WCE, the same Pilcam SB2 or SB3 WCE device as in the first embodiment was used. Data were analyzed using STATA software (version 13; Stata Corp, College Station, TX, USA).

In order to train / verify the CNN system of the second embodiment, all patient information attached to the image was anonymized prior to the development of the algorithm. None of the endoscopists involved in the CNN system of Embodiment 2 had access to identifiable patient information. Since the training / validation of this CNN system was a retrospective study using anonymized data, we adopted an opt-out approach to patient consent. This study was approved by the Japan Medical Association Ethics Committee (ID: JMA-IIA00283), Sendai Kousei Hospital (No. 30-5), Tokyo University Hospital (No. 11931), and Hiroshima University Hospital (No. E-1246). Got

[Measurement and statistics of results]
First, rectangular border boxes (hereinafter referred to as "true boxes") were manually added with thick lines to all areas of elevated lesions in the image of the validation dataset. In addition, the trained CNN system of the second embodiment imparts a rectangular boundary box (hereinafter referred to as “CNN box”) to the area of the detected elevated lesion in the image of the verification data set with a thin line. , The probability score (PS: range 0-1) of the area of the elevated lesion was output. The higher the probability score, the higher the probability that the trained CNN system of Embodiment 2 contains elevated lesions in the area.

The inventors evaluated the CNN box in descending order of the probability score of each image for the ability of the CNN system of the second embodiment to determine whether or not each image contained an elevated lesion. CNN boxes, probabilistic scores for elevated lesions, and categories of elevated lesions were determined as a result of CNN when the CNN box clearly surrounded the elevated lesions.

If it is difficult to judge visually due to the large number of CNN boxes drawn, if the overlap between the CNN box and the true box is equal to or greater than 0.05 Intersection over Unions (IoU). , CNN's conclusion. Note that IoU is an evaluation method for measuring the accuracy of an object detector, and is calculated by dividing the overlapping area of two boxes by the combined area of two boxes.
IoU = (overlap area) / (area where both are combined)
If the CNN box did not apply to the above rules, the CNN box with the next lowest probability score was evaluated in order.

When multiple true boxes are displayed in one image, if one of the CNN boxes overlaps the true box, it was determined that the CNN box is the result of the CNN. For images without elevated lesions, the CNN box with the highest probability score was determined as a result of the CNN. Three endoscopists performed these tasks on all images.

The area under the curve (AUC) is plotted to plot the receiver operating characteristic (ROC) curve by varying the cutoff value of the probability score and to assess the degree of elevated lesion identification by the trained CNN system of Embodiment 2. Was calculated (see FIG. 7). Then, as in the case of the first embodiment, the sensitivity, which is the ability to detect the elevated lesions of the CNN system of the second embodiment, using various cutoff values for the probability score including the score according to the Youden index. , Specificity and accuracy were calculated.

The secondary result in Embodiment 2 is the classification of elevated lesions into 5 categories by CNN and the detection of elevated lesions in individual patient analysis. Regarding the accuracy of classification, the concordance rate of classification between CNN and endoscopists was examined. An individual patient analysis of the detection rate of elevated lesions defined that detection by CNN was correct if the CNN detected at least one elevated lesion image in multiple images of the same patient.

Furthermore, after performing the CNN process, 10,000 clinically normal images in the validation data set were re-evaluated. From the images that were considered normal, some CNN boxes that appeared to be true elevated lesions were extracted. This lesion may have been overlooked by a doctor. This is based on the consensus of three endoscopists.

Table 6 shows the patient characteristics of the data set used for training and verification of the CNN system of the second embodiment, and the details of the training data set and the verification data set. The validation dataset includes 7,507 images containing elevated lesions from 73 patients (male, 65.8%, mean age, 60.1 years, standard deviation, 18.7 years) and 20. It consisted of 10,000 lesion-free images from human patients (male, 60.0%, mean age, 51.9 years, standard deviation, 11.4 years).

The CNN constructed in the second embodiment completed the analysis of all images in 530.462 seconds, and the average speed per image was 0.0303 seconds. The AUC of the CNN of Embodiment 2 used to detect elevated lesions was 0.911 (95% confidence interval (Cl), 0.9069-0.9155) (see FIG. 7).

According to the Youden index, the optimal cutoff value for the probability score was 0.317. Therefore, regions with a probability score of 0.317 or higher were recognized as elevated lesions detected by CNN. Using that cutoff value, the sensitivity and specificity of CNNs are 90.7% (95% CI, 90.0% -91.4%) and 79.8% (95% CI, 79.0%-, respectively). 80.6%) (see Table 7).

In a subgroup analysis of the raised lesion category, CNN sensitivities were 86.5%, 92.0%, 95.8%, 77 for detection of polyps, nodules, epithelial tumors, submucosal tumors, and vascular structures. It was 0.0% and 94.4%. 8A-8E show representative regions correctly detected and classified by the CNN of Embodiment 2, respectively, into five categories: polyps, nodules, epithelial tumors, submucosal tumors and vascular structures.

In the analysis of individual patients, the detection rate of elevated lesions was 98.6% (72/73). By category of elevated lesions, the detection rates of polyps, nodules, epithelial tumors, submucosal tumors, and vascular structures per patient were 96.7% (29/30), 100% (14/14), and 100%. It was (14/14), 100% (11/11), and 100% (4/4). However, all three images of the polyp of one patient shown in FIGS. 9A-9C could not be detected by the CNN of Embodiment 2. In this image, all probability scores for the CNN box were less than 0.317 and were not detected as elevated lesions by CNN. Also, of the false positive images (n = 2,019) where CNN provided a CNN box with a probability score of 0.317 or higher, although it does not appear to have elevated lesions, two are truly elevated by an endoscopist. It was suggested to be a lesion (see FIG. 10).

Table 8 shows the labeling of elevated lesions by CNN and a specialist endoscopist. CNN and endoscopist labeling concordance rates for polyps, nodules, epithelial tumors, submucosal tumors, and vascular structures were 42.0%, 83.0%, 82.2%, 44.5%, and 48. It was 0.0%.

As described above, in CNN in the second embodiment, the category based on CEST is applied, and although there are differences in sensitivity between categories such as polyps, nodules, epithelial tumors, submucosal tumors, and vascular structure, the sensitivity is high. , It was clarified that it can be detected and classified with a good detection rate.

[Embodiment 3]
WCE has become an indispensable tool for investigating small bowel disease, and the main indication is gastrointestinal bleeding (OGIB) of unknown cause for which no clear source of bleeding is found. When screening WCE images, doctors interpret 10,000 images per patient for 30-120 minutes. Therefore, it is important for WCE image analysis whether or not blood components can be automatically detected. As a means for automatically detecting a blood component in such a WCE image, for example, a "red region estimation display function" (Suspected Blood Indicator; hereinafter simply referred to as "SBI") is known (see Non-Patent Document 15). .. SBI is an image selection tool included in RAPID CE interpretation software (Medtronic, Minnesota, Minneapolis, USA) that tags potentially bleeding areas with red pixels.

In the third embodiment, in comparison with the above-mentioned SBI, a method for diagnosing small intestinal bleeding by WCE image using a CNN system, a diagnostic support system, a diagnostic support program, and a computer-readable recording medium storing the diagnostic support program. Will be described. When detecting the blood component of the small intestine, it is possible to estimate the amount of blood, and in this case, it is also possible to estimate the blood volume from the distribution range of blood or the like. However, in the following, the case of detecting the presence or absence of blood components, that is, the presence or absence of bleeding will be illustrated.

[About dataset]
WCE images from November 2009 to August 2015 were retroactively acquired at a single institution (University of Tokyo Hospital, Japan) to which some of the inventors belong. During that period, WCE was performed using the Pilcam SB2 or SB3 WCE device as in Embodiment 3. Two endoscopists classified images containing blood components in the lumen and images of normal small intestinal mucosa without considering the SBI results. The blood component of the lumen is defined as active bleeding or a clot.

As a training dataset for the CNN system of Embodiment 3, 27,847 images (6,503 images containing the blood components of 29 patients and 21,344 images of the normal small intestinal mucosa of 12 patients). Image) was collected. Similarly, as a data set for verification of the CNN system, 10,208 images from 25 patients were prepared separately from the training data set. Of these images, 208 images from 5 patients showed bleeding in the small intestine, and 10,000 images from 20 patients were from normal small intestinal mucosa. .. The outline of the flowchart of the CNN system of the third embodiment is shown in FIG.

In order to train / verify the CNN system of the third embodiment, all patient information attached to the image was anonymized prior to the development of the algorithm. None of the endoscopists involved in the CNN of Embodiment 3 had access to identifiable patient information. Since the training / validation of the CNN system of Embodiment 3 was a retrospective study using anonymized data, an opt-out approach was adopted for patient consent. This study was approved by the University of Tokyo Ethics Committee (No. 11931) and the Japan Medical Association (ID JMA-IIA00283).

[Training / Verification / Algorithm]
The algorithm of the CNN system used in Embodiment 3 was developed using ResNet50 (https://arxiv.org/abs/1512.03385), which is a 50-layer deep neural network architecture. It was then trained using the Caffe framework first developed at the Berkeley Vision Learning Center to train and validate the newly developed CNN system. Then, using SGD (Stochastic Gradient Descent), all layers of the network were stochastically optimized with a global learning rate of 0.0001. All images have been resized to 224 x 224 pixels for compatibility with ResNet50.

[Measurement and statistics of results]
The main results in the CNN system of Embodiment 3 are the area under the curve (AUC) of the receiver operating characteristic (ROC) curve, and the CNN system between images of blood components and images of normal mucous membranes. Includes the accuracy of the discriminating ability of. The trained CNN system of Embodiment 3 output consecutive numbers between 0 and 1 as probability scores for blood components per image. The higher the probability score, the higher the probability that the CNN system will contain blood components in the image. The validation test of the CNN system in Embodiment 3 was performed using a single still image, plotting the ROC curve by varying the threshold of the probability score, and calculating the AUC to assess the degree of discrimination. ..

In Embodiment 3, the threshold of the probability score is simply set to 0.5 for final classification by the CNN system to identify the CNN system between images containing blood components and images of normal mucous membranes. The sensitivity, specificity and accuracy of the ability were calculated. In addition, the validation set evaluated the sensitivity, specificity and accuracy of SBI discrimination between images containing blood components and images of normal mucosa by examining 10,208 images. Differences in performance between the CNN system of Embodiment 3 and SBI were compared using McNemmer's test. The data obtained were statistically analyzed using STATA software (version 13; Stata Corp, College Station, TX, USA).

The validation dataset consisted of 10,208 images from 25 patients (male, 56%, mean age, 53.4 years, standard deviation, 12.4 years). The trained CNN system of Embodiment 3 took 250 seconds to evaluate these images. This is equal to the speed of 40.8 images per second. The AUC of the CNN system of Embodiment 3 for identifying images containing blood components was 0.9998 (95% CI (confidence interval), 0.9996-1.0000; see FIG. 12). Table 9 also shows the respective sensitivities, specificities and accuracy calculated by increasing the probability score cutoff value by 0.1 from 0.1 to 0.9.

With a cutoff value of 0.5 probability score, the sensitivity, specificity and accuracy of the CNN system of Embodiment 3 are 96.63% (95% CI, 93.19-98.64%) and 99.96% (99.96%, respectively). It was 95% CI, 99.90-99.99%) and 99.89% (95% CI, 99.81-99.95%). The cutoff value 0.21 of the probability score in Table 15 is the optimum cutoff value calculated according to the Youden index, but in this verification data set, the accuracy of the Youden index cutoff value is 0. It was lower than the accuracy at the simple cutoff value of 5. In addition, FIG. 13 shows an image containing representative blood (FIG. 13A) and a normal mucosa image (FIG. 13B) correctly classified by the CNN system of Embodiment 3. The probability values obtained by the CNN system of the third embodiment of FIGS. 13A and 13B are shown in Table 10 below.

On the other hand, the sensitivity, specificity and accuracy of SBI were 76.92% (95% CI, 70.59-82.47%) and 99.82% (95% CI, 99.72-99.89%), respectively. And 99.35% (95% CI, 99.18-99.50%). All of these were significantly lower than the CNN system (p <0.01). Table 11 shows the difference in classification between the CNN system of the third embodiment and the SBI.

FIG. 14 shows seven false negative images classified as normal mucosa by the CNN of Embodiment 3. Of these, the four images shown in FIG. 14A were correctly classified as containing blood components by SBI, and the three images in FIG. 14B were erroneously classified as normal mucosa by SBI. The classifications obtained by the CNN system and SBI of the third embodiment of FIGS. 14A and 14B are shown in Table 12 below, and the relationship between the classifications of the CNN system and SBI is shown in Table 13, respectively.

As described above, according to the trained CNN system of the third embodiment, it was possible to distinguish between an image containing a blood component and a normal mucosal image with a high accuracy of 99.9% (AUC, 0.9998). In addition, a direct comparison with SBI showed that the trained CNN system of Embodiment 3 could be classified more accurately than SBI. Even at a simple cutoff point of 0.5, the trained CNN system of Embodiment 3 was superior to SBI in both sensitivity and specificity. This result shows that the trained CNN system of Embodiment 3 can be used as a highly accurate screening tool for WCE.

[Embodiment 4]
Endoscopy by WCE is performed by externally receiving and recording images taken while the capsule is moving in the digestive tract, and it is possible to take a picture of the entire small intestine at one time. It is also possible to shoot as. However, screening for moving WCE images is more burdensome for physicians than for still images. In the moving image of the present invention, in addition to those continuously shot by a so-called video camera, when still images are continuously shot with a very short shooting interval and a series of still images are continuously reproduced. Also includes those that can be recognized as movies.

The QuickView mode is known for the purpose of shortening the screening time in the WCE image of such a moving image. QuickView mode is an image selection tool installed in RAPID CE interpretation software (Medtronic, Minnesota, Minneapolis, USA). According to this QuickView mode, it is reported that it is not suitable for the initial screening due to the high error rate for important diseases despite its relatively high sensitivity and ability to shorten the reading time. (See Non-Patent Document 16).

Further, in the first embodiment, the WCE still image is used and the WCE moving image is not used in the screening by the WCE image. In the fourth embodiment, the diagnostic support method for various diseases of the small intestine using the CNN system, the diagnostic support system, the diagnostic support program, and the computer reading that stores the diagnostic support program are compared with the QuickView mode described above. A possible recording medium will be described. When detecting the blood component of the small intestine, it is possible to estimate the amount of blood, but the case of detecting the presence or absence of the blood component, that is, the presence or absence of bleeding will be described below as in the case of the third embodiment.

[About dataset]
The training data set for the CNN system of Experimental Example 4 includes mucosal disorders (n = 5360 sheets), nodules (n = 11,907 sheets), vasodilatory disease (n = 2,237 sheets), and polyps (n = 10, pcs). 704), epithelial tumor (n = 6,514), submucosal tumor (n = 1,875), vascular structure (n = 393), blood component (n = 6,503), and CEST Still images of normal images (n = 21,344 sheets) according to classification were used. These training still images were collected from patients from October 2009 to May 2018 at the institutions to which some of the inventors belong (Tokyo University Hospital, Hiroshima University Hospital and Sendai Kousei Hospital, Japan). The WCE test was performed using a Pilcam SB2 or SB3 device (Medtronic, Minneapolis, Minnesota, USA).

The verification data set includes 379 photographs taken between June 2018 and May 2019 at the institutions to which some of the inventors belong (Tokyo University Hospital, Hiroshima University Hospital and Sendai Kousei Hospital, Japan). WCE video images were acquired retroactively. The WCE test was performed using a Pilcam SB3 device. The training data set and the verification data set are completely independent.

Also, in order to train / verify the CNN system of Embodiment 4, all patient information attached to the image was anonymized prior to the development of the algorithm. None of the endoscopists involved in the CNN of Embodiment 4 had access to identifiable patient information. Since the training / validation of the CNN system of Embodiment 4 was a retrospective study using anonymized data, an opt-out approach was adopted for patient consent. This study was approved by the University of Tokyo Ethics Committee (No. 11931), Japan Medical Association (ID JMA-IIA00283), Hiroshima University Hospital (No. E-1246), and Sendai Kousei Hospital (No. 30-5). Obtained.

[Training / Verification / Algorithm]
The algorithm of the CNN system used in Embodiment 4 was the same SSD and ResNet50 architecture as in Embodiment 3 and was subsequently trained using the Caffe framework. Then, using SGD, all layers of the network were stochastically optimized with a global learning rate of 0.0001.

In a preliminary evaluation using images from previous studies, in the CNN system of Embodiment 4, mucosal damage and nodules can be detected when images containing mucosal damage and nodules are trained with images containing other abnormalities. The SSD that detects mucosal damage and the SSD that detects nodules were separated from the other SSDs because they were shown to be less sexual. That is, the CNN system of the fourth embodiment constructed and used a composite CNN system using the following four subsystems.
(1) SSD that detects mucosal damage,
(2) SSD that detects nodules,
(3) SSD that detects other abnormalities (vasodilation, polyps, submucosal tumors, vascular structure and epithelial tumors), and
(4) ResNet50 for detecting blood components.

For training the composite CNN system of Embodiment 4, all abnormalities except the blood components of the training dataset image dataset were manually performed by 6 endoscopists who have reviewed more than 400 WCE images. A rectangular border box (true box) was added in (n = 38,181 sheets). For the calculation, an Intel Core i7-7700K central processing unit and a GeForce GTX 1070 graphics processing unit were used.

[Measurement and statistics of results]
The main analytical results in the CNN system of Embodiment 4 include the detection of various abnormalities in the small intestine. The main analysis is primarily patient-specific disease analysis. If at least one particular anomaly image was obtained from a moving image of a patient, the detection result for that type of anomaly was defined as correct for that patient. For example, if the CNN acquired at least one mucosal damage image from a patient containing multiple mucosal damage images, it was determined that the CNN was able to accurately detect the mucosal damage in that patient.

In addition to the composite CNN system, the detection rate by QuickView mode installed in RAPID CE interpretation software v8.3 was also evaluated. QuickView mode picks up images containing significant lesions and enables fast-forward review of WCE video. In RAPID CE interpretation software v8.3, the sampling rate (threshold ratio) can be set between 2% and 80% in QuickView mode. In the fourth embodiment, the sampling rate in the QuickView mode was set according to the average acquisition rate of images by CNN. By setting the threshold in this way, the CNN and QuickView modes were able to reduce the image to the same extent, which allowed a direct comparison of detectability between the two systems.

As a sub-analysis, a patient-by-patient analysis was performed based on strict criteria for the detection rate of various abnormalities, taking into account the number of abnormalities (single or multiple) of each type. In this analysis, the correct detection criterion is that patients with multiple lesions of a particular abnormality need to detect multiple lesions, not just a single lesion. For example, if CNN detects only one ulcer in a patient with multiple ulcers, it is determined by strict criteria that CNN cannot properly detect the ulcer in that patient. Previous studies have suggested that information on single or multiple lesions may be useful in investigating the etiology and management of small bowel disease (see Non-Patent Document 17), thus substituting this rigorous criterion. Adopted as an analysis.

Detectability between CNN and QuickView modes was compared using McNemmer's test. Abnormalities were diagnosed in the original laboratory by two endoscopists, and a consensus review was conducted by two other endoscopists, ie, at the University of Tokyo Hospital as a gold standard, prior to searching by the two systems. It was. These studies were limited to the small intestine section of the small intestine WCE full video. The size of mucosal damage was classified as erosion (≤5 mm) or ulceration (> 5 mm). The types of vasodilators were classified into types 1a or 1b according to Yano Yamamoto's classification (see Non-Patent Document 18). Type 1a lesions are characterized by punctate erythema (<1 mm) with or without exudation, and type 1b lesions are characterized by patchy erythema (2-3 mm) with or without exudation.

The measurement of secondary results was a classification of CNN-induced abnormalities into the following four categories:
(1) Mucosal damage,
(2) Vasodilation,
(3) Nodules, polyps, epithelial tumors, submucosal tumors, and elevated lesions including vascular structure, and (4) blood components.

In the evaluation of the main result (ie, abnormality detection rate), if an abnormality was detected, it was defined as correct regardless of the labeling when the lesion was picked up. On the other hand, in the evaluation of secondary results, the classification agreement between CNN and endoscopist (eg, gold standard) was examined for each patient analysis. If at least one particular anomaly image was taken from a moving image of a patient and labeled as the correct classification, then that type of anomaly classification was defined as correct for each patient. The data were statistically analyzed using STATA software similar to that described above.

Table 14 shows the patient characteristics of the verification data set used for the verification of the CNN system of the fourth embodiment. The validation dataset consisted of WCE moving images of the small intestine of 379 patients (male: 57%, mean age: 62.3 years, standard deviation: 17.5 years). The most common indication for WCE was unexplained gastrointestinal bleeding (39%). In Table 14, duplicate data is allowed, the value of the part to which "±" is given indicates "mean value ± standard deviation", and the value in parentheses indicates the percentage. However, in Table 14, data duplication is allowed.

The average reading speed of the CNN system of the fourth embodiment was 0.09 seconds per image. Since this CNN system picked up 1,135,104 images (22.5%) from 5,050,226 verification images, the sampling rate in QuickView mode was set to 23%. The average number of all images of the small intestine is 13,325 per video, and the average number of images picked up by the CNN system of Embodiment 4 is 2,295 per video (average rate: 22.5%, Standard deviation: 8.7%). Representative images of various abnormalities correctly detected by the CNN system of Embodiment 4 are shown in FIG. 15 together with the respective definitive diagnosis results. The rectangular frame in FIG. 15 indicates the region of the diseased part given by the endoscopist, and the numerical value shows the probability score given by CNN.

In addition, the difference in the detection rate of each disease between the CNN system and the Quick View mode is shown in FIG. 16, and the disease between the CNN system and the Quick View mode in the case of a strict standard also considering a single lesion or a plurality of lesions is shown in FIG. The difference in the detection rates of the above is shown in FIG. In FIG. 16, FIG. 16A is a graph showing the difference in the detection rate of the whole disease, and FIG. 16B is a graph showing the difference in the detection rate of each of the four types of diseases such as mucosal disorder, vasodilation, elevated lesion and blood component. Further, FIG. 16C is a graph showing the difference in the detection rate for each disease when the diseases shown in FIG. 16B are further subdivided. Further, FIGS. 17A to 17C are graphs corresponding to FIGS. 16A to 16C in the case of a strict standard considering a single lesion or a plurality of lesions, respectively. Further, in FIGS. 16 and 17, “*” indicates p <0.05, “**” indicates P <0.001, and “NS” indicates no significant difference.

Mucosal disorders, vasodilation, elevated lesions, and blood components were detected in 94, 29, 81, and 23 patients, respectively (see Table 14). Overall, the detection rate of patient-specific anomalies by CNN was significantly higher than that by QuickView mode (99%: 89%, p <0.001), as shown in FIG. Specifically, the detection rates of erosion, ulcer, vasodilator type 1a, vasodilator type 1b, polyps, nodules, submucosal tumors, vascular structure, epithelial tumors and blood components by the CNN system of Embodiment 4 are 100, respectively. % (70/70), 100% (24/24), 95% (20/21), 100% (8/8), 100% (22/22), 100% (21/21), 94% ( 15/16), 100% (13/13), 100% (9/9) and 100% (23/23), 90%, 96%, 100%, 88%, respectively, in QuickView mode. It was 82%, 91%, 75%, 54%, 100% and 96%. From this result, it was shown that the detection rate of erosions, polyps, and vascular structures by the CNN system of the fourth embodiment was significantly higher (p <0.05) than the detection rate by the QuickView mode.

In a patient-by-patient analysis based on rigorous criteria considering the number of lesions (single or multiple), endoscopists found 53% (37/70) of patients with rash and 42% (37/70) of patients with ulcer 10/24), 14% (3/21) of patients with vasodilator 1a, 13% (1/8) of patients with vasodilator 1b, 41% (9/22) of patients with polyps, nodules 81% (17/21) of patients with, 6% (1/16) of patients with submucosal tumors, 23% (3/13) of patients with vascular structure, 22% of patients with epithelial tumors (2/9) and 100% (23/23) of patients with blood components could be detected.

Overall, the detection rate of CNN anomalies per patient based on strict criteria was significantly higher than in QuickView mode (98% vs. 85%, p <0.001) (Fig. 17A). Specifically, the detection rate of erosion, ulcer, vasodilation type 1a, vasodilation type 1b, polyps, nodules, submucosal tumors, vascular structure, epithelial tumors, and blood volume by CNN is 99% (69 / 69 /). 70), 100% (24/24), 90% (19/21), 100% (8/8), 100% (22/22), 100% (21/21), 94% (15/16) , 100% (13/13), 100% (9/9), and 100% (23/23), and those in QuickView mode are 87%, 96%, 100%, 88%, and 73%, respectively. , 81%, 75%, 38%, 100%, 96%. As a result, it was confirmed that the detection rate of erosion, polyps, nodules, and vascular structure by CNN of Embodiment 4 according to strict criteria was significantly higher than the detection rate by QuickView mode (p <0.05). ..

FIG. 18 shows four false negative images that could not be detected by CNN in a patient-by-patient analysis based on strict criteria. FIG. 18A is an example of erosion (n = 1), FIGS. 18B and 18C are examples of vasodilator type 1a (n = 2), and FIG. 18D submucosal tumor (n = 1). All of these were acquired correctly in QuickView mode.

　表１５に示した結果によると、実施形態４のＣＮＮによる結果と内視鏡専門医による診断結果の一致率は、粘膜障害の場合が９５．７％、血管拡張症の場合が７５．９％、隆起性病変の場合が９８．８％、血液成分の場合が１００％であった。 Table 15 shows the correspondence between the results of the CNN of the fourth embodiment and the diagnosis results of the endoscopist. The numerical values in parentheses in Table 15 indicate the ratio to the whole.

According to the results shown in Table 15, the concordance rate between the result by CNN of the fourth embodiment and the diagnosis result by the endoscopist was 95.7% in the case of mucosal disorder and 75.9% in the case of vasodilation. Elevated lesions accounted for 98.8%, and blood components accounted for 100%.

As described above, according to the trained CNN system of the fourth embodiment, various abnormalities in WCE moving images captured at a plurality of facilities can be detected with high sensitivity, and by direct comparison with the existing QuickView mode, it is possible to detect them. The detection rate of per-patient anomalies by the trained CNN system of Embodiment 4 was found to be significantly higher than in QuickView mode (99% vs. 89%). That is, it has been found that the trained CNN system of Embodiment 4 may be superior to the existing QuickView mode and helps reduce the burden on the physician without reducing the detection rate of anomalies. ..

[Embodiment 5]
In the fifth embodiment, the method for diagnosing a disease by the endoscopic image of the present invention, the diagnosis support system, the diagnosis support program, and a computer-readable recording medium storing the diagnosis support program are used at an early stage using ME-NBI images. An example applied in the case of gastric cancer will be described.

[About dataset]
At the clinic to which one of the inventors belongs, a total of 349 lesions were selected from 745 lesions of differentiated early gastric cancer resected by endoscopy between April 2013 and March 2018. From these lesions, 5,227 ME-NBI images were collected as a training dataset using the maximal water immersion method with sufficient quality to enable diagnosis by MESDA-G ( FIG. 19). ME-NBI images of gastric adenocarcinoma of fundic gland (GAFG) and diffuse early gastric cancer, as well as unanalyzable, low-quality images, can accurately diagnose ME-NBI findings in these cases. Excluded due to low sex.

In FIG. 19, A is a differentiated cancer (0-IIc, mub1), B is a differentiated cancer (0-IIc, tub2), C is a differentiated cancer (0-IIa, tub1), D to F indicates gastric gland mucosa, GI indicates pyloric gland mucosa, J and K indicate patchy redness, L indicates adenoma, M indicates yellow tumor, N indicates local atrophy, and O indicates ulcer scar. Here, 0-IIc indicates a surface recessed type, 0-IIa indicates a surface raised type, tub1 indicates a well-differentiated adenocarcinoma, and tub2 indicates a moderately differentiated adenocarcinoma.

Histopathological diagnosis for all patients with early gastric cancer was made by a pathologist specializing in gastrointestinal pathology. The main histological types were determined according to the Japanese classification of gastric cancer, the 3rd English version. In addition, 2,647 ME-NBI images of non-cancerous mucosa or non-cancerous lesions obtained under the same conditions were collected. These ME-NBI images included gastric gland mucosa, pyloric mucosa, patchy redness, adenomas, yellow tumors, localized atrophy and scars of ulcers (see Figure 19) and were endoscopically diagnosed. However, no pathological examination of the biopsy tissue was performed.

All images are high resolution endoscopes (GIF-H260Z or GIF-H290Z; Olympus Medical System, Tokyo, Japan) and standard endoscope video systems (EVIS LUCERA CV260 / CLV-260, EVIS LUCERA ELITE CV-290). / CLV-290SL; collected using the Olympus Medical System). For all test patients, maximum magnification ME-NBI using the water immersion method was performed by three experienced endoscopists. The video processor was always used with the structural enhancements set to the B8 level of ME-NBI and the NBI color mode fixed at level 1.

[Training / Verification / Algorithm]
The CNN system of Embodiment 5 was developed by transfer learning utilizing Deep Residual Network (ResNet-50, Non-Patent Document 21), which is a state-of-the-art CNN architecture pre-trained in an ImageNet database containing more than 14 million images. Was done. The same Caffe framework as in Embodiment 1 was used for training, verification, and testing of CNNs. In the CNN system of Embodiment 5, transfer learning is used to replace the final classification layer with another fully connected layer, retrain using the training dataset, and fine-tuning the parameters of all layers. It was adjusted. Each image was resized to 224 x 224 pixels. A rotated image was also used to increase the number of images. In order to improve the classification performance of the CNN stem of Embodiment 5, a rotated image was performed strictly only for the training dataset. All layers of the CNN were fine-tuned with a global learning rate of 0.001 using stochastic gradient descent. These values were set by trial and error so that all data were compatible with ResNet-50.

The CNN system of Embodiment 5 was trained and validated with a dataset of 5,574 ME-NBI images (early gastric cancer: 3977 images of 267 cases, non-cancerous images: 1777 images). During the training, the training data set was randomly divided into a training data set (4460 images) and a verification data set (1114 images) and trained at a ratio of 8: 2, and the CNN system of the fifth embodiment was constructed (FIG. 20). reference)

[Measurement and statistics of results]
Individual tests of 2,300 ME-NBI images (early gastric cancer: 1430 images of 82 cases, non-cancer images: 870 images) to evaluate the accuracy of the diagnosis (diagnosis of cancer or non-cancer) The dataset was applied to the CNN system of Embodiment 5 (see FIG. 20). The test data set was not expanded and the obtained images were used as they were. In validation and testing, the trained CNN system of Embodiment 5 generated a contiguous number between 0 and 1 corresponding to the probability value depending on whether each image was cancerous or non-cancerous.

[Evaluation algorithm]
Table 16 shows the definitions of the evaluation criteria (accuracy, sensitivity, specificity, PPV (positive predictive value) for early gastric cancer, NPV (negative predictive value) for early gastric cancer, false positives and false negatives). Further, in order to evaluate the accuracy of the CNN system of the fifth embodiment, the area (AUC) below the receiver operating characteristic curve (ROC) was obtained. The overall test rate was defined from the start to the end of the analysis of the test image by time measurement incorporated in the CNN system of Embodiment 5.

Further, by applying the Gradient-weighted Class Activation Map (Grad-CAM, Non-Patent Document 22) in order to determine which region of the image is most important for the classification result, the CNN system of the fifth embodiment can input the input image. I tried to understand how to recognize. Grad-CAM produces a coarse localization map that emphasizes important areas in the image to predict the target concept (in this case gastric cancer). Here, a heat map image, that is, a heat map was created from the position identification data of the localization map.

All statistical analysis was performed at EZR (EasyR; Saitama Medical Center, Jichi Medical University, Saitama, Japan), which is a graphical user interface of R (The R Foundation for Statistical Computing, Vienna, Austria). Continuous data were compared using the Mann-Whitney U test. Variable categorical analysis was performed using Fisher's exact test. A p-value <0.05 was considered to show a statistically significant difference.

In constructing the CNN system of Embodiment 5, it was reviewed and approved by the clinical trial review committee (approval number: # 18-229) of Juntendo University School of Medicine. The analysis used anonymous clinical data obtained after each patient consented to oral consent treatment, so patients did not have to consent to the study and could not identify individuals from the data presented. Was done.

From April 2013 to March 2018, a total of 349 patients with early gastric cancer were registered. According to the time frame of the ME-NBI procedure, the data obtained from these early gastric cancers were divided into two sets, a training data set (n = 267) and a test data set (n = 82). The test dataset contained data that was later collected. Table 17 shows the characteristics of patients and lesions used in the training and test datasets. Compared to the training dataset, the test dataset includes H. et al. Cases of Helicobacter pylori-negative gastric cancer (including uninfected, post-eradication cases) and cases with a small tumor diameter and lesion in the lower part (L) of the stomach were included.

Table 18 shows the results of the VS classification system for early gastric cancer in the training and test data sets. There are no major differences in borderline, MVP (microvascular pattern), MSP (microsurface pattern) and diagnosis between the training and test datasets. There were 18 lesions that were not diagnosed with cancer by endoscopy. Of these, 8 lesions, 7/267 (2.6%) in the training dataset and 1/82 (1.2%) in the test dataset, were erroneously diagnosed as non-cancer due to lack of boundaries. .. The remaining 10 lesions, 6/267 (2.2%) in the training dataset and 4/82 (4.9%) in the test dataset, were mistakenly non-cancerous due to the presence of normal MVPs and MSPs. Diagnosed, but borderline was present. Compared to the training dataset, the test dataset may have contained more lesions of normal MVP and MSP, but the difference is not significant.

The overall test speed was 38.3 sheets / second (0.026 seconds / sheet). The performance of the CNN system of Embodiment 5 is shown in Table 19. The accuracy was 98.7%, and 2271 out of 2300 images were diagnosed correctly. Sensitivity, specificity, PPV, NPV, false positive rate and false negative rate are 98% (1401/1430), 100% (870/870), 100% (1401/1401), 96.8% (870/899), respectively. ), 0% (0/870) and 2% (29/1430). When the receiver operating characteristic curve (ROC) by the CNN system of the fifth embodiment was evaluated, the area under the curve (AUC) was 99% (see FIG. 21).

A total of 12 cases of early gastric cancer shown in 29 ME-NBI images were misdiagnosed as non-cancer. In 6 cases, it was difficult for even an experienced endoscopist to distinguish from intestinal metaplasia or gastritis (Table 20, FIGS. 22A-C). In addition, lesions located in the middle or lower stomach, with a relatively small size and represented as 0-IIc, were found more frequently in the test dataset. In terms of infiltration depth, all cases were diagnosed with intramucosal cancer and most of them were H. pylori negative. Images of the other 6 cases were of poor quality for diagnosis due to hemorrhage, low magnification, and out-of-focus (FIGS. 22D, E). False positive images were not displayed in the test dataset.

Note that in FIG. 22, A to C are examples of images diagnosed as false negatives by the CNN system of Embodiment 5, and the lesions are diagnosed as intestinal metaplasia or gastritis. Of these, A is a differentiated cancer (0-IIc, tub1, after eradication of H. pyrrolili), showing normal MVP + MSP with a borderline, and B is a differentiated cancer (0-IIc, tub2, H. After eradication of pirori), showing normal MVP + MSP with borderline, C is differentiated cancer (0-IIa, tube1, after eradication of H. pirori), showing normal MVP + MSP with borderline. Note that D is an example of an image having bleeding diagnosed as false negative, and E is an example of an image having a low magnification field and out of focus also diagnosed as false negative. Here, 0-IIc indicates a surface recessed type, 0-IIa indicates a surface raised type, tub1 indicates a well-differentiated adenocarcinoma, and tub2 indicates a moderately differentiated adenocarcinoma.

An example of a heat map is shown in FIG. In FIG. 23, A is a differentiated cancer (0-IIc, tub1), B is a differentiated cancer (0-IIc, tub1), C is a patchy redness, and D is an adenoma. ~ D are heat maps corresponding to A to D, respectively. Here, 0-IIc indicates a surface recess type, and tube1 indicates a well-differentiated adenocarcinoma.

In the early gastric cancer image, the area determined to be cancer by the CNN system of Embodiment 5 was displayed in red, which coincided with the area determined to be cancer by the endoscopist. The image of the gastric adenoma and the image showing the patchy redness are determined to be non-cancerous by the CNN system of Embodiment 5 and are not displayed in red on the heat map. Similarly, these non-cancerous areas coincided with the areas determined by the endoscopist to be non-cancerous.

The CNN system of the fifth embodiment showed higher diagnostic accuracy than those obtained conventionally. The most important difference from the conventional one was the ME-NBI observation method. Because the maximum magnification water immersion method used in the CNN system of Embodiment 5 can eliminate halation and produce a perfectly focused, clear image of uniform quality suitable for endoscopic diagnosis. , These images are ideal for diagnostic support by the CNN system. Also, in order to diagnose early gastric cancer recorded in the video, it is necessary to analyze more than 30 images per second with the CNN system. The CNN system of Embodiment 5 can analyze 38.3 images per second, thus overcoming this technical limitation. Therefore, it is expected that the CNN system of the fifth embodiment can be applied to the diagnosis by the video image by the procedure such as ME-NBI.

[Embodiment 6]
In the sixth embodiment, ordinary white light imaging (WLI) is performed on a method for supporting a diagnosis of a disease using an endoscopic image of the present invention, a diagnosis support system, a diagnosis support program, and a computer-readable recording medium storing the diagnosis support program. , Non-enlarged narrow band imaging (NBI) and indigocarmine dye application (Indigo) diagnostic support method, diagnostic support system, diagnostic support program and diagnostic support method for diagnosing the depth of gastric cancer using three types of endoscopic images. A computer-readable recording medium that stores this diagnostic support program will be described.

The stomach is roughly divided from the inner surface side into a mucosal layer (M), a submucosal layer (SM), a mucosal muscle layer (MP), a subserosal layer (SS), and a serosa (SE). Of these, those classified as early gastric cancer are those in which the tip of the lesion remains in the mucosal layer (M) or submucosal layer (SM), and the tip of the lesion is deeper than the submucosal layer (SM). Those that have invaded the area are classified as advanced gastric cancer. According to the guidelines of the Japanese Society of Gastric Cancer, among gastric cancers, curative resection can be achieved by endoscopic submucosal dissection (ESD) because the lesion of gastric cancer remains in the mucosal layer (M). Invasion into the submucosa (SM) is less than 500 μm (SM1), and surgical operation is required for deeper invasion of gastric cancer. Therefore, in order to determine whether ESD is optimal for early gastric cancer, the infiltration position (depth) of the tip of the lesion of gastric cancer is less than 500 μm (SM1) or 500 μm or more in the submucosa. Diagnosis of whether it is (SM2) is indispensable.

[About dataset]
In constructing the CNN system of the sixth embodiment, the approval of the ethics committee of the University of Tokyo (No. 11931) and the Japan Medical Association (ID JMA-IIA00283) was obtained. At the clinic to which one of the inventors belongs, a retrospective review of endoscopy and pathology reports from January 2013 to June 2019, preoperative endoscopy, followed by endoscopy Cases in which endoscopic resection or surgery was performed and the final pathological diagnosis was gastric cancer were extracted. An image of a typical gastric cancer is shown in FIG. In FIG. 24, (a) is an example of an endoscopic image of gastric cancer invading deep submucosa, and (b) is an example of an endoscopic image of gastric cancer perched in the mucosa. Among the extracted cases, those with fundic gland-type gastric cancer, those with previous treatment history for gastric cancer, unanalyzable and low-quality images, and those with multiple lesions in one endoscopic image Excluded.

Evis Lucera Spectrum system or Evis Lucera Elite system (Olympus, Tokyo, Japan) is mainly used for endoscopic surgery, and high resolution or high definition endoscopes (GIF-Q240, GIF-Q240Z, GIF-H260, GIF-Q260, GIF-H260Z, GIF-PQ260, GIF-XP260N, GIF-H290, GIF-H290Z, GIF-XP290N, Olympus, Tokyo, Japan) were used. All images were extracted that could show at least half of the entire lesion of interest. If the video file is accessible, if possible, with conventional white light imaging (WLI), non-magnification narrowband imaging (NBI), and 0.1% indigocarmine dyeing imaging (Indigo). If possible, a total of two or more close-up images and distant-view images were extracted from the video.

The collected images were randomly divided into training and test datasets at a ratio of 4: 1 by computerized randomization. Facebook's PyTorch (https://pytorch.org/) deep learning framework was used to train, validate and test the AI system of Embodiment 6. The AI systems of Embodiment 6, which use WLI images, NBI images, and Indigo images, respectively, are defined as AI system (WLI), AI system (NBI), and AI system (Indigo), respectively.

Here, a total of 16,557 images were extracted from 1084 cases of gastric cancer that underwent endoscopic resection or surgery from January 2013 to June 2019 at the University of Tokyo Hospital. Details of the training image and test image datasets are shown in Table 21.

The training dataset for the CNN algorithm includes 8217 WLI images (6030 for training and 2241 for verification), including 884 lesions (428 for training and 184 for verification), and 629 lesions (for training). 2701 NBI images (1889 for training and 812 for verification) including 445 and 184 for verification, and 2656 Indigo images (291 for training and 125 for verification) including 416 lesions (291 for training and 125 for verification). It consisted of 1909 sheets for training and 747 sheets for verification). In addition, the test data set consists of a total of 1715 WLI images including 236 lesions of gastric cancer, 575 NBI images including 158 lesions, and 639 Indigo images including 111 lesions. It was. The images of the training and test datasets were mutually exclusive.

[Training / Verification / Algorithm]
In Embodiment 6, three independent AI systems were developed to predict the depth of gastric cancer invasion using WLI images, NBI images and Indigo images, respectively. The AI system of the sixth embodiment was developed by transfer learning utilizing ResNet-50, which is a state-of-the-art CNN architecture, as in the case of the fifth embodiment. In the CNN system of Embodiment 6, transfer learning is used to replace the final classification layer with another fully connected layer, retrain using the training dataset, and parameterizing all layers. Fine-tuned. All images were resized to 224 x 224 pixels. To increase the number of images, the images were expanded by vertical inversion, horizontal inversion, and scaling. All layers of the CNN were trained using a stochastic gradient descent algorithm with parameters of batch size: 64, global learning rate: 0.001, and epoch count: 100.

The trained CNN-based AI system of Embodiment 6 is programmed to output probability scores (range, 0-1) of "M or SM1" and "SM2 and above". "M" indicates the case where the tip of the lesion stays in the mucosal layer, "SM1" indicates the case where the tip of the lesion stays in the submucosal depth of less than 500 μm, and "SM2 or more" indicates the case of the lesion. Not only when the depth in the submucosal layer at the tip is 500 μm or more, but also when it invades into the muscularis mucosae (MP), subserosal layer (SS), serosa (SE), or to other organs. It also includes the case of infiltration (SI). In addition, "positive" was defined as a histologically proven cancer infiltrating SM2 or higher.

[Measurement and statistics of results]
Sensitivity, specificity, accuracy, positive predictive value and negative predictive value were calculated on an image-based and lesion-based basis for the three CNN-based AI systems of Embodiment 6. Regarding the lesion-based accuracy, the correct answer for each lesion in the test image was the correct answer when the majority of the images were correct for each lesion.

All data were statistically analyzed using JMP version 14.2 (SAS Institute Inc., Cary, North Carolina, USA). Each data was presented as frequency and percentage, or mean (standard deviation), as needed, and the number of lesions was also shown in each type of data. A comparison of lesion-based accuracy between the three AI systems using WLI, NBI, and Indigo images was performed by Wilcoxon rank sum test. The Wilcoxon rank sum test was performed on numerical data variables between the training and test datasets. Fisher's exact test was also performed on all categorical variables. All p-values were by two-sided testing and p <0.05 was considered statistically significant. In addition, a receiver operating characteristic curve (ROC) was used to evaluate the cutoff value for the three AI systems based on the CNN of Embodiment 6.

Table 22 shows the detailed clinical features of the patients and lesions included in the training and test datasets. The background factors between the WLI training and test datasets were 75.9% and 74.6% male (p-value = 0.67) with an average age of 70.4. There was no substantial difference between the ages of 71.7 years and 71.7 years (p-value = 0.10). The distribution of depth of invasion is also shown in Table 22.

The receiver operating characteristic curve (ROC) of the probability of invasion classification for the WLI test image is shown in FIG. 25A. The area under the curve (AUC) of the AI system (WLI) was 0.9590. According to the Youden index, the optimal cutoff value for the probability score was 0.5448. Diagnosis by the AI system (WLI) was made based on this probability score. The performance of the AI system (WLI) is shown in Table 23.

The image-based sensitivity, specificity, accuracy, positive predictive value and negative predictive value of the AI system (WLI) were 89.2%, 98.7%, 94.4%, 98.3% and 91. It was 7%. In addition, the AI system (WLI) lesion-based sensitivity, specificity, accuracy, positive predictive value and negative predictive value were 84.4%, 99.4%, 94.5%, 98.5% and 98.5%, respectively. It was 92.9%.

Further, as shown in FIG. 25B, the AUC of the AI system (NBI) was 0.9048, and the optimum cutoff value of the probability score was 0.4031. Further, as shown in FIG. 25C, the AUC of the AI system (Indigo) was 0.9491, and the optimum cutoff value of the probability score was 0.6094.

The image-based accuracy of the AI system (NBI) and AI system (Indigo) was 93.9% and 94.2%, respectively. Also, the lesion-based accuracy of the AI system (NBI) and (Indigo) was 94.3% and 95.5%, respectively. That is, there was no significant difference in lesion-based accuracy between the AI system (WLI), AI system (NBI) and AI system (Indigo) regarding the classification of invasion depth (p = 0.41), all of which were highly accurate. It was confirmed that the depth of invasion can be diagnosed. Therefore, according to the three types of AI systems of the sixth embodiment, it is highly determined whether the depth of penetration of the tip of the lesion of gastric cancer is less than 500 μm (SM1) or 500 μm or more (SM2) in the submucosa. It was confirmed that the diagnosis could be made accurately, and it was confirmed that it was possible to accurately determine whether or not ESD was optimal.

[Embodiment 7]
A method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the seventh embodiment will be described with reference to FIG. In the seventh embodiment, the method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using the CNN system of the first to sixth embodiments can be used. In S1, the CNN system is trained / verified using a first endoscopic image of the digestive organs and a definitive positive or negative diagnostic result of the disease of the digestive organs corresponding to the first endoscopic image. To do.

In S2, the CNN system trained / validated in S1 is based on a second endoscopic image of the gastrointestinal tract, with at least one positive and / or negative of the gastrointestinal tract disease and a positive probability score. Is output. This second endoscopic image shows a newly observed or input endoscopic image.

Further, in S2, the second endoscope image is an image being taken by the endoscope, an image transmitted via a communication network, an image provided by a remote control system or a cloud-type system, and a computer reading. It may be at least one of images or moving images recorded on a possible recording medium.

[Embodiment 8]
The disease diagnosis support system based on the endoscopic image of the digestive organ, the diagnostic support program based on the endoscopic image of the digestive organ, and the computer-readable recording medium of the eighth embodiment will be described with reference to FIG. 27. In the eighth embodiment, the method for supporting the diagnosis of a disease by the endoscopic image of the digestive organ described in the seventh embodiment can be used.

The disease diagnosis support system 1 using an endoscopic image of the digestive organs includes an endoscopic image input unit 10, an output unit 30, a computer 20 incorporating a CNN program, and an output unit 30. The computer 20 has a first storage area 21 for storing a first endoscopic image of the digestive organs, and a positive or negative gastrointestinal disorder, a past disease, or a severe condition corresponding to the first endoscopic image. It includes a second storage area 22 for storing at least one definitive diagnosis result of information corresponding to the level of degree or the imaged site, and a third storage area 23 for storing the CNN program. The CNN program stored in the third storage area 23 is divided into a first endoscopic image stored in the first storage area 21 and a definitive diagnosis result stored in the second storage area 22. Based on the second endoscopic image of the digestive organs that has been trained / verified and input from the endoscopic image input unit 10, the digestive organs of the digestive organ disease with respect to the second endoscopic image. The positive and / or negative of the disease and at least one of the positive probability scores are output to the output unit 30.

Further, the second endoscope image to be stored in the third storage area is provided by an image being photographed by the endoscope, an image transmitted via a communication network, a remote control system, or a cloud-type system. It may be at least one of an image, an image recorded on a computer-readable recording medium, or a moving image.

The disease diagnosis support system based on the endoscopic image of the digestive organs of the eighth embodiment includes a diagnostic support program based on the endoscopic image of the digestive organs for operating a computer as each means. In addition, the diagnostic support program using endoscopic images of the digestive organs can be stored in a computer-readable recording medium.

10 ... Endoscopic image input unit 20 ... Computer 21 ... First storage area 22 ... Second storage area 23 ... Third storage area 30 ... Output unit

Claims (34)

The first endoscopic image of the digestive system and
At least one definitive diagnosis result of the information corresponding to the positive or negative of the digestive organ disease corresponding to the first endoscopic image, and
Train a convolutional neural network system using
The trained convolutional neural network system detects the disease of the digestive organ and makes the disease positive based on the second endoscopic image of the digestive organ input from the endoscopic image input unit. A method for supporting the diagnosis of diseases by endoscopic images of digestive organs using a convolutional neural network system that outputs at least one of a corresponding region and a probability score.
The endoscopic image is a wireless capsule endoscopic image of the small intestine, and the trained convolutional neural network system detects it in a second wireless capsule endoscopic image input from the endoscopic image input unit. A method for supporting the diagnosis of a disease by endoscopic images of the digestive organs using a convolutional neural network system, which is characterized by outputting an area of an elevated lesion as the disease. The digestive organ using the convolutional neural network system according to claim 1, wherein the trained convolutional neural network system further displays a probability score of the disease in the second endoscopic image. A method for supporting the diagnosis of diseases using endoscopic images. The trained convolutional neural network system comprises the area of the elevated lesion displayed in the second endoscopic image based on the positive or negative definitive diagnosis of the disease in the small intestine.
The area of the elevated lesion in the second endoscopic image detected by the trained convolutional neural network system is displayed.
By overlapping the area of the raised lesion based on the definitive diagnosis result displayed in the second endoscopic image with the detected area of the raised lesion, the trained convolutional neural network system A method for supporting diagnosis of a disease by using an endoscopic image of a digestive organ using the convolutional neural network system according to claim 2, wherein the correctness of the diagnosis result is determined. The overlap is
(1) When it is 80% or more of the area of the elevated lesion based on the definitive diagnosis result, or
(2) When there are a plurality of positive regions of the disease detected by the trained convolutional neural network system, and any one region overlaps with the region of the elevated lesion based on the definitive diagnosis result. ,
The method for supporting diagnosis of a disease using an endoscopic image of a digestive organ using the convolutional neural network system according to claim 3, wherein the diagnosis of the trained convolutional neural network system is determined to be correct. The trained convolutional neural network system is characterized in determining that the detected elevated lesion is any of a polyp, a nodule, an epithelial tumor, a submucosal tumor and a vascular structure. A method for supporting diagnosis of a disease by endoscopic images of digestive organs using the convolutional neural network system according to any one of -4. The first endoscopic image of the digestive system and
At least one definitive diagnosis result of the information corresponding to the positive or negative of the digestive organ disease corresponding to the first endoscopic image, and
Train a convolutional neural network system using
The trained convolutional neural network system detects the disease of the digestive organ and makes the disease positive based on the second endoscopic image of the digestive organ input from the endoscopic image input unit. A method for supporting the diagnosis of diseases by endoscopic images of digestive organs using a convolutional neural network system that outputs at least one of a corresponding region and a probability score.
The endoscopic image is a wireless capsule endoscopic image of the small intestine, and the trained convolutional neural network system detects it in a second wireless capsule endoscopic image input from the endoscopic image input unit. It is characterized in that the probability score of bleeding as the disease is output.
A method for supporting the diagnosis of diseases by endoscopic images of the digestive organs using a convolutional neural network system. The first endoscopic image of the digestive system and
The definitive diagnosis result of the information corresponding to the positive or negative of the digestive organ disease corresponding to the first endoscopic image, and
Train a convolutional neural network system using
The trained convolutional neural network system detects the disease of the digestive organ and makes the disease positive based on the second endoscopic image of the digestive organ input from the endoscopic image input unit. A method for supporting the diagnosis of diseases by endoscopic images of digestive organs using a convolutional neural network system that outputs at least one of a corresponding region and a probability score.
The first endoscopic image is a still image of a wireless capsule endoscope of the small intestine.
The second endoscopic image is a moving image of a wireless capsule endoscope of the small intestine.
The trained convolutional neural network system displays a region of the disease detected in a second wireless capsule endoscopic image input from the endoscopic image input unit. A method for supporting diagnosis of diseases by endoscopic images of the digestive organs using the system. An endoscopic image of a digestive organ using the convolutional neural network system according to claim 7, wherein the area of the disease is at least one of mucosal damage, vasodilation, elevated lesions, and bleeding. Diagnosis support method for diseases caused by. The area of mucosal damage is at least one of erosions and ulcers.
The region of vasodilation is at least one of vasodilator type 1a and vasodilator type 1b.
Within the digestive organs using the convolutional neural network system according to claim 8, wherein the area of the elevated lesion is at least one of a polyp, a nodule, a submucosal tumor, a vascular structure and an epithelial tumor. A method for supporting the diagnosis of diseases using endoscopic images. The trained convolutional neural network system
The first convolutional neural network system part, whose definitive diagnostic results were trained by still images of mucosal damage,
A second convolutional neural network system part where the definitive diagnosis was trained with still images of vasodilation,
A third convolutional neural network system part where the definitive diagnosis was trained by still images of elevated lesions,
The fourth convolutional neural network system part, whose definitive diagnostic results were trained by still images of bleeding,
Support for diagnosing a disease by endoscopic images of the digestive organs using the convolutional neural network system according to any one of claims 7-9, which comprises using one consisting of the complex convolutional neural network system of the above. Method. The first endoscopic image of the digestive system and
At least one definitive diagnosis result of the information corresponding to the positive or negative of the digestive organ disease corresponding to the first endoscopic image, and
Train a convolutional neural network system using
The trained convolutional neural network system detects the disease of the digestive organ and makes the disease positive based on the second endoscopic image of the digestive organ input from the endoscopic image input unit. A method for supporting the diagnosis of diseases by endoscopic images of digestive organs using a convolutional neural network system that outputs at least one of a corresponding region and a probability score.
The endoscopic image is an endoscopic image obtained by inundation method narrow band imaging of the stomach, and the trained convolutional neural network system is used as the disease of the endoscopic image input from the endoscopic image input unit. Characterized by outputting the area of early gastric cancer,
A method for supporting the diagnosis of diseases by endoscopic images of the digestive organs using a convolutional neural network system. The digestion using the convolutional neural network system according to claim 11, wherein the trained convolutional neural network network system has a function of displaying a region of early gastric cancer as the disease on a heat map. A method for supporting diagnosis of diseases using endoscopic images of organs. The first endoscopic image of the digestive system and
At least one definitive diagnosis result of the information corresponding to the positive or negative of the digestive organ disease corresponding to the first endoscopic image, and
Train a convolutional neural network system using
The trained convolutional neural network system detects the disease of the digestive organ and makes the disease positive based on the second endoscopic image of the digestive organ input from the endoscopic image input unit. A method for supporting the diagnosis of diseases by endoscopic images of digestive organs using a convolutional neural network system that outputs at least one of a corresponding region and a probability score.
The endoscopic image is at least one selected from a white light image of the stomach, a narrow band imaging image and an indigo carmine dye spray image.
The trained convolutional neural network system is characterized by outputting the invasion depth of the disease as the disease of the endoscopic image input from the endoscopic image input unit.
A method for supporting the diagnosis of diseases by endoscopic images of the digestive organs using a convolutional neural network system. According to an endoscopic image of a digestive organ using the convolutional neural network system according to claim 13, which outputs whether the submucosal infiltration is less than 500 μm or more than 500 μm as the invasion depth. Disease diagnosis support method. Any of claims 1-14, wherein the convolutional neural network is further combined with three-dimensional information from an X-ray computed tomography apparatus, an ultrasonic computed tomography apparatus, or a magnetic resonance imaging diagnostic apparatus. A method for supporting diagnosis of a disease by endoscopic images of the digestive organs using the convolutional neural network system described in. The second endoscopic image is an image being taken by the endoscope, an image transmitted via a communication network, an image provided by a remote control system or a cloud-type system, or a computer-readable recording medium. Diagnosis of a disease by an endoscopic image of the digestive organs using the convolutional neural network system according to any one of claims 1-14, characterized in that it is at least one of the images or moving images recorded in. Support method. It is a disease diagnosis support system using an endoscopic image of the digestive organs using a convolutional neural network system having an endoscopic image input unit, an output unit, and a computer incorporating a convolutional neural network.
The computer
A first storage area for storing the first endoscopic image of the digestive organs,
A second storage area corresponding to the first endoscopic image and storing a definitive diagnosis result of information corresponding to the positive and negative of the disease in the digestive organs.
A third storage area for storing the convolutional neural network program,
With
The convolutional neural network program
It is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area.
Based on the second endoscopic image of the digestive organs input from the endoscopic image input unit, the information corresponding to the positive or negative of the disease of the digestive organs with respect to the second endoscopic image is described. It is supposed to be output to the output section,
The endoscopic image is a wireless capsule endoscopic image of the small intestine, and the trained convolutional neural network program is a prominence of the wireless capsule endoscopic image input from the endoscopic image input unit as the disease. A disease diagnosis support system using endoscopic images of the digestive organs using a convolutional neural network system, which is characterized by outputting the area of sexual lesions. The digestive organ using the convolutional neural network system according to claim 17, wherein the trained convolutional neural network program further displays a probability score of the disease in the second endoscopic image. Disease diagnosis support system using endoscopic images. The convolutional neural network program
In the second endoscopic image, the area of the elevated lesion displayed based on the positive or negative definitive diagnosis of the disease in the small intestine and was detected by the trained convolutional neural network system. Display the area of the elevated lesion and
The correctness of the diagnosis result is determined by the overlap between the region of the raised lesion based on the definitive diagnosis result displayed in the second endoscopic image and the detected region of the raised lesion. The disease diagnosis support system based on an endoscopic image of a digestive organ using the convolutional neural network system according to claim 17, wherein the system comprises. The overlap is
(1) When it is 80% or more of the area of the elevated lesion based on the definitive diagnosis result, or
(2) When there are a plurality of positive regions of the disease detected by the trained convolutional neural network system, and any one region overlaps with the region of the elevated lesion based on the definitive diagnosis result. ,
The disease diagnosis support system using an endoscopic image of a digestive organ using the convolutional neural network system according to claim 19, wherein the diagnosis of the trained convolutional neural network system is determined to be correct. The trained convolutional neural network program is characterized by displaying in the second image whether the elevated lesion is a polyp, a nodule, an epithelial tumor, a submucosal tumor or a vascular structure. The disease diagnosis support system based on an endoscopic image of a digestive organ using the convolutional neural network system according to claim 17. It is a disease diagnosis support system using an endoscopic image of the digestive organs using a convolutional neural network system having an endoscopic image input unit, an output unit, and a computer incorporating a convolutional neural network.
The computer
A first storage area for storing the first endoscopic image of the digestive organs,
A second storage area corresponding to the first endoscopic image and storing a definitive diagnosis result of information corresponding to the positive and negative of the disease in the digestive organs.
A third storage area for storing the convolutional neural network program,
With
The convolutional neural network program
It is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area.
Based on the second endoscopic image of the digestive organs input from the endoscopic image input unit, the information corresponding to the positive or negative of the disease of the digestive organs with respect to the second endoscopic image is described. It is supposed to be output to the output section,
The endoscopic image is a wireless capsule endoscopic image of the small intestine, and the trained convolutional neural network program displays a probability score of bleeding as the disease in the second image. A disease diagnosis support system using endoscopic images of the digestive organs using a convolutional neural network system. It is a disease diagnosis support system using an endoscopic image of the digestive organs using a convolutional neural network system having an endoscopic image input unit, an output unit, and a computer incorporating a convolutional neural network.
The computer
A first storage area for storing the first endoscopic image of the digestive organs,
A second storage area corresponding to the first endoscopic image and storing a definitive diagnosis result of information corresponding to the positive and negative of the disease in the digestive organs.
A third storage area for storing the convolutional neural network program,
With
The convolutional neural network program
It is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area.
Based on the second endoscopic image of the digestive organs input from the endoscopic image input unit, the information corresponding to the positive or negative of the disease of the digestive organs with respect to the second endoscopic image is described. It is supposed to be output to the output section,
The first endoscopic image is a still image of a wireless capsule endoscope of the small intestine.
The second endoscopic image is a moving image of a wireless capsule endoscope of the small intestine.
The trained convolutional neural network program displays a region of the disease detected in a second wireless capsule endoscopic image input from the endoscopic image input unit. A disease diagnosis support system using endoscopic images of the digestive organs using the system. Endoscopic image of the digestive organs using the convolutional neural network system according to claim 23, wherein the area of the disease is at least one of mucosal damage, vasodilation, elevated lesions, and bleeding. Disease diagnosis support system by. The area of mucosal damage is at least one of erosions and ulcers.
The region of vasodilation is at least one of vasodilator type 1a and vasodilator type 1b.
Within the digestive organs using the convolutional neural network system of claim 24, wherein the area of the elevated lesion is at least one of a polyp, a nodule, a submucosal tumor, a vascular structure and an epithelial tumor. Disease diagnosis support system using endoscopic images. The trained convolutional neural network system
The first convolutional neural network system part, whose definitive diagnostic results were trained by still images of mucosal damage,
A second convolutional neural network system part where the definitive diagnosis was trained with still images of vasodilation,
A third convolutional neural network system part where the definitive diagnosis was trained by still images of elevated lesions,
The fourth convolutional neural network system part, whose definitive diagnostic results were trained by still images of bleeding,
A disease diagnosis support system based on an endoscopic image of a digestive organ using the convolutional neural network system according to any one of claims 23 to 25, which comprises the composite convolutional neural network system of the above. It is a disease diagnosis support system using an endoscopic image of the digestive organs using a convolutional neural network system having an endoscopic image input unit, an output unit, and a computer incorporating a convolutional neural network.
The computer
A first storage area for storing the first endoscopic image of the digestive organs,
A second storage area corresponding to the first endoscopic image and storing a definitive diagnosis result of information corresponding to the positive and negative of the disease in the digestive organs.
A third storage area for storing the convolutional neural network program,
With
The convolutional neural network program
It is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area.
Based on the second endoscopic image of the digestive organs input from the endoscopic image input unit, the information corresponding to the positive or negative of the disease of the digestive organs with respect to the second endoscopic image is described. It is supposed to be output to the output section,
The endoscopic image is an endoscopic image obtained by inundation method narrow band imaging of the stomach, and the trained convolutional neural network program is used as the disease of the endoscopic image input from the endoscopic image input unit. A disease diagnosis support system using endoscopic images of the digestive organs using a convolutional neural network system, which is characterized by outputting the area of early gastric cancer. The digestive organ using the convolutional neural network system according to claim 27, wherein the trained convolutional neural network program has a function of displaying a region of early gastric cancer as the disease on a heat map. Disease diagnosis support system using endoscopic images. It is a disease diagnosis support system using an endoscopic image of the digestive organs using a convolutional neural network system having an endoscopic image input unit, an output unit, and a computer incorporating a convolutional neural network.
The computer
A first storage area for storing the first endoscopic image of the digestive organs,
A second storage area corresponding to the first endoscopic image and storing a definitive diagnosis result of information corresponding to the positive and negative of the disease in the digestive organs.
A third storage area for storing the convolutional neural network program,
With
The convolutional neural network program
It is trained based on the first endoscopic image stored in the first storage unit and the definitive diagnosis result stored in the second storage area.
Based on the second endoscopic image of the digestive organs input from the endoscopic image input unit, the information corresponding to the positive or negative of the disease of the digestive organs with respect to the second endoscopic image is described. It is supposed to be output to the output section,
The endoscopic image is at least one selected from a white light image of the stomach, a narrow band imaging image and an indigocarmine dye spray image, and the trained convolutional neural network program is from the endoscopic image input section. A disease diagnosis support system using an endoscopic image of a digestive organ using a convolutional neural network system, which outputs the depth of invasion of the disease as the disease of the input endoscopic image. 29. The trained convolutional neural network program is characterized by having a function of outputting whether the submucosal invasion is less than 500 μm or more than 500 μm as the invasion depth of the disease. A disease diagnosis support system using an endoscopic image of the digestive organs using the convolutional neural network system described in. Any of claims 17-30, wherein the convolutional neural network program is further combined with three-dimensional information from an X-ray computed tomography apparatus, an ultrasonic computed tomography apparatus, or a magnetic resonance imaging diagnostic apparatus. A disease diagnosis support system using endoscopic images of the digestive organs using the convolutional neural network system described in Crab. The second endoscopic image is an image being taken by the endoscope, an image transmitted via a communication network, an image provided by a remote control system or a cloud-type system, or a computer-readable recording medium. Diagnosis of a disease by an endoscopic image of the digestive organs using the convolutional neural network system according to any one of claims 17-31, characterized in that it is at least one of the images or moving images recorded in. Support system. A convolutional neural network system is used, which is for operating a computer as each means in the disease diagnosis support system based on the endoscopic image of the digestive organ according to any one of claims 17-32. A diagnostic support program using endoscopic images of the digestive organs. A computer-readable recording medium comprising recording a diagnostic support program using an endoscopic image of a digestive organ using the convolutional neural network system according to claim 33.

Images