TWI863845B - Method and system for auxiliary medical diagnosis for gastritis - Google Patents

Abstract

Provided are a method and a system for Medical auxiliary diagnostic. The medical auxiliary diagnostic system includes a sampling system, a user device, and an image analysis system. The sampling system collects and stores medical images, including a first dataset and a second dataset. The user device is connected to the sampling system, reading medical images from the sampling system or uploading medical images. The image analysis system is connected to both the sampling system and the user device to analyze medical images based on requests from the user device, thereby generating auxiliary images for assisting in diagnosis. The image analysis system performs deep learning based on the first dataset and the second dataset. Based on the results of the deep learning, the image analysis system performs inferences on a test image to generate auxiliary diagnostic images for assisting in identifying precancerous lesions. The first dataset is derived from a non-specific region with an average incidence rate, while the second dataset is derived from a specific region with an incidence rate higher than the average incidence rate. Both the first dataset and the second dataset include upper gastrointestinal endoscopic images.

Description

Medically assisted diagnosis system and method for medically assisted diagnosis of gastritis

The present invention relates to medical image recognition technology, and in particular to a system and method for diagnosing gastric precancerous lesions and Helicobacter pylori infection using artificial intelligence.

Gastric cancer is a major global health problem. In 2020 alone, there were more than 1 million new cases and approximately 769,000 deaths. In terms of incidence, gastric cancer is ranked as the fifth most common cancer and the fourth leading cause of cancer death worldwide. The main cause of gastric cancer is Helicobacter pylori infection, which can be prevented with short-term antibiotic treatment. However, even if the above infection has been successfully eradicated, individuals may still develop precancerous lesions such as atrophic gastritis and intestinal metaplasia. Therefore, patients need to continue to receive endoscopic surveillance for early detection and treatment. In general, histological evaluation is recognized as one of the most effective methods for diagnosing gastric precancerous lesions and Helicobacter pylori infection, allowing for stratification of gastric cancer risk. However, the bottleneck of this method is that it is time-consuming, requires experts, and has uncertainties in the ideal number and location of gastroscopic slices.

According to statistics, Matsu, located off the coast of Taiwan, is a high-risk area for gastric cancer. In order to eliminate the threat of gastric cancer to local people, relevant units have been carrying out a large-scale Helicobacter pylori eradication program for nearly 20 years. Although the above program has reduced the incidence of gastric cancer by about 50%, sporadic cases of gastric cancer continue to occur in the area. It is generally believed that if appropriate population stratification can be further carried out according to individual risks, better disease prevention and control benefits should be obtained. Therefore, a medical assistance system combined with modern computer computing technology is to be developed.

In view of this, the present invention proposes a medical auxiliary diagnosis system based on artificial intelligence (AI) to assist frontline doctors in diagnosing precancerous gastric lesions and Helicobacter pylori infection in the real world.

In one embodiment, a medical auxiliary diagnosis system includes a sampling system, a user device, and an image analysis system. The sampling system is used to collect and store medical images, wherein a first data set and a second data set are stored. The user device is connected to the sampling system and is configured to read medical images from the sampling system or upload medical images. The image analysis system is connected to the sampling system and the user device and is configured to analyze medical images according to the requirements of the user device to generate auxiliary images that can assist in diagnosis. The image analysis system reads the first data set and the second data set from the sampling system to perform deep learning. The image analysis system makes inferences on a test image provided by the image analysis system based on the result of the deep learning to generate an auxiliary diagnostic image that can assist in identifying precancerous lesions and Helicobacter pylori infection. The first data set comes from an unspecified region and has an average incidence rate. The second data set comes from a specific region and has an incidence rate higher than the average incidence rate. The first data set and the second data set include upper gastrointestinal endoscopic images.

In one embodiment, the image analysis system includes a training module, a first model, a second model, and a third model. The training module performs the deep learning based on the first data set and the second data set. The first model is trained by the training module and can be used to identify the gastric area and the non-gastric area in a sample image. The second model is trained by the training module and can further classify the gastric area identified in the sample image into the gastric sinus, the gastric body, and the gastric fundus. The third model is trained by the training module and can determine whether there is a precancerous lesion or Helicobacter pylori infection in the gastric sinus and the gastric body in the sample image. The sample image is an upper gastrointestinal tract endoscopic image from the first data set or the second data set. The precancerous lesions include atrophic gastritis or intestinal metaplasia.

Furthermore, the image analysis system may include a preprocessing module configured to preprocess an input image to generate a normalized image. When the preprocessing module preprocesses the input image, the preprocessing module converts the input image into a grayscale image, and converts the grayscale image into a binary map using an Otsu threshold. The preprocessing module performs edge detection on the binary map to identify a target area in the input image, and crops the input image to obtain a cropped image that only retains the target area. The preprocessing module resizes the cropped image to a preset dimension to generate the normalized image corresponding to the input image. Before the training module performs the deep learning on the first dataset and the second dataset, the preprocessing module is used to normalize the images in the first dataset and the second dataset.

Furthermore, when the preprocessing module preprocesses the input image, the preprocessing module may perform a contrast limited adaptive histogram equalization (CLAHE) operation on the input image to enhance image details and highlight gastric mucosal features.

In a further embodiment, the image analysis system includes an inference module, which is configured to use a first model, a second model, and a third model to infer the image to be tested to determine the risk of gastric cancer. The inference module uses the first model to identify the gastric area and the non-gastric area in the image to be tested. The inference module uses the second model to further classify the gastric area in the image to be tested into the gastric sinus, the gastric body, and the gastric fundus. The inference module uses the third model to determine whether the precancerous lesion exists in the gastric sinus and the gastric body in the image to be tested. Finally, the inference module uses a gradient-weighted class activation mapping (Grad-CAM) operation to superimpose a visual effect on the image to be tested according to the probability of disease, thereby generating the auxiliary diagnostic image. The image to be tested is an upper gastrointestinal endoscopic image, which is collected and stored by the sampling system and provided to the image analysis system through the user device so that the inference module can perform analysis. Before analyzing the image to be tested, the inference module can also normalize the image to be tested through the preprocessing module.

In another embodiment, the first model can be trained by using the dense network DenseNet201 as a model template and fine-tuning parameters by the training module. When the training module trains the first model, the first data set and the second data set are read from the sampling system, and 70% of them are allocated as a training set, 20% are allocated as a validation set, and 10% are allocated as a test set.

In another embodiment, the second model is formed by using a dense network DenseNet121 as a model template and fine-tuning parameters of the training module, and includes a feature extraction module and a classification module. The feature extraction module includes a multi-layer neural network, which is configured to extract a feature map from the input data through a convolution operation. The classification module includes one or more fully connected layers, coupled to the output end of the feature extraction module, and configured to calculate the probability value of the input data of the feature extraction module belonging to the gastric fundus, gastric sinus or gastric body based on the feature map generated by the feature extraction module. When the training module trains the second model, it reads the first data set and the second data set from the sampling system, and allocates 80% of the first data set as the training set, 20% as the validation set, and 10% of the second data set as the test set. Finally, the classification module uses the SoftMax activation function to calculate the probability value of the input data of the feature extraction module belonging to the gastric fundus, gastric sinus or gastric body.

In another embodiment, the third model is trained by fine-tuning parameters of the training module using the Vision Transformer as a model template. The translation patch layer in the Vision Transformer can translate an input image to four diagonals to generate four displacement images, which are then stacked with the input image to form a translation stack map. The image cutting layer is coupled to the translation patch layer, cuts the translation stack map into multiple patches, and flattens them into a one-dimensional sequence, where each patch corresponds to a feature vector of a fixed length. The position embedding layer is coupled to the image cutting layer, receives multiple patches output by the image cutting layer, and adds position information to each patch. The multi-layer encoder is coupled to the position embedding layer, and performs self-attention operation on the input multiple patches to extract a feature vector matrix corresponding to the input image. The multi-layer perception head is coupled to the multi-layer encoder, and includes a multi-layer fully connected layer, which is configured to pool the feature vector matrix to obtain a probability distribution matrix that can represent the entire input image. The multi-layer encoder also performs local self-attention operation for each patch and its neighboring patches, so that the feature vector matrix emphasizes local correlation. The multi-layer perception head uses a Sigmoid activation function to calculate the probability distribution matrix, wherein each element in the probability distribution matrix represents a pathological severity.

In another embodiment, the present invention proposes a method for medically assisting the diagnosis of gastritis. First, a first data set and a second data set are provided, wherein the first data set and the second data set contain upper gastrointestinal endoscopic images. The method first performs image preprocessing on the first data set and the second data set, and then performs a deep learning on the first data set and the second data set. Finally, based on the result of the deep learning, an image to be tested is inferred to generate an auxiliary image for assisting in identifying whether the image to be tested contains a precancerous lesion or Helicobacter pylori infection. The first data set comes from an unspecified region and has an average morbidity. The second data set comes from a specific region and has a morbidity higher than the average morbidity.

In summary, the medical image analysis system mentioned in the present invention is characterized in that it can perform pathological identification on endoscopic images, generate lesion heat maps, and assist experts and scholars to improve diagnostic accuracy, reduce misdiagnosis rates, and optimize the allocation of subsequent treatment resources.

100: Medical Assisted Diagnosis System

110: Sampling system

112: Image acquisition device

114: Image Server

120: User device

130: Image analysis system

131: First Model

132: Second Model

133: The third model

134: Preprocessing module

136: Training module

138: Inference module

200: Assist medical diagnosis process

201-209: Steps

300: Image preprocessing process

301-309: Steps

400: Image estimation process

401-407: Steps

500: Visual Converter

501: Feature extraction module

502: Classification module

$IN: Input image

$OUT: output result

510: Convolution module

512: Max pooling layer

514: The first dense block

520: Volume module

522: Average pooling layer

524: The second dense block

530: Volume module

532: Average pooling layer

534: The third dense block

540: Volume module

542: Average pooling layer

544: The fourth dense block

550: Global pooling layer

552: First fully connected layer

554: Second fully connected layer

600: Visual converter model

602: Input image

604:Probability distribution matrix

610:Translate patch layer

620: Image cutting layer

630: Position embedding layer

640:Multi-layer encoder

650: Multi-layer sensing head

710: Original gastric sinus image

712: Enhanced gastric sinus imaging

720: Original gastric image

722: Enhanced gastric image

730: Original gastric fundus image

732: Enhanced gastric fundus image

810: Original image

820: Heatmap overlay

The diagrams described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation on the present invention. In the diagrams: FIG. 1 is a medical auxiliary diagnosis system of an embodiment of the present invention; FIG. 2 is a flowchart of auxiliary medical diagnosis of an embodiment of the present invention; FIG. 3 is a flowchart of image preprocessing of an embodiment of the present invention; FIG. 4 is a flowchart of image inference of an embodiment of the present invention; FIG. 5 is a visual converter of an embodiment of the present invention; FIG. 6 is an embodiment of the training process of the third model of the present invention; FIG. 7 is an image enhancement result of an embodiment of the present invention; and FIG. 8 is a heat map superposition result of an embodiment of the present invention.

The following will combine the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative labor are within the scope of protection of the present invention.

Atrophic gastritis and intestinal metaplasia are histological symptoms associated with gastric cancer risk and are precancerous lesions. The embodiment of the present invention provides a medical auxiliary diagnosis system 100, which uses routinely collected endoscopic images for analysis to assist experts in diagnosing gastric precancerous lesions. The basic structure of the embodiment of the present invention is illustrated in Figure 1 below.

FIG1 is a medical auxiliary diagnosis system 100 of an embodiment of the present invention, which can substitute the results of histological analysis into an artificial intelligence algorithm to generate a visual image that can assist in the diagnosis of precancerous lesions in the stomach.

In the medical-assisted diagnosis system 100 of this embodiment, two data sources with different attributes are used. The first data source is the data of a large medical center with a national average incidence rate, and the other data source is the data of a specific regional hospital with a high incidence rate. The specific content of the data described in this embodiment is gastric endoscopic images. After these gastric endoscopic images are processed by a deep learning algorithm after specific preprocessing, the process of endoscopic biopsy and histological evaluation by human experts can be simulated. The medical-assisted diagnosis system 100 of the present invention can be applied to specific hospitals with a high incidence rate to assist in the implementation of end-to-end remote medical services. The medical auxiliary diagnosis system 100 of the present invention has verified the robustness of system predictability in practice and has been registered at ClinicalTrials.gov NCT05762991.

The medical auxiliary diagnosis system 100 of the present invention applies an artificial intelligence model to assist in the identification of gastric endoscopic images. The process of applying the artificial intelligence model for training in the present invention generally includes the following steps. Data collection: Collect data for training the model. These data may be in the form of images, text, audio, etc., depending on the problem to be solved by the model. Data preprocessing: Perform preprocessing operations such as cleaning, standardization (normalization), feature extraction, etc. on the collected data to ensure the quality and applicability of the data. Data segmentation: Divide the data into a training set, a validation set, and a test set. The training set is used to train the model, the validation set is used to adjust the model's hyperparameters and evaluate the model's performance, and the test set is used to finally evaluate the model's generalization ability. Model template selection: Select a model template suitable for solving the problem, such as deep neural network, decision tree, support vector machine, etc. Model training: Use the training set data to train the model, and adjust the model parameters through optimization algorithms (such as gradient descent) so that the model can minimize the loss function. Model validation: During the training process, or at the end of each training cycle (epoch), use the validation set data to evaluate the performance of the model to guide the direction of model training and adjust the model's hyperparameters. The validation set can help monitor whether the model is overfitting or underfitting, and adjust the model's hyperparameters, such as learning rate, regularization parameter, etc., based on the monitoring results. The validation results are usually not used directly to update the model's parameters, but are used as a guiding evaluation to ensure that the model can perform well on unseen data. Hyperparameter tuning: Optimize the performance of the model by trying different hyperparameter combinations, such as learning rate, batch size, number of layers, etc. Model evaluation: After the model is trained, use the test set data to evaluate the model's performance to determine the model's generalization ability and the effect of practical application. The test set data usually uses a data set with known answers to facilitate comparison of machine learning recognition results and statistics of accuracy. Model deployment: Deploy the trained model to actual applications to solve practical problems.

In terms of data collection, the first data set source used by the medical auxiliary diagnosis system 100 of the embodiment of the present invention can be the centralized research database of the National Taiwan University Hospital Medical System. The database includes ten hospitals located on the main island of Taiwan. In order to meet the requirements of real-world evidence, since 2006, all electronic medical records of these hospitals, such as medical records, laboratory data, examination reports, pathological results, and medical images, have been anonymously transmitted to a comprehensive medical database called NTUH-iMD. In this embodiment, the first data set includes endoscopic images of patients, as well as histological examination results of the gastric sinus, gastric body, and gastric mucosa. The first dataset is the main basis for model training of the medical auxiliary diagnosis system 100.

On the other hand, the second data set can be from a local hospital in the Matsu Islands. Statistics show that the incidence of gastric cancer among local residents is very high. Since 2004, the relevant units have invited local residents over 30 years old to participate in the Helicobacter pylori screening program and provided endoscopic examinations to participants. These examinations established a database of potential pathological lesions and Helicobacter pylori infection, and the relevant units then used clinical trial methodology to statistically calculate the prevalence and severity of precancerous gastric lesions and the presence or absence of Helicobacter pylori infection from these databases to establish a second data set. In this embodiment, the second data set can be used to verify and test the model established from the first data set.

In summary, the sample source of the first dataset covers a wide range of the total population of the country, and the morbidity value is the average value of the whole population. The sample source of the second dataset is limited to the places with high morbidity, so it is suitable for verifying and testing the recognition results of the first dataset in the artificial intelligence training process. In this embodiment, the images in the first dataset and the second dataset mainly include gastric endoscopic images of the gastric body, gastric sinus, and gastric fundus.

In order to establish standardized histological data for subsequent training analysis by the medical auxiliary diagnosis system 100, the biopsy sample collection method of the first data and the second data can be improved according to the Sydney Protocol. The Sydney Protocol is a standardized protocol for tissue biopsy of gastric mucosa, which was originally developed by a group of gastric disease experts in Australia to standardize the location and processing methods of gastric mucosal biopsy to improve the accuracy and consistency of diagnosis. In this embodiment, gastric mucosal biopsy specimens can be obtained from the gastric sinus (at the greater and lesser curvatures of the stomach) and the gastric body (at the greater and lesser curvatures of the stomach). By standardizing the sampling procedure, experts can obtain gastric mucosal biopsy samples with consistent conditions. In other words, the sampling location is consistent for all subjects. Experts do not need to know the clinical status of the participants to objectively perform histological evaluation, thereby improving the reliability of statistical data. In this embodiment, the classification of pathological specimens includes acute inflammation (polymorphonuclear infiltration), chronic inflammation (lymphoplasmic cell infiltration), atrophic gastritis (glandular tissue loss and fibrous replacement) or intestinal metaplasia (presence of goblet cells and absorptive cells), and the severity of each classification is defined as none, mild, moderate or significant. According to the above classification, the severity of precancerous lesions can be determined by the Operative Link for Gastritis Assessment of Atrophic Gastritis (OLGA) and the Operative Link for Gastritis Assessment of Intestinal Metaplasia (OLGIM) as staging criteria, ranging from stage 0 to stage 4.

In terms of system architecture, the medical auxiliary diagnosis system 100 of the present invention includes a sampling system 110, a user device 120 and an image analysis system 130, which are connected to each other via a network.

The sampling system 110 in this embodiment generally refers to a hospital or clinic with detection instruments, including an image acquisition device 112 and an image server 114. The image acquisition device 112 generally refers to an X-ray machine, a tomography scan, an endoscopy, or various instruments and equipment that can generate patient examination data. The image server 114 is a picture archiving and communication system (PACS). In the medical field, medical images are transmitted in the Digital Imaging and Communications in Medicine (DICOM) format. Therefore, various information collected from the patient by the image acquisition device 112 is centrally stored in the image server 114 in the DICOM format. The user device 120 can be a clinic computer in the hospital or a mobile device held by medical staff, and can view the data stored in the image server 114 through a specific program interface and access mechanism. It should be understood that the number of sampling systems 110 can be plural, distributed in hospitals across the country, not limited to large medical centers or rural clinics. The network connection between the sampling system 110, the user device 120 and the image analysis system 130 is not limited to a wired network or a wireless network.

The image analysis system 130 represents the main server for executing artificial intelligence analysis. The image analysis system 130 includes a first model 131, a second model 132, a third model 133, a preprocessing module 134, a training module 136, and an inference module 138. The image analysis system 130 can be a physical host installed in a large medical center for use by staff. In order to facilitate medical personnel scattered across the country to access the image analysis system 130 through the user device 120, the image analysis system 130 can adopt a public cloud architecture. That is, the image analysis system 130 can also be a cloud system composed of multi-layered virtual services. The public cloud architecture is divided by function, and can be divided into at least a front-end interface and a back-end platform (not shown). The front-end interface can be provided in the form of a web service for users to operate through a browser on the user device 120. The back-end platform allows the sampling system 110 or the user device 120 to upload selected image information to the image analysis system 130 for processing, training, and inference. The components in the image analysis system 130 do not necessarily need to be physically located inside a large medical center, but are distributed in the virtual services of various cloud vendors. For example, each component in the image analysis system 130 can adopt a customized Software as a Service (SaaS) or an existing Platform as a Service (PaaS) solution on the market. Since the public cloud architecture is a known technology with extremely flexible and diverse solutions, this embodiment will not be described in detail. Therefore, it can be understood that the image analysis system 130 of the present invention, and each component module therein, is not limited to a physical implementation method, and the sampling system 110 and the image analysis system 130 are not limited to being located in the same or different locations.

The pre-processing module 134 is configured to receive image data provided by the image server 114 and pre-process it. The detailed pre-processing process will be described in FIG. 3.

The training module 136 is configured to perform deep learning for the analysis method required by the present invention based on the preprocessing result of the preprocessing module 134, and finally train the first model 131, the second model 132, and the third model 133, etc., which can achieve specific purposes. It can be understood that the artificial intelligence model referred to in the present invention generally refers to a combination of digital data of a neural network algorithm and specific parameters, which can be executed under a specific hardware and operating system architecture to generate corresponding output results based on input data. In one embodiment, the training module 136 can be a computing system that supports a convolutional neural network (CNN), which is driven by a specific operating system and software to perform artificial intelligence learning functions. For example, the training module 136 itself may include code written in Python, a deep learning framework based on Tensorflow, and an NVIDIA DGX A100 graphics card.

The inference module 138 is configured to use the first model 131, the second model 132 and the third model 133 to perform inference recognition for the input image to obtain a recognition result. The input image may be provided by the user device 120. After the image analysis system 130 receives the input image provided by the user device 120, it may be pre-processed by the pre-processing module 134 before being transmitted to the inference module 138 to perform the inference recognition process.

The first model 131 is trained by the training module 136 to recognize the stomach area and the non-stomach area in the upper gastrointestinal tract endoscopy image.

The second model 132 is trained by the training module 136 to further classify the stomach regions identified in the first model 131 into three categories: gastric sinus, gastric body, and gastric fundus.

The third model 133 is trained by the training module 136 to judge whether there are precancerous lesions in the gastric sinus and gastric body area, such as atrophic gastritis and intestinal metaplasia, as well as Helicobacter pylori infection.

The medical assisted diagnosis system 100 proposed by the present invention can improve the accessibility of medical care for the community and provide real-time evidence-based information for patient management in areas with a high incidence of gastric cancer. The following flowchart illustrates the implementation process of each module.

FIG2 is an embodiment of the artificial intelligence lesion-assisted medical diagnosis process 200 of the present invention. This embodiment is based on the medical-assisted diagnosis system 100 architecture of FIG1 and is implemented by the sampling system 110, the user device 120 and the image analysis system 130.

In step 201, a first dataset with an average incidence and a second dataset with a high incidence are provided in the image server 114. In one embodiment, the first dataset includes endoscopic images and histological test results of patients with atrophic gastritis and intestinal metaplasia provided by a large medical center. The second dataset generally refers to endoscopic images and histological test results of patients with atrophic gastritis and intestinal metaplasia from a specific community hospital with a high incidence. In remote rural or outlying island areas, such as Matsu, due to the lifestyle and medical resource gap, the risk difference of residents suffering from gastric disease is statistically significant, and is suitable as a source of the second dataset. The gastric sinus and gastric body image data in the first and second datasets can be collected by medical personnel from various places using image acquisition devices 112 in accordance with the modified Sydney scheme, and have standard format files stored in the image server 114.

In a further embodiment, the image analysis system 130 may also include an image database (not shown) for centrally storing the first data set collected by a large medical center and the second data set collected by regional hospitals across the country. The implementation of the image database is not limited to a local storage device, but can be a public cloud solution. In other words, the embodiment of the present invention does not limit the number and location of the image server 114. The data transmission between the components can be implemented in accordance with known network protocols and will not be repeated in this embodiment.

Since the data sources and methods of obtaining data from sampling systems 110 across the country may vary greatly, the original images in the data need to be pre-processed before the deep learning stage to remove unnecessary noise, such as patient information, timestamps, or watermarks. In addition, pre-processing also helps to make the information learned by the subsequent training module 136 more consistent.

In step 203, image preprocessing is performed on the first data set and the second data set. The preprocessing step can be preprocessed at the sampling system 110 end, or processed by the preprocessing module 134 in the image analysis system 130. When the first data and the second data are sent to the preprocessing module 134 to perform the preprocessing step, the preprocessing module 134 normalizes the first data and the second data and converts them into normalized images. These normalized images are then transmitted to the training module 136 for training. The detailed image preprocessing steps will be described in detail in the flow chart of Figure 3.

In step 205, the first data set and the second data set are converted into normalized images by the preprocessing module 134 and input into the training module 136 for deep learning, and then the inference model is established.

When the training module 136 is running, the first data set and the second data set can be divided into a training data set, a validation data set, and a test data set. The training module 136 uses the cutoff value determined during the training and validation process for training, and learns to classify the normalized image according to the proportion of small blocks diagnosed as specific conditions. The training module 136 then evaluates the performance of the learned model on the test data set. In this embodiment, in order to accelerate the maturity of the model, the training module 136 adopts the transfer learning method to make good use of the existing knowledge in the known model by fine-tuning the known model.

The embodiments of the present invention develop a first model 131, a second model 132, and a third model 133 by fine-tuning known models, such as ResNet and densely connected networks (DenseNet), and overcoming the difficulty of integration. ResNet neural network is a technology released in 2016. It can reversely propagate gradients during training through "short-circuit connections" between the front layer and the back layer, thereby training a deeper convolutional neural network. The DenseNet model was released in 2018 and is a further improvement of the ResNet and the original DenseNet neural network architecture. The feature of DenseNet is that each layer in its block will be stacked with the feature maps of all previous layers in the dimension of the channel, and then used as the input of the next layer. Therefore, an N-layer convolutional neural network has N connections, which can enhance feature propagation and achieve feature reusability, while reducing model parameters and computational complexity, and improving training efficiency.

The first model 131, the second model 132 and the third model 133 in this embodiment are all neural network models trained by the training module 136 based on the first data set and the second data set.

The first model 131 is trained to identify gastric locations and non-gastric locations in gastric endoscopic images. The second model 132 is trained to further classify gastric locations into gastric body, gastric sinus, and pylorus. The third model 133 can further determine whether atrophic gastritis and precancerous lesions of intestinal metaplasia occur based on the gastric images of the corresponding parts provided by the second model 132, and predict their severity.

The model templates used by the training module 136 to train the first model 131, the second model 132, and the third model 133 may include derivative versions of various well-known neural network architectures, such as EfficientNet-b0, EfficiientNet-b4, EfficientNet-b6, AlexNet, VGG11, VGG19, ResNet18, ResNet152, DenseNet121, Vision Transformer, and DenseNet201. When the training is finally completed, the inference model with the best discriminative ability is selected and deployed to the image analysis system 130 for external services.

In the embodiment of training 131 and the second model 132, the image analysis system 130 can use data from a large medical center, such as the National Taiwan University Hospital, randomly shuffle the data by patient and use 80% for training and 20% for verification. In addition, in terms of testing, data from hospitals in high-risk areas, such as Matsu data, can be used to evaluate the accuracy and stability of the first model 131 or the second model 132.

In order to train the third model 133 to be a model that can predict the severity of pathology (atrophic gastritis and intestinal metaplasia), the data provided to the training module 136 must first exclude patients with missing pathological section results and select image data of the gastric sinus and gastric body as input values. The work of selecting the gastric area can be performed using the first model 131 and the second model 132 that have been trained in the above embodiment.

When training the third model 133, whether predicting the severity of atrophic gastritis or intestinal metaplasia, the allocation ratio of the training set, validation set, and test set can be flexibly set and dynamically adjusted according to the execution performance. For example, in the first training method, the training module 136 can select a remote hospital or a high-risk data set, such as Matsu data, from the image server 114 as input data. And 70% of the input data is allocated as a training set, 20% is allocated as a test set, and 10% is allocated as a validation set. In the second training method, the National Taiwan University Hospital data is further added for mixed training. In the implementation of mixed training, the data of large medical centers, such as National Taiwan University Hospital, can be allocated to the training set and the validation set in a ratio of 8:2. Alternatively, all the data of National Taiwan University Hospital can be allocated to the training set, and 10% of the data from Matsu can be selected and allocated to the validation set. Alternatively, all the data of National Taiwan University Hospital can be mixed with 70% of the data from Matsu and then allocated to the training set, while 10% of the data from Matsu can be allocated to the validation set. Experiments show that when the data from two different sources are mixed in the training set, the trained model has better performance and higher generalization ability.

In this embodiment, in order to train the first model 131 to distinguish between stomach and non-stomach images, the training module 136 can read a large number of original gastroscopic images from the image server 114 as an input data set, and use the convolutional neural network to learn the characteristics of the stomach position. The training module 136 can allocate 70% of the input data set as the training set of the model, 20% as the validation set, and 10% as the test set. Experimental tests show that when the first model 131 uses DenseNet201 as a model template, the training results have better performance. DenseNet201 is a deep neural network architecture, which is part of the DenseNet series. It is composed of densely connected blocks (Dense Blocks) and has a very deep network structure with a total of 201 layers. DenseNet201 is trained using the known ImageNet dataset and is mainly used for image classification tasks. DenseNet201 has stronger feature extraction capabilities than shallower networks. Its core idea is dense connection, that is, the input of each layer is connected to the output of all previous layers. This design allows information to be transmitted and utilized more fully, effectively alleviates the gradient vanishing problem, promotes the reuse of features, and thus improves the efficiency and performance of the network. The network structure of DenseNet201 is relatively complex, including multiple densely connected blocks and transition layers. The layers in the densely connected blocks share and transmit information through direct connections, while the transition layers are used to adjust the size and number of feature maps to help the network adapt to different input sizes and complexities. DenseNet201 performs well in many computer vision tasks, such as image classification, object detection, and semantic segmentation. It has excellent performance in extracting image features, is widely used in various practical applications, and has achieved remarkable results in many competitions and academic research.

The second model 132 is a model that can classify the gastric area, so that the inference module 138 can distinguish the gastric image into the gastric sinus, gastric body, and gastric fundus after execution. In order to train the second model 132 to achieve the above functions, the training module 136 can read a large number of original images collected by large medical centers and community hospitals from the image server 114. Among these original images, 80% of the images from large medical centers are allocated as training sets and 20% are allocated as validation sets. 10% of the original images from community hospitals are allocated as test sets. The training set and validation set use data obtained from large medical centers, in which the incidence rate tends to the national average. The test set mainly uses data obtained from specific community hospitals, which are known to have an incidence rate higher than the national average. Experimental results show that the training results of the DenseNet121 model can achieve better performance, so DenseNet121 was selected to deploy the second model 132. Similar to DenseNet201, DenseNet121 is also part of the DenseNet series, consisting of multiple densely connected blocks, with a total of 121 layers.

The third model 133 is a model that can predict and identify precancerous gastric diseases. In the process of training the third model 133 by the training module 136 of this embodiment, all the data of large medical centers and 70% of the data of community hospitals are used. In the verification process of the third model 133, 10% of the data of community hospitals are used. In the testing process of the third model 133, 20% of the data of community hospitals are used. For example, the training module 136 can only use the Mazu dataset to train the third model 133, and try to apply multiple model templates, including 11 convolutional neural network models such as EfficientNet-b0, EfficientNet-b4, EfficientNet-b6, AlexNet, VGG11, VGG19, ResNet18, ResNet50, ResNet152, DenseNet121, DenseNet201, and Vision Transformer (VIT) and other architecture models for training. The training results of the above-mentioned various model templates were evaluated based on the Micro-AUC indicator. It can be seen that the visual converter model is about 13-37% more effective than various convolutional neural network models. Therefore, the third model 133 can use the visual converter model to diagnose, verify, and test precancerous lesions of atrophic gastritis and intestinal metaplasia.

The results of each model trained by the training module 136 can be used to determine whether to deploy it to the image analysis system 130 through a performance evaluation program. When evaluating the ability of the trained model, the present embodiment can use sensitivity, specificity and area under the receiver operating characteristic (AUROC) curve. In the baseline characteristics of the subjects, the classification data is expressed as a percentage (%), and the continuous data is expressed as a mean (standard deviation). These indicators can verify whether various inference models can correctly identify positive and negative cases and judge overall unbalanced data sets. In the embodiment of the present invention, the training module 136 may use 95% confidence intervals (CIs) to evaluate the statistical significance of the trained first model 131, the second model 132, and the third model 133. In the process of the training module 136 training the first model 131, the second model 132, and the third model 133, multiple model templates may be trained simultaneously, and finally, based on the AUROCs shown by each model template, the model with the highest discriminative ability is selected and deployed in the image analysis system 130 as the inference model for actual execution application. In other words, the first model 131, the second model 132, and the third model 133 deployed in the image analysis system 130 are all selected by the training module 136 through AUROC after training based on various model templates and input data.

The third model 133 trained in this embodiment can verify the prediction value in a real environment. For example, researchers can compare the results of endoscopic examination and histological evaluation of patients with positive pepsinogen test results with the analysis results of the medical auxiliary diagnosis system 100. The actual test confirmed that the medical auxiliary diagnosis system 100 of this embodiment can correctly identify 96.2% of the images of the anatomical location and 97.5% of the images of the gastric area. In terms of histological prediction, the PPV and NPV of diagnosing gastric precancerous lesions are 0.583 (95% CI: 0.468-0.690) and 0.881 (0.815-0.925), respectively.

In a further embodiment, in order to improve the recognition performance of the third model 133, before training, the pre-processing module 134 may also perform a contrast limited adaptive histogram equalization (CLAHE) operation on the input image data in advance to enhance image details and highlight gastric mucosal features. CLAHE is an image processing technique that aims to enhance the contrast of an image. It is a variant of histogram equalization (HE) and is used to solve the problem of excessive noise enhancement that may be caused by HE when local contrast enhancement is performed. The working principle of CLAHE is to divide the image into many local regions and then apply HE to each region, thereby achieving local contrast enhancement in the entire image. However, unlike standard HE, CLAHE limits the contrast enhancement in local areas to avoid over-enhancement. This is achieved by cropping and redistributing the histogram of pixels in the local area. An important parameter of CLAHE is "contrast limit", which is used to control the degree of equalization of pixel brightness values in the local area. The larger the contrast limit, the more obvious the contrast enhancement effect, but it may lead to over-enhancement. Therefore, appropriately adjusting the contrast limit can achieve the best CLAHE effect.

The trained first model 131 can determine the location of the stomach in the gastric endoscopic image, and the second model 132 can classify the gastric fundus, gastric sinus, and gastric body. The third model 133 can predict the severity of the pathology (atrophic gastritis and intestinal metaplasia) based on the image identified by the second model 132. In a further derived embodiment, through different input images and parameter adjustments, the second model 132 can also be trained to classify the hypopharynx, esophagus, and duodenum, and the third model 133 can also be further trained to detect Helicobacter pylori in the gastric fundus, gastric body, and pylorus.

In step 207, the inference module 138 receives the sample data and performs artificial intelligence inference using the inference model established in step 205. When the medical auxiliary diagnosis system 100 is open to the outside world, the user of the user device 120 can upload the sample data of a new patient via the cloud at any time for the inference module 138 in the image analysis system 130 to perform artificial intelligence inference. The inference module 138 will use the knowledge learned by the first model 131, the second model 132, and the third model 133 to effectively distinguish various features in the sample data in sequence, and finally determine whether there is a risk of gastric cancer. In practice, the hardware and operating system architecture used by the inference module 138 can be the same as or shared with the training module 136, and the first model 131, the second model 132, and the third model 133 can be loaded to execute the corresponding artificial intelligence inference service. The further detailed steps of the inference module 138 for inference will be described in detail in FIG. 4.

In step 209, the image analysis system 130 receives histological samples with known test results to improve image preprocessing parameters. In other words, when the user uses the medical-assisted diagnosis system 100 in a real environment, the user can actively provide the image analysis system 130 with known test results to help verify or improve the model performance. The inference module 138 of the medical-assisted diagnosis system 100 achieves a sensitivity and specificity of about 80% when diagnosing precancerous lesions.

When the image analysis system 130 receives these histological samples with known test results, it can also perform step 203 to pre-process the input histological sample images, and then the inference module 138 executes the inference program. By applying the analysis results of the first model 131, the second model 132 and the third model 133 to these positive individuals, the PPV and NPV performance of the first model 131, the second model 132, or the third model 133, and their corresponding 95% confidence intervals can be verified.

FIG3 is a schematic diagram of an image preprocessing process 300 performed by the preprocessing module 134 of the present invention. This embodiment further illustrates the decomposition steps of step 202. The image format transmitted from the sampling system 110 or the user device 120 to the image analysis system 130 may have different resolutions and information formats due to different types and brands of instruments. For example, traditional gastroendoscopic images have a variety of size specifications, which may include various aspect ratios and resolutions such as 720*480, 640*480, or 512*384, and there are often noises such as timestamps or watermarks in the images. Before the deep learning stage is performed, the original image needs to be preprocessed to solve the problem of inconsistent image specifications or noise interference. The preprocessing procedure helps the training module 136 to process only mucosal appearance images containing key information when performing deep learning.

In step 301, the pre-processing module 134 converts the input image received from the sampling system 110 or the user device 120 into a grayscale image. The input image is generally a color endoscopic image, which is first converted into a grayscale image in this step for subsequent processing.

In step 303, the grayscale image is converted into a binary map using the Otsu's method threshold. Otsu's method is an image processing technique used to automatically determine the binarization threshold of an image. First, the grayscale histogram of the image is analyzed to find a threshold value to divide the image into two categories (foreground and background) so that the intra-class variance between the two categories is minimized and the inter-class variance is maximized under the threshold value. The minimum intra-class variance is the minimum variance of the same category. The maximum inter-class variance is the maximum variance of different categories. In other words, the threshold value calculated by the Otsu's method can make the difference between different categories in the image as large as possible and the difference between the same category as small as possible.

When the pre-processing module 134 calculates the threshold using the Otsu method, the grayscale histogram of the image is first calculated to count the number of pixels at each grayscale level. Then, for each possible threshold (i.e., 0 to 255), the image is divided into two categories, and the number of pixels and the average grayscale value of each category are calculated. For each possible threshold, the intra-class variance and the inter-class variance are calculated respectively. Finally, among all the thresholds, the threshold with the largest inter-class variance is selected as the optimal threshold and used for image binarization. The Otsu method is commonly used in fields such as image segmentation, edge detection, and target recognition, especially when a large number of images need to be processed automatically. It is a simple but effective image processing technique that can help improve the accuracy and efficiency of image processing.

In step 305, the pre-processing module 134 performs edge detection on the binary map to identify the target area in the input image. Based on the edges of the objects in the binary map, the pre-processing module 134 can identify the area with the largest area (determined by the maximum and minimum coordinates of the x-axis and y-axis).

In step 307, the pre-processing module 134 crops the input image according to the recognition result of step 305, and only retains the target area image. In other words, the range of the input image that does not belong to the target area is cropped and discarded, leaving an image file with a specific aspect ratio. In one embodiment, the aspect ratio of the cropped image file is set to one to one, that is, a square image.

In step 309, the pre-processing module 134 normalizes the cropped image size of the target area to a preset dimension. For example, the pre-processing module 134 may resize the cropped image to a square pixel size of 256×256 or 128×128 through an image scaling algorithm. These pre-processing steps ensure that the image is properly prepared for subsequent deep learning training.

FIG4 is an artificial intelligence pathological image estimation process 400 of one embodiment of the present invention.

In step 401, the inference module 138 removes non-stomach images from the pre-processed images using the first model 131. In this embodiment, the first model 131 is the DenseNet201 model trained in step 205 of FIG. 2.

In step 403, the estimation module 138 uses the second model 132 to perform regional classification on the stomach image.

In step 405, the estimation module 138 uses the third model 133 to perform histological prediction on each regional stomach image.

In step 407, a heat map is superimposed on the stomach image according to the prediction result. In order to have a deeper understanding of the inference process of the first model 131, the second model 132, and the third model 133, the present embodiment also uses the gradient-weighted class activation mapping (Grad-CAM) operation to generate a heat map to visualize the important regions in the input image. Grad-CAM is a visual presentation method for deep learning model predictions, which explains the prediction basis of the model for a specific category by visualizing the importance of each feature map in the neural network. Grad-CAM uses the gradient information in the back propagation to calculate the importance of each feature map to the final prediction of the model. These gradients can reflect the contribution of the activation degree of each feature map to a specific category during the prediction process. Grad-CAM operation can be achieved by the pre-processing module 134 in the image analysis system 130, or it can be processed by the inference module 138. For example, when the inference module 138 makes inferences, it can generate a class activation map based on the prediction results and gradient information of a specific category to show which areas in the input image are important for the model's final prediction. These areas are usually associated with the areas that the model focuses on when predicting a specific category. By superimposing the class activation map with the original image, an additional color bias visual effect is generated on the input image, which can be used to emphasize the image areas that deserve attention. This helps to understand the inference process of the model and to determine whether the artificial intelligence recognition effect is consistent with the experience of expert scholars.

Prior art for diagnosing precancerous gastric diseases in high-risk populations often relies too heavily on large, homogeneous data sources, such as images screened from large hospitals with abundant resources. The medical-assisted diagnosis system 100 proposed in the present invention not only integrates an artificial intelligence model to simulate the medical judgment thinking process, but also incorporates medical data from resource-limited areas into the analysis. The implementation method of the present invention brings several advantages. For example, the quality of traditional endoscopic images, such as proximity and blur, affects the step-by-step process of image classification. An embodiment of the present invention provides a heat map overlay effect that can assist endoscopists in judging the inferred results produced by the medical-assisted diagnosis system 100. In traditional practices, sampling angle errors of endoscopic images will affect the discrimination ability of the artificial intelligence system. In addition to following the histological standard sampling procedure, the present invention also generates normalized image data through an image preprocessing procedure to effectively reduce error variables. When using the third model 133 to infer precancerous gastric diseases, the embodiment of the present invention also enhances the details of the mucosal image, effectively assisting the endoscopic image analysis work. In addition, the artificial intelligence prediction model of this embodiment takes precancerous conditions into consideration when performing inference analysis on the sample. After the actual implementation results were tested, it was found that the overall morbidity had a higher negative prediction value. This means that the medical auxiliary diagnosis system 100 of the present invention can effectively reduce the requirement for endoscopic monitoring of low-risk gastric cancer patients, saving unnecessary medical costs.

The medical-assisted diagnosis system 100 of the present invention is a comprehensive solution, and its technical features cover data collection, model development, and field implementation. Compared with labeling only during the model development process, the medical-assisted diagnosis system 100 can improve the recognition accuracy in implementation based on the feedback of the doctor's clinical experience. The medical-assisted diagnosis system 100 also includes the concept of end-to-end services, which can extract information from endoscopic images stored in the image server 114. The image analysis system 130 can flexibly use image diagnosis or detection related artificial intelligence algorithms. The medical auxiliary diagnosis system 100 of the present invention uses data in a particular way. Traditional training data mainly comes from large medical centers. When researchers classify anatomical locations and gastric regions, the training data provided by community hospitals can be used universally. However, when performing histological grading, it is necessary to have a high sensitivity to subtle mucosal changes in the sample. Technical parameter or image quality errors in image data from different sources can easily cause identification errors.

In order to solve the above-mentioned technical bottlenecks, the embodiment of the present invention incorporates data from community hospitals in the training process, and uses a model with a self-attention mechanism to capture the relationship between patches, thereby making the prediction results more reliable. The medical-assisted diagnosis system 100 of the present invention uses federated learning between areas of different data quality and distribution to enhance the generalization ability of the model. Federated learning is a distributed training method for machine learning that aims to achieve centralized training of models while protecting data privacy. Compared with traditional centralized training, federated learning can push the training process to the local device where the data is located, instead of collecting the data on a centralized server. In federated learning, each participant trains a local model locally, using local data for training. Then, updates to the local model are anonymously aggregated on the central server to form a global model. The central server distributes updates to the global model to all participants, and repeats this process until the model converges or a stopping condition is reached. One of the main advantages of federated learning is that it protects the user's data privacy, because the data is always kept locally and does not need to be transmitted to the central server. This approach helps avoid the risk of data leakage and privacy violations. At the same time, federated learning also helps solve the problem of data dispersion and imbalance, because each participant can use their own local data for training without the need to centralize the data in one place.

The medical auxiliary diagnosis system 100 of the present invention also provides a complete histological prediction for each gastric fundus and gastric body image, which exceeds the requirements of the traditional gastric mucosal lesion scoring system (Operative Link for Gastritis Assessment; OLGA) or gastric mucosal glandular metaplasia scoring system (Operative Link for Gastritis Assessment of Intestinal Metaplasia; OLGIM). The AI-generated information generated by this system can be used as material for further long-term research to continuously enhance the ability to predict gastric cancer risk.

In summary, the medical auxiliary diagnosis system 100 of the present invention can be used as a valuable deep learning service system in remote medical care, especially for diagnosing precancerous gastric conditions and Helicobacter pylori infection in areas with limited resources. The system improves the accessibility of medical services and enables limited resources to be optimally allocated to those who really need procedures.

FIG5 is a second model 132 of an embodiment of the present invention. In practice, the second model 132 is modified based on the DenseNet121 neural network architecture. The input image $IN is processed by the feature extraction module 501 and the classification module 502 to generate the output result $OUT.

The main function of the feature extraction module 501 is to extract meaningful features from the input image $IN. In this embodiment, the feature extraction module 501 generates a high-level summary that can represent the input image $IN through multiple layers of neural network iterations for classification by the classification module 502. In this embodiment, the feature extraction module 501 includes a plurality of operation modules, which are connected in sequence according to the data flow order of processing the input image $IN, including a convolution module 510, a maximum pooling layer 512, a first dense block 514, a convolution module 520, an average pooling layer 522, a second dense block 524, a convolution module 530, an average pooling layer 532, a third dense block 534, a convolution module 540, an average pooling layer 542, and a fourth dense block 544.

Convolution module 510, convolution module 520, convolution module 530, and convolution module 540 generally refer to convolution layers in a neural network, which can perform convolution operations to extract key features from input data. The convolution operation is to slide a small filter (also called a convolution kernel or filter) on the input data, calculate the dot product between the filter and the input data, and generate an output feature map. The parameters of the filter can be updated and optimized through iterative training. By performing multiple iterative convolution operations on the input image $IN, the feature extraction module 501 can extract various features from the data, such as edges, textures, shapes, etc., which are the basis for the classification module 502 to perform subsequent classification.

The visual converter 500 of this embodiment is an improved version of the DenseNet architecture. The calculation result of each convolution module is processed by pooling and dense blocks before entering the next convolution module. For example, the output of the convolution module 510 is processed by the maximum pooling layer 512 and the first dense block 514 before being passed to the convolution module 520. And so on.

As shown in FIG5 , the maximum pooling layer 512 is coupled to the output of the convolution module 510 and is configured to reduce the spatial size of the feature map output by the convolution module 510 while retaining important features. In other words, the maximum value in each region of the feature map is retained, while other values are discarded, thereby achieving compression of the feature map. The convolution module 510 helps reduce the number of parameters and the amount of computation of the model, while improving the position invariance of the features, so that the second model 132 has better robustness to small changes in the input image.

The first dense block 514 is coupled to the output end of the maximum pooling layer 512, and can be directly connected to the subsequent convolution module 520, the convolution module 530, and the convolution module 540, and receives the feature map fed back by the subsequent convolution module 520, the convolution module 530 (connection line is not shown), and the convolution module 540 (connection line is not shown). The first dense block 514 concatenates the feature map output by the convolution module 510 through the maximum pooling layer 512 and the feature map fed back by the convolution module 520, the convolution module 530, and the convolution module 540 in the channel dimension to form the input value of the convolution module 520. The setting of dense blocks is the core feature of DenseNet, which is used to promote information flow and feature reuse. Since the output of each layer of convolutional modules is connected in series to the input of all previous convolutional modules, the expressive power of features can be enhanced, the gradient vanishing problem can be alleviated, and the model can be easier to train and optimize.

The average pooling layer 522 is coupled to the output end of the convolution module 520 and is used to reduce the spatial size of the feature map, thereby reducing the number of parameters and the amount of calculation of the model. The average pooling layer 522 averages the pixel values in the input area and reduces the spatial size of the feature map, thereby reducing the number of parameters and the amount of calculation of the model. This helps to alleviate the overfitting problem of the model and improve the computational efficiency of the model. Unlike the maximum pooling layer 512, the average pooling layer 522 retains the information of all pixel values in the input area, not just the maximum value. This means that the average pooling layer 522 pays more attention to the average expression of the overall features rather than the extraction of the most significant local features. Furthermore, the average pooling layer 522 can smooth the local changes in the feature map and reduce the impact of noise on the feature map.

The second dense block 524 is coupled to the output end of the average pooling layer 522, and can be directly connected to the subsequent convolution module 530 and the convolution module 540, and receive the feature map fed back by the subsequent convolution module 530 (connection line not shown) and the convolution module 540 (connection line not shown). The operation of the subsequent average pooling layer 532 and the average pooling layer 542 in Figure 5 is the same as that of the average pooling layer 522, the operation of the second dense block 524, the third dense block 534, and the fourth dense block 544 is the same as that of the first dense block 514, and the operation of the convolution module 530 and the convolution module 540 is the same as that of the convolution module 520, which will not be repeated. After iterative calculations of the above components, the feature extraction module 501 generates a feature map of #IN and transmits it to the classification module 502.

In summary, the feature extraction module 501 of FIG5 implements a four-layer convolutional neural network for feature extraction, but does not implement a fully connected layer. The hyperparameters used in the classification module 502 can be weighed by the hyperband algorithm (Hyperband) to find the best hyperparameters as the setting basis for the subsequent classification module 502. Hyperband is an efficient algorithm for optimizing hyperparameters. It combines random search and resource allocation strategies to effectively find the best hyperparameter combination with different combinations under limited computing resources.

In this embodiment, the input image $IN can be a gastric endoscopic image provided by the image server 114, such as the first data and the second data described in FIG2. These images are pre-processed by the pre-processing module 134 and then transmitted to the feature extraction module 501 for training. The initial weights required for the feature extraction module 501 to perform convolutional neural network operations can be weights pre-trained by the known image network ImageNet model. The weight values will be gradually fine-tuned as the feature extraction module 501 learns the image.

The main function of the classification module 502 is to classify the input image $IN. The classification module 502 can calculate the probability that the input image $IN belongs to each category based on the feature map generated by the feature extraction module 501. In deep learning, the classification module 502 usually sets a fully connected layer after the feature extraction module 501 to implement the classification function. The classification module 502 of this embodiment includes a global pooling layer 550, a first fully connected layer 552 and a second fully connected layer 554.

The global pooling layer 550 (Global Pooling) is a pooling operation that can reduce the spatial size of the feature map provided by the feature extraction module 501 to a fixed-size vector. Unlike local pooling, global pooling does not consider the region, but directly acts on the entire feature map. In one embodiment, the global pooling layer 550 performs a global average pooling operation, which performs average pooling on each channel in the entire feature map to obtain the average value of each channel. The final vector represents the average feature of the entire feature map. The global pooling layer 550 integrates the information of the entire feature map into a fixed-size vector, which can improve the computational efficiency and generalization ability of the model.

The global pooling layer 550 is connected to the first fully connected layer 552 and the second fully connected layer 554. The first fully connected layer 552 is a 64-layer fully connected layer, and the second fully connected layer 554 is a 480-layer fully connected layer. A fully connected layer, also known as a densely connected layer, is a common structure in a deep learning model. The first fully connected layer 552 and the second fully connected layer 554 are located at the end of the visual converter 500 and are used to integrate and combine the features of the previous layers to output the final prediction result.

In the first fully connected layer 552 and the second fully connected layer 554, each neuron is connected to all neurons in the previous layer, and each connection has a weight, so that all information of the previous layer can be transmitted to the fully connected layer. Each neuron in the first fully connected layer 552 and the second fully connected layer 554 can learn different aspects or features of the input data, so that the model can learn more complex nonlinear relationships and model the input data more accurately. Since each neuron in the first fully connected layer 552 and the second fully connected layer 554 is connected to all neurons in the previous layer, the number of parameters is large and overfitting is prone to occur. The embodiment of the present invention uses two fully connected layers of different orders to be connected in series to appropriately adjust the calculation amount of the classification module 502.

In one embodiment, the classification module 502 may use the SoftMax activation function to output the probability values of the three stomach positions. The SoftMax activation function can map a vector with arbitrary real values to a probability distribution, so it is very suitable for solving classification problems. For example, the output results of the first fully connected layer 552 and the second fully connected layer 554 are usually matrix vectors. After being converted by the Softmax activation function, the probability distribution of multiple answers can be displayed.

In practice, the formula for the SoftMax activation function is as follows:

Among them, the denominator is the sum of the exponential function values of all elements _X1 to _XK in a K-dimensional real vector, and the numerator is the natural number exponential value of the i-th element Xi. After division, the probability of occurrence of the i-th element Xi is obtained. The function of the SoftMax activation function is to convert the input K-dimensional vector into a probability distribution. After each element Xi is exponentialized with a natural number, it can be normalized to a probability value. The output value of the SoftMax activation function ranges from 0 to 1, and the sum of all outputs is 1, so it can be interpreted as a probability distribution. If each element represents a category, the SoftMax activation function can be applied to the probability prediction of multi-classification problems.

The visual converter 500 of this embodiment can adopt transfer training method (Transfer Learning), using known model templates with fine-tuning parameters to accelerate model training. In this embodiment, the hyperparameters used to train the visual converter 500 are as follows. The batch size (Batch size) is set to 32. The maximum number of iterations (Epoch) is set to 50. The loss function (Loss function) uses Categorical cross-entropy for multi-category classification tasks. The optimizer (Optimizer) uses Adam. The initial value of the learning rate (Learning rate) is set to 0.001, in which the callback function (ReduceLROnPlateau) is used and the patience (Patience) is set to 3, and the step factor (Step Factor) is set to 0.2, thereby observing the validation loss (Validation loss).

The callback function ReduceLROnPlateau is used to dynamically adjust the learning rate according to the performance on the validation set to help the optimization algorithm converge faster or avoid falling into a local minimum. Validation loss is a performance indicator used to evaluate the difference between the model's prediction results on the validation set and the true label. Validation loss can be calculated using the cross entropy loss function. A lower validation loss means that the model's prediction on the validation set is closer to the actual situation, that is, the model is more accurate.

FIG6 is an embodiment of the visual converter model 600 of the present invention, which is used to illustrate the training process of the third model 133.

In 2017, the Google team proposed a transformer neural network with a self-attention mechanism, which enabled the model to be trained in parallel and obtain global information, becoming a classic model for natural language processing. The Vision Transformer (ViT) was born in 2021. Based on the transformer network, it further broke through the barrier between natural language processing and computer vision. The Vision Transformer is suitable for image classification and other computer vision tasks. ViT also uses a self-attention mechanism to capture global and local information in the image, so the operation is not limited by the size of the convolution kernel. The basic structure of the ViT model is a stack of multiple transformer modules, each of which consists of multiple attention heads. Each attention head compares each position in the input feature map with all other positions and assigns a weight to each position based on their relevance. In this way, the model can learn the relationship between different positions in the image, thereby effectively capturing global and local information. The ViT model has a simple structure and is easy to expand to different image sizes and tasks. In addition, in order to process the position information in the image, ViT also introduces position encoding technology to embed the coordinate information of each position into the feature representation. Because the ViT model processes information of different scales simultaneously through multiple attention heads, the generalization ability of the model can be improved. The ViT model has excellent performance in drawing vision tasks, including image classification, object detection, semantic segmentation, etc. This embodiment applies the self-attention mechanism to the third model 133, thereby effectively identifying whether there are precancerous lesions of atrophic gastritis and intestinal metaplasia in the input image.

Compared to convolutional neural networks (CNNs), Vit generally lacks translational relevance and locality information, and requires large-scale data sets for transfer learning to achieve considerable results. To solve this problem, the embodiment of the present invention integrates shifted patch tokenization and locality self-attention mechanisms into the traditional Vit process to form a visual converter model 600 as shown in FIG6 .

Figure 6 decomposes the visual converter model into multiple processing procedures and explains them in layers. It can be understood that the contents of each layer are essentially logical execution steps and do not limit the hardware or software implementation method.

In FIG. 6 , the translation patch layer 610 receives the input image 602 and shifts the input image 602 along the four diagonal lines (upper left, upper right, lower left, and lower right) to form four shifted images. The translation patch layer 610 then stacks the four shifted images with the input image 602 to form a translation stacked image.

The image cutting layer 620 follows the output of the translation patch layer 610, cutting the above translation stack into multiple patches and flattening it into a one-dimensional sequence. Each patch represents a local area in the input image 602. The image cutting layer 620 may also include linear projection and normalization operations, so that each patch corresponds to a feature vector of a fixed length, similar to a token in natural language processing.

After receiving multiple patches output by the image segmentation layer 620, the position embedding layer 630 adds position information to each patch. In practice, the position embedding layer 630 can add the position code of each patch to the feature vector of the patch.

The multi-layer encoder 640 is connected to the output end of the position embedding layer 630 to extract features from the input multiple patches. The multi-layer encoder 640 may include a multi-head attention module to form a multi-layer iterative architecture. By continuously performing self-attention operations and normalization operations, the important features of each patch are gradually learned and extracted. One or more multi-head attention modules in the multi-layer encoder 640 can perform local self-attention operations. The local self-attention operation only calculates the correlation between each patch and its neighboring patches, and does not process non-neighboring patches. In this way, the multi-layer encoder 640 can prioritize the local important areas with greater correlation, so that the model can more effectively capture the feature vector matrix in the image.

The multi-layer perception head 650 is connected to the output of the multi-layer encoder 640. As the last processing stage of the visual converter model 600, the feature vector matrix generated by the multi-layer encoder 640 after processing the patch can be pooled to obtain a probability distribution matrix 604 that can represent the entire input image 602. In the visual converter model 600 of this embodiment, the multi-layer perception head 650 can be one or more fully connected layers located at the top of the deep learning model, which is used to convert the feature representation learned by the model into the final task prediction or classification result. The design of the multi-layer sensor head 650 depends on the specific task and model structure, and can usually be customized by adjusting parameters such as the number of layers, number of neurons, and activation functions.

In the implementation of the visual converter model 600, the open source dataset of colorectal cancer tissue slice images (5000, 150, 150, 3) can be used to pre-train the model for 20 training cycles to obtain the initial weight vector matrix. Then, the initial weight vector matrix is input into the visual converter model 600 for further fine-tuning by using the learning transfer method. The number of fully connected layers of the multi-layer perception head 650 can be adjusted according to the number of categories to be classified. In this embodiment, the disease predicted by the visual converter model 600 is in a multi-label form, so the multi-layer perception head 650 can use the Sigmoid activation function to classify and output the severity of the pathology (severity 0, severity 1, severity 2).

The function formula of Sigmoid is as follows:

Where x represents the input value. The Sigmoid function is often used in the output layer of binary classification problems. Because it maps the input value to between 0 and 1, the output can be interpreted as a probability value, indicating the probability of the positive class. In the gradient descent algorithm, the derivative calculation of the Sigmoid function is relatively simple, which makes it more efficient when calculating the gradient in the back propagation algorithm. In another embodiment, the multi-layer perception head 650 can also use the ReLU (Rectified Linear Unit) activation function.

The parameters used by the visual converter model 600 are as follows. The batch size is set to 32 and the maximum training epoch is set to 50. The loss function is binary cross entropy, which is used for binary classification problems and minimizes the difference between the predicted probability and the actual label. In binary classification problems, the label value is usually 0 or 1, each corresponding to a category. When the model's prediction result is consistent with the actual label, the value of the loss function should be closer to 0. If the model's prediction result is inconsistent with the actual label, the value of the loss function will increase. The binary cross entropy loss function encourages the model to produce probability outputs close to the true labels during training, and provides good gradient information during backpropagation, which helps optimize model parameters.

The optimizer of this embodiment can use Adam, an adaptive learning rate optimization algorithm. The Adam optimizer combines the characteristics of momentum gradient descent (Momentum) and adaptive learning rate adjustment (Adaptive Learning Rate), and can effectively update model parameters during the training process.

The initial value of the learning rate in the visual converter model 600 is set to 0.0001, and the ReduceLROnPlateau function is used, the patience value is set to 3, and the coefficient Factor is set to 0.2. That is, when the evidence loss does not decrease for more than 3 training cycles, the learning rate is reduced according to the coefficient, so that the visual converter model 600 learns more details and information.

In addition, the visual converter model 600 also adopts an early stopping mechanism and gives a patience value of 6. During the training process, if the verification loss has not decreased after 6 training cycles, the visual converter model 600 stops training to avoid unnecessary subsequent training time.

In summary, the third model 133 of the present invention is trained using the embodiment of the visual converter model 600. When the inference module 138 performs diagnosis on the input image, the trained third model 133 is used to determine whether atrophic gastritis and precancerous lesions of intestinal metaplasia occur and predict their severity. The inference module 138 can also visualize the inference results by overlaying heat maps to facilitate the user to read the diagnosis results.

FIG7 is the image enhancement result of an embodiment of the present invention. The preprocessing module 134 performs CLAHE operation on the input image data to enhance the image details and highlight the gastric mucosal features. As shown in FIG7, the original gastric sinus image 710, the original gastric body image 720 and the original gastric fundus image 730 are processed by the preprocessing module 134 to become the enhanced gastric sinus image 712, the enhanced gastric body image 722 and the enhanced gastric fundus image 732. Since the contrast of the image is enhanced, it is more helpful for the subsequent prediction operation.

FIG8 is a heat map superposition result of an embodiment of the present invention. When the inference module 138 infers the original image 810, it can generate a heat map by using a gradient-weighted class activation mapping (Grad-CAM) operation. By superimposing the heat map with the original image, the original image 810 becomes a heat map superposition 820, in which the color deviation visual effect represents the area worthy of attention. Thus, the heat map superposition 820 helps to become a reference for clinical diagnosis.

The medical assisted diagnosis system 100 proposed in the present invention not only integrates an artificial intelligence model to simulate the medical judgment thinking process, but also incorporates medical data from resource-limited areas into the analysis. The implementation method of the present invention brings several advantages. For example, the quality of traditional endoscopic images, such as proximity and blur, will affect the step-by-step process of image classification. The embodiment of the present invention provides a heat map overlay effect, which can assist endoscopists in judging the inference results produced by the medical assisted diagnosis system 100. In traditional practices, the sampling angle error of endoscopic images will affect the judgment ability of the artificial intelligence system. In addition to following the standard histological sampling procedure, the present invention also generates normalized image data through an image preprocessing procedure to effectively reduce error variables. When using the third model 133 to infer precancerous gastric diseases, the embodiment of the present invention also enhances the details of the mucosal image, effectively assisting the endoscopic image analysis work. In addition, the artificial intelligence prediction model of this embodiment takes precancerous conditions into consideration when inferring and analyzing the sample. After the actual implementation results were tested, it was found that the overall prevalence had a higher negative prediction value. This means that the medical auxiliary diagnosis system 100 of the present invention can effectively reduce the requirements for endoscopic monitoring of low-risk gastric cancer patients, saving unnecessary medical costs

It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the phrase "includes a..." do not exclude the existence of other identical elements in the process, method, article or device including the element.

The above describes the embodiments of the present invention in combination with the drawings, but the present invention is not limited to the above specific embodiments. The above specific embodiments are only illustrative and not restrictive. Under the inspiration of the present invention, ordinary technical personnel in this field can make many forms without departing from the purpose of the present invention and the scope of protection of the patent application, all of which are within the scope of protection of the present invention.

100: Medical Assisted Diagnosis System

110: Sampling system

112: Image acquisition device

114: Image Server

120: User device

130: Image analysis system

131: First Model

132: Second Model

133: The third model

134: Preprocessing module

136: Training module

138: Inference module

Claims (14)

A medical auxiliary diagnosis system is used to provide medical auxiliary diagnosis services, comprising: a sampling system for collecting and storing medical images, wherein the sampling system stores a first data set and a second data set; a user device connected to the sampling system and configured to read the medical image from the sampling system or upload the medical image; and an image analysis system connected to the sampling system and the user device and configured to analyze the medical image according to the requirements of the user device. The invention relates to a method for generating an auxiliary image for assisting diagnosis, wherein: the image analysis system reads the first data set and the second data set from the sampling system to perform a deep learning; the image analysis system infers a test image provided by the image analysis system according to the result of the deep learning to generate an auxiliary diagnostic image for assisting in identifying precancerous lesions; the first data set comes from an unspecified region and has an average morbidity rate; the second data set comes from a specific region, the incidence rate is higher than the average incidence rate; and the first data set and the second data set include upper gastrointestinal endoscopic images; wherein the image analysis system comprises: a training module, used to perform the deep learning based on the first data set and the second data set; a first model, which is trained by the training module and can be used to identify the gastric area and the non-gastric area in a sample image; a second model, which is trained by the training module and can identify the gastric area and the non-gastric area in the sample image The stomach regions identified in the image are further classified into the gastric sinus, gastric body, and gastric fundus; and a third model, trained by the training module, can determine whether there is a precancerous lesion or Helicobacter pylori infection in the gastric sinus and gastric body in the sample image, wherein: the sample image is an upper gastrointestinal tract endoscopic image from the first data set or the second data set; and the precancerous lesion includes atrophic gastritis or intestinal metaplasia and Helicobacter pylori infection. The medical auxiliary diagnosis system as described in claim 1, wherein the image analysis system further comprises a preprocessing module configured to preprocess an input image to generate a normalized image, wherein: when the preprocessing module preprocesses the input image, the preprocessing module converts the input image into a grayscale image; the preprocessing module converts the grayscale image into a binary map using the Otsu threshold; the preprocessing module performs edge detection on the binary map To identify a target area in the input image; the pre-processing module crops the input image to obtain a cropped image that only retains the target area; and the pre-processing module adjusts the size of the cropped image to a preset dimension to generate the normalized image corresponding to the input image; wherein, before the training module performs the deep learning on the first data set and the second data set, the pre-processing module is used to normalize the images in the first data set and the second data set. A medical auxiliary diagnosis system as described in claim 2, wherein when the preprocessing module preprocesses the input image: the preprocessing module is also configured to perform a contrast limited adaptive histogram equalization (CLAHE) operation on the input image to enhance image details and highlight gastric mucosal features. The medical auxiliary diagnosis system as described in claim 2, wherein the image analysis system further comprises: an inference module, which is configured to use the first model, the second model, and the third model to infer the image to be tested to determine the risk of gastric cancer, wherein: the inference module uses the first model to identify the gastric area and the non-gastric area in the image to be tested; the inference module uses the second model to further classify the gastric area in the image to be tested into the gastric sinus, the gastric body, and the gastric fundus; the inference module uses the third model to determine whether the gastric sinus and the gastric body in the image to be tested have the precancerous lesions and Helicobacter pylori infection; and the inference module uses the gradient-weighted class activation mapping (Gradient-weighted Class Activation Mapping) Mapping; Grad-CAM) operation, superimposing a visual effect on the image to be tested according to the probability of disease, and generating the auxiliary diagnosis image; wherein the image to be tested is an upper gastrointestinal tract endoscopy image, which is collected and stored by the sampling system and provided to the image analysis system through the user device so that the inference module can analyze it; before the inference module analyzes the image to be tested, the pre-processing module normalizes the image to be tested. A medical auxiliary diagnosis system as described in claim 1, wherein: the first model is trained by fine-tuning parameters of the training module using the dense network DenseNet201 as a model template; and when the training module trains the first model, the first data set and the second data set are read from the sampling system, and 70% of them are allocated as a training set, 20% are allocated as a validation set, and 10% are allocated as a test set. The medical auxiliary diagnosis system as described in claim 1, wherein the second model is trained by fine-tuning parameters of the training module using the dense network DenseNet121 as a model template, and comprises: a feature extraction module, comprising a multi-layer neural network, configured to extract a feature map from input data through convolution operations; and a classification module, comprising one or more fully connected layers, coupled to the output end of the feature extraction module, configured to calculate the feature map generated by the feature extraction module. The probability value of the input data of the feature extraction module belonging to the gastric fundus, gastric sinus or gastric body; wherein, when the training module trains the second model, the first data set and the second data set are read from the sampling system, and 80% of the first data set is allocated as a training set, 20% is allocated as a validation set, and 10% of the second data set is allocated as a test set; and the classification module uses the SoftMax activation function to calculate the probability value of the input data of the feature extraction module belonging to the gastric fundus, gastric sinus or gastric body. The medical auxiliary diagnosis system as described in claim 1, wherein the third model is trained by fine-tuning parameters of a training module using a visual converter as a model template, and comprises: a translation patch layer, configured to translate an input image to four diagonal lines to generate four displacement images, and then stack them with the input image to form a translation stacking map; an image cutting layer, coupled to the translation patch layer, cutting the translation stacking map into multiple patches, flattening them into a one-dimensional sequence, wherein each patch corresponds to a feature vector of a fixed length; a position embedding layer, coupled to the image cutting layer, receiving multiple patches output by the image cutting layer, and adding position information to each patch; A multi-layer encoder coupled to the position embedding layer performs a self-attention operation on the input multiple patches to extract a feature vector matrix corresponding to the input image; and a multi-layer perception head coupled to the multi-layer encoder, including a multi-layer fully connected layer, configured to pool the feature vector matrix to obtain a probability distribution matrix that can represent the entire input image, wherein: the multi-layer encoder also performs a local self-attention operation for each patch and its neighboring patches, so that the feature vector matrix emphasizes local correlation; and the multi-layer perception head uses a Sigmoid activation function to calculate the probability distribution matrix, wherein each element in the probability distribution matrix represents a pathological severity. A method for medically assisting the diagnosis of gastritis, comprising: providing a first data set and a second data set, wherein the first data set and the second data set include upper gastrointestinal endoscopic images; performing image preprocessing on the first data set and the second data set; performing deep learning on the first data set and the second data set; and inferring a test image based on the result of the deep learning to generate an auxiliary image for assisting in identifying whether the test image includes a precancerous lesion, wherein: the first data set comes from an unspecified region and has an average morbidity; and the second data set comes from a specific region and has a morbidity higher than the average morbidity; wherein the deep learning The learning step includes: training a first model based on the first data set and the second data set for identifying the gastric region and the non-gastric region in a sample image; training a second model based on the first data set and the second data set for further classifying the gastric region identified in the sample image into the gastric sinus, the gastric body, and the gastric fundus; and training a third model based on the first data set and the second data set for determining whether there is a precancerous lesion in the gastric sinus and the gastric body in the sample image, wherein: The sample image is an upper gastrointestinal tract endoscopic image from the first data set or the second data set; and the precancerous lesion includes atrophic gastritis or intestinal metaplasia. The method for medically assisting in diagnosing gastritis as described in claim 8 further comprises a preprocessing step, the preprocessing step comprising: converting an input image into a grayscale image; converting the grayscale image into a binary map using an Otsu threshold; performing edge detection on the binary map to identify a target region in the input image; cropping the input image to obtain a region that retains only the target region; A cropped image of the target area; and adjusting the size of the cropped image to a preset dimension to generate a normalized image corresponding to the input image; wherein the method for medically assisted diagnosis of gastritis further comprises, before performing the deep learning step on the first dataset and the second dataset, normalizing the images in the first dataset and the second dataset through the preprocessing step. The method for medically aided diagnosis of gastritis as described in claim 9, wherein the preprocessing step further comprises: performing a contrast limited adaptive histogram equalization (CLAHE) operation on the input image to enhance image details and highlight gastric mucosal features. The method for medically assisting in diagnosing gastritis as described in claim 9 further comprises: performing the preprocessing step to normalize the image to be tested; using the first model to identify the gastric region and non-gastric region in the image to be tested; using the second model to further classify the gastric region in the image to be tested into the gastric sinus, gastric body, and gastric fundus; using the third model to determine whether the gastric sinus and gastric body in the image to be tested have the precancerous lesion or Helicobacter pylori infection; and using the Gradient-weighted Class Activation Mapping (Grad-CAM) operation to superimpose a visual effect on the image to be tested according to the probability of disease, thereby generating an auxiliary diagnosis image. The method for medically assisting in diagnosing gastritis as described in claim 8, wherein the step of training the first model comprises: allocating 70% of the first data set and the second data set as training sets, 20% as validation sets, and 10% as test sets; and using the dense network DenseNet201 as a model template, substituting the above training set, validation set, and test set into a convolutional neural network to calculate and fine-tune parameters to generate the first model. The method for medically assisting in diagnosing gastritis as described in claim 8, wherein the step of training the second model comprises: allocating 80% of the first data set as a training set, 20% as a validation set, and 10% of the second data set as a test set; and using the dense network DenseNet121 as a model template, substituting the above training set, validation set, and test set into a convolutional neural network to calculate and fine-tune the parameters to generate the second model, wherein: the second model comprises a SoftMax activation function for calculating the probability distribution of the recognition result of the sample image. A method for medically assisting in the diagnosis of gastritis as described in claim 8, wherein the third model is based on a visual converter as a model template and is trained through a learning transfer method, the steps comprising: reading an input image from an open source data set of colorectal cancer tissue slice images; translating the input image to four diagonals to generate four displacement images, and then stacking them with the input image to form a translation stack image; cutting the translation stack image into a plurality of patches, flattening them into a one-dimensional sequence, wherein each patch corresponds to a feature of a fixed length; eigenvector; adding position information to each patch; performing self-attention operation on each patch to extract a feature vector matrix corresponding to the input image; wherein the self-attention operation includes local self-attention operation, so that the feature vector matrix emphasizes local correlation; and pooling the feature vector matrix to obtain a probability distribution matrix that can represent the entire input image, wherein: The probability distribution matrix is calculated using a Sigmoid activation function, wherein each element in the probability distribution matrix represents a pathological severity.