TWI856531B - Computer program and computer readable medium for dry mouth evaluation or oral contamination evaluation by image - Google Patents

Abstract

The present invention relates to a computer program and computer readable medium for dry mouth or oral contamination evaluation by image hereof, and that is characterized in being installed on a computer and perform the following steps: take an image of open mouth with protruding tongue to be evaluated, an image of close mouth with lip retractor to be evaluated, and an image of open mouth without protruding tongue as an evaluation object, and respectively use a trained artificial intelligence model executes the evaluation, and obtains an evaluation result of an image of open mouth with protruding tongue, an evaluation result of an image of close mouth with lip retractor, and an evaluation result of an image of open mouth without protruding tongue, and then the program outputs a dry mouth evaluation result or an oral contamination evaluation result that based on at least two of the three image evaluation results. Thus, it is convenient for the user to get advice for the need of care assistance or the need of referral to a professional for intervention as soon as possible according to the computer program evaluation result.

Description

Computer program and computer readable medium for evaluating oral dryness or contamination based on images

The present invention relates to a computer program and a computer-readable medium for evaluating oral dryness or contamination, and in particular to a computer program and a computer-readable medium for evaluating oral dryness or contamination based on images.

The mouth is not only the entrance for eating, but also one of the entrances for the respiratory tract, thus becoming an important entrance for bacterial infection. Oral dryness and oral contamination both increase the risk of oral and respiratory infections. Medical research shows that oral contamination and infection can also increase blood sugar instability and increase the risk of inflammation and infection throughout the body. The moisture content of the lips, back of the tongue, gums and mucosa, as well as the degree of saliva adhesion to oral tissues, are all used in clinical practice for preliminary screening of dry mouth. In addition, if there are crusts (peeled epithelial membranes), oral ulcers, trauma or bleeding on the upper jaw, back of the tongue, lips and oral mucosa, it is easy to promote the growth of bacteria in the mouth, contaminate the mouth, and is a factor promoting oral and respiratory infections. Professionals need to intervene as soon as possible to help improve.

When the mouth is dry, there is insufficient saliva in the mouth, and the self-cleaning function is reduced, which can easily lead to food residues in the mouth and pharynx, and oral microorganisms can easily proliferate and damage the tissues in the mouth. When the mouth is dry, there is insufficient saliva in the mouth, and the removable dentures are not easy to fit tightly with the oral mucosa. The adhesion of the removable dentures is significantly reduced, causing the dentures to shake unsteadily when chewing, which can easily cause trauma, ulcers and bleeding when eating. Traumatic ulcers and bleeding will also further promote the proliferation of oral microorganisms. When the mouth is dry, there is insufficient saliva in the mouth, and it is difficult to form a food bolus suitable for swallowing when chewing food, which makes chewing and swallowing difficult, increasing the risk of food residue in the throat and aspiration. It is often necessary to increase the number of chewing times and chewing time to soften the food bolus before swallowing, making eating difficult and easy to get tired, affecting the quality of food. If it leads to malnutrition, it will affect immunity and increase the risk of infection. Discovering dry mouth and intervening to improve it as soon as possible can guide care skills as early as possible, massage the mouth to increase saliva secretion, use moisturizers to reduce saliva evaporation; remove the scabs formed by dry mouth and peel off the epithelium (crust), restore the touch, temperature and taste of the epithelial tissue, and improve the mobility of the epithelial tissue to enhance the feasibility of oral feeding. Removing the scab can also prevent the scab from being accidentally inhaled into the respiratory tract when it falls off, causing the risk of aspiration pneumonia. Visual methods and scales are often used in clinical practice for manual assessment to detect dry mouth. However, the visual assessment of dry mouth often results in errors due to the experience or negligence of the viewer.

Cleanliness and hygiene are the cornerstones of medical care and care. In the assessment of oral contamination, the degree of tongue coating is positively correlated with the number of bacteria in the mouth. If the tongue coating is assessed visually, when there is too much tongue coating, oral cleaning needs to be strengthened. If there are crusts (oral exfoliated epithelial membrane), oral ulcers, trauma or bleeding on the upper jaw, back of the tongue, lips and oral mucosa, it is easy to increase the reproduction of microorganisms in the mouth. If crusts, oral ulcers, trauma or bleeding are found during visual inspection, doctors, dentists, nurses, caregivers and other medical and care service professionals should be asked to intervene and assist as soon as possible. Oral bleeding is often caused by side effects of medication, or by inflammation or dryness. It should be treated as soon as possible, check whether daily medication needs to be adjusted, and professional intervention should be provided to assist with local oral medication, oral cleaning, and moisturizing.

Patents related to oral imaging include, for example, Patent Publication No. 202217843A of the Republic of China, Patent Announcement No. I764315 of the Republic of China, and Patent Publication No. WO2020029915A1 filed under the Patent Cooperation Treaty.

The aforementioned patent case 202217843A provides a remote tongue diagnosis method, computer program product, and device based on deep learning, which mainly inputs the photographed photo into multiple local detection convolutional neural networks to obtain multiple categories of classification results related to the tongue in the photographed photo, and then obtains medical instructions corresponding to the classification results of the multiple categories.

The aforementioned I764315 patent provides a tongue image detection device, which mainly uses a camera module to obtain at least one detection image of a tongue, and separates a plurality of detection areas in the detection image through a processing module, and obtains a plurality of tongue image feature points from the detection areas, and then uses a deep learning algorithm to determine whether the tongue image feature points in the detection areas are consistent with any of the plurality of symptom data, and the processing module further generates a diagnosis data according to the symptom data.

The aforementioned WO2020029915A1 patent provides an artificial intelligence-based TCM tongue image segmentation device, method and storage medium, which mainly processes the RGB pixels of the tongue surface image through the SLIC algorithm to generate N super-pixel regions, extracts the feature group of each super-pixel region, and then uses the pre-trained tongue classifier to classify the feature group of each super-pixel region, thereby identifying the tongue region and non-tongue region.

However, the aforementioned WO2020029915A1 patent mainly isolates the image of the tongue, and the back of the tongue still needs to be judged by medical personnel. Although the aforementioned 202217843A patent and the aforementioned I764315 patent do judge the back of the tongue, there is no further combined judgment for other parts of the oral cavity.

For dry mouth and oral contamination, which can easily increase the risk of oral and respiratory infections, not only the condition of the back of the tongue, but also the condition of the lips, gums and mucosa, as well as the degree of saliva adhesion to oral tissues, are all evaluated.

The aforementioned patents are not used to evaluate oral dryness or oral contamination, and it is difficult to accurately evaluate oral dryness or oral contamination simply by evaluating the tongue dorsum in the aforementioned patents.

Therefore, the inventors propose a computer program for evaluating oral dryness or contamination based on images, which is installed on a computer and executes the following steps: Step A. Take an image of an open mouth with tongue sticking out to be evaluated, an image of a mouth with lip expansion and mouth opening device closed to be evaluated, and an image of an open mouth without tongue sticking out to be evaluated as an evaluation target; execute at least two of Step B, Step C, and Step D; Step B An artificial intelligence model is obtained through deep learning based on a normal open-mouth tongue-out image and an abnormal open-mouth tongue-out image, wherein the abnormal open-mouth tongue-out image includes an assessment item of oral dryness or oral contamination symptoms, and deep learning enables the artificial intelligence model to identify the assessment item through image recognition based on image comparison, and then the open-mouth tongue-out image to be assessed is used as the training The artificial intelligence model performs the evaluation and obtains a tongue-out image evaluation result; the step C. The artificial intelligence model is obtained through deep learning based on a normal lip-expanding mouth-opening device mouth-closing image and an abnormal lip-expanding mouth-opening device mouth-closing image, wherein the abnormal lip-expanding mouth-opening device mouth-closing image includes the evaluation item of oral dryness or oral contamination symptoms, and the lip-expanding mouth-closing device mouth-closing image to be evaluated is used as the evaluation item of the lip-expanding mouth-closing device mouth-closing image to be evaluated. The artificial intelligence model performs the evaluation and obtains a lip expansion and mouth opening device closed mouth image evaluation result; the step D. The artificial intelligence model is obtained through deep learning based on a normal open mouth without tongue sticking out image and an abnormal open mouth without tongue sticking out image, wherein the abnormal open mouth without tongue sticking out image includes the evaluation item with oral dryness or oral contamination symptoms, and the open mouth without tongue sticking out image to be evaluated is used by the artificial intelligence model to obtain the evaluation result of the lip expansion and mouth opening device closed mouth image; the step D. The artificial intelligence model is obtained through deep learning based on a normal open mouth without tongue sticking out image and an abnormal open mouth without tongue sticking out image, wherein the abnormal open mouth without tongue sticking out image includes the evaluation item with oral dryness or oral contamination symptoms, and the open mouth without tongue sticking out image to be evaluated is used by the artificial intelligence model to obtain the evaluation result of the lip expansion and mouth opening device closed mouth image. The artificial intelligence model is used to perform an evaluation to obtain an open mouth without tongue sticking out image evaluation result; Step E. correspondingly, according to at least two of the tongue sticking out image evaluation result, the lip expander and mouth opening device closed mouth image evaluation result, and the open mouth without tongue sticking out image evaluation result, the artificial intelligence model outputs an oral dryness evaluation result or an oral contamination evaluation result; wherein the artificial intelligence model When the evaluation is performed by the type, an evaluation score is given to the evaluation item in the image of open mouth and tongue sticking out, the image of lip-opening and mouth-opening device closed mouth, and the image of open mouth without tongue sticking out, and the evaluation score is used as the evaluation result of the image of tongue sticking out, the evaluation result of the image of lip-opening and mouth-opening device closed mouth, and the evaluation result of the image of open mouth without tongue sticking out. Then the evaluation score is compared with a corresponding threshold value. The outputs are the oral dryness assessment result and the oral contamination assessment result respectively; the assessment item of the artificial intelligence model for the open-mouth tongue-out image to be assessed is one of the following or a combination thereof: tongue coating adhesion state, visibility of tongue papillae, color, crust, trauma, ulcer, bleeding, saliva stringiness, saliva bubbles, saliva clumping state, tongue dorsum wrinkles, tongue dorsum crack lines, oral Mucosal wrinkles, oral mucosal cracks and food residues; the artificial intelligence model divides the tongue back image of the open mouth and tongue sticking out image to be evaluated into multiple tongue back regions, and assigns a region score according to the tongue coating adhesion state of each tongue back region. The sum of all the region scores is the evaluation score, and the evaluation score is compared with the threshold to evaluate the tongue coating adhesion state.

The open-mouth and tongue-sticking images to be evaluated include partial or complete images of the following parts: upper lip, upper teeth, back of tongue, lower lip, lower teeth and corners of mouth.

The tongue dorsum is divided into nine grids.

The lip expander mouth opening device closed mouth image to be evaluated includes partial or complete images of the following parts: upper lip, upper gum, upper teeth row, upper gum peri-mucosa, lower lip, lower gum, lower gum peri-mucosa and lower teeth row.

The evaluation item of the artificial intelligence model for the lip expander and mouth opening device closed mouth image to be evaluated is one of the following or a combination thereof: color, crust, wound, ulcer, bleeding, food residue, dental plaque, tooth decay and tooth fracture.

The artificial intelligence model evaluates the lip expander and mouth opening device closure image to be evaluated in a single area, or divides the lip expander and mouth opening device closure image to be evaluated into multiple areas for area-by-area evaluation.

The open-mouthed, non-tongue-out images to be evaluated include partial or complete images of the following areas: upper lip, upper teeth, upper jaw, oropharynx, tongue dorsum, oral mucosa, lower lip, lower teeth, and corners of mouth.

The evaluation item of the artificial intelligence model for the open-mouth image to be evaluated is one of the following or a combination thereof: color, crust, wound, ulcer, bleeding, tongue dorsum wrinkles, tongue dorsum dry cracks, oral mucosal wrinkles, oral mucosal dry cracks, saliva stringiness, saliva bubbles, saliva clumping and sticky state, food residue, dental calculus, tooth decay and tooth fracture.

The artificial intelligence model evaluates the image of the open mouth without sticking out the tongue in a single area, or divides the image of the open mouth without sticking out the tongue into multiple areas for evaluation.

Among them, a lip expander and mouth opener is used to assist in expanding the lips and opening the mouth to take the image of the open mouth without sticking out the tongue to be evaluated.

The artificial intelligence model is obtained through deep learning based on a normal throat image and an abnormal throat image, wherein the abnormal throat image includes the evaluation item of having dry mouth or oral contamination symptoms, and the artificial intelligence model further evaluates a throat image to be evaluated, and gives the evaluation score to the evaluation item of the throat image to be evaluated, thereby obtaining a throat image evaluation result. The artificial intelligence model outputs the dry mouth evaluation result or the oral contamination evaluation result based on at least two of the tongue-out image evaluation result, the lip-opening mouth-closing image evaluation result, the open mouth without tongue-out image evaluation result, and the throat image evaluation result.

The artificial intelligence model further utilizes at least two of the tongue-out image assessment result, the lip-opening and mouth-closing image assessment result, the open-mouth image assessment result, and the throat image assessment result, combined with a medical diagnosis message, to output a recommendation message.

The pharyngeal images to be evaluated include partial or complete images of the following parts: pharyngeal mucosa, tonsils, tongue base, epiglottis, vallecula epiglottis, pyriformis, laryngeal vestibule and vocal cords.

The evaluation item of the artificial intelligence model in the throat image to be evaluated is one of the following: color, crust, wound, ulcer, bleeding, mucosal folds, mucosal cracks, saliva strings, saliva bubbles, saliva clumping and sticky state, saliva residue, food residue.

The inventor of the present invention also proposes a computer-readable medium that records the aforementioned computer program for evaluating oral dryness or contamination based on images.

According to the above technical features, the following effects can be achieved preferably:

1. Based on at least two of the three evaluation results of tongue sticking out image evaluation, lip expansion and mouth opening device closed mouth image evaluation, and open mouth without tongue sticking out image evaluation, oral dryness evaluation results or oral contamination evaluation results are obtained, so that users can take necessary measures as soon as possible according to the evaluation results, such as care assistance or professional referral.

2. Evaluate based on at least two of the following three evaluation results: tongue-sticking-out imaging evaluation results, lip-opening-mouth-closing imaging evaluation results, and open-mouth-but-not-sticking-out imaging evaluation results. You can even further add pharyngeal imaging evaluation results to avoid inaccurate judgments based on a single evaluation result.

3. In addition to the evaluation results obtained by the artificial intelligence model itself, it can also combine medical diagnosis information to output recommended information so that users or caregivers can better understand what actions to take.

4. For those who have difficulty going out for medical treatment, the present invention provides an effective tool for assessing oral dryness and oral contamination at home.

5. Reduce intensive contact during epidemics.

6. The evaluation results are easy to store and share. Nursing and medical units can integrate care and multi-professional collaboration to provide patients with immediate care and timely medical intervention.

7. Use artificial intelligence to reduce the demand for professional manpower, help implement regular health risk management for high-risk groups for oral or respiratory infections, and detect cases requiring professional intervention of medical and care services as early as possible.

1: Artificial intelligence model

A: Open mouth and tongue sticking out image to be evaluated

A’: Tongue sticking out image assessment results

A1: Upper lip

A2: Dorsal tongue

A21: Dorsal tongue division

A22: Tongue coating

A23: Saliva vesicles

A24: Dry cracks on the back of the tongue

A3: Lower lip

A4: Quarrel

B: Image of lip expander and mouth opening device closing the mouth to be evaluated

B’: Image evaluation results of lip expander and mouth opener closure

B1: Lip expander and mouth opener

B2: Upper gum

B21: Ulcers

B3: Upper teeth row

B4: Upper gum perigingival mucosa

B5: Lower gum

B6: Lower teeth row

B7: lower gum perigingival mucosa

C, C0: open mouth without tongue sticking out image to be evaluated

C’: Image evaluation results of mouth opening without tongue sticking out

C1: Upper lip

C2: Upper teeth row

C3: maxillary

C4: Dorsal tongue

C5: Lower lip

C6: Lower teeth row

C7: Quarrel

C71: Trauma

D: Throat imaging to be evaluated

D’: Throat imaging assessment results

D1: Piriform fossa

D2: I would hate it

D3: The Valley of Disgust

D4: Vocal cords

D5: Laryngeal vestibule

E: Oral dryness assessment results

F: Oral contamination assessment results

G: Medical diagnosis information

H: Suggestion message

[Figure 1] is a system block diagram of an embodiment of the present invention.

[Figure 2] is the first schematic diagram of the embodiment of the present invention, showing a normal image of an open mouth and tongue sticking out to be evaluated.

[Figure 3] is the second schematic diagram of the embodiment of the present invention, showing that in the image of the open mouth and tongue sticking out to be evaluated, there is tongue coating attached to the back of the tongue.

[Figure 4] is the third schematic diagram of the embodiment of the present invention, showing the tongue dorsum with cracked lines and saliva bubbles in the image of the open mouth and tongue sticking out to be evaluated.

[Fifth Figure] is the fourth schematic diagram of the embodiment of the present invention, showing a normal image of the lip expansion mouth opening device to be evaluated when the mouth is closed.

[Figure 6] is the fifth schematic diagram of the embodiment of the present invention, showing the upper gum ulcer in the closed mouth image of the lip expander mouth opening device to be evaluated.

[Figure 7] is the sixth schematic diagram of the embodiment of the present invention, showing a normal image of the person to be evaluated with the mouth open and the tongue not sticking out, without tilting the head back.

[Figure 8] is the seventh schematic diagram of the embodiment of the present invention, showing a normal image of the person to be evaluated with the mouth open and the tongue not sticking out, with the head tilted back.

[Figure 9] is the implementation diagram 8 of the embodiment of the present invention, showing the wound at the corner of the mouth in the image of the mouth open without sticking out the tongue when the head is not tilted back.

[Figure 10] is a photograph of an embodiment of the present invention, showing the throat image to be evaluated.

Combining the above technical features, the main functions of the computer program and computer-readable medium for evaluating oral dryness or contamination based on images of the present invention will be clearly presented in the following embodiments.

Please refer to the first and second figures, which disclose the computer program for evaluating oral dryness or contamination based on images according to the embodiment of the present invention. In actual implementation, the computer program for evaluating oral dryness or contamination based on images can also be recorded in a computer-readable medium, and after the computer-readable medium is connected to a computer, it is executed on the computer.

After the computer program for evaluating oral dryness or contamination based on images is installed on the computer, the following steps are executed: a trained artificial intelligence model 1 performs evaluation with an open mouth and tongue-sticking image A to be evaluated, an expanded lip and mouth-opening device closed mouth image B to be evaluated, an open mouth and non-tongue-sticking image C to be evaluated, and a throat image D to be evaluated as an evaluation target, and outputs a tongue-sticking image evaluation result A’, a lip-sticking device closed mouth image evaluation result B’, an open mouth and non-tongue-sticking image evaluation result C’, and a throat image evaluation result D’.

In actual implementation, at least two corresponding evaluation results can be obtained based on at least two of the images for subsequent evaluation.

Please refer to the first to fourth figures. In order to evaluate the open-mouth tongue-sticking image A to be evaluated and obtain the tongue-sticking image evaluation result A', the training process of the artificial intelligence model 1 needs to be first obtained through deep learning based on a normal open-mouth tongue-sticking image and an abnormal open-mouth tongue-sticking image. The abnormal open-mouth tongue-sticking image includes an evaluation item of having oral dryness or oral contamination symptoms.

The normal open mouth and tongue sticking out image, the abnormal open mouth and tongue sticking out image and the open mouth and tongue sticking out image A to be evaluated may include partial or complete images of the upper lip A1, upper teeth, back of tongue A2, lower lip A3, lower teeth and corner of mouth A4.

The evaluation items of the normal open-mouth tongue-out image, the abnormal open-mouth tongue-out image and the open-mouth tongue-out image to be evaluated A include, for example, tongue coating adhesion, visibility of tongue papillae, color, crust, wound, ulcer, bleeding, saliva stringiness, saliva vesicles A23, saliva clumping and sticking state, tongue dorsum wrinkles, tongue dorsum fissure lines A24, oral mucosal wrinkles, oral mucosal fissure lines and food residues, etc.

Taking the evaluation item of tongue coating attachment state as an example, the artificial intelligence model 1 can first divide the tongue back image of the tongue-opening image A to be evaluated into multiple tongue back areas A21. The preferred embodiment of the present invention takes the nine-square grid as an example. In actual implementation, it can also be, for example, 2 grids, 3 grids, 4 grids, ..., 18 grids, etc., or more grids, and even the image of the entire tongue back A2 is used as a single tongue back area A21. It is not limited to dividing by mutually perpendicular line segments, for example, it can be divided into 3 grids by Y-shaped line segments, etc.

Next, a division score is assigned according to the tongue coating adhesion state of each tongue back division A21, and the sum of all the division scores is an evaluation score. The evaluation score is compared with a threshold value to evaluate the tongue coating adhesion state and serve as the tongue sticking out image evaluation result A’.

For example, the scores of the areas are defined as 0, 1, and 2 points respectively from tongue coating A22 with the least amount of coating to tongue coating A23 with the most amount of coating. If the nine tongue dorsum areas A21 are all given the highest score of 2, the highest total score is 18 points.

At this time, the threshold corresponding to the tongue coating adhesion state can be defined as half of 18 points. When the total score of a certain user is higher than the threshold, the artificial intelligence model 1 can combine the evaluation results of other images to judge that the user's oral contamination level is higher, and use it as the oral contamination evaluation result F.

In actual implementation, the doctor may score or annotate the abnormal open-mouth tongue-out image or other abnormal images used to train the artificial intelligence model 1 in advance. In terms of scoring, the doctor may establish a scoring table based on the abnormal open-mouth tongue-out image. The artificial intelligence model 1 is established by comparing the normal open-mouth tongue-out image with the abnormal open-mouth tongue-out image, identifying the evaluation item using image recognition technology, and then performing deep learning based on the scoring table to complete the training.

Similarly, the artificial intelligence model 1 can also judge that the user's oral dryness is higher based on, for example, the presence of saliva bubbles A23 and tongue dorsum cracks A24 on the tongue dorsum A2, combined with the evaluation results of at least one other image, and use this as the oral dryness evaluation result E.

Please refer to the first, fifth and sixth figures, in order to evaluate the lip expander and mouth opener closure image B to be evaluated and obtain the lip expander and mouth opener closure image evaluation result B', the training process of the artificial intelligence model 1 needs to be first obtained through deep learning based on a normal lip expander and mouth opener closure image and an abnormal lip expander and mouth opener closure image, and the abnormal lip expander and mouth opener closure image includes the evaluation item with oral dryness or oral contamination symptoms.

The normal lip retractor mouth closing image, the abnormal lip retractor mouth closing image and the lip retractor mouth closing image to be evaluated B use a lip retractor (Lip retractor) B1 to assist in mouth opening, and include partial or complete images of the upper lip, upper gum B2, upper teeth B3, upper gum peri-mucosa B4, lower lip, lower gum B5, lower teeth B6 and lower gum peri-mucosa B7.

The evaluation items of the normal lip expander and mouth opening device closure image, the abnormal lip expander and mouth opening device closure image and the lip expander and mouth opening device closure image B to be evaluated include, for example, color, crust, wound, ulcer B21, bleeding, food residue, dental calculus, tooth decay and tooth fracture.

Similarly, the artificial intelligence model 1 can evaluate the lip expander and mouth opening device closure image B to be evaluated in a single area, or divide the lip expander and mouth opening device closure image B to be evaluated into multiple areas for area-by-area evaluation, so as to calculate the evaluation score according to, for example, the color depth from 0 to 2 points, 2 points for the presence of crust, 0 points for the absence of crust, etc., and obtain the lip expander and mouth opening device closure image evaluation result B'.

The artificial intelligence model 1 can judge that the user's oral contamination level is higher based on the presence of ulcer B21 on the upper gum B2 and combined with the evaluation results of at least one other image, and use this as the oral contamination evaluation result F.

Please refer to the first and seventh figures. In order to evaluate the open-mouth-without-tongue image C to be evaluated and obtain the open-mouth-without-tongue image evaluation result C', the training process of the artificial intelligence model 1 needs to be first obtained through deep learning based on a normal open-mouth-without-tongue image and an abnormal open-mouth-without-tongue image. The abnormal open-mouth-without-tongue image includes the evaluation item with oral dryness or oral contamination symptoms.

The normal open mouth without tongue sticking out image, the abnormal open mouth without tongue sticking out image and the open mouth without tongue sticking out image C to be evaluated may include partial or complete images of the upper lip C1, upper teeth C2, upper jaw C3, oropharynx, tongue dorsum C4, oral mucosa, lower lip C5, lower teeth C6 and corner of mouth C7.

Please refer to the first and eighth figures. When shooting the image C of the open mouth without tongue sticking out to be evaluated, in addition to using the lip expander and mouth opener B1 as shown in the fifth figure to assist in expanding the lips and opening the mouth, the head can also be tilted upward to form the image C0 of the open mouth without tongue sticking out to be evaluated in the eighth figure, so as to show a larger area of the upper jaw C3, so that the artificial intelligence model 1 can more accurately evaluate the state of the upper jaw C3.

Please refer to the first, seventh and ninth figures. The evaluation items of the normal open mouth without tongue sticking out image, the abnormal open mouth without tongue sticking out image and the open mouth without tongue sticking out image C to be evaluated include, for example, color, crust, wound C71, ulcer, bleeding, tongue dorsum wrinkles, tongue dorsum dry cracks, oral mucosal wrinkles, oral mucosal dry cracks, saliva stringiness, saliva bubbles, saliva clumping and sticky state, food residues, dental calculus, tooth decay and tooth cracks, etc.

Similarly, the artificial intelligence model 1 can evaluate the open-mouth-but-tongue-out image C to be evaluated in a single area, or divide the open-mouth-but-tongue-out image C to be evaluated into multiple areas for area-by-area evaluation, and obtain the open-mouth-but-tongue-out image evaluation result C'.

The artificial intelligence model 1 can judge that the user's oral dryness is higher based on the wound C71 at the corner of the mouth C7 and combined with the evaluation results of at least one other image, and use it as the oral dryness evaluation result E.

Please refer to the first and tenth figures. In order to evaluate the throat image D to be evaluated and obtain the throat image evaluation result D', the training process of the artificial intelligence model 1 needs to be first obtained through deep learning based on a normal throat image and an abnormal throat image. The abnormal throat image includes the evaluation item with oral dryness or oral contamination symptoms.

The normal throat image, the abnormal throat image and the throat image to be evaluated D may include partial or complete images of the throat mucosa, pyriform sinus D1, tonsils, tongue root, epiglottis D2, vallecula D3, vocal cords D4 and laryngeal vestibule D5.

The evaluation items of the normal throat image, the abnormal throat image and the throat image D to be evaluated include, for example, color, crust, wound, ulcer, bleeding, mucosal folds, mucosal cracks, saliva strings, saliva bubbles, saliva clumping and sticky state, saliva residue, food residue, etc.

Similarly, the artificial intelligence model 1 can evaluate the pharyngeal image D to be evaluated in a single area, or divide the pharyngeal image D to be evaluated into multiple areas for area-by-area evaluation, and obtain the pharyngeal image evaluation result D’.

The artificial intelligence model 1 can judge that the user has a higher risk of respiratory infection due to accidental swallowing and inhalation based on the presence of food residue or saliva residue in the pyriform fossa D1 and the evaluation results of at least one other image, and must increase the prevention level or improvement level of oral dryness and oral contamination, and provide the oral dryness evaluation result E and the oral contamination evaluation result F, but this scenario is not drawn in the diagram of the present invention.

Preferably, each of the evaluation items has a corresponding threshold value, and the evaluation score obtained according to the selected evaluation item is compared with the corresponding threshold value to be used as the oral dryness evaluation result E or the oral contamination evaluation result F. For example, the lip expander and mouth opener closed mouth image B to be evaluated and the open mouth without tongue sticking out image C to be evaluated are selected for evaluation. Two of the three evaluation items in the lip expander and mouth opener closed mouth image B to be evaluated have evaluation scores exceeding the threshold value, and only one of the five evaluation items in the open mouth without tongue sticking out image C to be evaluated has evaluation scores exceeding the threshold value. Then, based on the fact that only three of the eight evaluation items have evaluation scores exceeding the threshold value, the artificial intelligence model 1 can judge that the user has a lower degree of oral dryness or oral contamination.

In addition to obtaining the oral dryness assessment result E or the oral contamination assessment result F, a medical diagnosis message G can also be combined to output a recommendation message H.

For example, if the user has previously visited a doctor and obtained the medical diagnosis message G of swallowing disorder, and when the user returns home and uses the computer program for evaluating oral dryness or contamination based on images to self-assess, the artificial intelligence model 1 can output the recommendation message H based on the presence of saliva or food residues in the pyriform fossa D1 and the medical diagnosis message G of swallowing disorder, such as the need to strengthen eating and swallowing assistance and training, the need to check the eating and swallowing status, the need to seek medical assistance, etc., so that the user or caregiver can better understand what actions to take.

When the user's swallowing ability improves and there is no saliva or food residue in the pyriform fossa D1, the user continues to use the computer program for self-assessment based on images to assess oral dryness or contamination. The artificial intelligence model 1 can combine the medical diagnosis message G of swallowing disorders and output the recommendation message H, such as "the swallowing ability has improved, please continue to maintain regular monitoring and health management."

Please refer to the first figure again. The evaluation is performed by using at least two of the tongue-out image evaluation result A’, the lip-opening mouth-closing image evaluation result B’, the open-mouth image evaluation result C’ and the throat image evaluation result D’ to avoid inaccurate judgment based on a single evaluation result.

The open mouth and tongue sticking out image A to be evaluated, the lip expansion and mouth opening device closed mouth image B to be evaluated, the open mouth and tongue sticking out image C to be evaluated, and the throat image D to be evaluated can be taken by using a mobile phone, camera, endoscope, ultrasound and other equipment, and input into the artificial intelligence model 1 by using the computer. The user can understand whether necessary response is needed as soon as possible, such as necessary care assistance or referral to a professional, based on the oral dryness assessment result E or the oral contamination assessment result F obtained, as well as the further recommended message H.

In medical institutions, preliminary screening can also be performed based on the oral dryness assessment result E or the oral contamination assessment result F, so that medical personnel can make further diagnosis and treatment.

Combined with the description of the above embodiments, the operation, use and effects of the present invention can be fully understood. However, the above embodiments are only the preferred embodiments of the present invention and cannot be used to limit the scope of implementation of the present invention. In other words, simple equivalent changes and modifications made according to the scope of the patent application and the content of the invention description are all within the scope of the present invention.

1: Artificial intelligence model

A: Open mouth and tongue sticking out image to be evaluated

A’: Tongue sticking out image assessment results

B: Image of lip expander and mouth opening device closing the mouth to be evaluated

B’: Image evaluation results of lip expander and mouth opener closure

C: Image of open mouth without tongue sticking out to be evaluated

C’: Image evaluation results of mouth opening without tongue sticking out

D: Throat imaging to be evaluated

D’: Throat imaging assessment results

E: Oral dryness assessment results

F: Oral contamination assessment results

G: Medical diagnosis information

H: Suggestion message

Claims (15)

A computer program for evaluating oral dryness or contamination based on images is used to be installed on a computer and execute the following steps: Step A. Take an image of open mouth with tongue sticking out to be evaluated, an image of mouth opening with lip expansion and mouth opening device closed to be evaluated, and an image of open mouth without tongue sticking out to be evaluated as an evaluation target; execute at least two of Step B, Step C and Step D; Step B. An artificial intelligence model The method is obtained by deep learning based on a normal open-mouth tongue-out image and an abnormal open-mouth tongue-out image, wherein the abnormal open-mouth tongue-out image includes an evaluation item of oral dryness or oral contamination symptoms. The deep learning is based on image comparison to enable the artificial intelligence model to identify the evaluation item by image recognition, and then the open-mouth tongue-out image to be evaluated is used by the trained artificial intelligence model. The step C. the artificial intelligence model is obtained by deep learning based on a normal lip-expanding mouth-opening device closing mouth image and an abnormal lip-expanding mouth-opening device closing mouth image, wherein the abnormal lip-expanding mouth-opening device closing mouth image includes the evaluation item of oral dryness or oral contamination symptoms, and the lip-expanding mouth-opening device closing mouth image to be evaluated is used to evaluate the lip-expanding mouth-opening device closing mouth image. The artificial intelligence model is obtained by deep learning based on a normal open mouth without tongue sticking out image and an abnormal open mouth without tongue sticking out image, wherein the abnormal open mouth without tongue sticking out image includes the evaluation item with oral dryness or oral contamination symptoms, and the open mouth without tongue sticking out image to be evaluated is evaluated by the artificial intelligence model. The artificial intelligence model performs an evaluation to obtain an open mouth without tongue sticking out image evaluation result; Step E. Correspondingly, according to at least two of the tongue sticking out image evaluation result, the lip expander mouth opening device closed mouth image evaluation result, and the open mouth without tongue sticking out image evaluation result, the artificial intelligence model outputs an oral dryness evaluation result or an oral contamination evaluation result; Wherein, the artificial intelligence model performs When conducting the evaluation, an evaluation score is given to the evaluation item in the image of open mouth with tongue sticking out, the image of mouth opening with lip expansion and mouth opening device closed, and the image of open mouth without tongue sticking out, and the evaluation score is used as the evaluation result of the tongue sticking out image, the evaluation result of the image of mouth opening with lip expansion and mouth opening device closed, and the evaluation result of the image of open mouth without tongue sticking out, respectively. Then, the evaluation score is compared with a corresponding threshold value, and the evaluation is divided into The output is the oral dryness assessment result and the oral contamination assessment result respectively; the assessment item of the artificial intelligence model for the open-mouth and tongue-out image to be assessed is one of the following or a combination thereof: tongue coating adhesion state, tongue papilla visibility, color, crust, trauma, ulcer, bleeding, saliva stringiness, saliva vesicles, saliva clumping state, tongue dorsum wrinkles, tongue dorsum crack lines, oral mucosa membrane folds, oral mucosal cracks and food residues; the artificial intelligence model divides the tongue back image of the open mouth and tongue sticking out image to be evaluated into multiple tongue back regions, and assigns a region score according to the tongue coating adhesion state of each tongue back region. The sum of all the region scores is the evaluation score, and the evaluation score is compared with the threshold to evaluate the tongue coating adhesion state. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the image of the open mouth and tongue sticking out to be evaluated includes partial or complete images of the following parts: upper lip, upper teeth, back of tongue, lower lip, lower teeth and corners of mouth. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the tongue dorsum is divided into nine-square grids. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the lip expander mouth opening device closed mouth image to be evaluated includes partial or complete images of the following parts: upper lip, upper gum, upper teeth row, upper gum peri-mucosa, lower lip, lower gum, lower gum peri-mucosa and lower teeth row. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the evaluation item of the lip expander mouth opening device closed mouth image to be evaluated by the artificial intelligence model is one of the following or a combination thereof: color, crust, trauma, ulcer, bleeding, food residue, dental calculus, tooth decay and tooth fracture. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the artificial intelligence model evaluates the lip expander and mouth opening device closed mouth image to be evaluated as a single area, or divides the lip expander and mouth opening device closed mouth image to be evaluated into multiple areas for area-by-area evaluation. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the image to be evaluated of an open mouth without tongue protrusion includes partial or complete images of the following parts: upper lip, upper teeth, upper palate, oropharynx, back of tongue, oral mucosa, lower lip, lower teeth and corners of mouth. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the evaluation item of the open-mouth image to be evaluated by the artificial intelligence model is one of the following or a combination thereof: color, crust, wound, ulcer, bleeding, tongue dorsum wrinkles, tongue dorsum fissure lines, oral mucosal wrinkles, oral mucosal fissure lines, saliva strings, saliva bubbles, saliva clumping and sticky state, food residues, dental calculus, tooth decay and tooth fracture. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the artificial intelligence model evaluates the open-mouth image to be evaluated as a single area, or divides the open-mouth image to be evaluated into multiple areas for area-by-area evaluation. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein a lip expander and mouth opener is used to assist in expanding the lips and opening the mouth to capture the image of the open mouth to be evaluated without sticking out the tongue. A computer program for evaluating oral dryness or contamination based on images as described in claim 1, wherein the artificial intelligence model is obtained through deep learning based on a normal throat image and an abnormal throat image, the abnormal throat image includes the evaluation item with symptoms of oral dryness or oral contamination, the artificial intelligence model further evaluates a throat image to be evaluated, and gives the evaluation score to the evaluation item of the throat image to be evaluated, thereby obtaining a throat image evaluation result, and the artificial intelligence model outputs the oral dryness evaluation result or the oral contamination evaluation result based on at least two of the tongue-out image evaluation result, the lip-opening mouth-opening device closed mouth image evaluation result, the open mouth without tongue-out image evaluation result, and the throat image evaluation result. A computer program for evaluating oral dryness or contamination based on images as described in claim 11, wherein the artificial intelligence model further utilizes at least two of the tongue-out image evaluation result, the lip-opening mouth-closing image evaluation result, the open-mouth-without-tongue image evaluation result, and the throat image evaluation result, combined with a medical diagnosis message, to output a recommendation message. A computer program for evaluating oral dryness or contamination based on images as described in claim 11, wherein the pharyngeal image to be evaluated includes partial or complete images of the following parts: pharyngeal mucosa, tonsils, tongue base, epiglottis, vallecula epiglottis, pyriformis, laryngeal vestibule and vocal cords. A computer program for evaluating oral dryness or contamination based on images as described in claim 11, wherein the evaluation item of the pharyngeal image to be evaluated by the artificial intelligence model is one of the following: color, crust, wound, ulcer, bleeding, mucosal folds, mucosal cracks, saliva strings, saliva bubbles, saliva clumping and sticky state, saliva residue and food residue. A computer readable medium recording a computer program for evaluating oral dryness or contamination based on images as described in any one of claims 1 to 14.