WO2023058994A1 - Method and device for orthodontic treatment result prediction based on deep learning - Google Patents

Abstract

Disclosed are a method and device for orthodontic treatment result prediction based on deep learning. According to the present invention, the orthodontic treatment result prediction device comprises a processor and memory connected to the processor. The memory stores program instructions that are executed by the processor to: data-pre-process and standardize dental images before and after orthodontic extraction treatment and dental images before and after non-orthodontic extraction treatment, the dental images being images that have been collected in advance; configure a pair of the standardized dental images before and after orthodontic extraction treatment as a first data set; configure a pair of the standardized dental images before and after non-orthodontic extraction treatment as a second data set; use the first data set to train a first deep-learning model for predicting results of orthodontic extraction treatments; use the second data set to train a second deep-learning model for predicting results of non-orthodontic extraction treatments; and use the trained first deep-learning model or second deep-learning model to generate a predictive image of a prescribed patient's teeth after orthodontic extraction treatment or non-orthodontic extraction treatment.

Description

Method and apparatus for predicting orthodontic treatment results based on deep learning

The present invention relates to a deep learning-based orthodontic treatment result prediction method and apparatus.

Malocclusion refers to a condition in which the arrangement of teeth is not aligned for some reason or the state of upper and lower meshing is out of the normal range.

Malocclusion can cause aesthetic and functional problems or increase the incidence of tooth decay and various diseases. Orthodontic treatment helps to create or maintain healthy oral tissue by removing physiological and psychological disorders caused by malocclusion, and can create a harmonious face.

If each treatment result for various treatment methods can be predicted in the pre-treatment diagnosis and treatment plan establishment stage in order to obtain a successful orthodontic treatment effect, it can be of great help in establishing a treatment plan that meets the patient's needs.

Orthodontic treatment is a series of numerous medical ‘judgments’, such as consultation, diagnosis, establishment of an initial treatment plan, and large or small changes in the contents of treatment according to numerous variables at each visit. In particular, the results of orthodontic treatment can vary greatly depending on what kind of judgment the medical staff makes when establishing a treatment plan.

When establishing a treatment plan, one of the variables that can have a significant impact on the outcome of orthodontic treatment can be considered is whether or not the tooth is extracted and the decision of the tooth to be extracted.

Since there are large individual differences in teeth, gum bone, degree of growth, maxillary and mandibular skeletal conditions and growth, it is necessary to establish an orthodontic treatment plan tailored to each individual through precise examination and accurate analysis and prediction. An orthodontic treatment plan with tooth extraction is established considering many factors, but depending on the patient's case, it is greatly influenced by the subjective judgment of the attending physician rather than an objective unified standard.

In order to establish an efficient and accurate orthodontic treatment plan, there is a need for a predictive method that can provide a more objective basis for the orthodontist's judgment.

Deep Learning, one of the fields of Artificial Intelligence (AI) Neural Network technology, is used to cluster or classify data through a deeply layered/hierarchical model that represents a probability distribution similar to the input data population.

Recently, applications of artificial intelligence and deep learning algorithms are being attempted in various fields of medicine and dentistry.

In particular, deep learning algorithms in dentistry are used in areas such as tooth localization/numbering, detection of dental caries/periodontal disease/periapical disease/oral cancerous lesion, localization of cephalometric landmarks, image quality enhancement, prediction and compensation of deformation error in additive manufacturing of prosthesis, etc. its application is being attempted.

In order to solve the problems of the prior art, the present invention proposes a deep learning-based orthodontic treatment result prediction method and apparatus that can help establish an effective orthodontic treatment plan.

In order to achieve the above object, according to an embodiment of the present invention, an apparatus for predicting a result of orthodontic treatment based on deep learning, comprising: a processor; and a memory connected to the processor, wherein the memory pre-processes and standardizes pre-collected dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment, and standardizes the standardized dental images before and after orthodontic treatment. Construct a pair of dental images as a first data set, configure a pair of dental images before and after the standardized non-extraction orthodontic treatment as a second data set, and use the first data set to predict orthodontic treatment results 1 Learning a deep learning model, learning a second deep learning model for predicting non-extraction orthodontic treatment results using the second data set, and using the first deep learning model or the second deep learning model for which learning has been completed Provided is an orthodontic treatment result prediction device that stores program instructions executed by the processor to generate a predictive dental image after tooth extraction or non-extraction orthodontic treatment of a predetermined patient.

The program instructions adjust the degree of blackness of the collected dental images before and after orthodontic treatment and the dental images before and after non-extraction orthodontic treatment, centered on a preset reference point, the collected dental images before and after orthodontic treatment, The collected dental images before and after non-extraction orthodontic treatment are overlapped, respectively, and after the overlap, a region of interest of each of the collected dental images before and after orthodontic treatment and the dental images before and after non-extraction orthodontic treatment may be extracted.

The preset reference point may be the frontal cranial fossa by the stable bone superimposition method.

The region of interest may include upper and lower lip soft tissues and anterior teeth.

The program commands may convert the collected dental images before and after orthodontic treatment and the contrast of the dental images before and after non-extraction orthodontic treatment into black and white images to represent standardized brightness.

When constructing the first data set and the second data set, orthognathic surgery or facial soft tissue surgery, presence of artifacts on radiographs, prosthetic treatment or conservative treatment using metal objects during orthodontic treatment, impacted teeth, remaining at maturity Dental images with deciduous teeth, the presence of unerupted permanent teeth, and poor radiographic resolution can be excluded.

The first deep learning model and the second deep learning model may be composed of a CGAN model including a generative adversarial network (GAN) and a convolutional neural network (CNN).

The first deep learning model and the second deep learning model are a virtual dental image generated by taking the pre-orthodontic dental image generated by the adversarial neural network as an input and a real image after orthodontic treatment after a predetermined time has elapsed from the dental image before orthodontic treatment. It may be repeatedly learned until the correlation coefficient with the dental image is equal to or greater than a preset threshold value.

According to another aspect of the present invention, as a deep learning-based orthodontic treatment result prediction method of a device including a processor and a memory, data preprocessing of pre-collected dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment Standardizing by doing; constructing a pair of dental images before and after the standardized orthodontic treatment as a first data set; constructing a pair of dental images before and after the standardized non-extraction orthodontic treatment as a second data set; learning a first deep learning model for predicting an orthodontic treatment result using the first data set; learning a second deep learning model for predicting non-extraction orthodontic treatment results using the second data set; and generating a predictive dental image after extraction or non-extraction orthodontic treatment of a predetermined patient by using the first deep learning model or the second deep learning model that has been learned.

According to another aspect of the present invention, a computer program stored in a computer readable non-transitory recording medium for performing the above orthodontic treatment result prediction is provided.

According to the present invention, it is possible to predict an image (X-ray) after orthodontic treatment based on a tooth image before orthodontic treatment, and through this, an objective basis can be provided when an orthodontist establishes an orthodontic treatment plan, and also It has the advantage of facilitating communication with patients by providing visual data for each treatment plan during patient consultation.

1 is a diagram showing the configuration of a device for predicting a result of orthodontic treatment based on deep learning according to a preferred embodiment of the present invention.

2 is a flowchart of a process for predicting a result of orthodontic treatment according to an embodiment of the present invention.

3 is a diagram illustrating a data pre-processing process according to an embodiment of the present invention.

4 shows dental images before and after adjusting the degree of blackening according to the present embodiment.

5 is a diagram for explaining a process of overlapping dental images before and after orthodontic treatment according to the present embodiment.

6 is a view showing results of extracting a region of interest at the same location in a state in which dental images before and after orthodontic treatment are overlapped.

7 to 8 are views illustrating a process of constructing a data set and learning a deep learning model through dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment.

9 shows the verification results of the first deep learning model for predicting the result of orthodontic extraction according to the present embodiment.

10 shows the verification results of the second deep learning model for predicting non-extraction orthodontic treatment results according to the present embodiment.

11 and 12 show statistical verification of results after orthodontic treatment and after orthodontic treatment by applying a dental image of an actual patient before orthodontic treatment to a trained extraction/non-extraction dental image deep learning model.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The present invention proposes a method for predicting results after orthodontic treatment in case of tooth extraction and non-extraction using deep learning among artificial intelligence techniques.

1 is a diagram showing the configuration of a device for predicting a result of orthodontic treatment based on deep learning according to a preferred embodiment of the present invention.

As shown in FIG. 1 , the device according to the present embodiment may include a processor 100 and a memory 102 .

The processor 100 may include a central processing unit (CPU) capable of executing a computer program or other virtual machines.

Memory 102 may include a non-volatile storage device such as a non-removable hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, and the like. Memory 102 may also include volatile memory, such as various random access memories.

According to one embodiment of the present invention, memory 102 stores program instructions for predicting orthodontic treatment results.

2 is a flowchart of a process for predicting a result of orthodontic treatment according to an embodiment of the present invention.

The process of FIG. 2 is defined as a process performed by program instructions installed in the device according to the present embodiment.

Referring to FIG. 2 , the apparatus according to the present embodiment provides dental images before and after orthodontic treatment of patients with tooth extraction (hereinafter, referred to as 'dental images before orthodontic treatment and dental images after orthodontic treatment') and before and after orthodontic treatment of non-extraction patients. Dental images (hereinafter referred to as 'dental images before non-extraction orthodontic treatment and dental images after non-extraction orthodontic treatment') are collected (step 200).

The dental images collected in step 200 are images captured by a standardized device with a time difference before and after orthodontic treatment for individual patients who have actually performed extraction and non-extraction orthodontic treatment.

For example, the collected dental images are a dental X-ray image before orthodontic treatment and a dental X-ray image after orthodontic treatment of a patient who received comprehensive orthodontic treatment with fixed orthodontic devices attached to the entire upper and lower jaw dentition.

Thereafter, data preprocessing is performed to standardize the collected dental images before and after orthodontic treatment and dental images before and after orthodontic treatment to be suitable for deep learning model learning (step 202).

3 is a diagram illustrating a data pre-processing process according to an embodiment of the present invention.

Referring to FIG. 3 , the apparatus according to the present embodiment adjusts the degree of blackening of the collected dental images (step 300).

In step 300, the degree of blackening is uniformly adjusted so that the region of interest (eg, upper and lower lip soft tissue and upper and lower anterior teeth) is clearly visible.

4 shows dental images before and after adjusting the degree of blackening according to the present embodiment.

Next, for data pre-processing, the device according to the present embodiment uses the structural method for dental images before and after orthodontic treatment, centering on the frontal region with relatively little growth change and bone change during the orthodontic treatment period. Superposition is performed (step 302).

5 is a diagram for explaining a process of overlapping dental images before and after orthodontic treatment according to the present embodiment.

Referring to FIG. 5 , a difference due to a change in head position is compensated for in two dental images before and after orthodontic treatment taken at a time difference through an overlapping process, and images changed by orthodontic treatment are easily recognized.

In FIG. 5, the red circle represents the frontal cranial fossa, which is the criterion for overlapping the stable bone overlapping method.

After overlapping the images, as shown in FIG. 6 , a region of interest at the same location is extracted in a state in which dental images before and after orthodontic treatment are overlapped (step 304).

The region of interest according to the present embodiment includes upper and lower lip soft tissues and anterior teeth to be obtained in the analysis, and a red rectangle in FIG. 6 represents an extracted region of interest.

As described above, after the upper and lower lip soft tissue and anterior teeth are selected as the region of interest, the angle and size of the face are adjusted according to the region of interest to match the irregular facial contour and teeth shown in the dental image before and after orthodontic treatment. (Step 306).

The apparatus according to the present embodiment may perform black-and-white image conversion to represent brightness and darkness of dental images, which may be different from each other, with standardized brightness.

For example, the device according to the present embodiment converts a dental image into an image having standardized contrast using the following equation.

은 n번째 관측된 환자의 치과 영상이고, DN은 치과 영상의 명암을 0 ~ 255 사이로 표현하는 디지털 넘버를 나타낸다.here,

is a dental image of the nth observed patient, and DN represents a digital number representing the contrast of the dental image between 0 and 255.

i and j mean indices indicating left and right and top and bottom positions of pixels in a 256×256 dental image, Max() is a function for obtaining the maximum value, Min() is a function for obtaining the minimum value, and n represents the number of data pairs.

Data pre-processing according to the present embodiment is separately performed for dental images before and after non-extraction orthodontic treatment and dental images before and after orthodontic treatment.

Referring back to FIG. 2 , after the data preprocessing is performed, a first data set is constructed as a pair of a standardized dental image before orthodontic extraction and a dental image after orthodontic treatment, and a standardized dental image before non-extraction orthodontic treatment A second data set is configured with a pair of images and dental images after non-extraction orthodontic treatment (step 204).

)의 치과 영상(B_n)을 데이터 쌍으로 하여 데이터 셋을 구성한다. In step 204, a standardized dental image (A _n ) of a first time (t) before orthodontic treatment for a patient who has undergone orthodontic treatment and a second time point (A n ) for which prediction is desired after a certain time (

) of dental images (B _n ) as data pairs to form a data set.

The data set according to the present embodiment can be expressed by the following equation.

According to this embodiment, standardized dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment are X-ray images, black and white images with a × b resolution, and deep learning model learning. To do this, the number of horizontal and vertical pixels is converted to a multiple of 2 to form a new data set.

For example, the original dental image has a size of 360 × 360, and after data preprocessing, the dental image converted for deep learning learning data is converted to have a size of 256 × 256, which is a multiple of 2. In addition, it is converted so that the horizontal and vertical sizes are the same as the size of a natural number that is a multiple of 2.

When constructing such a data set, dental images under the following conditions are excluded.

(1) Orthognathic surgery or facial soft tissue surgery

(2) Presence of artifacts (metal orthodontic devices, implants, miniplates, miniscrews, metal needles, etc.) on radiographs

(3) Prosthetic treatment or conservative treatment using metal during orthodontic treatment (crown or inlay restoration, amalgam, root canal filling, etc.)

(4) Impacted teeth, remaining mature primary teeth, and non-eupruded permanent teeth

(5) Poor radiographic resolution

7 to 8 are views illustrating a process of constructing a data set and learning a deep learning model through dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment.

In FIG. 7, the input of the deep learning model is the aforementioned first data set, and in FIG. 8, it is the second data set.

Referring back to FIG. 2 , the apparatus according to the present embodiment learns a deep learning model using the first data set and the second data set (step 206).

7 to 8, the deep learning model according to this embodiment is composed of a CGAN model composed of a Generative Adversarial Network (GAN) and a Convolution Neural Network (CNN), each The loss function of the neural network is trained to be minimal.

Hereinafter, the learning process of the deep learning model will be described in detail.

As shown in FIGS. 7 and 8 , the deep learning model according to the present embodiment is a deep learning model for orthodontic treatment (first deep learning model) and a deep learning model for orthodontic treatment without extraction (second deep learning model). ) can be configured independently.

The GAN of each deep learning model includes a generator and a discriminator, and the generator takes a standardized dental image corresponding to a specific time (t) before orthodontic treatment as an input and determines a predetermined time after orthodontic treatment (Δ After t) has elapsed, a virtual dental image is generated.

The discriminator compares the real dental image after orthodontic treatment with the virtual dental image, and repeatedly generates the virtual dental image and compares it with the real dental image until the GAN loss function (L _GAN ) is minimized.

The loss function of GAN can be determined using the equation below.

는 교정치료 이후(

)에 촬영된 실제 치과 영상을 나타내며, G(X)는 교정치료 이후(

)에 예측된 가상 치과 영상을 나타낸다. E represents the expected value, G represents the generator of GAN, and D represents the discriminator of GAN. A ₁ represents a dental image corresponding to a specific time (t) before orthodontic treatment,

after orthodontic treatment (

), and G(X) is after orthodontic treatment (

) represents the predicted virtual dental image.

In addition, the similarity between the virtual dental image and the real dental image can be determined using a CNN model together.

The apparatus according to the present embodiment calculates the loss function of the CNN model using Equation 4.

는 교정치료 이후(

)에 촬영된 실제 치과 영상을 나타내고, G(X)는 가상 치과 영상을 나타낸다. 또한, ∥ ∥는 교정후 실제 치과 영상과 예측된 가상 치과 영상의 픽셀별 거리를 계산하는 함수를 나타낸다. Here, E represents the expected value,

after orthodontic treatment (

) represents a real dental image captured in , and G(X) represents a virtual dental image. In addition, ? ? denotes a function for calculating a pixel-by-pixel distance between a real dental image after correction and a predicted virtual dental image.

The deep learning model according to the present embodiment may be trained to minimize loss functions of the GAN model and the CNN model.

According to the discrimination result of the discriminator, the generator generates a virtual dental image so that the difference from the actual dental image after the actual orthodontic treatment is minimized.

If this is expressed as an equation, it is equivalent to Equation 5.

The apparatus according to the present embodiment may obtain a predicted virtual dental image G(X) with a time difference Δt before and after orthodontic treatment through the deep learning model.

After calculating the degree of similarity by statistically comparing the predicted virtual dental image with the actual dental image, learning of the deep learning model is completed when a preset threshold is reached.

Here, the threshold value is determined when the correlation coefficient is maximum using a correlation coefficient (CC) of a real dental image and a virtual dental image.

The correlation coefficient according to this embodiment is as follows.

Here, CC is a quantitative representation of the relationship between the real dental image and the virtual dental image, and the closer to 1, the higher the prediction accuracy, and the closer to 0, the lower the prediction accuracy.

는 교정치료 후의 시간, o는 실제 치과 영상(예를 들어, 발치 교정치료 또는 비발치 교정치료 후 상당한 시간이 경과한 후의 치과 영상)을 나타내고, f는 학습된 딥러닝 모델을 이용하여 예측된 가상 치과 영상을 나타낸다. and,

is the time after orthodontic treatment, o represents an actual dental image (eg, a dental image after a significant period of time has elapsed after orthodontic treatment or non-extraction orthodontic treatment), and f is a predicted virtual dentist using a trained deep learning model. represents an image.

는 실제 치과 영상의 평균값,

는 가상 치과 영상의 평균값을 나타내며, n은 치과 영상의 전체 화소수를 나타낸다. i is an individual pixel of the dental image,

is the average value of actual dental images,

represents the average value of the virtual dental image, and n represents the total number of pixels in the dental image.

As described above, after learning of the deep learning model is completed, the apparatus according to the present embodiment performs a verification process (step 208).

9 shows the verification results of the first deep learning model for predicting the result of orthodontic extraction according to the present embodiment.

9A is a real dental image before orthodontic treatment, and FIG. 9B is an actual dental image after orthodontic treatment. Referring to FIGS. 9A to 9B , after orthodontic treatment, the patient's upper and lower anterior teeth moved backward, the dentition was changed, and the labial inclination of the upper and lower anterior teeth decreased.

9C is a dental image predicted by the first deep learning model after a lapse of a predetermined time. When this is compared with FIG. 9B , it can be confirmed that the dental image after orthodontic treatment is predicted with very high accuracy.

10 shows the verification results of the second deep learning model for predicting non-extraction orthodontic treatment results according to the present embodiment.

10A is a real dental image before non-extraction orthodontic treatment, FIG. 10B is an actual dental image after non-extraction orthodontic treatment, and FIG. 10C shows a dental image predicted by a second deep learning model, with the deep learning model taking FIG. 10A as an input. This is a predicted dental image after a predetermined time has elapsed.

When comparing FIGS. 10B and 10C , it can be confirmed that the result after non-extraction orthodontic treatment is predicted with very high accuracy.

The apparatus according to the present embodiment predicts results that may appear when orthodontic treatment or non-extraction orthodontic treatment is performed through a deep learning model that has been learned and verified (step 210).

11 and 12 show statistical verification of results after orthodontic treatment and after orthodontic treatment by applying a dental image of an actual patient before orthodontic treatment to a trained extraction/non-extraction dental image deep learning model.

11 and 12 show predictions of expected treatment results through the deep learning model when orthodontic treatment and non-extraction orthodontic treatment are performed, respectively, in a state before orthodontic treatment.

11 and 12 show CC values between an actual dental image after orthodontic treatment and a dental image predicted through the deep learning model.

As shown in the figure, it can be seen that the deep learning model according to the present embodiment shows a very high correlation between the predicted dental image and the actual dental image, and accurate prediction is possible in both extraction and non-extraction orthodontic treatment.

The embodiments of the present invention described above have been disclosed for illustrative purposes, and those skilled in the art having ordinary knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions will be considered to fall within the scope of the following claims.

Claims (10)

As a device for predicting results of orthodontic treatment based on deep learning, processor; and Including a memory connected to the processor, the memory, Data pre-processing and standardization of pre-collected dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment, A pair of dental images before and after the standardized orthodontic treatment is configured as a first data set, Constructing a pair of dental images before and after the standardized non-extraction orthodontic treatment as a second data set, Learning a first deep learning model for predicting orthodontic treatment results using the first data set; Learning a second deep learning model for predicting non-extraction orthodontic treatment results using the second data set; To generate a predictive dental image after tooth extraction or non-extraction orthodontic treatment of a predetermined patient using the first deep learning model or the second deep learning model for which learning has been completed, Orthodontic treatment result prediction device for storing program instructions executed by the processor. According to claim 1, The program instructions are Adjusting the blackness of the collected dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment, overlapping the collected dental images before and after orthodontic treatment and the collected dental images before and after non-extraction orthodontic treatment, respectively, around a preset reference point; Orthodontic treatment result prediction device for extracting a region of interest of each of the collected dental images before and after orthodontic treatment and the dental images before and after non-extraction orthodontic treatment after the overlapping. According to claim 2, The preset reference point is the orthodontic treatment result prediction device, which is the frontal region by the stable bone superimposition method. According to claim 2, The region of interest is an orthodontic treatment result prediction device including upper and lower lip soft tissue and anterior teeth. According to claim 1, The program instructions are Orthodontic treatment result prediction device for converting the contrast of the collected dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment into black and white images to represent standardized brightness. According to claim 1, When constructing the first data set and the second data set, orthognathic surgery or facial soft tissue surgery, presence of artifacts on radiographs, prosthetic treatment or conservative treatment using metal objects during orthodontic treatment, impacted teeth, remaining at maturity A device for predicting orthodontic treatment results excluding deciduous teeth, unerupted permanent teeth, and dental images with poor radiographic resolution. According to claim 1, The first deep learning model and the second deep learning model are composed of a CGAN model including a generative adversarial network (GAN) and a convolutional neural network (CNN) Orthodontic treatment result prediction device. According to claim 7, The first deep learning model and the second deep learning model are a virtual dental image generated by taking the pre-orthodontic dental image generated by the adversarial neural network as an input and a real image after orthodontic treatment after a predetermined time has elapsed from the dental image before orthodontic treatment. An orthodontic treatment result prediction device that is repeatedly learned until a correlation coefficient with a dental image is equal to or greater than a preset threshold. A deep learning-based orthodontic treatment result prediction method of a device including a processor and a memory, Data pre-processing and standardization of pre-collected dental images before and after orthodontic treatment and dental images before and after non-extraction orthodontic treatment; constructing a pair of dental images before and after the standardized orthodontic treatment as a first data set; constructing a pair of dental images before and after the standardized non-extraction orthodontic treatment as a second data set; learning a first deep learning model for predicting an orthodontic treatment result using the first data set; learning a second deep learning model for predicting non-extraction orthodontic treatment results using the second data set; and Orthodontic treatment result prediction method comprising the step of generating a predicted dental image after extraction or non-extraction orthodontic treatment of a predetermined patient by using the first deep learning model or the second deep learning model for which learning has been completed. A computer program stored in a computer readable non-transitory recording medium that performs the orthodontic treatment result prediction according to claim 9.

Images