WO2023018259A1 - Diagnosis method and apparatus for remotely diagnosing skin disease by using augmented reality and virtual reality - Google Patents

Abstract

A diagnosis method and apparatus for remotely diagnosing a skin disease by using augmented reality and virtual reality are disclosed. The diagnosis method performed on a server to remotely diagnose a skin disease may comprise the steps of: receiving 3D data and image data on a skin lesion site of a subject from a terminal of the subject; generating a 3D content for the skin lesion site, on the basis of the received 3D data and image data; transmitting the generated 3D content to a terminal of a diagnostician; receiving diagnosis data on the skin lesion site from the terminal of the diagnostician; and transmitting the diagnostic data to the terminal of the subject.

Description

Diagnosis method and apparatus for remotely diagnosing skin diseases using augmented reality and virtual reality

The embodiments below relate to a technology for remotely diagnosing a skin disease using augmented reality and virtual reality.

In the case of epidemics like these days, face-to-face treatment is difficult in dermatology, where promotion is important. Accordingly, the demand for non-face-to-face treatment is increasing due to the characteristics of dermatology treatment that does not require expensive equipment such as X-Ray or CT (computerized tomography).

Referring to the case of dermatology treatment in countries where telemedicine is already allowed, when a user (patient) takes a picture with a camera and uploads it through a dermatology website or mobile phone application, the doctor analyzes the picture and diagnoses it. are losing There are many difficulties in making an accurate diagnosis because the analysis of lesions through photographs varies greatly depending on the shooting environment or the type of camera. In addition, since the diagnosis result is delivered in writing, it is difficult for the user to feel that he or she has actually received treatment. Therefore, despite the remote medical treatment, there is a need for research to improve the accuracy of diagnosis and to ensure proper diagnosis.

According to an embodiment, a diagnosis method for remotely diagnosing a skin disease performed in a server includes receiving three-dimensional data and image data of a skin lesion of a subject from a terminal of a subject; generating 3D contents (three dimensional contents) for the skin lesion area based on the received 3D data and image data; Transmitting the generated 3D content to a terminal of a diagnoser; receiving diagnosis data on the skin lesion site from a terminal of the diagnoser; and transmitting the diagnostic data to a terminal of the subject.

A server according to an embodiment includes a memory and a processor, the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor causes the server to: Receives 3D data and image data for the skin lesion of the subject from a terminal of the subject, generates 3D content for the skin lesion based on the received 3D data and image data, and generates the 3D content The server may be controlled to transmit to the terminal of the diagnoser, receive diagnosis data on the skin lesion from the terminal of the diagnoser, and transmit the diagnosis data to the terminal of the subject.

According to an embodiment, a terminal of a subject includes a memory and a processor, the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the processor stores the instructions of the subject A terminal photographs the subject's skin lesion area using a camera provided in the subject's terminal, and transmits 3D data and image data of the subject's skin lesion area where the subject's skin lesion area was captured. Transmission to a server, receiving diagnostic data from the server in response to the transmission, and controlling the terminal of the subject to output the diagnostic data.

1 is a diagram showing an overview of a diagnostic system according to an embodiment.

2 is a flowchart illustrating a diagnosis method performed in a server according to an exemplary embodiment.

3 is a flowchart illustrating a diagnosis method performed in a terminal of a subject according to an exemplary embodiment.

4 is a flowchart illustrating a diagnosis method performed between entities according to an exemplary embodiment.

5 is a diagram illustrating a configuration of a server according to an embodiment.

6 is a diagram illustrating a configuration of a terminal of a subject according to an embodiment.

7 is a diagram for explaining a process of obtaining 3D data on a skin lesion site according to an embodiment.

8 is a diagram for explaining providing tracking result information on a skin lesion based on virtual reality according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

1 is a diagram showing an overview of a diagnostic system according to an embodiment.

The diagnosis system for remotely diagnosing skin diseases described herein may provide a remote medical treatment system based on augmented reality (AR) and/or virtual reality (VR) for patients with skin diseases. In one embodiment, the diagnosis system converts images collected by a subject's terminal into 3D contents (three dimensional contents) in a server, transmits the 3D contents to a diagnoser's terminal, and based on diagnostic data input to the diagnoser's terminal Diagnosis results and progress of the subject's skin lesions can be provided. The diagnosis system generates 3D content based on image data and 3D data of a subject's skin lesion area collected through a terminal of a subject (or patient, treated person, examinee, and diagnosed person), and generates 3D content. It can be transmitted to the terminal of the diagnoser (or doctor, medical practitioner, and medical staff). Here, the terminal of the subject may be, for example, a mobile terminal (eg, a smart phone, tablet PC, laptop, or PDA) possessed by the subject, and the terminal of the diagnoser may be AR glasses or a head mounted display (HMD).

The diagnosis system described in this specification replaces the diagnosis process conventionally performed when visiting a dermatologist with a mobile device application and an AR/VR program, so that patients can be diagnosed regardless of place and time, and skin implemented in three dimensions. A more accurate diagnosis result may be provided to the subject based on the 3D contents including the lesion.

By observing skin lesions based on virtual reality or augmented reality, doctors can implement a remote medical care environment close to reality, improve the accuracy of medical treatment, and improve patient satisfaction.

Referring to FIG. 1 , the subject may photograph the subject's skin lesion area 115 through the subject's terminal 110 . In this embodiment, it is assumed that the skin lesion site 115 is, for example, an arm with a skin lesion, but the scope of the embodiment is not limited thereto. The terminal 110 of the subject may photograph and scan the skin lesion area 115 and transmit skin lesion data of the result of the photographing and scanning to the server 120 . The terminal 110 of the subject may generate image data and 3D data through photographing and scanning of the skin lesion site 115, and transmit the skin lesion data including the image data and 3D data to the server 120. . The image data may include texture data of the skin lesion, and the 3D data may include geometric data of the skin lesion site 115 including the skin lesion. In one embodiment, the image data may be obtained by a color camera provided in the terminal 110 of the subject, and the 3D data may be acquired by a lidar sensor and/or a depth sensor provided in the terminal 110 of the subject. (e.g. ToF sensor). The subject's terminal 110 may generate 3D data of the skin lesion area 115 based on virtual reality technology and provide it to the diagnoser's terminal 130 through the server 120 .

FIG. 7 is a diagram for explaining a process of obtaining 3D data on a skin lesion site according to an embodiment. Referring to FIG. An example of image data 710 including texture data and 3D data 720 obtained by reconstructing the skin lesion area 115 with a 3D mesh and texture is shown. The terminal 110 of the subject may separate and store 3D mesh data and texture data, and transmit them to the server 120 .

Returning to FIG. 1 , when acquiring skin lesion data, the subject's terminal 110 may provide the subject with a shooting guide for obtaining 3D data for each body part. In one embodiment, body parts to be diagnosed for skin diseases may be divided into a preset number (eg, 35), and the subject may select a body part for skin lesion diagnosis. Each body part may have its own skeletal structure data and mesh preset data for its own body part. The subject's terminal 110 may provide a shooting guide for the skin lesion site 115 based on the skeletal structure data and mesh preset data. In addition, the subject's terminal 110 may provide various operation modes (eg, proximity scan mode, long distance scan mode) for scan resolution in relation to acquisition of 3D data. The terminal 110 of the subject removes the background area except for the skin lesion area 115 from the acquired skin lesion data, adjusts at least one of contrast/saturation/color, and crops the skin lesion area 115 Alternatively, image preprocessing such as warping may be performed.

The server 120 may generate 3D content based on the image data and 3D data, and transmit the generated 3D content to the terminal 130 worn by the diagnoser. The server 120 may reconstruct the subject's skin lesion into 3D content based on the image data and 3D data received from the subject's terminal 110 and analyze the skin lesion. The server 120 may extract the subject's skin lesion area through image analysis. The server 120 detects three-dimensional irregularities such as swelling and protrusion on the skin surface based on the texture of the skin lesion site 115 shown in the image data and the three-dimensional shape of the skin lesion site 115 shown in the three-dimensional data. can be extracted. The server 120 may distinguish skin lesions by using differences in tone and pattern between the skin lesion area and the surrounding normal skin area. The server 120 may distinguish a skin lesion area based on a vector difference from a bone center to a mesh point, a coordinate position gap difference between neighboring meshes, and the like.

In some embodiments, image data and 3D data are transferred from the subject's terminal 110 to the diagnoser's terminal 130 without passing through the server 120, and the image data and the 3D data are transferred from the diagnoser's terminal 130. 3D content may be generated based on.

The diagnoser's terminal 130 may output the received 3D content, receive an input from the diagnoser, and generate diagnostic data. The diagnosis person's terminal 130 may provide 3D content to the diagnosis person through a program or application capable of reproducing and controlling augmented reality content and/or virtual reality content. The diagnoser's terminal 130 can realistically provide the diagnoser with 3D content on the subject's skin lesion through a VR device or an AR device, and diagnose based on the augmented three-dimensional skin lesion through the 3D content. can be done The diagnoser's terminal 130 may provide a function for the diagnoser to observe a skin lesion in a three-dimensional form at 360 degrees and a function for enlarging/rotating/moving/extracting the skin lesion area. Through these functions, the diagnoser can observe the subject's 3D lesion in detail. In one embodiment, the diagnostician may select and control functions applicable to the 3D lesion through a separate controller or gesture recognition.

A diagnoser may input diagnosis data based on a diagnosis result. The diagnosis data may include diagnosis contents of a subject's skin lesion. The terminal 130 of the diagnoser may transmit diagnosis data to the server 120 . For example, a diagnoser may diagnose a skin lesion and input candidate disease information for the skin lesion and information on the progress of the skin lesion, and the input information may be transmitted to the server 120 as diagnosis data. Diagnosis data may include, for example, 3D augmented data that can be simulated with the name of a skin lesion, severity, and stage of progression of the skin lesion (eg, remission, progress, exacerbation, and cure).

The server 120 may transmit diagnostic data to the terminal 110 of the subject, and the terminal 110 of the subject may output the received diagnostic data. In some embodiments, diagnostic data may be transmitted from the terminal 130 of the diagnoser to the terminal 110 of the subject without going through the server 120 . Based on the diagnosis data, the terminal 110 of the subject may provide detailed information on the skin disease to the terminal 110 of the subject through augmented reality technology. For example, the subject's terminal 110 may provide the subject with candidate disease information on skin lesions or information on the progression of skin lesions to the subject through augmented reality technology.

8 is a diagram for explaining the provision of tracking result information on skin lesions based on virtual reality according to an embodiment. Referring to FIG. 8 , 3D augmentation provided to the subject through the terminal 110 of the subject An example of lesion data is shown. The 3D augmented lesion data 810 distinguishes and displays the actual skin lesion 812 from the actual skin lesion site 115 of the user, and displays information related to the skin lesion 812 (eg, diagnosis date, skin disease diagnosis name, skin A lesion size (width) 814 may be displayed in an area adjacent to the skin lesion 812 . In addition, the terminal 110 of the subject will provide the subject with 3D augmented lesion data 820 including information 824 on the progression of the skin lesion 822 after time has passed based on augmented reality technology. may be In this way, the diagnosis system may provide 3D augmented lesion data to a subject, and the subject may predict future progression of the lesion based on the 3D augmented lesion data and perform self-monitoring. The terminal 110 of the subject may analyze the 3-dimensional change caused by the normal skin condition and the skin disease and provide the analysis result to the subject. According to the embodiment, when the subject's terminal 110 provides 3D augmented lesion data, to change the size and shape of the augmented content according to the distance and positional relationship between the subject's terminal 110 and the subject's lesion body part. A subject may photograph a reference object (eg, coin) or an identification mark (eg, a predefined QR code) along with the skin lesion area 115 . The terminal 110 of the subject may estimate the size and location of the skin lesion based on the size or shape of the identified reference object/identification mark.

Returning to FIG. 1 , the subject's terminal 110 may provide the subject with a preliminary diagnosis result of a skin lesion and a comparative analysis result of previous diagnosis data based on the received diagnosis data. In some embodiments, a provisional diagnosis result for a skin lesion may be diagnosed by a diagnoser or may be determined by a skin disease diagnosis algorithm operated in a separate skin disease diagnosis server. The skin disease diagnosis server may be included in the server 120 and operated according to an embodiment. In one embodiment, when it is determined that the symptoms of the skin lesion are getting worse, the terminal 110 of the subject may recommend a visit to a hospital to the subject or provide a guide for inducing proper treatment. When it is determined that the symptoms of the skin lesion are gradually alleviated, the terminal 110 of the subject may provide the subject with an expected image or 3D augmented data of a state when the subject is completely cured.

As described above, in the diagnosis system, the subject's terminal 110, the server 120, and the diagnoser's terminal 130 may provide face-to-face diagnosis and a high-quality remote medical treatment environment close to face-to-face treatment. The terminal 110 of the subject may provide the subject with augmented data on the skin lesion, synthesize the received augmented data and the subject's skin, and provide the subject with a predicted result of a change in the skin lesion in the future. In addition, the diagnoser's terminal 130 enables a more accurate diagnosis of skin lesions during remote medical treatment by providing realistic three-dimensional data on the subject's skin lesions to the diagnoser.

2 is a flowchart illustrating a diagnosis method performed in a server according to an exemplary embodiment.

Referring to FIG. 2 , in step 210, a server (eg, the server 120 of FIG. 1 ) obtains a 3D image of the subject's skin lesion from the subject's terminal (eg, the subject's terminal 110 of FIG. 1 ). Data and image data can be received. The terminal of the subject generates 3D data by 3D scanning the affected area and skin lesion using at least one of a Lidar, a color sensor, and a depth camera provided in the terminal of the subject. can do. Here, the skin lesion site may be a site including the skin lesion, and the 3D data may be a 3D image or image including the skin lesion site. The image data may be a two-dimensional image or image including a skin lesion site.

The server may generate 3D content for the skin lesion area based on the 3D data and image data received in step 220 . The server may create 3D content for the skin lesion by constructing a 3D mesh and texture. The 3D content is content generated by analyzing and reconstructing skin lesions of a subject included in image data and 3D data, and may be at least one of virtual reality content and augmented reality content. Based on the 3D data and the image data, the server may distinguish a normal skin region and a lesional skin region from the skin lesion region included in the 3D data and the image data. For example, the server extracts areas where 3D irregularities such as swelling and protrusions are included in the skin lesion area, and based on the difference in tone and pattern between the areas where 3D irregularities appear and the normal skin area. Thus, a normal skin area and a lesioned skin area can be distinguished from the skin lesion area included in the 3D data and the image data.

The server may also amplify the 3-dimensional irregularity of the skin surface, the difference in color tone, and the 3-dimensional mesh on the data of the normal skin region and the lesion skin region. For example, the server may generate 3D content in which the normal skin area and the lesioned skin area are clearly separated by amplifying the difference in brightness between the normal skin area and the lesioned skin area.

The server may transmit the 3D content generated in step 230 to the diagnoser's terminal (eg, the diagnoser's terminal 130 of FIG. 1 ). After receiving the 3D contents from the server, the terminal of the diagnoser may output the 3D contents, receive the diagnoser's input based on the 3D contents, generate diagnostic data based on the diagnostician's input, and transmit the diagnostic data to the server. there is. For example, the diagnostician's terminal may output 3D content through at least one of a display based on augmented reality and a display based on virtual reality. The diagnoser may diagnose a subject's skin disease based on the 3D content output based on the augmented reality environment or the virtual reality environment, and input the diagnoser's input to the diagnoser's terminal based on the diagnosed content.

In step 240, the server may receive diagnostic data on the skin lesion site from the diagnoser's terminal. In step 250, the server may transmit diagnostic data to the subject's terminal. The subject's terminal may output at least one of information about the candidate disease name of the skin lesion, the candidate cause of the skin lesion, the severity of the skin lesion, and the candidate disease of the skin lesion included in the diagnostic data received from the server. Here, candidate disease names may include disease names that may correspond to skin lesions of the subject, and candidate causes of skin lesions may include candidates that may cause skin lesions. Information on the candidate disease of the skin lesion may include at least one of symptoms and treatment methods of the candidate disease.

The server may generate progress data of skin lesions based on the 3D content. The server may transmit progress data of the skin lesion to the terminal of the subject. The subject's terminal may receive progress data of the skin lesion and output the received progress data of the skin lesion. The progress data of the skin lesion may be generated based on 3D data and image data of the subject's skin lesion area accumulated in the server, that is, received by the server from the subject's terminal in the meantime. The progress data of the skin lesion may indicate a state in which the subject's skin lesion has changed in the meantime. The progress data may include three-dimensional data simulated for each stage of the skin lesion of the current subject by dividing the skin lesion into stages of cure, alleviation, progress, and exacerbation. The subject's terminal may provide the progress of each stage of the skin lesion by overlapping and outputting each image data in augmented reality on the skin lesion site on the screen capturing the skin lesion site.

3 is a flowchart illustrating a diagnosis method performed in a terminal of a subject according to an exemplary embodiment.

Referring to FIG. 3 , the subject's terminal (eg, the subject's terminal 110 of FIG. 1 ) may take a picture of the subject's skin lesion in step 310 using a camera provided in the subject's terminal. The subject's terminal may output a screen for receiving a selection for the subject's skin lesion site and may receive a selection for the subject's skin lesion site. A screen for receiving a selection of a skin lesion site of the subject may include, for example, an image in which a body part is divided based on a predetermined criterion. The subject may select and input a body part corresponding to the subject's skin lesion part from among the divided body part images based on a predetermined criterion by using at least one method of touching or clicking. The subject's terminal may capture a skin lesion site of the subject corresponding to the selection of the subject's skin lesion site. In addition, the terminal of the subject may output a guideline for photographing the subject's skin lesion area after receiving a selection input for the subject's skin lesion area. For example, in order to collect image data and 3D data suitable for generating 3D content, the subject's terminal may output a guideline providing a guide for the subject to photograph a skin lesion site. The subject's terminal may photograph the subject's skin lesion area in a proximity mode and a perspective mode. In the close-up mode, skin lesions are photographed within 10 cm (cm), and in the perspective mode, the overall shape of the body is photographed. In the close-up mode, the morphological changes of the skin surface can be photographed more closely. Image data and 3D data captured in the proximity mode may include a larger amount of data than image data and 3D data captured in the perspective mode based on mesh points. Proximity mode prevents image data and 3D data from including unnecessary parts such as the background except for skin lesions, and prevents the size of 3D content from becoming excessively large by setting presets for texture composition. It can be prevented. In order to intuitively provide the subject with the shooting progress, the subject's terminal may distinguish and output a part where the shooting is completed and a part where the shooting is not completed based on augmented reality.

Depending on the embodiment, the subject's terminal may generate basic medical examination data on skin lesions. The subject's terminal may receive an input of the subject for medical examination, and may generate medical examination data based on the subject's input. In this case, the subject's terminal may also transmit medical examination data to the server, and the server may also transmit medical examination data to the diagnoser's terminal. The diagnoser's terminal can output medical examination data together with 3D contents.

In step 320, the subject's terminal may transmit 3D data and image data of the subject's skin lesion, which are captured, to a server (eg, the server 120 of FIG. 1). The server may generate 3D content based on the 3D data and image data and transmit the 3D content to the terminal of the diagnoser (eg, the terminal 130 of the diagnoser of FIG. 1 ). The diagnoser's terminal may generate diagnostic data based on the 3D content and transmit it to the server. The server may transmit diagnostic data to the subject's terminal, and the subject's terminal may receive the diagnostic data from the server in step 330 in response to transmission of the 3D data and image data. Here, the diagnosis data may include at least one of a name of a candidate disease of the skin lesion, a candidate cause of the skin lesion, severity of the skin lesion, information on the candidate disease of the skin lesion, progress of the skin lesion, and expected course of the skin lesion. In addition, the diagnosis data may include a difference between a normal skin condition and a skin lesion of the subject by enlarging a site where at least one symptom of swelling or protrusion on the skin surface appears.

In step 340, the subject's terminal may output diagnostic data. The subject's terminal may also receive progress data of the skin lesion from the server and output the progress data of the skin lesion. The progress data of the skin lesion may be output by overlapping or enumerating accumulated 3D contents for the subject's skin lesion. In addition, the subject's terminal may synthesize and output progress data to the subject's skin lesion site. The subject may check the change and progress of the subject's skin lesion in at least one of 2D and 3D based on the output progress data of the skin lesion. The progress data of the skin lesion may be at least one of augmented reality content and virtual reality content.

4 is a flowchart illustrating a diagnosis method performed between entities according to an exemplary embodiment.

Referring to FIG. 4 , the subject's terminal 405 (eg, the subject's terminal 110 of FIG. 1 ) may photograph the subject's skin lesion in step 420 . The terminal 405 of the subject may generate 3D data and image data for the skin lesion area by photographing the area including the affected area. In step 425, the subject's terminal 405 may transmit 3D data and image data about the skin lesion site to the server 410 (eg, the server 120 of FIG. 1). The server 410 may generate 3D content based on the 3D data and image data in step 430 . In step 435, the server 410 may transmit 3D content to the diagnoser's terminal 415 (eg, the diagnoser's terminal 130 of FIG. 1). The diagnoser's terminal 415 may output 3D content in step 440 . The diagnostician's terminal 415 is an HMD and may be a smart glasses (eg, AR glasses) or a virtual reality (VR) device. The diagnoser's terminal 415 may output 3D content in augmented reality or virtual reality through an augmented reality display or a virtual reality display. The output 3D contents may be rotated, moved, or enlarged on the 3D contents by manipulation of the examiner, or a filter for emphasizing the skin lesions may be applied.

In step 445, the diagnostician's terminal 415 may generate diagnostic data. The diagnosing user's terminal 415 may receive a diagnosing user's input for diagnostic data through a built-in or external input interface of the diagnosing user's terminal 415 and generate diagnostic data based on the diagnosing user's input. For example, the terminal 415 of the diagnoser may recognize at least one of a gesture, movement, and motion of the diagnoser through a controller or a camera to receive an input of the diagnoser, a computing device connected to the terminal 415 of the diagnoser, and a smart device. The diagnostician's input may be received through at least one of a device, a microphone, and a physical input device.

In step 450 , the diagnoser's terminal 415 may transmit diagnostic data to the server 410 . Server 410 may generate progress data in step 455 . The progress data may include accumulated data of 3D contents generated during the period in which the subject was diagnosed, and data predicting changes in skin lesions in the future. In step 460, the server 410 may transmit progress data and diagnosis data to the terminal 405 of the subject. The terminal 405 of the subject may output diagnosis data and progress data in step 465 . The subject's terminal 405 may output diagnosis data and progress data to at least one of a 2D-based display and a 3D-based display. The subject's terminal 405 may output diagnosis data and progress data through an augmented reality or virtual reality-based display. The subject's terminal 405 augments progress data on the subject's actual skin lesion so that the subject can observe the course and progress of the skin lesion more reliably.

The order shown in FIG. 4 is not limited to the shown order and may be performed in a different order or in parallel. Depending on the embodiment, the subject's terminal 405 and the diagnoser's terminal 415 may provide a video medical treatment function to each other.

5 is a diagram illustrating a configuration of a server according to an embodiment.

Referring to FIG. 5 , a server 500 (eg, the server 120 of FIG. 1 ) may correspond to the server described herein. The server 500 may include a communicator 510 , a processor 520 and a memory 530 . Also, according to embodiments, the server 500 may include a database 540 .

The memory 530 is connected to the processor 520 and may store instructions executable by the processor 520, data to be calculated by the processor 520, or data processed by the processor 520. Memory 530 may include non-transitory computer readable media such as high-speed random access memory and/or non-volatile computer readable storage media (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid state memory devices). can include

The communicator 510 provides an interface for communication with external devices (eg, a terminal of a diagnoser and a terminal of a subject). The database 540 may store data necessary for the server 500 to perform a diagnosis method. For example, the database 540 may store image data and 3D data received from the subject's terminal. In addition, the database 540 may store 3D content generated based on image data and 3D data, accumulate 3D content, and store progress data predicting future skin lesions.

The processor 520 may control the server 500 so that the server 500 may perform one or more operations related to the operations of the server described herein. For example, the processor 520 may allow the server 500 to receive 3D data and image data of the subject's skin lesion from the subject's terminal, and based on the received 3D data and image data, determine the skin lesion. The server 500 may be controlled to generate 3D content for the user. The processor 520 causes the server 500 to distinguish a normal skin area from a lesion skin area from the skin lesion area included in the 3D data and the image data, based on the 3D data and the image data. You can control it. In addition, the processor 520 transmits the 3D content generated by the server 500 to the terminal of the examiner, receives diagnostic data on the skin lesion from the terminal of the examiner, and transmits the diagnostic data to the terminal of the subject ( 500) can be controlled.

6 is a diagram illustrating a configuration of a terminal of a subject according to an embodiment.

Referring to FIG. 6 , the subject's terminal 600 (eg, the subject's terminal 110 of FIG. 1 ) may correspond to the subject's terminal described herein. The subject's terminal 600 may include a processor 610 , a memory 620 and a communicator 630 . Also, according to the embodiment, the terminal 600 of the subject may further include a sensor 640 and a display 650 .

The memory 620 is connected to the processor 610 and may store instructions executable by the processor 610 , data to be calculated by the processor 610 , or data processed by the processor 610 . Memory 620 may include non-transitory computer readable media such as high-speed random access memory and/or non-volatile computer readable storage media (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid state memory devices). can include

The communicator 630 provides an interface for communication with an external device (eg, a server). The display 650 may output diagnostic data and progress data. The display 650 may include a display for outputting a 2D-based video or image, and a display for outputting an augmented reality, virtual reality, or 3D-based video or image.

The sensor 640 may capture the subject's skin lesion area. The sensor 640 may include, for example, at least one of a Lidar, a color sensor, and a depth camera.

The processor 610 may control the terminal 600 of the subject so that the terminal 600 of the subject may perform one or more operations related to the operation of the terminal of the subject described herein. For example, the processor 610 may control the terminal 600 of the subject so that the terminal 600 of the subject uses a camera provided in the terminal of the subject to capture a skin lesion site of the subject. The processor 610 controls the subject's terminal 600 so that the subject's terminal 600 receives a selection for the subject's skin lesion area and photographs the subject's skin lesion area corresponding to the subject's skin lesion area selection. You can control it.

The processor 610 transmits 3-dimensional data and image data of the subject's skin lesion site, in which the subject's terminal 600 has been photographed, to the server, and receives diagnostic data from the server in response to the transmission. and control the terminal 600 of the subject to output diagnostic data. The processor 610 may control the terminal 600 of the subject so that the terminal 600 of the subject receives progress data of the skin lesion from the server and outputs progress data of the skin lesion.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. may be permanently or temporarily embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. may be Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (13)

In the diagnosis method of remotely diagnosing a skin disease performed in a server, Receiving three-dimensional data and image data of a skin lesion of the subject from a terminal of the subject; generating 3D contents (3 dimensional contents) for the skin lesion area based on the received 3D data and image data; Transmitting the generated 3D content to a terminal of a diagnoser; receiving diagnosis data on the skin lesion site from a terminal of the diagnoser; and Transmitting the diagnostic data to the subject's terminal A diagnostic method comprising a. According to claim 1, The generating step is Based on the 3D data and image data, distinguishing a normal skin region and a lesional skin region from the skin lesion region included in the 3D data and image data A diagnostic method comprising a. According to claim 1, The target person's terminal, Diagnosis of generating the 3D data by scanning the skin lesion in 3D using at least one of a Lidar, a color sensor, and a depth camera provided in the terminal of the subject. method. According to claim 1, The 3D content, At least one of virtual reality content and augmented reality content; The terminal of the diagnoser, After receiving the 3D content from the server, outputting the 3D content, receiving an input of a diagnoser based on the 3D content, generating diagnostic data based on the input of the diagnoser, and transmitting the diagnostic data to the server sent to, diagnostic method. According to claim 1, The target person's terminal, Outputting at least one of information about a candidate disease name of a skin lesion, a candidate cause of the skin lesion, a severity of the skin lesion, and a candidate disease of the skin lesion included in the diagnostic data received from the server. According to claim 1, generating progress data of skin lesions based on the 3D content; and Transmitting progress data of the skin lesion to a terminal of the subject A diagnostic method further comprising. in the server, including memory and processor; the memory stores instructions executable by the processor; When the instructions are executed by the processor, the processor causes the server to: Receiving three-dimensional data and image data of the subject's skin lesion from the subject's terminal; Based on the received 3D data and image data, 3D content for the skin lesion is generated; Transmitting the generated 3D content to a terminal of a diagnoser; Receiving diagnostic data on the skin lesion site from the terminal of the diagnoser; A server for controlling the server to transmit the diagnostic data to the terminal of the subject. According to claim 7, The processor is the server, Based on the 3-dimensional data and image data, the server controls the server to distinguish a normal skin region and a lesion skin region from the skin lesion region included in the 3-dimensional data and image data. According to claim 7, The terminal of the diagnoser, After receiving the 3D content from the server, outputting the 3D content, receiving an input of a diagnoser based on the 3D content, generating diagnostic data based on the input of the diagnoser, and transmitting the diagnostic data to the server to the server. In the terminal of the subject, including memory and processor; the memory stores instructions executable by the processor; When the instructions are executed by the processor, the processor determines that the terminal of the subject, Using a camera provided in the terminal of the subject, photographing the subject's skin lesion area, Transmitting three-dimensional data and image data of the subject's skin lesion, in which the subject's skin lesion is photographed, to a server; receiving diagnostic data from the server in response to the transmission; A terminal of the subject for controlling the terminal of the subject to output the diagnostic data. According to claim 10, The processor is the terminal of the subject, Receiving a selection for the skin lesion site of the subject, The subject's terminal for controlling the subject's terminal to photograph the subject's skin lesion area corresponding to the selection of the subject's skin lesion area. According to claim 10, The diagnostic data, At least one of the candidate disease name of the skin lesion, the candidate cause of the skin lesion, the severity of the skin lesion, information on the candidate disease of the skin lesion, the progression of the skin lesion, and the expected course of the skin lesion, the subject's terminal. According to claim 10, The processor is the terminal of the subject, Receiving progress data of skin lesions from the server; A terminal of the subject for controlling the terminal of the subject to output progress data of the skin lesion.

Images