WO2020091385A1 - Virtual reality device for medical training including microsurgery and injection - Google Patents

Abstract

The present invention relates to a virtual reality device and method for medical training. To this end, provided are: a reference plane data generation step of generating phantom reference plane data on the basis of phantom markers formed on a phantom in real image data and generating medical appliance reference plane data on the basis of medical appliance markers formed on a medical appliance; a matching step of generating matching data by matching the phantom reference plane data and the medical appliance reference plane data to each other; a rendering step of generating and outputting phantom rendering data, which is 3D rendering data for the phantom, and medical appliance rendering data, which is 3D rendering data for the medical appliance, on the basis of the phantom reference plane data, the medical appliance reference plane data, and the matching data; and a medical appliance movement step of updating the medical appliance reference plane data according to the motion of the medical appliance and generating and outputting medical appliance rendering data in reference to the updated medical appliance reference plane data.

Description

Medical training virtual reality device including microsurgery and injection

The present invention relates to a medical training virtual reality device and method, including microsurgery and injection.

The surgical time required to become a skilled surgeon amounts to 1.5 million hours. However, due to the resident's statutory working hours, sufficient training of up to 150,000 hours is physically impossible. Residents are limited to 80 hours per week in the United States, 80 hours per week in the United States, and 48 hours per week in the EU in the United States. Especially in the case of the first year of residency, the opportunity to participate in the surgery is gradually disappearing.

The lack of opportunities for these residents to participate in surgery can be directly linked to medical accidents. Especially in the case of microsurgery, this tendency is high. Microsurgery is mainly performed in ophthalmology such as cataracts, otolaryngology such as mastoid surgery, and cosmetic plastic surgery such as skin / urinary / plastic surgery. Most of these microsurgery are more likely to result in medical accidents, as there is little opportunity to participate in surgical operations during the major. For example, the fact that 40% of domestic medical accidents are concentrated in dermatology, urinary surgery, and plastic surgery proves this.

Accordingly, simulation-based microsurgical training and injection training were required. According to the simulation-based microsurgery training and injection training, it is possible to repeatedly perform professional surgical training without the risk of a human accident, and to solve the problem of insufficient professional educators through numerical objective evaluation.

However, such a simulation-based microsurgical training system has not been able to provide a surgical environment close to reality, including injection simulation, and accordingly, simulation of microsurgery and injection is progressing only to a system in which image and reality are not matched. .

Accordingly, an object of the present invention is a virtual training device and method for medical training, including microsurgery and injection, in which virtual reality and phantom / medical device reality are precisely matched in a simulation of microsurgery or injection to provide a realistic user experience. To provide.

Hereinafter, specific means for achieving the object of the present invention will be described.

An object of the present invention, a memory module for storing a program code configured to receive real image data of a phantom and a medical device for simulation of microsurgery and output virtual reality image data of microsurgery; And a processing module configured to output the virtual reality image data by using the real image data, which is pose data of the second image data input by processing the program code. The program code includes: a reference plane generating phantom reference plane data based on the phantom marker configured on the phantom in the real image data, and generating medical instrument reference plane data based on the medical instrument marker configured on the medical instrument. Data generation step; A matching step of generating matching data by matching the phantom reference surface data and the medical device reference surface data; A rendering step of generating and outputting phantom rendering data, which is 3D rendering data for the phantom, and medical equipment rendering data, which is 3D rendering data for the medical device, based on the phantom reference surface data, the medical device reference surface data, and the registration data; A medical device movement step of updating the medical device reference surface data according to the movement of the medical device, and generating and outputting the medical device rendering data based on the updated medical device reference surface data; And an evaluation step of evaluating the movement of the medical device according to a predetermined curriculum and generating and outputting evaluation data; and configured to be performed on a computer, and outputting the virtual reality image data at a specific magnification. Characterized in that the output, it can be achieved by providing a medical training virtual reality device.

Another object of the present invention is a medical training virtual reality device configured to receive real image data of a phantom and a medical instrument for microsurgery simulation and to output virtual reality image data of microsurgery, from the real image data. A reference plane data generating step of generating phantom reference plane data based on the phantom marker configured on the phantom and generating medical instrument reference plane data based on the medical instrument marker configured on the medical instrument; A matching step in which the medical training virtual reality device matches the phantom reference surface data and the medical device reference surface data to generate matching data; The medical training virtual reality device, 3D rendering data for the phantom based on the phantom reference surface data, the medical device reference surface data and the matching data, and medical device rendering data as 3D rendering data for the medical device A rendering step of generating and outputting; A medical device movement step of the medical training virtual reality device updating the medical device reference surface data according to the movement of the medical device, and generating and outputting the medical device rendering data based on the updated medical device reference surface data; And an evaluation step in which the medical training virtual reality device evaluates the movement of the medical device according to a predetermined curriculum and generates and outputs evaluation data, and is configured to be performed on a computer, and outputs the virtual reality image data. It can be achieved by providing a microsurgical virtual reality method, characterized in that the output as a microscope image enlarged at a specific magnification.

An object of the present invention, a memory module for storing a program code configured to receive real image data of a phantom and a medical device for injection simulation and output virtual reality image data of the injection; And a processing module configured to output the virtual reality image data by using the real image data, which is pose data of the second image data input by processing the program code. The program code includes: a reference plane generating phantom reference plane data based on the phantom marker configured on the phantom in the real image data, and generating medical instrument reference plane data based on the medical instrument marker configured on the medical instrument. Data generation step; A matching step of generating matching data by matching the phantom reference surface data and the medical device reference surface data; A rendering step of generating and outputting phantom rendering data, which is 3D rendering data for the phantom, and medical equipment rendering data, which is 3D rendering data for the medical device, based on the phantom reference surface data, the medical device reference surface data, and the registration data; A medical device movement step of updating the medical device reference surface data according to the movement of the medical device, and generating and outputting the medical device rendering data based on the updated medical device reference surface data; And an evaluation step of evaluating the movement of the medical device according to a predetermined curriculum and generating and outputting evaluation data; and configured to be performed on a computer, and outputting the virtual reality image data at a specific magnification. Characterized in that the output, it can be achieved by providing an injection virtual reality device.

Another object of the present invention is an injection virtual reality device configured to receive virtual image data of a phantom for injection simulation and a medical device and output virtual reality image data of the injection, to the phantom in the real image data. A reference plane data generating step of generating phantom reference plane data based on the configured phantom marker and generating medical instrument reference plane data based on the medical instrument marker configured on the medical instrument; A matching step in which the injection virtual reality device matches the phantom reference surface data and the medical device reference surface data to generate matching data; The injection virtual reality device, based on the phantom reference surface data, the medical device reference surface data and the matching data, the phantom rendering data, which is 3D rendering data for the phantom, and medical device rendering data, which is 3D rendering data for the medical device. A rendering step of generating and outputting; A medical device movement step of the injection virtual reality device updating the medical device reference surface data according to the movement of the medical device, and generating and outputting the medical device rendering data based on the updated medical device reference surface data; And an evaluation step in which the injection virtual reality device evaluates the movement of the medical device according to a predetermined curriculum and generates and outputs evaluation data. The output of the virtual reality image data is configured to be performed on a computer. It can be achieved by providing an injection virtual reality method, characterized in that the output as a microscope image enlarged at a specific magnification.

Another object of the present invention is a memory module for storing a program code configured to receive real image data of a phantom and a medical device for microsurgery or injection simulation and to output virtual reality image data of the phantom and the medical device. ; And a processing module that processes the program code and outputs the virtual reality image data to a connected output module. The program code includes phantom marker data based on the phantom marker configured in the phantom in the real image data. A marker detection step of generating and generating medical device marker data based on the medical device marker configured in the medical device; Divide the real image data into a plurality of grid cells, generate a plurality of bounding boxes composed of at least one of the grid cells, and predict class for the phantom and the medical device for each of the grid cells to obtain class prediction data. Generates, a bounding box reliability multiplied by the overlapping data and a probability that the phantom or the medical device will be included in the bounding box for each of the bounding boxes, and based on the product of the class prediction data and the bounding box reliability, the bounding box An object detection step of outputting a class reliability for the, and generating reference vector data for the phantom and the medical device, which are objects determined based on the class reliability for the bounding box; A matching step of generating matching data by performing virtual object matching and spatial location reflection in a simulation space based on the reference vector data of the phantom and the reference vector data of the medical device; Phantom rendering data, which is 3D rendering data for the phantom, and medical device rendering data, which is 3D rendering data for the medical device, are generated based on the reference vector data of the phantom, the reference vector data of the medical device, and the matching data. And a rendering step of outputting the virtual reality image data; And updating the reference vector data of the medical device according to the movement of the medical device, and outputting the virtual reality image data by updating the medical device rendering data based on the updated reference vector data of the medical device. step; It is configured to be performed on a computer, including, the overlapping data, the phantom marker data or the medical device marker data refers to the data about the degree of overlap with the bounding box, the object detection step of generating the class prediction data The class prediction data for the grid cell is generated by a learned class reliability prediction CNN (Convolutional Neural Network) using the grid cell as input data and the class for the phantom and the medical device as output data. In the object detection step, the generation of the reference vector data includes: a reference vector learned by using the bounding box in which the class is determined and the class reliability of the bounding box as input data and the reference vector data as output data. Predictive Convolutional Neur al Network) to generate the reference vector data for the bounding box of the class, and in the matching step, the environment renders the phantom rendering data and the medical device having the same view point as the real image data. The data is the matched data, which is the matched data, the state is the difference vector from the matched data having the same view point as the real image data, and the agent is the location of the reference vector data and The reference vector by the reinforcement learning module that was previously learned by using a matching module for correcting the direction, an action as a correction to the reference vector data, and a reward as a reduction rate of the difference vector. Data is corrected, and the virtual reality image data in the rendering step and the simulation step The output of can be achieved by providing a medical training virtual reality device and method, including microsurgery and injection, characterized by being output to a client connected to an HMD, an AR HMD, and a display in the form of a microscope enlarged at a specific magnification. .

As described above, according to the present invention, the following effects are obtained.

First, according to an embodiment of the present invention, an effect that enables to learn the microsurgical environment or the injection environment empirically is generated.

Second, according to an embodiment of the present invention, the effect of being able to accurately evaluate and feedback the simulation results with a numerical objective evaluation is generated.

The following drawings attached to the present specification illustrate preferred embodiments of the present invention, and serve to further understand the technical idea of the present invention together with the detailed description of the present invention, so the present invention is limited only to those described in those drawings. And should not be interpreted.

1 is a schematic diagram showing the driving of the medical training virtual reality device 1 according to an embodiment of the present invention,

Figure 2 is a schematic diagram showing the detailed configuration of a medical training virtual reality device 1 according to an embodiment of the present invention,

Figure 3 is a flow chart showing a method of microsurgical virtual reality according to an embodiment of the present invention,

4 is a schematic diagram showing a medical training virtual reality device 1 according to a modification of the present invention.

Hereinafter, exemplary embodiments in which a person having ordinary skill in the art to which the present invention pertains may easily implement the present invention will be described in detail with reference to the accompanying drawings. However, in the detailed description of the operation principle for the preferred embodiment of the present invention, if it is determined that a detailed description of related known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

In addition, the same reference numerals are used for parts having similar functions and functions throughout the drawings. Throughout the specification, when a specific part is said to be connected to another part, this includes not only the case of being directly connected, but also the case of being connected indirectly with another element in between. In addition, the inclusion of specific components does not exclude other components unless specifically stated otherwise, and means that other components may be further included.

Medical training virtual reality devices and methods, including microsurgery and injection

Regarding the medical training virtual reality device 1 according to an embodiment of the present invention, FIG. 1 is a schematic diagram showing the operation of the medical training virtual reality device 1 according to an embodiment of the present invention. As shown in FIG. 1, the medical training virtual reality device 1 according to an embodiment of the present invention is connected to the camera module 2, a head mounted display (HMD) 3 or other display through a wired / wireless network, and the phantom The medical training virtual reality is output to the HMD 3 through the image data of the (10) and the medical device (11).

The camera module 2 may be configured to generate a real image data 200 by photographing a pre-installed phantom 10 and a medical instrument 11 for microsurgery / injection. The generated real image data 200 is transmitted to the medical training virtual reality device 1 and used for registration, rendering, and microsurgery / injection evaluation of phantoms and medical devices. For example, an optical camera such as an OptiTrack Flex 13 model capable of precise motion capture of 0.5 mm or less may be used for the camera module 2 according to an embodiment of the present invention.

The HMD 3 is a display worn on the user's head and provides 3D virtual reality to the user through the virtual reality image data output from the medical training virtual reality device 1. Typical products include Oculus Rift, PlayStation VR, Gear VR, and HTC Vive. In the present invention, HMD is mainly described for convenience of description, but the scope of the present invention is not limited to HMD and may include AR HMD, general display, and the like.

The phantom 10 refers to a body mimic that is a target of simulation of a specific microsurgery / specific injection. The phantom 10 according to an embodiment of the present invention may include a plurality of markers, the first marker 20. For example, in the case of a cataract surgery simulation, a torso in a lying down form may be constructed, and a phantom of a tactile feel similar to an actual skin texture may be constructed. In addition, for example, in the case of a cataract surgery simulation, at least one hole in which one end of the medical instrument 11 is inserted into the position of the eyeball may be configured to provide a tactile feel similar to that of a real surgery. In addition, for example, in the case of intravenous injection, a plurality of layers having different textures on the phantom may be configured to provide a tactile feel similar to an actual injection.

The medical device 11 refers to a medical device for simulation of a specific microsurgery / specific injection, and the medical device 11 according to an embodiment of the present invention may include a plurality of markers, the second marker 21. have. For example, for cataract surgery simulation, Discission Knife, Nucleus Spatula, Cataract Knife, Capsul Extractive Forceps, Capsulorhexis Forceps, Infusion Handpiece, Irrigation / aspiration Cannula, Lens Loop, IOL Implantation Forceps, Cataract Scissors Membrance, Micro-needle Holder, etc. The medical device 11 may be configured in the form of a handpiece simulating. In addition, for example, in the case of intravenous injection, a simulation product in which an intravenous syringe is simulated may be used as a medical device.

Regarding the detailed configuration of the medical training virtual reality device 1, FIG. 2 is a schematic diagram showing the detailed configuration of the medical training virtual reality device 1 according to an embodiment of the present invention. 2, the medical training virtual reality device 1 according to an embodiment of the present invention receives the real image data 200 and outputs the virtual reality image data 300, the phantom detection module 100 ), A medical device detection module 101, a phantom output module 102, a medical device output module 103, a matching module 104, a surgical effect output module 105, and a surgical evaluation module 106. .

The phantom detection module 100 is a component that generates position data based on a specific point such as a reference plane vector and a camera module 2 of the phantom based on a plurality of first markers 20 configured in the phantom 10. According to an embodiment of the present invention, the phantom detection module 100 matches 3D position data of the first marker 20 pre-stored and position data of the first marker 20 received from the real image data 200. The camera module 2 may be configured to generate reference vector data of the phantom 10 including the position data of the phantom 10 and the angle data of the phantom 10.

The medical device detection module 101 is based on a plurality of second markers 21 configured in the medical device 11, position data based on a specific point such as a reference plane vector and the camera module 2 of the medical device 11 It is a configuration that generates. According to an embodiment of the present invention, the medical device detection module 101 matches 3D position data of the pre-stored second marker 21 with the position data of the second marker 21 received from the real image data 200. The camera module 2 may be configured to generate reference surface vector data of the medical device 11 including the position data of the medical device 11 and the angle data of the medical device 11.

The matching module 104 matches the reference plane of the phantom 10 and the reference plane of the medical instrument 11 based on the reference plane vector data of the phantom 10 and the reference plane vector data of the medical instrument 11 to generate matching data It is a configuration. The registration data according to an embodiment of the present invention may be composed of vector data between the reference plane of the phantom 10 and the reference plane of the medical instrument 11.

The phantom output module 102 is a module that 3D renders a phantom based on the reference plane vector data of the phantom 10 and the matching data. For example, in the case of cataract surgery simulation, the phantom output module 102 may be configured to output 3D rendering of the eye in an enlarged state by a microscope.

The medical device output module 103 is a module for 3D rendering the medical device 11 with the phantom 10 based on the reference plane vector data of the medical device 11 and the matching data. For example, in the case of cataract surgery simulation, the medical device output module 103 may be configured to output a medical device in an enlarged state by a microscope in 3D rendering in relation to the eyeball.

The surgical effect output module 105 is a model that outputs a surgical effect generated according to the movement of the medical device 11. The surgical effect output module 105 may include a rigid body & destruction simulation model, a cloth simulation model, a soft body & tear simulation model, and a fluid simulation model. In addition, it may be configured to output the surgical effect according to the specific curriculum of the medical device 11, including the curriculum data, the surgical effect to proceed to the next curriculum, success and failure of the curriculum.

The surgical evaluation module 106 is a module for evaluating the motion of the medical device 11 for a specific step based on the curriculum data and evaluating and outputting the motion of the medical device 11 of the corresponding user as success or failure.

3 is a flowchart illustrating a medical training virtual reality method according to an embodiment of the present invention. As shown in Figure 3, the medical training virtual reality method according to an embodiment of the present invention is a real image data receiving step (S10), a phantom reference plane vector generating step (S11), a medical device reference plane vector generating step (S12), Matching and 3D rendering data output step (S13), medical device movement step (S14), and may include an evaluation step (S15).

In the real image data receiving step (S10), the medical training virtual reality device 1 receives the real image data 200 of the phantom 10 and the medical device 11 from the camera module 2.

In the phantom reference plane vector generation step (S11), the medical training virtual reality device 1 generates a reference plane vector of the phantom 10 using the first marker 20 of the phantom 10 of the real image data 200. to be.

In the medical device reference plane vector generation step (S12), the medical training virtual reality device 1 uses the second marker 21 of the medical device 11 of the real image data 200 to obtain the reference plane vector of the medical device 11. It is a step to create.

In the matching and 3D rendering data output step (S13), the medical training virtual reality device 1 generates registration data using the reference plane vector of the phantom 10 and the reference plane vector of the medical device 11, and the phantom 10 and This is a step of outputting 3D rendering data generated by 3D rendering of the medical device 11 to a display such as the HMD 3.

In the medical device movement step (S14), the user moves the medical device 11, the reference plane vector of the medical device 11 and 3D rendering data of the medical device 11 are updated and displayed on the display according to the motion of the medical device 11. This is the output stage.

The evaluation step S15 evaluates the motion of the user's medical device 11 according to a pre-stored curriculum based on the updated reference plane vector of the medical device 11 and the 3D rendering data of the medical device 11 to evaluate data. It is a step of generating and outputting the generated evaluation data.

4 is a schematic view showing a medical training virtual reality device 1 according to a modification of the present invention. As shown in FIG. 4, the medical training virtual reality device 1 according to a modification of the present invention receives the real image data 200 and the motion detection sensor data 201 and evaluates the virtual reality image data 300 and It can be configured to output data 301. In addition, for this, the medical training virtual reality device 1 includes a marker detection module 400, an object detection module 401, a detailed adjustment module 402, a matching module 403, a simulation module 404, and an object output module ( 405), an effect output module 406, and an evaluation module 407.

The marker detection module 400 receives the real image data 200 and detects a plurality of markers (identification markers and position markers) configured in the phantom 10 and the medical device 11 based on the real image data 200. It is a module that generates phantom marker data (identification data and marker position data) and medical device marker data (identification data and marker position data). The marker according to an embodiment of the present invention may include an identification marker and a location marker, and may include a plurality of markers to be configured to detect a change in the axis of the corresponding object. The identification marker means a marker capable of identifying which object is for each object, and may be distinguished by color or shape, code, listed shape, and attachment location.

The object detection module 401 may be configured to receive the real image data 200, detect an object based on the real image data 200, configure a bounding box, classify objects, and generate reference plane vector data of the object. Can be. The object detection module 401 according to an embodiment of the present invention divides the real image data 200 received from the camera module 2 into a plurality of nxn grid cells, and is composed of at least one grid cell. A plurality of bounding boxes are generated, and a class prediction data is generated by predicting one class (what object, ie, what kind of phantom, and which medical device) for each grid cell, and each bounding box creates a bounding box It may be configured to output reliability, generate class reliability for a corresponding bounding box based on the class prediction data and the bounding box reliability, and predict reference vector data based on the class reliability of the bounding box. (

Specifically, data for a specific grid cell includes object position data (position x, y), object size data (width w, height h) for a plurality of bounding boxes containing the grid cell, and each bounding box included. The bounding box reliability for the class may include class prediction data predicted from the corresponding grid cell. At this time, the reliability of the bounding box is the overlapping data of the bounding box and the reference box when the reference box generated by the probability that the corresponding bounding box includes the object and the reference box generated by the marker position data is ground truth (intersection area width / union area) Area) and may be generated by the object detection module 401. In addition, the class prediction data uses a specific grid cell as input data and a specific class (each class of phantom and medical devices) as output data to predict the learned class reliability to a convolutional neural network (CNN) object detection module 401. ), It may mean a class predicted value output for the corresponding grid cell. Also, the object detection module 401 may be configured to output a class reliability for a specific bounding box based on the product of the bounding box reliability and the class prediction data. Each class of phantoms and medical instruments according to an embodiment of the present invention includes, for example, ophthalmic microsurgery phantom, otorhinolaryngology microsurgery phantom, lower extremity injection phantom, upper limb injection phantom, ophthalmic microsurgery medical instrument, otolaryngology microsurgery medical instrument, Injection medical devices, and the like. In addition, the class reliability prediction CNN according to an embodiment of the present invention limits the class to data class-labeled by an identification marker identifying an object among markers, and a class identified by an identification marker among output nodes of the class reliability prediction CNN It can be configured to null for an output node that predicts a class other than.

The object detection module 401 determines a specific object as a specific class (each class of a phantom and a medical device) based on class reliability for a specific bounding box, and outputs reference vector data of the bounding box in which the class is determined. Can be configured. The reference vector data means a vector value that is a reference for rendering in order to precisely render the object, and may include a position (x, y, z) and direction of the 3D axis. The reference vector data in the object detection module 401 according to an embodiment of the present invention is a reference vector prediction CNN (Convolutional) by which the class bounding box and class reliability are input data and the reference vector data is output data. Neural Network (included in the object detection module 401) may mean a reference vector prediction value output for a corresponding bounding box of the corresponding class.

According to the object detection module 401 according to an embodiment of the present invention, the configuration of the bounding box is not predicted, but the class reliability of the bounding box is generated. And generate an effect that can generate reference vector data. According to this, when a plurality of medical devices are used in one medical training, a virtual reality rendering effect is generated.

According to the object detection module 401 according to another embodiment of the present invention, when a medical training virtual reality is used by a user (inference step), the reference vector data of the phantom and the reference vector data of the medical device are rendered closest to each other. The distance between the phantom region and the rendered medical device region may be used to re-learn the reference vector prediction CNN.

The detailed adjustment module 402 receives the motion detection sensor data 201 input from a motion detection sensor such as an IR sensor, a microwave sensor, etc. configured at a specific location outside, and based on the motion detection sensor data 201, a medical device ( It is a module that detects the displacement of 11) and compares the displacement of the medical instrument 11 with the displacement of the reference vector data output by the object detection module 401 to fine-tune the reference vector data.

The matching module 403 is a module that performs virtual object matching and spatial location reflection in a simulation space based on the reference vector data of the phantom and the reference vector data of the medical device. For example, the matching module 403 generates a virtual matching space using the reference vector data of the phantom as a reference axis, projects the reference vector data of the medical device into the space, and converts it into a simulation space. Through this, it can be configured to execute a task of matching a real object (phantom, medical device) to a simulation space.

The matching module 403 according to an embodiment of the present invention uses 2D real image data 200 input by the camera module 2 and object rendering data generated by the object output module 405 at the same view point. It can be configured to compare and correct the reference vector data of each object so that the difference between the real image data 200 and the object rendering data is reduced. With respect to a method for the matching module 403 to find the same view point in the object rendering data as the real image data 200 according to an embodiment of the present invention, the reference vector data in the real image data 200 of a specific object The view point of the object rendering data in which the reference vector data in the simulation environment is identically output may be calculated as the same view point as the real image data 200.

The matching module 403 according to an embodiment of the present invention may include a reinforcement learning module that corrects reference vector data of each object, and the reinforcement learning module has an environment in which the real image data 200 and the same view are identical. Difference vector between object rendering data that has points and object view data whose state is the same as real image data 200, and agent corrects for reference vector data of each object (3D position x , y, z and direction), the matching module 403, the action performs correction (3D position x, y, z and direction) for the reference vector data of each object, and the reward is recalled. As a reduction rate of the difference vector, it may mean a reinforcement learning module that is trained to act in the direction in which the agent takes the reward. For precise matching, objects of reality and virtual reality (phantom and medical devices) must match. However, between the object of the virtual reality produced by 3D rendering and the object of reality produced by injection molding or 3D printing, a fine error is inevitable in the manufacturing and processing process. In the general case, it is not a big problem, but in the case of microsurgery or injection, an error of 1 mm also causes a very large difference, and is a factor that interferes with the immersion and realism of the practitioner. According to the matching module 403 according to an embodiment of the present invention, since the objects of the reality and the virtual reality are consistently matched by the reinforcement learning, the effect of solving the above problems of the virtual reality medical training is generated.

The simulation module 404 is a module for generating simulation data through the movement of the reference vector data of the medical device and the positional relationship between the reference vector data of the phantom and the reference vector data of the medical device.

The object output module 405 is an object rendering data that is a 3D rendered object based on the reference vector data of the medical device and the reference vector data of the phantom matched by the matching module 403 and simulated by the simulation module 404. And a module for generating and outputting effect data. For example, in the case of a cataract surgery simulation, the object output module 405 may be configured to output 3D rendering of the eye in an enlarged state by a microscope. In addition, for example, in the case of a cataract surgery simulation, the object output module 405 may be configured to output a 3D rendered surgical instrument in a state enlarged by a microscope in relation to the eyeball.

The effect output module 406 is a model that outputs effect data, which is a surgical effect generated according to the movement of the medical device 11. The effect output module 406 may include a rigid body & destruction simulation model, a cloth simulation model, a soft body & tear simulation model, and a fluid simulation model. In addition, it may be configured to output the surgical effect according to the specific curriculum of the medical device 11, including the curriculum data, the surgical effect proceeding to the next curriculum, success and failure of the curriculum.

The evaluation module 407 evaluates the movement of the medical device 11 for a specific step based on the curriculum data, and evaluates the movement of the medical device 11 of the corresponding user as success or failure to output the evaluation data 301 It is a module.

The object rendering data and effect data output from the object output module 405 and the effect output module 406 may be configured to be output to HMD, general display, beam projection, etc. connected to the object output module 405 and the effect output module 406. Can be. In addition, the evaluation data may be output to a client of a trainee who has undergone medical training and a client of a leader through a web server connected to the evaluation module 407.

As described above, those skilled in the art to which the present invention pertains will understand that the present invention may be implemented in other specific forms without changing its technical spirit or essential features. Therefore, the above-described embodiments should be understood as illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the detailed description, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts should be interpreted to be included in the scope of the present invention.

The features and advantages described in this specification are not all inclusive, and many additional features and advantages will become apparent to those skilled in the art in view of the drawings, specifications, and claims. Moreover, it should be noted that the language used herein has been chosen primarily for readability and for teaching purposes, and may not be selected to describe or limit the subject matter of the present invention.

The foregoing description of embodiments of the invention has been presented for purposes of illustration. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art can understand that many modifications and variations are possible in light of the above disclosure.

Therefore, the scope of the present invention is not limited by the detailed description, but by any claims of the application based thereon. Accordingly, the disclosure of the embodiments of the present invention is exemplary and does not limit the scope of the present invention as set forth in the claims below.

Claims (1)

A memory module for storing program code configured to receive real image data of a phantom and a medical device for microsurgery or injection simulation and to output virtual reality image data of the phantom and the medical device; And A processing module that processes the program code and outputs the virtual reality image data to a connected output module; Including, The program code, A marker detection step of generating phantom marker data based on the phantom marker configured on the phantom in the real image data, and generating medical instrument marker data based on the medical instrument marker configured on the medical instrument; Divide the real image data into a plurality of grid cells, generate a plurality of bounding boxes composed of at least one of the grid cells, and predict class for the phantom and the medical device for each of the grid cells to obtain class prediction data. Generates, a bounding box reliability multiplied by the overlapping data and a probability that the phantom or the medical device will be included in the bounding box for each of the bounding boxes, and based on the product of the class prediction data and the bounding box reliability, the bounding box An object detection step of outputting a class reliability for the, and generating reference vector data for the phantom and the medical device, which are objects determined based on the class reliability for the bounding box; A matching step of generating matching data by performing virtual object matching and spatial location reflection in a simulation space based on the reference vector data of the phantom and the reference vector data of the medical device; Phantom rendering data, which is 3D rendering data for the phantom, and medical device rendering data, which is 3D rendering data for the medical device, are generated based on the reference vector data of the phantom, the reference vector data of the medical device, and the matching data. And a rendering step of outputting the virtual reality image data; And A simulation step of updating the reference vector data of the medical device according to the movement of the medical device, and outputting the virtual reality image data by updating the medical device rendering data based on the updated reference vector data of the medical device. ; It is configured to be performed on a computer, including, The overlapping data refers to data on the degree to which the phantom marker data or the medical device marker data overlaps the bounding box, In the object detection step, the generation of the class prediction data is performed by a previously learned class reliability prediction CNN (Convolutional Neural Network) using the grid cell as input data and the class for the phantom and the medical device as output data. The class prediction data for the grid cell is generated, The generation of the reference vector data in the object detection step includes: learning the reference vector prediction CNN using the bounding box in which the class is determined and the class reliability of the bounding box as input data and the reference vector data as output data. The reference vector data for the bounding box of the class is generated by (Convolutional Neural Network), In the matching step, an environment is set as the matching data, which is data matching the phantom rendering data and the medical device rendering data having the same view point as the real image data, and a state is the real image data. Set as the difference vector from the matching data having the same view point as and, the agent (Agent) as a matching module that performs the correction of the position and direction for the reference vector data, the action (Action) to the reference vector data The reference vector data is corrected by a reinforcement learning module that is pre-learned using a correction as a correction factor and a reward as a reduction rate of the difference vector, Characterized in that the output of the virtual reality image data in the rendering step and the simulation step is output to a client connected to an HMD, an AR HMD, and a display in a microscopic interface enlarged at a specific magnification, Medical training virtual reality device including microsurgery and injection.

Images