KR20220020634A - Augmented reality navigation system for wrist arthroscopic surgery using skin markers and method thereof - Google Patents

Abstract

The present invention relates to an augmented reality navigation system for wrist arthroscopic surgery using a skin marker and a method therefor.
According to the present invention, in the augmented reality navigation system for wrist arthroscopy using skin markers, the subject to be measured before finger traction by using MRI or CT in a state where a plurality of skin markers are attached to the hand of the subject to be measured. A first capture unit that captures the hand bones of a first extractor that generates a three-dimensional image using the captured image and extracts positions of a plurality of skin markers from the three-dimensional image, a reference marker is attached to the plurality of skin markers A second photographing unit that captures the hand bones of the measurement object using a depth camera while pulling the measurement object's finger, and extracts the positions of a plurality of skin markers to which reference markers are attached from the depth image photographed through the depth camera a second extraction unit, an image correction unit for correcting a three-dimensional image using the positions of the plurality of skin markers extracted from the first extraction unit and the positions of the plurality of skin markers extracted from the second extraction unit, and A control unit for generating an augmented reality image by synthesizing the photographed hand image of the subject to be measured and the corrected 3D image, and displaying the augmented reality image.

Description

Augmented reality navigation system and method for wrist arthroscopic surgery using skin markers

The present invention relates to an augmented reality navigation system for wrist arthroscopy surgery using a skin marker and a method therefor, and a wrist using a skin marker for expanding a portion to be photographed using augmented reality during surgery using an arthroscopy and providing it to a user It relates to an augmented reality navigation system for arthroscopic surgery and a method therefor.

The surgical navigation system uses a tracker attached to the patient and surgical tool to identify the current position of the surgical tool, and uses pre-imaged image information such as CT and MRI to display the surgical tool on the internal image of the human body reproduced on the monitor. It is a system that shows the location of

As a related prior art, AR-based navigation technology that simply visualizes the rigid registration result of the existing navigation system through augmented reality (AR) technology using preoperative images (CT/MRI) There is this.

However, the prior art performs only rigid body registration of the pre-operative image, and, despite being a technique that targets organs that are deformed by force, such as the brain and liver, the degree of visualizing the main parts without considering the deformation of the organs There is a limit to being limited to .

In addition, according to the prior art, since the surgical arthroscopy and the navigation system used in arthroscopic surgery cannot be used at the same time, there is an inconvenience of alternately checking the CT/MRI image of the patient matched with the arthroscopic image.

Further, in the case of arthroscopic surgery, the movement path of the field of view and the arthroscopy is narrow, so the operation time is delayed, and there is a disadvantage that the internal tissue may be damaged by the movement of the surgical tool.

The technology that is the background of the present invention is disclosed in Korean Patent Publication No. 10-2020-0065297 (published on June 9, 2020).

The technical problem to be achieved by the present invention is to provide an augmented reality navigation system for wrist arthroscopy surgery using a skin marker and a method therefor to extend the part photographed using augmented reality during arthroscopic surgery and provide the user with will be.

According to an embodiment of the present invention for achieving this technical task, in the augmented reality navigation system for wrist arthroscopic surgery using skin markers, MRI or CT is used while a plurality of skin markers are attached to the hand of a subject to be measured. A three-dimensional image generating unit for generating a three-dimensional image of the hand bones of the subject to be measured before finger traction of the subject to be measured, a first extraction unit for extracting the plurality of skin markers from the three-dimensional image, the plurality of skin markers A reference marker is attached to the , and a depth image generator for generating a depth image of the hand bone of the measurement target by using a depth camera in a state in which the finger of the measurement target is pulled, a plurality of the reference markers attached from the depth image a second extraction unit for extracting the positions of the skin markers of the third An image correction unit for correcting a dimensional image, and a controller for generating an augmented reality image by synthesizing the corrected 3D image with the hand image of the subject to be measured taken by arthroscopy, and displaying the augmented reality image.

The image compensator may correct the 3D image through a link manipulator model so that the first positions of the plurality of skin markers match the second positions.

If the hand of the subject to be measured is photographed before the arthroscopy is inserted into the body of the subject to be measured, the controller may display different colors for each hand bone in the hand image captured by the arthroscopy.

When the arthroscopy is inserted into the body of the subject to be measured and the inner side of the hand of the subject to be measured is photographed, the control unit determines the type of hand bone displayed in the image of the inside of the hand photographed by the arthroscopy, and corresponds to the type of hand bone Thus, the peripheral region of the hand bone may be expanded and displayed, and a boundary line for displaying the outer portion of the hand bone may be displayed on the extended hand bone image.

When the arthroscope moves inside the body of the subject to be measured, the controller may provide the 3D image that is changed according to the movement of the arthroscope.

According to another embodiment of the present invention, in a method for providing augmented reality navigation using an augmented reality navigation system for wrist arthroscopic surgery, in a state in which a plurality of skin markers are attached to the hand of a subject to be measured, MRI or CT is used to Generating a three-dimensional image of the hand bones of the subject to be measured before finger retraction of the subject to be measured, extracting first positions of the plurality of skin markers from the three-dimensional image, and attaching a reference marker to the plurality of skin markers and generating a depth image of the hand bone of the measurement subject using a depth camera in a state in which the finger of the measurement subject is pulled, the second position of the plurality of skin markers to which the reference marker is attached from the depth image extracting, correcting the three-dimensional image using first positions of the plurality of skin markers extracted from the three-dimensional image and second positions of the plurality of skin markers extracted from the depth image, and joints and generating an augmented reality image by synthesizing the corrected three-dimensional image with the hand image of the subject to be measured taken by the mirror, and displaying the augmented reality image.

As described above, according to the present invention, by expanding the narrow arthroscopy field of view during wrist arthroscopy using augmented reality images, the accuracy of wrist arthroscopy can be increased and side effects can be reduced.

1 is a configuration diagram for explaining the configuration of an augmented reality navigation system according to an embodiment of the present invention.
2 is a flowchart illustrating a method for providing augmented reality navigation according to an embodiment of the present invention.
FIG. 3 is a diagram for explaining step S210 of FIG. 2 .
4A and 4B are diagrams for explaining step S250 of FIG. 2 .
5 is a view showing a screen provided when the outside of the hand is photographed through an arthroscopy according to an embodiment of the present invention.
6 is a view showing a screen provided when the inside of the hand is photographed through an arthroscopy according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily carry out the present invention. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

Then, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them.

1 is a configuration diagram for explaining the configuration of an augmented reality navigation system according to an embodiment of the present invention.

As shown in FIG. 1 , the augmented reality navigation system 100 according to an embodiment of the present invention includes a 3D image generator 110 , a first extractor 120 , a depth image generator 130 , and a second extractor 140 , an image correction unit 150 , and a control unit 160 .

First, a skin marker is attached to the wrist or back of the hand of a subject to be measured, and in this case, the skin marker is manufactured in the form of a hexagon having a predetermined size. And, the inside of the skin marker contains vitamin D to increase the discriminability during MRI or CT scan.

Next, the 3D image generating unit 110 uses MRI or CT to photograph the hand bones before finger traction of the subject to be measured in a state in which a plurality of skin markers are attached to the hand of the subject to be measured.

Here, computed tomography (CT) may include a method of imaging internal structures by projecting X-rays around a cross section of the human body and measuring the amount of X-rays that decrease as they pass through the human body. .

In addition, magnetic resonance imaging (MRI) may include a method of acquiring an arbitrary tomography image of a living body using a magnetic field generated by magnetic force.

Next, the first extractor 120 generates a three-dimensional image by using the plurality of images captured by the three-dimensional image generator 110 , and extracts positions of a plurality of skin markers from the three-dimensional image.

Next, the depth image generating unit 130 captures the hand bones of the subject to be measured by using the depth camera in a state where the finger of the subject to be measured is pulled.

In this case, a reference marker is attached to the upper surface of the plurality of skin markers.

In addition, the depth camera is a camera capable of measuring the distance between the camera and an object using images captured through a plurality of lenses, and according to an embodiment of the present invention, it is possible to three-dimensionally photograph the hand bones in a towed state. there is.

And, it can be used by attaching a dot projector to the front part of the depth camera.

At this time, the dot projector is to disperse one laser into 30,000 or more lasers to obtain three-dimensional coordinates. Create 3D coordinates.

Next, the second extractor 140 extracts positions of a plurality of skin markers to which reference markers are attached from the depth image captured by the depth image generator 130 .

Next, the image compensator 150 moves the plurality of skin markers extracted from the first extractor 120 to positions of the plurality of skin markers extracted from the second extractor 140 , and uses a link manipulator ) to correct the 3D image by applying it to the model.

Here, the link manipulator model simulates the movement of the joints of each bone, and in the embodiment of the present invention, a plurality of skin markers extracted from the first extraction unit 120 are combined with the second extraction unit 140 . It is used to move to the location of a plurality of skin markers extracted from

Next, the control unit 160 generates an augmented reality image by synthesizing the hand image of the subject to be measured and the corrected 3D image taken from the arthroscopy (not shown), and displays the augmented reality image.

At this time, if the hand of the subject to be measured is photographed before the arthroscopy is inserted into the body of the subject to be measured, the control unit 160 displays the hand bones in different colors on the hand image captured by the arthroscopy.

And, the arthroscopy is used for joint surgery of the subject to be measured, and the end of the arthroscope includes a camera for photographing the hand of the subject to be measured.

In addition, when the arthroscopy is inserted into the body of the subject to be measured and the inside of the hand of the subject is photographed, the control unit 160 determines the type of hand bone displayed on the image of the inside of the hand photographed by the arthroscopy, and responds to the type of hand bone The peripheral region of the hand bone is expanded and displayed, and a boundary line for displaying the outer part of the hand bone is displayed on the extended hand bone image.

Also, when the arthroscopy moves inside the subject's body, the controller 160 provides a 3D image that is changed according to the movement of the arthroscopy.

Hereinafter, an augmented reality navigation providing method for providing an augmented reality navigation system for wrist arthroscopic surgery using a skin marker according to an embodiment of the present invention will be described with reference to FIGS. 2 to 6 .

2 is a flowchart illustrating a method for providing augmented reality navigation according to an embodiment of the present invention.

First, the three-dimensional image generating unit 110 uses MRI or CT in a state in which a plurality of skin markers are attached to the hand of the subject to be measured, and photographs the hand bones of the subject to be measured before finger traction of the subject to be measured ( S210 ).

That is, the 3D image generating unit 110 uses MRI or CT to photograph the target's finger before arthroscopic surgery.

FIG. 3 is a diagram for explaining step S210 of FIG. 2 .

As shown in FIG. 3 , the image captured by the 3D image generator 110 includes a plurality of skin markers.

Although the number of skin markers is shown as three in FIG. 3 , the number of skin markers may be changed according to the size of the hand of the subject to be measured and a setting value of the augmented reality navigation system 100 .

Next, the first extractor 120 generates a three-dimensional image by using the captured image, and extracts a plurality of skin markers from the three-dimensional image, respectively (S220).

That is, as shown in FIG. 3 , the first extraction unit 120 extracts three-dimensional coordinates corresponding to positions of a plurality of skin markers.

Next, in a state in which the reference markers are attached to the plurality of skin markers, the depth image generating unit 130 captures the hand bones of the subject to be measured using a depth camera in a state in which the subject's finger is pulled ( S230 ).

At this time, the depth image generating unit 130 applies a dot projector to the lens of the front part of the depth camera to photograph the hand bones of the subject to be measured.

Next, the second extractor 140 extracts a plurality of skin markers to which reference markers are attached by using the depth image captured by the depth image generator 130 ( S240 ).

At this time, the second extraction unit 140 extracts the positions of the plurality of skin markers to which the reference markers are attached as three-dimensional coordinates.

Next, the image corrector 150 uses the positions of the plurality of skin markers extracted through the first extraction unit 120 and the positions of the plurality of skin markers extracted through the second extraction unit 140 in 3D. The image is corrected (S250).

4A and 4B are diagrams for explaining step S250 of FIG. 2 .

As shown in FIGS. 4A and 4B , a portion indicated in red is an image captured by the 3D image generator 110 , and a portion indicated in gray is an image captured by the depth image generator 130 .

That is, as shown in FIG. 4A , when the 3D image is not corrected, the position of the current hand bone of the subject to be measured and the hand bone in the image acquired by the 3D image generator 110 may exist at different positions.

Accordingly, the image correction unit 150 converts the positions of the plurality of skin markers extracted through the first extraction unit 120 to the positions of the plurality of skin markers to which the reference markers extracted through the second extraction unit 140 are attached. Correct.

In this case, the image correction unit 150 corrects the 3D image using the link manipulator model described above.

Then, as shown in FIG. 4B , the image correction unit 150 corrects the position of the hand bone of the subject to be measured to be the same as the position of the hand bone in the image acquired by the 3D image generator 110 .

Next, the controller 160 generates an augmented reality image by synthesizing the hand image of the subject to be measured and the corrected 3D image taken from the arthroscopy, and displays the generated augmented reality image (S260).

5 is a view showing a screen provided when the outside of the hand is photographed through an arthroscopy according to an embodiment of the present invention, and FIG. 6 is a case in which the inside of the hand is photographed through an arthroscope according to an embodiment of the present invention It is a diagram showing the provided screen.

At this time, as shown in FIG. 5 , if the hand of the subject to be measured is photographed before the arthroscopy is inserted into the body of the subject to be measured, the controller 160 displays different colors for each hand bone in the hand image captured by the arthroscopy. Display.

That is, as shown in FIG. 5 , the controller 160 displays different hand bones such as purple, green, and yellow.

In addition, when the arthroscopy is inserted into the body of the subject to be measured and the inner side of the hand of the subject is photographed, the controller 160 determines the type of hand bone present in the image on the inner side of the hand by using the image on the inner side of the hand photographed through the arthroscopy. do.

In this case, the hand bone may be determined as the scaphoid bone or the radial bone according to the photographing position of the arthroscopy.

In addition, as shown in FIG. 6 , the controller 160 expands and displays the peripheral region of the hand bone in response to the type of hand bone on the inner image of the hand photographed from the arthroscopy, and a boundary line for displaying the outer portion of the hand bone on the expanded hand bone image are displayed together.

That is, as shown in FIG. 6 , the control unit 160 provides the ligament and the boundary between the navicular and radial bones through the display line.

In this case, the controller 160 may change the region of the image of the hand bone that is expanded and displayed according to the determined type of hand bone.

In addition, when the arthroscopy moves to another position inside the body of the subject to be measured, the controller 160 provides a 3D image that is changed according to the movement of the arthroscope.

As described above, according to an embodiment of the present invention, by expanding the narrow arthroscopy field of view during wrist arthroscopy by using an augmented reality image, the accuracy of wrist arthroscopy can be increased and side effects can be reduced.

Although the present invention has been described with reference to the embodiment shown in the drawings, this is merely exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: augmented reality navigation system,
110: a three-dimensional image generator, 120: a first extractor,
130: a depth image generator, 140: a second extractor;
150: image correction unit, 160: control unit

Claims (10)

In the augmented reality navigation system of wrist arthroscopic surgery using skin markers,
A three-dimensional image generating unit for generating a three-dimensional image of the hand bone of the subject to be measured before finger traction of the subject by using MRI or CT in a state in which a plurality of skin markers are attached to the hand of the subject to be measured;
a first extraction unit for extracting the plurality of skin markers from the three-dimensional image;
A reference marker is attached to the plurality of skin markers, and a depth image generator for generating a depth image of the hand bone of the measurement target by using a depth camera in a state in which the finger of the measurement target is pulled;
a second extraction unit for extracting positions of a plurality of skin markers to which the reference markers are attached from the depth image;
An image correction unit for correcting the 3D image by using the first positions of the plurality of skin markers extracted from the 3D image and the second positions of the plurality of skin markers extracted from the depth image, and
Augmented reality navigation system comprising a controller for generating an augmented reality image by synthesizing the corrected 3D image with the hand image of the subject to be measured taken by arthroscopy, and displaying the augmented reality image. According to claim 1,
The image correction unit,
An augmented reality navigation system in which the first positions of the plurality of skin markers match the second positions by correcting the three-dimensional image through a link manipulator model. 3. The method of claim 2,
When the hand of the subject to be measured is photographed before the arthroscope is inserted into the body of the subject to be measured,
The control unit is
Augmented reality navigation system for displaying each hand bone in different colors on the hand image taken by the arthroscopy. 4. The method of claim 3,
When the arthroscopy is inserted into the body of the subject to be measured and photographed the inside of the hand of the subject to be measured,
The control unit is
Determining the type of hand bone displayed on the inner image of the hand taken by the arthroscopy,
In response to the type of the hand bone, the peripheral area of the hand bone is expanded and displayed,
An augmented reality navigation system for displaying a boundary line for displaying an outer part of the hand bone together on the extended hand bone image. 5. The method of claim 4,
The control unit is
When the arthroscopy moves inside the body of the subject to be measured, the augmented reality navigation system provides the three-dimensional image that is changed according to the movement of the arthroscopy. In a method for providing augmented reality navigation using an augmented reality navigation system for wrist arthroscopy,
In a state in which a plurality of skin markers are attached to the hand of the subject to be measured, using MRI or CT to generate a three-dimensional image of the hand bone of the subject to be measured before finger traction of the subject to be measured;
extracting the first positions of the plurality of skin markers from the three-dimensional image;
A reference marker is attached to the plurality of skin markers, and generating a depth image of the hand bone of the measurement target by using a depth camera in a state where the finger of the measurement target is pulled;
extracting second positions of the plurality of skin markers to which the reference markers are attached from the depth image;
correcting the 3D image using first positions of the plurality of skin markers extracted from the 3D image and second positions of the plurality of skin markers extracted from the depth image, and
Augmented reality navigation providing method comprising the step of generating an augmented reality image by synthesizing the corrected 3D image with the hand image of the subject to be measured taken by arthroscopy, and displaying the augmented reality image. 7. The method of claim 6,
The step of correcting the image,
A method of providing augmented reality navigation in which the first positions of the plurality of skin markers match the second positions by correcting the three-dimensional image through a link manipulator model. 8. The method of claim 7,
When the hand of the subject to be measured is photographed before the arthroscope is inserted into the body of the subject to be measured,
The step of displaying the augmented reality image,
Augmented reality navigation providing method for displaying each hand bone in different colors on the hand image taken by the arthroscopy. 9. The method of claim 8,
When the arthroscopy is inserted into the body of the subject to be measured and photographed the inside of the hand of the subject to be measured,
The step of displaying the augmented reality image,
Determining the type of hand bone displayed on the inner hand image taken by the arthroscopy, expanding and displaying the peripheral region of the hand bone in response to the type of hand bone, and adding a boundary line for displaying the outer portion of the hand bone on the expanded hand bone image A method of providing augmented reality navigation for display. 10. The method of claim 9,
The step of displaying the augmented reality image,
When the arthroscopy moves inside the body of the subject to be measured, augmented reality navigation providing method for providing the 3D image that is changed according to the movement of the arthroscope.

Images