TWI697317B - Digital image reality alignment kit and method applied to mixed reality system for surgical navigation - Google Patents

Abstract

The present invention relates to a digital image reality alignment kit and method, applied to a mixed reality based surgical navigation system, comprising: a plurality of moveable markers, having a plurality of movable marker coordinate frames correspondingly and a part of the plurality of moveable markers are configured on a surgical instrument; a positioning marker, configured in proximity to a surgical area having a surgical coordinate frame and configured to have a determined positioning marker coordinate frame and have a determined relative position and orientation with respect to the part of the plurality of moveable markers and the surgical area to transform the surgical area coordinate to the positioning marker coordinate; and a mixed reality device, detecting the positioning marker and having a mixed reality device coordinate frame to transform the positioning marker coordinate to the mixed reality device coordinate accordingly to project digital surgical area image onto the surgical area.

Description

Digital image reality pairing applied to mixed reality system integrated with surgical navigation Bit package and method

The present invention relates to a mixed reality surgical navigation system, in particular to an alignment kit used for aligning and projecting digital images of the affected part to the affected part and its alignment method.

Today’s surgery still has many high-risk operations, such as craniotomy, arterial thrombus screening, minimally invasive surgery, or spine surgery. These operations also face difficulties in positioning, difficult operations, and high-tech. Problems such as sex, high complexity, etc., and these problems are mainly due to the fact that the doctor cannot directly see the affected part visually, only a small part of the affected part or its outer tissue, which results in extremely high surgical risks .

Therefore, before the implementation of these high-risk operations, physicians must use X-ray imaging technology, computer tomography (CT) technology, computer tomography angiography (CTA) technology, digital subtraction angiography (DSA) technology, and maximum density projection ( MIP) technology, or magnetic resonance imaging (MRI) technology, etc., take two-dimensional slice images of the affected area in advance, and then watch and refer to the above images from time to time before and during the operation, and use their professional knowledge and experience to speculate and imagine The actual condition of the affected area is used to perform the operation. The difficulty of these high-risk surgical operations and the psychological pressure on the doctor can be imagined.

Therefore, in order for the doctor to be able to see as the truth, and to make it easy before and during the operation It is easy and intuitive to grasp the actual situation of the replacement. Therefore, the applicant in this case proposes a real-time dynamic surgical navigation system based on mixed reality. Through the application of mixed reality (MR) technology, the blood vessels, nerves, brain tissues, lesions and Surgical instruments, etc., are directly projected on the affected area, so that the doctor can intuitively grasp the actual condition of the affected area and the operation process at any time before and during the operation, thereby reducing the difficulty of the operation and reducing the pressure on the doctor.

However, in the above-mentioned mixed reality real-time dynamic surgical navigation system, how to achieve accurate alignment, real-time dynamic alignment, and real-time dynamic alignment in an open three-dimensional space between the affected part and the affected part image Positioning is another technical problem that needs to be overcome.

For this reason, the applicant, after careful experimentation and research, and with a spirit of perseverance, finally conceived the case "The digital image reality alignment kit and method for the mixed reality system integrated with surgical navigation", which can overcome The above shortcomings, the following is a brief description of the present invention.

In view of the positioning and alignment problems in the real-time dynamic surgical navigation system of mixed reality, the present invention uses two different function tracking markers to provide the mobile MR wearable device and the fixed optical tracker to perform real-time position. Tracking can determine the projection position of the MR real-time dynamic stereo 3D image, and provide the MR real-time dynamic stereo 3D image in different viewing angles.

Through the alignment technology proposed by the present invention, the MR device can accurately project the stereoscopic 3D images containing the actual conditions of the blood vessels, nerves and brain tissues of the patient's affected area on the affected area in real-time 3D visualization, and display it to the affected area through the MR device The doctor, no matter how the doctor moves during the operation, the stereoscopic 3D image displayed by the MR device can always be accurately positioned on the affected area, and With the change of the doctor's perspective, the 3D images of the affected part in different perspectives are displayed in real-time and dynamically correspondingly, and can be simultaneously provided to multiple doctors standing in different positions and different perspectives for viewing, allowing multiple doctors to synchronize Participate in complex surgery.

The alignment technology proposed by the present invention can be used as the core technology in the MR surgical navigation system. Finally, the MR glasses can display the pictures of the affected part, the patient and the surgical instruments in real time, assisting the physician to more intuitively plan the safe surgical path, 3D surgical navigation and Reduce the risk of cerebral artery damage during brain surgery.

Accordingly, the present invention provides a digital image reality alignment kit, which is applied to a mixed reality surgical navigation system, including: a plurality of moving markers, which have corresponding plural moving marker coordinate systems, and a part of these moving markers are for setting On the surgical instrument; the positioning mark, which is set near the affected part with the coordinate system of the affected part, has a definite positioning mark coordinate system after being set, and has a definite relative position between the moving marks of another part and the affected part , To convert the affected part coordinate system to the positioning mark coordinate system; and a mixed reality device, which detects the positioning mark and has a mixed reality device coordinate system, and accordingly converts the positioning mark coordinate system to the mixed reality Install the coordinate system and project the digital image of the affected part on the affected part.

Preferably, the digital image reality alignment kit further includes: a position tracker, which senses the movement marks, and has a definite position tracker coordinate system after setting, so that the movement marks of the part are located The corresponding moving marker coordinate systems and the digital affected part image coordinate system of the digital affected part image are converted to the affected part coordinate system based on the position tracker coordinate system.

Preferably, the digital image reality alignment kit further includes: a first registration device, which is arranged beside the affected part and has a first surface and a second surface opposite to the first surface. The first surface is for the positioning mark and the second surface is for the movement marks of the other part. When the first registration device is set beside the affected part, the affected part, the positioning mark, and the other part The relative positions of the moving marks are determined accordingly; and the second registration device is arranged beside the affected part and has plural surfaces, each surface is provided for the positioning mark, and one of the plural surfaces is provided The moving marks of the other part are arranged, and when the second registration device is set beside the affected part, the relative positions of the affected part, the positioning mark, and the moving marks of the other part are determined accordingly.

Preferably, the positioning mark, the movement marks of the other part, and the affected part have a certain relative position after being set, so the positioning mark coordinates and the movement marks of the other part are The corresponding coordinate systems of the moving markers and the coordinate systems of the affected part have a definite coordinate conversion relationship between each other as system design values.

Preferably, the digital image coordinates of the affected part are converted to the coordinate system of the affected part through the moving marker coordinate systems corresponding to the moving markers of the other part based on the coordinate system of the position tracker.

Preferably, the digital affected part image coordinates are converted to the position tracker coordinate system through an image registration algorithm, and the positioning mark coordinates are converted to the mixed reality device through a pinhole perspective projection positioning algorithm Coordinate system.

The present invention further provides a digital image reality alignment method, which is applied to a mixed reality surgical navigation system, including: providing a plurality of moving markers with a plural moving marker coordinate system, a positioning marker with a positioning marker coordinate system, and a mixed reality The mixed reality device of the device coordinate system and the affected part with the affected part coordinate system; one part of these moving marks is set on the surgical instrument; the positioning mark and the other part of the moving marks are set beside the affected part, And determine the relative position of the affected part, the positioning mark, and the movement mark of the other part to convert the coordinate system of the affected part to the positioning mark coordinate system; and detect the positioning mark through the mixed reality device, The coordinate system of the positioning mark is converted to the coordinate system of the mixed reality device, and the digital image of the affected part is projected on the affected part.

Preferably, the digital image real-world alignment method further includes one of the following: providing a position tracker with a position tracker coordinate system and the digital affected part image with a digital affected part image coordinate system; determining the position tracking The location of the device, and sense the movement marks through the position tracker, so that the movement mark coordinate system corresponding to the movement marks of the part and the digital image coordinate system of the affected part are based on the position tracker coordinate system Converting to the coordinate system of the affected part for the reference; and by determining the relative position of the affected part, the positioning mark, and the movement mark of the other part to each other, the coordinate system of the positioning mark and the movement mark of the other part are determined. Corresponding to these moving mark coordinate systems and the coordinate conversion relationship between the affected part coordinate systems.

100‧‧‧Mixed reality system integrated with surgical navigation system

106‧‧‧Locating mark

108‧‧‧Location Tracker

110‧‧‧Move the mark

112‧‧‧Surgical instruments

114‧‧‧Computer

116‧‧‧Computer Unit Module

118‧‧‧Flat Panel Display

200‧‧‧MR device

202‧‧‧MR display

204‧‧‧MR sensor

400‧‧‧First registration device

HX‧‧‧Three-dimensional cerebrovascular distribution image

SX‧‧‧Three-dimensional spine imaging

UR‧‧‧User

UH‧‧‧User head

PH‧‧‧Patient head

PW‧‧‧Patient's back waist

IM‧‧‧Affected part imaging model

SA‧‧‧affected part

PP‧‧‧Patient

UR1‧‧‧User

UR2‧‧‧User

UR3‧‧‧User

2001‧‧‧MR device

2002‧‧‧MR device

2003‧‧‧MR device

1061‧‧‧Locating mark

1062‧‧‧Locating mark

1063‧‧‧Locating mark

410‧‧‧Second Registration Device

ST: marker group

F _image : The coordinate system of the image model of the affected part

F _Probe : Surgical instrument coordinate system (moving mark coordinate system)

F _opt : position tracker coordinate system

F _patient : the coordinate system of the affected part

F _marker : Positioning marker coordinate system

F _hc : MR device coordinate system

:患部影像模型座標系→位置追蹤器座標系

: The coordinate system of the image model of the affected part → the coordinate system of the position tracker

:患部座標系→定位標記座標系

: Coordinate system of affected part → Coordinate system of positioning mark

:定位標記座標系→MR裝置座標系

: Positioning mark coordinate system → MR device coordinate system

:手術器械座標系→位置追蹤器座標系

: Surgical instrument coordinate system → position tracker coordinate system

:位置追蹤器座標系→患部座標系

: Position tracker coordinate system → affected part coordinate system

Figure 1 is a schematic diagram showing the architecture of the mixed reality system for surgical navigation applied by the kit and method of the present invention; Figure 2 is a schematic diagram showing the MR device proposed by the system of the present invention; Figure 3 is a schematic diagram showing the positioning mark proposed by the present invention Schematic diagram; Figure 4 is a schematic diagram of the position tracker proposed by the present invention; Figure 5 is a schematic diagram of the mobile marker proposed by the present invention; Figure 6 is a schematic diagram of the surgical instrument provided with a mobile marker provided by the present invention; Figure 7 shows a schematic diagram of the multiple coordinate systems included in the mixed reality surgical navigation system used in the present invention and their coordinate conversion; Figure 8 shows the image model of the affected part and the affected part involved in the alignment method proposed by the present invention Fig. 9 shows a schematic diagram of the coordinate conversion between the moving mark and the positioning mark of the registration device beside the affected part of the present invention; Fig. 10 shows the schematic diagram of the coordinate system definition of the characteristic pattern included in the positioning mark proposed by the present invention Figure 11 shows a schematic diagram of the coordinate conversion between the positioning mark of the present invention and the MR device; Figure 12 shows a schematic diagram of the coordinate conversion between the surgical instrument of the present invention and the affected part

Figure 13 is a schematic front view of the registration device used in the present invention; Figure 14 is a schematic back view of the registration device used in the present invention; Figures 15 and 16 are showing the user on the MR device of the present invention A schematic diagram of the overall mixed reality image viewed; and Figure 17 is a schematic diagram showing a scenario where the kit and method of the present invention provide multi-user operation.

The present invention will be fully understood by the following examples, so that those who are familiar with the art can complete it, but the implementation of the present invention is not limited by the following examples; The size, size, and scale are not limited. When the present invention is actually implemented, the size, size, shape and scale are not limited by the drawings of the present invention; the following will be combined with the drawings in the embodiments of the present invention to describe the present invention The technical solutions in the embodiments are described clearly and completely, but the described embodiments are only part of the implementation of the present invention Examples, not all examples. Based on the embodiments of the present invention, all other embodiments obtained by persons with ordinary knowledge in the field without creative activities shall fall within the protection scope of the present invention.

The term "preferred" as used herein is non-exclusive, and should be understood as "preferably but not limited to". Any steps described or recorded in any specification or claim can be executed in any order, and are not limited to those described in the claim The scope of the present invention should only be determined by the appended claims and their equivalent solutions, and should not be determined by the examples of implementation examples; when the term "including" and its variations appear in the specification and claims in this article, it is An open-ended term has no restrictive meaning and does not exclude other features or steps.

The applicant in this case applied for the title of the invention "Mixed Reality System Integrated with Surgical Navigation System" on the same day as the application date of this case on August 30, 108. The Republic of China Invention Patent Application No. 108131367, which The content here is fully incorporated into this case, as if it were put forward here.

In the present invention, Mixed Reality (MR) refers to the technology that allows the virtual digital content on the screen to be combined and interacted with the real environment scene by calculating the position and angle of the camera image and adding image analysis technology. Jiaco uses various sensors to sense environmental parameters, and calculates the position and direction of the virtual image in real space in real time. The virtual image is projected or superimposed on the real object through the display; the real environment and the virtual environment should be used as At the two ends of a continuous system, the presentation closer to the real environment is also called Augmented Reality (AR), the presentation closer to the virtual environment is also called Virtual Reality (VR), and MR is also visible It is a combination of AR and VR.

Figure 1 shows the architecture of the mixed reality surgical navigation system used in the present invention Schematic diagram; Figure 2 is a schematic diagram showing the MR device proposed by the system of the present invention; Figure 3 is a schematic diagram showing the positioning mark proposed by the present invention; as disclosed in Figure 1, the mixed reality surgical navigation system 100 of the present invention includes: MR device 200, MR display 202, multiple MR sensors 204, positioning mark 106, position tracker 108, movement mark (position mark) 110, surgical instrument 112, computer 114, computing unit module 116, flat-panel display 118 and other components, in The MR device 200 of this embodiment adopts but is not limited to the Microsoft HoloLens device, which can be worn by the user UR on the head UH. In this embodiment, the MR display 202 and the MR sensor 204 are preferably configured on the same device, that is, the MR device 200 , As shown in Figure 2; the positioning mark 106 is a flat mark containing a characteristic pattern, which can be read and identified by the MR sensor 204, as shown in Figure 3.

Fig. 4 is a schematic diagram showing the position tracker proposed by the present invention; Fig. 5 is a schematic diagram showing the mobile marker proposed by the present invention; and Fig. 6 is a schematic diagram showing the surgical instrument equipped with a mobile marker proposed by the present invention; position tracking The position tracker 108 and the moving marker 110 are matched with each other to track the current position of various surgical instruments 112 in real time. In this embodiment, the position tracker 108 is preferably an infrared active optical tracker. As shown in FIG. 4, the moving marker 110 is preferably a corresponding infrared passive reflection mark, which is designed and made into a sphere with a diameter of about 1 cm, which can be embedded on the surgical instrument 112. As shown in Figures 5 and 6, when the position tracker 108 After the infrared scan is activated, it can accurately track the current spatial coordinates of each moving marker 110 in a pyramid-like three-dimensional space and continuously report back to the system, so that the system can update the real-time space of surgical instruments at any time position.

Figure 7 shows a schematic diagram of the multiple coordinate systems included in the mixed reality surgical navigation system used in the present invention and their coordinate conversion; Figure 7 shows the image model IM of the affected part, surgical instruments 112, moving markers 110, and positions Tracker 108, affected area SA, patient PP, location marker 106. MR device 200, etc., in which the image model IM of the affected part is preferably made in advance before the operation, and is a series of virtual (virtual) three-dimensional images of the affected part constructed from a series of preferably but not limited to computer tomography (CT) slice images Preferably, the image content may include images of nerves, tissues, structures, organs, etc. of the inner layer of the affected area, which is regarded as a virtual augmented of the actual affected area.

In the above-mentioned mixed reality surgical navigation system, there are 6 independent coordinate systems at the same time, including the image model coordinate system F _{image of the} affected part, the moving marker coordinate system F _Probe representing the coordinate system of the surgical instrument, and the position tracker coordinate system F _opt , The coordinate system of the affected part is F _patient , the positioning marker coordinate system F _marker , the MR device coordinate system F _hc, etc., as listed in the following table:

In the above-mentioned coordinate system, the coordinate values of the affected part image model in the affected part image model coordinate system F _image are known values. After the position tracker, affected part, positioning mark, etc. are located and determined, the position tracker, affected part, The coordinate values of the positioning markers in the position tracker coordinate system F _opt , the affected part coordinate system F _patient , and the positioning marker coordinate system F _marker will also be determined and become known, and the affected part image model coordinate system F _image , position tracker The coordinate conversion relationship between the coordinate system F _opt , the affected part coordinate system F _patient , and the positioning marker coordinate system F _marker are also determined. Only the moving markers installed on the surgical instruments and the coordinate values of the MR device are changed, but By using the position tracker positioning mark and the positioning mark as the anchor conversion point, the image model of the affected part can be accurately aligned and projected on the affected part.

For the correspondence between the image model of the affected part and the affected part, it is better to use an image registration algorithm such as Interactive Closest Point (ICP) to correctly align the characteristic points of the affected part to the corresponding points of the image model. The coordinate conversion relationship between the model coordinate system F _{image of the} affected part image model and the position tracker coordinate system F _opt . Since the conversion relationship between the position tracker coordinate system F _opt and the affected part coordinate system F _patient is known, the position tracker The coordinate system F _{opt is} converted to the affected part coordinate system F _patient to complete the basic correspondence between the affected part image model of the position tracker and the affected part position.

For the correspondence between the surgical instrument and the affected part, the position tracker can track the moving mark to know the coordinate value of the surgical instrument in the position tracker coordinate system F _opt , which is equivalent to converting the surgical instrument coordinate system F _Probe to the position The tracker coordinate system is F _opt . Since the conversion relationship between the position tracker coordinate system F _opt and the affected part coordinate system F _patient is known, the surgical instrument coordinate system F _{Probe is} converted to the affected part coordinate system F _patient to complete the surgical instrument and The basic correspondence between the location of the affected part.

For the correspondence between the MR device worn by the doctor and the affected part, the movement mark of the registration device (Registration device) is tracked through the position tracker, and the coordinate system conversion relationship between the movement mark of the registration device and the location mark is the preset design value According to this, the affected part coordinate system F _patient can be converted to the positioning marker coordinate system F _marker through the position tracker coordinate system F _opt , and the MR device preferably captures the positioning mark through the MR sensor, and then uses the pinhole perspective projection positioning algorithm , The real-time coordinate conversion relationship between the positioning marker coordinate system F _marker and the MR device coordinate system F _hc can be calculated in real time, and the image model of the affected part can be accurately projected on the affected part through the MR device.

The coordinate conversion between the above coordinate systems is listed in the following table:

The focus of the above-mentioned coordinate conversion is how to perform coordinate conversion between the surgical instrument, the affected part and the MR device, so as to accurately align the image model of the affected part on the affected part.

Figure 8 discloses a schematic diagram of the coordinate conversion between the image model of the affected part and the affected part in the alignment method proposed by the present invention; Figure 9 discloses a schematic diagram of the coordinate conversion between the moving marker and the positioning mark beside the affected part of the present invention; Figure 10 discloses the present invention The schematic diagram of the coordinate system definition of the characteristic pattern included in the positioning mark proposed by the invention; Figure 11 shows a schematic diagram of the coordinate conversion between the positioning mark of the present invention and the MR device; Figure 12 shows a schematic diagram of the coordinate conversion between the surgical instrument of the present invention and the affected part.

The coordinate conversion relationship between the image model of the affected part and the affected part is as equation (1):

The coordinate conversion between the image model of the affected part and the affected part roughly includes five steps as follows:

Step 1. As shown in Figure 8, the main task before the operation is to capture images of the affected part, such as CTA images. After the capture is completed, the CTA images are reconstructed in 3D according to the imaging machine coordinate system to create the disease. The image model coordinates of the affected part are F _image .

，藉由與病患相對位置已知的病患(移動標記)座標系F _patient ，由F _opt 得

，進一步得到方程式(2)。 Step 2: Select three surface points of the patient's head model on F _image , and then use the positioning device F _probe with a moving mark to select the three points at the same position on the patient's head, and the three image points Make the initial alignment with the coordinates of the three clicked points, and then click the N point coordinates on the patient’s head, and use optimization methods such as ICP (Iterative Closest Point) algorithm to convert the N point coordinates to After the image coordinate system F _image and the corresponding image point coordinate have the smallest position error average value, get

, Using the patient (movement marker) coordinate system F _patient whose relative position is known to the _patient , and get F _opt

, And further obtain equation (2).

，進一步得到方程式(3)。 Step 3. As shown in Figure 9, the measurement through the position tracker can be obtained from the patient coordinate system F _patient and the positioning marker coordinate system F _marker

, And further obtain equation (3).

Step 4: As shown in Figure 10, define the coordinate system according to the characteristic pattern.

，透過先前座標系轉換得方程式(4)，最後得到方程式(5)。 Step 5. As shown in Figure 11, the camera F _hc locates F _marker through monocular vision to obtain

, Through the conversion of the previous coordinate system to obtain equation (4), and finally obtain equation (5).

The coordinate conversion between the surgical instrument and the affected part roughly includes four steps as follows:

，再經由病患座標系F _patient 由F _opt 得

，透過座標轉換得方程式(6)。 Step 1: As shown in Figure 12, the surgical instrument design value F _probe is obtained by F _opt

, And then obtain from F _opt through the patient coordinate system F _patient

, Equation (6) can be obtained through coordinate conversion.

，又由器械座標系F _probe 與定位標記座標系F _marker 進一步得到方程式(7) Step 2: As shown in Figure 9 can be obtained from the patient coordinate system F _patient and the positioning marker coordinate system F _marker

, And further get the equation (7) from the instrument coordinate system F _probe and the positioning marker coordinate system F _marker

，透過方程式(7)近一步得到方程式(8)，最後得方程式(9)。 Step 3: As shown in Figure 10, the MR device F _hc locates the F _marker through monocular vision.

, Through equation (7) to get equation (8) one step closer, and finally equation (9).

Figure 13 is a schematic diagram showing the front of the registration device used in the present invention; Figure 14 is a schematic diagram showing the back of the positioning mark used in the present invention; in order to have a definite correspondence between the positioning mark and the moving mark, Therefore, the present invention combines the moving mark and the positioning mark placed next to the affected part to be fixed and arranged on the front and back of a registration device, so that the positioning mark and the moving mark have a definite and unchanging positional correspondence relationship, and An anchored connection relationship; as shown in Figure 13, the first registration device 400 is pasted with a positioning mark 106 containing a characteristic pattern on the front side. As shown in Figure 14, the back of the first registration device 400 is provided with multiple A mark group ST composed of a moving mark 110, the moving mark 110 is preferably a corresponding infrared passive reflection mark, which is designed and made into a sphere with a diameter of about 1 cm. In clinical surgical applications, the first registration device 400 is changed to be fixed on the instrument fixed horizontal frame beside the operating table.

Figures 15 and 16 are schematic diagrams showing the overall mixed reality image viewed by the user on the MR device of the system of the present invention; Figure 15 mainly shows a three-dimensional cerebrovascular image The model is a mixed reality image displayed through the MR device after the alignment method of the present invention has been implemented to align the affected part; in this embodiment, the affected part is the head, so when the user wears the MR device, the visual The obtained image includes the patient’s head PH, and the three-dimensional cerebrovascular distribution image HX displayed on the lens of the MR device after being aligned with the affected area, because the three-dimensional cerebrovascular distribution image HX has been processed by the alignment method of the present invention with the affected area, that is, the patient The head PH is accurately aligned, so the user can directly see the mixed reality of the three-dimensional cerebrovascular distribution image HX superimposed on the affected area on the transparent display.

Figure 16 mainly shows a three-dimensional image model of the spine. After implementing the alignment method of the present invention and the affected area, the mixed reality image displayed by the MR device; in this embodiment, the affected area is the back waist, so the user wears it After the MR device, the visually observed image includes the patient's back waist PW and the three-dimensional spine image SX displayed on the lens after being aligned with the affected area. Because the three-dimensional spine image SX has been calculated and processed by the system, it has been compared with the patient's back waist. The PW is accurately aligned, so the user can directly see the mixed reality of the three-dimensional cerebrovascular distribution image HX superimposed on the affected area on the transparent display of the MR device.

Figure 17 is a schematic diagram showing the situation where the kit and method of the present invention are provided to multiple users; when multiple doctors need to participate in the operation, the second registration device 410 can be used to set multiple Positioning marks 1061, 1062, and 1063, and making each positioning mark 1061, 1062, and 1063 roughly correspond to the sight of each user UR1, UR2, and UR3, each user UR1, UR2, and UR3 wear one corresponding MR devices 2001, 2002 and 2003, and the second registration device 410 is also provided with a mark set ST composed of a plurality of moving marks 110 in the direction facing the position tracker 108, and a plurality of positioning marks 1061, 1062, and 1063 It is possible to share a set of marks ST composed of multiple mobile marks 110, and after determining multiple positioning After marking the relative positions between the markers 1061, 1062, and 1063, the marker set ST and the affected area SA, the kit and method proposed by the present invention can align the affected area model image with the affected area SA through MR devices 2001, 2002 and 2003, It is displayed to each user UR1, UR2 and UR3 who stand in different positions, orientations, and angles, and have different perspectives, allowing multiple doctors to simultaneously participate in complex operations.

The present invention uses two types of tracking markers with different functions. The first type is fixed positioning markers mainly provided for tracking by movable MR devices, and the second type is movable moving markers mainly provided for fixed optical trackers for real-time tracking. Position tracking, through the application of these two types of markers, the system can determine the projection position of the MR real-time dynamic 3D image, and provide MR real-time dynamic 3D images from different perspectives.

Through the alignment technology proposed in the present invention, the MR device can accurately project the three-dimensional 3D images containing the actual conditions of the blood vessels, nerves and tissues of the patient's affected area on the affected area in real-time 3D visualization and display it to the affected area through the MR device. The doctor, no matter how the doctor moves during the operation, the stereoscopic 3D image displayed by the MR device can always be accurately positioned on the affected area, and can change with the doctor’s perspective, instantly and dynamically showing the affected area in different perspectives. Stereoscopic 3D images can be simultaneously provided to multiple doctors standing in different positions and different perspectives for viewing, allowing multiple doctors to simultaneously participate in complex operations.

In the present invention, the first type of positioning mark containing the characteristic pattern is seen through the camera on the MR/AR glasses. The positioner uses infrared rays to sense the second type of movement mark of the passive spherical mark, and calculates the distance between the camera and the characteristic pattern and the spherical mark. After the position and direction are related, the virtual instrument and virtual medical image are displayed on the real instrument and patient through the display on the MR/AR glasses to achieve the purpose of assisting surgical navigation.

The alignment technology proposed by the present invention can be used as the core technology in the MR surgical navigation system. Finally, the MR glasses can display the pictures of the affected part, the patient and the surgical instruments in real time, assisting the physician to more intuitively plan the safe surgical path, 3D surgical navigation and Reduce the risk of cerebral artery damage during brain surgery.

Embodiment 1: A digital image reality alignment kit, which is applied to a mixed reality surgical navigation system, includes: a plurality of moving markers having corresponding plural moving marker coordinate systems, and a part of these moving markers are provided for On surgical instruments; positioning marks, which are set near the affected part with the coordinate system of the affected part, have a definite positioning mark coordinate system after being set, and have a definite relative position between the moving marks of another part and the affected part, To convert the coordinate system of the affected part to the positioning mark coordinate system; and a mixed reality device that detects the positioning mark and has a mixed reality device coordinate system, and accordingly converts the positioning mark coordinate system to the mixed reality device Coordinate system, and project a digital image of the affected part on the affected part.

Embodiment 2: The digital image reality alignment kit as described in Embodiment 1 further includes: a position tracker, which senses the movement marks, and has a certain position tracker coordinate system after setting, so that the part of the The moving mark coordinate systems corresponding to the moving marks and the digital affected part image coordinate system of the digital affected part image are converted to the affected part coordinate system based on the position tracker coordinate system.

Embodiment 3: The digital image reality alignment kit as described in Embodiment 1 further includes: a registration device, which is arranged beside the affected part and has a first surface and a second surface opposite to the first surface. One side is for the positioning mark and the second side is for the movement marks of the other part. When the registration device is set beside the affected part, the affected part, the positioning mark, and the other part The relative positions of the moving markers are determined accordingly.

Embodiment 4: The digital image reality alignment kit as described in Embodiment 1, wherein the positioning mark, the moving marks of the other part, and the affected part have a certain relative position after being set up, Therefore, the positioning mark coordinate system, the movement mark coordinate system corresponding to the movement marks of the other part, and the affected part coordinate system have a definite coordinate conversion relationship among each other as system design values.

Embodiment 5: The digital image reality alignment kit as described in Embodiment 2, wherein the image coordinates of the digital affected part are based on the coordinate system of the position tracker through the movement marks corresponding to the other part The marker coordinate system is moved to switch to the affected part coordinate system.

Embodiment 6: The digital image reality alignment kit as described in Embodiment 2, wherein the digital affected part image coordinate system is converted to the position tracker coordinate system through an image registration algorithm, and the positioning mark coordinate system is Transform to the coordinate system of the mixed reality device through a pinhole perspective projection positioning algorithm.

Embodiment 7: The digital image reality alignment kit as described in Embodiment 2, wherein the digital affected part image coordinate system, the position tracker coordinate system, the movement marker coordinate systems, the positioning marker coordinate system, and the mixed reality The coordinate system of the environment device and the coordinate system of the affected part are selected from the two-dimensional cassette coordinate system, the two-dimensional cylindrical coordinate system, the two-dimensional spherical coordinate system, the three-dimensional cassette coordinate system, the three-dimensional cylindrical coordinate system, the three-dimensional spherical coordinate system, and others. Combine one of them.

Embodiment 8: The digital image reality alignment kit as described in Embodiment 1, wherein the digital image of the affected part is constructed in advance before the operation is performed.

Embodiment 9: A digital image reality alignment method, applied to a mixed reality surgical navigation system, including: providing a plurality of moving markers with a plural moving marker coordinate system, a positioning marker with a positioning marker coordinate system, and a mixed reality Mixed reality installation of device coordinate system Set the location and the affected part with the coordinate system of the affected part; set a part of the moving marks on the surgical instrument; set the positioning mark and the other part of the moving marks beside the affected part, and determine the affected part and the positioning mark , And the relative position of the movement marks of the other part to convert the affected part coordinate system to the positioning mark coordinate system; and detect the positioning mark through the mixed reality device and convert the positioning mark coordinate system Go to the coordinate system of the mixed reality device, and project the digital image of the affected part on the affected part.

Embodiment 10: The digital image reality alignment method as described in Embodiment 9, further comprising one of the following: providing a position tracker with a position tracker coordinate system, and a digital affected part image with a digital affected part image coordinate system ; Determine the location of the position tracker, and sense the movement markers through the position tracker to the part of the movement markers corresponding to the movement marker coordinate system and the digital image coordinate system of the affected part, with the The position tracker coordinates are converted to the coordinate system of the affected part as a reference; and by determining the relative positions of the affected part, the positioning mark, and the movement mark of the other part, the coordinate system of the positioning mark and the other part are determined The moving mark coordinate systems corresponding to the moving marks and the coordinate conversion relationship between the affected part coordinate systems.

In the description of the present invention, the directions or position relations indicated by the terms "coaxial", "bottom", "top", "upper", and "inner" are the positions or position relations shown in the diagrams as The basis is only to facilitate the description of the present invention and simplify the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

In the present invention, unless otherwise clearly defined and limited, the terms "installation", "setup", "connection", and "fixed" should be understood in a broad sense. For example, it may be a fixed connection. It can also be detachably connected or combined into a whole; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two components or the connection of two components. The interaction relationship, unless otherwise clearly defined, for those with ordinary knowledge in the field, the specific meaning of the above terms in the present invention can be understood based on the actual situation.

The foregoing descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Even though the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still implement the foregoing The technical solutions described in the examples are modified, or some of the technical features are equally replaced. Any modification, equivalent replacement, improvement, etc., made within the spirit and principle of the present invention shall be included in the protection scope of the present invention within.

The various embodiments of the present invention can be combined or replaced arbitrarily to derive more implementation modes, but they do not deviate from the scope of protection of the present invention. The protection scope of the present invention is defined by the scope of the patent application The recorder shall prevail.

106‧‧‧Locating mark

108‧‧‧Location Tracker

110‧‧‧Move the mark

112‧‧‧Surgical instruments

200‧‧‧MR device

IM‧‧‧Affected part imaging model

SA‧‧‧affected part

PP‧‧‧Patient

Claims (10)

A digital image reality alignment kit, which is applied to a mixed reality system integrated with a surgical navigation system, includes: a plurality of moving markers with corresponding plural moving marker coordinate systems, and a part of these moving markers are for setting On a surgical instrument; a positioning mark, which is set near an affected part with a coordinate system of the affected part, has a certain positioning mark coordinate system after being set, and between the moving marks of another part and the affected part Having a definite relative position to convert the coordinate system of the affected part to the positioning mark coordinate system; and a mixed reality device that detects the positioning mark and has a mixed reality device coordinate system, based on which the positioning mark coordinate system The system is converted to the coordinate system of the mixed reality device, and a digital image of the affected part is projected on the affected part. The digital image reality alignment kit described in claim 1 further includes: a position tracker, which senses the movement markers, and has a certain position tracker coordinate system after setting, so that the part of the The moving mark coordinate system corresponding to the moving marks and a digital affected part image coordinate system of the digital affected part image are converted to the affected part coordinate system based on the position tracker coordinate system. The digital image reality alignment kit according to claim 1, further comprising: a first registration device, which is arranged next to the affected part and has a first side and a second side opposite to the first side The first surface is for the positioning marks and the second surface is for the movement marks of the other part. When the first registration device is set beside the affected part, the patient The relative positions of the moving marks of the part, the positioning mark, and the other part are determined accordingly; and a second registration device, which is arranged next to the affected part and has a plurality of surfaces, and each surface is provided for the positioning Mark setting, one of the plurality of surfaces is for setting the movement marks of the other part, when the second registration device is set beside the affected part, the affected part, the positioning mark, and the movement marks of the other part The relative position between each other is therefore determined. Such as the digital image reality alignment kit described in claim 1, wherein the positioning mark, the movement marks of the other part, and the affected part have a definite relative position between each other after being set, so the The positioning mark coordinate system, the movement mark coordinate systems corresponding to the movement marks of the other part, and the affected part coordinate system have a definite coordinate conversion relationship among each other as system design values. The digital image reality alignment kit according to claim 2, wherein the image coordinates of the digital affected part are based on the coordinate system of the position tracker and the movement marks corresponding to the movement marks of the other part The coordinate system is converted to the coordinate system of the affected part. Such as the digital image reality alignment kit described in claim 2, wherein the image coordinates of the digital affected part are converted to the position tracker coordinate system through an image registration algorithm, and the positioning marker coordinates are transmitted through A pinhole perspective projection positioning algorithm is converted to the coordinate system of the mixed reality device. Such as the digital image reality alignment kit described in item 2 of the request, wherein the digital affected part image coordinate system, the position tracker coordinate system, the movement marker coordinate system, the positioning marker coordinate system, The coordinate system of the mixed reality device and the coordinate system of the affected part are selected from a two-dimensional cassette coordinate system, a two-dimensional cylindrical coordinate system, a two-dimensional spherical coordinate system, a three-dimensional cassette coordinate system, and a three-dimensional cylindrical coordinate system , A three-dimensional sphere coordinate system, and one of its combinations. The digital image reality alignment kit described in claim 1, wherein the digital image of the affected part is constructed in advance before the operation. A digital image reality alignment method, which is applied to a mixed reality system integrated with a surgical navigation system, includes: providing a plurality of mobile markers with a plurality of mobile marker coordinate systems, a positioning marker with a positioning marker, a positioning marker with a coordinate system A mixed reality device coordinate system, a mixed reality device, and an affected part with an affected part coordinate system; a part of the movement marks are set on a surgical instrument; the positioning mark and the other part of the movement The mark is arranged beside the affected part, and the relative positions of the affected part, the positioning mark, and the movement mark of the other part are determined to convert the coordinate system of the affected part to the positioning mark coordinate system; and through the mixed reality The device detects the location mark, converts the location mark coordinate system to the mixed reality device coordinate system, and projects a digital image of the affected part on the affected part. The digital image reality alignment method described in claim 9 further includes one of the following: providing a position tracker with a position tracker coordinate system, and the digital affected part with a digital affected part image coordinate system Image; determine the location of the position tracker, and sense the movement markers through the position tracker to the part of the movement marker corresponding to the movement marker coordinate system and the number The image coordinate system of the affected part is converted to the coordinate system of the affected part based on the coordinate system of the position tracker; and the positioning mark is determined by determining the relative position of the affected part, the positioning mark, and the movement mark of the other part with each other The coordinate system, the movement mark coordinate systems corresponding to the movement marks of the other part, and the coordinate conversion relationship between the affected part coordinate systems.

Images