CN117898834A - Method for guiding endoscopic surgery, computer-readable storage medium, control device and computer program product, electronic device, navigation system and robotic system - Google Patents

Abstract

The present invention relates to a method for guiding endoscopic surgery, a corresponding computer-readable storage medium, a control device, a computer program product, and an electronic device, a navigation system, and a robot system for navigating endoscopic surgery. The method comprises: an endoscopic image acquisition step; an enhanced information acquisition step: acquiring at least two of the following three types of enhanced information: a) a spliced image obtained by splicing multiple images of an endoscope; b) a three-dimensional image of a patient's physiological structure; c) a mark on a three-dimensional image or a two-dimensional image of a patient's physiological structure; an endoscopic enhanced image fusion step: fusing the acquired current image of the endoscope with at least two types of enhanced information; and a step of displaying a view including at least a portion of the endoscopic enhanced image. The present invention fuses the current image of the endoscope with at least two of the above three types of enhanced information, so that the endoscopic image is enhanced in multiple dimensions, thereby achieving all-round and multi-dimensional guidance for the operator.

Description

Method for guiding endoscopic surgery, computer-readable storage medium, control device, computer program product, electronic device, navigation system, and robotic system

Technical Field

The present invention relates to the technical field of medical equipment, and in particular to a surgical navigation system, and more specifically to a method, electronic equipment, a navigation system and a robot system for guiding endoscopic surgery.

Background technique

In traditional endoscopic surgery, the endoscope is used for observation, tool operation and control. However, the doctor's field of vision is limited to the small range of the endoscope field of view, and the area outside the endoscope field of view cannot be seen, which results in the endoscope not being able to reach the affected area correctly or the operation being inadequate, resulting in failure to achieve the intended surgical goal.

In particular, endoscopic spinal surgery, due to its minimally invasive characteristics, is an effective means of treating nerve compression symptoms without pyramidal structural instability, and has been widely clinically recognized and promoted in recent years. However, due to the physiological structure of the intervertebral foramen and the special configuration of the spinal endoscope, doctors have to face the following challenges in the process of becoming familiar with and ultimately mastering endoscopic spinal surgery techniques:

First, the field of view under the microscope is limited. The spinal endoscope is used in the working channel sleeve, which causes the doctor's field of view to be blocked by the working channel sleeve during use. When the doctor pulls up the spinal endoscope and moves it away from the observation target point according to common thinking, hoping to observe a larger range of physiological structures, the first thing he sees is the tube wall of the working channel sleeve. If the doctor pulls up the working channel sleeve away from the target point, the soft tissue on the outside of the working channel sleeve will gather inward and invade the channel. What the doctor will see is the soft tissue on the outside of the working channel sleeve, which will block the structure of the target point that the doctor wants to observe. Therefore, he still cannot achieve the purpose of observing a larger range of physiological structures. "Seeing the trees but not the forest" is a figurative metaphor for the difficulty of limited field of view under spinal endoscopic surgery by spinal endoscopic surgeons.

Second, it is easy to get lost. In order to overcome the challenge of limited visual field under the microscope, the tip of the spinal endoscope is designed to have an angled bevel, and the lens of the imaging device is mounted on the angled bevel and arranged eccentrically relative to the axis of the endoscope, so that a wider range can be observed by rotating the endoscope around the axis.

However, it is precisely because of this configuration that beginners who are not familiar with the soft tissue structure under the endoscope will often get lost due to turning the endoscope.

Third, hand-eye coordination is difficult. The configuration of the spinal endoscope is that the optical hard endoscope module of the endoscope (also called the imaging device of the endoscope) and the channel of the surgical tool are integrated into the same endoscope insertion tube. When the doctor axially rotates the spinal endoscope to observe the target position, the surgical tool also rotates with the spinal endoscope. Because the actual position of the optical hard endoscope module and the surgical tool cannot change relative to each other due to the special configuration of the spinal endoscope, the intuitive result is that the doctor observes that the position of the surgical tool on the screen has not changed, but the actual situation at this time is that the position of the surgical tool relative to the physiological structure under the microscope has changed due to the axial rotation of the spinal endoscope. However, the doctor cannot "see through" the human body and intuitively see the position change of the tool relative to the surrounding physiological structure, but can only observe through the screen displaying the spinal endoscope image. Therefore, doctors are very likely to have "hand-eye coordination" difficulties. After each rotation of the endoscope, they need to re-adapt to the movement direction of the tool on the screen. The mind is always in a state of operation, which is very easy to cause fatigue.

Fourth, the "learning curve" is steep. Because of the difficulties faced in the above three endoscopic operations, spinal endoscopists need superior physiological and anatomical knowledge, excellent spatial imagination and more surgical case accumulation to overcome the so-called "learning curve". Usually, doctors need 30 to 50 surgeries or even more to master spinal endoscopy technology. This means that it often takes a long time and a high cost to train a qualified spinal endoscopy surgeon, which also limits the promotion and popularization of spinal endoscopy technology.

Moreover, traditional navigation technology is mostly used to guide the insertion of pedicle screws in spinal fixation surgery to track and locate tools and implants in real time relative to the patient's physiological structure, assisting doctors in achieving the clinical goal of "precision, safety, and minimally invasive". In recent years, navigation technology, especially electromagnetic navigation technology, has begun to be applied to spinal endoscopic surgery. The specific implementation method is: the real-time spatial position of the endoscopic tracer is tracked by the electromagnetic tracker in the navigation system, and the real-time position and field of view of the endoscope are calculated using the processor included in the navigation system, and it is projected onto a spatial three-dimensional image model or a two-dimensional perspective image model that has been matched with the patient's actual physiological structure through the navigation registration process, and this relative position relationship is displayed on the display of the navigation system. This solution has the following problem, that is, on the electromagnetic navigation interface, the doctor's simulated observation angle is consistent with the position of the X-ray source when the three-dimensional image or two-dimensional perspective image is obtained, and is located somewhere outside the patient's body. On the display that displays the endoscopic image of the spinal endoscope, the actual observation angle is located behind the lens of the endoscope hard endoscope module, inside the patient's body. Therefore, since the observation angles of the navigation view and the spinal endoscope image view are different, every time the doctor moves or rotates the spinal endoscope to change the position of the lens of the optical hard endoscope module of the spinal endoscope or adjusts the lens orientation, the doctor must combine his understanding of the anatomical structure and transform and unify the different observation angles through the thinking process, so as to "brain" the accurate position and direction of the physiological structure under the endoscope in the patient's body. As a result, the doctor's mind is always in a state of operation, which makes it easier to get tired.

Therefore, this navigation solution that tracks the endoscope like a common surgical tool and visualizes its spatial position and orientation is just a simple and direct application of navigation technology in endoscopic surgery, and does not effectively solve clinical needs such as limited visual field and difficulty in hand-eye coordination under the endoscope. Even for the need related to "positioning" of "easy to get lost under the endoscope", the current electromagnetic navigation technology has not substantially solved the problem that has long plagued spinal endoscopic surgeons.

How to leverage the advantages of the navigation system to solve the challenges of "limited visual field under the microscope", "easy to get lost", "difficult hand-eye coordination" and the steep "learning curve" encountered in current endoscopic surgeries such as spinal endoscopic surgery, so that beginners can also complete surgical operations accurately and efficiently with the assistance of the navigation system, is an urgent problem to be solved in this field.

Summary of the invention

The object of the present invention is to solve at least one of the above-mentioned problems and defects in the prior art as well as other technical problems.

In one aspect, the present invention provides a method for guiding endoscopic surgery, the method comprising the following steps:

Endoscopic image acquisition steps: acquiring an endoscopic image;

Enhanced information acquisition step: acquiring at least two of the following three types of enhanced information:

a) a stitched image obtained by stitching multiple images of the endoscope;

b) Three-dimensional images of the patient’s physiological structure;

c) marking on a three-dimensional image or a two-dimensional image of a patient's physiological structure;

Endoscopic enhanced image fusion step: fusing the acquired current image of the endoscope with at least two types of enhanced information acquired in the enhanced information acquisition step to obtain an endoscopic enhanced image; and

Displaying step: displaying a view including at least a portion of the endoscopically enhanced image.

In the scheme of this example, by fusing the current image of the endoscope with at least two of the above three types of enhanced information, the endoscopic image is enhanced in multiple dimensions (at least two) of soft tissue in a larger field of view around the end of the endoscope, bone structures that are invisible to the naked eye, and planning orientation information. Since the endoscopic enhanced image fuses the spliced images of a larger range around the end of the endoscope, it provides the operator with a larger field of view; since the three-dimensional image of the bone structure is fused, the operator can see the bone structure that was originally invisible and thus grasp the global orientation; since the guidance instructions related to the marker are fused, it is convenient for the operator to quickly locate the target structure position and determine the direction and orientation of the surgical tool. Even if there is an enhancement of the endoscopic image in the prior art, it is only enhanced in a limited way, and the enhancement of the endoscopic image is not achieved in multiple dimensions from at least two aspects at the same time. In the present invention, the operator can intuitively observe the current image under the endoscopic field of view and the soft tissue, bone structure, target position and direction information around it, so as to achieve all-round and multi-dimensional guidance for the operator. This solution perfectly solves the problems encountered in endoscopic surgery, such as "limited visual field under the microscope", "easy to get lost", "difficult hand-eye coordination" and "steep learning curve".

According to an example, the endoscopic enhanced image fusion step is performed by means of a navigation system, wherein the method further comprises before the endoscopic enhanced image fusion step:

Imaging orientation acquisition step: acquiring the orientation of the imaging device of the endoscope under the navigation system; and enhanced information orientation acquisition step: acquiring the orientation of the required fused spliced image, three-dimensional image or marker under the navigation system;

In the endoscopic enhanced image fusion step, the fusion is performed according to the orientation of the imaging device and the orientation of the spliced image, three-dimensional image or marker to be fused.

In this example, with the help of the navigation system, the images of the imaging device to be fused and the enhanced information are unified into the navigation coordinate system. The images obtained by this fusion method have higher fusion accuracy and higher navigation accuracy and navigation effect.

According to one example, the method further includes: a global enhanced image fusion step, wherein in the global enhanced image fusion step, at least two of the stitched image, the three-dimensional image and the guidance instructions associated with the marker are fused to obtain a global enhanced image, wherein the global enhanced image indicates the current field of view position of the imaging device and/or the position of the view of the endoscopic enhanced image; and a step of displaying the global enhanced image, wherein the edges of the current field of view position of the imaging device and the edges of the view of the endoscopic enhanced image are indicated by lines of different colors and/or line types on the global enhanced image.

In this example, the endoscopic enhanced view is displayed simultaneously with the global enhanced view as a part of the global enhanced image, and the correlation information between the two is displayed in the global enhanced image, that is, the edge of the current field of view of the imaging device and the edge of the view of the endoscopic enhanced image are prominently indicated by lines of different colors and/or line types on the global enhanced image, so that the operator can know the position of the endoscopic real-time image in the global image and the information relative to the surrounding soft tissue, bone structure, target position and direction while watching the larger endoscopic real-time image, thereby realizing all-round guidance for the operator.

According to an example, the type of enhancement information acquired and fused is selected based on operator input. This example enables the surgeon to flexibly select enhancement information for the endoscopic image according to his needs and preferences.

According to an example, the range of the endoscopic enhanced image displayed in the view of the endoscopic enhanced image can be determined according to the input of the operator. In this example, the range of the endoscopic enhanced image displayed in the view of the endoscopic enhanced image (corresponding to the size of the current image of the endoscope in the endoscopic enhanced view) can be determined according to the input of the operator. In other words, the operator can change the size ratio of the current image of the endoscope in the endoscopic enhanced view by interacting with the system as needed, such as zooming in or out the current image of the endoscope. When the doctor needs to observe a detail in the visual field under the microscope in more detail, the real-time image of the endoscope can be enlarged (correspondingly, the range that can be displayed in the view window of the entire endoscopic enhanced image becomes smaller), while the proportional relationship between the content displayed in the surrounding enhanced information and the real-time image of the endoscope remains unchanged, and the enhanced information beyond the edge of the "endoscopic enhanced view" is no longer displayed on the "endoscopic enhanced view".

According to an example, in the case where the fused enhancement information includes a stitched image, the enhancement information acquisition step includes a stitching step, in which multiple endoscopic images are stitched according to the orientation of an imaging device corresponding to at least one endoscopic image to obtain a stitched image. In this solution, since each image is stitched according to the orientation of an imaging device corresponding to at least one image, less calculations need to be processed compared to various existing stitching methods (e.g., based on algorithms), and the stitching speed is greatly improved, which is particularly important for real-time observation of endoscopic images during surgery.

According to an example, in the case where the fused enhanced information includes a stitched image, the enhanced information acquisition step further includes: a distortion calibration step before the stitching step, wherein the image of the endoscope to be stitched is subjected to distortion calibration in the distortion calibration step; and a processing step after the stitching step, wherein a plane image is generated according to the image obtained in the stitching step, wherein the plane image is fused with the current image of the endoscope as a stitched image in the endoscope enhanced image fusion step. In this example, the distortion effect of the endoscope is removed by the distortion calibration before stitching, so that the stitched image is closer to the real image. And in the processing step after stitching, the visual deformation of the stitched image caused by the different viewing angles is eliminated or reduced. The two distortion processing methods are combined to make the stitched image closer to the real surrounding scene, which is convenient for the operator to observe.

According to one example, when the fused enhanced information includes a marker, the enhanced information position acquisition step includes a marker position acquisition step, in which the position of the marker under the navigation system is acquired with the help of the alignment of the three-dimensional image or the two-dimensional image under the navigation system; wherein in the endoscope enhanced image fusion step, according to the position of the marker under the navigation system, the guidance instructions related to the marker are fused with the current image of the endoscope.

According to one example, the method enables an operator to select one or more of a directional marker and a marker indicating a physiological structure. This enables the operator to make a selection based on his or her preferences or actual scenarios. According to one example, the directional marker includes one or more of directional markers toward the dorsal, ventral, cranial, and caudal sides of the patient. According to one example, the patient's physiological structure is a spine, and the marker indicating the physiological structure includes an image model of a marked target (e.g., one or more of a protruding or free intervertebral disc, osteophyte, and ossified yellow ligament). The image model can be removed or separated from the image of the patient's physiological structure, in which case the image model is the marker indicating the physiological structure.

According to one example, the marker indicating the physiological structure includes physiological structure marker points; preferably, the patient's physiological structure is a spine, and the physiological structure marker points are marker points indicating one or more of a protruding or free intervertebral disc, osteophytes and ossified yellow ligaments, or are physiological structure marker points that do not displace during endoscopic surgery, such as one or more of the ventral side of the articular process, the pedicle of the anterior vertebra, the pedicle of the posterior vertebra, and the intervertebral disc; preferably, there are multiple physiological structure marker points.

According to an example, when the fused enhancement information includes a three-dimensional image of the patient's physiological structure, the enhancement information orientation acquisition step includes the step of registering the three-dimensional image of the patient's physiological structure to the navigation system to acquire the orientation of the three-dimensional image. Preferably, the three-dimensional image of the patient's physiological structure includes a preoperative three-dimensional image or an intraoperative three-dimensional image.

According to one example, the endoscope is a spinal endoscope.

The navigation method of the present invention is particularly beneficial for spinal endoscopes (such as transforaminal endoscopic endoscopes). As introduced in the background technology, the spinal endoscope is used in the working channel sleeve, which further limits the doctor's endoscopic field of view. The imaging device of the endoscope and the channel of the surgical tool are integrated in the same endoscope insertion tube, which makes the doctor prone to hand-eye coordination difficulties. How to prevent doctors from being affected by the special use of the spinal endoscope (the endoscope is used in the working channel sleeve) and the limited viewing angle of the endoscope optical hard lens module during spinal endoscopic surgery, reduce the "hand-eye coordination difficulties" and the problem of disorientation caused by the rotation of the endoscope, there has been no convenient and reliable solution. The method of the present invention solves the above problems by providing multi-dimensional enhanced information for endoscopic images, which can largely avoid doctors' misoperation and increase the reliability of spinal surgery.

According to an example, the imaging position acquisition step is performed by acquiring the position of a tracer having a fixed positional relationship relative to the endoscope from a tracking device of a navigation system. The method conveniently realizes the position tracking of the imaging device of the endoscope by means of the navigation system.

According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the methods in the above examples are executed.

According to yet another aspect of the present invention, a control device is provided. The control device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the methods in the above examples are executed when the processor runs the program.

According to yet another aspect of the present invention, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the steps of the methods in the above examples are implemented.

According to another aspect of the present invention, there is also provided an electronic device for endoscopic surgery navigation, characterized in that the electronic device comprises a display device and a processor, the processor having a data interface, wherein the data interface can be connected to an endoscope, so that the processor can obtain an image of the endoscope; and wherein the processor is configured to obtain one or more of a three-dimensional image of a patient's physiological structure and a mark on the three-dimensional image or two-dimensional image of the patient's physiological structure; wherein the processor is configured to display a view of at least a portion of an endoscopic enhanced image on the display device for at least a period of time when the processor is running, wherein the endoscopic enhanced image is an image that fuses a current image of the endoscope with at least two of the following three types of enhanced information:

a) a stitched image obtained by stitching multiple images of the endoscope;

b) Three-dimensional images of the patient’s physiological structure;

c) Guidance instructions related to the marking.

According to another aspect of the present invention, there is also provided a navigation system for endoscopic surgery, the navigation system comprising a tracking device suitable for tracking a tracer having a fixed positional relationship relative to an endoscope, a display device and a processor, wherein the processor is suitable for being connected to the tracking device and the endoscope, wherein the methods in the above-mentioned examples are executed when the processor is running, and the display in the method is realized through the display device.

Because the navigation system integrates a larger range of stitched images around the endoscope, it provides the operator with a wider field of view; it integrates the three-dimensional image of the bone structure, allowing the operator to see the bone structure that is otherwise invisible and grasp the overall orientation; it integrates directional marks or marks indicating physiological structures or their guiding instructions to facilitate the operator to quickly locate the target position and grasp the orientation of the surgical tools, thus having better navigation effects and navigation accuracy.

According to another aspect of the present invention, a robot system is provided, the robot system comprising a robot arm and the above navigation system. In other words, the concept of the present invention can also be implemented in the navigation system of the robot system.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in detail below by way of exemplary embodiments with reference to the accompanying drawings.

FIG. 1 exemplarily shows a flow chart of a method for guiding endoscopic surgery according to the present invention.

FIG. 2 is a schematic diagram showing a principle of a navigation system for spinal endoscopic surgery according to an exemplary embodiment of the present invention.

FIG. 3 exemplarily shows an endoscopic enhanced image obtained by fusing a stitched image, a three-dimensional image of a patient's physiological structure, and a current image of an endoscope.

FIG. 4 exemplarily shows a view in which the endoscopy enhanced image of FIG. 3 is displayed on a window of a display device.

FIG5 exemplarily shows a view of an endoscopically enhanced image displayed on a window of a display device according to another exemplary embodiment, wherein the endoscopically enhanced image fuses a stitched image, a three-dimensional image of the patient's physiological structure, physiological structure markers, and a current image of the endoscope.

FIG. 6 exemplarily shows a display interface of a navigation system having an endoscopic enhanced image view and a global enhanced image.

FIG. 7 exemplarily shows a global enhanced image that is a fusion of a stitched image, a three-dimensional image of a patient's physiological structure, and a direction mark, wherein the current field of view position of the imaging device and the position of the view of the endoscopic enhanced image are indicated.

FIG8 exemplarily shows a global enhanced image that fuses a stitched image, a three-dimensional image of a patient's physiological structure, and physiological structure markers, wherein the current field of view position of the imaging device and the position of the view of the endoscopic enhanced image are indicated.

FIG. 9 exemplarily shows a flow chart of an enhancement information acquisition step including a stitching step to acquire a stitched image.

FIG. 10 exemplarily shows a schematic diagram of the principle of moving the endoscope to obtain multiple endoscopic images of a larger range.

FIG. 11 is a schematic diagram showing an exemplary principle of rotating an endoscope to obtain a stitched image.

FIG. 12 shows an example of an operator marking a direction on a preoperative 3D image or an intraoperative 3D image.

FIG. 13 shows an example of an operator marking a direction on an image after performing quasi-three-dimensional fitting on a two-dimensional perspective image.

FIG. 14 shows an example of an operator completing physiological structure marking on a preoperative 3D image or an intraoperative 3D image.

FIG. 15 shows an example of an operator completing physiological structure marking on an intraoperative anteroposterior image.

FIG. 16 shows an example of an operator completing physiological structure marking on an intraoperative lateral image.

It should be noted that the drawings are only schematic. They only show those parts or steps required to illustrate the present invention, while other parts or steps may be omitted or only briefly mentioned. In addition to the parts or steps shown in the drawings, the present invention may also include other parts or steps.

Detailed ways

The technical solution of the present invention is further specifically described below by examples and in conjunction with the accompanying drawings. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall concept of the present invention and should not be construed as limiting the present invention.

As a specific embodiment, the following describes the specific steps of the method of image-guided endoscopic surgery using an endoscope (e.g., a spinal endoscope, a neuroendoscope, etc.) under a navigation system of the present invention, as well as the electronic equipment, navigation system, and robot system (or positioning and navigation system) involved. In the following detailed description, many specific details and steps are described in a very specific and detailed manner to provide a comprehensive understanding of the embodiment. However, it should be understood that one or more other embodiments may also be implemented without these specific details and steps.

It should be noted that, although the term "imaging device" and "image" of an endoscope is used herein, those skilled in the art can understand that "imaging device" is a broad concept, including functions such as video recording, video acquisition and image acquisition, and "image" is a broad concept, including video, dynamic continuous images and static images. In this article, the "imaging device" of an endoscope can be an endoscope module used for video imaging of the endoscope, and what is acquired in the image acquisition step of the present invention is a framed image of the image from the endoscope.

In a specific embodiment of the present invention, the navigation system used includes a tracking device 1, a control device 2 and a display device 3 as shown in FIG2. The tracking device 1 can be an optical tracking device (such as an NDI navigator), and a tracer 4 can be set on the endoscope 5 accordingly. As a specific example, the control device 2 can be a general-purpose computer, a special-purpose computer, an embedded processor, or any other suitable programmable data processing device such as a single-chip microcomputer or a chip. The control device 2 can include a processor and a memory for storing programs, but can also include only a processor, in which case the processor can be attached to a memory storing programs. In other words, the control device includes at least a processor. The control device 2 (or processor) and the display device 3 can be integrated into one or separately arranged. The control device or processor has a data interface, which can include a data interface that can be connected to an endoscope, so that the control device/processor can obtain the image of the endoscope in real time. The control device or processor also includes a data interface that can be connected to the tracking device 1 of the navigation system, so that the position and direction of the tracked target, such as the tracer 4 on the endoscope, can be obtained from the tracking device 1 in real time. As an example, the endoscope 5 and/or the tracer 4 can also be regarded as part of the navigation system of the present invention.

Usually, the end of the endoscope enters the tissue structure and bone structure of the patient to observe and/or operate it, and the proximal end of the endoscope (the end close to the operator, that is, the end opposite to the end of the insertion tube of the endoscope) is located outside the patient's body for the operator to manipulate. In the navigation system, by setting a tracer 4 suitable for being tracked by the tracking device 1 at the proximal end of the endoscope located outside the patient's body, the navigation system can obtain the position and direction of the tracer 4 on the endoscope 5 in the navigation coordinate system in real time. As an example of the present invention, the imaging device of the endoscope, that is, the distal lens of the optical hard lens module, is set at the end of the insertion tube of the endoscope. By calibrating the relative position relationship of the end of the insertion tube of the endoscope with respect to the tracer 4 on the endoscope 5, the relative position relationship of the imaging device with respect to the tracer 4 can be obtained, so that the orientation of the tracer 4 can be known through the tracking device 1 in the navigation system, and then the position and direction of the imaging device in the navigation coordinate system can be known. This calibration is usually performed before performing endoscopic surgery and its navigation, and is also called the calibration of the external parameters of the endoscope.

Specifically, before performing endoscopic surgery and its navigation, the relative position relationship of the imaging device of the endoscope, i.e., the hard mirror module, with respect to the tracer 4 can be calibrated first. Preferably, in this embodiment, the calibration is performed by a calibration tool (not shown in the figure) without the need to obtain images with the help of an endoscope, which reduces the workload of image processing. The navigation system of the present invention optionally includes the calibration tool. A plurality of calibration holes may be formed on the calibration tool, and these calibration holes have bottom surfaces with different inclination angles and/or different apertures to calibrate endoscopes with different terminal bevel inclination angles and/or barrel diameters. Specifically, the calibration tool also includes another tracer fixed on its bracket, and the navigation system can know the position of the tracer on the calibration tool in the navigation coordinate system. Moreover, the positional relationship of each calibration hole relative to the tracer on the calibration tool can be known according to the design size of the calibration tool, and thus the positions of these calibration holes in the navigation system are known. Insert the end of the insertion tube of an endoscope to be calibrated into the matching calibration hole, and fit the inclined surface of the end of the insertion tube with the inclined bottom surface of the calibration hole to achieve the positioning of the endoscope. Since the navigation system can also know the position of the tracer 4 on the endoscope in the navigation coordinate system, the relative position relationship between the end of the endoscope insertion tube and the tracer 4 on the endoscope can be known to complete the calibration process.

During the operation, the position (position and direction) of the tracer 4 on the endoscope can be obtained in real time by the tracking device 1, so that the position and direction of the endoscope end can be obtained indirectly in real time, that is, the position of the imaging device can be obtained. The imaging position acquisition step in the method according to the present invention is performed in this way.

In this specific embodiment of the present invention, before performing endoscopic surgery and its navigation, the imaging parameters of the endoscope, also known as internal parameters, can also be calibrated. Exemplarily, the calibration of the internal parameters can use a calibration plate, for example, using the endoscope to photograph the calibration plate at different azimuth angles, save the image and record the specifications of the calibration plate. Then, the internal parameters of the endoscope are calibrated, including the distortion matrix and/or the rotation and displacement vectors. The internal parameters calibrated here will be used in the distortion calibration process to be described later.

The following will describe in detail the steps of a specific embodiment of the method for guiding endoscopic surgery of the present invention in conjunction with Figure 1. Before the operation begins, the internal and external parameters of the endoscope can be calibrated as shown above. And before the operation and navigation begin, the navigation system is aligned, that is, the navigation coordinate system is determined. The method of the present invention can be used in scenarios where two-dimensional images are used for navigation, and can also be used in scenarios where three-dimensional images are used for navigation. The alignment method of the navigation system is a known method and will not be described in detail here.

During the endoscopic surgery, the control device automatically acquires the real-time image of the endoscope. And, the type of enhancement information for enhancing the real-time image of the endoscope is determined according to the input of the operator. The input of the operator can be selected by the operator according to his own needs, for example, the instruction can be input through various input components such as keyboard, mouse, handle, foot switch, etc. In the present invention, the following scheme is provided to the operator, that is, at least two of the following three types of enhancement information can be fused on the endoscopic image, namely, a spliced image obtained by splicing multiple images around the end of the endoscope, or a three-dimensional image of the patient's physiological structure, and, for example, a direction mark or a mark indicating the physiological structure made by the operator on the three-dimensional image or two-dimensional image of the patient's physiological structure during the surgical planning process.

Among the three types of enhanced information mentioned above, the stitched image (indicated by reference numeral 6 in FIGS. 3 and 4 ) may also be referred to as a “complete three-dimensional map” or a “panoramic image”. However, it should be noted that the “complete” or “panoramic” here only refers to an image of a larger range relative to the limited endoscopic field of view of the endoscope, and does not necessarily mean a 360-degree panoramic image. This can be obtained, for example, by the doctor according to his observation needs during the operation, by flexibly translating (for example, as shown in FIG. 10 ) and/or rotating (for example, as shown in FIG. 11 ) the endoscope within a larger range that needs to be observed. FIGS. 10 and 11 also exemplarily show the conical field of view 51 of the endoscope 5 at three positions. The conical field of view 51 represents the field of view that the imaging device of the endoscope can see.

The stitched image around the end of the endoscope enables the doctor to observe a larger field of view (or a global field of view), and can better display the soft tissue around the endoscope end, such as nerves, dura mater, blood vessels, intervertebral discs, etc., so that the doctor can quickly determine the direction of the endoscope and the spatial position relationship of key physiological structures relative to the current image of the endoscope at this moment, eliminating the defect of the limited field of view of the endoscope. The acquisition and fusion of the stitched image will be further described below.

The three-dimensional image of the patient's physiological structure displays the three-dimensional bone structure from the CT or CBCT image. Therefore, when it is fused to the endoscopic image, the doctor can observe the three-dimensional bone structure that is covered by soft tissue and cannot be observed under the endoscopic field of view on the endoscopic enhanced image (the dotted part pointed to by the figure mark 7 in Figure 4 represents the three-dimensional image of the three-dimensional bone structure), thereby obtaining the global position information of the endoscope tip and the surgical tool tip, for example, in the entire spinal structure.

For the above-mentioned marks on the three-dimensional image or two-dimensional image of the patient's physiological structure, it can be marked by the operator in the surgical plan, for example, including the direction, physiological structure landmarks, and image models of marked targets (such as protruding or free intervertebral discs, osteophytes, or ossified yellow ligaments) marked in the surgical plan, and/or target information. When the guidance instructions related to the mark (which may include the mark itself, such as direction marks, physiological structure landmarks, image models of physiological structure targets, and indicators pointing to the mark such as arrows, which are collectively referred to as guidance instructions related to the mark) are fused to the endoscopic image, there are orientation guidance instructions on the displayed endoscopic image, such as direction marks, physiological structure marks, target image models, etc. Therefore, the doctor can quickly determine the orientation of the physiological structure under the endoscope at this moment, thereby guiding the operation of the endoscopic tool. The making of the mark and the acquisition and fusion of its orientation will be further described below.

In summary, in this solution of the present invention, by fusing the current image of the endoscope (the framed image from the real-time image of the endoscope) with at least two of the above three types of enhanced information, the endoscopic image is enhanced from multiple angles of bony structure, soft tissue and/or planning mark position indication information. The fusion of a larger range of spliced images around the endoscope provides the operator with a wider field of view; the fusion of the three-dimensional image of the bony structure allows the operator to see the bony structure that was not originally visible and grasp the global orientation; the fusion of direction marks or marks indicating physiological structures or their guiding instructions facilitates the operator to quickly locate the target position and determine the direction. It perfectly solves the problems of "limited field of view under the microscope", "easy to get lost", "difficult hand-eye coordination" and "steep learning curve" encountered in endoscopic surgery.

Specifically, as shown in FIG1 , the method of this specific embodiment of the present invention includes an endoscopic image acquisition step, where the endoscopic image can be a frame of the endoscopic image acquired in real time by the control device. At the same time, the control device acquires and records the position and direction of the imaging device of the endoscope corresponding to at least one image of the endoscope (for example, corresponding to each image), that is, the imaging orientation acquisition step in FIG1 . In this imaging orientation acquisition step, the processor acquires the position of the tracer 4 located outside the patient's body and disposed at the distal end of the endoscope 5 from the tracking device 1 of the navigation system, and indirectly acquires the position and direction of the imaging device in combination with the external parameters calibrated above. The endoscopic image acquisition step here includes the acquisition of the current image of the endoscope fused in the endoscopic enhanced image fusion.

As shown in FIG1 , the method of the present invention further includes an enhanced information acquisition step, wherein the type of enhanced information to be acquired is determined based on the input of the operator. The input component for the operator to input may include, for example, a selection box, a dialog box, etc. displayed on a display device. The operator may, for example, select two types of enhanced information, namely, a stitched image and a three-dimensional image of the patient's physiological structure, or may select two types of enhanced information, namely, a stitched image and a mark, or may select two types of enhanced information, namely, a three-dimensional image of the patient's physiological structure and a mark, or three types of enhanced information, namely, a stitched image, a three-dimensional image of the patient's physiological structure, and a mark. The present invention can acquire corresponding enhanced information based on the above selections, and correspondingly acquire the orientation information of these enhanced information, so as to be used for fusing the endoscopic enhanced image in the fusion step according to the orientation of the imaging device and the orientation of the stitched image, three-dimensional image, or mark to be fused.

Specifically, when the enhanced information to be fused includes a stitched image, the enhanced information acquisition step includes a step of stitching the images of multiple endoscopes. FIG. 9 shows an enhanced information acquisition step including a stitching step. In the image acquisition step shown in FIG. 9 , the control device automatically acquires multiple images around the end of the endoscope. At the same time, the control device automatically acquires and records the position and direction of the imaging device of the endoscope corresponding to at least one image of the endoscope (e.g., corresponding to each image), i.e., the imaging orientation acquisition step in FIG. 9 . The above steps are repeated at multiple positions and orientations of the endoscope, so that after, for example, the endoscope is translated (as shown in FIG. 10 ) and/or rotated (as shown in FIG. 11 ) within a larger range to be observed, images of the imaging device in multiple orientations are obtained; at the same time or thereafter, the multiple images of the endoscope can be calibrated for distortion in combination with the internal parameters of the imaging device calibrated above to remove the distortion effect of the imaging device, i.e., the distortion calibration step in FIG. 9 . Afterwards, the images are stitched according to the position and direction (i.e., orientation) of the endoscope corresponding to at least one image, so as to obtain a stitched image.

In this scheme, since the images are stitched together according to the position of the imaging device corresponding to at least one image to obtain the stitched image, compared with various existing stitching methods (e.g., based on algorithms), the required calculations are less, and the stitching speed is greatly improved, which is particularly important for real-time observation during surgery. In addition, the stitching accuracy of this scheme is higher, and the computing power requirements of the processor are lower. It should be noted that in the imaging position acquisition step, the position and/or direction of the imaging device under the navigation system corresponding to at least one image in the stitched images are acquired. Exemplarily, the position of the imaging device under the navigation system can be acquired corresponding to each acquired image, but it is also possible to acquire only the position and/or direction of the imaging device corresponding to part of the images or even only one image. For example, in some cases, it is only necessary to acquire the position and/or direction of the imaging device corresponding to certain images at intervals in time, or it is only necessary to acquire the position and/or direction of the imaging device when the image to be stitched is acquired.

As shown in FIG9 , the enhanced information acquisition step may also include a processing step after the stitching step, in which a planar image is generated based on the image obtained in the stitching step to reduce visual distortion, and then the planar image is used as a stitched image to be fused with the current image of the endoscope in the endoscope enhanced image fusion step (for example, it is placed around the current image of the endoscope). This method eliminates or reduces the distortion of the stitched image caused by different viewing angles.

Since in the above stitching step, the images are stitched according to the orientation of the imaging device corresponding to at least one image to obtain the stitched image, the orientation information of the stitched image under the navigation system is known. Therefore, in the enhanced information orientation acquisition step, the orientation of the stitched image under the navigation system can be acquired, so as to be used to fuse the stitched image with the endoscope current image and other enhanced information according to the orientation in the subsequent endoscope enhanced image fusion step.

When the enhanced information to be fused includes the markers mentioned above, the enhanced information position acquisition step includes a marker position acquisition step, in which the position of the marker under the navigation system is acquired with the help of the alignment of the three-dimensional image or the two-dimensional image under the navigation system; wherein in the endoscope enhanced image fusion step, according to the position of the marker under the navigation system, the guidance instructions related to the marker are fused with the current image of the endoscope and other enhanced information.

Specifically, in this article, the image on which the operator makes a mark can be a preoperative three-dimensional image, such as a preoperative CT image, or an image obtained during surgery, such as an intraoperative CBCT image or an intraoperative two-dimensional fluoroscopic image. When the navigation system uses the preoperative CT image for navigation, the operator makes a mark on the preoperative three-dimensional image. When the navigation system uses the intraoperative CBCT image or the intraoperative two-dimensional fluoroscopic image for navigation, the operator makes a mark on the image after obtaining it during surgery.

In the present invention, the operator can choose to make various marks, such as directional marks or marks indicating physiological structures, thereby providing the operator with a variety of choices and possibilities. In the example where the patient's physiological structure is the spine, the mark indicating the physiological structure can, for example, include an image of the target tissue or target bony structure (such as a protruding or free intervertebral disc, an ossified yellow ligament, or osteophytes generated by degeneration) removed by the operator during surgical planning, or it can be a physiological structure marker. For the operator, if the target tissue or bony structure can be distinguished in the three-dimensional image used in the surgical planning (for example, osteophytes can be distinguished in CT images, or protruding or free intervertebral discs can be distinguished in MRI images), then its image model is directly formed in the surgical planning. The image model is used as the marker to enhance information and is fused into the endoscopic enhanced image. If the target tissue or target bony structure cannot be displayed in the three-dimensional image used for surgical planning, for example, a CT image is used in which the intervertebral disc cannot be displayed, the doctor can refer to the MRI image to mark points indicating the target tissue or target bony structure in the surgical planning (for example, marking points indicating one or more of a protruding or free intervertebral disc, osteophytes and ossified yellow ligament), that is, physiological structure marking points.

Physiological structure markers may also include physiological structure markers that do not shift during endoscopic surgery. As shown in FIG. 14 , the operator may also mark (for example, using preoperative planning software) certain physiological structure markers in or around the intervertebral foramen that do not shift during the entire spinal endoscopic surgery on the three-dimensional image, including but not limited to physiological structure points such as the ventral side of the articular process, the pedicles of the anterior vertebral body, the pedicles of the posterior vertebral body, and the intervertebral disc. As shown in FIGS. 15 and 16 , when using intraoperative two-dimensional fluoroscopic images, the doctor uses the intraoperative planning software to select physiological structure markers on the intraoperative anteroposterior image (FIG. 15) and lateral image (FIG. 16), including but not limited to physiological structure points such as the ventral side of the articular process, the pedicles of the anterior vertebral body, the pedicles of the posterior vertebral body, and the intervertebral disc.

As shown in Figures 12 and 13, the operator can make direction marks on the preoperative or intraoperative three-dimensional image (Figure 12), such as arrow marks toward the dorsal, ventral, cephalic and caudal sides of the patient. The operator can also mark the direction on the intraoperative two-dimensional image. Preferably, in order to facilitate the operator's marking, the intraoperative two-dimensional image (for example, the intraoperative anteroposterior and lateral images) can also be fitted to a three-dimensional pattern first, and the operator marks the direction on the fitted image (as shown in Figure 13). Compared with the method of marking directly on the intraoperative two-dimensional image, such as the anteroposterior image and the lateral image, this method can solve the problem of easily marking the head and tail in reverse when marking on the two-dimensional image.

Those skilled in the art will appreciate that, although some specific examples of direction marks and examples of marks indicating physiological structures are described herein, the selection of direction marks or marks indicating physiological structures can be freely selected according to the preferences and needs of the operator and is not limited to these examples. Moreover, although in the embodiments shown in FIGS. 12-16 , four direction marks or physiological structure marking points are shown, the number thereof can be set to one, two, three, or more than three as needed.

For the enhancement method of making marks on the preoperative three-dimensional image, the processor first obtains the preoperative three-dimensional image and simultaneously obtains the marks made by the operator on the preoperative three-dimensional image as described above (such as direction marks or marks indicating physiological structures). Then, the preoperative three-dimensional image is registered to the navigation system, and the coordinates of each mark under the navigation system are determined according to the registration relationship determined in the registration step, thereby completing the step of obtaining the mark position.

For the enhancement method of making a mark on the intraoperative 3D image or the intraoperative 2D image, usually, the registration of the image to the navigation system is completed at the same time as the intraoperative 3D image or the intraoperative 2D image is acquired. Therefore, in this case, the operator makes a mark on the intraoperative 3D image or the intraoperative 2D image after the registration is completed. Since the coordinates of the registered image under the navigation system are known, the coordinates, i.e., the position of the mark made in the image under the navigation system can be obtained accordingly, thereby completing the step of obtaining the mark position.

As described above, in the case of using intraoperative two-dimensional images, quasi-three-dimensional fitting can be performed on the two-dimensional images of at least two body positions to obtain a three-dimensional-like image to facilitate the operator's marking, and the mark obtained by the processor afterwards is the mark made by the operator on the fitted image. Those skilled in the art can understand that the image used for quasi-three-dimensional fitting can be an intraoperative anteroposterior image and an intraoperative lateral image, or an image of other body positions, which can be selected according to the needs of the operator or the actual situation of the part of the patient to be operated on.

The operator can input or select the various forms of marks described above in various ways. This can be input through the interface of the display device, for example, through the touch interface of the display device and using the preoperative planning software or intraoperative planning software to select the direction marks of the head, tail, ventral, and dorsal sides, or the selection of various physiological structure marks, as shown in Figures 12 and 13. It can also be input through the keyboard and/or mouse of the electronic device, as shown in Figure 14.

It should be noted that, although in the description herein, the mark is formed by an operator making a mark on the image of the patient's physiological structure, it can be understood that in some other embodiments, the mark may not be made by the operator, for example, the mark is automatically formed when the image is formed, which can be generated when the image is acquired by pre-setting a marker on a certain body part of the patient or a position such as an operating table.

In the case where the fused enhanced information includes a three-dimensional image of the patient's physiological structure, the enhanced information acquisition step acquires the three-dimensional image. And because the three-dimensional image used for navigation has determined position information under the navigation system through registration, the position of the three-dimensional image under the navigation system can be easily acquired.

After obtaining the enhanced information (stitched images, three-dimensional images or markers) required to be fused and the orientation of each enhanced information under the navigation system as described above, the fusion of the endoscopic enhanced image can be performed in combination with the acquired orientation information of the endoscopic imaging device. Since each enhanced information required to be fused has a certain orientation relationship with the imaging device under the navigation system, and thus each enhanced information required to be fused has a certain orientation relationship with the image acquired by the imaging device under the navigation system, the fusion of each enhanced information and the current image of the endoscope can be performed based on these orientation relationships.

It should be understood that although the above steps are listed and described in sequence in the flowchart of FIG. 1 , the claims, and the description herein, it does not mean that there is a specific order relationship between the various steps in these steps. For example, the enhanced information acquisition step and the enhanced information orientation acquisition step can be performed before, after, or simultaneously with the endoscopic image acquisition step and/or the imaging orientation acquisition step. The global enhanced image fusion step (described below) can also be performed before, after, or simultaneously with the endoscopic enhanced image fusion step.

FIG3 shows an example of the obtained endoscopic enhanced image, in which a stitched image 6, a three-dimensional image 7 of the patient's physiological structure and a current image 10 of the endoscope are fused. FIG4 exemplarily shows a view (also called an endoscopic enhanced view) in which the endoscopic enhanced image in FIG3 is displayed on a window of a display device, wherein the current image of the endoscope (i.e., the real-time image) is located in the central area of the window. The endoscopic image displayed on this view is larger, and the surrounding enhancement information is shown at the same time. Exemplarily, the range of the endoscopic enhanced image displayed in the view of the endoscopic enhanced image (corresponding to the proportion of the current image of the endoscope in the endoscopic enhanced view) can be determined according to the input of the operator. In other words, the operator can change the proportion of the current image of the endoscope in the endoscopic enhanced view by interacting with the system, for example, zooming in or out the current image of the endoscope. When the doctor needs to observe a detail in the endoscopic field of view more carefully, the real-time endoscopic image can be enlarged (correspondingly, the portion of the endoscopic enhanced image displayed in the view window becomes smaller), while the proportion of the content displayed in the surrounding enhanced information to the real-time endoscopic image remains unchanged, and the enhanced information beyond the edge of the "endoscopic enhanced view" is no longer displayed on the "endoscopic enhanced view". The operator's input can be performed in the form of a zoom icon input component on the display device, for example. However, those skilled in the art can understand that the operator's input method for determining the size of the real-time endoscopic image in the view is not limited to the zoom icon method, and can also be performed in other ways, such as providing the operator with several different size options, or determining it by the operator inputting a numerical value. The input component displayed on the display interface can be, for example, a tab, or a dialog box for the operator to input a numerical value.

FIG5 exemplarily shows a view of an endoscopic enhanced image displayed on a window of a display device according to another exemplary embodiment, wherein the endoscopic enhanced image fuses the current image 10 of the endoscope and three types of enhanced information, namely, the stitched image 6, the three-dimensional image 7 of the patient's physiological structure and the physiological structure markers shown in the figure. In this example, the ventral side of the articular process, the pedicles and intervertebral discs of the anterior vertebra are shown, and the pedicles of the posterior vertebra are not displayed because they are located outside the edge of the view. Guidance instructions related to the physiological structure markers can also be fused to the endoscopic enhanced image and displayed. For example, when the physiological structure marker is located outside the view range of the endoscopic enhanced image, the guidance instruction can be an arrow pointing to the physiological structure marker (for example, an arrow pointing to the pedicles of the posterior vertebra that are not displayed in FIG5).

Since it is hoped that the current endoscopic image, i.e., the real-time image, on the endoscopic enhanced view is as large as possible to facilitate the operator's observation, the enhanced information that can be displayed is very local and limited. In order to provide the operator with better global guidance, the present invention may also include a global enhanced image. Specifically, at least two of the spliced image, the three-dimensional image, and the guidance instructions related to the mark are fused to obtain a global enhanced image and display it, wherein the current visual field position of the imaging device and the position of the view of the endoscopic enhanced image are indicated in the global enhanced image. As shown in FIG6 , the endoscopic enhanced image is shown on the left side of the display device, and the global enhanced image is shown on the lower right side. Specifically, FIG7 exemplarily shows a global enhanced image fused with a spliced image 6, a three-dimensional image 7 of the patient's physiological structure, and direction marks pointing to the dorsal, ventral, caudal, and cephalic sides. FIG8 exemplarily shows a global enhanced image fused with a spliced image 6, a three-dimensional image 7 of the patient's physiological structure, and physiological structure markers such as an intervertebral disc. The fusion method of the global enhanced image is the same as the fusion method of the enhanced information described above, and each of the acquired enhanced information is fused according to the orientation information of each acquired enhanced information under the navigation system. The specific process is not repeated here.

In the global enhanced images shown in FIGS. 7 and 8 , the edge of the current field of view position of the imaging device (i.e., the position of the current endoscope image in the global enhanced image) is indicated by a dotted line 8, and the edge of the view corresponding to the endoscope enhanced image is framed by a dotted line 9. Specifically, the method of the present invention also includes a step of indicating the current field of view position of the imaging device in the global enhanced image, in which the area corresponding to the current field of view position of the imaging device in the global enhanced image is determined by means of the current position and direction of the imaging device, and a mark 8 indicating the edge of the area is generated and displayed in the display step. It can be understood by those skilled in the art that other forms can be used instead of dotted lines to indicate the edge of the current field of view and the edge of the view of the endoscope enhanced image, and both can also be represented by lines of different colors and/or line types.

In the above scheme, the endoscopic enhanced view is displayed simultaneously with the global enhanced view as a part of the global enhanced image, and the correlation information between the two is displayed in the global enhanced image, that is, the position of the edge of the real-time endoscopic image and the edge of the endoscopic enhanced view in the global enhanced image is displayed, so that the operator can know the position of the endoscopic field of view in the entire world and the surrounding soft tissue, bone structure, target position and direction information while watching the larger real-time endoscopic image, thereby realizing all-round guidance for the operator.

It should also be noted that the "step of displaying the endoscopic enhanced image" or "step of displaying the global enhanced image" described herein does not mean that the image is always displayed. For example, the endoscopic enhanced image or the global enhanced image may be displayed only in a certain period of time according to the needs of the operator. For example, when the operator wants to observe the enhanced image, he or she may trigger the display of the corresponding enhanced image by, for example, using a foot switch.

The present invention also provides an electronic device for navigation of endoscopic surgery, the electronic device includes a display device 3 and a processor as described above, in the specific example shown in FIG2, the processor is included in the control device 2, i.e., the illustrated host. It will be appreciated by those skilled in the art that the processor and the display device may be integrated or split. The control device 2 or the processor has a data interface. The control device or the processor is electrically connected to the tracking device 1 and the endoscope 5 of the navigation system through the data interface to obtain the position and direction of the imaging device of the endoscope and to obtain the image of the endoscope in the corresponding position and direction. The data interface of the processor also enables the processor to obtain one or more of the three-dimensional image of the patient's physiological structure and the mark on the three-dimensional image or two-dimensional image of the patient's physiological structure. When the processor is running, a computer program (the computer program may be stored in a memory included in the control device or in other memories) is executed to execute the method of the present invention and display a view of at least a portion of the endoscopic enhanced image on the display device 3 for at least a period of time, wherein the endoscopic enhanced image is an image that fuses the current image of the endoscope with at least two of the following three types of enhanced information:

a) a stitched image obtained by stitching multiple images of the endoscope;

b) Three-dimensional images of the patient’s physiological structure;

c) Guidance instructions related to the marking.

As an example, a global enhanced image is also displayed on the display interface of the display device 3, and the global enhanced image is an image obtained by fusing at least two of the stitched image 6, the three-dimensional image 7 and the guidance instructions, wherein the global enhanced image indicates the current field of view position 8 of the imaging device and/or the view position 9 of the endoscopic enhanced image.

As an example, one or more of the following images or views can also be displayed on the display interface of the display device 3: a view on a fitted two-dimensional perspective section, a section view of a sagittal plane, a section view of a coronal plane, a section view of an axial plane, a real-time image of an endoscope, etc. Thus, the operator can conveniently obtain more comprehensive navigation information. Those skilled in the art can understand that other sections, views of orientations, or any suitable views can also be displayed as needed.

The present invention also provides a computer-readable storage medium having a computer program stored thereon, and the computer program can perform the steps of the above method of the present invention when executed by a processor. In addition, the present invention also provides a control device, which may include a memory, a processor, and a program stored on the memory and capable of running on the processor, wherein the steps of the above method of the present invention are performed when the processor runs the program. The present invention also provides a computer program product, including a computer program, and the computer program implements the steps of the above method of the present invention when executed by a processor.

Those skilled in the art will appreciate that the steps of the methods or algorithms described herein may be implemented directly using hardware, software modules executed by a processor, or a combination of the two. The software modules may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

In the above embodiments, the method can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present invention is generated in whole or in part. The computer here can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state hard disk Solid State Disk), etc.

Those skilled in the art will appreciate that the memory of the control device of the present invention may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device separate from the processor.

The processor of the control device can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

Although some embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the general inventive concept, the scope of which is defined in the claims and their equivalents.

Claims (29)

1. A method for guiding an endoscopic procedure, the method comprising the steps of:

An endoscopic image acquisition step: acquiring an image of an endoscope;

Enhancement information acquisition: at least two of the following three types of enhancement information are acquired:

a) Splicing a plurality of images of the endoscope to obtain a spliced image;

b) A three-dimensional image of a patient's physiological structure;

c) A marker on a three-dimensional image or a two-dimensional image of a patient's physiological structure;

And (3) an endoscope enhanced image fusion step: fusing the obtained current image of the endoscope with at least two enhancement information obtained in the enhancement information obtaining step to obtain an enhanced image of the endoscope; and

And a display step: a view including at least a portion of the endoscopically enhanced image is displayed.

2. The method of claim 1, wherein the endoscopically enhanced image fusion step is performed by means of a navigation system, wherein the method further comprises, prior to the endoscopically enhanced image fusion step:

An imaging azimuth acquisition step: acquiring the azimuth of the imaging device of the endoscope under the navigation system; and

An enhanced information azimuth acquisition step: acquiring a spliced image to be fused, the three-dimensional image or the azimuth of the mark under the navigation system;

In the step of fusion of the endoscope enhanced images, the fusion is carried out according to the orientation of the imaging device and the orientation of the spliced image, the three-dimensional image or the mark to be fused.

3. The method according to claim 1 or 2, further comprising the step of:

global enhanced image fusion step: fusing at least two of the stitched image, the three-dimensional image, and a guidance indication associated with the marker to obtain a global enhanced image, wherein the global enhanced image indicates a current field of view position of an imaging device and/or a position of the view of the endoscope enhanced image; and

A step of displaying the global enhanced image;

Wherein the edge of the current field of view position of the imaging device and the edge of the view of the endoscopically enhanced image are represented by lines of different colors and/or lineages on the global enhanced image.

4. A method according to claim 1 or 2, characterized in that the type of enhancement information acquired and fused is selected based on operator input.

5. A method according to claim 1 or 2, wherein the range of the endoscopically enhanced image displayed in the view of the endoscopically enhanced image is determinable from operator input.

6. The method according to claim 2, wherein in the case where the fused enhancement information includes a stitched image, the enhancement information acquiring step includes:

And a stitching step, in which a plurality of images of the endoscope are stitched according to the orientation of the imaging device corresponding to at least one endoscope image to obtain the stitched image.

7. The method of claim 6, wherein in the case where the fused enhancement information includes a stitched image, the enhancement information obtaining step further comprises:

a distortion calibration step prior to the stitching step, wherein in the distortion calibration step, the image of the endoscope to be stitched is distortion calibrated; and

And a processing step subsequent to the stitching step, in which a planar image is generated from the image obtained in the stitching step, wherein the planar image is fused with a current image of an endoscope as the stitched image in the endoscope-enhanced image fusion step.

8. The method according to claim 2, wherein in case the fused enhancement information comprises the marker, the enhancement information orientation obtaining step comprises a marker orientation obtaining step in which the orientation of the marker under the navigation system is obtained by means of registration of the three-dimensional image or the two-dimensional image under the navigation system;

In the step of the endoscope enhanced image fusion, the guiding instruction related to the mark is fused with the current image of the endoscope according to the position of the mark under the navigation system.

9. The method of claim 8, wherein the method enables an operator to select one or more of a directional marker and a marker indicative of a physiological structure.

10. The method of claim 9, wherein the patient physiological structure is the spine and the indicia indicative of physiological structure comprises an image model of one or more of a herniated or free intervertebral disc, osteophyte, and ossified ligamentum flavum.

11. The method of claim 9, wherein the marker indicative of physiological structure comprises a physiological structure marker dot; preferably, the patient physiological structure is the spine and the physiological structure marker points are marker points indicating one or more of a herniated or free intervertebral disc, osteophyte and ossified ligamentum flavum, or physiological structure marker points that are not displaced during endoscopic surgery, such as one or more of the pedicles of the articular process ventral, anterior, posterior, and intervertebral discs; preferably, the physiological structure marker points are a plurality of.

12. The method of claim 9, wherein the direction markers comprise one or more of a dorsal, ventral, cephalad, and caudal direction marker toward the patient.

13. The method of claim 2, wherein in the event that the fused enhancement information comprises a three-dimensional image of the patient's physiological structure, the enhancement information position acquisition step comprises the step of registering the three-dimensional image of the patient's physiological structure under a navigation system to acquire the position of the three-dimensional image.

14. The method of claim 13, wherein the three-dimensional image of the patient's physiological structure comprises a pre-operative three-dimensional image or an intra-operative three-dimensional image.

15. The method of any one of claims 1-14, wherein the endoscope is a spinal endoscope.

16. Method according to claim 2, wherein the imaging position acquisition step is performed by means of acquiring the position of a tracer (4) having a fixed positional relationship with respect to the endoscope from a tracking device (1) of a navigation system.

17. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, performs the steps of the method according to any one of claims 1-16.

18. A control device comprising a memory, a processor and a program stored on the memory and executable on the processor, characterized in that the steps of the method according to any one of claims 1-16 are performed when the processor runs the program.

19. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any of claims 1-16.

20. An electronic device for navigation of an endoscopic procedure, characterized in that it comprises a display means (3) and a processor with a data interface,

Wherein the data interface is connectable to an endoscope (5) such that the processor is capable of acquiring an image of the endoscope; and

Wherein the processor is configured to be capable of acquiring one or more of a three-dimensional image of the patient's physiological structure and a marker on the three-dimensional or two-dimensional image of the patient's physiological structure;

Wherein the processor is configured to display a view of at least a portion of an endoscope enhancement image on the display device (3) for at least a period of time when the processor is running, wherein the endoscope enhancement image is an image fusing a current image of an endoscope with at least two of the following three types of enhancement information:

a) Splicing a plurality of images of the endoscope to obtain a spliced image;

b) A three-dimensional image of a patient's physiological structure;

c) A guidance indication associated with the marker.

21. Electronic device according to claim 20, characterized in that the data interface is connectable to a tracking means (1) of a navigation system.

22. The electronic device according to claim 20, characterized in that a global enhancement image is also displayed on the display means (3), the global enhancement image being an image obtained by fusing at least two of the stitched image, the three-dimensional image and the guidance indication, the edge of the current field of view position of the imaging means and the edge of the view of the endoscope enhancement image being represented by lines of different colors and/or lines on the global enhancement image.

23. The electronic device of any one of claims 20-22, wherein one or more of the following is further displayed on a display interface of the display apparatus: view on a fitted two-dimensional perspective section, section view of sagittal plane, section view of coronal plane, section view of axial plane, and real-time image of endoscope.

24. The electronic device of any of claims 20-22, further comprising an input means for an operator to input instructions to select the type of enhancement information that is fused.

25. The electronic device of any one of claims 20-22, further comprising an input means for an operator to input instructions to determine a range of an endoscopically enhanced image displayed in the view of endoscopically enhanced images.

26. The electronic device of any of claims 20-22, wherein the indicia comprises one or more of a directional indicia, indicia indicative of a physiological structure;

Preferably, the patient physiological structure is the spine, the marker indicative of physiological structure comprises an image model of one or more of a herniated or free intervertebral disc, osteophyte and ossified ligamentum flavum, or the marker indicative of physiological structure comprises a physiological structure marker point which is a marker point indicative of one or more of a herniated or free intervertebral disc, osteophyte and ossified ligamentum flavum, or is a physiological structure marker point which is not displaced during endoscopic surgery, such as one or more of the pedicles of the articular process ventral, anterior segment vertebral body, posterior segment vertebral body, intervertebral disc;

preferably, the direction markers comprise one or more of a dorsal, ventral, cephalad and caudal direction marker towards the patient.

27. A navigation system for endoscopic surgery, the navigation system comprising:

-a tracking device (1), the tracking device (1) being adapted to track a tracer (4) having a fixed positional relationship with respect to the endoscope (5); and

-A display device (3) and-a processor adapted to be connected to the tracking device (1) and the endoscope (5), wherein the method of any one of claims 1-16 is performed when the processor is run, and wherein the display in the method is effected by the display device (3).

28. A navigation system for endoscopic surgery, the navigation system comprising:

-a tracking device (1), the tracking device (1) being adapted to track a tracer (4) having a fixed positional relationship with respect to the endoscope (5); and

Electronic device according to any of claims 20-26, wherein a processor in the electronic device is adapted to be connected to the tracking means (1) and the endoscope (5).

29. A robotic system comprising a robotic arm and a navigation system according to any one of claims 27-28.