WO2024149161A1 - Catheter robot and registration method therefor, and readable storage medium - Google Patents

Abstract

Provided in the present application are a registration method for a catheter robot, a catheter robot, and a readable storage medium. The method comprises: by using a sensor arranged on a catheter or on an instrument carried by the catheter, acquiring an actual path point of the catheter and/or the instrument; according to a first transformation matrix, acquiring a simulated path point of the catheter and/or the instrument that corresponds to the actual path point in an anatomical model; determining, from among a plurality of pipe center line segments obtained by dividing a pipe center line, a target pipe center line segment corresponding to the simulated path point; calculating the centroid of the target pipe center line segment; and if the centroid is different from all of a plurality of skeleton points comprised in the target pipe center line segment, determining a connection line between the centroid and the simulated path point, selecting a skeleton point, the distance between which and the connection line is the shortest, from among the plurality of skeleton points, and taking the selected skeleton point as a matching point of the actual path point. In this way, the present application can improve the accuracy of point cloud registration, thereby improving the accuracy of navigation during catheterization.

Description

Catheter robot and registration method thereof, and readable storage medium

This application claims the priority of the Chinese patent application filed with the China Patent Office on January 9, 2023, with application number CN 202310026323.7 and application name “Catheter robot and its alignment method, readable storage medium”, the entire contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the technical field of surgical robots, and in particular, to a catheter robot, a catheter robot registration method, and a readable storage medium.

Background technique

Minimally invasive medical technology intends to reduce the amount of tissue damaged during medical procedures to reduce patient recovery time, discomfort and harmful side effects. In minimally invasive medical technology, it is often necessary to insert a catheter through a natural orifice in the patient's anatomy or through a surgical incision, and with the cooperation of an intraoperative navigation system, pass through the complex structure of the human body to reach or approach the target.

In most cases, the catheter moves along the human body's ducts (such as bronchi, blood vessels, ureters, etc.) and does not shuttle freely in a way that damages the ducts. Therefore, it can be considered that the movement path of the catheter/instrument is confined within the human body's ducts and is similar to the centerline of the human body's ducts. Especially for smaller ducts, due to the limited radial range of motion, the movement path is more similar to the centerline.

Based on this principle, the intraoperative navigation system can use point cloud registration to detect and display the position of the catheter tip and/or the instrument tip carried by the catheter in real time. Specifically, the path data of the catheter/instrument movement can be collected by sensors and matched with the pipeline centerline data in the patient's anatomical model. The coordinate transformation between the two can be determined through point cloud registration, thereby obtaining the real-time position of the catheter/instrument.

Point cloud registration in related technologies uses the shortest Euclidean distance criterion to find corresponding point pairs, and then calculates coordinate transformation based on the corresponding point pairs. However, the inventors found that due to the complexity of human body pipes and the influence of patient movement, the use of the shortest Euclidean distance criterion may often result in incorrect corresponding point pairs. For example, the point on the pipe centerline corresponding to the path point of the catheter is not in the pipe where the catheter is currently located. As a result, the accuracy of point cloud registration is not high, which affects the accuracy of navigation during catheterization.

Summary of the invention

The embodiments of the present application provide a catheter robot alignment method, apparatus, and terminal device, which can solve the problem in the related art that it is difficult to accurately align path data.

In a first aspect, an embodiment of the present application provides a catheter robot, which includes a robotic arm, a catheter instrument engaged with a power unit of the robotic arm, and a processor communicatively connected to the robotic arm, wherein the catheter instrument includes an instrument box configured to be engaged with the power unit and a catheter connected to the instrument box, and the catheter and/or the instrument carried by the catheter are provided with a sensor for measuring the position of the catheter and/or the instrument, and the processor is configured to perform the following steps: using the sensor to obtain actual path points of the catheter and/or the instrument; obtaining simulated path points of the catheter and/or the instrument corresponding to the actual path points in an anatomical model according to a first transformation matrix, wherein the anatomical model includes a pipeline centerline; determining a target pipeline centerline segment corresponding to the simulated path point among multiple pipeline centerline segments into which the pipeline centerline is divided; calculating the centroid of the target pipeline centerline segment; if the centroid is different from multiple skeleton points included in the target pipeline centerline segment, determining a connecting line between the centroid and the simulated path point, and selecting one of the multiple skeleton points with the shortest distance to the connecting line as a matching point of the actual path point.

The processor is configured to perform the following steps after determining the connection line between the centroid and the simulated path point: if the centroid is one of multiple skeleton points, obtain the first perpendicular foot of the simulated path point to the projection line, and obtain the second perpendicular foot of each skeleton point to the projection line, the projection line is a straight line passing through the two end points of the target pipeline centerline segment; select the skeleton point corresponding to the second perpendicular foot with the shortest distance to the first perpendicular foot as the matching point of the actual path point.

Wherein, the processor is configured to perform the following steps: update the point pair set using the first point pair consisting of the actual path point and its matching point; if the point pair set satisfies the first condition, use the point pair set to update the first transformation matrix, wherein the first condition includes that the number of point pairs in the point pair set is greater than a preset value.

The processor is configured to perform the following steps in updating the point pair set using the matching point pair consisting of the actual path point and its matching point: determining whether there is a second point pair including the matching point in the point pair set; if not, adding the first point pair to the point pair set; if so, The first distance is calculated for the matching points, and the second distance is calculated based on the path points and the matching points in the second point pair. If the first distance is less than the second distance, the second point pair is replaced by the first point pair.

The processor is configured to perform the following steps in updating the first transformation matrix using the point pair set: substituting the point pairs in the point pair set into the objective function to calculate the latest second transformation matrix; and updating the first transformation matrix according to the latest second transformation matrix.

The first transformation matrix is a weighted average of the latest second transformation matrix and at least part of the historical second transformation matrix, wherein the weight of the latest second transformation matrix is greater than the weight of the historical second transformation matrix.

Wherein, the processor is configured to perform the following steps before determining a target pipeline centerline segment corresponding to a simulation path point among a plurality of pipeline centerline segments into which the pipeline centerline is divided: extracting characteristic skeleton points from a plurality of skeleton points included in the pipeline centerline, wherein the position of the skeleton points is represented by a code, the code of the skeleton point s _{= x1} + _x2R1 + _x3R1R2 , wherein _x1 , _{x2 , x3} are the coordinates of the skeleton points in the first, second, and third dimensions of the coding order, respectively, _{and R1} and _R2 are the sizes of the anatomical model in the first and second dimensions of the coding order, respectively; dividing the pipeline centerline into a plurality of pipeline centerline segments according to the characteristic skeleton points.

Wherein, the processor is configured to perform the following steps in extracting characteristic skeleton points from multiple skeleton points included in the pipeline centerline: determine the starting point; starting from the starting point, use the region growing method to search for skeleton points and count the connectivity number of the skeleton points, where the connectivity number is the number of all child nodes in the neighborhood of the skeleton point; if the connectivity number of the skeleton point is greater than 1, and the connectivity number of the parent node is equal to 1, then the skeleton point is a bifurcation point among the characteristic skeleton points; if the connectivity number of the skeleton point is equal to 1, and the connectivity number of the parent node is equal to 0, then the skeleton point is an end point among the characteristic skeleton points.

The processor is configured to perform the following steps in selecting a starting point: randomly selecting an integral layer; counting the total number of skeleton points in a specified number of layers on both sides of the body layer in a direction perpendicular to the cross section to determine the direction in which the airway extends; and finding the skeleton point of the tracheal entrance as the starting point based on the total number of skeleton points in the body layer and the direction in which the airway extends.

The processor is configured to perform the following steps: counting the number of skeleton points in the pipeline centerline segment; and deleting the pipeline centerline segment whose number is less than a preset threshold.

In a second aspect, the present application provides a catheter robot registration method, the method comprising: The invention is summarized as follows: using a sensor disposed on the catheter and/or an instrument carried by the catheter to obtain actual path points of the catheter and/or the instrument; obtaining simulated path points of the catheter and/or the instrument corresponding to the actual path points in an anatomical model according to a first transformation matrix, wherein the anatomical model includes a pipeline centerline; determining a target pipeline centerline segment corresponding to the simulated path point from a plurality of pipeline centerline segments into which the pipeline centerline is divided, wherein the target pipeline centerline segment includes a plurality of skeleton points; calculating the centroid of the target pipeline centerline segment; if the centroid is different from a plurality of skeleton points included in the target pipeline centerline segment, determining a connecting line between the centroid and the simulated path point, and selecting one of the plurality of skeleton points with the shortest distance to the connecting line as a matching point of the actual path point.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the following method: obtaining actual path points of the catheter and/or the instrument using sensors disposed on the catheter and/or the instrument carried by the catheter; obtaining simulated path points of the catheter and/or the instrument corresponding to the actual path points in an anatomical model according to a first transformation matrix, wherein the anatomical model includes a pipeline centerline; determining a target pipeline centerline segment corresponding to the simulated path point from a plurality of pipeline centerline segments into which the pipeline centerline is divided, wherein the target pipeline centerline segment includes a plurality of skeleton points; calculating the centroid of the target pipeline centerline segment; if the centroid is different from the plurality of skeleton points included in the target pipeline centerline segment, determining a connecting line between the centroid and the simulated path point, and selecting one of the plurality of skeleton points with the shortest distance to the connecting line as a matching point of the actual path point.

Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

In order to adapt to the complex human body environment, the present application first determines the target pipeline centerline segment corresponding to the simulated path point to reduce the error that the matching point is not in the pipeline where the catheter is currently located, and selects a skeleton point with the shortest distance between the line connecting the simulated path point and the centroid of the target pipeline centerline segment. The skeleton point can be approximately considered as the intersection of the line and the target pipeline centerline segment. This intersection is used as the matching point of the actual path point, which can more accurately reflect the matching relationship between the path point of the catheter moving in the pipeline and the pipeline centerline, improve the accuracy of point cloud registration, that is, obtain a more accurate first transformation matrix, thereby improving the accuracy of navigation during catheterization.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

FIG1 is a schematic diagram of a catheter robot provided in one embodiment of the present application;

FIG2 is a schematic diagram of a catheter device and a power unit provided in one embodiment of the present application;

FIG3 is a schematic diagram of a flow chart of a catheter robot registration method provided in an embodiment of the present application;

FIG4 is a schematic diagram of determining the target pipeline centerline segment corresponding to the simulation path point in one embodiment of the present application;

FIG5 is a schematic diagram of determining matching points when the centroid is not a skeleton point on the target pipeline centerline segment in one embodiment of the present application;

FIG6 is a schematic diagram showing a centroid of a skeleton point on a target pipeline centerline segment in an embodiment of the present application;

FIG7 is a schematic diagram of determining matching points when the centroid is a skeleton point on a target pipeline centerline segment in one embodiment of the present application;

FIG8 is a schematic diagram of a specific process of S17 in FIG3 ;

FIG9 is a schematic diagram of a specific process of S18 in FIG3 ;

FIG10 is a schematic diagram of a process of intraoperative navigation provided by an embodiment of the present application;

FIG11 is a schematic diagram of a process for segmenting a pipeline centerline according to an embodiment of the present application;

FIG12 is a schematic diagram of using a single digital code to represent a pixel position in an embodiment of the present application;

FIG13 is a schematic diagram of a specific process of S21 in FIG11 ;

FIG14 is a schematic diagram of a specific process of S211 in FIG12 ;

FIG15 is a schematic diagram of the structure of a control system of a catheter robot provided in one embodiment of the present application;

FIG. 16 is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application.

Detailed ways

The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

The following describes the catheter robot registration method, the catheter robot, and the computer-readable storage medium proposed according to the embodiments of the present application in conjunction with the accompanying drawings.

FIG1 shows a catheter system 1000 provided in an embodiment of the present application. The catheter system 1000 includes an imaging vehicle 100, a trolley 200 and a main controller 300 respectively connected to the imaging vehicle 100, a catheter instrument 400 that can be coupled to the trolley 200, a sensor system 500 connected to the trolley 200, and a control system 600 for realizing control between the catheter instrument 400, the main controller 300, the sensor system 500 and the imaging vehicle 100. Among them, the main controller 300 can be connected to the trolley 200 by wire or wirelessly. When the operator performs various procedures on the patient next to the trolley 200, the control instruction can be triggered by operating the main controller 300, and the catheter instrument 400 can be controlled to advance, retract, bend and turn, etc. through the drive of the trolley 200.

The trolley 200 can usually be moved to the side of the operating table to engage the catheter instrument 400 and control the catheter instrument 400 to move up and down in the vertical direction, or to move in the horizontal direction, or to move in non-vertical and non-horizontal directions under the control command, thereby providing a better preoperative preparation angle for the operation of the catheter instrument 400. The control command can be a command triggered by the operator by operating the main controller 300, or a command triggered by the operator directly clicking or pressing a button set on the trolley 200. Of course, in other embodiments, the control command can be a command triggered by the operator by operating the main controller 300. The command may also be a voice control or a command triggered by a force feedback mechanism.

As shown in FIG1 , further, the trolley 200 may include a base 210, a sliding seat body 220 that can be lifted and moved along the base 210, and two mechanical arms 230 fixedly connected to the sliding seat body 220. The mechanical arm 230 may include a plurality of arm segments connected at a joint, and the plurality of arm segments provide the mechanical arm 230 with a plurality of degrees of freedom, for example, seven degrees of freedom corresponding to seven arm segments. A power unit (not shown in the figure) is installed at the end of the mechanical arm 230, and the power unit of the mechanical arm 230 is used to engage the catheter instrument 400, and under the driving action of the power unit, the end of the catheter instrument 400 is controlled to bend and turn accordingly. Among them, the two mechanical arms 230 may be structures that are completely the same or partially the same, one mechanical arm 230 is used to engage the inner catheter instrument 410, and the other mechanical arm 230 is used to engage the outer catheter instrument 420. During installation, the outer catheter device 420 may be installed first, and when the outer catheter device 420 is installed, the catheter of the inner catheter device 410 is inserted into the catheter of the outer catheter device 420 .

The sensor system 500 has one or more subsystems for receiving information about the catheter device 400. The subsystems may include: a position sensor system; a shape sensor system for determining the position, orientation, speed, velocity, pose, and/or shape of the tip of the catheter device 400 and/or along one or more segments of a catheter that may constitute the catheter device 400; and/or a visualization system for capturing images from the tip of the catheter device 400.

The imaging vehicle 100 may be provided with a display system 110 and a flushing system (not shown in the figure), etc. The display system 110 is used to display images or representations of the surgical site and the catheter instrument 400 generated by the subsystem of the sensor system 500. Real-time images of the surgical site and the catheter instrument 400 captured by the visualization system may also be displayed. Image data from imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound may also be used to present images of the surgical site recorded preoperatively or intraoperatively. The preoperative or intraoperative image data may be A virtual navigation image may also be displayed in a manner that is presented as a two-dimensional, three-dimensional, or four-dimensional (e.g., time-based or rate-based) image and/or as an image from a model created based on a preoperative or intraoperative image dataset. In the virtual navigation image, the actual position of the catheter device 400 is registered with the preoperative image to present a virtual image of the catheter device 400 within the surgical site to the operator from the outside.

The control system 600 includes at least one memory and at least one processor. It is understandable that the control system 600 can be integrated into the trolley 200 or the imaging trolley 100, or can be independently provided. The control system 600 can support wireless communication protocols such as IEEE 802.11, IrDA, Bluetooth, HomeRF, DECT, and wireless telemetry. The control system 600 can transmit one or more signals indicating the movement of the catheter instrument 400 by the power unit to move the catheter instrument 400. The catheter instrument 400 can extend to a surgical location in the body through an opening of the patient's natural cavity or a surgical incision.

Further, the control system 600 may include a mechanical control system (not shown in the figure) and an image processing system (not shown in the figure), wherein the mechanical control system is used to control the movement of the catheter instrument 400, and therefore, can be integrated into the trolley 200. The image processing system is used for virtual navigation path planning, and therefore, can be integrated into the imaging vehicle 100. Of course, the various subsystems of the control system 600 are not limited to the specific situations listed above, and can also be reasonably set according to actual conditions. Among them, the image processing system can image the surgical site based on the image of the surgical site recorded before or during the operation, using the above imaging technology. The software used in combination with manual input can also convert the recorded image into a two-dimensional or three-dimensional synthetic image of a part or the entire anatomical organ or segment. During the virtual navigation procedure, the sensor system 500 can be used to calculate the position of the catheter instrument 400 relative to the patient's anatomical structure, which can be used to generate an external tracking image and an internal virtual image of the patient's anatomical structure, so as to realize the actual position of the catheter instrument 400 and the preoperative image registration, so that the virtual image of the catheter instrument 400 in the surgical site can be presented to the operator from the outside.

The structural composition of the inner catheter device 410 and the outer catheter device 420 is substantially the same, and each comprises a slender and flexible inner catheter 41 and an outer catheter 42, wherein the diameter of the outer catheter 42 is slightly larger than that of the inner catheter 41, so that the inner catheter 41 can pass through the outer catheter 42 and provide a certain support for the inner catheter 41, so that the inner catheter 41 can reach the target position in the patient's body, so as to facilitate operations such as tissue or cell sampling from the target position.

Certain movements of the main controller 300 may cause corresponding movements of the catheter device 400. For example, when the operator operates the direction lever of the main controller 300 to move upward or downward, the movement of the direction lever of the main controller 300 may be mapped to the corresponding pitch movement of the end of the catheter device 400; when the operator operates the direction lever of the main controller 300 to move left or right, the movement of the direction lever of the main controller 300 may be mapped to the corresponding yaw movement of the end of the catheter device 400. In this embodiment, the main controller 300 may control the end of the catheter device 400 to move within a 360° spatial range.

FIG2 shows a catheter device 400 provided in an embodiment of the present application. The catheter device 400 is configured to engage with the power unit 240 of the mechanical arm 230, and the catheter device 400 includes an instrument box 45 configured to engage with the power unit 240 and a catheter 48 connected to the instrument box 45. The "engagement" refers to a state in which when the instrument box 45 is installed in the power unit 240, the driving force of the power unit 240 can be transmitted to the instrument box 45 and the catheter 48 can move normally. For example, under the driving force of the power unit 240, the end of the catheter 48 can be bent and turned.

The end in this application may also be referred to as the distal end or the head, which refers to the end away from the instrument box 45 ; the front end may also be referred to as the proximal end or the tail, which refers to the end close to the instrument box 45 .

The processor of the control system 600 is configured to perform the following steps to implement the catheter robot registration method provided in an embodiment of the present application. As shown in FIG3 , the method includes:

S11: Using sensors to obtain actual path points of the catheter and/or instrument.

The actual path point is the path point of the end of the catheter and/or the instrument carried by the catheter obtained by the sensor, which is generally described by the coordinates of the end of the catheter and/or the instrument in the world coordinate system, or the surgical environment coordinate system. The sensor may include a position sensor and/or a shape sensor.

The position sensor can be set on the catheter and/or the instrument. A position sensor can be a component in the electromagnetic positioning system. The electromagnetic positioning system can further include a magnetic field generating component and a magnetic field detecting component. The magnetic field generating component is used to generate a magnetic field. The position sensor will cause a change in the magnetic field in the magnetic field. The magnetic field detecting component can detect the change in the magnetic field and thus detect the position of the position sensor relative to the magnetic field/magnetic field generating component. Combined with the coordinate conversion relationship between the position sensor and the catheter end/instrument end obtained by the pre-set or calibrated coordinate conversion relationship, the position of the catheter end/instrument end relative to the magnetic field/magnetic field generating component can be calculated. Combined with the coordinate conversion relationship between the magnetic field/magnetic field generating component and the world coordinate system, the position of the catheter end/instrument end in the world coordinate system, that is, the actual path point, can be calculated.

The position of the catheter tip/instrument tip can be acquired by a shape sensor system. For example, the shape sensor can include an optical fiber aligned with the catheter, and the optical fiber bend sensor formed by the optical fiber can feedback the shape of the catheter, based on which the position of the catheter tip/instrument tip relative to the base of the shape sensor can be calculated. Combined with the position of the base of the shape sensor in the world coordinate system, the position of the catheter tip/instrument tip in the world coordinate system, i.e., the actual path point, can be calculated.

An actual path point reflects the position of the catheter end/device end when the sensor feedbacks. The actual path points obtained by multiple feedbacks from the same sensor are arranged according to the feedback time to obtain the movement path of the catheter end/device end. The set of these actual path points can be called a path point set or a path point cloud.

S12: Acquire simulated path points of the catheter and/or instrument corresponding to the actual path points in the anatomical model according to the first transformation matrix.

Scanning the target area of the patient before surgery can obtain medical images (such as CT, MRI, etc.), and the preoperative images are reconstructed in three dimensions to obtain three-dimensional preoperative images. The three-dimensional preoperative images are appropriately processed to obtain anatomical models. Image processing can include image segmentation and skeleton extraction. Image segmentation is used to extract the human ducts involved in the surgery from the three-dimensional images. To facilitate intraoperative navigation, the skeleton of the human duct is extracted to obtain the duct centerline. The duct centerline is the centerline of the human duct, which can also be called the skeleton of the human duct. It is a curve used to describe some geometric features of the human duct. It is located in the middle part of the human duct and has the same topological structure as the human duct. Its width is generally a single pixel. The duct centerline can provide the information required for intraoperative navigation. At the same time, compared with the original human duct, its data volume is significantly reduced, and it is more convenient to process, which is conducive to real-time navigation.

The pipeline centerline is a three-dimensional curve. In practical applications, the pipeline centerline is often stored and used in the form of multiple three-dimensional points. In other words, the pipeline centerline includes these three-dimensional points, which can be called skeleton points. The set of skeleton points can be called a skeleton point set or a skeleton point cloud.

The anatomical model is a three-dimensional binary image, in which the pixel value of each pixel (also referred to as a voxel) is 0 or 1, which is used to indicate whether the pixel is a skeleton point. In the embodiments of the present application, the pixel value of the skeleton point is 1 and the pixel value of the non-skeleton point is 0. In fact, in other embodiments, the values may be opposite, and there is no limitation on this.

The range of pixel values is determined by the image bit depth supported by the display system, and the common range is 0 to 255 or 0 to 1023. When displaying an anatomical model, in order to intuitively distinguish between skeleton points and non-skeleton points, the pixel values of the anatomical model are generally mapped. For example, the pixel values of skeleton points are mapped to the maximum value (255 or 1023) and displayed as white, and the pixel values of non-skeleton points are mapped to 0 and displayed as black. Of course, it can also be reversed, that is, skeleton points are displayed as black and non-skeleton points are displayed as white.

To facilitate subsequent registration, the pipeline centerline can be preprocessed. The preprocessing can include extracting characteristic skeleton points from the skeleton point set and/or resampling the pipeline centerline. The characteristic skeleton points generally include The bifurcation points and end points are used to segment the centerline of the pipeline. The number of skeleton points in the neighborhood of a skeleton point can be counted to determine whether the skeleton point is a feature skeleton point. The present application provides a method for extracting feature skeleton points based on coding, see the subsequent embodiments for details. The bifurcation points and end points reflect the structural characteristics of the human body pipeline itself. In addition, for segments that are too long/too curved, new feature points can be introduced to further segment them to improve the registration accuracy.

To avoid damaging the connectivity of the pipeline centerline, resampling is generally upsampling. Specific resampling parameters, such as resampling interval and interpolation number, can be determined according to actual needs. If the preprocessing includes extracting feature skeleton points and resampling, there is no restriction on the order of the two.

The first transformation matrix may be a transformation matrix between the world coordinate system and the anatomical model. The first transformation matrix may be used to transform actual path points from the world coordinate system to the coordinate system of the anatomical model to obtain simulated path points.

The simulated path point reflects the position of the catheter end/instrument end in the anatomical model. Combined with the display parameters of the surgical site, the position of the catheter end/instrument end in the displayed surgical site can be obtained, thereby integrating the displayed surgical site with the catheter/instrument therein to achieve intraoperative navigation. The surgical site can be displayed in the form of a preoperative image and/or an intraoperative image. If the preoperative image data is used to display the surgical site, since the anatomical model is obtained by processing the three-dimensional preoperative image, the position of the catheter end/instrument end in the surgical site can be obtained based on the coordinate transformation relationship of the displayed surgical site relative to the three-dimensional preoperative image. If the intraoperative image data is used to display the surgical site, it is necessary to align the intraoperative image with the three-dimensional preoperative image/anatomical model to obtain the coordinate transformation relationship of the displayed surgical site relative to the three-dimensional preoperative image, and then obtain the position of the catheter end/instrument end in the surgical site.

To achieve intraoperative navigation, the present application performs point cloud registration on the skeleton point set and the path point set to calculate the transformation matrix between the world coordinate system and the anatomical model, namely, the first transformation matrix.

The representative algorithm for point cloud registration is the Iterative Closest Point (ICP) algorithm. The core idea is to minimize the distance between two point sets and make the two point sets close to each other through iteration. The following is a brief introduction to the principle of ICP.

The basic method of ICP includes two steps: 1. Match point clouds Q and P to find the corresponding point pairs between them; 2. Calculate the transformation matrix between point clouds Q and P based on the corresponding point pairs. A corresponding point pair consists of two points, one from point cloud P and the other from point cloud Q. These two points are matching points of each other, and these two points are considered to be corresponding, that is, the two points are essentially equivalent.

If the real and accurate corresponding point pairs can be found directly, then the above process only needs to be performed once to directly calculate a sufficiently accurate transformation matrix. However, it is difficult to directly find accurate corresponding point pairs in practical applications, so ICP will match according to the principle of the closest distance (generally Euclidean distance), that is, for each point in point cloud P, find the point closest to it in point cloud Q as its matching point. Then calculate a transformation matrix based on these point pairs. After completing a round of calculations, ICP will determine whether the iteration stop condition is met. If not, the transformation matrix obtained from this round of calculations will be used to transform and update point cloud P, and then the point cloud Q and the transformed point cloud P will be used to repeat the above process until the iteration stops.

Since ICP is sensitive to initial values, if the initial difference between point clouds P and Q is large, using ICP directly may not guarantee convergence. Therefore, point cloud registration can be divided into two parts: coarse registration and fine registration. First, coarse registration is used to obtain an initial transformation matrix, and then ICP is used for fine registration. At this time, the first round of ICP is to match point cloud Q with point cloud P transformed by the initial transformation matrix.

When the catheter moves along the human body, there is often a range of motion in the radial direction of the catheter, which may cause the catheter to deviate from the center line of the catheter. The thicker the catheter, the larger the range of motion. In addition, due to the influence of organ movement, the shape of the catheter's moving path and the skeleton center line may be different. Therefore, for the registration of the path point set and the skeleton point set, it is difficult to accurately find the optimal matching point using the classic ICP. Yes, the obtained registration result often falls into the local optimum. For this reason, this application proposes an improved ICP fine registration method, which is described in detail later. Before executing the ICP fine registration method provided by this application, a coarse registration can be performed to obtain the initial value of the first transformation matrix.

During the operation, it is not necessary to match each actual path point. For example, when the ICP iteration is finished, only the virtual path point needs to be obtained to realize the intraoperative navigation, and there is no need to perform the subsequent registration steps. Therefore, after this step, it can be determined whether the registration conditions are met. If they are met, the subsequent steps are performed, otherwise they are not performed. The registration conditions can include ICP iteration in progress, ICP iteration started, etc.

S13: Determine a target pipeline centerline segment corresponding to the simulation path point among the plurality of pipeline centerline segments into which the pipeline centerline is divided.

According to the characteristic skeleton points, the pipeline centerline can be divided into multiple pipeline centerline segments. Each pipeline centerline segment has two endpoints, one of which can be set as the starting point and the other as the end point. The specific starting point and end point can be set according to anatomical characteristics, the motion path of the catheter robot, actual needs, etc. For example, if the human body pipeline is an airway, since the airway is a multi-level bifurcated tree structure, generally the bifurcation point with a higher level (i.e., closer to the trachea) of the two endpoints is used as the starting point, and the bifurcation point/endpoint with a lower level (i.e., closer to the alveoli) is used as the end point. Of course, it can also be set in reverse according to needs.

Based on the characteristics of the catheter robot moving in the human body's pipes, the target pipe centerline segment can be determined first to improve the accuracy of subsequent matching. The target pipe centerline segment is the pipe centerline segment where the estimated simulation path point is located.

First, the simulation path point is within the target pipeline centerline segment, which means that the simulation path point must be between the two endpoints of the target pipeline centerline segment. Converted into a relative position relationship in space, for the pipeline centerline segment s _i s _i+1 , only when the simulation path point t _n is on the side of the starting point s _i that deviates from the end point s _i+1 , and at the same time, on the side of the end point s _i+1 that deviates from the starting point s _i , can it be the target pipeline centerline segment. Therefore, it can be calculated according to the following formula and

Only satisfied and The pipeline centerline segment that meets the requirements is the segment where the simulation path point _tn may be located (referred to as the possible segment), and the pipeline centerline segment is preliminarily screened based on this. and If the target pipeline centerline segment of the previous simulation path point tn _-1 is used as the target pipeline centerline segment of the simulation path point _tn .

For the possible segments initially screened, if the number is 1, they are directly used as the target pipeline centerline segments; if the number is greater than 1, further screening is required. Specifically, the third distance between the simulated path point tn _and the projection line segment of the possible segment can be calculated: The one with the smallest third distance is selected as the target pipeline centerline segment. For pipeline centerline segment s _i s _i+1 , its projection line segment is the straight line segment between the starting point s _i and the end point s _i+1 .

Combined with the accompanying drawings, an example is given, as shown in Figure 4, the dotted line in the figure represents the pipeline centerline, and in the three pipeline centerline segments a, b, and c, for the simulated path point A, and The possible segments are a and b. By comparing the third distance between the projection line segments from the simulated path point A to a and b, it can be obtained that the target pipeline centerline segment corresponding to the simulated path point A is a.

If the simulated path point is close to the bifurcation of the human body pipeline, there may be more than one minimum third distance, or there may be a third distance very close to the minimum third distance, and the above method of determining the target pipeline centerline segment may have a certain error. For this reason, the above determination method can be supplemented.

Specifically, for the possible segments initially screened, determine whether they need to be screened again. If necessary, screen the possible segments again to obtain candidate segments, calculate the moving direction of the catheter, and select the one closest to the moving direction from the candidate segments as the target pipeline centerline segment; otherwise, directly select The one with the third smallest distance is taken as the target pipeline centerline segment.

When determining whether to screen again, there are two methods: based on the third distance and based on the position of the simulated path point. These two methods can be used independently, or combined in an AND or OR manner.

Based on the third distance, a very small range can be set. If, in addition to the minimum third distance, the difference between other third distances and the minimum third distance falls within this range, it is necessary to screen again, and the pipeline centerline segments corresponding to the third distances whose difference falls within this range are the candidate segments. The difference here can be an absolute value, that is, the difference itself, or a relative value, that is, the ratio of the difference to the minimum third distance.

Based on the position of the simulated path point, it can be determined whether there is an endpoint among the endpoints of the possible segments, which belongs to the bifurcation point and the distance between it and the simulated path point is less than the threshold value. If it exists, all the segments to which the endpoint belongs are selected as candidate segments. The distance between the simulated path point and the endpoint can be the Euclidean distance between the two, or the ratio of the Euclidean distance to the length of the projected line segment to which the endpoint belongs, or the projected distance between the simulated path point and the endpoint (the Euclidean distance between the foot of the perpendicular of the simulated path point on the projected line segment and the endpoint), or the ratio of the projected distance to the length of the projected line segment to which the endpoint belongs, or the Euclidean distance or projected distance between the corresponding point of the simulated path point (i.e., the intersection of the line connecting the simulated path point to the centroid of the pipeline centerline segment and the pipeline centerline segment) and the endpoint, or the ratio of the distance to the length of the projected line segment to which the endpoint belongs.

The vector from the previous simulated path point _tnh to the current simulated path point _tn can be calculated as the moving direction of the catheter, where h is a positive integer and can be set based on experiments, experience, etc. The direction of the candidate segment can be a vector pointing from its starting point to its end point. Then, the angle between the moving direction of the catheter and the candidate segment is calculated by the vector inner product method, and the one with the smallest angle is selected as the target pipeline centerline segment.

S14: Calculate the centroid of the target pipeline centerline segment.

The centroid, i.e., the center of the shape, may be the average coordinates of all skeleton points constituting the target pipeline centerline segment. Since the target pipeline centerline segment is obtained by processing the medical image, its density can be considered uniform, so the centroid of the target pipeline centerline segment is also the centroid.

If the centroid is different from the multiple skeleton points constituting the target pipeline centerline segment, jump to S15; if the centroid is one of the multiple skeleton points, jump to S16.

S15: Determine the connection line between the centroid and the simulated path point, and select one of the multiple skeleton points with the shortest distance to the connection line as the matching point of the actual path point.

Since the pipeline centerline reflects the shape of the human body pipeline, it is generally stored and used in the form of a point set. The pipeline centerline often cannot be expressed by an analytical expression, and the analytical solution of the intersection point cannot be obtained. Even if it can be obtained, it may not be exactly the skeleton point. Therefore, in this application, a skeleton point on the target pipeline centerline segment with the shortest distance to the connecting line is approximated as the intersection point between the target pipeline centerline segment and the connecting line.

In this application, the intersection points between the target pipeline centerline segments and the connecting lines are mainly used as matching points. This registration method is more consistent with the movement mode of the catheter robot in the human body pipeline, and the matching points found are more accurate.

For example, as shown in Figure 5, the dotted line in the figure represents the target pipeline centerline segment, _sc is the centroid of the target pipeline centerline segment, and the solid line represents the path of the catheter robot in the anatomical model. The intersection of the line connecting the simulated path point _tn and the centroid _sc and the target pipeline centerline segment is approximately the skeleton point _qk , and the skeleton point _qk is the matching point of the actual path point corresponding to tn.

There is a special case: the centroid of a target pipeline centerline segment is exactly a skeleton point on it. As shown in Figure 6, the dotted line in the figure represents the target pipeline centerline segment, the solid line represents the path of the catheter robot in the anatomical model, _sc is the centroid of the target pipeline centerline segment and _sc is a skeleton point on the target pipeline centerline segment. In this case, no matter where the simulation path point is, The intersection point of the connecting line and the target pipeline centerline segment must be the centroid _sc , that is, the matching points of all path points corresponding to the target pipeline centerline segment are the centroid _sc , which obviously violates the original intention of matching. Therefore, in this case, the intersection point cannot be used as the matching point.

S16: Obtain the first perpendicular foot from the simulated path point to the projection line, and obtain the second perpendicular foot from each skeleton point to the projection line, where the projection line is a straight line passing through the two end points of the target pipeline centerline segment; select the skeleton point corresponding to the second perpendicular foot with the shortest distance to the first perpendicular foot as the matching point of the actual path point.

If the centroid of the target pipeline centerline segment happens to be a skeleton point on it, the matching point is selected according to the principle of the shortest projection distance. Specifically, the first perpendicular foot from the simulated path point to the projection line is obtained, and the projection line is a straight line passing through the two end points of the target pipeline centerline segment, and the second perpendicular foot from each skeleton point to the projection line is obtained. The skeleton point corresponding to the second perpendicular foot with the shortest distance between the first perpendicular foot is selected as the matching point of the actual path point. Since the second perpendicular foot and the fourth distance of each skeleton point are determined after the segmentation is completed, in order to shorten the calculation time, the fourth distance between the second perpendicular foot of all skeleton points on the pipeline centerline and the start/end point of the pipeline centerline segment to which they belong can be calculated and stored in advance. When the projection distance is needed later, it is only necessary to call the fourth distance of all skeleton points on the target pipeline centerline segment, and calculate the fifth distance between the first perpendicular foot and the start/end point of the target pipeline centerline segment. The difference between the fourth distance and the fifth distance can be calculated to obtain the projection distance.

For example, as shown in Figure 7, the solid line in the figure represents the path of the catheter robot in the anatomical model, the dotted line represents the target pipeline centerline segment, the dotted line represents the projection line, the first perpendicular foot from the simulated path point _tn to the projection line is _dtn , and the fifth distance between the starting point _si is _Dtn , the second perpendicular foot from the skeleton point _qk to the projection line is _dqk , and the fourth distance between the starting point _si is _Dqk , and the projection distance between the simulated path point _tn and the skeleton point _qk is | _Dtn - _Dqk |.

S17: Update the point pair set using the first point pair consisting of the actual path point and its matching point.

Since the transformation matrix generally has 6 degrees of freedom, a single point pair cannot find a unique solution, so A point pair set is used to temporarily store multiple point pairs to calculate the transformation matrix. After the matching is completed, the obtained point pairs can be used to try to update the point pair set. One way to update is to directly add the first point pair to the point pair set. However, due to the complexity of the catheter robot's movement, multiple path points may match the same skeleton point, which is not conducive to subsequent calculations. To solve this problem, the first point pair can be judged before adding it to the point pair set, as follows.

As shown in FIG8 , in some embodiments, this step specifically includes:

S171: Determine whether there is a second point pair including a matching point in the point pair set.

If it does not exist, jump to S172; if it does exist, jump to S173.

S172: Add the first point pair to the point pair set.

S173: Calculate a first distance based on the actual path point and the matching point, calculate a second distance based on the path point and the matching point in the second point pair, and if the first distance is smaller than the second distance, replace the second point pair with the first point pair.

The first distance may be referred to as the distance of the first point pair, and the second distance may be referred to as the distance of the second point pair. The first distance may be the Euclidean distance between the simulated path point _tn corresponding to the current actual path point _pn and the matching point, and the second distance may be the Euclidean distance between the simulated path point corresponding to the path point in the second point pair and the matching point. Alternatively, the first distance may be the projected distance between the simulated path point _tn corresponding to the current actual path point _pn and the matching point, and the second distance may be the projected distance between the simulated path point corresponding to the path point in the second point pair and the matching point.

If the first distance is greater than or equal to the second distance, the point pair set is not modified.

Then, it can be determined whether the updated point pair set meets the first condition, which is the condition for updating the transformation matrix. The first condition may include that the number of point pairs in the point pair set is greater than a preset value. In theory, only three point pairs are needed to calculate the unique solution of the transformation matrix. In order to reduce the influence of errors, multiple point pairs are often used in practical applications to estimate the optimal solution of the transformation matrix by the least squares method. The preset value is set The minimum number of point pairs required to estimate the transformation matrix. The default value must be greater than or equal to 3, and the specific value can be set as needed, such as 100, 200, etc.

If the point pair set satisfies the first condition, jump to S18.

S18: Update the first transformation matrix using the point pair set.

As shown in FIG. 9 , in some embodiments, this step specifically includes:

S181: Substitute the point pairs in the point pair set into the objective function to calculate the latest second transformation matrix.

An objective function can be designed in advance, which generally reflects the error of matching point pairs. Then, each point pair in the point pair set is substituted into the objective function, and the transformation matrix that can minimize the objective function is estimated as the latest second transformation matrix, or the second transformation matrix of this round of iteration.

The commonly used objective function of ICP can be used:

Where N is the total number of point pairs in the point pair set, i is the sequence number of the point pair in the point pair set, i = 1, 2, ..., N, is the coordinate of the matching point in the anatomical model in the i-th point pair, is the coordinate of the actual path point in the i-th point pair in the world coordinate system, and T is the transformation matrix.

Since the human body's pipes are thick or thin, the path in the thicker pipe has a larger offset range relative to the center line. In order to obtain a more accurate transformation matrix, the radius parameter of the human body pipe can be introduced into the objective function. The improved objective function is:

Wherein, rn _is the average radius of the target pipeline centerline segment n corresponding to the simulated path point corresponding to the actual path point in the i-th point pair.

S182: Update the first transformation matrix according to the latest second transformation matrix.

The latest second transformation matrix can be directly used as the updated first transformation matrix. In order to adapt to the situation of dynamic local registration, the historical second transformation matrix can be introduced when calculating the first transformation matrix. Optionally, the first transformation matrix is a weighted average of the latest second transformation matrix and at least part of the historical second transformation matrix, wherein the weight of the latest second transformation matrix is greater than the weight of the historical second transformation matrix.

For example, a specific calculation formula of the first transformation matrix is:

Wherein, M represents the serial number of the latest completed iteration, m represents the serial number of the historical iteration (iteration before the latest iteration), m=1, 2, ..., M-1.

After each round of ICP iteration is completed, the updated first transformation matrix is used to transform the actual path points.

Through the implementation of this embodiment, in order to adapt to the complex human body environment, the present application first determines the target pipeline centerline segment corresponding to the simulated path point to reduce the error that the matching point is not in the pipeline where the catheter is currently located, and selects a skeleton point with the shortest distance between the line connecting the simulated path point and the centroid of the target pipeline centerline segment. The skeleton point can be approximately considered as the intersection of the line and the target pipeline centerline segment. This intersection is used as the matching point of the actual path point, which can more accurately reflect the matching relationship between the path point of the catheter moving in the pipeline and the pipeline centerline, improve the accuracy of point cloud registration, that is, obtain a more accurate first transformation matrix, thereby improving the accuracy of navigation during catheterization.

In conjunction with the accompanying drawings, the complete process of intraoperative navigation is illustrated below based on the anatomical model extracted from a three-dimensional lung CT image (obtained by three-dimensional reconstruction of multiple lung tomographic CT images) and the airway centerline therein, wherein the same parts as those in the previous embodiment are not repeated.

The airway includes the trachea and bronchi, which are a tree-like structure that branches out step by step. Therefore, it can also be called the airway tree. The trachea splits into the left and right main bronchi at the end. The main bronchi are distributed deep into the left and right lungs. In the left and right lungs, they split into lobar bronchi (usually three in the left lung and two in the right lung). The lobar bronchi are divided into It splits into segmental bronchi, which gradually divide into alveoli.

As shown in FIG10 , the process of intraoperative navigation in one embodiment of the present application includes:

S101: Acquire an anatomical model including an airway centerline.

S102: Resample the airway centerline.

S103: Extracting bifurcation points and end points from a plurality of skeleton points included in the airway centerline.

S104: Divide the airway centerline into multiple segments using bifurcation points and end points.

S105: Roughly align the world coordinate system and the anatomical model.

Generally, a coarse registration is performed by collecting the coordinates of a small number of points with obvious features (such as the main carina, the first and second carinas of the left and right lobes, etc.) in the world coordinate system and the anatomical model.

S106: Using the sensor to obtain the actual path point of the catheter end.

S107: Use the first transformation matrix to perform coordinate transformation on the actual path points to obtain corresponding simulated path points.

S108: Determine whether the registration condition is met.

If satisfied, jump to S109; otherwise jump to S106.

S109: Determine the target segment corresponding to the simulated path point.

S110: Calculate the centroid of the target segment.

S111: Determine whether the centroid is a skeleton point on the target segment.

If yes, jump to S112, otherwise jump to S113.

S112: Obtain the first perpendicular from the simulated path point to the projection line, and obtain the second perpendicular from each skeleton point included in the target segment to the projection line, where the projection line is a straight line passing through the two end points of the target segment; select the skeleton point corresponding to the second perpendicular with the shortest distance to the first perpendicular as the matching point of the actual path point.

Jump to S114.

S113: Determine the connection line between the centroid and the simulated path point, and select one of the multiple skeleton points included in the target segment with the shortest distance to the connection line as the matching point of the actual path point.

S114: Determine whether there is a second point pair including a matching point in the point pair set.

If it does not exist, jump to S115; otherwise jump to S116.

S115: Add the first point pair consisting of the actual path point and its matching point to the point pair set.

Jump to S118.

S116: Calculate a first distance based on the actual path point and the matching point, calculate a second distance based on the path point and the matching point in the second point pair, and compare the first distance with the second distance.

If the first distance is less than the second distance, then jump to S117; otherwise jump to S118.

S117: Replace the second point pair with the first point pair.

S118: Determine whether the point pair set satisfies the first condition.

If satisfied, jump to S119; otherwise jump to S106.

S119: Substitute the point pairs in the point pair set into the objective function to calculate the latest second transformation matrix.

S120: Calculate a weighted average of the latest second transformation matrix and at least part of the historical second transformation matrix as an updated first transformation matrix.

Jump to S106.

The process of S106 - S120 may be referred to as dynamic fine registration of the world coordinate system and the anatomical model.

As shown in FIG. 11 , in one embodiment of the present invention, the process of segmenting the pipeline centerline specifically includes:

S21: Extract feature skeleton points from multiple skeleton points constituting the center line of the pipeline.

In conventional 3D images, the position of a pixel is represented by a 3D coordinate (x, y, z). Three numbers are needed to represent the position of the pixel, which takes up a large storage space. In addition, since the three numbers need to be searched during the search process, In comparison, it also has an adverse effect on the search speed. In order to reduce the space occupied by storing three-dimensional images such as anatomical models and to speed up the search speed, the positions of the skeleton points in this embodiment are represented by a single digital code.

Specifically, the encoding of each pixel (including skeleton _points ) in the anatomical model _is s= _x1 + _x2R1 + _x3R1R2 , where _x1 , _x2 , _x3 are the coordinates of the pixel in the first, second and third dimensions of the encoding order _, respectively, and _R1 and _R2 are the sizes of the anatomical model in the first and second dimensions of the encoding order, respectively.

Take the accompanying drawings as _an example, as shown in Figure 12, the three dimensions of the anatomical model are X, Y, and Z, and the encoding order is X, Y, and Z, then s= _x1 + _x2R1 + _{x3R1R2 , x1} is the coordinate of the pixel on the X-axis, _x2 is the coordinate of the pixel on the Y-axis, _x3 is the coordinate of the pixel on the Z-axis, _R1 is the size of the anatomical model on the X-axis, that is, the total number of pixels, _{and R2} is the size of the anatomical model on the Y-axis.

In the related art, feature points are generally extracted from a two-dimensional skeleton image by first using a table lookup method to screen out candidate feature points, and then extracting feature points from them. The principle of the table lookup method is: the two-dimensional skeleton image is a binary image, so considering the 8 neighborhoods of a pixel, the 3×3 area including itself has a total of 2 ⁹ =512 possible arrangements, each arrangement represents a distribution of skeleton points, and which arrangements represent that the pixel is an endpoint/intersection point is known. Design a 3×3 convolution kernel, in which each position is set to a different value, such as 1, 2, 4, 8, 16, 32, 64, 128, 256, and the value after convolution is a scalar, which corresponds one-to-one to the arrangement of the 3×3 area centered on the pixel, and calculate the convolution values corresponding to the endpoints and intersections and store them as a feature point correspondence table. When using it, you only need to use the convolution kernel to convolve the two-dimensional skeleton image, and then search for the convolution value in the feature point correspondence table in the obtained convolution image to find all the candidate feature points at once.

However, when the table lookup method is applied to a three-dimensional skeleton image (such as the anatomical model in this application), since the neighborhood is expanded from the 8 neighborhoods in two dimensions to the 26 neighborhoods, and the 3×3 area is expanded to the 3×3×3 area, the possible permutation categories increase from 2 ⁹ to 2 ²⁷ = 134217728, the generation and search of the table will become very complicated. The table lookup method is not suitable for extracting feature skeleton points from 3D skeleton images.

As shown in FIG. 13 , in one embodiment of the present application, a method for extracting characteristic skeleton points based on a region growing method is proposed, which specifically includes:

S211: Determine the starting point.

The starting point, also called the seed point, is the skeleton point that serves as the starting point for regional growth. According to the structural characteristics of the airway tree, the starting point of the airway tree, that is, the skeleton point of the tracheal entrance, is generally selected as the starting point. In the related art, it is generally necessary to manually select the starting point. In this application, a method for automatically selecting the starting point is proposed, which is as follows.

As shown in FIG. 14 , in one embodiment of the present application, this step specifically includes:

S2111: Randomly select an integral layer.

The three-dimensional CT image is reconstructed from multiple slice CT images, where each slice CT image corresponds to an anatomical layer. The anatomical model is obtained by image processing of the three-dimensional CT image. Accordingly, the anatomical model consists of multiple layers, each layer corresponding to the processing result of a slice CT image. One dimension of the anatomical model (usually z) is used to distinguish different layers.

S2112: Count the total number of skeleton points in a specified number of layers on both sides of the body layer in a direction perpendicular to the cross section to determine the direction in which the airway extends.

The tomographic CT image is a cross-sectional image reconstructed by projection. According to the structural characteristics of the airway, it can be known that at the entrance of the trachea, there are only a few skeleton points on the side with the same direction of airway extension (only the trachea has one skeleton point per layer), and then the number of skeleton points increases rapidly (including the bronchus). The side opposite to the direction of airway extension has no skeleton points or only a few skeleton points, and then there will be no skeleton points; the number of skeleton points near the end of the airway tree is significantly reduced, but the number of skeleton points in a single layer is generally greater than the entrance of the trachea. Based on this, the direction of airway extension can be determined.

S2113: Find the bone structure of the tracheal entrance based on the total number of skeleton points in the body layer and the direction of airway extension. The rack point is used as the starting point.

Starting from a randomly selected body layer, search toward the tracheal entrance according to the direction of airway extension until the skeleton point of the tracheal entrance is found.

S212: Starting from the starting point, use the region growing method to search for skeleton points and count the number of connections of the skeleton points.

The connectivity number is the number of all child nodes in the neighborhood of the skeleton point. In the traditional three-dimensional coordinate system, the relative coordinate relationship between the skeleton point and the pixel points in its neighborhood is known, that is, if the three-dimensional coordinates of a skeleton point are known, the three-dimensional coordinates of all the pixel points in its neighborhood can be determined. Substituting this relative coordinate relationship into the encoding calculation formula s = x ₁ +x ₂ R ₁ +x ₃ R ₁ R ₂ , the relative encoding relationship between the skeleton point and the pixel points in its neighborhood can be obtained.

Starting from the starting point, search for skeleton points in the neighborhood of the starting point. The skeleton points found are all child nodes of the starting point, and the starting point is the parent node of these skeleton points. Similarly, for each skeleton point except the starting point, there must be a parent node in its neighborhood. The skeleton points found in its neighborhood except the parent node are the child nodes of the skeleton point. The number of child nodes is counted to get the connectivity number of the skeleton point.

S213: Determine a characteristic skeleton point according to the number of connections between the skeleton point and its parent node.

If the connectivity number of a skeleton point is greater than 1, and the connectivity number of the parent node is equal to 1, the skeleton point is a bifurcation point among the characteristic skeleton points. If the connectivity number of a skeleton point is equal to 1, and the connectivity number of the parent node is equal to 0, the skeleton point is an end point among the characteristic skeleton points.

S22: Divide the pipeline centerline into a plurality of pipeline centerline segments according to the characteristic skeleton points.

The parent feature point is determined for each feature skeleton point except the starting point. The two endpoints of each pipeline centerline segment are a feature skeleton point and its parent feature point. The process of determining the parent feature point for a feature skeleton point includes: searching along the direction of the parent node of the feature skeleton point, that is, finding the parent node of the feature skeleton point, then finding the parent node of the parent node, and so on. The first feature skeleton point found is the parent feature point of the feature skeleton point.

Since the pipeline centerline may have burrs, resulting in errors in the extracted feature skeleton points and the divided pipeline centerline segments, the following two steps are optionally performed to filter out the erroneous segments and feature skeleton points.

S23: Count the number of skeleton points in the pipeline centerline segment.

S24: deleting pipeline centerline segments whose number is less than a preset threshold.

A parameter for indicating the level can be set for the feature skeleton point. The level parameter of each feature skeleton point is the level parameter of its parent feature point plus 1. The smaller the level parameter, the higher the level of the feature skeleton point and the closer it is to the trachea. The preset threshold can be determined based on the level parameter of the endpoint of the pipeline center segment. Generally speaking, the larger the level parameter of the feature skeleton point, the shorter the bronchial segment corresponding to the feature skeleton point, and accordingly, the smaller the preset threshold. Optionally, a level range is set. The level range generally excludes larger level parameters, and only the pipeline centerline segments whose endpoints are within the level range are filtered.

If a pipeline centerline segment is deleted, all skeleton points except the starting point _si (i.e., the one with the smaller level parameter of the two endpoints) are changed to non-skeleton points, and all pipeline centerline segments with non-skeleton points as starting points are deleted. If the number of pipeline centerline segments with _si as the starting point is equal to 1 after deletion, _si is modified to a normal skeleton point. If the number of pipeline centerline segments with _si as the starting point is equal to 0 after deletion, _si is modified to an end point.

The embodiment of the present application also provides a control system for a catheter robot. Please refer to Figure 15, which shows a schematic diagram of the structure of the control system of the catheter system provided by an embodiment of the present application. As shown in Figure 15, the control system 600 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, and the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the memory 61 stores computer program instructions that can be executed by the processor 60, and when the processor 60 executes the computer program instructions, it executes the alignment method of the catheter robot provided in any of the aforementioned embodiments of the present application.

The memory 61 may include a high-speed random access memory (RAM). Memory), and may also include non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

The bus 62 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 61 is used to store programs, and the processor 60 executes the programs after receiving the execution instructions. The alignment method of the catheter robot disclosed in any of the embodiments of the present application may be applied to the processor 60, or implemented by the processor 60.

The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 60 or the instruction in the form of software. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

The control system of the catheter robot provided in the embodiment of the present application is the same as the catheter provided in the embodiment of the present application. The robot registration method is based on the same inventive concept and has the same beneficial effects as the method adopted, operated or implemented by the robot.

An embodiment of the present application also provides a computer-readable storage medium corresponding to the catheter robot alignment method provided in the aforementioned embodiment. Please refer to Figure 16, which shows a computer-readable storage medium 6 on which computer program instructions are stored. When the computer program instructions are executed by the processor, they will implement the catheter robot alignment method provided in any of the aforementioned embodiments.

It should be noted that examples of the computer-readable storage medium may include, but are not limited to, optical disks, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the catheter robot registration method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

It should be noted:

In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

Similarly, it should be understood that in order to simplify the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosure should not be considered The method of describing the invention is to be interpreted as reflecting a schematic representation that the claimed application requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of the invention.

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules based on needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

In the embodiments provided by the present invention, it should be understood that the disclosed devices/terminal equipment and methods can be implemented in other ways. For example, the device/terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected based on actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated module/unit is implemented in the form of a software functional unit and used as an independent product When the product is sold or used, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or combinations thereof.

It should also be understood that the term “and/or” used in the specification and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if the described condition or event is detected" May be interpreted to mean "upon determination" or "in response to determination" or "upon detection of a described condition or event" or "in response to detection of a described condition or event" depending on the context.

In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims (12)

A catheter robot, wherein the catheter robot comprises a mechanical arm, a catheter instrument engaged with a power unit of the mechanical arm, and a processor communicatively connected to the mechanical arm, the catheter instrument comprises an instrument box configured to be engaged with the power unit and a catheter connected to the instrument box, the catheter and/or the instrument carried by the catheter are provided with a sensor for measuring the position of the catheter and/or the instrument, and the processor is configured to perform the following steps: utilizing the sensor to obtain actual path points of the catheter and/or the instrument; Acquire simulated path points of the catheter and/or the instrument corresponding to the actual path points in the anatomical model according to the first transformation matrix, wherein the anatomical model includes a pipeline centerline; Determine a target pipeline centerline segment corresponding to the simulation path point among the plurality of pipeline centerline segments into which the pipeline centerline is divided; Calculating the centroid of the target pipeline centerline segment; If the centroid is different from the multiple skeleton points included in the target pipeline centerline segment, a line between the centroid and the simulated path point is determined, and one of the multiple skeleton points with the shortest distance to the line is selected as the matching point of the actual path point. The catheter robot according to claim 1, wherein the processor is configured to perform the following steps after calculating the centroid of the target pipeline centerline segment: If the centroid is one of the multiple skeleton points, obtain the first perpendicular foot from the simulated path point to the projection line, and obtain the second perpendicular foot from each skeleton point to the projection line, where the projection line is a straight line passing through the two end points of the target pipeline centerline segment; select the skeleton point corresponding to the second perpendicular foot with the shortest distance to the first perpendicular foot as the matching point of the actual path point. The catheter robot according to claim 2, wherein the processor is configured to perform the following steps: Updating the point pair set using the first point pair consisting of the actual path point and its matching point; If the point pair set satisfies a first condition, the point pair set is used to update the first transformation matrix, wherein the first condition includes that the number of point pairs in the point pair set is greater than a preset value. The catheter robot according to claim 3, wherein the processor is configured to perform the following steps in the updating of the point pair set using the matching point pairs consisting of the actual path points and their matching points: Determine whether there is a second point pair including the matching point in the point pair set; If it does not exist, the first point pair is added to the point pair set; if it exists, the first distance is calculated based on the actual path point and the matching point, and the second distance is calculated based on the path point in the second point pair and the matching point; if the first distance is less than the second distance, the second point pair is replaced by the first point pair. The catheter robot according to claim 3, wherein the processor is configured to perform the following steps in the updating of the first transformation matrix using the point pair set: Substituting the point pairs in the point pair set into the objective function to calculate the latest second transformation matrix; The first transformation matrix is updated according to the latest second transformation matrix. The catheter robot according to claim 5, wherein the first transformation matrix is a weighted average of the latest second transformation matrix and at least part of the historical second transformation matrix, wherein the weight of the latest second transformation matrix is greater than the weight of the historical second transformation matrix. The catheter robot according to claim 1, wherein the processor is configured to perform the following steps before determining the target pipeline centerline segment corresponding to the simulated path point among the plurality of pipeline centerline segments into which the pipeline centerline is divided: Extracting characteristic skeleton points from a plurality of skeleton points included in the pipeline centerline, wherein the positions of the skeleton points are represented by codes, the codes of the skeleton points being s=x ₁ +x ₂ R ₁ +x ₃ R ₁ R ₂ , wherein x ₁ , x ₂ , x ₃ are coordinates of the skeleton points in the first, second, and third dimensions of the coding order, respectively, and R ₁ , R ₂ are sizes of the anatomical model in the first and second dimensions of the coding order, respectively; The pipeline centerline is divided into the plurality of pipeline centerline segments according to the characteristic skeleton points. The catheter robot according to claim 7, wherein the processor is configured to perform the following steps in extracting a feature skeleton point from a plurality of skeleton points of the pipeline centerline: Determine the starting point; Starting from the starting point, the skeleton points are searched using the region growing method and the skeleton points are counted. Connectivity number, which is the number of all child nodes in the neighborhood of the skeleton point; If the connectivity number of the skeleton point is greater than 1 and the connectivity number of the parent node is equal to 1, then the skeleton point is a bifurcation point among the characteristic skeleton points; if the connectivity number of the skeleton point is equal to 1 and the connectivity number of the parent node is equal to 0, then the skeleton point is an end point among the characteristic skeleton points. The catheter robot according to claim 8, wherein the processor is configured to perform the following steps in the determining the starting point: Randomly select an integral layer; Counting the total number of the skeleton points in a specified number of layers on both sides of the body layer in a direction perpendicular to the cross section to determine the direction in which the airway extends; The skeleton point of the tracheal entrance is found as the starting point according to the total number of the skeleton points in the body layer and the direction in which the airway extends. The catheter robot according to claim 7, wherein the processor is configured to perform the following steps: Counting the number of skeleton points in the pipeline centerline segment; The pipeline centerline segments whose number is less than a preset threshold are deleted. A catheter robot registration method, comprising: Using sensors disposed on the catheter and/or the device carried by the catheter to obtain actual path points of the catheter and/or the device; Acquire simulated path points of the catheter and/or the instrument corresponding to the actual path points in the anatomical model according to the first transformation matrix, wherein the anatomical model includes a pipeline centerline; Determine a target pipeline centerline segment corresponding to the simulation path point from among the plurality of pipeline centerline segments into which the pipeline centerline is divided, wherein the target pipeline centerline segment includes a plurality of skeleton points; Calculating the centroid of the target pipeline centerline segment; If the centroid is different from the multiple skeleton points, a line between the centroid and the simulated path point is determined, and one of the multiple skeleton points having the shortest distance to the line is selected as a matching point of the actual path point. A computer-readable storage medium, wherein the computer-readable storage medium stores Computer program instructions, the computer program instructions are configured to be loaded by a processor and executed to implement the steps of the method described below: Using sensors disposed on the catheter and/or the device carried by the catheter to obtain actual path points of the catheter and/or the device; Acquire simulated path points of the catheter and/or the instrument corresponding to the actual path points in the anatomical model according to the first transformation matrix, wherein the anatomical model includes a pipeline centerline; Determine a target pipeline centerline segment corresponding to the simulation path point from among the plurality of pipeline centerline segments into which the pipeline centerline is divided, wherein the target pipeline centerline segment includes a plurality of skeleton points; Calculating the centroid of the target pipeline centerline segment; If the centroid is different from the multiple skeleton points, a line between the centroid and the simulated path point is determined, and one of the multiple skeleton points having the shortest distance to the line is selected as a matching point of the actual path point.