WO2024174779A1

WO2024174779A1 - Endoscope registration method and device, and endoscope calibration system

Info

Publication number: WO2024174779A1
Application number: PCT/CN2024/072895
Authority: WO
Inventors: 戴知宇; 高元倩
Original assignee: Shenzhen Edge Medical Co Ltd
Current assignee: Shenzhen Edge Medical Co Ltd
Priority date: 2023-02-23
Filing date: 2024-01-18
Publication date: 2024-08-29
Anticipated expiration: 2025-08-23
Also published as: CN115908121B; CN115908121A

Abstract

The present invention relates to an endoscope registration method and device, and an endoscope calibration system. The registration method comprises: acquiring a three-dimensional tomography image; according to coordinates of each identification member in the image, obtaining the position of a checkerboard in the image; according to the pose of each first electromagnetic sensor in an electromagnetic space, obtaining the pose of the checkerboard and a first transformation relationship between the image and the electromagnetic space; calibrating an endoscope to obtain a second transformation relationship between a camera coordinate system of the endoscope and a coordinate system of a second electromagnetic sensor; according to the first transformation relationship, the second transformation relationship, and the pose of the second electromagnetic sensor in the electromagnetic space, obtaining the pose of a virtual camera corresponding to the endoscope in the image, and obtaining a virtual field of view according to the position of the checkerboard in the image; performing registration on a real field of view and the virtual field of view to obtain a third transformation relationship; and according to the first transformation relationship, the second transformation relationship, and the third transformation relationship, obtaining a transformation relationship between the pose of the virtual camera in the image and the pose of the second electromagnetic sensor in the electromagnetic space during photographing by the endoscope.

Description

Endoscope registration method, device and calibration system

本申请要求于2023年02月23日提交中国专利局、申请号为CN 202310154245.9、申请名称为“内窥镜配准方法及装置和标定系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims the priority of the Chinese patent application filed with the China Patent Office on February 23, 2023, with application number CN 202310154245.9 and application name “Endoscopic alignment method, device and calibration system”, the entire contents of which are incorporated by reference in this application.

Technical Field

本申请涉及医疗器械技术领域，尤其涉及一种内窥镜配准方法、一种内窥镜配准装置以及一种内窥镜标定系统。The present application relates to the field of medical device technology, and in particular to an endoscope registration method, an endoscope registration device, and an endoscope calibration system.

Background Art

现代的微创或无创手术中，一种常用的术中导航方法是基于电磁定位，具体包括：基于电磁定位得到当前时刻内窥镜的位姿，得到术前的三维医学影像中与当前时刻的内窥镜对应的虚拟相机拍摄到的虚拟视野，与当前时刻内窥镜拍摄到的真实视野叠加，实现术中导航。In modern minimally invasive or non-invasive surgeries, a commonly used intraoperative navigation method is based on electromagnetic positioning, which specifically includes: obtaining the posture of the endoscope at the current moment based on electromagnetic positioning, obtaining the virtual field of view captured by the virtual camera corresponding to the endoscope at the current moment in the preoperative three-dimensional medical image, and superimposing it with the real field of view captured by the endoscope at the current moment to achieve intraoperative navigation.

通常电磁传感器和内窥镜的镜头都设置在送入人体内部的医疗器械的末端，且两者的位置关系相对固定。而在手术过程中，会采用电磁传感器反馈的位姿数据作为内窥镜位姿的依据。然而，因为内窥镜和电磁传感器之间存在相对位移和偏转，若直接利用电磁传感器的位姿数据进行定位会产生误差，导致内窥镜的实际位姿参数不准确，进而导致导航时内窥镜的真实视野和虚拟的三维影像不匹配，这种真实视野与虚拟的三维影像的不匹配可能会引发严重的医疗事故。Usually, the electromagnetic sensor and the endoscope lens are both set at the end of the medical device inserted into the human body, and the positional relationship between the two is relatively fixed. During the operation, the posture data fed back by the electromagnetic sensor will be used as the basis for the posture of the endoscope. However, because there is a relative displacement and deflection between the endoscope and the electromagnetic sensor, if the posture data of the electromagnetic sensor is directly used for positioning, errors will occur, resulting in inaccurate actual posture parameters of the endoscope, which will lead to a mismatch between the real field of view of the endoscope and the virtual three-dimensional image during navigation. This mismatch between the real field of view and the virtual three-dimensional image may cause serious medical accidents.

目前关于电磁传感器和内窥镜的标定主要是通过棋盘格作为媒介，内窥镜通过拍摄多帧图像，采集得到棋盘格的图像信息，而后利用张正友法求解出内窥镜的相机内外参数，得到棋盘格与内窥镜之间的坐标变换关系。电磁传感器依据棋盘格的网格尺寸大小和数目，构建棋盘格的电磁坐标系。同时，在内窥镜的拍摄过程中，固定在医疗器械末端的电磁传感器随着内窥镜同步运动，记录每帧图像下电磁传感器的坐标值，从而得到棋盘格与电磁传感器之间的坐标变换关系；最终求解得到电磁传感器与内窥镜之间的坐标变换关系，从而完成两者的标定流程。At present, the calibration of electromagnetic sensors and endoscopes is mainly carried out through the chessboard as a medium. The endoscope captures multiple frames of images to collect the image information of the chessboard, and then uses Zhang Zhengyou's method to solve the internal and external parameters of the endoscope camera to obtain the coordinate transformation relationship between the chessboard and the endoscope. The electromagnetic sensor constructs the electromagnetic coordinate system of the chessboard according to the size and number of the grids of the chessboard. At the same time, during the shooting process of the endoscope, the electromagnetic sensor fixed at the end of the medical device is synchronized with the endoscope. The coordinate value of the electromagnetic sensor in each frame of the image is recorded to obtain the coordinate transformation relationship between the chessboard and the electromagnetic sensor; finally, the coordinate transformation relationship between the electromagnetic sensor and the endoscope is solved to complete the calibration process of the two.

但是发明人发现这一标定过程只是确定了电磁传感器与内窥镜之间的坐标变换关系，而没有实现内窥镜的真实视野与虚拟视野的匹配。However, the inventors found that this calibration process only determined the coordinate transformation relationship between the electromagnetic sensor and the endoscope, but did not achieve the matching of the real field of view and the virtual field of view of the endoscope.

发明内容Summary of the invention

为克服现有技术存在的不足，本申请实施例提供一种内窥镜配准方法、一种内窥镜配准装置以及一种内窥镜标定系统。In order to overcome the deficiencies of the prior art, the embodiments of the present application provide an endoscope registration method, an endoscope registration device and an endoscope calibration system.

具体地，本申请实施例提供的一种内窥镜配准方法，例如包括步骤：Specifically, an endoscope registration method provided in an embodiment of the present application includes the following steps:

(a)获取包含棋盘格标定工具的三维断层扫描影像，所述棋盘格标定工具包括印有已知尺寸的棋盘格的标定板和多个在所述三维断层扫描影像中可识别的标识件，所述标定板上设有多个用于放置第一电磁传感器的第一安装件，所述棋盘格、所述标识件和所述第一安装件之间的相对位置已知；(a) acquiring a three-dimensional tomographic image including a checkerboard calibration tool, wherein the checkerboard calibration tool comprises a calibration plate printed with a checkerboard of known size and a plurality of identification members identifiable in the three-dimensional tomographic image, wherein the calibration plate is provided with a plurality of first mounting members for placing a first electromagnetic sensor, and the relative positions of the checkerboard, the identification members and the first mounting members are known;

(b)根据所述标识件在所述三维断层扫描影像中的坐标得到所述棋盘格在所述三维断层扫描影像中的位置；(b) obtaining the position of the chessboard in the three-dimensional tomographic image according to the coordinates of the identification element in the three-dimensional tomographic image;

(c)根据从电磁定位仪获取的所述第一电磁传感器在电磁空间中的位姿得到所述棋盘格在所述电磁空间中的位姿以及所述三维断层扫描影像相对于所述电磁空间的第一变换关系，其中所述第一电磁传感器放置于所述第一安装件；(c) obtaining the position of the chessboard in the electromagnetic space and a first transformation relationship of the three-dimensional tomography image relative to the electromagnetic space according to the position of the first electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator, wherein the first electromagnetic sensor is placed on the first mounting member;

(d)基于使用固定有第二电磁传感器的内窥镜在多个不同角度下分别对所述棋盘格标定工具进行拍照的结果对所述内窥镜进行标定，得到所述内窥镜的相机内参、以及所述内窥镜的相机坐标系相对于所述第二电磁传感器的坐标系的第二变换关系； (d) calibrating the endoscope based on the results of taking pictures of the checkerboard calibration tool at multiple different angles using an endoscope fixed with a second electromagnetic sensor, to obtain the camera internal parameters of the endoscope and a second transformation relationship between the camera coordinate system of the endoscope and the coordinate system of the second electromagnetic sensor;

(e)根据所述第一变换关系、所述第二变换关系、以及使用所述内窥镜拍照时从所述电磁定位仪获取的所述第二电磁传感器在所述电磁空间中的位姿得到对应于所述内窥镜的虚拟相机在所述三维断层扫描影像中的位姿，将所述虚拟相机的内参设置为所述内窥镜的相机内参，并结合所述棋盘格在所述三维断层扫描影像中的位置得到虚拟视野；(e) obtaining the position of a virtual camera corresponding to the endoscope in the three-dimensional tomographic image according to the first transformation relationship, the second transformation relationship, and the position of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when taking pictures with the endoscope, setting the internal parameters of the virtual camera as the camera internal parameters of the endoscope, and obtaining a virtual field of view in combination with the position of the chessboard in the three-dimensional tomographic image;

(f)对真实视野和所述虚拟视野进行配准，得到所述虚拟视野相对于所述真实视野的第三变换关系，所述真实视野为使用所述内窥镜拍照得到的图像；以及，(f) registering the real field of view with the virtual field of view to obtain a third transformation relationship between the virtual field of view and the real field of view, wherein the real field of view is an image obtained by taking a photo with the endoscope; and

(g)根据所述第一变换关系、所述第二变换关系、所述第三变换关系得到所述虚拟相机在所述三维断层扫描影像中的位姿相对于所述内窥镜拍照时从所述电磁定位仪获取的所述第二电磁传感器在所述电磁空间中的位姿的变换关系。(g) According to the first transformation relationship, the second transformation relationship, and the third transformation relationship, a transformation relationship is obtained between the posture of the virtual camera in the three-dimensional tomography image and the posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when the endoscope takes pictures.

另一方面，本申请实施例提供的一种内窥镜配准装置，例如包括：处理器和连接所述处理器的存储器；其中所述存储器存储有被所述处理器执行的指令，且所述指令被所述处理器执行时实现如下所述的内窥镜配准方法：On the other hand, an endoscope registration device provided in an embodiment of the present application, for example, includes: a processor and a memory connected to the processor; wherein the memory stores instructions executed by the processor, and when the instructions are executed by the processor, the endoscope registration method described below is implemented:

(c)根据从电磁定位仪获取的所述第一电磁传感器在电磁空间中的位姿得到所述棋盘格在所述电磁空间中的位姿以及所述三维断层扫描影像相对于所述电磁空间的第一变换关系，其中所述第一电磁传感器放置于所述第一安装件；(c) obtaining the position of the chessboard in the electromagnetic space and the relative position of the three-dimensional tomographic image to the electromagnetic space according to the position of the first electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator; The first transformation relationship of the electromagnetic space, wherein the first electromagnetic sensor is placed on the first mounting member;

(d)基于使用固定有第二电磁传感器的内窥镜在多个不同角度下分别对所述棋盘格标定工具进行拍照的结果对所述内窥镜进行标定，得到所述内窥镜的相机内参、以及所述内窥镜的相机坐标系相对于所述第二电磁传感器的坐标系的第二变换关系；(d) calibrating the endoscope based on the results of taking pictures of the checkerboard calibration tool at multiple different angles using an endoscope fixed with a second electromagnetic sensor, to obtain the camera internal parameters of the endoscope and a second transformation relationship between the camera coordinate system of the endoscope and the coordinate system of the second electromagnetic sensor;

再一方面，本申请实施例提供的一种内窥镜标定系统，例如包括：前述实施例的内窥镜配准装置以及棋盘格标定工具，所述棋盘格标定工具包括印有已知尺寸的棋盘格的标定板和多个在所述三维断层扫描影像中可识别的标识件，所述标定板上设有多个用于放置第一电磁传感器的第一安装件，所述棋盘格、所述标识件和所述第一安装件之间的相对位置已知。 On the other hand, an embodiment of the present application provides an endoscope calibration system, for example, including: the endoscope alignment device of the aforementioned embodiment and a checkerboard calibration tool, the checkerboard calibration tool including a calibration plate printed with a checkerboard of known size and a plurality of identification members identifiable in the three-dimensional tomographic image, the calibration plate is provided with a plurality of first mounting members for placing a first electromagnetic sensor, and the relative positions among the checkerboard, the identification members and the first mounting members are known.

综上所述，本申请各个实施例的内窥镜配准方法、内窥镜配准装置和内窥镜标定系统通过采用改进后的棋盘格标定工具作为媒介，除了对第二电磁传感器与内窥镜进行配准之外，还对真实视野和虚拟视野进行配准，实现第二电磁传感器与虚拟相机之间的配准。并且采用第一安装件来放置第一电磁传感器，提高棋盘格在电磁空间中的位姿的精度，结合对真实视野和虚拟视野之间的精确配准，提高了第二电磁传感器与虚拟相机之间的配准的精度，从而有利于术中导航。In summary, the endoscope registration method, endoscope registration device and endoscope calibration system of each embodiment of the present application use the improved chessboard calibration tool as a medium, in addition to aligning the second electromagnetic sensor with the endoscope, also aligning the real field of view with the virtual field of view, to achieve the alignment between the second electromagnetic sensor and the virtual camera. In addition, the first mounting member is used to place the first electromagnetic sensor, improve the accuracy of the chessboard's position in the electromagnetic space, and combine the precise alignment between the real field of view and the virtual field of view to improve the accuracy of the alignment between the second electromagnetic sensor and the virtual camera, thereby facilitating intraoperative navigation.

BRIEF DESCRIPTION OF THE DRAWINGS

为了更清楚地说明本申请实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

图1为本申请实施例提供的一种内窥镜配准方法的流程示意图。FIG1 is a schematic flow chart of an endoscope registration method provided in an embodiment of the present application.

图2为本申请实施例提供的一种棋盘格标定工具的结构示意图。FIG. 2 is a schematic diagram of the structure of a chessboard calibration tool provided in an embodiment of the present application.

图3为图2所示棋盘格标定工具的一种剖视示意图。FIG. 3 is a schematic cross-sectional view of the checkerboard calibration tool shown in FIG. 2 .

图4为图1所示内窥镜配准方法中的相关过程状态示意图。FIG. 4 is a schematic diagram of relevant process states in the endoscope registration method shown in FIG. 1 .

图5为图1所示内窥镜配准方法中使用的一种双目相机的对极几何约束示意图。FIG. 5 is a schematic diagram of epipolar geometry constraints of a binocular camera used in the endoscope registration method shown in FIG. 1 .

图6为本申请实施例提供的一种内窥镜配准装置的模块示意图。FIG. 6 is a schematic diagram of a module of an endoscope registration device provided in an embodiment of the present application.

图7为本申请实施例提供的另一种内窥镜配准装置的结构示意图。FIG. 7 is a schematic diagram of the structure of another endoscope registration device provided in an embodiment of the present application.

图8为本申请实施例提供的一种内窥镜标定系统的架构示意图。FIG8 is a schematic diagram of the architecture of an endoscope calibration system provided in an embodiment of the present application.

图9为本申请实施例提供的另一种内窥镜标定系统的结构示意图。 FIG. 9 is a schematic diagram of the structure of another endoscope calibration system provided in an embodiment of the present application.

DETAILED DESCRIPTION

下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

参见图1，本申请实施例提供的一种内窥镜配准方法，包括以下步骤S11～S17。Referring to FIG. 1 , an endoscope registration method provided in an embodiment of the present application includes the following steps S11 to S17 .

步骤S11：获取包含棋盘格标定工具的三维断层扫描影像，所述棋盘格标定工具包括印有已知尺寸的棋盘格的标定板和多个在所述三维断层扫描影像中可识别的标识件，所述标定板上设有多个用于放置第一电磁传感器的第一安装件，所述棋盘格、所述标识件和所述第一安装件之间的相对位置已知。Step S11: Acquire a three-dimensional tomographic image including a checkerboard calibration tool, wherein the checkerboard calibration tool includes a calibration plate printed with a checkerboard of known size and a plurality of identification members identifiable in the three-dimensional tomographic image, wherein the calibration plate is provided with a plurality of first mounting members for placing a first electromagnetic sensor, and the relative positions of the checkerboard, the identification members and the first mounting members are known.

本申请实施例为了改进内窥镜与电磁传感器标定的方式，提出了一种新的棋盘格标定工具50作为媒介。举例说明，如图2和图3所示，棋盘格标定工具50可以包括印有已知尺寸的棋盘格的标定板51和多个标识件54、且所述标定板51上设有多个用于放置电磁传感器的容置孔55，所述棋盘格、所述标识件54和所述容置孔55之间的相对位置已知。具体而言，所述棋盘格典型地包括多个黑白相间的网格53，各个网格53的尺寸大小已知；所述标识件54可以分别固定设置在标定板51的各个角落处，所述标识件54和所述容置孔55的数量均可以是图2及图3所示的四个，但本申请实施例并不以此为限，也可以是三个或者更多个。所述容置孔55可以是设置在标定板51的各个角落处凹陷的圆柱形孔，以方便固定电磁传感器；所述标识件54可以是埋入标定板51的各个角落处对应所述容置孔55的位置，其可以是高Hu(Hounsfield Unit，亨斯菲尔德单位)值的聚氯乙烯(Polyvinyl chloride，PVC) 珠，当然也可以换成其他与标定板51的HU值明显不同的材料，以方便在三维CT影像中建立所述棋盘格的坐标系。此外，值得一提的是，在一些实施例中，所述标识件54也可以与所述标定板51分离设置，其形状也可以更改，只要其能够在三维断层扫描影像中可识别、且与所述容置孔55的相对位置已知即可；所述容置孔55也可以替换成其他形式的第一安装件例如夹具以夹持电磁传感器，只要能实现电磁传感器的固持/夹持作用即可。本实施例后续使用的设备典型地涉及电磁定位仪、多个电磁传感器(包含第一电磁传感器和第二电磁传感器)、内窥镜以及所述棋盘格标定工具50。In order to improve the calibration method of the endoscope and the electromagnetic sensor, the embodiment of the present application proposes a new checkerboard calibration tool 50 as a medium. For example, as shown in Figures 2 and 3, the checkerboard calibration tool 50 may include a calibration plate 51 printed with a checkerboard of known size and a plurality of identification members 54, and the calibration plate 51 is provided with a plurality of receiving holes 55 for placing electromagnetic sensors, and the relative positions between the checkerboard, the identification members 54 and the receiving holes 55 are known. Specifically, the checkerboard typically includes a plurality of black and white grids 53, and the size of each grid 53 is known; the identification members 54 can be fixedly arranged at each corner of the calibration plate 51, and the number of the identification members 54 and the receiving holes 55 can be four as shown in Figures 2 and 3, but the embodiment of the present application is not limited thereto, and can also be three or more. The receiving holes 55 may be cylindrical holes recessed at the corners of the calibration plate 51 to facilitate fixing the electromagnetic sensor; the identification member 54 may be embedded in the corners of the calibration plate 51 corresponding to the receiving holes 55, and may be a polyvinyl chloride (PVC) with a high Hu (Hounsfield Unit) value. The beads can of course be replaced with other materials with significantly different HU values from the calibration plate 51, so as to facilitate the establishment of the coordinate system of the checkerboard in the three-dimensional CT image. In addition, it is worth mentioning that in some embodiments, the identification member 54 can also be set separately from the calibration plate 51, and its shape can also be changed, as long as it can be identified in the three-dimensional tomography image and its relative position with the receiving hole 55 is known; the receiving hole 55 can also be replaced with other forms of first mounting members such as clamps to clamp the electromagnetic sensor, as long as the holding/clamping function of the electromagnetic sensor can be achieved. The equipment used subsequently in this embodiment typically involves an electromagnetic locator, a plurality of electromagnetic sensors (including a first electromagnetic sensor and a second electromagnetic sensor), an endoscope, and the checkerboard calibration tool 50.

具体地，在步骤S11中，可以将所述棋盘格标定工具50固定在手术床上，并通过CT扫描(Computer Tomography，计算机断层扫描)，得到包含所述棋盘格标定工具50的三维CT影像作为所述三维断层扫描影像。三维CT影像的坐标系例如设为S_i(x_i,y_i,z_i)，以下简称CT坐标系S_i(x_i,y_i,z_i)。可选的，棋盘格标定工具50可以是固定在手术床上和患者一起进行CT扫描(手术前或手术中皆可)，这样的话得到的三维CT影像中除了所述棋盘格标定工具50还包括患者的手术部位，在此情况下，若电磁定位仪和患者的位姿不出现未知的变化，最终标定得到的虚拟相机相对于第二电磁传感器的变换关系可以直接用于术中导航。Specifically, in step S11, the checkerboard calibration tool 50 can be fixed on the operating table, and a 3D CT image including the checkerboard calibration tool 50 can be obtained as the 3D tomography image by CT scanning (Computer Tomography). The coordinate system of the 3D CT image is, for example, set to S _i ( _xi , _yi , z _i ), hereinafter referred to as the CT coordinate system S _i ( _xi , _yi , z _i ). Optionally, the checkerboard calibration tool 50 can be fixed on the operating table and CT scanned together with the patient (either before or during surgery). In this case, the obtained 3D CT image includes the surgical site of the patient in addition to the checkerboard calibration tool 50. In this case, if the posture of the electromagnetic locator and the patient does not undergo unknown changes, the transformation relationship of the virtual camera obtained by the final calibration relative to the second electromagnetic sensor can be directly used for intraoperative navigation.

步骤S12：根据所述标识件在所述三维断层扫描影像中的坐标得到所述棋盘格在所述三维断层扫描影像中的位置。Step S12: obtaining the position of the chessboard in the three-dimensional tomography image according to the coordinates of the identification element in the three-dimensional tomography image.

具体地，在步骤S12中，可以根据三维CT影像结合阈值(例如HU阈值)分割，提取出各个标识件54在三维CT影像中的坐标，例如其中心点的CT坐标。进一步地，根据各个标识件54的CT坐标结合所述棋盘格标定工具50的棋盘格的每个网格53的真实尺寸大小，可以得到每个网格53的角点在三维CT影像中的坐标，也即CT坐标。此处的网格53的角点为网格53的顶点。Specifically, in step S12, the coordinates of each identification element 54 in the three-dimensional CT image, such as the CT coordinates of its center point, can be extracted based on the segmentation of the three-dimensional CT image in combination with a threshold value (such as a HU threshold value). Further, based on the CT coordinates of each identification element 54 and the actual size of each grid 53 of the chessboard calibration tool 50, the corner point of each grid 53 in the three-dimensional CT image can be obtained. The coordinates in the CT image are also called CT coordinates. The corner points of the grid 53 here are the vertices of the grid 53.

步骤S13：根据从电磁定位仪获取的所述第一电磁传感器在电磁空间中的位姿得到所述棋盘格在所述电磁空间中的位姿以及所述三维断层扫描影像相对于所述电磁空间的第一变换关系，其中所述第一电磁传感器放置于所述第一安装件。Step S13: obtaining the position of the chessboard in the electromagnetic space and a first transformation relationship of the three-dimensional tomography image relative to the electromagnetic space according to the position of the first electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator, wherein the first electromagnetic sensor is placed on the first mounting member.

具体地，在步骤S13中，可以由电磁定位仪构建电磁空间，其世界坐标系可以设为S_w(x_w,y_w,z_w)。取四个电磁传感器作为所述第一电磁传感器置入棋盘格标定工具50的各个容置孔55中，再基于从所述电磁定位仪获取的所述第一电磁传感器在所述电磁空间中的位姿可以得到所述棋盘格标定工具50的棋盘格在所述电磁空间中的位姿，所述棋盘格的电磁坐标系可以设为S_t(x_t,y_t,z_t)。Specifically, in step S13, the electromagnetic space can be constructed by the electromagnetic locator, and its world coordinate system can be set to S _w (x _w ,y _w ,z _w ). Four electromagnetic sensors are taken as the first electromagnetic sensors and placed in the respective receiving holes 55 of the chessboard calibration tool 50 , and then the posture of the chessboard of the chessboard calibration tool 50 in the electromagnetic space can be obtained based on the posture of the first electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator, and the electromagnetic coordinate system of the chessboard can be set to S _t (x _t ,y _t ,z _t ).

再者，可以根据从所述电磁定位仪获取的所述第一电磁传感器在所述电磁空间中的位姿和各个标识件54在三维CT影像中的CT坐标，进行点匹配，从而可以求出三维CT影像相对于所述电磁空间的所述第一变换关系，即CT坐标系S_i(x_i,y_i,z_i)相对于世界坐标系S_w(x_w,y_w,z_w)的旋转矩阵R_wi和平移向量T_wi，此过程也可以称为空间配准。值得一提的是，如果棋盘格标定工具50是固定在手术床上和患者一起进行扫描(手术前或手术中皆可)，这样的话得到的三维CT影像中除了棋盘格标定工具50还包括患者的手术部位，这里的空间配准也就是患者的手术部位的CT影像与电磁空间的配准。Furthermore, point matching can be performed based on the position of the first electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator and the CT coordinates of each identification member 54 in the three-dimensional CT image, so that the first transformation relationship of the three-dimensional CT image relative to the electromagnetic space can be obtained, that is, the rotation matrix R _wi and the translation vector T _wi of the CT coordinate system S _i ( _xi , _yi , z _i ) relative to the world coordinate system S _w ( _xw , _yw , _zw ). This process can also be called spatial registration. It is worth mentioning that if the checkerboard calibration tool 50 is fixed on the operating table and scanned together with the patient (before or during surgery), then the three-dimensional CT image obtained includes the patient's surgical site in addition to the checkerboard calibration tool 50. The spatial registration here is the registration of the CT image of the patient's surgical site with the electromagnetic space.

步骤S14：基于使用固定有第二电磁传感器的内窥镜在多个不同角度下分别对所述棋盘格标定工具进行拍照的结果对所述内窥镜进行标定，得到所述内窥镜的相机内参、以及所述内窥镜的相机坐标系相对于所述第二电磁传感器的坐标系的第二变换关系。 Step S14: Calibrate the endoscope based on the results of taking pictures of the chessboard calibration tool at multiple different angles using an endoscope fixed with a second electromagnetic sensor, and obtain the camera internal parameters of the endoscope and the second transformation relationship between the camera coordinate system of the endoscope and the coordinate system of the second electromagnetic sensor.

具体地，在步骤S14中，将内窥镜与作为所述第二电磁传感器的电磁传感器两者固定，基于所述第二电磁传感器构建的坐标系可以为传感器坐标系S_s(x_s,y_s,z_s)，基于内窥镜构建的相机坐标系可以为S_c(x_c,y_c,z_c)，各个坐标系如图4所示。接下来，可以进行内窥镜与第二电磁传感器的标定过程。Specifically, in step S14, the endoscope and the electromagnetic sensor serving as the second electromagnetic sensor are fixed, the coordinate system constructed based on the second electromagnetic sensor may be the sensor coordinate system S _s (x _s , y _s , z _s ), and the camera coordinate system constructed based on the endoscope may be _Sc (x _c , y _c , z _c ), and each coordinate system is shown in Fig. 4. Next, a calibration process of the endoscope and the second electromagnetic sensor may be performed.

承上述，通过内窥镜在多个不同角度下分别对所述棋盘格标定工具50进行拍照以采集多个不同角度的棋盘格图像信息，利用张正友标定法求解得到内窥镜的相机内参K，以及不同角度下的外部参数其中，R_tc和T_tc分别是相机坐标系S_c(x_c,y_c,z_c)相对于所述棋盘格在所述电磁空间中的位姿(对应棋盘格的电磁坐标系S_t(x_t,y_t,z_t))的旋转矩阵和平移向量。所述相机内数K例如是相机内参矩阵，其典型地为(dx,dy,r,u,v,f)；其中，dx、dy表示一个像素的物理尺寸，f表示焦距，r表示图像物理坐标的扭曲因子，u和v(单位为像素)表示图像原点相对于光心成像点的横纵偏移量。Based on the above, the chessboard calibration tool 50 is photographed at multiple different angles by an endoscope to collect chessboard image information at multiple different angles, and the Zhang Zhengyou calibration method is used to solve the camera internal parameter K of the endoscope and the external parameters at different angles. Wherein, R _tc and T _tc are respectively the rotation matrix and translation vector of the camera coordinate system _Sc ( _xc , _yc , _zc ) relative to the posture of the chessboard in the electromagnetic space (corresponding to the electromagnetic coordinate system _St ( _xt , _yt , _zt ) of the chessboard). The camera intrinsic number K is, for example, a camera intrinsic parameter matrix, which is typically (dx, dy, r, u, v, f); wherein dx and dy represent the physical size of a pixel, f represents the focal length, r represents the distortion factor of the image physical coordinates, and u and v (in pixels) represent the horizontal and vertical offsets of the image origin relative to the optical center imaging point.

再者，由于所述第二电磁传感器跟随内窥镜运动，因而同步采集了所述第二电磁传感器在内窥镜不同角度下拍照时在所述电磁空间中的位姿，从而可以得到所述棋盘格在所述电磁空间中的位姿(对应棋盘格的电磁坐标系S_t(x_t,y_t,z_t))相对于传感器坐标系S_s(x_s,y_s,z_s)的旋转矩阵R_st和平移向量T_st。Furthermore, since the second electromagnetic sensor moves with the endoscope, the position and posture of the second electromagnetic sensor in the electromagnetic space when the endoscope takes pictures at different angles is synchronously collected, so that the rotation matrix Rst and translation vector Tst of the _position and posture of the chessboard in the electromagnetic space (corresponding to the electromagnetic coordinate system _St ( _xt , _yt , _zt ) of the chessboard) relative to the sensor coordinate system _Ss ( _xs , _ys , _zs ) can be _obtained .

根据上述的旋转矩阵R_tc、R_st和平移向量T_tc、T_st，可以求出所述内窥镜的相机坐标系S_c(x_c,y_c,z_c)相对于传感器坐标系S_s(x_s,y_s,z_s)的所述第二变换关系，完成对所述内窥镜的标定。若设第二电磁传感器在所述电磁空间中的位置和姿态分别为p_s、R_s，内窥镜在所述电磁空间中的位置和姿态分别为p_c、R_c，则：
According to the above rotation matrices R _tc , R _st and translation vectors T _tc , T _st , the second transformation relationship of the camera coordinate system S _c (x _c , y _c , z _c ) of the endoscope relative to the sensor coordinate system S _s (x _s , y _s , z _s ) can be obtained to complete the calibration of the endoscope. If the position and posture of the second electromagnetic sensor in the electromagnetic space are p _s and R _s respectively, and the position and posture of the endoscope in the electromagnetic space are p _c and R _c respectively, then:

其中，R_c、p_c共同定义内窥镜在所述电磁空间中的位姿，R_s、p_s共同定义所述第二电磁传感器在所述电磁空间中的位姿。Among them, R _c and p _c jointly define the position of the endoscope in the electromagnetic space, and R _s and p _s jointly define The present invention relates to a method for determining a position of the second electromagnetic sensor in the electromagnetic space.

由于所述第二电磁传感器可以输出当前位姿下与X、Y、Z三轴的夹角α、β、γ，从而得到第二电磁传感器在世界坐标系S_w(x_w,y_w,z_w)中的姿态，通过公式(1)中的变换矩阵，可以求得内窥镜的姿态。R_c具体表示内窥镜在世界坐标系S_w(x_w,y_w,z_w)中的姿态，即内窥镜的相机坐标系S_c(x_c,y_c,z_c)相对于世界坐标系S_w(x_w,y_w,z_w)的旋转；R_s具体表示第二电磁传感器在世界坐标系S_w(x_w,y_w,z_w)中的姿态，即传感器坐标系S_s(x_s,y_s,z_s)相对于世界坐标系S_w(x_w,y_w,z_w)的旋转，由夹角α、β、γ可以计算得到。Since the second electromagnetic sensor can output the angles α, β, and γ between the X, Y, and Z axes in the current position, the posture of the second electromagnetic sensor in the world coordinate system _Sw ( _xw , _yw , _zw ) can be obtained. The posture of the endoscope can be obtained through the transformation matrix in formula (1). _Rc specifically represents the posture of the endoscope in the world coordinate system _Sw ( _xw , _yw , _zw ), that is, the rotation of the camera coordinate system _Sc ( _xc , _yc , _zc ) of the endoscope relative to the world coordinate system _Sw ( _xw , _yw , _zw ); _Rs specifically represents the posture of the second electromagnetic sensor in the world coordinate system _Sw ( _xw , _yw , _zw ), that is, the rotation of the sensor coordinate system _Ss ( _xs , _ys , _zs ) relative to the world coordinate system _Sw ( _xw , _yw , _zw ), which can be calculated by the angles α, β, and γ.

步骤S15：根据所述第一变换关系、所述第二变换关系、以及使用所述内窥镜拍照时从所述电磁定位仪获取的所述第二电磁传感器在所述电磁空间中的位姿得到对应于所述内窥镜的虚拟相机在所述三维断层扫描影像中的位姿，将所述虚拟相机的内参设置为所述内窥镜的相机内参，并结合所述棋盘格在所述三维断层扫描影像中的位置得到虚拟视野。Step S15: According to the first transformation relationship, the second transformation relationship, and the posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when taking pictures with the endoscope, the posture of the virtual camera corresponding to the endoscope in the three-dimensional tomography image is obtained, the internal parameters of the virtual camera are set as the camera internal parameters of the endoscope, and the virtual field of view is obtained in combination with the position of the chessboard in the three-dimensional tomography image.

具体地，在步骤S15中，可以针对CT影像进行三维重建，得到棋盘格标定工具50的棋盘格的三维模型。再者，根据上述步骤S13及S14得到的第一变换关系和第二变换关系、并使用所述内窥镜拍照时从所述电磁定位仪获取的第二电磁传感器在所述电磁空间中的位姿可以将内窥镜在电磁空间中的位姿变换到CT空间下，从而得到对应于所述内窥镜的虚拟相机在三维CT影像中的位姿(参见图4的右侧部分)。此外，将所述虚拟相机的内参设置为所述内窥镜的相机内参K。设内窥镜在电磁空间中的位置和姿态分别为p_c、R_c，虚拟相机在三维CT影像(也即CT空间)中的位置和姿态分别为p_i、R_i，则
Specifically, in step S15, three-dimensional reconstruction can be performed on the CT image to obtain a three-dimensional model of the chessboard of the chessboard calibration tool 50. Furthermore, according to the first transformation relationship and the second transformation relationship obtained in the above steps S13 and S14, and the position and posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when taking pictures with the endoscope, the position and posture of the endoscope in the electromagnetic space can be transformed to the CT space, thereby obtaining the position and posture of the virtual camera corresponding to the endoscope in the three-dimensional CT image (see the right part of Figure 4). In addition, the internal parameter of the virtual camera is set to the camera internal parameter K of the endoscope. Assume that the position and posture of the endoscope in the electromagnetic space are p _c and R _c respectively, and the position and posture of the virtual camera in the three-dimensional CT image (that is, CT space) are p _i and R _i respectively, then

结合公式(1)，公式(2)变换为： Combining formula (1), formula (2) is transformed into:

再者，利用所述虚拟相机在三维CT影像中的位姿，结合所述棋盘格在三维CT像中的位置可以得到虚拟视野。至此，可以得到初步配准的真实视野和虚拟视野。此处的真实视野为使用内窥镜拍照得到的图像；虚拟视野为所述虚拟相机在三维CT影像中“拍照”得到的图像，可以根据三维CT影像和虚拟相机的外部参数及内参计算得到。Furthermore, the virtual field of view can be obtained by using the position of the virtual camera in the three-dimensional CT image and combining it with the position of the chessboard in the three-dimensional CT image. At this point, the real field of view and the virtual field of view that are preliminarily aligned can be obtained. The real field of view here is the image obtained by taking pictures with an endoscope; the virtual field of view is the image obtained by "taking pictures" of the virtual camera in the three-dimensional CT image. The external parameters of the three-dimensional CT image and the virtual camera can be used to calculate the virtual field of view. And the internal parameters are calculated.

步骤S16：对真实视野和所述虚拟视野进行配准，得到所述虚拟视野相对于所述真实视野的第三变换关系，所述真实视野为使用所述内窥镜拍照得到的图像。Step S16: aligning the real field of view with the virtual field of view to obtain a third transformation relationship between the virtual field of view and the real field of view, where the real field of view is an image obtained by taking a photo with the endoscope.

由于初步配准精度可能存在误差，为了尽可能使得真实视野和虚拟视野保持一致，因而可以进一步基于真实视野对虚拟视野进行精确配准。Since there may be errors in the initial registration accuracy, in order to make the real field of view and the virtual field of view consistent as much as possible, the virtual field of view may be further accurately registered based on the real field of view.

承上述，由于真实视野和虚拟视野两个视野实际上是很接近的，类似于双目相机的画面，故而可以利用对极几何约束完成两个相机的匹配。双目相机的对极几何约束示意图如图5所示。在图5中，P为空间点(表示空间中的物体)，p₁、p₂分别为P点在真实视野与虚拟视野中的成像点，e₁、e₂为极点(也即光心O₁和O₂连线与成像面的交点)，P、O₁、O₂组成的平面为极平面，p₁和e₁的连线以及p₂和e₂的连线为极线，也就是极平面与成像面的交线。As mentioned above, since the real field of view and the virtual field of view are actually very close, similar to the image of a binocular camera, the epipolar geometry constraint can be used to complete the matching of the two cameras. The schematic diagram of the epipolar geometry constraint of the binocular camera is shown in Figure 5. In Figure 5, P is a spatial point (representing an object in space), _p1 and _p2 are the imaging points of point P in the real field of view and the virtual field of view, respectively, _e1 and _e2 are poles (that is, the intersection of the line connecting the optical centers _O1 and _O2 and the imaging surface), the plane formed by P, _O1 , and _O2 is the epipolar plane, and the line connecting _p1 and _e1 and the line connecting _p2 and _e2 are epipolar lines, that is, the intersection of the epipolar plane and the imaging surface.

具体地，在步骤S16中，可以随机采样多对匹配像素点。这里的像素点可以是所述棋盘格上的网格角点，像素点可以来自一对或者多对真实视野和虚拟视野，不同的真实视野的位姿不同。设一对匹配像素点分别为p₁和p₂，p₁来自于真实视野，p₂来自于与真实视野对应的虚拟视野，则这两个像素点的位置为：
s₁p₁＝KPs₂p₂＝K(R₁₂P+T₁₂) Specifically, in step S16, multiple pairs of matching pixel points may be randomly sampled. The pixel points here may be grid corner points on the chessboard, and the pixel points may come from one or more pairs of real fields of view and virtual fields of view, and different real fields of view have different postures. Assume that a pair of matching pixel points are _p1 and _p2 , _p1 comes from the real field of view, and _p2 comes from the virtual field of view corresponding to the real field of view, then the positions of the two pixel points are:
s ₁ p ₁ =KPs ₂ p ₂ =K(R ₁₂ P+T ₁₂ )

其中，K是相机内参矩阵(对应内窥镜的相机内参)，R₁₂、T₁₂是点p₁到点p₂的坐标转换矩阵，也即所述第三变换关系中的旋转矩阵和平移向量，s₁是点P在真实视野对应的相机坐标系中的深度，s₂是点P在虚拟视野对应的相机坐标系中的深度，最后可以得到对极约束公式：
Among them, K is the camera intrinsic parameter matrix (corresponding to the camera intrinsic parameter of the endoscope), R ₁₂ and T ₁₂ are the coordinate transformation matrices from point p ₁ to point p ₂ , that is, the rotation matrix and translation vector in the third transformation relationship, s ₁ is the depth of point P in the camera coordinate system corresponding to the real field of view, and s ₂ is the depth of point P in the camera coordinate system corresponding to the virtual field of view. Finally, the epipolar constraint formula can be obtained:

代入采样的多对匹配像素点，最终可以求出旋转矩阵R₁₂和平移向量T₁₂，得到目标变换矩阵 Substituting multiple pairs of sampled matching pixels, we can finally find the rotation matrix R ₁₂ and the translation vector T ₁₂ to get the target transformation matrix

步骤S17：根据所述第一变换关系、所述第二变换关系、所述第三变换关系得到所述虚拟相机在所述三维断层扫描影像中的位姿相对于所述内窥镜拍照时从所述电磁定位仪获取的所述第二电磁传感器在所述电磁空间中的位姿的变换关系。Step S17: According to the first transformation relationship, the second transformation relationship, and the third transformation relationship, obtain the transformation relationship of the posture of the virtual camera in the three-dimensional tomography image relative to the posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when the endoscope takes pictures.

具体地，在步骤S17中，由于目标变换矩阵是由真实视野变换到虚拟视野下的，而实际上则是需要将虚拟视野调整至与真实视野一致，故需要对所述目标变换矩阵取逆，得到目标逆变换矩阵 Specifically, in step S17, since the target transformation matrix It is transformed from the real field of view to the virtual field of view. In fact, the virtual field of view needs to be adjusted to be consistent with the real field of view. Therefore, it is necessary to invert the target transformation matrix to obtain the target inverse transformation matrix

之后，基于目标逆变换矩阵和上述步骤S13及S14得到的第一及第二变换关系及可以得到所述虚拟相机在三维CT影像中的位姿相对于内窥镜拍照时从所述电磁定位仪获取的第二电磁传感器在所述电磁空间中的位姿之间的变换关系当已知与内窥镜固定的第二电磁传感器的位姿时，可以通过下列公式(4)将内窥镜在电磁空间的位姿变换至虚拟的CT空间下，从而引导虚拟相机与真实内窥镜的观察视角对齐，进而可以保证真实视野和虚拟视野的精准匹配。Afterwards, based on the target inverse transformation matrix and the first and second transformation relationships obtained in the above steps S13 and S14 and The transformation relationship between the position of the virtual camera in the three-dimensional CT image and the position of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when the endoscope is photographed can be obtained. When the position of the second electromagnetic sensor fixed to the endoscope is known, the position of the endoscope in the electromagnetic space can be transformed into the virtual CT space by the following formula (4), thereby guiding the virtual camera to align with the observation angle of the real endoscope. This can ensure the precise matching of real vision and virtual vision.

设已知与内窥镜固定的第二电磁传感器在电磁空间中的位置和姿态分别为p_s、R_s，虚拟相机在三维CT影像中的位置和姿态分别为p_i、R_i，则两者之间的变换关系为：
Assume that the position and posture of the second electromagnetic sensor fixed to the endoscope in the electromagnetic space are known to be _ps and _Rs , respectively, and the position and posture of the virtual camera in the three-dimensional CT image are known to be p _i and R _i , respectively. The transformation relationship between the two is:

综上所述，本申请实施例的内窥镜配准方法通过采用改进后的棋盘格标定工具作为媒介，除了对第二电磁传感器与内窥镜进行配准之外，还对真实视野和虚拟视野进行配准，实现第二电磁传感器与虚拟相机之间的配准。并且采用第一安装件来放置第一电磁传感器，提高棋盘格在电磁空间中的位姿的精度，结合对真实视野和虚拟视野之间的精确配准，提高了第二电磁传感器与虚拟相机之间的配准的精度，从而有利于术中导航。In summary, the endoscope registration method of the embodiment of the present application uses the improved chessboard calibration tool as a medium, and in addition to registering the second electromagnetic sensor with the endoscope, it also registers the real field of view and the virtual field of view to achieve the registration between the second electromagnetic sensor and the virtual camera. In addition, the first mounting member is used to place the first electromagnetic sensor, improve the accuracy of the chessboard's position in the electromagnetic space, and combine the precise registration between the real field of view and the virtual field of view to improve the accuracy of the registration between the second electromagnetic sensor and the virtual camera, thereby facilitating intraoperative navigation.

参见图6，本申请实施例提供的一种内窥镜配准装置，包括：影像获取模块110、棋盘格位置确定模块120、棋盘格位姿及空间变换关系确定模块130、内窥镜标定模块140、虚拟相机及虚拟视野确定模块150、视野变换关系确定模块160和位姿变换关系确定模块170。6 , an endoscope alignment device provided in an embodiment of the present application includes: an image acquisition module 110, a chessboard position determination module 120, a chessboard posture and spatial transformation relationship determination module 130, an endoscope calibration module 140, a virtual camera and virtual field of view determination module 150, a field of view transformation relationship determination module 160 and a posture transformation relationship determination module 170.

其中，所述影像获取模块110例如用于获取包含棋盘格标定工具的三维断层扫描影像，所述棋盘格标定工具包括印有已知尺寸的棋盘格的标定板和多个在所述三维断层扫描影像中可识别的标识件，所述标定板上设有多个用于放置第一电磁传感器的第一安装件，所述棋盘格、所述标识件和所述第一安装件之间的相对位置已知；所述棋盘格位置确定模块120例如用于根据所述标识件在所述三维断层扫描影像中的坐标得到所述棋盘格在所述三维断层扫描影像中的位置；所述棋盘格位姿及空间变换关系确定模块130例如用于根据从电磁定位仪获取的所述第一电磁传感器在电磁空间中的位姿得到所述棋盘格在所述电磁空间中的位姿以及所述三维断层扫描影像相对于所述电磁空间的第一变换关系，其中所述第一电磁传感器放置于所述第一安装件；所述内窥镜标定模块140例如用于基于使用固定有第二电磁传感器的内窥镜在多个不同角度下分别对所述棋盘格标定工具进行拍照的结果对所述内窥镜进行标定，得到所述内窥镜的相机内参、以及所述内窥镜的相机坐标系相对于所述第二电磁传感器的坐标系的第二变换关系；所述虚拟相机及虚拟视野确定模块150例如用于根据所述第一变换关系、所述第二变换关系、以及使用所述内窥镜拍照时从所述电磁定位仪获取的所述第二电磁传感器在所述电磁空间中的位姿得到对应于所述内窥镜的虚拟相机在所述三维断层扫描影像中的位姿，将所述虚拟相机的内参设置为所述内窥镜的相机内参，并结合所述棋盘格在所述三维断层扫描影像中的位置得到虚拟视野；所述视野变换关系确定模块160例如用于对真实视野和所述虚拟视野进行配准，得到所述虚拟视野相对于所述真实视野的第三变换关系，所述真实视野为使用所述内窥镜拍照得到的图像；以及，所述位姿变换关系确定模块170例如用于根据所述第一变换关系、所述第二变换关系、所述第三变换关系得到所述虚拟相机在所述三维断层扫描影像中的位姿相对于所述内窥镜拍照时从所述电磁定位仪获取的所述第二电磁传感器在所述电磁空间中的位姿的变换关系。The image acquisition module 110 is used, for example, to acquire a three-dimensional tomographic image including a checkerboard calibration tool, wherein the checkerboard calibration tool includes a calibration plate printed with a checkerboard of known size and a plurality of identification members identifiable in the three-dimensional tomographic image, wherein the calibration plate is provided with a plurality of first mounting members for placing a first electromagnetic sensor, and the relative positions of the checkerboard, the identification members and the first mounting members are known; the checkerboard position determination module 120 is used, for example, to obtain the coordinates of the checkerboard in the three-dimensional tomographic image according to the coordinates of the identification members in the three-dimensional tomographic image. The position in the scanned image; the chessboard posture and spatial transformation relationship determination module 130 is used, for example, to obtain the posture of the chessboard in the electromagnetic space and the first transformation relationship of the three-dimensional tomographic scanning image relative to the electromagnetic space according to the posture of the first electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator, wherein the first electromagnetic sensor is placed on the first mounting member; the endoscope calibration module 140 is used, for example, to calibrate the endoscope based on the results of taking pictures of the chessboard calibration tool at multiple different angles using an endoscope fixed with a second electromagnetic sensor, to obtain the camera internal parameters of the endoscope, and the second transformation relationship of the camera coordinate system of the endoscope relative to the coordinate system of the second electromagnetic sensor; the virtual camera and virtual field of view determination module 150 is used, for example, to obtain the second transformation relationship of the camera coordinate system of the endoscope relative to the coordinate system of the second electromagnetic sensor according to the first transformation relationship, the second transformation relationship, and the first transformation relationship of the first electromagnetic sensor when taking pictures using the endoscope The position and posture of the second electromagnetic sensor in the electromagnetic space obtained by the positioner is used to obtain the position and posture of the virtual camera of the endoscope in the three-dimensional tomography image, the internal parameters of the virtual camera are set as the camera internal parameters of the endoscope, and the virtual field of view is obtained in combination with the position of the chessboard in the three-dimensional tomography image; the field of view transformation relationship determination module 160 is used, for example, to align the real field of view and the virtual field of view to obtain a third transformation relationship of the virtual field of view relative to the real field of view, and the real field of view is an image obtained by taking pictures with the endoscope; and the posture transformation relationship determination module 170 is used, for example, to obtain the transformation relationship of the posture of the virtual camera in the three-dimensional tomography image relative to the posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when the endoscope takes pictures according to the first transformation relationship, the second transformation relationship, and the third transformation relationship.

至于所述影像获取模块110、所述棋盘格位置确定模块120、所述棋盘格位姿及空间变换关系确定模块130、所述内窥镜标定模块140、所述虚拟相机及虚拟视野确定模块150、所述视野变换关系确定模块160和所述位姿变换关系确定模块170的具体功能细节可参考前述内窥镜配准方法的相关实施例中的详细描述，在此不再赘述。此外，值得一提的是，所述影像获取模块110、所述棋盘格位置确定模块120、所述棋盘格位姿及空间变换关系确定模块130、所述内窥镜标定模块140、所述虚拟相机及虚拟视野确定模块150、所述视野变换关系确定模块160和所述位姿变换关系确定模块170可以为软件模块，存储于非易失性存储器中且由处理器执行相关操作以进行前述实施例中的步骤S11、S12、S13、S14、S15、S16及S17。As for the specific functional details of the image acquisition module 110, the chessboard position determination module 120, the chessboard posture and spatial transformation relationship determination module 130, the endoscope calibration module 140, the virtual camera and virtual field of view determination module 150, the field of view transformation relationship determination module 160 and the posture transformation relationship determination module 170, reference may be made to the relevant embodiments of the aforementioned endoscope registration method. In addition, it is worth mentioning that the image acquisition module 110, the chessboard position determination module 120, the chessboard posture and space transformation relationship determination module 130, the endoscope calibration module 140, the virtual camera and virtual field of view determination module 150, the field of view transformation relationship determination module 160 and the posture transformation relationship determination module 170 can be software modules, which are stored in a non-volatile memory and the processor performs related operations to perform steps S11, S12, S13, S14, S15, S16 and S17 in the aforementioned embodiments.

参见图7，本申请实施例提供的一种内窥镜配准装置10，包括：处理器11和连接所述处理器11的存储器13。其中，所述存储器13存储有被所述处理器11执行的指令，且所述指令被所述处理器11执行时实现前述任一实施例所述的内窥镜配准方法。7 , an endoscope registration device 10 provided in an embodiment of the present application includes: a processor 11 and a memory 13 connected to the processor 11. The memory 13 stores instructions executed by the processor 11, and when the instructions are executed by the processor 11, the endoscope registration method described in any of the above embodiments is implemented.

此外，本申请其他实施例还提供的一种计算机可读存储介质，其为非易失性存储器且存储有程序代码，当所述程序代码被一个或多个处理器执行时，例如使得所述一个或多个处理器执行前述任一实施例所述的内窥镜配准方法。In addition, other embodiments of the present application also provide a computer-readable storage medium, which is a non-volatile memory and stores program code. When the program code is executed by one or more processors, for example, the one or more processors execute the endoscope alignment method described in any of the aforementioned embodiments.

参见图8，本申请实施例提供的一种内窥镜标定系统，包括：内窥镜配准装置10、电磁定位仪20、计算机三维断层扫描机40、棋盘格标定工具50、多个第一电磁传感器60、内窥镜70和第二电磁传感器80。Referring to Figure 8, an endoscope calibration system provided in an embodiment of the present application includes: an endoscope alignment device 10, an electromagnetic locator 20, a computer three-dimensional tomography scanner 40, a chessboard calibration tool 50, a plurality of first electromagnetic sensors 60, an endoscope 70 and a second electromagnetic sensor 80.

其中，所述内窥镜配准装置10用于执行前述任一实施例所述的内窥镜配准方法，所述电磁定位仪20和所述计算机三维断层扫描机40分别通信连接所述内窥镜配准装置10，所述计算机三维断层扫描机40(简称CT机)用于提供三维断层扫描影像例如扫描输出包含棋盘格标定工具的CT影像，所述电磁定位仪20用于构建电磁空间以及感测所述第一电磁传感器60和所述第二电磁传感器80在所述电磁空间中的位姿。再者，各个第一电磁传感器60可以放置于所述棋盘格标定工具50的各个第一安装件，所述第二电磁传感器 80可以在利用所述内窥镜70在多个不同角度下分别对所述棋盘格标定工具50进行拍照时与所述内窥镜70固定在一起。The endoscope alignment device 10 is used to execute the endoscope alignment method described in any of the aforementioned embodiments, the electromagnetic locator 20 and the computer three-dimensional tomography scanner 40 are respectively connected to the endoscope alignment device 10 for communication, the computer three-dimensional tomography scanner 40 (CT machine for short) is used to provide three-dimensional tomography images, such as scanning output of CT images containing chessboard calibration tools, and the electromagnetic locator 20 is used to construct an electromagnetic space and sense the positions of the first electromagnetic sensor 60 and the second electromagnetic sensor 80 in the electromagnetic space. Furthermore, each first electromagnetic sensor 60 can be placed on each first mounting member of the chessboard calibration tool 50, and the second electromagnetic sensor 80 can be placed on each first mounting member of the chessboard calibration tool 50. 80 can be fixed together with the endoscope 70 when the endoscope 70 is used to take pictures of the chessboard calibration tool 50 at multiple different angles.

参见图9，本申请实施例提供的另一种内窥镜标定系统，包括：内窥镜配准装置10和棋盘格标定工具50。其中，所述内窥镜配准装置10例如为图7所述的内窥镜配准装置10。所述棋盘格标定工具50例如图2和图3所示，包括印有已知尺寸的棋盘格的标定板51和多个在三维断层扫描影像中可识别的标识件54，所述标定板51上设有多个用于放置第一电磁传感器的第一安装件(例如容置孔55或夹具)，所述棋盘格、所述标识件54和所述第一安装件之间的相对位置已知。Referring to FIG9 , another endoscope calibration system provided in an embodiment of the present application includes: an endoscope alignment device 10 and a checkerboard calibration tool 50. The endoscope alignment device 10 is, for example, the endoscope alignment device 10 described in FIG7 . The checkerboard calibration tool 50, for example, as shown in FIG2 and FIG3 , includes a calibration plate 51 printed with a checkerboard of known size and a plurality of identification members 54 identifiable in a three-dimensional tomographic image, the calibration plate 51 is provided with a plurality of first mounting members (such as accommodating holes 55 or clamps) for placing a first electromagnetic sensor, and the relative positions among the checkerboard, the identification member 54 and the first mounting member are known.

以上所述实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和/或方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元/模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多路单元或模块可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and/or methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units/modules is only a logical function division. There may be other division methods in actual implementation, such as multiple units or modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

所述作为分离部件说明的单元/模块可以是或者也可以不是物理上分开的，作为单元/模块显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多路网络单元上。可以根据实际的需要选择其中的部分或者全部单元/模块来实现本实施例方案的目的。The units/modules described as separate components may or may not be physically separate, and the components shown as units/modules may or may not be physical units, i.e., they may be located in one place or distributed over multiple network units. It is necessary to select some or all of the units/modules to achieve the purpose of the solution of this embodiment.

另外，在本申请各个实施例中的各功能单元/模块可以集成在一个处理单元/模块中，也可以是各个单元/模块单独物理存在，也可以两个或两个以上单元/模块集成在一个单元/模块中。上述集成的单元/模块既可以采用硬件的形式实现，也可以采用硬件加软件功能单元/模块的形式实现。In addition, each functional unit/module in each embodiment of the present application may be integrated into one processing unit/module, or each unit/module may exist physically separately, or two or more units/modules may be integrated into one unit/module. The above-mentioned integrated unit/module may be implemented in the form of hardware or in the form of hardware plus software functional units/modules.

上述以软件功能单元/模块的形式实现的集成的单元/模块，可以存储在一个计算机可读取存储介质中。上述软件功能单元存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)的一个或多个处理器执行本申请各个实施例所述方法的部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，简称ROM)、随机存取存储器(Random Access Memory，简称RAM)、磁碟或者光盘等各种可以存储程序代码的介质。The above-mentioned integrated unit/module implemented in the form of a software functional unit/module can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for enabling one or more processors of a computer device (which can be a personal computer, server, or network device, etc.) to execute some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

以上所述实施例仅表达了本申请的几种实施例，其描述较为具体和详细，但并不能因此而理解为对本申请专利申请保护范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请创造构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利申请的保护范围应以所附权利要求为准。 The above-described embodiments only express several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of protection of the present patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the inventive concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be subject to the attached claims.

Claims

An endoscope registration method, comprising:

Acquire a three-dimensional tomographic image including a checkerboard calibration tool, wherein the checkerboard calibration tool comprises a calibration plate printed with a checkerboard of known size and a plurality of identification members identifiable in the three-dimensional tomographic image, wherein the calibration plate is provided with a plurality of first mounting members for placing first electromagnetic sensors, and the relative positions among the checkerboard, the identification members and the first mounting members are known;

Obtaining the position of the chessboard in the three-dimensional tomography image according to the coordinates of the identification element in the three-dimensional tomography image;

Obtaining the position of the chessboard in the electromagnetic space and a first transformation relationship of the three-dimensional tomography image relative to the electromagnetic space according to the position of the first electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator, wherein the first electromagnetic sensor is placed on the first mounting member;

The endoscope is calibrated based on the results of taking pictures of the checkerboard calibration tool at multiple different angles using an endoscope fixed with a second electromagnetic sensor, to obtain the camera internal parameters of the endoscope and a second transformation relationship between the camera coordinate system of the endoscope and the coordinate system of the second electromagnetic sensor;

According to the first transformation relationship, the second transformation relationship, and the posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when taking pictures with the endoscope, the posture of the virtual camera corresponding to the endoscope in the three-dimensional tomographic image is obtained, the internal parameters of the virtual camera are set as the camera internal parameters of the endoscope, and the virtual field of view is obtained in combination with the position of the chessboard in the three-dimensional tomographic image;

Registering the real field of view with the virtual field of view to obtain a third transformation relationship between the virtual field of view and the real field of view, wherein the real field of view is an image obtained by taking a photo with the endoscope;

According to the first transformation relationship, the second transformation relationship, and the third transformation relationship, the transformation relationship of the position of the virtual camera in the three-dimensional tomography image relative to the position of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when the endoscope takes pictures is obtained.

The endoscopic registration method according to claim 1, wherein the three-dimensional tomographic image includes a surgical site of the patient.

The endoscope registration method according to claim 1, wherein obtaining the position of the chessboard in the three-dimensional tomographic image according to the coordinates of the multiple identification elements in the three-dimensional tomographic image specifically comprises:

Extracting the coordinates of the identification element in the three-dimensional tomographic image according to the three-dimensional tomographic image combined with threshold segmentation;

According to the coordinates of the identification element in the three-dimensional tomographic image and the size of each grid of the chessboard, the coordinates of multiple grid corner points of the chessboard in the three-dimensional tomographic image are obtained.

The endoscope registration method according to claim 1, wherein the position of the virtual camera corresponding to the endoscope in the three-dimensional tomographic image is obtained according to the first transformation relationship, the second transformation relationship, and the position of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when taking pictures with the endoscope, specifically comprising:

The position and posture of the virtual camera in the three-dimensional tomography image is calculated according to the following formula:

Wherein, _Ri represents the posture of the virtual camera in the three-dimensional tomography image, and _pi represents the position of the virtual camera in the three-dimensional tomography image, so as to jointly define the virtual camera. The position of the machine in the three-dimensional tomographic image; represents the first transformation relationship, and R _wi and T _wi represent the rotation matrix and translation vector of the three-dimensional tomography image relative to the electromagnetic space respectively; represents the second transformation relationship, R _tc , T _tc respectively represent the rotation matrix and translation vector of the camera coordinate system relative to the posture of the chessboard in the electromagnetic space, R _st , T _st respectively represent the rotation matrix and translation vector of the posture of the chessboard in the electromagnetic space relative to the coordinate system of the second electromagnetic sensor; R _s represents the posture of the second electromagnetic sensor in the electromagnetic space, and p _s represents the position of the second electromagnetic sensor in the electromagnetic space, so as to jointly define the posture of the second electromagnetic sensor in the electromagnetic space.

The endoscope registration method according to claim 4, wherein the real field of view and the virtual field of view are registered to obtain a third transformation relationship between the virtual field of view and the real field of view, specifically comprising:

Acquire multiple pairs of matching pixel points of at least one pair of the real field of view and the virtual field of view;

Determine a target transformation matrix of the virtual field of view relative to the real field of view based on epipolar geometry constraints and the plurality of pairs of matching pixels;

The epipolar geometry constraint condition includes the following formula:

Wherein, p ₁ and p ₂ represent a pair of matching pixel points of the real field of view and the virtual field of view respectively, K represents the camera intrinsic parameter of the endoscope, and R ₁₂ and T ₁₂ represent the rotation matrix and the translation vector in the target transformation matrix respectively.

The endoscope registration method according to claim 5, wherein the transformation relationship between the position and posture of the virtual camera in the three-dimensional tomography image relative to the position and posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when the endoscope takes a picture is obtained according to the first transformation relationship, the second transformation relationship, and the third transformation relationship, specifically includes:

Inverting the target transformation matrix to obtain a target inverse transformation matrix;

According to the first transformation relationship, the second transformation relationship, and the target inverse transformation matrix, the transformation relationship between the posture of the virtual camera in the three-dimensional tomography image and the posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when the endoscope takes pictures is obtained.

An endoscope alignment method as described in any one of claims 1 to 6, wherein the identification members are fixedly disposed at multiple corners of the calibration plate.

The endoscope alignment method as described in claim 7, wherein the first mounting member is a receiving hole or a clamp formed respectively at the multiple corners of the calibration plate.

An endoscope registration device, comprising: a processor and a memory connected to the processor; wherein the memory stores instructions executed by the processor, and when the instructions are executed by the processor, the endoscope registration method described below is implemented:

Acquire a three-dimensional tomographic image including a chessboard calibration tool, wherein the chessboard calibration tool comprises a calibration plate printed with a chessboard of known size and a plurality of identification members identifiable in the three-dimensional tomographic image, wherein the calibration plate is provided with a plurality of first mounting members for placing first electromagnetic sensors, and the relative positions among the chessboard, the identification members and the first mounting members are known;

The endoscope is calibrated based on the results of taking pictures of the chessboard calibration tool at multiple different angles using an endoscope fixed with a second electromagnetic sensor, and the endoscope is obtained. camera intrinsic parameters, and a second transformation relationship between the camera coordinate system of the endoscope and the coordinate system of the second electromagnetic sensor;

The endoscopic registration device as described in claim 9, wherein the three-dimensional tomographic image includes a surgical site of the patient.

The endoscope registration device according to claim 9, wherein obtaining the position of the chessboard in the three-dimensional tomographic image according to the coordinates of the plurality of identification elements in the three-dimensional tomographic image specifically comprises:

The endoscope registration device according to claim 9, wherein the position of the virtual camera corresponding to the endoscope in the three-dimensional tomographic image is obtained according to the first transformation relationship, the second transformation relationship, and the position of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when taking pictures with the endoscope, specifically including:

Wherein, _Ri represents the posture of the virtual camera in the three-dimensional tomographic image, and _pi represents the position of the virtual camera in the three-dimensional tomographic image, so as to jointly define the posture of the virtual camera in the three-dimensional tomographic image; represents the first transformation relationship, and R _wi and T _wi represent the rotation matrix and translation vector of the three-dimensional tomography image relative to the electromagnetic space respectively; represents the second transformation relationship, R _tc , T _tc respectively represent the rotation matrix and translation vector of the camera coordinate system relative to the posture of the chessboard in the electromagnetic space, R _st , T _st respectively represent the rotation matrix and translation vector of the posture of the chessboard in the electromagnetic space relative to the coordinate system of the second electromagnetic sensor; R _s represents the posture of the second electromagnetic sensor in the electromagnetic space, and p _s represents the position of the second electromagnetic sensor in the electromagnetic space, so as to jointly define the posture of the second electromagnetic sensor in the electromagnetic space.

The endoscope registration device according to claim 12, wherein the real field of view and the virtual field of view are registered to obtain a third transformation relationship between the virtual field of view and the real field of view, specifically comprising:

The epipolar geometry constraint condition includes the following formula:

The endoscope registration device according to claim 13, wherein the transformation relationship between the position and posture of the virtual camera in the three-dimensional tomography image relative to the position and posture of the second electromagnetic sensor in the electromagnetic space obtained from the electromagnetic locator when the endoscope takes a picture is obtained according to the first transformation relationship, the second transformation relationship, and the third transformation relationship, specifically includes:

An endoscope alignment device as described in any one of claims 9 to 14, wherein the identification members are respectively fixedly arranged at multiple corners of the calibration plate.

The endoscope alignment method as described in claim 15, wherein the first mounting member is a receiving hole or a clamp respectively formed at the multiple corners of the calibration plate.

An endoscope calibration system, comprising: an endoscope alignment device as described in claim 9 and a checkerboard calibration tool, wherein the checkerboard calibration tool comprises a calibration plate printed with a checkerboard of known size and a plurality of identification members identifiable in the three-dimensional tomographic image, the calibration plate is provided with a plurality of first mounting members for placing a first electromagnetic sensor, and the relative positions among the checkerboard, the identification members and the first mounting members are known.