CN100559397C - Method and system for affine registration of two-dimensional images during surgery and three-dimensional images before surgery - Google Patents

Abstract

The system and method for three-dimensional (3D) medical figure registration before the sequence that the invention discloses a kind of operation two dimension (2D) medical image that is used for making target signature and the operation of described target signature.The 3D rendering of target signature is converted into first skeletal graph.The 2D image of target signature is converted into second skeletal graph.Carry out the coarse alignment of the figure coupling of first and second skeletal graph, and make first and second skeletal graph registration with the acquisition figure.

Description

Method and system for affine registration of two-dimensional images during surgery and three-dimensional images before surgery

Cross References to Related Applications

This application claims the benefit of US Provisional Application Serial No. 60/537,820, filed January 21, 2004, which is incorporated by reference in its entirety.

technical field

The present invention relates to methods and systems for registering intra-operative images with pre-operative images, and more particularly to feature-based methods and systems for registering intra-operative two-dimensional images with pre-operative three-dimensional images.

Background technique

Imaging of human tissues and organs is an important tool used to aid in the diagnosis and treatment of many medical conditions. Imaging modalities such as magnetic resonance imaging (MRI) and computed tomography (CT) produce high-quality three-dimensional (3D) images. These imaging modalities are typically used to image patients prior to surgery.

Indeed, in breast, prostate, and brain tumor surgery, due to the ability to perform dynamic imaging, parametric modeling, and diffusion or other functional MR or CT imaging with impractical acquisition times for interactive intraoperative imaging method, preoperative imaging better delineates tumor extent. Two-dimensional (2D) fluorescence images are typically acquired during surgery. Although these images are useful for providing real-time monitoring of the interventional device, they do not have the image quality and tissue contrast of closed magnet MR and CT.

Interventional fluorescence imaging or intraoperative fluorescence imaging is used to guide instruments for diagnostic or minimally invasive therapeutic interventions. Interventional and surgical procedures require that updates regarding patient anatomy or changing positions of moving organs are available to physicians. Live imaging during the intervention (without registration) establishes the necessary relationship between patient and image. The lower image quality of fluorescence images prevents their use in various procedures. There is a need for a registration process that utilizes high quality pre-operative volumes/images from conventional high-field magnet MRI systems or CT systems to augment fluoroscopic intra-operative images (volumes are considered as three-dimensional images and are hereinafter referred to as image).

Contents of the invention

The present invention relates to a system and method for registering a sequence of intraoperative two-dimensional (2D) medical images of a target feature with preoperative three-dimensional (3D) medical images of the target feature. The 3D image of the target feature is converted into a first skeletal graphic. The 2D image of the target feature is converted into a second skeletal graphic. Graphic matching of the first and second skeletal graphics is performed to obtain a coarse alignment of the graphics, and the first and second skeletal graphics are registered.

Description of drawings

Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings, wherein like reference numerals indicate like elements:

1 is a schematic diagram of a system architecture of a typical magnetic resonance imaging (MRI) system according to the present invention;

2 is a schematic diagram of a typical C-arm gantry used to obtain two-dimensional intraoperative images according to the present invention;

3 is a schematic block diagram illustrating a system for registering a three-dimensional pre-operative image with a two-dimensional intra-operative image according to the present invention;

Figures 4a-b are flowcharts illustrating a process for registering pre-operative and intra-operative images according to the present invention;

Figure 5 is a representation of a skeletal graphic and corresponding 3D graphic of an object feature according to the present invention;

Figure 6 is a representation of how contrast agent flow is monitored in a series of vessels over a first time period in accordance with the present invention; and

Fig. 7 is a representation of how contrast medium flow is monitored in a series of vessels over a second time period in accordance with the present invention.

Detailed ways

The present invention relates to a method for registering preoperative three-dimensional images with intraoperative two-dimensional images to help guide interventional procedures. Different modalities, such as magnetic resonance imaging (MRI) or computed tomography, can be used to obtain three-dimensional images. Examples of such devices that can be used to obtain images are MAGNETOM Symphony and SONATOM Sensation, both manufactured by Siemens AG.

Fig. 1 shows a diagram of the components of a typical MRI system that can be used to obtain high-quality preoperative 3D images according to the present invention. An MRI system is located in scanning room 100 . Magnet 108 generates a first magnetic field for the imaging process. Gradient coils 110 are within the magnet 108 for generating gradients in the magnetic field in the X, Y and Z directions. A radio frequency (RF) coil 112 is within the gradient coil 110 . The RF coil 112 generates the second magnetic field necessary to rotate the spins by 90° or 180°. The RF coil 112 also detects signals from spins within the body.

A patient 102 is positioned within a magnet 108 by a computer-controlled patient table 106 . The treatment table 106 has a positioning accuracy of 1 mm. Scanning chamber 100 is surrounded by RF shielding 104 . Shielding 104 prevents high power RF pulses from radiating outside the hospital. It also prevents various RF signals from TV and radio stations from being detected by the MRI system. Some scan rooms are also surrounded by magnetic shields containing magnetic fields that extend too far into the hospital. In newer magnets, the magnetic shield is the main part of the magnet.

The central element of the MRI system is the computer 126 . Computer 126 controls all components on the MRI system. The RF components under the control of computer 126 are radio frequency source 138 and pulse programmer 134 . The radio frequency source 138 generates a sine wave of the desired frequency. The pulse programmer 134 shapes the RF pulses into apodized sinc pulses. The RF amplifier 136 increases the pulse power from milliwatts to kilowatts. Computer 126 also controls gradient pulse programmer 122, which sets the shape and amplitude of each of the three gradient fields. The gradient amplifier 120 increases the power of the gradient pulse to a level sufficient to drive the gradient coil 110 .

An operator of the MRI system provides input to computer 126 via console 128 . Imaging sequences are selected and customized from the console 128 . An operator may view the image on a video display located on console 128 or may make a hard copy of the image on a film printer (not shown).

According to the present invention, the patient will be imaged by a high modality imaging system, such as the described MRI system, prior to surgery. Additional lower quality 2D images are acquired during surgery. Typically, an x-ray system is used to acquire these images. An example of such a system is AXIOM Artis MP, manufactured by Siemens AG.

Fig. 2 is a schematic diagram of a fluorescent X-ray imaging system. C-arm 212 carries X-ray source 208 and image intensifier 210 in orbit (eg, a circular path) around the patient. As known to those skilled in the art, an x-ray image is a radiographic projection of an object onto a plane. It can be viewed as a two-dimensional function of x and y, so that it records at each point (x, y) all absorptions along rays radiated from the X-ray source 208 to the point (x, y) on the image plane total amount. 3D angiographic data are acquired by rotating the C-arm gantry of the X-ray unit about its isocenter with the patient present.

3 is a block diagram of an exemplary system for registering preoperative 3D images with intraoperative 2D images according to the present invention. The present invention utilizes pre-operative and intra-operative images to provide more useful and inexpensive registered images of target features, such as blood vessels within specific tissue regions, that are the subject of minimally invasive therapeutic interventions. For example, tumors can be imaged using a closed MRI system or CT system before surgery and a fluoroscopy system during surgery. Images are registered and stitched to provide structural and functional information about the tumor and affected tissue regions. Subsequent images taken during surgery using the fluoroscopy system can then be stitched over time with pre-operative images to assist doctors. Where deformation is detected intra-operatively, the invention can also be used to modify the pre-operative image to mimic the deformation prior to registration with the intra-operative image.

Three-dimensional (3D) images of the desired tissue region or organ are obtained by using a closed MRI system (Figure 1) or a CT system (not shown) such as the 1.5T MAGNETOM Sonata scanner commercially available from Siemens Medical Solutions . Data is collected from images of tissue regions or organs and stored for further processing by processor 304 . In particular, target features, such as a portion of a blood vessel associated with a particular tissue region, are segmented from the image and a skeleton graph is stored, as will be described in more detail below. This image is acquired prior to any surgical procedure. Other organs or internal structures can also be imaged if desired.

A two-dimensional (2D) image of the same desired target feature is then acquired by using the fluoroscopy system 302 . During the surgical procedure, initial images are acquired and stored in processor 304 . Rigorous registration of the 2D images from the fluoroscopy system with the preoperative 3D images from the closed MRI was performed. Preferably, the 3D pre-operative image and the 2D intra-operative image are in a relatively similar state. For example, an internal organ being imaged should be in approximately the same state (eg, position and perspective) for both imaging sessions to ensure proper registration.

As described above, 3D image data and 2D image data are input to the processor 304 . Processor 304 may include a graphical user interface (GUI) that allows a user to manually draw a boundary or outline around a region of interest in an image. Alternatively, segmentation algorithms can be used to distinguish regions of interest and outline images without user interaction. Segmentation algorithms known to those skilled in the art can be used. Processor 304 includes a database 306 that stores images.

A display 310 is included for displaying images as well as displaying registered images. Also included is an interface device or device 308, such as a keyboard, mouse, or other device known in the art.

4a-4b are flowcharts illustrating a method for registering a pre-operative 3D image with an intra-operative 2D image according to the present invention. The patient undergoes an MRI or CT scan prior to the surgical procedure to obtain a preoperative 3D image of the target feature (step 402). 3D images from scans are stored for later use. According to the invention, the target feature is a blood vessel. By looking at blood vessels and surrounding tissue, various conditions can be diagnosed. For example, such methods can be used to detect cancer cells in the liver, brain conditions, and heart conditions.

Object features are segmented from the 3D image (step 404). Segmenting blood vessels is relatively easy since a contrast agent is usually injected into the target area during an interventional procedure. The flow of contrast agent through blood vessels is also used to extract 3D blood vessels from the fluorescence image sequence, as will be described in further detail below. The 3D image provides a high resolution view of the desired area and allows the user to observe the soft tissue in the area and enable depth measurements. Next, a centerline is extracted from the segmented 3D image (step 406). According to the invention, a parallel thinning algorithm for 3D object thinning is used to identify centerlines. An example of such an algorithm is described in the article "A Parallel Thinning Algorithm or 3-DPictures" by Tsao and Fu (Computer Graphics and Image Processing, 17:315-331, 1981), which is incorporated by reference in its entirety. Extracting the centerline provides a set of voxels located on the median axis of the vessel. Test connectivity for identifying vessel segments and branches. For example, a voxel that is part of a blood vessel segment typically has only two neighbors, while a branch voxel usually has many more neighbors, as follows:

xx xxxxxx

xxxxxo

xxxxxxxx

where o is a branch voxel and x is an adjacent voxel.

The 3D skeleton tree is obtained according to the thinning algorithm and stored in the database 306 as a 3D graph (step 408). An example of such a representation is shown in FIG. 5 . A skeletal graph is generated by representing each vessel segment as a node in the skeletal graph. Additional information about vessel segments is also stored in database 306 . Such information may include, as node attributes, the length and diameter of the vessel segment. Connectivity information between vessel segments is represented as edges in the skeleton graph. Due to the inherent structure of blood vessels, skeleton graphics are always in tree format. More particularly, the graph is typically a finite rooted tree. The skeletal graph is converted to a directed Acyclic graph.

Next, the patient is scanned using the 2D fluoroscopy system at the beginning of the procedure (step 410). Preferably, the patient is positioned at substantially the same location as when the patient was scanned by the 3D imaging system prior to surgery and the angle and perspective of the scan is similar. Images from 2D scans are also stored.

Object features are then segmented from the 2D image (step 412). This is achieved by subtracting features from the zero-contrast image. Once the target features have been segmented, it is desirable to obtain a 3D representation of the 2D image. However, this is not an easy task. Since 2D images do not provide any indication of object depth in the image, it is difficult to tell whether overlapping objects are connected. For example, where the target feature is a series of vessels, it is not possible to discern in the 2D image whether certain vessel segments overlap and thus are separate blood flows, or whether overlapping segments indicate bifurcations of connected vessels.

To make this discrimination, the flow of contrast agent through the blood vessel is studied for a given period of time (step 414). Contrast agent flow through blood vessels can be achieved by using a method similar to that used for matching rooted directed acyclic graphs. Such a method is described in "Skeleton Based Shape Matching and Retrieval" by H. Sundar et al. (the Proceedings, Shape Modeling and Application Conference, SMI 2003), which is incorporated by reference in its entirety. A map of the blood vessel is generated by recording the arrival and departure times of the flow of contrast medium from each intersection or branch point of the blood vessel. By monitoring this flow, depth determinations can be made.

In accordance with the present invention, the arrival time and departure time of the contrast flow from each junction or branch point are recorded (step 416). Figure 6 is an illustration of how the flow of contrast media through a blood vessel is tracked. At a first time point, the onset of the contrast agent is determined. Next, the flow of contrast agent through the vessel is tracked at predetermined time intervals. This monitoring is repeated at predetermined time intervals so that the flow of contrast medium can be analyzed to determine the structure of the blood vessel. Image 602 shows a region of tissue containing blood vessels in the absence of contrast agent. Image 604 shows the same region of tissue where a contrast agent has been injected into a blood vessel and monitored over a predetermined period of time.

As contrast agent flows through a particular segment of a vessel, the vessel is highlighted to indicate whether the particular segment is connected to an adjacent segment or whether the two segments overlap. Although FIG. 6 shows blood vessels in varying gray scale brightness to illustrate this relationship, those skilled in the art will understand that other representations, such as different color schemes, may be used. Each branch of blood vessels 606-614 has a different brightness.

FIG. 7 shows the same tissue region 602 and an image 704 showing the contrast agent flow at a later point in time than image 604 . From this image it can be seen that blood vessels 606 and 608 overlap and are not connected. Such a determination aids in determining depth measurements in tissue regions and in registering the 2D fluorescence image with the 3D image.

Once the contrast agent flow through the vessel has been analyzed, a 3D graph is generated from the 2D skeletal tree (step 418). The 3D graph shows the structure of vessels in the tissue region and clarifies ambiguities regarding overlapping vessels and false intersections. Next, graph matching of the 3D graph is performed by coarsely aligning the skeleton trees from the 2D and 3D images (step 420). Methods commonly used for matching rooted directed acyclic graphs can be used to implement this step. An example of such a method is described in "Skeleton Based Shape Matching and Retrieval" by H. Sundar et al., supra.

Once the coarse alignment is obtained, the iterative closest point (ICP) algorithm is used to refine the registration of the skeletal trees (step 422). Single plane fluorescence images were used to perform 2D-3D registration. Graph matching not only provides coarse alignment; it also provides vessel consistency between 2D and 3D skeletons. The consistency is used as a constraint when applying the ICP algorithm to the centerline (set of points). This helps to avoid local minima during the registration process making the algorithm very robust.

Additionally, the entire sequence of 2D images acquired during monitoring of the contrast flow is used during 2D-3D registration. This information is mainly used to avoid ambiguities in monoplane projection of 3D images. The registration parameters are optimized by using the sum of the squared differences of all corresponding points in the 2D features and the projections of the 3D features (step 424). The optimization is performed on six parameters, namely three translation and three rotation parameters.

While an embodiment of a method for registering a 3D pre-operative image with a 2D intra-operative image has been described, it should be noted that modifications and variations may be made by those skilled in the art in light of the above teachings. For example, the present invention mainly relates to strict registration of images, however deformable registration can also be performed. In such cases, the optimization step will include parameters for controlling the deformation. Once a preliminary registration between 2D and 3D images is achieved, the registration can be maintained by tracking features in the 2D images. Any known motion tracking algorithm can be used for this purpose.

Shape models can also be used by obtaining them from 3D images or from previous 2D image sequences. This shape model can be used to guide skeleton extraction and for optimization steps.

It is therefore to be understood that changes may be made in the particular embodiments of the invention disclosed which are within the scope and spirit of the invention as defined by the appended claims. Having thus described the invention with the details and particularity required by the patent laws, what is required and desired protected by Letters Patent is set forth in the appended claims.

Claims (24)

1. the method for three-dimensional medical figure registration before the operation of the sequence of an operation two-dimensional medical images that is used for making target signature and described target signature, this method may further comprise the steps:

The 3-D view of described target signature is converted to first skeletal graph;

By from the two dimensional image of described target signature, cutting apart described target signature and the contrast agent flow of research by described target signature, the two dimensional image of described target signature is converted to second skeletal graph;

The figure coupling of carrying out described first and second skeletal graph is to obtain the coarse alignment of figure; And

Make described first and second skeletal graph registration.

2. according to the process of claim 1 wherein that described target signature is the blood vessel in the particular organization zone.

3. according to the process of claim 1 wherein that the step that 3-D view with described target signature is converted to first skeletal graph further may further comprise the steps:

From described 3-D view, cut apart described target signature; And

From the 3-D view of being cut apart, extract center line.

4. according to the method for claim 3, wherein use the parallel thinning algorithm to extract described center line.

5. according to the process of claim 1 wherein that the step of research contrast agent flow further may further comprise the steps:

Be recorded in the time of arrival and the time departure of each place, point of crossing contrast agent flow of described target signature;

Determine that based on described time of arrival and time departure described point of crossing is connected point of crossing or overlapping point of crossing.

6. according to the process of claim 1 wherein that the step of carrying out the figure coupling comprises further using that the root directed acyclic graph is arranged.

7. according to the process of claim 1 wherein the step of described first and second skeletal graph registration further be may further comprise the steps:

Use the iterative closest point algorithms registration of refining.

8. according to the process of claim 1 wherein by following the tracks of the registration that described target signature in the described two dimensional image keeps described first and second skeletal graph.

9. according to the process of claim 1 wherein that the monoplane fluoroscopic image is used to carry out the registration of described two dimension and 3-D view.

10. according to the process of claim 1 wherein that the sequence of two dimensional image is continuous in time.

11. according to the process of claim 1 wherein that described target signature is relevant with the human organ.

12. according to the method for claim 11, wherein said organ is a liver.

13. the system of the sequence of an operation two dimensional image that is used for making target signature and the operation forward three-dimensional viewing registration of described target signature, described system comprises:

Be used to make the two-dimensional imaging system of target signature imaging;

Be used to store the database of the 3-D view of described target signature;

Be used to receive and handle the processor of described two dimensional image, this processor is carried out following steps:

I). the 3-D view of described target signature is converted to first skeletal graph;

Ii). by from described two dimensional image, cutting apart described target signature and the contrast agent flow of research by described target signature, the two dimensional image of described target signature is converted to second skeletal graph;

Iii). the figure coupling of carrying out described first and second skeletal graph is to obtain the coarse alignment of figure; And

Iv). make described first and second skeletal graph registration; And

Be used to show the display of registering images.

14. according to the system of claim 13, wherein said target signature is the blood vessel in the particular organization zone.

15. according to the system of claim 13, the step that wherein 3-D view of described target signature is converted to first skeletal graph further may further comprise the steps:

From described 3-D view, cut apart described target signature; And

From the 3-D view of being cut apart, extract center line.

16., wherein use the parallel thinning algorithm to extract described center line according to the system of claim 15.

17. according to the system of claim 13, the step of wherein studying contrast agent flow further may further comprise the steps:

Be recorded in the time of arrival and the time departure of each place, point of crossing contrast agent flow of described target signature;

Determine that based on described time of arrival and time departure described point of crossing is connected point of crossing or overlapping point of crossing.

18. according to the system of claim 13, the step of wherein carrying out the figure coupling further comprises using the root directed acyclic graph.

19., the step of described first and second skeletal graph registration further be may further comprise the steps according to the system of claim 13:

Use the iterative closest point algorithms registration of refining.

20. according to the system of claim 13, wherein by following the tracks of the registration that described target signature in the described two dimensional image keeps described first and second skeletal graph.

21. according to the system of claim 13, wherein the monoplane fluoroscopic image is used to carry out the registration of described two dimension and 3-D view.

22. according to the system of claim 13, wherein the sequence of two dimensional image is continuous in time.

23. according to the system of claim 13, wherein said target signature is relevant with the human organ.

24. according to the system of claim 23, wherein said organ is a liver.

Images