WO2024169341A1 - Registration method for multimodality image-guided radiotherapy - Google Patents

Abstract

Disclosed in the present invention is a registration method for multimodality image-guided radiotherapy. The method comprises: acquiring multimodality images to be registered; performing point cloud processing on a target organ in each image to be registered, and using a deformation relationship learned from non-rigid registration to perform organ anatomical structure point clouds-based registration on the images to be registered, so as to obtain preliminarily deformed images to be registered; inputting the preliminarily deformed images to be registered into a trained deep convolutional network to obtain a registered image, the deep convolutional network reflecting the deformation relationship between the multimodality images to be registered and a template image; and formulating a radiotherapy plan by using the registered image. The present invention improves the accuracy of image registration and medical interpretability.

Description

A registration method for multimodal image-guided radiotherapy Technical Field

The present invention relates to the technical field of medical image processing, and more specifically, to a registration method for multimodal image-guided radiotherapy.

Background Art

Image-guided radiotherapy is a personalized radiotherapy method that is performed under the guidance of medical images of the human body. This method can show the different sizes, shapes and locations of cancer lesions. Doctors can develop a radiotherapy plan based on this, which can accurately irradiate the cancer lesions while ensuring that the surrounding healthy tissues receive low doses of radiation as much as possible to avoid damaging healthy tissues.

Medical images of different modalities can display different information about the human body from microscopic to macroscopic scales and from the inside to the outside. Using multimodal medical image registration technology to match medical images of different modalities to the same spatial and temporal scales can help doctors accurately delineate the location of the patient's lesions and develop more accurate radiotherapy plans. At present, this technology has overcome the problems of medical image information being unable to match and time and space information being difficult to calibrate to a certain extent, but traditional methods rely too much on manually designed feature selectors and feature matchers, and manually designed feature matchers are not robust enough in multimodal image registration, and feature selectors and feature matchers that perform well on single-modality images are usually not adaptable to images of multiple modalities.

After analysis, the existing technology mainly has the following defects:

1) The preparation of radiotherapy plans takes a long time. During the radiotherapy operation, the radiologist cannot make real-time changes to the radiotherapy operation plan based on the patient's movement.

2) The boundaries and specific details of cancer lesions are difficult to define on a single modality image, and if multiple modalities of images are integrated to formulate radiotherapy plans, it requires a high level of doctor experience.

3) The existing multimodal image registration methods based on deep learning have problems such as high distortion and poor interpretability.

Summary of the invention

The purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a registration method for multimodal image-guided radiotherapy. The method comprises the following steps:

Acquire multimodal images to be registered;

Performing point cloud processing on the target organ of the image to be registered, and registering the image to be registered based on the organ anatomical structure point cloud using the deformation relationship learned by non-rigid registration to obtain a preliminarily deformed image to be registered;

Inputting the preliminarily deformed image to be registered into a trained deep convolutional network to obtain a registered image, wherein the deep convolutional network reflects the deformation relationship between the multimodal image to be registered and the template image;

A radiation therapy plan is formulated using the registered images.

Compared with the prior art, the advantages of the present invention are that it proposes an image registration method based on deformation vector field constraints and attention mechanism of anatomical point cloud, which can automatically extract features and match them in deep space, that is, it realizes fully automatic registration of multimodal images and end-to-end optimization. In addition, the present invention combines the anatomical structure point cloud of the organ itself to constrain the image registration in the multimodal image-guided radiotherapy system, thereby improving the accuracy of image registration and medical interpretability.

Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

FIG1 is a flow chart of a registration method for multi-modality image-guided radiotherapy according to an embodiment of the present invention;

FIG2 is a schematic diagram of a process of a registration method for multi-modality image-guided radiotherapy according to an embodiment of the present invention;

FIG3 is a schematic diagram of the architecture of a deep convolutional network based on a self-attention mechanism according to an embodiment of the present invention;

FIG4 is a schematic diagram of two connected Swin-Transformer blocks according to one embodiment of the present invention;

FIG. 5 is a schematic diagram of registration effects under three evaluation indicators according to an embodiment of the present invention.

DETAILED DESCRIPTION

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

The present invention provides a multimodal image registration method based on deformation vector field constraints and attention mechanism of anatomical point cloud. The method first uses a public multimodal image-guided radiotherapy image data set for training. The designed training network is, for example, a deep convolutional network based on a self-attention mechanism. The image features are extracted in combination with the attention mechanism, and the image is feature matched by a feature matcher. The image deformation vector field is generated by feature matching, thereby obtaining a deformation vector field that is closer to the real situation and achieving a more accurate registration effect.

Specifically, in combination with FIG. 1 and FIG. 2, the provided multimodal image-guided radiotherapy registration method includes: The following steps are included.

Step S110 , collecting data of medical image-guided radiotherapy of different modalities.

Medical images of different modalities include magnetic resonance imaging, ultrasound imaging, computed tomography, cone beam computed tomography or other types of medical images. Data of medical image-guided radiotherapy of different modalities can be obtained from public datasets or collected by the user.

Step S120 , preprocessing the multimodal images to add labels of different target organs.

In one embodiment, step S120 includes:

Step S121: normalizing the size of the multimodal image data, including voxel coordinate direction, distance between voxels, etc.

Step S122: label the organs and add identifiable organ marker labels.

Step S130 , converting the target organ label into a point cloud, and based on the point cloud, using a non-rigid registration method to perform a coarse registration based on the organ anatomical structure point cloud.

In one embodiment, step S130 includes:

Step S131: Use the marching cubes method to construct the human organ anatomical structure point cloud and the target area point cloud.

For example, assuming that the value of any intersection point between a three-dimensional plane and a hexahedral voxel is expressed as f( _xi , _yj , _zk ), the central difference method is used to calculate the gradient _Gx , _Gy , _Gz of the voxel vertex where the data point is located, where a, b, and c represent the distances between two adjacent hexahedrons, respectively.

Step S132: After respectively calculating the gradients G _x , G _y , and G _z of the eight vertices of the voxel using the marching cubes method, the gradients of the vertices of the central triangle patch are calculated using linear interpolation to draw the isosurface.

Step S133: extracting the coordinates of the isosurface vertices and defining them as a point cloud, and using a non-rigid registration method to perform registration based on the organ anatomical structure point cloud on the defined target point cloud.

For example, suppose the template is represented as Where V represents the vertex, there are m; is an edge, there are n of them, the point cloud to be deformed is M = (v, e), where v represents m movable vertices; e represents n edges, and X represents the deformation vector field. Deform the point cloud to be deformed and make it meet the following conditions as much as possible:
minE(X)＝min{L _d (X)+αL _s (X)+βL _l (X)} (4)

Among them, L _d (X) is the distance term, L _s (X) is the rigid term, L _l (X) is the key point term, α, β are weights, and the goal is to minimize the function and find the deformation vector field that meets the conditions.

In one embodiment, the distance term L _d (X) is expressed as:

_vi = [x, y, z, 1] ^T (6)

Among them, _wi is the weight, _vi is the i-th point in the point cloud M to be deformed, dist(S,v) represents the distance between point _vi and the nearest point on the template image, _Xi is a transformation matrix of size 3×4, and x, y, and z represent the coordinate values of point _vi respectively.

In one embodiment, the stiffness term _Ls (X) is expressed as:

Use rigid constraints on the same edge The affine transformation of the two points on is penalized, where G = diag(1,1,1,γ), γ can be used to weight the difference between the rotation and tilt parts of the deformation and the translation part of the deformation, and γ can be preset based on experience or simulation.

In one embodiment, the key point term L _l (X) is expressed as:

Where l represents the feature point in the template image (or reference image), Represents a given feature point set. The last term of the loss function is the constraint of the feature point. If a feature point set is given This loss can take into account the distance between the feature points to be registered and the feature points of the template image. If a given feature point does not exist, the function will cover all feature points.

Specifically, the registration process based on the organ anatomical structure point cloud using the non-rigid registration method includes the following steps:

1) Initialize X ⁰

In the first layer loop, the parameters α and β are updated.

2) For each iteration, the rigid term weight is α ⁱ⁺¹ <α ⁱ , and the key point weight is β ⁱ⁺¹ <β ⁱ .

Keep looping: until ||X ^j+1 -X ^j ||<ε, where ε is the set threshold.

The second loop is to solve the parameter X, including:

(1) Find preliminary correspondences through the nearest point finding algorithm;

(2) Use the downhill method or other gradient-based methods to solve ^Xj , and use ^Xj as the preliminary correspondence and the optimal deformation vector field of ^αi .

In this step S130, the template image and the image to be registered are initially deformed. The multimodal images are roughly registered using the human organ anatomical structure point cloud method, which provides the interpretability of the image registration task in image-guided radiotherapy. And the roughly registered deformation vector field is interpolated using linear interpolation to obtain a deformation vector field that is consistent with the original image size.

Step S140, using a deep convolutional network, performing deep learning-based precise registration on the obtained multimodal images with the same label to obtain a final registered image.

FIG3 is an architecture diagram of a deep convolutional network as a registration network, which is used to further obtain the deformation field between the image to be registered and the template image. The deep convolutional network as a whole includes an encoder and a decoder, wherein the encoder is a self-attention encoder. This cascaded approach connects the point cloud registration of step S130 and the registration method based on a deep neural network, further improving the accuracy of image registration. In addition, a convolutional network based on a self-attention mechanism is used for fine image registration in multimodal image-guided therapy. It should be noted that other types of deep convolutional network architectures may also be used.

In one embodiment, step S140 includes:

Step S141: using a convolutional image feature extraction model based on a self-attention mechanism, inputting the template image preliminarily deformed in step S130 and the image to be registered into a self-attention encoder for feature encoding;

Step S142: input the fused features into the deep convolution layer for decoding prediction, perform prediction after fusing features of different scales, and output a predicted image with the same size as the original image;

Step S143: Compare the predicted result with the original image and back-propagate it to the self-attention encoder to satisfy the following conditions:

Where p represents the position of the voxel in the image, _Im and f represent the image to be registered and the template image, and and represents the average voxel value of the adjacent voxels of the image to be registered and the template image with _pi as the center point, φ represents the deformation vector field from the image to be registered to the template image, _pi represents the central voxel of the current operation, and R represents the image domain.

In order to further understand the fine registration process based on deep convolutional networks, as shown in Figure 3, it mainly includes the following steps:

Step S31: Segment the image to be registered and the template image obtained in S130 after point cloud guided deformation into non-overlapping voxel blocks (patches).

For example, the original image size is 160×192×160, and the final size of each voxel block is 2×4×4×4.

Step S32: Flatten each voxel block and input it into a linear mapping partition encoder for position encoding.

Step S33: Input the encoded token into the Swin-Transformer encoder based on the self-attention mechanism. After passing through the encoder, the adjacent 2×2×2 tokens will be merged, the number of tokens will be reduced by 8 times, and the feature dimension will be expanded by 8 times.

Step S34: Input the expanded feature terms into the linear layer and output the features of every two dimensions.

Step S35: After four stages of Swin-Transformer encoders and three partition splicings, the encoder finally outputs 5×6×5×8.

Step S36: The decoder includes an upsampling layer and a convolution layer with a convolution kernel of 3×3. During the decoding process, the feature map in each upsampling layer will be spliced with the corresponding features in the encoding process through jump links, and two three-dimensional convolution layers will be connected after upsampling.

Step S37: After a three-dimensional convolution, the original image to be registered, the template image and the image after a downsampling are spliced with the feature map obtained in step S36 to integrate the global position information. The spliced features are convolved 16 times with a convolution kernel of 3×3 and the deformation vector field is output.

Figure 4 is a schematic diagram of two connected Swin-Transformer blocks. Each Swin-Transformer block contains a layer normalization layer, a window-based multi-head attention, and a multi-layer perceptron. The specific processing process includes the following steps:

Step S41: The input features are normalized by layers and then input into the window-based multi-head self-attention module.

Step S42: Add the result obtained in step S41 to the original feature, perform layer normalization, pass through a multi-layer perceptron, and then add it to the result of step S41.

Step S43: After the result obtained in step S42 is layer-normalized, it is input into the multi-head self-attention module based on the sliding window, and the result is added to the result obtained in step S42.

Step S44: After the result obtained in step S43 is layer-normalized, it is input into a multilayer perceptron, and the result is added to the result obtained in step S43.

It should be understood that the above is a training process described in a specific embodiment, and after the training of the deep convolutional network is completed, it can be used for actual medical image registration. For example, the actual application process includes: obtaining multi-modal images to be registered; performing point cloud processing on the target organ of the image to be registered, and using the deformation relationship learned by non-rigid registration to register the image to be registered based on the organ anatomical structure point cloud to obtain a preliminary deformed image to be registered; inputting the preliminary deformed image to be registered into the trained deep convolutional network to obtain a registered image, and the deep convolutional network reflects the deformation relationship between the multi-modal image to be registered and the template image; and using the registered image to formulate a radiotherapy plan.

To further verify the present invention, the data set was verified on the image-guided breast cancer radiotherapy dataset, with a total of 14 cases. The data of two patients were used as the test data set, and the rest of the data were used as the training set. The results of the comparison between the traditional convolutional neural network registration method and the linear interpolation image registration method are shown in Figure 5, where for each indicator, the rightmost figure corresponds to the present invention. It can be seen that the deep convolutional network method based on the human anatomical structure point cloud pre-registration cascade self-attention mechanism proposed in the present invention can effectively realize image-guided breast cancer radiotherapy registration, and has higher accuracy and signal-to-noise ratio than the existing methods.

In summary, compared with the prior art, the present invention has the following advantages:

1) By building a deep convolutional network model based on the self-attention mechanism to register medical images, and combining the guidance of the human anatomical structure point cloud, the fully automatic registration of multimodal images is realized. From a technical perspective, the present invention takes into account the application of human anatomical structure in multimodal image registration by clinicians, takes into account the guiding role of anatomical key points in the registration problem, improves the interpretability of the registration results, and improves the accuracy of the registration task.

2) A cascade registration method was used. The front end used the anatomical information of human organs for rough registration, and the back end used a deep convolutional neural network based on the self-attention mechanism for fine registration, which improved the accuracy of multimodal image-guided radiotherapy. This method of using anatomical point cloud and actual image multimodal fusion makes full use of the anatomical information of human organs and reduces the dependence of deep network models on the amount of clinical data.

3) Using a deep convolutional network based on anatomical point cloud deformation vector field constraints and attention mechanism for multi-modal The multimodal images are registered to solve the problems of insufficient cross-modal image feature extraction and low accuracy in multimodal image-guided radiotherapy. Traditional registration methods generally use iterative algorithms, which consume a lot of running time. The present invention adopts a one-step registration method to ultimately achieve fully automatic real-time registration in multimodal image-guided radiotherapy.

The present invention may be a system, a method and/or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing/processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device.

The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

Reference is made herein to flowcharts and illustrations of methods, apparatus (systems) and computer program products according to embodiments of the present invention. The flowchart and/or block diagrams describe various aspects of the present invention. It should be understood that each block of the flowchart and/or block diagram and the combination of each block in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions/actions specified in one or more boxes in the flowchart and/or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

The flowchart and block diagram in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and/or flowchart, and the combination of the boxes in the block diagram and/or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims (10)

A registration method for multimodal image-guided radiotherapy comprises the following steps: Acquire multimodal images to be registered; The target organ of the image to be registered is processed into a point cloud, and the deformation relationship learned by non-rigid registration is used to register the image to be registered based on the organ anatomical structure point cloud to obtain a preliminarily deformed image to be registered; Inputting the preliminarily deformed image to be registered into a trained deep convolutional network to obtain a registered image, wherein the deep convolutional network reflects the deformation relationship between the multimodal image to be registered and the template image; A radiation therapy plan is formulated using the registered images. The method according to claim 1, characterized in that the deformation relationship learned by the non-rigid registration is obtained using the following optimization objectives:
minE(X)＝min{L _d (X)+αL _s (X)+βL _l (X)} Among them, X represents the deformation vector field, L _d (X) is the distance term, L _s (X) is the rigidity term, L _l (X) is the key point term, and α and β are the weights of the corresponding terms. The method according to claim 2, characterized in that the distance term is expressed as:

_vi = [x, y, z, 1] ^T Where, _wi is the weight, _vi is the i-th point in the point cloud M to be deformed, dist(S,v) represents the distance between point _vi and the nearest point on the template image S, and _Xi is a transformation matrix of size 3×4; The rigid term is expressed as:
Among them, G = diag (1, 1, 1, γ), γ is the set weight value; The key point item is expressed as:
Among them, l represents the feature point in the template image, Represents a given set of feature points. The method according to claim 1 is characterized in that the deep convolutional network includes an encoder and a decoder, the encoder is a Swin-Transformer encoder, the decoder includes an upsampling layer and a convolution layer, and in the decoding process, the feature map in each upsampling layer is spliced with the corresponding feature in the encoding process through a jump link, and the two three-dimensional convolutional layers are connected after the upsampling in the encoding process. The method according to claim 4, characterized in that training the deep convolutional network comprises the following steps: Inputting the template image that has undergone preliminary deformation and the image to be registered into the encoder for feature encoding; After fusing the features extracted by the encoder, the fused features are decoded by the decoder, and prediction is performed after fusing features of different scales, and a predicted image having the same size as the original image is output; The predicted image is compared with the original image and back-propagated to the encoder, and the deep convolutional network is trained using a set loss function. The method according to claim 1, characterized in that the loss function for training the deep convolutional network is set to:
Where p represents the position of the voxel in the image, _Im and f represent the image to be registered and the template image, and and represents the average voxel value of the adjacent voxels of the image to be registered and the template image with _pi as the center point, φ represents the deformation vector field from the image to be registered to the template image, _pi represents the central voxel of the current operation, and R represents the image domain. The method according to claim 3 is characterized in that the use of the deformation relationship learned by the non-rigid registration to register the image to be registered based on the organ anatomical structure point cloud comprises: The moving cube method is used to construct the human organ anatomical structure point cloud and the target area point cloud, in which the central difference method is used to calculate the gradient of the voxel vertex where each data point is located; After using the marching cube method to calculate the gradients of the eight vertices of the voxel, use linear interpolation to calculate the gradients of the central triangle patch vertices and draw the isosurface; The coordinates of the isosurface vertices are extracted and defined as a point cloud, and the defined target point cloud is registered based on the organ anatomical structure point cloud using non-rigid registration. The method according to claim 1 is characterized in that the multimodal medical imaging includes magnetic resonance imaging, ultrasound imaging, computed tomography and cone beam computed tomography. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8. A computer device comprises a memory and a processor, wherein a computer program that can be run on the processor is stored in the memory, and wherein the processor implements the steps of any one of the methods of claims 1 to 8 when executing the computer program.