WO2025035380A1 - Pet imaging method and apparatus based on optical flow registration, and device and storage medium - Google Patents

Abstract

The present application relates to a PET imaging method and apparatus based on optical flow registration, and a device and a storage medium. The method comprises: correcting the position of an MR image, and extracting a first feature map of the PET image and a second feature map of a corrected MR image; performing feature enhancement processing of a spatial dimension and a channel dimension on both the first feature map and the second feature map, and fusing feature enhancement processing results of the spatial dimension and the channel dimension to generate a first multi-modal enhanced feature map of the PET image and a second multi-modal enhanced feature map of the corrected MR image; and fusing the first multi-modal enhanced feature map and the second multi-modal enhanced feature map by means of a cross-modal feature fusion module and using a multi-head cross-attention mechanism, so as to generate a multi-modal fused feature map, and on the basis of the multi-modal fused feature map, generating a reconstructed PET image. By means of the present application, the features of PET and MR can be effectively fused to obtain a clearer reconstructed PET image.

Description

PET imaging method, device, equipment and storage medium based on optical flow registration Technical Field

The present application belongs to the field of medical imaging technology, and in particular relates to a PET imaging method, device, equipment and storage medium based on optical flow registration.

Background Art

PET (Positron Emission Tomography) is an important medical imaging technology used to obtain functional and metabolic information inside organisms. It reveals the metabolic activities of tissues and organs by detecting the distribution and concentration of radioactive isotope-labeled biological molecules in the body. During PET imaging, patients receive a labeled radioactive drug, usually a glucose molecule with a positron-emitting radioactive isotope (fluorine-18 labeled glucose, FDG). After these drugs are injected into the patient's body, they are metabolized and absorbed by the body's tissues and organs. When the positron decays, two high-energy positrons are released, which meet and annihilate with electrons in the body. During the annihilation process, two gamma photons in opposite directions are produced, which can be captured by the detector.

The use of radioisotopes in PET scanning and the associated radiation dose are potential risks to patients and medical staff, and the issue of PET radiation dose has received increasing attention. However, as the radiation dose and scanning time decrease, more noise will appear in the imaging process, resulting in poor PET image quality.

In order to solve the above problems, Yuru He et al. published an article titled "Dynamic PET Image Denoising With Deep Learning-Based Joint Filtering" in the IEEE Access journal in 2021. The study improved the network framework based on the structure of the spatial variant linear representation model (SVLRM). The study used a CNN with 12 convolutional layers to denoise PET noisy images. At the network input, the article fused PET and MR (Magnetic Resonance) images. The information of the two modalities is combined by splicing in the channel dimension. Subsequently, the fused input is processed by a network model with shared weights, and the final denoised PET image is output. However, due to the possible misalignment problem between the PET and MR modalities, this article did not effectively solve this problem in the multimodal fusion process; in addition, there is room for improvement in the information fusion process.

At the same time, the existing PET imaging technology still has the following shortcomings:

1. Long PET scanning time: During the PET scanning process, continuous image acquisition of the patient is required, which makes the entire scanning process relatively long, causing inconvenience to the patient and may increase discomfort during the scanning process.

2. Slow image reconstruction speed: The amount of data collected by full sampling is large, and complex image reconstruction algorithms are required, resulting in slow image reconstruction speed and prolonging the diagnosis process.

3. Motion artifacts: The long scanning process is easily affected by the patient's movement, resulting in image artifacts, thereby reducing image quality and affecting the doctor's accurate judgment of the patient's condition.

4. Lack of multimodal data usage and registration: Most algorithms perform denoising based on only a single PET modality, ignoring the possibility of using MR prior knowledge to guide PET image denoising, which limits the algorithm's ability to utilize multimodal information.

5. Simple processing of multimodal fusion process: Existing algorithms usually simply splice multimodal data in the channel dimension in multimodal fusion, lacking in-depth exploration of the effective interaction of multimodal features. This simple splicing method may not fully utilize the correlation and complementarity between multimodal data, resulting in unsatisfactory fusion results.

Summary of the invention

The present application provides a PET imaging method, apparatus, device and storage medium based on optical flow registration, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

In order to solve the above problems, this application provides the following technical solutions:

A PET imaging method based on optical flow registration, comprising:

PET images and MR images of the irradiated area were collected separately;

Acquire optical flow information of the PET image and the MR image through an optical flow registration network, correct the position of the MR image according to the optical flow information, and then extract a first feature map and a second feature map of the PET image and the corrected MR image respectively;

Performing feature enhancement processing of the spatial dimension and the channel dimension on the first feature map and the second feature map respectively by a spatial channel feature enhancement module, and fusing the feature enhancement processing results of the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image;

The first multimodal enhanced feature map and the second multimodal enhanced feature map are fused by a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and a PET reconstructed image is generated according to the multimodal fusion feature map.

The technical solution adopted by the embodiment of the present application also includes: before obtaining the optical flow information of the PET image and the MR image through the optical flow registration network, it also includes:

The Canny operator is used to extract contour information from the PET image and the MR image respectively.

The technical solution adopted by the embodiment of the present application also includes: the use of the Canny operator to extract contour information from the PET image and the MR image respectively is specifically as follows:

The PET image and the MR image are subjected to image corrosion and dilation processing by using morphological image processing technology, small objects and thin lines in the PET image and the MR image are eliminated, the basic contour shape of the radiation part is retained, and finally a binary mask composed of 0 and 1 is generated to represent the contour information of the PET image and the MR image respectively.

The technical solution adopted by the embodiment of the present application also includes: the optical flow registration network is a The U-net model includes an encoder and a decoder. The optical flow information of the PET image and the MR image is obtained through the optical flow registration network. After the position of the MR image is corrected according to the optical flow information, the first feature map and the second feature map of the PET image and the corrected MR image are respectively extracted as follows:

The contour information of the PET image and the MR image is input into the optical flow registration network, the encoder part of the optical flow registration network uses three 1x1 convolution layers to extract feature maps, and uses a 3×3 convolution layer to downsample the extracted feature maps; the decoder part of the optical flow registration network uses upsampling and 1x1 convolution layers to expand and compress the extracted feature maps to restore the size of the optical flow map, and generates a weight mask with a channel number of 64, generates the weight mask by applying a Sigmoid activation function, and outputs a first feature map F _pet and a second feature map F _mr of the PET image and the corrected MR image.

The technical solution adopted in the embodiment of the present application also includes: the spatial channel feature enhancement module performs feature enhancement processing on the first feature map and the second feature map in the spatial dimension and the channel dimension respectively, and fuses the feature enhancement processing results in the spatial dimension and the channel dimension to generate the first multimodal enhanced feature map of the PET image and the second multimodal enhanced feature map of the corrected MR image. Specifically,

The spatial channel feature enhancement module includes a spatial enhancement module and a channel enhancement module. The spatial enhancement module compresses the first feature map F _pet and the second feature map F _mr through two convolutional layers to obtain two matrices S1 and S2. The matrices S1 and S2 are connected in the spatial dimension to obtain a single-dimensional matrix S containing attention information of the PET and MR image streams. The first feature map F _pet and the second feature map F _mr are activated by a Sigmoid function and element-by-element operations are performed to obtain the first spatial enhancement feature map F _s-pet and the second spatial enhancement feature map F _s-mr of the PET image and the corrected MR image in the spatial dimension respectively.

The channel enhancement module uses global average pooling to compress the first feature map F _pet and the second feature map F _mr , and creates a channel descriptor, and calculates the encoding of the PET image and the MR image through the channel descriptor Vectors C1 and C2, connecting the encoding vectors C1 and C2, and creating a fused embedding vector through a fully connected layer, and then activating the embedding vector with a Sigmoid function and performing element-by-element operations to obtain the first channel enhanced feature map Fc _-pet and the second channel enhanced feature map Fc _-mr of the PET image and the corrected MR image respectively;

The first spatial enhancement feature map and the first channel enhancement feature map of the PET image and the second spatial enhancement feature map and the second channel enhancement feature map of the corrected MR image are fused respectively to generate a first multimodal enhancement feature map F _sc-pet of the PET image and a second multimodal enhancement feature map F _sc-mr of the corrected MR image.

The technical solution adopted in the embodiment of the present application also includes: after the cross-modal feature fusion module adopts a multi-head cross attention mechanism to fuse the first multimodal enhanced feature map and the second multimodal enhanced feature map, the multimodal fusion feature map is generated specifically as follows:

The cross-modal feature fusion module adopts a multi-head cross attention mechanism and a Transformer architecture to perform a linear projection operation on the first multimodal enhanced feature map F _sc-pet and the second multimodal enhanced feature map F _sc-mr to generate a query matrix Q, a key matrix K and a value matrix V for PET images and MR images, respectively, and perform an attention operation between the vectorized features of the two modalities:

In the above formula, σ represents the sigmoid function. represents the square root of the feature dimension, K _pet , V _pet represent the query matrix Q, key matrix K and value matrix V of PET images respectively. K _mr , V _mr represent the query matrix Q, key matrix K and value matrix V of MR images respectively;

After compressing the attention operation results of the two modalities through the average pooling operation, the attention maps of the two modalities are spliced in the channel dimension to generate a multimodal fusion feature map.

The technical solution adopted by the embodiment of the present application also includes: generating a PET reconstructed image according to the multimodal fusion feature map is specifically:

The multimodal fusion feature map is input into a decoder through a jump connection, and a PET reconstructed image is output through the decoder.

Another technical solution adopted by the embodiment of the present application is: a PET imaging device based on optical flow registration, comprising:

Image acquisition module: used to respectively acquire PET images and MR images of the radiation site;

An optical flow registration module: used for acquiring optical flow information of the PET image and the MR image through an optical flow registration network, and extracting a first feature map and a second feature map of the PET image and the corrected MR image respectively after correcting the position of the MR image according to the optical flow information;

A feature enhancement module is used to perform feature enhancement processing of the first feature map and the second feature map in the spatial dimension and the channel dimension respectively through the spatial channel feature enhancement module, and fuse the feature enhancement processing results in the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image;

Feature fusion module: used to fuse the first multimodal enhanced feature map and the second multimodal enhanced feature map through a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and generate a PET reconstructed image based on the multimodal fusion feature map.

Another technical solution adopted by the embodiment of the present application is: a device, the device includes a processor and a memory coupled to the processor, wherein:

The memory stores program instructions for implementing the PET imaging method based on optical flow registration;

The processor is used to execute the program instructions stored in the memory to control a PET imaging method based on optical flow registration.

Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the PET imaging method based on optical flow registration.

Compared with the prior art, the beneficial effects produced by the embodiments of the present application are: the PET imaging method, device, equipment and storage medium based on optical flow registration in the embodiments of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a PET imaging method based on optical flow registration according to an embodiment of the present application;

FIG2 is a schematic structural diagram of a PET imaging device based on optical flow registration according to an embodiment of the present application;

FIG3 is a schematic diagram of the device structure of an embodiment of the present application;

FIG. 4 is a schematic diagram of the structure of a storage medium according to an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications in the embodiments of the present application (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components in a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the terms "including" and "having" and any variations thereof are intended to cover Exclusive inclusion. For example, a process, method, system, product or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to the process, method, product or apparatus.

Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

Please refer to FIG1 , which is a flow chart of a PET imaging method based on optical flow registration according to an embodiment of the present application. The PET imaging method based on optical flow registration according to an embodiment of the present application comprises the following steps:

S100: Acquire PET images and MR images of the irradiated part respectively;

S110: Use the Canny operator to extract contour information from the PET image and the MR image respectively;

In this step, the lower and upper thresholds of contour information extraction are set to 20 and 50, respectively. The contour information extraction process is as follows: First, morphological image processing technology is used to perform image erosion and dilation on the PET image and the MR image to eliminate small objects and thin lines in the PET image and the MR image, while retaining the basic contour shape of the radiation site. During the processing, a circular kernel of size 5×5 is used to ensure isotropic erosion and dilation processes to maintain the integrity of the contour shape and prevent unnecessary distortion. Finally, a binary mask consisting of 0 and 1 is generated to represent the contour information I _pet and I _mr of the PET image and the MR image, respectively.

S120: inputting the contour information of the PET image and the MR image into the optical flow registration network, the optical flow registration network calculates the optical flow information according to the contour information of the PET image and the MR image, corrects the position of the MR image according to the optical flow information, and extracts the first feature map and the second feature map of the PET image and the corrected MR image respectively;

In this step, the optical flow registration network is a U-net-based model that uses the PET image as a reference and uses the inverse image transformation to correct the position of the MR image to achieve the effect of PET/MR dual-modality registration and alignment. Specifically, the optical flow registration network consists of two parts: an encoder and a decoder. The encoder part uses three The 1x1 convolution layer extracts the feature map, and the 3×3 convolution layer is used to downsample the extracted feature map by 2 times, thereby gradually reducing the size of the extracted feature map from 256×256 to 32×32. The decoder part uses upsampling and 1x1 convolution layers to expand and compress the extracted feature map to restore the size of the optical flow map, and finally generates a weight mask with 64 channels. Finally, the weight mask is generated by applying the Sigmoid activation function, and the first feature map F _pet and the second feature map F _mr of the PET image and the corrected MR image are output. Specifically, the optical flow registration network parameter settings are shown in Table 1 below:

Table 1 PWC optical flow registration network parameters

The encoder parameters are shown in Table 2 below:

Table 2 Encoder parameter settings

S130: inputting the first feature map and the second feature map into a spatial channel feature enhancement module, and performing feature enhancement processing on the first feature map and the second feature map in the spatial dimension and the channel dimension respectively by the spatial channel feature enhancement module, so as to obtain a first spatial enhancement feature map in the spatial dimension and a first channel enhancement feature map in the channel dimension of the PET image, and a second spatial enhancement feature map in the spatial dimension and a second channel enhancement feature map in the channel dimension of the corrected MR image respectively;

In this step, the spatial channel feature enhancement module (SC-FEM) includes a spatial enhancement module and a channel enhancement module. The spatial enhancement module can improve the network's attention to important areas such as high-density noise. In the spatial enhancement module, the first feature map F _pet and the second feature map F _mr of the input are compressed by two convolutional layers K1 and K2 to obtain two matrices S1 and S2. The matrices S1 and S2 are connected in the spatial dimension to obtain a single-dimensional matrix S containing the attention information of the PET and MR image streams. The first spatial enhancement feature map F _s-pet and the second spatial enhancement feature map F _s-mr of the PET image and the corrected MR image in the spatial dimension are finally obtained by activating the input feature map with a Sigmoid function and performing element-by-element operations. The parameter settings of the spatial enhancement module are shown in Table 3 below:

Table 3. Parameter settings of spatial enhancement module

In the channel enhancement module, global average pooling is used to compress each H×W input feature map into a 1×1 value, and a channel descriptor is created. The encoding vectors C1 and C2 of the PET image and MR image are calculated through the channel descriptor, and then the encoding vectors C1 and C2 are connected, and a fused embedding vector is created through a fully connected layer. Finally, the first channel enhanced feature map Fc _-pet and the second channel enhanced feature map Fc _-mr of the PET image and the corrected MR image are obtained by activating the embedding vector with a Sigmoid function and performing element-by-element operations. Specifically, the channel enhancement module parameter settings are shown in Table 4 below:

Table 4 Channel enhancement module parameter settings

S140: Fusing the first spatial enhancement feature map and the first channel enhancement feature map of the PET image and the second spatial enhancement feature map and the second channel enhancement feature map of the corrected MR image, respectively. generating a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image;

In this step, by fusing the first spatial enhancement feature map and the first channel enhancement feature map of the PET image and the second spatial enhancement feature map and the second channel enhancement feature map of the corrected MR image, a first multimodal enhancement feature map F _sc-pet and a second multimodal enhancement feature map F _sc-mr combining the two modal information of PET and MR are generated, thereby comprehensively considering the two modal information and making the feature enhancement process more effective. Specifically, the first multimodal enhancement feature map F _sc-pet and the second multimodal enhancement feature map F _sc-mr can be defined by the following formula:
F _sc-pet =F _pet +F _s-pet +F _c-pet (1)
F _sc-mr ＝F _mr +F _s-mr +F _c-mr (2)

S150: inputting the first multimodal enhanced feature map and the second multimodal enhanced feature map into a cross-modal feature fusion module, and the cross-modal feature fusion module fuses the first multimodal enhanced feature map and the second multimodal enhanced feature map using a multi-head cross attention mechanism to generate a multimodal fusion feature map;

In this step, the cross-modal feature fusion module adopts a multi-head cross-attention mechanism (CM-FFM) and a Transformer architecture to obtain the query matrix Q (queries), key matrix K (keys) and value matrix V (values) by calculating the dot product between the PET and MR images, thereby estimating the correlation between the multimodal enhanced feature maps, and calculating the cross-attention weights based on the softmax function through scaling and normalization. Specifically, the first multimodal enhanced feature map F _sc-pet and the second multimodal enhanced feature map F _sc-mr are first linearly projected to generate the query matrix Q, key matrix K and value matrix V of the PET image and the MR image, and then the attention operation is performed between the vectorized features of the two modalities. The calculation formula is as follows:

In the above formula, σ represents the sigmoid function, which is used to map the input to the range of 0 to 1, ensuring that the attention weight is within a reasonable range to represent the correlation between the two modalities and prevent the weight from deviating too much from the normal range. Represents the square root of the feature dimension and is used to scale the attention weights to ensure that the attention scores are not too large when calculating the dot product, thereby maintaining the stability of the model. K _pet , V _pet represent the query matrix Q, key matrix K and value matrix V of PET images respectively. K _pet and V _pet are both obtained by projecting the first multimodal enhanced feature map F _sc-pet , and represent information extracted from the PET image. _Kmr , _Vmr represent the query matrix Q, key matrix K and value matrix V of MR images respectively. K _mr and V _mr are both obtained by projecting the second multimodal enhanced feature map F _sc-mr , and represent the information extracted from the MR image.

After compressing the attention operation results of the two modalities through the average pooling operation, the attention maps of the two modalities are spliced in the channel dimension to generate a multi-modal fusion feature map, thereby providing rich and favorable image details for PET image denoising. Specifically, the multi-head cross attention mechanism parameter settings are shown in Table 5 below:

Table 5 Multi-head cross attention mechanism parameter settings

S160: input the multimodal fusion feature map into the decoder through the skip connection, and output the denoised PET reconstructed image through the decoder;

In this step, in the decoder part, considering that transposed convolution often produces chessboard artifacts during upsampling and involves a large number of trainable parameters, the embodiment of the present application selects a bilinear upsampling method to reduce The magnification factor is 2, and a convolutional layer is used at each stage, which helps to restore the multi-level feature map to the original image resolution and generate a relatively smooth feature map. In order to enhance the generalization ability of the model and improve the expression ability of the activation function, the embodiment of the present application introduces batch normalization in each convolutional layer during the downsampling and upsampling of the image, and introduces jump connections at the encoding and decoding ends to retain the most favorable image details from the two modalities. It can be understood that in other embodiments of the present application, CT images can also be used as a substitute for MR images to denoise PET images. Specifically, the decoder parameter settings are shown in Table 6 below:

Table 6 Decoder parameter settings

Based on the above, the PET imaging method based on optical flow registration in the embodiment of the present application utilizes the contour information of the PET image and the MR image, and aligns the PET image and the MR image through the optical flow correction network, so that the multimodal data can be better aligned, effectively reducing the image distortion and artifacts caused by the registration problem, and improving the clarity and accuracy of the image; by introducing the spatial channel feature enhancement module, the module uses matrices and vectors to represent spatial attention and channel attention, which is calculated not only based on the information of the current modality, but also combined with the influence of other modalities to ensure the effectiveness of feature extraction at each downsampling stage, which can better capture the important features in the image and improve the accuracy and robustness of feature extraction. By adopting a multi-head cross-attention mechanism to capture the diversity of feature information across modalities, it is possible to more fully utilize the correlation and complementarity between multimodal data, and can effectively fuse the features of PET and MR to obtain a better Add clear PET reconstruction images.

Please refer to FIG2 , which is a schematic diagram of the structure of a PET imaging device based on optical flow registration according to an embodiment of the present application. The PET imaging device 40 based on optical flow registration according to an embodiment of the present application comprises:

Image acquisition module 41: used to respectively acquire PET images and MR images of the radiation site;

An optical flow registration module 42 is used to obtain optical flow information of the PET image and the MR image through an optical flow registration network, and after correcting the position of the MR image according to the optical flow information, extract a first feature map and a second feature map of the PET image and the corrected MR image respectively;

The feature enhancement module 43 is used to perform feature enhancement processing of the first feature map and the second feature map in the spatial dimension and the channel dimension respectively through the spatial channel feature enhancement module, and fuse the feature enhancement processing results in the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image;

Feature fusion module 44: used to fuse the first multimodal enhanced feature map and the second multimodal enhanced feature map through a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and generate a PET reconstructed image according to the multimodal fusion feature map.

Please refer to FIG3 , which is a schematic diagram of the device structure of an embodiment of the present application. The device 50 includes:

A memory 51 storing executable program instructions;

A processor 52 connected to the memory 51;

The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: respectively collect PET images and MR images of the radiation site; obtain the optical flow information of the PET image and the MR image through the optical flow registration network, correct the position of the MR image according to the optical flow information, and respectively extract the first feature map and the second feature map of the PET image and the corrected MR image; respectively perform feature enhancement processing of the spatial dimension and the channel dimension on the first feature map and the second feature map through the spatial channel feature enhancement module, and fuse the feature enhancement processing results of the spatial dimension and the channel dimension to generate The first multimodal enhanced feature map of the PET image and the second multimodal enhanced feature map of the corrected MR image; after fusing the first multimodal enhanced feature map and the second multimodal enhanced feature map through a cross-modal feature fusion module using a multi-head cross-attention mechanism, a multimodal fusion feature map is generated, and a PET reconstructed image is generated according to the multimodal fusion feature map.

The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

Please refer to Figure 4, which is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores a program instruction 61 that can implement the following steps: respectively collect PET images and MR images of the radiation site; obtain the optical flow information of the PET image and the MR image through the optical flow registration network, and extract the first feature map and the second feature map of the PET image and the corrected MR image respectively after correcting the position of the MR image according to the optical flow information; perform feature enhancement processing of the spatial dimension and the channel dimension on the first feature map and the second feature map respectively through the spatial channel feature enhancement module, and fuse the feature enhancement processing results of the spatial dimension and the channel dimension to generate the first multimodal enhancement feature map of the PET image and the second multimodal enhancement feature map of the corrected MR image; after fusing the first multimodal enhancement feature map and the second multimodal enhancement feature map through the cross-modal feature fusion module using a multi-head cross attention mechanism, a multimodal fusion feature map is generated, and a PET reconstructed image is generated according to the multimodal fusion feature map. The program instructions 61 may be stored in the above storage medium in the form of a software product, including several instructions for enabling a device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute the entire method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program instructions, or terminal devices such as computers, servers, mobile phones, tablets, etc. Among them, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims (10)

A PET imaging method based on optical flow registration, characterized by comprising: PET images and MR images of the irradiated area were collected separately; Acquire optical flow information of the PET image and the MR image through an optical flow registration network, correct the position of the MR image according to the optical flow information, and then extract a first feature map and a second feature map of the PET image and the corrected MR image respectively; Performing feature enhancement processing of the spatial dimension and the channel dimension on the first feature map and the second feature map respectively by a spatial channel feature enhancement module, and fusing the feature enhancement processing results of the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image; The first multimodal enhanced feature map and the second multimodal enhanced feature map are fused by a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and a PET reconstructed image is generated according to the multimodal fusion feature map. The PET imaging method based on optical flow registration according to claim 1 is characterized in that before acquiring the optical flow information of the PET image and the MR image through the optical flow registration network, it also includes: The Canny operator is used to extract contour information from the PET image and the MR image respectively. The PET imaging method based on optical flow registration according to claim 2 is characterized in that the use of the Canny operator to extract contour information from the PET image and the MR image respectively is specifically: The PET image and the MR image are subjected to image corrosion and dilation processing by using morphological image processing technology, small objects and thin lines in the PET image and the MR image are eliminated, the basic contour shape of the radiation part is retained, and finally a binary mask composed of 0 and 1 is generated to represent the contour information of the PET image and the MR image respectively. The PET imaging method based on optical flow registration according to any one of claims 1 to 3 is characterized in that the optical flow registration network is a U-net-based model, including an encoder and a decoder, and the optical flow information of the PET image and the MR image is obtained through the optical flow registration network, and after the position of the MR image is corrected according to the optical flow information, the first feature map and the second feature map of the PET image and the corrected MR image are respectively extracted, specifically: The contour information of the PET image and the MR image is input into the optical flow registration network, the encoder part of the optical flow registration network uses three 1x1 convolution layers to extract feature maps, and uses a 3×3 convolution layer to downsample the extracted feature maps; the decoder part of the optical flow registration network uses upsampling and 1x1 convolution layers to expand and compress the extracted feature maps to restore the size of the optical flow map, and generates a weight mask with a channel number of 64, generates the weight mask by applying a Sigmoid activation function, and outputs a first feature map F _pet and a second feature map F _mr of the PET image and the corrected MR image. The PET imaging method based on optical flow registration according to claim 4 is characterized in that the spatial channel feature enhancement module performs feature enhancement processing of the first feature map and the second feature map in the spatial dimension and the channel dimension respectively, and fuses the feature enhancement processing results of the spatial dimension and the channel dimension to generate the first multimodal enhanced feature map of the PET image and the second multimodal enhanced feature map of the corrected MR image, specifically: The spatial channel feature enhancement module includes a spatial enhancement module and a channel enhancement module. The spatial enhancement module compresses the first feature map F _pet and the second feature map F _mr through two convolutional layers to obtain two matrices S1 and S2. The matrices S1 and S2 are connected in the spatial dimension to obtain a single-dimensional matrix S containing attention information of the PET and MR image streams. The first feature map F _pet and the second feature map F _mr are activated by a Sigmoid function and element-by-element operations are performed to obtain the first spatial enhancement feature map F _s-pet and the second spatial enhancement feature map F _s-mr of the PET image and the corrected MR image in the spatial dimension respectively. The channel enhancement module compresses the first feature map F _pet and the second feature map F _mr using global average pooling, and creates a channel descriptor, calculates the encoding vectors C1 and C2 of the PET image and the MR image through the channel descriptor, connects the encoding vectors C1 and C2, and creates a fused embedding vector through a fully connected layer, and then activates the embedding vector with a Sigmoid function and performs element-by-element operations to obtain the first channel enhanced feature map F _c-pet and the second channel enhanced feature map F _c-mr of the PET image and the corrected MR image, respectively; The first spatial enhancement feature map and the first channel enhancement feature map of the PET image and the second spatial enhancement feature map and the second channel enhancement feature map of the corrected MR image are fused respectively to generate a first multimodal enhancement feature map F _sc-pet of the PET image and a second multimodal enhancement feature map F _sc-mr of the corrected MR image. The PET imaging method based on optical flow registration according to claim 5 is characterized in that the multimodal fusion feature map is generated by fusing the first multimodal enhanced feature map and the second multimodal enhanced feature map using a multi-head cross attention mechanism through the cross-modal feature fusion module, and the multimodal fusion feature map is specifically: The cross-modal feature fusion module adopts a multi-head cross attention mechanism and a Transformer architecture to perform a linear projection operation on the first multimodal enhanced feature map F _sc-pet and the second multimodal enhanced feature map F _sc-mr to generate a query matrix Q, a key matrix K and a value matrix V for PET images and MR images, respectively, and perform an attention operation between the vectorized features of the two modalities:

In the above formula, σ represents the sigmoid function. represents the square root of the feature dimension, K _pet , V _pet represent the query matrix Q, key matrix K and value matrix V of PET images respectively. K _mr , V _mr represent the query matrix Q, key matrix K and value matrix V of MR images respectively; After compressing the attention operation results of the two modalities through the average pooling operation, the attention maps of the two modalities are spliced in the channel dimension to generate a multimodal fusion feature map. The PET imaging method based on optical flow registration according to claim 6 is characterized in that generating a PET reconstructed image according to the multimodal fusion feature map is specifically: The multimodal fusion feature map is input into a decoder through a jump connection, and a PET reconstructed image is output through the decoder. A PET imaging device based on optical flow registration, characterized by comprising: Image acquisition module: used to respectively acquire PET images and MR images of the radiation site; An optical flow registration module: used for acquiring optical flow information of the PET image and the MR image through an optical flow registration network, and extracting a first feature map and a second feature map of the PET image and the corrected MR image respectively after correcting the position of the MR image according to the optical flow information; A feature enhancement module is used to perform feature enhancement processing of the first feature map and the second feature map in the spatial dimension and the channel dimension respectively through the spatial channel feature enhancement module, and fuse the feature enhancement processing results in the spatial dimension and the channel dimension to generate a first multimodal enhanced feature map of the PET image and a second multimodal enhanced feature map of the corrected MR image; Feature fusion module: used to fuse the first multimodal enhanced feature map and the second multimodal enhanced feature map through a cross-modal feature fusion module using a multi-head cross-attention mechanism to generate a multimodal fusion feature map, and generate a PET reconstructed image based on the multimodal fusion feature map. A device, characterized in that the device comprises a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the PET imaging method based on optical flow registration according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control a PET imaging method based on optical flow registration. A storage medium, characterized in that it stores program instructions executable by a processor, wherein the program instructions are used to execute the PET imaging method based on optical flow registration as described in any one of claims 1 to 7.