WO2022027595A1 - Method for reconstructing low-dose image by using multiscale feature sensing deep network - Google Patents

Abstract

A method for reconstructing a low-dose image by using a multiscale feature sensing deep network. The method comprises: constructing an end-to-end network model, wherein the network model comprises a plurality of cascaded multi-size feature sensing modules, an input end and an output end of the network model are connected, and each multi-size feature sensing module includes a backbone structure, a primary branch and a secondary branch which are used for performing feature extraction on input feature maps of different scales, wherein the secondary branch adjusts the sizes of the processed feature maps to be consistent with that of the primary branch and transmits the processed feature maps to the primary branch, so as to guide the feature extraction performed by the primary branch; the primary branch adjusts the sizes of the processed feature maps to be consistent with that of the backbone structure and transmits the processed feature maps to the backbone structure, so as to guide the feature extraction performed by the backbone structure; and inputting a low-dose image to be constructed into the trained network model, so as to obtain a reconstructed image. By means of the method, a reconstructed image with a better artifact correction effect can be obtained.

Description

Reconstruction of low-dose images using multi-scale feature-aware deep networks technical field

The present invention relates to the technical field of medical image processing, and more particularly, to a method for reconstructing a low-dose image using a multi-scale feature-aware deep network.

Background technique

Computed tomography (CT) technology is widely used in the detection of tissues in the human body. Taking an oral CT image as an example, the oral CT instrument emits X-ray beams from enough angles during the scanning process to obtain fully sampled projection data, and then calculates and reconstructs a clear intraoral image. In recent years, the public has paid extensive attention to the damage to the human body caused by X-ray radiation in CT examinations, and patients wish to receive as little radiation as possible during examinations. During the inspection process, the exposure time of the X-ray tube can be shortened by reducing the sampling angle, thereby reducing the radiation dose. However, there will be obvious artifacts and noise in the image reconstructed from the sparsely sampled projection data, which will cause the image quality to decrease significantly and affect the clinical diagnosis. Therefore, studying the generation of oral CT images with a near-normal dose effect from low-dose oral CT images can effectively reduce the radiation dose received by patients and obtain high-quality CT images that meet the needs of clinical diagnosis, which is conducive to expanding CT technology. scope of application.

However, the current low-dose image reconstruction methods mainly have the following defects: when the complete projection data is sampled, the radiation dose to the patient is relatively large; when the complete projection data is sampled, the X-ray tube exposure time is long, and the equipment loss is large; the existing low-dose image The reconstruction effect still needs to be improved. For example, Xieshipeng et al. published an article "Artifact Removal using Improved GoogLeNet for Sparse-view CT Reconstruction" in the journal Scientific Reports in 2018 (using improved GoogLeNet to remove artifacts for sparse CT reconstruction). The article adds residual learning on the basis of GoogleNet, so that the network can learn the characteristics of artifacts. The network in the article contains 3 convolutional layers and 8 inception modules, each inception module contains 3 convolution kernels of different sizes, with sizes of 1×1, 3×3, 5×5, to improve feature extraction accuracy. The network is firstly a two-layer convolutional layer used to initially extract the artifact features of the input image, and then eight inception modules are used to process the artifact features more finely. The last layer of the network convolutional layer fuses all feature maps to form The artifact distribution map is output, and the sparsely reconstructed CT image is used to subtract the artifact distribution map to obtain the final corrected image. However, the effect of using this method to obtain reconstructed images still needs to be further improved to restore more complete and clearer image details.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the above-mentioned defects of the prior art, and provide a method for reconstructing a low-dose image using a multi-scale feature-aware depth network, and further process the low-dose image reconstructed from sparsely sampled projection data to obtain high quality image, the obtained image is close to the effect of reconstruction using fully sampled projection data.

According to an aspect of the present invention, a method for reconstructing a low-dose image using a multi-scale feature-aware deep network is provided. The method includes the following steps:

Build an end-to-end network model with low-dose images as input and normal-dose images as output, the network model includes multiple cascaded multi-scale feature perception modules, and the input and output of the network model have connections, each The multi-scale feature perception module includes a backbone structure, a first-level branch and a second-level branch for feature extraction of input feature maps of different scales, wherein the second-level branch adjusts the processed feature map size to the same size as the first-level branch. The branch is consistent and passed to the first-level branch to guide the feature extraction of the first-level branch; the first-level branch adjusts the size of the processed feature map to be consistent with the backbone structure and transmits it to the backbone structure to guide the backbone structure feature extraction; input the low-dose image to be reconstructed into the trained network model to obtain a reconstructed image.

Compared with the prior art, the present invention has the advantage that the feature map is processed into different scales by using a multi-scale feature perception module, and then the feature information is learned on different scales, so that the extracted feature information is more comprehensive. In addition, the processed feature information of different branches is transmitted layer by layer from bottom to top, and the feature information extracted by the upper branch can be corrected, thereby improving the accuracy of feature learning and making the artifact correction in the reconstructed image more accurate. , the human tissue structure information is more complete.

Other features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

1 is a flowchart of a method for reconstructing a low-dose image using a multi-scale feature-aware deep network according to an embodiment of the present invention;

2 is a structural diagram of a multi-scale feature perception module according to an embodiment of the present invention;

3 is an overall framework diagram of a deep network model according to an embodiment of the present invention;

4 is a schematic diagram of a residual block in a multi-scale feature perception module according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a fully sampled normal dose oral CT image according to an embodiment of the present invention;

6 is a schematic diagram of a sparsely sampled low-dose oral CT image according to an embodiment of the present invention;

FIG. 7 is an effect diagram of a reconstructed oral CT image according to an embodiment of the present invention.

detailed description

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

In all examples shown and discussed herein, any specific values should be construed as illustrative only and not limiting. Accordingly, other instances of the exemplary embodiment may have different values.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

The present invention provides a technical solution for reconstructing sparsely sampled low-dose maps using a multi-scale feature-aware deep network, which can be applied to various types of image data reconstruction, such as CT images, MRI images, PET images, and 3D images. In order to clearly understand the idea of the present invention, in the following description, an oral CT image will be used as an example for illustration.

In short, the present invention provides a deep network model including a multi-scale feature perception module (or multi-scale feature perception module) to correct artifacts in sparsely sampled low-dose oral CT images to improve image quality. The deep network model is an end-to-end network, and the main body consists of multiple multi-scale feature perception modules and a long skip connection connecting the input and output. For example, the multi-scale feature perception module is divided into a backbone structure and two branches according to the level. Two branches are used to reduce the input feature map to different sizes, so as to extract and learn the features of artifacts on different sizes. Among them, the size of the feature map processed by the second-level branch of the lowest level will be adjusted to be consistent with the first-level branch of the upper layer, and then input into the first-level branch to correct and guide the extraction and learning of the features of the first-level branch. . The module backbone structure learns on the input feature map size. Similarly, the feature map processed by the first-level branch will also adjust the size to be consistent with the backbone feature map, and then pass it to the backbone to make the feature map information more refined by the backbone. Accurate extraction and learning. Multiple multi-scale feature perception modules can be connected to facilitate the deep network model to generate higher quality CT images.

Specifically, as shown in FIG. 1 , the method for reconstructing a low-dose image using a multi-scale feature-aware deep network provided by an embodiment of the present invention includes the following steps:

In step S110, a multi-scale feature perception module is constructed to perform feature extraction on feature maps of different scales.

Specifically, as shown in FIG. 2 , the multi-scale feature sensing module is divided into a backbone and two branches according to the level, which are respectively called the backbone structure (or backbone for short), the first-level branch and the second-level branch from top to bottom. . Through one and two inverse shuffling operations, the input feature map size is reduced to 1/2 and 1/4, respectively, and the number of feature map channels is increased to 4 times and 16 times, respectively, and then the reduced size feature maps are input to In the two-level branch.

In the lowest-level branch (ie, the second-level branch), a layer of convolutional layer is used to fuse the feature information on some channels to reduce the number of channels and improve computational efficiency, and then use a residual block to further learn features. For example, as shown in Figure 3, the residual block contains 3 convolutional layers, and the activation function LeakyReLU is used for activation after each convolutional layer. The processing result of the third convolution layer in the residual block is fused with the input of the residual block, and then output through the activation function LeakyReLU, so as to realize the transfer of low-level feature information to high-level, and improve the stability of network training. After the output of the residual block is processed by the convolution layer, the shuffling operation is used to aggregate the feature information on different channels, the number of channels is reduced to 1/4 and the size of the feature map is enlarged by 2 times, and then the feature information is passed to the upper branch. middle.

In the first-level branch, a convolutional layer is also used to fuse the feature information on some channels to reduce the number of channels. Different from the secondary branch, the primary branch first cascades the feature information from the secondary branch and fuses the information through the convolution layer, and then inputs it into the residual block for processing. The feature information is used to guide and correct the feature extraction and learning of the first-level branch. The output of the residual block is also processed by the convolution layer, and then the feature information on different channels is aggregated through the shuffling operation, the number of channels is reduced to 1/4 and the size of the feature map is enlarged to 2 times the original size, and then the feature information is passed to the backbone. among.

The backbone structure is different from other branches. The backbone can be divided into two parts according to the cascading position of the feature information of the lower branch. Each of the two parts has 3 layers of convolutional layers, and the convolutional layers of the former part are followed by activation functions. ReLU, the latter part is followed by the activation function LeakyReLU. The feature information extracted and learned from the former part is fused with the feature information of the lower branch for correction, and then input to the latter part for subsequent processing. The processed feature information is fused with the input of the multi-scale feature perception module to obtain the final result, which is then output outside the module.

In this step, a multi-size feature perception module is constructed for learning feature maps of different sizes, which can extract more accurate feature information.

It should be noted that those skilled in the art can also make appropriate modifications or changes to the above-mentioned multi-scale feature perception module, for example, the residual blocks that use more convolution layers, first-level branches and second-level branches use the same Or a different structure, use other activation functions, or include more levels of branches, etc. In addition, the cascade identified in FIG. 2 is used to indicate the location where the feature information of the subordinate branch is received, and is also called a cascade layer for the convenience of understanding the overall architecture of the network.

Step S120, constructing an end-to-end network model including multiple multi-scale feature perception modules.

For example, referring to the overall framework of the network model shown in Figure 4, the network model includes two convolution layers (the size of the convolution kernel can be set to 3 × 3), two multi-scale feature perception modules and a connection network model input and long jump connections at the output.

Since artifacts and noise in low-dose oral CT images can be regarded as additive interference factors, adding long skip connections enables the network model to learn the features of residual images (ie, artifacts, noise). The first convolutional layer in the network model is used to perform preliminary feature extraction on the input image, generate a feature map with 64 channels, for example, and then input it into the multi-scale feature perception module. Two multi-scale feature perception modules are used for more accurate extraction and learning of image feature information. The last layer of convolutional layer fuses the information of different channels in the feature map output by the multi-scale feature perception module to obtain a single-channel image, and fuses the single-channel image with the input image to obtain the final output of the network model. Shadow corrected image.

It should be noted that by connecting multiple multi-scale feature perception modules (not limited to two), the performance of the network model can be adjusted, so that the network model can be applied to different tasks, thereby improving the generalization ability.

Step S130, designing the loss function of the network model.

Assume that the low-dose oral CT image dataset is D1={x ₁ ,x ₂ ,...,x _n }, and the normal-dose CT image dataset is D2={y ₁ ,y ₂ ,...,y _n }, where n is The total number of image samples. Preferably, the loss function of the network model is divided into reconstruction loss and perceptual loss.

In one embodiment, the reconstruction loss uses L1loss to calculate the difference between the generated image and the normal dose image, which can be expressed as:

Wherein Net represents the network model of the present invention, and Net( _xi ) represents the output result of the network model.

In one embodiment, the perceptual loss refers to using the network model VGG19 to generate high-level features of the image and the normal dose image, and then calculating the L2loss between the two, which can be expressed as:

where VGG ₁₉ (Net(x _i )) and VGG ₁₉ (y _i ) represent the output results of the generated image and normal dose image input to the VGG19 network, respectively.

The purpose of training the network model is to minimize the value of the above two loss functions as the optimization goal, and update the parameters of the network model.

It should be noted that those skilled in the art can also make appropriate variations or modifications to the above loss function. For example, it is not necessary to include perceptual loss or replace VGG19 network with VGG16 network.

Step S140, train the network model with the optimized loss function as the target.

Use paired sparsely sampled low-dose oral CT images and fully sampled normal-dose oral CT images to construct training datasets and test sets, and use the Adam algorithm to iteratively optimize the parameters of each layer of the network model (such as weights, biases, etc.) during the training process. ). After training, the performance of the network model can be further tested using the test set.

Step S150, the low-dose image is input into the trained network model to obtain a reconstructed image.

Using the trained network model, the mapping between the low-dose oral CT image and the normal-dose oral CT image can be realized, that is, the sparsely sampled low-dose image is input into the trained network model, and the reconstructed image can be obtained.

In order to further verify the effect of the present invention, experiments were carried out, and the experimental results are shown in Fig. 5 to Fig. 7, wherein Fig. 5 is a normal-dose oral CT image of complete sampling, Fig. 6 is a sparsely sampled low-dose oral CT image, Fig. 7 is the rendering of the reconstructed oral CT image. It can be seen that the reconstructed image obtained by the present invention has clearer details and a more complete structure. In addition, the CT reconstruction image quality evaluation parameters are shown in Table 1 below.

Table 1. CT reconstruction image quality evaluation parameter table (PSNR, SSIM, NMSE)

  PSNR SSIM NMSE Low-dose oral CT images 28.63 0.8555 0.0232 Reconstructed image of the present invention 35.86 0.9125 0.0044

It can be seen from Table 1 that the processing results of the present invention are significantly improved in terms of PSNR (peak signal-to-noise ratio) and SSIM (structural similarity). The lower score on the mean square error) index indicates that the processing of the present invention reduces the error with the normal dose CT image, and the result is more accurate, which is close to the normal dose CT image.

To sum up, the multi-scale feature perception module of the present invention processes the feature map into different scales, and then learns the feature information on different scales, so that the extracted feature information is more comprehensive. In addition, the processed feature information of different branches is transmitted from bottom to top layer by layer. After the feature information extracted by the lower layer branch is fused with the feature information of the upper layer branch, the feature information extracted by the upper layer branch can be corrected, which improves the performance of the upper layer branch. The accuracy of feature learning makes the artifact correction in the images output by the network model more accurate, and the human tissue structure information is more complete. In addition, shuffling and inverse shuffling operations are applied in the multi-size feature perception module, and the number of feature map channels is correspondingly increased and decreased when reducing and increasing the size of the feature map, so as to better preserve pixel information and avoid information loss.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims (10)

A method for reconstructing low-dose images using multi-scale feature-aware deep networks, comprising the following steps: Build an end-to-end network model with low-dose images as input and normal-dose images as output, the network model includes multiple cascaded multi-scale feature perception modules, and the input and output of the network model have connections, each The multi-scale feature perception module includes a backbone structure, a first-level branch and a second-level branch for feature extraction of input feature maps of different scales, wherein the second-level branch adjusts the processed feature map size to the same size as the first-level branch. The branch is consistent and passed to the first-level branch to guide the feature extraction of the first-level branch; the first-level branch adjusts the size of the processed feature map to be consistent with the backbone structure and transmits it to the backbone structure to guide the backbone structure feature extraction; The low-dose image to be reconstructed is input to the trained network model, and the reconstructed image is obtained. The method of claim 1, wherein the secondary branch comprises a first convolutional layer, a first residual block and a second convolutional layer connected in sequence; the primary branch comprises a first convolutional layer connected in sequence Three convolutional layers, a first convolutional layer, a fourth convolutional layer, a second residual block and a fifth convolutional layer, the first convolutional layer is connected with the output of the second branch; the backbone structure It includes a first feature extraction network, a second cascade layer, and a second feature extraction network connected in sequence. The first feature extraction network includes a plurality of convolutional layers and each convolutional layer is activated by ReLU. The second feature The extraction network includes a plurality of convolutional layers and each convolutional layer is activated by LeakyReLU, and the output of the second feature extraction network is fused with the input of the first feature extraction network as the output of the backbone structure. The method of claim 2, wherein the first residual block and the second residual block have the same structure, including a plurality of convolutional layers, and the output of the last convolutional layer is the same as the corresponding residual Input fusion of difference blocks. The method of claim 1, wherein a loss function for training the network model comprises a reconstruction loss and a perceptual loss, the reconstruction loss is used to characterize the gap between the low-dose image and the normal-dose image, and the perceptual loss is represented by Loss between high-level features characterizing low-dose and normal-dose images. The method of claim 4, wherein the reconstruction loss is expressed as:

Among them, the low-dose image dataset is D1={x ₁ ,x ₂ ,...,x _n }, and the normal-dose image dataset is D2={y ₁ ,y ₂ ,...,y _n }, where n is the total number of image samples , Net represents the network model, and Net( _xi ) represents the output result of the network model. The method of claim 4, wherein the perceptual loss is expressed as:

Wherein VGG ₁₉ (Net) x _i )) and VGG ₁₉ (y _i ) represent the output results after inputting the generated image and normal dose image of the network model to the VGG19 network, respectively. The method according to claim 1, wherein the input feature map of the first-level branch is obtained by performing an inverse shuffling operation on the input feature map of the backbone structure, and the input feature map of the second-level branch is a pair of The input feature map of the backbone structure is obtained through two inverse shuffling operations. The method of claim 1, the network model comprising a first convolutional layer, a first multi-scale feature perception module, a second multi-scale feature perception module, and a second convolutional layer connected in sequence, the second volume The output of the convolutional layer is fused with the input of the first convolutional layer as the output of the network model. A computer-readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the steps of the method of claim 1 . A computer device, comprising a memory and a processor, a computer program that can be run on the processor is stored in the memory, and characterized in that, when the processor executes the program, the method of claim 1 is implemented. step.

Images