CN116630201A - Image reconstruction model building and applying method and device - Google Patents

Abstract

The invention relates to an image reconstruction model establishing and applying method and device, which are applied to the technical field of image processing and comprise the following steps: based on the traditional GAN network structure, L1 loss, perception loss and total variation loss are introduced, the L1 loss can enable the GAN network to obtain better imaging effect when training paired pictures, deep information between a real image and a network generated image is approximated through the perception loss, detail information of output characteristics is enhanced, unnecessary details are removed through the total variation loss, important details such as edges are reserved, the image is smoother, a new framework for efficiently recovering image quality from sparse photoacoustic data is built through the L1 loss, the perception loss and the addition of the total variation loss, when images reconstructed by a small amount of undersampled data or limited view scanning are provided, the trained network can enhance the visibility of structures in any direction, and further better effects are obtained on the outline edges and the internal details of the images.

Description

Method and device for establishing and applying an image reconstruction model

technical field

The invention relates to the technical field of image processing, in particular to an image reconstruction model establishment and application method and device.

Background technique

In the field of traditional photoacoustic image reconstruction research, the initial method is to use reconstruction algorithms such as time reversal, Fourier transform, etc. After the rise of deep convolutional neural networks, Neda Davoudi et al. innovatively combined deep convolution The neural network combined with its new scanner has obtained better image reconstruction quality, but in photoacoustic imaging, the number of sensors and cost have always been a problem that needs to be solved. Under low-number sensor imaging conditions such as 16 and 32 Under-sampled images, the imaging effect after reconstruction with ordinary convolutional neural networks such as CNN, resnet, unet, etc. is not good, and more information is lost during the imaging process.

Contents of the invention

In view of this, the object of the present invention is to provide an image reconstruction model establishment, application method and device to solve the problem of under-sampled images obtained under low-number sensor imaging conditions such as 16 and 32 in the prior art. Ordinary convolutional neural networks such as CNN, resnet, unet, etc. have poor imaging effects after reconstruction, and a lot of information is lost during the imaging process.

According to a first aspect of an embodiment of the present invention, a method for establishing an image reconstruction model is provided, the method comprising:

Obtain multiple sets of undersampled images and corresponding real images to obtain training set data;

Select any group of undersampled images and their corresponding real images in the training set data, and input the undersampled images into the GAN network structure to obtain network generated images;

Select any pixel point in the image generated by the network, obtain two pixel points adjacent to the pixel point in the horizontal direction and vertical direction, and calculate the value of the pixel point through the two pixel points adjacent to the pixel point in the horizontal direction and vertical direction Difference value, summing the difference values of all pixels in the image generated by the network to obtain the total variation loss;

The loss value of the network generated image and the real image is represented by L1 loss, perceptual loss, BCE loss and total variation loss, with the minimum loss value of the network generated image and the real image as the goal, through the training set data pair The GAN network structure is trained, and the parameters of the GAN network structure are adjusted until the loss value no longer decreases, and an image reconstruction model is established.

Preferably,

The GAN network structure is trained by the training set data until the loss value no longer declines, and the image reconstruction model is established including:

The generator of the GAN network structure is trained by the training set data until the loss value of the generator no longer declines to obtain a trained generator;

The discriminator of the GAN network structure is trained by the training set data until the loss value of the discriminator no longer declines to obtain a trained discriminator;

An image reconstruction model is established through the trained generator and discriminator.

Preferably,

During the establishment of the image reconstruction model, the generator and the discriminator are trained alternately, when the generator is trained, the parameters of the discriminator are fixed, and when the discriminator is trained, the generated The parameter of the device is fixed.

Preferably,

The generator of the GAN network structure is trained by the training set data until the loss value of the generator no longer declines, and the trained generator includes:

The subsampled image is input into the generator of the GAN network structure to obtain the image generated by the network;

The image generated by the network and the under-sampled image are input into the discriminator of the GAN network structure to obtain the first similarity matrix;

The perceptual loss is obtained through the network generated image and the real image, the L1 loss is obtained through the preset L1 loss function, and the first BCE loss is obtained through the first similarity matrix and the N-order 1 matrix;

The generator loss is obtained by the perceptual loss, the L1 loss, the first BCE loss and the total variation loss, and the generator loss is minimized, and the generator is trained through the training set data until the generator The loss no longer drops, and a trained generator is obtained.

Preferably,

The generator loss obtained through the perceptual loss, L1 loss, first BCE loss and total variation loss includes:

The damping coefficient is introduced, and the damping coefficient of each iteration is calculated according to the preset damping coefficient calculation formula;

The sum of the perceptual loss, L1 loss and total variation loss of each iteration is multiplied by the damping coefficient of this iteration to obtain the damping loss;

adding the damping loss to the first BCE loss of this iteration to obtain the generator loss;

Preferably,

The discriminator of the GAN network structure is trained through the training set data until the loss value of the discriminator no longer decreases, and the trained discriminator includes:

Inputting the undersampled image and the real image into the discriminator together to obtain a second similarity matrix;

A second BCE loss is obtained through the first similarity matrix and an N-order 0 matrix, and a third BCE loss is obtained through the second similarity matrix and an N-order 1 matrix;

The discriminator loss is obtained through the second BCE loss and the third BCE loss, and the discriminator loss is minimized, and the discriminator is trained through the training set data until the discriminator loss no longer decreases , to get the trained discriminator.

Preferably,

The perceptual loss obtained by generating images and real images through the network includes:

The network-generated image and the real image are respectively input into the feature extraction module of the generator, and the feature extraction module outputs the network feature and the real feature respectively, and the perceptual loss is obtained through the network feature and the real feature.

According to the second aspect of the embodiments of the present invention, an image reconstruction model application method is provided, the method is based on the image reconstruction model established above, and the method includes:

Acquiring the undersampled image to be processed, and inputting the undersampled image to be processed into an image reconstruction model;

The image reconstruction model enhances internal detail information of the under-sampled image to be processed based on perceptual loss, enhances edge contours of the under-sampled image to be processed based on L1 loss, and smoothes the output image based on total variation loss, Output an enhanced image.

According to a third aspect of the embodiments of the present invention, an image reconstruction model establishment device is provided, the device comprising:

Data acquisition module: used to acquire multiple sets of undersampled images and corresponding real images to obtain training set data;

Network image acquisition module: used to select any group of undersampled images and their corresponding real images in the training set data, and input the undersampled images into the GAN network structure to obtain network generated images;

Total variation loss acquisition module: used to select any pixel point in the image generated by the network, obtain two pixel points adjacent to the pixel point in the horizontal direction and vertical direction, and obtain two pixel points adjacent to the pixel point in the horizontal direction and vertical direction. Two pixel points calculate the difference value of the pixel point, sum the difference values of all the pixel points in the image generated by the network, and obtain the total variation loss;

Model building module: used to represent the loss value of the network generated image and the real image through L1 loss, perceptual loss, BCE loss and total variation loss, with the minimum loss value of the network generated image and the real image as the goal, through The training set data trains the GAN network structure, adjusts the parameters of the GAN network structure until the loss value no longer decreases, and establishes an image reconstruction model.

According to a fourth aspect of the embodiments of the present invention, there is provided an image reconstruction model application device, the device comprising:

Input module: used to acquire the under-sampled image to be processed, and input the under-sampled image to be processed into the image reconstruction model;

Output module: for the image reconstruction model to enhance internal detail information of the under-sampled image to be processed based on perceptual loss, to enhance the edge profile of the under-sampled image to be processed based on L1 loss, and to enhance the edge profile of the under-sampled image to be processed based on the total variation loss The output image is smoothed and the enhanced image is output.

The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

This application introduces L1 loss, perceptual loss and total variation loss based on the traditional GAN network structure during the training process of the model. The L1 regularization item can enable the GAN network to obtain better imaging effects when training paired pictures , Approximating the deep information between the real image and the network-generated image through perceptual loss, that is, perceptual information, can enhance the detailed information of the output features, remove unnecessary details through the total variational loss, while retaining important details such as edges, so that The image is smoother. Through the addition of L1 loss, perceptual loss and total variation loss, a new framework for efficiently recovering image quality from sparse photoacoustic data, when provided with images reconstructed from a small amount of undersampled data or limited viewing angle scans, training A good network can enhance the visibility of structures in any direction, and then achieve better results in the contour edges and internal details of the image, and restore the expected image quality as much as possible.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

Fig. 1 is a schematic flowchart of a method for establishing an image reconstruction model according to an exemplary embodiment;

Fig. 2 is a schematic diagram showing the principle of generator training according to another exemplary embodiment;

Fig. 3 is a schematic diagram showing the principle of discriminator training according to another exemplary embodiment;

Fig. 4 is a schematic structural diagram of a VGG16 feature extraction module shown according to another exemplary embodiment;

Fig. 5 is a schematic flowchart of a method for applying an image reconstruction model according to another exemplary embodiment;

Fig. 6 is a system schematic diagram of an apparatus for establishing an image reconstruction model according to another exemplary embodiment;

Fig. 7 is a system schematic diagram of an image reconstruction model application device according to another exemplary embodiment;

In the drawings: 101-data acquisition module, 102-network image acquisition module, 103-total variation loss acquisition module, 104-model building module, 201-input module, 202-output module.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with aspects of the invention as recited in the appended claims.

Embodiment one

Fig. 1 is a schematic flowchart of a method for establishing an image reconstruction model according to an exemplary embodiment. As shown in Fig. 1, the method includes:

S1, obtaining multiple sets of under-sampled images and corresponding real images to obtain training set data;

S2, selecting any group of undersampled images and their corresponding real images in the training set data, and inputting the undersampled images into the GAN network structure to obtain network-generated images;

S3, select any pixel in the image generated by the network, obtain two pixels adjacent to the pixel in the horizontal direction and vertical direction, and calculate the pixel through the two adjacent pixels in the horizontal direction and vertical direction The difference value of the point, the difference value of all pixel points in the image generated by the network is summed to obtain the total variation loss;

S4, expressing the loss value of the image generated by the network and the real image by L1 loss, perceptual loss, BCE loss and total variation loss, aiming at the minimum loss value of the image generated by the network and the real image, through the training set Data trains the GAN network structure, adjusts the parameters of the GAN network structure, until the loss value no longer declines, and establishes an image reconstruction model;

It can be understood that in this application, pairs of under-sampled pictures and 512 high-precision sampling pictures are input into the GAN network structure for model training. It is worth emphasizing that the input images in this application are photoacoustic images. This application focuses on optical A new model is proposed for the reconstruction of acoustic images. The practical application of photoacoustic tomography (PAT) usually involves suboptimal sampling of tomographic data. In the process of data acquisition, in order to obtain high-quality images , it is usually necessary to use a larger number of transducers for measurement, but more transducers will bring higher production costs, hardware complexity, and greater challenges to the computing power of the device, so it is necessary to exchange The number of transducers is limited, and measurements are performed under sparse viewing angles. Photoacoustic imaging under sparse viewing angles can effectively reduce system complexity, reduce costs, and reduce the amount of data that needs to be processed to speed up imaging, but it will also lead to reconstruction of light. The quality of the acoustic image is degraded, and artifacts and blurring appear, so it is necessary to propose a new model to preserve more detailed information and border outlines during the reconstruction process of the photoacoustic image; the existing GAN (generated confrontation network) is the main The structure includes a generator G (Generator) and a discriminator D (Discriminator). The generator generates new network images based on undersampled images, and the discriminator performs similarity judgments on network-generated images and high-precision sampling images. This application is in On the basis of the existing GAN network structure, L1loss is introduced. The L1 regularization item can enable the GAN network to obtain better imaging results when training pairs of pictures. At the same time, in order to solve the problem of sparse perspective and blurred edges in the picture , this application also cites the perceptual loss function on the basis of the GAN network structure. The perceptual loss is used for real-time super-resolution tasks and style transfer tasks. It was later applied to more fields, and there are many works in the direction of image denoising. When it comes to perceptual loss, in the process of extracting features, the shallower layer usually extracts low-frequency information such as edges, colors, and brightness, while the deeper layer of the network extracts some high-frequency information such as detailed textures, and the deeper network layer extracts some information with discriminative features. That is to say, the deeper the network layer, the more abstract and advanced the features extracted, and the perceptual loss can realize the focus on deep features, so as to achieve better results in the contour edges and internal details of the image. The total variation loss is eliminated. The most effective application of total variation in image processing is image denoising and restoration. The total variation of a noise-contaminated image is significantly larger than that of a noise-free image. Limiting the total variation is Will limit the noise, used on the image, the total variation loss (TV loss) can make the image smooth, it is based on the principle that the signal with too many and possibly false details has a high total variation, that is, the signal's The integral of the absolute gradient is high. According to this principle, the total variation of the signal is reduced to closely match the original signal, removing unnecessary details while retaining important details such as edges; TV Loss stands for Total Variation Loss (Total Variation Loss) Variational loss), calculates the total variation of the image generated by the network, TV Loss is often used as a regular term in the overall function to constrain network learning, and can effectively promote the spatial smoothness of network output results. In digital image processing, its definition is usually as follows:

In the formula, x _{i, j} represents any pixel in the image generated by the network. The meaning of the above formula is to calculate the relationship between each pixel x _{i, j} and the horizontal direction (the width W of the image), the vertical direction (the height of the image H) The square of the difference between the next adjacent pixel x _i,j-1 and x _i+1,j , and then the square root, summing all pixels to get the total variation loss;

Through the addition of L1 loss, perceptual loss, and total variational loss, a new framework for efficiently recovering image quality from sparse photoacoustic data, when provided with images reconstructed from small amounts of undersampled data or limited viewing angle scans, the trained network can enhance The visibility of structures in any direction, and thus better results in the contour edges and internal details of the image, restore the expected image quality as much as possible.

Preferably,

The GAN network structure is trained by the training set data until the loss value no longer declines, and the image reconstruction model is established including:

The generator of the GAN network structure is trained by the training set data until the loss value of the generator no longer declines to obtain a trained generator;

The discriminator of the GAN network structure is trained by the training set data until the loss value of the discriminator no longer declines to obtain a trained discriminator;

Establishing an image reconstruction model through the trained generator and discriminator;

It is understandable that in the process of building the model, that is, during the training process, it is necessary to perform confrontational loop training on the generator and the discriminator of the GAN network structure, and use the gradient descent method until the respective loss values no longer decrease. Complete the training of the generator and the discriminator, and introduce the concepts of L1 loss and perceptual loss during the training process, and the trained generator and discriminator composition are more advantageous for image reconstruction than the existing GAN network structure LP-GAN network structure.

Preferably,

During the establishment of the image reconstruction model, the generator and the discriminator are trained alternately, when the generator is trained, the parameters of the discriminator are fixed, and when the discriminator is trained, the generated The parameters of the device are fixed;

It is understandable that since the generator and the discriminator are alternately trained against each other, in order to avoid the interference caused by the parameter change of the discriminator during the training process of the generator, the parameters of the discriminator are fixed during the training process of the generator. A set of data first completes the training of the generator, adjusts the parameters of the generator, and then completes the training of the discriminator based on this set of data, adjusts the parameters of the discriminator, and then selects the next set of data to train the generator and the discriminator respectively , until the respective losses of the generator and the discriminator no longer decrease.

Preferably,

The generator of the GAN network structure is trained by the training set data until the loss value of the generator no longer declines, and the trained generator includes:

The subsampled image is input into the generator of the GAN network structure to obtain the image generated by the network;

The image generated by the network and the under-sampled image are input into the discriminator of the GAN network structure to obtain the first similarity matrix;

The perceptual loss is obtained through the network generated image and the real image, the L1 loss is obtained through the preset L1 loss function, and the first BCE loss is obtained through the first similarity matrix and the N-order 1 matrix;

The generator loss is obtained by the perceptual loss, the L1 loss, the first BCE loss and the total variation loss, and the generator loss is minimized, and the generator is trained through the training set data until the generator The loss no longer drops, and the trained generator is obtained;

It can be understood that, as shown in the schematic diagram of the training process of the generator shown in Figure 2, the generator uses Unet with residual connections as the backbone network, uses Fourier loss, perceptual loss and L1 loss as the gradient descent target, and the parameters j is the feature representation of layer j in the pre-trained neural network, N is the number of feature layers, any standard gradient-based learning rule can be used for gradient-based updates, we used momentum in our experiments, sampled from an undersampled image Batch m samples {x ⁽¹⁾ , ..., x ^(m) }, and then obtain the corresponding 512 high-precision pictures (real images) {z ⁽¹⁾ , ..., z ^(m) }, First, the undersampled image is input into the generator of the GAN network structure, the generator generates a new network generated image based on the undersampled image, and the new network generated image and the undersampled image are input to the discriminator together, and the discriminator obtains The first similarity matrix of the image generated by the network and the undersampled image, the perceptual loss PerceptualLoss is obtained through the image generated by the network and the real image, and the L1 loss is obtained through the preset L1 loss function. The expression of the L1 loss is as follows:

L1Loss＝|z ⁽ⁱ⁾ -G(x ⁽ⁱ⁾ )|

In the formula, G represents the generator;

Set the real image value to 1, the output is D(G(x ⁽ⁱ⁾ ), that is, the first similarity matrix, and the first BCE loss is obtained through the first similarity matrix and the N-order 1 matrix, and the first BCE The expression of the loss is as follows:

BCE loss＝-1log(D(G(x ⁽ⁱ⁾ )))

In the formula, D represents the discriminator; the generator loss is obtained through the perceptual loss PerceptualLoss, L1 loss and the first BCE loss, and the generator loss expression is as follows:

In the formula, α, β are adjustable weight coefficients; with the goal of minimizing the loss of the generator, the generator is trained using the training set data through the gradient descent method until the loss of the generator no longer decreases, Get a trained generator.

Preferably,

The generator loss obtained through the perceptual loss, L1 loss, first BCE loss and total variation loss includes:

The damping coefficient is introduced, and the damping coefficient of each iteration is calculated according to the preset damping coefficient calculation formula;

The sum of the perceptual loss, L1 loss and total variation loss of each iteration is multiplied by the damping coefficient of this iteration to obtain the damping loss;

adding the damping loss to the first BCE loss of this iteration to obtain the generator loss;

It is understandable that on the basis of the above scheme, the concept of damping coefficient is introduced. The damping coefficient is an important parameter for adjusting the loss value of the entire LP-GAN network structure. The problem of convergence of the loss value is solved by adjusting the value of the damping coefficient , in common physical application scenarios, a fixed damping strategy is often used. If the damping coefficient is set to a larger value, it may cause the algorithm to be difficult to converge due to oscillations. If the damping coefficient is set to a higher value If the value is small, the details of the image generated by the algorithm are poor, and it is easy to reach the local optimum. In this embodiment, the damping coefficient is introduced into the model training process, and the loss value of the overall training is limited to less than 0.1, in order to make up for the lost part The detailed information of the image, highlighting the detailed part of the image, the definition of the damping coefficient is as follows:

In the formula, x represents the number of iterations, that is, during each iteration, the damping coefficient y will change accordingly. When the number of iterations is 100epoch, the damping coefficient y is 0.5; after calculating the damping coefficient y of each iteration, the damping The coefficient y is multiplied by the sum of the total variation loss, L1 loss and perceptual loss of each iteration, and the loss is converged through the damping coefficient y. When the training is started for the first time, the damping coefficient is 0. At this time, only the BCE loss is involved in the calculation. The total contour information of the image can be better obtained. When the gradual damping coefficient becomes larger and larger, and the sum of BCE loss and total variation loss, L1 loss and perceptual loss is calculated, more detailed information inside the image can be obtained. , therefore, after adding the damping coefficient, it can balance the influence of different factors in the processing process, thereby improving the quality of image processing; the application of damping coefficient in the field of image processing is to balance the different factors in the processing process By adjusting the damping coefficient, factors such as smoothness, image quality, segmentation results or sparsity can be controlled to meet the needs of specific tasks and improve the quality of image processing results; the advantages of the model after adding the damping coefficient:

Balance smoothing and detail preservation:

The damping coefficient can help balance the trade-off between smoothing and detail preservation. In image processing tasks, sometimes it is necessary to smooth the image to remove noise or reduce details, and sometimes it is necessary to retain important details. By adjusting the damping coefficient, you can more precisely Control the degree of smoothing, so that smoothing and detail preservation can be better balanced;

Control image quality and visual effects:

The damping coefficient can be used to control the quality and visual effect of the image. In tasks such as image compression and enhancement, the damping coefficient can affect the quality and perception of the result. By adjusting the damping coefficient, you can find the best between compression ratio and image quality balance point, or a good compromise between enhancement and image fidelity;

Control sparsity and reconstruction quality:

In the image processing method based on sparse representation, the damping coefficient can be used to balance the trade-off between sparsity and reconstruction quality. By adjusting the damping coefficient, the stability of the sparse representation and the quality of the reconstructed image can be controlled to obtain better image processing effect;

In summary, the application of damping coefficients in image processing can provide a better trade-off between smoothing and detail preservation, more precise control of image quality and visual effects, improved continuity of segmentation results, and better sparsity and reconstruction. The balance of mass, these advantages make the damping coefficient a valuable tool in the model training process, which can improve the quality and effectiveness of image processing tasks.

Preferably,

The discriminator of the GAN network structure is trained through the training set data until the loss value of the discriminator no longer decreases, and the trained discriminator includes:

Inputting the undersampled image and the real image into the discriminator together to obtain a second similarity matrix;

A second BCE loss is obtained through the first similarity matrix and an N-order 0 matrix, and a third BCE loss is obtained through the second similarity matrix and an N-order 1 matrix;

The discriminator loss is obtained through the second BCE loss and the third BCE loss, and the discriminator loss is minimized, and the discriminator is trained through the training set data until the discriminator loss no longer decreases , get the trained discriminator;

It is understandable that the discriminator uses a simple patchGAN. After the input image undergoes multi-layer convolution, an N×N matrix is obtained instead of the binary classification number of the original GAN, and then BCEloss is obtained with the label. When training the discriminator, it is required to go through the network virtual The NXN matrix and N-order 0 matrix BCEloss obtained by the discriminator for the output image are as small as possible, and the NXN matrix and N-order 1 matrix BCEloss obtained by the label through the discriminator are also as small as possible, as shown in Figure 3, based on the above The same set of data for training the generator, input the undersampled image and the real image into the discriminator together to obtain a second similarity matrix, obtained by the first similarity matrix and the N-order 0 matrix The second BCE loss, the third BCE loss is obtained by the second similarity matrix and the N-order 1 matrix, wherein the second BCE loss sets the real image value to 0, and the output is D(x ⁽ⁱ⁾ ), the second BCE The expression of the loss is as follows:

BCE loss＝-log(1-D(x ⁽ⁱ⁾ ))

The third BCE loss sets the real image value to 1, and the output is D(z ⁽ⁱ⁾ ), the expression of the third BCE loss is as follows:

BCE loss＝-1log(D(z ⁽ⁱ⁾ ))

The second BCE loss is added to the third BCE loss to obtain the discriminator loss, and the expression of the discriminator loss is as follows:

Aiming at the minimum loss of the discriminator, the discriminator is trained by using the training set data using a gradient descent method until the loss of the discriminator no longer decreases, and a trained discriminator is obtained.

Preferably,

The perceptual loss obtained by generating images and real images through the network includes:

The image generated by the network and the real image are respectively input into the feature extraction module of the generator, and the feature extraction module outputs the network feature and the real feature respectively, and obtains the perceptual loss through the network feature and the real feature;

It can be understood that the perceptual loss is through a fixed network (usually using pre-trained VGG16 or VGG19), respectively using the real image (Ground Truth) and the network generated image (Prediciton) output by the generator as its input to obtain the corresponding Output features: feature_gt, feature_pre, and then use feature_gt and feature_pre to construct a loss (usually L2 loss), which approximates the deep information between the real image and the network-generated result, that is, perceptual information. Compared with ordinary L2 loss, it can be enhanced. The details of the output features, (can be simply understood as: the fixed network here is regarded as a function f, feature_gt=f(Ground Truth), feature_pre=f(Prediciton), our purpose is to minimize the difference between feature_gt and feature_pre Difference, that is, minimize the perceptual loss formed by feature_gt and feature_pre;

A fixed network is set (this embodiment uses the pre-trained VGG16 on ImageNet), the network parameters are fixed and do not update;

Using the real image (Ground Truth) and the network-generated image (Prediciton) as its input, the corresponding output features are obtained: feature_gt, feature_pre, using feature_gt and feature_pre to construct the loss, usually not only using a single layer of a fixed network (such as VGG16) to extract features, Instead, it uses the combination of shallow, deeper, and deeper layers in its network structure to extract features and construct losses. For example, the output features of the 3rd, 5th, and 7th convolutional layers of the feature extraction module of VGG16 are accumulated. , the VGG16 feature extraction module structure is shown in Figure 4: In this application, the output of the four activation layers shown in the box is used to construct the perceptual loss, and the expression of the perceptual loss is as follows:

In the formula, F represents the feature extraction module.

Embodiment two

Fig. 5 is a schematic flowchart of a method for applying an image reconstruction model according to another exemplary embodiment, including:

S201. Acquire the undersampled image to be processed, and input the undersampled image to be processed into an image reconstruction model;

S202, the image reconstruction model enhances the internal detail information of the under-sampled image to be processed based on perceptual loss, enhances the edge profile of the under-sampled image to be processed based on L1 loss, and makes the output image based on total variation loss Smooth, output enhanced image;

It can be understood that this application also provides an image reconstruction model application method. After the model is obtained through the above-mentioned model building method, the low-sampled image can be input into the model, and the model is based on perceptual loss enhancement. Describe the internal detail information of the under-sampled image to be processed, enhance the edge profile of the under-sampled image to be processed based on L1 loss, make the output image space smooth based on the total variation loss, and output the enhanced image. The model of this application is used to enhance image, can illuminate arbitrarily oriented structures and improve image reconstruction even when scanning with undersampled data or limited views, as confirmed by qualitative analysis and quantitative evaluation, the solution of the present application can achieve 0.7 The Structural Similarity Index, which highlights the ability of the network to generate higher-resolution photoacoustic images from low-quality data sets, this research method has the potential for medical imaging applications, because it can improve the quality and accuracy of image reconstruction while maximizing Minimize the need for experiments, this method retains the key information in the original data, and shows high adaptability and scalability when dealing with various sparse data types, its practical significance lies in its ability to promote Photoacoustic imaging.

Embodiment three:

Fig. 6 is a system schematic diagram of a device for establishing an image reconstruction model according to another exemplary embodiment, and the device includes:

Data acquisition module 101: used to acquire multiple sets of undersampled images and corresponding real images to obtain training set data;

Network image acquisition module 102: used to select any group of undersampled images and their corresponding real images in the training set data, and input the undersampled images into the GAN network structure to obtain network generated images;

Total variation loss acquisition module 103: used to select any pixel point in the image generated by the network, and obtain two pixel points adjacent to the pixel point in the horizontal direction and vertical direction, and pass the adjacent pixel point in the horizontal direction and vertical direction Calculate the difference value of the two pixels of the pixel, and sum the difference values of all pixels in the image generated by the network to obtain the total variation loss;

Model building module 104: used to represent the loss value of the network-generated image and the real image through L1 loss, perceptual loss, BCE loss and total variation loss, with the minimum loss value of the network-generated image and the real image as the goal, The GAN network structure is trained by the training set data, and the parameters of the GAN network structure are adjusted until the loss value no longer declines, and an image reconstruction model is established;

It can be understood that the present application also provides an apparatus for establishing an image reconstruction model, which is used to execute the method for establishing an image reconstruction model, including: using the data acquisition module 101 to acquire multiple groups of under-sampled images and corresponding The real image of the training set data is obtained; the network image acquisition module 102 is used to select any group of undersampled images and their corresponding real images in the training set data, and the undersampled images are input into the GAN network structure, Obtain the image generated by the network; the total variation loss acquisition module 103 is used to select any pixel point in the image generated by the network, and obtain two pixels adjacent to the pixel point in the horizontal direction and the vertical direction, through the horizontal direction And calculate the difference value of the pixel point for two adjacent pixels in the vertical direction, and sum the difference values of all pixels in the image generated by the network to obtain the total variation loss; the model building module 104 is used to pass L1 Loss, perceptual loss, BCE loss and total variation loss represent the loss value of the network-generated image and the real image, with the minimum loss value between the network-generated image and the real image as the goal, through the training set data to the GAN network The structure is trained, and the parameters of the GAN network structure are adjusted until the loss value no longer decreases, and an image reconstruction model is established.

Embodiment four

Fig. 7 is a system schematic diagram of an image reconstruction model application device according to another exemplary embodiment, and the device includes:

Input module 201: used to acquire the under-sampled image to be processed, and input the under-sampled image to be processed into an image reconstruction model;

Output module 202: used for the image reconstruction model to enhance the internal detail information of the under-sampled image to be processed based on perceptual loss, enhance the edge profile of the under-sampled image to be processed based on L1 loss, and based on the total variation The loss smoothes the output image and outputs an enhanced image;

It can be understood that the present application also provides an image reconstruction model application device, which is used to execute the image reconstruction model application method, including: using the input module 201 to obtain the under-sampled image to be processed , input the undersampled image to be processed into the image reconstruction model; use the output module 202 to use the image reconstruction model to enhance the internal detail information of the undersampled image to be processed based on perceptual loss, based on L1 loss The edge profile of the under-sampled image to be processed is enhanced, and the output image is smoothed based on the total variation loss, and the enhanced image is output.

It can be understood that, the same or similar parts in the above embodiments can be referred to each other, and the content that is not described in detail in some embodiments can be referred to the same or similar content in other embodiments.

It should be noted that, in the description of the present invention, terms such as "first" and "second" are only used for description purposes, and should not be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" means at least two.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which shall It is understood by those skilled in the art to which the embodiments of the present invention pertain.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. During execution, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (10)

1. A method for building an image reconstruction model, the method comprising:

acquiring a plurality of groups of undersampled images and corresponding real images to obtain training set data;

selecting any group of undersampled images and corresponding real images in the training set data, and inputting the undersampled images into a GAN network structure to obtain network generated images;

selecting any pixel point from the network generated image, acquiring two pixel points adjacent to the pixel point in the horizontal direction and the vertical direction, calculating the difference value of the pixel point through the two pixel points adjacent to the pixel point in the horizontal direction and the vertical direction, and summing the difference values of all the pixel points in the network generated image to obtain total variation loss;

and representing the loss values of the network generated image and the real image through L1 loss, perception loss, BCE loss and total variation loss, taking the minimum loss value of the network generated image and the real image as a target, training the GAN network structure through the training set data, and adjusting parameters of the GAN network structure until the loss value is not reduced any more, and establishing an image reconstruction model.

2. The method of claim 1, wherein the step of determining the position of the substrate comprises,

training the GAN network structure through the training set data until the loss value no longer decreases, and establishing the image reconstruction model includes:

training the generator of the GAN network structure through the training set data until the loss value of the generator is no longer reduced, so as to obtain a trained generator;

training the discriminators of the GAN network structure through the training set data until the loss value of the discriminators is no longer reduced, so as to obtain trained discriminators;

and establishing an image reconstruction model through the trained generator and the discriminator.

3. The method of claim 2, wherein the step of determining the position of the substrate comprises,

in the image reconstruction model building process, the generator and the discriminator perform training alternately, when the generator is trained, the parameters of the discriminator are fixed, and when the discriminator is trained, the parameters of the generator are fixed.

4. The method of claim 2, wherein the step of determining the position of the substrate comprises,

training the generator of the GAN network structure through the training set data until the loss value of the generator is no longer reduced, and obtaining the trained generator includes:

Inputting the undersampled image into a generator of a GAN network structure to obtain a network generated image;

the network generated image and the undersampled image are input into a discriminator of the GAN network structure together to obtain a first similarity matrix;

generating an image and a real image through the network to obtain perceived loss, obtaining L1 loss through a preset L1 loss function, and obtaining first BCE loss through the first similarity matrix and an N-order 1 matrix;

and obtaining generator loss through the perception loss, the L1 loss, the first BCE loss and the total variation loss, and training the generator through the training set data by taking the minimum generator loss as a target until the generator loss is not reduced any more, so as to obtain a trained generator.

5. The method of claim 4, wherein the step of determining the position of the first electrode is performed,

the deriving the generator penalty from the perceptual penalty, L1 penalty, first BCE penalty, and total variation penalty comprises:

introducing a damping coefficient, and calculating the damping coefficient of each iteration according to a preset damping coefficient calculation formula;

multiplying the sum of the perception loss, the L1 loss and the total variation loss of each iteration by the damping coefficient of the iteration to obtain damping loss;

And adding the damping loss to the first BCE loss of the iteration to obtain a generator loss.

6. The method of claim 5, wherein the step of determining the position of the probe is performed,

training the discriminators of the GAN network structure through the training set data until the loss value of the discriminators is no longer reduced, and obtaining the trained discriminators comprises the following steps:

inputting the undersampled image and the real image into the discriminator together to obtain a second similarity matrix;

obtaining a second BCE loss through the first similarity matrix and an N-order 0 matrix, and obtaining a third BCE loss through the second similarity matrix and an N-order 1 matrix;

and obtaining a discriminator loss through the second BCE loss and the third BCE loss, and training the discriminator through the training set data by taking the minimum discriminator loss as a target until the discriminator loss is not reduced any more, so as to obtain a trained discriminator.

7. The method of claim 4, wherein the step of determining the position of the first electrode is performed,

the generating the image through the network and obtaining the perceived loss of the real image comprise:

and respectively inputting the network generated image and the real image into a feature extraction module of the generator, wherein the feature extraction module respectively outputs network features and real features, and obtains perception loss through the network features and the real features.

8. A method of applying an image reconstruction model, the method being based on an image reconstruction model established according to any one of claims 1-6, the method comprising:

acquiring the undersampled image to be processed, and inputting the undersampled image to be processed into an image reconstruction model;

the image reconstruction model enhances internal detail information of the undersampled image to be processed based on a perceptual loss, enhances an edge contour of the undersampled image to be processed based on an L1 loss, and smoothes an output image based on a total variation loss, outputting an enhanced image.

9. An image reconstruction model building apparatus, characterized in that the apparatus comprises:

and a data acquisition module: the method comprises the steps of obtaining a plurality of groups of undersampled images and corresponding real images to obtain training set data;

a network image acquisition module: the method comprises the steps of selecting any group of undersampled images and corresponding real images in training set data, inputting the undersampled images into a GAN network structure, and obtaining network generated images;

total variation loss acquisition module: the method comprises the steps of selecting any one pixel point from a network generated image, obtaining two pixel points adjacent to the pixel point in the horizontal direction and the vertical direction, calculating the difference value of the pixel point through the two pixel points adjacent to the pixel point in the horizontal direction and the vertical direction, and summing the difference values of all the pixel points in the network generated image to obtain total variation loss;

And a model building module: the method is used for representing the loss values of the network generated image and the real image through L1 loss, perception loss, BCE loss and total variation loss, taking the minimum loss values of the network generated image and the real image as targets, training the GAN network structure through the training set data, and adjusting parameters of the GAN network structure until the loss values are not reduced any more, and establishing an image reconstruction model.

10. An image reconstruction model application apparatus, the apparatus comprising:

an input module: the method comprises the steps of acquiring an undersampled image to be processed, and inputting the undersampled image to be processed into an image reconstruction model;

and an output module: the image reconstruction model enhances internal detail information of the undersampled image to be processed based on a perceptual loss, enhances an edge contour of the undersampled image to be processed based on an L1 loss, and smoothes an output image based on a total variation loss, outputting an enhanced image.