WO2025152229A1 - Scanning light field self-supervised network reconstruction method and apparatus, electronic device, and medium - Google Patents

Abstract

The present application relates to the technical field of computational imaging, and in particular to a scanning light field self-supervised network reconstruction method and apparatus, an electronic device, and a medium. The method comprises: acquiring scanning light field data and a point spread function of an optical system capturing the scanning light field data; preprocessing the scanning light field data to obtain preprocessed data; and constructing a self-supervised reconstruction network on the basis of the preprocessed data and the point spread function until a preset iteration stop condition is reached, to obtain a scanning light field self-supervised network reconstruction result. Thus, the present application solves the problems in the related art that the need to provide high-resolution ground truth images leads to increased costs, longer times consumed, and reduced reconstruction efficiency, and the existence of serious reconstruction artifacts at the original object plane easily leads to loss or distortion of details on object surfaces and reduced quality and accuracy of reconstruction results.

Description

Scanning light field self-supervised network reconstruction method, device, electronic device and medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with application number 202410074899.5 and application date January 18, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference.

Technical Field

The present application relates to the field of computational imaging technology, and in particular to a method, device, electronic device and medium for self-supervised network reconstruction of a scanned light field.

Background Art

Light fields or scanned light fields have attracted much attention because they can achieve large-scale rapid 3D imaging with a single or a small number of captured images. They are widely used in biological applications including 3D calcium imaging, among which reconstruction is an important step in scanning light field imaging and one of the main factors affecting image quality. In related technologies, the 3D structure can be restored from the captured image according to the point spread function and light propagation model of the optical system through traditional deconvolution methods based on the light propagation model, or through network reconstruction methods based on deep learning, using light field or scanned light field data as input and high-resolution true value images as supervision, so that the network can learn the reconstruction process.

However, in the related art, due to the need to provide high-resolution true-value images, other microscopic methods need to be used to photograph the same set of samples, which increases costs, takes a long time, and reduces reconstruction efficiency. In addition, due to the existence of serious reconstruction artifacts at the original object plane and other problems, it is easy to cause the details of the object surface to be lost or distorted, thereby affecting the quality and accuracy of the reconstruction results.

Summary of the invention

The present application provides a scanning light field self-supervised network reconstruction method, device, electronic device and storage medium to solve the problems in the related art that the need to provide a high-resolution true value image leads to increased costs, long time consumption, reduced reconstruction efficiency, and the presence of serious reconstruction artifacts at the original object plane, which easily leads to loss or distortion of details on the object surface, reducing the quality and accuracy of the reconstruction result.

A first aspect of the present application provides a method for self-supervised network reconstruction of a scanned light field, comprising the following steps: acquiring scanned light field data and a point spread function of an optical system that captures the scanned light field data; preprocessing the scanned light field data to obtain preprocessed data; and constructing a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result.

Optionally, in one embodiment of the present application, the scanned light field data is preprocessed to obtain preprocessed data, including: rearranging the pixel points of the scanned light field data based on the position behind the microlens of the optical system to generate multi-angle data, wherein all pixel points at the same position behind the microlens are arranged in sequence according to spatial order.

Optionally, in one embodiment of the present application, preprocessing the scanned light field data to obtain preprocessed data further includes: performing a linear transformation on the multi-angle data so that the pixel values are within a preset range to obtain changed data.

Optionally, in one embodiment of the present application, a self-supervised reconstruction network is constructed based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result, including: selecting any image processing network, inputting multi-angle data and forwarding it to obtain a network output; randomly selecting several angles of the point spread function, and forward projecting the network output.

Optionally, in one embodiment of the present application, a self-supervised reconstruction network is constructed based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result, and also includes: calculating the mean square error between the network output and the corresponding angle of the forward projection; calculating the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output; weighting the mean square error, the second norm and the axial continuity constraint, calculating the self-supervised loss function of the self-supervised reconstruction network, and transmitting back to update the network parameters to obtain the scanned light field self-supervised network reconstruction result.

A second aspect of the present application provides a scanning light field self-supervised network reconstruction device, including: an acquisition module, used to acquire scanning light field data and a point spread function of an optical system that shoots the scanning light field data; a processing module, used to preprocess the scanning light field data to obtain preprocessed data; and a reconstruction module, used to construct a self-supervised reconstruction network according to the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanning light field self-supervised network reconstruction result.

Optionally, in one embodiment of the present application, the processing module includes: a generation unit, used to rearrange the pixel points of the scanned light field data based on the position behind the microlens of the optical system to generate multi-angle data, wherein all pixel points at the same position behind the microlens are arranged in sequence according to spatial order.

Optionally, in one embodiment of the present application, the processing module further includes: a transformation unit, configured to perform a linear transformation on the multi-angle data so that the pixel value is within a preset range to obtain changed data.

Optionally, in one embodiment of the present application, the reconstruction module includes: a forward transmission unit, used to select any image processing network, input the multi-angle data and forward transmit it to obtain the network output; a projection unit, used to randomly select several angles of the point spread function and obtain a forward projection of the network output.

Optionally, in one embodiment of the present application, the reconstruction module further includes: a first calculation unit, used to calculate the mean square error between the network output and the corresponding angle of the forward projection; a second calculation unit, used to calculate the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output; a weighting unit, used to weight the mean square error, the second norm and the axial continuity constraint to calculate the self-supervised reconstruction network. The loss function is then fed back to update the network parameters to obtain the scanned light field self-supervised network reconstruction result.

The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scanning light field self-supervised network reconstruction method as described in the above embodiment.

The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned scanning light field self-supervised network reconstruction method.

The embodiment of the present application can pre-process the scanned light field data by obtaining the scanned light field data and the point spread function of the optical system that shoots the scanned light field data, and construct a self-supervised reconstruction network based on the pre-processed data and the point spread function until the preset iteration stop condition is reached, thereby obtaining the scanned light field self-supervised network reconstruction result, which can be continuously calibrated and adjusted to improve the accuracy of the data, thereby improving the stability and reliability of the system, and reducing the need for manual intervention, avoiding the influence of subjective factors and operational errors. Thus, the problems in the related art that the need to provide a high-resolution true value image leads to increased costs, longer time consumption, and reduced reconstruction efficiency, and that the existence of serious reconstruction artifacts at the original object plane easily leads to loss or distortion of details on the object surface, reducing the quality and accuracy of the reconstruction results, etc.

Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

FIG1 is a schematic diagram of the structure of a scanning light field self-supervised network reconstruction method according to an embodiment of the present application;

FIG2 is a flow chart of a scanning light field self-supervised network reconstruction method provided according to an embodiment of the present application;

FIG3 is a schematic diagram showing a comparison between an original image of a scanned light field and an image reconstructed through a network according to an embodiment of the present application;

FIG4 is a schematic diagram of the principle of random multi-angle forward transmission and self-supervised loss function calculation process according to an embodiment of the present application;

FIG5 is a schematic diagram of the structure of a scanning light field self-supervisory network reconstruction device provided according to an embodiment of the present application;

FIG6 is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application.

DETAILED DESCRIPTION

The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described are exemplary and are intended to be used to explain the present application, but should not be construed as limiting the present application.

The following describes the scanning light field self-supervised network reconstruction method, device, electronic device and storage medium of the embodiment of the present application with reference to the accompanying drawings. In view of the related technologies mentioned in the background technology center, the need to provide high-resolution true value images leads to increased costs, longer time consumption, and reduced reconstruction efficiency. In addition, due to the existence of serious reconstruction artifacts at the original object plane, it is easy to cause the details of the object surface to be lost or distorted, thereby reducing the quality and accuracy of the reconstruction results. The present application provides a scanning light field self-supervised network reconstruction method, in which the scanning light field data can be pre-processed by obtaining the scanning light field data and the point spread function of the optical system that shoots the scanning light field data, and a self-supervised reconstruction network is constructed based on the pre-processed data and the point spread function until the preset iteration stop condition is reached, thereby obtaining the scanning light field self-supervised network reconstruction result, which can be continuously calibrated and adjusted to improve the accuracy of the data, thereby improving the stability and reliability of the system, and reducing the need for manual intervention, avoiding the influence of subjective factors and operational errors. Thereby, the problems in the related technology of the need to provide high-resolution true-value images, which increases costs, takes a long time, reduces reconstruction efficiency, and the existence of serious reconstruction artifacts at the original object plane, which easily leads to loss or distortion of details on the object surface, thereby reducing the quality and accuracy of the reconstruction results, are solved.

Before explaining the scanning light field self-supervised network reconstruction method provided in the embodiment of the present application, the structure of the scanning light field self-supervised network reconstruction method involved in the embodiment of the present application is first explained.

As shown in FIG1 , the structure of the scanning light field self-supervised network reconstruction method includes: a scanning light field data acquisition unit, a data preprocessing unit, a self-supervised reconstruction network training unit and a self-supervised reconstruction network testing unit.

Among them, a scanning light field data unit is obtained to provide data for training and testing of the network, and a point spread function of the optical system that captured the data is passed to a random multi-angle forward transmission module for self-supervised reconstruction network training, wherein the data and the corresponding point spread function can be captured by a scanning optical microscope or downloaded from a public dataset;

The data preprocessing unit includes data rearrangement function, data normalization function and data amplification function.

Among them, the data rearrangement function is used to rearrange the pixel points of the scanned light field data according to the position behind the microlens, and all the pixel points at the same position behind the microlens are arranged in sequence according to the spatial order to form 3D multi-angle data; the data normalization function is used to perform linear transformation on the multi-angle data so that the pixel value is between 0 and 1 to obtain normalized data; the data augmentation function is used to perform random angle rotation, flipping, cropping and other operations on the normalized data to obtain the final data after augmentation, and the final data will be used for self-supervised reconstruction network training or self-supervised reconstruction network testing;

The self-supervised reconstruction network training unit includes the network input forward transmission function, the random multi-angle forward transmission function and the self-supervised loss function return transmission function. Completing the above three steps is called an iteration.

Among them, the network input forward transmission function receives the final data output by the data preprocessing unit through the input layer, passes it through the network, and obtains the network output through the output layer; the random multi-angle forward transmission function is used to randomly select several angles of the point spread function, convolve the point spread function of the corresponding angle with the network output to obtain the forward projection, and then The size of the forward projection is adjusted according to the scanning magnification and the number of pixels behind the microlens; the self-supervised loss function feedback function is used to first calculate the value of the self-supervised loss function, and then perform gradient feedback to update the network parameters, where the self-supervised loss function value is formed by the mean square error between the adjusted forward projection and the corresponding angle of the network input, the second norm of the second-order derivative of the network output, and the axial continuity constraint of the network output. Usually, a threshold of the number of iterations is set. When it does not exceed this value, it is considered that the network training has not ended. When the network training has not ended, it enters the next iteration, that is, the next batch of data is forwarded to the network input, forwarded at random multiple angles, and back-transmitted to the self-supervised loss function;

The self-supervised reconstruction network testing unit is performed after the self-supervised reconstruction network training is completed, and is used to test the final performance of the selected data on the network.

Among them, the selected data often does not overlap with the data used by the self-supervised reconstruction network training unit, and the size is not necessarily the same. If the size is different, the selected data should be cropped with overlap and cropped into several images that meet the size requirements of the network input layer. Let the network predict separately and then splice the prediction results.

Next, the scanning light field self-supervised network reconstruction method of the embodiment of the present application is described in detail.

Specifically, FIG2 is a flow chart of a scanning light field self-supervised network reconstruction method provided in an embodiment of the present application.

As shown in FIG2 , the scanned light field self-supervised network reconstruction method includes the following steps:

In step S201 , scanning light field data and a point spread function of an optical system for capturing the scanning light field data are obtained.

It can be understood that the light field refers to the changing distribution of information such as light intensity, direction and phase in space and time, and the scanned light field data refers to a series of images or measurements obtained by sampling the light field from different angles and positions.

Specifically, the embodiments of the present application can obtain scanning light field data and the point spread function of the optical system that captures the scanning light field data through a scanning optical microscope. For example, through a scanning optical microscope, image information of cells at different angles and positions can be obtained, including light source brightness, light direction, etc., and the point spread function of the optical system that captures the scanning light field data can be obtained through calculation and measurement.

The embodiment of the present application improves the comprehensiveness of image information and provides a data basis for subsequent processing by acquiring scanning light field data and a point spread function of an optical system that captures the scanning light field data.

In step S202, the scanned light field data is preprocessed to obtain preprocessed data.

It can be understood that preprocessing includes data rearrangement, data normalization, data amplification, etc.

Specifically, the embodiment of the present application can rearrange the pixel points of the scanned light field data according to their positions behind the microlens, and all the pixel points at the same position behind the microlens are arranged in sequence according to the spatial order to form 3D multi-angle data. The multi-angle data can be linearly transformed so that the pixel value is between 0 and 1 to obtain normalized data. The normalized data can be subjected to random angle rotation, flipping, cropping and other operations to obtain the final data after amplification.

The embodiment of the present application can pre-process the scanned light field data, such as data rearrangement, data normalization, data By performing amplification and other processing, the preprocessed data is obtained, thereby improving the precision and accuracy of the data, improving the clarity of the image, and further improving the authenticity of the reconstruction results.

Optionally, in one embodiment of the present application, the scanned light field data is preprocessed to obtain preprocessed data, including: rearranging the pixel points of the scanned light field data based on the position behind the microlens of the optical system to generate multi-angle data, wherein all pixel points at the same position behind the microlens are arranged in sequence according to spatial order.

Specifically, the embodiment of the present application can rearrange the pixel points of the scanned light field data based on the position behind the microlens of the optical system, and all the pixel points at the same position behind the microlens are arranged in sequence in spatial order, thereby forming 3D multi-angle data. For example, the scanned light field data includes 5 viewing angles scanned from left to right in the horizontal direction, each viewing angle has 10×10 pixels, and for the first viewing angle, after rearrangement, the 10 pixel points in the first row will correspond to a specific spatial position. Similarly, the 10 pixel points in the second row will correspond to the next spatial position, and so on. Subsequently, for other viewing angles, the pixel points will be arranged in sequence according to the same spatial position.

The embodiment of the present application can rearrange the pixel points of the scanned light field data based on the position behind the microlens of the optical system to generate multi-angle data, and can obtain 3D multi-angle data based on the rearrangement of the position behind the microlens, thereby providing more comprehensive and accurate information, reducing artifacts, improving detail restoration, and ensuring the accuracy of the reconstruction results.

Optionally, in one embodiment of the present application, preprocessing the scanned light field data to obtain preprocessed data further includes: performing a linear transformation on the multi-angle data so that the pixel values are within a preset range to obtain changed data.

It can be understood that the preset range refers to a preset pixel value range of data, such as scaling the pixel value to between 0 and 1.

Specifically, the embodiments of the present application can perform a linear transformation on multi-angle data so that the pixel value is between a preset range to obtain the changed data. For example, if the pixel value needs to be scaled to between 0 and 1, a linear transformation can be performed to map it from the current pixel value range to the target range to obtain the changed data.

The embodiment of the present application performs a linear transformation on multi-angle data so that the pixel value is within a preset range to obtain the changed data, which can effectively improve the contrast and display effect of the image, help identify and analyze the details in the image, and reduce visual inconsistency.

In step S203, a self-supervised reconstruction network is constructed according to the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result.

It can be understood that the preset iteration stop condition may be that the number of iterations reaches a preset iteration threshold, such as the number of iterations reaches 50,000 times.

Specifically, the embodiment of the present application can use the preprocessed data for supervised reconstruction network training or supervised reconstruction network testing, wherein the self-supervised reconstruction network training includes network input forward transmission, random multi-angle forward transmission and self-supervised loss function back transmission. Completion of the above three steps is called an iteration. When the network training is not completed, it enters the next iteration. That is, the next batch of data is forwarded through the network, randomly transmitted at multiple angles, and transmitted back through the self-supervised loss function; the self-supervised reconstruction network test is performed after the self-supervised reconstruction network training is completed, and is used to test the final performance of the selected data on the network.

For example, in combination with what is shown in FIG3 , an embodiment of the present application can construct a self-supervised reconstruction network based on the preprocessed data and the point spread function until a preset iteration stop condition is reached, thereby obtaining a scanned light field self-supervised network reconstruction result, wherein the image reconstructed by the network is a 3D image, but is displayed as a projection along the z-axis. Specifically, the embodiment of the present application can use a scanning light field instrument (scanning magnification is 3, the number of pixels behind the microlens is 13×13) to shoot zebrafish embryo data, and then rearrange, normalize and amplify the data to generate 1,000 multi-angle images with a size of 60×60×169 for self-supervised reconstruction network training; use the Keras deep learning framework based on Tensorflow and the Python programming language to build a self-supervised reconstruction network. Specifically, after the input layer, bilinear interpolation is used to adjust the network input to the target size, and then pass through a U-Net, and further, through the training network, wherein the initial learning rate is 1× ^10-4 , the training batch size is 3, and the Adam optimizer is used for back-propagation iterative optimization, and a total of 50,000 iterations are trained, and the learning rate is reduced to half of the original every 10,000 iterations, so that the multi-angle test image can be input into the trained self-supervised reconstruction network to obtain a reconstructed image.

The embodiments of the present application can utilize preprocessed data and point spread functions to gradually improve the ability to reconstruct light field images during the training process. Through iterative training, the adaptability and generalization ability to different samples and scenes can be improved, thereby improving the accuracy and efficiency of reconstruction.

Optionally, in one embodiment of the present application, a self-supervised reconstruction network is constructed based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result, including: selecting any image processing network, inputting multi-angle data and forwarding it to obtain a network output; randomly selecting several angles of the point spread function, and forward projecting the network output.

During the actual implementation process, the embodiment of the present application can randomly select two point spread function angles for forward projection, convolve the point spread function of the corresponding angle with the network output to obtain the forward projection, and can adjust the size of the forward projection according to the scanning magnification and the number of pixels behind the microlens.

Specifically, in conjunction with FIG. 4 , the embodiment of the present application adds a random multi-angle forward transmission module and uses the information of the point spread function to simulate the optical forward transmission process of the network output. The specific steps are as follows:

Step S1: Randomly select an angle.

It can be understood that the core of the light field or scanning light field microscopy system is the microlens array in front of the camera. There are Nnum×Nnum camera photosensitive pixels behind each microlens, also known as angles. Because pixels at different positions correspond to different angles, and the angle information at the edge of the microlens is small and the light intensity is low, only the angles within the circle with the microlens optical center as the center and a radius of r are used, where r is generally one or two pixels less than half of Nnum. It can also be specified based on computing resources and actual needs, and N angles are selected from the angles within the circle for subsequent calculations. The selection method can be as follows:

(1) N angles are randomly selected in each iteration;

(2) Each iteration has M fixed angles that do not change with the iteration, and N-M angles are randomly selected, and they do not overlap with the fixed M angles. For example, M = 1, the angle where the optical center of the microlens is fixed, and N-1 angles other than the angle where the optical center of the microlens is randomly selected;

(3) N-M angles are randomly selected in each iteration, and the other M angles are randomly selected from the N angles selected in the previous iteration, and they are guaranteed not to overlap with the N-M angles selected in this iteration;

(4) Divide the circle formed by arranging all the selectable angles into N equal parts, and randomly select an angle from each of the N parts in each iteration;

It is worth noting that the above selection method may cause slightly different network convergence speeds in different embodiments, but the final effect is similar in theory, and the selection of some angles for calculation is to save memory and video memory overhead and avoid memory or video memory overflow during network training. After a sufficient number of training iterations, due to the randomness of angle selection, the network optimization result should theoretically be the same as the result of selecting all angles for calculation each time.

Step S2: Convolve the point spread function of the corresponding angle with the network output.

Among them, for each angle selected in step S1, the part of the point spread function corresponding to the angle, that is, a 3D vector, is convolved with the 3D network output to obtain the forward projection. In the actual execution process, the frequency domain operation can be used instead of the spatial domain convolution, which can be shown as follows:
Proj _i =iFFTshift{iFFT[FFTshift(FFT(Output))·FFTshift(FFT(PSF _i ))]}

Among them, Proj _i is the i-th angular component of the forward projection, which is a 2D image; PSF _i is the i-th angular component of the point spread function; Output is the network output; FFT is the fast Fourier transform; FFTshift is the operation of moving the spectrum so that the zero frequency is in the center; iFFT is the inverse operation of FFT; iFFTshift is the inverse operation of FFTshift.

Furthermore, Proj ₁ , ..., Proj _N can be concatenated into a 3D vector Proj_tmp to obtain a forward projection of N angles.

Step S3: adjusting the size of the forward projection according to the scanning magnification and the number of pixels behind the microlens.

Specifically, the relationship between the adjusted forward projection and the forward projection obtained by splicing in step S2 can be determined by the following expression:
Proj＝ImResize(Proj_tmp,scanning/Nnum)

Among them, Proj is the adjusted forward projection; Proj_tmp is the forward projection obtained by splicing in step S2; ImResize(Img,scale) is a method of adjusting the image size, such as bilinear interpolation, nearest neighbor interpolation, etc., which adjusts Img to scale times the original size; scanning is the scanning magnification; Nnum is the number of pixels in one column or one row behind the microlens.

It is worth noting that in step S1 to step S3, the embodiment of the present application completes the optical forward transmission simulation of the network output Output. If the part corresponding to N angles is selected from the network input and recorded as Input, then Input and Proj have the same size.

The embodiment of the present application randomly selects the point spread function angle for forward projection, and adjusts the size of the forward projection according to the scanning magnification and the number of pixels behind the microlens, which can improve the comprehensiveness and diversity of the projection results, thereby improving the applicability of the scanning light field technology.

Optionally, in one embodiment of the present application, a self-supervised reconstruction network is constructed based on the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result, and also includes: calculating the mean square error between the network output and the corresponding angle of the forward projection; calculating the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output; weighting the mean square error, the second norm and the axial continuity constraint, calculating the self-supervised loss function of the self-supervised reconstruction network, and transmitting back to update the network parameters to obtain the scanned light field self-supervised network reconstruction result.

Specifically, in combination with FIG4 , the embodiment of the present application can calculate the random multi-angle forward transmission and self-supervised loss function, and the specific steps are as follows:

Step S4: Calculate the self-supervised loss function.

Specifically, the function value can be determined by the following expression:
L(Proj,Input,Output)=αMSE(Proj,Input)+βHess(Output)+γCont(Output)

Among them, Proj is the adjusted forward projection, Input is the part of the network input corresponding to the angle, Output is the network output, L is the self-supervised loss function, MSE is the mean square error between the forward projection and the network input, Hess is the second norm of the second-order derivative of the network output, Cont is the axial continuity constraint of the network output, α is the weight of the mean square error, β is the weight of the second norm of the second-order derivative, and γ is the weight of the continuity constraint.

Among them, the axial continuity constraint of the network output can be determined by the following expression:

Among them, Output is the network output, which is a three-dimensional vector, the third dimension represents the axial direction, Cont is the axial continuity constraint of the network output, N is the number of axial pixels output by the network, and Sum(·) is the sum of each component of the two-dimensional vector.

The embodiments of the present application can quantify the error between the network reconstruction result and the real light field data by calculating the mean square error between the network output and the forward projection, and can gradually reduce the mean square error by optimizing the self-supervised loss function, thereby improving the accuracy of the reconstruction result. By considering the second norm of the second-order derivative of the network output and the axial continuity constraint, it helps to suppress noise and artifacts in the reconstruction result and improve the smoothness and stability of the reconstruction result.

According to the scanning light field self-supervised network reconstruction method proposed in the embodiment of the present application, the scanning light field data can be pre-processed by obtaining the scanning light field data and the point spread function of the optical system that shoots the scanning light field data, and a self-supervised reconstruction network is constructed according to the pre-processed data and the point spread function until a preset iteration stop condition is reached, thereby obtaining the scanning light field self-supervised network reconstruction result, which can be continuously calibrated and adjusted to improve the accuracy of the data, thereby improving The stability and reliability of the system are improved, and the need for manual intervention is reduced, avoiding the influence of subjective factors and operational errors. This solves the problems in related technologies, such as the need to provide high-resolution true-value images, which increases costs, takes a long time, reduces reconstruction efficiency, and the existence of serious reconstruction artifacts at the original object plane, which easily leads to loss or distortion of object surface details, reducing the quality and accuracy of the reconstruction results.

Next, the scanning light field self-supervised network reconstruction device proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

FIG5 is a schematic diagram of the structure of a scanning light field self-supervisory network reconstruction device according to an embodiment of the present application.

As shown in FIG. 5 , the scanning light field self-supervised network reconstruction device 10 includes: an acquisition module 100 , a processing module 200 and a reconstruction module 300 .

Specifically, the acquisition module 100 is used to acquire the scanning light field data and the point spread function of the optical system that captures the scanning light field data.

The processing module 200 is used to pre-process the scanned light field data to obtain pre-processed data.

The reconstruction module 300 is used to construct a self-supervised reconstruction network according to the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result.

Optionally, in one embodiment of the present application, the processing module 200 includes: a generating unit.

The generating unit is used to rearrange the pixel points of the scanned light field data based on the position behind the microlens of the optical system to generate multi-angle data, wherein all the pixel points at the same position behind the microlens are arranged in sequence according to the spatial order.

Optionally, in one embodiment of the present application, the processing module 200 further includes: a transformation unit.

The transformation unit is used to perform linear transformation on the multi-angle data so that the pixel value is within a preset range to obtain the changed data.

Optionally, in one embodiment of the present application, the reconstruction module 300 includes: a forward transmission unit and a projection unit.

The forward transmission unit is used to select any image processing network, input multi-angle data and forward transmit it to obtain network output;

The projection unit is used to randomly select several angles of the point spread function and obtain the forward projection of the network output.

Optionally, in one embodiment of the present application, the reconstruction module 300 further includes: a first calculation unit, a second calculation unit and a weighting unit.

Wherein, the first calculation unit is used to calculate the mean square error between the network output and the corresponding angle of the forward projection;

A second calculation unit is used to calculate the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output;

The weighting unit is used to weight the mean square error, the second norm and the axial continuity constraint, calculate the self-supervised loss function of the self-supervised reconstruction network, and transmit back to update the network parameters to obtain the scanned light field self-supervised network reconstruction result.

It should be noted that the above explanation of the embodiment of the scanning light field self-supervised network reconstruction method is also applicable to this embodiment. The scanning light field self-supervised network reconstruction device of the embodiment will not be described in detail here.

According to the scanning light field self-supervised network reconstruction device proposed in the embodiment of the present application, the scanning light field data can be pre-processed by obtaining the scanning light field data and the point spread function of the optical system that shoots the scanning light field data, and a self-supervised reconstruction network is constructed according to the pre-processed data and the point spread function until the preset iteration stop condition is reached, thereby obtaining the scanning light field self-supervised network reconstruction result, which can be continuously calibrated and adjusted to improve the accuracy of the data, thereby improving the stability and reliability of the system, and reducing the need for manual intervention, avoiding the influence of subjective factors and operational errors. Thus, the problems in the related art that the need to provide a high-resolution true value image leads to increased costs, longer time consumption, and reduced reconstruction efficiency, and that the existence of serious reconstruction artifacts at the original object plane easily leads to loss or distortion of details on the object surface, reducing the quality and accuracy of the reconstruction results, etc.

FIG6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

When the processor 602 executes the program, the scanning light field self-supervised network reconstruction method provided in the above embodiment is implemented.

Furthermore, the electronic device further comprises:

The communication interface 603 is used for communication between the memory 601 and the processor 602 .

The memory 601 is used to store computer programs that can be executed on the processor 602 .

Memory 601 may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk storage.

If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in FIG6, but it does not mean that there is only one bus or one type of bus.

Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

Processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

The present application also provides a computer-readable storage medium on which a computer program is stored. The above scanned light field self-supervised network reconstruction method is implemented when the processor is executed.

In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

The logic and/or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

It should be understood that various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the embodiment, the N steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it may be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims (12)

A scanning light field self-supervised network reconstruction method, characterized by comprising the following steps: Acquire scanning light field data and a point spread function of an optical system that photographs the scanning light field data; Preprocessing the scanned light field data to obtain preprocessed data; and A self-supervised reconstruction network is constructed according to the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result. The scanning light field self-supervised network reconstruction method according to claim 1 is characterized in that the preprocessing of the scanning light field data to obtain the preprocessed data comprises: The pixels of the scanned light field data are rearranged based on the positions behind the microlens of the optical system to generate multi-angle data, wherein all the pixels at the same position behind the microlens are arranged in sequence according to a spatial order. The scanned light field self-supervised network reconstruction method according to claim 2 is characterized in that the preprocessing of the scanned light field data to obtain preprocessed data further comprises: A linear transformation is performed on the multi-angle data so that the pixel value is within a preset range to obtain the transformed data. The scanning light field self-supervised network reconstruction method according to claim 2 is characterized in that the self-supervised reconstruction network is constructed according to the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain the scanning light field self-supervised network reconstruction result, comprising: Select any image processing network, input the multi-angle data and forward it to obtain network output; Several angles of the point spread function are randomly selected and forward projected onto the network output. The scanning light field self-supervised network reconstruction method according to claim 4 is characterized in that the self-supervised reconstruction network is constructed according to the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain the scanning light field self-supervised network reconstruction result, and further comprises: Calculating the mean square error between the network output and the corresponding angle of the forward projection; Calculating the second norm of the second derivative of the network output and the axial continuity constraint of the network output; The mean square error, the second norm and the axial continuity constraint are weighted, the self-supervised loss function of the self-supervised reconstruction network is calculated, and the updated network parameters are fed back to obtain the scanned light field self-supervised network reconstruction result. A scanning light field self-supervised network reconstruction device, characterized by comprising: An acquisition module, used for acquiring scanning light field data and a point spread function of an optical system for photographing the scanning light field data; a processing module, used for preprocessing the scanned light field data to obtain preprocessed data; and The reconstruction module is used to construct a self-supervised reconstruction network according to the preprocessed data and the point spread function until a preset iteration stop condition is reached to obtain a scanned light field self-supervised network reconstruction result. The scanning light field self-supervised network reconstruction device according to claim 6, characterized in that the processing module comprises: A generating unit is used to rearrange the pixel points of the scanned light field data based on the position behind the microlens of the optical system to generate multi-angle data, wherein all the pixel points at the same position behind the microlens are arranged in sequence according to the spatial order. The scanning light field self-supervised network reconstruction device according to claim 7, characterized in that the processing module further comprises: The transformation unit is used to perform linear transformation on the multi-angle data so that the pixel value is within a preset range to obtain the transformed data. The scanning light field self-supervised network reconstruction device according to claim 7, characterized in that the reconstruction module comprises: A forward transmission unit, used for selecting any image processing network, inputting the multi-angle data and forwarding it to obtain a network output; The projection unit is used to randomly select several angles of the point spread function to obtain a forward projection of the network output. The scanning light field self-supervised network reconstruction device according to claim 9, characterized in that the reconstruction module further comprises: A first calculation unit, configured to calculate a mean square error between the network output and a corresponding angle of the forward projection; A second calculation unit, used for calculating the second norm of the second-order derivative of the network output and the axial continuity constraint of the network output; A weighting unit is used to weight the mean square error, the second norm and the axial continuity constraint, calculate the self-supervised loss function of the self-supervised reconstruction network, and transmit back to update the network parameters to obtain the scanned light field self-supervised network reconstruction result. An electronic device, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the scanning light field self-supervised network reconstruction method as described in any one of claims 1 to 5. A computer-readable storage medium having a computer program stored thereon, characterized in that the program is executed by a processor to implement the scanning light field self-supervised network reconstruction method as described in any one of claims 1 to 5.