TWI848575B - Image reconstruction method, electronic device, and storage medium - Google Patents

Abstract

The present application provides an image reconstruction method, an electronic device, and a storage medium. The method includes: obtaining an image to be reconstructed of an object captured by a target camera device and a sample image of the object; training a radial basis function network based on the image to be reconstructed and the sample image; converting the radial basis function network into an image reconstruction model based on a deconvolution algorithm. The image reconstruction model is used for reconstructing images captured by the target camera device. This application can assist image reconstruction and improve the clarity of the image to be reconstructed.

Description

Image reconstruction method, electronic device and storage medium

The present invention relates to the field of image reconstruction technology, and in particular to an image reconstruction method, electronic equipment and storage medium.

In order to increase the screen-to-body ratio of mobile phones and provide users with a better user experience, the design of the under-screen camera on the front of the mobile phone is constantly developing in the direction of miniaturization. The under-screen camera is installed below the display screen. The gap between the light-emitting diodes in the display area of the display screen above the corresponding position of the under-screen camera rearranges the display pixels so that external light can be projected to the under-screen camera through the gap between the display pixels. However, since the gap between the display pixels is very small, the light projected to the under-screen camera will diffract, causing the point light source to diffuse when projected to the under-screen camera, causing the image captured by the under-screen camera to degrade, such as blurring in the image captured by the under-screen camera.

In view of the above, it is necessary to provide an image reconstruction method, electronic equipment and storage medium that can assist image reconstruction and improve the clarity of the image to be reconstructed.

The image reconstruction method includes: obtaining an image to be reconstructed of an object photographed by a target photographing device and a sample image of the object; training a radial basis function network based on the image to be reconstructed and the sample image; converting the radial basis function network into an image reconstruction model based on a deconvolution algorithm, and the image reconstruction model is used to reconstruct the image photographed by the target photographing device.

Optionally, the radial basis function network includes an input layer, a hidden layer and an output layer, wherein: the input layer is used as the input end of the radial basis function network; the hidden layer uses the radial basis function as the basis function, and the hidden layer is fully connected to the output layer; the output layer outputs the output result of the radial basis function network based on the accumulation function.

Optionally, the training of the radial basis function network based on the image to be reconstructed and the sample image includes: initializing the structural parameters and loss function of the radial basis function network.

Optionally, the training of the radial basis function network based on the image to be reconstructed and the sample image further includes: optimizing the structural parameters of the radial basis function network until the loss function corresponding to the output result of the radial basis function network converges to a preset value after the sample image is used as the input of the radial basis function network.

Optionally, the radial basis function network is used to simulate the point spread function model corresponding to the target shooting device.

Optionally, the loss function comprises a two-norm.

Optionally, the optimization algorithm used to optimize the radial basis function network includes a gradient descent algorithm.

Optionally, the deconvolution algorithm comprises a Weiner deconvolution algorithm.

The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, the image reconstruction method or the image reconstruction method is implemented.

The electronic device includes a memory and at least one processor, wherein at least one instruction is stored in the memory, and when the at least one instruction is executed by the at least one processor, the image reconstruction method is implemented.

Compared with the prior art, the image reconstruction method provided by the embodiment of the present application obtains the image to be reconstructed of the object photographed by the target photographing device and the sample image of the object; trains a radial basis function network based on the image to be reconstructed and the sample image; converts the radial basis function network into an image reconstruction model based on the deconvolution algorithm, and the image reconstruction model is used to reconstruct the image photographed by the target photographing device. A radial basis network can be trained to degenerate a sample image without a screen into an image to be reconstructed that is close to a superimposed screen using multiple radial functions and error functions. The radial basis network is converted into an image reconstruction model based on the deconvolution algorithm. The image reconstruction model is used to reconstruct images taken by other target shooting devices, which can improve the clarity of the image to be reconstructed and the efficiency of image reconstruction, and can also reduce the cost of image reconstruction.

3: Electronic equipment

30: Image reconstruction system

31: Storage

32: Processor

S1~S3: Steps

In order to more clearly illustrate the technical solutions in the embodiments of this application or the known technology, the drawings required for the embodiments or the known technology description will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative labor.

Figure 1 is a flow chart of the image reconstruction method provided by the embodiment of this application.

Figure 2 is a diagram showing an example of the network structure of a radial basis function network provided in an embodiment of the present application.

Figure 3 is a second example of the network structure of the radial basis function network provided by the embodiment of this application.

Figure 4 is an example diagram of the image reconstruction method provided by the embodiment of this application.

Figure 5 is a schematic diagram of the electronic device provided in the embodiment of this application.

The following specific implementation method will be combined with the above-mentioned drawings to further illustrate this application.

In order to more clearly understand the above-mentioned purpose, features and advantages of this application, the following is a detailed description of this application in conjunction with the attached drawings and specific embodiments. It should be noted that the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

In the following description, many specific details are explained to facilitate a full understanding of this application. The described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative labor are within the scope of protection of this application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application. The terms used in this specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

In one embodiment, in order to increase the screen-to-body ratio of a mobile phone and provide a better user experience, the design of the under-screen camera on the front of the mobile phone is constantly developing in a miniaturized direction. The under-screen camera is installed below the display screen, and the gaps between the LEDs in the display area of the display screen above the corresponding position of the under-screen camera rearrange the display pixels, so that external light can be projected onto the under-screen camera through the gaps between the display pixels. However, due to the small gap between the display pixels, the light projected onto the under-screen camera will diffract, causing the point light source to diffuse when projected onto the under-screen camera, causing the image captured by the under-screen camera to degrade, such as blurring of the image captured by the under-screen camera.

In order to eliminate the blur of the image taken by the under-screen camera caused by light diffraction, the image taken by the under-screen camera needs to be reconstructed. The commonly used technology directly uses a deep learning network to construct a large processing network. Although it can often present a good reconstruction effect, this method is very time-consuming.

In order to solve the above problems, the image reconstruction method provided in the embodiment of the present application obtains the image to be reconstructed of the object captured by the target shooting device and the sample image of the object; trains a radial basis function network based on the image to be reconstructed and the sample image; and converts the radial basis function network into an image reconstruction model based on the deconvolution algorithm, and the image reconstruction model is used to reconstruct the image captured by the target shooting device. A radial basis network can be trained to degenerate a sample image without a screen into an image to be reconstructed that is close to a superimposed screen using multiple radial functions and error functions. The radial basis network is converted into an image reconstruction model based on the deconvolution algorithm. The image reconstruction model is used to reconstruct images taken by other target shooting devices, which can improve the clarity of the image to be reconstructed and the efficiency of image reconstruction, and can also reduce the cost of image reconstruction.

Refer to Figure 1, which is a flow chart of the image reconstruction method of the preferred embodiment of this application.

In this embodiment, the image reconstruction method can be applied to an electronic device (such as the electronic device 3 shown in FIG. 5 ), and the image reconstruction function provided by the method of the embodiment of the present application is integrated on the electronic device, or is run in the electronic device in the form of a software development kit (SDK), and the electronic device can be a computer, a server, a laptop, etc.

As shown in Figure 1, the image reconstruction method specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

Step S1, obtaining the image to be reconstructed of the object captured by the target shooting device, and the sample image of the object.

In one embodiment, the target shooting device may be an under-display camera (UDC), and the object may be any scene or object. The sample image represents a standard image (ground truth image) of the object, and the sample image may be obtained by shooting the object with a high-resolution camera.

Specifically, the image to be reconstructed and the sample image are images of the same object captured at the same angle and distance, and the image to be reconstructed and the sample image have the same size and dimensions, with only a different resolution.

Step S2, training a radial basis function network based on the image to be reconstructed and the sample image.

In one embodiment, the image to be reconstructed taken by the target device (e.g., an under-screen camera) will appear blurred, which is the result of the convolution integral of the ambient light source and the point spread function (PSF). By analogy with the point spread function corresponding to the target device, the image to be reconstructed can be reconstructed based on the inverse transformation.

Therefore, the sample image is used as the input of the radial basis function network, and the convolution integral is simulated through the radial basis function network, so that the output image of the radial basis function network is infinitely close to the image to be reconstructed, and the combination of the neurons and network weights of the radial basis function network will be close to the point spread function model corresponding to the target device.

In one embodiment, the radial basis function network is used to simulate the point spread function model corresponding to the target shooting device.

In one embodiment, the radial basis function (RBF) is a three-layer network structure that can project n-dimensional input data into m-dimensional space, where n and m both represent positive integers greater than or equal to 1. Therefore, an image (e.g., a sample image) can be converted into an n-dimensional vector, and the n-dimensional vector corresponding to the image is used as the input data of the radial basis function network to obtain the image corresponding to the output m-dimensional vector.

The radial basis function network includes an input layer, a hidden layer and an output layer, wherein: the input layer is used as the input end of the radial basis function network; the hidden layer uses the radial basis function as the basis function, and the hidden layer is fully connected to the output layer; the output layer outputs the output result of the radial basis function network based on the accumulation function. The radial basis function includes a Gaussian radial basis function.

In one embodiment, the training of the radial basis function network based on the image to be reconstructed and the sample image includes: initializing the structural parameters and loss function of the radial basis function network. The initialization includes but is not limited to random initialization, the structural parameters include but are not limited to the connection weights between the neurons of the hidden layer and the output layer of the radial basis network, and the loss function includes but is not limited to the two-norm.

In one embodiment, the loss function is used to indicate the difference between the output image of the radial basis function network and the image to be reconstructed. When the loss function converges to a preset value (e.g., 0.02), it can be determined that the output image is infinitely close to the image to be reconstructed.

In one embodiment, as shown in FIG2 , a network structure example of a radial basis function network provided in the embodiment of the present application is shown in FIG1 , wherein the black filled circles on the left represent multiple neurons in the input layer, the gray filled circles in the middle represent multiple neurons kj in the hidden layer, and the white filled circles on the right represent multiple neurons in the output layer.

As shown in _{Figure 2, each hidden layer neuron in the middle of the radial basis function network is connected to all neurons in the output layer on the right, and has corresponding connection weights, such as weight w0,0} , weight w0,1 _, etc. in the figure.

，其中，i、j、v、m、n的取值都為大於或等於1的正整數，μ _p 表示高斯徑向基函數的第p個中心點，σ表示x _i 的標準方差。 Among them, x =( x ₁ , x ₂ ,..., x _i ,... x _n ) represents the input data, y =( y ₁ , y ₂ ,..., y _v ,... y _m ) represents the output result, and based on the Gaussian radial basis function, we can get:

, where i , j , v , m , and n are all positive integers greater than or equal to 1, μp represents the pth center point of the Gaussian radial basis function, _and σ represents the standard deviation _of xi .

In one embodiment, the structural parameters also include the value of the center point.

In one embodiment, as shown in FIG3, a network structure example of a radial basis function network provided in the embodiment of the present application is shown in FIG2. Compared with FIG2, the network structure in FIG3 is a simplified example diagram. Among them, δ _n represents the nth neuron in the hidden layer, and the Gaussian radial basis function corresponding to each neuron in the hidden layer is simplified to a Gaussian impulse function.

In one embodiment, the training of the radial basis function network based on the image to be reconstructed and the sample image further includes: optimizing the structural parameters of the radial basis function network until the loss function corresponding to the output result of the radial basis function network converges to a preset value after the sample image is used as the input of the radial basis function network.

In one embodiment, since a random initialization method is used when initializing the structural parameters and loss function of the radial basis function network, it is impossible to ensure that the initialized radial basis function network achieves the expected loss function convergence effect, so it is necessary to optimize the radial basis function network, and the optimization algorithm used to optimize the radial basis function network includes a gradient descent algorithm.

，使用梯 度下降演算法對權重進行優化時的反覆運算更新可以表示為：w _i,t+1=w _i,t -η△w _i ，w _i,t 表示第t次反覆運算更新後得到的權重，t表示大於或等於1的正整數，η表示梯度下降演算法的學習率(Learning Rate)。 Specifically, referring to the weights w =( w ₁ , w ₂ ,..., w _i ,... w _n ) in Figure 3 , the two-norm loss function can be expressed as: L ( x , w ) =( y - y' ) ² =( y -Σ _i w _i x _i ) ² , and the gradient descent algorithm can be expressed as:

, the repeated operation updates when using the gradient descent algorithm to optimize the weights can be expressed as: w _{i,t +1} = w _i,t -η△ w _i , w _i,t represents the weight obtained after the tth repeated operation update, t represents a positive integer greater than or equal to 1, and η represents the learning rate of the gradient descent algorithm.

In one embodiment, the gradient descent algorithm is used to make the loss function continuously converge until it converges to a preset value, thereby obtaining the corresponding expected radial basis function network.

Step S3, based on the deconvolution algorithm, the radial basis function network is converted into an image reconstruction model, and the image reconstruction model is used to reconstruct the image taken by the target shooting device.

In one embodiment, the radial basis function network is used in the above method to analogize the point diffusion function of the image to be reconstructed by degrading the sample image. In order to reconstruct and restore other images to be reconstructed, the radial basis function needs to be deconvolved, thereby converting the radial basis function network into an image reconstruction model.

，其中h表示模糊核，Q表示雜訊，

表示卷積。Weiner反卷積演算法的原理包括使用一個Wiener卷積核G，使得G

y與x之間的差異達到最小。 In one embodiment, the deconvolution algorithm includes a Weiner deconvolution algorithm. The Weiner deconvolution algorithm is a non-blind linear image restoration algorithm. Specifically, the transformation relationship of a clear sample image x degenerated into a blurred image y to be reconstructed is considered as:

, where h is the blur kernel, Q is the noise,

Denotes convolution. The principle of the Weiner deconvolution algorithm involves using a Wiener convolution kernel G such that G

The difference between y and x is minimized.

In one embodiment, the specific action mode of the blur kernel and the noise can be known from the radial basis function network model in the above step S2. Therefore, the non-blind Weiner deconvolution algorithm can be directly performed on the radial basis function network model to obtain the image reconstruction model. The specific formula used is a common method in this field and will not be described again.

In one embodiment, the image reconstruction method provided by the present application only uses sample images and graphics to be reconstructed to train the network and model. The training cost is low and the amount of computation in the training process is small. It can be directly integrated into the target shooting device, for example, into a mobile phone including an under-screen camera. When the user takes a picture using the under-screen camera, the reconstructed image of the captured image can be quickly obtained, so that the user can preview the reconstructed image conveniently.

In one embodiment, as shown in FIG. 4 , it is an example diagram of the image reconstruction method provided by the embodiment of the present application. Among them, the x image is a high-definition sample image, the y image is a blurred image to be reconstructed, and the x' represents an image with improved clarity after the image to be reconstructed is reconstructed. It can be seen that the image reconstruction method provided by the embodiment of the present application can effectively reduce the blurriness of the image to be reconstructed and improve the clarity of the image to be reconstructed.

In one embodiment, the image reconstruction method provided by the present application obtains the image to be reconstructed of the object captured by the target shooting device and the sample image of the object; trains a radial basis function network based on the image to be reconstructed and the sample image; and converts the radial basis function network into an image reconstruction model based on the deconvolution algorithm, and the image reconstruction model is used to reconstruct the image captured by the target shooting device. A radial basis network can be trained to degenerate a sample image without a screen into an image to be reconstructed that is close to a superimposed screen using multiple radial functions and error functions. The radial basis network is converted into an image reconstruction model based on the deconvolution algorithm. The image reconstruction model is used to reconstruct images taken by other target shooting devices, which can improve the clarity of the image to be reconstructed and the efficiency of image reconstruction, and can also reduce the cost of image reconstruction.

The above-mentioned Figure 1 introduces the image reconstruction method of the present application in detail. The following, in conjunction with Figure 5, introduces the functional modules of the software system for implementing the image reconstruction method and the hardware device architecture for implementing the image reconstruction method.

It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

See Figure 5, which is a schematic diagram of the structure of the electronic device provided in the preferred embodiment of this application.

In the preferred embodiment of the present application, the electronic device 3 includes a memory 31 and at least one processor 32. Those skilled in the art should understand that the structure of the electronic device shown in FIG. 5 does not constitute a limitation of the embodiment of the present application, and it can be a bus structure or a star structure. The electronic device 3 can also include more or less other hardware or software than shown in the figure, or different component layouts.

In some embodiments, the electronic device 3 includes a terminal that can automatically perform numerical calculations and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, dedicated integrated circuits, programmable logic gate arrays, digital signal processors, and embedded devices.

It should be noted that the electronic device 3 is only an example. Other existing or future electronic products that are suitable for this application should also be included in the protection scope of this application and are included here by reference.

In some embodiments, the memory 31 is used to store program codes and various data. For example, the memory 31 can be used to store the image reconstruction system 30 installed in the electronic device 3, and realize high-speed and automatic access to programs or data during the operation of the electronic device 3. The storage device 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable storage medium that can be used to carry or store data.

In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital signal processing chips, graphics processors, and combinations of various control chips. The at least one processor 32 is the control core (Control Unit) of the electronic device 3, and uses various interfaces and lines to connect the various components of the entire electronic device 3, and executes or runs programs or modules stored in the memory 31, and calls the data stored in the memory 31 to execute various functions of the electronic device 3 and process data, such as executing the image reconstruction function shown in Figure 1.

In some embodiments, the image reconstruction system 30 runs in the electronic device 3. The image reconstruction system 30 may include a plurality of functional modules composed of program code segments. The program code of each program segment in the image reconstruction system 30 may be stored in the memory 31 of the electronic device 3 and executed by at least one processor 32 to realize the image reconstruction function shown in FIG1.

In this embodiment, the image reconstruction system 30 can be divided into multiple functional modules according to the functions it performs. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory.

Although not shown, the electronic device 3 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 32 through a power management device, so as to manage charging, discharging, and power consumption through the power management device. The power source may also include one or more DC or AC power sources, recharging devices, power fault test circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated here.

It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling an electronic device (which can be a server, a personal computer, etc.) or a processor to execute a part of the method described in each embodiment of the present application.

The memory 31 stores program codes, and the at least one processor 32 can call the program codes stored in the memory 31 to execute related functions. The program codes stored in the memory 31 can be executed by the at least one processor 32, thereby realizing the functions of each module to achieve the purpose of image reconstruction.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of this embodiment.

In addition, each functional module in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments and that the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure marks in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, not to indicate any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of this application and are not limiting. Although this application is described in detail with reference to the above preferred embodiments, ordinary technicians in this field should understand that the technical solution of this application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of this application.

S1~S3: Steps

Claims (9)

An image reconstruction method is applied to an electronic device, wherein the method comprises: obtaining an image to be reconstructed of an object photographed by a target photographing device, and a sample image of the object, wherein the image to be reconstructed and the sample image are images of the same object photographed at the same angle and distance, and the image to be reconstructed and the sample image have the same size and dimensions, and only the resolution is different; based on the image to be reconstructed and the sample image, a radial basis function network is trained, and the radial basis function network is used to analogize a point diffusion function model corresponding to the target photographing device; based on a deconvolution algorithm, the radial basis function network is converted into an image reconstruction model, and the image reconstruction model is used to reconstruct the image photographed by the target photographing device. The image reconstruction method as described in claim 1, wherein the radial basis function network includes an input layer, a hidden layer and an output layer, wherein: the input layer is used as the input end of the radial basis function network; the hidden layer uses the radial basis function as the basis function, and the hidden layer is fully connected to the output layer; the output layer outputs the output result of the radial basis function network based on the accumulation function. The image reconstruction method as described in claim 1, wherein the training of the radial basis function network based on the image to be reconstructed and the sample image includes: initializing the structural parameters and loss function of the radial basis function network. The image reconstruction method as described in claim 3, wherein the training of the radial basis function network based on the image to be reconstructed and the sample image further comprises: optimizing the structural parameters of the radial basis function network until the loss function corresponding to the output result of the radial basis function network converges to a preset value after the sample image is used as the input of the radial basis function network. An image reconstruction method as described in claim 3, wherein the loss function includes a binary norm. An image reconstruction method as described in claim 3, wherein the optimization algorithm used to optimize the radial basis function network includes a gradient descent algorithm. An image reconstruction method as described in claim 1, wherein the deconvolution algorithm includes a Weiner deconvolution algorithm. A computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, an image reconstruction method as described in any one of claims 1 to 7 is implemented. An electronic device, wherein the electronic device includes a memory and at least one processor, wherein at least one instruction is stored in the memory, and when the at least one instruction is executed by the at least one processor, the image reconstruction method as described in any one of claims 1 to 7 is implemented.