CN112184550A - Neural network training method, image fusion method, apparatus, equipment and medium - Google Patents

Abstract

The disclosure relates to the technical field of image processing, and discloses a neural network training method, an image fusion device, equipment and a medium. The method comprises the following steps: designing a first sub-network and a second sub-network with the same network structure, wherein any sub-network comprises a neural network of a primary feature extraction module, a high-level feature extraction module and a coupling feedback module; the primary characteristic module is used for extracting low-level characteristics of the underexposed low-resolution image and the overexposed low-resolution image; the high-level feature extraction module is used for further extracting high-level features of the underexposed low-resolution image and the overexposed low-resolution image from the corresponding low-level features of the underexposed low-resolution image and the overexposed low-resolution image; the feedback module is coupled to alternately fuse the low-level features and the high-level features corresponding to the under-exposed low-resolution image and the over-exposed low-resolution image. By the technical scheme, multi-exposure fusion processing and super-resolution processing of the image are simultaneously performed by using one neural network, and the image processing speed and the processing accuracy are improved.

Description

Neural network training method, image fusion method, apparatus, equipment and medium

technical field

The present disclosure relates to the technical field of image processing, and in particular, to a neural network training method, an image fusion method, an apparatus, a device and a medium.

Background technique

With the development of technology, people are more and more accustomed to using photos to record the details of their lives. However, due to the hardware limitations of the camera sensor, the captured images usually have various distortions, which make the images very different from the real natural scene. Compared with the real scene, the images captured by the camera tend to have the characteristics of low dynamic range (Low DynamicRange, LDR) and low resolution (Low-Resolution, LR). In order to narrow the difference between the captured image and the actual shooting scene, the image needs to be processed.

At present, multi-exposure image fusion (MEF) is mainly used to correct the low dynamic range of the image, and Image Super-Resolution (ISR) is used to correct the low resolution of the image. The multi-exposure image fusion technology aims to fuse several LDR images with different exposure levels to generate an image with High Dynamic Range (HDR). Image super-resolution technology aims to reconstruct an LR image into a High-Resolution (HR) image.

However, the actual situation is that a shot image has two characteristics of LDR and LR at the same time, and the multi-exposure image fusion technology and image super-resolution technology are two independent image processing technologies, which means that a shot image needs to be processed successively. Exposure image fusion processing and image super-resolution processing. Moreover, the sequence of execution of the two image processing techniques may sometimes affect the final image processing result. Therefore, the existing image processing methods are not only complicated in the processing process, but also have unsatisfactory image processing effects.

SUMMARY OF THE INVENTION

In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a neural network training method, an image fusion method, an apparatus, a device and a medium.

In a first aspect, the present disclosure provides a neural network training method, the neural network includes a first sub-network and a second sub-network with the same network structure, and any sub-network includes a primary feature extraction module and a high-level feature extraction module and a coupled feedback module; the method includes:

Obtain an underexposed low-resolution image and an overexposed low-resolution image;

Input the under-exposed low-resolution image and the over-exposed low-resolution image into the initial feature extraction module in the first sub-network and the second sub-network, respectively, to generate under-exposed low-level features and Overexposure low-level features;

Input the under-exposed low-level features and the over-exposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network, respectively, to generate under-exposed high-level features and over-exposed low-level features high-level features;

Input the under-exposed low-level features, the under-exposed high-level features, and the over-exposed high-level features into the coupling feedback module in the first sub-network to generate the coupling corresponding to the first sub-network feedback results;

Input the over-exposed low-level features, the over-exposed high-level features, and the under-exposed high-level features into the coupling feedback module in the second sub-network to generate the coupling corresponding to the second sub-network feedback results;

based on the underexposed low-resolution image, the underexposed high-level feature, and the coupling feedback result corresponding to the first sub-network, and the overexposed low-resolution image, the overexposed high-level feature, and the The coupling feedback result corresponding to the second sub-network adjusts the parameters of the neural network.

In some embodiments, the neural network includes a plurality of the coupled feedback modules, and each of the coupled feedback modules does not share model parameters.

In some embodiments, each of the coupled feedback modules is processed serially;

The under-exposure low-level feature, the under-exposure high-level feature, and the over-exposure high-level feature are input into the coupling feedback module in the first sub-network to generate a corresponding corresponding to the first sub-network. The coupled feedback results include:

Input the under-exposed low-level features, the under-exposed high-level features, and the over-exposed high-level features into the first coupling feedback module in the first sub-network to generate the corresponding Coupling feedback results;

With respect to any subsequent coupling feedback module except the first coupling feedback module in the first sub-network, the underexposed low-level feature, the coupling feedback module of the previous adjacent coupling feedback module of the subsequent coupling feedback module The coupling feedback result and the coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in the second sub-network are input to the subsequent coupling feedback module to generate the coupling feedback result corresponding to the first sub-network ;

Inputting the over-exposed low-level features, the over-exposed high-level features and the under-exposed high-level features into the coupling feedback module in the second sub-network to generate the corresponding second sub-network The coupled feedback results include:

Input the overexposure low-level features, the overexposure high-level features and the underexposure high-level features into the first coupling feedback module in the second sub-network to generate the corresponding Coupling feedback results;

For any subsequent coupling feedback module except the first coupling feedback module in the second sub-network, the overexposure low-level feature, the coupling feedback module of the previous adjacent coupling feedback module of the subsequent coupling feedback module The coupling feedback result and the coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in the first sub-network are input to the subsequent coupling feedback module to generate the coupling feedback result corresponding to the second sub-network .

In some embodiments, the coupled feedback module includes at least two coupling sub-modules and at least two feature map groups, wherein each feature map group includes a filter, a deconvolution layer and a convolution layer Floor;

The first described connection sub-module is located before each of the feature map groups;

Any other connection submodules except the first connection submodule are located between any two adjacent feature map groups, and any two other connection submodules are located in different positions.

In some embodiments, based on the under-exposed low-resolution image, the under-exposed high-level feature and a coupling feedback result corresponding to the first sub-network, and the over-exposed low-resolution image, the Overexposing the high-level features and the coupling feedback result corresponding to the second sub-network, and adjusting the parameters of the neural network includes:

respectively performing an upsampling operation on the underexposed low-resolution image and the overexposed low-resolution image;

The image corresponding to the under-exposed high-level feature and the image corresponding to the coupling feedback result corresponding to the first sub-network are respectively added to the up-sampled under-exposed low-resolution image to generate an under-exposed high-resolution image a fused exposure high-resolution image corresponding to the image and the second sub-network;

The image corresponding to the over-exposure high-level feature and the image corresponding to the coupling feedback result corresponding to the second sub-network are respectively added to the up-sampled over-exposed low-resolution image to generate an over-exposure high-resolution image a fused exposure high-resolution image corresponding to the image and the second sub-network;

Based on the under-exposed high-resolution image, the fused-exposure high-resolution image corresponding to the first sub-network, the over-exposed high-resolution image, and the fused-exposure high-resolution image corresponding to the second sub-network, adjust the parameters of the neural network.

In some embodiments, the parameters of the neural network are adjusted through a loss function shown in the following formula:

分别表示每一部分损失函数值对应的权重，

和

分别表示所述第一子网络中的所述高层特征提取模块和所述耦合反馈模块对应的损失函数值，

和

分别表示所述第二子网络中的所述高层特征提取模块和所述耦合反馈模块对应的损失函数值，L_MS表示基于图像的结构相似性指数确定的两个图像之间的损失值，

和

分别表示所述过曝高分辨率图像和过曝高分辨率参考图像，

和

分别表示所述欠曝高分辨率图像和欠曝高分辨率参考图像，

和I_gt分别表示第t个所述第二子网络对应的融合曝光高分辨率图像、第t个所述第一子网络对应的融合曝光高分辨率图像和融合曝光高分辨率参考图像，T表示所述耦合反馈模块的数量。where L _total represents the total loss function value, λ _o , λ _u and

respectively represent the weight corresponding to each part of the loss function value,

and

respectively represent the loss function values corresponding to the high-level feature extraction module and the coupled feedback module in the first sub-network,

and

respectively represent the loss function values corresponding to the high-level feature extraction module and the coupled feedback module in the second sub-network, L _MS represents the loss value between two images determined based on the structural similarity index of the images,

and

represent the overexposed high-resolution image and the overexposed high-resolution reference image, respectively,

and

represent the underexposed high-resolution image and the underexposed high-resolution reference image, respectively,

and I _gt respectively represent the fused exposure high-resolution image corresponding to the t-th second sub-network, the fused-exposure high-resolution image corresponding to the t-th first sub-network, and the fused-exposure high-resolution reference image, T Indicates the number of the coupled feedback modules.

In a second aspect, the present disclosure provides an image fusion method, the method comprising:

Obtain an underexposed low-resolution image and an overexposed low-resolution image;

Inputting the under-exposed low-resolution image and the over-exposed low-resolution image into a pre-trained neural network to generate a first fused-exposure high-resolution image and a second fused-exposure high-resolution image; wherein the neural network Obtained by training the neural network training method in any embodiment of the present disclosure;

An image fusion result is generated based on the first fused exposure high-resolution image and the second fused exposure high-resolution image.

In some embodiments, generating an image fusion result based on the first fused exposure high-resolution image and the second fused exposure high-resolution image includes:

Using the first weight and the second weight respectively, the first fusion exposure high-resolution image and the second fusion exposure high-resolution image are weighted and summed to generate the image fusion result.

In a third aspect, the present disclosure provides a neural network training device, the neural network includes a first sub-network and a second sub-network with the same network structure, and any sub-network includes a primary feature extraction module and a high-level feature extraction module and a coupled feedback module; the device includes:

an image acquisition unit for acquiring an under-exposed low-resolution image and an over-exposed low-resolution image;

A low-level feature generation unit, configured to input the under-exposed low-resolution image and the over-exposed low-resolution image into the initial feature extraction module in the first sub-network and the second sub-network respectively , to generate under-exposed low-level features and over-exposed low-level features;

The high-level feature generation unit is configured to input the under-exposed low-level features and the over-exposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network, respectively, to generate Underexposed high-level features and overexposed high-level features;

a first coupling feedback result generating unit, configured to input the under-exposed low-level feature, the under-exposed high-level feature, and the over-exposed high-level feature into the coupling feedback module in the first sub-network, generating a coupling feedback result corresponding to the first sub-network;

A second coupling feedback result generating unit, configured to input the overexposed low-level feature, the overexposed high-level feature, and the underexposed high-level feature into the coupling feedback module in the second sub-network, generating a coupling feedback result corresponding to the second sub-network;

A parameter adjustment unit, configured to base on the under-exposed low-resolution image, the under-exposed high-level feature and the coupling feedback result corresponding to the first sub-network, as well as the over-exposed low-resolution image, the over-exposed high-level feature The high-level feature and the corresponding coupling feedback result of the second sub-network adjust the parameters of the neural network.

In some embodiments, the neural network includes multiple coupled feedback modules, and each coupled feedback module does not share model parameters.

In some embodiments, the coupled feedback modules are processed serially;

Correspondingly, the first coupling feedback result generating unit is specifically used for:

Input the under-exposed low-level features, under-exposed high-level features and over-exposed high-level features into the first coupling feedback module in the first sub-network to generate the coupling feedback result corresponding to the first sub-network;

For any subsequent coupling feedback module except the first coupling feedback module in the first sub-network, the underexposure low-level features, the coupling feedback result of the previous adjacent coupling feedback module of the subsequent coupling feedback module, and the The coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in the two sub-networks is input to the subsequent coupling feedback module to generate the coupling feedback result corresponding to the first sub-network;

Correspondingly, the second coupling feedback result generating unit is specifically used for:

Input the over-exposed low-level features, over-exposed high-level features and under-exposed high-level features into the first coupling feedback module in the second sub-network to generate the coupling feedback result corresponding to the second sub-network;

For any subsequent coupling feedback module except the first coupling feedback module in the second sub-network, the low-level features, the coupling feedback result of the previous adjacent coupling feedback module of the subsequent coupling feedback module, and the first coupling feedback module will be overexposed. The coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in a sub-network is input to the subsequent coupling feedback module to generate the coupling feedback result corresponding to the second sub-network.

In some embodiments, the coupled feedback module includes at least two connection sub-modules and at least two feature map groups, wherein each feature map group includes a filter, a deconvolution layer, and a convolution layer;

The first link submodule is located before each feature map group;

Any other linking submodules except the first linking submodule are located between any two adjacent feature map groups, and any two other linking submodules are located at different positions.

In some embodiments, the parameter adjustment unit is specifically used to:

Perform up-sampling operations on the under-exposed low-resolution image and the over-exposed low-resolution image respectively;

The image corresponding to the under-exposed high-level feature and the image corresponding to the coupling feedback result corresponding to the first sub-network are added to the up-sampled under-exposed low-resolution image to generate the under-exposed high-resolution image and the second sub-network. Corresponding fused exposure high-resolution images;

The image corresponding to the overexposed high-level feature and the image corresponding to the coupling feedback result corresponding to the second sub-network are added to the up-sampled over-exposed low-resolution image to generate the over-exposed high-resolution image and the second sub-network. Corresponding fused exposure high-resolution images;

The parameters of the neural network are adjusted based on the under-exposed high-resolution image, the fused-exposure high-resolution image corresponding to the first sub-network, the over-exposed high-resolution image, and the fused-exposure high-resolution image corresponding to the second sub-network.

Further, the parameter adjustment unit is specifically used for:

Adjust the parameters of the neural network through the loss function shown in the following formula:

分别表示每一部分损失函数值对应的权重，

和

分别表示第一子网络中的高层特征提取模块和耦合反馈模块对应的损失函数值，

和

分别表示第二子网络中的高层特征提取模块和耦合反馈模块对应的损失函数值，L_MS表示基于图像的结构相似性指数确定的两个图像之间的损失值，

和

分别表示过曝高分辨率图像和过曝高分辨率参考图像，

和

分别表示欠曝高分辨率图像和欠曝高分辨率参考图像，

和I_gt分别表示第t个第二子网络对应的融合曝光高分辨率图像、第t个第一子网络对应的融合曝光高分辨率图像和融合曝光高分辨率参考图像，T表示耦合反馈模块的数量。where L _total represents the total loss function value, λ _o , λ _u and

respectively represent the weight corresponding to each part of the loss function value,

and

respectively represent the loss function values corresponding to the high-level feature extraction module and the coupled feedback module in the first sub-network,

and

respectively represent the loss function values corresponding to the high-level feature extraction module and the coupled feedback module in the second sub-network, L _MS represents the loss value between the two images determined based on the structural similarity index of the image,

and

represent the overexposed high-resolution image and the overexposed high-resolution reference image, respectively,

and

represent the underexposed high-resolution image and the underexposed high-resolution reference image, respectively,

and I _gt represent the fused exposure high-resolution image corresponding to the t-th second sub-network, the fused-exposure high-resolution image corresponding to the t-th first sub-network, and the fused-exposure high-resolution reference image, respectively, and T represents the coupling feedback module quantity.

In a fourth aspect, the present disclosure provides an image fusion device, the device comprising:

an image acquisition unit for acquiring an under-exposed low-resolution image and an over-exposed low-resolution image;

A fusion exposure high-resolution image generation unit, configured to input the under-exposed low-resolution image and the over-exposed low-resolution image into a pre-trained neural network to generate a first fusion exposure high-resolution image and a second fusion exposure high-resolution image A high-resolution image; wherein, the neural network is obtained by training any embodiment of the neural network training method in the present disclosure;

An image fusion result generating unit, configured to generate an image fusion result based on the first fusion exposure high-resolution image and the second fusion exposure high-resolution image.

In some embodiments, the image fusion result generating unit is specifically used for:

Using the first weight and the second weight respectively, the first fusion exposure high-resolution image and the second fusion exposure high-resolution image are weighted and summed to generate an image fusion result.

In a fifth aspect, the present disclosure provides an electronic device comprising:

one or more processors;

storage means for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned embodiments of the neural network training method or the image fusion method.

In a fourth aspect, the present disclosure provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements any one of the foregoing neural network training methods or image fusion methods.

In the technical solution provided by the embodiments of the present disclosure, by designing a neural network that includes a first sub-network and a second sub-network with the same network structure, and any sub-network includes a primary feature extraction module, a high-level feature extraction module, and a coupled feedback module, And the primary feature module is used to extract the low-level features of the under-exposed low-resolution image and the over-exposed low-resolution image, and the high-level feature extraction module is used to extract the corresponding low-level features from the under-exposed low-resolution image and the over-exposed low-resolution image. The high-level features are further extracted from the features, and the low-resolution images are initially mapped to high-resolution features. Through the coupling feedback module, the low-level features and high-level features corresponding to the under-exposed low-resolution image and the over-exposed low-resolution image are cross-fused to realize the multi-exposure fusion of the over-exposed image and the under-exposed image, and further improve the image quality. It can obtain an image with high resolution and high dynamic range at the same time, and achieve the purpose of multi-exposure fusion processing and super-resolution processing of images at the same time, which not only simplifies the processing flow of captured images, but also improves the speed of image processing. , and the complementary properties between multi-exposure fusion and super-resolution are used to further improve the image processing accuracy.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the accompanying drawings that are required to be used in the description of the embodiments or the prior art will be briefly introduced below. In other words, on the premise of no creative labor, other drawings can also be obtained from these drawings.

1 is a network architecture diagram of a neural network provided by an embodiment of the present disclosure;

2 is a network architecture diagram of a high-level feature extraction module in a neural network provided by an embodiment of the present disclosure;

3 is a network architecture diagram of a coupled feedback module in a neural network provided by an embodiment of the present disclosure;

4 is a network architecture diagram of a neural network for neural network training provided by an embodiment of the present disclosure;

5 is a flowchart of a neural network training method provided by an embodiment of the present disclosure;

6 is a flowchart of an image fusion method provided by an embodiment of the present disclosure;

7 is a schematic structural diagram of a neural network training apparatus provided by an embodiment of the present disclosure;

FIG. 8 is a schematic structural diagram of an image fusion apparatus provided by an embodiment of the present disclosure;

FIG. 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be described in further detail below. It should be noted that the embodiments of the present disclosure and the features in the embodiments may be combined with each other under the condition of no conflict.

Many specific details are set forth in the following description to facilitate a full understanding of the present disclosure, but the present disclosure can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, and Not all examples.

The neural network training solution provided by the embodiments of the present disclosure can be applied to the application scenario of fusion processing of images with low dynamic range and low resolution, and is especially suitable for overexposed low-resolution images (referred to as overexposed low resolution images for short). high-resolution images) and underexposed low-resolution images (referred to as underexposed low-resolution images) for image fusion processing.

FIG. 1 is a block diagram of a network structure of a neural network for image fusion provided by an embodiment of the present application. As shown in FIG. 1 , the neural network includes a first sub-network 110 and a second sub-network 120 which have the same network structure but do not share model parameters. The first sub-network 110 includes an initial feature extraction block (Initial Feature Extraction Block, FEB) 111 , a high-level feature extraction block (super-resolution block, SRB) 112 , and a coupled feedback block (CFB) 113 . The second network sub 120 includes a primary feature extraction module 121 , a high-level feature extraction module 122 and a coupled feedback module 123 . The numbers of the coupling feedback modules 113 in the first sub-network 110 and the coupling feedback modules 123 in the second sub-network 120 are the same, and the number is equal to or greater than one. The input data of the neural network is an over-exposed low-resolution image and an under-exposed low-resolution image, and the two input images only need to be input into one sub-network respectively, and the specific input correspondence is not limited. In the embodiment of the present disclosure, the underexposed low-resolution image is input to the first sub-network 110 and the over-exposed low-resolution image is input to the second sub-network 120 as an example for description. The above FEB and SRB are used to extract high-level features from the input image, which helps to enhance the image resolution; the above CFB is located after the SRB and is used to absorb the features learned by the SRB of the two sub-networks, thereby merging a picture Images with high-resolution (HR) and high dynamic range (HDR).

和过曝低分辨率图像

的基本特征，获得对应的欠曝低层次特征

和过曝低层次特征

利用公式表征初级特征提取过程为：

和

其中的f_FEB()表示初级特征提取模块的操作。在一些实施例中，f_FEB()包含了一系列3×3和1×1卷积核的卷积层。The primary feature extraction module 111 and the primary feature extraction module 121 are respectively used to extract the input underexposed low resolution image

and overexposed low-resolution images

The basic features of , obtain the corresponding underexposure low-level features

and overexposed low-level features

Using the formula to characterize the primary feature extraction process is:

and

where _fFEB () represents the operation of the primary feature extraction module. In some embodiments, _fFEB ( ) consists of a series of convolutional layers with 3×3 and 1×1 convolution kernels.

和过曝低层次特征

进行进一步的特征提取操作，来提取欠曝低分辨率图像

和过曝低分辨率图像

的高层次的特征，获得欠曝高层次特征G^u和过曝高层次特征G^o。由于高层次的特征包含更高层的语义特征，其更能表征图像中小而复杂的目标，从而丰富图像中的细节信息，所以欠曝高层次特征G^u和过曝高层次特征G^o能够提高相应图像的分辨率，实现超分辨效果。在一些实施例中，参见图2，利用SRFBN网络中的反馈模块作为高层特征提取模块112(122)的主要结构，它包含了数个以稠密连接(dense connection)方式连续连接的特征映射组210。每个特征映射组210中至少含有一个上采样操作(Deconv)和一个下采样操作(Conv)。通过连续上下采样的方式，在确保特征的大小不变的情况下，逐步地从低层次特征F_in中提取出更高层次的特征G，以改善图像的分辨率。利用公式表征高层特征提取过程为：SRB输出的高层次的特征可以被表示为：

和

其中，f_SRB()代表高层特征提取模块中的操作。The high-level feature extraction module 112 and the high-level feature extraction module 122 are respectively used for underexposing the input low-level features

and overexposed low-level features

Perform further feature extraction operations to extract underexposed low-resolution images

and overexposed low-resolution images

The high-level features of , obtain under-exposed high-level features ^{Gu and over-exposed high-level features G o .} Since high-level features contain higher-level semantic features, they can better represent small and complex objects in the image, thereby enriching the detailed information in the image, so the under-exposed high-level features ^{Gu and over-exposed high-level features G o can} improve the corresponding The resolution of the image to achieve super-resolution effect. In some embodiments, referring to FIG. 2, the feedback module in the SRFBN network is used as the main structure of the high-level feature extraction module 112 (122), which includes several feature map groups 210 continuously connected in a dense connection manner . Each feature map group 210 contains at least one up-sampling operation (Deconv) and one down-sampling operation (Conv). Through continuous up- and down-sampling, while ensuring that the size of the feature remains unchanged, a higher-level feature G is gradually extracted from the low-level feature F _in to improve the resolution of the image. Using the formula to characterize the high-level feature extraction process is: The high-level features output by SRB can be expressed as:

and

where f _SRB ( ) represents the operations in the high-level feature extraction module.

The coupled feedback module CFB is the core component of the neural network, and its purpose is to simultaneously achieve super-resolution and multi-exposure image fusion through a complex network structure. The input data of the coupled feedback module CFB contains three, namely low-level features and high-level features in the same sub-network, and high-level features in another sub-network. Here, the role of the two input data in the same sub-network is to further improve the image resolution and enhance the image super-resolution effect; the role of the input data in the other sub-network is to improve the image fusion effect and realize multi-exposure image fusion.

欠曝高层次特征G^u和过曝高层次特征G^o，生成第一子网络110对应的耦合反馈结果。耦合反馈模块123用于融合输入的过曝低层次特征

过曝高层次特征G^o和欠曝高层次特征G^u，生成第二子网络120对应的耦合反馈结果。这里的耦合反馈结果均是同时实现多曝光融合和超分辨的图像特征。In some embodiments, each sub-network of the neural network includes a coupled feedback module CFB. Then, the coupled feedback module 113 is used to fuse the input underexposed low-level features

The underexposed high-level feature ^{Gu and the overexposed high-level feature G o are} used to generate a coupling feedback result corresponding to the first sub-network 110 . The coupled feedback module 123 is used to fuse the input overexposed low-level features

The overexposed high-level feature ^Go and the underexposed high ^- level feature Gu are used to generate a coupling feedback result corresponding to the second sub-network 120 . The coupled feedback results here are all image features that simultaneously achieve multi-exposure fusion and super-resolution.

In some embodiments, each sub-network in the neural network includes multiple coupled feedback modules CFB, and the multiple CFBs are processed in parallel. In this embodiment, the input data of each CFB in the same sub-network is the same, and the output coupling feedback results need to be further fused (such as weighted summation, etc.) to obtain a coupling feedback result.

欠曝高层次特征G^u和过曝高层次特征G^o，输入第一子网络110中的第一个耦合反馈模块113，生成第一子网络对应的耦合反馈结果

针对第一子网络中除了第一个耦合反馈模块113之外的任一个后续耦合反馈模块113(序号为t)，将欠曝低层次特征

该后续耦合反馈模块的前一相邻耦合反馈模块(序号为t-1)的耦合反馈结果

以及第二子网络120中与该前一相邻耦合反馈模块对应的耦合反馈模块的耦合反馈结果

输入该后续耦合反馈模块113，生成第一子网络110对应的耦合反馈结果

按照该过程，经过所有CFB的运算，可获得第一子网络110对应的最终的耦合反馈结果

同样地，生成第二子网络120对应的耦合反馈结果的过程为：将过曝低层次特征

过曝高层次特征G^o和欠曝高层次特征G^u，输入第二子网络120中的第一个耦合反馈模块123，生成第二子网络120对应的耦合反馈结果

针对第二子网络120中除了第一个耦合反馈模块123之外的任一个后续耦合反馈模块123(序号为t)，将过曝低层次特征

该后续耦合反馈模块的前一相邻耦合反馈模块的耦合反馈结果

以及第一子网络110中与该前一相邻耦合反馈模块对应的耦合反馈模块的耦合反馈结果

输入该后续耦合反馈模块123，生成第二子网络120对应的耦合反馈结果

按照该过程，经过所有CFB的运算，可获得第二子网络120对应的最终的耦合反馈结果

将上述过程用公式表征为：

和

其中，f_CFB()表示耦合反馈模块的操作。In some embodiments, each sub-network in the neural network includes multiple coupled feedback modules CFB, and the multiple CFBs are cyclically connected in series, as shown in FIG. 1 . In this embodiment, it is assumed that there are T CFBs in each sub-network, then, the process of generating the coupling feedback result corresponding to the first sub-network 110 is:

The underexposed high-level feature ^{Gu and the overexposed high-level feature G o are} input to the first coupling feedback module 113 in the first sub-network 110 to generate the coupling feedback result corresponding to the first sub-network

For any subsequent coupling feedback module 113 (serial number t) in the first sub-network except the first coupling feedback module 113, the underexposed low-level features are

The coupling feedback result of the previous adjacent coupling feedback module (serial number t-1) of the subsequent coupling feedback module

and the coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in the second sub-network 120

Input the subsequent coupling feedback module 113 to generate the coupling feedback result corresponding to the first sub-network 110

According to this process, after all CFB operations, the final coupling feedback result corresponding to the first sub-network 110 can be obtained

Similarly, the process of generating the coupling feedback result corresponding to the second sub-network 120 is as follows: overexpose low-level features

The overexposed high-level feature G ^o and the underexposed high-level feature ^Gu are input to the first coupling feedback module 123 in the second sub-network 120 to generate the coupling feedback result corresponding to the second sub-network 120

For any subsequent coupling feedback module 123 (serial number is t) in the second sub-network 120 except for the first coupling feedback module 123, the overexposure low-level features

The coupling feedback result of the previous adjacent coupling feedback module of the subsequent coupling feedback module

and the coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in the first sub-network 110

Input the subsequent coupling feedback module 123 to generate the coupling feedback result corresponding to the second sub-network 120

According to this process, after all CFB operations, the final coupling feedback result corresponding to the second sub-network 120 can be obtained

The above process is represented by the formula as:

and

where f _CFB ( ) represents the operation of the coupled feedback block.

In some embodiments, the number of the coupled feedback modules 113 and the coupled feedback modules 123 is three. This can better balance the computational speed and model accuracy of the neural network.

In some embodiments, the internal network structure of each coupled feedback module CFB is the same, but does not share model parameters. Referring to FIG. 3 , the t-th coupling feedback module 123 in the second sub-network 120 is used as an example to illustrate its internal structure and the interconnection with other modules. The coupling feedback module 123 includes at least two coupling sub-modules 310 and at least two feature map groups 320 . Same as FIG. 2 , the plurality of feature map groups 320 are continuously connected in a dense manner, and each feature map group 320 includes a filter, a deconvolution layer Deconv and a convolution layer Conv to achieve continuous up and down sampling. The first connection sub-module 310 is located before all feature map groups 320; any other connection sub-module 310 except the first connection sub-module is located between any two adjacent feature map groups 320, and any The two other bonding submodules 310 are located at different locations.

第t-1个CFB提取的耦合反馈结果

以及第一子网络110中第t-1个CFB提取的耦合反馈结果

其中，用于反馈的特征

是从同一个子网络中获取到的反馈信息，故其主要功能是修正过曝低层次特征

以进一步提高超分辨的效果；而用于反馈的特征

是从另一个子网络中得到的反馈信息，其主要功能是带来互补的信息，以提高多曝光图像融合的效果。The t-th CFB has three input data, which are low-level features of overexposure

Coupled feedback results of the t-1th CFB extraction

and the coupling feedback result extracted by the t-1th CFB in the first sub-network 110

Among them, the features used for feedback

is the feedback information obtained from the same sub-network, so its main function is to correct the low-level features of overexposure

to further improve the effect of super-resolution; and the features used for feedback

is the feedback information obtained from another sub-network, and its main function is to bring complementary information to improve the effect of multi-exposure image fusion.

其中，

表示基于三个输入特征进行滤波融合后的低分辨率的特征，M_in表示一系列1×1的滤波器，

表示内部元素的联结。之后，基于滤波融合后的特征

利用一系列特征映射组320来重复执行上采样Deconv和下采样Conv的操作，每个上采样操作可获得高分辨率的特征

每次下采样可获得低分辨率的特征

最终以递进式地提取出更有效的高层次特征。The processing procedure of the above-mentioned t-th coupling feedback module 123 is as follows: first, in the dimension of the number of channels, the connection sub-module 310 is used to connect the features of the three inputs. Then, the concatenated results are fused with a series of 1×1 filters:

in,

Represents the low-resolution feature after filtering and fusion based on three input features, M _in represents a series of 1×1 filters,

Represents a join of inner elements. After that, based on the filtered and fused features

The operations of upsampling Deconv and downsampling Conv are repeatedly performed using a series of feature map groups 320, each upsampling operation can obtain high-resolution features

Low-resolution features are obtained with each downsampling

Finally, more effective high-level features are extracted progressively.

的主要功能是带来互补的信息以提高多曝光图像融合的效果的说明，同时考虑到随着特征映射组的数量的增加，模块内部对于特征

的记忆逐渐被遗忘，使得

产生的影响在逐渐下降，导致后续融合的效果不佳，所以，本实施例中为了增强

对于网络的影响，除了将

作为每个CFB的输入数据之外，还将它植入特征映射组320之间来重新激活CFB模块对其的记忆，即在CFB的特征映射组之间增加联结子模块310，增加的联结子模块310的输入数据均包含

具体实施时，在每个CFB中设置至少两个联结子模块310，并且除第一个联结子模块之外的其他联结子模块设置在不同的特征映射组320之间。如果对神经网络运算速度无要求，可以设置超过两个的联结子模块310，甚至可以在每两个特征映射组310之间均增加一个联结子模块310，这样能够更大程度上提高融合效果。如果对神经网络的运算速度和运算精度都有较高的要求，那么为了均衡速度和精度，可以只设置两个联结子模块310，第二个联结子模块310设置在多个特征映射组320的中间位置。例如，假设总的特征映射组个数是N，反馈特征

和

被联结起来构成一张新的低分辨率LR的特征图

其中，

表示向下取整的操作。该新的低分辨率的特征图

将替代

作为后续的特征映射组的输入特征。During the operation of the feature map group 320, based on the above

The main function of is to bring complementary information to improve the effect of multi-exposure image fusion, taking into account that as the number of feature map groups increases, the internal

memory is gradually forgotten, making

The resulting influence is gradually decreasing, resulting in a poor effect of subsequent fusion. Therefore, in this embodiment, in order to enhance the

The impact on the network, in addition to the

In addition to the input data of each CFB, it is also implanted between the feature map groups 320 to reactivate the memory of the CFB module. The input data of module 310 all contain

During specific implementation, at least two connection sub-modules 310 are set in each CFB, and other connection sub-modules except the first connection sub-module are set between different feature map groups 320 . If there is no requirement for the operation speed of the neural network, more than two connection sub-modules 310 can be set, and even a connection sub-module 310 can be added between every two feature map groups 310, which can improve the fusion effect to a greater extent. If there are high requirements for the operation speed and operation accuracy of the neural network, in order to balance the speed and accuracy, only two connection sub-modules 310 may be provided, and the second connection sub-module 310 is arranged in the plurality of feature mapping groups 320. in the middle. For example, assuming that the total number of feature map groups is N, the feedback features

and

are concatenated to form a new low-resolution LR feature map

in,

Represents an operation to round down. The new low-resolution feature map

will replace

as input features for subsequent feature map groups.

其中，M_out()表示用一系列1×1的滤波器进行卷积的操作。Finally, after the operations of N feature map groups 320, the LR feature maps of each feature map group 320 are aggregated together, and fused by a series of 1×1 filters to obtain the final output result of the CFB

Among them, M _out ( ) represents the operation of convolution with a series of 1×1 filters.

和第二融合曝光高分辨率图像

任一融合曝光高分辨率图像均具有高动态范围HDR和高分辨率HR的特点。需要说明的是，在多个CFB循环连接的实施例中，每个CFB均会输出一个耦合反馈结果，但是考虑到各个CFB的串行反馈处理会逐步提升图像融合和超分辨的效果，故最后一个CFB所得的耦合反馈结果是综合效果最好的。基于此，在获得第一融合曝光高分辨率图像

和第二融合曝光高分辨率图像

的过程中，将每个子网络中最后一个CFB所得的耦合反馈结果对应的重建的图像作为输入之一。另外，耦合反馈结果的特征尺寸要大于原始输入图像的图像尺寸，故可先利用诸如双三次插值的上采样操作来扩大原始输入图像的图像尺寸，再将上采样的结果作为获得融合曝光高分辨率图像的另一输入。上述过程用公式表征为：

和

其中，f_UP()和f_REC()分别表示上采样操作和图像重建操作。In some embodiments, the first sub-network 110 and the second sub-network 120 include an image reconstruction module (REC) 114 and an image reconstruction module 124, respectively, for reconstructing the coupling feedback result (feature) obtained by at least one CFB for the image. Then, multiple CFBs can obtain multiple reconstructed images. On this basis, the original input image of the neural network can be further fused to obtain the first fusion exposure high-resolution image

and second fused exposure high resolution image

Any fused exposure high-resolution image is characterized by high dynamic range HDR and high resolution HR. It should be noted that, in the embodiment in which multiple CFBs are cyclically connected, each CFB will output a coupling feedback result, but considering that the serial feedback processing of each CFB will gradually improve the effect of image fusion and super-resolution, so finally The coupled feedback result obtained by a CFB is the best overall result. Based on this, the first fusion exposure high-resolution image is obtained

and second fused exposure high resolution image

During the process, the reconstructed image corresponding to the coupling feedback result obtained by the last CFB in each sub-network is used as one of the inputs. In addition, the feature size of the coupled feedback result is larger than the image size of the original input image, so an upsampling operation such as bicubic interpolation can be used to enlarge the image size of the original input image, and then the result of the upsampling can be used to obtain high-resolution fusion exposure. Another input for the rate image. The above process is represented by the formula:

and

where f _UP ( ) and f _REC ( ) represent the upsampling operation and the image reconstruction operation, respectively.

Based on the above description, the parameter settings of each part of the neural network provided by the embodiments of the present disclosure can be exemplified as follows:

和欠曝高分辨率图像

输出第一子网络110中其他耦合反馈模块对应的融合曝光高分辨率图像

和

以及输出第二子网络120中其他耦合反馈模块对应的融合曝光高分辨率图像

和

FIG. 4 is a network architecture diagram of a neural network for neural network training provided by an embodiment of the present disclosure. Based on the architecture of the neural network in Figure 1, there are multi-level features in the neural network, such as low-level features, high-level features, and at least one coupled feedback result (feature), all of which are used to simultaneously achieve multi-exposure image fusion and superimposition. Discrimination techniques, so, to ensure the validity of each resulting feature, a hierarchical loss function constraint is employed during neural network training. The layered loss function requires the images of each layer to be calculated, so the network architecture of the neural network used for neural network training in Figure 4 is compared with the network architecture used for image fusion prediction in Figure 1, adding multiple images Output branches, such as outputting overexposed high-resolution images corresponding to high-level features

and underexposed high-resolution images

Output the fused exposure high-resolution images corresponding to other coupling feedback modules in the first sub-network 110

and

and outputting the fused exposure high-resolution images corresponding to other coupling feedback modules in the second sub-network 120

and

FIG. 5 is a flowchart of a neural network training method provided by an embodiment of the present disclosure. The neural network training method is implemented based on the neural network architecture in FIG. 4 , and the explanations of the same or corresponding contents as those of the above-mentioned embodiments are not repeated here. The neural network training method provided by the embodiments of the present disclosure may be performed by a neural network training apparatus, which may be implemented in software and/or hardware, and the apparatus may be integrated into an electronic device with certain computing capabilities, such as a notebook computer, Desktop computers, servers or supercomputers, etc. Referring to Figure 5, the neural network training method specifically includes:

S110. Acquire an under-exposed low-resolution image and an over-exposed low-resolution image.

和一张过曝低分辨率图像

欠曝低分辨率图像是指拍摄曝光度小于第一预设曝光度阈值、且图像分辨率低于预设分辨率阈值的图像。过曝低分辨率图像是指拍摄曝光度高于第二预设曝光度阈值、且图像分辨率低于上述预设分辨率阈值的图像。这里的第一预设曝光度阈值小于第二预设曝光度阈值，并且第一预设曝光度阈值、第二预设曝光度阈值和预设分辨率阈值分别是预先确定的曝光度和图像分辨率。Specifically, multiple network trainings are required in the entire neural network training process, and each network training needs to obtain a training image group, then the entire training process needs to obtain multiple training image groups, and each training image group needs to be obtained. The training process is the same. In this embodiment, only one training process is described. One of the above training image sets contains an underexposed low-resolution image

and an overexposed low-res image

The underexposed low-resolution image refers to an image whose exposure is lower than the first preset exposure threshold and the image resolution is lower than the preset resolution threshold. An overexposed low-resolution image refers to an image whose exposure is higher than the second preset exposure threshold and whose image resolution is lower than the above-mentioned preset resolution threshold. Here, the first preset exposure threshold is smaller than the second preset exposure threshold, and the first preset exposure threshold, the second preset exposure threshold and the preset resolution threshold are the predetermined exposure and image resolution, respectively. Rate.

S120. Input the under-exposed low-resolution image and the over-exposed low-resolution image into the initial feature extraction modules in the first sub-network and the second sub-network, respectively, to generate under-exposed low-level features and over-exposed low-level features.

输入第一子网络中的初始特征提取模块FEB，得到欠曝低层次特征

将过曝低分辨率图像

输入第二子网络中的初始特征提取模块FEB，得到过曝低层次特征

Specifically, underexposing low-resolution images

Input the initial feature extraction module FEB in the first sub-network to obtain underexposed low-level features

will overexpose low resolution images

Input the initial feature extraction module FEB in the second sub-network to obtain overexposed low-level features

S130. Input the under-exposed low-level features and over-exposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network respectively, to generate under-exposed high-level features and over-exposed high-level features.

输入第一子网络中的高层特征提取模块SRB，得到欠曝高层次特征G^u。将过曝低层次特征

输入第二子网络中的高层特征提取模块SRB，得到过曝高层次特征G^o。Specifically, underexposing low-level features

Input the high-level feature extraction module SRB in the first sub-network to obtain the underexposed high ^- level feature Gu. will overexpose low-level features

Input the high-level feature extraction module SRB in the second sub-network to obtain the overexposed high-level feature G ^o .

S140. Input the under-exposed low-level features, under-exposed high-level features, and over-exposed high-level features into a coupling feedback module in the first sub-network to generate a coupling feedback result corresponding to the first sub-network.

欠曝高层次特征G^u和过曝高层次特征G^o为基础的输入特征，通过第一子网络中的至少一个耦合反馈模块CFB的处理，生成第一子网络对应的至少一个耦合反馈结果。Specifically, to underexpose low-level features

The input features based on the underexposed high-level feature ^{Gu and the overexposed high-level feature G o are} processed by at least one coupling feedback module CFB in the first sub-network to generate at least one coupling feedback result corresponding to the first sub-network.

In some embodiments, S140 may be implemented as: inputting the under-exposed low-level features, under-exposed high-level features, and over-exposed high-level features into the first coupling feedback module in the first sub-network to generate a first sub-network corresponding to for any subsequent coupling feedback module except the first coupling feedback module in the first sub-network, underexposing the low-level features, the coupling feedback module of the previous adjacent coupling feedback module of the subsequent coupling feedback module The feedback result and the coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in the second sub-network are input to the subsequent coupling feedback module to generate the coupling feedback result corresponding to the first sub-network. In this embodiment, the neural network includes a plurality of coupling feedback modules CFB (T are taken as an example), and each coupling feedback module is processed in series.

欠曝高层次特征G^u和过曝高层次特征G^o，输入该第一个CFB后，输出第一子网络对应的第一个耦合反馈结果

然后，对于第一子网络中除了第一个CFB之外的某个后续CFB(假设为第t个，且t<T)，将欠曝低层次特征

该第t个CFB的前一相邻CFB(即第t-1个CFB)的耦合反馈结果

以及第二子网络中第t-1个CFB的耦合反馈结果

输入该第t个CFB后，输出第一子网络对应的第t个耦合反馈结果

以此类推，通过迭代反馈，可获得第一子网络中任一后续CFB输出的耦合反馈结果。Referring to FIG. 4 , the process of generating at least one coupled feedback result corresponding to the first sub-network is as follows: first, for the first CFB, underexposing the low-level features

Under-exposed high-level feature ^{Gu and over-exposed high-level feature G o ,} after inputting the first CFB, output the first coupling feedback result corresponding to the first sub-network

Then, for a certain subsequent CFB in the first sub-network other than the first CFB (assumed to be the t-th, and t<T), the low-level features will be underexposed

The coupling feedback result of the previous adjacent CFB of the t-th CFB (ie, the t-1-th CFB)

and the coupled feedback result of the t-1th CFB in the second sub-network

After inputting the t-th CFB, output the t-th coupling feedback result corresponding to the first sub-network

By analogy, through iterative feedback, the coupling feedback result of any subsequent CFB output in the first sub-network can be obtained.

S150. Input the overexposed low-level feature, the overexposed high-level feature, and the underexposed high-level feature into the coupling feedback module in the second sub-network to generate a coupling feedback result corresponding to the second sub-network.

过曝高层次特征G^o和欠曝高层次特征G^u为基础的输入特征，通过第二子网络中的至少一个耦合反馈模块CFB的处理，生成第二子网络对应的至少一个耦合反馈结果。Specifically, to overexpose low-level features

The input features based on the overexposed high ^- level feature ^Go and the underexposed high-level feature Gu are processed by at least one coupling feedback module CFB in the second sub-network to generate at least one coupling feedback result corresponding to the second sub-network.

In some embodiments, S150 may be implemented as: inputting the overexposed low-level feature, overexposed high-level feature, and underexposed high-level feature into the first coupling feedback module in the second sub-network to generate a corresponding second sub-network For any subsequent coupling feedback module in the second sub-network except the first coupling feedback module, the coupling feedback module of the previous adjacent coupling feedback module of the subsequent coupling feedback module will be overexposed. The feedback result and the coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in the first sub-network are input to the subsequent coupling feedback module to generate the coupling feedback result corresponding to the second sub-network. In this embodiment, the neural network includes a plurality of coupling feedback modules CFB (T are taken as an example), and each coupling feedback module is processed in series.

过曝高层次特征G^o和欠曝高层次特征G^u，输入该第一个CFB后，输出第二子网络对应的第一个耦合反馈结果

然后，对于第二子网络中除了第一个CFB之外的某个后续CFB(假设为第t个，且t<T)，将过曝低层次特征

第t-1个CFB的耦合反馈结果

以及第一子网络中第t-1个CFB的耦合反馈结果

输入该第t个CFB后，输出第二子网络对应的第t个耦合反馈结果

以此类推，通过迭代反馈，可获得第二子网络中任一后续CFB输出的耦合反馈结果。Referring to FIG. 4 , the process of generating at least one coupled feedback result corresponding to the second sub-network is as follows: first, for the first CFB, the low-level features are overexposed

Overexposed high-level feature G ^o and underexposed high-level feature ^Gu , after inputting the first CFB, output the first coupling feedback result corresponding to the second sub-network

Then, for some subsequent CFB other than the first CFB in the second sub-network (assumed to be the t-th, and t<T), the low-level features will be overexposed

Coupling feedback results of the t-1th CFB

and the coupled feedback result of the t-1th CFB in the first sub-network

After inputting the t-th CFB, output the t-th coupling feedback result corresponding to the second sub-network

By analogy, through iterative feedback, the coupling feedback result of any subsequent CFB output in the second sub-network can be obtained.

S160. Based on the coupling feedback result corresponding to the underexposed low-resolution image, the underexposed high-level feature and the first sub-network, and the coupling feedback result corresponding to the over-exposed low-resolution image, the overexposed high-level feature and the second sub-network, Adjust the parameters of the neural network.

欠曝高层次特征G^u、第一子网络对应的各个耦合反馈结果

过曝低分辨率图像

过曝高层次特征G^o和第二子网络对应的各个耦合反馈结果

来确定神经网络各层输出的图像，并利用这些输出的图像来计算该次训练的损失值，进而利用该损失值来调整神经网络中的模型参数。Specifically, according to the above description, in the embodiment of the present disclosure, a layered loss function is used to train the above neural network, so it needs to be based on the underexposed low-resolution image

Under-exposed high-level feature ^Gu and each coupling feedback result corresponding to the first sub-network

Overexposed low-resolution images

Overexposure high-level feature G ^o and each coupling feedback result corresponding to the second sub-network

To determine the images output by each layer of the neural network, and use these output images to calculate the loss value of this training, and then use the loss value to adjust the model parameters in the neural network.

In some embodiments, S160 may be implemented as:

A. Perform up-sampling operations on the under-exposed low-resolution image and the over-exposed low-resolution image respectively.

和过曝低分辨率图像

但是，因为高层次特征和耦合反馈结果均增加了超分辨的更多图像细节信息，其特征大小均大于原始输入图像，故需要将原始输入图像上采样以扩大图像尺寸。例如，分别对欠曝低分辨率图像

和过曝低分辨率图像

进行双三次插值上采样的操作，获得上采样后的欠曝低分辨率图像

和上采样后的过曝低分辨率图像

Specifically, in order to further improve the image fusion effect, the images output by each layer of the neural network in the embodiment of the present disclosure need to be fused with the original input image, that is, the underexposed low-resolution image

and overexposed low-resolution images

However, because both high-level features and coupled feedback results add more image detail information for super-resolution, and their feature sizes are larger than the original input image, it is necessary to upsample the original input image to enlarge the image size. For example, for underexposed low-resolution images separately

and overexposed low-resolution images

Perform a bicubic interpolation and upsampling operation to obtain an underexposed low-resolution image after upsampling

and upsampled overexposed low-resolution images

B. Add the image corresponding to the under-exposed high-level feature and the image corresponding to the coupling feedback result corresponding to the first sub-network to the up-sampled under-exposed low-resolution image to generate an under-exposed high-resolution image and a second under-exposed high-resolution image. The sub-network corresponds to the fused exposure high-resolution image.

均施加图像重建模块REC的操作，获得相应的图像。然后，将欠曝高层次特征G^u对应的图像和上采样后的欠曝低分辨率图像

相加，得到欠曝高分辨率图像

并且，将上采样后的欠曝低分辨率图像

分别与第一子网络中的每个耦合反馈结果

对应的图像相加，得到第一子网络对应的各个融合曝光高分辨率图像

Specifically, first, each coupling feedback result corresponding to the underexposed high ^- level feature Gu and the first sub-network is

The operation of the image reconstruction module REC is applied to obtain the corresponding image. Then, the image corresponding to the under ^- exposed high-level feature Gu and the up-sampled under-exposed low-resolution image

Add to get underexposed high-resolution images

And, the upsampled underexposed low-resolution image

Separate feedback results with each coupling in the first sub-network

The corresponding images are added to obtain each fusion exposure high-resolution image corresponding to the first sub-network

C. Add the image corresponding to the over-exposed high-level feature and the image corresponding to the coupling feedback result corresponding to the second sub-network respectively with the up-sampled over-exposed low-resolution image to generate the over-exposed high-resolution image and the second sub-network. The sub-network corresponds to the fused exposure high-resolution image.

均施加图像重建模块REC的操作，获得相应的图像。然后，将过曝高层次特征G^o对应的图像和上采样后的过曝低分辨率图像

相加，得到过曝高分辨率图像

并且，将上采样后的过曝低分辨率图像

分别与第二子网络中的每个耦合反馈结果

对应的图像相加，得到第二子网络对应的各个融合曝光高分辨率图像

Specifically, first, each coupling feedback result corresponding to the overexposed high-level feature ^Go and the second sub-network is

The operation of the image reconstruction module REC is applied to obtain the corresponding image. Then, the image corresponding to the overexposed high-level feature Go and the upsampled ^overexposed low-resolution image

Add up to get an overexposed high-resolution image

And, the upsampled overexposed low-resolution image

Separate feedback results with each coupling in the second sub-network

The corresponding images are added to obtain each fusion exposure high-resolution image corresponding to the second sub-network

D. Adjust the parameters of the neural network based on the under-exposed high-resolution image, the fused-exposure high-resolution image corresponding to the first sub-network, the over-exposed high-resolution image, and the fused-exposure high-resolution image corresponding to the second sub-network.

第一子网络对应的各融合曝光高分辨率图像

过曝高分辨率图像

和第二子网络对应的各融合曝光高分辨率图像

来计算该次训练的损失值，并利用该损失值的反向传播来调整神经网络中的模型参数。Specifically, using the underexposed high-resolution image obtained by the above process

High-resolution images of each fusion exposure corresponding to the first sub-network

Overexposed high-resolution images

Each fused exposure high-resolution image corresponding to the second sub-network

to calculate the loss value of this training, and use the back-propagation of the loss value to adjust the model parameters in the neural network.

In some embodiments, step D can be implemented as: adjusting the parameters of the neural network through the loss function shown in the following formula (1):

分别表示每一部分损失函数值对应的权重，

和

分别表示第一子网络中的高层特征提取模块和耦合反馈模块对应的损失函数值，

和

分别表示第二子网络中的高层特征提取模块和耦合反馈模块对应的损失函数值，L_MS表示基于图像的结构相似性指数确定的两个图像之间的损失值，

和

分别表示过曝高分辨率图像和过曝高分辨率参考图像，

和

分别表示欠曝高分辨率图像和欠曝高分辨率参考图像，

和I_gt分别表示第t个第二子网络对应的融合曝光高分辨率图像、第t个第一子网络对应的融合曝光高分辨率图像和融合曝光高分辨率参考图像，T表示耦合反馈模块的数量。where L _total represents the total loss function value, λ _o , λ _u and

respectively represent the weight corresponding to each part of the loss function value,

and

respectively represent the loss function values corresponding to the high-level feature extraction module and the coupled feedback module in the first sub-network,

and

respectively represent the loss function values corresponding to the high-level feature extraction module and the coupled feedback module in the second sub-network, L _MS represents the loss value between the two images determined based on the structural similarity index of the image,

and

represent the overexposed high-resolution image and the overexposed high-resolution reference image, respectively,

and

represent the underexposed high-resolution image and the underexposed high-resolution reference image, respectively,

and I _gt represent the fused exposure high-resolution image corresponding to the t-th second sub-network, the fused-exposure high-resolution image corresponding to the t-th first sub-network, and the fused-exposure high-resolution reference image, respectively, and T represents the coupling feedback module quantity.

Each of the above reference images is the true value image corresponding to the corresponding neural network output image, and is a target image that is expected to be as close as possible to the image generated by the above neural network. The above loss value L _MS representing the image-level structural similarity index (SSIM) between two images (X and Y) can be determined as follows:

和

用于保证高层特征提取模块SRB的有效性，最后一部分损失函数

用来确保耦合反馈模块CFB的有效性。即，前两个损失函数是为了确保超分辨的效果，而后一部分的构建用于同时保证超分辨和多曝光图像融合的效果。同时，前两个损失函数对于最后一部分的损失函数来说也是重要的基础。整个神经网络通过最小化公式(1)中定义的损失函数，以端到端的方式进行训练。The loss function in the above formula (1) can be divided into two parts. The first two loss functions

and

Used to ensure the effectiveness of the high-level feature extraction module SRB, the last part of the loss function

Used to ensure the effectiveness of the coupled feedback module CFB. That is, the first two loss functions are to ensure the effect of super-resolution, and the latter part is constructed to ensure the effect of super-resolution and multi-exposure image fusion at the same time. At the same time, the first two loss functions are also important foundations for the last part of the loss function. The entire neural network is trained in an end-to-end fashion by minimizing the loss function defined in Equation (1).

It should be noted that the execution order of S140 and S150 is not limited. S140 may be executed first and then S150 may be executed, or S150 may be executed first and then S140 may be executed, or S140 and S150 may be executed in parallel.

In the above-mentioned technical solutions of the embodiments of the present disclosure, by inputting the obtained underexposed low-resolution images and overexposed low-resolution images into the initial feature extraction modules in the first sub-network and the second sub-network, respectively, an underexposed low-resolution image is generated. Features and overexposed low-level features; input the under-exposed low-level features and over-exposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network, respectively, to generate under-exposed high-level features and over-exposed high-level features Features; input the under-exposed low-level features, under-exposed high-level features, and over-exposed high-level features into the coupling feedback module in the first sub-network to generate the coupling feedback results corresponding to the first sub-network; The over-exposed high-level features and under-exposed high-level features are input to the coupling feedback module in the second sub-network to generate the coupling feedback results corresponding to the second sub-network; based on the under-exposed low-resolution image, under-exposed high-level features and the first The coupling feedback results corresponding to the sub-network, as well as the over-exposure low-resolution images, over-exposure high-level features, and coupling feedback results corresponding to the second sub-network, adjust the parameters of the neural network. The end-to-end training of the neural network coupled with the multi-exposure fusion technology and the super-resolution technology is realized, and a neural network with more accurate parameters of each part of the module is obtained. speed, and take advantage of the complementary properties between multi-exposure fusion and super-resolution to further improve image processing accuracy.

FIG. 6 is a flowchart of an image fusion method provided by an embodiment of the present disclosure. The image fusion method is implemented based on the neural network architecture in FIG. 1 , and the explanations of the same or corresponding contents as those of the above-mentioned embodiments are not repeated here. The image fusion method provided in the embodiments of the present disclosure may be performed by an image fusion apparatus, which may be implemented in software and/or hardware, and the apparatus may be integrated into an electronic device with certain computing capabilities, such as a mobile phone, a tablet computer, a Laptops, desktops, servers or supercomputers, etc. Referring to Figure 6, the image fusion method includes:

S210. Acquire an under-exposed low-resolution image and an over-exposed low-resolution image.

Specifically, two extreme exposure images for the same shooting scene and the same shooting object, that is, an underexposed low-resolution image and an overexposed low-resolution image are acquired.

S220: Input the under-exposed low-resolution image and the over-exposed low-resolution image into a pre-trained neural network to generate a first fused-exposure high-resolution image and a second fused-exposure high-resolution image.

对应的第一融合曝光高分辨率图像

过曝低分辨率图像

对应的第二融合曝光高分辨率图像

Specifically, in the application process of the neural network, it only needs to input the under-exposed low-resolution image and the over-exposed low-resolution image, and after the processing of the network, it can output two images, namely the under-exposed low-resolution image.

Corresponding first fused exposure high-resolution image

Overexposed low-resolution images

Corresponding second fused exposure high-resolution image

S230. Generate an image fusion result based on the first fusion exposure high-resolution image and the second fusion exposure high-resolution image.

和第二融合曝光高分辨率图像

虽然都是超分辨和多曝光融合的图像，但是因其对应的输入图像不同，两个输出图像之间也有差异。为了进一步提高融合精度，本公开实施例中需要对

和

进行进一步的综合处理，获得最终的输出图像，即图像融合结果。Specifically, the first fusion exposure high-resolution image

and second fused exposure high resolution image

Although they are both super-resolution and multi-exposure fusion images, there are differences between the two output images because of their corresponding input images. In order to further improve the fusion accuracy, in the embodiments of the present disclosure, it is necessary to

and

Further comprehensive processing is performed to obtain the final output image, that is, the image fusion result.

和

进行加权处理，故需要预先确定两个加权权重，即第一权重和第二权重。这两个权重的取值与欠曝低分辨率图像和过曝低分辨率图像的曝光度和拍摄场景等有关。例如，可将0.5作为第一权重和第二权重的默认值。那么，可按照如下公式(2)来生成图像融合结果：In some embodiments, S230 may be implemented as: using the first weight and the second weight, respectively, to perform a weighted summation process on the first fused exposure high-resolution image and the second fused exposure high-resolution image to generate an image fusion result. Specifically, in this embodiment, the

and

For weighting processing, two weighting weights need to be pre-determined, namely the first weight and the second weight. The values of these two weights are related to the exposure and shooting scene of the underexposed low-resolution image and the overexposed low-resolution image. For example, 0.5 can be used as the default value for the first weight and the second weight. Then, the image fusion result can be generated according to the following formula (2):

Among them, I _out , _wo and _wu represent the image fusion result, the second weight and the first weight, respectively.

According to the above technical solutions of the embodiments of the present disclosure, the first fused exposure high-resolution image and the second fused exposure high-resolution image are generated by inputting the captured under-exposed low-resolution image and over-exposed low-resolution image into a pre-trained neural network. and generate an image fusion result based on the first fusion exposure high-resolution image and the second fusion exposure high-resolution image. Realize the use of a neural network coupled with multi-exposure fusion technology and super-resolution technology to process two low-resolution images with extreme exposure, and generate an image fusion result with high-resolution HR and high dynamic range HDR, which simplifies the captured image. The processing flow improves the image processing speed and processing accuracy.

FIG. 7 is a schematic structural diagram of a neural network training apparatus provided by an embodiment of the present disclosure. The neural network includes a first sub-network and a second sub-network with the same network structure, and any sub-network includes a primary feature extraction module, a high-level feature extraction module and a coupled feedback module. Referring to Figure 7, the device specifically includes:

An image acquisition unit 710, configured to acquire an under-exposed low-resolution image and an over-exposed low-resolution image;

The low-level feature generation unit 720 is configured to input the under-exposed low-resolution image and the over-exposed low-resolution image into the initial feature extraction modules in the first sub-network and the second sub-network, respectively, to generate under-exposed low-level features and over-exposed low-resolution images. Expose low-level features;

The high-level feature generation unit 730 is configured to input the under-exposed low-level features and over-exposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network respectively, to generate under-exposed high-level features and over-exposed high-level features. Hierarchical features;

The first coupling feedback result generating unit 740 is configured to input the under-exposed low-level features, under-exposed high-level features, and over-exposed high-level features into the coupling feedback module in the first sub-network to generate coupling feedback corresponding to the first sub-network result;

The second coupling feedback result generating unit 750 is configured to input the over-exposed low-level features, over-exposed high-level features, and under-exposed high-level features into the coupling feedback module in the second sub-network to generate coupling feedback corresponding to the second sub-network result;

The parameter adjustment unit 760 is used for coupling feedback results based on the underexposed low-resolution image, the underexposed high-level feature and the corresponding first sub-network, and the over-exposed low-resolution image, the overexposed high-level feature corresponding to the second sub-network The results of the coupled feedback to adjust the parameters of the neural network.

In some embodiments, the neural network includes multiple coupled feedback modules, and each coupled feedback module does not share model parameters.

In some embodiments, the coupled feedback modules are processed serially;

Correspondingly, the first coupling feedback result generating unit 740 is specifically configured to:

Input the under-exposed low-level features, under-exposed high-level features and over-exposed high-level features into the first coupling feedback module in the first sub-network to generate the coupling feedback result corresponding to the first sub-network;

For any subsequent coupling feedback module except the first coupling feedback module in the first sub-network, the underexposure low-level features, the coupling feedback result of the previous adjacent coupling feedback module of the subsequent coupling feedback module, and the The coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in the two sub-networks is input to the subsequent coupling feedback module to generate the coupling feedback result corresponding to the first sub-network;

Correspondingly, the second coupling feedback result generating unit 750 is specifically configured to:

Input the over-exposed low-level features, over-exposed high-level features and under-exposed high-level features into the first coupling feedback module in the second sub-network to generate the coupling feedback result corresponding to the second sub-network;

For any subsequent coupling feedback module except the first coupling feedback module in the second sub-network, the low-level features, the coupling feedback result of the previous adjacent coupling feedback module of the subsequent coupling feedback module, and the first coupling feedback module will be overexposed. The coupling feedback result of the coupling feedback module corresponding to the previous adjacent coupling feedback module in a sub-network is input to the subsequent coupling feedback module to generate the coupling feedback result corresponding to the second sub-network.

In some embodiments, the coupled feedback module includes at least two connection sub-modules and at least two feature map groups, wherein each feature map group includes a filter, a deconvolution layer, and a convolution layer;

The first link submodule is located before each feature map group;

Any other linking submodules except the first linking submodule are located between any two adjacent feature map groups, and any two other linking submodules are located at different positions.

In some embodiments, the parameter adjustment unit 760 is specifically used to:

Perform up-sampling operations on the under-exposed low-resolution image and the over-exposed low-resolution image respectively;

The image corresponding to the under-exposed high-level feature and the image corresponding to the coupling feedback result corresponding to the first sub-network are added to the up-sampled under-exposed low-resolution image to generate the under-exposed high-resolution image and the second sub-network. Corresponding fused exposure high-resolution images;

The image corresponding to the overexposed high-level feature and the image corresponding to the coupling feedback result corresponding to the second sub-network are added to the up-sampled over-exposed low-resolution image to generate the over-exposed high-resolution image and the second sub-network. Corresponding fused exposure high-resolution images;

The parameters of the neural network are adjusted based on the under-exposed high-resolution image, the fused-exposure high-resolution image corresponding to the first sub-network, the over-exposed high-resolution image, and the fused-exposure high-resolution image corresponding to the second sub-network.

Further, the parameter adjustment unit 760 is specifically used for:

Adjust the parameters of the neural network through the loss function shown in the following formula:

分别表示每一部分损失函数值对应的权重，

和

分别表示第一子网络中的高层特征提取模块和耦合反馈模块对应的损失函数值，

和

分别表示第二子网络中的高层特征提取模块和耦合反馈模块对应的损失函数值，L_MS表示基于图像的结构相似性指数确定的两个图像之间的损失值，

和

分别表示过曝高分辨率图像和过曝高分辨率参考图像，

和

分别表示欠曝高分辨率图像和欠曝高分辨率参考图像，

和I_gt分别表示第t个第二子网络对应的融合曝光高分辨率图像、第t个第一子网络对应的融合曝光高分辨率图像和融合曝光高分辨率参考图像，T表示耦合反馈模块的数量。where L _total represents the total loss function value, λ _o , λ _u and

respectively represent the weight corresponding to each part of the loss function value,

and

respectively represent the loss function values corresponding to the high-level feature extraction module and the coupled feedback module in the first sub-network,

and

respectively represent the loss function values corresponding to the high-level feature extraction module and the coupled feedback module in the second sub-network, L _MS represents the loss value between the two images determined based on the structural similarity index of the image,

and

represent the overexposed high-resolution image and the overexposed high-resolution reference image, respectively,

and

represent the underexposed high-resolution image and the underexposed high-resolution reference image, respectively,

and I _gt represent the fused exposure high-resolution image corresponding to the t-th second sub-network, the fused-exposure high-resolution image corresponding to the t-th first sub-network, and the fused-exposure high-resolution reference image, respectively, and T represents the coupling feedback module quantity.

Through the neural network training device provided by the embodiment of the present disclosure, the multi-exposure fusion processing and super-resolution processing of images are simultaneously performed by using a neural network, which not only simplifies the processing flow of captured images, improves the image processing speed, but also utilizes Complementary properties between multi-exposure fusion and super-resolution further improve image processing accuracy.

The neural network training apparatus provided by the embodiment of the present disclosure can execute the neural network training method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.

FIG. 8 is a schematic structural diagram of an image fusion apparatus provided by an embodiment of the present disclosure. Referring to Figure 8, the device specifically includes:

An image acquisition unit 810, configured to acquire an under-exposed low-resolution image and an over-exposed low-resolution image;

The fused exposure high-resolution image generation unit 820 is configured to input the under-exposed low-resolution image and the over-exposed low-resolution image into the pre-trained neural network to generate a first fused-exposure high-resolution image and a second fused-exposure high-resolution image Image; wherein, the neural network is obtained by training the neural network training method in any embodiment of the present disclosure;

The image fusion result generating unit 830 is configured to generate an image fusion result based on the first fusion exposure high-resolution image and the second fusion exposure high-resolution image.

In some embodiments, the image fusion result generating unit 830 is specifically configured to:

Using the first weight and the second weight respectively, the first fusion exposure high-resolution image and the second fusion exposure high-resolution image are weighted and summed to generate an image fusion result.

The image fusion device provided by the embodiment of the present disclosure realizes the simultaneous multi-exposure fusion processing and super-resolution processing of images by using a neural network, which not only simplifies the processing flow of captured images, improves the image processing speed, but also utilizes multiple Complementary properties between exposure fusion and super-resolution further improve image processing accuracy.

The image fusion apparatus provided by the embodiment of the present disclosure can execute the image fusion method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.

It is worth noting that in the above embodiments of the neural network training apparatus, the units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, each function The specific names of the units are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present disclosure.

Referring to FIG. 9, this embodiment provides an electronic device, which includes: one or more processors 920; and a storage device 910 for storing one or more programs, when the one or more programs are processed by one or more The processor 920 executes, so that one or more processors 920 implement the neural network training method provided by the embodiment of the present invention, where the neural network includes a first sub-network and a second sub-network with the same network structure, and any sub-network includes A primary feature extraction module, a high-level feature extraction module and a coupled feedback module; the method includes:

Obtain an underexposed low-resolution image and an overexposed low-resolution image;

Input the under-exposed low-resolution image and the over-exposed low-resolution image into the initial feature extraction modules in the first sub-network and the second sub-network, respectively, to generate under-exposed low-level features and over-exposed low-level features;

Input the under-exposed low-level features and over-exposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network, respectively, to generate under-exposed high-level features and over-exposed high-level features;

Input the under-exposed low-level features, under-exposed high-level features and over-exposed high-level features into the coupling feedback module in the first sub-network to generate the coupling feedback result corresponding to the first sub-network;

Input the over-exposed low-level features, over-exposed high-level features and under-exposed high-level features into the coupling feedback module in the second sub-network to generate the coupling feedback result corresponding to the second sub-network;

Based on the coupled feedback results corresponding to the underexposed low-resolution image, the underexposed high-level feature and the first sub-network, and the coupled feedback results corresponding to the over-exposed low-resolution image, the overexposed high-level feature and the second sub-network, adjust the neural parameters of the network.

Of course, those skilled in the art can understand that the processor 920 can also implement the technical solution of the neural network training method provided by any embodiment of the present invention.

The electronic device shown in FIG. 9 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present invention.

As shown in FIG. 9 , the electronic device includes a processor 920 , a storage device 910 , an input device 930 and an output device 940 ; the number of processors 920 in the electronic device may be one or more. In FIG. 9 , one processor 920 is used as the For example, the processor 920 , the storage device 910 , the input device 930 and the output device 940 in the electronic device may be connected by a bus or other means, and the connection by the bus 950 is taken as an example in FIG. 9 .

As a computer-readable storage medium, the storage device 910 may be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the neural network training method in the embodiment of the present invention.

The storage device 910 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Additionally, storage device 910 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some examples, storage device 910 may further include memory located remotely from processor 920, which may be connected to the electronic device through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 930 may be used to receive input numerical or character information, and generate key signal input related to user setting and function control of the electronic device. The output device 940 may include a display device such as a display screen.

The embodiment of the present invention also provides another electronic device, which includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that One or more processors implement the image fusion method provided by the embodiment of the present invention, including:

Obtain an underexposed low-resolution image and an overexposed low-resolution image;

Input the under-exposed low-resolution image and the over-exposed low-resolution image into a pre-trained neural network to generate a first fused-exposure high-resolution image and a second fused-exposure high-resolution image; wherein the neural network is implemented by any one of the present disclosure The neural network training method in the example is trained;

An image fusion result is generated based on the first fused exposure high-resolution image and the second fused exposure high-resolution image.

Of course, those skilled in the art can understand that the processor can also implement the technical solution of the image fusion method provided by any embodiment of the present invention. The hardware structure and function of the electronic device can be explained with reference to the content in FIG. 9 .

Embodiments of the present disclosure also provide a storage medium containing computer-executable instructions, the computer-executable instructions, when executed by a computer processor, are used to execute a neural network training method, where the neural network includes a first sub-system with the same network structure network and a second sub-network, and any sub-network includes a primary feature extraction module, a high-level feature extraction module and a coupled feedback module; the method includes:

Obtain an underexposed low-resolution image and an overexposed low-resolution image;

Input the under-exposed low-resolution image and the over-exposed low-resolution image into the initial feature extraction modules in the first sub-network and the second sub-network, respectively, to generate under-exposed low-level features and over-exposed low-level features;

Input the under-exposed low-level features and over-exposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network, respectively, to generate under-exposed high-level features and over-exposed high-level features;

Input the under-exposed low-level features, under-exposed high-level features and over-exposed high-level features into the coupling feedback module in the first sub-network to generate the coupling feedback result corresponding to the first sub-network;

Input the over-exposed low-level features, over-exposed high-level features and under-exposed high-level features into the coupling feedback module in the second sub-network to generate the coupling feedback result corresponding to the second sub-network;

Based on the coupled feedback results corresponding to the underexposed low-resolution image, the underexposed high-level feature and the first sub-network, and the coupled feedback results corresponding to the over-exposed low-resolution image, the overexposed high-level feature and the second sub-network, adjust the neural parameters of the network.

Of course, a storage medium containing computer-executable instructions provided by the embodiments of the present invention, the computer-executable instructions are not limited to the above method operations, and can also perform the relevant tasks in the neural network training method provided by any embodiment of the present invention. operate.

The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable mediums. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (a non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave, with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .

Program code embodied on a computer readable medium may be transmitted using any suitable medium, including - but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional procedural languages, or a combination thereof. Programming Language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

The embodiment of the present invention also provides another computer-readable storage medium, and the computer-executable instructions are used to execute an image fusion method when executed by a computer processor, including:

Obtain an underexposed low-resolution image and an overexposed low-resolution image;

Input the under-exposed low-resolution image and the over-exposed low-resolution image into a pre-trained neural network to generate a first fused-exposure high-resolution image and a second fused-exposure high-resolution image; wherein the neural network is implemented by any one of the present disclosure The neural network training method in the example is trained;

An image fusion result is generated based on the first fused exposure high-resolution image and the second fused exposure high-resolution image.

Of course, a storage medium containing computer-executable instructions provided by an embodiment of the present invention, the computer-executable instructions of which are not limited to the above method operations, and can also perform related operations in the image fusion method provided by any embodiment of the present invention . For the introduction of the storage medium, please refer to the content explanation in the above-mentioned embodiment.

It should be noted that the terminology used in the present disclosure is only for describing specific embodiments, rather than limiting the scope of the present application. As shown in this disclosure and the claims, unless the context clearly dictates otherwise, the words "a", "an", "an" and/or "the" are not intended to be specific in the singular and may include the plural. The term "and/or" includes any and all combinations of one or more of the associated listed items. Relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply that any such entity or operation exists between them. The actual relationship or sequence. The terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method or device comprising a list of elements includes not only those elements, but also other elements not expressly listed, Alternatively, elements inherent to such a process, method or apparatus may also be included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, or device that includes the element.

The above descriptions are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (12)

1. A neural network training method is characterized in that the neural network comprises a first sub-network and a second sub-network which have the same network structure, and any sub-network comprises a primary feature extraction module, a high-level feature extraction module and a coupling feedback module; the method comprises the following steps:

acquiring an under-exposed low-resolution image and an over-exposed low-resolution image;

inputting the underexposed low-resolution image and the overexposed low-resolution image into the initial feature extraction modules in the first sub-network and the second sub-network respectively to generate an underexposed low-level feature and an overexposed low-level feature;

inputting the underexposed low-level features and the overexposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network respectively to generate the underexposed high-level features and the overexposed high-level features;

inputting the underexposed low-level features, the underexposed high-level features and the overexposed high-level features into the coupling feedback module in the first sub-network to generate a coupling feedback result corresponding to the first sub-network;

inputting the over-exposed low-level features, the over-exposed high-level features and the under-exposed high-level features into the coupling feedback module in the second sub-network to generate a coupling feedback result corresponding to the second sub-network;

and adjusting parameters of the neural network based on the underexposed low-resolution image, the underexposed high-level features and the coupling feedback results corresponding to the first sub-network, and the overexposed low-resolution image, the overexposed high-level features and the coupling feedback results corresponding to the second sub-network.

2. The method of claim 1, wherein the neural network comprises a plurality of the coupled feedback modules, and wherein each of the coupled feedback modules does not share model parameters.

3. The method of claim 2, wherein each of the coupled feedback modules processes serially;

the inputting the underexposed low-level features, the underexposed high-level features, and the overexposed high-level features into the coupling feedback module in the first sub-network, and the generating of the coupling feedback result corresponding to the first sub-network includes:

inputting the underexposed low-level features, the underexposed high-level features and the overexposed high-level features into a first coupling feedback module in the first sub-network to generate a coupling feedback result corresponding to the first sub-network;

for any subsequent coupled feedback module in the first sub-network except the first coupled feedback module, inputting the underexposed low-level feature, the coupled feedback result of a previous adjacent coupled feedback module of the subsequent coupled feedback module, and the coupled feedback result of the coupled feedback module corresponding to the previous adjacent coupled feedback module in the second sub-network into the subsequent coupled feedback module, and generating the coupled feedback result corresponding to the first sub-network;

the inputting the overexposed low-level features, the overexposed high-level features, and the underexposed high-level features into the coupling feedback module in the second sub-network, and the generating of the coupling feedback result corresponding to the second sub-network includes:

inputting the over-exposed low-level features, the over-exposed high-level features and the under-exposed high-level features into a first coupling feedback module in the second sub-network to generate a coupling feedback result corresponding to the second sub-network;

and for any subsequent coupled feedback module except the first coupled feedback module in the second sub-network, inputting the over-exposed low-level feature, the coupled feedback result of a previous adjacent coupled feedback module of the subsequent coupled feedback module, and the coupled feedback result of the coupled feedback module corresponding to the previous adjacent coupled feedback module in the first sub-network into the subsequent coupled feedback module, and generating the coupled feedback result corresponding to the second sub-network.

4. The method of any one of claims 1 to 3, wherein the coupled feedback module comprises at least two concatenated sub-modules and at least two sets of signature maps, wherein each set of signature maps comprises a filter, an deconvolution layer and a convolution layer;

a first one of said join submodules precedes each of said sets of feature maps;

any other coupling submodule than the first coupling submodule is located between any two adjacent feature map sets, and any two other coupling submodules are located at different positions.

5. The method of claim 1, wherein the adjusting parameters of the neural network based on the under-exposed low resolution image, the under-exposed high level features and the coupled feedback results corresponding to the first sub-network, and the over-exposed low resolution image, the over-exposed high level features and the coupled feedback results corresponding to the second sub-network comprises:

respectively carrying out up-sampling operation on the under-exposed low-resolution image and the over-exposed low-resolution image;

adding the image corresponding to the under-exposed high-level features and the image corresponding to the coupling feedback result corresponding to the first sub-network to the up-sampled under-exposed low-resolution image respectively to generate an under-exposed high-resolution image and a fusion exposed high-resolution image corresponding to the second sub-network;

adding the image corresponding to the overexposure high-level feature and the image corresponding to the coupling feedback result corresponding to the second sub-network to the upsampled overexposure low-resolution image respectively to generate an overexposure high-resolution image and a fusion exposure high-resolution image corresponding to the second sub-network;

and adjusting parameters of the neural network based on the underexposed high-resolution image, the fusion exposed high-resolution image corresponding to the first sub-network, the overexposed high-resolution image and the fusion exposed high-resolution image corresponding to the second sub-network.

6. The method of claim 5, wherein adjusting parameters of the neural network based on the under-exposed high resolution image, the fused-exposed high resolution image corresponding to the first sub-network, the over-exposed high resolution image, and the fused-exposed high resolution image corresponding to the second sub-network comprises:

adjusting parameters of the neural network by a loss function as shown in the following equation:

wherein L is_totalExpressing the value of the total loss function, λ_o、λ_uAnd

the weight corresponding to each partial loss function value is respectively represented,

and

respectively representing the high-level feature extraction module and the coupled feedback module in the first subnetworkThe value of the loss function to which the block corresponds,

and

respectively represent the loss function values, L, corresponding to the high-level feature extraction module and the coupling feedback module in the second sub-network_MSRepresenting a loss value between two images determined based on the structural similarity index of the images,

and

respectively representing the overexposed high resolution image and the overexposed high resolution reference image,

and

respectively representing the under-exposed high resolution image and the under-exposed high resolution reference image,

and I_gtAnd the fusion exposure high-resolution images corresponding to the tth second sub-network, the fusion exposure high-resolution images corresponding to the tth first sub-network and the fusion exposure high-resolution reference images are respectively represented, and T represents the number of the coupling feedback modules.

7. An image fusion method, comprising:

acquiring an under-exposed low-resolution image and an over-exposed low-resolution image;

inputting the underexposed low-resolution image and the overexposed low-resolution image into a pre-trained neural network to generate a first fusion exposure high-resolution image and a second fusion exposure high-resolution image; wherein the neural network is trained by the neural network training method of any one of claims 1-6;

and generating an image fusion result based on the first fusion exposure high-resolution image and the second fusion exposure high-resolution image.

8. The method of claim 7, wherein generating an image fusion result based on the first and second fused-exposure high-resolution images comprises:

and respectively utilizing a first weight and a second weight to carry out weighted summation processing on the first fusion exposure high-resolution image and the second fusion exposure high-resolution image so as to generate the image fusion result.

9. A neural network training device is characterized in that the neural network comprises a first sub-network and a second sub-network which have the same network structure, and any sub-network comprises a primary feature extraction module, a high-level feature extraction module and a coupling feedback module; the device comprises:

an image acquisition unit for acquiring an under-exposed low-resolution image and an over-exposed low-resolution image;

a low-level feature generation unit, configured to input the under-exposed low-resolution image and the over-exposed low-resolution image into the initial feature extraction modules in the first sub-network and the second sub-network, respectively, and generate an under-exposed low-level feature and an over-exposed low-level feature;

a high-level feature generation unit configured to input the underexposed low-level features and the overexposed low-level features into the high-level feature extraction modules in the first sub-network and the second sub-network, respectively, and generate the underexposed high-level features and the overexposed high-level features;

a first coupling feedback result generating unit, configured to input the underexposed low-level features, the underexposed high-level features, and the overexposed high-level features into the coupling feedback module in the first sub-network, and generate a coupling feedback result corresponding to the first sub-network;

a second coupling feedback result generating unit, configured to input the overexposed low-level feature, the overexposed high-level feature, and the underexposed high-level feature into the coupling feedback module in the second sub-network, and generate a coupling feedback result corresponding to the second sub-network;

and the parameter adjusting unit is used for adjusting the parameters of the neural network based on the underexposed low-resolution image, the underexposed high-level feature and the coupling feedback result corresponding to the first sub-network, and the overexposed low-resolution image, the overexposed high-level feature and the coupling feedback result corresponding to the second sub-network.

10. An image fusion apparatus, comprising:

an image acquisition unit for acquiring an under-exposed low-resolution image and an over-exposed low-resolution image;

the fusion exposure high-resolution image generating unit is used for inputting the underexposure low-resolution image and the overexposure low-resolution image into a pre-trained neural network to generate a first fusion exposure high-resolution image and a second fusion exposure high-resolution image; wherein the neural network is trained by the neural network training method of any one of claims 1-6;

an image fusion result generating unit configured to generate an image fusion result based on the first fusion-exposed high-resolution image and the second fusion-exposed high-resolution image.

11. An electronic device, characterized in that the electronic device comprises:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the neural network training method of any one of claims 1-6 or the image fusion method of any one of claims 7-8.

12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the neural network training method of any one of claims 1 to 6 or the image fusion method of any one of claims 7 to 8.

Images