WO2024188171A1 - Image processing method and related device thereof - Google Patents

Abstract

The present application discloses an image processing method and a related device thereof, capable of implementing better fusion of a plurality of low-dynamic-range (LDR) images so that a finally obtained high-dynamic-range (HDR) image does not have any artifacts. The method of the present application comprises: when an HDR image of a target object needs to be acquired, a first LDR image of the target object and a second LDR image of the target object can be collected first, and the first LDR image and the second LDR image are input into a target model; thus, the target model can perform image block matching on the first LDR image and the second LDR image, so as to obtain a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image; and then, the target model can fuse the first LDR image and the second LDR image by using the correspondence, so as to obtain and output the HDR image of the target object.

Description

Image processing method and related device

This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on March 15, 2023, with application number 202310277464.6 and invention name “An image processing method and related equipment”, the entire contents of which are incorporated by reference in this application.

Technical Field

The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular, to an image processing method and related equipment.

Background Art

As a key issue in computer vision applications, high dynamic range (HDR) imaging has received more and more attention. With the rapid development of AI technology, more and more device manufacturers have built-in neural network models in AI technology into their devices to obtain high-quality HDR images through the models.

In the related art, a target object in a scene may be photographed at different exposure rates to obtain multiple low dynamic range (LDR) images of the target object. Then, the multiple LDR images may be input into a neural network model to fuse the multiple LDR images through the neural network model to obtain an HDR image of the target object.

In the above process, when shooting multiple LDR images, the target object may move in the scene, resulting in differences between the contents presented by the multiple LDR images. Therefore, the HDR image directly obtained by fusing the multiple LDR images is prone to artifacts.

Summary of the invention

The embodiment of the present application provides an image processing method and related equipment, which can achieve better fusion of multiple LDR images, so that the final HDR image does not have artifacts.

A first aspect of an embodiment of the present application provides an image processing method, which can be implemented by a target model. The method includes:

When it is necessary to obtain an HDR image of a target object, the target object may be photographed at different exposure rates, thereby acquiring a first LDR image of the target object and a second LDR image of the target object. It should be noted that the exposure rate used for photographing the second LDR image may be greater than the exposure rate used for photographing the first LDR image, or may be less than the exposure rate used for photographing the first LDR image.

After obtaining the first LDR image of the target object and the second LDR image of the target object, the first LDR image of the target object and the second LDR image of the target object can be input into the target model. After receiving the first LDR image of the target object and the second LDR image of the target object, the target model can perform image block matching on a plurality of first image blocks of the first LDR image of the target object and a plurality of second image blocks of the second LDR image of the target object, thereby establishing a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks.

After obtaining a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks, the target model can use the one-to-one correspondence between the multiple first image blocks and the multiple second image blocks to fuse the first LDR image and the second LDR image, thereby obtaining and outputting an HDR image of the target object.

It can be seen from the above method that when it is necessary to obtain an HDR image of the target object, the first LDR image of the target object and the second LDR image of the target object can be first collected, and the first LDR image and the second LDR image can be input into the target model. Then, the target model can match the image blocks of the first LDR image and the second LDR image, so as to obtain a one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image. Then, the target model can use the corresponding relationship to fuse the first LDR image and the second LDR image, so as to obtain and output the HDR image of the target object. In the above process, in the process of obtaining the one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image, it is equivalent to aligning the multiple first image blocks of the first LDR image with the multiple second image blocks of the second LDR image one by one in terms of content. Then, using this corresponding relationship as a guide to achieve the fusion between the first LDR image and the second LDR image, a better fusion can be achieved, so that the HDR image of the target object finally obtained does not have artifacts.

In a possible implementation, based on the first LDR image and the second LDR image, acquiring the correspondence between the plurality of first image blocks of the first LDR image and the plurality of second image blocks of the second LDR image comprises: based on the first LDR image and the second LDR image, acquiring the correspondence between the plurality of first image blocks of the first LDR image and the plurality of second image blocks of the second LDR image; The similarity between a plurality of first image blocks of an LDR image and a plurality of second image blocks of a second LDR image; based on the similarity, obtaining a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks. In the aforementioned implementation, after receiving the first LDR image of the target object and the second LDR image of the target object, the target model may perform a series of processing on the first LDR image and the second LDR image, thereby obtaining the similarity between the plurality of first image blocks of the first LDR image and the plurality of second image blocks of the second LDR image. After obtaining the similarity between the plurality of first image blocks and the plurality of second image blocks, the target model may use the similarity between the plurality of first image blocks and the plurality of second image blocks to accurately construct a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks.

In a possible implementation, based on the first LDR image and the second LDR image, obtaining the similarity between the plurality of first image blocks of the first LDR image and the plurality of second image blocks of the second LDR image includes: extracting features from the first LDR image and the second LDR image to obtain first features of the plurality of first image blocks of the first LDR image and second features of the plurality of second image blocks of the second LDR image; and calculating the first features of the plurality of first image blocks and the second features of the plurality of second image blocks to obtain the similarity between the plurality of first image blocks and the plurality of second image blocks. In the aforementioned implementation, after receiving the first LDR image of the target object and the second LDR image of the target object, the target model extracts features from the first LDR image and the second LDR image respectively, thereby correspondingly obtaining first features of the plurality of first image blocks of the first LDR image and second features of the plurality of second image blocks of the second LDR image. After obtaining the first features of the plurality of first image blocks and the second features of the plurality of second image blocks, the target model may also calculate the first features of the plurality of first image blocks and the second features of the plurality of second image blocks to accurately obtain the similarity between the plurality of first image blocks and the plurality of second image blocks.

In a possible implementation, the plurality of first image blocks include a third image block, and based on the similarity, obtaining a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks includes: based on the similarity between the third image block and the plurality of second image blocks, determining the second image block with the greatest similarity among the plurality of second image blocks as the fourth image block; and establishing a correspondence between the third image block and the fourth image block. In the aforementioned implementation, for any first image block of the plurality of first image blocks (i.e., the aforementioned third image block), the target model may select the second image block with the greatest similarity among the plurality of second image blocks based on the similarity between the first image block and the plurality of second image blocks as the second image block most similar to the first image block (i.e., the aforementioned fourth image block). Then, the target model may establish a correspondence between the first image block and the second image block most similar to the first image block. In addition, the target model may also perform the same operation as that performed on the first image block on the remaining first image blocks except the first image block among the plurality of first image blocks, so that a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks may be finally obtained.

In a possible implementation, based on the correspondence, the first LDR image and the second LDR image are fused to obtain a high dynamic range HDR image of the target object, including: extracting features from the first LDR image and the second LDR image to obtain a third feature of multiple first image blocks of the first LDR image and a fourth feature of multiple second image blocks of the second LDR image; adjusting the order of the fourth features of the multiple second image blocks based on the correspondence to obtain the fourth features of the multiple second image blocks after the order is adjusted; processing the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the order is adjusted to obtain the HDR image of the target object. In the aforementioned implementation, after receiving the first LDR image and the second LDR image, the target model may also extract features from the first LDR image and the second LDR image to obtain the third features of the multiple first image blocks of the first LDR image and the fourth features of the multiple second image blocks of the second LDR image. After obtaining the fourth features of the multiple second image blocks and the one-to-one correspondence between the multiple first image blocks and the multiple second image blocks, since the correspondence indicates a new ordering of the fourth features of the multiple second image blocks, the target model can adjust the ordering of the fourth features of the multiple second image blocks based on the correspondence, thereby obtaining the fourth features of the multiple second image blocks after the ordering is adjusted. Then, after obtaining the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the ordering is adjusted, the target model can perform a series of processing on the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the ordering is adjusted, thereby finally obtaining and externally outputting an HDR image of the target object.

In one possible implementation, processing the third feature of the plurality of first image blocks and the fourth feature of the plurality of second image blocks after adjustment and sorting to obtain an HDR image of the target object includes: processing the third feature of the plurality of first image blocks, the fourth feature of the plurality of second image blocks, and the fourth feature of the plurality of second image blocks after adjustment and sorting to obtain an HDR image of the target object. In the aforementioned implementation, after obtaining the third feature of the plurality of first image blocks, the fourth feature of the plurality of second image blocks, and the fourth feature of the plurality of second image blocks after adjustment and sorting, the target model may perform a series of processing on the third feature of the plurality of first image blocks, the fourth feature of the plurality of second image blocks, and the fourth feature of the plurality of second image blocks after adjustment and sorting, so as to finally obtain and output the HDR image of the target object externally.

In one possible implementation, the aforementioned processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing. In the aforementioned implementation, the target model can be based on the self-attention mechanism and the interactive attention mechanism to implement the first LDR image and the second LDR image. Image fusion. In the fusion process, the detail information of the first LDR image and the second LDR image can be effectively taken into account, so that the final HDR image maintains good details and has no artifacts.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

A second aspect of an embodiment of the present application provides a model training method, characterized in that the method includes: obtaining a first LDR image of a target object and a second LDR image of the target object, the first LDR image and the second LDR image being images obtained by photographing the target object based on different exposures; processing the first LDR image and the second LDR image by a model to be trained to obtain a high dynamic range HDR image of the target object, wherein the model to be trained is used to: obtain a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image based on the first LDR image and the second LDR image; based on the correspondence, fuse the first LDR image and the second LDR image to obtain an HDR image of the target object; and train the model to be trained based on the HDR image to obtain a target model.

The target model obtained by training the above method has an image processing function (for example, a function of fusing multiple LDR images into an HDR image, etc.). Specifically, when it is necessary to obtain an HDR image of the target object, the first LDR image of the target object and the second LDR image of the target object can be first collected, and the first LDR image and the second LDR image can be input into the target model. Then, the target model can match the image blocks of the first LDR image and the second LDR image, thereby obtaining a one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image. Then, the target model can use the corresponding relationship to fuse the first LDR image and the second LDR image, thereby obtaining and outputting the HDR image of the target object. In the above process, in the process of obtaining the one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image, it is equivalent to aligning the multiple first image blocks of the first LDR image with the multiple second image blocks of the second LDR image one by one in terms of content. Then, by using this correspondence as a guide to achieve the fusion between the first LDR image and the second LDR image, a better fusion can be achieved, so that the HDR image of the target object finally obtained does not have artifacts.

In one possible implementation, the model to be trained is used to: obtain the similarity between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image based on the first LDR image and the second LDR image; and obtain a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks based on the similarity.

In one possible implementation, the model to be trained is used to: extract features from a first LDR image and a second LDR image to obtain first features of multiple first image blocks of the first LDR image and second features of multiple second image blocks of the second LDR image; and calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain similarities between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, multiple first image blocks include a third image block, and the model to be trained is used to: based on the similarity between the third image block and multiple second image blocks, determine the second image block with the greatest similarity among the multiple second image blocks as the fourth image block; and establish a corresponding relationship between the third image block and the fourth image block.

In one possible implementation, the model to be trained is used to: extract features from the first LDR image and the second LDR image to obtain third features of multiple first image blocks of the first LDR image and fourth features of multiple second image blocks of the second LDR image; adjust the order of the fourth features of the multiple second image blocks based on the corresponding relationship to obtain fourth features of the multiple second image blocks after the order is adjusted; and process the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the order is adjusted to obtain an HDR image of the target object.

In one possible implementation, the model to be trained is used to process the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting to obtain an HDR image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

A third aspect of an embodiment of the present application provides an image processing method, the method comprising: obtaining a first noisy image and a second noisy image of a target object, the first noisy image and the second noisy image being images obtained by photographing the target object based on different exposures; based on the first noisy image and the second noisy image, obtaining a one-to-one correspondence between multiple first image blocks of the first noisy image and multiple second image blocks of the second noisy image; based on the correspondence, fusing the first noisy image and the second noisy image to obtain a denoised image of the target object.

In one possible implementation, based on the first noisy image and the second noisy image, obtaining the correspondence between multiple first image blocks of the first noisy image and multiple second image blocks of the second noisy image includes: based on the first noisy image and the second noisy image, obtaining the similarity between the multiple first image blocks of the first noisy image and the multiple second image blocks of the second noisy image; based on the similarity, obtaining a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, based on the first noisy image and the second noisy image, obtaining the similarity between multiple first image blocks of the first noisy image and multiple second image blocks of the second noisy image includes: performing feature extraction on the first noisy image and the second noisy image to obtain first features of multiple first image blocks of the first noisy image and second features of multiple second image blocks of the second noisy image; and calculating the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain the similarity between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, the multiple first image blocks include a third image block, and based on the similarity, obtaining a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks includes: based on the similarity between the third image block and the multiple second image blocks, determining the second image block with the greatest similarity among the multiple second image blocks as the fourth image block; and establishing a correspondence between the third image block and the fourth image block.

In one possible implementation, based on the correspondence, the first noisy image and the second noisy image are fused to obtain a high dynamic range denoised image of the target object, including: extracting features from the first noisy image and the second noisy image to obtain a third feature of multiple first image blocks of the first noisy image and a fourth feature of multiple second image blocks of the second noisy image; based on the correspondence, adjusting the order of the fourth features of the multiple second image blocks to obtain the fourth features of the adjusted and sorted multiple second image blocks; processing the third features of the multiple first image blocks and the fourth features of the adjusted and sorted multiple second image blocks to obtain the denoised image of the target object.

In one possible implementation, processing the third feature of the multiple first image blocks and the fourth feature of the multiple second image blocks after adjustment and sorting to obtain a denoised image of the target object includes: processing the third feature of the multiple first image blocks, the fourth feature of the multiple second image blocks, and the fourth feature of the multiple second image blocks after adjustment and sorting to obtain a denoised image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

A fourth aspect of an embodiment of the present application provides an image processing method, the method comprising: acquiring a first low-resolution image of a target object and a second low-resolution image of the target object, the first low-resolution image and the second low-resolution image being images obtained by photographing the target object based on different exposures; based on the first low-resolution image and the second low-resolution image, acquiring a one-to-one correspondence between multiple first image blocks of the first low-resolution image and multiple second image blocks of the second low-resolution image; based on the correspondence, fusing the first low-resolution image and the second low-resolution image to obtain a high-resolution image of the target object.

In one possible implementation, based on the first low-resolution image and the second low-resolution image, obtaining the correspondence between multiple first image blocks of the first low-resolution image and multiple second image blocks of the second low-resolution image includes: based on the first low-resolution image and the second low-resolution image, obtaining the similarity between the multiple first image blocks of the first low-resolution image and the multiple second image blocks of the second low-resolution image; based on the similarity, obtaining a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, based on the first low-resolution image and the second low-resolution image, obtaining the similarity between multiple first image blocks of the first low-resolution image and multiple second image blocks of the second low-resolution image includes: performing feature extraction on the first low-resolution image and the second low-resolution image to obtain first features of the multiple first image blocks of the first low-resolution image and second features of the multiple second image blocks of the second low-resolution image; and calculating the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain the similarity between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, the multiple first image blocks include a third image block, and based on the similarity, obtaining a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks includes: based on the similarity between the third image block and the multiple second image blocks, determining the second image block with the greatest similarity among the multiple second image blocks as the fourth image block; and establishing a correspondence between the third image block and the fourth image block.

In one possible implementation, based on the correspondence, the first low-resolution image and the second low-resolution image are fused to obtain a high-dynamic range high-resolution image of the target object, including: extracting features from the first low-resolution image and the second low-resolution image to obtain a third feature of multiple first image blocks of the first low-resolution image and a fourth feature of multiple second image blocks of the second low-resolution image; based on the correspondence, adjusting the order of the fourth features of the multiple second image blocks to obtain the fourth features of the adjusted and sorted multiple second image blocks; processing the third features of the multiple first image blocks and the fourth features of the adjusted and sorted multiple second image blocks to obtain a high-resolution image of the target object.

In one possible implementation, processing the third feature of the multiple first image blocks and the fourth feature of the multiple second image blocks after adjustment and sorting to obtain a high-resolution image of the target object includes: processing the third feature of the multiple first image blocks, the fourth feature of the multiple second image blocks, and the fourth feature of the multiple second image blocks after adjustment and sorting to obtain a high-resolution image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

A fifth aspect of an embodiment of the present application provides an image processing device, which includes a target model, and the device includes: a first acquisition module, used to acquire a first LDR image of a target object and a second LDR image of the target object, the first LDR image and the second LDR image being images obtained by photographing the target object based on different exposures; a second acquisition module, used to acquire, based on the first LDR image and the second LDR image, a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image; and a fusion module, used to fuse the first LDR image and the second LDR image based on the correspondence to obtain an HDR image of the target object.

It can be seen from the above device that when it is necessary to obtain an HDR image of the target object, the first LDR image of the target object and the second LDR image of the target object can be first collected, and the first LDR image and the second LDR image can be input into the target model. Then, the target model can match the image blocks of the first LDR image and the second LDR image, so as to obtain a one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image. Then, the target model can use the corresponding relationship to fuse the first LDR image and the second LDR image, so as to obtain and output the HDR image of the target object. In the above process, in the process of obtaining the one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image, it is equivalent to aligning the multiple first image blocks of the first LDR image with the multiple second image blocks of the second LDR image one by one in terms of content. Then, using this corresponding relationship as a guide to achieve the fusion between the first LDR image and the second LDR image, a higher quality fusion can be achieved, so that the HDR image of the target object finally obtained does not have artifacts.

In one possible implementation, the second acquisition module is used to: acquire the similarity between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image based on the first LDR image and the second LDR image; and acquire a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks based on the similarity.

In one possible implementation, the second acquisition module is used to: perform feature extraction on the first LDR image and the second LDR image to obtain first features of multiple first image blocks of the first LDR image and second features of multiple second image blocks of the second LDR image; and calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain similarities between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, the multiple first image blocks include a third image block, and the second acquisition module is used to: based on the similarity between the third image block and the multiple second image blocks, determine the second image block with the greatest similarity among the multiple second image blocks as the fourth image block; and establish a corresponding relationship between the third image block and the fourth image block.

In a possible implementation, the fusion module is used to extract features from the first LDR image and the second LDR image to obtain third features of multiple first image blocks of the first LDR image and fourth features of multiple second image blocks of the second LDR image; based on the corresponding relationship, adjust the order of the fourth features of the multiple second image blocks to obtain the adjusted fourth features of the multiple second image blocks; The third feature of the plurality of first image blocks and the fourth feature of the plurality of second image blocks after adjustment and sorting are processed to obtain an HDR image of the target object.

In one possible implementation, the fusion module is used to process the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting to obtain an HDR image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

A sixth aspect of an embodiment of the present application provides a model training device, which includes: an acquisition module, used to acquire a first LDR image of a target object and a second LDR image of the target object, the first LDR image and the second LDR image being images obtained by photographing the target object based on different exposures; a processing module, used to process the first LDR image and the second LDR image through a model to be trained to obtain a high dynamic range HDR image of the target object, wherein the model to be trained is used to: based on the first LDR image and the second LDR image, obtain a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image; based on the correspondence, fuse the first LDR image and the second LDR image to obtain an HDR image of the target object; and a training module, used to train the model to be trained based on the HDR image to obtain a target model.

The target model obtained by training the above-mentioned device has an image processing function (for example, a function of fusing multiple LDR images into an HDR image, etc.). Specifically, when it is necessary to obtain an HDR image of the target object, the first LDR image of the target object and the second LDR image of the target object can be first collected, and the first LDR image and the second LDR image can be input into the target model. Then, the target model can match the image blocks of the first LDR image and the second LDR image, thereby obtaining a one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image. Then, the target model can use the corresponding relationship to fuse the first LDR image and the second LDR image, thereby obtaining and outputting the HDR image of the target object. In the above process, in the process of obtaining the one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image, it is equivalent to aligning the multiple first image blocks of the first LDR image with the multiple second image blocks of the second LDR image one by one in terms of content. Then, by using this correspondence as a guide to achieve the fusion between the first LDR image and the second LDR image, a better fusion can be achieved, so that the HDR image of the target object finally obtained does not have artifacts.

In one possible implementation, the model to be trained is used to: obtain the similarity between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image based on the first LDR image and the second LDR image; and obtain a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks based on the similarity.

In one possible implementation, the model to be trained is used to: extract features from a first LDR image and a second LDR image to obtain first features of multiple first image blocks of the first LDR image and second features of multiple second image blocks of the second LDR image; and calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain similarities between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, multiple first image blocks include a third image block, and the model to be trained is used to: based on the similarity between the third image block and multiple second image blocks, determine the second image block with the greatest similarity among the multiple second image blocks as the fourth image block; and establish a corresponding relationship between the third image block and the fourth image block.

In one possible implementation, the model to be trained is used to: extract features from the first LDR image and the second LDR image to obtain third features of multiple first image blocks of the first LDR image and fourth features of multiple second image blocks of the second LDR image; adjust the order of the fourth features of the multiple second image blocks based on the corresponding relationship to obtain fourth features of the multiple second image blocks after the order is adjusted; and process the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the order is adjusted to obtain an HDR image of the target object.

In one possible implementation, the model to be trained is used to process the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting to obtain an HDR image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

A seventh aspect of an embodiment of the present application provides an image processing device, which includes a target model, and the device includes: a first acquisition module, used to acquire a first noisy image of a target object and a second noisy image of the target object, the first noisy image and the second noisy image being images obtained by photographing the target object based on different exposures; a second acquisition module, used to acquire, based on the first noisy image and the second noisy image, a one-to-one correspondence between multiple first image blocks of the first noisy image and multiple second image blocks of the second noisy image; and a fusion module, used to fuse the first noisy image and the second noisy image based on the correspondence to obtain a denoised image of the target object.

In one possible implementation, the second acquisition module is used to: acquire the similarity between multiple first image blocks of the first noisy image and multiple second image blocks of the second noisy image based on the first noisy image and the second noisy image; and acquire a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks based on the similarity.

In one possible implementation, the second acquisition module is used to: perform feature extraction on the first noisy image and the second noisy image to obtain first features of multiple first image blocks of the first noisy image and second features of multiple second image blocks of the second noisy image; calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain the similarity between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, the multiple first image blocks include a third image block, and the second acquisition module is used to: based on the similarity between the third image block and the multiple second image blocks, determine the second image block with the greatest similarity among the multiple second image blocks as the fourth image block; and establish a corresponding relationship between the third image block and the fourth image block.

In one possible implementation, a fusion module is used to extract features from a first noisy image and a second noisy image to obtain a third feature of multiple first image blocks of the first noisy image and a fourth feature of multiple second image blocks of the second noisy image; based on the corresponding relationship, the sorting of the fourth features of the multiple second image blocks is adjusted to obtain the fourth features of the multiple second image blocks after the adjusted sorting; and the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the adjusted sorting are processed to obtain a denoised image of the target object.

In one possible implementation, the fusion module is used to process the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting to obtain a denoised image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

An eighth aspect of an embodiment of the present application provides an image processing device, which includes a target model, and the device includes: a first acquisition module, used to acquire a first low-resolution image of a target object and a second low-resolution image of the target object, the first low-resolution image and the second low-resolution image being images obtained by photographing the target object based on different exposures; a second acquisition module, used to acquire, based on the first low-resolution image and the second low-resolution image, a one-to-one correspondence between multiple first image blocks of the first low-resolution image and multiple second image blocks of the second low-resolution image; and a fusion module, used to fuse the first low-resolution image and the second low-resolution image based on the correspondence to obtain a high-resolution image of the target object.

In one possible implementation, the second acquisition module is used to: acquire the similarity between multiple first image blocks of the first low-resolution image and multiple second image blocks of the second low-resolution image based on the first low-resolution image and the second low-resolution image; and acquire a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks based on the similarity.

In one possible implementation, the second acquisition module is used to: perform feature extraction on the first low-resolution image and the second low-resolution image to obtain first features of multiple first image blocks of the first low-resolution image and second features of multiple second image blocks of the second low-resolution image; and calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain similarities between the multiple first image blocks and the multiple second image blocks.

In a possible implementation, the plurality of first image blocks include a third image block, and the second acquisition module is used to: based on the third image block The similarity between the third image block and the plurality of second image blocks is determined, among the plurality of second image blocks, the second image block with the greatest similarity is determined as the fourth image block; and a corresponding relationship between the third image block and the fourth image block is established.

In one possible implementation, a fusion module is used to extract features from a first low-resolution image and a second low-resolution image to obtain a third feature of multiple first image blocks of the first low-resolution image and a fourth feature of multiple second image blocks of the second low-resolution image; based on the corresponding relationship, adjust the order of the fourth features of the multiple second image blocks to obtain the fourth features of the multiple second image blocks after the order is adjusted; and process the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the order is adjusted to obtain a high-resolution image of the target object.

In one possible implementation, the fusion module is used to process the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting to obtain a high-resolution image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

A ninth aspect of an embodiment of the present application provides an image processing device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the image processing device executes the method described in the first aspect, any possible implementation of the first aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

The tenth aspect of an embodiment of the present application provides a model training device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes the method described in the second aspect or any possible implementation method of the second aspect.

The eleventh aspect of an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

The twelfth aspect of an embodiment of the present application provides a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes a method as described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

In a possible implementation manner, the processor is coupled to the memory through an interface.

In a possible implementation, the chip system also includes a memory, in which a computer program or computer instructions are stored.

A thirteenth aspect of an embodiment of the present application provides a computer storage medium, which stores a computer program. When the program is executed by a computer, the computer implements the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

A fourteenth aspect of an embodiment of the present application provides a computer program product, which stores instructions, which, when executed by a computer, enable the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.

In an embodiment of the present application, when it is necessary to obtain an HDR image of the target object, a first LDR image of the target object and a second LDR image of the target object may be first collected, and the first LDR image and the second LDR image may be input into the target model. Then, the target model may perform image block matching on the first LDR image and the second LDR image, thereby obtaining a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image. Then, the target model may use the correspondence to fuse the first LDR image and the second LDR image, thereby obtaining and outputting the HDR image of the target object. In the aforementioned process, in the process of obtaining a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image, it is equivalent to matching multiple first image blocks of the first LDR image with multiple second image blocks of the second LDR image. The image block is aligned one by one with the second image blocks of the second LDR image in terms of content. Then, using this correspondence as a guide to achieve fusion between the first LDR image and the second LDR image, a better fusion can be achieved, so that the HDR image of the target object finally obtained does not have artifacts.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a structure of an artificial intelligence main framework;

FIG2a is a schematic diagram of a structure of an image processing system provided in an embodiment of the present application;

FIG2b is another schematic diagram of the structure of the image processing system provided in an embodiment of the present application;

FIG2c is a schematic diagram of an image processing related device provided in an embodiment of the present application;

FIG3 is a schematic diagram of the architecture of the system 100 provided in an embodiment of the present application;

FIG4 is a schematic diagram of a structure of a target model provided in an embodiment of the present application;

FIG5 is a schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;

FIG6 is a schematic diagram of a block search network provided in an embodiment of the present application;

FIG7 is another schematic diagram of the structure of a block search network provided in an embodiment of the present application;

FIG8 is a schematic diagram of a structure of a fusion transformer network provided in an embodiment of the present application;

FIG9 is another schematic diagram of the structure of the fusion transformer network provided in an embodiment of the present application;

FIG10 is a schematic diagram of a structure of a local reconstruction transformer network provided in an embodiment of the present application;

FIG11 is a schematic diagram of a structure of a self-attention mechanism module or an interactive attention mechanism module provided in an embodiment of the present application;

FIG12 is a schematic diagram of a structure of a transformer module provided in an embodiment of the present application;

FIG13a is a schematic diagram of a comparison result provided in an embodiment of the present application;

FIG13b is another schematic diagram of the comparison results provided in an embodiment of the present application;

FIG14 is another schematic diagram of comparison results provided in an embodiment of the present application;

FIG15 is another schematic diagram of comparison results provided in an embodiment of the present application;

FIG16 is a flow chart of a model training method provided in an embodiment of the present application;

FIG17 is a schematic diagram of a structure of an image processing device provided in an embodiment of the present application;

FIG18 is a schematic diagram of a structure of a model training device provided in an embodiment of the present application;

FIG19 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application;

FIG20 is a schematic diagram of a structure of a training device provided in an embodiment of the present application;

FIG. 21 is a schematic diagram of the structure of a chip provided in an embodiment of the present application.

DETAILED DESCRIPTION

The embodiment of the present application provides an image processing method and related equipment, which can achieve better fusion of multiple LDR images, so that the final HDR image does not have artifacts.

The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

As a key issue in computer vision applications, HDR imaging has received more and more attention. With the rapid development of AI technology, more and more device manufacturers are embedding neural network models in AI technology into their devices to obtain high-quality HDR images through the models.

In the related art, a target object in a scene may be photographed at different exposure rates to obtain multiple LDR images of the target object. Then, the multiple LDR images may be input into a neural network model to fuse the multiple LDR images through the neural network model to obtain an HDR image of the target object. For example, three LDR images of the same scene may be collected first. The three LDR images are photographed at three exposure levels. After the three LDR images are processed by the neural network model, an HDR image with optimized image indicators such as color, brightness and contrast may be obtained.

In the above process, when taking multiple LDR images, the target object may move in the scene, resulting in differences between the contents presented by the multiple LDR images taken. Therefore, the neural network model is directly based on the HDR image obtained by fusing these multiple LDR images, which is prone to artifacts.

Furthermore, in order to suppress the generation of artifacts, other related technologies usually allow the neural network model to ignore some information in the image during the process of fusing multiple LDR images. In this way, the fused HDR image will not have artifacts, but the details of the HDR image itself are often not good enough, that is, the quality of the HDR image is not high.

In order to solve the above problems, the embodiment of the present application provides an image processing method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by sensing the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for data processing is a common application of artificial intelligence.

First, the overall workflow of the artificial intelligence system is described. Please refer to Figure 1. Figure 1 is a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

(1) Infrastructure

The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

(2) Data

The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

(4) General capabilities

After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

(5) Smart products and industry applications

Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

Next, several application scenarios of this application are introduced.

FIG2a is a schematic diagram of a structure of an image processing system provided in an embodiment of the present application, wherein the image processing system includes a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, a personal computer or an information processing center. The user device is the initiator of image processing, and as the initiator of the image processing request, the request is usually initiated by the user through the user device.

The above-mentioned data processing device can be a device or server with data processing function such as a cloud server, a network server, an application server and a management server. The data processing device receives the image processing request from the intelligent terminal through the interactive interface, and then performs image processing in the form of machine learning, deep learning, search, reasoning, decision-making, etc. through the memory for storing data and the processor for data processing. The memory in the data processing device can be a general term, including local storage and a database for storing historical data. The database can be on the data processing device or on other network servers.

In the image processing system shown in FIG2a, the user device can receive the user's instruction, for example, the user device can obtain multiple images input/selected by the user, and then initiate a request to the data processing device, so that the data processing device performs an image fusion application on the multiple images obtained by the user device, thereby obtaining corresponding fusion results for the multiple images. Exemplarily, the user device can obtain multiple LDR images input by the user, and then initiate an image fusion request to the data processing device, so that the data processing device performs a series of processing on the multiple LDR images based on the image fusion request, thereby obtaining the processing results of the multiple LDR images, that is, the HDR image obtained based on the fusion of the multiple LDR images.

In FIG. 2 a , the data processing device may execute the image processing method according to the embodiment of the present application.

Figure 2b is another structural schematic diagram of the image processing system provided in an embodiment of the present application. In Figure 2b, the user device directly serves as a data processing device. The user device can directly obtain input from the user and directly process it by the hardware of the user device itself. The specific process is similar to that of Figure 2a. Please refer to the above description and will not be repeated here.

In the image processing system shown in FIG2b , the user device can receive instructions from the user. For example, the user device can obtain multiple LDR images input by the user, and then perform a series of processing on the multiple LDR images to obtain processing results of the multiple LDR images, that is, an HDR image obtained based on the fusion of the multiple LDR images.

In FIG. 2b , the user equipment itself can execute the image processing method of the embodiment of the present application.

FIG. 2c is a schematic diagram of an image processing related device provided in an embodiment of the present application.

The user device in the above Figures 2a and 2b can specifically be the local device 301 or the local device 302 in Figure 2c, and the data processing device in Figure 2a can specifically be the execution device 210 in Figure 2c, wherein the data storage system 250 can store the data to be processed of the execution device 210, and the data storage system 250 can be integrated on the execution device 210, and can also be set on the cloud or other network servers.

The processors in Figures 2a and 2b can perform data training/machine learning/deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to execute image processing applications on the image, thereby obtaining corresponding processing results.

Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of the present application. In Figure 3, the execution device 110 is configured with an input/output (I/O) interface 112 for data interaction with an external device. The user can input data to the I/O interface 112 through the client device 140. The input data may include: various tasks to be scheduled, callable resources and other parameters in the embodiment of the present application.

When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

Finally, the I/O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

It is worth noting that the training device 120 can generate corresponding target models/rules based on different training data for different goals or different tasks, and the corresponding target models/rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from the training samples collected by the data collection device 160.

In the case shown in FIG. 3 , the user can manually give input data, and the manual giving can be operated through the interface provided by the I/O interface 112. In another case, the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I/O interface 112 and the output results of the output I/O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I/O interface 112 directly stores the input data of the input I/O interface 112 and the output results of the output I/O interface 112 as new sample data in the database 130.

It is worth noting that FIG3 is only a schematic diagram of a system architecture provided in an embodiment of the present application. The positional relationship between the data storage system 150 and the execution device 110 does not constitute any limitation. For example, in FIG3 , the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110. As shown in FIG3 , a neural network may be obtained by training according to the training device 120.

The embodiment of the present application also provides a chip, which includes a neural network processor NPU. The chip can be set in the execution device 110 as shown in Figure 3 to complete the calculation work of the calculation module 111. The chip can also be set in the training device 120 as shown in Figure 3 to complete the training work of the training device 120 and output the target model/rule.

Neural network processor NPU, NPU is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of NPU is the operation circuit, and the controller controls the operation circuit to extract data from the memory (weight memory or input memory) and perform operations.

In some implementations, the arithmetic circuit includes multiple processing units (process engines, PEs) internally. In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory and performs matrix operations with matrix B. The partial results or final results of the matrix are stored in the accumulator.

The vector calculation unit can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit can be used for network calculations of non-convolutional/non-FC layers in neural networks, such as pooling, batch normalization, local response normalization, etc.

In some implementations, the vector computation unit can store the processed output vector to a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the computation circuit, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector computation unit generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit, such as for use in a subsequent layer in a neural network.

The unified memory is used to store input data and output data.

The weight data is directly transferred from the external memory to the input memory and/or the unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.

The bus interface unit (BIU) is used to enable interaction between the main CPU, DMAC and instruction fetch memory through the bus.

An instruction fetch buffer connected to the controller, used to store instructions used by the controller;

The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.

Generally, the unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memories.

Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

(1) Neural Network

A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid Function. A neural network is a network formed by connecting many of the above single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

The work of each layer in the neural network can be described by the mathematical expression y=a(Wx+b): From a physical level, the work of each layer in the neural network can be understood as completing the transformation from the input space to the output space (i.e., the row space to the column space of the matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimension increase/reduction; 2. Zoom in/out; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are completed by Wx, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network is to finally obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, or more specifically, learning the weight matrix.

Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

(2) Back propagation algorithm

Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

The method provided in the present application is described below from the training side of the neural network and the application side of the neural network.

The model training method provided in the embodiment of the present application involves the processing of data sequences, and can be specifically applied to methods such as data training, machine learning, and deep learning, and symbolizes and formalizes intelligent information modeling, extraction, preprocessing, and training of training data (for example, the first LDR image of the target object and the second LDR image of the target object in the model training method provided in the embodiment of the present application), and finally obtains a trained neural network (for example, the target model in the model training method provided in the embodiment of the present application); and the image processing method provided in the embodiment of the present application can use the above-mentioned trained neural network to input input data (for example, the first LDR image of the target object and the second LDR image of the target object in the image processing method provided in the embodiment of the present application) into the trained neural network to obtain output data (for example, the HDR image of the target object in the image processing method provided in the embodiment of the present application). It should be noted that the model training method and image processing method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

The image processing method provided in the embodiment of the present application can be implemented by a target model. The structure of the target model is briefly introduced below. FIG4 is a schematic diagram of the structure of the target model provided in the embodiment of the present application. As shown in FIG4, the target model includes a block search network based on speech similarity, a fusion transformer network based on self-attention and interactive attention mechanisms, and a local Reconstruct the transformer network, wherein the input end of the block search network and the first input end of the fused transformer network serve as the input end of the entire target model, the output end of the block search network is connected to the second input end of the fused transformer network, the output end of the fused transformer network is connected to the input end of the local reconstruction transformer network, and the output end of the local reconstruction transformer network serves as the output end of the entire target model. In order to understand the workflow of the target model shown in FIG4 , the workflow is introduced below in conjunction with FIG5 , which is a flowchart of an image processing method provided in an embodiment of the present application. As shown in FIG5 , the method includes:

501. Obtain a first LDR image of a target object and a second LDR image of the target object, where the first LDR image and the second LDR image are images obtained by photographing the target object at different exposure levels.

In this embodiment, when it is necessary to obtain an HDR image of the target object, the target object can be photographed using different exposure rates, thereby acquiring a first LDR image of the target object (also referred to as a reference image of the target object) and a second LDR image of the target object (also referred to as a support image of the target object). It should be noted that the target object may refer to an object in a scene, or an area in a scene containing an object, etc. For example, in a park scene, the target object may refer to a boy in the park, or may refer to the grass where the boy is in the park, etc.

It is worth noting that the number of the second LDR images can be one or more. If multiple second LDR images are collected, then the multiple second LDR images are captured using multiple exposure rates (the multiple exposure rates are different from each other). For any second LDR image among the multiple second LDR images, the exposure rate used to capture the second LDR image can be greater than the exposure rate used to capture the first LDR image, or can be less than the exposure rate used to capture the first LDR image.

After obtaining the first LDR image of the target object and the second LDR image of the target object, the first LDR image of the target object and the second LDR image of the target object can be input into the target model, so that the target model performs a series of processing on the first LDR image of the target object and the second LDR image of the target object, thereby obtaining an HDR image of the target object.

502. Based on the first LDR image and the second LDR image, obtain a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image.

After receiving a first LDR image of the target object and a second LDR image of the target object, the target model can perform block matching on multiple first image blocks of the first LDR image of the target object and multiple second image blocks of the second LDR image of the target object, thereby establishing a one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image.

Specifically, the target model can obtain a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks in the following manner:

(1) After receiving a first LDR image of a target object and a second LDR image of the target object, a block search network of a target model may perform a series of processing on the first LDR image and the second LDR image, thereby obtaining similarities between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image.

(2) After obtaining the similarities between the multiple first image blocks and the multiple second image blocks, the block search network can use the similarities between the multiple first image blocks and the multiple second image blocks to construct a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks.

More specifically, as shown in FIG6 (FIG6 is a schematic diagram of the structure of a block search network provided in an embodiment of the present application), the block search network includes a feature extraction module and a block search module. Then, the block search network can obtain the similarity between the plurality of first image blocks and the plurality of second image blocks in the following manner:

(1.1) After receiving the first LDR image of the target object and the second LDR image of the target object, the feature extraction module of the block search network (the module may include a classification backbone network, such as GhostNet, VGG network, etc.) may cooperate with the block search module to perform feature extraction on the first LDR image and the second LDR image respectively, thereby obtaining first features of multiple first image blocks of the first LDR image (also referred to as semantic features of multiple first image blocks, etc.) and second features of multiple second image blocks of the second LDR image (also referred to as semantic features of multiple second image blocks, etc.).

For example, as shown in Figure 7 (Figure 7 is another structural schematic diagram of the block search network provided in an embodiment of the present application), it is assumed that three LDR images are collected, namely support image 1, reference image and support image 2, and these three LDR images are shot using exposure rate 1, exposure rate 2 and exposure rate 3, respectively, wherein exposure rate 1>exposure rate 2>exposure rate 3. After inputting support image 1, reference image and support image 2 into the block search network, the feature extraction module can first extract the overall semantic features of support image 1, the size of the overall semantic features of support image 1 is c (channel) × H (height) × W (width), and send the overall semantic features of support image 1 to the block search module. Similarly, the feature extraction module can also first extract the overall semantic features of the reference image, the size of the overall semantic features of the reference image is c×H×W, and send the overall semantic features of the reference image to the block search module. Similarly, the feature extraction module can also first extract the overall semantic features of support image 2, the size of the overall semantic features of support image 2 is c×H×W, and send the overall semantic features of support image 2 to the block search module. The features are sent to the block search module.

It can be understood that the supporting image 1 is composed of N2 supporting image blocks 1 (N can be 8 or 16, etc.), the reference image is composed of N2 reference image blocks, and the supporting image block 2 is composed of N2 supporting image blocks 2. Accordingly, the overall semantic features of the supporting image 1 are composed of the semantic features of the N2 supporting image blocks 1, the overall semantic features of the reference image are composed of the semantic features of the N2 reference image blocks, and the overall semantic features of the supporting image 2 are composed of the semantic features of the N2 supporting image blocks 2.

Then, the block search module can divide the overall semantic features of the support image 1 into N2 semantic features of the support image blocks 1, and the size of the semantic features of each support image block 1 is c×(H/N)×(W/N). Similarly, the block search module can also divide the overall semantic features of the reference image into N2 semantic features of the reference image blocks, and the size of the semantic features of each reference image block is c×(H/N)×(W/N). Similarly, the block search module can also divide the overall semantic features of the support image 2 into N2 semantic features of the support image blocks 2, and the size of the semantic features of each support image block 2 is c×(H/N)×(W/N).

(1.2) After obtaining the first features of the multiple first image blocks and the second features of the multiple second image blocks, the block search module can also calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks, so as to obtain the similarity between the multiple first image blocks and the multiple second image blocks (which can also be called the cosine similarity between the multiple first image blocks and the multiple second image blocks, etc.).

Still as in the above example, the block search module can also calculate the cosine similarity of the semantic features of the N2 supporting image blocks 1 and the semantic features of the N2 reference image blocks, thereby obtaining a similarity matrix 1 with N2 rows and N2 columns, and the similarity matrix 1 includes the similarities between the N2 supporting image blocks 1 and the N2 reference image blocks. Similarly, the block search module can also calculate the cosine similarity of the semantic features of the N2 supporting image blocks 2 and the semantic features of the N2 reference image blocks, thereby obtaining a similarity matrix 2 with N2 rows and N2 columns, and the similarity matrix 2 includes the similarities between the N2 supporting image blocks 2 and the N2 reference image blocks.

More specifically, the block search network can obtain the one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks in the following manner:

(2.1) For ease of explanation, any one of the plurality of first image blocks (i.e., the aforementioned third image block) is referred to below. The block search module may select, based on the similarity between the first image block and the plurality of second image blocks, a second image block with the greatest similarity among the plurality of second image blocks as the second image block most similar to the first image block (i.e., the aforementioned fourth image block).

(2.2) Then, the block search module can construct a correspondence between the first image block and the second image block that is most similar to the first image block. In addition, the block search module can also perform the same operation as that performed on the first image block on the remaining first image blocks except the first image block among the plurality of first image blocks, so that a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks can be finally obtained, and the correspondence is sent to the fusion transformer network.

Still as in the above example, in the similarity matrix 1, the first row represents the similarity between the first reference image block and the N2 supporting images 1, and the block search module can extract the maximum similarity in the first row as the correspondence between the first reference image block and the supporting image 1 that is most similar to the first reference image block. Similarly, the block search module can also extract the correspondence between the second reference image block and the supporting image 1 that is most similar to the second reference image block in the second row, ..., until the correspondence between the N2th reference image block and the supporting image 1 that is most similar to the N2th reference image block is extracted in the N2th row.

Similarly, in the similarity matrix 2, the first row represents the similarity between the first reference image block and the N2 supporting images 2, and the block search module can extract the maximum similarity in the first row as the correspondence between the first reference image block and the supporting image 2 most similar to the first reference image block. Similarly, the block search module can also extract the correspondence between the second reference image block and the supporting image 2 most similar to the second reference image block in the second row, ..., until the correspondence between the N2th reference image block and the supporting image 2 most similar to the N2th reference image block is extracted in the N2th row.

In this way, the block search module can obtain the one-to-one correspondence between the N2 reference image blocks and the N2 support images 1, and the one-to-one correspondence between the N2 reference image blocks and the N2 support images 2. In addition, the block search module can also perform a reshape operation to present the one-to-one correspondence between the N2 reference image blocks and the N2 support images 1 as a similarity matrix 3 with N rows and N columns, and present the one-to-one correspondence between the N2 reference image blocks and the N2 support images 2 as a similarity matrix 4 with N rows and N columns, and send them to the fusion transformer network.

503. Based on the corresponding relationship, the first LDR image and the second LDR image are fused to obtain an HDR image of the target object.

After obtaining a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks, the target model can use the one-to-one correspondence between the multiple first image blocks and the multiple second image blocks to fuse the first LDR image and the second LDR image, thereby obtaining and outputting an HDR image of the target object.

Specifically, as shown in FIG8 (FIG. 8 is a schematic diagram of the structure of a fusion transformer network provided in an embodiment of the present application), The fusion transformer network of the target model includes feature extraction module, block alignment module, self-attention mechanism module, interactive attention mechanism module and splicing module. Then, the target model can obtain the HDR image of the target object in the following ways:

(1) After receiving the first LDR image and the second LDR image, a feature extraction module (e.g., a convolutional network, etc.) fused with a transformer network may perform feature extraction on the first LDR image and the second LDR image, thereby obtaining third features of the plurality of first image blocks of the first LDR image (also referred to as depth features of the plurality of first image blocks, etc.), and fourth features of the plurality of second image blocks of the second LDR image (also referred to as depth features of the plurality of second image blocks, etc.).

Still as in the above example, as shown in Figure 9 (Figure 9 is another structural diagram of the fused transformer network provided in an embodiment of the present application), after the support image 1, the reference image and the support image 2 are input into the fused transformer network, the feature extraction module can first extract the overall depth feature of the support image 1, the size of the overall depth feature of the support image 1 is c×H×W, and send the overall depth feature of the support image 1 to the block alignment module. Similarly, the feature extraction module can also first extract the overall depth feature of the reference image, the size of the overall depth feature of the reference image is c×H×W, and send the overall depth feature of the reference image to the block alignment module. Similarly, the feature extraction module can also first extract the overall depth feature of the support image 2, the size of the overall depth feature of the support image 2 is c×H×W, and send the overall depth feature of the support image 2 to the block alignment module.

It can be understood that the overall depth feature of the support image 1 is composed of the depth features of N2 support image blocks 1, the overall depth feature of the reference image is composed of the depth features of N2 reference image blocks, and the overall depth feature of the support image 2 is composed of the depth features of N2 support image blocks 2.

(2) After obtaining the fourth features of the plurality of second image blocks and the one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks, since the correspondence indicates a new ordering of the fourth features of the plurality of second image blocks, the block alignment module can adjust the ordering of the fourth features of the plurality of second image blocks based on the correspondence, thereby obtaining the fourth features of the plurality of second image blocks after the ordering is adjusted. Then, the block alignment module can send the third features of the plurality of first image blocks, the fourth features of the plurality of second image blocks, and the fourth features of the plurality of second image blocks after the ordering is adjusted to the self-attention mechanism module and the interactive attention mechanism module.

Still as in the above example, after the block alignment module obtains the overall depth features of the support image 1, the overall depth features of the support image 2, the similarity matrix 3 and the similarity matrix 4, since in the overall depth features of the support image 1, the depth features of the N2 support image blocks 1 are set according to the original order (that is, in the support image 1, the original order of the N2 support image blocks 1), and the similarity matrix 3 indicates the new order of the depth features of the N2 support image blocks 1, the block alignment module can adjust the order of the depth features of the N2 support image blocks 1 according to the indication of the similarity matrix 3, thereby obtaining the overall depth features of the support image 1 after the adjusted order. Then, the block alignment module can divide the overall depth features of the support image 1 after the adjusted order into the semantic features of the N2 support image blocks 1 after the adjusted order, and the size of each semantic feature of the support image block 1 after the adjusted order is c×(H/N)×(W/N).

Similarly, since the depth features of the N2 supporting image blocks 2 are arranged according to the original order in the overall depth features of the supporting image 2, and the similarity matrix 4 indicates the new order of the depth features of the N2 supporting image blocks 2, the block alignment module can adjust the order of the depth features of the N2 supporting image blocks 2 according to the indication of the similarity matrix 4, thereby obtaining the overall depth features of the supporting image 2 after the adjusted order. Then, the block alignment module can divide the overall depth features of the supporting image 2 after the adjusted order into the semantic features of the N2 supporting image blocks 2 after the adjusted order, and the size of each semantic feature of the supporting image block 2 after the adjusted order is c×(H/N)×(W/N).

In addition, the block alignment module and the block search module may also divide the overall depth features of the support image 1 into N2 depth features of the support image blocks 1, and the size of the depth features of each support image block 1 is c×(H/N)×(W/N). Similarly, the block search module may also divide the overall depth features of the reference image into N2 depth features of the reference image blocks, and the size of the depth features of each reference image block is c×(H/N)×(W/N). Similarly, the block search module may also divide the overall depth features of the support image 2 into N2 depth features of the support image blocks 2, and the size of the depth features of each support image block 2 is c×(H/N)×(W/N).

Then, the block alignment module can send the depth features of the N2 supporting image blocks 1, the depth features of the N2 reference image blocks, the depth features of the N2 supporting image blocks 2, the adjusted and sorted semantic features of the N2 supporting image blocks 1, and the adjusted and sorted semantic features of the N2 supporting image blocks 2 to the self-attention mechanism module and the interactive attention mechanism module.

(3) After obtaining the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting, the self-attention mechanism module and the interactive attention mechanism module can cooperate with the local reconstruction transformer network to perform a series of processing on the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting, so as to obtain and output the HDR image of the target object.

More specifically, as shown in FIG10 (FIG10 is a schematic diagram of the structure of the local reconstruction transformer network provided in an embodiment of the present application), the local reconstruction transformer network of the target model includes a convolution module, a transformer module, an addition module and an activation module. Then, the target model can obtain an HDR image of the target object in the following manner:

(3.1) After obtaining the third features of multiple first image blocks, the fourth features of multiple second image blocks, and the fourth features of multiple second image blocks after adjustment and sorting, the self-attention mechanism module can perform a series of processing on the third features of multiple first image blocks and send the obtained processing results to the splicing module. The interactive attention mechanism module can perform a series of processing on the third features of multiple first image blocks and the fourth features of multiple second image blocks, and send the obtained processing results to the splicing module. At the same time, the interactive attention mechanism module can also perform a series of processing on the third features of multiple first image blocks and the fourth features of multiple second image blocks after adjustment and sorting, and send the obtained processing results to the splicing module. Then, the splicing module can splice all the received processing results and send the obtained splicing results to the local reconstruction transformer network.

Still as in the above example, the self-attention module can process the depth features of the N2 reference image blocks to obtain processing result 1. The interactive attention module 1 can process the depth features of the N2 reference image blocks and the depth features of the N2 supporting image blocks 1 to obtain processing result 2. The interactive attention module 1 can process the depth features of the N2 reference image blocks and the depth features of the N2 supporting image blocks 2 to obtain processing result 2. The interactive attention module 3 can process the depth features of the N2 reference image blocks and the depth features of the adjusted and sorted N2 supporting image blocks 1 to obtain processing result 4. The interactive attention module 4 can process the depth features of the N2 reference image blocks and the depth features of the adjusted and sorted N2 supporting image blocks 2 to obtain processing result 5. Then, the stitching module can stitch processing results 1 to 5 to obtain the corresponding stitching result.

(3.2) In the local reconstruction transformer network, the splicing result is processed by the convolution module, transformer module, addition module and activation module in the local reconstruction transformer network respectively, and finally the HDR image of the target object can be obtained and output externally.

More specifically, as shown in FIG11 (FIG11 is a structural diagram of a self-attention mechanism module or an interactive attention mechanism module provided in an embodiment of the present application), in a fused transformer network, the structure of the self-attention mechanism module and the structure of the interactive attention module can be the same, and any one of these two modules may include: a normalization unit, a multi-head attention mechanism unit, an addition unit, and a multi-layer perceptron unit, etc. It can be seen that both modules can implement normalization processing, processing based on a multi-head attention mechanism, addition processing, processing based on a multi-layer perceptron, etc.

More specifically, as shown in FIG. 12 (FIG. 12 is a schematic diagram of the structure of the transformer module provided in an embodiment of the present application), in the local reconstruction transformer network, the transformer module may include: a multi-head self-attention mechanism unit and a multi-layer perceptron unit. It can be seen that the transformer unit can implement processing based on the multi-head self-attention mechanism and processing based on the multi-layer perceptron, etc.

It should be understood that in this embodiment, only the example of the target model being able to fuse multiple LDR images into an HDR image is used for schematic introduction. In actual applications, the target model can also fuse multiple low-resolution images into a high-resolution image (for example, fuse the first low-resolution image and the second low-resolution image into a high-resolution image), or fuse multiple noisy images into a denoised image, etc. (for example, fuse the first noisy image and the second noisy image into a denoised image). These fusion processes can refer to steps 501 to 503. It is only necessary to replace the LDR image with a low-resolution image (for example, the first low-resolution image and the second low-resolution image) or a noisy image (for example, the first noisy image and the second noisy image), and replace the HDR image with a high-resolution image or a denoised image. No further details will be given here.

In addition, the target model provided in the embodiment of the present application (i.e., IFT in Table 1) and the model provided by the related art (i.e., the remaining models except IFT in Table 1, for example, Sen, Hu, etc.) can be compared on a certain data set, and the comparison results are shown in Table 1:

Table 1

Based on Table 1, it can be seen that, in the test on this data set, the embodiment of the present application has better PSNR-μ/PSNR-L/HDR-VDP-2 (the larger the better) than the related art, and as shown in Figures 13a and 13b (Figure 13a is a schematic diagram of the comparison results provided by the embodiment of the present application, and Figure 13b is another schematic diagram of the comparison results provided by the embodiment of the present application), the embodiment of the present application can restore details more accurately and without artifacts compared to the related art. In areas with relatively large motion in the foreground, other methods will produce certain artifacts and cannot restore color details well, while the embodiment of the present application is visually most similar to GT.

Furthermore, the target model provided in the embodiment of the present application (i.e., IFT in Table 1) and the model provided by the related art (i.e., the remaining models except IFT in Table 1, for example, Sen, Hu, etc.) can be compared on another data set. The comparison results are shown in Table 2:

Table 2

Based on Table 2, it can be seen that, in the test on this data set, the embodiment of the present application has better PSNR-μ/PSNR-L/HDR-VDP-2 (the larger the better) than the related art, and in terms of visual effects, as shown in Figures 14 and 15 (Figure 14 is another schematic diagram of the comparison results provided by the embodiment of the present application, and Figure 15 is another schematic diagram of the comparison results provided by the embodiment of the present application), the embodiment of the present application can restore details more accurately and without generating artifacts compared to the related art.

In an embodiment of the present application, when it is necessary to obtain an HDR image of a target object, a first LDR image of the target object and a second LDR image of the target object may be first collected, and the first LDR image and the second LDR image may be input into a target model. Then, the target model may perform image block matching on the first LDR image and the second LDR image, thereby obtaining a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image. Then, the target model may use the correspondence to fuse the first LDR image and the second LDR image, thereby obtaining and outputting an HDR image of the target object. In the aforementioned process, in the process of obtaining a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image, it is equivalent to aligning a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image one-to-one in terms of content. Then, using this correspondence as a guide to achieve the fusion between the first LDR image and the second LDR image, a higher quality fusion may be achieved, so that the HDR image of the target object finally obtained does not have artifacts.

Furthermore, in an embodiment of the present application, the target model includes a fusion transformer network based on a self-attention mechanism and an interactive attention mechanism. When fusing the first LDR image and the second LDR image, the network can effectively take into account the detail information of the first LDR image and the second LDR image themselves, so that the final HDR image maintains good details and has no artifacts.

The above is a detailed description of the image processing method provided by the embodiment of the present application. The following is an introduction to the model training method provided by the embodiment of the present application. FIG16 is a flow chart of the model training method provided by the embodiment of the present application. As shown in FIG16 , the method includes:

1601. Acquire a first LDR image of a target object and a second LDR image of the target object, where the first LDR image and the second LDR image are images obtained by photographing the target object at different exposure levels.

In this embodiment, when it is necessary to train the model to be trained, a batch of training data may be first obtained, the batch of training data including a first LDR image of the target object and a second LDR image of the target object, the first LDR image and the second LDR image being images obtained by photographing the target object at different exposures. It should be noted that, for the first LDR image and the second LDR image, the real HDR image of the target object is known.

1602. Process the first LDR image and the second LDR image through the model to be trained to obtain a high dynamic range HDR image of the target object, wherein the model to be trained is used to: obtain a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image based on the first LDR image and the second LDR image; and fuse the first LDR image and the second LDR image based on the correspondence to obtain the HDR image of the target object.

After obtaining the first LDR image and the second LDR image, the first LDR image and the second LDR image can be input into the model to be trained. Then, the model to be trained can perform a series of processing on the first LDR image and the second LDR image, thereby obtaining a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image, and using the one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks, fuse the first LDR image and the second LDR image, thereby obtaining a (predicted) HDR image of the target object.

In one possible implementation, the model to be trained is used to: obtain the similarity between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image based on the first LDR image and the second LDR image; and obtain a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks based on the similarity.

In one possible implementation, the model to be trained is used to: extract features from a first LDR image and a second LDR image to obtain first features of multiple first image blocks of the first LDR image and second features of multiple second image blocks of the second LDR image; and calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain similarities between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, multiple first image blocks include a third image block, and the model to be trained is used to: based on the similarity between the third image block and multiple second image blocks, determine the second image block with the greatest similarity among the multiple second image blocks as the fourth image block; and establish a corresponding relationship between the third image block and the fourth image block.

In one possible implementation, the model to be trained is used to: perform feature extraction on the first LDR image and the second LDR image to obtain a third feature of multiple first image blocks of the first LDR image and a fourth feature of multiple second image blocks of the second LDR image; adjust the order of the fourth features of the multiple second image blocks based on the corresponding relationship to obtain the fourth features of the multiple second image blocks after the order is adjusted; and process the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after the order is adjusted to obtain an HDR image of the target object.

In one possible implementation, the aforementioned processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, Intention mechanism processing, splicing processing, convolution processing, transformer network-based processing, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

It should be noted that, for the introduction of step 1602, reference may be made to the relevant description of steps 502 to 503 in the embodiment shown in FIG. 5, which will not be repeated here.

1603. Based on the HDR image, the model to be trained is trained to obtain a target model.

After obtaining the HDR image of the target object, since the real HDR image of the target object is known, the HDR image and the real HDR image can be calculated by a preset loss function to obtain a target loss, which is used to indicate the difference between the HDR image and the real HDR image. Then, the target loss can be used to update the parameters of the model to be trained, thereby obtaining the model to be trained after the updated parameters. Then, the next batch of training data can be used to continue training the model to be trained after the updated parameters until the model training conditions are met (for example, the target loss converges, etc.), thereby obtaining the target model in the embodiment shown in Figure 5.

It should be understood that in this embodiment, only the example of the model to be trained being able to fuse multiple LDR images into an HDR image is used for schematic introduction. In actual applications, the model to be trained can also fuse multiple low-resolution images into a high-resolution image (for example, fuse the first low-resolution image and the second low-resolution image into a high-resolution image), or fuse multiple noisy images into a denoised image, etc. (for example, fuse the first noisy image and the second noisy image into a denoised image). These fusion processes and corresponding model training processes can refer to steps 1601 to 1603. It is only necessary to replace the LDR image with a low-resolution image (for example, the first low-resolution image and the second low-resolution image) or a noisy image (for example, the first noisy image and the second noisy image), and replace the HDR image with a high-resolution image or a denoised image. No further details will be given here.

The target model obtained by training in the embodiment of the present application has an image processing function (for example, a function of fusing multiple LDR images into an HDR image, etc.). Specifically, when it is necessary to obtain an HDR image of a target object, a first LDR image of the target object and a second LDR image of the target object can be first collected, and the first LDR image and the second LDR image can be input into the target model. Then, the target model can match the image blocks of the first LDR image and the second LDR image, thereby obtaining a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image. Then, the target model can use the corresponding relationship to fuse the first LDR image and the second LDR image, thereby obtaining and outputting an HDR image of the target object. In the aforementioned process, in the process of obtaining a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image, it is equivalent to aligning multiple first image blocks of the first LDR image with multiple second image blocks of the second LDR image in terms of content. Then, by using this correspondence as a guide to achieve the fusion between the first LDR image and the second LDR image, a better fusion can be achieved, so that the HDR image of the target object finally obtained does not have artifacts.

The above is a detailed description of the model training method provided in the embodiment of the present application. The following is an introduction to the image processing device and the model training device provided in the embodiment of the present application. FIG. 17 is a structural schematic diagram of the image processing device provided in the embodiment of the present application. As shown in FIG. 17 , the device includes:

A first acquisition module 1701 is used to acquire a first LDR image of a target object and a second LDR image of the target object, where the first LDR image and the second LDR image are images obtained by photographing the target object at different exposure levels;

A second acquisition module 1702 is used to acquire a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image based on the first LDR image and the second LDR image;

The fusion module 1703 is used to fuse the first LDR image and the second LDR image based on the corresponding relationship to obtain an HDR image of the target object.

In an embodiment of the present application, when it is necessary to obtain an HDR image of the target object, a first LDR image of the target object and a second LDR image of the target object may be first collected, and the first LDR image and the second LDR image may be input into the target model. Then, the target model may perform image block matching on the first LDR image and the second LDR image, thereby obtaining a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image. Then, the target model may use the correspondence to fuse the first LDR image and the second LDR image, thereby obtaining and outputting the HDR image of the target object. In the aforementioned process, in the process of obtaining a one-to-one correspondence between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image, it is equivalent to aligning the multiple first image blocks of the first LDR image with the multiple second image blocks of the second LDR image one-to-one in terms of content. Then, using this correspondence as a guide to implement The fusion between the first LDR image and the second LDR image can achieve better quality fusion, so that the final HDR image of the target object does not have artifacts.

In one possible implementation, the second acquisition module 1702 is used to: acquire the similarity between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image based on the first LDR image and the second LDR image; and acquire a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks based on the similarity.

In one possible implementation, the second acquisition module 1702 is used to: perform feature extraction on the first LDR image and the second LDR image to obtain first features of multiple first image blocks of the first LDR image and second features of multiple second image blocks of the second LDR image; calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain similarities between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, the plurality of first image blocks include a third image block, and the second acquisition module 1702 is configured to: based on the similarity between the third image block and the plurality of second image blocks, determine the second image block with the greatest similarity among the plurality of second image blocks as the fourth image block; and establish a correspondence between the third image block and the fourth image block.

In one possible implementation, the fusion module 1703 is used to extract features from the first LDR image and the second LDR image to obtain third features of multiple first image blocks of the first LDR image and fourth features of multiple second image blocks of the second LDR image; based on the corresponding relationship, adjust the order of the fourth features of the multiple second image blocks to obtain the fourth features of the multiple second image blocks after the adjustment; and process the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the adjustment to obtain an HDR image of the target object.

In one possible implementation, the fusion module 1703 is used to process the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting to obtain an HDR image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

It should be understood that in this embodiment, only the example of the target model being able to fuse multiple LDR images into an HDR image is used for schematic introduction. In actual applications, the target model can also fuse multiple low-resolution images into a high-resolution image (for example, fusing the first low-resolution image and the second low-resolution image into a high-resolution image), or fuse multiple noisy images into a denoised image, etc. (for example, fusing the first noisy image and the second noisy image into a denoised image). It only requires replacing the LDR image with a low-resolution image (for example, the first low-resolution image and the second low-resolution image) or a noisy image (for example, the first noisy image and the second noisy image), and replacing the HDR image with a high-resolution image or a denoised image. No further details will be given here.

FIG18 is a schematic diagram of a structure of a model training device provided in an embodiment of the present application. As shown in FIG18 , the device includes:

An acquisition module 1801 is used to acquire a first LDR image of a target object and a second LDR image of the target object, where the first LDR image and the second LDR image are images obtained by photographing the target object at different exposure levels;

The processing module 1802 is used to process the first LDR image and the second LDR image through the model to be trained to obtain a high dynamic range HDR image of the target object, wherein the model to be trained is used to: obtain a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image based on the first LDR image and the second LDR image; and fuse the first LDR image and the second LDR image based on the correspondence to obtain the HDR image of the target object;

The training module 1803 is used to train the model to be trained based on the HDR image to obtain a target model.

The target model obtained by training in the embodiment of the present application has an image processing function (for example, a function of fusing multiple LDR images into an HDR image, etc.). Specifically, when it is necessary to obtain an HDR image of the target object, the first LDR image of the target object and the second LDR image of the target object can be first collected, and the first LDR image and the second LDR image can be input into the target model. Then, the target model can perform image block matching on the first LDR image and the second LDR image to obtain a one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image. Then, the target model can use the corresponding relationship to fuse the first LDR image and the second LDR image to obtain and output the HDR image of the target object. In the aforementioned process, in the process of obtaining the one-to-one correspondence between the multiple first image blocks of the first LDR image and the multiple second image blocks of the second LDR image, it is equivalent to matching the multiple first image blocks of the first LDR image with the multiple second image blocks of the second LDR image. The image block is aligned one by one with the second image blocks of the second LDR image in terms of content. Then, using this correspondence as a guide to achieve fusion between the first LDR image and the second LDR image, a better fusion can be achieved, so that the HDR image of the target object finally obtained does not have artifacts.

In one possible implementation, the model to be trained is used to: obtain the similarity between multiple first image blocks of the first LDR image and multiple second image blocks of the second LDR image based on the first LDR image and the second LDR image; and obtain a one-to-one correspondence between the multiple first image blocks and the multiple second image blocks based on the similarity.

In one possible implementation, the model to be trained is used to: extract features from a first LDR image and a second LDR image to obtain first features of multiple first image blocks of the first LDR image and second features of multiple second image blocks of the second LDR image; and calculate the first features of the multiple first image blocks and the second features of the multiple second image blocks to obtain similarities between the multiple first image blocks and the multiple second image blocks.

In one possible implementation, multiple first image blocks include a third image block, and the model to be trained is used to: based on the similarity between the third image block and multiple second image blocks, determine the second image block with the greatest similarity among the multiple second image blocks as the fourth image block; and establish a corresponding relationship between the third image block and the fourth image block.

In one possible implementation, the model to be trained is used to: extract features from the first LDR image and the second LDR image to obtain third features of multiple first image blocks of the first LDR image and fourth features of multiple second image blocks of the second LDR image; adjust the order of the fourth features of the multiple second image blocks based on the corresponding relationship to obtain fourth features of the multiple second image blocks after the order is adjusted; and process the third features of the multiple first image blocks and the fourth features of the multiple second image blocks after the order is adjusted to obtain an HDR image of the target object.

In one possible implementation, the model to be trained is used to process the third features of the multiple first image blocks, the fourth features of the multiple second image blocks, and the fourth features of the multiple second image blocks after adjustment and sorting to obtain an HDR image of the target object.

In one possible implementation, the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing.

In one possible implementation, the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron.

In one possible implementation, the transformer network-based processing includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron.

It should be understood that in this embodiment, only the example of the model to be trained being able to fuse multiple LDR images into an HDR image is used for schematic introduction. In actual applications, the model to be trained can also fuse multiple low-resolution images into a high-resolution image (for example, fuse the first low-resolution image and the second low-resolution image into a high-resolution image), or fuse multiple noisy images into a denoised image, etc. (for example, fuse the first noisy image and the second noisy image into a denoised image). It only needs to replace the LDR image with a low-resolution image (for example, the first low-resolution image and the second low-resolution image) or a noisy image (for example, the first noisy image and the second noisy image), and replace the HDR image with a high-resolution image or a denoised image. No further details will be given here.

It should be noted that the information interaction, execution process, etc. between the modules/units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. The specific contents can be referred to the description in the method embodiment shown above in the embodiment of the present application, and will not be repeated here.

The embodiment of the present application also relates to an execution device, and FIG. 19 is a structural schematic diagram of the execution device provided by the embodiment of the present application. As shown in FIG. 19, the execution device 1900 can be specifically manifested as a mobile phone, a tablet, a laptop computer, an intelligent wearable device, a server, etc., which is not limited here. Among them, the image processing device described in the corresponding embodiment of FIG. 17 can be deployed on the execution device 1900 to implement the image processing function in the corresponding embodiment of FIG. 5. Specifically, the execution device 1900 includes: a receiver 1901, a transmitter 1902, a processor 1903 and a memory 1904 (wherein the number of processors 1903 in the execution device 1900 can be one or more, and FIG. 19 takes one processor as an example), wherein the processor 1903 may include an application processor 19031 and a communication processor 19032. In some embodiments of the present application, the receiver 1901, the transmitter 1902, the processor 1903 and the memory 1904 may be connected via a bus or other means.

The memory 1904 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1903. A portion of the memory 1904 may also include a non-volatile random access memory (NVRAM). The memory 1904 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

The processor 1903 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

The method disclosed in the above embodiment of the present application can be applied to the processor 1903, or implemented by the processor 1903. The processor 1903 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1903 or the instruction in the form of software. The above processor 1903 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1903 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1904, and the processor 1903 reads the information in the memory 1904 and completes the steps of the above method in combination with its hardware.

The receiver 1901 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1902 can be used to output digital or character information through the first interface; the transmitter 1902 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1902 can also include a display device such as a display screen.

In an embodiment of the present application, in one case, the processor 1903 is used to obtain an HDR image of the target object through the target model in the embodiment corresponding to FIG. 5 .

The embodiment of the present application also relates to a training device, and FIG. 20 is a schematic diagram of the structure of the training device provided by the embodiment of the present application. As shown in FIG. 20, the training device 2000 is implemented by one or more servers. The training device 2000 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 2020 (for example, one or more processors) and a memory 2032, and one or more storage media 2030 (for example, one or more mass storage devices) storing application programs 2042 or data 2044. Among them, the memory 2032 and the storage medium 2030 can be short-term storage or permanent storage. The program stored in the storage medium 2030 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 2020 can be configured to communicate with the storage medium 2030 and execute a series of instruction operations in the storage medium 2030 on the training device 2000.

The training device 2000 may also include one or more power supplies 2026, one or more wired or wireless network interfaces 2050, one or more input and output interfaces 2058; or, one or more operating systems 2041, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

Specifically, the training device can execute the model training method in the embodiment corresponding to Figure 16.

An embodiment of the present application also relates to a computer storage medium, in which a program for signal processing is stored. When the program is run on a computer, the computer executes the steps executed by the aforementioned execution device, or the computer executes the steps executed by the aforementioned training device.

An embodiment of the present application also relates to a computer program product, which stores instructions, which, when executed by a computer, enable the computer to execute the steps executed by the aforementioned execution device, or enable the computer to execute the steps executed by the aforementioned training device.

The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

Specifically, please refer to FIG. 21 , which is a schematic diagram of a structure of a chip provided in an embodiment of the present application. The chip may be a neural network processor NPU 2100, which is mounted on a host CPU (Host CPU) as a coprocessor, and the Host CPU allocates any The core part of the NPU is the operation circuit 2103, which is controlled by the controller 2104 to extract the matrix data in the memory and perform multiplication operations.

In some implementations, the operation circuit 2103 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2103 is a two-dimensional systolic array. The operation circuit 2103 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2103 is a general-purpose matrix processor.

For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2102 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2101 and performs matrix operations with matrix B, and the partial results or final results of the matrix are stored in the accumulator 2108.

The unified memory 2106 is used to store input data and output data. The weight data is directly transferred to the weight memory 2102 through the direct memory access controller (DMAC) 2105. The input data is also transferred to the unified memory 2106 through the DMAC.

BIU stands for Bus Interface Unit, which is used for the interaction between AXI bus, DMAC and instruction fetch buffer (IFB) 2109.

The bus interface unit 2113 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2109 to obtain instructions from the external memory, and is also used for the storage unit access controller 2105 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2106 or to transfer weight data to the weight memory 2102 or to transfer input data to the input memory 2101.

The vector calculation unit 2107 includes multiple operation processing units, and when necessary, further processes the output of the operation circuit 2103, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional/fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of the predicted label plane, etc.

In some implementations, the vector calculation unit 2107 can store the processed output vector to the unified memory 2106. For example, the vector calculation unit 2107 can apply a linear function; or a nonlinear function to the output of the operation circuit 2103, such as linear interpolation of the predicted label plane extracted by the convolution layer, and then, for example, a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 2107 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2103, for example, for use in a subsequent layer in a neural network.

An instruction fetch buffer 2109 connected to the controller 2104 is used to store instructions used by the controller 2104;

Unified memory 2106, input memory 2101, weight memory 2102 and instruction fetch memory 2109 are all on-chip memories. External memories are private to the NPU hardware architecture.

The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above programs.

It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims (23)

An image processing method, characterized in that the method is implemented by a target model, and the method comprises: Acquire a first low dynamic range (LDR) image of a target object and a second LDR image of the target object, wherein the first LDR image and the second LDR image are images obtained by photographing the target object at different exposures; Based on the first LDR image and the second LDR image, acquiring a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image; Based on the corresponding relationship, the first LDR image and the second LDR image are fused to obtain a high dynamic range HDR image of the target object. The method according to claim 1, characterized in that the acquiring, based on the first LDR image and the second LDR image, the correspondence between the plurality of first image blocks of the first LDR image and the plurality of second image blocks of the second LDR image comprises: Based on the first LDR image and the second LDR image, obtaining similarities between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image; Based on the similarity, a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks is obtained. The method according to claim 2, characterized in that the obtaining, based on the first LDR image and the second LDR image, the similarity between the plurality of first image blocks of the first LDR image and the plurality of second image blocks of the second LDR image comprises: Performing feature extraction on the first LDR image and the second LDR image to obtain first features of a plurality of first image blocks of the first LDR image and second features of a plurality of second image blocks of the second LDR image; The first features of the plurality of first image blocks and the second features of the plurality of second image blocks are calculated to obtain similarities between the plurality of first image blocks and the plurality of second image blocks. The method according to claim 2 or 3, characterized in that the plurality of first image blocks include a third image block, and obtaining a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks based on the similarity comprises: Based on the similarity between the third image block and the plurality of second image blocks, determining, among the plurality of second image blocks, a second image block with the greatest similarity as a fourth image block; A corresponding relationship between the third image block and the fourth image block is established. The method according to any one of claims 1 to 4, characterized in that, based on the corresponding relationship, fusing the first LDR image and the second LDR image to obtain a high dynamic range HDR image of the target object comprises: Performing feature extraction on the first LDR image and the second LDR image to obtain third features of a plurality of first image blocks of the first LDR image and fourth features of a plurality of second image blocks of the second LDR image; Based on the corresponding relationship, adjusting the order of the fourth features of the plurality of second image blocks to obtain the fourth features of the plurality of second image blocks after the order is adjusted; The third feature of the plurality of first image blocks and the fourth feature of the plurality of second image blocks after adjustment and sorting are processed to obtain an HDR image of the target object. The method according to claim 5, characterized in that the step of processing the third feature of the plurality of first image blocks and the fourth feature of the plurality of second image blocks after adjustment and sorting to obtain the HDR image of the target object comprises: The third feature of the plurality of first image blocks, the fourth feature of the plurality of second image blocks, and the fourth feature of the plurality of second image blocks after adjustment and sorting are processed to obtain an HDR image of the target object. The method according to claim 6 is characterized in that the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing. The method according to claim 7 is characterized in that the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron. The method according to claim 7 or 8 is characterized in that the processing based on the transformer network includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron. A model training method, characterized in that the method comprises: Acquire a first LDR image of a target object and a second LDR image of the target object, wherein the first LDR image and the second LDR image are images obtained by photographing the target object at different exposure levels; The first LDR image and the second LDR image are processed by a to-be-trained model to obtain a high dynamic range HDR image of the target object, wherein the to-be-trained model is used to: obtain a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image based on the first LDR image and the second LDR image; and fuse the first LDR image and the second LDR image based on the correspondence to obtain the HDR image of the target object; Based on the HDR image, the model to be trained is trained to obtain a target model. The method according to claim 10, characterized in that the model to be trained is used to: Based on the first LDR image and the second LDR image, obtaining similarities between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image; Based on the similarity, a one-to-one correspondence between the plurality of first image blocks and the plurality of second image blocks is obtained. The method according to claim 11, characterized in that the model to be trained is used for: Performing feature extraction on the first LDR image and the second LDR image to obtain first features of a plurality of first image blocks of the first LDR image and second features of a plurality of second image blocks of the second LDR image; The first features of the plurality of first image blocks and the second features of the plurality of second image blocks are calculated to obtain similarities between the plurality of first image blocks and the plurality of second image blocks. The method according to claim 11 or 12, characterized in that the plurality of first image blocks include a third image block, and the model to be trained is used for: Based on the similarity between the third image block and the plurality of second image blocks, determining, among the plurality of second image blocks, a second image block with the greatest similarity as a fourth image block; A corresponding relationship between the third image block and the fourth image block is established. The method according to any one of claims 10 to 13, characterized in that the model to be trained is used to: Performing feature extraction on the first LDR image and the second LDR image to obtain third features of a plurality of first image blocks of the first LDR image and fourth features of a plurality of second image blocks of the second LDR image; Based on the corresponding relationship, adjusting the order of the fourth features of the plurality of second image blocks to obtain the fourth features of the plurality of second image blocks after the order is adjusted; The third feature of the plurality of first image blocks and the fourth feature of the plurality of second image blocks after adjustment and sorting are processed to obtain an HDR image of the target object. The method according to claim 14, characterized in that the model to be trained is used to: The third feature of the plurality of first image blocks, the fourth feature of the plurality of second image blocks, and the fourth feature of the plurality of second image blocks after adjustment and sorting are processed to obtain an HDR image of the target object. The method according to claim 15 is characterized in that the processing includes at least one of the following: processing based on a self-attention mechanism, processing based on an interactive attention mechanism, splicing processing, convolution processing, processing based on a transformer network, addition processing, and activation processing. The method according to claim 16 is characterized in that the processing based on the self-attention mechanism or the processing based on the interactive attention mechanism includes at least one of the following: normalization processing, processing based on a multi-head attention mechanism, addition processing, and processing based on a multi-layer perceptron. The method according to claim 16 or 17 is characterized in that the processing based on the transformer network includes at least one of the following: processing based on a multi-head self-attention mechanism and processing based on a multi-layer perceptron. An image processing device, characterized in that the device includes a target model, and the device comprises: A first acquisition module is used to acquire a first LDR image of a target object and a second LDR image of the target object, wherein the first LDR image and the second LDR image are images obtained by photographing the target object at different exposure levels; a second acquisition module, configured to acquire, based on the first LDR image and the second LDR image, a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image; A fusion module is used to fuse the first LDR image and the second LDR image based on the corresponding relationship to obtain an HDR image of the target object. A model training device, characterized in that the device comprises: An acquisition module, configured to acquire a first LDR image of a target object and a second LDR image of the target object, wherein the first LDR image and the second LDR image are images obtained by photographing the target object at different exposure levels; a processing module, configured to process the first LDR image and the second LDR image through a model to be trained to obtain a high dynamic range HDR image of the target object, wherein the model to be trained is configured to: based on the first LDR image and the second LDR image, obtain a one-to-one correspondence between a plurality of first image blocks of the first LDR image and a plurality of second image blocks of the second LDR image; based on the correspondence, fuse the first LDR image and the second LDR image to obtain the HDR image of the target object; The training module is used to train the model to be trained based on the HDR image to obtain a target model. An image processing device, characterized in that the device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes, and when the codes are executed, the image processing device executes the method according to any one of claims 1 to 18. A computer storage medium, characterized in that the computer storage medium stores one or more instructions, and when the instructions are executed by one or more computers, the one or more computers implement any one of the methods of claims 1 to 18. A computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the method according to any one of claims 1 to 18.