WO2019119842A1 - Image fusion method and apparatus, electronic device, and computer readable storage medium - Google Patents

Abstract

Embodiments of the present application provide an image fusion method and apparatus, an electronic device, and a computer readable storage medium. The method comprises: first obtaining each frame of image which is obtained by means of at least two exposures within an image collection period; determining an infrared-sensible brightness image based on one of the collected frames of image; performing infrared removal on an image to be processed in the collected frames of image to obtain a visible color image; and finally fusing the infrared-sensible brightness image and the visible color image to obtain a fused image. In the solution provided by the embodiments of the present application, the frames of image obtained by means of at least two exposures within an image collection period can be collected by an image sensor, and image collection and fusion can be completed as long as an image sensor exists in a device, so as to improve the image quality in the case of low illumination. Therefore, the device in the solution provided by the embodiments of the present application has good adaptability, and is convenient for application.

Description

Image fusion method, device, electronic device and computer readable storage medium

This application claims the priority of the Chinese Patent Application entitled "Image Fusion Method, Apparatus, Electronic Device, and Computer Readable Storage Medium", filed on December 20, 2017, with the application number of 201711381018.0, the entire contents of which are hereby incorporated by reference. The citations are incorporated herein by reference.

Technical field

The present application relates to the field of image acquisition technologies, and in particular, to an image fusion method, apparatus, electronic device, and computer readable storage medium.

Background technique

The fusion in the image fusion technology can be understood as: merging the visible light image with the non-visible image such as the infrared image to obtain the fused image; wherein the fused image is a dual-band image, relative to the visible light image and the non-single band In any of the visible light images, the fused image can reflect more image information.

In the prior art, the image fusion technology mainly refers to a split-light fusion technology. The basic flow of image fusion is as follows: the incident light is divided into a visible light signal and a non-visible light signal by a light splitting device such as a beam splitting prism, and then the two sensors are respectively based on visible light. The signal and the non-visible signal respectively generate a visible light image and a non-visible light image, and finally the visible light image and the non-visible light image are fused to obtain a fused image.

It can be understood that the above-mentioned spectroscopic fusion technology must be adapted to a device having two image sensors. If there is only one image sensor in the device, the above-mentioned process of splitting and merging cannot be completed. Therefore, the device of the prior art split-light fusion technology has poor adaptability.

Summary of the invention

An object of the embodiments of the present application is to provide an image fusion method, apparatus, electronic device, and computer readable storage medium to improve device adaptability of an image fusion technology. The specific technical solutions are as follows:

To achieve the above objective, in a first aspect, an embodiment of the present application provides an image fusion method, where the method includes:

Obtaining each frame image obtained by at least two exposures in one image acquisition period; determining an infrared illuminance image based on one of the frame images; and de-infrareding the image to be processed in each frame image Processing, obtaining a visible light color image; fusing the infrared illuminance image with the visible light color image to obtain a fused image.

Optionally, the method is applied to an image fusion device, where each frame image is collected by the image fusion device; the method further includes: performing an exposure time corresponding to a preset exposure within the image collection period Performing infrared fill light; the step of determining an infrared illuminance image based on one of the frames of the frames, comprising: determining an infrared illuminance image based on the image obtained by the first preset exposure.

Optionally, the exposure parameter corresponding to the preset exposure is not greater than a target maximum, wherein the exposure parameter is an exposure duration and/or a gain, and the target maximum is a preset exposure The maximum value of the exposure parameters corresponding to each of the other exposures.

Optionally, the step of performing infrared fill light in the exposure time corresponding to the preset exposure within the image acquisition period comprises: performing a preset time in the image collection period according to the following control manner The infrared fill light is performed during the exposure time corresponding to the exposure: the start time of the infrared fill light is not earlier than the exposure start time of the first preset exposure, and the end time of the infrared fill light is not later than the exposure of the preset preset exposure End time.

Optionally, when the number of exposures in the image acquisition period is greater than two times, the first preset exposure is the first exposure or the last exposure of the at least two exposures.

Optionally, the step of determining an infrared illuminating image based on one of the frame images comprises: demosaicing one of the frame images to generate a sensible infrared brightness image.

Optionally, the image to be processed is the remaining image in the image of each frame, and the number of remaining images in each frame image is 1, and the image to be processed in the image of each frame is de-infrared. Processing, the step of obtaining a visible light color image, comprising: performing de-infrared processing on the remaining images in the frame images to obtain a visible light color image.

Optionally, the step of performing de-infrared processing on the remaining images in the image of each frame to obtain a visible light color image includes: performing an IR channel of the target image if the target image includes an IR channel Interpolating, generating the target image after the interpolation process, wherein the target image is the remaining image in each frame image; for each pixel in the target image after the interpolation process, updating is performed as follows Visible light color image: If the pixel has an R value, the R value of the pixel is updated: the difference between the R value of the pixel and the IR parameter value of the pixel; if the pixel has a G value, the G value of the pixel is updated: The difference between the G value of the pixel and the IR parameter value of the pixel; if the pixel has a B value, the B value of the pixel is updated: a difference between the B value of the pixel and the IR parameter value of the pixel; wherein, the pixel The IR parameter value is the product of the IR value of the pixel and the preset correction value.

Optionally, the image to be processed is the remaining image in each frame image, the number of remaining images in each frame image is at least 2, and the exposures of the remaining images in the frame images respectively are correspondingly exposed. The durations are different; the step of performing de-infrared processing on the image to be processed in the image of each frame to obtain a visible light color image includes: performing wide dynamic synthesis processing on the remaining images in the image of each frame to obtain a wide dynamic An image; de-infrared processing the wide dynamic image to obtain a visible light color image.

Optionally, the method is applied to an image fusion device, where each frame image is collected by the image fusion device; an optical lens of the image fusion device is provided with a filter, and the filter is filtered by the filter. The region includes [T1, T2]; wherein, 600 nm ≤ T1 ≤ 800 nm, 750 nm ≤ T2 ≤ 1100 nm, and T1 < T2.

In a second aspect, an embodiment of the present application provides an image fusion device, where the device includes: an obtaining module, configured to obtain each frame image obtained by at least two exposures in one image acquisition period; The second determining module is configured to perform de-infrared processing on the image to be processed in the image of each frame to obtain a visible light color image; and a fusion module, to determine an infrared brightness image based on one of the frame images; And merging the infrared illuminance image with the visible light color image to obtain a fused image.

Optionally, the device is applied to an image fusion device, and the frame images are collected by the image fusion device; the device further includes: an infrared fill light module, configured to be preset in the image collection period The infrared light is added to the exposure time corresponding to the second exposure; the first determining module is specifically configured to: determine the infrared brightness image based on the image obtained by the preset exposure.

Optionally, the exposure parameter corresponding to the preset exposure is not greater than a target maximum, wherein the exposure parameter is an exposure duration and/or a gain, and the target maximum is a preset exposure The maximum value of the exposure parameters corresponding to each of the other exposures.

Optionally, the infrared fill light module is specifically configured to: perform infrared fill light in an exposure time corresponding to the preset exposure time in the image acquisition period according to the following control manner: the start time of the infrared fill light is not The end time of the infrared fill light is earlier than the exposure end time of the first preset exposure, earlier than the exposure start time of the preset exposure.

Optionally, when the number of exposures in the image acquisition period is greater than two times, the first preset exposure is the first exposure or the last exposure of the at least two exposures.

Optionally, the first determining module is specifically configured to: perform demosaic processing on one of the frames of the frames to generate an infrared brightness image.

Optionally, the image to be processed is the remaining image in the image of each frame, and the number of the remaining images in the frame image is 1. The second determining module is specifically configured to: The remaining images in the image are subjected to de-infrared processing to obtain a visible light color image.

Optionally, the second determining module includes: an interpolation submodule, configured to interpolate an IR channel of the target image, where the target image includes an IR channel, to generate the target image after the interpolation process The target image is the remaining image in the image of each frame; the update sub-module is configured to update each pixel in the target image after the interpolation process as follows to obtain a visible light color image: The pixel has an R value, and the R value of the pixel is updated: the difference between the R value of the pixel and the IR parameter value of the pixel; if the pixel has a G value, the G value of the pixel is updated: the G value of the pixel The difference from the IR parameter value of the pixel; if the pixel has a B value, the B value of the pixel is updated: the difference between the B value of the pixel and the IR parameter value of the pixel; wherein the IR parameter value of the pixel is The product of the IR value of the pixel and the preset correction value.

Optionally, the image to be processed is the remaining image in each frame image, the number of remaining images in each frame image is at least 2, and the exposures of the remaining images in the frame images respectively are correspondingly exposed. The second determining module includes: a first processing submodule, configured to perform wide dynamic synthesis processing on the remaining images in the frame images to obtain a wide dynamic image; and a second processing submodule, configured to: The wide dynamic image is subjected to de-infrared processing to obtain a visible light color image.

Optionally, the device is applied to an image fusion device, where each frame image is collected by the image fusion device; an optical lens of the image fusion device is provided with a filter, and the filter is filtered by the filter. The region includes [T1, T2]; wherein, 600 nm ≤ T1 ≤ 800 nm, 750 nm ≤ T2 ≤ 1100 nm, and T1 < T2.

In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory is used to store program code, and the processor is configured to execute any of the foregoing when executing the program code stored on the memory. Method steps as described in the image fusion method.

In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, where the computer readable storage medium stores a computer program, where the computer program is executed by a processor to implement any of the image fusion methods described above. Method steps.

In a fifth aspect, an embodiment of the present application provides a computer program, where the computer program is configured to perform the method steps described in any of the image fusion methods described above at runtime.

It can be seen that, in the solution provided by the embodiment of the present application, each frame image obtained by at least two exposures in one image acquisition period is first obtained; and then one of the frames of each frame image is collected to determine the infrared brightness. And determining a visible light color image based on the remaining images in the acquired frame images; finally, the determined infrared brightness image is merged with the visible light color image to obtain a fused image. Compared with the prior art, in the solution provided by the embodiment of the present application, each frame image obtained by at least two exposures in one image acquisition period may be collected by one image sensor, so that only one image sensor exists in the device. The image collection and fusion can be completed to improve the image quality in the case of low illumination. Therefore, the device provided by the embodiment of the present application has good adaptability and is convenient to apply; from another perspective, for image acquisition and integration. For the device to which the solution provided by the embodiment of the present application is applied, only one sensor can be disposed in the device, and the spectroscopic device is not required, and the structure is simple and the device cost is low.

DRAWINGS

In order to more clearly illustrate the embodiments of the present application and the technical solutions of the prior art, the following description of the embodiments and the drawings used in the prior art will be briefly introduced. Obviously, the drawings in the following description are only Some embodiments of the application may also be used to obtain other figures from those of ordinary skill in the art without departing from the scope of the invention.

FIG. 1 is a schematic flowchart diagram of an image fusion method according to an embodiment of the present application.

FIG. 2 is a schematic diagram of a format of an RGBIR image sensor according to an embodiment of the present application.

FIG. 3 is a schematic flowchart diagram of an image fusion method according to another embodiment of the present application.

FIG. 4 is a schematic diagram of spectral response according to an embodiment of the present application.

FIG. 5 is a schematic diagram showing the relationship between exposure and infrared fill light according to an embodiment of the present application.

FIG. 6 is a schematic diagram showing another relationship between exposure and infrared fill light according to an embodiment of the present application.

FIG. 7 is a schematic structural diagram of an image fusion device according to an embodiment of the present application.

FIG. 8 is a schematic structural diagram of an image fusion device according to an embodiment of the present application.

FIG. 9 is a schematic structural diagram of an image fusion device according to another embodiment of the present disclosure.

FIG. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

In order to make the objects, technical solutions, and advantages of the present application more comprehensible, the present application will be further described in detail below with reference to the accompanying drawings. It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

The technical terms involved in this application are briefly introduced below.

The image acquisition period, in this document, refers to the time period corresponding to each frame image obtained by successive multiple exposures. The time of the image acquisition period is usually not too long. For example, an image acquisition period is 40 ms (milliseconds). Taking an image sensor as an example, the image sensor can generate an image by using the incident light signal obtained by each exposure, and after multiple exposures, a multi-frame image can be obtained. If a multi-frame image is used to obtain a frame of the fused image, the multi-frame is obtained. The sum of the exposure times corresponding to the images may be the image acquisition period described above.

In addition, for the video shooting process, each video image in the video can be regarded as a fused image in the present application file, that is, each frame of the video image is obtained by fusion of multi-frame original images obtained by image sensor imaging. Therefore, in the video technology field, the image capturing period may be: a start exposure time of the first frame original image corresponding to the previous frame video image, and a start exposure time of the first frame original image corresponding to the next frame video image. The time passed.

Visible light is an electromagnetic wave that can be perceived by the human eye. The visible spectrum has no precise range. The wavelength of electromagnetic waves that the human eye can perceive is between 400 and 760 nm (nanometer). Infrared light is an electromagnetic wave having a wavelength between 760 nm and 1 mm (mm), which is not visible to the human eye.

A visible light color image may refer to a color image in which only a visible light signal is perceived, and the color image is only sensitive to a visible light band.

The infrared brightness image may be a brightness image that senses the infrared light signal. It should be noted that the infrared brightness image is not limited to a brightness image that only senses the infrared light signal, and may also be an infrared light signal. And brightness images of other band optical signals.

In order to solve the problems mentioned in the above background, the embodiments of the present application provide an image fusion method, apparatus, electronic device, and computer readable storage medium to improve device adaptability of an image fusion technology.

An image fusion method provided by the embodiment of the present application is described in detail below.

The image fusion method provided by the embodiment of the present application may be applied to an image fusion device, and the image fusion device may be a device having an image collection function, such as a camera. In addition, the image fusion device may not have an image collection function. However, the device that is in communication with the image capturing device can receive the image that is collected and sent by the image capturing device, which is reasonable. The embodiment of the present application does not limit the specific form of the image fusion device.

An image fusion method provided by the embodiment of the present application is as shown in FIG. 1 , and the foregoing method includes:

S101: Obtain each frame image obtained by at least two exposures in one image acquisition period.

With regard to the obtaining form of each frame image in step S101, as described above, in one case, the image fusion device may be a device having an image capturing function, and step S101 may be: at least two in an image capturing period. Each frame image is obtained by the sub-exposure acquisition, that is, each frame image obtained in step S101 is acquired by the image fusion device itself. Among them, one exposure gives one frame of image. For example, the image fusion device is a camera that performs 3 exposures in one image acquisition cycle, and the camera captures 3 frames of images.

It should be noted that, in the embodiment of the present application, each frame image acquired by the image fusion device in one image acquisition period may not be obtained by imaging by one image sensor in the image fusion device, for example, a camera with dual cameras.

In addition, in another case, step S101 may also be: receiving images of each frame acquired by other devices through at least two exposures in one image acquisition period.

In another case, as described above, the image fusion device is a device that does not have an image acquisition function but communicates with other image acquisition devices, and step S101 may be: receiving the image collection device and transmitting within an image collection period. The resulting frame images are acquired by at least two exposures. For example, the monitoring front-end camera acquires three frames of images in one image acquisition period, and transmits the three-frame images to the image fusion device of the monitoring back end, that is, the image fusion device that monitors the back end obtains the above three frames of images.

The number of images in each frame image obtained above is the same as the number of exposures in the above-described one image acquisition period, and only one frame of exposure image can be obtained for each exposure. The number of exposures corresponding to an image acquisition period may be preset. For example, the preset number of exposures is two, and the above step S101 may specifically obtain two frames of images obtained by two exposures in one image acquisition period.

It should be noted that, in the embodiment of the present application, each of the obtained frame images is obtained by imaging by an image sensor, and the image sensor may be a common image sensor, but in order to ensure that each frame image obtained above is included. With as many infrared components as possible, the image sensor described above can be an RGBIR image sensor, such as the OVIR2 image sensor of the OV4682 produced by Ominivision.

Exemplarily, it is assumed that the image fusion device itself is provided with an image acquisition component for acquiring the obtained frame images, and the image sensor used in the image acquisition component is an RGBIR image sensor.

S102: Determine an infrared brightness image based on one of the obtained frames of each frame image.

Since the image obtained by each exposure is a brightness image that can be perceived as an infrared light signal, as an alternative implementation of the embodiment of the present application, one frame of the image can be randomly selected in each frame image obtained, and the image is determined to be a sense. Infrared brightness image, for example, three frames of images are collected in one image acquisition period, and the image fusion device randomly selects the second frame image, and determines the second frame image as an infrared brightness image; as another embodiment of the present application, Alternatively, the image of a certain frame may be determined to be an infrared brightness image. For example, three frames of images are acquired in one image acquisition period, and the last frame image collected is determined to be an infrared brightness image.

Because the image quality of the image sensor is generally poor, the quality of each frame image obtained above is poor. Therefore, in order to improve the image quality of the infrared brightness image, another alternative implementation manner of the embodiment of the present application is provided. The image processing may be performed on one of the obtained frame images to generate an infrared illuminance image. Of course, the specific processing method used for image processing of the frame image is not limited, as long as it is Image processing methods that improve image quality are all reasonable.

As an alternative manner of the embodiment of the present application, in order to obtain a sensible infrared brightness image with clear and real image details, the image processing mode may be a demosaic process, that is, among the obtained frame images. One frame is subjected to demosaic processing to generate an infrared illuminance image. That is, the step of determining the infrared brightness image based on one of the obtained frames of the frame (S102) may include:

Step X: Perform demosaic processing on one of the obtained frame images to generate an infrared illuminance image.

It can be understood by those skilled in the art that in the image directly imaged by the image sensor, the signals of each channel are staggered, and the RGBIR image sensor is taken as an example, as shown in FIG. 2, R (red, red), G (green, green). ), B (blue, blue) and IR (Infrared Spectroscopy (infrared) channel signals are staggered, and when you directly zoom in and view the image obtained by the image sensor, you will find mosaic in the image, and the resolution is not good, so you need to go Mosaic processing to generate images with real details.

For convenience of description, one of the frames of the image obtained here is referred to as a target image, and then the above-described demosaic processing of the target image to generate the infrared brightness image includes steps 1 and 2:

Step 1: Interpolate the R, G, B, and IR channels of the target image respectively to obtain the R value (R channel value), G value (G channel value), and B value (B channel) corresponding to each pixel in the target image. Value) and IR value (IR channel value).

Specifically, the interpolation method used in the interpolation in step 1 may be a bilinear interpolation algorithm, a bicubic interpolation algorithm, etc., and the interpolation algorithm used in this embodiment is not limited in this embodiment.

Step 2: The R channel value, the G channel value, the B channel value, and the IR (Infrared Spectroscopy) value in the target image obtained after the interpolation are averaged to obtain a sensed infrared brightness image after demosaic processing.

That is, step 2 can obtain a sensible infrared brightness image that includes only the brightness signal, and the brightness value of each pixel in the infrared brightness image is: corresponding channel value after interpolation in the target image. average of. Taking the image corresponding to the RGBIR image sensor as an example, the luminance values of each pixel in the image are: the R channel value after interpolation of the pixel, the G channel value after interpolation, the B channel value after interpolation, and the interpolation. The average value of the IR channel value; for example, the luminance value of the pixel coordinate (x, y) in the infrared illuminance image is equal to the interpolated R channel value of the pixel coordinate (x, y) in the target image, and the interpolation is performed. The G channel value, the interpolated B channel value, and the average of the interpolated IR channel values.

Of course, the above steps 1 and 2 are only exemplary descriptions of the step X, and do not constitute a specific limitation on the embodiments of the present application. Those skilled in the art may complete the step X based on other specific technical means.

S103: De-infrared processing the image to be processed in each frame image to obtain a visible light color image.

The visible light color image determined in step S103 is an image that does not include an infrared component, so it is necessary to perform de-infrared processing on the image to be processed in each frame image to obtain a visible light color image having a true color reproduction degree.

In an implementation manner, the image to be processed may be the rest of the images in each frame image. First, it should be noted that the remaining images mentioned herein are used in the above-mentioned frame images to be removed in step S102. The image remaining after one frame of the luminance image. For example, the image of each frame obtained above includes images a to c, wherein the image a is used to determine the infrared brightness image, and the images b and c are the remaining images; for example, the image of each frame obtained includes the image. d and e, where the image d is used to determine the infrared brightness image, then the image e is the remaining image.

It can be understood that the number of the remaining images involved in the foregoing may be one or at least two. Therefore, in the embodiment of the present application, in one case, when the number of remaining images in the image of each frame is 1, the above De-infrared processing the image to be processed in each frame image to obtain a visible light color image (S103), which may include:

De-infrared processing is performed on the remaining images in the obtained frame images to obtain a visible light color image.

After de-infrared processing is performed on the remaining images in the frame images obtained as described above, the infrared components in the image are removed, and a visible light color image is obtained. Certainly, the manner of de-infrared processing the image may refer to the prior art. The specific implementation manner of the de-infrared processing may not be limited in the embodiment of the present application.

As an alternative implementation in this case, as shown in FIG. 3, based on the method embodiment shown in FIG. 1, the above-mentioned images in the obtained frame images are de-infrared processed to obtain visible light colors. The steps of the image may include:

S1031: If the target image includes an IR channel, the IR channel of the target image is interpolated to generate an interpolated target image, wherein the target image is the remaining image in each frame image.

For example, the image sensor used for imaging to obtain the target image is an RGBIR image sensor, and the target image includes an IR channel; it can be understood that after interpolating the IR channel of the target image, each pixel in the target image after the interpolation process corresponds to IR value. For the same reason, the interpolation method used in the interpolation in step S1031 may be a bilinear interpolation algorithm, a bicubic interpolation algorithm, etc., and the interpolation algorithm adopted in this embodiment is not limited in this embodiment.

S1032: For each pixel in the target image after the interpolation process, updating is performed as follows to obtain a visible light color image: if the pixel has an R value, the R value of the pixel is updated: the R value of the pixel and the IR of the pixel. The difference between the parameter values; if the pixel has a G value, the G value of the pixel is updated: the difference between the G value of the pixel and the IR parameter value of the pixel; if the pixel has a B value, the B value of the pixel is updated. The difference between the B value of the pixel and the IR parameter value of the pixel; wherein the IR parameter value of the pixel is the product of the IR value of the pixel and the preset correction value.

The preset correction value may be any integer or decimal of 0 to 1024. The specific value of the preset correction value may be set according to an actual situation. The embodiment of the present application does not limit the value of the preset correction value. In general, the preset correction value may be set to 1, the step S1032 may be specifically: for each pixel in the target image after the interpolation processing, updating according to the following manner to obtain a visible light color image: if the pixel has an R value Update the R value of the pixel: the difference between the R value of the pixel and the IR value of the pixel; if the pixel has a G value, update the G value of the pixel: the G value of the pixel and the IR value of the pixel The difference value; if the pixel has a B value, the B value of the pixel is updated: the difference between the B value of the pixel and the IR value of the pixel. Of course, those skilled in the art can understand that the value of the preset correction value is not limited thereto.

Specifically, after the step S1031 is executed, each pixel of the target image corresponds to an IR value, but the image fusion device does not separately interpolate the R channel, the G channel, and the B channel of the target image, so the pixels in the target image are It may correspond to only the IR value, or in addition to the IR value, it also has an R value, a G value or a B value.

Exemplarily, if the preset correction value is 1, and the IR parameter value of the pixel is the IR value of the pixel, step S1032 can be understood as:

For each pixel in the target image after interpolation, update as follows:

If the pixel has an R value, the R value of the pixel is updated: the difference between the R value of the pixel and the IR value of the pixel; if the pixel has a G value, the G value of the pixel is updated: the G value of the pixel The difference from the IR value of the pixel; if the pixel has a B value, the B value of the pixel is updated: the difference between the B value of the pixel and the IR value of the pixel; of course, if the pixel only has an IR value, Then no update processing is performed on the pixel.

At this time, a color image having only three channels of RGB can be obtained, which can be used as a visible light color image.

It should be noted that after de-infrared processing is performed on the remaining images in the obtained frame images, not every pixel has an R value, a G value, and a B value, so To improve the image quality, after the de-infrared processing, the R channel, the G channel, and the B channel of the remaining images in the above-mentioned frames are separately interpolated to obtain a visible light color image.

For the same reason, the interpolation method used for the interpolation of the R channel, the G channel, and the B channel of the remaining images in the foregoing frame images may be a bilinear interpolation algorithm, a bicubic interpolation algorithm, etc. The interpolation algorithm used is not limited.

In addition, the implementation of the de-infrared processing of the remaining images described herein may be performed by other implementations provided in the prior art, which is not limited herein.

In another case, when the number of remaining images in each of the frame images is at least 2, the exposure time lengths of the remaining images in the obtained frame images are different.

It can be understood that, in order to enable wide dynamic synthesis processing, a wide dynamic image is obtained, and the exposure time lengths of the remaining images are different. Specifically, a control unit may be disposed in the image fusion device to control the corresponding image of each frame. The length of exposure. For example, if the number of the remaining images is 2, the 3 frames of images collected by the image fusion device may be preset, and the exposure time of the second frame image is 32 ms, except for the first frame image to determine the infrared brightness image. The exposure time corresponding to the three-frame image is 2 ms.

In this case, the step of performing de-infrared processing on the image to be processed in each frame image to obtain a visible light color image (S103) may include the following steps a and b:

Step a: performing wide dynamic synthesis processing on the remaining images in the frame images obtained above to obtain a wide dynamic image.

A high dynamic range (HDR) image, which is also called a wide dynamic range image, which has no partial overexposure phenomenon compared to a low dynamic range image, and can reflect more image details, so in the embodiment of the present application, In order to obtain a visible light color image with more image details, the wide dynamic synthesis processing can be performed on the remaining images of the plurality of frames to obtain a wide dynamic image. Of course, the specific implementation of the wide dynamic synthesis processing on the multi-frame image belongs to the prior art, and the embodiments of the present application are not described in detail herein.

Step b: De-infrared processing the wide dynamic image to obtain a visible light color image.

For the specific implementation of the de-infrared processing of the wide dynamic image, reference may be made to the specific implementation of the de-infrared processing of the remaining images of one frame in the method embodiment shown in FIG. 3, where the embodiment of the present application is Do not repeat them.

In another implementation manner, the image to be processed may be all the images in each frame image. For example, each of the obtained frame images includes images a to c, wherein the image a is used to determine the infrared brightness image, and the image a ~c is the image to be processed.

Since each frame image is obtained by at least two exposures, and one frame of image is obtained by one exposure, the number of images to be processed mentioned above is at least two, and the exposure time lengths corresponding to the images to be processed are different. Therefore, in the embodiment of the present application, the specific implementation manner of performing the de-infrared processing on the image to be processed in each frame image to obtain the visible light color image (S103) may refer to step a and step b in step S103, and the difference is only for each frame. The remaining images in the image are replaced with all the images in each frame image, and are not described here.

S104: Fusion the infrared brightness image with the visible color image to obtain a fused image.

In the embodiment of the present application, the implementation of the infrared brightness image and the visible light color image may be various. As an implementation manner of the embodiment of the present application, the infrared brightness image and the visible color image are sensed. The step of obtaining the fused image may include the following steps a1 to a4:

Step a1: Calculate the luminance signal of each pixel in the visible light color image by the following formula:

Y=(R+G+B)/3;

Where Y represents the luminance signal value of the pixel in the visible light color image, R represents the R channel value of the pixel corresponding to Y, G represents the G channel value of the pixel corresponding to Y, and B represents the B channel value of the pixel corresponding to Y.

Step a2: calculating, for each pixel in the visible light color image, a ratio of an R channel value, a G channel value, and a B channel value of the pixel to a luminance signal value Y corresponding to the pixel, that is, K1=R/Y, K2=G /Y, K3=B/Y.

Step a3: performing color noise reduction processing on K1, K2, and K3 corresponding to all pixels in the visible light color image, for example, using Gaussian filtering processing to obtain K1', K2', K3' after the color noise reduction processing corresponding to each pixel.

Step a4: merging the luminance signal value Y' of each pixel in the infrared illuminance image with the K1', K2', K3' of the corresponding pixel in the visible light color image by using the following formula to obtain a fused image:

R'=K1'*Y';

G'=K2’*Y’;

B’=K3’*Y’;

Where R', G', and B' represent the R channel value, G channel value, and B channel value of the pixel in the fused image, respectively; K1', K2', and K3' respectively represent the visible light color image after color noise reduction processing K1, K2, K3 of the corresponding pixel; Y' represents the luminance signal value of the corresponding pixel in the infrared brightness image.

As another implementation manner of the embodiment of the present application, the step of fusing the infrared brightness image and the visible color image to obtain a fused image may include the following steps b1 b b4:

Step b1: Converting the RGB color signal in the visible light color image into a YUV (a color coding standard) signal.

Certainly, the specific implementation manner of converting the RGB color signal into the YUV signal belongs to the prior art, and the embodiments of the present application are not described in detail herein.

Step b2: Extract the UV component, that is, the color component, in the Y UV signal.

Step b3: Perform color noise removal processing on the extracted UV component, for example, Gaussian filtering noise reduction to obtain a processed UV component.

Step b4: combining the processed UV component with the luminance signal of the infrared illuminating image to form a new YUV signal, and the image corresponding to the new YUV signal may be used as the final fused image; The YUV signal is then converted into a new RGB signal, and the image corresponding to the new RGB signal is used as the final fused image.

In addition, similar to this implementation, the RGB color signal in the visible light color image may be converted into an HSV (a color coding standard) signal for image fusion, which is not limited herein.

In addition, in order to ensure the accurate restoration of the color after removing the infrared component, thereby improving the image fusion quality, when the above method is applied to the image fusion device, and the obtained frame images are collected by the image fusion device; The optical lens of the image fusion device may be provided with a filter, and the spectral region filtered by the filter may include [T1, T2]; wherein 600 nm ≤ T1 ≤ 800 nm, 750 nm ≤ T2 ≤ 1100 nm, and T1 < T2.

Referring to FIG. 4, it can be understood that the R, G, B, and IR channels have different responses in the near-infrared band (650 nm to 1100 nm), and the infrared component removal effect is poor in order to avoid the difference in response of each channel in some spectral regions. The problem is that the optical lens of the image fusion device is provided with a filter to filter out the spectral regions with large differences in response.

Specifically, the image fusion device may be provided with an image acquisition unit, and the image acquisition unit includes an optical lens, a filter disposed on the optical lens, and an image sensor. The filter can be integrated on the optical lens by a coating technology; in addition, the filter can be a band stop filter or a lower cost bimodal filter. It should be noted that the filter When the sheet is a bimodal filter, the spectral region filtered by the filter may further include a spectral region of [T3, +∞), 850 nm ≤ T3 ≤ 1100 nm, and T2 < T3.

Compared with the prior art, in the solution provided by the embodiment, each frame image obtained by at least two exposures in one image acquisition period may be collected by one image sensor, so that only one image sensor exists in the device. The image collection and fusion can be completed to improve the image quality in the case of low illumination. The device provided by the embodiment has good adaptability and is convenient to apply; from another perspective, for image acquisition and integration, For the device to which the solution provided in this embodiment is applied, only one sensor can be disposed in the device, and the spectroscopic device is not required, and the structure is simple and the device cost is low.

In order to obtain a fused image with a high signal-to-noise ratio and a higher quality, as an alternative implementation manner of the embodiment of the present application, the foregoing method is applied to an image fusion device, and the obtained frame images are collected by the image fusion device. In the case of any of the foregoing method embodiments, the method may further include:

The infrared fill light is performed during the exposure time corresponding to the preset exposure within the image acquisition period.

In the process of one exposure, the image fusion device generates an image by using the incident light signal captured by the optical lens. If the infrared light is not applied, the incident light signal captured by the optical lens includes only the ambient incident light signal, and the infrared compensation is performed. In the case of light, the incident light signal captured by the optical lens includes an ambient incident light signal and an infrared fill light signal.

The image fusion device performs infrared fill light in the preset exposure time. Specifically, the control unit may be set in the image fusion device to control the infrared fill light and the image acquisition unit, so that the fill time of the infrared fill light is completed. It is within a certain exposure time preset in the image acquisition unit.

It should be noted that the image fusion device performs infrared fill light during the preset exposure time, which can increase the quality of the aforementioned infrared brightness image, but if the image capture period is other than the preset exposure time Infrared fill light increases the difficulty of obtaining the above visible color image.

Therefore, in order to improve the quality of the infrared brightness image, the difficulty of obtaining the visible light color image is not increased. As an optional implementation manner of the embodiment of the present application, the preset exposure corresponding to the image acquisition period corresponds to The step of performing infrared fill light during the exposure time may include:

According to the following control mode, the infrared fill light is performed during the exposure time corresponding to the preset exposure in the image acquisition period: the start time of the infrared fill light is not earlier than the exposure start time of the first preset exposure, and the infrared fill light is The end time is not later than the exposure end time of the first preset exposure.

Exemplarily, the image fusion device is provided with a control unit, and the control unit controls to activate the infrared fill light at the start time of the preset exposure in the image acquisition period, and controls to turn off the infrared fill light at the end time of the preset exposure. The infrared fill light is completely synchronized with the preset preset exposure, that is, the infrared fill light starts when the first preset exposure starts, and the infrared fill light ends when the preset preset exposure ends.

In the embodiment of the present application, the fill light intensity of the infrared fill light can be set according to actual conditions, and the embodiment of the present application does not limit the fill light intensity of the infrared fill light. In addition, the exposure duration corresponding to the one-time exposure of the infrared fill light can be determined according to the actual fill light parameter, and the embodiment of the present application also does not limit the exposure time length corresponding to the one-time exposure of the infrared fill light.

In addition, the wavelength band of the infrared light used for the infrared fill light may not be limited, but for the image sensor to obtain the maximum response, the embodiment of the present application may perform infrared fill light using infrared light having a wavelength band of 850 nm to 900 nm.

In this case, the step of determining the infrared brightness image based on one of the obtained frames of the frame (S102) may include:

Based on the image obtained by the above-mentioned preset exposure, the infrared brightness image is determined.

It can be understood that, in the embodiment of the present application, the image used to determine the infrared brightness image is obtained by exposure in the presence of infrared fill light, and the remaining images in the above frame images are in the absence of infrared fill light. Obtained under conditions of exposure.

The video capture process is as follows. As described above, each frame of the video is the fused image in the embodiment of the present application. Since the video frame in the video is continuously collected, the infrared fill light is added. It is a stroboscopic fill light, and the period of stroboscopic fill light is the same as the acquisition period of each frame of image.

It can be understood that the infrared fill light performed during the exposure process of the first preset exposure enhances the brightness of the image obtained by the preset exposure, so that the brightness of the image obtained by the preset exposure is maintained at an appropriate level. In the range of the brightness, in the embodiment of the present application, the exposure parameter corresponding to the preset exposure may be no greater than the target maximum, wherein the exposure parameter is the exposure duration and/or the gain, and the target maximum is the preset. The maximum value of the exposure parameters corresponding to the other exposures other than the secondary exposure.

Taking the exposure parameter as the exposure duration as an example, assuming that three exposures are performed within one image acquisition period, the image obtained by the third exposure in the preset image acquisition period is used to generate an infrared luminance image, three times in the image acquisition period. The exposure durations of the exposure are: x milliseconds, y milliseconds, and z milliseconds. Assuming x>y, there must be x≥z; for example, the exposure durations of the three exposures in the image acquisition period are: 25 milliseconds, 5 milliseconds, and 10 milliseconds.

Assuming that two exposures are performed during an image acquisition period, the image obtained by the first exposure in the preset image acquisition period is used to generate an infrared brightness image, and the exposure durations of the two exposures in the image acquisition period are: For m milliseconds and n milliseconds, n ≥ m must exist; for example, the exposure durations of the two exposures in the image acquisition period are: 10 milliseconds and 30 milliseconds, respectively.

In addition, when the number of exposures in the image capturing period is greater than two times, the first preset exposure may be the first exposure or the last exposure of the at least two exposures.

It can be understood that when the number of exposures in the image capturing period is greater than two times, at least three frames of images can be obtained. Among the at least three frames, one frame is used to generate an infrared brightness image, and the remaining frame images are used to generate visible light colors. The image, so the remaining frame images here need to be images continuously acquired by the image sensor. To achieve this requirement, the first preset exposure may be the first exposure or the last exposure of the at least two exposures.

For example, a wide dynamic range image may be first generated using the remaining frame images described above, and then a visible light color image may be generated using the generated wide dynamic range image. Since the generation of a wide dynamic range image requires a multi-frame image that is continuously acquired, the remaining frame images here need to be images that are continuously acquired by the image sensor.

For the present embodiment, as an example, as shown in FIG. 5, one image acquisition period includes two exposures, that is, double shutter exposure in FIG. 5, one odd exposure in FIG. 5, and an adjacent even exposure. Corresponding to an image acquisition period, and the corresponding image obtained by the even exposure is used to determine the infrared brightness image. From the brightness curve of the infrared light in the figure, it can be seen that the rising edge of the infrared fill light can be later than the start time of the even exposure. But not early; for the same reason, the falling edge can be earlier than the end of the even exposure, but not late; that is, infrared fill light should not be early or delayed in the even exposure. It can be understood that in the continuous acquisition process of the video frame, the infrared lamp only performs infrared filling light for the even exposure, and forms a stroboscopic fill light effect.

As shown in FIG. 6, an image acquisition period includes three exposures, that is, A exposure in FIG. 6, and adjacent B exposure and C exposure, and C exposure corresponds to the obtained image to determine an infrared brightness image. From the brightness curve of the infrared light in the figure, it can be seen that the rising edge of the infrared fill light can be later than the start time of the C exposure, but not earlier; similarly, the falling edge can be earlier than the end of the C exposure, but not later; that is, infrared Fill light should not be early or delayed in the C exposure. It can be understood that in the continuous acquisition process of the video frame, the infrared lamp only performs infrared filling light for the Cth exposure to form a stroboscopic fill light effect, wherein the third time in FIG. 6 is the third time. The sixth time.

It can be understood that, in this embodiment, the image for determining the infrared brightness image is obtained by exposure in the presence of infrared fill light, and the infrared brightness image is enhanced by infrared fill light, and has a better signal to noise ratio. By merging the infrared brightness image with the visible color image, a fused image with better quality can be obtained.

The following briefly describes an embodiment of the present application by a specific example.

In order to more clearly demonstrate the process of obtaining a fused image by the image fusion device, in this example, the image collection device is divided into a plurality of units, and the image fusion process is completed by each unit; of course, the image fusion device is not divided in this example. The limitations of the application are to be construed as merely illustrative.

As shown in FIG. 7, the image fusion device may include an infrared fill light unit (such as a fill light), a control unit, an image acquisition unit, a preprocessing unit, and a fusion processing unit, wherein the preprocessing unit and the fusion processing unit may be regarded as Is an image synthesis unit.

It should be noted that the control unit may send an exposure control signal to the image acquisition unit to control the image acquisition unit to acquire multiple frames of images in one image acquisition period, and may control the exposure duration of each exposure through the exposure control signal; The fill light control signal may be sent to the infrared fill light unit, so that the infrared fill light unit ensures that the infrared fill light is performed within a preset one exposure time.

Specifically, the process of obtaining a fused image by the image fusion device is as follows:

The RGBIR image sensor in the image acquisition unit obtains images a, b, and c by three consecutive exposures in one image acquisition period, and in the process of obtaining image c in the third exposure, the infrared fill light unit performs infrared fill light. The image c is made imaged based on ambient incident light and infrared fill light.

Then, the pre-processing unit performs wide dynamic synthesis processing on the images a and b to obtain a wide dynamic image, and performs de-infrared processing on the wide dynamic image to obtain a visible light color image. At the same time, the pre-processing unit also performs demosaic processing on the image c to obtain an infrared illuminance image.

Finally, the fusion processing unit obtains the visible light color image and the infrared illuminance image from the preprocessing unit, and fuses the visible light color image and the sensible infrared brightness image to obtain a fused image.

Corresponding to the method embodiment shown in FIG. 1 , the embodiment of the present application further provides an image fusion device. As shown in FIG. 8 , the device includes:

The obtaining module 110 is configured to obtain each frame image obtained by at least two exposures in one image acquisition period; and the first determining module 120 is configured to determine the infrared brightness image based on one of the frame images; The second determining module 130 is configured to perform de-infrared processing on the image to be processed in the image of each frame to obtain a visible light color image, and the fusion module 140 is configured to fuse the infrared brightness image with the visible color image. Get a fused image.

As an optional implementation of the embodiment of the present application, the device is applied to an image fusion device, and the image of each frame is collected by the image fusion device; the device may further include:

The infrared fill light module is configured to perform infrared fill light in an exposure time corresponding to the preset exposure time in the image acquisition period; the first determining module 120 may be specifically configured to: based on the preset time The resulting image is exposed to determine an infrared brightness image.

Specifically, the exposure parameter corresponding to the preset exposure may be not greater than a target maximum, wherein the exposure parameter is an exposure duration and/or a gain, and the target maximum is in addition to the preset exposure The maximum value of the exposure parameters corresponding to each of the other exposures.

As an optional implementation manner of the embodiment of the present application, the infrared fill light module may be specifically configured to: perform infrared in an exposure time corresponding to a preset exposure in the image collection period according to the following control manner; Fill light: The start time of the infrared fill light is not earlier than the exposure start time of the first preset exposure, and the end time of the infrared fill light is not later than the exposure end time of the first preset exposure.

Specifically, when the number of exposures in the image acquisition period is greater than two times, the first preset exposure may be the first exposure or the last exposure of the at least two exposures.

Specifically, the first determining module 120 may be specifically configured to: perform demosaic processing on one of the frames of the frames to generate an infrared brightness image.

As an optional implementation manner of the embodiment of the present application, the to-be-processed image is the remaining image in the frame image, and when the number of the remaining images in the frame image is 1, the second determination The module 130 may be specifically configured to perform de-infrared processing on the remaining images in the frame images to obtain a visible light color image.

In this implementation, corresponding to the method embodiment shown in FIG. 3, specifically, as shown in FIG. 9, the second determining module 130 may include: an interpolation sub-module 1301, configured to include an IR channel in the target image. In the case of the target image, the target image is interpolated, and the target image is the remaining image in the frame image; the update sub-module 1302 is configured to After the interpolation process, each pixel in the target image is updated as follows to obtain a visible light color image: if the pixel has an R value, the R value of the pixel is updated: the R value of the pixel and the IR parameter of the pixel. The difference between the values; if the pixel has a G value, the G value of the pixel is updated: the difference between the G value of the pixel and the IR parameter value of the pixel; if the pixel has a B value, the B value of the pixel is updated. The difference between the B value of the pixel and the IR parameter value of the pixel; wherein the IR parameter value of the pixel is the product of the IR value of the pixel and the preset correction value.

As another optional implementation manner of the embodiment of the present application, the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is at least 2, and each of the The second determining module 130 may include: a first processing sub-module, configured to perform wide dynamic synthesis processing on the remaining images in the frame images of the frame images. Obtaining a wide dynamic image; and a second processing sub-module for performing de-infrared processing on the wide dynamic image to obtain a visible light color image.

Specifically, the device may be applied to an image fusion device, where each frame image is collected by the image fusion device; an optical lens of the image fusion device may be provided with a filter, and the filter is filtered. The spectral region includes [T1, T2]; wherein, 600 nm ≤ T1 ≤ 800 nm, 750 nm ≤ T2 ≤ 1100 nm, and T1 < T2.

It can be seen from the above that, compared with the prior art, in the solution provided by the embodiment, each frame image obtained by at least two exposures in one image acquisition period may be collected by one image sensor, so as long as it is in the device. There is an image sensor, which can complete the image collection and fusion to improve the image quality under low illumination conditions. The device provided by this embodiment has good adaptability and is easy to apply; from another perspective, for image acquisition and For a device integrated with the solution provided by the embodiment, only one sensor can be disposed in the device, and the spectroscopic device is not required, and the structure is simple and the device cost is low.

Corresponding to the method embodiment shown in FIG. 1 or FIG. 3, the embodiment of the present application further provides an electronic device. As shown in FIG. 10, the electronic device includes a memory 210 and a processor 220.

The memory 210 is configured to store program code. When the processor 220 is configured to execute the program code stored on the memory 210, the following steps are performed: obtaining each frame image obtained by at least two exposures in one image acquisition period; Determining an infrared brightness image based on one of the obtained frames of each frame image; performing de-infrared processing on the image to be processed in each obtained frame image to obtain a visible light color image; and sensing the infrared brightness image and the visible light color image Fusion to obtain fused images.

As an optional implementation manner of the embodiment of the present application, the method is applied to an image fusion device, where each frame image is collected by the image fusion device; the processor 220 is further configured to perform the following steps: Performing infrared fill light corresponding to the exposure time corresponding to the preset exposure in the period; and determining, according to one of the frames of the frames, the step of determining the infrared brightness image, comprising: based on the preset exposure The resulting image determines the infrared brightness image.

As an optional implementation manner of the embodiment of the present application, the exposure parameter corresponding to the preset exposure is not greater than a target maximum, wherein the exposure parameter is an exposure duration and/or a gain, and the target maximum The maximum value of the exposure parameters corresponding to each of the other exposures except the preset exposure.

As an optional implementation manner of the embodiment of the present application, the step of performing infrared light filling in the exposure time corresponding to the preset exposure in the image collection period may include: following the following control manners; The infrared fill light is performed in the exposure time corresponding to the preset exposure in the image acquisition period: the start time of the infrared fill light is not earlier than the exposure start time of the first preset exposure, and the end time of the infrared fill light is not late At the end of exposure of the first predetermined exposure.

As an optional implementation manner of the embodiment of the present application, when the number of exposures in the image collection period is greater than two times, the first preset exposure is the first exposure or the last of the at least two exposures. One exposure.

As an optional implementation manner of the embodiment of the present application, the step of determining an infrared illuminance image based on one of the frame images may include: performing one of the frame images De-mosaic processing to generate an infrared brightness image.

As an optional implementation manner of the embodiment of the present application, the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is 1, and the frames are The step of performing de-infrared processing on the image to be processed in the image to obtain a visible light color image includes: performing de-infrared processing on the remaining images in the frame images to obtain a visible light color image.

As an optional implementation manner of the embodiment of the present application, the step of performing de-infrared processing on the remaining images in the frame images to obtain a visible light color image may include: if the target image includes an IR channel Interpolating an IR channel of the target image to generate the target image after the interpolation process, wherein the target image is the remaining image in each frame image; and each of the target images after the interpolation process One pixel is updated as follows to obtain a visible light color image: if the pixel has an R value, the R value of the pixel is updated: the difference between the R value of the pixel and the IR parameter value of the pixel; if the pixel has G Value, update the G value of the pixel: the difference between the G value of the pixel and the IR parameter value of the pixel; if the pixel has a B value, update the B value of the pixel: the B value of the pixel and the pixel The difference of the IR parameter values; wherein the IR parameter value of the pixel is the product of the IR value of the pixel and the preset correction value.

As an optional implementation manner of the embodiment of the present application, the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is at least 2, and the image of each frame is The exposure time lengths of the remaining images in the respective frames are different; the step of performing de-infrared processing on the image to be processed in the image of each frame to obtain a visible light color image may include: resting the remaining images in the frames The image is subjected to wide dynamic synthesis processing to obtain a wide dynamic image; the wide dynamic image is subjected to de-infrared processing to obtain a visible light color image.

As an optional implementation manner of the embodiment of the present application, the method is applied to an image fusion device, where each frame image is collected by the image fusion device; and the optical lens of the image fusion device is provided with a filter. The spectral region filtered by the filter includes [T1, T2]; wherein, 600 nm ≤ T1 ≤ 800 nm, 750 nm ≤ T2 ≤ 1100 nm, and T1 < T2.

For the specific implementation of the various steps of the method and related explanations, reference may be made to the method embodiments and other method embodiments shown in FIG. 1 and FIG. 3, and details are not described herein.

The above memory may include a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (Ne twork processor, NP for short), or a digital signal processor (DSP). ), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component.

It can be seen from the above that, compared with the prior art, in the solution provided by the embodiment, each frame image obtained by at least two exposures in one image acquisition period may be collected by one image sensor, so as long as it is in the device. There is an image sensor, which can complete the image collection and fusion to improve the image quality under low illumination conditions. The device provided by this embodiment has good adaptability and is easy to apply; from another perspective, for image acquisition and For a device integrated with the solution provided by the embodiment, only one sensor can be disposed in the device, and the spectroscopic device is not required, and the structure is simple and the device cost is low.

In still another embodiment provided by the present application, there is provided a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the following steps are performed: obtaining a Each frame image obtained by at least two exposures in an image acquisition period; determining an infrared brightness image based on one of the obtained frame images; performing de-infrared processing on the image to be processed in each obtained frame image Obtaining a visible light color image; fusing the infrared brightness image with the visible color image to obtain a fused image.

As an optional implementation manner of the embodiment of the present application, the method is applied to an image fusion device, where each frame image is collected by the image fusion device; when the computer program is executed by the processor, the following steps may be implemented: Performing infrared fill light in an exposure time corresponding to the preset exposure in the image acquisition period; and determining, according to one of the frame images, the step of determining an infrared brightness image, comprising: The image obtained by the preset exposure is used to determine the infrared brightness image.

As an optional implementation manner of the embodiment of the present application, the exposure parameter corresponding to the preset exposure is not greater than a target maximum, wherein the exposure parameter is an exposure duration and/or a gain, and the target maximum The maximum value of the exposure parameters corresponding to each of the other exposures except the preset exposure.

As an optional implementation manner of the embodiment of the present application, the step of performing infrared light filling in the exposure time corresponding to the preset exposure in the image collection period may include: following the following control manners; The infrared fill light is performed in the exposure time corresponding to the preset exposure in the image acquisition period: the start time of the infrared fill light is not earlier than the exposure start time of the first preset exposure, and the end time of the infrared fill light is not late At the end of exposure of the first predetermined exposure.

As an optional implementation manner of the embodiment of the present application, when the number of exposures in the image collection period is greater than two times, the first preset exposure is the first exposure or the last of the at least two exposures. One exposure.

As an optional implementation manner of the embodiment of the present application, the step of determining an infrared illuminance image based on one of the frame images may include: performing one of the frame images De-mosaic processing to generate an infrared brightness image.

As an optional implementation manner of the embodiment of the present application, the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is 1, and the frames are The step of performing de-infrared processing on the image to be processed in the image to obtain a visible light color image includes: performing de-infrared processing on the remaining images in the frame images to obtain a visible light color image.

As an optional implementation manner of the embodiment of the present application, the step of performing de-infrared processing on the remaining images in the frame images to obtain a visible light color image may include: if the target image includes an IR channel Interpolating an IR channel of the target image to generate the target image after the interpolation process, wherein the target image is the remaining image in each frame image; and each of the target images after the interpolation process One pixel is updated as follows to obtain a visible light color image: if the pixel has an R value, the R value of the pixel is updated: the difference between the R value of the pixel and the IR parameter value of the pixel; if the pixel has G Value, update the G value of the pixel: the difference between the G value of the pixel and the IR parameter value of the pixel; if the pixel has a B value, update the B value of the pixel: the B value of the pixel and the pixel The difference of the IR parameter values; wherein the IR parameter value of the pixel is the product of the IR value of the pixel and the preset correction value.

As an optional implementation manner of the embodiment of the present application, the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is at least 2, and the image of each frame is The exposure time lengths of the remaining images in the respective frames are different; the step of performing de-infrared processing on the image to be processed in the image of each frame to obtain a visible light color image may include: resting the remaining images in the frames The image is subjected to wide dynamic synthesis processing to obtain a wide dynamic image; the wide dynamic image is subjected to de-infrared processing to obtain a visible light color image.

As an optional implementation manner of the embodiment of the present application, the method is applied to an image fusion device, where each frame image is collected by the image fusion device; and the optical lens of the image fusion device is provided with a filter. The spectral region filtered by the filter includes [T1, T2]; wherein, 600 nm ≤ T1 ≤ 800 nm, 750 nm ≤ T2 ≤ 1100 nm, and T1 < T2.

It can be seen from the above that, compared with the prior art, in the solution provided by the embodiment, each frame image obtained by at least two exposures in one image acquisition period may be collected by one image sensor, so as long as it is in the device. There is an image sensor, which can complete the image collection and fusion to improve the image quality under low illumination conditions. The device provided by this embodiment has good adaptability and is easy to apply; from another perspective, for image acquisition and For a device integrated with the solution provided by the embodiment, only one sensor can be disposed in the device, and the spectroscopic device is not required, and the structure is simple and the device cost is low.

In still another embodiment provided by the present application, there is also provided a computer program for implementing the image fusion method of any of the above embodiments at runtime.

It can be seen from the above that, compared with the prior art, in the solution provided by the embodiment, each frame image obtained by at least two exposures in one image acquisition period may be collected by one image sensor, so as long as it is in the device. There is an image sensor, which can complete the image collection and fusion to improve the image quality under low illumination conditions. The device provided by this embodiment has good adaptability and is easy to apply; from another perspective, for image acquisition and For a device integrated with the solution provided by the embodiment, only one sensor can be disposed in the device, and the spectroscopic device is not required, and the structure is simple and the device cost is low.

It should be noted that, in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply such entities or operations. There is any such actual relationship or order between them. Furthermore, the term "comprises" or "comprises" or "comprises" or any other variations thereof is intended to encompass a non-exclusive inclusion, such that a process, method, article, or device that comprises a plurality of elements includes not only those elements but also Other elements, or elements that are inherent to such a process, method, item, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device that comprises the element.

The various embodiments in the present specification are described in a related manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, the electronic device, and the computer readable storage medium embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The above is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., which are made within the spirit and principles of the present application, should be included in the present application. Within the scope of protection.

Claims (23)

An image fusion method, the method comprising: Obtaining each frame image obtained by at least two exposures in one image acquisition period; Determining an infrared brightness image based on one of the frames of the frames; De-infrared processing the image to be processed in each frame image to obtain a visible light color image; The sensible infrared brightness image is fused with the visible light color image to obtain a fused image. The method according to claim 1, wherein the method is applied to an image fusion device, and the frame images are collected by the image fusion device; The method further includes: Performing infrared fill light during an exposure time corresponding to the preset exposure within the image acquisition period; The step of determining an infrared brightness image based on one of the frames of the frames includes: The infrared brightness image is determined based on the image obtained by the preset exposure. The method according to claim 2, wherein the exposure parameter corresponding to the preset exposure is not greater than a target maximum value, The exposure parameter is an exposure duration and/or a gain, and the target maximum value is a maximum value of exposure indexes corresponding to the other exposures except the first preset exposure. The method according to claim 2, wherein the step of performing infrared fill light in the exposure time corresponding to the preset exposure within the image acquisition period comprises: According to the following control mode, infrared fill light is performed during the exposure time corresponding to the preset exposure within the image acquisition period: The start time of the infrared fill light is not earlier than the exposure start time of the first preset exposure, and the end time of the infrared fill light is not later than the exposure end time of the first preset exposure. The method according to claim 2, wherein said first exposure is the first exposure or the last of said at least two exposures when said number of exposures in said image acquisition period is greater than two One exposure. The method according to claim 1, wherein the step of determining an infrared illuminance image based on one of the frames of the frames comprises: Performing a demosaic process on one of the frames of the frames to generate an infrared illuminance image. The method according to any one of claims 1 to 6, wherein the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is 1. The step of performing de-infrared processing on the image to be processed in the image of each frame to obtain a visible light color image includes: De-infrared processing is performed on the remaining images in the frame images to obtain a visible light color image. The method according to claim 7, wherein the step of performing de-infrared processing on the remaining images in the frame images to obtain a visible light color image comprises: If the target image includes an IR channel, interpolating the IR channel of the target image to generate the target image after the interpolation process, wherein the target image is the remaining image in each frame image; For each pixel in the target image after the interpolation process, the image is updated as follows to obtain a visible light color image: If the pixel has an R value, the R value of the pixel is updated: the difference between the R value of the pixel and the IR parameter value of the pixel; if the pixel has a G value, the G value of the pixel is updated: G of the pixel The difference between the value and the IR parameter value of the pixel; if the pixel has a B value, the B value of the pixel is updated: the difference between the B value of the pixel and the IR parameter value of the pixel; wherein, the IR parameter value of the pixel The product of the IR value of the pixel and the preset correction value. The method according to any one of claims 1 to 6, wherein the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is at least 2, and The exposure durations corresponding to the remaining images in each frame image are different; The step of performing de-infrared processing on the image to be processed in the image of each frame to obtain a visible light color image includes: Performing wide dynamic synthesis processing on the remaining images in the frame images to obtain a wide dynamic image; The wide dynamic image is subjected to de-infrared processing to obtain a visible light color image. The method according to any one of claims 1 to 6, wherein the method is applied to an image fusion device, and the frame images are collected by the image fusion device; The optical lens of the image fusion device is provided with a filter, and the spectral region filtered by the filter includes [T1, T2]; wherein, 600 nm ≤ T1 ≤ 800 nm, 750 nm ≤ T2 ≤ 1100 nm, and T1 < T2. An image fusion device, characterized in that the device comprises: Obtaining a module for obtaining each frame image obtained by at least two exposures in one image acquisition period; a first determining module, configured to determine an infrared brightness image based on one of the frames of the frames; a second determining module, configured to perform de-infrared processing on the image to be processed in the image of each frame to obtain a visible light color image; And a fusion module, configured to fuse the infrared brightness image with the visible color image to obtain a fused image. The apparatus according to claim 11, wherein said apparatus is applied to an image fusion device, and said frame images are acquired by said image fusion device; The device also includes: The infrared fill light module is configured to perform infrared fill light during an exposure time corresponding to the preset exposure in the image acquisition period; The first determining module is specifically configured to: The infrared brightness image is determined based on the image obtained by the preset exposure. The device according to claim 12, wherein the exposure parameter corresponding to the preset exposure is not greater than a target maximum value, The exposure parameter is an exposure duration and/or a gain, and the target maximum value is a maximum value of exposure indexes corresponding to the other exposures except the first preset exposure. The device according to claim 12, wherein the infrared fill light module is specifically configured to: According to the following control mode, infrared fill light is performed during the exposure time corresponding to the preset exposure within the image acquisition period: The start time of the infrared fill light is not earlier than the exposure start time of the first preset exposure, and the end time of the infrared fill light is not later than the exposure end time of the first preset exposure. The apparatus according to claim 12, wherein said first exposure is the first exposure or the last of said at least two exposures when said number of exposures in said image acquisition period is greater than two One exposure. The device according to claim 11, wherein the first determining module is specifically configured to: Performing a demosaic process on one of the frames of the frames to generate an infrared illuminance image. The device according to any one of claims 11 to 16, wherein the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is 1. The second determining module is specifically configured to: De-infrared processing is performed on the remaining images in the frame images to obtain a visible light color image. The apparatus according to claim 17, wherein the second determining module comprises: An interpolation sub-module, configured to interpolate an IR channel of the target image, where the target image includes an IR channel, to generate the target image after the interpolation process, wherein the target image is the image of each frame The rest of the images; And an update submodule, configured to update each pixel in the target image after the interpolation process as follows to obtain a visible light color image: If the pixel has an R value, the R value of the pixel is updated: the difference between the R value of the pixel and the IR parameter value of the pixel; if the pixel has a G value, the G value of the pixel is updated: G of the pixel The difference between the value and the IR parameter value of the pixel; if the pixel has a B value, the B value of the pixel is updated: the difference between the B value of the pixel and the IR parameter value of the pixel; wherein, the IR parameter value of the pixel The product of the IR value of the pixel and the preset correction value. The device according to any one of claims 11 to 16, wherein the image to be processed is the remaining image in each frame image, and the number of remaining images in each frame image is at least 2, and The exposure durations corresponding to the remaining images in each frame image are different; The second determining module includes: a first processing sub-module, configured to perform wide dynamic synthesis processing on the remaining images in the frame images to obtain a wide dynamic image; And a second processing submodule, configured to perform de-infrared processing on the wide dynamic image to obtain a visible light color image. The device according to any one of claims 11 to 16, wherein the device is applied to an image fusion device, and the frame images are collected by the image fusion device; The optical lens of the image fusion device is provided with a filter, and the spectral region filtered by the filter includes [T1, T2]; wherein, 600 nm ≤ T1 ≤ 800 nm, 750 nm ≤ T2 ≤ 1100 nm, and T1 < T2. An electronic device, comprising a processor and a memory, Wherein, the memory is used to store the program code; The processor, when used to execute the program code stored on the memory, implements the method steps of any one of claims 1 to 10. A computer readable storage medium, wherein the computer readable storage medium stores a computer program, the computer program being executed by a processor to implement the method steps of any one of claims 1 to 10. A computer program for performing the method of any of claims 1-10 at runtime.

Images