WO2018178104A1 - Method and device for acquiring and restoring digital images with extended dynamics - Google Patents

Abstract

The invention concerns a method for acquiring digital images with an extended dynamic range and an associated method for restoring digital images, as well as corresponding devices. The method for acquiring digital images comprises steps of: - determining (30) a plurality of different integration times, - and for each of the determined integration times, obtaining (32) an initial digital image acquired with the integration time, each initial digital image having a corresponding signal level, - for each initial digital image, applying (34) a clipping function with an associated clipping threshold to obtain a linear signal level according to the lighting, to obtain a processed digital image, and - combining (36) processed digital images to obtain a final image with extended dynamic range, said combination comprising a summation of the signals of the processed digital images, the final image having a linear signal level by fragments according to the lighting.

Description

 Method and device for acquiring and restoring digital images with extended dynamics

 The present invention relates to a method for acquiring digital images with extended dynamics. It also relates to an associated device, a method and a device for rendering associated digital images.

 It is located in the field of image processing with wide dynamic range, known as "High Dynamic Range" or HDR in English.

 In some applications, the dynamic acquisition of conventional digital images is not sufficient for a sufficiently accurate representation of the physical data represented.

 In fact, a digital image is conventionally obtained by an image acquisition device provided with sensors composed of thousands of pixels, each pixel being able to transform photons received during an integration time into an electrical signal, said electrical signal being transformed into a digital image sample value. The numerical values in the dynamic range of representation are therefore representative of the amount of light received and integrated by the pixels. Low numerical values correspond to a small amount of photons, therefore of light, in other words to dark areas, whereas high values correspond to light areas.

 In addition to the useful signal of the image, several noises come to "pollute" this image.

When the useful signal is weak, the noise becomes significant and the useful signal can be embedded in the noise. The SNR (Signal to Noise Ratio) is the ratio of the useful signal level to the noise level.

 There are techniques for processing a plurality of digital images making it possible to obtain a digital image whose dynamics are increased, but such techniques generally require a large computational power.

 Such image processing techniques are not applicable in certain applications, for example in on-board image capture on autonomous mobile devices having a limited computing power, for example drones or vehicles designed to move on the road. surface of stars, also called "rovers" in English.

In particular, a problem arises in the case of space applications, in the context of the acquisition of images by autonomous robots, equipped with communication systems. In such applications, the onboard computational power is low, and the communication bandwidth available for transmitting data to a remotely located receiver system may also be low. It would therefore be useful to develop a method allowing the acquisition of digital images with extended dynamics, and a good signal-to-noise ratio in the range of representation and requiring only low computational means.

 For this purpose, the invention proposes a digital image acquisition method implemented in a system comprising a digital image acquisition device, said digital image acquisition device being provided with sensors, each sensor being able to transform an illumination detected during an integration time into an electrical signal, said electrical signal being transformed into a digital image sample radiometry value. The method comprises the following steps:

 -determination of a plurality of different integration times,

 and for each of the determined integration times, obtaining an initial digital image acquired with said integration time, each initial digital image having a corresponding signal level,

 for each initial digital image, application of a clipping function with an associated clipping threshold to obtain a linear signal level according to the illumination, to obtain a processed digital image, and

 combining the processed digital images to obtain an extended dynamic range final image, said combination comprising a summation of the processed digital image signals, the final image having a piecewise linear signal level as a function of illumination.

 Advantageously, the digital image acquisition method of the invention is simple and requires little computational resources, and makes it possible to obtain a final digital image of great dynamics.

 The digital image acquisition method according to the invention may have one or more of the following characteristics, taken independently or in any technically acceptable combination.

 The step of clipping is to compare, for each acquired digital image, the value of each sample with the predetermined clipping threshold value, and when said sample value is greater than the clipping threshold value, replacing said sample value with said clipping threshold value.

 For each acquired image, the same clipping threshold value is applied.

The step of determining the integration times comprises a determination of a minimum integration time and / or a maximum integration time, and the calculation of the other integration times as a function of the maximum integration time. and a number of images to process. The step of determining integration time comprises determining the integration times as a function of a quality objective of the final digital image and a number of images to be processed.

 The quality objective of the digital image is a minimum signal to noise ratio objective and the determination of the integration times implements the minimum signal-to-noise ratio, the number of images processed and a saturation level of a captor.

 According to another aspect, the invention relates to a computer program comprising software instructions which, when implemented by a programmable device, implement a method of acquiring digital images as briefly described below. above.

 According to another aspect, the invention relates to an information recording medium adapted to store a computer program comprising software instructions which, when implemented by a programmable device, implement a method of acquisition of digital images as briefly described above.

 According to another aspect, the invention relates to a digital image acquisition device implemented in a system comprising a digital image acquisition device, said digital image acquisition device being provided with sensors, each sensor being adapted to transform a detected illumination during an integration time into an electrical signal, said electrical signal being converted into a digital image sample radiometry value. The digital image acquisition device comprises a programmable device comprising at least one processor adapted to implement calculations, comprising:

 a module for determining a plurality of different integration times,

a control module for acquiring a corresponding initial digital image for each of the determined integration times, each initial digital image having a corresponding signal level,

 for each digital image acquired, an application module of a clipping function with an associated clipping threshold to obtain a linear signal level according to the illumination, in order to obtain a processed digital image,

a combination module of the processed digital images to obtain an extended dynamic range final image, said combination comprising a summation of the processed digital image signals, the final image having a piecewise linear signal level according to the illumination. In another aspect, the invention relates to a method for rendering digital images from a final digital image of extended dynamic range obtained by an acquisition method as briefly described above. The restitution process comprises steps of:

 obtaining a plurality of different integration times and clipping thresholds used to obtain the final digital image of extended dynamic, corresponding to a number N of initial digital images;

 -calculating N final thresholds from integration times and clipping thresholds;

obtaining at least one rendered image having a linear signal level with the illumination from the final digital image and the final calculated thresholds.

 The digital image rendering method according to the invention may have one or more of the following characteristics, taken independently or in any technically acceptable combination.

 The step of obtaining at least one rendered image comprises calculating a single image having a linear signal level.

 The step of obtaining at least one rendered image comprises the calculation of N restored images corresponding to the N initial digital images.

 According to another aspect, the invention relates to a computer program comprising software instructions which, when implemented by a programmable device, implement a method of rendering digital images as briefly described above. .

 According to another aspect, the invention relates to an information recording medium adapted to store a computer program comprising software instructions which, when implemented by a programmable device, implement a restitution process. digital images as briefly described above.

 According to another aspect, the invention relates to a device for restoring digital images from a final digital image of extended dynamic range obtained by acquisition device as briefly described above. The device comprises a programmable device comprising at least one processor adapted to implement modules of:

 obtaining a plurality of different integration times and clipping thresholds used to obtain the final digital image of extended dynamic, corresponding to a number N of initial digital images;

-calculating N final thresholds from integration times and clipping thresholds; obtaining at least one rendered image having a linear signal level with the illumination from the final digital image and the final calculated thresholds.

Other features and advantages of the invention will emerge from the description given below, by way of indication and in no way limiting, with reference to the appended figures, among which:

 FIG 1 is a diagram showing the functional blocks of a programmable device capable of implementing the invention;

 FIG. 2 is a block diagram of the main steps of an extended dynamic image acquisition method according to one embodiment of the invention;

 FIG. 3 illustrates the signal levels corresponding to images acquired as a function of the quantity of light integrated by the sensors of the image acquisition device, with different integration times;

 FIG. 4 schematically illustrates the signal level of the final image;

 FIG. 5 is a block diagram of the main steps of restoring image signals acquired from a digital image with an extended dynamic according to one embodiment of the invention.

FIG. 1 diagrammatically illustrates a digital image acquisition / reproduction system 1 capable of implementing the digital image acquisition method with an extended dynamic according to the invention.

 The system 1 comprises an image acquisition system 2, which can for example be embedded on board an autonomous mobile platform, for example a robot.

 The image acquisition system 2 comprises an image acquisition device 4, provided with sensors, each sensor being able to transform photons received during an integration time into an electrical signal, said electrical signal being transformed into a digital value of digital image sample.

 Preferably, digital output sensors of the CMOS type are used.

 The image acquisition device 4 is connected to a programmable device 6, able to implement the image acquisition method according to the invention, and in particular to control the acquisition of images by the device. acquiring images 4.

In one embodiment, the programmable device 6 is implemented in the form of programmable logic components, such as an FPGA (English Field-Programmable Gate Array), or in the form of dedicated integrated circuits, ASIC type (English Application-Specific Integrated Circuit). Programmable device 6 is connected to a memory type storage device for storing digital images.

 The image acquisition system 2 optionally comprises a communication module 8 able to communicate remotely with a rendering system 10, also comprising a communication module 12, according to a dedicated communication protocol. For example, a satellite communication is implemented.

 Thus, the image acquisition system 2 is capable of transmitting digital data, in particular digital images, to the rendering system 10.

 In one embodiment, the rendering system 10 is a programmable device capable of implementing the invention, for example a computer. It comprises a central processing unit 16, or CPU, able to execute computer program instructions when the programmable device is powered up. The programmable device also comprises information storage means 16, for example registers or memories, capable of storing executable code instructions enabling the implementation of programs comprising code instructions able to implement the method according to FIG. the invention.

 Optionally, the programmable device 10 comprises a screen 18 and a means 20 for inputting commands from an operator, for example a keyboard, optionally an additional pointing means 22, such as a mouse, for selecting graphic elements displayed on the screen. screen 18.

 The various functional blocks 12 to 22 of the programmable device 10 described above are connected via a communication bus 24.

 Figure 2 is a block diagram of the main steps of a digital image acquisition method with extended dynamics according to one embodiment of the invention.

 In a first step 30, a number N of digital image acquisitions IM, to be performed by the image acquisition device 2, and an integration time T, associated with each image acquisition, are determined. IM ,.

 The number N is an integer strictly greater than 1.

 In one embodiment, the number N is predetermined, for example provided by an operator and stored, as well as the associated integration times.

 In a variant, N is determined according to the storage capacities in the image acquisition system 2.

For example, N = 4 image acquisitions made with four integration times T ₁ , T ₂ , T ₃ and T ₄ . In one embodiment, T ₁ = ms, T ₂ = ₂ ms, T ₃ = ₃ ms, T ₄ = ₄ ms. In one embodiment, the integration time T ₄ is the longest, and the other integration times are fixed by a calculation rule with respect to T ₄ , for example:

 Of course, this embodiment is given as an example.

 Other calculation rules can be applied depending on the longest integration time and / or the shortest integration time.

 For example, a constant time increment is applied to the shortest integration time.

 According to a variant, each integration time T 1 is determined separately.

 According to another variant, the number N and the integration times T are calculated as a function of environmental capture conditions or of a fixed quality objective, for example in terms of the minimum signal-to-noise ratio to be obtained.

Preferably, the integration times are chosen so that the amount of motion in the scene observed between the shortest integration time T ₁ and the longest integration time T _N is negligible.

 Such is the case, for example, if the image capture device is fixed and if the brightness of the observed scene is stable throughout the duration of the images, that the image acquisitions are performed in series by a single sensor. or in parallel by several sensors.

 Step 30 is followed by a step 32 of acquisition of each digital image IM, by adjusting the integration time of the sensors of the image acquisition device to the integration time T, previously determined.

 A digital image is formed of a two-dimensional array of image samples, each image sample having an associated digital value called a signal (corresponding to the sum of the wanted signal and noise). The useful signal corresponds to a radiometry value. In a dynamic range of representation, the radiometry value is representative of the illumination, or in other words of the quantity of light, received and integrated by the sensor or sensors associated with said sample.

 The low radiometric values correspond to the dark areas, and the high radiometric values to the illuminated areas.

For example, each digital sample image value is represented on P bits, P being for example between 8 and 10, the corresponding dynamic representation range being the range of integer values between 0 and 2 ^P -1. In a variant, a digital image is composed of several such three-dimensional matrices of image samples, each matrix corresponding to a color, for example red, green, blue.

The shortest integration time, T ₁ , makes it possible to acquire a digital image having a good representation of the light areas.

The longest integration time T _N makes it possible to acquire a digital image having a good representation of the dark areas.

 For each initial digital image IM ,, step 32 is followed by a clipping step 34, making it possible to use the corresponding sensor in its linear zone. The linear zone of the sensor is the area in which the useful signal level is proportional to the number of photons.

 The step 34 implements a clipping function applied to each initial image IM ,, and consisting of replacing all the values of the signal of IM, greater than or equal to a clipping threshold Seuil_i by the value Seuil_i

 Yes

If not,

 for 1 <k≤L, 1≤l≤C, L, C being the respective dimensions, in number of rows and number of columns, of the image matrix IM ,.

 At the end of step 34, for each index i, a digital image IM ', called the processed image, is obtained.

 In one embodiment, the clipping threshold is constant and equal to a predetermined Sat threshold value for all acquired digital images.

 For example, the threshold value Sat will be the value up to which the signal is linear as a function of the quantity of photons received by the pixel.

 Preferably, this value Sat is close to the saturation of the sensor, so as to use almost the entire linear zone of the sensor.

 Advantageously, this embodiment comprises simple computational operations, easily achievable by an FPGA. In practice, only the values of IMj (k, l) greater than the clipping threshold Seuil_i are modified.

FIG. 3 schematically illustrates the signal level of each image after the clipping step 34 as a function of the amount of light integrated by the sensor (s) of the image acquisition device, for a set of four times of integration T ₁ , T ₂ , T ₃ , T ₄ with T ₁ <T ₂ <T ₃ <T ₄ .

In FIG. 3 4 graphs have been illustrated, each having on the abscissa the illumination (denoted "light"), and on the ordinate the signal level, which is a numerical value. The signal level S _i corresponds to the clipped image IM ' ₁ , the signal level S ₂ corresponds to the clipped image IM' ₂ and so on. The integration time corresponding to each image acquisition, noted T _int , is also specified for each graph.

As illustrated in FIG. 3, the longer the integration time, the lower the illumination value necessary to saturate the pixel. The saturation steps L _{sa_i} are inversely proportional to the integration times T _{N-i + 1} .

The signal levels S, are linear between 0 and L _sa y, then constant, thanks to the selection of the threshold threshold values Threshold.

Returning to FIG. 2, following the step 34 of clipping, a step 36 of summation of the clipped digital images IM '' is applied, and a final image of extended representation range, denoted IM _F , is obtained:

When each image IM 'is encoded on P bits, the final image IM _F of extended representation range is coded on P' = P + log ₂ (N) bits.

For example, if N = 4, the signal values of IM _F are represented on P + 2 bits.

Thus, for example, values represented on 10 bits are passed to values represented on 12 bits.

The final digital image thus obtained IM _F is then stored or transmitted to a rendering device during a storage or transmission step 38.

In one embodiment, the final digital image IM _F is transmitted by wireless communication means, for example by radio communications or satellite communications, to a remote receiving device, which receives and stores or processes the image. final digital extended dynamic range IM _F.

FIG. 4 illustrates the signal level Σ of the final image IM _F thus obtained by summation, as a function of the quantity of integrated light.

 The schematic representation of FIG. 4 comprises a graph showing, on the abscissa, the amount of integrated light (denoted "light"), and the ordinate the signal level, which is a numerical value.

The signal level Σ is the sum of the respective signal levels of the clipped images:

The signal level Σ is piecewise linear, depending on the saturation levels L _{sa_i} corresponding. A final threshold value corresponds to each saturation stage L _{sa_i} Advantageously, each final threshold value can be calculated as a function of the clipping threshold values Seuil_i as follows:

 And then :

 The formulas are simplified in the particular cases where all the threshold thresholds ThresholdJ are equal to a predetermined value Sat:

 The final thresholds Σ ,. are used either to render each of the N initial images, or to create a single image of extended dynamic.

The signal-to-noise ratio of the final digital image IM _F , noted here SNR (IM _F ), can be calculated theoretically as a function of the illumination received. This makes it possible to determine integration times T, making it possible to ensure a quality objective for the final image, for example to ensure that the SNR (IM _F ) of the final digital image is greater than a signal-to-image ratio. minimum noise, SNR _min , for a final digital image obtained from N initial images.

The SNR of the final image IM _F is calculated as a function of the N digital images processed:

The signal of the processed digital images can be expressed as a function of the response of the sensor R _sensor , the illumination received by the pixel of the sensor and the

En considérant que le bruit majoritaire des capteurs actuels est le bruit photonique, (le bruit photonique est égal à la racine carrée du signal en électrons), au premier ordre, on peut donc écrire le SNR comme suit : integration time T _i [s]:

Considering that the majority noise of the current sensors is the photonic noise, (the photonic noise is equal to the square root of the signal in electrons), in the first order, we can write the SNR as follows:

We can deduce :

To ensure a minimum signal-to-noise ratio SNR (IM _F ), according to one embodiment, the shortest integration time, denoted TV, is set.

Considering the saturation level in electrons, note Sat ^e- , and the desired minimum SNR level, denoted SNR _min , we obtain the successive clipping times by the following equations:

For example, Sat ^e- = 10000 electrons. The value of SNRmin is set according to the intended application.

 FIG. 5 is a block diagram of the main restitution steps, in one embodiment, from a final image of extended dynamic range obtained by the method described above.

The rendering method comprises a first step 40 for receiving or reading in memory the final digital image IM _F.

 Step 40 is followed by a step 42 for obtaining the parameters used: the number N of acquisitions, the associated integration times, the clipping thresholds Threshold_i used.

Step 42 is followed by a step 44 of determining the final thresholds corresponding to each saturation step L _{sa_i} of the signal level, using the calculation formulas given by (Eq 5) to (Eq 8).

 Step 44 is followed by a step 46 of obtaining a linearized final image.

In one embodiment, darkness radiometry levels, which are specific to the sensors used and which depend on the integration time, are also considered. We denote V _{obs_i} the level of darkness corresponding to the acquisition of the image IM, with the integration time Ti.

The IM signal _F (k, l) of the pixels of the final image is piecewise linear with the illumination, or, in other words, the IM signal _F (k, 1) can be expressed as the affine function of the amount of illumination. To make the interpretation of images easier, it may be interesting to linearize the final image. We note the signal of the

pixel (k, l) linearized.

For example, for 4 integration times T ₁ <T ₂ <T ₃ <T ₄ with clipping thresholds equal to a value Sat, the linearized signal is calculated as follows.

 In a simplified embodiment of step 46, the darkness levels are neglected.

 In this case, the previous relationships are simplified:

In FIG. 4, the signal level corresponding to FIG.

the final image of linearized extended dynamic range.

 Alternatively or additionally, step 44 is followed by step 48 of restoring the N acquired images, in their initial dynamic range, from the signal

des pixels de l'image finale.

pixels of the final image.

 In the embodiment without taking into account the darkness levels, the following formulas are applied to determine the signal value of each pixel of each of the final rendered images.

 As a variant, it is of course possible to take into account the darkness levels, in a manner analogous to that explained above with reference to step 46.

 Of course, the formulas have been detailed in the embodiment with N = 4 images acquired, but the principle applies analogously to an N number of images acquired, with any N.

 In addition, on the same principle, it is possible to determine the formulas to be applied for rendering a linearized final image or for restoring N final images corresponding to the images acquired when using different threshold values Threshold J.

Claims

1. - Digital image acquisition method implemented in a system comprising a digital image acquisition device, said digital image acquisition device being provided with sensors, each sensor being able to transform a detected illumination during an integration time in an electrical signal, said electrical signal being converted into a digital image sample radiometry value, characterized in that it comprises the following steps: -determining (30) a plurality of different integration times (T _1; T ₂ , T ₃ , T ₄ ), and for each of the integration times (T _1, T ₂ , T ₃ , T ₄ ) determined, obtaining (32) an initial digital image acquired with said integration time, each initial digital image having a level of corresponding signal, for each initial digital image, application (34) of a clipping function with an associated clipping threshold (Threshold_ 1, Threshold_2, Threshold_3, Threshold_4) to obtain a signal level (S _1; S ₂ , S ₃ , S ₄ ) linearly according to the illumination, to obtain a processed digital image, and  -combining (36) the processed digital images to obtain an extended dynamic range final image, said combination including a summation of the processed digital image signals, the final image having a piecewise linear signal level (Σ) as a function of illuminance. 2. The method of acquiring digital images as claimed in claim 1, characterized in that the clipping step (34) consists in comparing, for each digital image acquired, the value of each sample at the threshold value. of clipping (Threshold_1, Threshold_2, Threshold_3, Threshold_4), and when said sample value is greater than the clipping threshold value (Threshold_1, Threshold_2, Threshold_3, Threshold_4), to replace said sample value by said clipping threshold value. 3. A digital image acquisition method according to claim 2, characterized in that for each of the acquired images, the same clipping threshold value is applied. 4. A digital image acquisition method according to one of claims 1 to 3, characterized in that the step of determining (30) the integration times comprises a determination of a minimum integration time and / or an integration time maximum, and the calculation of other integration times as a function of the maximum integration time and a number of images to be processed. 5. - digital image acquisition method according to one of claims 1 to 3, characterized in that the step of determining integration time comprises the determination of the integration times according to a goal of quality of the final digital image and a number of images to be processed. 6. Digital image acquisition device implemented in a system comprising a digital image acquisition device, said digital image acquisition device being provided with sensors, each sensor being capable of transforming an illumination. detected during an integration time in an electrical signal, said electrical signal being converted into a digital image sample radiometry value, characterized in that it comprises a programmable device comprising at least one processor adapted to implement calculations, including:  a module for determining a plurality of different integration times, a control module for acquiring a corresponding initial digital image for each of the determined integration times, each initial digital image having a corresponding signal level; ,  for each digital image acquired, an application module of a clipping function with an associated clipping threshold to obtain a linear signal level according to the illumination, in order to obtain a processed digital image,  a combination module of the processed digital images to obtain an extended dynamic range final image, said combination comprising a summation of the processed digital image signals, the final image having a piecewise linear signal level according to the illumination. 7. - Method for restoring digital images from a final digital image of extended dynamic range obtained by an acquisition method according to one of claims 1 to 5, characterized in that it comprises steps of :  obtaining (42) a plurality of different integration times and clipping thresholds used to obtain the final digital image of extended dynamic, corresponding to a number N of initial digital images; calculating (44) N final thresholds (Σ ₁ , ..., Σ ₄ ) from the integration times and the clipping thresholds; obtaining (46, 48) at least one rendered image having a linear signal level with the illumination from the final digital image and the final calculated thresholds. 8. The method of restoring digital images as claimed in claim 7, characterized in that the step of obtaining at least one rendered image (46) comprises computing a single image of extended dynamic range having a level linear signal. 9. - Method of restoring digital images according to claim 7, characterized in that the step of obtaining at least one restored image (48) comprises the calculation of N restored images corresponding to N initial digital images. 10. - Device for restoring digital images from a final digital image of extended dynamic range obtained by acquisition device according to claim 6, characterized in that it comprises a programmable device comprising at least one adapted processor. to implement modules of:  obtaining a plurality of different integration times and clipping thresholds used to obtain the final digital image of extended dynamic, corresponding to a number N of initial digital images;  -calculating N final thresholds from integration times and clipping thresholds; obtaining at least one rendered image having a linear signal level with the illumination from the final digital image and the final calculated thresholds.