WO2013165377A1

WO2013165377A1 - A system and method of modifying the dynamic range

Info

Publication number: WO2013165377A1
Application number: PCT/US2012/035896
Authority: WO
Inventors: Ramin Samadani
Original assignee: Hewlett Packard Development Co LP
Current assignee: Hewlett Packard Development Co LP
Priority date: 2012-04-30
Filing date: 2012-04-30
Publication date: 2013-11-07
Anticipated expiration: 2014-10-30

Description

A SYSTEM AND METHOD OF MODIFYING THE DYNAMIC RANGE

BACKGROUND

[0001] Images taken by a camera may be captured at a single exposure level that provides a limited range of contrast that results in a loss of detail at the lightest and darkest regions of the pictures. High dynamic range (HDR) refers to the set of imaging processing techniques used to provide improved dynamic range between the lightest and darkest areas captured in a scene. One technique used to improve the dynamic range is to take multiple pictures at different exposures and then to intelligently merge the pictures to create a single photograph that has an improved range of contrast.

[0002] A problem with this technique is the motion that occurs between the times when the pictures having different exposures are captured. This motion can be, for example, motion by the camera or motion of the image being captured. It can be difficult to compensate for the motion between captured images and incorrect compensation for the motion results in artifacts which decrease the image quality. In addition, it is computationally expensive to compensate for motion between the frames.

BRIEF DESCRIPTION OF DRAWINGS

[0003] The figures depict implementations/embodiments of the invention and not the invention itself. Some embodiments are described, by way of example, with respect to the following Figures.

[0004] Figure 1 shows a flow diagram for a method of producing a high dynamic range video according to an example of the invention;

[0005] Figure 2A shows six consecutive frames of a video input sequence captured at high frame rate showing substantial flicker with changing color and brightness according to an example of the invention;

[0006] Figure 2B shows the waveform associated with the fluorescent lighting that creates the variable output shown in Figure 2A according to an example of the invention;

[0007] Figure 3 shows a rectified version waveform shown in Figure 2B where the waveform is a 60 Hz AC electrical signal according to an example of the invention;

[0008] Figure 4 shows a system for producing a high dynamic range video according to an example of the invention; [0009] Figure 5 shows a computer system for implementing the method shown in Figure 1 described in accordance with examples of the invention.

[0010] The drawings referred to in this Brief Description should not be understood as being drawn to scale unless specifically noted.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0011] For simplicity and illustrative purposes, the principles of the embodiments are described by referring mainly to examples thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one of ordinary skill in the art, that the embodiments may be practiced without limitation to these specific details. Also, different embodiments may be used together. In some instances, well known methods and structures have not been described in detail so as not to unnecessarily obscure the description of the embodiments.

[0012] High dynamic range image processing techniques produce images with improved contrast in the darker and lighter regions of the captured image. However, a remaining problem in HDR reconstruction is compensating for motion between frames. Problems with motion between captured exposures in HDR can be minimized by using a high frame rate capture as this can reduce the

displacement between captured image frames. However, high frame capture can create flicker in environments where artificial ambient lights (i.e. fluorescent lights) exist. Flicker can reduce image quality by introducing unwanted variations in color and brightness that affect the image output.

[0013] If we have fluorescent light - the ambient light produced by the fluorescent is naturally flickering at a steady frequency twice the electrical power grid frequency. The described method uses the "problem" of flicker that occurs at high frame rates to our advantage. When there is flicker in ambient lights captured at high frame rates, it may provide a natural change in exposure, depending on the high frame rate capture sampling rate. One HDR technique, improves the image dynamic range by capturing intermediate image frames at different exposures. We use the flicker to our advantage by selecting special high frame rate sampling rates and locking the exposure control to the flicker levels so that we can maximize light capture while ensuring diversity in the exposure levels of our HFR capture.

[0014] Figure 1 shows a flow diagram for a method of producing a high dynamic range video according to an example of the invention. The method 100 for producing a high dynamic range video comprises the steps of: determining the affect of flicker in each video frame using a color statistic value of the captured video frames (step 120); determining a flicker frequency from the affect of flicker in each video frame (step 130); given the flicker frequency, selecting a video camera exposure frequency that provides diversity in light intensities (step 140);

synchronizing the video camera exposure control to the flicker frequency (step 150); and adapting the video camera exposure control to the amount of flicker occurring in each video frame to capture a plurality of video frames (step 160). The described method has the following advantages: 1) motion artifacts are minimized, 2) flicker is used as an advantage to provide diversity in affective exposure sampling, 3) captures more light.

[0015] Referring to Figure 1 , a step in the described method is determining the affect of flicker in each video frame using a color statistic value of the captured video frames (step 120). In one example, the video is captured at a high frame rate so that the video frames are captured at a first frame rate, where the first frame rate is greater than an output video frame rate. In one example, video is captured at a high frame rate (say 600 fps) and output at a lower rate (say 30 fps). The high frame rate capture (say 600 fps) by the video capture device helps to minimize motion between video frames which can deteriorate the results of the HDR.

[0016] One complication of capturing images at high frame rate is that flicker from ambient indoor illumination is also captured. Figure 2 shows six consecutive frames of a video sequence captured at a high frame rate (in this example 600 fps). The sequence shows substantial flicker with changing color and brightness. The image captured in the video camera capture sequence is a color chart lit by at least an ambient artificial light source having substantial flicker according to an example of the invention. Referring to the color chart shown in the captured videos, the color square in the right uppermost corner should be white in each image. However, the right uppermost corner square which should be pure white -shifts in color. For one example, the "white" color square changes and first gets bluer and then yellower. These color changes occur because of the flicker associated with fluorescent lighting.

[0017] The method 100 for producing a high dynamic range video comprises the steps of: determining the affect of flicker in each video frame using a color statistic value of the captured video frames (step 120). The term color channel statistic value is used to describe a measureable color value quantity that varies as the light varies and that can be measured in the RGB color channel. The values can be quantities measured for a particular color channel(s) that are indicative of a color value (i.e. color, brightness, luminance) .

[0018] Various types of color statistics may be used. In one example, the color statistic is the average color channel pixel value. In this case the average of each color channel will be calculated. The values tor each channel will be calculated and divided by the number of pixels. In another example, the color channel statistic value is the mean (the color statistic) of each color channel for the first frame. In another example, the median value may be used as the color statistic. The median value will also change as the color of the fluorescent light changes, however, the median value tends to be more less noisy. In another example, say if computational speed is a priority, instead of doing an average value of all of the pixels for a channel in the frame, alternatively an average value of a judicious sub-sampling of the pixels in the video frame may be used. In one example, the sampling could be a random sampling or alternatively a sampling on a grid of pre-determined points. Further, sequential estimation methods could be used to further improve the computational speed of the described method.

[0019] The step of determining the affect of flicker in each video frame using a color statistic value of the captured video frames (step 120) is achieved in part by measuring values on the curve shown in Figure 2B which shows variations (flicker) in the light output at a high sampling rate to capture the information about the expected flicker frequency. Measurements of the current pixel values are taken for each color channel (i.e. light intensity per color channel) dependent upon the desired color statistic. Data values for each color channel are collected and once a pattern is determined, a phase lock loop or other similar method is used in combination with knowledge of the average frame intensity per frame (or other color statistic value) to predict the affect of flicker in each frame. This provides the nominal frequency of electricity.

[0020] The described method is useful where there is an artificial light source (such as fluorescent lights) that is time variant with the frequency of electricity. However, for the case where there are no artificial light sources or for the case where the light of the artificial light source output is not strongly time variant (for example, incandescent lighting)- flicker is not a significant problem, even at high frame rates. Thus, it may be useful to determine whether flicker is occurring before the described method is applied. In one example, the method is adaptive in the sense that it can detect a certain amount of flicker and determine whether application of the described method is warranted. Alternatively a periodic flicker detector could be used at given intervals that automatically disables the described method when flicker is not occurring.

[0021] Referring to Figure 1, the described method 100 includes the step of determining a flicker frequency from the affect of flicker in each video frame (step 130). Although other methods may be used, in one example, a method described in the application "System and Method for Minimizing Flicker", having serial number 13/460,480, filed on April 30, 2012, which is hereby incorporated by reference can be used to minimize flicker but it can also be used to determine the frequency of electricity or flicker.

[0022] Selecting a video camera exposure frequency is very important in that different high frame rate sampling frequencies result in different numbers (diversities) of light intensities. Referring to Figure 3 shows a rectified version waveform shown in Figure 2B where the waveform is a 60 Hz AC electrical signal applied to the ambient artificial fluorescent lights. Referring to Figure 3, flicker in the ambient artificial lights when captured at high frame rates provides a natural change in exposure of the captured frames. Once you determine that flicker is occurring responsive to fluorescent lights, then you know the frequency of flicker as the fluorescent lighting varies with the waveform of the electricity. [0023] We use flicker to our advantage by selecting special high frame rate sampling rates and locking the exposure control to the flicker levels so that we can maximize light capture while ensuring diversity in the exposure rates of our high frame rate capture. A diversity in light intensities (step 140) is desirable. Diversity in this context means a sampling rate that results in a plurality of samples at a plurality of intensity values. A higher number of samples and intensity values is usually preferred as this improves the amount of information available for processing. Choosing exposure frequencies that do not provide diverse light intensities will not provide the desired result. A goal of the frequency selection is to use the fact that there is flicker and variations in light intensities to select a frequency which provides additional light intensity information to improve the dynamic range.

[0024] For example, an HFR (high frame rate) sampling frequency of 120 fps in a 60 Hz country is a poor choice, since there will be only one exposure value whose value depends on the relative phase between the capture and the flicker. If a sampling rate of 120 Hz is chosen then referring to Figure 3, only a single exposure value having a light intensity value of zero is achieved (see points 302a, 302b, 302c, 302d). If a 120 Hz rate is selected, then the fluorescent light will have gotten darker/lighter and then darker again so - each sample will measuring intensity at the same point in the cycle again. Exposing with this light at a 120 Hz rate, will result in the same flicker level. In contrast, if a sampling rate of 307 Hz is chosen - a much wider diversity of light intensities is achieved naturally through the flicker (see points 304a-304k where multiple light intensities result from this frequency rate). Thus, if the goal is a diversity of light intensities being sampled - then choosing a HFR sampling frequency of 307 Hz is more desirable then choosing a HFR sampling frequency of 120 Hz.

[0025] As previously stated, diversity of light intensities can improve the output image results. In one example, there may be a limited number of flicker cycles may be used for the HDR processing. For this case, the described method may maximize diversity for a given limited number of flicker cycles. In one case, you may maximize the sampling to maximize the number of light intensities in the limited number of flicker cycles. In another example, the method may optimize the sampling to maximize the number of light intensity samples within certain time periods.

[0026] Referring to Figure 1 , the described method includes the step of: synchronizing the video camera exposure control to the flicker frequency (step 150). In one example, synchronization of the camera exposure control to the flicker level is achieved using digital phase locked loop. Synchronization timing is implemented to use flicker to the methods advantage - so that the maximum of light is captured. This results in synchronization of the changing camera contrast capture so that it is in phase with the flicker, resulting in maximum light capture. In one example, synchronization of the camera exposure control levels corresponds to the flicker levels -the comparatively lower exposure levels when the light intensity levels are naturally lower in the ambient flicker. This way, capture is not out of phase with the flicker and we capture the most light. [0027] Referring to Figure 1 , the described method includes the step of adapting the video camera exposure control to the amount of flicker occurring in each video frame to capture a plurality of video frames (step 160). The described method takes advantage of natural varying light intensities of the ambient artificial lighting to improve the capture of information corresponding to the bright and dark regions of the captured image. The information is captured at different exposure levels and merged using HDR techniques.

[0028] In one example, we adapt the video camera to align the different exposure values with the flicker cycles. This helps improve the image quality since at high frame rates, the amount of light available decreases and frames are noisier compared to video frames captured at standard frame rates. The reason for the different exposures is to get more light in some exposures frames and less light in others. Say for example, the system does not pay attention to the flicker and took long exposures during the ambient dark period. Then the system is working against the advantages of the natural signal variations in light intensity. The goal of the method is to determine when the flicker variations are happening and

synchronize to it in order to control the exposures by combining the amount of extra exposure with amount of flicker that is sensed. In general, we want long exposures during high ambient flicker light levels (see exposure length 360) and short exposures (see exposure length 362) when there is low ambient flicker light levels. Exposures should be synchronized with the flicker and also the amount of change should be dependent upon how much the flicker is naturally changing.

[0029] After the data is captured at different light intensities (different exposures), so that HDR techniques may be used to combine the image

information for the captured frames that have different exposure levels to reconstruct a plurality of output frames. For example, the HDR techniques described in http://www.amazon.com/High-Dvnamic-Ranae-imaaing-

Acguisition/dp/0125852630 - could be used to reconstruct the output frames.

[0030] Figure 4 shows a system for producing a high dynamic range video according to an example of the invention. The Dynamic Range Modification system 400 shown in Figure 4 is comprised of: a frequency determination component 410, an exposure frequency selection component 420, a synchronization component 430, and an adaptation component 440.

[0031] Video from a scene illuminated by fluorescent lighting 442 is captured by a video camera 444 at a high frame rate. The captured video 448 is input into the dynamic range modification system 400. Because the video is captured by the high frame rate camera, the video capture has color and brightness fluctuations due to flicker similar to the video captured in Figure 2A. In one example, the frequency determination component 410. In one example, the frequency determination component determines flicker frequency from the affect of flicker in each video frame in each video frame is determined using a color statistic value of a plurality of captured video frames. The exposure modification system 400 also includes an exposure frequency selection component 420 which given the flicker frequency, selects a video camera exposure frequency that provides diversity in light intensities. After a video camera exposure is selected, the synchronization component 430 synchronizing the video camera exposure control to the flicker frequency and the adaptation component 440 adapts the video camera exposure control to the amount of flicker occurring in each video frame to capture a plurality of video frames. [0032] Figure 5 shows a computer system for implementing the method shown in Figure 1 described in accordance with examples of the present invention.

The computing apparatus 500 includes one or more processor(s) 502 that may implement or execute some or all of the steps described in the method 300.

Commands and data from the processor 502 are communicated over a

communication bus 504. The computing apparatus 500 also includes a main memory 506, such as a random access memory (RAM), where the program code for the processor 502, may be executed during runtime, and a secondary memory 508. The secondary memory 508 includes, for example, one or more hard drives 510 and/or a removable storage drive 512, representing a removable flash memory card, etc., where a copy of the program code for the method 100 may be stored. The removable storage drive 512 reads from and/or writes to a removable storage unit 514 in a well-known manner.

[0033] These methods, functions and other steps described may be embodied as machine readable instructions stored on one or more computer readable mediums, which may be non-transitory. Exemplary non-transitory computer readable storage devices that may be used to implement the present invention include but are not limited to conventional computer system RAM, ROM, EPROM, EEPROM and magnetic or optical disks or tapes. Concrete examples of the foregoing include distribution of the programs on a CD ROM or via Internet download. In a sense, the Internet itself is a computer readable medium. The same is true of computer networks in general. It is therefore to be understood that any interfacing device and/or system capable of executing the functions of the above-described examples are encompassed by the present invention. [0034] Although shown stored on main memory 506, any of the memory components described 506, 508, 514 may also store an operating system 530, such as Mac OS, MS Windows, Unix, or Linux; network applications 532; and a video sequence control component 534. The operating system 530 may be multi- participant, multiprocessing, multitasking, multithreading, real-time and the like. The operating system 530 may also perform basic tasks such as recognizing input from input devices, such as a keyboard or a keypad; sending output to the display 520; controlling peripheral devices, such as disk drives, printers, image capture device; and managing traffic on the one or more buses 504. The network applications 532 includes various components for establishing and maintaining network connections, such as software for implementing communication protocols including TCP/IP, HTTP, Ethernet, USB, and FireWire.

[0035] The computing apparatus 500 may also include an input devices 516, such as a keyboard, a keypad, functional keys, etc., a pointing device, such as a tracking ball, cursors, mouse 518, etc., and a display(s) 520. A display adaptor 522 may interface with the communication bus 504 and the display 520 and may receive display data from the processor 502 and convert the display data into display commands for the display 520.

[0036] The processor(s) 502 may communicate over a network, for instance, a cellular network, the Internet, LAN, etc., through one or more network interlaces 524 such as a Local Area Network LAN, a wireless 402.11x LAN, a 3G mobile WAN or a WiMax WAN. In addition, an interface 526 may be used to receive an image or sequence of images from imaging components 528, such as the image capture device.

[0037] The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the invention. The foregoing descriptions of specific embodiments of the present invention are presented for purposes of illustration and description. They are not intended to be exhaustive of or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations are possible in view of the above teachings. The embodiments are shown and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents:

Claims

What is Claimed is:

1.A method for modifying the dynamic range, comprising the steps of:

determining the affect of flicker in each video frame using a color statistic value of a plurality of captured video frames (120);

determining a flicker frequency from the affect of flicker in each video frame (130);

given the flicker frequency, selecting a video camera exposure frequency that provides diversity in light intensities (140);

synchronizing the video camera exposure control to the flicker frequency

(150); and

adapting the video camera exposure control to the amount of flicker occurring in each video frame to capture a plurality of video frames (160).

2. The method recited in claim 1 further including the step of capturing video frames at a first frame rate, wherein the first frame rate is greater than an output video frame rate.

3. The method recited in claim 1 further including the step of applying HDR reconstruction to the captured video frames to form a plurality of output high dynamic range video frames.

4. The method recited in claim 1 wherein the adaption of the camera exposure control is incremental to the amount of flicker that is occurring.

5. The method recited in claim 1 wherein optimizing for diversity is achieved within a limited time window.

6. The method recited in claim 1 where the color statistic is the mean of the RGB value.

7. The method recited in claim 1 wherein synchronization is achieved using a phase lock loop.

8. A non-transitory computer readable storage medium having computer readable program instructions stored thereon for causing a computer system to perform instructions, the instructions comprising the steps of:

steps of:

synchronizing the video camera exposure control to the flicker frequency (150); and adapting the video camera exposure control to the amount of flicker occurring in each video frame to capture a plurality of video frames (160).

9. The computer readable medium recited in claim 8 further including the step of capturing video frames at a first frame rate, wherein the first frame rate is greater than an output video frame rate.

10. The computer readable medium recited in claim 8 further including the step of applying HDR reconstruction to the captured video frames to form a plurality of output high dynamic range video frames.

11. The computer readable medium recited in claim 8 wherein optimizing for diversity is achieved within a limited time window.

12. The computer readable medium recited in claim 8 where the color statistic is the mean of the RGB value.

13. A dynamic range modification system comprising:

a frequency determination component 410 for determining the affect of flicker in each video frame using a color statistic value of a plurality of captured video frames and for determining a flicker frequency from the affect of flicker in each video frame;

an exposure frequency selection component 420 for selecting a video camera exposure frequency that provides diversity in light intensities; a synchronization component 430 for synchronizing the video camera exposure control to the flicker frequency; and

an adaptation component 440 for adapting the video camera exposure control to the amount of flicker occurring in each video frame to capture a plurality of video frames.

14. The system recited in claim 13 where the color statistic is the mean of the RGB value.

15. The system recited in claim 13 wherein synchronization is achieved using a phase lock loop.