Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/745—Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination

Definitions

• High dynamic range refers to the set of imaging processing techniques used to provide improved dynamic range between the lightest and darkest areas captured in a scene.
• One technique used to improve the dynamic range is to take multiple pictures at different exposures and then to intelligently merge the pictures to create a single photograph that has an improved range of contrast.
• a problem with this technique is the motion that occurs between the times when the pictures having different exposures are captured.
• This motion can be, for example, motion by the camera or motion of the image being captured. It can be difficult to compensate for the motion between captured images and incorrect compensation for the motion results in artifacts which decrease the image quality. In addition, it is computationally expensive to compensate for motion between the frames.
• Figure 1 shows a flow diagram for a method of producing a high dynamic range video according to an example of the invention
• Figure 2A shows six consecutive frames of a video input sequence captured at high frame rate showing substantial flicker with changing color and brightness according to an example of the invention
• Figure 2B shows the waveform associated with the fluorescent lighting that creates the variable output shown in Figure 2A according to an example of the invention
• Figure 3 shows a rectified version waveform shown in Figure 2B where the waveform is a 60 Hz AC electrical signal according to an example of the invention
• Figure 4 shows a system for producing a high dynamic range video according to an example of the invention
• Figure 5 shows a computer system for implementing the method shown in Figure 1 described in accordance with examples of the invention.
• High dynamic range image processing techniques produce images with improved contrast in the darker and lighter regions of the captured image.
• a remaining problem in HDR reconstruction is compensating for motion between frames. Problems with motion between captured exposures in HDR can be minimized by using a high frame rate capture as this can reduce the
• the ambient light produced by the fluorescent is naturally flickering at a steady frequency twice the electrical power grid frequency.
• the described method uses the "problem" of flicker that occurs at high frame rates to our advantage.
• flicker When there is flicker in ambient lights captured at high frame rates, it may provide a natural change in exposure, depending on the high frame rate capture sampling rate.
• One HDR technique improves the image dynamic range by capturing intermediate image frames at different exposures.
• We use the flicker to our advantage by selecting special high frame rate sampling rates and locking the exposure control to the flicker levels so that we can maximize light capture while ensuring diversity in the exposure levels of our HFR capture.
• FIG. 1 shows a flow diagram for a method of producing a high dynamic range video according to an example of the invention.
• the method 100 for producing a high dynamic range video comprises the steps of: determining the affect of flicker in each video frame using a color statistic value of the captured video frames (step 120); determining a flicker frequency from the affect of flicker in each video frame (step 130); given the flicker frequency, selecting a video camera exposure frequency that provides diversity in light intensities (step 140);
• step 150 synchronizing the video camera exposure control to the flicker frequency
• step 160 adapting the video camera exposure control to the amount of flicker occurring in each video frame to capture a plurality of video frames.
• the described method has the following advantages: 1) motion artifacts are minimized, 2) flicker is used as an advantage to provide diversity in affective exposure sampling, 3) captures more light.
• a step in the described method is determining the affect of flicker in each video frame using a color statistic value of the captured video frames (step 120).
• the video is captured at a high frame rate so that the video frames are captured at a first frame rate, where the first frame rate is greater than an output video frame rate.
• video is captured at a high frame rate (say 600 fps) and output at a lower rate (say 30 fps).
• the high frame rate capture (say 600 fps) by the video capture device helps to minimize motion between video frames which can deteriorate the results of the HDR.
• FIG. 2 shows six consecutive frames of a video sequence captured at a high frame rate (in this example 600 fps).
• the sequence shows substantial flicker with changing color and brightness.
• the image captured in the video camera capture sequence is a color chart lit by at least an ambient artificial light source having substantial flicker according to an example of the invention. Referring to the color chart shown in the captured videos, the color square in the right uppermost corner should be white in each image. However, the right uppermost corner square which should be pure white -shifts in color. For one example, the "white" color square changes and first gets bluer and then yellower. These color changes occur because of the flicker associated with fluorescent lighting.
• the method 100 for producing a high dynamic range video comprises the steps of: determining the affect of flicker in each video frame using a color statistic value of the captured video frames (step 120).
• color channel statistic value is used to describe a measureable color value quantity that varies as the light varies and that can be measured in the RGB color channel.
• the values can be quantities measured for a particular color channel(s) that are indicative of a color value (i.e. color, brightness, luminance) .
• the color statistic is the average color channel pixel value. In this case the average of each color channel will be calculated. The values tor each channel will be calculated and divided by the number of pixels.
• the color channel statistic value is the mean (the color statistic) of each color channel for the first frame.
• the median value may be used as the color statistic. The median value will also change as the color of the fluorescent light changes, however, the median value tends to be more less noisy.
• the sampling could be a random sampling or alternatively a sampling on a grid of pre-determined points. Further, sequential estimation methods could be used to further improve the computational speed of the described method.
• the step of determining the affect of flicker in each video frame using a color statistic value of the captured video frames is achieved in part by measuring values on the curve shown in Figure 2B which shows variations (flicker) in the light output at a high sampling rate to capture the information about the expected flicker frequency. Measurements of the current pixel values are taken for each color channel (i.e. light intensity per color channel) dependent upon the desired color statistic. Data values for each color channel are collected and once a pattern is determined, a phase lock loop or other similar method is used in combination with knowledge of the average frame intensity per frame (or other color statistic value) to predict the affect of flicker in each frame. This provides the nominal frequency of electricity.
• the described method is useful where there is an artificial light source (such as fluorescent lights) that is time variant with the frequency of electricity.
• an artificial light source such as fluorescent lights
• the light of the artificial light source output is not strongly time variant (for example, incandescent lighting)- flicker is not a significant problem, even at high frame rates.
• the method is adaptive in the sense that it can detect a certain amount of flicker and determine whether application of the described method is warranted.
• a periodic flicker detector could be used at given intervals that automatically disables the described method when flicker is not occurring.
• the described method 100 includes the step of determining a flicker frequency from the affect of flicker in each video frame (step 130).
• a method described in the application "System and Method for Minimizing Flicker", having serial number 13/460,480, filed on April 30, 2012, which is hereby incorporated by reference can be used to minimize flicker but it can also be used to determine the frequency of electricity or flicker.
• FIG. 3 shows a rectified version waveform shown in Figure 2B where the waveform is a 60 Hz AC electrical signal applied to the ambient artificial fluorescent lights.
• flicker in the ambient artificial lights when captured at high frame rates provides a natural change in exposure of the captured frames. Once you determine that flicker is occurring responsive to fluorescent lights, then you know the frequency of flicker as the fluorescent lighting varies with the waveform of the electricity.
• We use flicker to our advantage by selecting special high frame rate sampling rates and locking the exposure control to the flicker levels so that we can maximize light capture while ensuring diversity in the exposure rates of our high frame rate capture.
• a diversity in light intensities is desirable. Diversity in this context means a sampling rate that results in a plurality of samples at a plurality of intensity values. A higher number of samples and intensity values is usually preferred as this improves the amount of information available for processing. Choosing exposure frequencies that do not provide diverse light intensities will not provide the desired result. A goal of the frequency selection is to use the fact that there is flicker and variations in light intensities to select a frequency which provides additional light intensity information to improve the dynamic range.
• an HFR (high frame rate) sampling frequency of 120 fps in a 60 Hz country is a poor choice, since there will be only one exposure value whose value depends on the relative phase between the capture and the flicker. If a sampling rate of 120 Hz is chosen then referring to Figure 3, only a single exposure value having a light intensity value of zero is achieved (see points 302a, 302b, 302c, 302d). If a 120 Hz rate is selected, then the fluorescent light will have gotten darker/lighter and then darker again so - each sample will measuring intensity at the same point in the cycle again. Exposing with this light at a 120 Hz rate, will result in the same flicker level.
• the described method may maximize diversity for a given limited number of flicker cycles.
• you may maximize the sampling to maximize the number of light intensities in the limited number of flicker cycles.
• the method may optimize the sampling to maximize the number of light intensity samples within certain time periods.
• the described method includes the step of: synchronizing the video camera exposure control to the flicker frequency (step 150).
• synchronization of the camera exposure control to the flicker level is achieved using digital phase locked loop. Synchronization timing is implemented to use flicker to the methods advantage - so that the maximum of light is captured. This results in synchronization of the changing camera contrast capture so that it is in phase with the flicker, resulting in maximum light capture.
• synchronization of the camera exposure control levels corresponds to the flicker levels -the comparatively lower exposure levels when the light intensity levels are naturally lower in the ambient flicker. This way, capture is not out of phase with the flicker and we capture the most light.
• the described method includes the step of adapting the video camera exposure control to the amount of flicker occurring in each video frame to capture a plurality of video frames (step 160).
• the described method takes advantage of natural varying light intensities of the ambient artificial lighting to improve the capture of information corresponding to the bright and dark regions of the captured image.
• the information is captured at different exposure levels and merged using HDR techniques.
• Exposures should be synchronized with the flicker and also the amount of change should be dependent upon how much the flicker is naturally changing.
• Acguisition/dp/0125852630 - could be used to reconstruct the output frames.
• FIG. 4 shows a system for producing a high dynamic range video according to an example of the invention.
• the Dynamic Range Modification system 400 shown in Figure 4 is comprised of: a frequency determination component 410, an exposure frequency selection component 420, a synchronization component 430, and an adaptation component 440.
• Video from a scene illuminated by fluorescent lighting 442 is captured by a video camera 444 at a high frame rate.
• the captured video 448 is input into the dynamic range modification system 400. Because the video is captured by the high frame rate camera, the video capture has color and brightness fluctuations due to flicker similar to the video captured in Figure 2A.
• the frequency determination component 410 determines flicker frequency from the affect of flicker in each video frame in each video frame is determined using a color statistic value of a plurality of captured video frames.
• the exposure modification system 400 also includes an exposure frequency selection component 420 which given the flicker frequency, selects a video camera exposure frequency that provides diversity in light intensities.
• Figure 5 shows a computer system for implementing the method shown in Figure 1 described in accordance with examples of the present invention.
• the computing apparatus 500 includes one or more processor(s) 502 that may implement or execute some or all of the steps described in the method 300.
• the computing apparatus 500 also includes a main memory 506, such as a random access memory (RAM), where the program code for the processor 502, may be executed during runtime, and a secondary memory 508.
• the secondary memory 508 includes, for example, one or more hard drives 510 and/or a removable storage drive 512, representing a removable flash memory card, etc., where a copy of the program code for the method 100 may be stored.
• the removable storage drive 512 reads from and/or writes to a removable storage unit 514 in a well-known manner.
• any of the memory components described 506, 508, 514 may also store an operating system 530, such as Mac OS, MS Windows, Unix, or Linux; network applications 532; and a video sequence control component 534.
• the operating system 530 may be multi- participant, multiprocessing, multitasking, multithreading, real-time and the like.
• the operating system 530 may also perform basic tasks such as recognizing input from input devices, such as a keyboard or a keypad; sending output to the display 520; controlling peripheral devices, such as disk drives, printers, image capture device; and managing traffic on the one or more buses 504.
• the network applications 532 includes various components for establishing and maintaining network connections, such as software for implementing communication protocols including TCP/IP, HTTP, Ethernet, USB, and FireWire.
• the computing apparatus 500 may also include an input devices 516, such as a keyboard, a keypad, functional keys, etc., a pointing device, such as a tracking ball, cursors, mouse 518, etc., and a display(s) 520.
• a display adaptor 522 may interface with the communication bus 504 and the display 520 and may receive display data from the processor 502 and convert the display data into display commands for the display 520.
• the processor(s) 502 may communicate over a network, for instance, a cellular network, the Internet, LAN, etc., through one or more network interlaces 524 such as a Local Area Network LAN, a wireless 402.11x LAN, a 3G mobile WAN or a WiMax WAN.
• a network for instance, a cellular network, the Internet, LAN, etc.
• network interlaces 524 such as a Local Area Network LAN, a wireless 402.11x LAN, a 3G mobile WAN or a WiMax WAN.
• an interface 526 may be used to receive an image or sequence of images from imaging components 528, such as the image capture device.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Studio Devices (AREA)

Priority Applications (1)

Applications Claiming Priority (1)

Publications (1)

Family

ID=49514644

Family Applications (1)

Country Status (1)

Cited By (3)

Citations (4)

• 2012
  • 2012-04-30 WO PCT/US2012/035896 patent/WO2013165377A1/fr not_active Ceased

Patent Citations (4)

Similar Documents

Legal Events