KR102396062B1 - Apparatus for Acquiring Multiple Exposure Images Using Optical Shutter - Google Patents

Abstract

The present invention provides a multiple exposure image acquisition device, comprising an optical shutter for controlling the amount of transmitted light, an image sensor for acquiring an image by accumulating charges according to the amount of light transmitted through the optical shutter, and controlling the image acquisition of the image sensor an electronic shutter, and an optical shutter control unit for controlling the optical shutter so that the image sensor acquires a plurality of consecutive images having exposures that vary according to the amount of transmitted light, but by synthesizing the plurality of consecutive images to achieve high dynamics A range of images can be created.

Description

Apparatus for Acquiring Multiple Exposure Images Using Optical Shutter

The present invention relates to a technique for acquiring a multiple exposure image from a camera and acquiring an image of a high dynamic range using the acquired multiple exposure image.

Real-time video recording devices or video recording devices for surveillance, such as video recording devices of conventional surveillance camera systems, video recording devices of automobiles, video recording devices of autonomous vehicles, and video recording devices of autonomous drones, mainly due to single exposure video recording, It is not possible to actively cope with various weather conditions, illuminance, backlight, and changes in the direction of lighting, so there are many cases in which it is difficult to accurately identify objects.

For example, a surveillance system using CCTV is used for a wide range of purposes, such as for the purpose of preventing crime or safety accidents by identifying the surrounding conditions through weather, traffic conditions, building, vehicle, and face identification, and for managing various facilities. there is.

In particular, in the case of uneven lighting at night, strong external light, or a situation where strong outdoor light and shade areas exist at the same time, when the camera exposure of the CCTV system is adjusted to one location, object identification in areas with different light conditions, such as lighting becomes difficult For example, when the headlights of a vehicle illuminate at night, when a person's face appears dark due to backlight, when the light is low, such as in an underground parking lot, or when the shadow of a building on a bright day causes the outline, color, face, or character of objects to appear Problems that are difficult to recognize often occur.

In order to solve problems such as problems occurring in the CCTV system, a method of extending the dynamic range of the image to be acquired is used.

The dynamic range of the luminance input of a commonly used camera image sensor is around 60dB, but the dynamic range of human vision for the actual scene reaches 140dB. Therefore, within the limited dynamic range of the camera image sensor, the shutter speed, aperture value, ISO sensitivity, etc. are adjusted in order to bring the identification object into the processable dynamic range.

The image sensor internally adjusts auto exposure to find the optimal exposure value for the average luminance situation, but for very bright or very dark areas outside of the allowable exposure band, the image sensor may not produce a saturated output or low signal-to-noise ratio. send the output Once the data of the overexposed or underexposed area deviating from the standard exposure does not have original brightness information, it cannot be restored by increasing the resolution of the sensor or correcting the image quality.

Therefore, in order to realize a high-performance camera system that expresses luminance information in a wide area, a WDR display is used for visually perceived expression together with a high-sensitivity WDR (Wide Dynamic Range) image sensor, or the current WDR image sensor and high-performance image correction technology Although relatively good multi-exposure image synthesis results are obtained by using the equipment, the problem of high component cost follows.

On the other hand, there are two representative methods for capturing multiple exposure images that can be processed in real time. The first method is to shoot multiple exposure images using a dual capture image sensor made by spatially arranging two exposure pixels on one image sensor. The dual capture image sensor supports high-resolution video due to its low pixel density. The dynamic range is relatively narrow compared to the WDR image sensor. Another method is to receive light incident from a subject with one lens, split the beam, and then photograph the split light with two image sensors in multiple exposures. However, unlike a general camera, it uses two image sensors, as well as requires the use of expensive additional parts such as a beam splitter and ND filter, and there are also problems with follow-up management and restrictions on miniaturization.

Japanese Patent Laid-Open No. 2005-065077

Accordingly, an object of the present invention is to solve the above problems, and to provide an apparatus capable of easily acquiring a multiple exposure image using a single low dynamic range image sensor.

Another object of the present invention is to provide an apparatus capable of acquiring a multiple exposure image using an image sensor of a low dynamic range and generating a high dynamic range image by using the image sensor.

Another object of the present invention is to provide an apparatus capable of generating a high dynamic range image in real time using a multiple exposure image acquired through an image sensor of a low dynamic range.

Another object of the present invention is to provide an apparatus capable of acquiring a multi-exposure image capable of easily generating a high dynamic range image by simplifying an image processing process.

The apparatus for acquiring multiple exposure images according to an embodiment of the present invention includes an optical shutter that adjusts the amount of transmitted light, an image sensor that acquires an image by accumulating charges according to the amount of light transmitted through the optical shutter, and image acquisition of the image sensor and an optical shutter control unit for controlling the optical shutter so that the image sensor acquires a plurality of consecutive images having exposures that vary according to the amount of transmitted light; Thus, it is possible to generate an image with a high dynamic range.

According to an embodiment, the plurality of consecutive images may be acquired at substantially the same aperture value, the same image sensor sensitivity, and the same electronic shutter speed.

According to an embodiment, the plurality of consecutive images may be images acquired in a state in which at least one of a light transmittance and a light transmittance time of the optical shutter is changed.

In some embodiments, the optical shutter control unit may adjust the light transmittance by varying the level of the voltage applied to the optical shutter, and may adjust the light transmission time by varying the pulse width of the voltage applied to the optical shutter. .

According to an embodiment, the plurality of consecutive images may be continuously acquired in synchronization with an image frame rate of the image sensor, and an image of which exposure is changed at an operation timing of the electronic shutter may be acquired.

According to an embodiment, the apparatus for acquiring a multiple exposure image may further include an electronic shutter controller configured to control an opening/closing time and an opening/closing time of the electronic shutter.

According to an embodiment, the plurality of consecutive images are continuously acquired in synchronization with the image frame rate of the image sensor, and the electronic shutter is read by the image sensor while the light transmittance is changed in synchronization with the frame rate. It can be obtained by working before and after the reading time of every other order of the betting times.

According to an embodiment, the interval between the electronic shutter operation timing before and after the read-out time may be shorter than the interval between the electronic shutter operation timing after the read-out time and the next electronic shutter operation timing.

According to an embodiment, the plurality of consecutive images are continuously acquired in synchronization with an image frame rate of the image sensor, and in a state in which the electronic shutter is fully opened, the optical shutter controller is It can be obtained by applying voltages of different pulse widths to the optical shutters, respectively, before and after the read-out time of each order.

According to an embodiment, the plurality of consecutive images are continuously acquired in synchronization with an image frame rate of the image sensor, and in a state in which the electronic shutter is fully opened, the optical shutter controller is It can be obtained by applying different pulse widths and different levels of voltages to the optical shutters, respectively, before and after the bi-sequential readout time.

According to an embodiment, the plurality of consecutive images are continuously acquired in synchronization with an image frame rate of the image sensor, and in a state in which the electronic shutter is fully opened, the optical shutter controller is The same pulse width and voltages of different levels may be respectively applied to the optical shutters before and after the bi-sequential readout time.

According to an embodiment, the plurality of consecutive images are continuously acquired in synchronization with an image frame rate of the image sensor, and in a state in which the electronic shutter is fully opened, the optical shutter controller is The graph of the light transmittance of the optical shutter is obtained by applying a voltage pulse of an asymmetric gradient function type to the optical shutter so that the graph of the light transmittance of the optical shutter has a rising shape and a falling shape, respectively, before and after the reading time of every other sequence. The absolute value of the slope of the transmittance graph may be greater than the absolute value of the slope of the falling light transmittance graph.

According to an embodiment, the optical shutter may include an electro-optic element whose light transmittance or the light transmittance time is adjusted according to the strength or duration of the electric field.

According to an embodiment, the multi-exposure image acquisition apparatus may generate an image having a high dynamic range by synthesizing the plurality of consecutive images.

According to an embodiment, an image recording apparatus of a vehicle using the multiple exposure image acquisition device, an image photographing apparatus for monitoring of a vehicle, an image photographing apparatus for a side mirror of a vehicle, an image photographing apparatus of a drone, an image photographing apparatus of a surveillance camera system, Any one of the image capturing apparatus of the robot may be applied to an apparatus for generating an image of a high dynamic range.

The device according to an embodiment of the present invention has the effect of acquiring multiple exposure images at low cost using one image sensor without using a plurality of image sensors or a wide-band image sensor.

In addition, when the apparatus according to an embodiment of the present invention acquires a multi-exposure image using one image sensor, a ghost effect during image synthesis occurs because the shooting timing of the high-exposure image and the low-exposure image is different. has the effect of reducing

In addition, when acquiring a multiple exposure image using a single image sensor, the device according to an embodiment of the present invention provides multiple exposure systems with substantially the same image exposure meter (shutter time, aperture value, ISO sensitivity) of the camera. Since an image of exposure can be acquired, there is an effect of reducing a blurring effect that occurs when synthesizing a high dynamic range image.

In addition, the apparatus according to an embodiment of the present invention easily acquires a multiple-exposure image using one low dynamic range image sensor, and uses the acquired multiple-exposure image to achieve a high dynamic range in real time. There is an effect that can create an image (HDR image) of

1 is a block diagram illustrating a multi-exposure image acquisition apparatus for generating an image of a high dynamic range according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a control signal generation timing of each component for acquiring a multiple exposure image in the multiple exposure image acquisition apparatus shown in FIG. 1 .
3 is a block diagram of an apparatus for acquiring a multiple exposure image according to another embodiment of the present invention, and shows the configuration of an apparatus for acquiring a multiple exposure image by synchronizing an optical shutter with a vertical synchronization signal of an image sensor.
FIG. 4 is a diagram illustrating a control signal generation timing of each component for acquiring a multiple exposure image in the multiple exposure image acquisition apparatus shown in FIG. 3 .
FIG. 5 is a diagram illustrating control signal generation timings of each component for acquiring a multiple exposure image by controlling the opening and closing timings of the optical shutter and the operation timing of the electronic shutter in the multiple exposure image acquisition device shown in FIG. 1 .
FIG. 6 is a diagram illustrating control signal generation timing of each component for acquiring a multiple exposure image by adjusting the light transmission time of an optical shutter under a full-time electronic shutter condition in the multiple exposure image acquisition device shown in FIG. 1 .
7 is a control signal generation timing of each component for acquiring a multiple exposure image by adjusting the light transmittance and light transmission time of the optical shutter under the full-time electronic shutter condition in the multiple exposure image acquisition device shown in FIG. It is also
8 is a control signal generation timing of each component for acquiring a multiple exposure image by adjusting the light transmittance and light transmission time of the optical shutter under the full-time electronic shutter condition in the multiple exposure image acquisition device shown in FIG. it's another way
FIG. 9 is an operation timing diagram for acquiring a multiple exposure image using a voltage pulse in the form of an asymmetric gradient function in the multiple exposure image acquisition apparatus shown in FIG. 1 .
10 is a diagram illustrating an example of generating an image of a high dynamic range by synthesizing a plurality of multiple exposure images acquired using the multiple exposure image acquisition apparatus according to an embodiment of the present invention.
11 is a block diagram of a multi-exposure image acquisition apparatus for generating images of a high dynamic range by acquiring images of three different exposures according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one element from another, for example, without departing from the scope of the inventive concept, a first element may be termed a second element and similarly a second element A component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Other expressions describing the relationship between components, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described herein exists, but one or more other features It should be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, are common to those of ordinary skill in the art to which the technology disclosed herein belongs, unless specifically defined otherwise herein. has the same meaning as understood as Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. shouldn't

The suffixes "module" and "part" for components used in this specification are given or mixed in consideration of only the ease of writing the specification, and do not have a meaning or role distinct from each other by themselves. It may refer to a functional or structural combination of hardware for performing a method according to an embodiment of the present invention or software capable of driving the hardware.

Hereinafter, a preferred embodiment of the present invention, that is, a technique for acquiring a multiple exposure image using an optical shutter will be described in detail with reference to the accompanying drawings.

The technique disclosed herein is applied to a technique for acquiring a multiple exposure image using an optical shutter and a technique for generating a high dynamic range image using the acquired multiple exposure image. However, the technology disclosed herein is not limited thereto, and may be applied to an apparatus and method for acquiring all multiple exposure images to which the technical spirit of the technology can be applied, and an apparatus and method for generating a high dynamic range image.

1 is a block diagram illustrating a multi-exposure image acquisition apparatus for generating an image of a high dynamic range according to an embodiment of the present invention.

As shown in FIG. 1 , the apparatus for acquiring multiple exposure images according to an embodiment of the present invention includes an optical shutter 100 , an image sensor 110 , an electronic shutter control unit 120 , an optical shutter control unit 130 , and an HDR image. The generator 140 may be included.

Hereinafter, the function of each component will be described in detail with reference to FIG. 1 .

The optical shutter 100 may adjust the transmission amount of light incident to the image sensor 110 .

The image sensor 110 may acquire an image by accumulating charges according to the amount of light transmitted through the optical shutter 100 .

The electronic shutter control unit 120 may control an opening/closing time and an opening/closing time (opening/closing timing) of the electronic shutter of the image sensor 110 .

The optical shutter control unit 130 may control opening and closing of the optical shutter 100 to adjust the amount of light transmitted through the optical shutter 100 . Accordingly, the image sensor 110 may successively acquire and output a plurality of consecutive images having different exposure degrees depending on the amount of transmitted light, that is, a high-exposure image and a low-exposure image.

Here, the control of opening and closing of the optical shutter 100 means adjusting the light transmission amount of the optical shutter 100 , and the optical shutter control unit 130 controls the level of the voltage input to the optical shutter 100 . and by controlling the voltage pulse width, that is, the duration of the voltage pulse, the amount of light transmission may be adjusted. Accordingly, the image sensor 110 may acquire a plurality of consecutive images having different degrees of exposure according to the amount of light incident through the optical shutter 100 .

Meanwhile, the optical shutter control unit 130 may include a voltage control unit (not shown) that adjusts the level of the voltage input to the optical shutter 100 .

The HDR image generator 140 may generate an image having a high dynamic range by synthesizing a plurality of consecutive images having different exposure degrees.

Here, the plurality of consecutive images having different exposure degrees have substantially the same aperture value, the same image sensor sensitivity (ISO), and the image sensor 110 except for the amount of light transmission adjusted by the optical shutter 100 . It may have been taken continuously under the same electronic shutter speed conditions for

That is, according to the multi-exposure image acquisition apparatus according to the embodiment of the present invention, when continuously shooting images, the image exposure meter (shutter time, aperture value, ISO sensitivity) of the camera is fixed and only the amount of light through the optical shutter 100 is fixed. Because it is adjusted, it is possible to continuously shoot images that differ only in the brightness and saturation region, but with the same sharpness and blurring of the boundary of objects. Accordingly, when synthesizing an image of a high dynamic range using continuously captured images having different exposure degrees, an error problem (synthesis boundary distortion) due to a difference in boundary information can be solved.

The electronic shutter control unit 120 uses the opening/closing timing signal for the optical shutter 100 generated by the optical shutter control unit 130 to open/close time and open/close time (open/close timing) for the electronic shutter of the image sensor 110 . ), it is possible to control the image acquisition timing of the image sensor 110 .

The amount of light transmission may vary depending on at least one of a light transmittance and a light transmission time of the optical shutter 100 . Accordingly, a plurality of consecutive images having different exposure degrees may be acquired in a state in which the light transmission amount is changed.

Here, the optical shutter control unit 130 adjusts the light transmittance of the optical shutter 100 by varying the level of the voltage applied to the optical shutter 100 , and pulses the voltage applied to the optical shutter 100 . The light transmission time of the optical shutter 100 may be adjusted by varying the width, that is, the voltage pulse application time.

Although a liquid crystal glass lens is employed as the optical shutter 100 in the present embodiment, the optical shutter 100 has a light transmittance or light transmission time adjusted according to the strength of the electric field or the duration of the electric field. Any possible electro-optical device may be employed.

Hereinafter, a method of generating a high dynamic range image (HDR image) using the multiple exposure image acquisition apparatus according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 2 .

FIG. 2 is a diagram illustrating a control signal generation timing of each component for acquiring a multiple-exposure image in the multiple-exposure image acquisition apparatus shown in FIG. 1 .

The specification of the main hardware used in the manufacture of the embodiment of the multiple exposure image acquisition apparatus for acquiring the multiple exposure image shown in FIG. 2 and the method of setting the hardware will be described as follows.

Specifications of the camera module including the image sensor 110 are as follows.

- Model Name: Edmund Optics EO-0413

- Image sensor resolution: 752X480

- Image sensor electronic shutter method: global shutter method

- Image sensor video frame rate: 100 frame/second or less

A liquid crystal shutter glass is used as the optical shutter 100 and the specifications are as follows.

- Model name: Samsung Electronics' SSG-4100GB

- Liquid crystal transient time (on-off): about 4ms

- Relationship between voltage and light transmittance: 0V-40%, 5V-5%, 10V-1% or less

In addition, the inventor of the present invention set the internal function of the camera module as follows to shoot (acquire) a multiple exposure image.

- Exposure: Manually set the opening time of the electronic shutter to 3ms fixed exposure

- Master gain :　X1 (All RGB channels are set the same)

- Auto white balance adjustment: Auto exposure control, auto gain control, and auto white balance control are all disabled

- Trigger mode: Falling edge triggering

- Frame rate: set to 60Hz according to the setting of the electronic shutter control unit 120

In addition, a setting method for the multiple exposure image acquisition apparatus including the hardware is as follows.

First, the optical shutter control unit 130 sets a voltage input to the optical shutter 100 so as to have target light transmittances for a low-exposure image and a high-exposure image, respectively. In this embodiment, the input voltage to the optical shutter 100 is set to 5V to obtain a low-exposure image, so that the light transmittance is 5%, and the optical shutter 100 is applied to obtain a high-exposure image. Set the input voltage to 0V so that the light transmittance is 40%.

Next, the optical shutter control unit 130 sets a frame rate of a high dynamic range image (HDR image) to be finally synthesized. In this embodiment, the frame rate of the HDR image is set to 30 frames/second, that is, 30 Hz.

The electronic shutter control unit 120 generates a trigger signal at a rising edge and a falling edge of the voltage pulse signal output from the optical shutter control unit 130 . At this time, since the HDR image frame rate set by the optical shutter controller 130 is 30 Hz, the trigger signal generated by the electronic shutter controller 120 is set to 60 Hz. Meanwhile, the pulse width of the trigger signal may be set to 5 ms in consideration of a transient time of the optical shutter 100 .

Next, when the trigger mode of the camera module is set to a falling edge trigger mode as described above, the light transmittance change of the optical shutter 100 occurs, and as much as the pulse width (5ms) of the trigger signal After a time has elapsed, the electronic shutter for the image sensor 110 is activated to alternately acquire images of different exposures.

Hereinafter, a method for generating a high dynamic range image (HDR image) will be described with reference to the hardware setting of the multiple exposure image acquisition apparatus and FIGS. 1 to 2 .

Referring to FIG. 2 , (a) shows the voltage level input from the optical shutter control unit 130 to the optical shutter 100. Finally, if you want to generate an image with a high dynamic range with a frame rate of 30 frames/second, , the optical shutter control unit 130 alternately applies a low-level voltage and a high-level voltage to the optical shutter 100 at a cycle of 1/60 of a second (60Hz), but in an odd-numbered cycle, a voltage level of 0V voltage is applied, and a voltage having a voltage level of 5V is applied in an even-numbered period.

Here, even if the optical shutter control unit 130 applies a voltage pulse of 50% duty cycle to the optical shutter 100 with a period of 1/30 of a second (30 Hz), a high dynamic range image with a frame rate of 30 is finally obtained. can

(b) shows the light transmittance of the optical shutter 100 that changes according to the voltage level applied to the optical shutter 100, and shows transmittances 211 and 213 of 40% in odd-numbered periods, and even numbers In the th period, transmittances 212 and 214 of 5% are shown.

(c) shows a trigger pulse output from the electronic shutter control unit 120 to open and close the electronic shutter with respect to the image sensor 110 , the electronic shutter control unit 120 outputs from the optical shutter control unit 130 Since a trigger pulse is generated in synchronization with a rising edge and a falling edge of the voltage pulse signal, in this embodiment, the trigger pulse is set to 60 Hz, and the pulse width is the It can be set in consideration of the transient time of the optical shutter 100 . In this embodiment, since the transient time is 4 ms, the width of the trigger pulse is set to 5 ms. Here, the trigger pulse of 60 Hz indicates that the image frame rate of the image sensor 110 is 60 frames/second.

(d) shows the operation timing of the electronic shutter, and the opening timing of the electronic shutter is synchronized with the trigger pulse and the electronic shutter is opened for a predetermined time for each falling edge of the trigger pulse, for example, 3 ms (open)

As shown in (e), the image sensor 110 may acquire high-exposure images 221 and 223 and low-exposure images 222 and 224 in odd-numbered frames and even-numbered frames, respectively.

Accordingly, the apparatus for acquiring multiple exposure images according to an embodiment of the present invention may generate an HDR image frame by synthesizing the consecutively acquired pairs of images of different exposures. That is, the n-th HDR image frame is generated by synthesizing the high-exposure image frame 221 and the low-exposure image frame 222 , and by synthesizing the high-exposure image frame 223 and the low-exposure image frame 224 (n+ 1) By generating the first HDR image frame, an HDR image having a frame rate of 30 frames/second may be generated. In the detailed description, n is a natural number.

The embodiment shown in FIG. 2 is a case in which a signal for determining the frame rate of the image sensor 110 is generated using an external trigger signal generated by the electronic shutter control unit 120. In another embodiment, the camera without the external trigger signal The optical shutter 100 may be synchronized with a vertical synchronous signal of the image sensor 110 that determines the frame rate inside the module.

3 is a block diagram of an apparatus for acquiring a multiple exposure image according to another embodiment of the present invention, and shows the configuration of an apparatus for acquiring a multiple exposure image by synchronizing an optical shutter with a vertical synchronization signal of an image sensor.

In FIG. 3 , the same reference numerals are used for the reference symbols of each component when the components are the same as those of FIG. 1 .

As shown in FIG. 3 , the apparatus for acquiring multiple exposure images according to another embodiment of the present invention includes an optical shutter 100 , an image sensor 110 , an optical shutter controller 130 , and an HDR image generator 140 . The operation of the optical shutter 100 , the image sensor 110 , and the HDR image generator 140 is the same as the description of the components of FIG. 1 .

However, the optical shutter control unit 130 controls the opening and closing of the optical shutter 100 in synchronization with a vertical synchronization signal for determining the frame rate of the image sensor 110 , thereby reducing the amount of light passing through the optical shutter 100 . The amount of penetration can be adjusted. That is, the optical shutter control unit 130 adjusts the amount of light transmission in synchronization with the period of the trigger signal of the electronic shutter to the image sensor 110 , so that the image sensor 110 controls the high-exposure image and the low-exposure image. Images can be acquired alternately.

FIG. 4 is a diagram illustrating a control signal generation timing of each component for acquiring a multiple exposure image in the multiple exposure image acquisition apparatus shown in FIG. 3 .

As described above, the trigger signal of the electronic shutter shown in FIG. 4 is generated in synchronization with the vertical synchronization signal of the image sensor 110, and the optical shutter control unit 130 sets the target transmittance in synchronization with the vertical synchronization signal. By applying an achievable voltage level to the optical shutter 100 , the image sensor 110 may alternately acquire a high-exposure image and a low-exposure image.

For example, as shown in FIG. 4 , if the image acquisition frame rate of the image sensor 110 is 60 frames/second, the multiple exposure image acquisition apparatus according to the present invention includes the In a state of high light transmittance, the electronic shutter is operated for a certain period of time, for example, 3 ms, at a specific time in the 1/60 second image acquisition section to acquire a high-exposure image at every odd-numbered frame acquisition timing, and the optical shutter 100 In a state where the light transmittance of is low, the electronic shutter is operated for a certain time at a specific time within the 1/60 second image acquisition section to acquire a low-exposure image at every even-numbered frame acquisition timing. Thereafter, the HDR image generating unit 140 may generate an image having a high dynamic range of one frame by synthesizing the acquired high-exposure image and the low-exposure image. That is, when the frame rate of the image sensor 110 is 60, a video having a high dynamic range having a frame rate of 30 can be obtained.

Hereinafter, another method of acquiring a multiple exposure image by controlling the opening and closing of the optical shutter and the operation timing of the electronic shutter, and using this to generate a high dynamic range image (HDR image) will be described with reference to FIG. 5 .

FIG. 5 is a diagram illustrating control signal generation timing of each component for acquiring a multiple exposure image by controlling opening/closing timing of an optical shutter and operation timing of an electronic shutter in the multiple exposure image acquisition device shown in FIG. 1 .

Referring to FIG. 5 , the voltage level input from the optical shutter controller 130 to the optical shutter 100 is shown. Finally, if an image of a high dynamic range having a frame rate of 30 frames/second is to be generated, the optical shutter The controller 130 alternately applies a low-level voltage and a high-level voltage to the optical shutter 100 at a cycle of 1/60 of a second (60Hz), but applies a voltage having a voltage level of 0V in an odd-numbered cycle, , a voltage having a voltage level of 5V is applied in the even-numbered period.

Accordingly, as shown in (f), the light transmittance of the optical shutter 100 changes according to the applied voltage level, and shows a transmittance of 40% (high transmittance) in an odd-numbered period, and an even-numbered period The cycle shows a transmittance of 5% (low transmittance).

As shown in (g), the electronic shutter control unit 120 performs the electronic shutter of the image sensor 110 in advance at the end of the continuous odd-numbered image frame acquisition period and at the beginning of the even-numbered image frame acquisition period. An electronic shutter trigger signal may be generated so that the electronic shutter is opened (501, 502) for a predetermined period of time, for example, 3 ms.

In this embodiment, the electronic shutter operates close to the odd-numbered read-out time of the image sensor 110, and unlike the embodiment, the electronic shutter operates close to the read-out time of the even-numbered frame It is also possible to adjust the electronic shutter operation timing to

On the other hand, as shown in (g), when the electronic shutter operates close to the odd-numbered reading time of the image sensor 110 (501, 502), an interval 504 between the adjacent electronic shutter operation timings ) must be shorter than the time interval 505 between the electronic shutter opening timing 502 in the even-numbered frame and the electronic shutter opening timing 503 in the next odd-numbered frame so that the electronic shutter is the odd number of the image sensor 110 . You can make it work close to the second read time.

Accordingly, as shown in (h), the image sensor 110 may successively acquire a high-exposure image and a low-exposure image in odd-numbered frames and even-numbered frames, respectively, and the HDR image generator 140 ) may generate an HDR image frame by synthesizing the successively acquired pairs of images of different exposures. That is, the n-th HDR image frame is generated by synthesizing successive high-exposure image frames and low-exposure image frames, and then, the (n+1)-th HDR image is synthesized by synthesizing successive high-exposure image frames and low-exposure image frames. By generating a frame, an HDR image having a frame rate of 30 frames/second may be generated.

As described above, when an image frame is acquired at a point close to before and after the reading time of the odd-numbered frame or the reading time of the even-numbered frame with respect to the image sensor 110, when the acquired image frame is synthesized, the image There is an effect that can minimize the synthesis boundary error caused by the movement of the subject caused by the difference in frame acquisition time.

As the interval 504 between the adjacent electronic shutter operation timings in FIG. 5G is narrowed as much as possible within the limit allowed by the image sensor 110, the difference in acquisition timing of continuously acquired image frames can be reduced. The synthesis boundary error may be further reduced.

According to another embodiment, in a state in which the electronic shutter is fully opened, the optical shutter 100 is opened for different durations adjacent to before and after the read time for every read time for an odd-numbered frame of the image sensor 110 . By applying a predetermined voltage to the optical shutter 100 so as to be opened, the image sensor 110 may successively acquire a high-exposure image and a low-exposure image alternately.

Hereinafter, another method of acquiring a multiple exposure image by controlling the light transmission time of the optical shutter under the full-time electronic shutter condition and using this to generate a high dynamic range image (HDR image) will be described with reference to FIG. 6 . .

FIG. 6 is a diagram illustrating control signal generation timings of each component for acquiring a multiple exposure image by adjusting the light transmission time of an optical shutter under a full-time electronic shutter condition in the multiple exposure image acquisition device shown in FIG. 1 .

Referring to FIG. 6 , when the frame rate of the image sensor 110 is 60 frames/second, the multi-exposure image acquisition device according to the present invention has the electronic shutter fully opened through the electronic shutter control unit 120 ( In (j)), the optical shutter 100 may be opened for a predetermined time spanning the reading time of the odd-numbered frame of the image sensor 110 (shown in (i)).

At this time, if the image of the odd-numbered frame is acquired with high exposure and the image of the even-numbered frame is acquired with low exposure, the time 601 spanning the odd-numbered frame among the opening times of the optical shutter 100 is the even-numbered It can be set to be longer than the time spanning the frame 602 (shown in (i)).

At this time, the actual opening time of the electronic shutter becomes equal to the opening time of the optical shutter 100, and may be as shown in (k) (603, 604).

Accordingly, as shown in (l), the image sensor 110 may successively acquire a high-exposure image and a low-exposure image in an odd-numbered frame and an even-numbered frame, respectively, and the HDR image generator 140 ) may generate an HDR image frame by synthesizing the successively acquired pairs of images of different exposures.

As described above, when the high-exposure image and the low-exposure image are continuously acquired by adjusting the opening timing of the optical shutter 100 in a state in which the electronic shutter is fully opened, when synthesizing the acquired image frames, There is an effect that can minimize the synthesis boundary error caused by the movement of the subject caused by the difference in the image frame acquisition time.

Here, the optical shutter control unit 130 does not precisely control the light transmittance of the optical shutter 100 by adjusting the voltage level applied to the optical shutter 100 , but simply sets the optical shutter 100 to the maximum. By applying a voltage to have a light transmittance of or by applying a voltage so that the optical shutter 100 has a minimum light transmittance, the optical shutter 100 is controlled in an on state and an off state , it is possible to easily obtain a target exposure for an image to be acquired by controlling only the time when the optical shutter 100 is turned on and the duration that the optical shutter 100 is turned on.

According to an embodiment, in a state in which the electronic shutter is fully opened, the readout at every readout time for the odd-numbered frame of the image sensor 110 so that the optical shutter 100 is opened with different durations and different light transmittances By applying a voltage to the optical shutter 100 adjacent to before and after a period of time, the image sensor 110 may successively acquire a high-exposure image and a low-exposure image alternately.

Hereinafter, refer to FIG. 7 for a method of acquiring a multiple exposure image by adjusting the light transmittance and light transmission time of the optical shutter under the full-time electronic shutter condition, and using this to generate a high dynamic range image (HDR image) to explain

7 is a control signal generation timing of each component for acquiring a multiple exposure image by adjusting the light transmittance and light transmission time of the optical shutter under the full-time electronic shutter condition in the multiple exposure image acquisition device shown in FIG. It is also

Referring to FIG. 7 , the multiple exposure image acquisition method according to the present embodiment is mostly the same as the multiple exposure image acquisition method shown in FIG. 6 , and an optical shutter for adjusting the light transmittance and light transmission time of the optical shutter 100 . Only the control method of the controller 130 is different.

The image sensor 110 acquires the image of the odd-numbered frame with high exposure, and when it wants to acquire the image of the even-numbered frame with low exposure, the optical shutter 100 ), but set so that the time spanning the odd-numbered frame among the opening times of the optical shutter 100 is longer than the time spanning the even-numbered frame, and at the same time, the light transmittance during the time spanning the odd-numbered frame spans the even-numbered frame It may be set to be greater than the light transmittance over time (shown in (m)). The method for synthesizing the HDR image by synthesizing the acquired multiple exposure image is the same as the description of FIG. 6 , and thus will be omitted.

As described above, when the high-exposure image and the low-exposure image are continuously acquired by adjusting the opening timing and light transmittance of the optical shutter 100 in a state in which the electronic shutter is fully opened, multiple exposure images are acquired. Therefore, the dynamic range for the luminance is further expanded, the error with respect to the shooting timing is lowered, and the settability for the exposure ratio is improved.

Here, the optical shutter control unit 130 precisely adjusts the light transmittance of the optical shutter 100 by adjusting the voltage level applied to the optical shutter 100 , while the optical shutter 100 is turned on. ) and the duration of being turned on together, the target exposure for the image to be acquired can be obtained more precisely.

According to an embodiment, in a state in which the electronic shutter is fully opened, the reading is performed at every reading time of the odd-numbered frame of the image sensor 110 so that the optical shutter 100 is opened with the same duration and different light transmittance. By applying a voltage to the optical shutter 100 adjacent to before and after a period of time, the image sensor 110 may successively acquire a high-exposure image and a low-exposure image alternately.

Hereinafter, a multiple exposure image is acquired by controlling the light transmittance and light transmission time of the optical shutter under the full-time electronic shutter condition, and FIG. 8 is used to describe another method of generating a high dynamic range image (HDR image). Refer to and explain.

8 is a control signal generation timing of each component for acquiring a multiple exposure image by adjusting the light transmittance and light transmission time of the optical shutter under the full-time electronic shutter condition in the multiple exposure image acquisition device shown in FIG. it's another way

Referring to FIG. 8 , the multiple exposure image acquisition method according to the present embodiment is mostly the same as the multiple exposure image acquisition method shown in FIG. 7 , and the optical shutter control unit 130 for adjusting the light transmission time of the optical shutter 100 . ) differs only in the control method, so a detailed description is omitted.

As described in the embodiment of FIG. 8, when the high-exposure image and the low-exposure image are continuously acquired by adjusting the opening timing and light transmittance of the optical shutter 100 in a state in which the electronic shutter is fully opened, In generating an image of a high dynamic range by synthesizing the acquired multiple exposure images, there is an effect that the synthesis boundary distortion is very low, the error with respect to the photographing timing is lowered, and the setting of the exposure ratio is improved.

Hereinafter, a method of acquiring a multiple exposure image by controlling an optical shutter with a voltage level waveform in the form of an asymmetric ramp function and using this to generate a high dynamic range image (HDR image) will be described with reference to FIG. 9 . Explain.

FIG. 9 is an operation timing diagram for acquiring a multiple exposure image using a voltage pulse in the form of an asymmetric gradient function in the multiple exposure image acquisition apparatus shown in FIG. 1 .

Referring to FIG. 9 , as shown in (o) of the drawing, the optical shutter control unit 130 applies a voltage in the form of an asymmetric slope function to the optical shutter 100, so that the light transmittance having a high ascending slope in the high exposure section. After adjusting the start point of the light transmittance waveform in the read-out time section so that the high-exposure shooting condition is achieved with the waveform 910 and the target low-exposure shooting condition can be achieved, in the low-exposure section The low-exposure shooting condition may be achieved with the light transmittance waveform 920 having a low descending slope.

Accordingly, in a state in which the electronic shutter control unit 120 fully opens the electronic shutter, the optical shutter control unit 130 performs the above-described asymmetry at each reading time of the odd-numbered frame of the image sensor 110 over the reading time. A voltage in the form of a gradient function is applied to the optical shutter 100 to obtain a target light transmittance waveform, and the image sensor 100 acquires an image of an odd-numbered frame with high exposure, and Images can be acquired with low exposure.

The method of synthesizing the HDR image by synthesizing the successive images of different exposures is the same as described above, and thus will be omitted.

As described above, when the optical shutter 100 is controlled using the voltage pulse of the asymmetric gradient function, the exposure of the optical shutter 100 can be finely adjusted, and the response speed of the optical shutter is limited. It is possible to improve the exposure setting error, and also has the effect of accurately adjusting the ratio of the amount of light between the high-exposure image and the low-exposure image.

That is, the transmittance of the optical shutter 100 can be gradually changed by varying the voltage level applied to the optical shutter 100 with an asymmetric ramp function waveform such as a triangular wave or an S wave, so that a target high exposure Alternatively, the error can be minimized for the low exposure exposure setting.

In addition, when the opening and closing of the optical shutter 100 is controlled using the voltage pulse in the form of the asymmetric gradient function, the error in the shooting timing is lowered in acquiring a multiple exposure image, and the setting of the exposure ratio is improved When synthesizing the plurality of image frames having the acquired different exposure values, it is possible to minimize the synthesis boundary error (distortion) due to the movement of the subject due to the difference in the image frame acquisition time point.

Hereinafter, an example of synthesizing a plurality of multiple exposure images acquired through the multiple exposure image acquisition apparatus according to an embodiment of the present invention using HDR image synthesis software provided in the HDR image generator is shown in FIG. 10 . Refer to and explain.

10 is a diagram illustrating an example of generating an image of a high dynamic range by synthesizing a plurality of multiple exposure images acquired using the multiple exposure image acquisition apparatus according to an embodiment of the present invention.

Referring to FIG. 10 , using image synthesis software, one high-exposure image 1000, which is an odd-numbered image frame, and an even-numbered image frame, a low image, acquired through a multiple-exposure image acquisition device using image synthesis software. By synthesizing one exposure image 1010, one high dynamic range image was synthesized (Tone Mapped Image, 1020).

Here, the multi-exposure image acquisition environment used for image synthesis is a daylight condition with an illuminance of 10,000 Lux or more, and a luminance ratio of a dark place to a bright place is 10,000:1 or more.

Therefore, as shown in FIG. 10 , using the multiple exposure image acquisition apparatus according to an embodiment of the present invention, it is possible to easily create an image of a high dynamic range by synthesizing two images of different exposures. there is.

On the other hand, in addition to the synthesis of the two images of different exposures described above, images of three different exposures, for example, high exposure, middle exposure, and low exposure, are synthesized. An image with a further extended dynamic range may be easily generated.

Hereinafter, a method of generating an image with an extended dynamic range (ie, an image with a high dynamic range) by synthesizing three images of different exposures will be described in detail with reference to FIG. 11 .

11 is a block diagram of a multi-exposure image acquisition apparatus for generating images of a high dynamic range by acquiring images of three different exposures according to an embodiment of the present invention.

First, the optical shutter control unit 130 is input to the optical shutter 100 to achieve each target light transmission amount for the low-exposure image, medium-exposure image, and high-exposure image in synchronization with the image acquisition cycle of the image sensor 100 . Set the voltage.

Next, the image sensor 110 acquires the low-exposure image, the medium-exposure image, and the high-exposure image in each of the three consecutive frames, and the HDR image generator 140 has different exposure degrees of the three acquired images. By synthesizing an image having

Here, if the frame rate of the image sensor 110 is 60 frames/second, the optical shutter control unit 130 controls the image sensor 110 to perform three consecutive frames of three different exposures. By controlling the optical shutter 100 to acquire an image, the frame rate of the finally output high dynamic range image becomes 20 frames/second.

In the present embodiment, a case of synthesizing a high dynamic range image by acquiring three images having different exposures is exemplified, but the number of images of different exposures used for synthesizing the high dynamic range image is not limited. .

In the above-described embodiments in this specification, the frame rate of the image sensor is described as 60 frames/second. Therefore, the finally synthesized high dynamic range image is 30 frames/second when two images of different exposures are synthesized. The frame rate is , and when synthesizing three images with different exposures, the frame rate is 20 frames/second. However, as the frame rate of the image sensor increases, the frame rate of the finally synthesized high dynamic range image also increases.

On the other hand, the multi-exposure image acquisition device according to the concept of the present invention can improve the performance of equipment with poor image analysis accuracy due to the low dynamic range of the camera, so it is suitable for recognizing objects or analyzing screens through image capturing. can be applied

For example, the multi-exposure image acquisition device according to the concept of the present invention overcomes rapid exposure changes that may occur during flight such as clouds, building shadows, sun exposure, etc. It can also be applied to CCTV security systems that need to analyze specific objects such as objects, people, and cars through analysis.

In addition, the multi-exposure image acquisition device according to the concept of the present invention captures a front image to recognize a lane of a road and a vehicle of the other party, and then maintains a lane, avoids obstacles, avoids collisions with vehicles in front, etc. It can be applied to the field of autonomous driving vehicles, the field of a driving image recording device of a vehicle such as a black box, or the field of an automotive imaging device technology such as a side mirror or rearview mirror of a vehicle using a camera.

The multi-exposure image acquisition apparatus according to the concept of the present invention may be applied not only to the above-described application technology fields but also to all application technology fields for recognizing an object or analyzing a screen through image capturing.

The scope of the present invention is not construed as being limited by the number and types of components of the multiple exposure image acquisition device.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is merely exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

Claims (14)

An optical shutter for controlling the amount of transmitted light, an image sensor for acquiring an image by accumulating an electric charge according to the amount of light transmitted through the optical shutter, and an electronic for controlling the opening/closing time and opening/closing time of the electronic shutter of the image sensor A multiple exposure image acquisition apparatus comprising: a shutter control unit; and an optical shutter control unit configured to control a light transmission amount by controlling a voltage level and a voltage pulse width input to the optical shutter,
An HDR image generator for generating a High Dynamic Range (HDR) image by synthesizing a pair of odd-numbered and even-numbered frame images output from the image sensor with different degrees of exposure;
The optical shutter controller generates a voltage pulse signal in which a relatively low-level voltage and a high-level voltage are regularly located so that the high-exposure image and the low-exposure image are alternately output in frame units from the image sensor, and is applied to the optical shutter. do,
The electronic shutter control unit generates a trigger pulse having a predetermined pulse width synchronized with a rising edge and a falling edge of the voltage pulse signal input from the optical shutter control unit, and applies it to the electronic shutter;
According to the opening timing of the electronic shutter, which is opened for a predetermined time at each falling edge of the trigger pulse, the image sensor alternately outputs images with different exposure levels in the odd-numbered frame and the even-numbered frame in frame units. Multiple exposure image acquisition device. The electronic shutter according to claim 1, wherein in order to minimize a synthesis boundary error due to subject movement, the reading time of the odd-numbered frame and the reading time of the subsequent even-numbered frame with respect to the image sensor have a relatively short value. Multiple exposure image acquisition device, characterized in that setting the opening timing of. The method according to claim 1, wherein the images of the odd-numbered frame and the even-numbered frame output with different degrees of exposure,
In a state in which the electronic shutter is fully opened, the optical shutter control unit is configured such that the graph of the light transmittance of the optical shutter has a rising shape and a falling shape, respectively, before and after the reading time of every other sequence among the reading times of the image sensor. It is obtained by applying a voltage pulse of an asymmetric slope function type to the optical shutter, wherein the absolute value of the slope of the ascending type light transmittance graph is greater than the absolute value of the slope of the falling type light transmittance graph, characterized in that Multiple exposure image acquisition device. delete delete delete delete delete delete delete delete delete delete delete

Images