KR20140061234A - Image generating apparatus and method for generating imgae - Google Patents

Abstract

An image generating apparatus is disclosed. According to another aspect of the present invention, there is provided an image generating apparatus including a polarizing filter for polarizing incident light, a polarizing rotator for changing polarization characteristics of the incident light, a main lens for transmitting the incident light, An image sensor for sensing the incident light transmitted through the microlens array, and a control unit for processing the sensed value sensed by the image sensor to generate an image, .

Description

IMAGE GENERATING APPARATUS AND METHOD FOR GENERATING IMAGE < RTI ID = 0.0 >

The present invention relates to an image generating apparatus, and more particularly, to an image generating apparatus and an image generating method capable of simultaneously obtaining a conventional image and an integrated image using an integrated image technology and a micro lens array.

Integral Photography is a technique for obtaining multiple views in one shot using a micro lens array (MLA). Although the history of integrated imaging technology is long, commercialization has only recently taken place. Integrated imaging technology brings potential innovations by adding new features to the camera. What is interesting in the integrated imaging technology is that high-quality depth information can be obtained from a photographed image.

However, integrated video cameras that can be used today do not show the merit of surpassing the limits of conventional cameras. Problems of the integrated video camera include the following.

First, spatial resolution is low. Currently, integrated video cameras have a resolution of 1.2 megapixels, while conventional low-cost cameras have a resolution of 20 megapixels.

Second, it has limitations of zoom and iris control. A plenoptic integrated image camera must meet the F-number matching condition, and the integrated image camera has a fixed F-number.

Third, the obtained image has jittering edge and pattern noise.

In order to overcome the shortcomings of the current integrated image camera, a hybrid camera combining traditional camera and integrated image camera technology can be considered. The present invention will present this technique. In the present specification, various hardware systems will be introduced that use the same optical system to obtain integrated images and conventional images continuously or simultaneously using various hardware structures. Also, a software package for automatically correcting the hybrid camera is provided, and a high-quality depth map is calculated using the integrated image obtained by the hybrid camera. Due to the hardware structure of the proposed hybrid camera, the output depth map can be used with conventional full resolution images. Thus, the depth map can be used to modify or enhance conventional camera images.

1 and 2 are diagrams showing a conventional technology of a plenoptic camera.

Plenoptic camera technology using two MLAs is generally known. One of them is shown in Fig. 1 and mechanically moves the MLA in the camera sensor direction. When the MLA moves in the direction of the sensor, the camera behaves like a traditional full resolution camera. On the other hand, when the MLA moves away from the sensor, the camera operates like an integrated video camera. This is because, as shown in FIG. 1, when the MLA moves away from the sensor, multiple focuses are formed on the sensor, whereas when the MLA approaches the sensor, it converges to a single focus. However, such a technology implementation is time consuming and has reliability problems.

Another implementation is to use an active birefringent MLA as in FIG. A polarizing filter is placed in front of the main lens as shown in Fig. First, light having a specific polarization component is blocked through a polarizing filter. The active birefringence MLA is turned on or off according to the external potential value. These principles are explained in detail later. If the MLA is turned off, the camera operates with a traditional full resolution camera. On the other hand, when the MLA is turned on, the camera operates like an integrated video camera. The embodiment as shown in FIG. 2 uses an active MLA, which requires a switching time and is susceptible to ambient temperature. Moreover, there is a problem that the amount of light is reduced compared to a conventional camera due to the polarization filter.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is an object of the present invention to provide an image generating apparatus and an image generating method capable of capturing both an integrated image and a traditional full resolution image while providing a sufficient amount of light using the MLA .

According to an aspect of the present invention, there is provided an image forming apparatus including a polarizing filter for polarizing incident light, a polarizing rotator for changing polarization characteristics of the incident light, a main lens for transmitting the incident light, A microlens array for retransmitting the incident light transmitted through the main lens; an image sensor for sensing the incident light transmitted through the microlens array; To generate an image.

At this time, the polarizing filter and the polarization rotator may be positioned behind the main lens.

At this time, the polarizing filter and the polarizing rotator may be positioned behind the microlens array.

The polarizing filter and the polarization rotator may be coupled to at least a part of the image sensor.

According to another aspect of the present invention, there is provided an image generating apparatus including a main lens, a first active birefringent microlens array for transmitting incident light transmitted through the main lens, A second active birefringent microlens array for retransmitting the incident light transmitted through the lens array; an image sensor for detecting incident light transmitted through the second active birefringent microlens array; and an image sensor for processing the sensed value sensed by the image sensor, And a control unit for generating the control signal.

The image generating apparatus may further include a polarizing filter for polarizing the incident light.

The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may differ in at least one of a lens shape, a focus and a diaphragm value.

In the first active birefringent microlens array and the second active birefringent microlens array, birefringence may occur when a potential difference is generated.

The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may be individually voltage-impressed.

The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may have different optical axes.

According to an aspect of the present invention, there is provided a method of generating an image, the method comprising: polarizing incident light; varying polarization characteristics of the incident light; transmitting the incident light; Retransmitting the incident light, sensing the incident light transmitted through the microlens array, and processing an image sensed by the image sensor to generate an image.

At this time, the polarizing filter and the polarization rotator may be positioned behind the main lens.

The polarizing filter and the polarizing rotator may be positioned behind the microlens array.

In addition, the polarizing filter and the polarizing rotator may be positioned and coupled to at least a part of the image sensor.

According to another aspect of the present invention, there is provided a method of generating an image, comprising: transmitting incident light through a main lens; transmitting incident light transmitted through the main lens through a first active birefringent microlens array; Refracting the incident light transmitted through the first active birefringent microlens array through the lens array, sensing incident light transmitted through the second active birefringent microlens array through the image sensor, And processing the value to generate an image.

The image generating method may further include polarizing the incident light through a polarizing filter.

The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may differ in at least one of a lens shape, a focus and a diaphragm value.

Further, in the first active birefringent microlens array and the second active birefringent microlens array, birefringence may occur when a potential difference is generated.

In addition, the microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may be individually voltage-impressed.

The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may have different optical axes.

According to various embodiments of the present invention as described above, it is possible to capture both an integrated image and a conventional full resolution image while providing a sufficient amount of light using the MLA, and to provide an image generating apparatus and an image generating method .

Figures 1 and 2 illustrate a prior art plenoptik camera technology configuration,
3 to 6 are reference views for explaining the MLA structure of the image generating apparatus according to the present invention,
7 is a block diagram showing the configuration of an image generating apparatus according to an embodiment of the present invention;
8 to 10 are views showing a prototype having the configuration of the above-described image generating apparatus,
Figures 11 and 12 show an MLA structure having a plurality of layers,
13 is a block diagram showing the configuration of an image generating apparatus according to another embodiment of the present invention;
14 to 17 are views showing a prototype having the configuration of the above-described image generating apparatus,
18 is a view showing a microlens array composed of different kinds of microlenses,
Figure 19 is a diagram showing the overall schema of the software package of the present invention,
20 and 21 are flowcharts of image generation methods according to various embodiments of the present invention.

Various embodiments of the present invention will now be described in more detail with reference to the accompanying drawings.

3 to 6 are reference views for explaining the MLA structure of the image generating apparatus according to the present invention.

To design the prototype of a camera according to the present invention, two physical phenomena are used, which is commonly used by the makers of switchable 2D-3D lens displays.

The first phenomenon is birefringence. Birefringence is an optical property of a material having a refractive index that depends on the tilt of the polarization of light relative to the tilting of the optical axis of the material. The refractive index of the birefringent material for light with parallel polarization on the optical axis of the material is called the parallel axis of the optical axis. On the other hand, the refractive index of a birefringent material for light with vertical polarization at the optical axis of the material usually represents the optical axis. FIG. 3 shows the birefringence phenomenon, and shows ordinary rays and extraordinary rays.

The second phenomenon is the special electro-optical properties of liquid crystals (LC). Since the external potential can rotate the optical axis of the liquid crystal in which birefringence occurs, the refractive index can be controlled. When such a liquid crystal is used, it is possible to manufacture an active MLA which effectively turns on or off by adjusting the electric potential applied to the liquid crystal.

4 shows a passive MLA structure. Passive MLA is a material that causes birefringence. The light transmitted through the MLA is usually split into a light beam and an extraordinary ray. However, since it is passive, the transmitted light can not be changed.

5 shows a case where the active MLA is turned on. However, when the potential difference of the liquid crystal is zero, the incident light can be divided into ordinary light and abnormal light in different directions and transmitted.

6 shows a case where the active MLA is turned off. However, if the potential difference of the liquid crystal is set to a predetermined value (> 0), the incident light can be transmitted through ordinary rays in the same direction.

Based on the above-described phenomenon, various structures that can be used to fabricate a hybrid 2D-3D plenoptic camera having various properties can be proposed.

7 is a block diagram showing the configuration of an image generating apparatus according to an embodiment of the present invention.

7, an image generating apparatus 100 according to an exemplary embodiment of the present invention includes a polarization filter 110, a polarization rotator 120, a main lens 130, a microlens array 140, an image sensor 150, , And a control unit (160).

The polarizing filter 110 has a polarization axis in a predetermined direction and allows polarized light in the same direction to pass therethrough. For example, it can transmit horizontally polarized light or transmit vertically polarized light.

The polarization rotator 120 changes the polarization characteristics of the polarized light. Here, the characteristic of the polarized light may be the phase or the direction of the polarized light. That is, when the polarized light is incident, it serves to delay the wavelength of the incident light. When the polarization direction of the linearly polarized light is changed, the circularly polarized light is converted. When the phase of the circularly polarized light is changed, the direction of the circularly polarized light is reversed.

The main lens 130 transmits the reflected light reflected from the subject. The main lens 130 may be implemented by a general-purpose lens or a wide-angle lens.

The microlens array 140 transmits the light incident through the main lens 130. The microlens array 140 may include a plurality of microlenses connected horizontally to form a microlens array. Each microlens constituting the microlens array re-transmits the light transmitted through the main lens 130 individually. The re-transmitted light is incident on the image sensor 150.

The image sensor 150 detects the light transmitted through the microlens array 140. The image sensor 150 may be composed of a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD). The image sensor 150 accumulates light through the photodiode PD of the pixel array and outputs an electric signal according to the accumulated light amount.

The image sensor 150 may include a photodiode PD, a transfer transistor TX, a reset transistor RX, and a floating diffusion node FD. The photodiode PD generates and accumulates photo charges corresponding to the optical image of the subject. The transfer transistor TX transmits the optical telephone generated in the photodiode PD to the floating diffusion node FD in response to the transmission signal. The reset transistor discharges the charge stored in the floating diffusion node FD in response to the reset signal. Before the reset signal is applied, the charges stored in the floating diffusion node FD are output. In the case of the CDS image sensor, CDS (Correlated Double Sampling) processing is performed. Then, the ADC converts the analog signal subjected to CDS processing into a digital signal.

The control unit 160 controls the overall operation of the image generating apparatus 100 and processes the sensed values sensed by the image sensor 150 to generate an image. The control unit 160 includes a hardware configuration such as a CPU and a cache memory, an operating system, and a software configuration of an application that performs a specific purpose. A control command for each component for operation to be described later is read from the memory according to the system clock, and an electric signal is generated according to the read control command to operate each component of the hardware.

The light reflected by the subject and incident on the main lens 130 is again incident on the microlens 150 and the light transmitted through the microlens 150 is detected by the image sensor 160 and output as an electric signal. The light transmitted through each microlens 150 is sensed in the sensing area of the image sensor 170. A prototype having such a configuration will be described below.

8 to 10 are views showing a prototype having the configuration of the above-described image generating apparatus.

The prototype of FIG. 8 uses a fixed birefringence MLA, which transmits light energy only for light having a specific polarization direction. A polarizing filter and a polarizing rotator that can turn on and turn off the MLA can be used. In the embodiment of FIG. 8, the polarizing filter and the polarizing rotator are disposed in front of the main lens, but may be located behind the main lens or behind the MLA. The polarizing rotator is analogous to mechanically rotating polarizing filters or twisted nematic liquid crystal switches. The first prototype architecture enables capturing the integrated images and the traditional full resolution images in chronological order.

The second prototype shown in Figure 9 is similar to the first prototype except that the polarizing filter is coupled to the camera sensor. Since several pixels of the sensor have different polarization filters, some pixels of the sensor act like pixels of an integrated video camera, while others behave like pixels of a conventional camera. As a result, the second prototype can simultaneously acquire integrated images and traditional images in one shot. It should be noted that the pattern of the polarizing filter of the Bayer pattern and the second prototype is selected in a different way.

In the third prototype shown in Fig. 10, all pixels of the image sensor are coupled to a polarization rotator and a polarization filter. The polarization rotator combined with the polarizing filter operates like a valve that transmits only light having a specific polarization direction. Since it is possible to control such valves individually for all camera sensors, it is possible to dynamically determine whether a pixel will acquire an integrated image or a traditional image. Therefore, the third prototype can obtain the integrated image, and can obtain the conventional full resolution image continuously or simultaneously according to the set value.

The first, second, and third prototypes have a common problem: each pixel receives up to half the amount of light compared to a traditional camera. This is because all of these prototypes include polarizing filters. However, when a plurality of the active birefringence MLA are used, it becomes possible to avoid the use of the polarizing filter.

11 and 12 are diagrams showing an MLA structure having a plurality of layers.

In an MLA having a plurality of optical energies, the liquid crystal has to have a vertical optical axis mutually. As a result, if the MLA is not activated, it forms an optical system equivalent to a single MLA. On the other hand, when the MLA is activated, it forms a system without optical energy.

13 is a block diagram showing the configuration of an image generating apparatus according to another embodiment of the present invention.

13, an image generating apparatus 100 'according to another embodiment of the present invention includes a main lens 130, a first active birefringent microlens array 140-1, a second active birefringent microlens array 140-2, an image sensor 150, and a control unit 160. [

Since each configuration is the same as described above, a duplicate description will be omitted.

The first active birefringent microlens array 140-1 transmits incident light transmitted through the main lens 130 and the second active birefringent microlens array 140-2 transmits the incident light transmitted through the first active birefringent microlens array 140- 1) of the incident light.

Also, the image sensor 150 detects incident light transmitted through the second active birefringent microlens array.

14 to 17 are views showing a prototype having the configuration of the above-described image generating apparatus.

The fourth prototype does not use a polarizing filter. The fourth prototype can continuously acquire integrated images and traditional full resolution images. The fourth prototype connects two or more birefringence active MLAs to form an MLA set. When the voltage is applied, the MLA set is turned on, and when the voltage is not applied, the MLA set is turned off. The MLA set operates in the integrated video mode in the turn-on state, and the traditional full-resolution camera in the turn-off state.

The fifth prototype shown in Fig. 15 has an operation mode for generating various integrated images. Each of the modes of operation represents a trade-off between the spatial resolution and the respective resolution. The prototype has an MLA that exhibits active birefringence that results in multiple results between the main lens and the sensor. Each MLA has a unique parameter such as a microlens shape, a focus length, an arrangement, and an aperture value. To switch the camera to a specific integrated imaging mode, all MLAs except one should be turned off. On the other hand, to switch the camera to full resolution traditional camera mode, all MLA must be turned off.

The sixth prototype shown in FIG. 16 can independently turn-on and turn-on all microlenses of the active birefringence MLA. This is made possible by installing individual control modules in front of every MLA. Such a prototype pixel group operates in the conventional mode while the other group is operating in the integrated image mode.

The seventh prototype shown in Fig. 17 is functionally similar to the previous prototype except that it is based on a passive birefringence MLA. Each microarray is controlled by a separate polarizing rotator and is located in front of all microlenses.

All MLA prototypes should not operate in purely turn-off and turn-on mode. Instead, the MLA has different optical energies. In such a case, the prototype in a different mode would be in a mode with a different tradeoff between spatial and angular resolution.

18 is a view showing a microlens array composed of different kinds of microlenses.

In the drawing, the MLA is made up of microlenses of the same kind, but it may be composed of different kinds of microlenses as shown in FIG. For example, different micro lenses of the same MLA may have different focus lengths, apertures, sizes, shapes, and other parameters. In addition, they may be made of different materials.

Also, software for the hybrid floptronic photo camera according to the present invention can be considered.

19 is a diagram showing the overall schema of the software package of the present invention.

The software package of the present invention consists of three modules. That is, the software package of the present invention includes a calibration module, a depth evaluation module, and an image fusion module.

The calibration module performs the calibration of the optical system of the hybrid camera using the average of the various white images. In a first step, the module calculates a vignetting compensation factor in a pixel way. The module then calculates the global parameters of the microlens grid, such as grid rotation, duration, and shift. Finally, the module performs local calculations of the microlenses with the subpixel accuracy. Then, the calibration parameter output for continuing use is stored by the depth evaluation module.

The depth evaluation module computes a continuous depth map from the resulting plenotic image of high quality. When a new plenoptic image arrives, the module first normalizes the image using calibration data. The bnet is removed and the microcentre is searched. Then, it calculates the smoothing and calculates the data condition of the target quadratic function used to find the depth map. Finally, the module minimizes the target function by evaluating the depth map by solving the linear function. The module outputs a depth map for use of the image fusion module.

The Image Fusion module uses a depth map to enhance or modify the traditional high-definition image. Since depth and color images are obtained using the same optical system, no prior image registration is required. There are three types of algorithms in the image fusion module. The first type is to mimic the photographic effects of high-end cameras such as out-focus blur and depth-dependent contrast. The second type is a camera with new interactive features such as digital refocusing and object level editing. The third type is a camera that performs high-level image processing such as object recognition and texture mapping.

The hybrid camera system proposed in the present invention can be applied to various devices. For example, devices such as compact cameras, smart phones, tablet PCs, and the like. Hybrid cameras can provide high-end cameras as well as interactive functions. In addition, it can be used variously in the medical field and the medical field.

Hereinafter, an image generating method according to various embodiments of the present invention will be described.

20 and 21 are flowcharts of image generation methods according to various embodiments of the present invention.

Referring to FIG. 20, an image forming method according to an exemplary embodiment of the present invention includes polarizing incident light (S2010), changing polarization characteristics of the incident light (S2020), transmitting the incident light (S2030) (S2040) of retransmitting the incident light transmitted through the main lens (S2040), sensing the incident light transmitted through the microlens array (S2050), and generating an image by processing a sensed value sensed by the image sensor (S2060).

At this time, the polarizing filter and the polarization rotator may be positioned behind the main lens.

The polarizing filter and the polarizing rotator may be positioned behind the microlens array.

In addition, the polarizing filter and the polarizing rotator may be positioned and coupled to at least a part of the image sensor.

21, an image generating method according to an exemplary embodiment of the present invention includes: transmitting incident light through a main lens (S2110); transmitting incident light transmitted through the main lens through a first active birefringent microlens array (S2120), retransmitting the incident light transmitted through the first active birefringent microlens array through the second active birefringent microlens array (S2130), transmitting the second active birefringent microlens array through the image sensor A step S2140 of detecting the incident light, and a step S2150 of processing an image sensed by the image sensor to generate an image.

The image generating method may further include polarizing the incident light through a polarizing filter.

The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may differ in at least one of a lens shape, a focus and a diaphragm value.

Further, in the first active birefringent microlens array and the second active birefringent microlens array, birefringence may occur when a potential difference is generated.

In addition, the microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may be individually voltage-impressed.

The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array may have different optical axes.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the invention as defined by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

100, 100 ': Image generating device
110: Polarization filter 120: Polarization rotator
130: main lens 140: MLA
150: image sensor 160:

Claims (20)

An image generating apparatus comprising:
A polarizing filter for polarizing incident light;
A polarization rotator changing a polarization characteristic of the incident light;
A main lens that transmits the incident light;
A microlens array for retransmitting the incident light transmitted through the main lens;
An image sensor for sensing the incident light transmitted through the microlens array; And
And a controller for processing the sensing value sensed by the image sensor to generate an image. The method according to claim 1,
The polarizing filter and the polarizing rotator may comprise:
And the second lens is located behind the main lens. The method according to claim 1,
The polarizing filter and the polarizing rotator may comprise:
Wherein the micro-lens array is located behind the micro-lens array. The method according to claim 1,
The polarizing filter and the polarizing rotator may comprise:
Wherein the image sensor is coupled to at least a portion of the image sensor. An image generating apparatus comprising:
A main lens;
A first active birefringent microlens array for transmitting incident light transmitted through the main lens;
A second active birefringent microlens array retransmitting the incident light transmitted through the first active birefringent microlens array;
An image sensor for sensing incident light transmitted through the second active birefringent microlens array; And
And a controller for processing the sensing value sensed by the image sensor to generate an image. 6. The method of claim 5,
And a polarizing filter for polarizing the incident light. 6. The method of claim 5,
Wherein the microlens constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array differ in at least one of a lens shape, a focus and a diaphragm value. 6. The method of claim 5,
Wherein the first active birefringent microlens array and the second active birefringent microlens array comprise:
And a birefringence occurs when a potential difference is generated. 6. The method of claim 5,
The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array are arranged such that,
Wherein the image forming apparatus is capable of applying a voltage separately. 6. The method of claim 5,
The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array are arranged such that,
And has different optical axes. In the image generating method,
Polarizing incident light;
Changing a polarization characteristic of the incident light;
Transmitting the incident light;
Retransmitting the incident light transmitted through the main lens;
Sensing the incident light transmitted through the microlens array; And
And processing the sensing value sensed by the image sensor to generate an image. 12. The method of claim 11,
The polarizing filter and the polarizing rotator may comprise:
Wherein the second lens is located behind the main lens. 12. The method of claim 11,
The polarizing filter and the polarizing rotator may comprise:
Wherein the micro-lens array is located behind the micro-lens array. 12. The method of claim 11,
The polarizing filter and the polarizing rotator may comprise:
Wherein the image sensor is located in at least a portion of the image sensor. In the image generating method,
Transmitting the incident light through the main lens;
Transmitting the incident light transmitted through the main lens through the first active birefringent microlens array;
Retransmitting the incident light transmitted through the first active birefringent microlens array through the second active birefringent microlens array;
Detecting an incident light transmitted through the second active birefringent microlens array through an image sensor; And
And processing the sensing value sensed by the image sensor to generate an image. 16. The method of claim 15,
And polarizing the incident light through a polarizing filter. 16. The method of claim 15,
Wherein the microlens constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array are different from each other in at least one of a lens shape, a focus and a diaphragm value. 16. The method of claim 15,
Wherein the first active birefringent microlens array and the second active birefringent microlens array comprise:
And a birefringence occurs when a potential difference is generated. 16. The method of claim 15,
The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array are arranged such that,
And a voltage can be separately applied. 16. The method of claim 15,
The microlenses constituting the first active birefringent microlens array and the microlenses constituting the second active birefringent microlens array are arranged such that,
And having different optical axes.

Images