KR20130106525A - Device for processing image and method thereof - Google Patents

Abstract

An image processing method according to an embodiment of the present invention may include obtaining four-dimensional light field information including a combination of positional information about a plurality of two-dimensional images; Firstly converting the obtained 4D optical field information into a 4D phase space function; And reconstructing the two-dimensional image by quadratic transforming the four-dimensional phase space function.

Description

Image processing apparatus and image processing method thereof

The present invention relates to an image processing apparatus, and more particularly, to an image processing apparatus capable of acquiring a 3D image including depth information by using image information obtained from a 4D optical field, and an image processing method thereof.

Imaging applications such as cameras, video cameras, microscopes, telescopes and many others have typically had extremely limited amounts of light collected. This is because most imaging devices do not record most of the information about the light distribution entering the device. For example, conventional cameras such as digital still cameras and video cameras do not record most of the information about the light distribution coming from the world. In these devices, the collected light often has to be manipulated for various applications such as focusing at different depths (distance from the imaging device), correcting lens aberrations, or manipulating the screen angle.

Among these, in the case of still-imaging applications, a typical imaging device for capturing a particular scene generally focuses on the subject or object in the scene, while other parts of the scene have the problem of out of focus.

In the case of video-imaging applications, many similar problems appear, but the problem is that the set of images used in the video application cannot capture the scene in focus.

Many imaging applications have suffered from aberrations due to the device (lens) used to collect light. Such aberrations may include, for example, spherical aberration, chromatic aberration, distortion, bending of the light field, oblique astigmatism, and coma. Correction of aberrations typically required the use of correction optics when tending to add volume, expense and weight to the imaging device. However, including additional optical systems is undesirable in terms of miniaturization and manufacturing cost of the imaging device.

In order to solve this problem, a method of compensating for the disadvantages and limitations of the device by refocusing the already taken image by software has been proposed in US Patent No. 7,936,392.

However, in the method disclosed in the above patent, two Fourier transforms are used, and an integral is applied four times to this equation, so that the calculation speed can be reduced. For example, it may take 3 to 4 seconds to acquire a single image by this method, and there is a problem in processing a real-time image.

According to an embodiment of the present invention, by converting the combination of the two-dimensional position information obtained in the four-dimensional light field to the Wigner function, and then radon transform it to obtain a three-dimensional image including the depth information with a reduced amount of calculation An image processing apparatus and an image processing method thereof may be provided.

It is to be understood that the technical objectives to be achieved by the embodiments are not limited to the technical matters mentioned above and that other technical subjects not mentioned are apparent to those skilled in the art to which the embodiments proposed from the following description belong, It can be understood.

An image processing method according to an embodiment of the present invention may include obtaining four-dimensional light field information including a combination of positional information about a plurality of two-dimensional images; Firstly converting the obtained 4D optical field information into a 4D phase space function; And reconstructing the two-dimensional image by quadratic transforming the four-dimensional phase space function.

In addition, obtaining the four-dimensional light field information includes detecting light from a scene that is delivered to different locations on a focal plane using a photo sensor array.

Reconstructing the two-dimensional image also includes reconstructing the detected light to form an image in which at least a portion of the two-dimensional image is constructed at different focal planes.

The converting into the 4D phase space function may include converting 4D light field information configured from the detected light into a Wigner distribution function.

In addition, the four-dimensional light field information is represented by L (u, v, x, y), the Wigner distribution function is represented by WDF (s, t, x, y), the s and t parameters are It is calculated by the following equation.

s = s`tan-1 ((u-x) / D), t = t`tan-1 ((v-y) / D)

Here, D is the distance value between the photosensor array and the microlens array or the distance from the two-dimensional (x, y) plane to the (u, v) plane.

S` and t` are information for matching the 4D optical field information with any constant including 1.

In addition, the Wigner distribution function WDF (s, t, x, y) is calculated by the following equation.

는 임의의 면에서의 2차원 이미지 분포를 의미하는 함수임.Where k is the wave vector of the light source,

Is a function of two-dimensional image distribution on any plane.

The reconstructing the two-dimensional image may include performing the quadratic transformation and performing a three-dimensional image including depth information or two-dimensional image corresponding to the detected light by the second transformation. Generating.

In addition, performing the quadratic transformation includes radon transforming the transformed Wigner distribution function.

The method may further include defining a scale factor for the radon transform.

In addition, the scale factor is determined through the focal length of the main lens and the wavelength of the light, and the rotation angle of the radon transform determines the depth direction of the generated 3D image.

On the other hand, the image processing apparatus according to an embodiment of the present invention, the image processing apparatus for generating an image from the scene, the main lens; A photosensor array for capturing a set of rays; A micro lens array between the main lens and the photo sensor array; And acquiring four-dimensional light field information consisting of a combination of positional information about a plurality of two-dimensional images using the set of light rays captured by the photosensor array, and based on the acquired four-dimensional light field information. And a data processor for reconstructing the dimensional image.

In addition, the data processor is an image acquisition unit for acquiring four-dimensional light field information consisting of a combination of position information for the plurality of two-dimensional images, and the acquired four-dimensional light field information means a four-dimensional phase space function And a Wigner function unit for converting the Wigner distribution function, and a Radon transform unit for reconstructing the 2D image by radon transforming the converted Wigner distribution function.

The image acquisition unit also detects light from the scene that is delivered to different locations on the focal plane using the photo sensor array.

The radon converter may reconstruct the detected light to generate an image in which at least a portion of the two-dimensional image is configured in a different focal plane.

In addition, the four-dimensional light field information is represented by L (u, v, x, y), the Wigner distribution function is represented by WDF (s, t, x, y), the s and t parameters are It is calculated by the following equation.

s = s`tan-1 ((u-x) / D), t = t`tan-1 ((v-y) / D)

Here, D is the distance value between the photosensor array and the microlens array or the distance from the two-dimensional (x, y) plane to the (u, v) plane.

S` and t` are information for matching the 4D optical field information with any constant including 1.

In addition, the Wigner function unit calculates the Wigner distribution function WDF (s, t, x, y) by the following equation.

는 임의의 면에서의 2차원 이미지 분포를 의미하는 함수임.Where k is the wave vector of the light source,

Is a function of two-dimensional image distribution on any plane.

The radon transform unit may generate a 3D image including the 2D image or the depth information corresponding to the detected light by radon transforming the Wigner distribution function.

In addition, the radon transform unit performs a radon transform based on a predefined scale factor, the scale factor is determined based on the focal length and the wavelength of light of the main lens, the rotation angle of the radon transform of the generated three-dimensional image Determine the depth direction.

According to an embodiment of the present invention, a three-dimensional image or a two-dimensional image including depth information may be obtained in real time using a combination of two-dimensional image information acquired in a four-dimensional light field.

In addition, according to an embodiment of the present invention, by obtaining a three-dimensional image including the depth information by using the Wigner function transform and the Radon transform, it is possible to increase the speed of the operation by reducing the number of integral operations, accordingly This has the effect of performing image processing.

1 illustrates an image processing apparatus 100 according to an embodiment of the present invention.
2 illustrates a method for mapping sensor pixel positions to light rays in L (u, v, s, t) space with respect to the collected data.
3 is a diagram illustrating a configuration of a data processor according to an exemplary embodiment of the present invention.
4 is a flowchart illustrating a step-by-step method for processing an image according to an exemplary embodiment.

The following merely illustrates the principles of the invention. Thus, those skilled in the art will be able to devise various apparatuses which, although not explicitly described or shown herein, embody the principles of the invention and are included in the concept and scope of the invention. Furthermore, all of the conditional terms and embodiments listed herein are, in principle, only intended for the purpose of enabling understanding of the concepts of the present invention, and are not to be construed as limited to such specifically recited embodiments and conditions do.

It is also to be understood that the detailed description, as well as the principles, aspects and embodiments of the invention, as well as specific embodiments thereof, are intended to cover structural and functional equivalents thereof. It is also to be understood that such equivalents include all elements contemplated to perform the same function irrespective of the currently known equivalents as well as the equivalents to be developed in the future, i.e., the structure.

2 is a diagram illustrating an image processing apparatus according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the image processing apparatus 100 splits the light beams passing through the main lens 110 and the main lens 110 to form an image of an object based on the direction of the light beams to the photo sensor array 130. A light field data capture unit 140 having a microlens array 120 and a photosensor array 130 to be directed; and a data processor 150 for processing the captured light field data.

In this case, the microlens array 120 and the photosensor array 130 may be implemented as the image sensor 160. Such an image processing apparatus may be used to acquire a refocused image or an image viewed from various angles (that is, to adjust the view of the image).

Rays from a single point on the subject 105 in the imaged scene may be reached at a single convergence point on the focal plane of the microlens array 120.

The microlens 122 at this convergence point separates these rays based on the direction of the light, producing a focused image of the aperture of the main lens 110 on the photosensor of the micro lens arena.

The photosensor array 130 detects light incident thereon and produces an output that is processed using one or more of the various components. The output optical data may be, for example, sent to the data processor 150 using the data together with the position information of each photosensor providing the data when generating an image of a scene including the subjects 105, 106, and 107. Is sent.

The data processor 150 may be implemented with, for example, a computer or other processing circuit selectively implemented in a common component (eg, one chip) or a different component. A part of the data processor 150 may be implemented in the light field data capture unit 140, and the other part may be implemented in an external computer.

The data processor 150 is implemented to process image data and to calculate an image of a scene including the subjects 105, 106, and 107.

The data processing unit 150 uses the detected light or the characteristic of the detected light together with the known direction in which the light reaches the microlens array (calculated using the known position of each photosensor) to achieve refocusing. Selectively refocus and / or correct data when forming an image that can be corrected.

The image processing apparatus 100 may be implemented in various ways depending on the application. For example, the microlens 120 is shown by way of several distinguishable microlenses by way of example, but the array is typically implemented with multiple (eg, thousands or millions) microlenses.

In addition to the microlens array 120, the optical rays may pass through the main lens 110 and may be configured in other forms, such as an optical coding mask. Those skilled in the art will appreciate that the main lens 110 and microlens array 120 can be implemented using various lenses and / or microlens arrays that are currently available or developed in the future. There will be.

The photosensor array 130 generally has several photosensors for each microlens in the microlens array 120. The size of each pixel of the photosensor array 130, that is, the pitch, is relatively finer than that of the microlens array 122. In addition, the microlenses in the microlens array 120 and the photosensors in the photosensor array 130 are generally positioned such that the light that goes through the respective microlenses to the photosensor array does not overlap with the light that passes through the adjacent microlenses. Can be.

The main lens 110 has a performance such as being moved horizontally along the optical axis to focus on the subject of interest at the desired depth 'd' as illustrated between the main lens 110 and the exemplary imaging subject 105. Has

Therefore, it is possible to refocus at each location of interest based on the obtained light field data.

By way of example, light rays from a single point of the subject 105 are shown for this description. These rays may reach a single convergence point in the microlens 122 on the focal plane of the microlens array 120.

The microlens 122 separates based on the direction of these rays to produce a focused image and light field data of the aperture of the main lens 110 on a set of pixels in the pixel array below the microlens.

Considering the two-plane light field "L" inside the image processing apparatus 100, the light field data L (u, v, s, t) intersects the main lens 110 at (u, v), and s, t) represents light traveling along a ray intersecting the plane of the microlens array 120. For example, in each micro lens (s, t), the intensity values passing through the position (u, v) of each sub-aperture of the main lens (110).

Here, the sub-aperture means the number of directional resolutions of the main lens 110. For example, when the number of sub-apertures is 196, each microlens array 120 may include 196 pixels.

Each photosensor in photosensor array 130 may be implemented to provide a value indicative of a set of rays directed to photosensors through main lens 110 and microlens array 120. That is, each photosensor produces an output in response to light incident on the photosensors, and the position of each photosensor relative to the microlens array 120 is used to provide direction information for the incident light.

The data processor 150 may generate a 3D image including depth information using light field data, that is, L (u, v, s, t). In this case, the data processor 150 may determine the direction of light on each photosensor using the position of each photosensor with respect to the microlens.

In addition, the data processor 150 may determine a depth of field of a subject in a scene in which the detected light is spread, and calculate a synthesized image focused on a focal plane different from the focal plane by using the depth of field and the direction of the detected light. have.

Hereinafter, as an image processing apparatus of the present invention, an image processing method performed by the data processing unit 150 will be described in more detail.

2 shows a method for the mapping of sensor pixel positions to light rays in a four-dimensional optical space L (u, v, x, y).

The method shown in FIG. 2 and described herein, for example, is applicable to FIG. 1, and each photosensor in photosensor array 130 may correspond to a sensor pixel.

The lower right image 1170 represents downsampling of the raw data read from the photosensor array 130, and some of the circular images 1150 of the image 1170 appear below a single microlens circled to the left. It may be a circular image 1150.

The lower left image 1180 is an enlargement of a portion of the raw data around the circled microlens image 1150, with one photosensor value 1140 circled within the microlens image 1150.

Since this circular image 1150 is an image of a lens aperture, the position of the selected pixel in the disk can provide (u, v) coordinates of the starting position of the depicted ray 110 on the main lens.

The location of the microlens image 1150 within the sensor image 1170 may provide (x, y) coordinates of the ray 110.

With reference to FIG. 1, each photosensor in the photosensor array can thus be implemented to provide a value representing a set of rays directed through the main lens 110 and microlens array 120 to the photosensor.

That is, each photosensor generates an output in response to light incident on the photosensors, and the position of each photosensor relative to the microlens array 120 may be used to provide direction information for the incident light.

On the other hand, the resolution of the microlens array 120 may be selected to suit the desired resolution for the final image of the particular application. The resolution of the photosensor array 130 may be selected so that each microlens covers as many photosensors as necessary to match the desired orientation resolution of the application, or the most sophisticated resolution of the photosensors that can be implemented.

3 is a diagram illustrating a configuration of a data processor 150 according to an embodiment of the present invention.

Referring to FIG. 3, the data processor 150 according to an exemplary embodiment of the present invention may include an image acquisition unit 151 for simultaneously detecting light from a scene to be delivered, and a four-dimensional light field including the detected light. The Wigner function unit 153 converts an argument into a Wigner function, and a Radon transform unit 155 for radon transforming the Wigner function converted through the Wigner function 153.

The image acquisition unit 151 may acquire image information regarding the sensor pixel position with respect to the light ray in the 4D optical space L (u, v, x, y).

Operations performed by the Wigner function unit 153 and the radon transform unit 155 may correspond to an output image generation operation according to an embodiment of the present invention.

In generating an output image, the Wigner function 153 and the radon transform unit 155 may perform a series of operations using the detected light to form an image in which at least a portion of the image is focused at different focal planes. have.

In detail, the Wigner function 153 may convert a factor of a four-dimensional light field composed of light into a Wigner function.

For example, when the image information regarding the photosensor pixel position of the four-dimensional light field is L (u, v, x, y), the Wigner function 153 may determine the four-dimensional light field value. Create a Wigner function (WDF (s, t, x, y)).

Converting four-dimensional light field information, which is a combination of the position information of the two-dimensional images, into a Wigner distribution function, is performed when Fresnel diffraction occurs when the light diffraction phenomenon is greater than or equal to a ratio of the area of the aperture and the wavelength. By using the characteristic, a plurality of two-dimensional position information can be converted into a four-dimensional phase space Wigner distribution function.

That is, the positional space information of (u, v) and (x, y) may be changed into an angle and a phase space variable of L (u, v, x, y) and expressed as a Wigner distribution function.

As described above, the parameter input to the Wigner function includes s, t, x, and y, where s and t may be calculated as in Equation 1 below.

Here, D may use a distance value between the photosensor array and the microlens array, but may mean a distance from the two-dimensional (x, y) plane to the (u, v) plane.

In addition, s` and t` generally use 1, but any other constant may be used, which serves to match 4D optical field information.

That is, s and t may represent angles of position coordinates (x, y) values.

At this time, the Wigner function unit 153 calculates a Wigner function value WDF (s, t, x, y) based on the four-dimensional light field value using Equation 2 below.

Here, k described in Equation 2 means the wavenumber vector of the light source.

는 임의의 면에서의 2차원 이미지 분포를 의미하는 함수이다. In addition, it is described in Formula 2

Is a function of the two-dimensional image distribution in any plane.

In this case, the image distribution value P_d (x) at an arbitrary distance 'd' may be defined as a coordinate rotation function of the Wigner distribution function as shown in Equation 3 below.

와 2차원 이미지 면의 x`으로 나타낼 수 있음을 보여준다.Equation 3 is a two-dimensional image distribution function and rotation conversion angle at a constant distance 'd'

And x 'of the two-dimensional image plane.

와의 관계는 아래 수학식 4와 같이 정의될 수 있다.Where the scale vector s and the rotation transformation angle

The relationship with may be defined as in Equation 4 below.

That is, the Wigner function value may be a distribution value of the two-dimensional plane of the position coordinate of the four-dimensional light field.

The radon converter 155 may generate an output image by radon transforming the Wigner function.

The radon converter 155 may generate an output image of the actual image by integrating the distribution value function calculated through the Wigner function along the radian line.

In this specification, since the Wigner function is to be calculated in radians in a 4D optical field, integration of the 2D function is required twice. The radon transform may be calculated by Equation 5 below.

Accordingly, the 2D image may be reconstructed by radon transforming the Wigner distribution function, and a 3D image including depth information may be configured. The reason for this is to use the optical property that the Radon transform of the Wigner distribution function expressed in phase space has a one-to-one relationship with the Fractional Fourier Transform, which simply defines the appropriate scale factor. Radon transform allows image reconstruction.

In this case, the defined scale factor may be determined based on the focal length of the main lens and the wavelength of light, and the rotation angle of the radon transform determines the depth direction.

In other words, when the Wigner distribution function is transformed into a Radon transform that is a rotational window integral with respect to an angle, an image distribution of a certain position is obtained, and a plane image at a constant position can be obtained by multiplying an appropriate scale constant.

As described above, in the image processing apparatus of the present invention, since only three integrations are required as compared with the conventional method of integrating four-dimensional image information four times, the amount of computation can be reduced.

That is, according to the embodiment of the present invention as described above, it is possible to obtain a three-dimensional image or two-dimensional image including depth information in real time by using a combination of two-dimensional image information.

In addition, according to an embodiment of the present invention, by obtaining a three-dimensional image including the depth information by using the Wigner function transform and the Radon transform, it is possible to increase the speed of the operation by reducing the number of integral operations, accordingly Image processing can be performed.

4 is a diagram illustrating step by step an image processing method according to an exemplary embodiment.

In step S11, image information is obtained by detecting image information about the sensor pixel position from a scene delivered to different positions on the focal plane using a photosensor array.

In step S12, when the image information about the sensor pixel position is the information about the position of the pixel in the four-dimensional light field, it is applied to the Wigner function value.

In step S13, an output image is generated by radon transforming the Wigner function value.

In this case, a 3D image including arbitrary depth information may be generated using the scale factor defined during the radon transformation.

In the above, the image processing method and the like according to an embodiment of the present invention have been described. The image processing method of the present invention may be provided stored as an electronic recording code in a computer-readable recording medium.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

100: image processing device
110: main lens
120: microlens array
130: photo sensor array
140: light field data capture unit
150: data processing unit
151: image acquisition unit
153: Wigner function
155: radon converter

Claims (18)

Acquiring four-dimensional light field information consisting of a combination of positional information about a plurality of two-dimensional images;
Firstly converting the obtained 4D optical field information into a 4D phase space function; And
Reconstructing the two-dimensional image by performing quadratic transformation of the four-dimensional phase space function. The method of claim 1,
Acquiring the four-dimensional light field information,
Detecting light from the scene being delivered to different locations on the focal plane using a photo sensor array. The method of claim 2,
Reconstructing the two-dimensional image,
Reconstructing the detected light to form an image in which at least a portion of the two-dimensional image is constructed in different focal planes. The method of claim 3, wherein
Converting to the four-dimensional phase space function,
And converting four-dimensional light field information configured from the detected light into a Wigner distribution function. The method of claim 3, wherein
The four-dimensional light field information is represented by L (u, v, x, y),
The Wigner distribution function is expressed as WDF (s, t, x, y),
The s and t parameters are calculated by the following equation.
s = s`tan-1 ((ux) / D), t = t`tan-1 ((vy) / D)
Here, D is the distance value between the photosensor array and the microlens array or the distance from the two-dimensional (x, y) plane to the (u, v) plane.
s` and t` are information for matching the 4D optical field information with any constant including 1. The method of claim 3, wherein
The Wigner distribution function WDF (s, t, x, y) is calculated by the following equation.

Where k is the wave vector of the light source,

Is a function of two-dimensional image distribution on any plane. 5. The method of claim 4,
Reconstructing the two-dimensional image,
Performing the quadratic transformation;
And generating a 3D image including the 2D image or the depth information corresponding to the detected light by the performed second transformation. 8. The method of claim 7,
Performing the quadratic transformation,
And radon transforming the converted Wigner distribution function. The method of claim 8,
And defining a scale factor for the radon transform. The method of claim 9,
The scale factor is determined by the focal length of the main lens and the wavelength of light,
And a rotation angle of the radon transform determines a depth direction of the generated 3D image. An image processing apparatus for generating an image from a scene,
Main lens;
A photosensor array for capturing a set of rays;
A micro lens array between the main lens and the photo sensor array; And
Acquire four-dimensional light field information consisting of a combination of positional information about a plurality of two-dimensional images using the set of light rays captured by the photosensor array, and based on the acquired four-dimensional light field information. An image processing apparatus comprising a data processing unit for reconstructing an image. 12. The method of claim 11,
Wherein the data processing unit comprises:
An image acquisition unit for acquiring four-dimensional optical field information including a combination of positional information about the plurality of two-dimensional images;
A Wigner function unit for converting the obtained 4D optical field information into a Wigner distribution function meaning a 4D phase space function;
And a radon transform unit configured to reconstruct the 2D image by radon transforming the converted Wigner distribution function. 13. The method of claim 12,
And the image acquisition unit detects light from a scene transmitted to different positions on a focal plane using the photo sensor array. The method of claim 13,
The radon conversion unit,
And reconstruct the detected light to generate an image in which at least a portion of the two-dimensional image is constructed in different focal planes. 13. The method of claim 12,
The four-dimensional light field information is represented by L (u, v, x, y),
The Wigner distribution function is expressed as WDF (s, t, x, y),
The s and t parameters are calculated by the following equation.
s = s`tan-1 ((ux) / D), t = t`tan-1 ((vy) / D)
Here, D is the distance value between the photosensor array and the microlens array or the distance from the two-dimensional (x, y) plane to the (u, v) plane.
S` and t` are information for matching the 4D optical field information with any constant including 1. 16. The method of claim 15,
The Wigner function unit calculates the Wigner distribution function WDF (s, t, x, y) by the following equation.

Where k is the wave vector of the light source,

Is a function of two-dimensional image distribution on any plane. The method of claim 14,
And the radon conversion unit generates a 3D image including a 2D image or depth information corresponding to the detected light by radon transforming the Wigner distribution function. 18. The method of claim 17,
The radon converter performs a radon transform based on a predefined scale factor,
The scale factor is determined by the focal length of the main lens and the wavelength of light,
And a rotation angle of the radon transform determines a depth direction of the generated 3D image.

Images