WO2019021591A1 - Image processing device, image processing method, program, and image processing system - Google Patents

Abstract

An imaging system (10) that uses a plurality of imaging devices (20-1 to 20-6) and biasing elements (21-1 to 21-6) to acquire a plurality of polarized images having polarization directions and different viewpoints. An image processing device (30) classifies pixels subjected to a pixel depth calculation having texture information and pixels having no texture information according to an epipolar plane image generated from the plurality of polarized images. The epipolar plane image is used to calculate the slopes of pixels having texture information. A polarized epipolar plane image is used to calculate slopes of pixels that do not have texture information. Conversion of depth-calculated slopes not only calculates pixel depths that have texture information, but also depths of pixels that do not have texture information.

Description

IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM, AND IMAGE PROCESSING SYSTEM

This technology relates to an image processing apparatus, an image processing method, a program, and an image processing system, and performs depth calculation for an area without texture.

Conventionally, an epipolar plane image is generated from images obtained by a plurality of imaging devices to calculate depth which is distance information to a subject. For example, in Non-Patent Document 1, calculation of depth is performed based on a gradient calculated from an epipolar plane image using texture information in images obtained by a plurality of imaging devices.

Transactions on Circuits and Systems for Video Technology, Yongbing Zhang, Huijin Lv, et al, DOI 10.1109 / TCSVT.2016.2555778, "Light Field Depth Estimation via Epipolar Plane Image Analysis and Locally Linear Embedding.", [Online], [Heisei July 1, 29 Search], Internet <URL: http://media.au.tsinghua.edu.cn/liuyebin_files/lfdepth.pdf>

By the way, the method of utilizing the texture information in the image can not detect the gradient from the epipolar plane image when there is no texture. Therefore, it is not possible to calculate the depth for an area without texture.

Therefore, it is an object of the present technology to provide an image processing apparatus, an image processing method, a program, and an image processing system for calculating the depth of an area without texture.

The first aspect of this technology is
According to another aspect of the present invention, there is provided an image processing apparatus including a depth calculation unit that performs depth calculation of a depth calculation target pixel based on polarization epipolar plane images generated from a plurality of polarization imaging images having different polarization directions and viewpoint positions.

In this technique, classification is performed on the basis of an epipolar plane image generated from a plurality of polarization imaging images, in which the depth calculation target pixel is either a pixel having texture information or a pixel having no texture information. For pixels with texture information, the gradient is calculated based on the epipolar plane image, and for pixels without texture information, the gradient is calculated based on the polarization epipolar plane image, and the calculated gradient is converted to depth Thus, the depth of the depth calculation target pixel is calculated. In addition, a gradient having a calculated reliability of the gradient equal to or higher than a predetermined reliability threshold may be converted into the depth.

The gradient based on polarization information is fit to a trigonometric function using pixels on a straight line based on the position of the pixel for depth calculation in the polarization epipolar plane image, and the waveform of the trigonometric function generated by the difference in pixels used for fitting The slope of the straight line that minimizes the difference is taken as the slope of the depth calculation target pixel. Further, the polarization directions of the plurality of polarization imaging images are set to four or more directions within the range of the angle difference less than 180 degrees.

In addition, the normal to the depth calculation target pixel is calculated using the pixel located in the direction of the gradient calculated using the polarization epipolar plane image, and interpolation processing is performed using the depth and the normal to the depth calculation target pixel. , Get depth at a higher resolution than pixel units.

When the gradient of the depth calculation target pixel can not be calculated, the depth of the depth calculation target pixel is calculated by interpolation processing using the depth and the normal line of the pixel close to the depth calculation target pixel. Further, the normal to the depth calculation target pixel is calculated using a pixel located in the direction of an arbitrary gradient from the depth calculation target pixel for which the gradient can not be calculated.

Furthermore, the gradient of the pixel for depth calculation calculated based on the first polarization epipolar plane image or the first epipolar plane image generated from a plurality of polarization imaging images having different viewpoint positions in the first direction is different from the first direction Either the second polarization epipolar plane image generated from a plurality of polarization imaging images different in viewpoint position in the second direction or the gradient of the depth calculation target pixel calculated based on the second epipolar plane image is selected and selected The slope may be converted to depth. In this case, if one of the gradients is a gradient calculated based on the texture information and the other gradient is a gradient calculated based on the polarization information, the gradient calculated based on the texture information is selected. Also, if the gradients are both calculated based on the texture information, and if the gradients are both calculated based on the polarization information, then a highly reliable gradient is selected.

The second aspect of this technology is
The present invention is an image processing method including performing depth calculation of a depth calculation target pixel based on polarization epipolar plane images generated from a plurality of polarization imaging images having different polarization directions and viewpoint positions.

The third aspect of this technology is
A program that causes a computer to execute image processing using a polarization image.
Generating a polarization epipolar plane image from a plurality of polarization imaging images different in polarization direction and viewpoint position;
And a program for causing the computer to execute the steps of calculating the depth of the depth calculation target pixel based on the generated polarization epipolar plane image.

Note that the program of the present technology is, for example, a storage medium, communication medium such as an optical disc, a magnetic disc, a semiconductor memory, etc., provided in a computer readable format to a general-purpose computer capable of executing various program codes. It is a program that can be provided by a medium or a communication medium such as a network. By providing such a program in a computer readable form, processing according to the program is realized on the computer.

The fourth aspect of this technology is
An imaging device for acquiring a plurality of polarization imaging images having different polarization directions and viewpoint positions;
It has an image processing device which performs depth calculation of a depth calculation target pixel from the plurality of polarization imaging images acquired by the imaging device,
The image processing apparatus is
A polarization epipolar plane image generation unit that generates a polarization epipolar plane image including pixels for depth calculation from the plurality of polarization imaging images;
And a depth calculation unit for calculating the depth of the depth calculation target pixel based on the polarization epipolar plane image generated by the polarization epipolar plane image generation unit.

According to this technique, the depth of the depth calculation target pixel is calculated based on polarization epipolar plane images generated from a plurality of polarization imaging images having different polarization directions and viewpoint positions. For this reason,
It becomes possible to calculate the depth based on the polarization information for the area without texture. The effects described in the present specification are merely examples and are not limited, and additional effects may be present.

It is the figure which illustrated the composition of the imaging system. It is the figure which illustrated the composition of the image processing device. It is the figure which illustrated the several polarization imaging drawing, the epipolar plane image, and the polarization epipolar plane image. It is the figure which illustrated the composition of the depth calculation part in a 1st embodiment. It is a figure for demonstrating calculation of the gradient of the pixel which has texture information. It is a figure for demonstrating the relationship between a polarization direction and a pixel value (luminance). It is the figure which illustrated the relationship between luminance and a polarization angle. It is a figure for demonstrating the case where gradient can be calculated using polarization information. It is a flowchart which shows operation | movement of 1st Embodiment. It is the flowchart which illustrated depth calculation operation. It is the figure which illustrated the composition of the depth calculation part in a 2nd embodiment. It is the figure which illustrated the relationship between polarization degree and zenith angle. It is a figure for demonstrating the depth interpolation process based on a depth and a normal line. It is a flowchart which shows operation | movement of 2nd Embodiment. It is the figure which illustrated the composition of the depth calculation part in a 3rd embodiment. It is a figure for demonstrating a depth interpolation process. It is a flowchart which shows operation | movement of 3rd Embodiment. It is the figure which illustrated the composition of the imaging device in a 4th embodiment. It is the figure which illustrated the polarization direction of the imaging device in a 4th embodiment. It is the figure which illustrated the composition of the depth calculation part in a 4th embodiment. It is a figure for demonstrating gradient selection operation | movement. It is a flowchart which shows operation | movement of 4th Embodiment. It is a block diagram showing an example of rough composition of a vehicle control system. It is explanatory drawing which shows an example of the installation position of a vehicle exterior information detection part and an imaging part.

Hereinafter, modes for carrying out the present technology will be described. The description will be made in the following order.
1. Configuration of imaging system First embodiment 3. Second embodiment 4. Third embodiment 5. Fourth embodiment 6. Other configuration and operation 7. Application example

<1. Configuration of Imaging System>
FIG. 1 illustrates the configuration of an imaging system using the image processing device of the present technology. The imaging system 10 is configured using an imaging device that acquires a plurality of polarization imaging images different in polarization direction and viewpoint position, and an image processing device that performs image processing using the acquired polarization imaging image. In an imaging device, polarizing elements, such as a polarizing plate or a polarizing filter, are provided in front of an imaging lens or an imaging element to obtain a plurality of polarized images. In the imaging device, for example, a plurality of imaging units are arranged linearly in parallel, and the polarization directions of the polarization elements are set in different directions in each imaging unit. When the polarization direction is rotated by 180 degrees, the polarization direction becomes equal to that before rotation, so the polarization direction of the polarizing element is set to an angle of 0 degrees or more and less than 180 degrees. In FIG. 1, for example, six imaging units 20-1 to 20-6 are linearly arranged in parallel. In addition, the polarization elements 21-1 to 21-6 of the imaging units 20-1 to 20-6 are set so that the angle difference of the polarization direction with the adjacent imaging units is, for example, 30 degrees and the angle of the polarization direction increases. ing. The six imaging units 20-1 to 20-6 perform imaging, for example, by including a rectangular columnar object OBa and a cylindrical object OB provided at a position farther than the object OBa in the imaging range. Polarized imaging images different in position and polarization direction from each other are acquired and output to the image processing device 30. The imaging apparatus acquires four or more polarization imaging images different in viewpoint position and polarization direction as described later.

The image processing apparatus 30 generates polarization epipolar plane images and epipolar plane images from a plurality of polarization imaging images having different polarization directions and viewpoint positions, and uses the generated polarization epipolar plane images to obtain distance information to the subject. Calculate the depth.

<2. First embodiment>
Next, a first embodiment of the image processing apparatus will be described. FIG. 2 illustrates the configuration of the image processing apparatus. The image processing apparatus 30 includes a preprocessing unit 31, a parameter holding unit 32, a polarization epipolar plane image generation unit 33, and a depth calculation unit 34.

The preprocessing unit 31 uses the parameters stored in the parameter storage unit 32 to perform preprocessing on the polarization imaging image acquired by the imaging device. The parameter holding unit 32 stores in advance internal parameters and external parameters obtained by performing calibration using a predetermined subject such as a checker board. The internal parameter is a parameter unique to each imaging unit, and is a focal length of the imaging unit, a lens distortion coefficient, or the like. The external parameter is a parameter for specifying the arrangement of the imaging unit, and indicates parallel movement and rotation. The preprocessing unit 31 performs distortion correction or the like of the polarization imaging image using the polarization imaging image and an internal parameter of the imaging unit that has acquired the polarization imaging image. Further, the preprocessing unit 31 performs registration of the polarization imaging image using the polarization imaging image processed using the internal parameter and the external parameter. Therefore, the pre-processed polarization captured image has no positional displacement in the vertical direction of the pixels indicating the desired position of the subject between the images, and causes lateral positional displacement in accordance with the difference in depth which is distance information to the subject. Image. The preprocessing unit 31 outputs the polarization imaging image after the preprocessing to the polarization epipolar plane image generating unit 33.

The polarization epipolar plane image generation unit 33 generates a plurality of polarization imaging images having different polarization directions and viewpoint positions so as to include an image indicating a desired position on the subject, in the arrangement direction of viewpoint positions, for example, in the horizontal direction in the configuration shown in FIG. An image is extracted, and an epipolar plane image is generated by arranging the extracted images in the order of the viewpoint position in the vertical direction orthogonal to the alignment direction, at the interval according to the interval of the viewpoint positions (the interval between the imaging units (Baseline)) Do. In addition, the polarization epipolar plane image generation unit 33 generates the polarization epipolar plane image by arranging the extracted images in the order of the viewpoint positions in the vertical direction orthogonal to the arrangement direction, at the intervals according to the intervals of the viewpoint positions. Each pixel of the polarization epipolar plane image has polarization direction information indicating the polarization direction of the polarization imaging image to be extracted.

FIG. 3 illustrates a plurality of polarization imaging images and an epipolar plane image and a polarization epipolar plane image. FIG. 3A illustrates the polarization directions of the imaging units 20-1 to 20-6. For example, the polarization element 21-1 provided in the imaging unit 20-1 has an angle indicating “0 °” indicating the polarization direction, and the polarization element 21-2 provided in the imaging unit 20-2 has a polarization direction. The angle is "30 °". Similarly, the polarization elements provided in the other imaging units are also set so that the angle difference between the adjacent imaging units in the polarization direction is 30 degrees and the polarization direction angle is increased. Further, a distance ds is set between the respective imaging units, and for example, with the imaging unit 20-1 as a reference, the imaging unit 20-6 is positioned at a distance 5ds.

(B) of FIG. 3 shows polarization captured images acquired by the imaging units 20-1 to 20-6, and indicates the position of the image indicating the desired position on the subject by a broken line. For example, the position PS1 of the side of the prismatic object OBa, the position PS2 of the side surface of the cylindrical object OBb provided at a position farther than the object OBa, and the position PS3 in the side of the prismatic object OBa The following description will be made assuming that is the desired position on the subject. In each polarization imaging image, the left end portion in the image at the position indicated by the broken line is taken as the reference pixel position in the lateral direction in the epipolar plane image and the polarization epipolar plane image.

(C) of FIG. 3 shows an epipolar plane image. In the epipolar plane image, the horizontal axis is the position in the horizontal direction in the polarization imaging image, and the vertical axis is the physical distance of the viewpoint position (physical distance between the imaging units).

FIG. 3 (d) shows a polarized epipolar plane image. In the polarization epipolar plane image, the horizontal axis is the position in the lateral direction in the polarization imaging image, and the vertical axis is the physical distance of the viewpoint position (physical distance between imaging units).

In the epipolar plane image shown in (c) of FIG. 3, the line connecting the points (PS1-1 to PS1-6) at the position PS1 in each polarization imaging image is a straight line since the intervals of the viewpoint positions are equal. Further, in the polarization epipolar plane image shown in (d) of FIG. 3, the line connecting the points (PS1-1 to PS1-6) at the position PS1 in each polarization imaging image is a straight line since the distance between the viewpoint positions is equal. . Also, a line connecting points PS2 and PS3 is a straight line. When the depth which is distance information from the imaging unit to the desired position is large, the inclination of the straight line with respect to the vertical axis is small, and when the depth is small, the inclination of the straight line with respect to the vertical axis is large. Therefore, at positions PS1 to PS3, when the depth to position PS1 is the smallest, the slope of the straight line connecting the points of position PS1 to the vertical axis is the largest, and when the depth to position PS2 is the largest, the inclination to the vertical axis The slope of the straight line connecting the points of position PS2 is the smallest.

Returning to FIG. 2, the polarization epipolar plane image generation unit 33 generates an epipolar plane image and a polarization epipolar plane image using the plurality of pre-processed polarization imaging images, and outputs the image to the depth calculation unit 34.

The depth calculation unit 34 calculates the depth using the epipolar plane image generated by the polarization epipolar plane image generation unit 33 and the polarization epipolar plane image. The depth calculating unit 34 calculates the depth at the position indicating the texture based on the epipolar plane image, and calculates the depth at the position not having the texture based on the polarized epipolar plane image.

FIG. 4 illustrates the configuration of the depth calculation unit. The depth calculation unit 34 includes a slope calculation unit 341 and a depth conversion unit 343.

The gradient calculation unit 341 calculates the gradient of each pixel in the epipolar plane image generated by the polarization epipolar plane image generation unit 33 and the polarization epipolar plane image. The gradient calculating unit 341 includes a pixel classification unit 3411, a texture gradient calculating unit 3412, and a polarization gradient calculating unit 3413.

The pixel classification unit 3411 classifies the pixels of the epipolar plane image into pixels having texture information and an image having no texture information. The pixel classification unit 3411 uses a first derivative filter such as a Sobel filter or a Prewitt filter, or a second derivative filter such as a Laplacian filter, for the determination target pixel, to generate the horizontal differential value Iu and the vertical differential value Is. Calculate Also, the pixel classification unit 3411 calculates the texture determination value G by performing the calculation of Expression (1) using the horizontal differential value Iu and the vertical differential value Is.

If the calculated texture determination value G is equal to or greater than a predetermined determination threshold σ, the pixel classification unit 3411 determines that the pixel has texture information, and the texture determination value G is smaller than the determination threshold σ. In this case, it is determined that the pixel does not have texture information.

The texture gradient calculation unit 3412 calculates the gradient of the image classified as the pixel having the texture information by the pixel classification unit 3411 using the texture information (color information and texture information of the texture). The texture gradient calculation unit 3412 calculates the gradient θt by performing the calculation of Equation (2) using, for example, the horizontal differential value Iu and the vertical differential value Is calculated by the pixel classification unit 3411.

Also, the texture gradient calculation unit 3412 may calculate the gradient θt with high accuracy by using pixels in the vicinity region of the gradient calculation target pixel. FIG. 5 is a diagram for explaining the case where the gradient of a pixel having texture information is calculated with high accuracy. For example, the optimal gradient is a direction in which the difference between the pixel value of the calculation target pixel O and the differential value of the differential value with respect to the pixel q in a certain direction is smallest in the vicinity region to the calculation target pixel O. Specifically, the direction satisfies the equation (3). In Expression (3), the evaluation value C _θ (O) is a value calculated by Expression (4), and the larger the evaluation value C _θ (0), the lower the reliability.

In equation (3), “Θ” is a set (0, 180) of gradient directions. In Expression (4), “S (θ)” is a set of pixels in the direction along the gradient θt in the vicinity region of the calculation target pixel O. Also, “I (O)” is the pixel value of the calculation target pixel, “Iu (O), Is (O)” is the differential value of the calculation target pixel, and “I (q)” is the pixel value of the pixel q in the vicinity region. , “Iu (q), Is (q)” are differential values of the pixel q in the near region. “Α” is a parameter for adjusting the weight of the pixel value difference and the gradient difference.

The polarization gradient calculation unit 3413 calculates the gradient using polarization information for an image classified by the pixel classification unit 3411 as a pixel having no texture information. Here, the relationship between the polarization direction and the pixel value (brightness) will be described. FIG. 6 is a diagram for explaining the relationship between the polarization direction and the pixel value (brightness). The light source LT is used to illuminate the subject OB, and the subject OB is imaged by the imaging device CM via the polarization element PL. In this case, it is known that, in the polarization imaging image generated by the imaging device CM, the luminance of the subject OB changes in accordance with the rotation of the polarization element PL. Here, the highest luminance when the polarizing element PL is rotated is Imax, and the lowest luminance is Imin. When the x-axis and y-axis in two-dimensional coordinates are the plane direction of the polarizing element PL, an angle on the xy plane with respect to the x-axis when the polarizing element PL is rotated is taken as a polarization angle υ. The polarization element PL returns to the original polarization state when rotated 180 degrees and has a cycle of 180 degrees. Further, in the case of the diffuse reflection model, the polarization angle と き when the maximum luminance Imax is observed is taken as the azimuth angle φ. If such a definition is made, the luminance I observed when the polarizing element PL is rotated can be expressed as Expression (5). That is, the change in luminance caused when the polarization direction of the polarizing element is changed at the position where the polarization angle and the azimuth angle are equal shows a waveform of a trigonometric function. Also, the phase of the trigonometric function waveform changes depending on the polarization angle and the azimuth angle. FIG. 7 illustrates the relationship between the luminance and the polarization angle.

When the gradient is calculated for the depth calculation target pixel, the pixels located in the gradient direction when the gradient is a true value are physically the same point viewed from different viewpoint positions. For this reason, when the pixel values of the pixels positioned in the direction of the gradient (which is also referred to as the “gradient of the depth calculation object pixel”) which is a true value are plotted in the polarization direction of the polarizing element, it becomes a waveform of trigonometric function (for example, Sin wave). Furthermore, even when the combination of the polarization directions when obtaining the waveform of the trigonometric function from the pixel values of the pixels located in the direction of the gradient of the true value is different, the phase and the amplitude of the waveform of the trigonometric function are equal. However, pixels located in directions different from the gradient of the true value are physically different points when viewed from different viewpoint positions, so when the orientation of the surface of the object changes smoothly, in the direction different from the gradient of the true value If the combination of the polarization directions in obtaining the waveform of the trigonometric function from the pixel values of the located pixels is different, the phase and the amplitude of the waveform of the trigonometric function are different. Therefore, the polarization gradient calculating unit 3413 sets a gradient that minimizes the difference in the waveform of the trigonometric function obtained in the combination in which the polarization directions are different as the gradient of the pixel for which the depth is to be calculated. In addition, in order to fit the change in brightness that occurs when the polarization direction is changed to the waveform of the trigonometric function, pixel values (brightness values) in three or more polarization directions are required, and combinations of polarization directions are different. Since the true value of the gradient is calculated based on the difference in the waveform of the trigonometric function, polarization imaging drawings having four or more polarization directions within the range of angle differences less than 180 degrees are used as the plurality of polarization imaging drawings.

FIG. 8 is a diagram for explaining the case where the gradient can be calculated using polarization information. (A) of FIG. 8 shows the gradient of the true value by a solid line, and shows the case where the gradient is incorrect by a broken line. Here, when six imaging units are used as shown in FIG. 1, the polarization direction is “0 °, 30 °, 60 °, 90 °, 120 °, 120 °, 150 °” in the longitudinal direction in the polarization epipolar plane image. A pixel value of 6 pixels is obtained.

The pixel value of each polarization direction when the gradient is incorrect indicates, for example, different positions of the subject as shown in FIG. 8B. Therefore, if the combination of the polarization directions when obtaining the waveform of the trigonometric function from the pixel values is different, the phase and the amplitude of the trigonometric function are different as shown in (c) of FIG. For example, a triangular function waveform obtained from pixel values of three pixels whose polarization directions are "0 °, 60 °, 120 °" and three pixel pixels whose polarization directions are "30 °, 90 °, 150 °" The waveforms of trigonometric functions determined from the values differ in phase and amplitude. However, since the pixel value in each polarization direction when the gradient is a true value indicates the same position of the subject as shown in (d) of FIG. 8, the polarization direction in obtaining the waveform of the trigonometric function from the pixel value Even if the combination of pixel values is different, as shown in (e) of FIG. 8, the phase and amplitude of the trigonometric function match. For example, a triangular function waveform obtained from pixel values of three pixels whose polarization directions are "0 °, 60 °, 120 °" and three pixel pixels whose polarization directions are "30 °, 90 °, 150 °" The waveforms of the trigonometric functions obtained from the values have the same phase and amplitude.

Therefore, the polarization gradient calculating unit 3413 calculates a gradient that minimizes the difference in the waveform of the trigonometric function when the combination of the polarization directions when obtaining the waveform of the trigonometric function is different from the pixel value. For example, the polarization gradient calculating unit 3413 generates a histogram in which the horizontal axis indicates the angle of inclination and the vertical axis indicates the difference, and sets the angle of inclination at which the difference is minimum as the gradient of the true value. In addition, the polarization gradient calculating unit 3413 may determine that the gradient can not be calculated if the difference between the maximum value and the minimum value of the difference is smaller than a preset threshold.

The combination of pixels used to detect the amplitude and phase of the waveform of the trigonometric function is not limited to the case where pixels are selected every other pixel. For example, the polarization gradient calculation unit 3413 may detect the amplitude and phase of the waveform of the trigonometric function for each set, with the pixel values at successive positions being a set. Also, the polarization gradient calculation unit 3413 may select pixels at random and make a set, and detect the amplitude and phase of the waveform of the trigonometric function for each set. Furthermore, by using pixels at distant positions, the pixels used for the combination can be less susceptible to noise compared to using pixels at successive positions. The influence of noise or the like can be reduced by increasing the angle difference in the polarization direction.

Also, the polarization gradient calculation unit 3413 detects the phase and amplitude of the waveform of the trigonometric function based on the pixel values of pixels randomly selected from pixels located in the direction of an arbitrary gradient, for example, to reduce the influence of noise. Is repeated a plurality of times to calculate the variance (for example, standard deviation) of the phase and amplitude of the waveform. Furthermore, the polarization gradient calculating unit 3413 repeatedly calculates the dispersion of the phase and amplitude of the waveform of the trigonometric function with the gradient within the range of 0 ° to 180 °. Further, the polarization gradient calculating unit 3413 generates a histogram in which the horizontal axis indicates the angle of the gradient and the vertical axis indicates the dispersion, and determines the gradient with the smallest dispersion as the gradient of the true value.

The polarization gradient calculating unit 3413 determines the angle difference between the angle at which the phase difference is minimum and the angle at which the difference in amplitude is minimum, or the angle between the angle at which the phase dispersion is minimum and the angle at which the amplitude dispersion is minimum. When the difference is larger than a preset threshold value, it may be determined that the pixel can not calculate the gradient. In addition, when the angle difference is within the threshold, the polarization gradient calculating unit 3413 may set either one of the angles as the true value of the gradient, and the difference between the angle and the amplitude at which the phase difference (dispersion) becomes minimum (dispersion The average value of weighted angles and the weighted average value or the like at which) is minimized may be used as the true value of the gradient.

Although three points are used in FIG. 8 to calculate the phase and amplitude of the waveform of the trigonometric function, the polarization gradient calculating unit 3413 may use more pixel values than three points. Furthermore, the difference or variance of the phase and amplitude of the waveform of the trigonometric function in the true value gradient may be used as the evaluation value of the gradient. In this case, the larger the difference or variance in the gradient of the true value, the smaller the reliability.

By the way, when the calculation target pixel of the gradient indicates a point on the surface where there is no change in the surface, the amplitude and the phase of the trigonometric function waveform are constant regardless of the gradient, and the phase is compared with the point on the changing surface. And the variance of the amplitude is too small to calculate the slope of the true value. Further, as will be described in the second embodiment described later, the normal direction can be calculated based on the pixel values of the polarization image with three or more polarization directions. Therefore, in the present technology, in pixels that do not have texture information, it is a pixel that can not calculate the gradient of the true value, and the amplitude and phase of the trigonometric function waveform are constant regardless of the gradient. A pixel with a small variance is taken as a pixel with normal and not calculated gradient. In addition, in pixels that do not have texture information, pixels that are different from normal pixels and have not calculated gradients, and pixels whose reliability of the gradient is lower than a predetermined reliability threshold value are pixels that have no normal gradients and that have not been calculated. I assume.

Returning to FIG. 4, the depth conversion unit 343 converts the gradients calculated by the texture gradient calculation unit 3412 and the polarization gradient calculation unit 3413 of the gradient calculation unit 341 into depths. The depth conversion unit 343 performs the calculation of Expression (6) using the gradient θp calculated by the polarization gradient calculation unit 3413, and calculates the depth D to the object position corresponding to the pixel for which the gradient θx is calculated. In equation (6), “f” is the focal length of the pixel for which the gradient is calculated, and the gradient θx is the gradient θt or the gradient θp. In addition, when the reliability of the gradient is equal to or greater than a predetermined reliability threshold, the depth conversion unit 343 can prevent the calculation of the depth with low reliability by converting the calculated gradient to a depth. .

FIG. 9 is a flowchart showing the operation of the first embodiment. In step ST1, the image processing apparatus acquires a plurality of polarization imaging images. For example, the image processing apparatus 30 acquires polarized captured images from a plurality of imaging units, and proceeds to step ST2.

In step ST2, the image processing apparatus performs pre-processing of the polarization image. The image processing apparatus 30 performs distortion correction and the like on each of the plurality of polarization imaging images acquired from the plurality of imaging units using an internal parameter corresponding to the imaging unit. Furthermore, the image processing apparatus 30 performs registration using the external parameter on each polarization imaging image processed using the internal parameter, and proceeds to step ST3.

In step ST3, the image processing apparatus generates an epipolar plane image and a polarization epipolar plane image. The image processing apparatus 30 generates, for example, an epipolar plane image and a polarization epipolar plane image by extracting an image of a depth calculation target line from the pre-processed polarization captured image and stacking the images in order of distance of viewpoint position. Each pixel in the polarization epipolar plane image has information indicating the polarization direction. The image processing device 30 generates an epipolar plane image and a polarization epipolar plane image, and proceeds to step ST4.

In step ST4, the image processing apparatus performs depth calculation. The image processing device 30 calculates the depth to the subject position indicated by the depth calculation target pixel in the depth calculation target line based on the epipolar plane image and the polarization epipolar plane image.

FIG. 10 is a flowchart illustrating the depth calculation operation in step ST4. In step ST11, the image processing apparatus performs pixel classification. The image processing device 30 calculates the texture determination value G using the pixel value. Furthermore, the image processing apparatus 30 determines whether the pixel has texture information or not based on the calculated texture determination value G, and proceeds to step ST12.

In step ST12, the image processing apparatus calculates the gradient using the texture information. The image processing apparatus 30 calculates the gradient θt using the texture information for the pixel determined to have the texture information, and proceeds to step ST13.

In step ST13, the image processing apparatus calculates a gradient using polarization information. The image processing apparatus 30 calculates the gradient θp using polarization information for the pixel determined to have no texture information, and proceeds to step ST14.

In step ST14, the image processing apparatus calculates the depth. The image processing device 30 converts the gradient θx calculated in step ST12 and step ST13 into depth. The gradient θx is the gradient θt or the gradient θp.

The image processing apparatus performs the above-described processing, and calculates the depth on the basis of the gradient calculated using the texture information, for example, for the position PS1 of the side having the texture in the object OBa. Also, for example, with respect to the position PS3 of the side surface having no texture in the object OBb, the depth is calculated based on the gradient calculated using the polarization information.

<3. Second embodiment>
Next, a second embodiment of the image processing apparatus will be described. In the second embodiment, the case where depth calculation is performed at a resolution higher than that of the first embodiment will be described.

The image processing apparatus according to the second embodiment has the preprocessing unit 31, the parameter holding unit 32, the polarization epipolar plane image generation unit 33, and the depth calculation unit 34, as in the first embodiment shown in FIG. doing.

The preprocessing unit 31 uses the parameters stored in the parameter storage unit 32 to perform preprocessing on polarization imaging images acquired by a plurality of imaging units. The parameter holding unit 32 stores in advance internal parameters and external parameters obtained by performing calibration using a predetermined subject such as a checker board. The preprocessing unit 31 performs distortion correction or the like of the polarization imaging image using the polarization imaging image and an internal parameter of the imaging unit that has acquired the polarization imaging image. Further, the preprocessing unit 31 performs registration of the polarization imaging image using the polarization imaging image processed using the internal parameter and the external parameter. The preprocessing unit 31 outputs the polarization imaging image after the preprocessing to the polarization epipolar plane image generating unit 33.

The polarization epipolar plane image generation unit 33 extracts an image in the arrangement direction of the plurality of imaging units, that is, the arrangement direction of the plurality of viewpoint positions so as to include an image indicating a desired position in the subject from the plurality of polarization imaging images Then, the epipolar plane image is generated by arranging the extracted images in the order of the imaging units in the vertical direction orthogonal to the alignment direction, at the intervals according to the intervals of the imaging units (the intervals of the viewpoint positions). In addition, the polarization epipolar plane image generation unit 33 generates the polarization epipolar plane image by arranging the extracted images in the order of the viewpoint positions in the vertical direction orthogonal to the arrangement direction, at the intervals according to the intervals of the viewpoint positions. The polarization epipolar plane image generation unit 33 generates an epipolar plane image and a polarization epipolar plane image using the plurality of pre-processed polarization imaging images, and outputs the epipolar plane image and the polarization epipolar plane image to the depth calculation unit 34.

In the second embodiment, the configuration of the depth calculation unit is different from that of the first embodiment. The depth calculating unit in the second embodiment calculates the normal from the pixel value of the pixel located in the direction of the gradient of the true value in the polarization epipolar plane image, and uses the calculated normal to execute the first embodiment. Perform depth calculation with higher resolution than the form.

FIG. 11 illustrates the configuration of the depth calculation unit according to the second embodiment. The depth calculation unit 34 includes a gradient calculation unit 341, a depth conversion unit 343, a normal calculation unit 344, and an integration processing unit 345.

The gradient calculation unit 341 calculates the gradient of each pixel in the epipolar plane image generated by the polarization epipolar plane image generation unit 33 and the polarization epipolar plane image. The gradient calculating unit 341 includes a pixel classification unit 3411, a texture gradient calculating unit 3412, and a polarization gradient calculating unit 3413.

The pixel classification unit 3411 classifies the pixels of the epipolar plane image into pixels having texture information and an image having no texture information. The pixel classification unit 3411 calculates the texture determination value G as described above for the determination target pixel. If the calculated texture determination value G is equal to or greater than a predetermined determination threshold σ, the pixel classification unit 3411 determines that the pixel has texture information, and the texture determination value G is smaller than the determination threshold σ. In this case, it is determined that the pixel does not have texture information.

The texture gradient calculation unit 3412 calculates the gradient θt as described above using the texture information with respect to the image classified as the pixel having the texture information by the pixel classification unit 3411. The texture gradient calculation unit 3412 outputs the calculated gradient to the depth conversion unit 343.

The polarization gradient calculation unit 3413 calculates the gradient using polarization information for pixels that do not have the texture information classified by the pixel classification unit 3411. As described above, the polarization gradient calculation unit 3413 calculates the waveform of the trigonometric function for each different combination of pixels located in the direction of the gradient in the polarization epipolar plane image, and determines the gradient with the smallest waveform difference as the true value. It is a gradient. The polarization gradient calculation unit 3413 outputs the calculated gradient of the true value to the depth conversion unit 343. In addition, the polarization gradient calculating unit 3413 outputs the pixel value of the pixel at which the difference in amplitude and the difference in phase are minimum, that is, the pixel value of the pixel positioned in the direction of the gradient of the true value to the normal calculation unit 344. Furthermore, the polarization gradient calculating unit 3413 outputs not only the pixel values of the pixels positioned in the direction of the gradient of the true value, but also the pixel values of the pixels positioned in the direction of the arbitrary gradient to the normal calculation unit 344.

The depth conversion unit 343 converts the gradients calculated by the texture gradient calculation unit 3412 and the polarization gradient calculation unit 3413 of the gradient calculation unit 341 into depths. The depth conversion unit 343 converts the gradient θp calculated by the texture gradient calculation unit 3412 and the polarization gradient calculation unit 3413 into depth as described above. The depth conversion unit 343 outputs the converted depth to the normal calculation unit 344 and the integration processing unit 345.

The normal line calculation unit 344 calculates a normal line using the pixel value (brightness) supplied from the polarization gradient calculation unit 3413. As described above, the pixel value observed when the polarization direction is rotated can be expressed as Expression (5).

In equation (5), the polarization angle 明 ら か is apparent when generating a polarization image, and the maximum luminance Imax, the minimum luminance Imin and the azimuth angle φ become variables. Therefore, since there are three variables, the normal line calculation unit 344 performs fitting to the function shown in equation (5) using the brightness of the polarization image with three or more polarization directions, and The azimuth angle φ at which the maximum brightness is obtained is determined based on the function indicating the relationship.

Further, the object surface normal is expressed in a polar coordinate system, and the normal information is an azimuth angle φ and a zenith angle θz. The zenith angle θz is an angle from the z-axis toward the normal, and the azimuth angle φ is an angle in the y-axis direction with respect to the x-axis as described above. Here, even if the minimum luminance Imin and the maximum luminance Imax obtained by rotating the polarizing element PL are used, the degree of polarization ρ can be calculated by performing the operation of equation (7).

The relationship between the degree of polarization and the zenith angle is known to have, for example, the characteristics shown in FIG. 12 from the Fresnel equation, and the zenith angle θz can be determined based on the degree of polarization か ら from the characteristics shown in FIG. The characteristic shown in FIG. 12 is an example, and the characteristic changes depending on the refractive index of the subject.

Therefore, the normal line calculation unit 344 obtains the relationship between the brightness and the polarization angle from the polarization direction and the brightness of the polarization image based on the pixel values of the polarization image with three or more polarization directions, and obtains the azimuth angle φ To determine The normal line calculation unit 344 calculates the degree of polarization ρ using the maximum brightness and the minimum brightness obtained from the relationship between the brightness and the polarization angle, and calculates it based on the characteristic curve indicating the relationship between the degree of polarization and the zenith angle. The zenith angle θz corresponding to the polarization degree ρ is determined. Thus, the normal line calculation unit 344 generates normal line information indicating the normal direction of the subject (the azimuth angle φ and the zenith angle θz) based on the pixel values of the polarization image having three or more polarization directions. Here, when calculating the normal using the pixel value of the pixel located in the direction of the gradient of the true value, the normal calculation unit 344 outputs normal information indicating the calculated normal direction to the integration processing unit 345. Do. Although the azimuth of the normal line has an indeterminacy of 180 °, if the gradient of the object surface is estimated based on the depth obtained by the depth conversion unit 343, the indeterminacy of the azimuth can be eliminated.

The integration processing unit 345 performs integration processing using the depth obtained by the depth conversion unit 343 and the normal calculated by the normal calculation unit 344 to obtain the depth with high resolution and high accuracy. The integration processing unit 345 is denser and higher than the pixel unit based on the depth and normal of the pixel for which the gradient of the true value is calculated based on the polarization information, or the normal of the pixel for which the gradient of the true value is not calculated. Calculate depth with high resolution and high accuracy.

FIG. 13 is a diagram for explaining depth interpolation processing based on the depth and the normal. For example, it is assumed that the depth D1 of the pixel position PX1 and the depth D2 of the pixel position PX2 are obtained by the depth conversion unit 343. Further, it is assumed that the normal line calculation unit 344 obtains the normal line F1 of the pixel position PX1 and the normal line F2 of the pixel position PX2. The integration processing unit 345 performs interpolation processing using the depths D1 and D2 and the normals F1 and F2, and calculates, for example, the depth D12 of the object position corresponding to the boundary position PX12 between the pixel position PX1 and the pixel position PX2.

The relationship between the three-dimensional vector of the normal, the zenith angle θz of the normal and the azimuth angle φ is as shown in equations (8) to (10).

Here, assuming that the arrangement direction of the pixel positions PX1 and PX2 is the x direction on the assumption that the orthogonal projection is performed, the relationships of the equations (11) and (12) hold. Therefore, the depth D12 can be calculated based on the equation (13).

In the case of perspective projection, the coordinate of the pixel position PX1 in the image coordinate system with the focal length of the imaging unit as “f” and the image center as the origin is (u1, v1), and the coordinate of the pixel position PX2 is (u2, v2) The normal vector based on the normal F1 is (Nx1, Ny1, Nz1), the normal vector based on the normal F2 is (Nx2, Ny2, Nz2), the pixel position PX1, PX2 is the u direction of the image coordinate system (rightward) In the case of the upper two points, the relationships of equations (14) and (15) hold. Therefore, the depth D12 can be calculated based on the equation (16).

Note that the depth D12 'indicates the depth at the boundary position PX12 calculated by linear interpolation using the depth D1 and the depth D2, and by performing the depth interpolation processing based on the normal indicated by the depth and the normal information The depth D12 can be calculated with high accuracy as compared to the case of calculating the depth by linear interpolation without using the normal line information.

In addition, the integration processing using the normal is, for example, a gradient with a normal which is a pixel for which normal information is generated and a depth is not obtained by performing integration processing in the same manner as in JP-A-2015-114307. The depth of the uncalculated pixel can be calculated with high accuracy.

As described above, the integration processing unit 345 performs integration processing using the depth and the normal line or the normal line, and calculates the depth with higher resolution and high accuracy than the pixel unit.

FIG. 14 is a flow chart showing the operation of the second embodiment. In step ST21, the image processing device performs pixel classification. The image processing device 30 calculates the texture determination value using the pixel value. Furthermore, the image processing apparatus 30 determines whether the pixel has texture information or not based on the calculated texture determination value, and proceeds to step ST22.

In step ST22, the image processing apparatus calculates the gradient using the texture information. The image processing apparatus 30 calculates the gradient using the texture information for the pixel determined to have the texture information, and proceeds to step ST23.

In step ST23, the image processing apparatus calculates a gradient using polarization information. The image processing apparatus 30 calculates the gradient of the true value using the polarization information for the pixel determined to have no texture information, and proceeds to step ST24.

In step ST24, the image processing apparatus calculates the depth. The image processing device 30 calculates the depth based on the gradients calculated in step ST12 and step ST13, and proceeds to step ST25.

In step ST25, the image processing apparatus generates normal line information. The image processing device 30 calculates the normal based on the polarization information when the gradient is calculated in step ST23, generates normal information indicating the calculated normal, and proceeds to step ST26.

In step ST26, the image processing apparatus performs integration processing. The image processing device 30 performs integration processing using the depth obtained in the pixel unit and the normal indicated by the normal information, and calculates the depth with a higher resolution and higher accuracy than the pixel unit.

Thus, according to the second embodiment, it is possible to calculate the depth with higher resolution and accuracy than the first embodiment.

<4. Third embodiment>
Next, a third embodiment of the image processing apparatus will be described. In the third embodiment, the depth can be calculated for a depth calculation target pixel whose gradient can not be calculated based on texture information or polarization information.

The image processing apparatus according to the third embodiment has the preprocessing unit 31, the parameter holding unit 32, the polarization epipolar plane image generation unit 33, and the depth calculation unit 34, as in the first embodiment shown in FIG. doing.

The preprocessing unit 31 uses the parameters stored in the parameter storage unit 32 to perform preprocessing on polarization imaging images acquired by a plurality of imaging units. The parameter holding unit 32 stores in advance internal parameters and external parameters obtained by performing calibration using a predetermined subject such as a checker board. The preprocessing unit 31 performs distortion correction or the like of the polarization imaging image using the polarization imaging image and an internal parameter of the imaging unit that has acquired the polarization imaging image. Further, the preprocessing unit 31 performs registration of the polarization imaging image using the polarization imaging image processed using the internal parameter and the external parameter. The preprocessing unit 31 outputs the polarization imaging image after the preprocessing to the polarization epipolar plane image generating unit 33.

The polarization epipolar plane image generation unit 33 extracts an image in the arrangement direction of the plurality of imaging units, that is, the arrangement direction of the plurality of viewpoint positions so as to include an image indicating a desired position in the subject from the plurality of polarization imaging images Then, the epipolar plane image is generated by arranging the extracted images in the order of the imaging units in the vertical direction orthogonal to the alignment direction, at the intervals according to the intervals of the imaging units (the intervals of the viewpoint positions). In addition, the polarization epipolar plane image generation unit 33 generates the polarization epipolar plane image by arranging the extracted images in the order of the viewpoint positions in the vertical direction orthogonal to the arrangement direction, at the intervals according to the intervals of the viewpoint positions. The polarization epipolar plane image generation unit 33 generates an epipolar plane image and a polarization epipolar plane image using the plurality of pre-processed polarization imaging images, and outputs the epipolar plane image and the polarization epipolar plane image to the depth calculation unit 34.

The third embodiment is different from the first and second embodiments in the configuration of the depth calculating unit. The depth calculation unit in the third embodiment further performs processing of calculating the depth of the depth calculation target pixel for which the gradient can not be calculated based on the texture information and the polarization information.

FIG. 15 illustrates the configuration of the depth calculator in the third embodiment. The depth calculation unit 34 includes a gradient calculation unit 341, a depth conversion unit 343, a normal calculation unit 344, an integration processing unit 345, and a depth interpolation unit 346.

The gradient calculation unit 341 calculates the gradient of each pixel in the epipolar plane image generated by the polarization epipolar plane image generation unit 33 and the polarization epipolar plane image. The gradient calculating unit 341 includes a pixel classification unit 3411, a texture gradient calculating unit 3412, and a polarization gradient calculating unit 3413.

The pixel classification unit 3411 classifies the pixels of the epipolar plane image into pixels having texture information and an image having no texture information. The pixel classification unit 3411 calculates the texture determination value G as described above for the determination target pixel. If the calculated texture determination value G is equal to or greater than a predetermined determination threshold σ, the pixel classification unit 3411 determines that the pixel has texture information, and the texture determination value G is smaller than the determination threshold σ. In this case, it is determined that the pixel does not have texture information.

The texture gradient calculation unit 3412 calculates the gradient using the texture information for the pixels having the texture information classified by the pixel classification unit 3411. The texture gradient calculating unit 3412 calculates the gradient θt as described above using, for example, the differential value calculated by the pixel classification unit 3411. The texture gradient calculation unit 3412 outputs the calculated gradient to the depth conversion unit 343.

The texture gradient calculation unit 3412 calculates the gradient θt as described above using the texture information with respect to the image classified as the pixel having the texture information by the pixel classification unit 3411. The texture gradient calculation unit 3412 outputs the calculated gradient to the depth conversion unit 343.

The polarization gradient calculation unit 3413 calculates the gradient using polarization information for pixels that do not have the texture information classified by the pixel classification unit 3411. As described above, the polarization gradient calculation unit 3413 calculates the waveform of the trigonometric function for each different combination of pixels located in the direction of the gradient in the polarization epipolar plane image, and determines the gradient with the smallest waveform difference as the true value. It is a gradient. The polarization gradient calculation unit 3413 outputs the calculated gradient of the true value to the depth conversion unit 343. In addition, the polarization gradient calculating unit 3413 outputs the pixel value of the pixel at which the difference in amplitude and the difference in phase are minimum, that is, the pixel value of the pixel positioned in the gradient direction of the true value to the normal calculation unit 344. In addition, the polarization gradient calculating unit 3413 outputs not only the pixel values of the pixels positioned in the direction of the gradient of the true value but also the pixel values of the pixels positioned in the direction of the arbitrary gradient to the normal calculation unit 344. Further, the polarization gradient calculating unit 3413 generates normal-no gradient non-computed pixel information indicating a normal-non-normalized gradient non-computed pixel and outputs the pixel information to the depth interpolation unit 346.

The depth conversion unit 343 converts the gradients calculated by the texture gradient calculation unit 3412 and the polarization gradient calculation unit 3413 of the gradient calculation unit 341 into depths. The depth conversion unit 343 converts the gradient θp calculated by the texture gradient calculation unit 3412 and the polarization gradient calculation unit 3413 into depth as described above. The depth conversion unit 343 outputs the converted depth to the normal calculation unit 344 and the integration processing unit 345.

Based on the pixel value (brightness) supplied from the polarization gradient calculating unit 3413, the normal line calculating unit 344 determines the azimuth angle φ at which the brightness is maximum. Further, the normal line calculation unit 344 determines the zenith angle θz corresponding to the degree of polarization ρ calculated using the maximum luminance and the minimum luminance obtained from the relationship between the luminance and the polarization angle. The normal line calculation unit 344 generates normal line information indicating the calculated azimuth angle φ and the zenith angle θz, and outputs the generated normal line information to the integration processing unit 345.

The integration processing unit 345 performs integration processing using the depth obtained by the depth conversion unit 343 and the normal calculated by the normal calculation unit 344 to calculate the depth with high resolution and high accuracy, and the depth interpolation unit 346. Output to

The depth interpolation unit 346 calculates the depth of the pixel indicated by the non-normal gradient non-computed pixel information supplied from the polarization gradient calculation unit 3413 by interpolation processing. As described above, the non-normal gradient uncomputed pixel is a pixel for which the gradient can not be calculated or a pixel with low reliability of the gradient or the normal, that is, a pixel with strong noise. Therefore, the depth interpolation unit 346 performs interpolation processing using the depth of the pixel which is located around the non-normalized gradient non-computed pixel and the depth is calculated by the integration processing unit 345, and the non-normalized gradient is not calculated. Calculate the depth of the pixel.

FIG. 16 is a diagram for explaining the depth interpolation processing of the depth interpolation unit 346. For example, the pixel P1 is the pixel closest to the left with respect to the pixel with no normal gradient but not calculated, and for which the depth is calculated. In addition, the pixel P2 is located in the right direction with respect to the normal line no gradient non-computed pixel Pt, and the closest pixel for which the depth is calculated is set as the pixel P2. In addition, the distance (for example, the number of pixels) from the normal line non-gradient uncomputed pixel Pt to the pixel P1 is “L1”, and the distance from the normal line no gradient non-computed pixel Pt to the pixel P2 (for example, pixel number) is “L2” Do. Further, the pixel P1 has a depth "D1", and the pixel P2 has a depth "D2". In this case, the depth interpolation unit 346 calculates the depth "Dt" of the no-normal-gradient uncomputed pixel Pt based on Expression (17).

Thus, the depth interpolation unit 346 performs interpolation processing using the depths of pixels located around the non-normal gradient non-computed pixel to calculate the depth of the non-normal gradient non-computed pixel.

FIG. 17 is a flow chart showing the operation of the third embodiment. In step ST31, the image processing apparatus performs pixel classification. The image processing device 30 calculates the texture determination value using the pixel value. Furthermore, the image processing apparatus 30 determines whether the pixel has texture information or not based on the calculated texture determination value, and proceeds to step ST32.

In step ST32, the image processing apparatus calculates the gradient using the texture information. The image processing apparatus 30 calculates the gradient using the texture information for the pixel determined to have the texture information, and proceeds to step ST33.

In step ST33, the image processing apparatus calculates a gradient using polarization information. The image processing apparatus 30 calculates a gradient using polarization information for a pixel determined to have no texture information, and proceeds to step ST34.

In step ST34, the image processing apparatus generates no normal line gradient uncalculated pixel information. The image processing device 30 discriminates the pixel with no normal line without calculating the normal line on the basis of the calculation result of the gradient in step ST33, the reliability of the calculated gradient, the amplitude and phase of the waveform of the trigonometric function used for calculating the gradient. Then, on the basis of the determination result, no normal line gradient uncalculated pixel information is generated, and the process proceeds to step ST35.

The image processing apparatus calculates the depth in step ST35. The image processing device 30 calculates the depth based on the gradients calculated in step ST32 and step ST33, and proceeds to step ST36.

In step ST36, the image processing apparatus generates normal line information. The image processing device 30 calculates the normal based on the polarization information when the gradient is calculated in step ST33, generates normal information indicating the calculated normal, and proceeds to step ST37.

In step ST37, the image processing apparatus performs integration processing. The image processing device 30 performs integration processing using the depth obtained in pixel units and the normal indicated by the normal information, calculates the depth with high resolution at a higher resolution than the pixel units, and proceeds to step ST38 .

In step ST38, the image processing apparatus performs depth interpolation processing. The image processing device 30 uses the depth of the non-normalized gradient non-calculated pixel indicated by the non-normalized gradient non-calculated pixel information generated in step ST34 as the depth located near the non-normalized gradient non-calculated pixel. Calculated by interpolation processing.

As described above, according to the third embodiment, the depth can be calculated even for a pixel whose depth can not be calculated in the first embodiment or the second embodiment.

<5. Fourth embodiment>
In the first to third embodiments described above, the case where depth calculation is performed based on a plurality of polarization imaging images having different viewpoint positions in one direction (for example, horizontal direction) has been described, but viewpoint positions are different in a plurality of directions. May be Next, in the fourth embodiment, a case will be described where depth calculation is performed based on a plurality of polarization imaging images whose viewpoint positions are different in a plurality of directions, for example, a plurality of polarization imaging images whose viewpoint positions are different in the horizontal direction and the vertical direction. .

FIG. 18 illustrates the configuration of the imaging apparatus according to the fourth embodiment. In the imaging apparatus, six imaging units are arranged in the horizontal direction, and six stages are stacked in the vertical direction. Polarizing elements 21- (1, 1) to 21- (6, 6) are provided in the imaging units 20- (1, 1) to 20- (6, 6).

FIG. 19 illustrates the polarization direction of the imaging device in the fourth embodiment. The polarization direction of the polarization element is arranged so that the polarization direction has a predetermined angle difference, for example, 30 °, with the imaging units adjacent vertically and horizontally. In FIG. 19, the polarization direction is “0 °, 30 °”. And 60 °, 90 °, 120 °, and 150 °.

The image processing apparatus according to the fourth embodiment includes the preprocessing unit 31, the parameter holding unit 32, the polarization epipolar plane image generation unit 33, and the depth calculation unit 34, as in the first embodiment shown in FIG. ing.

The preprocessing unit 31 uses the parameters stored in the parameter storage unit 32 to perform preprocessing on polarization imaging images acquired by a plurality of imaging units. The parameter holding unit 32 stores in advance internal parameters and external parameters obtained by performing calibration using a predetermined subject such as a checker board. The preprocessing unit 31 performs distortion correction or the like of the polarization imaging image using the polarization imaging image and an internal parameter of the imaging unit that has acquired the polarization imaging image. Further, the preprocessing unit 31 performs registration of the polarization imaging image using the polarization imaging image processed using the internal parameter and the external parameter. The preprocessing unit 31 outputs the polarization imaging image after the preprocessing to the polarization epipolar plane image generating unit 33.

The polarization epipolar plane image generation unit 33 generates a first polarization epipolar plane image and a first epipolar plane image from a plurality of polarization imaging images having different viewpoint positions in the first direction, and generates a viewpoint in a second direction different from the first direction. A second polarization epipolar plane image and a second epipolar plane image are generated from a plurality of polarization imaging images different in position. For example, the polarization epipolar plane image generation unit 33 generates an image in the arrangement direction of the plurality of imaging units, that is, the horizontal direction so as to include an image indicating a desired position in the subject from polarization imaging images acquired by the plurality The epipolar plane image is generated by arranging the extracted images in the order of the imaging units in the vertical direction orthogonal to the arrangement direction, in the interval according to the interval of the imaging units (the interval of the viewpoint position). In addition, the polarization epipolar plane image generation unit 33 generates the polarization epipolar plane image by arranging the extracted images in the order of the viewpoint positions in the vertical direction orthogonal to the arrangement direction, at the intervals according to the intervals of the viewpoint positions. Furthermore, the polarization epipolar plane image generation unit 33 arranges the plurality of imaging units in the alignment direction, that is, the vertical direction so as to include an image indicating a desired position in the subject from polarization imaging images acquired by the plurality of imaging units arranged in the vertical direction. The image is extracted, and the epipolar plane image is generated by arranging the extracted images in the order of the imaging units in the lateral direction orthogonal to the arrangement direction, in the interval according to the interval of the imaging units (the interval of the viewpoint position). In addition, the polarization epipolar plane image generation unit 33 generates the polarization epipolar plane image by arranging the extracted images in the order of the viewpoint position in the horizontal direction orthogonal to the arrangement direction, at an interval corresponding to the interval of the viewpoint position. The polarization epipolar plane image generation unit 33 generates a polarization epipolar plane image (hereinafter referred to as “laterally polarized epipolar plane image”) and an epipolar plane image (hereinafter referred to as “lateral epipolar plane image”) with epipolar lines in the lateral direction and epipolar lines. The polarization epipolar plane image (hereinafter referred to as “longitudinal polarization epipolar plane image”) and the epipolar plane image (hereinafter referred to as “longitudinal epipolar plane image”) whose vertical directions are the vertical directions are output to the depth calculation unit 34.

The fourth embodiment differs from the first to third embodiments in the configuration of the depth calculation unit. The depth calculation unit according to the fourth embodiment includes a gradient calculated using polarization imaging images acquired by a plurality of imaging units aligned in the horizontal direction, and a polarization imaging image acquired by a plurality of imaging units aligned in the vertical direction. Is used to calculate the depth based on the highly reliable gradient.

FIG. 20 illustrates the configuration of the depth calculation unit according to the fourth embodiment. The depth calculation unit 34 includes slope calculation units 341 H and 341 V, a slope selection unit 342, a depth conversion unit 343, a normal line calculation unit 344, an integration processing unit 345, and a depth interpolation unit 346.

The gradient calculation unit 341H calculates the gradient of the depth calculation target pixel in the lateral epipolar plane image generated by the polarization epipolar plane image generation unit 33. The gradient calculating unit 341H includes a pixel classification unit 3411H, a texture gradient calculating unit 3412H, and a polarization gradient calculating unit 3413H.

The pixel classification unit 3411H classifies the pixels of the horizontal epipolar plane image into a pixel having texture information and an image having no texture information. The pixel classification unit 3411H calculates the texture determination value G as described above for the determination target pixel. When the calculated texture determination value G is equal to or greater than a predetermined determination threshold σ, the pixel classification unit 3411H determines that the pixel has texture information, and the texture determination value G is smaller than the determination threshold σ. In this case, it is determined that the pixel does not have texture information.

The texture gradient calculation unit 3412H calculates the gradient θth using texture information for the pixels having the texture information classified by the pixel classification unit 3411H. The texture gradient calculation unit 3412H calculates the gradient θth as described above, for example, using the differential value calculated by the pixel classification unit 3411H. The texture gradient calculating unit 3412 H outputs the calculated gradient θth and the reliability Rth to the gradient selecting unit 342, using the dispersion when the gradient of the true value is calculated as the reliability Rth.

The polarization gradient calculation unit 3413H calculates the gradient using polarization information for the pixels that do not have the texture information classified by the pixel classification unit 3411H. The polarization gradient calculation unit 3413H calculates the waveform of the trigonometric function for each different combination of pixels positioned in the gradient direction as described above in the transversely polarized epipolar plane image as described above, and determines the gradient with the smallest waveform difference as the true. The gradient of the value is θph. Further, the polarization gradient calculation unit 3413H outputs the calculated gradient θph and the reliability Rph to the gradient selection unit 342, using the dispersion when the gradient θph of the true value is calculated as the reliability Rph.

The pixel classification unit 3411V classifies the pixels of the vertical epipolar plane image into pixels having texture information and an image having no texture information. The pixel classification unit 3411V calculates the texture determination value G as described above for the determination target pixel. When the calculated texture determination value G is equal to or greater than a predetermined determination threshold σ, the pixel classification unit 3411 V determines that the pixel has texture information, and the texture determination value G is smaller than the determination threshold σ. In this case, it is determined that the pixel does not have texture information.

The texture gradient calculation unit 3412 V calculates the gradient θ tv using texture information for pixels having texture information classified by the pixel classification unit 3411 V. The texture gradient calculation unit 3412 V calculates the gradient θ tv as described above, for example, using the differential value calculated by the pixel classification unit 3411 H. The texture gradient calculating unit 3412 V outputs the calculated gradient θ tv and the reliability R tv to the gradient selecting unit 342, using the variance when the gradient of the true value is calculated as the reliability Rtv.

The polarization gradient calculation unit 3413V calculates the gradient using polarization information for pixels that do not have the texture information classified by the pixel classification unit 3411V. The polarization gradient calculation unit 3413 V calculates the waveform of the trigonometric function for each different combination of pixels positioned in the gradient direction as described above in the longitudinal polarization epipolar plane image as described above, and determines the gradient with the smallest waveform difference as the true. The gradient of the value is θpv. Further, the polarization gradient calculating unit 3413 V outputs the calculated gradient θpv and the reliability Rpv to the gradient selecting unit 342 with the dispersion obtained when the true value gradient θpv is calculated as the reliability Rpv.

The gradient selection unit 342 is a pixel for which the depth is to be calculated based on the gradients θth, θtv, θph, θpv and the reliabilities Rth, Rtv, Rph, Rpv obtained from the texture gradient calculators 3412H, 3412V and the polarization gradient calculators 3413H, 3413V. The gradient is selected and output to the depth conversion unit 343. Also, when the selected gradient is calculated based on the polarization epipolar plane image, the gradient selecting unit 342 calculates the pixel value of the pixel located in the direction of an arbitrary gradient in the polarization epipolar plane image used for calculating the selected gradient. Are output to the normal calculation unit 344. Furthermore, the gradient selection unit 342 generates non-normalized gradient uncomputed pixel information indicating whether the depth calculation target pixel is a non-normalized gradient uncomputed pixel and outputs the pixel information to the depth interpolation unit 346.

FIG. 21 is a diagram for explaining the gradient selection operation. For example, it is determined that the depth calculation target pixel is a pixel having texture information based on the horizontal epipolar plane image, the gradient θth is calculated based on the texture information, and the pixel having texture information based on the vertical epipolar plane image If it is determined and the gradient θ tv is calculated based on the texture information, the gradient selection unit 342 compares the reliability Rth and the reliability Rtv, selects a gradient with high reliability, and outputs the gradient to the depth conversion unit 343. . In addition, when the reliability of the selected gradient is lower than the reliability threshold α, the gradient selecting unit 342 may set the calculated gradient as invalid and set the pixel for which the depth is calculated as the pixel without gradient calculation without normal.

The pixel whose depth is to be calculated is determined to be a pixel that does not have texture information based on the horizontal epipolar plane image, the gradient θph is calculated based on the polarization information, and has texture information based on the vertical epipolar plane image When it is determined that the pixel is not a pixel and the gradient θpv is calculated based on the polarization information, the gradient selecting unit 342 compares the reliability Rph with the reliability Rpv to select a gradient with high reliability, and the depth conversion unit Output to 343. In addition, when the reliability of the selected gradient is lower than the reliability threshold value β, the gradient selecting unit 342 may set the calculated pixel as invalid and set the pixel for which the depth is calculated as the non-normal gradient uncalculated pixel.

The pixel whose depth is to be calculated is determined to be a pixel having texture information based on the epipolar plane image in one direction (for example, the horizontal direction), and the gradient (for example, θth) is calculated based on the texture information. When it is determined that the pixel does not have texture information based on the epipolar plane image in the vertical direction) and the gradient (for example, θpv) is calculated based on the polarization information, the gradient selecting unit 342 determines that the gradient is highly reliable. The gradient (for example, θth) calculated based on the texture information is selected and output to the depth conversion unit 343. In addition, when the reliability of the gradient calculated based on the texture information is lower than the reliability threshold α, the gradient selecting unit 342 selects the gradient (for example, θpv) calculated based on the polarization information to the depth conversion unit 343. You may output it. Furthermore, when the gradient selection unit 342 selects the gradient calculated based on the polarization information instead of the gradient calculated based on the texture information, the reliability of the gradient calculated based on the polarization information is higher than the reliability threshold value β If it is low, the calculated gradient may be invalidated, and the pixel for which the depth is to be calculated may be the pixel without normal gradient without being calculated.

Also, although not shown, when the gradient is calculated based on the epipolar plane image or the polarized epipolar plane image in one direction, and the gradient is not calculated based on the epipolar plane image and the polarized epipolar plane image in the other direction, The gradient selection unit 342 outputs the calculated gradient to the depth conversion unit 343. In addition, when the calculated reliability of the gradient is lower than the reliability threshold, the gradient selecting unit 342 may set the pixel for which the depth is to be calculated as the pixel for which the normal without gradient is calculated.

The depth conversion unit 343 converts the gradient selected by the gradient selection unit 342 into depth. The depth conversion unit 343 converts the gradient selected by the gradient selection unit 342 into a depth as described above, and outputs the depth to the normal calculation unit 344 and the integration processing unit 345.

The normal line calculation unit 344 calculates the normal line of the pixel with normal line and gradient not calculated. Based on the pixel value (brightness) supplied from the gradient selection unit 342, the normal line calculation unit 344 determines the azimuth angle φ at which the brightness is maximum. Further, the normal line calculation unit 344 determines the zenith angle θz corresponding to the degree of polarization ρ calculated using the maximum luminance and the minimum luminance obtained from the relationship between the luminance and the polarization angle. The calculation of the normal when the gradient selected by the gradient selecting unit 342 is the gradient θpv calculated by the gradient calculating unit 341V is the pixel used for calculating the gradient θpH by the gradient calculating unit 341H as described above. The same process as in the case of calculating the normal, that is, the azimuth angle φ and the zenith angle θz from the values may be performed by rotating the coordinate axes by 90 degrees. The normal line calculation unit 344 generates normal line information indicating the azimuth angle φ and the zenith angle θz calculated for the normal line and gradient uncalculated pixel, and outputs the generated normal line information to the integration processing unit 345.

The integration processing unit 345 performs integration processing using the depth obtained by the depth conversion unit 343 and the normal calculated by the normal calculation unit 344 to calculate the depth with high resolution and high accuracy, and the depth interpolation unit 346. Output to

The depth interpolation unit 346 calculates the depth of the non-normalized gradient uncomputed pixel indicated by the non-normalized gradient uncomputed pixel information supplied from the gradient selecting unit 342 by interpolation processing. The depth interpolation unit 346 performs interpolation processing using the depth of the pixel located around the non-normalized gradient uncomputed pixel and for which the depth is calculated by the integration processing unit 345. Calculate the depth.

FIG. 22 is a flow chart showing the operation of the fourth embodiment. In step ST41, the image processing apparatus performs pixel classification. The image processing device 30 calculates the texture determination value using the pixel value. Furthermore, based on the calculated texture determination value, the image processing device 30 determines whether the pixel has texture information or not, for each of the horizontal epipolar plane image and the vertical epipolar plane image. Then, the process proceeds to step ST42.

In step ST42, the image processing apparatus calculates the gradient using the texture information. The image processing apparatus 30 calculates the gradient and the reliability of the pixels determined to have texture information for each of the horizontal epipolar plane image and the vertical epipolar plane image using texture information, and proceeds to step ST43.

In step ST43, the image processing apparatus calculates a gradient using polarization information. The image processing device 30 calculates the gradient and the reliability by using polarization information for a pixel determined to have no texture information in each of the horizontal polarization epipolar plane image and the vertical polarization epipolar plane image, and performs steps. Go to ST44.

In step ST44, the image processing apparatus selects a gradient. The image processing apparatus 30 selects a gradient to be used to calculate the depth from the gradients calculated in step ST42 and step ST43, and proceeds to step ST45.

In step ST45, the image processing apparatus generates no normal line gradient uncalculated pixel information. The image processing apparatus 30 calculates the gradient without normal based on the gradient calculation result in step ST42 and step ST43, the reliability of the gradient selected in step ST44, and the amplitude and phase of the waveform of the trigonometric function used for calculation of the gradient. The uncomputed pixel is determined, the normal line no gradient uncalculated pixel information is generated based on the determination result, and the process proceeds to step ST46.

In step ST46, the image processing apparatus calculates the depth. The image processing device 30 calculates the depth based on the gradient selected in step ST44 or the gradient selected in step ST44 and whose reliability is equal to or higher than the reliability threshold set in advance, and the process proceeds to step ST47. move on.

In step ST47, the image processing apparatus generates normal line information. The image processing device 30 calculates normals of the pixels with normals and gradients not calculated, and generates normal information indicating the calculated normals. In the image processing device 30, the gradient selected in step ST44 is a pixel calculated using the polarization epipolar plane image, the reliability of the gradient is lower than the reliability threshold, and the amplitude and phase of the trigonometric function waveform are gradients In the case of a pixel which becomes constant regardless of, that is, with a normal and a gradient uncalculated pixel, the pixel value of the pixel located in the direction of an arbitrary gradient in the polarization epipolar plane image used for calculating the selected gradient Is calculated, normal line information indicating the calculated normal line is generated, and the process proceeds to step ST48.

In step ST48, the image processing apparatus performs integration processing. The image processing device 30 performs integration processing using the depth obtained in pixel units and the normal line indicated by the normal information, calculates the depth having a resolution higher than that before integration processing, and proceeds to step ST49.

In step ST49, the image processing apparatus performs depth interpolation processing. The image processing device 30 calculates the depth of the non-normalized gradient non-computed pixel indicated by the non-normalized gradient non-computed pixel information generated at step ST45 by interpolation processing using the depth of the neighboring pixel.

As described above, according to the fourth embodiment, it is possible to calculate a high-reliability and high-reliability depth using a plurality of polarization imaging images obtained with the viewpoint position as a two-dimensional position.

<6. Other configuration and operation>
By the way, although the above-mentioned imaging system 10 illustrated the case where a plurality of imaging units with different polarization directions are linearly arranged in parallel, a single imaging device can be moved to switch the polarization direction of the polarization element. It is also good. For example, the imaging device is sequentially moved to the positions of the imaging devices 20-1 to 20-6 shown in FIG. 1, and the polarization direction of the polarizing element 21 is switched according to the movement of the imaging device , Images of the objects OBa and OBb without motion. In addition, external parameters corresponding to the movement of the imaging device are used. If an imaging system is configured in this manner, a plurality of polarization imaging images having different polarization directions and viewpoint positions can be obtained by performing imaging by moving one imaging device in the horizontal direction or the vertical direction without using a plurality of imaging devices. It is possible to calculate the depth with high resolution and accuracy.

<7. Application example>
The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on any type of mobile object such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot May be

FIG. 23 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile control system to which the technology according to the present disclosure can be applied.

Vehicle control system 12000 includes a plurality of electronic control units connected via communication network 12001. In the example shown in FIG. 23, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (Interface) 12053 are illustrated.

The driveline control unit 12010 controls the operation of devices related to the driveline of the vehicle according to various programs. For example, the drive system control unit 12010 includes a drive force generation device for generating a drive force of a vehicle such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, and a steering angle of the vehicle. adjusting steering mechanism, and functions as a control device of the braking device or the like to generate a braking force of the vehicle.

Body system control unit 12020 controls the operation of the camera settings device to the vehicle body in accordance with various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device of various lamps such as a headlamp, a back lamp, a brake lamp, a blinker or a fog lamp. In this case, the body system control unit 12020, the signal of the radio wave or various switches is transmitted from wireless controller to replace the key can be entered. Body system control unit 12020 receives an input of these radio or signal, the door lock device for a vehicle, the power window device, controls the lamp.

Outside vehicle information detection unit 12030 detects information outside the vehicle equipped with vehicle control system 12000. For example, an imaging unit 12031 is connected to the external information detection unit 12030. The out-of-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle, and receives the captured image. The external information detection unit 12030 may perform object detection processing or distance detection processing of a person, a vehicle, an obstacle, a sign, characters on a road surface, or the like based on the received image.

Imaging unit 12031 receives light, an optical sensor for outputting an electric signal corresponding to the received light amount of the light. The imaging unit 12031 can output an electric signal as an image or can output it as distance measurement information. The light image pickup unit 12031 is received may be a visible light, it may be invisible light such as infrared rays.

Vehicle information detection unit 12040 detects the vehicle information. For example, a driver state detection unit 12041 that detects a state of a driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera for imaging the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated or it may be determined whether the driver does not go to sleep.

The microcomputer 12051 calculates a control target value of the driving force generation device, the steering mechanism or the braking device based on the information inside and outside the vehicle acquired by the outside information detecting unit 12030 or the in-vehicle information detecting unit 12040, and a drive system control unit A control command can be output to 12010. For example, the microcomputer 12051 realizes functions of an advanced driver assistance system (ADAS) including collision avoidance or shock mitigation of a vehicle, follow-up traveling based on an inter-vehicle distance, vehicle speed maintenance traveling, vehicle collision warning, vehicle lane departure warning, etc. It is possible to perform coordinated control aiming at

The microcomputer 12051, the driving force generating device on the basis of the information around the vehicle acquired by the outside information detection unit 12030 or vehicle information detection unit 12040, by controlling the steering mechanism or braking device, the driver automatic operation such that autonomously traveling without depending on the operation can be carried out cooperative control for the purpose of.

Further, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the external information detection unit 12030. For example, the microcomputer 12051 controls the headlamps in response to the preceding vehicle or the position where the oncoming vehicle is detected outside the vehicle information detection unit 12030, the cooperative control for the purpose of achieving the anti-glare such as switching the high beam to the low beam It can be carried out.

Audio and image output unit 12052 transmits, to the passenger or outside of the vehicle, at least one of the output signal of the voice and image to be output device to inform a visually or aurally information. In the example of FIG. 23, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are illustrated as output devices. Display unit 12062, for example, may include at least one of the on-board display and head-up display.

FIG. 24 is a diagram illustrating an example of the installation position of the imaging unit 12031.

In FIG. 24, imaging units 12101, 12102, 12103, 12104, and 12105 are provided as the imaging unit 12031.

The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, on the front nose of the vehicle 12100, a side mirror, a rear bumper, a back door, an upper portion of a windshield of a vehicle interior, and the like. The imaging unit 12101 provided in the front nose and the imaging unit 12105 provided in the upper part of the windshield in the vehicle cabin mainly acquire an image in front of the vehicle 12100. The imaging units 12102 and 12103 included in the side mirror mainly acquire an image of the side of the vehicle 12100. The imaging unit 12104 provided in the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield in the passenger compartment is mainly used to detect a leading vehicle or a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

Note that FIG. 24 shows an example of the imaging range of the imaging units 12101 to 12104. Imaging range 12111 indicates an imaging range of the imaging unit 12101 provided in the front nose, imaging range 12112,12113 are each an imaging range of the imaging unit 12102,12103 provided on the side mirror, an imaging range 12114 is The imaging range of the imaging part 12104 provided in the rear bumper or the back door is shown. For example, by overlaying the image data captured by the imaging units 12101 to 12104, a bird's eye view of the vehicle 12100 viewed from above can be obtained.

At least one of the imaging unit 12101 through 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera including a plurality of imaging devices, or an imaging device having pixels for phase difference detection.

For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 measures the distance to each three-dimensional object in the imaging ranges 12111 to 12114, and the temporal change of this distance (relative velocity with respect to the vehicle 12100). In particular, it is possible to extract a three-dimensional object traveling at a predetermined speed (for example, 0 km / h or more) in substantially the same direction as the vehicle 12100 as a leading vehicle, in particular by finding the it can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in advance before the preceding vehicle, and can perform automatic brake control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. Automatic operation or the like for autonomously traveling without depending on the way of the driver operation can perform cooperative control for the purpose.

For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 converts three-dimensional object data relating to three-dimensional objects into two-dimensional vehicles such as two-wheeled vehicles, ordinary vehicles, large vehicles, classification and extracted, can be used for automatic avoidance of obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 into obstacles visible to the driver of the vehicle 12100 and obstacles difficult to see. Then, the microcomputer 12051 determines a collision risk which indicates the risk of collision with the obstacle, when a situation that might collide with the collision risk set value or more, through an audio speaker 12061, a display portion 12062 By outputting a warning to the driver or performing forcible deceleration or avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be performed.

At least one of the imaging unit 12101 to 12104, may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the imaging units 12101 to 12104. Such pedestrian recognition is, for example, a procedure for extracting feature points in images captured by the imaging units 12101 to 12104 as an infrared camera, and pattern matching processing on a series of feature points indicating the outline of an object to determine whether it is a pedestrian or not The procedure is to determine Microcomputer 12051 is, determines that the pedestrian in the captured image of the imaging unit 12101 to 12104 is present, recognizing the pedestrian, the sound image output unit 12052 is rectangular outline for enhancement to the recognized pedestrian to superimpose, controls the display unit 12062. The audio image output unit 12052 is, an icon or the like indicating a pedestrian may control the display unit 12062 to display the desired position.

The example of the vehicle control system to which the technology according to the present disclosure can be applied has been described above. The image processing apparatus according to the present disclosure may be applied to the external information detection unit 12030 among the configurations described above. In addition, the imaging device according to the present disclosure may be applied to the imaging units 12101, 12102, 12103, 12104, 12105 and the like. For example, if a plurality of polarization imaging images having different polarization directions and viewpoint positions are acquired by the imaging unit 12101 provided in the front nose and the imaging unit 12105 provided in the upper part of the windshield in the vehicle interior, the outside information detection unit 12030 The distance to the vehicle, the pedestrian in front, and the obstacle can be accurately calculated with high resolution. In addition, if a plurality of polarization imaging images having different polarization directions and viewpoint positions are acquired by the imaging units 12102 and 12103 provided in the side mirrors, the outside vehicle information detection unit 12030 is positioned on the side surface, for example, when performing parallel parking. The distance to the object can be accurately calculated with high resolution. The imaging units 12102 and 12103 included in the side mirrors perform imaging by switching the polarization direction according to the movement of the car and acquiring a plurality of polarization imaging images having different viewpoint positions and polarization directions, and performing parallel parking. It is also possible to accurately calculate the distance to an object located on the side with high resolution only by passing in front of the space. Furthermore, if a plurality of polarization imaging images different in polarization direction and viewpoint position are acquired by the imaging unit 12104 provided in the rear bumper or the back door, the outside vehicle information detection unit 12030 may back when performing distance to the following vehicle or right angle parking The distance to the object located at can be accurately calculated with high resolution.

The series of processes described in the specification can be performed by hardware, software, or a combination of both. In the case of executing processing by software, a program recording the processing sequence is installed and executed in a memory in a computer incorporated in dedicated hardware. Alternatively, the program can be installed and executed on a general-purpose computer that can execute various processes.

For example, the program can be recorded in advance on a hard disk or a solid state drive (SSD) as a recording medium, or a read only memory (ROM). Alternatively, the program may be a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disc, a digital versatile disc (DVD), a BD (Blu-Ray Disc (registered trademark)), a magnetic disc, a semiconductor memory card Etc. can be stored (recorded) temporarily or permanently on a removable recording medium such as Such removable recording media can be provided as so-called package software.

The program may be installed from the removable recording medium to the computer, or may be transferred from the download site to the computer wirelessly or by wire via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in such a manner, and install the program on a recording medium such as a built-in hard disk.

In addition, the effect described in this specification is an illustration to the last, is not limited, and may have an additional effect which is not described. In addition, the present technology should not be construed as being limited to the embodiments of the above-described technology. The embodiments of this technology disclose the present technology in the form of exemplification, and it is obvious that those skilled in the art can modify or substitute the embodiments within the scope of the present technology. That is, in order to determine the gist of the present technology, the claims should be taken into consideration.

In addition, the image processing apparatus of the present technology can also have the following configuration.
(1) An image processing apparatus including a depth calculation unit that calculates the depth of a pixel for which depth calculation is to be performed, based on polarization epipolar plane images generated from a plurality of polarization imaging images having different polarization directions and viewpoint positions.
(2) The depth calculation unit
A polarization gradient calculation unit that calculates the gradient of the depth calculation target pixel in the polarization epipolar plane image;
The image processing apparatus according to (1), further including: a depth conversion unit that converts the gradient calculated by the polarization gradient calculation unit into a depth.
(3) The image processing apparatus according to (2), wherein the depth conversion unit converts, to the depth, a gradient whose reliability of the gradient calculated by the polarization gradient calculation unit is equal to or higher than a reliability threshold set in advance.
(4) The polarization gradient calculation unit performs fitting to a trigonometric function using pixels on a straight line based on the position of the depth calculation target pixel in the polarization epipolar plane image, and occurs due to differences in pixels used for fitting The image processing apparatus according to any one of (2) and (3), wherein the gradient of the straight line at which the difference in waveform of the trigonometric function is the smallest is the gradient of the pixel for which the depth is to be calculated.
(5) The image processing apparatus according to (4), wherein the polarization direction is four or more in a range of an angle difference of less than 180 degrees.
(6) A normal line calculation unit that calculates a normal of the depth calculation target pixel using a pixel located in the direction of the gradient calculated by the polarization gradient calculation unit in the polarization epipolar plane image;
The image processing apparatus further includes an integrated processing unit that performs interpolation processing using the depth obtained by the depth conversion unit and the normal calculated by the normal calculation unit to obtain a depth that is denser than the pixel unit and has a higher resolution The image processing apparatus according to any one of 2) to 5).
(7) When the gradient calculation unit can not calculate the gradient of the depth calculation object pixel, the depth interpolation is performed to calculate the depth of the depth calculation object pixel by interpolation processing using the depth of the pixel adjacent to the depth calculation object pixel The image processing apparatus according to any one of (2) to (6), further including a part.
(8) It further has a normal line calculation unit which calculates a normal line using pixels located in the direction of an arbitrary gradient in the polarization epipolar plane image,
When the gradient calculation unit can not calculate the gradient of the depth calculation target pixel, the normal calculation unit calculates the depth using pixels located in a direction of an arbitrary gradient from the depth calculation target pixel for which the gradient can not be calculated. The image processing apparatus according to any one of (2) to (7), which calculates a normal of a target pixel.
(9) The depth calculation unit
A pixel classification unit that classifies whether the depth calculation target pixel is a pixel having texture information or a pixel not having the texture information based on an epipolar plane image generated from the plurality of polarization imaging images;
The image processing apparatus further includes a texture gradient calculation unit that calculates the gradient of the depth calculation target pixel in the epipolar plane image,
The texture gradient calculation unit calculates gradients of pixels having the texture information based on the epipolar plane image,
The image processing apparatus according to any one of (2) to (8), wherein the polarization gradient calculation unit calculates the gradient of the pixel not having the texture information based on the polarization epipolar plane image.
(10) The depth calculation unit
The polarization gradient calculator or the texture gradient calculator calculates the first polarization epipolar plane image or the first epipolar plane image generated from a plurality of polarization imaging images having different viewpoint positions in the first direction. The polarization based on the gradient of the depth calculation target pixel and a second polarization epipolar plane image or a second epipolar plane image generated from a plurality of polarization imaging images having different viewpoint positions in a second direction different from the first direction. The image processing apparatus further includes a gradient selection unit that selects any one of the gradients for the depth calculation target pixel calculated by the gradient calculation unit or the texture gradient calculation unit,
The image processing apparatus according to (9), wherein the depth conversion unit converts the gradient selected by the gradient selection unit into a depth.
(11) When the gradient selection unit is one of the gradients calculated by the texture gradient calculation unit and the other gradient is the gradient calculated by the polarization gradient calculation unit, the texture selection unit The image processing apparatus according to (9), wherein the calculated gradient is selected.
(12) The image processing apparatus according to (10), wherein the gradient selection unit selects a gradient with high reliability when the gradients are both calculated by the texture gradient calculation unit or the polarization gradient calculation unit.

In the image processing apparatus, the image processing method, the program, and the image processing system of this technology, the depth of the depth calculation target pixel is calculated based on polarization epipolar plane images generated from a plurality of polarization imaging images having different polarization directions and viewpoint positions. Be done. For this reason, it becomes possible to calculate the depth based on the polarization information even in the area without texture. Therefore, it is suitable for an apparatus using a function of acquiring a distance to a subject using a captured image, for example, an electronic apparatus mounted on a mobile object such as an automobile.

10: Imaging system 20-1 to 20-6, 20- (1, 1) to 20- (6, 6) ... imaging device 21-1 to 21-6, 21- (1, 1) to 21-(6, 6) ... Polarizing element 30 ... Image processing device 31 ... Preprocessing unit 32 ... Parameter holding unit 33 ... Polarized epipolar plane image generation unit 34 ... Depth calculation unit 341, 341 H, 341 V ... gradient calculation unit 342 ... gradient selection unit 343 ... depth conversion unit 344 ... normal calculation unit 345 ... integration processing unit 346 ... depth interpolation unit 3411, 3411 H , 3411 V: pixel classification unit 3412, 3412 H, 3412 V: texture gradient calculation unit 3413, 3413 H, 3413 V: polarization gradient calculation unit

Claims (17)

An image processing apparatus comprising: a depth calculation unit that performs depth calculation of a depth calculation target pixel based on polarization epipolar plane images generated from a plurality of polarization imaging images having different polarization directions and viewpoint positions. The depth calculation unit
A polarization gradient calculation unit that calculates the gradient of the depth calculation target pixel in the polarization epipolar plane image;
The image processing apparatus according to claim 1, further comprising: a depth conversion unit that converts the gradient calculated by the polarization gradient calculation unit into a depth. The image processing apparatus according to claim 2, wherein the depth conversion unit converts, to the depth, a gradient whose reliability of the gradient calculated by the polarization gradient calculation unit is equal to or more than a reliability threshold set in advance. The polarization gradient calculation unit performs fitting to a trigonometric function using pixels on a straight line based on the position of the depth calculation target pixel in the polarization epipolar plane image, and the trigonometric function generated due to a difference in pixels used for fitting The image processing apparatus according to claim 2, wherein the gradient of the straight line at which the waveform difference between the two is the smallest is the gradient of the depth calculation target pixel. The image processing apparatus according to claim 4, wherein the polarization direction is four or more directions within an angular difference range of less than 180 degrees. A normal line calculation unit that calculates a normal line of the depth calculation target pixel using a pixel located in the direction of the gradient calculated by the polarization gradient calculation unit in the polarization epipolar plane image;
The image processing apparatus further includes an integrated processing unit that performs interpolation processing using the depth obtained by the depth conversion unit and the normal calculated by the normal calculation unit to obtain a depth having a resolution higher than that of the pixel unit and having a higher resolution. An image processing apparatus according to Item 2. If the gradient calculation unit can not calculate the gradient of the depth calculation object pixel, the depth interpolation unit further calculates the depth of the depth calculation object pixel by interpolation processing using the depth of the pixel close to the depth calculation object pixel. The image processing apparatus according to claim 2. It further comprises a normal line calculation unit that calculates a normal line using pixels located in the direction of an arbitrary gradient in the polarization epipolar plane image,
When the gradient calculation unit can not calculate the gradient of the depth calculation target pixel, the normal calculation unit calculates the depth using pixels located in a direction of an arbitrary gradient from the depth calculation target pixel for which the gradient can not be calculated. The image processing apparatus according to claim 2, wherein the normal of the target pixel is calculated. The depth calculation unit
A pixel classification unit that classifies whether the depth calculation target pixel is a pixel having texture information or a pixel not having the texture information based on an epipolar plane image generated from the plurality of polarization imaging images;
The image processing apparatus further includes a texture gradient calculation unit that calculates the gradient of the depth calculation target pixel in the epipolar plane image,
The texture gradient calculation unit calculates gradients of pixels having the texture information based on the epipolar plane image,
The image processing apparatus according to claim 2, wherein the polarization gradient calculation unit calculates the gradient of the pixel not having the texture information based on the polarization epipolar plane image. The depth calculation unit
The polarization gradient calculator or the texture gradient calculator calculates the first polarization epipolar plane image or the first epipolar plane image generated from a plurality of polarization imaging images having different viewpoint positions in the first direction. The polarization based on the gradient of the depth calculation target pixel and a second polarization epipolar plane image or a second epipolar plane image generated from a plurality of polarization imaging images having different viewpoint positions in a second direction different from the first direction. The image processing apparatus further includes a gradient selection unit that selects any one of the gradients for the depth calculation target pixel calculated by the gradient calculation unit or the texture gradient calculation unit,
The image processing apparatus according to claim 9, wherein the depth conversion unit converts the gradient selected by the gradient selection unit into a depth. The gradient selecting unit is calculated by the texture gradient calculating unit when one gradient is the gradient calculated by the texture gradient calculating unit and the other gradient is the gradient calculated by the polarization gradient calculating unit. The image processing apparatus according to claim 10, wherein a gradient is selected. The image processing apparatus according to claim 11, wherein the gradient selection unit selects a gradient with high reliability when both of the gradients are calculated by the texture gradient calculation unit or the polarization gradient calculation unit. An image processing method comprising performing depth calculation of a depth calculation target pixel based on polarization epipolar plane images generated from a plurality of polarization imaging images having different polarization directions and viewpoint positions. A program that causes a computer to execute image processing using a polarization image.
Generating a polarization epipolar plane image from a plurality of polarization imaging images different in polarization direction and viewpoint position;
A program for causing the computer to execute the steps of calculating the depth of a pixel for which depth calculation is to be performed, based on the generated polarization epipolar plane image. An imaging device for acquiring a plurality of polarization imaging images having different polarization directions and viewpoint positions;
It has an image processing device which performs depth calculation of a depth calculation target pixel from the plurality of polarization imaging images acquired by the imaging device,
The image processing apparatus is
A polarization epipolar plane image generation unit that generates a polarization epipolar plane image including pixels for depth calculation from the plurality of polarization imaging images;
An image processing system comprising: a depth calculation unit that performs depth calculation of the depth calculation target pixel based on the polarization epipolar plane image generated by a polarization epipolar plane image generation unit. The imaging device is arranged in parallel in a first direction or a second direction different from the first direction or the first direction and an imaging unit for acquiring the polarization imaging image, and a plurality of polarization directions and viewpoint positions are different. The image processing system according to claim 15, which acquires a polarization imaging image. The image processing system according to claim 15, wherein the imaging device changes the polarization direction according to movement of a viewpoint position.

Images