WO2014192642A1 - Image processing device and image processing method - Google Patents

Abstract

An image processing unit (1) uses one input image of a group of multi-viewpoint input images each having a common partial region as a reference image to perform corresponding point search for estimating a pixel corresponding to one pixel on the reference image in an input image other than the reference image. The image processing unit is provided with: a first estimation unit (321) for, from a position corresponding to the one pixel of the input image other than the reference image, estimating the pixel corresponding to the one pixel by performing template matching at a first pitch in a direction corresponding to the direction formed by the viewpoint of the reference image and the view point of the input image other than the reference image; and a second estimation unit (322) for estimating the pixel corresponding to the one pixel by performing template matching on the estimated pixel at a second pitch smaller than the first pitch.

Description

Image processing apparatus and image processing method

The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method for searching for corresponding points.

There is an image processing method in which corresponding points between multi-viewpoint input images each having a common partial area are searched, a distance from each viewpoint to an object is calculated, and a parallax amount is corrected according to the distance. For example, Japanese Patent Laid-Open No. 2000-28355 (hereinafter referred to as Patent Document 1) discloses a technique for reducing a parallax estimation error by detecting a deviation of an epipolar line. Japanese Patent Laid-Open No. 9-153148 (hereinafter referred to as Patent Document 2) discloses a technique for searching for the most parallelized condition of epipolar lines by performing movement, rotation, magnification conversion, and the like. Japanese Patent Laying-Open No. 2012-107971 (hereinafter referred to as Patent Document 3) discloses a technique for searching for corresponding points in two stages: a coarse search and a fine search for nearby subjects.

JP 2000-28355 A JP-A-9-153148 JP 2012-107971 A

Although the above Patent Documents 1 to 3 improve the accuracy of the corresponding point search, they cannot meet the needs for higher speed.

The present invention has been made in view of such problems, and an object thereof is to provide an image processing apparatus and an image processing method capable of searching for corresponding points at high speed and with high accuracy.

In order to achieve the above object, according to an aspect of the present invention, an image processing apparatus uses, as a reference image, one input image in a group of multi-viewpoint input images each having a common partial area, An image processing apparatus for estimating a corresponding pixel in an input image other than the reference image, from a position corresponding to one pixel of the input image other than the reference image, By performing template matching at a first pitch in a direction corresponding to the direction formed by the viewpoint of the input image, a first estimation unit for estimating a pixel corresponding to the one pixel, and a first estimation unit A second estimation unit configured to estimate a pixel corresponding to the one pixel by performing template matching on the estimated pixel at a second pitch smaller than the first pitch.

Preferably, the second estimation unit performs template matching using a template having a size equal to or smaller than the size of the template used in the first estimation unit.

Preferably, the image processing apparatus is configured to estimate a shift amount in a decimal pixel based on a shift between the one pixel and a pixel corresponding to the one pixel estimated by the second estimation unit. The estimation part is further provided.

Preferably, a processing unit for executing processing for calculating a distance from the viewpoint of the reference image to the subject based on a shift between the one pixel and a pixel corresponding to the one pixel of the input image other than the reference image Is further provided.

More preferably, the processing unit calculates the distance from the viewpoint of the reference image to the subject by adding the deviations of each of the plurality of input images other than the reference image.

More preferably, the processing unit calculates a correction amount from a shift of each of the plurality of input images other than the reference image, and corrects each shift.

Preferably, the image processing apparatus performs super-resolution processing on an input image other than the reference image using a shift between the one pixel and a pixel corresponding to the one pixel of the input image other than the reference image as a parameter. A processing unit is further provided.

Preferably, the image processing apparatus further includes a preprocessing unit for performing distortion correction and parallelization processing as preprocessing for each of the input images.

According to another aspect of the present invention, an image processing method uses, as a reference image, one reference image of a multi-viewpoint input image group having a common partial area, and a reference image of one pixel on the reference image. A method of estimating a corresponding pixel in an input image other than the reference image, and a direction formed by the viewpoint of the reference image and the viewpoint of the input image other than the reference image from the position corresponding to the one pixel of the input image other than the reference image By performing template matching at a first pitch in a direction corresponding to the above, and estimating a pixel corresponding to the one pixel, and a second pitch smaller than the first pitch with respect to the estimated pixel. Estimating a pixel corresponding to the one pixel by performing template matching.

According to still another aspect of the present invention, an image processing program causes a computer to use one input image of a multi-viewpoint input image group having a common partial area as a reference image, and one pixel on the reference image. A program for executing a process for estimating a corresponding pixel in an input image other than the reference image, from a position corresponding to the one pixel in the input image other than the reference image, Performing a template matching at a first pitch in a direction corresponding to a direction formed with the viewpoint of the input image to estimate a pixel corresponding to the one pixel, and a first pitch with respect to the estimated pixel By performing template matching at a finer second pitch, the computer is caused to execute the step of estimating the pixel corresponding to the one pixel.

According to the present invention, it is possible to search for corresponding points at high speed and with high accuracy.

1 is a block diagram illustrating a basic configuration of an image processing apparatus according to an embodiment. It is a block diagram which shows the structure of the digital camera which actualized the image processing apparatus. It is a block diagram which shows the structure of the personal computer which actualized the image processing apparatus. It is the schematic showing the relationship between the lens contained in the camera which is an array camera, and the image pick-up element which is an image sensor. It is a figure showing the specific example of arrangement | positioning of the lens contained in a camera. It is the schematic showing the ideal state of the pitch change by the temperature change of an array lens. It is a figure showing the measurement result of the position shift of the lens by a temperature change. It is a figure for comparing the shift | offset | difference of a reference | standard lens and a surrounding lens when there is no temperature change. It is a figure for demonstrating the direction of the shift | offset | difference in the case of FIG. It is a figure for comparing the shift | offset | difference of a reference | standard lens and a surrounding lens in case there exists a temperature change. It is a figure for demonstrating the direction of the shift | offset | difference in the case of FIG. 3 is a flowchart showing an operation flow in the image processing apparatus according to the first embodiment. It is a figure for demonstrating the 1st example of the position shift estimation process in step S1 of FIG. It is a figure for demonstrating the 1st example of the position shift estimation process in step S1 of FIG. It is a figure for demonstrating the process which estimates the position shift in the sub pixel unit (decimal pixel) of step S2 of FIG. It is a figure for demonstrating the process which estimates the position shift in the sub pixel unit (decimal pixel) of step S2 of FIG. It is the figure which represented the deviation | shift amount of the corresponding pixel with respect to the reference | standard image for every input image as a pixel value (color shade). It is a figure showing the specific example of the distance image with the case where there is a temperature change (A) and the case where it does not exist (B). It is a figure showing the flow of the preparation process among the refocus processes of step S4 of FIG. It is a figure showing the flow of the refocus process. It is a figure showing the specific example of the refocus image. It is a figure showing the specific example of the refocus image. It is a figure showing the specific example of the refocus image. It is a flowchart showing the flow of operation | movement in the image processing apparatus concerning 2nd Embodiment. FIG. 25 is a diagram for describing a first example of a positional deviation estimation process in step S1 ′ of FIG. 24. FIG. 25 is a diagram for describing a second example of the positional deviation estimation process in step S1 ′ of FIG. 24. It is a figure showing the flow of the super-resolution process in step S5 of FIG. It is a figure for demonstrating the degradation information used by step # 33 of FIG. It is a figure for demonstrating the process in step # 33 of FIG. It is a figure for demonstrating the process in step # 33 of FIG. It is a figure for demonstrating the process in step # 33 of FIG. It is a figure for demonstrating the process in step # 33 of FIG. It is a figure for demonstrating the process in step # 33 of FIG. It is a flowchart showing an example of the residual calculation process of step # 33 of FIG. FIG. 25 is a diagram illustrating another example of super-resolution processing in step S5 of FIG. 24. 10 is a flowchart showing an operation flow in the image processing apparatus according to the third embodiment. It is a figure for demonstrating the synthetic | combination process in step S7 of FIG. It is a figure for demonstrating the other example of the synthetic | combination process in step S7 of FIG. 10 is a flowchart illustrating an operation flow in an image processing apparatus according to a fourth embodiment. It is a figure for demonstrating the method to correct | amend the positional offset amount in FIG.39 S8. It is a figure for demonstrating the method to correct | amend the positional offset amount in FIG.39 S8. It is a figure for demonstrating the method to correct | amend the positional offset amount in FIG.39 S8. It is a figure for demonstrating the method to extract the mistake data in step S9 of FIG. It is a figure for demonstrating the method to extract the mistake data in step S9 of FIG. It is a figure for demonstrating the method to extract the mistake data in step S9 of FIG. It is a figure showing the specific example of the distance image which extracted error data. It is a figure showing the other example of the lens contained in a camera. It is a figure showing the other example of the lens contained in a camera.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, these descriptions will not be repeated.

<System configuration>
FIG. 1 is a block diagram showing a basic configuration of an image processing apparatus 1 according to the present embodiment.

Referring to FIG. 1, the image processing apparatus 1 includes an imaging unit 2, an image processing unit 3, and an image output unit 4. In the image processing apparatus 1 illustrated in FIG. 1, the image capturing unit 2 captures an image of a subject to acquire an image (hereinafter also referred to as “input image”), and the image processing unit 3 performs an operation on the acquired input image. Image processing as will be described later is performed. Then, the image output unit 4 outputs the image after image processing to a display device or the like.

The imaging unit 2 captures an object (subject) and generates an input image. More specifically, the imaging unit 2 includes a camera 22 and an A / D (Analog to Digital) conversion unit 24 connected to the camera 22. The A / D converter 24 outputs an input image indicating the subject imaged by the camera 22.

The camera 22 is an optical system for imaging a subject, and is an array camera. That is, the camera 22 includes N lenses 22a-1 to 22a-n having different viewpoints arranged in a lattice shape (these are also referred to as lenses 22a) and an image sensor (image sensor) 22b. .

The A / D converter 24 converts a video signal (analog electrical signal) indicating a subject output from the image sensor 22b into a digital signal and outputs the digital signal. The imaging unit 2 may further include a control processing circuit for controlling each part of the camera.

The image processing unit 3 includes a position shift estimation unit 32 for searching for corresponding points by performing a position shift estimation process, and a processing unit 33 for performing image processing to be described later. Preferably, the image processing unit 3 includes a preprocessing unit 31 for executing distortion correction and parallelization processing as preprocessing for the input image.

The misregistration estimation unit 32 includes a first estimation unit 321 for performing a “coarse” estimation process in the first stage of the process of estimating the misregistration amount in pixels (integer pixels), and the misalignment amount in pixel units. And a second estimation unit 322 for performing a second-stage estimation process in the process of estimating. Preferably, the positional deviation estimation unit 32 further includes a third estimation unit 323 for performing a process of estimating the deviation amount in units of sub-pixels (decimal pixels).

The image processing apparatus 1 shown in FIG. 1 can be configured as a system in which each unit is embodied as an independent apparatus. However, for general purposes, the image processing apparatus 1 may be embodied as a digital camera or a personal computer described below. Many. Therefore, as an implementation example of the image processing apparatus 1 according to the present embodiment, an implementation example with a digital camera and an implementation example with a PC (personal computer) will be described.

FIG. 2 is a block diagram showing a configuration of a digital camera 100 that embodies the image processing apparatus 1 shown in FIG. 2, components corresponding to the respective blocks constituting the image processing apparatus 1 shown in FIG. 1 are denoted by the same reference numerals as those in FIG. 1.

Referring to FIG. 2, a digital camera 100 includes a CPU (Central Processing Unit) 102, a digital processing circuit 104, an image display unit 108, a card interface (I / F) 110, a storage unit 112, and a camera unit. 114.

The CPU 102 controls the entire digital camera 100 by executing a program stored in advance. The digital processing circuit 104 executes various digital processes including image processing according to the present embodiment. The digital processing circuit 104 is typically configured by a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an LSI (Large Scale Integration), an FPGA (Field-Programmable Gate Array), or the like. The digital processing circuit 104 includes an image processing circuit 106 for realizing the functions provided by the image processing unit 3 shown in FIG.

The image display unit 108 includes an input image provided by the camera unit 114, an output image generated by the digital processing circuit 104 (image processing circuit 106), various setting information related to the digital camera 100, and a control GUI (Graphical User Interface) screen is displayed.

The card interface (I / F) 110 is an interface for writing image data generated by the image processing circuit 106 to the storage unit 112 or reading image data or the like from the storage unit 112. The storage unit 112 is a storage device that stores image data generated by the image processing circuit 106 and various types of information (setting values such as control parameters and operation modes of the digital camera 100). The storage unit 112 includes a flash memory, an optical disk, a magnetic disk, and the like, and stores data in a nonvolatile manner.

The camera unit 114 generates an input image by imaging a subject.
A digital camera 100 shown in FIG. 2 is obtained by mounting the entire image processing apparatus 1 according to the present embodiment as a single apparatus. That is, the user can visually recognize a high-resolution image on the image display unit 108 by imaging the subject using the digital camera 100.

FIG. 3 is a block diagram showing a configuration of a personal computer 200 that embodies the image processing apparatus 1 shown in FIG. A personal computer 200 shown in FIG. 3 is obtained by mounting a part of the image processing apparatus 1 according to the present embodiment as a single apparatus. In the personal computer 200 shown in FIG. 3, the imaging unit 2 for acquiring an input image is not mounted, and an input image acquired by an arbitrary imaging unit 2 is input from the outside. Even such a configuration can be included in the image processing apparatus 1 according to the embodiment of the present invention. In FIG. 3 as well, components corresponding to blocks constituting the image processing apparatus 1 shown in FIG. 1 are denoted by the same reference numerals as those in FIG.

3, the personal computer 200 includes a personal computer main body 202, a monitor 206, a mouse 208, a keyboard 210, and an external storage device 212.

The personal computer main body 202 is typically a general-purpose computer according to a general-purpose architecture, and includes a CPU, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like as basic components. The personal computer main body 202 can execute an image processing program 204 for realizing a function provided by the image processing unit 3 shown in FIG. Such an image processing program 204 is stored and distributed in a storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or distributed from a server device via a network. The image processing program 204 is stored in a storage area such as a hard disk of the personal computer main body 202.

Such an image processing program 204 implements processing by calling necessary modules among program modules provided as part of an operating system (OS) executed by the personal computer main body 202 at a predetermined timing and order. It may be configured as follows. In this case, the image processing program 204 itself does not include a module provided by the OS, and image processing is realized in cooperation with the OS. Further, the image processing program 204 may be provided by being incorporated in a part of some program instead of a single program. Even in such a case, the image processing program 204 itself does not include a module that is commonly used in the program, and image processing is realized in cooperation with the program. Even such an image processing program 204 that does not include some modules does not depart from the spirit of the image processing apparatus 1 according to the present embodiment.

Of course, some or all of the functions provided by the image processing program 204 may be realized by dedicated hardware.

The monitor 206 displays a GUI screen provided by an operating system (OS), an image generated by the image processing program 204, and the like.

The mouse 208 and the keyboard 210 each accept a user operation and output the contents of the accepted user operation to the personal computer main body 202.

The external storage device 212 stores an input image acquired by some method, and outputs this input image to the personal computer main body 202. As the external storage device 212, a device that stores data in a nonvolatile manner such as a flash memory, an optical disk, or a magnetic disk is used.

FIG. 4 is a schematic diagram showing the relationship between the lens 22a included in the camera 22 that is an array camera and the image sensor 22b that is an image sensor. Referring to FIG. 4, the lens 22a forms an optical image by collecting incident light at a focal point. The imaging element 22b is disposed so as to be positioned at the focal point of the lens 22a. Therefore, the lens 22a forms an optical image on the image sensor 22b. The image sensor 22 b converts the optical image formed by the lens 22 a into an electrical signal and outputs the electrical signal to the A / D converter 24.

FIG. 5 is a diagram showing a specific example of the arrangement of the lenses 22 a included in the camera 22. In the example of FIG. 5, as an example, the camera 22 is assumed to be an array camera including nine lenses 22a-1 to 22a-9 (lenses A to I) arranged in a grid pattern. The intervals (base line lengths) between the lenses A to I in FIG. 5 are uniform in both the vertical direction and the horizontal direction. It should be noted that input images A, B, C, D, E, F, G, H, and I from the camera 22 when captured by the camera 22 represent input images from the lenses A to I, respectively. Shall.

[First Embodiment]
<Overview of operation>
(Explanation of environmental impact on lens)
6 to 11 are diagrams for explaining the influence of a temperature change as an example of a change in environmental conditions when the lens 22a is a plastic array lens. One of the influences of the temperature change of the lens 22a, which is a plastic array lens, is that the pitch (interval) between the lenses 22a changes and the pixel position shifts.

FIG. 6 is a schematic diagram showing an ideal state of a pitch change due to a temperature change of the array lens. That is, as shown in FIG. 6, the array lens ideally has a radial shape centered on the central lens E, that is, the parallax direction of the input image from each lens (the lens arrangement direction) as shown in FIG. ) Is expected to expand and contract evenly.

On the other hand, FIG. 7 is a diagram showing the measurement result of the positional deviation for each actual lens. 7 uses the 9-lens lens array arranged at 3 × 3 shown in FIG. 5, and each of the lenses A, C, G, and I arranged at the four corners is 43 ° C. on the basis of the state of 24 ° C. , 64 ° C., and 81 ° C. are shown the measurement results of misalignment. As shown in FIG. 7, it can be seen that the actual lens has a different deformation amount for each lens, and the entire array lens is deformed asymmetrically. Note that how to deform asymmetrically also varies from individual to individual.

8 and 10 are diagrams for comparing the deviation between the reference lens and the surrounding lenses using the lens E (center lens) in the lens array as a reference lens. FIG. 8 shows the deviation when there is no temperature change, and FIG. 10 shows the deviation when there is a temperature change.

Referring to FIG. 8, when there is no temperature change, there is no deviation at infinity, and the deviation becomes larger as the subject is closer. FIG. 9 is a diagram for explaining the direction of deviation in the case of FIG. That is, the deviation in the case of FIG. 8 appears in the direction along the arrangement direction of the lens 22a (the direction connecting the central pixel of the lens 22a and the central pixel of the reference lens) as shown in FIG. For example, in the case of the upper left lens A, the shift direction is the upper left direction. In FIG. 8, distortion correction and parallelization processing are performed as stereo calibration. The direction in which the parallax appears (epipole line) is assumed to be along the arrangement of the individual eyes. However, in the case of an array in which lenses with small distortion are arranged so that their optical axes are parallel, the correction effect by the distortion correction or parallelization processing is small, and therefore the processing need not be performed.

On the other hand, when there is a temperature change, as shown in FIG. 10, the influence of deformation appears unevenly for each lens 22a. That is, the direction and the amount of deviation differ for each lens 22a. FIG. 11 is a diagram for explaining the direction of deviation in the case of FIG. The broken-line arrows in FIG. 11 indicate the direction in which the deviation appears due to temperature change, and the solid-line arrows indicate the direction of parallax that appears depending on the distance from the camera 22 to the subject. Referring to FIG. 11, it can be seen that the direction in which the shift due to the temperature change (broken arrow) is close to the direction of the parallax that appears depending on the distance of the subject (solid arrow), but is slightly different. In each image, the deviations in these two directions are combined and appear. That is, the direction of deviation in the case of FIG. 10 is slightly different from the epipole line because the deformation is asymmetric.

(Overview of operation)
The image processing apparatus 1 according to the first embodiment uses the input image to perform refocus processing as an example of image processing even when there is a temperature change as an example of environmental change with respect to the camera 22. To do.

<Operation flow>
(Overall operation)
FIG. 12 is a flowchart showing an operation flow in the image processing apparatus 1 according to the present embodiment. Referring to FIG. 12, first, in the image processing apparatus 1, nine input images are acquired by executing a process of acquiring an input image with each of the lenses 22 a illustrated in FIG. 5. Here, for example, an image having a resolution of about 500 × 500 pixels is input.

When the input image is acquired, a process for estimating the positional deviation in pixel units (integer pixels) is executed (step S1). In step S1, a positional shift due to an environmental change (temperature change or the like) or deformation of a member is estimated. Next, preferably, a process for estimating a positional deviation in sub-pixel units (decimal pixels) is executed (step S2). The estimation in step S2 improves the accuracy of distance estimation described later, but is not essential. Then, the distance from the viewpoint to the subject is calculated in consideration of the positional deviation amount (step S3), and the refocus processing as image processing is executed by adding blur corresponding to the distance using the distance as a parameter. (Step S4). In the refocusing process, the distance only needs to be relatively grasped (if the difference in distance is recognized), the distance calculation in step S3 does not necessarily require calculation of the actual distance, but the relative distance What is necessary is just to calculate by the unit which can recognize distance, ie, the difference in distance. For example, the reciprocal of the distance or a numerical value obtained by converting the parallax may be used. In this description, such calculation is referred to as distance calculation. An image after the refocus processing is generated as an output image.

(Position misalignment estimation)
The position deviation estimation process in step S1 is a first stage estimation process in which template matching is performed with a rough pitch in a direction corresponding to a direction formed by the reference image viewpoint and the calculated image viewpoint with respect to the reference image. The process (# 1) and the estimation process (# 2) for performing template matching at a finer pitch than the pitch in the first stage are included as the second stage estimation process. FIGS. 13 and 14 are diagrams for explaining a first example and a second example of the positional deviation estimation processing in step S1, respectively.

For the first example, referring to FIG. 13, first, the image processing apparatus 1 uses one input image as a reference image, and an input image other than the reference image and having a temperature change as a reference image. . Then, the image processing apparatus 1 uses the template (rectangular frame in the figure) in the standard image to the position of the viewpoint (lens) of the standard image and the viewpoint (lens) of the reference image with respect to the corresponding position of the reference image. Coarse estimation (# 1) is performed in which the corresponding position is identified by matching with a coarse pitch (for example, about 4 pixel increments) in the direction of the position. This is because the deformation (deviation) due to temperature change and the direction in which the parallax appears are substantially the same as described above.

For example, the input image E from the lens E is set as a standard image, and the input image A from the lens A is set as a reference image. A case will be described in which the pixel position of the reference image corresponding to the position (x, y) = (100, 150) of the standard image is searched by the positional deviation estimation processing according to the first example. At this time, the image processing apparatus 1 performs (−4, −4), (0, 0), (4, 4), (8, 8), (12, 12) with respect to the corresponding position of the reference image. , (16, 16), (20, 20), template matching is performed with 8 pixels shifted by (24, 24). That is, the image processing apparatus 1 has the positions (x, y) = (96, 146), (100, 150), (104, 154), (108, 158), (112, 162), (116) of the reference image. , 166), (120, 170), (124, 174), template matching is performed. Then, the image processing apparatus 1 identifies the pixel having the highest degree of matching. The image processing apparatus 1 switches the direction of the pixel to be shifted according to the position of the parallax (lens) of the input image as the reference image (angle formed with the lens of the reference image). For example, when the reference image is the input image B from the lens B, the image processing apparatus 1 performs template matching while shifting the vertical image at a pitch of, for example, 4 pixels.

An example of the template matching process here is NCC (Normalized Cross Correlation). Other examples include SAD (Sum of Absolute Difference) and SSD (Sum of Squared Difference).

Next, the image processing apparatus 1 has a finer pitch around the position identified as the highest in the template in the first stage estimation process (# 1) than in the first stage estimation process (# 1). Precise estimation (# 2) is performed in which the corresponding position is specified by matching (for example, in increments of 1 pixel).

For example, when the position specified in the first stage estimation process (# 1) is (x, y) = (104, 154), the image processing apparatus 1 surrounds the pixel with the accuracy of one pixel. A search is performed in a planar manner (template matching), and matching pixels are identified as (x, y) = (102, 155).

“Rough pitch” is not limited to the direction of shifting pixels for template matching. As a second example, the size of the template may be different between the first stage estimation process (# 1) and the second stage estimation process (# 2). That is, referring to FIG. 14, in the first stage estimation process (# 1), the image processing apparatus 1 performs template matching using a 65 × 65 pixel size template when the input image is 500 × 500 pixels. Shall. In this case, preferably, the image processing apparatus 1 reduces the input image from 500 × 500 pixels to a size of ¼ horizontal and vertical to 125 × 125 pixel size, and a 17 × 17 pixel size template for the input image. Template matching may be performed using Thereby, the template matching process can be speeded up. Preferably, the image processing apparatus 1 extracts 125 pixels by 125 pixels every four pixels, instead of reducing the input image from 500 × 500 pixels to a size of ¼ length and width to 125 × 125 pixels. It may be converted into a low-pixel image having a pixel size, and template matching may be performed on the input image using a template having a size of 17 × 17 pixels. Thereby, the template matching process can be further accelerated.

Further, as a second example, the image processing apparatus 1 performs a planar search (template matching) around the pixel specified in the first stage estimation process (# 1) in the second stage estimation process (# 2). )), It may be searched in a pyramid shape using a reduced image. That is, referring to FIG. 14, as the first step of the second stage estimation process (# 2), the image processing apparatus 1 reduces the input image of 500 × 500 pixels to a size of 1/4 in the vertical and horizontal directions. The size is set to 125 × 125 pixels, and template matching is performed for nine locations of ± 1 pixel from the pixels specified using the 17 × 17 pixel size template for the input image. The template matching range corresponds to a range of ± 4 pixels from the specified pixel of the original image before reduction. Next, as the second step of the second-stage estimation process (# 2), the image processing apparatus 1 reduces the input image of 500 × 500 pixels to a size of 1/2 in the vertical and horizontal directions to a size of 250 × 250 pixels. Then, template matching is performed on nine locations of ± 1 pixel from the pixels specified by using a 17 × 17 pixel size template for the input image. The template matching range corresponds to a range of ± 2 pixels from the specified pixel of the original image before reduction. Next, as a third step of the second stage estimation process (# 2), the image processing apparatus 1 is specified using a 17 × 17 pixel size template without reducing a 500 × 500 pixel input image. Template matching is performed for nine locations of ± 1 pixel from the selected pixel. Similar to the first stage estimation process (# 1), the image processing apparatus 1 extracts the input image every several pixels and converts it into a low pixel image instead of generating a reduced image, thereby performing template matching. Processing can be further accelerated.

The image processing apparatus 1 changes the search direction and the search pitch in the positional deviation estimation process in step S1 to change the first stage estimation process (rough estimation (# 1)) and the second stage estimation process (fine estimation). By executing (# 2)), it is possible to reduce the number of times of template matching (the number of searches) without reducing the estimation accuracy. Therefore, it is possible to estimate the position shift (search for a corresponding pixel) at high speed at the pixel level.

(Subpixel estimation)
The process of estimating the position shift in sub-pixel units (decimal pixels) in step S2 is a pixel group including pixels in the vicinity of the pixels estimated as corresponding points in the position shift estimation process in step S1 (for example, surrounding 8 pixels). 9 pixels) including processing for calculating the degree of matching (for example, NCC value) of each template matching. FIG. 15 shows a specific example of the NCC values of the pixels estimated as corresponding points and the surrounding eight pixels.

In step S2, quadratic surface fitting is performed based on the degree of coincidence, and the positional deviation in units of subpixels is estimated. In the positional deviation estimation in sub-pixel units (decimal pixels) in step S2, as an example, the paper “Robust super-resolution processing based on pixel selection considering positional deviation amount” shown in FIG. 16 (electronic information communication) The method described in the academic journal, Vol. J92-D, No. 5, pp. 650-660, 2009, May 2009) can be employed. That is, as shown in FIG. 15 and FIG. 16, the coordinates of the pixel having the highest matching degree among the matching degrees for each pixel are specified as the shift amount.

In step S2, instead of the quadric surface fitting described above, other methods such as quadric surface fitting using the X coordinate and the Y coordinate may be employed.

(Distance calculation)
FIG. 17 is a diagram illustrating the shift amount of the corresponding pixel with respect to the reference image (input image E) for each input image as a pixel value (color density). In FIG. 17, with respect to an input image (input images A, C, G, and I) in which the direction of the viewpoint from the viewpoint of the reference image (input image E) is oblique, the shift amounts in the X and Y directions are different. It is represented by As an example, the image processing apparatus 1 adds the shift amount of each input image shown in FIG. 17 in the distance calculation processing in step S3.

FIG. 18A is a diagram showing the addition result of the shift amount of each input image represented by the pixel value in each diagram of FIG. As shown in FIG. 17, since the amount of shift differs for each input image, the overall offset amount differs. However, an image obtained by adding pixel values representing the shift of each input image as it is is shown in FIG. Become. In addition, as described above, for the input image in which a shift in the oblique direction occurs according to the position of the viewpoint (lens) as in the relationship of the input image A with respect to the input image E that is the reference image, the X direction and the Y direction are respectively added. Instead, an oblique shift according to the viewpoint (lens) direction may be calculated and added.

FIG. 18A shows that the shift amount is larger as the pixel value is higher (white color) and the shift amount is lower as the pixel value is lower (dark color). The closer the distance from the camera to the subject, the greater the amount of deviation, and the farther the distance, the smaller the amount of deviation. Therefore, in FIG. 18A, a subject having a high pixel value (white color) has a shorter distance from the camera to the subject, and a subject having a low pixel value (dark color) has a distance from the camera to the subject. The distance will be far. Accordingly, FIG. 18A can be said to be a distance image representing the distance from the camera 22 to the subject.

For comparison, FIG. 18B shows an image obtained by adding pixel values representing the shift of each input image when there is no temperature change. By comparing FIG. 18A and FIG. 18B, the relative amount of distance from the camera 22 to the subject (difference in distance) can be changed even if the lens 22a is affected by the temperature change. It can be seen that it can be recognized with the same accuracy as when there is no influence of.

(Refocus processing)
The refocus processing includes, as preparation processing, processing for correcting a distance image and processing for dividing the distance from the camera 22 to the subject in stages. FIG. 19 is a diagram showing the flow of the preparation process in the refocus process in step S4. Referring to FIG. 19, as a process for correcting the distance image, the image processing apparatus 1 generates a distance generated by using one input image (input image E) of the input images as a reference image (FIG. 19A). The image (FIG. 19B) is corrected so that the distance from the camera 22 is aligned in several steps for each subject (FIG. 19C). Here, for example, a joint bilateral filter is preferably used. As another example, an averaging filter may be used in which neighboring pixels are used and weighted as the color is closer. Thereby, the distance image of FIG. 19C is generated from the distance image of FIG. 19B.

Next, as an example, the image processing apparatus 1 divides the corrected distance into eight stages as a process of dividing the distance from the camera 22 to the subject. Thereby, the distance image of FIG. 19D is generated from the distance image of FIG. 19C.

FIG. 20 is a diagram showing the flow of refocus processing. The image processing apparatus 1 performs a refocus process, and creates a refocus image with blurring based on FIG. With reference to FIG. 20, for example, a case will be described in which a refocus image is created by focusing on a lower right subject (stuffed animal) of a captured image. In this case, the image processing apparatus 1 performs a refocus process to add stronger blur to a subject portion having a different distance from the subject to be focused. An example of a method for generating an image with blur is a method for applying a smoothing filter to an input image. Examples of the smoothing filter include a Gaussian filter and an averaging filter. In the following description, a case where a 3 × 3 averaging filter is used a plurality of times will be described as an example. Instead of using multiple times, the kernel size may be increased such as 5 × 5 or 7 × 7.

Note that preferably, the image processing apparatus 1 combines the images in order starting from the blur corresponding to the far distance. As a result, the same blur as when viewed by a person can be expressed on the image, and the display quality can be improved.

21 to 23 are diagrams showing specific examples of refocus images generated by performing the above-described processing in the image processing apparatus 1. FIG. 21 is an image in which a subject close to the lower left of the screen is focused from the camera 22, and FIG. 22 is an image in which the subject at the lower right of the screen is focused at a medium distance from the camera 22, FIG. 23 shows an image in which the object at the upper left of the screen is in focus and is far from the camera 22.

[Second Embodiment]
The image processing apparatus 1 according to the second embodiment performs super-resolution as an example of image processing using an input image even when there is a temperature change as an example of an environmental change with respect to the camera 22. Execute the process.

<Operation flow>
FIG. 24 is a flowchart showing an operation flow in the image processing apparatus 1 according to the present embodiment. Referring to FIG. 24, as in the first embodiment, nine input images are acquired by executing the process of acquiring the input image with each of the lenses 22a shown in FIG. Here, for example, an image having a resolution of about 500 × 500 pixels is input.

When the input image is acquired, a process for estimating the positional deviation in pixel units (integer pixels) is executed (step S1 '). Next, preferably, a process of estimating a positional deviation in sub-pixel units (decimal pixels) is executed (step S2 '). Then, super-resolution processing as image processing is executed in consideration of the amount of positional deviation (step S5), and a high-resolution image of about 1500 × 1500 pixels is generated as an output image.

(Position misalignment estimation)
The positional deviation estimation processing in step S1 ′ and step S2 ′ is the same as the positional deviation estimation processing in step S1 and step S2 in the image processing apparatus 1 according to the first embodiment. However, the image processing apparatus 1 according to the second embodiment performs the same processing by exchanging the base image and the reference image.

FIG. 25 and FIG. 26 are diagrams for explaining a first example and a second example of the positional deviation estimation processing in step S1 ', respectively. The first example and the second example correspond to the first example and the second example of the misregistration estimation process in the first embodiment, which are shown in FIGS. 13 and 14, respectively. is there.

That is, with reference to FIG. 25, even in the misregistration estimation process (step S1 ′) according to the second embodiment, the image processing apparatus 1 uses one input image as a reference image and an input image other than the reference image. With the input image having a temperature change as a reference image, the template in the standard image is compared with the position of the viewpoint (lens) of the standard image and the position of the viewpoint (lens) of the reference image with respect to the corresponding position of the reference image. Coarse estimation (# 1) is performed in which the corresponding position is specified by matching with a coarse pitch (for example, in units of about 4 pixels) in the formed direction. In the second embodiment, the image processing apparatus 1 uses the standard image and the reference image in reverse as described above. That is, the image processing apparatus 1 according to the second embodiment uses the input image E, which is the standard image in the first embodiment, as a reference image, and other input that is the reference image in the first embodiment. An image (for example, input image A) is used as a reference image. Then, the image processing apparatus 1 according to the second embodiment performs template matching similar to that in the first embodiment.

Referring to FIG. 26, as in the first embodiment, the template size may be made different between the first stage estimation process (# 1) and the second stage estimation process (# 2). Good. That is, in the first stage estimation process (# 1), when the input image is 500 × 500 pixels, the image processing apparatus 1 performs template matching using a 65 × 65 pixel size template. Further, as a second example, the image processing apparatus 1 performs a planar search (template matching) around the pixel specified in the first stage estimation process (# 1) in the second stage estimation process (# 2). ), As shown in FIG. 26 (similar to the first embodiment), a reduced image may be used to search in a pyramid shape. Also in this case, the image processing apparatus 1 according to the second embodiment uses the input image E, which is the standard image in the first embodiment, as a reference image, and the reference image in the first embodiment. The other input image (for example, input image A) is used as the reference image.

(Super-resolution processing)
FIG. 27 is a diagram showing the flow of super-resolution processing in step S5. In FIG. 27, as a specific example, the paper “Fast and Robust Multiframe Super Resolution” (IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 13, NO. 10, OCTOBER 2004 page. 1327-1344) is described. The flow of resolution processing is shown.

Referring to FIG. 27, in step # 31, one of the nine input images is subjected to an interpolation process such as a bilinear method, and the resolution of the input image is the resolution after the super-resolution process. As a result, an output candidate image as an initial image is generated.

In step # 32, a BTV (Bilateral Total Variation) amount as a constraint term for robust convergence to noise is calculated.

In step # 33, the generated output candidate images are compared with the nine input images, and a residual is calculated. In step # 34, the residual and the BTV amount calculated from the output candidate image generated in step # 31 are reduced, and the next output candidate image is generated.

The processes in steps # 31 to # 34 are repeated until the output candidate image converges, and the converged output candidate image is output as an output image after the super-resolution processing.

The number of iterations may be a predetermined number of times such as the number of times of convergence (for example, 200 times), or a convergence determination may be made for each series of processing, and may be repeated according to the result. .

FIG. 28 is a diagram for explaining the deterioration information used in step # 33. FIGS. 29 to 33 are diagrams for explaining the processing in step # 33.

Deterioration information refers to information representing the relationship of each input image with respect to a high-resolution image after super-resolution processing, and is represented, for example, in a matrix format. Degradation information includes a shift amount F (for the remaining small number of translated pixels) of each input image, a downsampling amount D (1/3 in each of the vertical and horizontal directions in the above example), and a blur amount H (PSF). ) Etc. are included.

Referring to FIG. 28, the deterioration information is defined by a matrix indicating the conversion when each of the input image and the high-resolution image after super-resolution processing is expressed as a one-dimensional vector.

In step # 33, first, referring to FIG. 29, an input size image DkHkFkXn is generated by applying deterioration information DkHkFk for the input image Yk to the output candidate image Xn generated from the input image Yk. (Step # 41). Specifically, referring to FIG. 30, first, the output candidate image Xn is subjected to filter processing using a matrix having a coefficient of PSF, which is a parameter representing the blur amount Hk, for the processing here (see FIG. 30). Step S11).

Then, down-sampling is performed in consideration of the deviation (step S12). As a result, an input size image DkHkFkXn is output. As an example, as shown on the right side of FIG. 30, when the shift amount F is a shift of 1/3 pixel in the X direction and the downsampling amount D is 1/3 vertical and horizontal, it is filtered. One pixel at which the average value of the pixels in the rectangular area of nine pixels at the position shifted by 1/3 pixel of the subsequent output candidate image Xn is shifted by 1/3 pixel of the input image DkHkFkXn Is output as a minute pixel value (step S13).

Next, referring to FIG. 31, the output candidate image DkHkFkXn, which is an input size image, is compared with the input image Yk, and an image sign (DkHkFkXn-Yk) of the difference is recorded (step # 42). In the example of FIG. 31, as an example, an example in which the sign function sign is applied to the difference between the output candidate image DkHkFkXn and the input image Yk is shown.

Next, as used in the convergence calculation of the super-resolution processing, the influence of which pixel in the super-resolution image is affected by the above-described difference, and the influence represented by the calculated difference on the super-resolution image. In order to return to the corresponding pixel, processing for returning the image sign (DkHkFkXn−Yk) of the difference to the image size after super-resolution is performed (step # 43). As described above, when the deterioration information Dk, Hk, Fk is defined by a matrix, the transposed matrix is used in step # 43. That is, referring to FIG. 32, by applying the transposed matrix Dk ^T Hk ^T Fk ^T of the degradation information DkHkFk for the input image Yk to the recorded difference image sign (DkHkFkXn−Yk), the post-super-resolution Return to image size. In detail, referring to FIG. 33, when the difference image sign (DkHkFkXn-Yk) is generated as described above (step S14), the processing here is up-sampled in consideration of the deviation (step S14). Step S15). As an example, as shown on the right side of FIG. 33, when the shift amount F is a shift of 1/3 pixel in the X direction and the downsampling amount D is 1/3 vertical and horizontal, the difference image sign. A rectangular area of 9 pixels at a position shifted by 1/3 pixel, where the pixel value (difference value) of 1 pixel at a predetermined position of (DkHkFkXn−Yk) is equally divided (9 equal parts in this example) Are output to each pixel. After this processing, a filter transpose matrix using PSF as a coefficient representing the blur amount Hk as a coefficient is applied to return the effect of the blur amount Hk (step S16), and the processed image is super-resolution. The difference image Dk ^T Hk ^T Fk ^T sign (DkHkFkXn−Yk) returned to the later image size is output (step S17). Through the above processing, a difference image Dk ^T Hk ^T Fk ^T sign (DkHkFkXn−Yk) for one input image Yk is created.

FIG. 34 is a flowchart showing an example of the residual calculation process in step # 33. Since there are nine input images in this example, the processes of steps S11 to S17 are repeated for the number of sheets as shown in FIG. 34 in the residual calculation process of step # 33. That is, with reference to FIG. 34, the above-described processing is performed as steps S11 to S17 for the first input image Y1, and an image having a difference in image size after super-resolution is output. The above processing is performed for Y2 as Steps S21 to S27, and an image having a difference in image size after super-resolution is output... The image of the difference in image size after super-resolution is output.

Then, the total value is output as a residual (step S100). In the above example, the processing order of the deviation amount F and the blur amount H is opposite to that described in the paper. However, since blur occurs in a certain large range of the image, the order is substantially changed even if the order is changed. There is no impact.

Note that the processing of the matrix of the blur amount H is equivalent to the following filter processing. That is, for the processing of the blur amount H, using the input pixel value I and the output pixel value I ′,
I ′ (x, y) = ΣΣ PSF (i, j) I (x + i, y + j)
It is expressed as Note that Σ is equal to the PSF filter size (for example, for a 7 × 7 filter, i and j are -3 to 3 respectively).

Regarding the processing of the transpose matrix of the blur amount H,
I ′ (x + i, y + j) = ΣΣPSF (i, j) I (x, y)
It is expressed as That is, the output pixel value I ′ is a total value obtained by processing all the pixels of the input pixel value I.

Note that the super-resolution processing is not limited to the processing shown in FIG. 27, and other processing is possible as long as it is a reconfigurable super-resolution processing that generates one image from a plurality of input images. May be adopted. FIG. 35 is a diagram showing another example of the super-resolution processing in step S5. That is, referring to FIG. 35, the constraint term calculated in step # 32 may be replaced with the BTV amount, for example, another one such as 4-neighbor Laplacian (step # 32 ').

Also, the residual calculation process in step # 33 may be a different calculation method. As an example, the difference between the output candidate image DkHkFkXn and the input image Yk may be directly used without using the sign function sign.

[Third Embodiment]
You may combine said 1st Embodiment and 2nd Embodiment. In other words, the image processing apparatus 1 according to the third embodiment uses an input image as an example of image processing even when there is a temperature change as an example of an environmental change with respect to the camera 22. Perform resolution processing and refocus processing.

FIG. 36 is a flowchart showing an operation flow in the image processing apparatus 1 according to the present embodiment. The image processing apparatus 1 according to the third embodiment performs a refocus process that is an image process according to the first embodiment and a super-resolution process that is an image process according to the second embodiment. By performing this operation on the input image, a refocus image and a super-resolution image are obtained and combined. That is, with reference to FIG. 36, steps S1 to S4 (FIG. 12) in the image processing apparatus 1 according to the first embodiment and steps in the image processing apparatus 1 according to the second embodiment. The processing of S1 ′ and S2 ′ (FIG. 24) is performed on the input image to obtain a refocus image and a super-resolution image. Note that, as shown in FIG. 36, the pixel size is different between the refocus image and the super-resolution image. Therefore, the image processing apparatus 1 performs resolution conversion such as bilinear interpolation on the refocused image obtained by the processing in steps S1 to S4 (step S6), and then combines it with the super-resolution image (step S7). ).

FIG. 37 is a diagram for explaining the composition processing in step S7. Referring to FIG. 37, image processing apparatus 1 performs resolution conversion in step S6 of the refocus image for pixels (regions) to which blur is added during the refocus processing in step S4 of the refocus image. Used pixels.

On the other hand, for the focused area in the refocus image, only the most focused area (pixel) is used as it is as the pixel obtained by the super-resolution processing in step S5. For the peripheral area of the most focused area (pixel), the pixels obtained by the super-resolution processing in step S5 in the refocus image are averaged using, for example, a 3 × 3 averaging filter. Using a 3 × 3 averaging filter during the refocusing process has substantially the same resolution as averaging a super-resolution image using a 9 × 9 averaging filter. Therefore, by averaging in this way, it is possible to smoothly change between the blurred area and the super-resolution area. As shown in FIG. 38, the peripheral area of the focused area (pixel) in the refocused image that is averaged using the 3 × 3 averaging filter in FIG. The focus image may be used after resolution conversion.

[Fourth Embodiment]
The image processing apparatus 1 according to the fourth embodiment is configured to perform image processing using an input image from the camera 22 even when there is a temperature change as an example of an environmental change with respect to the camera 22. Calculate the distance to the subject accurately.

<Operation flow>
FIG. 39 is a flowchart showing an operation flow in the image processing apparatus 1 according to the present embodiment. Referring to FIG. 39, the image processing apparatus 1 according to the fourth embodiment estimates the positional deviation in the same manner as the image processing apparatus 1 according to the first to third embodiments (step S1). Then, the misalignment amount is corrected (step S8).

40 to 42 are diagrams for explaining a method of correcting the positional deviation amount in step S8. The distance image representing the shift amount of the corresponding pixel with respect to the reference image (input image E) for each input image shown in FIG. 40 indicates that the shift amount is entirely different for each image. Therefore, as shown in FIG. 41, the image processing apparatus 1 calculates the average value of the shift amounts for each image, and specifies the minimum value from the 12 average values. Then, the difference from the minimum value for each image is set as the correction amount. The image processing apparatus 1 subtracts the correction amount from the shift amount for each image. FIG. 42 shows the result of correcting each figure of FIG. By performing this correction, the average values of the images are aligned.

39, the image processing apparatus 1 extracts erroneous data (noise) from the corrected image in step S8 (step S9). 43 to 45 are diagrams for explaining a method of extracting error data in step S9. The image processing apparatus 1 creates a histogram for the corresponding pixel positions of the 12 images after correction. Then, the image processing apparatus 1 determines that the pixel value (deviation amount) deviating from the histogram is an error. For each figure, the image processing apparatus 1 determines the entire image, for example, by creating a histogram at a pitch of 3 pixels.

FIG. 44 shows a histogram created for each corresponding pixel position with respect to the circle at the upper right of each of the 12 images in FIG. The image processing apparatus 1 specifies a location where the histogram value is the maximum of two consecutive pixel values in the histogram of FIG. 44, and determines pixel values other than that location as erroneous data. In the histogram of FIG. 44, the place where the histogram value is the maximum is specified as the place A having the pixel values 0 to 6, and the pixel having the pixel value indicated in this place A is correct (the pixel having an appropriate pixel value). It is determined. The histogram value at the location B near the pixel value 30 is determined to be erroneous data.

45 is a diagram showing the determination result of the error data for each of the 12 images in FIG. A white area in the figure represents an area having a pixel value determined to be erroneous data.

Returning again to FIG. 39, when the image processing apparatus 1 determines the error data (noise) for each input image by the method described above in step S9, the image processing apparatus 1 adds using the pixels excluding the pixels that are erroneous data in each figure. By performing the process, the distance is estimated (step S10). FIG. 46 shows a distance image obtained by adding the respective figures excluding the pixels shown in white in FIG.

In the addition processing in step S10, pixels that are determined to be correct for all input images are added as they are. However, for example, as shown in the histogram of FIG. 44, only 10 of 12 images are determined to be correct. 10 pixels are added and then multiplied by 12/10 to match the pixel weight.

When the distance image generated by adding all the images in FIG. 18A is compared with the distance image generated by the above processing in FIG. 46, the distance image in FIG. Pixel values appearing in the upper left and upper right are removed as erroneous data (noise). From this, it can be seen that the correct distance image can be created by adding after removing the erroneous data in step S10.

[Modification 1]
In the above description, it is assumed that the pixel shifts from the corresponding position assumed by the pixel due to the temperature change as an example of the change in the environmental condition. However, the environmental change is not limited to the temperature change. Another example is a change with time (elongation, deflection, etc.) of a mechanical part such as a holding part of the lens 22a (not shown). Even in this case, the image processing apparatus 1 can perform the corresponding point search in the same manner as in step S1. Also, as described in the first embodiment, the present invention can be applied even when calibration (distortion correction or parallelization processing) is not performed. That is, if there is some error factor, the image processing apparatus 1 can be adapted by searching the corresponding pixel by switching the search direction and the search pitch in two stages according to the error accuracy.

[Modification 2]
In the above description, the image processing apparatus 1 uses an input image from the camera 22 that is an array camera in which lenses are arranged as shown in FIG. 5, but the lens is not limited to that shown in FIG. 47 and 48 are diagrams showing other examples of lenses included in the camera 22. FIG. That is, in the above description, it is assumed that the array lenses A to P are held at the four corners. However, as shown in FIG. 47, the array lenses A to P may be held only at one of the four corners (for example, upper left). In this case, as shown in FIG. 47, the deformation of each lens is asymmetrical, and is deformed substantially radially around the held corner. However, the positional deviation of each lens with respect to the lens A closest to the held corner is the direction of parallax of the input image from each lens around the lens A (the lens It expands and contracts in the direction of placement. Therefore, even in such a case, the image processing apparatus 1 can search for corresponding points by performing the positional deviation estimation process similar to that in step S1.

Also, as shown in FIG. 48, when the number of lenses 22a included in the camera 22 is 16 (16 eyes), the direction of positional displacement of each lens 22a due to environmental changes such as temperature changes is the same as described with reference to FIG. The direction (parallax direction) between the reference lens and the target lens. Therefore, even in such a case, the image processing apparatus 1 can search for corresponding points by performing the positional deviation estimation process similar to that in step S1.

<Effect of Embodiment>
When the lens 22a included in the camera 22 which is an array camera is affected by an environmental change such as a temperature change, the lens 22a may expand and contract as a whole and the lens interval (pitch) may fluctuate. In particular, when the lens 22a and its holding member are materials that are easily affected by the environment, such as plastic, the variation easily appears. However, the variation does not necessarily occur equally. Therefore, the corresponding points necessary for image processing such as distance calculation processing, super-resolution processing, and refocus processing may shift asymmetrically.

The image processing apparatus 1 according to the present embodiment uses the feature that in the corresponding point search, the arrangement direction of each lens (the direction in which the parallax appears) and the direction of the pitch fluctuation due to the temperature change are approximately the same direction. , Search in the direction with coarse accuracy (pitch) as the first step, and search in a planar manner without limiting the direction with finer accuracy (pitch) than the first step around the searched pixels when matching as the second step . This makes it possible to search for corresponding points at high speed and with high accuracy.

The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 2 Imaging part, 3 Image processing part, 4 Image output part, 22 Camera, 22a lens, 22b Image sensor, 24 A / D conversion part, 31 Pre-processing part, 32 Position shift estimation part, 33 Processing part , 100 digital camera, 102 CPU, 104 digital processing circuit, 106 image processing circuit, 108 image display unit, 112 storage unit, 114 camera unit, 200 personal computer, 202 personal computer body, 204 image processing program, 206 monitor, 208 mouse , 210 keyboard, 212 external storage device, 321 first estimation unit, 322 second estimation unit, 323 third estimation unit.

Claims (10)

One input image of a multi-viewpoint input image group each having a common partial area is used as a reference image, and a corresponding pixel in an input image other than the reference image is estimated for one pixel on the reference image. An image processing apparatus,
Template matching is performed at a first pitch in a direction corresponding to a direction between the viewpoint of the reference image and the viewpoint of the input image other than the reference image from a position corresponding to the one pixel of the input image other than the reference image. A first estimation unit for estimating a pixel corresponding to the one pixel;
A second pixel for estimating a pixel corresponding to the one pixel is obtained by performing template matching on the pixel estimated by the first estimating unit at a second pitch smaller than the first pitch. An image processing apparatus comprising: an estimation unit. The image processing apparatus according to claim 1, wherein the second estimation unit performs template matching using a template having a size equal to or smaller than a size of a template used in the first estimation unit. A third estimation unit for estimating a shift amount in a decimal pixel based on a shift between the one pixel and a pixel corresponding to the one pixel estimated by the second estimation unit; The image processing apparatus according to claim 1. A processing unit for executing a process of calculating a distance from a viewpoint of the reference image to a subject based on a shift between the one pixel and a pixel corresponding to the one pixel of the input image other than the reference image; The image processing apparatus according to any one of claims 1 to 3, further comprising: The image processing apparatus according to claim 4, wherein the processing unit calculates a distance from a viewpoint of the reference image to the subject by adding the shifts of the plurality of input images other than the reference image. The image processing apparatus according to claim 5, wherein the processing unit calculates a correction amount from the shift of each of the plurality of input images other than the reference image, and corrects the shift. A processing unit for performing super-resolution processing on an input image other than the reference image, using a shift between the one pixel and a pixel corresponding to the one pixel of the input image other than the reference image as a parameter; The image processing apparatus according to any one of claims 1 to 3, further comprising: 8. The image processing apparatus according to claim 1, further comprising a preprocessing unit for performing distortion correction and parallelization processing as preprocessing for each of the input images. One input image of a multi-viewpoint input image group each having a common partial area is used as a reference image, and a corresponding pixel in an input image other than the reference image is estimated for one pixel on the reference image. A method,
Template matching is performed at a first pitch in a direction corresponding to a direction between the viewpoint of the reference image and the viewpoint of the input image other than the reference image from a position corresponding to the one pixel of the input image other than the reference image. A step of estimating a pixel corresponding to the one pixel;
An image processing method comprising: estimating a pixel corresponding to the one pixel by performing template matching on the estimated pixel at a second pitch smaller than the first pitch. In a computer, one input image of a multi-viewpoint input image group each having a common partial area is set as a reference image, and one pixel on the reference image corresponds to an input image other than the reference image. A program for executing a process for estimating
Template matching is performed at a first pitch in a direction corresponding to a direction between the viewpoint of the reference image and the viewpoint of the input image other than the reference image from a position corresponding to the one pixel of the input image other than the reference image. A step of estimating a pixel corresponding to the one pixel;
An image processing program that causes the computer to execute a step of estimating a pixel corresponding to the one pixel by performing template matching on the estimated pixel at a second pitch smaller than the first pitch. .

Images