WO2011121763A1 - Image processing apparatus and image capturing apparatus using same - Google Patents

Abstract

This invention is directed to provision of an image processing apparatus capable of obtaining high definition of images, while correcting the asymmetry of aberration. An image processing apparatus comprises: an image acquiring means for acquiring an input image; and an image recovering means for recovering the input image by use of an image recovery filter, which is generated or selected based on the transfer function of an image capturing system used for forming, as the input image, a subject image, thereby generating a recovered image. The image recovery filter causes the difference in absolute value of transfer function between two azimuthal directions used when obtaining the recovered image from the subject to be smaller than the difference in absolute value of transfer function between the two azimuthal directions of the image capturing system.

Description

Image processing apparatus and imaging apparatus using the same

The present invention relates to an image processing apparatus that performs image processing, and more particularly to an image processing apparatus that performs image restoration (restoration).

An image obtained by an imaging device such as a digital camera is deteriorated due to blur. The image blur is caused by spherical aberration, coma aberration, field curvature, astigmatism, and the like of the imaging optical system. These aberrations can be expressed by a point spread function (PSF, Point Spread Function). An optical transfer function (OTF, Optic Transfer Function) obtained by Fourier transforming a point spread function (hereinafter referred to as PSF) is information in the frequency space of aberration, and is represented by a complex number. The absolute value of the optical transfer function (hereinafter OTF), that is, the amplitude component is called MTF (ModulationＦTransfer Function), and the phase component is called PTF (PhaseＰＴTransfer Function). The OTF of the imaging optical system affects (deteriorates) the amplitude component and phase component of the image. For this reason, an image deteriorated by the influence of the OTF (hereinafter referred to as a deteriorated image) is an image in which each point of the subject is asymmetrically blurred like coma.

This will be described with reference to FIG. FIGS. 13A, 13B, and 13C are schematic diagrams showing the spread of the point spread function (PSF) on the plane perpendicular to the principal ray (the ray passing through the center of the pupil of the optical system). is there. In the plane shown in FIG. 13, lines perpendicular to each other passing through the optical axis are defined as axes x1 and x2, and an angle θ formed by an arbitrary straight line passing through the optical axis and the axis x1 in the plane shown in FIG. And When the origin of the coordinate axis in FIG. 13 is the principal ray imaging position, the azimuth angle θ is the azimuth direction. The azimuth direction is a general term for all directions including the sagittal direction and the meridional direction, and also including the other angle θ directions.

As already mentioned, the deterioration of the phase component (PTF) of aberration causes asymmetry in the PSF. In addition, the deterioration of the amplitude component (MTF) affects the size of the PSF spread for each azimuth direction. FIG. 13A is a diagram schematically showing a PSF in which coma aberration occurs. The PSF of the angle of view other than on the optical axis in the case of an optical system having a rotationally symmetric lens without the eccentricity of the optical axis is symmetric with respect to a straight line passing through the optical axis and the principal ray on the image plane. It has a symmetrical shape. In FIG. 13A, the PSF is line symmetric with respect to the axis x2.

FIG. 13B shows a PSF in a state where there is no phase shift. Although it has a symmetric shape in each azimuth direction, there is a difference in amplitude (MTF), so that the PSF spread is different in the axis x1 direction and the axis x2 direction. By the way, the PSF on the optical axis has a rotationally symmetric shape as shown in FIG. 13C because there is no phase shift and no azimuth dependency of amplitude deterioration if manufacturing errors are ignored. That is, as shown in FIGS. 13A and 13B, the PSF has an asymmetric shape due to the difference in the phase (PTF) in each azimuth direction and the difference in amplitude (MTF) between the azimuth directions. This is a factor that hinders the generation of high-definition images.

As a technique for correcting such image blurring, Patent Document 1 provides a parameter α for designing an image restoration filter as shown in Equation 1. By adjusting the adjustment parameter α, the image restoration filter changes from a filter (α = 0) that does nothing to an inverse filter (α = 1). As a result, the degree of image recovery can be adjusted with one parameter in the range from the original photographed image to the image recovered to the maximum.

ここで、Ｆ（ｕ，ｖ）、Ｇ（ｕ，ｖ）はそれぞれ、回復画像および劣化画像のフーリエ変換である。

Here, F (u, v) and G (u, v) are Fourier transforms of the restored image and the degraded image, respectively.

Other than that, a Wiener filter is known as a filter that corrects image blur and improves sharpness. Equation 2 shows the frequency characteristic M (u, v) of the Wiener filter.

ここで、Ｈ（ｕ、ｖ）は光学伝達関数（ＯＴＦ）である。｜Ｈ（ｕ，ｖ）｜はＯＴＦの絶対値（ＭＴＦ）である。ＳＮＲはノイズ信号の強度比である。

Here, H (u, v) is an optical transfer function (OTF). | H (u, v) | is the absolute value (MTF) of the OTF. SNR is the intensity ratio of the noise signal.

Hereinafter, a process of performing image restoration using an image restoration filter based on an optical transfer function (OTF) as described in the Wiener filter or Patent Document 1 is referred to as an image restoration process.

JP 2007-183842 A

However, the above-described Wiener filter and the image restoration filter of Patent Document 1 can correct the deterioration of the amplitude component and the phase component due to the imaging optical system, but cannot correct the difference in the amplitude component between the azimuth directions. In the Wiener filter, if there is a difference between the azimuth directions of the MTF before the recovery, the difference in the azimuth direction of the MTF after the recovery is expanded. This will be described with reference to FIG.

FIG. 14 is a diagram showing the MTF before the image restoration process and the MTF after the image restoration process using the Wiener filter. The broken line (a) and the solid line (b) are the MTFs in the first azimuth direction and the second azimuth direction before recovery, respectively. The broken line (c) and the solid line (d) are the MTFs in the first azimuth direction and the second azimuth direction after recovery, respectively. The first and second azimuth directions are, for example, the sagittal direction and the meridional direction. The Wiener filter is an image restoration filter that lowers the recovery gain (recovery degree) when the MTF is high and increases the recovery gain (recovery degree) when the MTF is low. Therefore, the broken line (a), which is the azimuth direction with a low MTF, has a lower recovery gain than the azimuth direction (b) with a high MTF. Therefore, the MTF (c) in the first azimuth direction after the recovery and the MTF (d) in the second azimuth direction after the difference between the MTF (a) in the first azimuth direction before the recovery and the MTF (b) in the second azimuth direction. ) Is enlarged. That is, asymmetric aberrations appear in the image despite the image restoration process. The same applies to the image restoration filter disclosed in Patent Document 1.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus capable of reducing asymmetric aberrations that can be caused by image restoration processing and obtaining a higher definition image.

In order to solve the above problems, the present invention
Image acquisition means for acquiring an input image;
Image recovery means for recovering the input image using an image recovery filter generated or selected based on a transfer function of an imaging system used to form a subject image as the input image and generating a recovered image; And
The image restoration filter has a difference in absolute value of a transfer function between two azimuth directions in obtaining a restored image from a subject smaller than a difference in absolute value of a transfer function between the two azimuth directions of the imaging system. It is characterized by doing.

The effect of the present invention is that a high-definition restored image with reduced asymmetric aberration can be generated.

Explanatory drawing of the image processing method which is an Example of this invention Explanatory drawing of the image restoration filter used with the image processing method of an Example 1st explanatory drawing about the correction amount of MTF between each azimuth direction 2nd explanatory drawing regarding the correction amount of MTF between each azimuth direction Block diagram showing the basic configuration of the imaging device Explanatory drawing regarding selection and correction of an image restoration filter in the first embodiment Flowchart showing the image processing procedure of the first embodiment. The figure which shows the MTF change before and after the image processing of Example 1. Illustration of edge enhancement filter Edge cross section with edge enhancement filter applied Explanatory drawing of the image processing system of Example 2 of this invention. Explanatory drawing of the correction information of Example 2 of this invention Illustration of azimuth direction The figure which shows the MTF change before and behind the conventional image restoration process Explanatory diagram of point spread function (PSF) Explanatory drawing of amplitude component (MTF) and phase component (PTF) of image

First, the flow of image processing according to the present invention will be described with reference to FIG. FIG. 1 shows processing steps from image input to output. First, in the image acquisition process of step S11, an image generated by the imaging system is acquired. Hereinafter, the image acquired in the image acquisition process is referred to as an input image.

Next, in step S12, an image restoration filter corresponding to the shooting condition of the input image acquired in step S11 is generated. This step S12 may be a step of selecting an appropriate filter from a plurality of image restoration filters prepared in advance, or may be a step of appropriately correcting the filter after selecting the filter. . In the image restoration processing step of step S13, image restoration processing (correction processing) is performed using the image restoration filter generated or selected based on the transfer function of the imaging system (optical transfer function of the imaging optical system) in step S12. Execute. More specifically, the phase component (PTF) of the input image is corrected to zero as the target value, and the amplitude component (MTF) is corrected so that the difference between the two azimuth directions decreases. In step S14, the corrected image corrected in step S13 is output as an output image.

It should be noted that other image processing steps may be inserted before, after, or during the process of FIG. The other image processing is processing such as electronic aberration correction such as distortion aberration correction and peripheral light amount correction, demosaicing, gamma conversion, and image compression. Next, each step shown in FIG. 1 will be described in more detail.

MTF is the amplitude component (absolute value) of the transfer function of the image pickup system (optical transfer function of the image pickup optical system), but when the subject (object in the object field) is a white point light source, the MTF is the spectrum of the image. Can be considered.

(Image acquisition process)
An image acquired by the image acquisition process (hereinafter referred to as an input image) is a digital image obtained by capturing an image with an image sensor via an imaging optical system. The digital image obtained here is compared with an object in the field by an optical transfer function (OTF) based on aberrations of the imaging optical system including the imaging optical system (lens) and various optical filters. It has deteriorated. The optical transfer function (OTF) is preferably a transfer function based on the aberration of the optical element of the imaging optical system as described above and other characteristics of the imaging device. The imaging optical system can also use a mirror (reflection surface) having a curvature in addition to the lens.

Also, the input image is represented by a color space. The color space is represented by RGB, for example. In addition to RGB, there are brightness, hue, saturation expressed by LCH, luminance, color difference signal expressed by YCbCr, and the like. Other color spaces include XYZ, Lab, Yuv, JCh, and color temperature. The present invention can be applied to values represented by these commonly used color spaces as color components.

Further, the input image may be a mosaic image having a signal value of one color component for each pixel, or a color interpolation process (demosaicing process) is performed on this mosaic image, and each pixel has a signal value of a plurality of color components. A demosaic image may be used. The mosaic image is also called a RAW image as an image before image processing such as color interpolation processing (demosaicing processing), gamma conversion, image compression such as JPEG, or the like. For example, when obtaining information on multiple color components with a single-chip image sensor, a color filter with a different spectral transmittance is arranged in each pixel to obtain a mosaic image having a signal value of one color component in each pixel. To do. By performing color interpolation processing on this mosaic image, an image having a plurality of color component signal values at each pixel can be acquired. In addition, when using a multi-plate, for example, three-plate image sensor, a color filter having a different spectral transmittance is arranged for each image sensor, and a demosaic image having image signal values of different color components for each image sensor is obtained. Will get. In this case, since each image sensor has a signal value of each color component for the corresponding pixel, each pixel has a signal value of a plurality of color components without performing color interpolation processing. Images can be acquired.

Also, various correction information for correcting the input image can be attached to the input image. The correction information includes information about the imaging state (imaging state information) such as the focal length (zoom position), aperture value, shooting distance (focus distance), exposure time, and ISO sensitivity of the lens. When a series of processing from imaging to image output is performed by a single imaging apparatus, it is also possible to obtain the input image within the apparatus without adding imaging state information and correction information to the input image. However, when a RAW image is acquired from an imaging device and image restoration processing or development processing is performed by an image processing device separate from the imaging device, imaging state information and correction information are attached to the image as described above. It is preferable. However, the present invention is not limited to this, and if the correction information is stored in the image processing apparatus in advance and the correction information can be selected from the imaging state information attached to the input image, the correction information is not necessarily attached to the image. .

Although the input image is described as a digital image obtained by taking an image with an imaging device via an imaging optical system, the input image may be a digital image obtained by an imaging system that does not include the imaging optical system. For example, a scanner (reading device) or an X-ray imaging device that performs imaging with an imaging element in close contact with the subject surface may be an image obtained by an imaging device that does not have an imaging optical system such as a lens. Although these do not have an imaging optical system, an image generated by image sampling by the imaging device is not a little deteriorated. The deterioration characteristics in this case are not due to the optical transfer function of the imaging optical system (the optical transfer function in a narrow sense), but are due to the system transfer function of the imaging system, and this system transfer function corresponds to the optical transfer function. It can be said. Therefore, the “optical transfer function” referred to in the embodiments of the present invention is an optical transfer function in a broad sense including the system transfer function of such an imaging system that does not include such an imaging optical system.

(Image restoration filter generation process)
Next, generation of an image restoration filter will be described with reference to FIGS. FIG. 2A is a schematic diagram of an image restoration filter in which convolution processing is performed on pixels of an input image in real space. The number of taps (cells) of the image restoration filter can be determined according to the aberration characteristics of the imaging system and the required restoration accuracy. When the image is two-dimensional, generally the number of taps corresponding to each pixel of the image Is a two-dimensional image restoration filter. FIG. 2A shows an 11 × 11 tap two-dimensional image restoration filter as an example. In general, as the number of taps of the image restoration filter increases, the restoration accuracy improves as the number of taps increases. Therefore, the number of taps is set according to the required image quality, image processing capability, aberration characteristics, and the like.

In FIG. 2 (A), values in each tap are omitted, but one section of this image restoration filter is shown in FIG. 2 (B). The distribution of the values (coefficient values) of each tap of the image restoration filter plays a role of ideally returning the signal value spatially spread by the aberration to the original one point during the convolution process.

In order to generate this image restoration filter, first, an optical transfer function (OTF) of the imaging optical system is calculated or measured. When the original image (degraded image) is an image obtained by a system that does not have an imaging optical system, the degradation characteristic can be expressed by an imaging system transfer function. An image restoration filter may be generated as OTF). The expression “optical transfer function (OTF)” used below includes the transfer function of an imaging system that does not have this imaging optical system.

Unlike the conventional image restoration filter, the image restoration filter used in the present invention has a function of correcting the difference between the azimuth directions of the MTF. Prior to describing the process of creating the image restoration filter of the present invention, a conventional winner filter will be described with reference to FIG.

FIG. 15A shows a meridional section of a point spread function (PSF) of a color component at a certain position on the image, and FIG. 16 shows the frequency characteristics thereof. FIG. 16M shows the MTF that is the amplitude component, and FIG. 16P shows the PTF that is the phase component. Further, the frequency characteristics corresponding to the PSFs in FIGS. 15A, 15B, and 15C are a broken line (a), a two-dot chain line (b), and a one-dot chain line (c) in FIG.

The PSF before correction shown in FIG. 15A has an asymmetric shape due to coma and the like, and has an MTF characteristic with a lower amplitude response at higher frequencies as indicated by a broken line (A) in FIG. A phase shift occurs as indicated by a broken line (a) in FIG. When this is corrected using an image restoration filter created by inverse Fourier transform of the reciprocal of the optical transfer function (OTF) (1 / OTF (u, v)), the PSF is ideally as shown in FIG. Is corrected like a delta function with no spread.

Note that the reciprocal of OTF is called an inverse filter, and the degree of recovery by the inverse filter (image restoration filter) is defined as the maximum degree of restoration in this specification.

The MTF corresponding to FIG. 15B is 1 over the entire frequency as shown by the two-dot chain line (b) in FIG. 16M, and the PTF is as shown by the two-dot chain line (b) in FIG. Is zero over the entire frequency.

However, as described above, in creating an image restoration filter, it is necessary to control the influence of noise amplification. The PSF recovered from the PSF in FIG. 15A by the Wiener filter shown in Equation 1 becomes a symmetrical shape by correcting the phase as shown in FIG. 15C, and the PSF is improved by improving the amplitude. The spread is small and sharp. The MTF corresponding to FIG. 15C has a reduced recovery gain as indicated by the one-dot chain line (c) in FIG. 16M, and the PTF is indicated by the one-dot chain line (c) in FIG. Is zero over the entire frequency. The reason why the PTF is corrected to 0 in spite of the suppression of the recovery gain will be described using Equation 3.

Assuming a white point light source as the subject, the frequency characteristic of the subject has no phase shift, and the amplitude characteristic is 1 over the entire frequency. The frequency characteristic of the image obtained through this imaging optical system is the optical transfer function (OTF) itself, and the image has a pixel value distribution in the form of PSF. That is, if the frequency characteristic of the input image is OTF and is multiplied by the frequency characteristic of the image restoration filter, the frequency characteristic of the restored image can be known. If this is expressed by an equation, as shown in Equation 3, H (u, v), which is an OTF, is canceled out, and the frequency characteristic of the recovered image becomes as shown on the right side.

右辺の｜Ｈ（ｕ，ｖ）｜はＯＴＦの絶対値（ＭＴＦ）であるため、回復度合いを決定するパラメータＳＮＲの値に関わらず位相成分は消滅することになる。従って、位相劣化成分が補正されることで、ＰＳＦは対称な形状に補正される。

Since | H (u, v) | on the right side is the absolute value (MTF) of the OTF, the phase component disappears regardless of the value of the parameter SNR that determines the degree of recovery. Therefore, the PSF is corrected to a symmetric shape by correcting the phase degradation component.

However, although the PSF is corrected symmetrically for each azimuth direction by correcting the phase degradation component for each azimuth direction, the amplitude component is different for each azimuth direction, so that it is not corrected for rotational symmetry. For example, the coma aberration, which is the line object in FIG. 13A, is corrected to be point-symmetric so as to be corrected to a PSF like the astigmatism in FIG. That is, for each azimuth direction, a PSF whose phase is corrected and whose amplitude is different is corrected only to a point-symmetric state. In the conventional image restoration method, the difference in MTF between the azimuth directions is not corrected but rather enlarged as described above with reference to FIG. Therefore, the conventional image restoration method cannot sufficiently correct the asymmetric aberration.

Next, the image restoration filter of the present invention having the function of correcting the asymmetric aberration of the present invention will be described. As can be seen from Equations 1 and 3, the rOTF portion of Equation 4 is the frequency characteristic after recovery of an image taken with a white point light source.

ここで、ｒＯＴＦは、任意の関数である。回復画像の位相劣化成分は零であることが望ましいので、ｒＯＴＦは位相成分を持たないようにすれば良く、ｒＯＴＦは実数部のみを有するため実質的にはｒＭＴＦと等しい。ｒＯＴＦは実部のみを有することが好ましいが、許容可能な範囲で虚数部に値を持たせたとしても本発明の変形の範囲内であることはいうまでも無い。つまり式４に表した画像回復フィルタとすることで、点光源に関わらずどのような被写体でも、あたかも光学伝達関数（ＯＴＦ）がｒＯＴＦの特性を持った撮像光学系で撮影された画像のようにすることができるということである。

Here, rOTF is an arbitrary function. Since it is desirable that the phase degradation component of the restored image is zero, it is sufficient that rOTF does not have a phase component. Since rOTF has only a real part, it is substantially equal to rMTF. Although rOTF preferably has only a real part, it goes without saying that even if an imaginary part has a value within an allowable range, it is within the scope of the present invention. In other words, by using the image restoration filter expressed by Equation 4, any subject regardless of the point light source, as if the image was captured by an imaging optical system having an optical transfer function (OTF) characteristic of rOTF. Is that you can.

Therefore, by using Equation 5 using OTF (rH (u, v)) common between the azimuth directions, an image captured with an imaging optical system having no MTF difference between the azimuth directions can be obtained. . That is, the image restoration filter used in this embodiment is a filter that reduces the MTF difference between the two azimuth directions of the imaging system.

This will be described with reference to FIG. FIG. 3 is a diagram showing changes in MTF in two azimuth directions before recovery and after recovery when processing is performed using the image recovery filter of the present invention when the subject is a point light source. The broken line (a) and the solid line (b) are the MTFs before recovery in the first and second azimuth directions, respectively, and the broken line (c) and the solid line (d) are the MTFs after the recovery in the first and second azimuth directions, respectively. Represents. The MTF before recovery differs for each azimuth direction as shown in FIGS. 3A and 3B, but the MTF after recovery is aligned between the azimuth directions as shown in FIGS. 3C and 3D. 3A and 3B correspond to MTFs in the meridional direction and the sagittal direction, for example. As described above, the image restoration filter can be restored while correcting the difference in MTF between the azimuth directions.

Further, in Equation 5, a common OTF (rH (u, v)) is used between the azimuth directions, but the difference in OTF for each azimuth direction of rH (u, v) is smaller than the difference in OTF before recovery. The rotational symmetry can be controlled by correcting as described above. An example is shown in FIG. Even if the recovered MTF between the two azimuth directions does not match as shown in FIGS. 4C and 4D, the MTF difference between the azimuth directions is reduced with respect to FIGS. 14C and 14D. PSF asymmetry will be reduced. In order to obtain an asymmetry correction effect, it is desirable to use a filter that recovers at least so as to be smaller than the difference in MTF between the azimuth directions before recovery.

In the image restoration filter of this embodiment, the difference in MTF (absolute value of transfer function) between two azimuth directions when obtaining a restored image from a subject is larger than the difference in MTF between the two azimuth directions of the imaging system. Configure to be smaller.

In other words, when the subject is a white point light source, the image restoration filter of the present embodiment has a difference in spectrum between the two azimuth directions in the restored image that is smaller than the difference in spectrum between the two azimuth directions in the input image. To recover.

In other words, the image restoration filter of the present embodiment is a corrected transfer function that is corrected so that the difference between the transfer function having different frequency characteristics in the two azimuth directions and the absolute value of the transfer function between the two azimuth directions is reduced. Is generated based on

As described above, by executing the image restoration process using the image restoration filter of the present invention, it is possible to correct the difference between the phase component of each aberration in the azimuth direction and the amplitude component between the azimuth directions, and the asymmetry of the aberration. Therefore, it becomes possible to obtain a higher definition image.

In the image restoration filter, the H (u, v) portion of Equation 5 is different for each azimuth direction, so that rH (u, v) may be common between the azimuth directions, but may be asymmetric. Has a coefficient array. That is, the cross-sectional view of FIG. 2 is different for each azimuth direction.

The optical transfer function (OTF) can include not only the imaging optical system but also a factor that degrades the optical transfer function (OTF) during the imaging process. For example, an optical low-pass filter having birefringence suppresses a high-frequency component with respect to a frequency characteristic of an optical transfer function (OTF). Further, the shape and aperture ratio of the pixel aperture of the image sensor also affect the frequency characteristics. In addition, there are spectral characteristics of a light source and spectral characteristics of various wavelength filters. It is desirable to create an image restoration filter based on an optical transfer function (OTF) in a broad sense including these.

(Image restoration process (correction process))
Next, a method for obtaining a corrected image using the generated image restoration filter will be described.

As already described, in the correction step, a corrected image (recovered image) can be obtained by convolving an image recovery filter with the deteriorated image. Here, convolution (convolution integration, product sum) processing is performed on each pixel included in the tap of the image restoration filter. Convolution makes the pixel coincide with the center of the image restoration filter in order to improve the signal value of that pixel. This is a process of taking the product of the image signal value and the coefficient value of the image restoration filter for each corresponding pixel of the image and the image restoration filter and replacing the sum as the signal value of the central pixel.

The advantage of applying an image restoration filter to the input image or performing convolution processing is that the image can be restored without performing Fourier transform or inverse Fourier transform of the image in the image restoration processing. In general, the load for convolution processing is smaller than the load for performing Fourier transform. Therefore, it is possible to reduce the processing burden when performing the image restoration process.

Although the number of vertical and horizontal taps of the image restoration filter has already been described, the number of vertical and horizontal taps does not necessarily have to be the same, and can be arbitrarily changed as long as they are taken into consideration when performing convolution processing. .

In addition, since the image restoration process of the present invention can process the reverse process for restoring the original image before the degradation with high accuracy when the image degradation process is linear, the input image is subjected to various adaptive nonlinear processes. Is preferably not performed. That is, it is more preferable to carry out with respect to the mosaic image (RAW image). However, the image restoration processing of the present invention can be applied regardless of whether the input image is a mosaic image or a demosaic image. The reason is that if the degradation process by the color interpolation process is linear, the image restoration process can be performed by considering this degradation function in the generation of the image restoration filter. In addition, when the required accuracy of recovery is low or only images that have been subjected to various image processing can be obtained, the effect of reducing blur asymmetry even if image recovery processing is performed on the demosaic image is obtained. Can do.

(Image output process)
The corrected image (recovered image) acquired by the above processing is output to a desired device. If it is an imaging device, it is output to a display unit or a recording medium. If other image processing or the like is performed on the image that has been subjected to the image restoration processing, the image may be output to a device that executes a later process.

As described above, the image processing of the present invention has been described in order for each process. However, if several processes can be processed at the same time, they can be processed together. It is also possible to add necessary processing steps as appropriate before and after each step. Furthermore, the formulas and equal signs used in the description do not limit the specific algorithm of the image processing of the present invention to this, and can be modified as necessary within a range where the object can be achieved.

Hereinafter, an embodiment to which the above-described image processing is applied will be described with reference to the drawings.

FIG. 5 is a schematic diagram of the configuration of the imaging apparatus according to the first embodiment. A subject image (not shown) is formed on the image sensor 102 by the imaging optical system 101. The imaging element 102 converts the imaged light into an electric signal (photoelectric conversion), and the A / D converter 103 converts the electric signal into a digital signal. The image processing unit 104 performs image processing on the digital signal (input image) together with predetermined processing. The predetermined processing here is processing such as electronic aberration correction such as magnification chromatic aberration correction, distortion aberration correction, and peripheral light amount correction, demosaicing, gamma conversion, and image compression.

First, the imaging state information of the imaging device is obtained from the state detection unit 107. The state detection unit 107 may obtain imaging state information directly from the system controller 110. For example, imaging state information regarding the imaging optical system 101 may be obtained from the imaging system control unit 106. Next, an image restoration filter corresponding to the imaging state is selected from the storage unit 108, and image restoration processing is performed on the image input to the image processing unit 104. The image restoration filter selected from the storage unit 108 according to the imaging state may be used as it is, or an image restoration filter prepared in advance and corrected to an image restoration filter more suitable for the imaging state is used. You can also.

Then, the output image processed by the image processing unit 104 is stored in the image recording medium 109 in a predetermined format. This output image is an image in which the asymmetry of the aberration is corrected and the sharpness is improved. Further, the display unit 105 may display an image that has undergone predetermined processing for display on the image after the image restoration processing, or no correction processing is performed for high-speed display, or simple correction. You may display the image which processed.

The above-described series of control is performed by the system controller 110, and mechanical driving of the imaging system is performed by the imaging system control unit 106 according to an instruction from the system controller 110. The aperture of the aperture 101a is controlled as an F number shooting state setting. The focus lens 101b is controlled in position by an unillustrated autofocus (AF) mechanism or manual manual focus mechanism in order to adjust the focus according to the shooting distance.

This imaging system may include an optical element such as a low-pass filter or an infrared cut filter. When using an element that affects the characteristics of an optical transfer function (OTF) such as a low-pass filter, the recovery process can be performed with higher accuracy if the influence of this element is taken into consideration when the image recovery filter is created. It is. Even in the infrared cut filter, it affects each PSF of the RGB channel, particularly the PSF of the R channel, which is an integral value of the point spread function (PSF) of the spectral wavelength. .

Further, the imaging optical system 101 is configured as a part of the imaging apparatus, but may be an interchangeable type as in a single-lens reflex camera. Functions such as aperture diameter control and manual focus may not be used depending on the purpose of the imaging apparatus.

In addition, since the optical transfer function (OTF) changes in accordance with the image height (image position) of the imaging system even in one shooting state, the image restoration processing of the present invention can be performed by changing it according to the image height. desirable.

The image processing unit 104 includes at least a calculation unit and a temporary storage unit (buffer). The image is temporarily written (stored) and read out from the storage unit as necessary for each step of the image processing. Further, the storage unit for temporarily storing is not limited to the temporary storage unit (buffer), but may be the storage unit 108, which is suitable for the data capacity and communication speed of the storage unit having a storage function. It can be appropriately selected and used. In addition, the storage unit 108 stores data such as an image restoration filter and correction information.

The selection and correction of the image restoration filter will be described with reference to FIG. FIG. 6 schematically illustrates imaging state information and a plurality of image restoration filters (black circles) stored in the storage unit 108 based on the imaging state information. The image restoration filter stored in the storage unit 108 is in an imaging state space centered on three imaging states of a focus position (state A), an aperture value (state B), and a subject distance (focusing distance) (state C). Are arranged discretely. The coordinates of each point (black circle) in the imaging state space indicate the image restoration filter stored in the storage unit 108. In FIG. 6, the image restoration filter is arranged at a grid point on a line orthogonal to each imaging state, but the image restoration filter may be arranged away from the grid point. The types of imaging states are not limited to the focal length, the aperture value, and the subject distance, and the number thereof may not be three, and a four-dimensional or more imaging state space with four or more imaging states is configured, The image restoration filter may be discretely arranged therein.

6, it is assumed that an imaging state indicated by a large white circle is an actual imaging state detected by the state detection unit 107. When there is an image restoration filter stored in advance at or near the position corresponding to the actual imaging state position, the image restoration filter can be selected and used for image restoration processing. One method for selecting an image restoration filter in the vicinity of a position corresponding to the actual imaging state is a distance (in the imaging state space between the actual imaging state and a plurality of imaging states in which the image restoration filter is stored ( The difference between the imaging states is calculated. This is a method of selecting the image restoration filter at the shortest distance. By this method, the image restoration filter at the position indicated by a small white circle in FIG. 6 is selected.

Also, as another method, there is a method of weighting the image restoration filter selection by the direction in the imaging state space. In other words, this is a method of selecting an image restoration filter having the highest value of the evaluation function using the product of the distance in the imaging state space and the weighted direction as the evaluation function.

Next, a method for generating a new image restoration filter by correcting the selected image restoration filter will be described. In correcting the image restoration filter, first, the distance (state difference amount) in the imaging state space between the actual imaging state and the imaging state in which the image restoration filter is stored is calculated, and the shortest distance (state difference amount) Select the image restoration filter at the position with the smallest). By selecting the image restoration filter having the smallest state difference amount, the correction amount of the image restoration filter can be reduced, and an image restoration filter close to the original image restoration filter in the imaging state can be generated.

In FIG. 6, the image restoration filter at the position indicated by a small white circle is selected. State difference amounts ΔA, ΔB, and ΔC between the imaging state corresponding to the selected image restoration filter and the actual imaging state are calculated. A state correction coefficient is calculated based on the state difference amount, and the selected image restoration filter is corrected using the state correction coefficient. Thereby, an image restoration filter corresponding to an actual imaging state can be generated.

Further, as another method, by selecting a plurality of image restoration filters located in the vicinity of the actual imaging state and performing interpolation processing according to the amount of state difference between the plurality of image restoration filters and the actual imaging state, An image restoration filter suitable for the imaging state can be generated. In this interpolation process, the coefficient values of the corresponding taps between the two-dimensional image restoration filters may be interpolated using linear interpolation, polynomial interpolation, spline interpolation, or the like.

Further, the optical transfer function (OTF) used for generating the image restoration filter can be obtained by calculation using an optical design tool or an optical analysis tool. Furthermore, the optical transfer function (OTF) in the actual state of the imaging optical system alone or the imaging apparatus can be measured and obtained.

FIG. 7 shows a specific flowchart of the image restoration processing of this embodiment executed by the image processing unit 104. The mark ● in the figure represents the step of storing pixel data such as an image at least temporarily.

The image processing unit 104 acquires an input image in an image acquisition process. Next, imaging state information is obtained from the state detection unit 107 (step S72). Then, an image restoration filter corresponding to the imaging state is selected from the storage unit 108 (step S73), and the restoration process is performed on the input image using the image restoration filter in the image restoration processing step (correction step) (step S74). ).

Next, other processing necessary for image formation is performed and the recovered image is output (step S76). Other processing here includes color interpolation processing (demosaicing processing), shading correction (peripheral light amount correction), distortion aberration correction, and the like if the correction image is a mosaic image. Further, various image processes including the other processes described here can be inserted before, after, or in the middle of the above flow as necessary.

Here, a more preferable example of the flow of image restoration processing will be described with reference to FIG. FIG. 8 shows changes in the MTF before and after the recovery process. The broken line (a) and the solid line (b) are the MTFs in the first azimuth direction and the second azimuth direction before the image restoration process, respectively, and the broken line (c) and the solid line (d) are the first azimuth direction and the solid line (d) after the restoration process. It is MTF of 1 azimuth direction and 2nd azimuth direction. As shown in FIG. 8, the image restoration processing is performed with a low degree of restoration for the two MTFs (a) and (b) in the azimuth direction before the restoration. As a result, as in (c) and (d), the MTF is low and the difference in the azimuth direction is corrected. In this state, the phase component of the aberration and the asymmetry of the aberration are corrected, but the sharpness is low. In other words, if the recovery degree of the recovered image when the image recovery filter is an inverse filter is the maximum recovery degree, the recovered image has a recovery degree lower than the maximum recovery degree. Preferably, in the Nyquist frequency, the average frequency of the MTF after recovery in the two azimuth directions is preferably 1.5 times or less of the maximum MTF before recovery. More preferably, it is preferably 1.2 times or less. More preferably, of the two azimuth directions, the first azimuth direction having a higher MTF recovers only the phase and does not substantially change the MTF. The second azimuth direction having a lower MTF recovers the phase, and the MTF in the second azimuth direction is preferably aligned with the MTF in the first azimuth direction. Edge enhancement processing is performed on the recovered image that has undergone such recovery processing.

Thereby, since the sharpness of only the edge portion can be improved, it is possible to suppress noise amplification rather than performing the recovery process on the entire image.

Edge enhancement processing will be described with reference to FIG. An example of the edge enhancement filter is shown in FIG. As shown in FIG. 9, a filter for performing edge enhancement can be generated by a difference between a filter that directly outputs an input image and a differential filter. As the differential filter, a Sobel filter that performs primary differentiation, a Laplacian filter that performs secondary differentiation, and the like are well known. The differential filter in FIG. 9 is a Laplacian filter. Since the edge enhancement filter performs processing based on the relationship between the pixel values of adjacent pixels, a filter having about 3 × 3 taps as shown in the figure is often used.

FIG. 10 shows the edge enhancement effect when the edge enhancement filter shown in FIG. 9 is used. 10A, 10B, and 10C are diagrams when the luminance of the edge portion in the image is viewed in a certain cross section. The horizontal axis represents coordinates, and the vertical axis represents amplitude. (A) of FIG. 10 is a brightness | luminance cross section of the edge part in an image, (B) what extracted the edge part with the differential filter and reversed the code | symbol to this. By adding (B) to the original image (A), the edge inclination can be sharply enhanced as shown in (C). Edge enhancement works only on sharp edges of edges, and sharpens, so the entire image is less affected by noise amplification, and the number of filter taps is relatively small, allowing high-speed processing. There is an advantage. Therefore, it is more preferable to perform the edge enhancement process after performing the image restoration process with a low degree of restoration. When combined with edge enhancement processing in this way, the edge enhancement processing may be included in the other necessary processing in FIG. Other processing that can enhance the edge portion of the image includes sharpness processing.

As described above, the preferred context of each processing step and the processing to be considered have been described. However, when there is a restriction from another viewpoint on the order of the processing steps, the present invention is not limited to this. It may be changed according to the required image quality. Moreover, although the example regarding an imaging device was shown, various deformation | transformation and a change are possible within the range of the summary.

FIG. 11A shows a configuration diagram of an image processing system that is Embodiment 2 of the present invention. The image processing apparatus 111 includes an information processing apparatus, and is loaded with image processing software (image processing program) 112 for causing the information processing apparatus to execute the image processing method described in the first embodiment.

The imaging device 113 includes a camera, a microscope, an endoscope, a scanner, and the like. The storage medium 114 stores an image (captured image data) generated by imaging such as a semiconductor memory, a hard disk, or a server on a network.

The image processing device 111 acquires image data from the imaging device 113 or the storage medium 114 and outputs output image (corrected image) data obtained by performing predetermined image processing to at least one of the output device 116, the imaging device 113, and the storage medium 114. Output to one. Further, the output destination can be a storage unit built in the image processing apparatus 111, and the output image data can be stored in the storage unit. An example of the output device 116 is a printer. A display device 115 as a monitor is connected to the image processing apparatus 111, and the user can perform an image processing operation through the display device 115 and can evaluate a recovery adjustment image (output image). The image processing software 112 has a development function and other image processing functions as needed in addition to the image recovery processing function and the recovery degree adjustment function.

FIG. 11B shows the configuration of another image processing system. As in the first embodiment, when the image processing of the first embodiment is performed by the imaging device 118 alone, the recovery adjustment image can be output directly from the imaging device 118 to the output device 119.

In addition, by installing the image processing apparatus that executes the image processing method of the first embodiment in the output device 119, the output device 119 sets an adjustment coefficient according to the feature amount of the image, and adjusts the degree of recovery. Is also possible. Furthermore, by adjusting the degree of recovery according to the degradation characteristics of the output image of the output device 119, a higher quality image can be provided.

Here, the contents of correction information for performing image processing including image restoration processing and adjustment of the degree of restoration will be described. FIG. 12 shows an example of the correction information, and the plurality of correction information is referred to as a correction information set. Each correction information will be described below.

"Correction control information"
The correction control information includes setting information indicating which of the imaging device 113, the image processing device 111, and the output device 116 performs the recovery process and the recovery degree adjustment process, and data to be transmitted to other devices according to the setting information Is selection information for selecting. For example, when only the restoration processing is performed by the imaging device 113 and the restoration degree is adjusted by the image processing device 111, it is not necessary to transmit the image restoration filter to the image processing device 111, but at least the photographed image and the restoration image or the restoration component information. (Difference information) needs to be transmitted.

"Imaging device information"
The imaging device information is identification information of the imaging device 113 corresponding to the product name. If the lens and the camera body are interchangeable, the identification information includes the combination.

"Imaging status information"
The imaging state information is information relating to the state of the imaging device 113 at the time of shooting. For example, the focal length (zoom position), aperture value, subject distance (focusing distance), ISO sensitivity, white balance setting, etc.

"Imaging device individual information"
The imaging device individual information is identification information of each imaging device with respect to the above imaging device information. Since the optical transfer function (OTF) of the imaging apparatus has individual variations due to variations in manufacturing errors, the individual imaging apparatus information is effective information for setting an optimum recovery degree adjustment parameter individually. The restoration degree adjustment parameter is a restoration strength adjustment coefficient μ and a color composition ratio adjustment coefficient ω.

"Image recovery filters"
The image restoration filter group is a set of image restoration filters used in image restoration processing. When a device that performs image restoration processing does not have an image restoration filter, it is necessary to transmit the image restoration filter from another device (apparatus).

"User Setting Information"
The user setting information is an adjustment parameter for adjusting the recovery degree according to the user's preference or a correction function of the adjustment parameter. The user can variably set the adjustment parameter, but if user setting information is used, a desired output image can always be obtained as an initial value. Moreover, it is preferable that the user setting information is updated by the learning function with the sharpness most preferred from the history of the user determining the adjustment parameter.

Furthermore, the imaging device provider (manufacturer) can also provide preset values according to some sharpness patterns via a network.

The above correction information set is preferably attached to individual image data. By attaching necessary correction information to the image data, an image restoration process can be performed by any device equipped with the image processing apparatus of the second embodiment. The contents of the correction information set can be selected automatically and manually as necessary.

The present invention is not limited to the above embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Therefore, in order to make the scope of the present invention public, the following claims are attached.

DESCRIPTION OF SYMBOLS 101 Imaging optical system 102 Imaging element 104 Image processing part 106 Imaging system control part 108 Storage part 110 System controller

Claims (6)

Image acquisition means for acquiring an input image;
Image recovery means for recovering the input image using an image recovery filter generated or selected based on a transfer function of an imaging system used to form a subject image as the input image and generating a recovered image; And
The image restoration filter has a difference in absolute value of a transfer function between two azimuth directions in obtaining a restored image from a subject smaller than a difference in absolute value of a transfer function between the two azimuth directions of the imaging system. An image processing apparatus. The image processing apparatus according to claim 1, wherein the image restoration means performs image restoration by convolving and integrating the image restoration filter with respect to pixels of the acquired image. The image restoration filter is generated based on a transfer function having different frequency characteristics in two azimuth directions and a corrected transfer function corrected so that a difference between the absolute values of the transfer functions between the two azimuth directions is reduced. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. An image sensor that photoelectrically converts a subject image formed by an imaging system to generate a captured image; and
An imaging apparatus comprising: the image processing apparatus according to claim 1, wherein the captured image is processed as the input image. Acquiring an image generated by the imaging system as an input image;
Recovering the input image using an image recovery filter generated or selected based on a transfer function of the imaging system, and generating a recovered image,
The image restoration filter is configured to make a difference between absolute values of transfer functions between two azimuth directions when obtaining a restored image from a subject smaller than a difference between transfer functions between the two azimuth directions of the imaging system. A featured image processing method. An image acquisition step of acquiring an image generated by the imaging system as an input image;
An image processing program for causing an information processing apparatus to execute an image recovery step of recovering the input image using an image recovery filter generated or selected based on a transfer function of the imaging system and generating a recovered image,
The image restoration filter has a difference in absolute value of a transfer function between two azimuth directions in obtaining a restored image from a subject smaller than a difference in absolute value of a transfer function between the two azimuth directions of the imaging system. An image processing program.

Images