WO2012150660A1 - Image capture device - Google Patents

Abstract

By executing a prescribed feature conversion process on pixel values (d) of pixels which are read with a plurality of pixels, an inflection point scattering correction unit (621) unifies photoelectric conversion features of each pixel with linear features common to all pixels. An interpolation assessment unit (622) receives sequential input of the pixel values (d) of the pixels from a black scattering correction unit (61), assesses whether each pixel is an inflection point defective pixel, and notifies a pixel interpolation unit (63) of the result of the assessment. The pixel interpolation unit (63) treats the inflection point defective pixels and the defective pixels as pixels to be interpolated, and interpolates the pixels to be interpolated, employing, in the interpolation, peripheral pixels of the pixels to be interpolated which are regular pixels other than the inflection point defective pixels and the defective pixels.

Description

Imaging device

The present invention relates to an image pickup apparatus including an image pickup element including a plurality of pixels having different photoelectric conversion characteristics at an inflection point.

In recent years, in order to expand the dynamic range, an image sensor composed of a plurality of pixels having different photoelectric conversion characteristics at an inflection point is known. As such an image pickup device, for example, the lower luminance side than the inflection point is linear photoelectric conversion characteristics (hereinafter referred to as “linear characteristics”), and the higher luminance side is logarithmic photoelectric conversion characteristics (hereinafter referred to as log characteristics). ). A photoelectric conversion characteristic composed of a linear characteristic and a log characteristic is called a linear log characteristic.

A pixel defect exists in the image sensor, and a pixel having this pixel defect is called a defective pixel. Further, in an image sensor having a linear log characteristic, if the inflection point is higher than a predetermined value, the pixel value may be saturated on the high luminance side. Further, if the inflection point is lower than a predetermined value, the pixel value may become negative on the low luminance side. A pixel having such an abnormal pixel value is referred to herein as an inflection point defective pixel.

For example, Patent Document 1 stores in advance first correction data that stores in advance address information of a single defective pixel in which no defective pixel exists in the peripheral pixels, and address information of defective pixels in which defective pixels exist in the periphery. And correcting the single defective pixel using the first correction data, and then correcting the defective pixel using the second correction data.

In Patent Document 2, in an imaging apparatus having pixels with linear log characteristics, when a pixel whose inflection point pixel value is higher than a predetermined inflection point threshold value outputs a pixel value higher than the predetermined pixel threshold value, Is an inflection point defective pixel, and the inflection point defective pixel is interpolated using peripheral pixels other than the inflection point defect pixel.

In Patent Document 3, in an imaging device having pixels with linear log characteristics, when a pixel whose inflection point has a pixel value lower than a predetermined inflection point threshold outputs a pixel value lower than the predetermined pixel threshold, Is a technique for interpolating an inflection point defective pixel using peripheral pixels other than the inflection point defect pixel.

However, in the method of Patent Document 1, the pixels stored in the first correction data and the second correction data are uniformly specified as pixels to be interpolated. It is not specified whether or not. Therefore, there is a problem that the number of pixels to be interpolated increases.

Further, as shown in FIG. 2 of Patent Document 2 and FIG. 2 of Patent Document 3, a defect interpolation unit that interpolates defective pixels using peripheral pixels, and an inflection point interpolation unit that interpolates inflection point defective pixels. It is arranged on the upstream side. Therefore, there is a problem that the inflection point defective pixel is interpolated using the pixel value of the defective pixel after interpolation.

JP 2005-333620 A Japanese Patent No. 4586942 Japanese Patent No. 4586941

An object of the present invention is to provide an imaging device that interpolates an interpolation target pixel in consideration of whether or not the pixel adjacent to the interpolation target pixel is a defective pixel as well as whether or not the pixel is an inflection point defective pixel. is there.

An imaging device according to one embodiment of the present invention includes an imaging element including a plurality of pixels having different photoelectric conversion characteristics with an inflection point as a boundary, and defective pixel information indicating a position of a defective pixel among pixels constituting the imaging element. A defective pixel information storage unit that stores in advance, an inflection point information storage unit that stores inflection point information indicating an inflection point pixel value that is a pixel value of an inflection point of each pixel, and a subject for each pixel. An interpolation determination unit that determines whether the pixel is an inflection point defective pixel based on the pixel value at the time of imaging and the inflection point pixel value, and the inflection point defective pixel and the defective pixel are set as interpolation target pixels. And a pixel interpolation unit that interpolates the interpolation target pixel using normal pixels other than the inflection point defective pixel and normal pixels other than the defective pixel.

1 is a block diagram of an imaging apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a detailed configuration of an image processing unit illustrated in FIG. 1. It is the graph which showed the photoelectric conversion characteristic of a certain pixel which constitutes an image sensor. It is the figure which showed an example of the lookup table. In the graph showing the photoelectric conversion characteristics of pixels having different inflection points, (A) shows the case where the inflection point pixel value S at the inflection point P1 is equal to or larger than a predetermined inflection point threshold Sth1, and (B) shows the change. The case where the inflection point pixel value S of the inflection point P1 is equal to or less than a predetermined inflection point threshold value Sth2 is shown. It is explanatory drawing of the process in which a pixel interpolation part produces | generates interpolation pixel arrangement | positioning information. It is explanatory drawing of the process in which a pixel interpolation part produces | generates interpolation pixel arrangement | positioning information, when an imaging sensor is a color. It is the graph which showed the problem of the interpolation process in case only an inflection point defective pixel is made into an interpolation object pixel, without considering a defective pixel. It is a screen figure which shows the effect of the interpolation process by the imaging device by embodiment of this invention.

FIG. 1 is a block diagram of an imaging apparatus according to an embodiment of the present invention. As shown in FIG. 1, the imaging apparatus 1 includes, for example, a digital camera, and includes a lens unit 2, an imaging sensor 3 (an example of an imaging element), an amplifier 4, an A / D conversion unit 5, an image processing unit 6, and an image memory 7. , A control unit 8, a monitor unit 9, and an operation unit 10.

The lens unit 2 includes an optical lens system that captures an optical image of a subject and guides it to the image sensor 3. As the optical lens system, for example, a zoom lens, a focus lens, other fixed lens blocks, and the like arranged in series along the optical axis L of the optical image of the subject can be employed. The lens unit 2 includes a diaphragm (not shown) for adjusting the amount of transmitted light, a shutter (not shown), and the like, and the driving of the diaphragm and the shutter is controlled under the control of the control unit 8.

The imaging sensor 3 includes a plurality of pixels arranged in a matrix of predetermined rows × predetermined columns and having different photoelectric conversion characteristics at an inflection point, and photoelectrically converts a light image formed in the lens unit 2. Image data composed of pixel values of R (red), G (green), and B (blue) color components having a level corresponding to the amount of light is generated and output to the amplifier 4.

Here, as the image sensor 3, an image sensor such as a CMOS image sensor, a VMIS (Threshold Voltage Modulation Image Sensor) image sensor, or a CCD image sensor may be employed.

Note that this is an example, and an imaging sensor 3 that captures a monochrome image may be used. Then, the imaging sensor 3 sequentially outputs pixel values read by each pixel so as to perform raster scanning from the upper left pixel toward the lower right pixel, for example.

Further, in this embodiment, as shown in FIG. 3, each pixel has a low luminance side photoelectric conversion characteristic as a linear characteristic and a high luminance side photoelectric conversion characteristic as a log characteristic. FIG. 3 is a graph showing the photoelectric conversion characteristics of a certain pixel constituting the image sensor 3. In addition, the vertical axis | shaft of FIG. 3 has shown the pixel value, and the horizontal axis has shown the luminance logarithmically.

In the graph G1, the low luminance side linear range 51 shows linear characteristics, and the high luminance side log range 52 shows log characteristics. The boundary between the linear range 51 and the log range 52 is the inflection point P1. In the linear range 51, the graph G1 draws a curve, and in the log range 52, the graph G1 draws a straight line because the horizontal axis is a logarithmic axis.

Thus, by setting the high luminance side to the log characteristic, the slope of the graph G1 in the log range 52 becomes smaller than the slope of the graph G1 in the linear range 51. On the other hand, when the pixel has only a linear characteristic, the photoelectric conversion characteristic changes as in the graph L1. Therefore, the luminance with respect to the saturated pixel value is higher in the graph G1 than in the graph L1. As a result, the dynamic range DL can be widened by adopting the linear log characteristics. Hereinafter, the pixel value of the inflection point P1 is described as the inflection point pixel value S.

Returning to FIG. 1, the amplifier 4 includes, for example, an AGC (auto gain control) circuit, a CDS (correlated double sampling) circuit, and the like, removes noise components from the image data output from the image sensor 3, and removes noise components. The removed image data is amplified. The A / D converter 5 converts the R, G, and B color image data amplified by the amplifier 4 into R, G, and B digital image data. In the present embodiment, the pixel value of the pixel received by each pixel of the image sensor 3 is converted into a pixel value having a 12-bit gradation value, for example.

The image processing unit 6 performs image processing as will be described later. The image memory 7 is composed of, for example, a RAM (Random Access Memory), and stores image data subjected to image processing by the image processing unit 6.

The control unit 8 includes a ROM that stores various control programs, a RAM that temporarily stores data, a central processing unit (CPU) that reads and executes control programs from the ROM, and the like. To manage.

The monitor unit 9 employs, for example, a color liquid crystal display disposed on the back surface of the housing of the imaging apparatus, and displays on the monitor an image taken by the imaging sensor 3 or an image stored in the image memory 7.

The operation unit 10 includes various operation switch groups such as a power switch, a release switch, a mode setting switch for setting various shooting modes, and a menu selection switch. When the release switch is pressed, a light image of the subject is picked up by the image pickup sensor 3, image data of the subject is acquired, predetermined image processing is performed on the acquired image data, and the image data is recorded in the image memory 7 or the like. Is done. The image data may be output as a digital signal from the image processing unit 6 without being recorded in the image memory 7, or may be D / A converted and output as an analog signal such as NTSC.

FIG. 2 is a block diagram showing a detailed configuration of the image processing unit 6 shown in FIG. The image processing unit 6 includes a black variation correction unit 61, a characteristic conversion unit 62, a pixel interpolation unit 63, a gradation conversion unit 64, a Bayer interpolation unit 65, and a tone curve correction unit 66. The black variation correction unit 61 corrects the variation in the black level of each pixel constituting the image sensor 3.

The characteristic conversion unit 62 includes an inflection point variation correction unit 621, an interpolation determination unit 622, and a memory unit 623, and converts the photoelectric conversion characteristic of each pixel into a predetermined reference photoelectric conversion characteristic. The black variation correcting unit 61 outputs one piece of image data in which R, G, and B color components are arranged in, for example, a Bayer array.

In this embodiment, linear characteristics are adopted as the reference photoelectric conversion characteristics, and the log characteristics are converted into linear characteristics.

The inflection point variation correcting unit 621 performs a predetermined characteristic conversion process on the pixel value d of the pixel read by each pixel, thereby unifying the photoelectric conversion characteristics of each pixel to a linear characteristic common to all pixels.

Here, the inflection point variation correcting unit 621 compares the pixel value d output from each pixel with the inflection point pixel value S of each pixel, so that the pixel value d belongs to the linear range 51. Or the pixel value of the log characteristic belonging to the log range 52 is determined. When the inflection point variation correcting unit 621 determines that the pixel value d has linear characteristics, the inflection point variation correcting unit 621 outputs the pixel value d as the pixel value d ′ to the pixel interpolating unit 63 as it is.

On the other hand, when the inflection point variation correcting unit 621 determines that the pixel value d has log characteristics, the inflection point variation correcting unit 621 performs a characteristic conversion process on the pixel value d and converts the pixel value d into a linear characteristic pixel value d ′. The pixel value d ′ is output to the pixel interpolation unit 63.

Since the inflection point pixel value S of each pixel is stored in advance in the memory unit 623, the inflection point variation correction unit 621 identifies the inflection point pixel value S of each pixel by referring to the memory unit 623. do it.

Specifically, when the pixel value d output from the black variation correcting unit 61 is d <S, the inflection point variation correcting unit 621 determines that the pixel value d is a linear characteristic, and d ≧ S. In this case, the pixel value d is determined to be log characteristics.

Here, the inflection point variation correcting unit 621 uses a look-up table in which a correspondence relationship when the pixel value d of the log characteristic is converted into the pixel value d ′ of the linear characteristic is used to determine the pixel value of the log characteristic. What is necessary is just to convert d into the pixel value d 'of a linear characteristic. FIG. 4 is a diagram showing an example of the lookup table T.

Here, the lookup table T is a lookup table common to all pixels. As shown in FIG. 4, the look-up table T is composed of n (n is an integer) cells each having addresses “0” to “n−1”. Each address corresponds to a relative value of the pixel value d with respect to the inflection point pixel value S, that is, dS. Each cell stores pixel values d1 to dn after conversion when a value obtained by subtracting the inflection point pixel value S from the pixel value d of log characteristics is converted into linear characteristics.

That is, the lookup table T stores values obtained by converting each pixel value d of the log characteristics shown in FIG. 3 so as to be on the graph L1.

Therefore, when the input pixel value d has a log characteristic, the inflection point variation correcting unit 621 is stored at the address d−S, which is a value obtained by subtracting the inflection point pixel value S from the pixel value d. The log characteristic is converted to a linear characteristic by outputting a value obtained by adding the inflection point pixel value S of the pixel to the value of the lookup table T as the converted pixel value d ′.

Here, the reason why each address of the lookup table T is made to correspond to dS is because the variation of the inflection point pixel value S of each pixel is taken into consideration. That is, in the present embodiment, it is assumed that the log characteristics of each pixel have substantially the same waveform although the inflection point pixel value S varies. Therefore, by making each address of the lookup table T correspond to dS instead of d, this lookup table T can be applied to all pixels.

The inflection point variation correcting unit 621 specifies the position of the pixel value d input from this order because the pixel value d of each pixel constituting one piece of image data is input in a predetermined order. Thus, a predetermined inflection point pixel value S can be specified for each pixel.

Returning to FIG. 2, the interpolation determination unit 622 sequentially receives the pixel value d of the target pixel from the black variation correction unit 61, determines whether the target pixel is an inflection point defective pixel, and performs pixel interpolation on the determination result. Notification to the unit 63. That is, the interpolation determination unit 622 determines whether or not the target pixel is an inflection point defective pixel using the pixel value d of the target pixel that has not been subjected to the characteristic conversion processing by the inflection point variation correction unit 621.

First, when the pixel value d of the target pixel is input, the interpolation determination unit 622 refers to the inflection point information stored in the memory unit 623 and specifies the inflection point pixel value S of the target pixel. Then, the interpolation determination unit 622 determines whether or not the inflection point pixel value S of the target pixel is equal to or greater than a predetermined inflection point threshold value Sth1 (an example of a first inflection point threshold value).

When the interpolation determination unit 622 determines that the inflection point pixel value S of the target pixel is equal to or greater than the inflection point threshold value Sth1 (S ≧ Sth1), the pixel value d of the target pixel is further set to a predetermined pixel threshold value Dth1 (first threshold value). It is determined whether or not it is an example of one pixel threshold.

Then, the interpolation determining unit 622 determines that the pixel of interest is an inflection point defective pixel when the pixel value d of the pixel of interest is determined to be greater than or equal to the pixel threshold Dth1 (d ≧ Dth1). That is, the interpolation determination unit 622 determines that the pixel of interest is an inflection point defective pixel when the target pixel satisfies the condition 1 of S ≧ Sth1 and d ≧ Dth1.

Further, the interpolation determination unit 622 determines whether or not the inflection point pixel value S of the target pixel is equal to or less than a predetermined inflection point threshold value Sth2 (an example of a second inflection point threshold value). When the interpolation determination unit 622 determines that the inflection point pixel value S of the target pixel is equal to or less than the inflection point threshold value Sth2 (S ≦ Sth2), the pixel value d of the target pixel is further set to a predetermined pixel threshold value Dth2 (first threshold value). An example of a two-pixel threshold value) It is determined whether or not it is equal to or less.

Then, when the pixel value d of the target pixel is determined to be equal to or less than the pixel threshold Dth2 (d ≦ Dth2), the interpolation determination unit 622 determines the target pixel as an inflection point defective pixel. That is, the interpolation determination unit 622 determines that the pixel of interest is an inflection point defective pixel when the target pixel satisfies the condition 2 of S ≦ Sth2 and d ≦ Dth2.

As described above, the interpolation determining unit 622 determines that the pixel of interest satisfies the condition 1 or the condition 2 as an inflection point defective pixel. Further, the interpolation determining unit 622 determines that the pixel of interest is not an inflection point defective pixel if the pixel of interest does not satisfy the conditions 1 and 2.

FIG. 5 is a graph showing photoelectric conversion characteristics of pixels having different inflection points P1, and FIG. 5A shows a case where the inflection point pixel value S at the inflection point P1 is equal to or greater than a predetermined inflection point threshold Sth1. FIG. 5B shows a case where the inflection point pixel value S of the inflection point P1 is equal to or less than a predetermined inflection point threshold Sth2. In FIG. 5A, the vertical axis represents the pixel value d, and the horizontal axis represents the luminance in logarithm.

When the pixel value d is greater than or equal to the pixel threshold value Dth1 in the pixel having the inflection point pixel value S or greater than the inflection point threshold value Sth1, this pixel value d can be regarded as saturated, and it is not preferable to employ it.

However, even if the inflection point pixel value S is a pixel equal to or greater than the inflection point threshold value Sth1, if the pixel value d is less than the pixel threshold value Dth1, the pixel value d is not saturated, and therefore it may be adopted. There is no.

In FIG. 5B, the vertical axis represents the pixel value d, and the horizontal axis represents the luminance in logarithm. In a pixel where the inflection point pixel value S is equal to or less than the inflection point threshold value Sth2, if the pixel value d is less than the pixel threshold value Dth2, the actual photoelectric conversion characteristic is assumed to be the characteristic indicated by the solid line in the figure. The photoelectric conversion characteristic becomes the characteristic indicated by the wavy line in the figure, and a correction error occurs. Therefore, since the pixel value d deviates from the assumed photoelectric conversion characteristic, it is not preferable to use the pixel value d.

However, even if the inflection point pixel value S is a pixel equal to or smaller than the inflection point threshold value Sth2, if the pixel value d is larger than the pixel threshold value Dth2, the pixel value d does not deviate from the assumed photoelectric conversion characteristics. There is no problem even if the value d is adopted.

Therefore, the interpolation determination unit 622 determines the inflection point defective pixel by performing the above processing. The pixel threshold value Dth1 may be a value near the saturation level and a value at which deterioration of image reproducibility starts.

Further, as the inflection point threshold value Sth1, an average value of the inflection point pixel values S of all the pixels constituting the image sensor 3 may be adopted, or the upper number of distribution of the inflection point pixel values S of all the pixels. A value such that% (for example, 10%) pixels exceed the inflection point threshold value Sth1 may be adopted.

As the inflection point threshold value Sth2, a value at least higher than the inflection point pixel value S of the pixel having the inflection point pixel value S that is the maximum among the pixels having the minus value of the inflection point pixel value S is adopted. do it.

Further, as the pixel threshold value Dth2, a value about the start level of the shift of the pixel having the maximum start level of the photoelectric conversion characteristic shift among the pixels having the negative inflection point pixel value S may be adopted.

The memory unit 623 is configured by a nonvolatile recording medium such as an EEPROM, for example, and stores defective pixel information and inflection point information (an example of a defective pixel information storage unit and an inflection point information storage unit). The defective pixel information is information indicating the position of the defective pixel among the pixels constituting the imaging sensor 3. Here, a defective pixel is a pixel that has become defective in the semiconductor manufacturing process and is not sensitive to light. As information indicating the position, horizontal and vertical coordinates are employed.

The inflection point information is information indicating the inflection point pixel value S of each pixel constituting the imaging sensor 3. Specifically, the inflection point information is information in which information indicating the position and the inflection point pixel value S are associated with each pixel. Note that the inflection point information includes the inflection point pixel value S for the defective pixel.

The pixel circuit of CMOS that constitutes each pixel includes a photodiode that accumulates signal charges according to received light, a floating diffusion that converts signal charges accumulated in the photodiodes into voltage signals, and signal charges that are accumulated in the photodiodes. Is transferred to the floating diffusion.

The inflection point pixel value S is determined by the opening degree of the gate of the transfer transistor. Therefore, the inflection point pixel value S is dominated by the gate voltage of the transfer transistor, the characteristics of the transfer transistor, and the like. On the other hand, in the defective pixel, only the photodiode has no sensitivity, and other circuit elements may be normal. Therefore, even in the case of a defective pixel, an inflection point pixel value S may be obtained as in the case of a normal pixel if current is injected from a current source and the photoelectric conversion characteristics are measured. Therefore, even a defective pixel may have an inflection point pixel value S.

Also, the memory unit 623 stores inflection point threshold values Sth1, Sth2, pixel threshold values Dth1, Dth2, and a lookup table T. Here, the inflection point threshold values Sth1, Sth2 and the pixel threshold values Dth1, Dth2 are values common to all pixels.

The pixel interpolation unit 63 uses the inflection point defective pixel and the defective pixel as an interpolation target pixel, and interpolates the target interpolation target pixel using normal pixels other than the inflection point defective pixel and the defective pixel among the peripheral pixels of the interpolation target pixel. To do. The target interpolation target pixel indicates one interpolation target pixel among the interpolation target pixels. The pixel interpolation unit 63 sequentially determines each interpolation target pixel as a target interpolation target pixel in accordance with, for example, the order of raster scanning.

Specifically, the pixel interpolation unit 63 generates interpolation pixel arrangement information indicating the position of the interpolation target pixel, and extracts a plurality of normal pixels in order of increasing distance from the target interpolation target pixel with reference to the interpolation pixel arrangement information. Then, the target interpolation target pixel is interpolated using the extracted normal pixel. Then, the pixel interpolation unit 63 outputs the pixel value of the interpolated pixel as d ″ to the gradation conversion unit 64. Note that the pixel interpolation unit 63 outputs the pixel value d ′ output from the inflection point variation correction unit 621 to the gradation conversion unit 64 as the pixel value d ″ for pixels that have not been interpolated.

FIG. 6 is an explanatory diagram of processing in which the pixel interpolation unit 63 generates interpolation pixel arrangement information. FIG. 6 shows a case where the image sensor 3 is monochrome. The arrangement information MA (an example of the first arrangement information), the arrangement information MB (an example of the second arrangement information), and the interpolation pixel arrangement information MC are information about each pixel in one column in the horizontal direction of the imaging sensor 3, respectively. A buffer for storing, and one field stores information on one pixel.

In the arrangement information MA, a determination result by the interpolation determination unit 622 is stored. The pixel interpolation unit 63 labels 1 for pixels determined to be inflection point defective pixels, 0 for other pixels, and displays each field of the arrangement information MA according to the determination result output from the interpolation determination unit 622. The 1 or 0 label is stored sequentially. In the example of FIG. 6, 1 is stored in the field with the “X” mark, and 0 is stored in the other fields.

In the arrangement information MB, a determination result as to whether or not it is a defective pixel is stored. When the pixel interpolation unit 63 refers to defective pixel information each time a determination result is input from the interpolation determination unit 622, determines whether or not the pixel corresponding to the determination result is a defective pixel, and determines that it is a defective pixel Stores a label of 1 in the field of the arrangement information MB corresponding to the pixel, and stores a label of 0 in the field of the arrangement information MB corresponding to the pixel when it is not determined as a missing pixel.

The interpolated pixel arrangement information MC stores a logical sum (or) of the label stored in each field of the arrangement information MA and the label stored in each field of the arrangement information.

When the information of all pixels in one column in the horizontal direction is stored in the placement information MA and MB, the pixel interpolation unit 63 stores the labels stored in the fields of the placement information MA and the fields of the placement information MB. The logical sum with the obtained label is obtained, and the obtained result is stored in each field of the interpolation pixel arrangement information MC.

In the example of FIG. 6, for the pixels P (-2) to P (3), a label of 1 is stored in at least one of the arrangement information MA and MB fields. Therefore, in the interpolation pixel arrangement information MC, 1 label is stored in each field of the pixels P (−2) to P (3).

The pixel interpolation unit 63 extracts two normal pixels located closest to each other on one side and the other side in the horizontal direction with respect to the target interpolation target pixel with reference to the interpolation pixel arrangement information MC, and extracts the two The target interpolation target pixel is interpolated using normal pixels.

Here, the pixel interpolation unit 63 assigns a higher weight value to a normal pixel that is closer to the target interpolation target pixel, and interpolates the target interpolation target pixel by weighted addition of the pixel values of the extracted normal pixels.

This will be specifically described using the interpolation pixel arrangement information MC in FIG. Pixel P (0) is the target interpolation target pixel. The pixel P (−3) is a normal pixel that is closest to the pixel P (0) on one side (left side) in the horizontal direction. The pixel P (4) is a normal pixel that is closest to the pixel P (0) on the other side (right side) in the horizontal direction.

The distance l1 between the pixel P (-3) and the pixel P (0) is 3, and the distance l2 between the pixel P (4) and the pixel P (0) is 4. Therefore, the pixel interpolation unit 63 sets a weight value of 4/7 for the pixel P (−3) and sets a weight value of 3/7 for the pixel P (0). That is, the pixel interpolation unit 63 sets the weight value of the pixel P (−3) to the distance l2 / (distance l1 + distance l2), and sets the weight value of the pixel P (4) to the distance l1 / (distance l1 + distance l2). ). Thereby, a higher weight value is set for a normal pixel that is closer to the target interpolation target pixel.

The pixel value of the pixel P (−3) is d ′ (− 3), the pixel value of the pixel P (4) is d ′ (4), and the pixel value of the pixel P (0) is d ″ (0). Then, the pixel value d (0) is obtained by the following equation, and the pixel P (0) is interpolated.
d ″ (0) = (4/7) · d ′ (− 3) + (3/4) · d ′ (4)

In the example of FIG. 6, the normal pixels located at the shortest distances on the left side and the right side are used. However, the present invention is not limited to this, and the target interpolation target pixel is determined using a plurality of normal pixels on each of the left side and the right side. Interpolation may be performed. Alternatively, interpolation may be performed using only one normal pixel, for example, the normal pixel P (−3) closest to the target interpolation target pixel P (0).

For example, in FIG. 6, the pixel P (-4) adjacent to the left side of the pixel P (-3) is also a normal pixel. Therefore, the pixel interpolation unit 63 may further add the pixel P (−4) to interpolate the target interpolation target pixel. If the pixel P (5) is also a normal pixel, the pixel P (5) may be further added to interpolate the target interpolation target pixel.

FIG. 6 shows the case where the image sensor 3 is monochrome, but in the case of color, the following mode may be adopted. FIG. 7 is an explanatory diagram of a process in which the pixel interpolation unit 63 generates the interpolation pixel arrangement information MC when the image sensor 3 is color. In this case, the pixel interpolation unit 63 may interpolate the target interpolation target pixel using the normal pixel of the same color pixel located at the shortest distance on each of the left side and the right side with respect to the target interpolation target pixel.

Specifically, the pixel interpolation unit 63 generates the array information MA and MB in the same manner as in the monochrome case. Further, the pixel interpolation unit 63 prepares interpolation pixel arrangement information MC for each of R, G, and B colors. If the target interpolation target pixel is R, a logical sum of labels of the R pixel field in the arrangement information MA and the R pixel field in the arrangement information MB is obtained in the R interpolation pixel arrangement information MC. , R interpolation pixel arrangement information MC is generated.

Then, the pixel interpolation unit 63 uses the normal pixel that is the same color as the target interpolation target pixel and is located at the shortest distance in the left and right sides in the R interpolation pixel arrangement information MC, and performs the target interpolation. The target pixel is corrected.

In the example of FIG. 7, the pixels P (−3), P (−1), P (1), and P (3) are labeled with 0 in the interpolation pixel arrangement information MC, but are different from the target interpolation target pixel. Therefore, it is not used for interpolation.

The pixel P (−4) is a normal pixel of the same color located at the shortest distance on the left side with respect to the target interpolation target pixel, and the pixel P (4) is the same color positioned at the shortest distance on the right side with respect to the target interpolation target pixel. Normal pixel. Therefore, the pixel interpolation unit 63 sets weight values for the pixel P (−4) and the pixel P (4) and interpolates the target interpolation target pixel, as in the case of monochrome.

In addition, the pixel interpolation unit 63 has two normal pixels located closest to each other on the left side and the right side with respect to the target interpolation target pixel, and one side (upper side) and the other side (lower side) in the vertical direction. May be extracted with reference to the interpolation pixel arrangement information, and the target interpolation target pixel may be interpolated using the extracted four normal pixels.

In this case, the pixel interpolation unit 63 may adopt a unit that stores information about all the pixels of the image sensor 3 as the arrangement information MA and MB and the interpolation pixel arrangement information MC. Thereby, normal pixels in the vertical direction can be searched.

In this case, the pixel interpolation unit 63 may obtain the weight value of each normal pixel using the above-described method so that the normal value closer to the target interpolation target pixel has a higher weight value.

In this case, the pixel interpolation unit 63 may interpolate the target interpolation target pixel using a plurality of normal pixels instead of one normal pixel in each of the upper, lower, left, and right directions.

The pixel interpolation unit 63 also includes a distance lH between the normal pixel PL that is the shortest distance on the left side of the target interpolation target pixel and the normal pixel PR that is the shortest distance on the right side of the target interpolation target pixel, and the target interpolation. Comparing the distance LV between the normal pixel PU that is the shortest distance above the target pixel and the normal pixel PD that is the shortest distance below the target interpolation target pixel, The target interpolation target pixel may be interpolated using the pixel.

For example, if the distance 1V is shorter than the distance 1H, the pixel interpolation unit 63 interpolates the target interpolation target pixel using two normal pixels PU and PD in the vertical direction, and the distance 1H is longer than the distance 1V. If it is short, the target interpolation target pixel may be interpolated using the two normal pixels PL and PR in the horizontal direction.

Further, the pixel interpolation unit 63 may interpolate the target interpolation target pixel by adding and averaging the pixel values of the normal pixels without setting a weight value for the normal pixels. For example, in the example of FIG.
d ″ (0) = (d ′ (− 3) + d ′ (4)) / 2

2, the gradation conversion unit 64 performs a histogram equalization process, a dynamic range compression process, and the like on the pixel value d ″ output from the pixel interpolation unit 63. The histogram equalization process is a process for uniformly distributing the gray level distribution in the image. Thereby, the contrast of an image can be clarified. As the dynamic range compression processing, for example, the technique described in Masami Ogata “Image Dynamic Range Compression Technology”, Nihon Kogyo Shuppan, Image Lab, 2004, June issue can be employed.

The Bayer interpolation unit 65 executes Bayer interpolation, which is an interpolation process for interpolating missing pixels in each color component that occurs because each pixel constituting the imaging sensor 3 has a Bayer array. As a result, image data representing one image is represented by three image data corresponding to the three color components R, G, and B. If the image sensor 3 is a monochrome image sensor, the Bayer interpolation unit 65 may be omitted.

The tone curve correction unit 66 performs gamma correction on the image value output from the Bayer interpolation unit 65. The pixel value output from the tone curve correction unit 66 is stored in the image memory 7 shown in FIG.

Next, an outline of the operation of the imaging apparatus according to the present embodiment will be described by taking a monochrome case as an example. First, when the operation unit 10 receives an imaging command from the user, the image data captured by the imaging sensor 3 is amplified to a predetermined level by the amplifier 4 and A / D conversion is performed by the A / D conversion unit 5. Input to the image processing unit 6.

The pixel value input to the image processing unit 6 is corrected in black level by the black variation correcting unit 61, and sequentially supplied to the inflection point variation correcting unit 621 and the interpolation determining unit 622 as the pixel value d. The pixel value d supplied to the inflection point variation correcting unit 621 has the photoelectric conversion characteristic unified to a linear characteristic common to all pixels, and is supplied to the pixel interpolating unit 63 as a pixel value d ′.

On the other hand, whether or not the pixel value d is an inflection point defective pixel is determined by the interpolation determination unit 622. Then, the determination result is supplied to the pixel interpolation unit 63.

The pixel interpolation unit 63 stores a 1 or 0 label in each field of the placement information MB shown in FIG. 6 according to the determination result by the interpolation determination unit 622, and at the same time, 1 or 0 in each field of the placement information MA according to the defective pixel information. Stores the label.

When the pixel interpolation unit 63 stores labels for one column in the horizontal direction in the arrangement information MA and the arrangement information MB, the pixel interpolation unit 63 obtains a logical sum of the arrangement information MA and the arrangement information MB, and generates the interpolation pixel arrangement information MC. To do. Then, the pixel interpolation unit 63 sequentially sets the pixels in which the label of 1 is stored in the interpolation pixel arrangement information MC as the target interpolation target pixel, and positions the normal pixel located at the shortest distance on the left side and the shortest distance on the right side. The target pixel to be interpolated is interpolated using the normal pixel. In the interpolation pixel arrangement information MC, the pixel value d ′ before interpolation of the interpolation target pixel is updated to the pixel value d ″ after interpolation.

Then, when the processing for one column in the horizontal direction is completed, the pixel interpolation unit 63 supplies each pixel value stored in each field of the interpolation pixel arrangement information MC to the gradation conversion unit 64 as a pixel value d ″.

The pixel value d ″ is subjected to histogram equalization processing and dynamic range compression processing by the gradation conversion unit 64, Bayer interpolation is performed by the Bayer interpolation unit 65, and gamma correction is performed by the tone curve correction unit 66. It is supplied to the memory 7 and the like. The above processing is repeated, and one piece of image data is stored in the image memory 7.

FIG. 8 is a graph showing problems in interpolation processing when only inflection point defective pixels are set as interpolation target pixels without considering defective pixels. The vertical axis indicates the pixel value, and the horizontal axis indicates the pixel value of one column in the horizontal direction.

In FIG. 8, pixels G (1), G (2), G (4), G (7) are inflection point defective pixels, and pixels G (3), G (5), G (6) are defective. Pixel. Pixels G ′ (1) to G ′ (7) are pixels after interpolation.

Since pixels G (1), G (2), and G (7) do not have defective pixels in the periphery, interpolation using defective pixels is not performed. Therefore, the interpolated pixels G ′ (1), G ′ (2), and G ′ (7) are not significantly different from the pixel values of the surrounding pixels, and the interpolation is successful.

On the other hand, since the pixel G (4) includes pixels G (3) and G (5) which are defective pixels, the pixel G ′ (3) or the pixel G (3) and the pixel G ′ (5) are present. Alternatively, interpolation is performed using the pixel G (5). For this reason, the pixel G (4) is dragged by the defective pixel, the pixel value is greatly reduced as compared with the surrounding pixels, and the interpolation fails.

As described above, when the interpolation target pixel is interpolated without considering the defective pixel, there is a problem that the pixel value after the interpolation is greatly influenced by the pixel value of the defective pixel, and accurate interpolation cannot be performed.

On the other hand, in this embodiment, interpolation target pixels are interpolated without using inflection point defective pixels and defective pixels. Therefore, the interpolation target pixel is interpolated using only normal pixels, so that accurate interpolation can be realized.

FIG. 9 is a screen diagram showing the effect of the interpolation processing by the imaging apparatus 1 according to the embodiment of the present invention. FIG. 9A shows a case where interpolation is performed without considering defective pixels, and FIG. 9B shows a case where interpolation is performed according to this embodiment.

As shown in FIG. 9A, when interpolation is performed using a defective pixel, it can be seen that the interpolation target pixel is affected by the defective pixel and numerous noises appear. On the other hand, according to this embodiment, since interpolation using defective pixels is not performed, it can be seen that almost no noise appears.

In the above embodiment, the normal pixels are extracted in consideration of the horizontal direction or the vertical direction with respect to the interpolation target pixel. However, the present invention is not limited to this, and the normal pixels may be extracted in consideration of the oblique direction. Good.

For example, searching for normal pixels in the eight directions of the upper, lower, left, right, upper left, upper right, lower left, and lower right with the pixel to be interpolated as the center, and using the normal pixels located at the shortest distance in each direction, The interpolation target pixel may be interpolated. Alternatively, the interpolation target pixels may be interpolated using a predetermined number (for example, two) of normal pixels in order of increasing distance among normal pixels located at the shortest distance in each of the eight directions.

Also, the normal pixel PU with the shortest distance in the upward direction, the normal pixel PD with the shortest distance in the downward direction, the normal pixel PL with the shortest distance in the left direction, the normal pixel PR with the shortest distance in the right direction, and the normal pixel with the shortest distance in the upper left direction. Of the normal pixel PB having the shortest distance in the upper right direction, the normal pixel PC having the shortest distance in the lower left direction, and the normal pixel PD having the shortest distance in the lower right direction, the distance 11 of the normal pixels PD and the normal pixels PL and PR The interpolation target pixel may be interpolated using two normal pixels having the smallest distance among the distance l2, the distance l3 between the normal pixels PA and PD, and the distance l4 between the normal pixels PB and PC.

(Summary of this embodiment)
(1) An imaging device according to the present embodiment includes an imaging device including a plurality of pixels having different photoelectric conversion characteristics at an inflection point, and a defective pixel indicating a position of a defective pixel among pixels constituting the imaging device. For each pixel, a defective pixel information storage unit that stores information in advance, an inflection point information storage unit that stores inflection point information indicating an inflection point pixel value that is a pixel value of an inflection point of each pixel, and An interpolation determination unit that determines whether or not the pixel is an inflection point defective pixel based on the pixel value when the subject is imaged and the inflection point pixel value, and the inflection point defective pixel and the defective pixel are to be interpolated A pixel interpolation unit that interpolates the interpolation target pixel using a normal pixel other than the inflection point defective pixel and a normal pixel other than the defective pixel.

According to this configuration, the interpolation target pixel is interpolated using not only the inflection point defective pixel but also normal pixels other than the defective pixel. Therefore, the pixel value after interpolation is prevented from being greatly affected by the pixel value of the defective pixel, and the interpolation accuracy can be increased.

(2) The pixel interpolation unit generates interpolation pixel arrangement information indicating the position of the interpolation target pixel, extracts normal pixels in the vicinity of the interpolation target pixel with reference to the interpolation pixel arrangement information, and extracts the extracted normal It is preferable to interpolate the interpolation target pixel using a pixel.

According to this configuration, the interpolation pixel arrangement information indicating the position of the interpolation target pixel is generated, and the normal pixel located in the vicinity of the interpolation target pixel is searched based on the generated information. Therefore, the normal pixel can be searched with high accuracy. it can.

(3) The pixel interpolation unit generates first arrangement information indicating the arrangement of the inflection point defective pixels based on the determination result of the interpolation determination unit, and based on the defective pixel information, It is preferable that second arrangement information indicating arrangement is generated, and the interpolation pixel arrangement information is generated from a logical sum of the first arrangement information and the second arrangement information.

According to this configuration, since the interpolation pixel arrangement information is calculated using the logical sum of the first arrangement information and the second arrangement information, the interpolation pixel arrangement information can be calculated quickly and accurately.

(4) The pixel interpolation unit extracts and extracts the normal pixels located close to each other on one side and the other side in the horizontal direction with respect to the interpolation target pixel with reference to the interpolation pixel arrangement information It is preferable that the interpolation target pixel is interpolated using at least the normal pixel.

According to this configuration, interpolation processing can be performed if interpolation pixel arrangement information for one column in the horizontal direction is obtained without waiting for interpolation pixel arrangement information for all pixels to be obtained. Therefore, interpolation processing can be performed at high speed.

(5) The pixel interpolating unit is close to the normal pixel located on each of one side and the other side in the horizontal direction and on one side and the other side in the vertical direction with respect to the interpolation target pixel. It is preferable that the normal pixels located are extracted with reference to the interpolation pixel arrangement information, and the interpolation target pixel is interpolated using at least one of the extracted normal pixels.

According to this configuration, since normal pixels are searched in consideration of not only the horizontal direction but also the vertical direction, the interpolation accuracy can be improved.

(6) The pixel interpolation unit compares the distance between the normal pixels extracted in the horizontal direction with the distance between the normal pixels extracted in the vertical direction, and uses at least the normal pixels in a short distance direction. It is preferable to interpolate the interpolation target pixel.

According to this configuration, the distance between the normal pixels searched in the horizontal direction and the distance between the normal pixels searched in the vertical direction are compared, and interpolation processing is performed using at least normal pixels in a direction closer to the distance. Interpolation accuracy can be improved.

(7) When interpolating the interpolation target pixel using the extracted normal pixel, the pixel interpolation unit preferably interpolates the interpolation target pixel using an average value of pixel values of the extracted normal pixels. .

According to this configuration, since the interpolation target pixel is interpolated using the average value of the pixel values of normal pixels, the interpolation process can be simplified.

(8) When interpolating the interpolation target pixel using the extracted normal pixel, the pixel interpolation unit assigns a higher weight value to the normal pixel that is closer to the interpolation target pixel, and extracts the normal pixel It is preferable to interpolate the pixel to be interpolated by weighting and adding the pixel values.

According to this configuration, the normal value closer to the interpolation target pixel has a higher weight value, so that the interpolation accuracy can be further increased.

(9) When the pixel whose inflection point pixel value is higher than a predetermined first inflection point threshold value outputs a pixel value higher than the predetermined first pixel threshold value, the interpolation determination unit converts the inflection point pixel value to the inflection point. It is preferable to determine as a point defect pixel.

According to this configuration, even if the pixel value of the inflection point pixel is higher than the first inflection point threshold value, when a pixel value lower than the first pixel value is output, the pixel is adopted as a normal pixel. The number of interpolation target pixels can be reduced and the resolution can be increased.

(10) When the pixel whose inflection point pixel value is lower than a predetermined second inflection point threshold value outputs a pixel value lower than the predetermined second pixel threshold value, the interpolation determination unit converts the inflection point pixel value to the inflection point. It is preferable to determine as a point defect pixel.

According to this configuration, even if the pixel value of the inflection point pixel is lower than the second inflection point threshold value, when a pixel value higher than the first pixel value is output, the pixel is adopted as a normal pixel. The number of interpolation target pixels can be reduced and the resolution can be increased.

This application is based on Japanese Patent Application No. 2011-102963 filed on May 2, 2011, the contents of which are included in the present application.

In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Therefore, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not covered by the claims. It is interpreted that it is included in

Claims (10)

An image sensor comprising a plurality of pixels having different photoelectric conversion characteristics at an inflection point;
A defective pixel information storage unit that preliminarily stores defective pixel information indicating a position of a defective pixel among pixels constituting the imaging element;
An inflection point information storage unit for preliminarily storing inflection point information indicating an inflection point pixel value that is a pixel value of an inflection point of each pixel;
For each pixel, an interpolation determination unit that determines whether the pixel is an inflection point defective pixel based on the pixel value when the subject is imaged and the inflection point pixel value;
The inflection point defective pixel and the defective pixel are set as interpolation target pixels, and the interpolation target pixel includes a pixel interpolation unit that performs interpolation using normal pixels other than the inflection point defective pixels and normal pixels other than the defective pixels. Imaging device. The pixel interpolation unit generates interpolation pixel arrangement information indicating a position of the interpolation target pixel, extracts normal pixels in the vicinity of the interpolation target pixel with reference to the interpolation pixel arrangement information, and uses the extracted normal pixels The imaging apparatus according to claim 1, wherein the interpolation target pixel is interpolated. The pixel interpolation unit generates first arrangement information indicating the arrangement of the inflection point defective pixels based on the determination result of the interpolation determination unit, and indicates the arrangement of the defective pixels based on the defective pixel information. The imaging apparatus according to claim 2, wherein second interpolation information is generated, and the interpolation pixel arrangement information is generated from a logical sum of the first arrangement information and the second arrangement information. The pixel interpolation unit extracts the normal pixels that are located close to each other on one side and the other side in the horizontal direction with respect to the interpolation target pixel with reference to the interpolation pixel arrangement information, and extracts the normal pixels The imaging apparatus according to claim 2, wherein the interpolation target pixel is interpolated using at least a pixel. The pixel interpolation unit is located close to the interpolation target pixel on each of one side and the other side in the horizontal direction, and on the one side and the other side in the vertical direction. The imaging apparatus according to claim 2, wherein normal pixels are extracted with reference to the interpolation pixel arrangement information, and the interpolation target pixel is interpolated using at least one of the extracted normal pixels. The pixel interpolation unit compares a distance between the normal pixels extracted in the horizontal direction with a distance between the normal pixels extracted in the vertical direction, and uses the normal pixels in a short distance direction to perform the interpolation. The imaging apparatus according to claim 5, wherein the target pixel is interpolated. The pixel interpolation unit, when interpolating the interpolation target pixel using the extracted normal pixel, interpolates the interpolation target pixel using an average value of pixel values of the extracted normal pixels. The imaging device according to any one of the above. When interpolating the interpolation target pixel using the extracted normal pixel, the pixel interpolation unit assigns a higher weight value to the normal pixel that is closer to the interpolation target pixel, and the extracted pixel value of the normal pixel 7. The imaging apparatus according to claim 2, wherein the interpolation target pixel is interpolated by weighting and adding. When the pixel whose inflection point pixel value is higher than a predetermined first inflection point threshold value outputs a pixel value higher than a predetermined first pixel threshold value, the interpolation determination unit identifies the pixel as the inflection point defective pixel. The imaging apparatus according to any one of claims 1 to 8, wherein When the pixel whose inflection point pixel value is lower than a predetermined second inflection point threshold value outputs a pixel value lower than a predetermined second pixel threshold value, the interpolation determination unit identifies the pixel as the inflection point defective pixel. The imaging device according to claim 1, wherein the imaging device is determined as follows.

Images