WO2017064746A1 - Imaging device, endoscopic device and imaging method - Google Patents

Abstract

This imaging device includes an imaging unit and an illumination unit. The imaging unit can capture images (Vb, Vg, Vr) including a first color image, a second color image having a longer wavelength than the first color, and a third color image having a longer wavelength than the second color, and can also capture the first color image and the third color image as a stereo image (Mb, Mr). The illumination unit irradiates a subject with patterned light having prescribed light quantity distribution in a first wavelength band Pb included in the wavelength band of the first color while not included in a second-color wavelength band, and in a second wavelength band Pr included in the wavelength band of the third color while not included in the second-color wavelength band.

Description

Imaging apparatus, endoscope apparatus, and imaging method

The present invention relates to an imaging device, an endoscope device, an imaging method, and the like.

Various methods for optically 3D measuring the surface shape of an object have been proposed. As one of the methods, a method of projecting active pattern illumination onto an object and performing stereo three-dimensional distance measurement by binocular stereoscopic vision is proposed. is there. In stereo measurement, matching processing is performed using a characteristic part (for example, an edge) of an image, and a distance to an object is obtained. By projecting the active pattern, a characteristic part is intentionally added to the subject, and distance measurement can be accurately performed even in a part without a characteristic part (for example, a flat part).

As a conventional technique of stereo measurement, for example, there are techniques disclosed in Patent Documents 1 and 2. Japanese Patent Application Laid-Open No. 2004-133620 discloses a technique for switching left and right imaging light paths with a mechanical shutter in time and acquiring left and right images in a time-sharing manner. Patent Document 2 discloses a technique of inserting an RG filter in the left half of a single imaging optical path and a GB filter in the right half, and separating the left and right images from the R image and the B image of the captured image. Yes. In Patent Document 2, in the case of normal observation, the RG filter and the GB filter are retracted from the imaging optical path, and an observation image is acquired.

JP 2010-128354 A JP 2013-3159 A

When the active pattern illumination as described above is used, there is a problem that a normal image cannot be obtained by flat illumination because the active pattern is reflected in the captured image. For example, in a non-destructive inspection using an endoscope apparatus, inspection may be performed by capturing an observation image that is not stereo shooting, and stereo measurement may be performed at a portion to be examined in detail. In such a case, for example, switching between flat illumination and active pattern illumination can be considered, but the illumination mechanism becomes complicated. Alternatively, by alternately switching between stereo shooting and normal observation, convenience can be improved when real-time stereo measurement can be performed while performing normal observation, but it is preferable that illumination switching is not accompanied at that time. .

According to some aspects of the present invention, it is possible to provide an imaging apparatus, an endoscope apparatus, an imaging method, and the like that can capture a normal observation image while performing illumination with active pattern illumination.

One embodiment of the present invention includes an image including a first color image, a second color image having a longer wavelength side than the first color, and a third color image having a longer wavelength side than the second color. An imaging unit that captures an image and can capture the first color image and the third color image as a stereo image, and is included in the wavelength band of the first color and included in the wavelength band of the second color A pattern illumination having a given light amount distribution in a second wavelength band that is included in the first wavelength band that is not included in the wavelength band of the third color and that is not included in the wavelength band of the second color. And an illumination unit.

According to one aspect of the present invention, a pattern illumination having a given light amount distribution in a first wavelength band included in the first color wavelength band and a second wavelength band included in the third color wavelength band is a subject. Is irradiated. Since the first wavelength band and the second wavelength band are not included in the wavelength band of the second color, the image of the second color is an image that is not affected by a given light amount distribution due to pattern illumination. This makes it possible to capture a normal observation image while performing illumination with active pattern illumination.

Further, another aspect of the present invention relates to an endoscope apparatus that includes the imaging apparatus described above.

According to still another aspect of the present invention, there is provided a first color image, a second color image having a longer wavelength side than the first color, and a third color image having a longer wavelength side than the second color. Among these, when the image of the first color and the image of the third color can be captured as a stereo image, the first color that is included in the wavelength band of the first color and not included in the wavelength band of the second color. A pattern illumination having a given light amount distribution in a second wavelength band that is included in the third wavelength band and not included in the second color wavelength band; and The present invention relates to an imaging method for imaging a captured image including an image of one color, an image of the second color, and an image of the third color.

An example of pattern illumination. The example of the waveform of the light quantity of pattern illumination, the reflection coefficient distribution of a to-be-photographed object, and a captured image. A first spectral characteristic example of pattern illumination and a first spectral characteristic example of a pupil in an observation mode and a stereo measurement mode. The relationship between a 1st spectral characteristic example and the captured image in observation mode and stereo measurement mode. The 2nd spectral characteristic example of pattern illumination and the 2nd spectral characteristic example of the pupil in observation mode and stereo measurement mode. The relationship between the 2nd spectral characteristic example and the captured image in observation mode and stereo measurement mode. Explanatory drawing of a correction process. Explanatory drawing of a correction process. The 1st structural example of an illumination part. Second configuration example of illumination unit The structural example of an imaging part. The structural example of an imaging part. The 1st detailed structural example of a fixed mask and a movable mask. The 1st detailed structural example of a fixed mask and a movable mask. The 2nd detailed structural example of a fixed mask and a movable mask. The 2nd detailed structural example of a fixed mask and a movable mask. An explanatory view of the principle of stereo measurement. The structural example of an endoscope apparatus (imaging apparatus). Sequence that switches between observation mode and stereo measurement mode.

Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

For example, an industrial endoscope apparatus will be described below as an application example of the present invention. However, the present invention is not limited to application to an industrial endoscope apparatus, and a stereo shooting method (with an imaging system having parallax). A method of detecting the phase difference between the two images and acquiring subject distance information), a three-dimensional measuring device that measures a three-dimensional shape, and an imaging device having a three-dimensional measuring function (for example, a medical endoscope device, Microscopes, industrial cameras, robot vision functions, etc. are applicable.

1. Pattern illumination In order to obtain a high-quality observation image, it is generally premised that the subject is illuminated uniformly. However, when performing stereo measurement, distance information to the subject cannot be obtained unless there is a feature that easily obtains a phase difference. Therefore, a method of irradiating the subject with pattern illumination that is intentionally characterized is used.

In the present embodiment, an observation mode for capturing an observation image using white light and a measurement mode for performing stereo measurement by the color phase difference method are switched and used. When conventional pattern illumination is used, it is necessary to have a function of switching the illumination itself so that uniform illumination is performed in the observation mode and pattern illumination is performed in the measurement mode.

Therefore, in the present embodiment, a method is used in which pattern illumination is always performed in the observation mode or in the measurement mode, and the pattern in the observation mode is erased by a spectral filter or image processing. Thereby, the illumination switching function can be made unnecessary. Hereinafter, this method will be described.

FIG. 1 schematically shows an example of pattern illumination. FIG. 1 shows a plan view of the pattern illumination PL (for example, a plan view when projected onto a plane perpendicular to the optical axis of the imaging system) and an example of the light quantity characteristic in the AA section. x is a position (coordinate) in a direction perpendicular to the optical axis.

As shown in FIG. 1, a small circle pattern DT whose brightness changes is an illumination pattern regularly arranged. The outside of the small circle pattern DT is illumination with flat brightness, and the inside of the small circle pattern DT is illumination darker than that.

When a subject image is captured by such pattern illumination PL, a waveform obtained by multiplying the amount of light of pattern illumination PL and the reflection coefficient distribution of the subject is obtained as an imaging waveform (sensor output) as shown in FIG. The sensor output in FIG. 2 represents a pixel value or a luminance value (or brightness of image formation on the sensor surface) at a pixel corresponding to the position x. A solid line indicates a sensor output when pattern illumination is performed, and a dotted line indicates a sensor output when flat illumination is performed.

2. First Spectral Characteristics Example FIG. 3 shows a first spectral characteristic example of pattern illumination and a first spectral characteristic example of a pupil in the observation mode and the stereo measurement mode. Dotted lines with B, G, R, and IR symbols represent spectral characteristics of the color filter of the image sensor. B, G, and R are spectral characteristics of the blue, green, and red filters of the image sensor, respectively. IR is an infrared sensitivity characteristic and passes through any of the blue, green, and red filters.

As shown in the upper diagram of FIG. 3, in the first spectral characteristic example of the pattern illumination, the wavelength bands are set to five {Pv1, Pb, Pv2, Pr, Pir} in association with the spectral characteristics of the color filter of the imaging sensor. To divide. The illumination light in the bands {Pv1, Pv2, Pir} is non-pattern light (flat light) with a uniform light amount distribution, and the illumination light in the bands {Pb, Pr} is pattern light. Light having a spectral characteristic obtained by combining these becomes illumination light to the subject.

The bands Pv1 and Pb are set to bands that pass through the blue filter of the image sensor but do not pass through the green filter, the band Pv is set to a band that passes through the green filter, and the bands Pr and Pir are red. A band that passes through the filter but does not pass through the green filter is set. The band Pir includes an infrared sensitivity characteristic IR. Note that the band Pir may not include the infrared sensitivity characteristic IR. Thus, the bands Pb and Pr are set to wavelength bands that do not interfere with the spectral characteristic G of the green filter of the image sensor. The bands Pb and Pr are set to narrow bands such as several nm to several tens of nm, for example.

More specifically, for the band Pb, it is desirable to select a wavelength range in which only the blue pixel (spectral characteristic B) of the image sensor can be acquired and the sensitivity is relatively good. For the band Pr, it is desirable to select a wavelength range in which only the red pixel (spectral characteristic R) of the imaging sensor can be acquired and the sensitivity is relatively good. That is, since the left pupil image and the right pupil image must be separated by different color pixels of the image sensor, {Pb, Pr} may be selected in a wavelength region that does not have a mutual light receiving sensitivity characteristic. Since the spectral components {Pv1, Pv2, Pir} of the non-pattern illumination light are components of a normal observation image, the red pixel, the green pixel, and the blue pixel (spectral characteristics R, It is desirable that G and B) can cover as many wavelength ranges as possible. For example, a standard laser light source with wavelengths of 450 nm and 660 nm is used for {Pb, Pr} so as to be a narrow-band light source, and the spectral components {Pv1, Pv2, Pir} may cover many wavelength components. This is one way.

As shown in the middle and lower diagrams of FIG. 3, the relationship between the spectral characteristics FL, FC, FR of the pupil and the spectral characteristics of the illumination light is set as in the following formula (1).

In the observation mode, images are taken with the pupil of the spectral characteristic FC. The spectral characteristic FC corresponds to the wavelength band {Pv1, Pv2, Pir}, and an observation image by flat illumination is obtained. On the other hand, in the stereo measurement mode, shooting is performed with the left pupil of the spectral characteristic FL and the right pupil of the spectral characteristic FR. The spectral characteristics FL and FR correspond to the wavelength bands Pb and Pr, and a stereo image by active pattern illumination is obtained.

The observation mode is a mode in which monocular imaging is performed with the central aperture 23 (spectral characteristic FC) of the fixed mask 20 as shown in FIGS. 11, 13, and 15, and the stereo measurement mode is shown in FIGS. 12, 14, and 16. In this manner, stereo imaging is performed with the left pupil aperture 21 (spectral characteristic FL) and right pupil aperture 22 (spectral characteristic FR) of the fixed mask 20 as described above. Details of these observation modes and stereo measurement modes will be described later.

FIG. 4 shows the relationship between the first spectral characteristic example and the captured images in the observation mode and the stereo measurement mode. In the waveform in the right diagram of FIG. 4, the dotted line virtually indicates the sensor output when flat illumination is performed.

In the following, the case where the active pattern illumination of the present embodiment is applied to the imaging unit of FIGS. 11 to 16 will be described as an example. However, the configuration of the imaging unit is not limited to FIGS. Any observation image can be used as long as a stereo image by spectral characteristics FL and FR can be taken.

The illumination light is the same regardless of the mode, and the active pattern illumination is projected onto the subject in both the observation mode and the stereo measurement mode.

In the observation mode, the reflected light by the illumination light having the spectral components {Pv1, Pv2, Pir} of the non-pattern light passes through the pupil center optical path (spectral characteristic FC), and a captured image {Vr, Vg, Vb} is obtained. Vr is a red image obtained by a pixel having a red filter of the image sensor, Vg is a green image obtained by a pixel having a green filter of the image sensor, and obtained by a pixel having a blue filter of the image sensor It is a blue image. In the observation mode, since the band of the active pattern is removed, an observation image that is not affected by the active pattern is obtained.

In the stereo measurement mode, the reflected light by the illumination light having the spectral component {Pb, Pr} of the pattern light passes through the left and right pupil optical paths, and a captured image {Mr, Mb} is obtained. Mr is an image by the left pupil optical path (spectral characteristic FL), and Mb is an image by the right pupil optical path (spectral characteristic FR). In the stereo measurement mode, since a pattern is intentionally formed even on a smooth subject surface, the captured image {Mr, Mb} can be easily matched and phase difference detection can be easily performed.

According to the above embodiment, the imaging device (endoscope device) captures the captured images (Vb, Vg, Vr) including the first color image, the second color image, and the third color image. An imaging unit capable of capturing a first color image and a third color image as a stereo image (Mb, Mr), and pattern illumination having a given light amount distribution in the first wavelength band Pb and the second wavelength band Pr And an illumination unit that irradiates the subject. The second color (green) image is an image on the longer wavelength side than the first color (blue), and the third color (red) image is an image on the longer wavelength side than the second color. The first wavelength band Pb is a band that is included in the first color wavelength band (spectral characteristic B band) and not included in the second color wavelength band (spectral characteristic G band). The second wavelength band Pr is a band that is included in the third color wavelength band (spectral characteristic R band) and not included in the second color wavelength band (spectral characteristic G band).

In this way, since the first wavelength band Pb and the second wavelength band Pr are not included in the wavelength band of the second color (green), at least the pattern of the given light quantity distribution is present in the second color image. Is not reflected. Thereby, it is possible to switch between stereo measurement and normal observation without switching between pattern illumination and flat illumination.

That is, the stereo image is composed of a first color (blue) image and a third color (red) image, and the first color image includes a pattern in the first wavelength band Pb, and the third color image. Shows a pattern in the second wavelength band Pr. Thereby, a stereo image intentionally characterized by a pattern is acquired, and high-precision stereo measurement can be performed by performing matching processing thereof. In addition, as described above, an observation image is captured through a spectral filter (spectral characteristic FC) that does not pass the first wavelength band Pb and the second wavelength band Pr, so that an observation image in which a pattern due to a given light amount distribution is not captured is obtained. Is obtained. Alternatively, as will be described later with reference to FIGS. 5 to 8, an observation image of the entire band of white light is photographed, and the first color (blue) and third color (red) images in which the pattern is captured are not captured. By correcting using two-color (green) images, an observation image from which the pattern has been removed is obtained. As described above, since a pattern with a given light quantity distribution is not provided in the wavelength band of the second color, it is possible to obtain an observation image as in flat illumination while performing pattern illumination.

Note that, here, the first wavelength band is the wavelength band Pb in FIG. 3 and the second wavelength band is the wavelength band Pr in FIG. 3, but the present invention is not limited to this. For example, the first wavelength band may be the wavelength band Pb1 in FIG. 5, and the second wavelength band may be the wavelength band Pr2 in FIG.

Here, a given light amount distribution (a light amount distribution of a given shape) is a distribution having a light / dark (light amount) boundary or a distribution having an edge portion where the light amount changes abruptly. A plurality of parts are provided in the irradiation area of the pattern illumination. By photographing a subject illuminated with pattern illumination having a given light intensity distribution, the boundaries of the given light intensity distribution and the shape and arrangement of edges are reflected in the captured image, thereby capturing the features required for stereo measurement. Can be given to images. In the present embodiment, this given light quantity distribution is given only to specific wavelength bands Pb and Pr. In FIG. 1, small circle patterns DT darker than the surroundings are regularly arranged in the wavelength bands Pb and Pr, but the given light quantity distribution is not limited to this. For example, the pattern DT may not be a small circle, the arrangement of the pattern DT may not be regular, and the inside of the pattern DT may be brighter than the outside.

In this embodiment, the imaging unit switches between a stereo mode for capturing a stereo image and a non-stereo mode for capturing a captured image with a single eye.

For example, in the imaging unit described later with reference to FIGS. 11 to 16, in the non-stereo mode (observation mode), monocular imaging is performed with the central aperture 23 (spectral characteristic FC) of the fixed mask 20, and in the stereo mode (stereo measurement mode), the image is fixed. Stereo imaging is performed with the left pupil aperture 21 (spectral characteristics FL) and right pupil aperture 22 (spectral characteristics FR) of the mask 20. Note that the imaging unit to which the pattern illumination of the present embodiment can be applied is not limited to this, and any imaging unit that captures the first color image and the third color image as a stereo image in the stereo mode may be used.

In this way, by switching between a stereo mode in which the first color image and the third color image are captured as a stereo image and a non-stereo mode in which the first to third color images are captured with a monocular as a captured image, It is possible to switch between stereo measurement and observation image capturing. In this case, by using the above-described pattern illumination, it is possible to switch between stereo measurement and observation image capturing while always performing pattern illumination.

In this embodiment, the monocular in the non-stereo mode uses the first wavelength band Pb and the second wavelength band Pr out of the wavelength bands (white light wavelength bands) of the first color, the second color, and the third color. Except the wavelength bands {Pv1, Pv2, Pir}.

Since the pattern based on the given light quantity distribution is attached to the first wavelength band Pb and the second wavelength band Pr, the wavelength band {Pv1, Pv2, Pir} through which the monocular passes is a pattern based on the given light quantity distribution. Is not attached. As a result, an observation image such as flat illumination can be taken despite pattern illumination in the non-stereo mode. In particular, when the first wavelength band Pb and the second wavelength band Pr are narrow bands as described above, the wavelength bands {Pv1, Pv2, Pir} excluding them are substantially the wavelength band of white light. Therefore, it is possible to obtain an image that is not inferior to a captured image obtained by illumination with white light.

In this embodiment, as will be described later with reference to FIG. 18 and the like, the imaging apparatus may include a phase difference detection unit 330 and an image output unit (color image generation unit 320). The phase difference detection unit 330 detects a phase difference between the first color image and the third color image captured in the stereo mode. The image output unit outputs an image for observation based on the captured image (first color to third color image) captured in the non-stereo mode.

As described above, since the pattern of pattern illumination is reflected in the first color image and the third color image, the phase difference detection unit 330 can detect the phase difference with high accuracy. In addition, since the pattern is not captured in the captured image captured in the non-stereo mode even though pattern illumination is performed, an image for observation can be output by the image output unit.

In this embodiment, the wavelength band of the first color wavelength band (spectral characteristic B) excluding the first wavelength band Pb, the second color wavelength band (spectral characteristic G), and the third color wavelength band. It is a flat light amount distribution in the wavelength band excluding the second wavelength band Pr in (spectral characteristic R).

In this way, since the illumination light has a flat light amount distribution other than the first wavelength band Pb and the second wavelength band Pr, at least the second color image is an image by flat illumination. As a result, as described above, it is possible to switch between stereo measurement and observation image capturing while always performing pattern illumination.

Here, the flat light quantity distribution (flat illumination) means that the light quantity distribution is constant (substantially constant) in the photographing region (field of view) photographed by the imaging unit. Specifically, the light amount distribution on a surface having a constant distance from the imaging unit (a surface perpendicular to the optical axis) is constant. Note that the light amount distribution need not be completely constant, and there may be a gradual light amount change without an abrupt light amount change (edge portion) such as pattern illumination. Alternatively, there may be a light amount distribution that can be considered in normal illumination, in which the light amount falls more slowly in the peripheral part of the imaging region than in the central part.

In this embodiment, the first color is blue, the second color is green, and the third color is red.

Note that the first color image, the second color image, and the third color image correspond to a blue image, a green image, and a red image, respectively, and these images have an image pickup element having a primary color filter. It is not limited to what was imaged (for example, image sensor of primary color Bayer arrangement). For example, a complementary color image may be captured by an imaging element having a complementary color filter, and a blue image, a green image, and a red image may be acquired from the complementary color image by conversion processing.

3. Second spectral characteristic example In the first spectral characteristic example, the case where the imaging bands of the observation mode and the stereo measurement mode are separated has been described. In the second spectral characteristic example, a case where the imaging bands of the observation mode and the measurement mode are not separated will be described. If you can always use illumination that combines non-patterned light and patterned light without separating the wavelength band, regardless of the mode, you can use the wavelength band effectively, and the illumination switching function is unnecessary, ensuring imaging sensitivity. Covering the color information of the subject is advantageous in terms of simplifying the illumination mechanism. However, if the wavelength bands are not separated, the captured image in the observation mode is also affected by the pattern light. Therefore, in order to obtain a high-quality and faithful observation image, it is necessary to remove or reduce the influence of the pattern light.

FIG. 5 shows a second spectral characteristic example of pattern illumination and a second spectral characteristic example of the pupil in the observation mode and the stereo measurement mode.

In the spectral characteristics of the pattern illumination, the wavelength band is divided into five parts {Pb1, Pb2, Pr1, Pr2, Pir} in association with the spectral characteristics of the color filter of the image sensor. The illumination light in the bands {Pb2, Pr1, Pir} is non-pattern light (flat light) with a uniform light quantity distribution, and the illumination light in the bands {Pb1, Pr2} is pattern light. Light having a spectral characteristic obtained by combining these becomes illumination light to the subject.

Pb1 is set in a band that passes through the blue filter but does not pass through the green filter, and Pb2 is set in a band that passes through both the blue filter and the green filter. Pr1 is set in a band that passes through both the red filter and the green filter, and Pr2 is set in a band that passes through the red filter but does not pass through the green filter. Pir is a band corresponding to the infrared sensitivity characteristic IR, and is a band that passes through all the red, green, and blue filters. Thus, the wavelength bands Pb1 and Pr2 are set to wavelength bands that do not interfere with the spectral characteristics G of the green filter of the image sensor.

In the observation mode, an image is taken with the pupil of the spectral characteristic FC, which includes the wavelength band {Pb1, Pb2, Pr1, Pr2, Pir}. For example, no filter is provided in the central aperture 23 of the fixed mask 20, and the central pupil optical path allows light in the entire band to pass. The relationship between the red image Vr, the green image Vg, and the blue image Vb constituting the color image captured in the observation mode and the wavelength band covered by them is expressed by the following equation (2).

In the stereo measurement mode, shooting is performed with the left pupil of the spectral characteristic FL and the right pupil of the spectral characteristic FR. The spectral characteristic FL corresponds to the wavelength band Pb1, and the spectral characteristic FR corresponds to the wavelength band Pr2. That is, the relationship between the red image Mr and the blue image Mb captured in the stereo measurement mode and the wavelength band covered by them is expressed by the following equation (3).

FIG. 6 shows the relationship between the second spectral characteristic example and the captured images in the observation mode and the stereo measurement mode. In the waveform in the right diagram of FIG. 6, the dotted line virtually indicates the sensor output when flat illumination is performed.

In the observation mode, the reflected light by the illumination light having the spectral components {Pb1, Pb2, Pr1, Pr2, Pir} passes through the pupil center optical path (spectral characteristic FC), and a captured image {Vr, Vg, Vb} is obtained. In the stereo measurement mode, the reflected light by the illumination light having the spectral components {Pb1, Pr2} of the pattern light passes through the left and right pupil optical paths, and a captured image {Mr, Mb} is obtained.

In the observation mode, the observation images Vb and Vr are affected by the pattern illumination, so that the images are intentionally changed in brightness. On the other hand, since the observation image Vg is composed of the wavelength range {Pb2, Pr1, Pir} that is uniform illumination, no intentional brightness change occurs, and the image profile reflects only the reflection coefficient of the subject.

4). Correction Processing When the second spectral characteristic example is used, the observation images Vr and Vb are profiles affected by the pattern, but assuming a profile (dotted line in FIG. 6) obtained when the illumination is uniform. Although the average brightness is different, the similarity (similarity) with the observed image Vg is partially high. This is because the observation image {Vr, Vg} is an image in which the spectral characteristics of the imaging sensor overlap in the band Pb2, and the observation image {Vb, Vg} is an image in which the spectral characteristics of the imaging sensor overlap in the band Pr1. This is because there is a considerable correlation between them. Therefore, if the observation image Vg not affected by the pattern is used, the observation images Vr and Vb can be corrected, and the influence of the pattern can be removed or reduced. That is, it is possible to restore an observation image as if it was taken with flat illumination. This correction process will be described below.

7 and 8 are explanatory diagrams of the correction process. In the following description, the correction of the blue image Vb will be described as an example, but the correction of the red image Vr can be similarly performed.

As shown in FIG. 7, the correlation value between the waveform of the blue image Vb and the green image Vg in the section of the width d centering on the arbitrary position XL on the sensor surface of the image sensor is calculated. This is performed for all positions x on the sensor surface (that is, all pixels of the captured image). For example, when ZNCC (Zero-mean Normalized Cross-Correlation) is used, the correlation value becomes 1 when the similarity is maximum, and the correlation value approaches 0 as the similarity becomes lower.

The correlation value is compared with the threshold Th, and when the correlation value is equal to or greater than the threshold Th, the flag value is set to “1”, and when the correlation value is smaller than the threshold Th, the flag value is set to “0”. That is, the flag value is obtained by binarizing the similarity with / without similarity. In the correction process, a pixel with a flag value “1” is determined as a valid pixel, and a pixel with a flag value “0” is determined as an invalid pixel.

Note that the correlation value is not limited to the case of obtaining all the pixels of the captured image. For example, the correlation value may be obtained for pixels in a predetermined region, or may be obtained for pixels thinned at a predetermined interval. In addition, the determination of the flag value is not limited to the above. For example, when using a correlation calculation in which the correlation value decreases as the similarity increases, the flag value is set to “1” when the correlation value is equal to or less than the threshold Th The flag value may be set to “0” when the correlation value is larger than the threshold Th.

Next, as shown in FIG. 8, when the flag value of the position XL of the blue image Vb is “1”, nothing is done and the pixel value Vb (XL) of the position XL of the blue image Vb is directly used as the correction value Vb ′ ( XL). When the flag value at the position XL of the blue image Vb is 0, pay attention to the section of the width w centering on the position XL of the blue image Vb and the green image Vg, and only the pixel having the flag value “1” in the section is selected. The section fitting process between the blue image Vb and the green image Vg is performed. In the example of FIG. 8, the effective pixel ranges of the target section with the width w are denoted as e1 and e2. The invalid pixel range with the flag value “0” is not used for the interval fitting process.

As the fitting process, for example, when the effective pixel range of the processing section is e1 and e2, the level of the green image Vg is changed, and the total absolute value of the difference between the green image Vg and the blue image Vb at each level is obtained. A method of superimposing such that the sum is minimized can be considered. Alternatively, a method is conceivable in which the gain of the green image Vg is changed, the sum of the absolute values of the differences between the green image Vg and the blue image Vb at each gain is obtained, and the sum is made so that the sum is minimized.

The pixel value Vg (XL) at the position XL of the green image Vg after the fitting process is set as a correction value Vb ′ (XL). The series of correction processes described above is performed at all positions x on the sensor surface (that is, all pixels of the captured image) to generate a corrected blue image Vb ′. Similarly, a corrected red image Vr ′ is generated. These corrected images and the already captured green image Vg are combined to reconstruct an RGB image as an observation image for display.

By performing the correction process as described above, a high-quality observation image can be generated even with illumination light including pattern light.

According to the embodiment described above, the monocular in the non-stereo mode passes the wavelength band including the wavelength bands of the first color (blue), the second color (green), and the third color (red).

In this way, in the non-stereo mode, the first color image and the third color image in which the pattern by the pattern illumination is captured, and the second color image in which the pattern by the pattern illumination is not captured are captured. Then, by using these three color images, it is possible to erase (reduce) the pattern by pattern illumination, and an observation image can be obtained.

In this embodiment, as will be described later with reference to FIG. 18 and the like, the imaging apparatus may include a phase difference detection unit 330 and an image output unit (color image generation unit 320). The phase difference detection unit 330 detects a phase difference between the first color image and the third color image captured in the stereo mode. The image output unit outputs an image for observation based on the captured image (first color to third color image) captured in the non-stereo mode. The image output unit corrects changes in the pixel values of the first color image and the third color image due to a given light amount distribution based on the second color image.

When the pattern illumination of this embodiment is used, the pattern by the pattern illumination is not reflected in the second color image. Accordingly, it is possible to correct the pixel values of the first color image and the third color image in which the pattern by the pattern illumination is captured using the second color image as a reference. In other words, it is considered that the profiles of the first to third color images are almost similar in a normal image (for example, an image of a subject that is normally captured by an industrial endoscope or an image captured of the natural world). . Therefore, the influence of pattern illumination can be corrected by correcting the profiles of the first color image and the third color image so as to be similar to the profile of the second color image in which no pattern is captured.

5). Illumination Unit An illumination unit that performs active pattern illumination according to this embodiment will be described below. In addition, although demonstrated taking the case of apparatuses, such as an endoscope scope in which the emission part of illumination light was separated from the light source part, a light source part is not necessarily limited to the apparatus using a light guide member from a light source part to an output end.

FIG. 9 shows a first configuration example of the illumination unit. 9 includes a white light source 401, a light guide member 402 (light guide member for non-pattern light), an illumination lens 403, a red laser light source 404, a blue laser light source 405, dichroic prisms 406 and 407 (mirrors in a broad sense). ), A light guide member 408 (light guide member for pattern light), a mask pattern 409, and a projection lens 410.

In this first configuration example, the illumination unit has two light guide members 402 and 408. One light guide member 402 guides the white light from the white light source 401 to the illumination lens 403 provided at the distal end of the scope unit. The guided white light is irradiated to the subject through the illumination lens 403. The other light guide member 408 guides blue laser light and red laser light to the projection lens 410 provided at the distal end of the scope portion. That is, the blue laser light from the blue laser light source 405 and the red laser light from the red laser light source 404 are incident on the light guide member 408 by optical path synthesis by the dichroic prisms 406 and 407. The laser light guided by the light guide member 408 passes through the mask pattern 409, and the added pattern is projected onto the subject 5 by the projection lens 410.

In this first configuration example, the non-pattern light and the pattern light are combined on the surface of the subject 5.

FIG. 10 shows a second configuration example of the illumination unit. 10 includes a white light source 451, a polarizing element 452, a blue laser light source 453, a red laser light source 454, dichroic prisms 455 and 456, a mask pattern 457, a polarizing element 458, a prism 459 (synthesis prism), and a light guide member 460. Projection lens 461.

In the second configuration example, the illumination unit has one light guide member 460. White light from the white light source 451 is converted into, for example, P-polarized light (polarized light perpendicular to S-polarized light) by the polarizing element 452, and the polarized white light enters the prism 459. The optical paths of the blue laser light from the blue laser light source 453 and the red laser light from the red laser light source 454 are synthesized by the dichroic prisms 455 and 456. The optical path synthesized laser light passes through the mask pattern 409, whereby a pattern is added. The laser light to which the pattern is added is converted into, for example, S-polarized light (polarized light parallel to the reflecting surface of the prism 459) by the polarizing element 452, and the polarized laser light enters the prism 459. White light that is non-patterned light and laser light that is patterned light are incident on the light guide member 460 by optical path synthesis by the prism 459. The light guide member 460 guides incident light to the projection lens 461 provided at the distal end portion of the scope portion. The guided non-pattern light and pattern light are projected onto the subject 5 by the projection lens 410.

In this second configuration example, pattern illumination is generated in advance before the light guide member 460 is incident, and is combined with non-patterned light and then incident on the light guide member 460. If the light guide member 460 is an optical fiber bundle (image guide) that can transmit the pattern as it is, the pattern of the generated pattern light is transmitted conveniently. When the light guide member 460 is a simple optical fiber bundle (light guide), there is a high possibility that the fiber arrangement is different between the incident end and the output end of the light guide member 460, so that the generated pattern light pattern is transmitted as it is. Can not. However, if it is sufficient that the pattern is a random pattern, the patterns at the incident end and the outgoing end may be appropriately changed.

6). Imaging Unit Hereinafter, a configuration example of an imaging unit capable of switching between the observation mode and the stereo measurement mode will be described.

In an inspection with an endoscopic device, for example, a scope is inserted into the inspection object and a normal image is taken to check for abnormalities. Measure the three-dimensional shape and examine whether further inspection is necessary. Thus, a normal observation image is taken with white light. As a method of achieving both such shooting with white light and stereo measurement, for example, performing stereo shooting with white light can be considered. However, when white light is used in stereo shooting, the image sensor is divided into left and right parts, and the left image and the right image need to be imaged in the respective regions. As a method for forming the left image and the right image in the same area of the image sensor, there is a color phase difference method. However, since the captured image becomes a color shift image, it cannot be used as an observation image.

From the above, in order to capture the left image and the right image in the same area of the image sensor with white light, time division switching (for example, Patent Document 1) is required. However, when the imaging system and the subject move relative to each other, there is a motion blur between the left image and the right image, so that the triangle measurement becomes inaccurate. In particular, motion blur is likely to occur when the camera cannot be fixed to the subject as in an endoscope.

As a technique for performing stereo measurement in a non-time-division manner using a color phase difference, for example, there is Patent Document 2 described above. However, Patent Document 2 applies stereo measurement to autofocus, and it is considered that high-speed switching with an observation image is not assumed. As described above, since there are two filters which are movable parts, it is considered disadvantageous in terms of high-speed switching.

Further, the configuration of Patent Document 2 has a problem in that it is difficult to increase the distance measurement accuracy because it is difficult to increase the distance between the pupils because the single optical path is divided into right and left in the middle. In the endoscope apparatus, since the pan focus is necessary, the diaphragm is small (F value is large). Therefore, the small diaphragm diameter is divided into right and left, and the distance between the pupils tends to be close.

In addition, it is necessary to mechanically move (switch) the shutter and the spectral filter for time division switching, including left and right time division switching in stereo. Since mechanical movements cause mistakes and failures, there is a problem that it is necessary to detect which state (position) of the shutter and the spectral filter is switched, and to repair any errors. When realizing such a detection function, detection and repair are easier when there are fewer types of errors. For example, in the configuration of Patent Document 2, multiple types of errors may occur, such as when both of the two spectral filters are not inserted into the pupil, or when only one of them is inserted into the pupil. Difficult to do.

11 and 12 show a configuration example of the imaging unit of the present embodiment that can solve the above-described problems. 11 and 12 are cross-sectional views (on a plane including the optical axis) of the image pickup unit, and a light amount of an image formed on the image pickup element (or a pixel value of an image picked up by the image pickup element). The relationship of the position x is shown. The position x is a position (coordinates) in a direction perpendicular to the optical axis of the imaging optical system, for example, a pixel position of the image sensor. Actually, it is a two-dimensional coordinate system, but here, a two-dimensional one-dimensional coordinate system in the parallax direction will be described.

The imaging unit includes an imaging optical system 10, a movable mask 30 (first mask), a fixed mask 20 (second mask), an imaging element 40 (imaging sensor, image sensor), and an illumination unit 60 (illumination device). . The imaging optical system 10 is a monocular optical system, and includes, for example, one or a plurality of lenses. Here, the case where the image pickup device 40 has RGB color filters of the Bayer array will be described as an example. However, the present invention is not limited to this, and may include, for example, a complementary color filter.

As shown in FIGS. 11 and 12, the reflected light from the subject 5 is imaged on the surface of the image sensor 40 by the imaging optical system 10. At this time, the fixed mask 20 divides the pupil center and the left and right pupils, and the movable mask 30 switches between the image formation by the pupil center and the image formation by the left and right pupils. These are imaged in the same area of the image sensor 40. d is the distance between the center line IC1 of the left pupil (the left eye aperture of the fixed mask 20) and the center line IC2 of the right pupil (the right eye aperture of the fixed mask 20), and the baseline length in stereo measurement It becomes. The straight line AXC is the optical axis of the imaging optical system 10. The center lines IC1 and IC2 are provided at an equal distance from the optical axis AXC of the single-lens imaging optical system 10, for example. The center lines IC1 and IC2 and the optical axis AXC are preferably in the same plane, but are not necessarily in the same plane.

The fixed mask 20 and the movable mask 30 are provided at the pupil position of the imaging optical system 10, for example. Alternatively, it may be provided on the imaging side with respect to the imaging optical system 10. The fixed mask 20 is fixed with respect to the imaging optical system 10, and the movable mask 30 is configured such that the position can be switched in a plane perpendicular to the optical axis AXC. The movable mask 30 has an observation mode (first mode, non-stereo mode, monocular mode) which is the first state shown in FIG. 11 and a stereo measurement mode (second mode) which is the second state shown in FIG. , Stereo mode), which can be switched at high speed.

The fixed mask 20 includes a plate-shaped light shielding portion (light shielding member) provided with three apertures (left eye aperture, right eye aperture, and central aperture), and a short wavelength ( Blue) spectral filter and a long wavelength (red) spectral filter provided in the right eye aperture. The portions other than the aperture hole are covered with a light shielding portion so that light does not pass through. The central aperture may be, for example, a through hole, or some spectral filter (for example, a broadband spectral filter that transmits at least white light) may be provided.

The movable mask 30 includes a plate-shaped light shielding portion (light shielding member) provided with three aperture holes. In each mode, the movable mask 30 is configured in such a size that the light blocking portion covers the central aperture hole or the left and right eye aperture holes among the three aperture holes of the fixed mask 20. The aperture is provided at a position overlapping the central aperture of the fixed mask 20 in the observation mode and at a position overlapping the left eye aperture and the right eye aperture in the stereo measurement mode. Hereinafter, for convenience, the movable mask 30 is also referred to as a left eye aperture, a right eye aperture, and a center aperture. 11 and 12 illustrate the case where the movable mask 30 is provided on the imaging side with respect to the fixed mask 20, the movable mask 30 may be provided on the objective side with respect to the fixed mask 20.

The illumination unit 60 is preferably provided so that the tip (illumination exit end) is positioned symmetrically with respect to the left pupil and the right pupil. However, the negative of the illumination unit 60 is not necessarily limited to the left pupil and the right pupil. May not be symmetrical. 11 and 12, the tip of the illumination unit 60 is disposed in front of the imaging optical system 10. However, the present invention is not limited to this. For example, the illumination unit 60 and the imaging optical system 10 at the tip of the imaging unit. May be arranged side by side.

Hereinafter, the spectral characteristics of the left eye aperture, the right eye aperture, and the center aperture of the fixed mask 20 are denoted as FL, FR, and FC. In addition, for the sake of easy understanding, the spectral filters provided in the respective apertures are also denoted by the same symbols FL, FR, and FC. In addition, each diaphragm hole of the movable mask 30 is not provided with a spectral filter (is an open hole), and allows the entire band to pass.

FIG. 11 shows the state of the observation mode, where the optical path at the center of the pupil is opened through the central aperture of the fixed mask 20 and the central aperture of the movable mask, and the optical path of the left and right pupils is blocked by the movable mask 30. (Light shielding). In this case, the image formed on the image sensor 40 is a formed image IC formed only by the pupil center, and a normal (monocular white light) captured image is obtained.

On the other hand, FIG. 12 shows a state of the stereo measurement mode, in which the left eye diaphragm hole of the fixed mask 20 and the left eye diaphragm hole of the movable mask 30 are overlapped, and the right eye diaphragm hole of the fixed mask 20 and the movable mask 30 are overlapped. The right eye aperture hole is overlapped. The optical path at the center of the pupil is blocked (shielded) by the movable mask 30. That is, in the optical path on the left pupil side, the imaging light is filtered by the short wavelength (blue) spectral filter FL (first filter), and an image IL based on the short wavelength component is formed on the image sensor 40. In the optical path on the right pupil side, the imaging light is filtered by a long wavelength (red) spectral filter FR (second filter), and an image IR based on the long wavelength component is formed on the same image sensor 40.

Therefore, in the stereo measurement mode, the image IL obtained from the blue pixels of the image sensor 40 is a short wavelength image, the image IR obtained from the red pixels of the image sensor 40 is a long wavelength image, and the images IL and IR from the two optical paths are obtained. Can be obtained separately. That is, in the stereo measurement mode, the left eye image IL and the right eye image IR having a phase difference can be obtained simultaneously and independently, and stereo measurement using the phase difference image becomes possible. In addition, the imaging unit of the present embodiment can take the observation mode that is the first state and the stereo measurement mode that is the second state, and can switch between these states at high speed. Thereby, for example, 3D measurement can be performed in real time while performing non-stereo normal observation.

7). First Detailed Configuration Example of Fixed Mask and Movable Mask FIGS. 13 and 14 show a first detailed configuration example of the fixed mask 20 and the movable mask 30. 13 and 14 are cross-sectional views of the imaging optical system 10, the fixed mask 20, and the movable mask 30, and a view of the fixed mask 20 and the movable mask 30 in the optical axis direction (a rear view viewed from the imaging side). ).

A diaphragm hole 21 having a short wavelength filter FL is opened in the optical path of the left pupil of the fixed mask 20, and a diaphragm hole 22 having a long wavelength spectral filter FR is formed in the optical path of the right pupil. The optical path is provided with an aperture (through hole) aperture 23. Note that the aperture 23 may be provided with a spectral filter FC that allows the bands Pv1, Pv2, and Pir of FIG. 3 to pass therethrough. The aperture holes 21 and 22 are opened in the light shielding portion 24 (light shielding member), and are, for example, holes having a size corresponding to the depth of field necessary for the imaging system (for example, circular holes, the size is a diameter). The centers of the aperture holes 21, 22, and 23 (for example, the center of a circle) coincide with (including substantially coincident with) the center lines IC1 and IC2 and the optical axis AXC, respectively. The light shielding unit 24 is provided so as to close the housing containing the imaging optical system 10 when viewed from the front (or the back), and is provided, for example, perpendicular to the optical axis AXC. It is a plate-like member.

The movable mask 30 has aperture holes 31, 32, 33 in an open state (through hole) and a light shielding portion 34 (light shielding member) in which the aperture holes 31, 32, 33 are opened. The aperture holes 31, 32, and 33 are holes that are slightly larger than the aperture holes 21, 22, and 23 of the fixed mask 20, for example. Alternatively, it may be a hole having a size corresponding to the depth of field necessary for the imaging system (for example, a circular hole having a diameter). The center of the aperture 33 (for example, the center of a circle) coincides with (including substantially coincides with) the optical axis AXC in the observation mode. The light shielding unit 34 is connected to a rotation shaft 35 perpendicular to the optical axis AXC, and is a plate-like member provided perpendicular to the optical axis AXC, for example. The shape of the light shielding part 34 is, for example, a fan shape (the fan base is connected to the shaft 35), but is not limited thereto, and may be any shape that can realize the states of FIGS.

The movable mask 30 is configured to rotate about a rotation axis 35 by a predetermined angle in a direction perpendicular to the optical axis AXC. For example, rotational movement can be realized by a piezo element or a motor. In the observation mode of FIG. 13, the movable mask 30 is rotated and tilted to the right eye by a predetermined angle, the pupil center optical path (aperture hole 23) of the fixed mask 20 is opened, and the left and right pupil optical paths (aperture holes 21, 22) is in a light shielding state. In the stereo measurement mode of FIG. 14, the movable mask 30 rotates and tilts to the left eye side by a predetermined angle, the pupil center optical path (aperture hole 23) of the fixed mask 20 enters a light-shielded state, and the left and right pupil optical paths (aperture hole 21). 22) is in an open state. By exposing the aperture 21 having the spectral filter FL, the left pupil passes only the short wavelength component, and by exposing the aperture 22 having the spectral filter FR, the right pupil passes only the long wavelength component.

In the above description, the case where two states are formed by rotating the movable mask 30 by a predetermined angle has been described. However, the present invention is not limited to this. For example, the movable mask 30 may be moved by a sliding operation to create two states. The rotation operation or the slide operation can be realized by, for example, a magnet mechanism or a piezoelectric mechanism, and an appropriate one may be selected in consideration of high speed and durability.

According to the above embodiment, the imaging device (endoscope device) includes the imaging element 40, the imaging optical system 10, the fixed mask 20, and the movable mask 30. The imaging optical system 10 forms an image of the subject 5 on the image sensor 40. The fixed mask 20 includes first to third apertures (diaphragm holes 21, 22, and 23) that divide the pupil of the imaging optical system 10, and a first wavelength band (Pb in FIG. 3 or Pb1 in FIG. 5). A first filter FL that passes therethrough and a second filter FR that passes the second wavelength band (Pr in FIG. 3 or Pr2 in FIG. 5) are included. The movable mask 30 includes a light shielding portion 34 and fourth to sixth openings (throttle holes 31, 32) provided in the light shielding portion 34 corresponding to the first to third openings (throttle holes 21, 22, 23). 33) and is movable with respect to the imaging optical system 10. The first filter FL is provided in the first opening (throttle hole 21). The second filter FR is provided in the second opening (throttle hole 22). The third opening (aperture hole 23) is provided on the optical axis AXC of the imaging optical system 10.

With this configuration, it is possible to switch between the observation mode and the stereo measurement mode as described with reference to FIGS. Moreover, since the parallax images in the color phase difference method can be acquired simultaneously (not in time division), accurate stereo measurement can be performed. In addition, since there is one movable mask 30 that is a movable part, it is possible to realize high-speed switching, simplification of the driving mechanism, and suppression of failures and errors in mode switching. In addition, the movable mask 30 has a simple configuration in which openings (diaphragm holes 31, 32, 33) are provided in the light-shielding portion 34, and troubles such as filter removal due to switching vibration can be suppressed. In addition, since the left and right pupils are clearly separated by the opening of the fixed mask 20 (diaphragm holes 21 and 22), it is easy to increase the baseline length (d in FIGS. 11 and 17) in stereo measurement, and accurate distance measurement is possible. Become.

Also, for example, when considering a comparative example in which an observation image of a single eye is photographed using one of the left and right pupils, photographing is performed with a pupil off the optical axis. In this respect, in this embodiment, the fixed mask 20 is provided with three openings (diaphragm holes 21, 22, and 23), and one of them is provided on the optical axis AXC, so that the observation image becomes a pupil center image. . Thereby, the vignetting of the light beam is reduced and an observation image having a wide viewing angle can be acquired. Further, high quality (for example, less distortion) imaging can be obtained.

Also, when considering the correspondence between the position on the subject 5 and the pixel position, the center (s / 2 position) of the phase difference (s in FIG. 12) in stereo measurement coincides with the light beam passing through the center of the pupil. That is, in the present embodiment, the same pixel in the observation image and the distance map corresponds to the same position on the subject 5. On the other hand, in the above comparative example, the observation image has a parallax on the left and is not the center of the pupil, so that the pixel on the subject image 5 and the distance map corresponding to the same position on the subject 5 corresponds. The present embodiment is more advantageous when, for example, displaying an observation image and three-dimensional information in an overlapping manner.

In the present embodiment, the first aperture (diaphragm aperture 21) corresponds to the left pupil, the second aperture (diaphragm aperture 22) corresponds to the right pupil, and the third aperture (diaphragm aperture 23) is at the pupil center. Correspond. Note that the first opening may correspond to the right pupil, and the second opening may correspond to the left pupil. Moreover, although the pupil in stereo measurement is separated into right and left for convenience, the separating direction of the pupil is not limited to right and left. In the present embodiment, the aperture is referred to as a diaphragm aperture. However, the aperture does not necessarily have a function as a diaphragm (a function of limiting the cross-sectional area of the light beam passing through the pupil). For example, the apertures 23 and 33 overlap in the observation mode, but when the aperture 23 is smaller, the aperture 23 has a function of aperture, and when the aperture 33 is smaller, the aperture 33 is the aperture. It has the function of.

Here, the pupil is for separating (or defining) the imaging optical path by the imaging optical system 10. The optical path is a path from the light that forms an image on the image sensor 40 to the image sensor 40 after entering from the objective side of the optical system. That is, the optical paths that pass through the imaging optical system 10 and the apertures 21 and 22 of the fixed mask 20 (the apertures 31 and 32 of the movable mask 30 in the stereo measurement mode) are the first and second optical paths. The optical path that passes through the imaging optical system 10 and the aperture 23 of the fixed mask 20 (or the aperture 33 of the movable mask 30 in the observation mode) is the third optical path.

Here, the mask is a member or component that shields light incident on the mask and allows part of the light to pass. In the fixed mask 20 and the movable mask 30 of the present embodiment, the light shielding portions 24 and 34 shield light, and the aperture holes 21, 22, 23, 31, 32, and 33 emit light (full band or partial band). Let it pass.

In the present embodiment, the imaging apparatus includes a movable mask control unit 340 (FIG. 18) that controls the movable mask 30. When the movable mask control unit 340 is viewed in the optical axis AXC direction in the non-stereo mode (observation mode), the light shielding unit 34 overlaps the first and second openings (diaphragm holes 21 and 22) and the sixth opening. The movable mask 30 is set in a first state (first position) where the (aperture hole 33) overlaps with the third opening (aperture hole 23). On the other hand, in the stereo mode (stereo measurement mode), when viewed in the direction of the optical axis AXC, the fourth and fifth apertures (diaphragm holes 31 and 32) become the first and second apertures (diaphragm apertures 21 and 22). The movable mask 30 is set in a second state (second position) where the light shielding portion 34 overlaps with the third opening (aperture hole 23).

By controlling the driving of the movable mask 30 as described above, switching control between the observation mode shown in FIGS. 11 and 13 and the stereo measurement mode shown in FIGS. 12 and 14 can be realized. In other words, when the movable mask 30 is set to the first state, the first and second openings are shielded by the light-shielding portion 34, so that only the third opening is photographed. Since no filter is inserted, an image for normal observation (white light image) can be taken. On the other hand, when the movable mask 30 is set to the second state, the first filter FL is fixed to the first opening and the second filter FR is fixed to the second opening. It is possible to take a parallax image in the phase difference method.

8). Second Detailed Configuration Example of Fixed Mask and Movable Mask FIGS. 15 and 16 show a second detailed configuration example of the fixed mask 20 and the movable mask 30. 15 and 16 are cross-sectional views of the imaging optical system 10, the fixed mask 20, and the movable mask 30, and a view of the fixed mask 20 and the movable mask 30 in the optical axis direction (a rear view viewed from the imaging side). ).

The movable mask 30 includes a light shielding part 34 and aperture holes 31 and 32 provided in the light shielding part 34. The aperture holes 31 and 32 are in an open state (through hole), and are arranged on the same circle around the rotation shaft 35. The aperture hole 31 has a shape extending in the circumferential direction of the same circle, and overlaps the aperture hole 23 of the fixed mask 20 in the observation mode and also overlaps the aperture hole 21 of the fixed mask 20 in the stereo measurement mode. ing.

The fixed mask 20 includes a light shielding part 24 and three aperture holes 21, 22, 23 provided in the light shielding part 24. The aperture holes 21 and 22 are provided with spectral filters FL and FR. The aperture hole 23 may be in an open state (through hole), or a spectral filter FC that allows the bands Pv1, Pv2, and Pir of FIG. 3 to pass therethrough may be provided. The aperture holes 21, 22, and 23 are arranged on the same circle around the rotation shaft 35.

In the observation mode, the aperture hole 23 at the center of the pupil of the fixed mask 20 is opened by the aperture hole 31 of the movable mask 30, and the aperture holes 21 and 22 of the left and right pupils of the fixed mask 20 are shielded by the light shielding portion 34 of the movable mask 30. A white light image is captured by a single eye. In the stereo measurement mode, the left and right pupil apertures 21 and 22 of the fixed mask 20 are opened by the aperture holes 31 and 32 of the movable mask 30, and the aperture hole 23 at the center of the pupil of the fixed mask 20 is the light shielding portion 34 of the movable mask 30. And a parallax image (red image, blue image) by the color phase difference method is captured.

According to the above embodiment, the imaging device (endoscope device) includes the imaging element 40, the imaging optical system 10, the fixed mask 20, and the movable mask 30. The imaging optical system 10 forms an image of the subject 5 on the image sensor 40. The fixed mask 20 includes first to third apertures (diaphragm holes 21, 22, and 23) that divide the pupil of the imaging optical system 10, and a first wavelength band (Pb in FIG. 3 or Pb1 in FIG. 5). A first filter FL that passes therethrough and a second filter FR that passes the second wavelength band (Pr in FIG. 3 or Pr2 in FIG. 5) are included. The movable mask 30 includes a light shielding part 34, a fourth opening (aperture hole 31) provided in the light shielding part 34 corresponding to the first and third openings (throttle holes 21 and 23), and a second opening. A fifth aperture (diaphragm hole 32) provided in the light shielding portion 34 corresponding to the (diaphragm hole 22) is provided, and is movable with respect to the imaging optical system 10. The first filter FL is provided in the first opening (throttle hole 21). The second filter FR is provided in the second opening (throttle hole 22). The third opening (aperture hole 23) is provided on the optical axis AXC of the imaging optical system 10.

Specifically, the imaging apparatus includes a movable mask control unit 340 that controls the movable mask 30. When the movable mask control unit 340 is viewed in the optical axis AXC direction in the non-stereo mode (observation mode), the light shielding unit 34 overlaps the first and second openings (diaphragm holes 21 and 22) and the fourth opening. The movable mask 30 is set in a first state where the (aperture hole 31) overlaps the third opening (aperture hole 23). On the other hand, in the stereo mode (stereo measurement mode), when viewed in the direction of the optical axis AXC, the fourth and fifth apertures (diaphragm holes 31 and 32) become the first and second apertures (diaphragm apertures 21 and 22). The movable mask 30 is set in a second state in which the light shielding portion 34 overlaps with the third opening (aperture hole 23).

Even with such a configuration, switching between the observation mode and the stereo measurement mode, simultaneous acquisition of parallax images in the stereo measurement mode, speeding up of mode switching, simplification of the driving mechanism of the movable mask 30, failure or error in mode switching Suppression, securing of baseline length in stereo measurement, etc. can be realized.

9. Principle of Stereo 3D Measurement The principle of stereo measurement in the stereo measurement mode will be described. As shown in FIG. 17, the optical paths of the left eye and the right eye are configured independently, and the reflected image from the subject 5 forms an image on the imaging sensor surface (light receiving surface) through these optical paths. A coordinate system X, Y, and Z of the three-dimensional space is defined as follows. That is, the X axis and the Y axis orthogonal to the X axis are set along the imaging sensor surface, and the Z axis is set in a direction toward the subject in a direction orthogonal to the imaging sensor surface and parallel to the optical axis AXC. The Z axis intersects the X axis and Y axis at the zero point. Note that the Y-axis is omitted here for convenience.

The distance between the imaging lens 10 and the imaging sensor surface is b, and the distance from the imaging lens 10 to the arbitrary point Q (x, z) of the subject 5 is z. The distance between the pupil center lines IC1 and IC2 and the Z axis is the same, and each is d / 2. That is, the baseline length in stereo measurement is d. The X coordinate of the corresponding point at which the arbitrary point Q (x, y) of the subject 5 is imaged on the imaging sensor surface by the imaging lens 10 is XL, and the arbitrary point Q (x, y) of the subject 5 is the imaging lens 10. The X coordinate of the corresponding point imaged on the imaging sensor surface is defined as XR. The following equation (4) can be obtained by using a similarity relationship between a plurality of partial right-angled triangles formed in the triangle surrounded by the arbitrary point Q (x, z) and the coordinates XL and XR.

Here, the following expressions (5) and (6) hold.

Thereby, the absolute value of the above equation (4) can be removed as in the following equation (7).

When the above equation (7) is solved for x, the following equation (8) is obtained.

Substituting x in the above equation (8) into the above equation (7), the following equation (9) is obtained, and z can be obtained.

D and b are known set values, and the unknowns XL and XR are obtained as follows. That is, the position XL on the imaging sensor surface is considered as a reference (the pixel position of the left image is regarded as XL), and the position XR corresponding to the position XL is detected by matching processing (correlation calculation). By calculating the distance z for each position XL, the shape of the subject can be measured. If the matching is not good, the distance z may not be obtained, but may be obtained by interpolation from the distance z of surrounding pixels, for example.

10. Endoscopic Device FIG. 18 shows a configuration example of the endoscope device (imaging device in a broad sense) of the present embodiment. The endoscope apparatus includes a scope unit 100 (imaging unit) and a main body unit 200 (control device). The scope unit 100 includes an imaging optical system 10, a fixed mask 20, a movable mask 30, an image sensor 40, a drive unit 50, and an illumination unit 60. The main body unit 200 includes a processing unit 210, a monitor display unit 220, and an imaging processing unit 230. The processing unit 210 includes a light source drive control unit 305, an image selection unit 310 (image frame selection unit), a color image generation unit 320 (image output unit), a phase difference detection unit 330, and a movable mask control unit 340 (movable mask drive control unit). ), A movable mask position detector 350, a distance information calculator 360, and a three-dimensional information generator 370.

The main body unit 200 may include an operation unit that operates the main body unit 200, an interface unit that is connected to an external device, and the like as components (not illustrated). The scope unit 100 may include, for example, an operation unit that operates the scope unit 100, a treatment instrument, and the like as components not shown.

As the endoscope apparatus, an industrial and medical so-called video scope (an endoscope apparatus incorporating an image sensor) can be assumed. The present invention can be applied to both a flexible mirror in which the scope unit 100 is configured to be bendable and a rigid mirror in which the scope unit 100 is configured in a stick shape. For example, in the case of an industrial flexible mirror, the main body 200 and the imaging unit 110 are configured as portable devices that can be carried, and are used for manufacturing inspection and maintenance inspection of industrial products, maintenance inspection of buildings and piping, and the like.

The driving unit 50 drives the movable mask 30 based on a control signal from the movable mask control unit 340, and switches between the first state (observation mode) and the second state (stereo measurement mode). For example, the drive unit 50 is configured by an actuator using a piezoelectric element or a magnet mechanism.

The imaging processing unit 230 performs imaging processing on the signal from the imaging element 40 and outputs a captured image (for example, a Bayer image). For example, correlated double sampling processing, gain control processing, A / D conversion processing, gamma correction, color correction, noise reduction, and the like are performed. The imaging processing unit 230 may be configured by, for example, a discrete IC such as an ASIC, or may be incorporated in the imaging device 40 (sensor chip) or the processing unit 210.

The monitor display unit 220 displays an image captured by the scope unit 100, 3D shape information of the subject 5, and the like. For example, the monitor display unit 220 includes a liquid crystal display, an EL (Electro-Luminescence) display, or the like.

Hereinafter, the operation of the endoscope apparatus will be described. The illumination unit 60 irradiates the subject 5 with the combined light of the non-pattern light and the pattern light described above. The light source drive control unit 305 optimally controls each light amount of the non-pattern light and the pattern light based on a signal from the imaging processing unit 230 (so-called dimming control). For example, the brightness of the captured image is obtained, and the amount of light is controlled so that the brightness is within a predetermined range.

The movable mask control unit 340 controls the driving unit 50 to switch the position of the movable mask 30. When the movable mask control unit 340 sets the movable mask 30 to the observation mode, the reflected light from the subject 5 is imaged on the image sensor 40 via the pupil center optical path. The imaging processing unit 230 reads the pixel value of the image formed on the imaging element 40, performs A / D conversion or the like, and outputs the image data to the image selection unit 310.

The image selection unit 310 detects that the movable mask 30 is in the observation mode based on the control signal from the movable mask control unit 340, and selects {Vr, Vg, Vb} from the captured image to generate a color image. To the unit 320. The color image generation unit 320 performs demosaicing processing (processing for generating an RGB image from a Bayer image) and various types of image processing, and outputs a three-plate RGB primary color image to the monitor display unit 220. The monitor display unit 220 displays the color image.

When the movable mask control unit 340 sets the movable mask 30 to the stereo measurement mode, the reflected light from the subject 5 is simultaneously imaged on the image sensor 40 via the left pupil optical path and the right pupil optical path. The imaging processing unit 230 reads the pixel value of the image formed on the imaging element 40, performs A / D conversion or the like, and outputs the image data to the image selection unit 310.

The image selection unit 310 detects that the movable mask 30 is in the stereo measurement mode based on a control signal from the movable mask control unit 340, selects {Mr, Mb} from the captured image, and detects the phase difference. To 330. The phase difference detection unit 330 performs matching processing on the two separated images Mr and Mb, and detects a phase difference (phase shift) for each pixel. Further, the phase difference detection unit 330 determines whether or not the phase difference detection is reliable. If it is determined that the phase difference detection is not reliable, an error flag is output for each pixel. Conventionally, a matching evaluation method for obtaining a shift amount (phase difference) between two similar waveforms is based on a normalized cross-correlation calculation method represented by ZNCC (Zero-meanＣＣNormalized Cross-Correlation), and a sum of absolute values of mutual differences. Various proposals such as SAD (Sum of Absolute Difference) have been proposed and can be used as appropriate.

It should be noted that the phase shift (phase difference) can be detected by using Vr and Mr, which are parallax images, although they are time-divisionally affected by subject blur and imaging system blur. When the reflection of the subject 5 has a small blue component and a large red component, even if the subject 5 is difficult to detect with Mr and Mb, measurement is possible if both Vr and Mr have a red component.

The phase difference detection unit 330 outputs the detected phase difference information and error flag to the distance information calculation unit 360. The distance information calculation unit 360 calculates the distance information of the subject 5 (for example, the distance z in FIG. 17) for each pixel, and outputs the distance information to the three-dimensional information generation unit 370. For example, the pixel on which the error flag is set may be regarded as a flat portion (region having a small edge component) of the subject 5 and may be interpolated from distance information of surrounding pixels, for example. The three-dimensional information generation unit 370 generates three-dimensional information from the distance information (or the distance information and the RGB image from the color image generation unit 320). As the three-dimensional information, various information such as a Z value map (distance map), a polygon, and a pseudo three-dimensional display image (for example, shape enhancement by shading or the like) can be assumed. The three-dimensional information generation unit 370 generates the generated three-dimensional image, the three-dimensional data, or a display image in which these and the observation image are superimposed as necessary, and outputs the generated image to the monitor display unit 220. The monitor display unit 220 displays the three-dimensional information.

The movable mask position detector 350 detects whether the movable mask 30 is in the observation mode position or the stereo measurement mode position using the image {Mr, Mb} obtained in the stereo measurement mode. If it is determined that the state of the movable mask 30 does not match the mode, a position error flag is output to the movable mask control unit 340. The movable mask control unit 340 receives the position error flag and corrects the movable mask 30 to a correct state (a state corresponding to image selection). For example, when it is determined that there is no color shift in the image {Mr, Mb} even though the movable mask control unit 340 outputs a control signal for the stereo measurement mode, the actual movable mask 30 is positioned in the observation mode. It has become. In this case, correction is performed to match the position of the control signal and the movable mask 30. If the correct state is not obtained even if the correction operation is performed, it is determined that some failure has occurred, and the entire function is stopped.

Since the movable mask 30 is composed of a mechanical mechanism, it is conceivable that a malfunction occurs in the switching operation. According to the present embodiment, since it is possible to detect whether the switching position is the observation mode or the measurement mode, it is possible to cope with a problem of the switching operation.

Detecting or judging whether the movable mask 30 is in the observation mode position or the stereo measurement mode position is performed as follows, for example. That is, after matching the levels (average level, etc.) in the judgment areas of the images Mr and Mb, judgment based on the sum of absolute difference values of the images Mr and Mb (first method) and the correlation between the images Mr and Mb The position error is determined by determination based on the number (second method) or the like.

In the first method, the absolute value of the difference value of the pixel value is obtained for each pixel, and it is integrated in all pixels or a partial pixel group. If the result exceeds a predetermined threshold, it is determined as an image in the stereo measurement mode, and if the result is less than the predetermined threshold, it is determined as an image in the observation mode. In the stereo measurement mode, the image Mr and the image Mb are basically images that have undergone color misregistration, so that the fact that a predetermined amount of difference value is obtained is used.

In the second method, the correlation coefficient in the predetermined range between the image Mr and the image Mb is calculated, and when the result is equal to or smaller than the predetermined threshold, it is determined as an image in the stereo measurement mode, and the result exceeds the predetermined threshold. In this case, it is determined that the image is an observation mode. In the stereo measurement mode, the image Mr and the image Mb are basically images that have undergone color misregistration, so the correlation coefficient is small, whereas in the observation mode, the image Mr and the image Mb are almost the same image, so the correlation coefficient is Take advantage of big things.

Note that the endoscope apparatus, the imaging apparatus, and the like of the present embodiment may include a processor and a memory. The processor here may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (GraphicsＧProcessing Unit) or a DSP (Digital Signal Processor) can be used. The processor may be an ASIC hardware circuit. The memory stores instructions that can be read by a computer. When the instructions are executed by the processor, each unit (for example, each unit of the processing unit 210) of the endoscope apparatus, the imaging apparatus, and the like according to the present embodiment. Etc.) will be realized. The memory here may be a semiconductor memory such as SRAM or DRAM, or a register or a hard disk. Further, the instruction here may be an instruction of an instruction set constituting the program, or an instruction for instructing an operation to the hardware circuit of the processor.

11. Mode Switching Sequence FIG. 19 shows a sequence (operation timing chart) for switching between the observation mode and the stereo measurement mode in moving image shooting.

In the stereo measurement mode described above, high-precision simultaneous measurement can be realized even for a moving subject. However, since it becomes a color shift image, it cannot be used for a high-quality observation image. Therefore, this problem can be solved by switching between the observation mode and the stereo measurement mode at high speed, and stereo measurement can be performed while displaying the observation image in a state almost in real time.

As shown in FIG. 19, switching of the state of the movable mask 30, imaging timing, and selection of a captured image are interlocked. As shown in A1 and A2, the mask state in the observation mode and the mask state in the stereo measurement mode are alternately repeated. As shown in A3 and A4, imaging is performed once in each mask state. As shown in A5, an image that is exposed and imaged by the image sensor 40 when in the mask state of the observation mode is selected as an observation image. As shown in A6, an image that is exposed and imaged by the image sensor 40 when in the mask state of the stereo measurement mode is selected as a measurement image.

By alternately repeating the observation mode and the stereo measurement mode in this way, an observation image and a measurement image can be obtained continuously in a state almost in real time, so both observation and measurement can be performed even when the subject 5 is moving. Can be realized. While displaying the image in the observation mode, if the information measured as necessary is superimposed and displayed there, visual inspection and quantitative inspection can be provided to the user at the same time, providing useful information It becomes possible.

As mentioned above, although embodiment and its modification which applied this invention were described, this invention is not limited to each embodiment and its modification as it is, and in the range which does not deviate from the summary of invention in an implementation stage. The component can be modified and embodied. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments and modifications. For example, some constituent elements may be deleted from all the constituent elements described in each embodiment or modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention. In addition, a term described together with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term anywhere in the specification or the drawings.

5 subject, 10 imaging optical system, 20 fixed mask,
21, 22, 23 Opening (aperture hole), 24 light shielding part, 30 movable mask,
31, 32, 33 Aperture (diaphragm hole), 34 Shading part, 35 Rotating shaft
40 image sensor, 50 drive unit, 60 illumination unit, 100 scope unit,
110 imaging unit, 200 main body unit, 210 processing unit, 220 monitor display unit,
230 imaging processing unit, 305 light source drive control unit, 310 image selection unit,
320 color image generation unit, 330 phase difference detection unit,
340 movable mask control unit, 350 movable mask position detection unit,
360 distance information calculation unit, 370 three-dimensional information generation unit, 401 white light source,
402 light guide member, 403 illumination lens, 404 red laser light source,
405 Blue laser light source, 406, 407 dichroic prism,
408 light guide member, 409 mask pattern, 410 projection lens,
451 white light source, 452 polarizing element, 453 blue laser light source,
454 red laser light source, 455,456 dichroic prism,
457 mask pattern, 458 polarizing element, 458 prism,
460 light guide member, 461 projection lens,
FL first filter, FR second filter, Mb, Mr image,
Pb, Pb1 first wavelength band, Pr, Pr2 second wavelength band,
s (XL) phase difference, Vb, Vg, Vr image

Claims (14)

Capturing a captured image including a first color image, a second color image having a longer wavelength than the first color, and a third color image having a longer wavelength than the second color; An imaging unit capable of imaging one color image and the third color image as a stereo image;
A first wavelength band that is included in the wavelength band of the first color and not included in the wavelength band of the second color, and that is included in the wavelength band of the third color and included in the wavelength band of the second color. An illumination unit that irradiates a subject with pattern illumination having a given light amount distribution in a second wavelength band that is not present;
An imaging apparatus comprising: In claim 1,
The imaging unit
An imaging apparatus characterized by switching between a stereo mode for capturing the stereo image and a non-stereo mode for capturing the captured image with a single eye. In claim 2,
The monocular in the non-stereo mode allows a wavelength band excluding the first wavelength band and the second wavelength band among the wavelength bands of the first color, the second color, and the third color to pass. An imaging device that is characterized. In claim 3,
A phase difference detector that detects a phase difference between the first color image and the third color image captured in the stereo mode;
An image output unit that outputs an image for observation based on the captured image captured in the non-stereo mode;
An imaging apparatus comprising: In claim 2,
The imaging apparatus according to claim 1, wherein the monocular in the non-stereo mode passes a wavelength band including the wavelength bands of the first color, the second color, and the third color. In claim 5,
A phase difference detector that detects a phase difference between the first color image and the third color image captured in the stereo mode;
An image output unit that outputs an image for observation based on the captured image captured in the non-stereo mode;
Including
The image output unit includes:
An imaging apparatus, wherein a change in pixel values of the first color image and the third color image due to the given light amount distribution is corrected based on the second color image. In any one of Claims 1 thru | or 6.
The pattern illumination includes a wavelength band excluding the first wavelength band among the wavelength bands of the first color, a wavelength band of the second color, and a second wavelength band of the wavelength bands of the third color. An image pickup apparatus having a flat light amount distribution in a wavelength band excluding. In any one of Claims 1 thru | or 7,
The imaging apparatus according to claim 1, wherein the first color is blue, the second color is green, and the third color is red. In any one of Claims 1 thru | or 8.
An image sensor;
An imaging optical system for imaging a subject on the image sensor;
First to third apertures that divide the pupil of the imaging optical system, a first filter that passes the first wavelength band, and a second filter that passes the second wavelength band A fixed mask,
A movable mask having a light shielding part and fourth to sixth openings provided in the light shielding part corresponding to the first to third openings, and movable with respect to the imaging optical system;
Including
The first filter is provided in the first opening;
The second filter is provided in the second opening;
The image pickup apparatus, wherein the third opening is provided on an optical axis of the imaging optical system. In claim 9,
A movable mask control unit for controlling the movable mask;
The movable mask controller is
In the non-stereo mode, when viewed in the optical axis direction, the movable portion is in a first state in which the light shielding portion overlaps the first and second openings and the sixth opening overlaps the third opening. Set the mask,
In the stereo mode, when viewed in the optical axis direction, the fourth and fifth openings overlap the first and second openings, and the light shielding portion overlaps the third opening. An imaging apparatus, wherein the movable mask is set. In any one of Claims 1 thru | or 8.
An image sensor;
An imaging optical system for imaging a subject on the image sensor;
First to third apertures that divide the pupil of the imaging optical system, a first filter that passes the first wavelength band, and a second filter that passes the second wavelength band A fixed mask,
A light shielding portion, a fourth opening provided in the light shielding portion corresponding to the first and third openings, and a fifth opening provided in the light shielding portion corresponding to the second opening. And a movable mask movable with respect to the imaging optical system,
Including
The first filter is provided in the first opening;
The second filter is provided in the second opening;
The image pickup apparatus, wherein the third opening is provided on an optical axis of the imaging optical system. In claim 11,
A movable mask control unit for controlling the movable mask;
The movable mask controller is
In the non-stereo mode, when viewed in the optical axis direction, the movable portion is in a first state in which the light shielding portion overlaps the first and second openings and the fourth opening overlaps the third opening. Set the mask,
In the stereo mode, when viewed in the optical axis direction, the fourth and fifth openings overlap the first and second openings, and the light shielding portion overlaps the third opening. An imaging apparatus, wherein the movable mask is set. An endoscope apparatus comprising the imaging apparatus according to any one of claims 1 to 12. Of the first color image, the second color image longer than the first color, and the third color image longer than the second color, the first color image and the When a third color image can be captured as a stereo image,
A first wavelength band that is included in the wavelength band of the first color and not included in the wavelength band of the second color, and that is included in the wavelength band of the third color and included in the wavelength band of the second color. Irradiating the subject with pattern illumination having a given light intensity distribution in the second wavelength band not present,
An imaging method, comprising: capturing a captured image including the first color image, the second color image, and the third color image.

Images