JP6260512B2 - Imaging apparatus, imaging method, and imaging program - Google Patents

Description

  The present invention relates to an imaging technique.

  Conventionally, in order to image a subject in an environment where there is almost no visible light such as at night, a method of irradiating the subject with infrared light using an infrared projector and imaging infrared light reflected from the subject has been used. This method is an effective imaging method when a light that emits visible light cannot be used.

  However, an image obtained by imaging a subject by this imaging method is a monochrome image. In monochrome images, it may be difficult to identify an object. If a color image can be captured even in an environment where there is no visible light, the object identification can be improved. For example, in a surveillance camera, it is desired to capture a color image even in an environment without visible light in order to improve the object identification.

  Patent Document 1 describes an imaging apparatus that can capture a color image even in an environment without visible light. In the imaging device described in Patent Document 1, an infrared projector is also used. If the technique described in Patent Document 1 is installed in the surveillance camera, it becomes possible to convert the subject into a color image and improve the object identification.

JP 2011-50049 A

  By the way, in an ordinary imaging apparatus that images a subject in an environment with visible light, it has been established how to adjust the color of a video signal. However, in an imaging device for night vision that irradiates a subject with infrared light in an environment where there is almost no visible light to produce a color image of the subject, it has been established how to adjust the color of the video signal. It has not been.

  That is, in order to obtain a video signal with good color reproducibility in an imaging device for night vision, a method for adjusting the color of the video signal has not been established, and it is necessary to establish an adjustment method. It has been.

  An object of the present invention is to provide a technique capable of obtaining a video signal with good color reproducibility in a state where infrared light is projected.

  A projection control unit that controls an infrared projector that projects infrared light; an imaging unit that captures an image of a subject in a state in which the infrared projector is controlled so as to project infrared light; and Each color in the color video signal output by the video output unit includes a video processing unit that generates a color video signal based on the imaging signal, a video output unit that outputs the color video signal, and a subject detection unit that detects a subject. An image pickup apparatus is provided in which the ratio of the pixel values between them differs according to the detection result of the subject detection unit.

Also, a projection control step for controlling an infrared projector that projects infrared light, and an imaging step for generating an imaging signal by imaging a subject in a state in which the infrared projector is controlled to project infrared light. A color image output step in the video output step, including a video processing step for generating a color video signal based on the imaging signal, a video output step for outputting the color video signal, and a subject detection step for detecting a subject. There is provided an imaging method characterized in that a ratio of pixel values between colors in a signal varies depending on a detection result by the subject detection unit.
In addition, a projection control means for controlling an infrared projector for projecting infrared light, an imaging signal generated by imaging a subject in a state where the infrared projector is controlled to project infrared light. A pixel value between each color in the color video signal output by the video output means, functioning as a video processing means for generating a color video signal based on the image output means, a video output means for outputting the color video signal, and a subject detection means for detecting a subject. The imaging program is characterized in that the ratio varies depending on the detection result by the subject detection unit.

  According to the present invention, it is possible to obtain a video signal with good color reproducibility in a state where infrared light is projected.

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to an embodiment. It is a figure which shows an example of the arrangement | sequence of the filter element in the color filter used for the imaging device of one Embodiment. It is a characteristic view which shows the spectral sensitivity characteristic of the wavelength of three primary color lights and relative sensitivity in the imaging part which comprises the imaging device of one Embodiment. It is a characteristic view showing the relationship between the wavelength and the relative detection rate when the reflectance of the three primary colors from a predetermined substance is multiplied by the light receiving sensitivity of silicon. It is a block diagram which shows the specific structure of the front signal process part 52 in FIG. It is a figure which shows the relationship between exposure and the flame | frame of a video signal when the imaging device of one Embodiment is operate | moving in normal mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in normal mode. It is a figure which shows the relationship between exposure and the flame | frame of a video signal when the imaging device of one Embodiment is operate | moving in intermediate | middle mode and night vision mode. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in 1st intermediate | middle mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in 1st intermediate | middle mode. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in 2nd intermediate mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in 2nd intermediate mode. It is a figure for demonstrating the addition process of the surrounding pixel when the imaging device of one Embodiment is operate | moving in night vision mode. It is a figure which shows the flame | frame in which the addition process of the surrounding pixel was performed. It is a figure for demonstrating the pre-signal process when the imaging device of one Embodiment is operate | moving in 1st night-vision mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in 1st night-vision mode. It is a figure for demonstrating the front signal process when the imaging device of one Embodiment is operate | moving in 2nd night-vision mode. It is a figure for demonstrating a demosaic process when the imaging device of one Embodiment is operate | moving in 2nd night-vision mode. It is a figure for demonstrating the example of the mode switching in the imaging device of one Embodiment. It is a figure which shows the state of each part when the imaging device of one Embodiment is set to each mode. It is a partial block diagram which shows the 1st modification of the imaging device of one Embodiment. It is a partial block diagram which shows the 2nd modification of the imaging device of one Embodiment. It is a partial block diagram which shows the 3rd modification of the imaging device of one Embodiment. It is a flowchart which shows a video signal processing method. It is a flowchart which shows the specific process of the normal mode shown to step S3 in FIG. It is a flowchart which shows the specific process of the intermediate | middle mode of step S4 in FIG. It is a flowchart which shows the specific process of night vision mode of step S5 in FIG. It is a flowchart which shows the process which a video signal processing program performs a computer. It is a figure which shows the white position on the vector scope of NTSC when white balance is adjusted in normal mode. It is a figure which shows the position where white moves on the vector scope of NTSC when the balance of a video signal is adjusted to a 1st state in night vision mode. It is a figure which shows the position which white moves on the vector scope of NTSC, when the balance of a video signal is adjusted to a 2nd state in night vision mode. It is a figure which shows how the value of R / G and B / G in normal mode changes in night vision mode.

  Hereinafter, an imaging apparatus, an imaging method, and an imaging program according to an embodiment will be described with reference to the accompanying drawings.

<Configuration of imaging device>
First, the overall configuration of an imaging apparatus according to an embodiment will be described with reference to FIG. An imaging apparatus according to an embodiment shown in FIG. 1 includes a normal mode suitable for an environment where there is sufficient visible light such as daytime, a night vision mode suitable for an environment where there is almost no visible light such as nighttime, and a visible mode. This is an imaging device capable of imaging in three modes, an intermediate mode suitable for an environment where light is slightly present.

  The intermediate mode is a first infrared light projection mode in which imaging is performed while projecting infrared light in an environment with little visible light. The night vision mode is a second infrared light projection mode in which imaging is performed while projecting infrared light in an environment where there is even less (almost) no visible light.

  The imaging device may include only one of the intermediate mode and the night vision mode. The imaging device may not have the normal mode. The imaging apparatus should just be equipped with the infrared light projection mode which images at least while projecting infrared rays.

  In FIG. 1, the light indicated by the alternate long and short dash line reflected from the subject is collected by the optical lens 1. Here, the optical lens 1 has visible light in an environment in which visible light is sufficiently present, and infrared light in which an object reflects infrared light emitted from an infrared projector 9 described below in an environment with little visible light. Is incident.

  Under an environment where there is a slight amount of visible light, the optical lens 1 is incident with light in which visible light and infrared light emitted from the infrared projector 9 are reflected by the subject.

  Although only one optical lens 1 is shown in FIG. 1 for the sake of simplicity, the imaging apparatus actually includes a plurality of optical lenses.

  An optical filter 2 is provided between the optical lens 1 and the imaging unit 3. The optical filter 2 has two parts, an infrared cut filter 21 and a dummy glass 22. The optical filter 2 includes a state in which an infrared cut filter 21 is inserted between the optical lens 1 and the imaging unit 3 and a state in which a dummy glass 22 is inserted between the optical lens 1 and the imaging unit 3 by the driving unit 8. It is driven to either state.

  The imaging unit 3 includes an imaging element 31 in which a plurality of light receiving elements (pixels) are arranged in the horizontal direction and the vertical direction, and any one of red (R), green (G), and blue (B) corresponding to each light receiving element. And a color filter 32 in which filter elements of these colors are arranged. The image sensor 31 may be a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

  As an example, as shown in FIG. 2, the color filter 32 has R, G, and B color filter elements arranged in an array called a Bayer array. The Bayer array is an example of a predetermined array of R, G, and B filter elements. In FIG. 2, the G filter element sandwiched between the R filter elements in each row is Gr, and the G filter element sandwiched between the B filter elements is Gb.

  In the Bayer array, horizontal rows in which R filter elements and Gr filter elements are alternately arranged and horizontal rows in which B filter elements and Gb filter elements are alternately arranged are vertically arranged. They are arranged alternately in the direction.

  FIG. 3 shows spectral sensitivity characteristics of the wavelengths of R light, G light, and B light and relative sensitivity in the imaging unit 3. The relative sensitivity is normalized to 1 at the maximum value. When the imaging apparatus is operated in the normal mode, it is necessary to cut infrared light having a wavelength of 700 nm or more in order to capture a good color image by visible light.

  Therefore, the drive unit 8 drives the optical filter 2 so that the infrared cut filter 21 is inserted between the optical lens 1 and the imaging unit 3 based on control by the control unit 7.

  As can be seen from FIG. 3, the imaging unit 3 has sensitivity even in an infrared light region having a wavelength of 700 nm or more. Therefore, when the imaging device is operated in the intermediate mode or the night vision mode, the driving unit 8 removes the infrared cut filter 21 between the optical lens 1 and the imaging unit 3 based on the control by the control unit 7 and the dummy glass. The optical filter 2 is driven so that 22 is inserted.

  In a state where the dummy glass 22 is inserted between the optical lens 1 and the imaging unit 3, infrared light having a wavelength of 700 nm or more is not cut. Therefore, the imaging apparatus can obtain R, G, and B color information by using the sensitivity of the portion surrounded by the dashed ellipse in FIG. The reason why the dummy glass 22 is inserted is to make the optical path length the same as the optical path length when the infrared cut filter 21 is inserted.

  The infrared projector 9 includes projectors 91, 92, and 93 that project infrared light having wavelengths IR1, IR2, and IR3, respectively. In the intermediate mode or the night vision mode, the light projecting control unit 71 in the control unit 7 performs control so as to selectively project infrared light with wavelengths IR1 to IR3 from the light projecting units 91 to 93 in a time division manner. .

  Incidentally, a silicon wafer is used for the image sensor 31. FIG. 4 shows the relationship between the wavelength and the relative detection rate when the reflectance at each wavelength is multiplied by the light receiving sensitivity of silicon when a material exhibiting each color of R, G, and B is irradiated with white light. Yes. Also in FIG. 4, the maximum value of the relative detection rate is normalized to 1.

  As shown in FIG. 4, in the infrared light region, for example, the reflected light at a wavelength of 780 nm is highly correlated with the reflected light of a material exhibiting an R color, and the reflected light at a wavelength of 870 nm is a reflection of a material exhibiting a B color. Correlation with light is high, and reflected light at a wavelength of 940 nm is highly correlated with reflected light of a material exhibiting G color.

  Therefore, in this embodiment, the wavelengths IR1, IR2, and IR3 of the infrared light projected by the light projecting units 91, 92, and 93 are set to 780 nm, 940 nm, and 870 nm. These wavelengths are examples of the wavelengths IR1 to IR3, and may be other than 780 nm, 940 nm, and 870 nm.

  The light projecting unit 91 irradiates the subject with infrared light having the wavelength IR1, and assigns an image signal obtained by imaging the light reflected from the subject to the R signal. The light projecting unit 93 irradiates the subject with infrared light having a wavelength IR2, and assigns a video signal obtained by imaging the light reflected from the subject to the G signal. The light projecting unit 92 irradiates the subject with infrared light having a wavelength IR3, and assigns a video signal obtained by imaging the light reflected from the subject to the B signal.

  In this way, in principle, even in the intermediate mode or the night vision mode, it is possible to reproduce the same color as when the subject is imaged in an environment where visible light exists in the normal mode.

  Although the color image is different from the actual color of the subject, a wavelength IR1 of 780 nm may be assigned to the R light, a wavelength IR3 of 870 nm may be assigned to the G light, and a wavelength IR2 of 940 nm may be assigned to the B light. The wavelengths IR1, IR2, and IR3 can be arbitrarily assigned to the R light, G light, and B light.

  In the present embodiment, the wavelengths IR1, IR2, and IR3 that best reproduce the color of the subject are assigned to R light, G light, and B light, respectively.

  The control unit 7 controls imaging in the imaging unit 3, each unit in the video processing unit 5, and the video output unit 6. In addition to the projection control unit 71, the control unit 7 uses a mode switching unit 72 that switches between the normal mode, the intermediate mode, and the night-vision mode, and a color gain setting unit 62 described later, and the three primary color data (R pixel value, G pixel value). , B pixel value), a color gain control unit 73 that controls to adjust and set the color gain.

  The mode switching unit 72 appropriately switches the operation in the video processing unit 5 as described later in correspondence with the normal mode, the intermediate mode, and the night vision mode. The video processing unit 5 and the control unit 7 may be integrated.

  The video signal captured by the imaging unit 3 is A / D converted by the A / D converter 4 and input to the video processing unit 5. The imaging unit 3 and the A / D converter 4 may be integrated.

  The video processing unit 5 includes switches 51 and 53, a previous signal processing unit 52, and a demosaic processing unit 54. The switches 51 and 53 may be physical switches, or may be conceptual switches for switching between operation and non-operation of the previous signal processing unit 52. A video signal is input from the video processing unit 5 to the control unit 7 in order to detect the brightness of the video being captured.

  As illustrated in FIG. 5, the previous signal processing unit 52 includes a surrounding pixel addition unit 521, a same-position pixel addition unit 522, and a synthesis unit 523.

  The video processing unit 5 generates R, G, and B primary color data and supplies them to the video output unit 6. The video output unit 6 outputs the three primary color data in a predetermined format to a display unit not shown.

  The video output unit 6 may output the R, G, and B signals as they are, or may convert the R, G, and B signals into luminance signals and color signals (or color difference signals) and output them. The video output unit 6 may output a composite video signal. The video output unit 6 may output a digital video signal or may output a video signal converted into an analog signal by a D / A converter.

  The video output unit 6 includes a color gain setting unit 62 that adjusts the balance (ratio) of R, G, and B by multiplying each of the three primary color data of R, G, and B by a predetermined color gain. The color gain setting unit 62 can set the color gain in order to adjust the white balance of the video signal in the normal mode. The color gain setting unit 62 may set a color gain in order to adjust the balance of R, G, and B in the night vision mode.

  Here, the color gain setting unit 62 is provided in the video output unit 6, but may be provided in the video processing unit 5, and the position thereof is arbitrary.

  Hereinafter, specific operations in the normal mode, the intermediate mode, and the night vision mode will be described.

<Normal mode>
In the normal mode, the control unit 7 causes the driving unit 8 to insert the infrared cut filter 21 between the optical lens 1 and the imaging unit 3. The light projection control unit 71 turns off infrared light projection by the infrared projector 9.

  The video signal imaged by the imaging unit 3 is converted into video data that is a digital signal by the A / D converter 4 and input to the video processing unit 5. In the normal mode, the mode switching unit 72 controls the switches 51 and 53 to be connected to the terminal Tb.

  FIG. 6A shows exposures Ex1, Ex2, Ex3,... Actually, the exposure time varies depending on conditions such as the shutter speed. Here, the exposures Ex1, Ex2, and Ex3 indicate the maximum exposure time.

  FIG. 6B shows the timing at which each video signal frame is obtained. Based on an exposure (not shown) before the exposure Ex1, a frame F0 of the video signal is obtained after a predetermined time. Based on the exposure Ex1, a frame F1 of the video signal is obtained after a predetermined time. Based on the exposure Ex2, a frame F2 of the video signal is obtained after a predetermined time. The same applies to exposure Ex3 and thereafter. The frame frequency of the video signal is set to 30 frames / second, for example.

  The frame frequency of the video signal may be set as appropriate, such as 30 frames / second or 60 frames / second for the NTSC system and 25 frames / second or 50 frames / second for the PAL system. The frame frequency of the video signal may be 24 frames / second used in movies.

  The video data of each frame output from the A / D converter 4 is input to the demosaic processing unit 54 via the switches 51 and 53. The demosaic processing unit 54 performs demosaic processing on the input video data of each frame. The video processing unit 5 performs various video processing such as white balance correction and gain correction in addition to demosaic processing, and outputs R, G, and B primary color data.

  The demosaic processing in the demosaic processing unit 54 will be described with reference to FIG. 7A shows an arbitrary frame Fm of video data. The frame Fm is a frame composed of pixels in the effective video period. The number of pixels of the video data is, for example, horizontal 640 pixels and vertical 480 pixels in the VGA standard. Here, for simplification, the number of pixels of the frame Fm is significantly reduced, and the frame Fm is conceptually illustrated.

  The video data generated using the Bayer array imaging unit 3 is data in which R, G, and B pixel data are mixed in the frame Fm. The demosaic processing unit 54 generates R interpolation pixel data Ri obtained by calculating R pixel data at a pixel position where no R pixel data is present using surrounding R pixel data. The demosaic processing unit 54 generates an R frame FmR in which all the pixels in one frame shown in FIG. 7B are made up of R pixel data.

  The demosaic processing unit 54 generates G interpolation pixel data Gi obtained by calculating G pixel data at a pixel position where no G pixel data exists using surrounding G pixel data. The demosaic processing unit 54 generates a G frame FmG in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 generates B interpolated pixel data Bi obtained by calculating B pixel data at a pixel position where no B pixel data exists using surrounding B pixel data. The demosaic processing unit 54 generates a B frame FmB in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 may use at least R pixel data when interpolating R pixel data, may use at least G pixel data when interpolating G pixel data, and B pixel data. When interpolation is performed, at least B pixel data may be used. In order to improve interpolation accuracy, the demosaic processing unit 54 may use pixel data of another color different from the color of the interpolation pixel data to be generated when interpolating R, G, and B pixel data. Good.

  Since there are pixels outside the effective video period in the imaging unit 3, R, G, and B pixel data can be interpolated even in pixels located at the upper, lower, left, and right ends of the frame Fm.

  The R frame FmR, G frame FmG, and B frame FmB generated by the demosaic processing unit 54 are output as R, G, and B primary color data. In FIG. 6, the R, G, and B pixel data has been described in units of frames in order to facilitate understanding, but actually, the R, G, and B pixel data are sequentially output for each pixel.

<Intermediate mode: first intermediate mode>
In the intermediate mode (first intermediate mode and second intermediate mode described later), the control unit 7 causes the driving unit 8 to insert the dummy glass 22 between the optical lens 1 and the imaging unit 3. The light projection controller 71 turns on the infrared light projection by the infrared light projector 9. The mode switching unit 72 controls the switches 51 and 53 to be connected to the terminal Ta.

  FIG. 8A shows a state of infrared light projection by the infrared projector 9. The control unit 7 divides one frame period of the normal mode into 1/3, and controls so as to project infrared light in the order of the light projecting units 91, 92, 93, for example.

  In the example shown in FIG. 8A, infrared light having a wavelength IR1 (780 nm) is irradiated onto the subject in the first ３ period of one frame. In the next 1/3 period of one frame, the subject is irradiated with infrared light having a wavelength IR2 (940 nm). In the last 1/3 period of one frame, the subject is irradiated with infrared light having a wavelength IR3 (870 nm). The order of projecting infrared light with wavelengths IR1 to IR3 is arbitrary.

  As shown in FIG. 8B, at the timing when infrared light having the wavelength IR1 is projected, the imaging unit 3 performs exposure Ex1R having a high correlation with the R light. At the timing when the infrared light having the wavelength IR2 is projected, the imaging unit 3 performs the exposure Ex1G having a high correlation with the G light. At the timing when the infrared light having the wavelength IR3 is projected, the imaging unit 3 performs exposure Ex1B having a high correlation with the B light.

  However, in the intermediate mode, since imaging is performed in an environment where there is a small amount of visible light, visible light and infrared light projected from the infrared projector 9 are mixed. Therefore, in the intermediate mode, the exposures Ex1R, Ex1G, Ex1B, Ex2R, Ex2G, Ex2B... Are exposures that combine exposure with visible light and exposure with infrared light.

  As shown in FIG. 8C, based on the exposures Ex1R, Ex1G, and Ex1B, a frame F1IR1 corresponding to the exposure Ex1R, a frame F1IR2 corresponding to the exposure Ex1G, and a frame F1IR3 corresponding to the exposure Ex1B are obtained after a predetermined time. It is done.

  Further, based on the exposures Ex2R, Ex2G, and Ex2B, a frame F2IR1 corresponding to the exposure Ex2R, a frame F2IR2 corresponding to the exposure Ex2G, and a frame F2IR3 corresponding to the exposure Ex2B are obtained after a predetermined time. The same applies to exposures Ex3R, Ex3G, and Ex3B.

  The frame frequency of the imaging signal in FIG. 8C is 90 frames / second. In the intermediate mode, since one frame of the video signal in the normal mode is time-divided and infrared light of wavelengths IR1 to IR3 is projected, in order to output a video signal in the same format as in the normal mode, ( The frame frequency of the imaging signal in c) is three times the frame frequency in the normal mode.

  As will be described later, one frame of a video signal having a frame frequency of 30 frames / second shown in FIG. 8D is generated based on the imaging signal of 3 frames shown in FIG. For example, a frame F1IR is generated based on the frames F1IR1, F1IR2, and F1IR3, and a frame F2IR is generated based on the frames F2IR1, F2IR2, and F2IR3.

  The operation in the intermediate mode for generating the video signal of each frame in FIG. 8D based on the three frames of the imaging signal in FIG. 8C will be specifically described.

  Video data of each frame corresponding to the imaging signal shown in FIG. 8C output from the A / D converter 4 is input to the previous signal processing unit 52 via the switch 51.

  The previous signal processing in the previous signal processing unit 52 will be described with reference to FIG. FIG. 9A shows an arbitrary frame FmIR1 of the video data generated at the timing when the infrared light having the wavelength IR1 is projected. The R, B, Gr, and Gb pixel data in the frame FmIR1 is attached with a subscript 1 indicating that it is generated in a state where infrared light having the wavelength IR1 is projected.

  FIG. 9B shows an arbitrary frame FmIR2 of video data generated at the timing when infrared light having a wavelength IR2 is projected. The R, B, Gr, and Gb pixel data in the frame FmIR2 is attached with a subscript 2 indicating that it is generated in a state where infrared light having a wavelength IR2 is projected.

  FIG. 9C shows an arbitrary frame FmIR3 of the video data generated at the timing when the infrared light having the wavelength IR3 is projected. The R, B, Gr, and Gb pixel data in the frame FmIR3 has a subscript 3 indicating that it is generated in a state where infrared light having the wavelength IR3 is projected.

  The frame FmIR1 shown in FIG. 9A is video data generated in a state where infrared light having a wavelength IR1 having a high correlation with the R light is projected. Therefore, the R pixel data is projected. The pixel data corresponding to the infrared light and the pixel data B and G are pixel data not corresponding to the projected infrared light. The hatching attached to the B, Gr, and Gb pixel data means that the pixel data does not correspond to the projected infrared light.

  The frame FmIR2 shown in FIG. 9B is video data generated in a state where infrared light having a wavelength IR2 having a high correlation with the G light is projected, so that the G pixel data is projected. The pixel data corresponding to the infrared light, and the R and B pixel data are pixel data not corresponding to the projected infrared light. The hatching attached to the R and B pixel data means that the pixel data does not correspond to the projected infrared light.

  The frame FmIR3 shown in FIG. 9C is video data generated in a state where infrared light having a wavelength IR3 having a high correlation with the B light is projected, so that the B pixel data is projected. The pixel data corresponding to the infrared light, and the R and G pixel data are pixel data not corresponding to the projected infrared light. The hatching attached to the R, Gr, and Gb pixel data means pixel data that does not correspond to the projected infrared light.

  The same-position pixel addition unit 522 in the previous signal processing unit 52 individually adds R, Gr, Gb, and B pixel data at the same pixel position according to the following equations (1) to (3) to obtain pixel data. R123, Gr123, Gb123, and B123 are generated. In the intermediate mode, the surrounding pixel addition unit 521 in the previous signal processing unit 52 does not operate.

R123 = ka × R1 + kb × R2 + kc × R3 (1)
G123 = kd × G1 + ke × G2 + kf × G3 (2)
B123 = kg × B1 + kh × B2 + ki × B3 (3)

  In the equations (1) to (3), R1, G1, and B1 are R, G, and B pixel data in the frame FmIR1, R2, G2, and B2 are R, G, and B pixel data in the frame FmIR2, R3, G3, B3 is R, G, B pixel data in the frame FmIR3. ka to ki are predetermined coefficients. G123 in Formula (2) is Gr123 or Gb123.

  At this time, the same-position pixel adding unit 522 adds R, Gr, Gb, and B pixel data at the same pixel position with hatching to the respective pixel data with R, Gr, Gb, and B that are not hatched. Is added.

  That is, the same-position pixel addition unit 522 adds the R pixel data at the same pixel position in the frames FmIR2 and FmIR3 to the R pixel data in the frame FmIR1 based on the equation (1) to generate the pixel data R123. To do. That is, red pixel data R123 is generated using only pixel data in a region corresponding to the red color filter in the light receiving element.

  The same-position pixel addition unit 522 adds the pixel data of Gr and Gb at the same pixel position in the frames FmIR1 and FmIR3 to the pixel data of Gr and Gb in the frame FmIR2 based on the equation (2), thereby obtaining pixel data G123. Is generated. That is, green pixel data G123 is generated using only pixel data in a region corresponding to the green color filter in the light receiving element.

  The same-position pixel addition unit 522 generates pixel data B123 by adding the B pixel data at the same pixel position in the frames FmIR1 and FmIR2 to the B pixel data in the frame FmIR3 based on the equation (3). That is, blue pixel data B123 is generated using only pixel data in a region corresponding to the blue color filter in the light receiving element.

  The combining unit 523 in the previous signal processing unit 52 generates a frame FmIR123 of the combined video signal shown in (d) of FIG. 9 based on the pixel data R123, Gr123, Gb123, and B123 generated at each pixel position. .

  Specifically, the combining unit 523 selects and combines the pixel data R123 in the frame FmIR1, the pixel data Gr123 and Gb123 in the frame FmIR2, and the pixel data B123 in the frame FmIR3. Thereby, the synthesis unit 523 generates a frame FmIR123 of the synthesized video signal.

  As described above, the synthesis unit 523 generates the frame FmIR123 in which the pixel data R123, Gr123, Gb123, and B123 are arranged so as to be the same as the arrangement of the filter elements in the color filter 32.

  In the first intermediate mode, video data of the frame FmIR123 is generated using pixel data that is not hatched and pixel data that is hatched.

  The reason why the pixel data at the same pixel position is added by the same position pixel addition unit 522 is as follows. Since the imaging is performed in an environment where visible light is present in the intermediate mode, the hatched pixel data includes respective color components based on exposure with visible light. Therefore, the sensitivity of each color can be increased by adding pixel data at the same pixel position.

  If the visible light and infrared light are mixed and there is a relatively large amount of visible light, exposure with visible light becomes dominant. In this case, the video data of the frame FmIR123 mainly includes components based on the video signal exposed with visible light. If visible light and infrared light are mixed and there is a relatively large amount of infrared light, exposure with infrared light becomes dominant. In this case, the video data of the frame FmIR123 mainly includes components based on the video signal exposed with infrared light.

  When visible light is relatively small, the relationship between the coefficients ka, kb, and kc in equation (1) is ka> kb, kc, and the relationship between the coefficients kd, ke, and kf in equation (2). , Kf> kd, ke, and in equation (3), the magnitude relationship between the coefficients kg, kh, ki is preferably kh> kg, ki. This is because the wavelength IR1 has a high correlation with the R light, the wavelength IR2 has a high correlation with the G light, and the wavelength IR3 has a high correlation with the B light.

  In this way, the R pixel data is mainly the R pixel data in the frame FmIR1, the G pixel data is the G pixel data in the frame FmIR2, and the B pixel data is mainly the B pixel data in the frame FmIR3. it can.

  The video data of the frame FmIR123 output from the previous signal processing unit 52 is input to the demosaic processing unit 54 via the switch 53. The demosaic processing unit 54 performs demosaic processing on the video data of the input frame FmIR123, as in the normal mode. The video processing unit 5 performs various video processing such as white balance correction and gain correction in addition to demosaic processing, and outputs R, G, and B primary color data.

  The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 10A shows a frame FmIR123. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data, and generates the R interpolation pixel data R123i. The demosaic processing unit 54 generates an R frame FmIR123R in which all the pixels in one frame shown in FIG.

  The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding G pixel data, and generates the G interpolation pixel data G123i. The demosaic processing unit 54 generates a G frame FmIR123G in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B123i. The demosaic processing unit 54 generates a B frame FmIR123B in which all the pixels in one frame shown in FIG.

  As can be understood by comparing the operation of the demosaic processing unit 54 shown in FIG. 7 in the normal mode with the operation of the demosaic processing unit 54 shown in FIG. 10 in the intermediate mode, they are substantially the same. The operation of the demosaic processing unit 54 does not change regardless of whether it is the normal mode or the intermediate mode.

  In the normal mode, the previous signal processing unit 52 is disabled, and in the intermediate mode, the previous signal processing unit 52 may be operated except for the surrounding pixel adding unit 521. In the normal mode and the intermediate mode, the demosaic processing unit 54 in the video processing unit 5 and the signal processing unit such as white balance correction and gain correction can be shared.

<Intermediate mode: second intermediate mode>
The operation in the second intermediate mode will be described with reference to FIGS. In the operation in the second intermediate mode, the description of the same part as the operation in the first intermediate mode is omitted. The frames FmIR1, FmIR2, and FmIR3 in (a) to (c) of FIG. 11 are the same as the frames FmIR1, FmIR2, and FmIR3 in (a) to (c) of FIG.

  The combining unit 523 selects and combines R1 which is R pixel data in the frame FmIR1, Gr2 and Gb2 which are G pixel data in the frame FmIR2, and B3 which is B pixel data in the frame FmIR3. As a result, the synthesis unit 523 generates a frame FmIR123 'of the synthesized video signal shown in FIG.

  That is, the frame FmIR123 'is video data obtained by collecting R, Gr, Gb, and B pixel data that are not hatched in the frames FmIR1, FmIR2, and FmIR3 into one frame.

  That is, in the frame FmIR123 ′, the pixel data for red using only the pixel data in the region corresponding to the red color filter in the state where the infrared light having the wavelength IR1 is projected, and the infrared light having the wavelength IR2 are projected. Only the pixel data for the green color using only the pixel data of the region corresponding to the green color filter in the selected state, and the pixel data of the region corresponding to the blue color filter in the state of projecting the infrared light having the wavelength IR3 are used. The pixel data is for blue.

  As described above, the synthesis unit 523 generates a frame FmIR123 'in which the pixel data R1, Gr2, Gb2, and B3 are arranged so that the arrangement is the same as the arrangement of the filter elements in the color filter 32.

  In the second intermediate mode, the same-position pixel adding unit 522 sets the coefficient ka in Equation (1) to 1, the coefficients kb and kc to 0, the coefficient ke in Equation (2) to 1, the coefficients kd and kf to 0, In equation (3), the coefficient ki is 1 and the coefficients kg and kh are 0.

  As a result, the R pixel data in the frame FmIR1, the Gr and Gb pixel data in the frame FmIR2, and the B pixel data in the frame FmIR3 have the same values.

  Therefore, similarly to the operation in the first intermediate mode, the combining unit 523 selects the R pixel data in the frame FmIR1, the Gr and Gb pixel data in the frame FmIR2, and the B pixel data in the frame FmIR3. FmIR123 'can be generated.

  In the second intermediate mode, the front signal processing unit 52 generates pixel data (hatched) with infrared light projected to generate pixel data having the same color as the pixel data. Video data of the frame FmIR123 ′ is generated using only (non-pixel data).

  According to the second intermediate mode, the sensitivity and color reproducibility are lower than those in the first intermediate mode, but the arithmetic processing can be simplified and the frame memory can be reduced.

  The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 12A shows the frame FmIR123 '. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data, and generates the R interpolation pixel data R1i. The demosaic processing unit 54 generates an R frame FmIR123′R in which all the pixels in one frame shown in FIG. 12B are made up of R pixel data.

  The demosaic processing unit 54 calculates G pixel data at a pixel position where no G pixel data exists using the surrounding G pixel data, and generates G interpolation pixel data G2i. The demosaic processing unit 54 generates a G frame FmIR123′G in which all the pixels in one frame shown in FIG.

  The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B3i. The demosaic processing unit 54 generates a B frame FmIR123′B in which all the pixels of one frame shown in FIG.

  As described above, in the intermediate mode, the pixel data for red is generated from the pixel data obtained from the region corresponding to the red color filter in the light receiving element, and obtained from the region corresponding to the green color filter in the light receiving element. Green pixel data is generated from the pixel data, and blue pixel data is generated from the pixel data obtained from the region corresponding to the blue color filter in the light receiving element.

<Night Vision Mode: First Night Vision Mode>
In the night-vision mode (the first night-vision mode and the second night-vision mode described later), as in the intermediate mode, the control unit 7 causes the drive unit 8 to place the dummy glass 22 between the optical lens 1 and the imaging unit 3. Insert it. The light projection controller 71 turns on the infrared light projection by the infrared light projector 9. The mode switching unit 72 controls the switches 51 and 53 to be connected to the terminal Ta.

  The schematic operation in the night vision mode is the same as that in FIG. However, in the night vision mode, since the imaging is performed in an environment where there is almost no visible light, the exposures Ex1R, Ex1G, Ex1B, Ex2R, Ex2G, Ex2B... In FIG. Is assumed.

  In an environment where there is almost no visible light and only infrared light, there is no difference in the characteristics of the filter elements in the color filter 32, so the imaging unit 3 can be regarded as a monochromatic imaging device. .

  Therefore, in the night vision mode, the surrounding pixel addition unit 521 in the previous signal processing unit 52 adds pixel data located in the periphery to each pixel data in order to improve the sensitivity of infrared light.

  Specifically, as illustrated in FIG. 13A, when the R pixel is the target pixel, the surrounding pixel adding unit 521 includes the G (Gr) positioned around the R pixel data of the target pixel. , Gb) and B pixel data of 8 pixels are added.

  In other words, in the intermediate mode, red pixel data was generated from the pixel data obtained from the region corresponding to the red color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. Pixel data for red is generated from the pixel data obtained from the above. In the example shown in FIG. 13, pixel data obtained from an area corresponding to nine pixels including the target pixel is used for each color.

  As shown in FIG. 13B, when the Gr pixel is the target pixel, the surrounding pixel adding unit 521 has eight R, Gb, and B pixels positioned around the pixel data of the target pixel Gr. Are added. As illustrated in FIG. 13C, when the Gb pixel is the target pixel, the surrounding pixel adding unit 521 has eight R, Gr, and B pixels positioned around the Gb pixel data of the target pixel. Are added.

  In other words, in the intermediate mode, the pixel data for green was generated from the pixel data obtained from the area corresponding to the green color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. The pixel data for green is generated from the pixel data obtained from the above.

  As illustrated in FIG. 13D, when the B pixel is the target pixel, the surrounding pixel adding unit 521 has R and G (Gr, Gb) positioned around the B pixel data of the target pixel. The pixel data of 8 pixels are added.

  In other words, in the intermediate mode, blue pixel data was generated from the pixel data obtained from the region corresponding to the blue color filter in the light receiving element, but in the night vision mode, a wider area than in the intermediate mode. The pixel data for blue is generated from the pixel data obtained from the above.

  The surrounding pixel adding unit 521 may simply add 9 pixels of the pixel data of the target pixel and the surrounding 8 pixel pixel data, or may apply a predetermined weight to the surrounding 8 pixel pixel data. May be added to the pixel data of the pixel of interest.

  By the way, there is an image pickup device that can read out a plurality of pixels called binning as a single pixel. When an image sensor having a binning function is used as the image sensor 31, an addition process by an image sensor having a binning function may be performed instead of the addition process by the surrounding pixel addition unit 521. Binning by the image sensor is substantially equivalent to addition processing by the surrounding pixel addition unit 521.

  The frames FmIR1, FmIR2, and FmIR3 in (a) to (c) of FIG. 14 are the same as the frames FmIR1, FmIR2, and FmIR3 in (a) to (c) of FIG. In (d) to (f) of FIG. 14, R1ad, Gr1ad, Gb1ad, B1ad, R2ad, Gr2ad, Gb2ad, B2ad, R3ad, Gr3ad, Gb3ad, and B3ad are converted into R, Gr, Gb, and B pixel data, respectively. On the other hand, it is added pixel data obtained by adding pixel data of surrounding eight pixels.

  The surrounding pixel addition unit 521 performs the addition processing shown in FIG. 13 on the pixel data of the frames FmIR1, FmIR2, and FmIR3, so that the frames FmIR1ad, FmIR2ad, and FmIR3ad shown in (d) to (f) of FIG. Is generated.

  The frames FmIR1ad, FmIR2ad, and FmIR3ad in (a) to (c) of FIG. 15 are the same as the frames FmIR1ad, FmIR2ad, and FmIR3ad in (d) to (f) of FIG.

  Similarly to the first intermediate mode, the same-position pixel addition unit 522 adds the pixel data of R2ad and R3ad at the same pixel position in the frames FmIR2ad and FmIR3ad to the pixel data of R1ad in the frame FmIR1ad based on Expression (1) Thus, pixel data R123ad is generated.

  The same-position pixel addition unit 522 adds the pixel data of Gr1ad, Gb1ad, Gr3ad, and Gb3ad at the same pixel position in the frames FmIR1ad and FmIR3ad to the pixel data of Gr2ad and Gb2ad in the frame FmIR2ad based on Expression (2). Pixel data Gr123ad and Gb123ad are generated.

  Based on Expression (3), the same-position pixel addition unit 522 adds the pixel data of B1ad and B2ad at the same pixel position in the frames FmIR1ad and FmIR2ad to the pixel data of B3ad in the frame FmIR3ad to generate pixel data B123ad To do.

  Similar to the first intermediate mode, the combining unit 523 selects and combines the pixel data R123ad in the frame FmIR1ad, the pixel data Gr123ad and Gb123ad in the frame FmIR2ad, and the pixel data B123ad in the frame FmIR3ad. As a result, the synthesis unit 523 generates a frame FmIR123ad of the synthesized video signal shown in FIG.

  The synthesizing unit 523 generates a frame FmIR123ad in which the pixel data R123ad, Gr123ad, Gb123ad, and B123ad are arranged so that the arrangement is the same as the arrangement of the filter elements in the color filter 32.

  FIG. 16A shows a frame FmIR123ad. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R pixel data, and generates the R interpolation pixel data R123adi. The demosaic processing unit 54 generates an R frame FmIR123adR in which all pixels in one frame shown in FIG. 16B are made up of R pixel data.

  The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding G pixel data, and generates the G interpolation pixel data G123adi. The demosaic processing unit 54 generates a G frame FmIR123adG in which all the pixels in one frame shown in FIG.

  The demosaic processing unit 54 calculates the B pixel data at the pixel position where the B pixel data does not exist using the surrounding B pixel data, and generates the B interpolation pixel data B123adi. The demosaic processing unit 54 generates a B frame FmIR123adB in which all the pixels in one frame shown in FIG.

  The first intermediate mode and the first night-vision mode are different in that the former does not operate the surrounding pixel adding unit 521 while the latter operates the surrounding pixel adding unit 521. The mode switching unit 72 may operate the surrounding pixel addition unit 521 in the night vision mode.

  The operation of the demosaic processing unit 54 in the night vision mode is substantially the same as the operation of the demosaic processing unit 54 in the normal mode and the intermediate mode. In the normal mode, the intermediate mode, and the night vision mode, the demosaic processing unit 54 in the video processing unit 5 and a signal processing unit such as white balance correction and gain correction can be shared.

<Night Vision Mode: Second Night Vision Mode>
The operation in the second night-vision mode will be described with reference to FIGS. In the operation in the second night vision mode, the description of the same part as the operation in the first night vision mode is omitted. The frames FmIR1ad, FmIR2ad, and FmIR3ad in (a) to (c) of FIG. 17 are the same as the FmIR1ad, FmIR2ad, and FmIR3ad in (a) to (c) of FIG.

  The combining unit 523 selects and combines R1ad that is R pixel data in the frame FmIR1ad, Gr2ad and Gb2ad that are G pixel data in the frame FmIR2, and B3ad that is B pixel data in the frame FmIR3. Thereby, the synthesis unit 523 generates a frame FmIR123′ad of the synthesized video signal shown in FIG.

  The synthesizing unit 523 generates a frame FmIR123′ad in which the pixel data R1ad, Gr2ad, Gb2ad, and B3ad are arranged so that the arrangement is the same as the arrangement of the filter elements in the color filter 32.

  As described with reference to FIG. 13, the pixel data R1ad for red in the frame FmIR123'ad is obtained from a wider area than the area used for generating the pixel data for red in the intermediate mode. It is generated from pixel data.

  Further, the pixel data Gr2ad for green in the frame FmIR123'ad is generated from pixel data obtained from an area wider than the area used for generating the pixel data for green in the intermediate mode. Yes.

  Further, the blue pixel data B3ad in the frame FmIR123'ad is generated from pixel data obtained from an area larger than the area used to generate the blue pixel data in the intermediate mode. Yes.

  In the second night-vision mode, as in the second intermediate mode, the same-position pixel addition unit 522 sets the coefficient ka in equation (1) to 1, the coefficients kb and kc to 0, and the coefficient ke in equation (2) to 1. The coefficients kd and kf are set to 0, the coefficient ki in the equation (3) is set to 1, and the coefficients kg and kh are set to 0.

  Thereby, the pixel data of R1ad in the frame FmIR1ad, the pixel data of Gr2ad and Gb2ad in the frame FmIR2ad, and the pixel data of B3ad in the frame FmIR3ad have the same values.

  Therefore, as in the operation in the first night vision mode, the combining unit 523 selects the pixel data of R1ad in the frame FmIR1ad, the pixel data of Gr2ad and Gb2ad in the frame FmIR2ad, and the pixel data of B3ad in the frame FmIR3ad. A frame FmIR123'ad can be generated.

  The demosaic process in the demosaic processing unit 54 will be described with reference to FIG. FIG. 18A shows a frame FmIR123′ad. The demosaic processing unit 54 calculates the R pixel data at the pixel position where the R pixel data does not exist using the surrounding R1ad pixel data, and generates the R interpolation pixel data R1adi. The demosaic processing unit 54 generates an R frame FmIR123′adR in which all the pixels of one frame shown in FIG. 18B are made up of R pixel data.

  The demosaic processing unit 54 calculates the G pixel data at the pixel position where the G pixel data does not exist using the surrounding Gr2ad and Gb2ad pixel data, and generates the G interpolation pixel data G2adi. The demosaic processing unit 54 performs interpolation to generate a G frame FmIR123′adG in which all the pixels of one frame shown in FIG.

  The demosaic processing unit 54 generates B interpolation pixel data B3adi obtained by calculating B pixel data at a pixel position where no B pixel data exists using surrounding B3ad pixel data. The demosaic processing unit 54 generates a B frame FmIR123′adB in which all the pixels in one frame shown in FIG.

  The second intermediate mode and the second night-vision mode differ in that the former does not operate the surrounding pixel adding unit 521 while the latter operates the surrounding pixel adding unit 521.

  Further, in the intermediate mode, pixel data for each color is generated from each pixel data obtained from the region corresponding to each color in the light receiving element, but in the night vision mode, the surrounding pixels are added. It can also be said that pixel data for each color is generated from pixel data obtained from a region wider than each region for generating pixel data for each color.

<Example of mode switching>
An example of mode switching by the mode switching unit 72 will be described with reference to FIG. FIG. 19A schematically shows, as an example, how the brightness of the surrounding environment changes as time passes from the daytime period to the nighttime period.

  As shown in FIG. 19 (a), the brightness decreases as time elapses from daytime to evening, and it becomes almost dark after time t3. The brightness shown in FIG. 19A substantially indicates the amount of visible light, and there is almost no visible light after time t3.

  The control unit 7 can determine the brightness of the surrounding environment based on the luminance level of the video signal (video data) input from the video processing unit 5. As shown in FIG. 19B, the mode switching unit 72 sets the normal mode when the brightness is equal to or higher than a predetermined threshold Th1 (first threshold), and sets the predetermined threshold Th2 (less than the threshold Th1). When it is equal to or greater than the second threshold), the intermediate mode is selected.

  The imaging apparatus according to the present embodiment automatically selects the normal mode from time t1 until time t1 when the brightness reaches the threshold value Th1, the intermediate mode from time t1 to time t2 when the brightness reaches the threshold value Th2, and the night vision mode after time t2. Switch automatically. In FIG. 19B, the intermediate mode may be either the first intermediate mode or the second intermediate mode, and the night vision mode may be either the first night vision mode or the second night vision mode.

  In FIG. 19A, the brightness immediately before time t3 when almost no visible light is present is set as the threshold value Th2, but the brightness at time t3 may be set as the threshold value Th2.

  As shown in FIG. 19 (c), the mode switching unit 72 has a period of time t1 in which the visible light is relatively large in the intermediate mode period, the first intermediate mode, and a period of time t2 in which the visible light is relatively small. The period may be set to the second intermediate mode. In FIG. 19C, the night-vision mode may be either the first night-vision mode or the second night-vision mode.

  In the imaging apparatus according to the present embodiment, the light projection control unit 71 controls on / off of the infrared projector 9, and the mode switching unit 72 switches the operation / non-operation of each unit in the video processing unit 5. Can be realized.

  As shown in FIG. 20, in the normal mode, the infrared projector 9 is off, the surrounding pixel adding unit 521, the same position pixel adding unit 522 and the synthesizing unit 523 are not operating, and the demosaic processing unit 54 is operating.

  In the first intermediate mode, the infrared projector 9 is on, the surrounding pixel adding unit 521 is not operating, and the same-position pixel adding unit 522, the combining unit 523, and the demosaic processing unit 54 are operating. In the second intermediate mode, the infrared projector 9 is on, the surrounding pixel adding unit 521 and the same-position pixel adding unit 522 are not operating, and the synthesizing unit 523 and the demosaic processing unit 54 are operating.

  The operation and non-operation in the same position pixel addition unit 522 can be easily switched by appropriately setting the values of the coefficients ka to ki in the expressions (1) to (3) as described above.

  In the first night-vision mode, the infrared projector 9 is on, and the surrounding pixel addition unit 521, the same-position pixel addition unit 522, the synthesis unit 523, and the demosaic processing unit 54 are all in an operating state. In the second night-vision mode, the infrared projector 9 is on, the same-position pixel adding unit 522 is not operating, and the surrounding pixel adding unit 521, the combining unit 523, and the demosaic processing unit 54 are operating.

  By the way, the surrounding pixel adding unit 521 sets the coefficient multiplied by the surrounding pixel data to a coefficient exceeding 0 (for example, 1) in the calculation formula for adding the surrounding pixel data to the pixel data of the target pixel. The surrounding pixel addition processing can be put into an operation state.

  Further, the surrounding pixel adding unit 521 can set the surrounding pixel addition processing to the non-operating state by setting the coefficient by which the surrounding pixel data is multiplied to 0 in the calculation formula.

  The operation and non-operation of the surrounding pixel addition unit 521 can be easily switched by appropriately setting the coefficient value.

<First Modification of Imaging Device>
The method by which the control unit 7 detects the brightness of the surrounding environment is not limited to the method based on the luminance level of the video signal.

  As shown in FIG. 21, the brightness sensor 11 may detect the brightness of the surrounding environment. In FIG. 21, the brightness of the surrounding environment may be determined based on both the luminance level of the video signal and the brightness detected by the brightness sensor 11.

<Second Modification of Imaging Device>
The control unit 7 does not directly detect the brightness of the surrounding environment, but roughly assumes the brightness of the surrounding environment based on the time (date) and time (time zone) in one year, and the mode switching unit 72 may be switched to each mode.

  As shown in FIG. 22, in the mode setting table 12, any one of the normal mode, the intermediate mode, and the night vision mode is set corresponding to the combination of the date and the time zone. A clock 74 in the control unit 7 manages the date and time. The controller 7 reads the mode set from the mode setting table 12 with reference to the date and time indicated by the clock 74.

  The light projection control unit 71 and the mode switching unit 72 control the imaging device so that the mode read from the mode setting table 12 is set.

<Third Modification of Imaging Device>
As illustrated in FIG. 23, the imaging device may be controlled so that the user manually selects a mode using the operation unit 13 and the projection control unit 71 and the mode switching unit 72 are in the selected mode. The operation unit 13 may be an operation button provided on the housing of the imaging apparatus or a remote controller.

<Video signal processing method>
A video signal processing method executed by the imaging apparatus shown in FIG. 1 will be described again with reference to FIG.

  In FIG. 24, when the imaging apparatus starts operation, the control unit 7 determines whether or not the brightness of the surrounding environment is equal to or greater than the threshold value Th1 in step S1. If it is equal to or greater than the threshold Th1 (YES), the control unit 7 causes the process in the normal mode to be executed in step S3. If it is not equal to or greater than the threshold value Th1 (NO), the control unit 7 determines whether or not the brightness of the surrounding environment is equal to or greater than the threshold value Th2 in step S2.

  If it is equal to or greater than the threshold value Th2 (YES), the control unit 7 causes the process in the intermediate mode to be executed in step S4. If it is not greater than or equal to the threshold Th2 (NO), the control unit 7 causes the process in the night-vision mode to be executed in step S5.

  Control part 7 returns processing to Step S1 after Steps S3-S5, and repeats after Step S1.

  FIG. 25 shows specific processing in the normal mode in step S3. In FIG. 25, the control part 7 (light projection control part 71) turns off the infrared light projector 9 in step S31. In step S32, the control unit 7 inserts the infrared cut filter 21. In step S33, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Tb. The order of steps S31 to S33 is arbitrary and may be simultaneous.

  In step S34, the control unit 7 causes the imaging unit 3 to image the subject. In step S <b> 35, the control unit 7 controls the video processing unit 5 so that the demosaic processing unit 54 performs demosaic processing on the frames constituting the video signal generated by the imaging unit 3 imaging the subject.

  FIG. 26 shows specific processing in the intermediate mode in step S4. 26, in step S41, the control unit 7 (light projection control unit 71) turns on the infrared projector 9 so that infrared light with wavelengths IR1 to IR3 is projected in a time-sharing manner from the light projecting units 91 to 93. To.

  The control part 7 inserts the dummy glass 22 in step S42. In step S43, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Ta. The order of steps S41 to S43 is arbitrary and may be simultaneous.

  In step S44, the control unit 7 causes the imaging unit 3 to image the subject. The imaging unit 3 projects infrared light with a wavelength IR1 associated with R, infrared light with a wavelength IR2 associated with G, and infrared light with a wavelength IR3 associated with B, respectively. Take a picture of the subject in the

In step S45, the control unit 7 (mode switching unit 72) controls the pre-signal processing unit 52 so that the surrounding pixel addition unit 521 is deactivated and the synthesis unit 523 is operated to generate a synthesized video signal.

  The frames constituting the video signal generated when the imaging unit 3 images the subject in the state where infrared light of wavelengths IR1, IR2, and IR3 are respectively projected are the first frame, the second frame, 3 frames are assumed.

  The synthesizing unit 523 converts the three primary color pixel data based on the R pixel data in the first frame, the G pixel data in the second frame, and the B pixel data in the third frame into color The filters 32 are arranged so as to have the same arrangement as the filter elements. The synthesizer 523 thus generates a synthesized video signal by synthesizing the first to third frames into one frame.

  In step S46, the control unit 7 controls the video processing unit 5 so that the demosaic processing unit 54 performs demosaic processing on the frame of the composite video signal.

  The demosaic processing unit 54 performs demosaic processing for generating an R frame, a G frame, and a B frame based on the frame of the composite video signal, and sequentially generates the demosaiced three primary color frames. .

  The demosaic processing unit 54 can generate an R frame by interpolating the R pixel data at a pixel position where the R pixel data does not exist. The demosaic processing unit 54 can generate the G frame by interpolating the G pixel data at the pixel position where the G pixel data does not exist. The demosaic processing unit 54 can generate the B frame by interpolating the B pixel data at the pixel position where the B pixel data does not exist.

  In the case of the first intermediate mode, the same position pixel addition unit 522 is operated in step S45, and in the case of the second intermediate mode, the same position pixel addition unit 522 is not operated in step S45. do it.

  FIG. 27 shows specific processing in the night-vision mode in step S5. In FIG. 27, in step S51, the control unit 7 (light projection control unit 71) turns on the infrared projector 9 so that infrared light with wavelengths IR1 to IR3 is projected in a time-sharing manner from the light projecting units 91 to 93. To.

  The control part 7 inserts the dummy glass 22 in step S52. In step S53, the control unit 7 (mode switching unit 72) connects the switches 51 and 53 to the terminal Ta. The order of steps S51 to S53 is arbitrary and may be simultaneous.

  In step S54, the control unit 7 causes the imaging unit 3 to image the subject. In step S55, the control unit 7 (mode switching unit 72) controls the previous signal processing unit 52 to operate the surrounding pixel addition unit 521 and the synthesis unit 523 to generate a synthesized video signal.

  In step S56, the control unit 7 controls the video processing unit 5 so that the demosaic processing unit 54 performs demosaic processing on the frame of the composite video signal.

  In the case of the first night vision mode, the same position pixel addition unit 522 is operated in step S55, and in the case of the second night vision mode, the same position pixel addition unit 522 is disabled in step S55. It may be an operation.

<Video signal processing program>
In FIG. 1, the control unit 7 or the video processing unit 5 and the control unit 7 are integrated by a computer (microcomputer), and the video signal processing program (computer program) is executed by the computer. It is also possible to realize the same operation as that of the imaging apparatus of this embodiment.

  With reference to FIG. 28, an example of a procedure executed by the computer when the control in the intermediate mode, which is step S4 in FIG. 24, is configured by the video signal processing program will be described. FIG. 28 shows processing that the video signal processing program causes the computer to execute.

  In FIG. 28, in step S401, the video signal processing program controls the infrared projector 9 so as to project infrared light of wavelengths IR1, IR2, and IR3 associated with R, G, and B to the computer. The step to perform is performed.

  The step shown in step S401 may be executed outside the video signal processing program. In FIG. 28, the step of inserting the dummy glass 22 is omitted. The step of inserting the dummy glass 22 may also be executed outside the video signal processing program.

  In step S402, the video signal processing program captures the first frame of the video signal generated by the imaging unit 3 imaging the subject in a state where infrared light having the wavelength IR1 is projected on the computer. A step of acquiring pixel data to be configured is executed.

  In step S403, the video signal processing program captures the second frame of the video signal generated when the imaging unit 3 images the subject in a state where infrared light having the wavelength IR2 is projected on the computer. A step of acquiring pixel data to be configured is executed.

  In step S404, the video signal processing program captures a third frame of the video signal generated when the imaging unit 3 images the subject in a state where infrared light having the wavelength IR3 is projected on the computer. A step of acquiring pixel data to be configured is executed. The order of steps S402 to S404 is arbitrary.

  In step S405, the video signal processing program causes the computer to arrange the R, G, and B pixel data so as to be in the same arrangement as the arrangement of the filter elements in the color filter 32, and synthesize the synthesized video into one frame. A step of generating a signal is executed.

  In the intermediate mode, the video signal processing program does not cause the computer to execute the step of adding surrounding pixels in step S405.

  In step S406, the video signal processing program causes the computer to execute a step of performing demosaic processing on the frame of the composite video signal to generate R, G, and B frames.

  Although illustration is omitted, when the control in the night vision mode, which is step S5 in FIG. 24, is configured by the video signal processing program, the step of adding surrounding pixels is executed in the computer in step S405 in FIG. You can do it.

  The video signal processing program may be a computer program recorded on a computer-readable recording medium. The video signal processing program may be provided in a state where it is recorded on a recording medium, or may be provided via a network such as the Internet so that the computer can download the video signal processing program. The computer-readable recording medium may be any non-temporary arbitrary recording medium such as a CD-ROM or a DVD-ROM.

<Color adjustment in night vision mode>
Here, how the color of the color video signal is adjusted in the night vision mode to obtain a video signal with good color reproducibility will be described. An example in which the color gain of each color is not changed according to the pixel value of the captured image, and an example in which the color gain of each color is changed according to the pixel value of the captured image using a technique such as auto white balance processing. explain. In the following description, when R, G, and B are simply written, R pixel values, G pixel values, and B pixel values are basically represented.

  First, in the normal mode in which the infrared projector 9 is controlled so as not to project infrared light, if the color gain of each color is not changed according to the pixel value of the photographed image, the adjustment method is used to adjust the color of the video signal. Explain what to do. In the normal mode, a general adjustment method established in a normal imaging apparatus that captures an image of an object in an environment with visible light may be employed.

  At the time of adjustment in the normal mode, the imaging apparatus is adjusted so that a video signal output when an achromatic object serving as a reference is imaged under a reference light source is white. The reference light source is, for example, a halogen lamp with a color temperature of 3100K. In general, for example, a standard white plate is used as the reference achromatic object.

  As the standard white plate, for example, “SRT-99-050 Reflectance Target”, a product manufactured by Labsphere, Inc., can be used. The standard white plate is provided by a plurality of companies and may be a product of any company. The standard white plate has a substantially constant reflectance from the visible light region to the near infrared region. In addition, said product has a reflectance of 90% or more in the wavelength range of 250 nm-2500 nm.

  When the color gain of each color is not changed according to the pixel value of the captured image in the normal mode, for example, the color gain control unit 73 turns white the video signal output when the standard white plate is imaged under the reference light source. As described above, the color gain setting unit 62 controls the color gain to be multiplied to the R, G, and B color data (pixel values).

  The ratio of the R, G, B color data output from the video output unit 6 is expressed as R: G: B = x: y: z. When the standard white plate is imaged under the reference light source in the normal mode, the white balance is adjusted by setting R: G: B = 1: 1: 1. The color gain for setting R: G: B = 1: 1: 1 at this time is a color gain value at which the video signal output when the standard white plate is imaged under the reference light source becomes white. At this time, at least the pixels in the region corresponding to the standard white plate may be adjusted so that R: G: B = 1: 1: 1.

  In some cases, the white balance is adjusted by fixing the color gain for G and changing the color gain for R and B. In this case, the white balance is adjusted by adjusting the color gain so that the R and G color data are equal and the B and G color data are equal. That is, white balance is adjusted by setting R / G = 1.0 and B / G = 1.0. Each color gain at this time is a color gain value at which the video signal output when the standard white plate is imaged under the reference light source becomes white.

When the color gain of each color is not changed according to the pixel value of the captured image in the normal mode, the color gain of the video signal output when the standard white plate is imaged under the reference light source is white as described above. The color gain setting unit 62 stores the color gain stored in the memory for each of the R pixel value, the G pixel value, and the B pixel value of the video signal input to the video output unit. Multiply Note that the R pixel value, the G pixel value, and the B pixel value when an achromatic object as a reference is imaged under a reference light source do not need to be completely equal. Good. Especially for monitoring applications, a deviation of several percent is within an acceptable range.
In the normal mode, the white balance is adjusted in this way, and as a result, each color of the video signal is also adjusted.

Next, an example in which the color gain of each color is changed according to the pixel value of the captured image in the normal mode will be described. In this case, a well-known auto white balance process is executed so that each color data in the average value of the pixel values of the entire screen is R: G: B = 1: 1: 1. At that time, the color gain for G is fixed, the color gain for R and B is changed, and auto white balance processing is executed so that R / G = 1.0 and B / G = 1.0. Also good. In any case, each color gain is set in accordance with the average value of the pixel values of the entire screen of the imaging signal that is captured so that the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen are equal. Just decide.
When executing the auto white balance processing in this way, the color gain control unit 73 determines each color gain so that the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen are equal. To do.

For example, when the R pixel value is 97, the G pixel value is 100, and the B pixel value is 107 in the average pixel value of the entire screen, the color gain control unit 73 sets the R color gain. 100/97, G color gain is 1.0, and B color gain is 100/107. Then, the color gain setting unit 62 multiplies each color data of the video signal input to the video output unit 6 by each color gain determined by the color gain control unit 73, thereby obtaining an average value of pixel values of the entire screen. The R pixel value, the G pixel value, and the B pixel value are equal. In this case, the color gain of each color changes according to the pixel value of the image being shot. Note that the actual color gain is set to a value that can be set in the specifications of the apparatus, the R color gain is 1.031, the G color gain is 1.000, and the B color gain is 0.935. Alternatively, it may be a number up to a settable position in the specification of the apparatus.
When the color gain of each color is changed according to the pixel value of the captured image in the normal mode, each color gain when the standard white plate is imaged under the reference light source is the color gain according to the pixel value of the captured image in the normal mode. It is theoretically equal to each color gain when the color gain is not changed.
Further, the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen do not need to be completely equal, and may be shifted as long as they are about several percent.
Various techniques relating to auto white balance processing may be employed. For example, when the ratio of the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen is greatly deviated from a predetermined threshold value, the target value is R: G: B = 1. You may change from 1: 1.

  Next, in order to obtain a video signal with good color reproducibility in the night vision mode in which the infrared projector 9 is controlled to project infrared light, how the color video signal input to the video output unit 6 is processed. Explain how to adjust.

  In the night vision mode, as described above, video signals obtained when infrared light of wavelengths IR1, IR2, and IR3 is irradiated on the subject are assigned to the R signal, G signal, and B signal, respectively. In principle, even in the night vision mode, the same color as that obtained when an object is imaged in an environment where visible light is present in the normal mode can be reproduced, but the exact same color cannot be reproduced.

  Therefore, in the present embodiment, an adjustment method different from that in the normal mode is employed in the night vision mode. In the night vision mode, an example in which the color gain of each color is not changed in accordance with the pixel value of the captured image and an example in which the color gain of each color is changed in accordance with the pixel value of the captured image will be described. In the following description, for convenience of explanation, the intensity of each infrared light is at least the standard when each standard gain plate is imaged by projecting each infrared light with the same gain for all three colors. It is assumed that the R pixel value, the G pixel value, and the B pixel value have the same intensity in the pixels in the region corresponding to the white plate. That is, it is assumed that the intensity of infrared light does not contribute to the degree of variation between the R pixel value, the G pixel value, and the B pixel value.

Note that the degree of variation between the R pixel value, the G pixel value, and the B pixel value is basically the degree of variation between the R pixel value, the G pixel value, and the B pixel value in the same screen. This is not necessarily the case as long as the image is continuously captured.
In addition, the variation degree of the R pixel value, the G pixel value, and the B pixel value is, for example, various dispersion degrees such as dispersion and standard deviation calculated from the R pixel value, the G pixel value, and the B pixel value. It can be expressed by an index indicating.
In addition, the degree of variation between the R pixel value, the G pixel value, and the B pixel value is also represented by a difference value between the largest value and the smallest value among the R pixel value, the G pixel value, and the B pixel value. It can also be expressed using various well-known statistical indexes.

First, a case where the color gain of each color is not changed according to the pixel value of the captured image in the night vision mode will be described. In this case, when the color gain setting unit 62 captures an image of the standard white plate, the degree of variation in the values of the R pixel value, the G pixel value, and the B pixel value is at least for the pixels in the region corresponding to the standard white plate. A predetermined color that is larger than the degree of variation in the values of the R pixel value, the G pixel value, and the B pixel value in the pixels corresponding to at least the standard white plate when the standard white plate is imaged under the reference light source in the normal mode. The gain is multiplied by the three primary color data of R, G, and B of the video signal input to the video output unit 6.
Then, the video output unit 6 outputs a color video signal in which the color gain is multiplied by the color gain setting unit 62 and the variation degree based on the pixel value of each color is larger than the predetermined variation degree.
That is, when the standard white plate is imaged under the reference light source in the normal mode, the R pixel value, the G pixel value, and the B pixel value are equal, but in the night vision mode, the color gain as described above is multiplied by each color data. The R pixel value, G pixel value, and B pixel value when the standard white plate is imaged are not equal.
When the color gain of each color is not changed according to the pixel value of the captured image in the night vision mode, such a color gain value is held in a memory or the like, and the color gain setting unit 62 is input to the video output unit. Each of the R pixel value, the G pixel value, and the B pixel value of the video signal may be multiplied by the color gain held in the memory.

  As described above, the variation degree of the R pixel value, the G pixel value, and the B pixel value in the pixel corresponding to the standard white plate in the imaging signal is obtained when the standard white plate is imaged under the reference light source in the normal mode. It only needs to be larger than the degree of variation of the R pixel value, the G pixel value, and the B pixel value. The pixel corresponding to the standard white plate in the imaging signal may be a single pixel or may be an average of pixel values of a plurality of pixels corresponding to the standard white plate.

A plurality of color gains in the night vision mode may be prepared according to the subject.
For example, as a result of experiments by the present inventor, when emphasizing the color of human skin, when a standard white plate is imaged by projecting infrared light of wavelengths IR1, IR2, IR3, at least the standard white plate It has been found that good color reproducibility can be obtained when the ratio of the R pixel value, the G pixel value, and the B pixel value in the corresponding region of the pixels is as follows.
That is, the ratio of the R pixel value, the G pixel value, and the B pixel value is such that B / G is larger than R / G, R / G is smaller than 1, and B / G is larger than 1. By setting the color gain in the color gain setting unit 62 so that the ratio of the color gain and the variation degree based on the pixel value of each color is larger than in the normal mode, good color reproducibility can be obtained. all right.
By setting the color gain in this way, it has been confirmed that the color of human skin is well reproduced when the infrared projector 9 projects infrared light with wavelengths IR1 to IR3. More specifically, it is preferable to set the color gain so that R / G = 0.66 and B / G = 1.46.

When the plastic color is important, when the standard white plate is imaged by projecting infrared light of wavelengths IR1, IR2, and IR3, at least in the region corresponding to the standard white plate, the R pixel value Color gain setting unit 62 so that the ratio of G / G pixel value to B pixel value is such that B / G is larger than R / G and R / G and B / G are a predetermined ratio larger than 1. It was found that good color reproducibility can be obtained by setting the color gain at and setting the degree of variation based on the pixel value of each color to be larger than that in the normal mode.
By using such a color gain, it has been confirmed that the color of the plastic is well reproduced when the infrared projector 9 projects infrared light with wavelengths IR1 to IR3. More specifically, it is preferable to set the color gain such that R / G = 1.35 and B / G = 2.04.

  In the case where importance is attached to the color of the plastic, values of R / G and B / G when a plurality of colors such as red, green, and blue are expressed with good color reproducibility in a balanced manner may be adopted. When emphasizing the reproducibility of a specific color among a plurality of colors, the values of R / G and B / G when the specific color is expressed with good color reproducibility may be employed.

As described above, when any color is important, each color gain in the night vision mode has wavelengths IR1 and IR2 as compared with the case where the color gain of each color is not changed according to the pixel value of the captured image in the normal mode. , When the standard white plate is imaged by irradiating infrared light of IR3, the variation degree of the R pixel value, the G pixel value, and the B pixel value is taken under the reference light source in the normal mode. The color gain is larger than the degree of variation of the R pixel value, G pixel value, and B pixel value.
Further, compared with the case where the color gain of each color is changed according to the pixel value of the captured image in the normal mode, each color gain in the night vision mode is irradiated with infrared light of wavelengths IR1, IR2, IR3 and a standard white plate The variation degree of the R pixel value, the G pixel value, and the B pixel value when the image is captured is determined by the R pixel value and the G value in the average value of the pixel values of the entire screen when the auto white balance process is executed in the normal mode. The color gain is larger than the degree of variation between the pixel value and the B pixel value.

It is also possible to adjust the color gain so that the best color reproducibility can be obtained while imaging an object that is actually a subject for colors other than skin and plastic, and then use the adjusted result for imaging. it can. As described above, even when importance is attached to colors other than skin and plastic, the degree of variation in the values of the R pixel value, the G pixel value, and the B pixel value is determined based on the R pixel value and the G pixel when the standard white plate is imaged. Good color reproducibility can be obtained if the value is larger than a predetermined degree of variation so that the value and the B pixel value are not equal.
In other words, when the degree of variation is expressed by standard deviation, good color reproducibility can be obtained if the standard deviation calculated from the R pixel value, G pixel value, and B pixel value is greater than zero.
In addition, when the degree of variation is expressed as a difference value between the largest value and the smallest value among the R pixel value, the G pixel value, and the B pixel value, it is preferable that the difference value be larger than 0. Color reproducibility is obtained.

Next, the case where the color gain of each color is changed according to the pixel value of the captured image in the night vision mode will be described.
In such a case, when compared with each pixel value in the case where the color gain of each color is not changed according to the pixel value of the captured image in the normal mode, the color gain control unit 73 performs the R pixel in the average value of the pixel values of the entire screen. The variation degree of the value, the G pixel value, and the B pixel value is based on the variation degree of the R pixel value, the G pixel value, and the B pixel value when the standard white plate is imaged under the reference light source in the normal mode. The color gain multiplied by the three primary color data of R, G, and B is controlled so as to be a large predetermined value.
That is, in the night vision mode, such color gain is multiplied by each color data, so that the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen are not basically equal.

  In addition, when compared with each pixel value when changing the color gain of each color according to the pixel value of the captured image in the normal mode, the color gain control unit 73 determines that the R pixel value and G in the average value of the pixel values of the entire screen are Each predetermined color in which the degree of variation between the pixel value and the B pixel value is greater than the degree of variation in the R pixel value, the G pixel value, and the B pixel value after the auto white balance process is executed in the normal mode It can be said that the color gain multiplied by the three primary color data of R, G, and B is controlled so as to be data.

The color gain control unit 73 acquires the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen of the color video signal for each frame or a plurality of frames, and the color as described above. The gain is calculated and the color gain is determined.
Then, the color gain setting unit 62 multiplies the determined color gain by the pixel value of each color of each pixel.
Then, the video output unit 6 outputs a color video signal in which the color gain is multiplied by the color gain setting unit 62 and the variation degree based on the pixel value of each color is larger than the predetermined variation degree.

For example, when emphasizing human skin color, as described above, the ratio of the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen is such that B / G is more than R / G. And the color gain in the color gain setting unit 62 is set so that R / G is smaller than 1 and B / G is a predetermined ratio larger than 1, and based on the pixel value of each color. Good color reproducibility can be obtained by making the degree of variation larger than in the normal mode. For example, it is preferable to set the color gain by using values such as R / G = 0.66 and B / G = 1.46 as target values.
Specifically, for example, when the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen are all 100, the color gain control unit 73 sets the R color gain to 0.66. And G color gain is 1.00 and B color gain is 1.46. Then, the color gain setting unit 62 multiplies each color data determined by the color gain control unit 73 with each color gain for the video signal input to the video output unit 6, thereby obtaining the average value of the pixel values of the entire screen. R / G = 0.66 and B / G = 1.46. Even when the plastic color is regarded as important, the color can be reproduced satisfactorily by setting the above-mentioned value as the target value.

  By doing as described above, a video signal with good color reproducibility can be obtained even in the night vision mode.

As described above, the color gain obtained by adjusting the white balance in the normal mode is different from the color gain in the night vision mode. The color gain control unit 73 switches processing in response to the mode switching unit 72 switching between the normal mode and the night vision mode.
In addition, a person detection unit (not shown) that detects a person using a known pattern matching technique is provided, and when a person is detected, the above-described gain setting that emphasizes the human skin color is used to detect the person. If not, other gain settings may be used.

In the night vision mode, another method may be used as a method in which the video output unit 6 outputs a color video signal in which the degree of variation based on the pixel value of each color is larger than a predetermined degree of variation. The method will be described below.
In the following description, for convenience of description, it is assumed that the color gain that the color gain setting unit 62 multiplies the video signal is the same as the color gain in the normal mode. That is, it is assumed that the color gain does not contribute to the degree of variation between the R pixel value, the G pixel value, and the B pixel value.
In the method described below, the degree of variation in the values of the R pixel value, the G pixel value, and the B pixel value is controlled by controlling the intensity when the infrared projector 9 projects infrared light with wavelengths IR1 to IR3. Or, the ratio of the R pixel value, the G pixel value, and the B pixel value is controlled.
An example in which the intensity of infrared light with wavelengths IR1 to IR3 is not changed according to the pixel value of the captured image and an example in which the intensity of each infrared light with wavelengths IR1 to IR3 is changed according to the pixel value of the captured image will be described. Do

First, a case where the intensity of each infrared light of the wavelengths IR1 to IR3 is not changed according to the pixel value of the captured image will be described. In this case, the light projection control unit 71 projects the degree of variation of the R pixel value, the G pixel value, and the B pixel value when each infrared light having the wavelengths IR1 to IR3 is projected and the standard white plate is imaged. When the standard white plate is imaged under the reference light source in the normal mode, a predetermined value that is larger than the degree of variation in the values of the R pixel value, the G pixel value, and the B pixel value in the pixels in the region corresponding to at least the standard white plate The infrared projector 9 is controlled so as to project each infrared light with the intensity.
By controlling the intensity of the infrared light in such a manner, the video output unit 6 outputs a color video signal in which the degree of variation based on the pixel value of each color is larger than the predetermined degree of variation.
That is, when the standard white plate is imaged under the reference light source in the normal mode, the R pixel value, the G pixel value, and the B pixel value are equal, but in the night vision mode, each of the wavelengths IR1 to IR3 has the above-described intensity. Since infrared light is projected, the R pixel value, the G pixel value, and the B pixel value are not equal.
When the intensity of each infrared light of the wavelengths IR1 to IR3 is not changed according to the pixel value of the photographed image in the night vision mode, the intensity (current value, etc.) of each infrared light for projecting at such an intensity is set. The light projection control unit 71 controls the infrared light projector 9 so that light is projected at the intensity.

A plurality of infrared light intensities may be prepared according to the subject.
For example, as described above, when emphasizing the color of human skin, when a standard white plate is imaged by projecting each infrared light of wavelengths IR1 to IR3, at least the region corresponding to the standard white plate In the pixel, the ratio of the R pixel value, the G pixel value, and the B pixel value is such that B / G is larger than R / G, R / G is smaller than 1, and B / G is larger than 1. By controlling the intensity of each infrared light so that it becomes a predetermined ratio and making the degree of variation based on the pixel value of each color larger than in the normal mode, the infrared light projector 9 causes infrared light with wavelengths IR1 to IR3 to be emitted. The color of the person's skin is well reproduced when the light is projected.
For example, when a color gain is set such that an R pixel value, a G pixel value, and a B pixel value are equal when a standard white plate is imaged under a reference light source, infrared light of IR1 corresponding to R Is projected at a lower intensity than IR2 infrared light corresponding to G, and IR3 infrared light corresponding to B is projected at an intensity stronger than IR2 infrared light corresponding to G. Thus, the video signals of the respective colors can be set to about R / G = 0.66 and B / G = 1.46.

  Further, as described above, when importance is attached to the color of the plastic, when a standard white plate is imaged by projecting each infrared light of wavelengths IR1 to IR3, at least in a pixel in a region corresponding to the standard white plate. , The ratio of the R pixel value, the G pixel value, and the B pixel value is such that B / G is larger than R / G and that R / G and B / G are a predetermined ratio larger than 1. By controlling the intensity of the external light and making the degree of variation based on the pixel values of each color larger than in the normal mode, the color of the plastic can be obtained when imaging is performed by projecting infrared light of wavelengths IR1 to IR3. It is reproduced well.

  In the case where importance is attached to the color of the plastic, values of R / G and B / G when a plurality of colors such as red, green, and blue are expressed with good color reproducibility in a balanced manner may be adopted. When emphasizing the reproducibility of a specific color among a plurality of colors, the values of R / G and B / G when the specific color is expressed with good color reproducibility may be employed.

As described above, in the case where any color is emphasized, the intensity of each infrared light in the night vision mode is compared with the case where the color gain of each color is not changed according to the pixel value of the captured image in the normal mode. When the standard white plate is imaged by irradiating infrared light of wavelengths IR1, IR2, and IR3, the degree of variation in the R pixel value, G pixel value, and B pixel value is standard white under the reference light source in the normal mode. The intensity is greater than the degree of variation of the R pixel value, G pixel value, and B pixel value when the plate is imaged.
Further, compared with the case where the color gain of each color is changed according to the pixel value of the captured image in the normal mode, the intensity of each infrared light in the night vision mode is irradiated with infrared light of wavelengths IR1, IR2, and IR3. The variation degree of the R pixel value, the G pixel value, and the B pixel value when the standard white plate is imaged is R in the average value of the pixel values of the entire screen when the auto white balance process is executed in the normal mode. The intensity is larger than the degree of variation of the pixel value, the G pixel value, and the B pixel value.

In addition to skin color and plastic color, adjust the intensity of each infrared light to obtain the best color reproducibility while imaging the subject you want to emphasize, and then use the adjusted result to capture the image. can do. At that time, the color gain may be adjusted at the same time, and may be adjusted by the intensity of the infrared light and the color gain. In this way, even when importance is placed on colors other than skin color and plastic color, the R pixel value, G pixel value, and B pixel value when the standard white plate is imaged are not equal to a predetermined degree of variation. When it is increased, good color reproducibility can be obtained. When the degree of variation is expressed in terms of standard deviation, good color reproducibility can be obtained if the standard deviation calculated from the R pixel value, G pixel value, and B pixel value is greater than zero.
In addition, when the degree of variation is expressed as a difference value between the largest value and the smallest value among the R pixel value, the G pixel value, and the B pixel value, it is preferable that the difference value be larger than 0. Color reproducibility is obtained.

Next, a case where the intensity of each infrared light with wavelengths IR1 to IR3 is changed according to the pixel value of the captured image in the night vision mode will be described.
In this case, the light projection control unit 71 projects values of the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen imaged by projecting each infrared light of the wavelengths IR1 to IR3. The wavelengths IR1 to IR1 are set so that the variation degree becomes a predetermined value larger than the variation degree of the R pixel value, the G pixel value, and the B pixel value when the standard white plate is imaged under the reference light source in the normal mode. The intensity of each infrared light of IR3 is determined.
That is, in the night vision mode, each infrared light having such an intensity is projected, so the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen are not basically equal.

Further, compared to the case where the color gain of each color is changed according to the pixel value of the captured image in the normal mode, the light projection control unit 71 projects the entire screen imaged by projecting each infrared light of the wavelengths IR1 to IR3. The average value of the pixel values is the variation degree of the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel value of the entire screen after the auto white balance process is executed in the normal mode. The intensity of each infrared light of the wavelengths IR1 to IR3 is determined so as to be a predetermined value larger than the degree of variation in the average value of the R pixel value, the G pixel value, and the B pixel value.
That is, when auto white balance processing is executed in the normal mode, the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen are basically equal, but in the night vision mode, as described above. Since infrared light is projected with a high intensity, the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen are not equal.
In the case of a method of adjusting by adjusting the intensity of infrared light, the control unit 71 performs control so as not to perform the auto white balance process during the night vision mode. This is because when the auto white balance process is performed, the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen are equal even if the intensity of each infrared light is changed.

For example, when emphasizing the color of human skin, as described above, for example, the ratio of the R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values of the entire screen is such that B / G is R The light projection control unit 71 sets the intensity of the infrared light so that the predetermined ratio is larger than / G, R / G is smaller than 1, and B / G is larger than 1. Good color reproducibility can be obtained by making the degree of variation based on the pixel value of each color larger than in the normal mode. For example, it is preferable to set the intensity of each infrared light with target values such as R / G = 0.66 and B / G = 1.46.
Specifically, for example, when shooting a subject whose R pixel value, G pixel value, and B pixel value are all 100 in the average value of the pixel values of the entire screen when shooting in the normal mode, the light projection control unit 71 Projects IR1 infrared light corresponding to R at a lower intensity than IR2 infrared light corresponding to G, and IR3 infrared light corresponding to B to IR2 red corresponding to G. By projecting light with a stronger intensity than external light, the video signal of each color can be set to about R / G = 0.66 and about B / G = 1.46. Even when the plastic color is regarded as important, the color can be reproduced satisfactorily by setting the above-mentioned value as the target value.

When changing the intensity of each infrared light of the wavelengths IR1 to IR3 according to the pixel value of the captured image in the night vision mode, the light projection control unit 71 performs the entire screen of the color video signal for each one or a plurality of frames. The R pixel value, the G pixel value, and the B pixel value in the average value of the pixel values are acquired, the intensity as described above is calculated, and the intensity of each infrared light is determined. The intensity is adjusted as needed while feeding back the R pixel value, the G pixel value, and the B pixel value.
The infrared projector 9 projects each infrared light of the wavelengths IR1 to IR3 with the determined intensity, and the video output unit 6 has a variation degree based on the pixel value of each color larger than a predetermined variation degree. A color video signal is output.

By doing so, a color video signal with good color reproducibility can be obtained even in the night vision mode.
In addition, a person detection unit (not shown) that detects a person using a known pattern matching technique is provided, and when a person is detected, the above-described intensity setting that emphasizes the skin color of the person is used to detect the person. If not, other strength settings may be used.

  The color gain setting unit 62 controls the color gain multiplied by R, G, and B of the color video signal, and the R pixel value, the G pixel value, and the B pixel value of the color video signal output from the video output unit 6 are controlled. A method of adjusting the balance and the intensity of infrared light having wavelengths IR1 to IR3 are controlled to adjust the balance of the R pixel value, G pixel value, and B pixel value of the color video signal output from the video output unit 6. A method may be combined to realize the degree of dispersion of the R pixel value, the G pixel value, and the B pixel value, and the ratio of the R pixel value, the G pixel value, and the B pixel value.

  Next, the color adjustment of the color video signal in the normal mode and the night vision mode described above will be described in further detail with reference to FIGS. The following description is useful for setting the intensity and color gain value of each infrared light in the initial stage. It is also useful for quality inspections of imaging devices.

  FIGS. 29 to 31 schematically show an NTSC vector scope as an example. The horizontal axis is the color difference signal BY, and the vertical axis is RY. Y is a luminance signal. The circumferential direction of the circle indicated by the vector scope indicates the hue. R, G, and B, and yellow (Ye), magenta (Mg), and cyan (Cy) are expressed by positions indicated by hatched circles.

  When the white balance is adjusted as described above in the normal mode, an index indicating a color video signal output when an achromatic object (for example, a standard white plate) is imaged is a vector as shown by a white circle in FIG. Located in the center of the scope circle.

  As in the case of emphasizing human skin color in the night vision mode, the degree of variation in the values of the R pixel value, the G pixel value, and the B pixel value is increased, and the R pixel value, the G pixel value, and the B pixel value are increased. Is a predetermined ratio of R / G = 0.66 and B / G = 1.46, the index indicating the color video signal output when an achromatic object is imaged is shown in FIG. Move to the position indicated by the circle with cross-hatching. Photographing in a state in which the color gain or the intensity of each infrared light is adjusted so that white in the normal mode is moved to a position indicated by a circle with cross-hatching in FIG. 30. Color is reproduced with good color reproducibility.

As in the case of emphasizing the color of plastic in the night vision mode, the degree of variation of the R pixel value, the G pixel value, and the B pixel value is increased, and the R pixel value, the G pixel value, and the B pixel value are When the ratio is set to a predetermined ratio of R / G = 1.35 and B / G = 2.04, the index indicating the color video signal output when an achromatic object is imaged is shown in FIG. Move to the position indicated by the hatched circle. Photographing in a state in which the color gain or the intensity of each infrared light is adjusted so that white in the normal mode is moved to a position indicated by a circle with cross hatching in FIG. Color is reproduced with good color reproducibility.
As described above, the initial setting, quality inspection, and the like of the imaging apparatus can be performed based on the vector scope.

  FIG. 32 shows how the R / G and B / G values change in the night vision mode when an achromatic object (for example, a standard white plate) serving as a reference for determining white in the normal mode is photographed. Indicates what to do. In the normal mode, as indicated by black circles, R / G = 1.0 and B / G = 1.0.

  When emphasizing the color of human skin in the night vision mode, the degree of variation of the R pixel value, the G pixel value, and the B pixel value is increased, and the R pixel value, the G pixel value, and the B pixel value are Therefore, the black circle moves to the position of the black square because R / G = 0.66 and B / G = 1.46.

When importance is attached to the color of plastic in the night vision mode, the degree of variation of the R pixel value, the G pixel value, and the B pixel value is increased, and the ratio of the R pixel value, the G pixel value, and the B pixel value is increased. Is set to a predetermined ratio of R / G = 1.35 and B / G = 2.04, the black circle moves to the position of the black triangle.
It is also possible to perform initial setting, quality inspection, and the like of the imaging device based on the above graph and the like.

The adjustment methods described above are summarized as follows.
The normal mode is a state in which light from a reference light source is irradiated to an achromatic object (for example, a standard white plate) that serves as a reference for determining white at the time of adjustment. The color gain control unit 73 multiplies the video signal of each color in the color video signal by a color gain such that the three primary color video signals generated by the video processing unit 5 become white when the object is imaged. The gain setting unit 62 is controlled.

  In the night vision mode, at the time of adjustment, the infrared light of wavelengths IR1 to IR3 from the infrared projector 9 is projected onto an achromatic object. When the imaging unit 3 captures the object, the balance of the video signal of each color in the color video signal is adjusted so that the color video signal generated by the video processing unit 5 reproduces a specific color other than white.

  The balance of the video signals of the respective colors may be adjusted by appropriately setting the color gain to be multiplied by the video signals of the respective colors by the color gain setting unit 62, or the infrared light with wavelengths IR1 to IR3 projected by the infrared projector 9 You may adjust by setting the intensity | strength of this suitably.

Moreover, it can also be performed as follows.
The imaging unit 3 captures an image of an achromatic object serving as a reference for determining white and irradiates light from the reference light source to generate a first imaging signal. The video processing unit 5 generates R, G, B first three primary color video signals based on the first imaging signal.

  The color gain control unit 73 determines a color gain to be multiplied by the R, G, B color video signals such that the first three primary color video signals are white. This completes the adjustment in the normal mode.

  In a state where infrared light having wavelengths IR1 to IR3 is projected onto the same object as described above, the imaging unit 3 captures the object and generates a second imaging signal. The video processing unit 5 generates R, G, B second three primary color video signals based on the second imaging signal. The imaging device adjusts the balance of the video signals of the respective colors in the three primary color video signals so that the second three primary color video signals reproduce colors other than white. Thus, the adjustment in the night vision mode is completed.

  According to the imaging device and the adjustment method of the imaging device of the present embodiment, a video signal with good color reproducibility can be obtained in the night vision mode.

  By the way, in the imaging device of the present embodiment, the configuration in which infrared light with wavelengths IR1 to IR3 is selectively projected in a time division manner during imaging in the night vision mode is shown. The structure which projects light simultaneously may be sufficient.

  For example, if the image pickup unit 3 is a single plate type having a single image pickup device, a configuration in which infrared light with wavelengths IR1 to IR3 is simultaneously projected can be obtained by using a microfilter or the like. In addition, if the imaging unit 3 is a three-plate type having three imaging elements, a dichroic prism or the like can be used to simultaneously project infrared light with wavelengths IR1 to IR3.

In the case of the single-plate type, in the normal mode, the R component of visible light is incident on the R pixel, the G component of visible light is incident on the G pixel, and the B component of visible light is incident on the B pixel. In the night vision mode, the infrared light having the wavelength IR1 is incident on the R pixel, the infrared light having the wavelength IR2 is incident on the G pixel, and the infrared light having the wavelength IR3 is incident on the B pixel. What is necessary is just to comprise.
In the case of the three-plate type, in the normal mode, the dichroic prism is used to capture the R component of visible light for the image sensor corresponding to R, the G component of visible light for the image sensor corresponding to G, and the image corresponding to B. The B component of visible light is incident on the element, and in the night vision mode, the dichroic prism causes infrared light having a wavelength IR1 to be applied to an image sensor corresponding to R, and red having a wavelength IR2 to an image sensor corresponding to G. What is necessary is just to comprise so that the infrared light of wavelength IR3 may inject into the image pick-up element corresponding to external light and B. FIG.

In the present embodiment, the degree of variation based on the R pixel value, the G pixel value, and the B pixel value is greater than the predetermined variation degree, and the ratio of the R pixel value, the G pixel value, and the B pixel value is as described above. The color video signal is output so that the ratio becomes the ratio, but what kind of substance is used and what ratio is arbitrary.
For example, you may change suitably according to the property of an image sensor, and the processing method which produces | generates a color video signal. The original color may be difficult to reproduce depending on the material of the material, but the original color of the subject or a color equivalent to it is good while looking at the color video signal generated by actually imaging while projecting infrared light It is good to adjust it to reproduce. At this time, if the degree of variation based on the R pixel value, the G pixel value, and the B pixel value is larger than a predetermined variation degree, a good color can be reproduced.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

For example, various adjustment methods are conceivable. As a result of adjustment, if the output color video signal has similar characteristics, the same effect can be exhibited.
In addition, settings such as color gain and infrared light intensity may be switchable between a setting that places importance on human skin color and a setting that places importance on plastic color. At this time, the operation of the color gain control unit 73 and the color gain setting unit 62 may be switched by detecting a state in which the human skin color should be emphasized, a state in which the plastic color is important. For example, a subject detection unit (not shown) for detecting a subject may be provided, and switching may be automatically performed according to the detection result. For example, the type of subject may be detected and automatically switched. That is, the color gain setting unit 62 changes the color gain multiplied by each pixel value of the color video signal according to the subject, and the ratio of the pixel values between the colors in the color video signal output by the video output unit 6 is determined by the subject detection unit. You may make it change according to the detection result by.
For example, a subject detection unit that detects a subject using a well-known pattern matching technique may be provided to detect whether or not the subject is a person and switch automatically.
For example, the subject detection unit may be a person detection unit or a face detection unit. When a face or person is detected, the color gain setting places importance on the skin color of the person, and in other cases, other colors The gain can be set. In addition, when a car or the like is detected using a known pattern matching technique, the setting may be made with emphasis on metal.
In addition, the color gain may be different between an area where a specific subject is detected in the screen and another area. For example, the area where the face is detected in the screen may be set as a color gain setting that emphasizes the color of human skin, and the other areas may be set as other color gain settings. At least the pixels in the area corresponding to the detected subject may be set so that the color of the subject is emphasized.

Further, similarly to the above, the operation of the light projection control unit 71 may be switched by detecting a state in which the human skin color should be emphasized, a state in which the plastic color is important. For example, a subject detection unit (not shown) for detecting a subject may be provided, and switching may be automatically performed according to the detection result. For example, the type of subject may be detected and automatically switched. That is, the light projection control unit 71 changes the intensity of each infrared light according to the subject, and the ratio of the pixel values between the colors in the color video signal output from the video output unit 6 depends on the detection result by the subject detection unit. May be changed.
For example, a subject detection unit that detects a subject using a well-known pattern matching technique may be provided to detect whether or not the subject is a person and switch automatically.
For example, the subject detection unit may be a person detection unit or a face detection unit. When a face or a person is detected, the intensity of infrared light that emphasizes the color of the person's skin is set. Otherwise, other Infrared light intensity may be used. In addition, when a car or the like is detected using a well-known pattern matching technique, the intensity of infrared light that places importance on metal may be used.
Further, as described above, the color gain to be multiplied by the color video signal is controlled by the color gain setting unit 62, and the R pixel value, G pixel value, and B pixel value of the color video signal output from the video output unit 6 are balanced. And a method of adjusting the balance of the R pixel value, the G pixel value, and the B pixel value of the color video signal output from the video output unit 6 by controlling the intensity of infrared light with wavelengths IR1 to IR3. May be combined to realize the degree of variation in the values of the R pixel value, the G pixel value, and the B pixel value, and the ratio of the R pixel value, the G pixel value, and the B pixel value.

  The video processing unit 5, the video output unit 6 and the control unit 7 can be realized by one or a plurality of hardware processors. It can also be realized by cooperation of one or a plurality of hardware processors and software. Moreover, you may combine each embodiment, an Example, a modification, etc.

3 Imaging unit 5 Image processing unit 9 Infrared projector 62 Color gain setting unit 71 Projection control unit 73 Color gain control unit

Claims (12)

A projection control unit that controls an infrared projector that projects infrared light;
An imaging unit that images an object and generates an imaging signal in a state where the infrared projector is controlled to project infrared light; and
A video processing unit that generates a color video signal based on the imaging signal;
A video output unit for outputting a color video signal;
A subject detection unit for detecting a subject, and
The subject detection unit detects whether the subject is a person,
When the subject detection unit detects that the subject is a person,
In the video output unit, the ratio of the pixel values between the colors is larger than the ratio of the red pixel value to the green pixel value, and the ratio of the blue pixel value to the green pixel value is larger than the green pixel value. A color video signal having a predetermined ratio in which a ratio of a red pixel value to a pixel value is smaller than 1 and a ratio of a blue pixel value to a green pixel value is greater than 1 is output. An imaging device.   A projection control unit that controls an infrared projector that projects infrared light;
  An imaging unit that images an object and generates an imaging signal in a state where the infrared projector is controlled to project infrared light; and
  A video processing unit that generates a color video signal based on the imaging signal;
  A video output unit for outputting a color video signal;
  A subject detection unit for detecting the subject;
  With
  The subject detection unit detects whether the subject is plastic,
  The ratio of pixel values between colors in the color video signal output by the video output unit varies depending on whether or not the subject detection unit detects plastic.
  An imaging apparatus characterized by that. When the subject detection unit detects a plastic subject,
The video output unit, the ratio between the pixel values of each color is larger than the value of the ratio of the red pixel value for the value a green pixel value of the ratio of the blue pixel values for the green pixel value, and, greater than the value of the ratio is 1 red pixel value for the green pixel value, and the value of the ratio of the blue pixel values to output a color image signal is one greater than the predetermined ratio green pixel value The imaging apparatus according to claim 2 .   A projection control unit that controls an infrared projector that projects infrared light;
  An imaging unit that images an object and generates an imaging signal in a state where the infrared projector is controlled to project infrared light; and
  A video processing unit that generates a color video signal based on the imaging signal;
  A video output unit for outputting a color video signal;
  A subject detection unit for detecting the subject;
  With
  The subject detection unit detects whether or not the subject is a car,
  The ratio of pixel values between colors in the color video signal output by the video output unit varies depending on whether or not the subject detection unit detects a car.
  An imaging apparatus characterized by that. A color gain setting unit for multiplying a pixel value of each color of the color video signal generated by the video processing unit by a color gain;
With
The video output unit outputs a color video signal in which a predetermined color gain is multiplied by the color gain setting unit, and a ratio of pixel values between colors is set to a ratio according to a detection result by the subject detection unit. the imaging apparatus according to any one of claims 1 to 4, characterized in. The light projection control unit is configured to output an intensity of each infrared light corresponding to each color of the color video signal so that the video output unit outputs a color video signal having a degree of variation based on a pixel value of each color larger than a predetermined degree of variation. Control
In the video output unit, the intensity of each infrared light corresponding to each color of the color video signal is controlled by the light projection control unit, and the ratio of pixel values between the colors is a ratio according to the detection result by the subject detection unit. the imaging apparatus according to any one of claims 1-5, characterized in that for outputting a color image signal with. The ratio of the pixel values between the colors is set such that the ratio of the pixel values between the colors corresponds to the detection result by the subject detection unit at least in the pixel corresponding to the subject detected by the subject detection unit. The imaging apparatus according to any one of claims 1 to 6 , wherein the imaging apparatus is characterized in that: The ratio between pixel values of each color in the average value of the pixel values of the entire screen, according to any one of claims 1 to 7, characterized in that there is a ratio in accordance with the detection result of the subject detection unit Imaging device. A projection control step for controlling an infrared projector that projects infrared light;
An imaging step of imaging an object and generating an imaging signal in a state where the infrared projector is controlled to project infrared light; and
A video processing step for generating a color video signal based on the imaging signal;
A video output step for outputting a color video signal;
A subject detection step for detecting a subject, and
The subject detection step detects whether the subject is a person,
When the subject detection step detects that the subject is a person,
In the video output step, the ratio of the pixel values between the colors is such that the ratio of the blue pixel value to the green pixel value is larger than the ratio of the red pixel value to the green pixel value, and A color video signal having a predetermined ratio in which a ratio of a red pixel value to a pixel value is smaller than 1 and a ratio of a blue pixel value to a green pixel value is greater than 1 is output. Imaging method.   A projection control step for controlling an infrared projector that projects infrared light;
  An imaging step of imaging an object and generating an imaging signal in a state where the infrared projector is controlled to project infrared light; and
  A video processing step for generating a color video signal based on the imaging signal;
  A video output step for outputting a color video signal;
  A subject detection step for detecting a subject;
  With
The subject detection step detects whether the subject is a car,
  The ratio of pixel values between colors in the color video signal output by the video output step varies depending on whether or not the subject detection step detects a car.
  An imaging method characterized by the above. Computer
Projection control means for controlling an infrared projector that projects infrared light,
Video processing means for generating a color video signal based on an imaging signal generated by imaging a subject in a state where the infrared projector is controlled to project infrared light;
Video output means for outputting a color video signal;
Subject detection means for detecting the subject;
Function as
The subject detection means detects whether the subject is a person,
When the subject detection means detects that the subject is a person,
In the video output means, the ratio of the pixel values between the colors is such that the ratio of the blue pixel value to the green pixel value is larger than the ratio of the red pixel value to the green pixel value, and A color video signal having a predetermined ratio in which a ratio of a red pixel value to a pixel value is smaller than 1 and a ratio of a blue pixel value to a green pixel value is greater than 1 is output. An imaging program.   Computer
  Projection control means for controlling an infrared projector that projects infrared light;
  Imaging means for imaging an object and generating an imaging signal in a state where the infrared projector is controlled to project infrared light; and
  Video processing means for generating a color video signal based on the imaging signal;
  Video output means for outputting a color video signal;
  Subject detection means for detecting the subject;
  Function as
  The subject detection means detects whether or not the subject is a car;
  The ratio of pixel values between colors in the color video signal output by the video output means varies depending on whether or not the subject detection means detects an automobile.
  An imaging program characterized by the above.

Images