WO2017042980A1 - Fluoroscopic apparatus and fluoroscopic endoscope apparatus - Google Patents

Abstract

This fluoroscopic apparatus comprises: a light source device capable of simultaneously emitting excitation light of a first wavelength range that is emitted to a fluorescent body and causes the fluorescent body to emit fluorescence, visible light of a second wavelength range different from the first wavelength range and visible light of a third wavelength range different from the first wavelength range and the second wavelength range; an imaging device in which a plurality of light receiving elements are arranged, the light receiving elements being provided with any of a first filter that selectively allows fluorescence to pass therethrough, a second filter that selectively allows visible light of the second wavelength range to pass therethrough and a third filter that selectively allows visible light of the third wavelength range to pass therethrough; and an image processing device that performs image processing on the basis of pixel signals indicating that the light receiving elements have detected light corresponding thereto.

Description

Fluorescence observation apparatus and fluorescence observation endoscope apparatus

The present invention relates to a fluorescence observation apparatus and a fluorescence observation endoscope apparatus.

Conventionally, in order to diagnose cancer and the like, a diagnostic method for determining the presence or absence of a lesion by previously administering a derivative-labeled antibody (fluorescent drug) called ICG (indocyanine green) into the body of a subject to be examined is known. ing. ICG is a fluorescent substance having affinity for lesions such as cancer, and is excited by light in the infrared region to emit fluorescence. Conventionally, various medical systems having a function of observing ICG fluorescence are proposed. Note that, for example, a practitioner such as a doctor determines the presence or absence of a lesion from the brightness of fluorescent emission observed by a medical system.

In a conventional medical system, as a configuration for observing fluorescence emission, light in the infrared region such as near infrared light is irradiated as excitation light for exciting the ICG, and fluorescence is emitted by the irradiated excitation light. It is equipped with a fluorescence observation device that contrasts a specific protein in the lesion that has emitted light.

For example, Patent Document 1 discloses an endoscope apparatus that can perform fluorescence observation using excitation light in addition to normal observation using visible light. In the endoscope apparatus disclosed in Patent Document 1, visible light and excitation light are irradiated on the object to be inspected from the distal end portion of the insertion portion, and visible light and excitation light reflected from the object to be inspected, and excitation by the excitation light. As a result, the fluorescence emitted by the ICG is guided to the camera head through the image guide fiber. In the endoscope apparatus disclosed in Patent Document 1, first, visible light, excitation light, and fluorescence guided to the camera head are converted into visible light and excitation light by a dichroic mirror provided in the camera head. And fluorescent. The visible light separated here is imaged by an imaging means. In the endoscope apparatus disclosed in Patent Document 1, the separated excitation light and fluorescence are removed (cut) by the excitation light cut filter provided in the camera head, and only the fluorescence is image intensities. Amplified by the fire and imaged by an imaging means different from the imaging means for visible light.

However, in the configuration of the endoscope apparatus disclosed in Patent Document 1, that is, the configuration in which light is separated by a dichroic mirror, visible light imaging and fluorescence imaging can be performed simultaneously by each imaging means. However, the configuration of the imaging unit, that is, the imaging unit including the dichroic mirror, the imaging unit, and the like becomes large. For this reason, the structure of the imaging part disclosed by patent document 1 cannot be mounted in the front-end | tip part of an endoscope apparatus. For this reason, it is difficult for the technique of the endoscope apparatus disclosed in Patent Document 1 to realize an endoscope apparatus having a configuration in which a wide range of observation is performed by a function of bending the distal end portion.

Conventionally, for example, as in Patent Document 2, an on-chip color filter that transmits light (visible light) in the red (R), green (G), and blue (B) wavelength bands used in normal photographing is used. In addition, a solid-state imaging device with an on-chip color filter that transmits light in the near-infrared wavelength band (near-infrared light) has been proposed. Therefore, it is conceivable to use the solid-state imaging device disclosed in Patent Document 2 for observation of fluorescence in an endoscope apparatus. In this case, unlike the endoscope device disclosed in Patent Document 1, it is not necessary to provide a dichroic mirror in the imaging unit, and the imaging unit can be mounted on the distal end portion of the endoscope device. It can be downsized.

Japanese Patent No. 3962122 Japanese Unexamined Patent Publication No. 2008-123194

However, the emission of fluorescence by ICG is very weak. For this reason, in the solid-state imaging device disclosed in Patent Document 2, an on-chip color filter that transmits near-infrared light (hereinafter referred to as “IR filter”) is attached to a pixel (hereinafter referred to as “IR pixel”). In order to detect fluorescence by ICG with high accuracy, it is necessary to make the detection amount (light reception amount) of visible light detected by the IR pixel sufficiently smaller than the detection amount (light reception amount) of fluorescence. Therefore, the IR filter attached to the IR pixel needs to have a characteristic of sufficiently attenuating visible light by increasing the attenuation factor with respect to visible light.

Here, an example of the IR filter characteristic necessary for the IR pixel to accurately detect fluorescence by ICG will be described. For example, the amount of visible light received by an IR pixel does not affect the detection of fluorescence, that is, the amount of visible light received that can be ignored in fluorescence detection is incident on the solid-state imaging device by 1/10 times the amount of received fluorescence. Assume that the ratio of the intensity of visible light to the intensity of fluorescence is assumed to be 15: 1, and the ratio of the sensitivity of visible light to fluorescence in the pixels provided in the solid-state imaging device is 2: 1. In this case, the IR filter must attenuate the incident visible light to 1 / (10 × 15 × 2) times = 0.3%.

However, when an IR filter is formed of an organic material that is generally used as a material for an on-chip color filter, several% of light in the visible wavelength band passes through the IR filter. For this reason, if the solid-state imaging device disclosed in Patent Document 2 is used for observing fluorescence in an endoscope apparatus, the fluorescence detection sensitivity is significantly deteriorated.

The present invention has been made based on the above-described problems, and is a fluorescence observation apparatus and fluorescence observation endoscopy that can realize downsizing of an imaging system and detection of fluorescence excited by a fluorescent substance with high accuracy. The object is to provide a mirror device.

The fluorescence observation apparatus according to the first aspect of the present invention includes excitation light in a first wavelength band that emits fluorescence by irradiating a phosphor, and visible light in a second wavelength band different from the first wavelength band. A light source device capable of simultaneously irradiating light and visible light having a third wavelength band different from the first wavelength band and the second wavelength band; and a first light source that selectively transmits the fluorescence. Or a third filter that selectively transmits visible light in the third wavelength band, or a second filter that selectively transmits visible light in the second wavelength band. An image pickup apparatus in which a plurality of light receiving elements are arranged, and an image processing apparatus that performs image processing based on a pixel signal in which light corresponding to each of the light receiving elements is detected.

According to the second aspect of the present invention, in the fluorescence observation apparatus according to the first aspect, the imaging device reflects light in the third wavelength band from the second wavelength band, and the wavelength of the fluorescence. A dielectric multilayer filter that transmits light in a band may be further included, and the dielectric multilayer filter may be disposed on the optical path of the first filter.

According to the third aspect of the present invention, in the fluorescence observation apparatus according to the first aspect, the intensity of the excitation light emitted by the light source device is higher than the intensity of each visible light emitted by the light source device. It may be strong.

According to a fourth aspect of the present invention, in the fluorescence observation apparatus according to the first aspect, the fluorescence observation apparatus is disposed on an optical path from the subject that irradiates the excitation light to the imaging apparatus, and the excitation light is the light receiving element. An excitation light cut filter that prevents the light from entering the light source may be further included.

According to a fifth aspect of the present invention, in the fluorescence observation apparatus according to the first aspect, the fluorescence includes a wavelength of near infrared light, and the visible light in the second wavelength band is green light. The visible light in the third wavelength band including a wavelength may include a wavelength of blue light.

According to a sixth aspect of the present invention, in the fluorescence observation device according to the first aspect, the image processing device converts light detected by the light receiving element from the first filter through the third filter. Image processing may be performed based on a signal level obtained by multiplying each of the corresponding pixel signals by a predetermined value.

According to a seventh aspect of the present invention, in the fluorescence observation apparatus according to the first aspect, the imaging device is configured to determine a wavelength band through which each of the third filter selectively transmits from the first filter. A plurality of light receiving elements provided with a fourth filter that selectively transmits visible light in different fourth wavelength bands are further disposed, and the light source device has the third wavelength band to the third wavelength band. Fluorescence imaging in which light is emitted and the image processing device performs the image processing based on each of the pixel signals corresponding to the light detected by the light receiving element from the first filter through the third filter Mode, the light source device emits light of the fourth wavelength band from the second wavelength band, and the image processing device transmits the light receiving element from the second filter through the fourth filter. Light detected by Normal mode and a mode switching device for switching the performing the image processing based on each of the pixel signals corresponding, may further include a.

According to an eighth aspect of the present invention, in the fluorescence observation apparatus according to the seventh aspect, the fluorescence includes a wavelength of near-infrared light, and the visible light in the second wavelength band is green light. The visible light of the third wavelength band may include a wavelength of blue light, and the visible light of the fourth wavelength band may include a wavelength of red light.

According to a ninth aspect of the present invention, in the fluorescence observation apparatus according to the seventh aspect, the image processing device converts the light detected by the light receiving element from the second filter through the fourth filter. For each of the corresponding pixel signals, an offset process may be performed based on each of the pixel signals corresponding to the light detected by the light receiving element via the first filter.

According to the tenth aspect of the present invention, in the fluorescence observation apparatus according to the first aspect, the first filter to the third filter may be formed of an organic material.

According to the eleventh aspect of the present invention, in the fluorescence observation apparatus according to the seventh aspect, the fourth filter may be formed of an organic material.

A fluorescence observation endoscope apparatus according to a twelfth aspect of the present invention is a fluorescence observation endoscope apparatus that performs a diagnosis using an endoscope by administering a fluorescent substance comprising an indocyanine green derivative-labeled antibody to a living body, Excitation light in a first wavelength band that emits fluorescence by irradiating the fluorescent substance administered to a living body, visible light in a second wavelength band different from the first wavelength band, and the first wavelength A light source device capable of simultaneously irradiating a visible light of a third wavelength band different from the band and the second wavelength band, a first filter that selectively transmits the fluorescence, and the second wavelength Imaging in which a plurality of light receiving elements provided with either a second filter that selectively transmits visible light in a band or a third filter that selectively transmits visible light in the third wavelength band are arranged Each of the device and the light receiving element It includes an image processing apparatus that performs image processing on the basis of the corresponding pixel signal which detected the light.

According to a thirteenth aspect of the present invention, in the fluorescence observation endoscope apparatus according to the twelfth aspect, the imaging device is disposed in an insertion portion that is inserted into the body of the living body in the fluorescence observation endoscope apparatus. May be.

According to a fourteenth aspect of the present invention, in the fluorescence observation endoscope apparatus according to the twelfth aspect, the fluorescence includes a wavelength of near infrared light, and the visible light in the second wavelength band is green. The visible light of the third wavelength band may include the wavelength of blue light.

According to a fifteenth aspect of the present invention, in the fluorescence observation endoscope apparatus according to any one of the twelfth aspect to the fourteenth aspect, the imaging apparatus is configured to move the first filter to the third filter. A plurality of light receiving elements provided with a fourth filter that selectively transmits visible light in a fourth wavelength band different from the wavelength band through which each of the filters selectively transmits, the light source device, The light of the third wavelength band is irradiated from the first wavelength band, and the image processing device responds to the light detected by the light receiving element from the first filter via the third filter. Fluorescence imaging mode for performing the image processing based on each of the pixel signals, the light source device irradiates light in the fourth wavelength band from the second wavelength band, and the image processing apparatus From the filter A normal shooting mode for performing the image processing based on each of the pixel signals corresponding to the light receiving element is detected through the fourth filter, the mode switching device for switching the may further include a.

According to a sixteenth aspect of the present invention, in the fluorescence observation endoscope apparatus according to the fifteenth aspect, the fluorescence includes a wavelength of near infrared light, and the visible light in the second wavelength band is green. The visible light of the third wavelength band may include the wavelength of blue light, and the visible light of the fourth wavelength band may include the wavelength of red light.

According to a seventeenth aspect of the present invention, in the fluorescence observation endoscope apparatus according to the fifteenth aspect, the image processing device detects the light receiving element from the second filter through the fourth filter. An offset process may be performed on each of the pixel signals corresponding to the received light based on each of the pixel signals corresponding to the light detected by the light receiving element via the first filter.

According to each aspect described above, it is possible to provide a fluorescence observation apparatus and a fluorescence observation endoscope apparatus that can realize downsizing of an imaging system and detection of fluorescence excited by a fluorescent substance with high accuracy.

It is the block diagram which showed schematic structure of the fluorescence observation endoscope apparatus in embodiment of this invention. It is the block diagram which showed schematic structure of the fluorescence observation endoscope apparatus of embodiment of this invention. It is the block diagram which showed schematic structure of the solid-state imaging device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped. It is sectional drawing which showed an example of the structure of the solid-state imaging device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped. It is the figure which showed typically an example of arrangement | positioning of the on-chip color filter stuck on the pixel of the solid-state imaging device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped. It is the figure which showed typically an example of arrangement | positioning of the dielectric multilayer filter corresponding to the pixel of the solid-state imaging device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped. It is the figure which showed an example of the characteristic of each filter corresponding to the pixel of the solid-state imaging device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped. It is the figure which showed an example of the characteristic of the dielectric multilayer filter corresponding to the pixel of the solid-state imaging device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped. It is the figure which showed typically an example of the characteristic of the light which the light source device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped light-emitted. It is the figure which showed typically an example of the characteristic of the light which the light source device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped light-emitted. It is the figure which showed typically an example of the operation | movement of the image process in the fluorescence observation endoscope apparatus of embodiment of this invention. It is the figure which showed typically an example of the operation | movement of the image process in the fluorescence observation endoscope apparatus of embodiment of this invention. It is the figure which showed typically an example of the operation | movement of the image process in the fluorescence observation endoscope apparatus of embodiment of this invention. It is the figure which showed typically an example of the operation | movement of the image process in the fluorescence observation endoscope apparatus of embodiment of this invention. It is the figure which showed typically an example of the operation | movement of the image process in the fluorescence observation endoscope apparatus of embodiment of this invention. It is the figure which showed typically an example of the operation | movement of the image process in the fluorescence observation endoscope apparatus of embodiment of this invention. It is the figure which showed typically an example of the operation | movement of the image process in the fluorescence observation endoscope apparatus of embodiment of this invention. It is the figure which showed typically an example of the operation | movement of the image process in the fluorescence observation endoscope apparatus of embodiment of this invention. It is sectional drawing which showed an example of another structure of the solid-state imaging device with which the fluorescence observation endoscope apparatus of embodiment of this invention was equipped.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, a case where the fluorescence observation apparatus of the present invention is configured as a fluorescence observation endoscope apparatus will be described. FIG. 1 is a configuration diagram showing a schematic configuration of a fluorescence observation endoscope apparatus according to an embodiment of the present invention.

1, the fluorescence observation endoscope apparatus 1 includes an endoscope scope unit 10, a light source device 20, an external processing unit 30, and a color monitor 40. The fluorescence observation endoscope apparatus 1 is, for example, an endoscope apparatus for laparoscopic surgery. In the fluorescence observation endoscope apparatus 1, the insertion portion of the endoscope scope unit 10 into the body is inserted into the abdomen 900 of the subject to be inspected, and the subject 901 to be inspected, such as a biological tissue in the subject of the subject to be examined. Take a picture. FIG. 1 shows a state in which the insertion portion of the endoscope scope portion 10 constituting the fluorescence observation endoscope apparatus 1 is inserted into the abdomen 900 of the subject to be examined and the subject 901 is imaged. .

The fluorescence observation endoscope apparatus 1 is used for a test subject in a state in which a derivative-labeled antibody (fluorescent drug) such as ICG has been administered into the body in advance. In the following description, it is assumed that ICG is administered as a fluorescent agent in the body of the subject to be examined.

The fluorescence observation endoscope apparatus 1 excites an imaging mode (hereinafter referred to as “normal imaging mode”) for imaging the object 901 by visible light and ICG administered by irradiation with excitation light such as near infrared light. And an imaging mode (hereinafter, referred to as “fluorescence imaging mode”) for imaging the object 901 by fluorescence emitted by fluorescence.

In the fluorescence observation endoscope apparatus 1, the endoscope scope unit 10 includes an insertion unit 11 and an operation unit 12. In the fluorescence observation endoscope device 1, the operation unit 12 of the endoscope scope unit 10 and the light source device 20 are connected by an optical signal cable 50. In the fluorescence observation endoscope apparatus 1, the operation unit 12 of the endoscope scope unit 10 and the external processing unit 30 are connected by an electric signal cable 60.

In the endoscope scope unit 10, the insertion unit 11 is inserted into the abdomen 900 or the like of the subject to be inspected, and an image of the inspection object 901 is taken. At this time, the illumination light guided through the optical signal cable 50 is irradiated from the distal end portion of the insertion portion 11 to the device 901 to be inspected. The endoscope scope unit 10 outputs an imaging signal corresponding to the captured image of the inspected object 901 to the external processing unit 30 via the electric signal cable 60.

The insertion unit 11 is inserted into the body of the subject from the abdomen 900 of the subject under test in which ICG has been administered in advance. In the endoscope scope unit 10, the insertion unit 11 includes an imaging unit 13 at the distal end. The imaging unit 13 generates an imaging signal obtained by converting an image of the inspection object 901 into an electrical signal. Then, the imaging unit 13 outputs the generated imaging signal to the external processing unit 30 via the insertion unit 11, the operation unit 12, and the electric signal cable 60.

The operation unit 12 is, for example, a support unit that controls the operation of the insertion unit 11 and the imaging unit 13 when operated by an examination practitioner (for example, a doctor performing laparoscopic surgery). In the endoscope scope unit 10, the operation unit 12 includes an imaging mode switching switch 14 for switching an imaging mode in the fluorescence observation endoscope apparatus 1. The imaging mode switching switch 14, for example, outputs an electrical signal indicating an imaging mode of the fluorescence observation endoscope apparatus 1 that is switched by an operation of the examination operator via the operation unit 12 and the electrical signal cable 60. Output to.

The light source device 20 emits illumination light that irradiates the inspection object 901 when the inspection object 901 is observed in the fluorescence observation endoscope apparatus 1. Illumination light emitted from the light source device 20 is guided to the operation unit 12 of the endoscope scope unit 10 through the optical signal cable 50, and irradiated to the object 901 from the imaging unit 13 provided at the distal end of the insertion unit 11. Is done. The light source device 20 emits illumination light in a wavelength band corresponding to the photographing mode in the fluorescence observation endoscope device 1. More specifically, when the fluorescence observation endoscope apparatus 1 images the object 901 in the normal imaging mode, the light source device 20 emits visible light as illumination light. In addition, when the fluorescence observation endoscope apparatus 1 images the object 901 in the fluorescence imaging mode, the light source device 20 emits illumination light including excitation light.

The external processing unit 30 performs predetermined image processing on the imaging signal of the inspected object 901 taken by the imaging unit 13 included in the endoscope scope unit 10 that is input via the electric signal cable 60. This is an image processing apparatus that generates an image including a photographed object to be inspected 901. The external processing unit 30 outputs the image signal of the image including the generated inspected object 901 to the color monitor 40 for display. In addition, the external processing unit 30 transmits a control signal (drive signal) when the imaging unit 13 captures the object 901 to the imaging unit 13 via the electric signal cable 60.

The color monitor 40 is a display device such as a liquid crystal display (LCD) that displays an image including the inspected object 901 according to the image signal input from the external processing unit 30.

With such a configuration, the fluorescence observation endoscope apparatus 1 performs imaging of the inspected object 901 with visible light and imaging of the inspected object 901 with fluorescence obtained by exciting the ICG administered to the subject to be inspected with excitation light. Do. That is, the fluorescence observation endoscope apparatus 1 performs photographing in the normal photographing mode and photographing in the fluorescent photographing mode. Then, the fluorescence observation endoscope apparatus 1 presents an image including the photographed object 901 to the examination practitioner.

Note that, as described above, since the fluorescence observation endoscope apparatus 1 is an endoscope apparatus for laparoscopic surgery, basically, external light such as visible light does not enter. Therefore, in the following description, when in the fluorescence imaging mode, the abdomen 900 and the insertion unit 11 are completely closed, and external light from the outside of the abdomen 900 in a working environment where the fluorescence observation endoscope apparatus 1 is used. (Environment light) is described as not leaking into the object 901 to be inspected. That is, in the fluorescence imaging mode, description will be made on the assumption that light other than illumination light emitted from the light source device 20 is not irradiated on the inspection object 901. However, in the normal imaging mode, preparations for observing the object 901 to be inspected may be made. Depending on the working environment in which the fluorescence observation endoscope apparatus 1 is used, ambient light from the outside of the abdomen 900 may be used. May leak. For this reason, in the normal photographing mode, it is assumed that ambient light including near infrared light leaks toward the object 901 to be inspected.

Next, a more detailed configuration of the fluorescence observation endoscope apparatus 1 will be described. FIG. 2 is a block diagram showing a schematic configuration of the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. In FIG. 2, the imaging unit 13 provided at the distal end portion of the endoscope scope unit 10 constituting the fluorescence observation endoscope apparatus 1 includes an imaging lens 130, an excitation light cut filter 131, an image sensor 132, and a light guide. 51.

The light guide 51 is a light guide cable such as an optical fiber that guides the illumination light emitted from the light source device 20 to the imaging unit 13. Illumination light emitted from the light source device 20 is guided to the imaging unit 13 through the optical signal cable 50, the operation unit 12, and the insertion unit 11 by the light guide 51, and irradiated on the object 901 from the tip of the light guide 51. Is done.

The imaging lens 130 emits reflected light and fluorescence from the inspected object 901 irradiated with incident light, that is, illumination light emitted from the light source device 20, to the image sensor 132 side, and images the inspected object 901. It is an optical lens that forms an image on the imaging surface of the sensor 132.

The excitation light cut filter 131 is an optical filter that reflects or absorbs and attenuates only the reflected light from the object 901 emitted from the imaging lens 130 and the light in the wavelength band of the excitation light included in the fluorescence. For example, the excitation light cut filter 131 attenuates light in a wavelength band around 700 nm to 800 nm, which is the wavelength band of excitation light. In the following description, the excitation light cut filter 131 attenuates the excitation light included in the incident reflected light and fluorescence to a level substantially close to “0”. "Is used for explanation. The excitation light cut filter 131 emits reflected light and fluorescence from the inspection object 901 from which the excitation light is cut to the image sensor 132.

The image sensor 132 is a solid-state imaging device that exposes (detects) incident light and outputs an electrical signal obtained by photoelectric conversion of the exposed light as an imaging signal. The image sensor 132 exposes the reflected light and fluorescence from the inspection object 901 from which the excitation light emitted from the excitation light cut filter 131 is cut, and generates an imaging signal corresponding to the exposed reflected light and fluorescence.

In the image sensor 132, a plurality of pixels each including a photoelectric conversion element (light receiving element) such as a photodiode that photoelectrically converts incident light is arranged in a matrix. The pixels arranged in the image sensor 132 include pixels (hereinafter referred to as “R pixels”) to which an on-chip color filter that transmits light (visible light) in the red (R) wavelength band is attached, and green ( G) a pixel to which an on-chip color filter that transmits light (visible light) in the wavelength band (visible light) (hereinafter referred to as “G pixel”) and blue (B) wavelength band light (visible light) are transmitted. Some pixels have an on-chip color filter attached (hereinafter referred to as “B pixel”). The pixels arranged in the image sensor 132 include pixels (hereinafter referred to as “IR pixels”) to which an on-chip color filter (hereinafter referred to as “IR filter”) that transmits near-infrared light is attached. . The IR pixel is a pixel that exposes the fluorescence emitted by the ICG administered to the subject by excitation light such as near-infrared light emitted by the light source device 20. A detailed description of the arrangement and configuration of each pixel in the image sensor 132 will be described later.

The image sensor 132 outputs an imaging signal corresponding to an electrical signal (hereinafter referred to as a “pixel signal”) photoelectrically converted by exposure of each pixel through the insertion unit 11, the operation unit 12, and the electrical signal cable 60. Output to the external processing unit 30 by the line 61. Note that the image sensor 132 performs analog / digital conversion (A / D conversion) on an analog pixel signal output from each pixel disposed in the image sensor 132, and a so-called digital value representing the magnitude of the pixel signal. The RAW data is output to the external processing unit 30 as an imaging signal. This RAW data is a parallel digital value representing the magnitude of each pixel signal.

The image sensor 132 reduces the number of signal lines of the imaging signal line 61 for transmitting RAW data as imaging signals to the external processing unit 30, so that RAW data (hereinafter, “parallel RAW data”) that is a parallel digital value is used. Are converted into serial digital value RAW data (hereinafter referred to as “serial RAW data”), and the serial RAW data is output to the external processing unit 30 as an imaging signal.

In FIG. 2, the light source device 20 constituting the fluorescence observation endoscope apparatus 1 includes a light emission control unit 21, four white light sources 221 to 224, and four dichroic mirrors 231 to dichroic mirrors. 234 and the light irradiation lens 24.

Each of the white light source 221 to the white light source 224 is a light source that emits white light. Each of the white light source 221 to the white light source 224 emits white light having an intensity according to the control from the light emission control unit 21. In the following description, when the white light source 221 to the white light source 224 are not distinguished, they are referred to as “white light source 220”.

Each of the dichroic mirror 231 to the dichroic mirror 234 corresponds to each of the white light source 221 to the white light source 224, and selects (separates) light in a specific wavelength band from the white light emitted from the corresponding white light source 220. To do. Each of the dichroic mirror 231 to the dichroic mirror 234 emits the separated light to the light irradiation lens 24.

More specifically, in the dichroic mirror 231, the light irradiation lens 24 emits light in the blue (B) wavelength band (for example, light in the wavelength band of 400 nm to 500 nm) in the white light emitted from the corresponding white light source 221. The blue (B) light (visible light) separated by reflecting in the arranged direction is emitted to the light irradiation lens 24. Further, the dichroic mirror 232 is a direction in which the light irradiation lens 24 is disposed in the light of the green (G) wavelength band (for example, light in the wavelength band of 500 nm to 600 nm) in the white light emitted from the corresponding white light source 222. The green (G) light (visible light) separated by reflecting the light is emitted to the light irradiation lens 24. Further, the dichroic mirror 233 emits light in the red (R) wavelength band (for example, light in the wavelength band of 600 nm to 700 nm) in the white light emitted from the corresponding white light source 223 in the direction in which the light irradiation lens 24 is disposed. The red (R) light (visible light) separated by being reflected is emitted to the light irradiation lens 24. Further, the dichroic mirror 234 reflects light in the wavelength band of excitation light (for example, light in the wavelength band of 700 nm to 800 nm) in the white light emitted from the corresponding white light source 224 in the direction in which the light irradiation lens 24 is disposed. Excitation light (near infrared light) for exciting the separated ICG is emitted to the light irradiation lens 24.

The light irradiation lens 24 is an optical lens that condenses light in a specific wavelength band emitted from each of the dichroic mirror 231 to the dichroic mirror 234 to the same extent as the diameter of the light guide 51. The light irradiation lens 24 emits the condensed light to the first end face of the light guide 51. Thereby, the light guide 51 guides the light emitted from the light irradiation lens 24 to the imaging unit 13 and emits it from the second end face disposed at the tip of the imaging unit 13 as illumination light emitted from the light source device 20 ( Irradiate the inspection object 901).

The light emission control unit 21 controls the emission of white light and the intensity of the emitted white light in each of the white light sources 220 based on the information on the photographing mode of the fluorescence observation endoscope apparatus 1 input from the external processing unit 30. To do. A detailed description of the control of each white light source 220 based on information on the shooting mode in the light emission control unit 21 will be described later.

In FIG. 2, the external processing unit 30 constituting the fluorescence observation endoscope apparatus 1 includes a mode determination unit 31, a deserializer 32, an image processing unit 33, and a digital / analog conversion unit (D / A conversion unit). 34.

The mode determination unit 31 outputs information on the imaging mode of the fluorescence observation endoscope apparatus 1 to each of the image processing unit 33 and the light emission control unit 21 in the light source device 20. More specifically, the mode determination unit 31 inspects the photographing mode changeover switch 14 included in the operation unit 12 included in the electric signal transmitted from the operation unit 12 through the mode signal line 62 passing through the electric signal cable 60. Information on the shooting mode switched by the operator is output to each of the image processing unit 33 and the light emission control unit 21 in the light source device 20.

The deserializer 32 restores the original parallel RAW data that has been analog / digital converted by the image sensor 132 from the imaging signal (serial RAW data) output from the image sensor 132 included in the imaging unit 13 and transmitted through the imaging signal line 61. To do. That is, the deserializer 32 performs serial / parallel conversion on the input serial RAW data to restore the parallel RAW data. Then, the deserializer 32 outputs the restored parallel RAW data to the image processing unit 33.

The image processing unit 33 performs image processing on the parallel raw data output from the deserializer 32 in accordance with the shooting mode information input from the mode determination unit 31, and the inspected object 901 captured by the image sensor 132. An image including is generated. The image processing unit 33 outputs image data including the generated inspected object 901 (hereinafter referred to as “image data”) to the digital / analog conversion unit 34. The image processing unit 33 includes a demosaicing unit 330, a white balance unit 331, and a gamma correction unit 332.

The demosaicing unit 330 generates red (R), green (G), blue (B), and near from the parallel RAW data input from the deserializer 32 according to the shooting mode information input from the mode determination unit 31. Image data is generated for each infrared light. That is, the demosaicing unit 330 generates respective image data including only pixel signals (digital values) of any one of R pixels, G pixels, B pixels, and IR pixels arranged in the image sensor 132.

More specifically, the demosaicing unit 330 performs image processing similar to so-called three-plate processing (hereinafter referred to as “demosaicing processing”) on the input parallel raw data (reconstructed parallel raw data). ), The pixel signals corresponding to the light of any one of red (R), green (G), blue (B), and near-infrared light are positioned at all the pixels arranged in the image sensor 132. Image data represented by (digital value) is generated. In the demosaicing process, a pixel signal (digital value) at a position where a pixel corresponding to light in another wavelength band is arranged from a pixel signal (digital value) corresponding to light in a certain wavelength band arranged in the image sensor 132. Are interpolated to generate image data for one frame consisting only of pixel signals (digital values) corresponding to light of the same wavelength band. The demosaicing unit 330 performs a demosaicing process for each light in each wavelength band.

The demosaicing unit 330 performs one-frame image data consisting only of a pixel signal (digital value) corresponding to the pixel signal output from the R pixel and a pixel signal corresponding to the pixel signal output from the G pixel by the demosaicing process. Image data of one frame including only (digital value) and image data corresponding to one frame including only a pixel signal (digital value) corresponding to the pixel signal output from the B pixel are generated. The demosaicing unit 330 also generates image data for one frame including only a pixel signal (digital value) corresponding to the pixel signal output from the IR pixel by the demosaicing process.

In the following description, image data for one frame consisting only of a pixel signal (digital value) corresponding to the pixel signal output from the R pixel is referred to as “R image data” and corresponds to the pixel signal output from the G pixel. Image data for one frame consisting only of pixel signals (digital values) is referred to as “G image data”, and image data for one frame consisting only of pixel signals (digital values) corresponding to pixel signals output by B pixels. This is called “B image data”. In the following description, R image data, G image data, and B image data are referred to as “RGB image data” when they are not distinguished from each other. In the following description, image data for one frame including only a pixel signal (digital value) corresponding to the pixel signal output from the IR pixel is referred to as “IR image data”.

Further, the demosaicing unit 330 determines the magnitude of the digital value of the pixel signal corresponding to the pixel at the same position in each of the generated R image data, G image data, and B image data based on the generated IR image data. Image processing to be corrected (hereinafter referred to as “offset processing”) is performed to generate final R image data, G image data, and B image data. In the offset process, IR image data is subtracted from each of R image data, G image data, and B image data. The demosaicing unit 330 performs offset processing on each image data.

Detailed description regarding each image processing performed in the demosaicing unit 330, that is, demosaicing processing for generating RGB image data and IR image data from parallel raw data, and offset processing based on the RGB image data based on the IR image data. Will be described later.

The demosaicing unit 330 outputs the generated RGB image data, that is, final RGB image data that has been subjected to digital value size correction (offset processing) based on the IR image data, to the white balance unit 331. To do.

The white balance unit 331 performs image processing (hereinafter referred to as “white balance processing”) for adjusting the white level of the image on the RGB image data input from the demosaicing unit 330. That is, when a white subject is photographed, the white balance unit 331 has the same digital signal magnitude corresponding to the pixel at the same position in each of the R image data, the G image data, and the B image data. Image processing is performed to adjust the digital value so that the value becomes. More specifically, when the RGB image data is a digital value representing a white subject, the magnitude of the digital value of the red (R) pixel signal at the same position included in the RGB image data and the green (G) The digital value of the pixel signal constituting each of the RGB image data so that the magnitude of the digital value of the pixel signal and the magnitude of the digital value of the blue (B) pixel signal are the same for the white subject. Image processing for multiplying the value by the gain value is performed for each pixel.

In the white balance processing, when the photographed subject is white, it is desirable that the digital value of each pixel signal of the RGB image data is the same value as the white subject. However, in the present invention, it is not stipulated that the magnitude of the digital value of the pixel signal constituting each of the RGB image data is the same value for a white subject. That is, in the present invention, the white balance deviation is allowed within a predetermined range. Note that the white balance deviation allowed in a predetermined range in the present invention is the same as a generally considered white balance deviation, and the amount of white balance deviation is not particularly defined.

The white balance unit 331 outputs each of the RGB image data subjected to white balance adjustment to the gamma correction unit 332.

The gamma correction unit 332 performs gamma offset processing that corrects the color when the image corresponding to each of the RGB image data input from the white balance unit 331 is output to the color monitor 40 and displayed. That is, the gamma correction unit 332 is generated based on the white balance processed RGB image data so that the color of the object 901 imaged by the image sensor 132 is correctly displayed on the color monitor 40, and the color monitor 40. The color nonlinearity between the image signal inputted to the image and the image actually displayed on the color monitor 40 is corrected.

The gamma correction unit 332 outputs each of the RGB image data subjected to gamma correction to the digital / analog conversion unit 34.

The digital / analog conversion unit 34 performs digital / analog conversion (D / A conversion) on each of the RGB image data (digital values) input from the gamma correction unit 332. The digital / analog conversion unit 34 outputs the digital / analog converted image signal (analog signal) to the color monitor 40 as a display image signal generated by the external processing unit 30, and the test object 901 is connected to the color monitor 40. Display the image that contains it.

With this configuration, the fluorescence observation endoscope apparatus 1 captures an image including the inspection object 901 with visible light in the normal imaging mode, and the fluorescence in which the inspection object 901 is excited with excitation light in the fluorescence imaging mode. Take an image containing And the fluorescence observation endoscope apparatus 1 displays the image | photographed image containing the to-be-inspected object 901 on the color monitor 40, and shows it to a tester.

In the fluorescence observation endoscope apparatus 1, the excitation light cut filter 131 and the image sensor 132 in the imaging unit 13 provided at the distal end portion of the endoscope scope unit 10, the light source device 20, and the external processing unit 30 are included. The demosaicing unit 330 and the white balance unit 331 in the image processing unit 33 provided correspond to the components constituting the fluorescence observation apparatus of the present invention.

Next, the image sensor 132 constituting the fluorescence observation apparatus of the present invention in the fluorescence observation endoscope apparatus 1 will be described. FIG. 3 is a block diagram showing a schematic configuration of the solid-state imaging device (image sensor 132) provided in the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. In FIG. 3, the image sensor 132 includes a pixel unit 1321, a reading unit 1322, an analog / digital conversion unit 1323, and a serializer 1324.

In the pixel portion 1321, a plurality of pixels 1320 are arranged. FIG. 3 shows an example of a pixel portion 1321 in which a plurality of pixels 1320 are two-dimensionally arranged in 7 rows and 8 columns. Each pixel 1320 arranged in the pixel portion 1321 generates a signal charge having a charge amount corresponding to the light intensity from the light incident on each arranged position, and accumulates the generated signal charge.

The reading unit 1322 controls reading of signal charges generated and accumulated by the respective pixels 1320 arranged in the pixel unit 1321. The reading unit 1322 includes, for example, a control circuit 13220, a vertical scanning circuit 13221, a horizontal scanning circuit 13222, and the like. The control circuit 13220 controls each component included in the reading unit 1322, that is, the entire reading unit 1322. The vertical scanning circuit 13221 drives each pixel 1320 in the pixel portion 1321 for each row in accordance with control from the control circuit 13220, and outputs a voltage signal corresponding to the signal charge accumulated in each pixel 1320 as a pixel signal. To the vertical signal line 1325. Accordingly, a pixel signal (analog signal) output from each pixel 1320 for each row is input to the analog / digital conversion unit 1323. The horizontal scanning circuit 13222 controls the analog / digital conversion unit 1323 for each column of the pixels 1320 in the pixel unit 1321 in accordance with the control from the control circuit 13220, and the analog / digital conversion unit 1323 converts the analog / digital conversion. The pixel signal (digital value) after the output is sequentially output to the horizontal signal line 1326 for each column of the pixels 1320 in the pixel portion 1321. Accordingly, a pixel signal having a digital value representing the magnitude of an analog pixel signal output from each pixel 1320 analog / digital converted by the analog / digital conversion unit 1323 is output from each pixel 1320 arranged in the pixel unit 1321. Each column is sequentially input to the serializer 1324.

The analog / digital conversion unit 1323 performs analog / digital conversion on the analog pixel signal output from each pixel 1320 in the pixel unit 1321 in accordance with control from the vertical scanning circuit 13221 in the reading unit 1322. In FIG. 3, an analog / digital conversion circuit (A / D conversion circuit) 13230 that outputs a digital value obtained by analog / digital conversion of a voltage value of an input analog signal is provided in each pixel 1320 arranged in the pixel portion 1321. An example of the analog / digital conversion unit 1323 having a configuration provided for each column is shown. Each of the analog / digital conversion circuits 13230 outputs a digital pixel signal obtained by analog / digital conversion of the voltage value of the analog pixel signal input from the pixel 1320 in the corresponding column. The analog / digital conversion unit 1323 converts the pixel signal of the digital value that each of the analog / digital conversion circuits 13230 performs analog / digital conversion to the horizontal signal line 1326 according to control from the horizontal scanning circuit 13222 in the reading unit 1322. Via the serializer 1324.

In FIG. 3, the analog / digital conversion circuit 13230 is provided for each column of pixels 1320 in the pixel portion 1321, that is, the analog / digital conversion unit 1323 having one analog / digital conversion circuit 13230 in one column. Indicated. However, the configuration in the analog / digital conversion unit 1323 is not limited to the configuration shown in FIG. For example, the analog / digital conversion unit 1323 may include a single analog / digital conversion circuit 13230 for a plurality of columns of the pixels 1320 in the pixel unit 1321. Further, a configuration may be adopted in which one analog / digital conversion circuit 13230 sequentially performs analog / digital conversion on the voltage value of the analog pixel signal output from the pixel 1320 in each column.

The serializer 1324 performs parallel / serial conversion on pixel signals (digital values) sequentially input from the analog / digital conversion unit 1323, that is, parallel RAW data. The serializer 1324 outputs serial RAW data subjected to parallel / serial conversion to the outside of the image sensor 132 as an imaging signal. At this time, the serializer 1324 outputs serial RAW data to the outside of the image sensor 132 as an imaging signal in accordance with, for example, a LVDS (Low voltage differential signaling) method that is a differential interface method. In this case, the serializer 1324 also performs LVDS termination processing and the like.

In addition, as described above, the pixels 1320 arranged in the pixel portion 1321 in the image sensor 132 include an R pixel, a G pixel, a B pixel, and an IR pixel. In the image sensor 132, the R pixel, the G pixel, the B pixel, and the IR pixel have different structures. Here, the structure of the image sensor 132 which comprises the fluorescence observation apparatus of this invention in the fluorescence observation endoscope apparatus 1 is demonstrated. FIG. 4 is a cross-sectional view showing an example of the structure of the solid-state imaging device (image sensor 132) provided in the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. FIG. 4 shows a partial vertical structure of the pixel portion 1321 in the case where the R pixel, the G pixel, or the B pixel and the IR pixel are formed in the image sensor 132. The image sensor 132 is a backside illumination (BSI) type solid-state imaging device.

The image sensor 132 is formed by laminating respective layers constituting the pixel 1320 on the support substrate 13200. More specifically, a wiring layer 13201, a photoelectric conversion layer 13202, a dielectric multilayer filter layer 13203, an on-chip color filter layer 13204, and a microlens 13205 are formed from the support substrate 13200 toward the light incident side. Yes.

The support substrate 13200 supports the semiconductor substrate on which the function of the image sensor 132 on which the wiring layer 13201, the photoelectric conversion layer 13202, the dielectric multilayer filter layer 13203, the on-chip color filter layer 13204, and the microlens 13205 are formed. For a semiconductor substrate.

The wiring layer 13201 is a layer for forming wiring in each circuit element included in the image sensor 132. FIG. 4 schematically shows a state in which wirings 13206 that connect circuit elements of the respective pixels 1320 are formed in the wiring layer 13201. Each pixel 1320 outputs a pixel signal (analog signal) corresponding to the drive signal from the vertical scanning circuit 13221 input through the wiring 13206 formed in the wiring layer 13201 to the vertical signal line 1325 through the wiring 13206.

The photoelectric conversion layer 13202 is a layer for forming a photoelectric conversion element constituting the pixel 1320. FIG. 4 schematically illustrates a state in which the photoelectric conversion element 13207 constituting each of the two pixels 1320 is formed on the photoelectric conversion layer 13202. Each photoelectric conversion element 13207 generates and accumulates signal charges corresponding to the light intensity of incident light.

The dielectric multilayer filter layer 13203 is a layer that forms a dielectric multilayer filter in which inorganic materials such as a dielectric that attenuates visible light incident on the photoelectric conversion element 13207 formed in the photoelectric conversion layer 13202 are laminated in multiple layers. is there. The image sensor 132 includes an R pixel, a G pixel, and a B pixel with an on-chip color filter that transmits visible light, and an IR pixel with an IR filter that transmits near-infrared light. . In the dielectric multilayer filter layer 13203, a dielectric multilayer filter is formed at a position corresponding to the IR pixel. In FIG. 4, a transparent material 13208 is formed on the dielectric multilayer filter layer 13203 at a position corresponding to the pixel 1320a that is an R pixel, G pixel, or B pixel, and a dielectric at a position corresponding to the pixel 1320b that is an IR pixel. A state in which a dielectric multilayer filter 13209 is formed on the multilayer filter layer 13203 is schematically shown.

The dielectric multilayer filter 13209 is an optical filter that attenuates visible light incident on the pixel 1320b (IR pixel). For example, a layer of silicon dioxide (SiO ₂ ) and a layer of titanium oxide (TiO ₂ ) are alternately formed. It is formed by stacking. The thickness of the dielectric multilayer filter 13209 is less than 1 μm. Therefore, it is possible to suppress the sensitivity to incident light and the deterioration of the oblique incidence characteristic with respect to light incident from an oblique direction. However, in the image sensor 132, the dielectric multilayer filter 13209 is not formed in the dielectric multilayer filter layer 13203 so as to attenuate the visible light incident on the pixel 1320b to a negligible level.

Here, if the dielectric multilayer filter 13209 is required to have a characteristic that attenuates visible light to 0.3%, for example, it is necessary to considerably increase the number of layers to be laminated. Will become very thick. Then, the distance from the microlens 13205 to the photoelectric conversion element 13207 is increased, and the sensitivity to light and the characteristic (oblique incident characteristic) with respect to light incident from an oblique direction are remarkably deteriorated. In other words, in the image sensor 132, the incidence of visible light on the pixel 1320b is limited, and the sensitivity to incident light and the remarkable deterioration in oblique incidence characteristics with respect to light incident from an oblique direction are also suppressed.

The transparent material 13208 has a characteristic of transmitting light incident on the pixel 1320a (R pixel, G pixel, or B pixel) as it is, and is formed of, for example, silicon dioxide (SiO ₂ ). The transparent material 13208 adjusts the level of the dielectric multilayer filter layer 13203 by adjusting the height of the dielectric multilayer filter layer 13203 by forming the dielectric multilayer filter 13209 at the position of the pixel 1320b (IR pixel). ) Have a role.

The on-chip color filter layer 13204 is a layer that forms an on-chip color filter that transmits light in a predetermined wavelength band and enters the photoelectric conversion element 13207 formed in the photoelectric conversion layer 13202. In FIG. 4, light (visible light) in the wavelength band of red (R), green (G), or blue (B) is transmitted to a position corresponding to the pixel 1320a (R pixel, G pixel, or B pixel). The on-chip color filter 13210 is formed, and the on-chip color filter 13211 that transmits light in the near-infrared wavelength band (near-infrared light) is formed at a position corresponding to the pixel 1320b (IR pixel). Is shown.

The microlens 13205 condenses incident light on each of the photoelectric conversion elements 13207 formed in the photoelectric conversion layer 13202. The microlens 13205 is formed at a position corresponding to each of the photoelectric conversion elements 13207. FIG. 4 schematically shows a state in which the respective microlenses 13205 corresponding to the two photoelectric conversion elements 13207 formed in the photoelectric conversion layer 13202 are formed.

Next, in the image sensor 132, the arrangement of each on-chip color filter formed on the on-chip color filter layer 13204, the transparent material 13208 and the dielectric multilayer filter 13209 formed on the dielectric multilayer filter layer 13203 will be described. . FIG. 5A is a diagram schematically illustrating an example of an arrangement of on-chip color filters to be attached to the pixels 1320 of the solid-state imaging device (image sensor 132) provided in the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. is there. FIG. 5B schematically shows an example of the arrangement of the dielectric multilayer filter 13209 corresponding to the pixel 1320 of the solid-state imaging device (image sensor 132) provided in the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. FIG.

First, the arrangement of on-chip color filters corresponding to the pixels 1320 arranged in the pixel portion 1321 of the image sensor 132 will be described with reference to FIG. 5A. As described above, the image sensor 132 includes a plurality of pixels 1320 that photoelectrically convert light in different wavelength bands. More specifically, the image sensor 132 photoelectrically converts R pixel that converts light (visible light) in the red (R) wavelength band and light (visible light) in the green (G) wavelength band. A G pixel, a B pixel that photoelectrically converts light (visible light) in the blue (B) wavelength band, and an IR pixel that photoelectrically converts light in the near infrared wavelength band (near infrared light) are arranged in a matrix. Several are arranged. The arrangement of the pixels 1320 shown in FIG. 5A is such that the R pixel, the G pixel, and the B pixel are arranged in a row in which the R pixel is arranged in the pixel portion 1321 arranged in a so-called Bayer array (the B pixel is In this arrangement, the G pixel in the arranged column) is changed to the IR pixel.

Therefore, as illustrated in FIG. 5A, each on-chip color filter layer 13204 of the pixel portion 1321 of the image sensor 132 includes each on-chip color filter that transmits light in a wavelength band that is photoelectrically converted by each pixel 1320. It is formed at the position of the corresponding pixel 1320.

More specifically, an on-chip color filter 13210 (shown in FIG. 5A) that transmits light in the red (R) wavelength band (for example, visible light in the wavelength band of 600 nm to 700 nm) is located at a position corresponding to the R pixel. "R" on-chip color filter 13210, hereinafter referred to as "on-chip color filter 13210R"). Further, an on-chip color filter 13210 (“G” shown in FIG. 5A) that transmits light in the green (G) wavelength band (for example, visible light in the wavelength band of 500 nm to 600 nm) is located at a position corresponding to the G pixel. On-chip color filter 13210, hereinafter referred to as “on-chip color filter 13210G”). In addition, an on-chip color filter 13210 (“B” shown in FIG. 5A) that transmits light in the blue (B) wavelength band (for example, visible light in the wavelength band of 400 nm to 500 nm) is disposed at a position corresponding to the B pixel. On-chip color filter 13210, hereinafter referred to as “on-chip color filter 13210B”). In addition, ICG administered to the subject is excited at a position corresponding to the IR pixel by light in the wavelength band of excitation light emitted from the light source device 20 (for example, near infrared light in the wavelength band of 700 nm to 800 nm). On-chip color filter 13211 that transmits fluorescence (for example, near-infrared light having a wavelength band exceeding 800 nm) (“IR” on-chip color filter 13211 shown in FIG. 5A, hereinafter “on-chip color filter”) 13211IR ") is formed.

Subsequently, the arrangement of the dielectric multilayer filter 13209 and the transparent material 13208 corresponding to the pixels 1320 arranged in the pixel portion 1321 of the image sensor 132 will be described using FIG. 5B. As described above, the dielectric multilayer filter 13209 or the transparent material 13208 is formed on the dielectric multilayer filter layer 13203 of the pixel portion 1321 of the image sensor 132 at a position corresponding to each pixel 1320.

More specifically, as shown in FIG. 5B, a dielectric multilayer filter 13209 is provided at a position corresponding to the IR pixel (see FIG. 5A) in the dielectric multilayer filter layer 13203 of the pixel portion 1321 of the image sensor 132. A transparent material 13208 is formed at positions corresponding to the R pixel, the G pixel, and the B pixel.

Next, in the image sensor 132, the light of the on-chip color filter formed on the on-chip color filter layer 13204 corresponding to each pixel 1320 and the dielectric multilayer filter 13209 formed on the dielectric multilayer filter layer 13203 are displayed. The characteristics with respect to the wavelength band will be described. Note that the transparent material 13208 formed in the dielectric multilayer filter layer 13203 has a characteristic of transmitting incident light as it is, and thus detailed description regarding characteristics of the transparent material 13208 with respect to the wavelength band of light is omitted.

FIG. 6A is a diagram illustrating an example of characteristics of each filter corresponding to the pixel 1320 of the solid-state imaging device (image sensor 132) provided in the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. FIG. 6B is a diagram illustrating an example of characteristics of the dielectric multilayer filter 13209 corresponding to the pixel 1320 of the solid-state imaging device (image sensor 132) included in the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. It is.

In FIG. 6A and FIG. 6B, the horizontal axis represents the wavelength of light, and the vertical axis represents the characteristics (hereinafter referred to as “spectral characteristics”) representing the transmittance of each filter. More specifically, in FIG. 6A, B1 represents the spectral characteristic of the on-chip color filter 13210B, G1 represents the spectral characteristic of the on-chip color filter 13210G, and R1 represents the spectral characteristic of the on-chip color filter 13210R. IR1 represents the spectral characteristic of the on-chip color filter 13211IR. In FIG. 6B, DM1 represents the spectral characteristic of the dielectric multilayer filter 13209. In FIG. 6A, IR1 + DM1 is a combination of the on-chip color filter 13211IR and the dielectric multilayer filter 13209, that is, in the image sensor 132, all optical filters formed at positions where IR pixels are arranged. The spectral characteristics combined with the characteristics are shown. FIG. 6A also shows the spectral characteristic EL1 of the excitation light cut filter 131 that attenuates or cuts the light in the wavelength band of the excitation light incident on the image sensor 132.

As shown in FIG. 6A, the spectral characteristic B1 of the on-chip color filter 13210B corresponding to the B pixel has high transmittance for light in the wavelength band of 400 nm to 500 nm, which is the wavelength band of blue (B). It is a characteristic. Therefore, the on-chip color filter 13210B selects (transmits) visible light in the blue (B) wavelength band of 400 nm to 500 nm and makes it incident on the photoelectric conversion element 13207 of the B pixel. When the center wavelength of visible light in the blue (B) wavelength band transmitted by the on-chip color filter 13210B corresponding to the B pixel is the transmission dominant wavelength, the transmission dominant wavelength of the on-chip color filter 13210B is, for example, 450 nm.

Further, as shown in FIG. 6A, the spectral characteristic G1 of the on-chip color filter 13210G corresponding to the G pixel has a high transmittance for light in the wavelength band of 500 nm to 600 nm, which is the wavelength band of green (G). It has characteristics. Accordingly, the on-chip color filter 13210G selects (transmits) visible light in the green (G) wavelength band of 500 nm to 600 nm and makes it incident on the photoelectric conversion element 13207 of the G pixel. Note that the transmission main wavelength of visible light in the green (G) wavelength band transmitted through the on-chip color filter 13210G corresponding to the G pixel is, for example, 540 nm.

As shown in FIG. 6A, the spectral characteristic R1 of the on-chip color filter 13210R corresponding to the R pixel has a high transmittance for light in the wavelength band of 600 nm to 700 nm, which is the red (R) wavelength band. It has characteristics. Therefore, the on-chip color filter 13210R selects (transmits) visible light in the red (R) wavelength band of 600 nm to 700 nm and makes it incident on the photoelectric conversion element 13207 of the R pixel. Note that the transmission main wavelength of visible light in the red (R) wavelength band transmitted through the on-chip color filter 13210R corresponding to the R pixel is, for example, 630 nm.

Further, as shown in FIG. 6A, the spectral characteristic IR1 of the on-chip color filter 13211IR corresponding to the IR pixel has a wavelength band exceeding 800 nm which is a wavelength band of fluorescence emitted by excitation light (near infrared light). This characteristic has a high transmittance for light. Accordingly, the on-chip color filter 13211IR selects (transmits) fluorescence (near infrared light) in a wavelength band exceeding 800 nm and makes it incident on the photoelectric conversion element 13207 of the IR pixel. Note that the transmission dominant wavelength of near-infrared light in the fluorescence wavelength band transmitted by the on-chip color filter 13211IR corresponding to the IR pixel is, for example, 830 nm. Therefore, the difference between the transmission main wavelength of the on-chip color filter 13211IR and the transmission main wavelengths of the on-chip color filter 13210B, the on-chip color filter 13210G, and the on-chip color filter 13210R is 380 nm, 290 nm, and 200 nm. As described above, in this embodiment, the difference between the transmission main wavelength of the on-chip color filter 13211IR and the transmission main wavelength of the other on-chip color filters is set to 150 nm or more, thereby suppressing the interference of incident light with each other. .

Since each of the on-chip color filter 13210 and the on-chip color filter 13211 is formed of an organic material, as illustrated in FIG. 6A, the wavelength band is not a selected wavelength band, that is, a wavelength band that is not the intended wavelength band. Visible light is also transmitted about several percent. Therefore, visible light that is not light in the wavelength band to be exposed also enters the photoelectric conversion element 13207 of each pixel 1320. For example, as shown in FIG. 6A, visible light having a wavelength band of 350 nm to 700 nm is also incident on the photoelectric conversion element 13207 of the IR pixel.

Therefore, the image sensor 132 has a structure as shown in FIG. 4 to attenuate light in the visible wavelength band incident on the photoelectric conversion element 13207 of the IR pixel. More specifically, the dielectric multilayer filter 13209 is formed in the dielectric multilayer filter layer 13203 to attenuate light in the visible wavelength band incident on the photoelectric conversion element 13207 of the IR pixel.

In FIG. 6B, the material of the medium on which light is incident (incident medium) is silicon dioxide (SiO ₂ ), and the material of the medium on which light is emitted (emitter medium) is silicon (Si). Results of simulating light transmittance in each wavelength band in dielectric multilayer filter 13209 having a structure in which eight layers of silicon dioxide (SiO ₂ ) layers and titanium oxide (TiO ₂ ) layers are alternately stacked. The spectral characteristics of are shown. Note that the film thickness (total film thickness) in the dielectric multilayer filter 13209 showing the spectral characteristic DM1 shown in FIG. 6B is 618 nm. That is, the total film thickness of the dielectric multilayer filter 13209 is less than 1 μm as described above. The dielectric multilayer filter 13209 attenuates visible light in the wavelength band of 400 nm to 600 nm, like the spectral characteristic DM1 shown in FIG. 6B. However, as shown in FIG. 6B, the dielectric multilayer filter 13209 does not have spectral characteristics until all visible light in the wavelength band of 400 nm to 600 nm is attenuated or removed (cut).

The image sensor 132 attenuates visible light contained in fluorescence incident on the photoelectric conversion element 13207 of the IR pixel by matching the dielectric multilayer filter 13209 with the on-chip color filter 13211IR. More specifically, as shown in the spectral characteristic IR1 + DM1 in FIG. 6A, visible light in the wavelength band of 400 nm to 600 nm in the spectral characteristic IR1 of only the on-chip color filter 13211IR is attenuated by the dielectric multilayer filter 13209. As a result, fluorescence in the wavelength band exceeding 800 nm, which attenuates the wavelength band of visible light, is incident on the photoelectric conversion element 13207 of the IR pixel arranged in the pixel portion 1321 of the image sensor 132.

Note that the dielectric multilayer filter 13209 is further multilayered, that is, by increasing the number of layers of silicon dioxide (SiO ₂ ) and titanium oxide (TiO ₂ ), the wavelength band of 400 nm to 600 nm is increased. Visible light can be further attenuated. That is, it is possible to increase the attenuation rate (light shielding rate) of visible light in the wavelength band of 400 nm to 600 nm in the dielectric multilayer filter 13209. However, in this case, as described above, the distance between the microlens 13205 and the photoelectric conversion element 13207 increases as the total film thickness increases due to the multilayer structure of the dielectric multilayer filter 13209. This increases the sensitivity of the image sensor 132 to light and the oblique incidence characteristics.

Therefore, in the present invention, as described above, by setting the total film thickness of the dielectric multilayer filter 13209 formed in the image sensor 132 to less than 1 μm, the IR pixel as shown in FIGS. 6A and 6B is used. In this case, slight visible light is allowed to enter the photoelectric conversion element 13207, and deterioration of sensitivity to light and oblique incidence characteristics in the image sensor 132 is suppressed. In other words, in the present invention, the total thickness of the dielectric multilayer filter 13209 is less than 1 μm while suppressing the attenuation rate (light shielding ratio) of visible light in the wavelength band of 400 nm to 600 nm in the dielectric multilayer filter 13209 to a necessary minimum. Thus, the sensitivity to light and the oblique incidence characteristic in the image sensor 132 are improved. In the present invention, fluorescence is detected with high accuracy by respective processes in the demosaicing unit 330 and the white balance unit 331 described later.

In the fluorescence observation endoscope apparatus 1, as described above, the reflected light and fluorescence from the inspected object 901 from which the excitation light emitted from the excitation light cut filter 131 is cut are incident on the image sensor 132. The excitation light cut filter 131 is configured by, for example, stacking a plurality of thin films in the same manner as the dielectric multilayer filter 13209. However, the excitation light cut filter 131 has a thickness of 700 nm by forming a multilayer (for example, 100 layers) than the dielectric multilayer filter 13209. The attenuation rate (light shielding rate) with respect to excitation light in the wavelength band of ˜800 nm is very high. For this reason, although the excitation light cut filter 131 is very thick, as shown in the spectral characteristic EL1 shown in FIG. 6A, the light in the wavelength band of 700 nm to 800 nm, which is the wavelength band of the incident excitation light, is almost Cut all and emit.

As described above, in the image sensor 132 provided in the imaging unit 13 of the endoscope scope unit 10 constituting the fluorescence observation endoscope apparatus 1, a plurality of pixels 1320 are arranged in a matrix in the pixel unit 1321, and each pixel is arranged. Each of the on-chip color filter 13210 and the transparent material 13208 or the on-chip color filter 13211 and the dielectric multilayer filter 13209 is formed at a position corresponding to 1320. Then, the image sensor 132 outputs serial RAW data representing the magnitude of the pixel signal obtained by photoelectric conversion by exposing light in the wavelength band corresponding to each pixel 1320 as an imaging signal.

Each on-chip color filter in the image sensor 132, that is, the arrangement of the pixels 1320, the configuration of the image sensor 132, and the characteristics of the on-chip color filter and the dielectric multilayer filter are the above-described arrangement, configuration, and characteristics. That is, the present invention is not limited to the arrangement, configuration, and characteristics shown in FIGS. 3 to 6A and 6B. For example, regarding the arrangement of IR pixels in the image sensor 132, the number of IR pixels arranged in the pixel unit 1321 may be reduced, and a G pixel may be arranged instead. In the image sensor 132 having this configuration, the sensitivity to visible light (particularly, visible light in the green (G) wavelength band) can be improved.

For example, with respect to the characteristics of the dielectric multilayer filter 13209 formed in the image sensor 132, if the attenuation rate (light shielding rate) for visible light in the wavelength band of 400 nm to 600 nm can be maintained at a necessary level, silicon dioxide (SiO _2). ) And titanium oxide (TiO ₂ ) layers may be reduced in number. Finally, for example, the dielectric multilayer filter layer 13203 may not be provided, that is, the dielectric multilayer filter 13209 and the transparent material 13208 may not be formed. In the image sensor 132 having these configurations, it is possible to further suppress deterioration in sensitivity to light and oblique incidence characteristics.

Next, the light source device 20 constituting the fluorescence observation apparatus of the present invention in the fluorescence observation endoscope apparatus 1 will be described. 7A and 7B are diagrams schematically illustrating an example of characteristics of light emitted from the light source device 20 included in the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. In FIG. 7A and FIG. 7B, the vertical axis represents the characteristic (hereinafter referred to as “spectral characteristic”) representing the wavelength band and the emission intensity of the illumination light emitted from the light source device 20 with the wavelength of light as the horizontal axis. ing.

FIG. 7A shows illumination emitted from the light source device 20 when the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, that is, when the examination operator switches the imaging mode changeover switch 14 to the normal imaging mode. The spectral characteristic of light is typically shown. In FIG. 7B, the light source device 20 emits light when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, that is, when the examiner switches the imaging mode switch 14 to the fluorescence imaging mode. The spectral characteristic of the illumination light to perform is shown typically. 7A and 7B show the spectral characteristics of each of the on-chip color filters shown in FIG. 6A, more specifically, the spectral characteristics B1, the spectral characteristics G1, the spectral characteristics R1, and the spectral characteristics IR1. It also shows.

As described above, the light source device 20 emits illumination light in a wavelength band corresponding to the imaging mode of the fluorescence observation endoscope device 1. First, the operation of the light source device 20 and the characteristics of illumination light when the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode will be described with reference to FIG. 7A. When the photographing mode of the fluorescence observation endoscope apparatus 1 is the normal photographing mode, the external processing unit 30 outputs information indicating that the photographing mode is the normal photographing mode to the light source device 20. Then, the light source device 20 emits visible light corresponding to information representing the normal imaging mode input from the external processing unit 30 as illumination light for imaging the object 901 to be inspected.

When the photographing mode of the fluorescence observation endoscope apparatus 1 is the normal photographing mode, the light emission control unit 21 provided in the light source device 20 causes the white light source 221, the white light source 222, and the white light source 223 to emit white light having the same intensity. The white light source 224 is controlled not to emit white light, that is, to be turned off. Each of the dichroic mirror 231 to the dichroic mirror 234 corresponding to each white light source 220 separates light in a specific wavelength band and emits it to the light irradiation lens 24.

In FIG. 7A, B2 is light in the blue (B) wavelength band (for example, visible light in the wavelength band of 400 nm to 500 nm) emitted from the dichroic mirror 231 corresponding to the white light source 221 and emitted to the light irradiation lens 24. The spectral characteristics of are shown. G2 is a spectrum of light in the green (G) wavelength band (for example, visible light in the wavelength band of 500 nm to 600 nm) emitted from the dichroic mirror 232 corresponding to the white light source 222 and emitted to the light irradiation lens 24. It represents a characteristic. R2 is a spectrum of light in the red (R) wavelength band (for example, visible light in the wavelength band of 600 nm to 700 nm) emitted from the dichroic mirror 233 corresponding to the white light source 223 and emitted to the light irradiation lens 24. It represents a characteristic.

The center wavelength of visible light in the blue (B) wavelength band emitted by the pair of the white light source 221 and the dichroic mirror 231 is, for example, 450 nm, and depends on the pair of the white light source 222 and the dichroic mirror 232. The center wavelength of visible light in the green (G) wavelength band that is emitted is, for example, 540 nm, and the center of visible light in the red (R) wavelength band that is emitted by the combination of the white light source 222 and the dichroic mirror 232. The wavelength of is 630 nm, for example.

As shown in FIG. 7A, the emission intensity in each of the spectral characteristic B2, the spectral characteristic G2, and the spectral characteristic R2 is the same value. Accordingly, white light, which is a combination of the light emitted from the dichroic mirror 231 to the dichroic mirror 233, is emitted from the light irradiation lens 24 to the light guide 51 as illumination light emitted from the light source device 20. . Then, the white light condensed by the light irradiation lens 24 is guided to the imaging unit 13 by the light guide 51, and is irradiated to the object 901 from the tip of the imaging unit 13.

The image sensor 132 provided in the imaging unit 13 exposes light in a wavelength band corresponding to the photoelectric conversion element 13207 of each pixel 1320 (R pixel, G pixel, and B pixel), and each pixel 1320 Signal charges corresponding to the light intensity of light (visible light) incident on the arranged position are generated and accumulated. Then, the image sensor 132 outputs serial RAW data representing the magnitude of the pixel signal exposed and photoelectrically converted by each pixel 1320 to the external processing unit 30 as an imaging signal.

When the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, the dichroic mirror 234 corresponding to the white light source 224 is light in the wavelength band of the excitation light (the light source 224 is turned off). For example, excitation light having a wavelength band of 700 nm to 800 nm is not emitted to the light irradiation lens 24. Therefore, the white light (illumination light) emitted from the front end of the imaging unit 13 to the object 901 is visible light that does not include light in the wavelength band of the excitation light.

However, as described above, when the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, it is assumed that ambient light including near infrared light leaks toward the object 901 to be inspected. Therefore, the imaging signal output to the external processing unit 30 by the image sensor 132 in the normal photographing mode includes a component related to near infrared light included in the ambient light.

Subsequently, the operation of the light source device 20 and the characteristics of illumination light when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode will be described with reference to FIG. 7B. When the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, the external processing unit 30 outputs information indicating that the imaging mode is the fluorescence imaging mode to the light source device 20. Then, the light source device 20 emits illumination light including excitation light corresponding to information indicating that it is the fluorescence imaging mode input from the external processing unit 30 as illumination light for imaging the object 901 to be inspected. .

When the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, the light emission control unit 21 provided in the light source device 20 causes the white light source 221 and the white light source 222 to emit white light having the same weak intensity. The white light source 223 is controlled to be turned off, and the white light source 224 emits white light whose intensity is higher than the emission intensity of the white light source 221 and the white light source 222 (for example, three times or more emission intensity). To control. Each of the dichroic mirror 231 to the dichroic mirror 234 corresponding to each white light source 220 separates light in a specific wavelength band and emits it to the light irradiation lens 24.

7B, B3 represents the spectral characteristic of visible light in the blue (B) wavelength band emitted from the dichroic mirror 231 corresponding to the white light source 221 and emitted to the light irradiation lens 24. G3 represents the spectral characteristic of visible light in the green (G) wavelength band emitted from the dichroic mirror 232 corresponding to the white light source 222 and emitted to the light irradiation lens 24. IR3 is a spectrum of light in the wavelength band of excitation light (for example, near-infrared light in the wavelength band of 700 nm to 800 nm) emitted from the dichroic mirror 234 corresponding to the white light source 224 and emitted to the light irradiation lens 24. It represents a characteristic. The center wavelength of near-infrared light in the wavelength band of excitation light emitted by the set of the white light source 224 and the dichroic mirror 234 is, for example, 750 nm.

As shown in FIG. 7B, the emission intensity in the spectral characteristic IR3 is a strong value relative to the emission intensity in the spectral characteristic B3 and the spectral characteristic G3. Then, the combined light emitted from the dichroic mirror 231 and the dichroic mirror 232 and the dichroic mirror 234 is emitted from the light irradiation lens 24 as illumination light emitted from the light source device 20. The light is emitted to the guide 51. Then, the light condensed by the light irradiation lens 24 is guided to the imaging unit 13 by the light guide 51, and is irradiated to the object 901 from the tip of the imaging unit 13.

The image sensor 132 included in the imaging unit 13 exposes light in a wavelength band corresponding to the photoelectric conversion element 13207 of each pixel 1320 (G pixel, B pixel, and IR pixel), and each pixel 1320 A signal charge corresponding to the light intensity of the light incident on the arranged position is generated and accumulated. Then, the image sensor 132 outputs serial RAW data representing the magnitude of the pixel signal exposed and photoelectrically converted by each pixel 1320 to the external processing unit 30 as an imaging signal.

When the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, the dichroic mirror 233 corresponding to the white light source 223 has a red (R) wavelength band because the white light source 223 is turned off. Light (for example, visible light having a wavelength band of 600 nm to 700 nm) is not emitted to the light irradiation lens 24. Therefore, the illumination light applied to the object 901 from the tip of the imaging unit 13 includes visible light other than visible light in the red (R) wavelength band and near infrared light in the wavelength band of excitation light. Light.

As described above, when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, it is assumed that the ambient light does not leak into the inspection object 901.

As described above, the light emission control unit 21 included in the light source device 20 configuring the fluorescence observation endoscope apparatus 1 is configured according to the imaging mode information of the fluorescence observation endoscope apparatus 1 input from the external processing unit 30. The white light source 221 to the white light source 224 control the light emission and emission intensity of the white light emitted. Thereby, the light source device 20 emits illumination light having a wavelength band corresponding to the photographing mode of the fluorescence observation endoscope device 1.

Next, the demosaicing unit 330 constituting the fluorescence observation apparatus of the present invention in the fluorescence observation endoscope apparatus 1 will be described. 8, FIG. 9A to FIG. 9C, FIG. 10, and FIG. 11A to FIG. 11C are diagrams schematically showing an example of the image processing operation in the fluorescence observation endoscope apparatus 1 according to the embodiment of the present invention. 8 and 9A to 9C show the case where the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, that is, when the examination operator switches the imaging mode changeover switch 14 to the normal imaging mode. The image processing which the mosaicing part 330 performs is shown typically. 10 and 11A to 11C, when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, that is, when the examiner switches the imaging mode switch 14 to the fluorescence imaging mode. 6 schematically shows image processing performed by the demosaicing unit 330 when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode. For ease of explanation, FIGS. 8, 9A to 9C, FIG. 10, and FIGS. 11A to 11C show a pixel portion in which a plurality of pixels 1320 are two-dimensionally arranged in 4 rows and 4 columns. An example of 1321 is shown.

First, the image processing operation in the demosaicing unit 330 when the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode will be described with reference to FIGS. 8 and 9A to 9C. When the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, the mode determination unit 31 included in the external processing unit 30 includes information indicating that the imaging mode is the normal imaging mode in the image processing unit 33. To the demosaicing unit 330. The demosaicing unit 330 then receives the parallel RAW data input from the deserializer 32 (the original parallel RAW data restored by the deserializer 32 in accordance with the information indicating the normal shooting mode input from the mode determination unit 31. ) To generate RGB image data (R image data, G image data, and B image data) and IR image data (a demosaicing process). Thereafter, the demosaicing unit 330 performs image processing (offset processing) for correcting the generated RGB image data based on the IR image data. FIG. 8 schematically shows a demosaicing process performed by the demosaicing unit 330 when the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode. FIGS. 9A to 9C show fluorescence observation endoscopes. An offset process performed by the demosaicing unit 330 when the shooting mode of the mirror device 1 is the normal shooting mode is schematically shown.

As described above, when the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, the image sensor 132 uses serial RAW data representing the magnitude of the pixel signal output from each pixel 1320 as an imaging signal. Output to the external processing unit 30. The deserializer 32 restores the input serial RAW data to the original parallel RAW data and outputs it to the demosaicing unit 330 in the image processing unit 33. More specifically, the deserializer 32 performs serial / parallel conversion on the input serial RAW data, and, as shown in FIG. Parallel RAW data restored to a state where each of 1320 (R pixel, G pixel, B pixel, or IR pixel) is arranged is generated, and the restored parallel RAW data is output to demosaicing section 330.

The demosaicing unit 330 performs a demosaicing process on the parallel raw data input from the deserializer 32 to generate R image data, G image data, B image data, and IR image data. More specifically, the demosaicing unit 330 obtains a pixel signal corresponding to red (R) visible light at a position where pixels other than the R pixel are arranged in the pixel unit 1321 by a demosaicing process corresponding to the R pixel. Interpolation is performed based on the pixel signal (digital value) output by the R pixel. Accordingly, the demosaicing unit 330 causes the pixel signals (digital values) at the positions of all the pixels 1320 arranged in the pixel unit 1321 to be pixel signals output by the R pixel, as illustrated in FIG. R image data for one frame, which is a pixel signal (digital value) corresponding to. Similarly, the demosaicing unit 330 receives a pixel signal corresponding to green (G) visible light at a position other than the G pixel in the pixel unit 1321 by a demosaicing process corresponding to the G pixel. Interpolation is performed based on the pixel signal (digital value) output by the pixel. Accordingly, the demosaicing unit 330 causes the pixel signals (digital values) at the positions of all the pixels 1320 arranged in the pixel unit 1321 to be the pixel signals output by the G pixel, as illustrated in FIG. G image data for one frame, which is a pixel signal (digital value) corresponding to. Similarly, the demosaicing unit 330 generates a pixel signal corresponding to visible light of blue (B) at a position where pixels other than the B pixel are arranged in the pixel unit 1321 by a demosaicing process corresponding to the B pixel. Interpolation is performed based on the pixel signal (digital value) output by the pixel. Accordingly, the demosaicing unit 330 causes the pixel signals (digital values) at the positions of all the pixels 1320 arranged in the pixel unit 1321 to be pixel signals output from the B pixel, as illustrated in FIG. B image data for one frame, which is a pixel signal (digital value) corresponding to. Similarly, the demosaicing unit 330 outputs a pixel signal corresponding to near-infrared light at a position other than the IR pixel in the pixel unit 1321 by the demosaicing process corresponding to the IR pixel. Interpolation is performed based on the pixel signal (digital value). Accordingly, the demosaicing unit 330 causes the pixel signals (digital values) at the positions of all the pixels 1320 arranged in the pixel unit 1321 to be pixel signals output from the IR pixels, as illustrated in FIG. IR image data for one frame, which is a pixel signal (digital value) corresponding to.

Subsequently, the demosaicing unit 330 performs offset processing based on the generated IR image data on each of the generated R image data, G image data, and B image data. FIG. 9A schematically shows an offset process performed on the R image data, FIG. 9B schematically shows an offset process performed on the G image data, and FIG. 4 schematically shows offset processing to be performed.

More specifically, the demosaicing unit 330 is generated from the pixel signal (digital value) at the position of each R pixel in the generated R image data ((b) in FIG. 8 = (a) in FIG. 9A). The pixel signal (digital value) of the IR pixel at the same position in the IR image data ((e) in FIG. 8 = (b) in FIG. 9A) is subtracted, and an offset process is performed as shown in (c) in FIG. 9A. The final R image data for one frame consisting only of the pixel signal R ′ (digital value) is generated. Similarly, the demosaicing unit 330 generates the IR from the pixel signal (digital value) at the position of each G pixel in the generated G image data ((c) in FIG. 8 = (a) in FIG. 9B). The pixel signal (digital value) of the IR pixel at the same position in the image data ((e) of FIG. 8 = (b) of FIG. 9B) is subtracted, and the offset processing is performed as shown in (c) of FIG. 9B. Final G image data for one frame including only the pixel signal G ′ (digital value) is generated. Similarly, the demosaicing unit 330 generates the IR generated from the pixel signal (digital value) at the position of each B pixel in the generated B image data ((d) in FIG. 8 = (a) in FIG. 9C). The pixel signal (digital value) of the IR pixel at the same position in the image data ((e) of FIG. 8 = (b) of FIG. 9C) is subtracted, and the offset processing as shown in (c) of FIG. 9C is performed. Final B image data for one frame consisting only of the pixel signal B ′ (digital value) is generated.

By such processing, the demosaicing unit 330 performs final R image data generated by subtracting IR image data from each of R image data, G image data, and B image data, G image data, Each of the image data is output to the white balance unit 331. Then, the white balance unit 331 performs white balance processing, and the gamma correction unit 332 performs gamma offset processing, so that the inspected object 901 photographed when the photographing mode of the fluorescence observation endoscope apparatus 1 is the normal photographing mode. Can be displayed on the color monitor 40 with the correct color.

In the fluorescence observation endoscope apparatus 1, the image sensor 132 captures an IR cut filter, which is an optical filter that is included in a normal image capturing apparatus and removes (cuts) near-infrared light that is unnecessary in photographing with visible light. It cannot be arranged on the lens 130 side. This is because the fluorescence observation endoscope apparatus 1 observes the fluorescence emitted by the ICG, and thus the fluorescence is observed when the IR cut filter is disposed on the light incident side of the image sensor 132. This is because it becomes impossible. However, in the fluorescence observation endoscope apparatus 1, the demosaicing unit 330 performs the above-described offset processing, so that it is not necessary for photographing with ambient light that leaks when photographing in the normal photographing mode, that is, photographing with visible light. Removes infrared light components. As a result, the fluorescence observation endoscope apparatus 1 improves the color reproducibility of the image of the taken inspected object 901 even when ambient light leaks when the inspected object 901 is imaged in the normal imaging mode. Can do.

Subsequently, an image processing operation in the demosaicing unit 330 when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode will be described with reference to FIGS. 10 and 11A to 11C. When the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, the mode determination unit 31 included in the external processing unit 30 includes information indicating that the imaging mode is the fluorescence imaging mode in the image processing unit 33. To the demosaicing unit 330. Then, the demosaicing unit 330 responds to the information indicating that the fluorescence imaging mode is input from the mode determination unit 31, as in the case where the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode. A demosaicing process for generating RGB image data and IR image data from the parallel raw data input from the deserializer 32 is performed. Thereafter, the demosaicing unit 330 performs an offset process for correcting the generated RGB image data based on the IR image data. FIG. 10 schematically shows a demosaicing process performed by the demosaicing unit 330 when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode. FIGS. 11A to 11C show fluorescence observation endoscopes. An offset process performed by the demosaicing unit 330 when the imaging mode of the mirror device 1 is the fluorescence imaging mode is schematically shown.

As described above, even when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, the image sensor 132 uses the serial RAW data representing the magnitude of the pixel signal output from each pixel 1320 as the imaging signal. To the external processing unit 30. Therefore, the deserializer 32 restores the input serial RAW data to the original parallel RAW data and outputs the original parallel RAW data to the demosaicing unit 330, as in the case where the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode. . That is, as shown in FIG. 10A, the deserializer 32 is restored to a state in which each of the R pixel, the G pixel, the B pixel, or the IR pixel is arranged at each position in the pixel unit 1321. The RAW data is output to the demosaicing unit 330.

Similar to the case where the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, the demosaicing unit 330 performs demosaicing processing on the parallel RAW data input from the deserializer 32 to obtain G image data and , B image data and IR image data are generated.

Similar to the case where the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, the demosaicing unit 330 performs the demosaicing process corresponding to the G pixel as shown in FIG. G image data for a frame is generated. Similarly, the demosaicing unit 330 performs the demosaicing process corresponding to the B pixel as shown in FIG. 10C as in the case where the photographing mode of the fluorescence observation endoscope apparatus 1 is the normal photographing mode. As described above, B image data for one frame is generated. Similarly, the demosaicing unit 330 performs the demosaicing process corresponding to the IR pixel as shown in FIG. 10D as in the case where the photographing mode of the fluorescence observation endoscope apparatus 1 is the normal photographing mode. As described above, one frame of IR image data is generated.

Subsequently, the demosaicing unit 330 generates the generated IR for each of the generated G image data and B image data in the same manner as when the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode. Perform offset processing based on image data. Further, the demosaicing unit 330 outputs the generated IR image data to the white balance unit 331 as R image data. FIG. 11A schematically illustrates offset processing performed on the G image data, and FIG. 11B schematically illustrates offset processing performed on the B image data. FIG. 11C schematically shows processing for handling IR image data as R image data.

More specifically, the demosaicing unit 330 generates the generated G image data ((b) in FIG. 10 = ((b) in FIG. 11A) as in the case where the photographing mode of the fluorescence observation endoscope apparatus 1 is the normal photographing mode. The IR image data generated from (a)) ((d) in FIG. 10 = (b) in FIG. 11A) is subtracted, and the pixel signal G ′ ( Final G image data for one frame consisting only of digital values) is generated. Similarly, similarly to the case where the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, the demosaicing unit 330 generates the generated B image data ((c) in FIG. 10 = (a in FIG. 11B)). )) Is subtracted from the IR image data (FIG. 10 (d) = FIG. 11B (b)), and the offset-processed pixel signal B ′ (digital) as shown in FIG. 11B (c). Final B image data for one frame consisting only of (value) is generated. Further, the demosaicing unit 330 obtains IR image data ((d) in FIG. 10 = (a) in FIG. 11C) from only the pixel signal R ′ (digital value) as shown in (b) in FIG. 11C. The final R image data for one frame.

Through such processing, the demosaicing unit 330 converts the final G image data and B image data generated by subtracting the IR image data from each of the G image data and the B image data, and the R image data into the R image data. The final R image data as image data is output to the white balance unit 331. Then, the white balance unit 331 performs white balance processing, and the gamma correction unit 332 performs gamma offset processing, so that the inspected object 901 photographed when the photographing mode of the fluorescence observation endoscope apparatus 1 is the fluorescent photographing mode. In this image, the portion where the ICG is excited and fluoresced can be made conspicuous and displayed on the color monitor 40. More specifically, it is possible to display on the color monitor 40 a reddish color where the ICG is excited to emit fluorescence.

Next, the white balance unit 331 constituting the fluorescence observation apparatus of the present invention in the fluorescence observation endoscope apparatus 1 will be described. As described above, when the white balance is photographed, the white balance unit 331 is the magnitude of the digital value of the pixel signal corresponding to the pixel at the same position in each of the R image data, the G image data, and the B image data. However, white balance processing is performed so that the same value is obtained. That is, white balance processing is performed so that a white subject is displayed in white on the color monitor 40.

When the imaging mode of the fluorescence observation endoscope apparatus 1 is the normal imaging mode, the light source device 20 has visible light in the blue (B) wavelength band and green (G) wavelength band as shown in FIG. 7A. The inspected object 901 is irradiated with white light whose emission intensity is the same for both visible light and visible light in the red (R) wavelength band. For this reason, when the white balance unit 331 performs white balance processing on the R image data, G image data, and B image data obtained by photographing the object 901 in the normal photographing mode, R The level is adjusted by multiplying the gain value so that the digital values of the pixel signals included in each of the image data, the G image data, and the B image data are approximately the same. Therefore, when this level adjustment is applied to all locations, the digital values of the pixel signals included in each of the R image data, the G image data, and the B image data in the locations other than the white subject are It becomes a digital value corresponding to the color, and is displayed on the color monitor 40 with the correct color.

On the other hand, when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, the light source device 20 has visible light in the blue (B) wavelength band and green (G) as shown in FIG. 7B. Each emission intensity with visible light in the wavelength band is the same weak value, does not include visible light in the red (R) wavelength band, and the emission intensity of near-infrared light in the excitation light wavelength band is blue (B). The object 901 is irradiated with light having a value stronger than the emission intensity of green (G), that is, light that is not white light. Here, in the white balance processing, it is assumed that a white subject is assumed and a fluorescent agent such as ICG is not assumed to be a subject. Therefore, as described above, the light source includes near infrared light in the wavelength band of the excitation light. However, the image sensor 132 is not irradiated with fluorescence, and the reflected light from the subject of the excitation light is also cut by the excitation light cut filter 131, so that the light reaching the image sensor 132 is blue (B). Only visible light in the wavelength band and visible light in the green (G) wavelength band are provided. In other words, the R image data (IR image data) in the white balance processing includes visible light in the blue (B) wavelength band and visible light in the green (G) wavelength band, and the on-chip color filter 13211IR and the dielectric multilayer filter. Only the leaked light component that has passed through 13209 is weak. For this reason, when the white balance unit 331 performs white balance processing on the R image data (IR image data), G image data, and B image data obtained by photographing the object 901 in the fluorescence photographing mode, a white subject is obtained. In the part, the digital value of the pixel signal included in the R image data (IR image data) is largely adjusted to a level equivalent to the digital value of the pixel signal at the same position included in each of the G image data or the B image data. (Multiply by a larger gain value) to reproduce a white subject. Therefore, when the fluorescence observation endoscope apparatus 1 is imaged in the fluorescence imaging mode, at the location where the ICG is excited to emit fluorescence, in addition to the above-described leakage light component, the fluorescence component further excited by the ICG Since the level adjustment is performed on the digital value of the image signal, the image signal of the R image data (IR image data) is a pixel at the same position included in each of the G image data or the B image data. The digital value is larger than the digital value of the signal.

That is, by making the digital value of the pixel signal included in the R image data (IR image data) where the ICG is excited and emitting fluorescence, a reddish color according to the fluorescence level. (Highlighted in red) and displayed on the color monitor 40. In other words, when the imaging mode of the fluorescence observation endoscope apparatus 1 is the fluorescence imaging mode, the white balance processing by the white balance unit 331 causes the R pixel (IR pixel) of the portion where the ICG is excited to emit fluorescence. By making the level of the pixel signal relatively higher than the level of the pixel signal of the G pixel or B pixel at the same position, the fluorescent light emission portion is highlighted in red and displayed on the color monitor 40.

It should be noted that at a location where the ICG is not excited and does not emit fluorescence, the digital value of the pixel signal included in the R image data (IR image data) is a digital value of a size that reproduces a white subject at most, that is, G Since it is a digital value equal to or less than the digital value of the pixel signal at the same position included in each of the image data or the B image data, it is emphasized in red and is not displayed on the color monitor 40.

In the present embodiment, as described above, the fluorescent emission portion is highlighted in red and displayed on the color monitor 40. However, the display method in the fluorescence observation apparatus of the present invention is not limited to this, and other Different display methods may be adopted based on this signal processing. For example, if the image processing unit 33 performs processing for transmitting data to the digital / analog conversion unit 34 by exchanging R image data (IR image data) and G image data only in the fluorescence photographing mode, the fluorescent light is emitted. The location is displayed in green with high sensitivity for humans, making it easier to identify the fluorescence emission location.

Here, an example of the value of the level displayed as a location where the ICG is excited to emit fluorescence will be described. In the following description, the level value of the pixel signal obtained by photoelectrically converting the light corresponding to each pixel 1320 will be described as an integer without a unit.

As described above, the optical filter in which the on-chip color filter 13211IR and the dielectric multilayer filter 13209 are combined attenuates visible light contained in incident fluorescent light, but transmits visible light to a small extent. (See FIG. 6A). That is, the pixel signal output from the IR pixel includes a small amount of transmitted visible light components (visible light in the blue (B) wavelength band and visible light in the green (G) wavelength band). Therefore, for example, the transmittance of light in the on-chip color filter 13210B for visible light in the blue (B) wavelength band, and the transmittance of light in the optical filter in which the on-chip color filter 13211IR and the dielectric multilayer filter 13209 are combined. Let us consider a case where the ratio is assumed to be 100: 1. For example, the transmittance of light in the on-chip color filter 13210G with respect to visible light in the green (G) wavelength band, and the transmittance of light in the optical filter in which the on-chip color filter 13211IR and the dielectric multilayer filter 13209 are combined. Let us consider a case where the ratio is assumed to be 100: 1. In other words, the transmittance with respect to visible light in the blue (B) wavelength band and visible light in the green (G) wavelength band in the optical filter including the on-chip color filter 13211IR and the dielectric multilayer filter 13209 is on-chip. Consider a case where it is assumed that the color filter 13210B and the on-chip color filter 13210G are each 1%. The state before the white balance unit 331 performs white balance processing, that is, the level of the pixel signal at the location of the white subject included in each of the B image data and G image data output from the demosaicing unit 330, That is, a case is assumed in which the pixel signal levels of the B pixel and the G pixel are each “4000”.

In this case, the level of the pixel signal at the location of the white subject included in the R image data (IR image data), that is, the level of the pixel signal at the location of the white subject in the R pixel (IR pixel) is the pixel of the B pixel. From the signal level and the pixel signal level of the G pixel, the following equation (1) is obtained.

(4000/100) + (4000/100) = 80 (1)

Therefore, the white balance unit 331 performs the white balance processing for the level “80” of the pixel signal of the R pixel (IR pixel) at the location of the white subject in the above equation (1), and the white balance of the R pixel (IR pixel). A white subject is reproduced by adjusting the level of the pixel signal at the location of the subject so as to be a multiple of the following equation (2).

4000/80 = 50 times (2)

In other words, a white subject is reproduced by multiplying the gain value by which the level of the pixel signal of the R subject (IR pixel) at the location of the white subject is multiplied by 50 as shown in the above equation (2). .

At this time, for example, in the image of the inspection object 901 photographed in the fluorescence photographing mode, the pixel signal levels of the B pixel and the G pixel at a location where the ICG is not excited to emit fluorescence emit “600”, respectively. The level of the pixel signal of the R pixel (IR pixel) at this location is expressed by the following equation (3).

600/100 + 600/100 = 1 2 (3)

For this reason, the white balance unit 331 adjusts the level of the pixel signal of the R pixel (IR pixel) by white balance processing from the above equation (3) and the above equation (2) to the following equation (4). To do.

12 × 50 = 600 (4)

As a result, in the part where the ICG is excited and not emitting fluorescence, the ratio of the level of the pixel signal of the B pixel, the G pixel, and the R pixel (IR pixel) is expressed by the following equation (5). , Reproduce the white subject.

B pixel: G pixel: R pixel (IR pixel) = 600: 600: 600
= 1: 1: 1 (5)

On the other hand, in the image of the inspected object 901 photographed in the fluorescence photographing mode, the intensity of visible light (light in the green wavelength band) incident on the image sensor 132 and the fluorescence at the location where the ICG is excited to emit fluorescence. A case is assumed in which the ratio of intensity is 15: 1 and the sensitivity ratio of visible light and fluorescence in each pixel 1320 arranged in the image sensor 132 is 2: 1. In this case, the level of the pixel signal of the R pixel (IR pixel) is expressed by the following formula with respect to the level of the pixel signal of the B pixel or G pixel by exposing (detecting) the fluorescence emitted by the ICG to emit fluorescence. Increase by the level shown in (6).

600 / (15 × 2) = 20 (6)

Therefore, the level of the pixel signal of the R pixel (IR pixel) is represented by the following equation (7) from the above equation (3) and the above equation (6).

12 + 20 = 32 (7)

For this reason, the white balance unit 331 changes the level of the pixel signal of the R pixel (IR pixel) to a multiple such as the following expression (8) from the above expression (7) and the above expression (2) by white balance processing. Adjust the level.

32 × 50 = 1600 (8)

As a result, at the location where the ICG is excited to emit fluorescence, the ratio of the pixel signal levels of the B pixel, G pixel, and R pixel (IR pixel) is as shown in the following equation (9). The portion where the ICG is excited and fluoresced becomes a reddish color (highlighted in red) and is displayed on the color monitor 40.

B pixel: G pixel: R pixel (IR pixel) = 600: 600: 1600
= 1: 1: 2.7 (9)

In the actual image of the inspection object 901, the level of the pixel signal of the B pixel and the level of the pixel signal of the G pixel are rarely the same level. Here, an example of a case closer to the actual image of the inspection object 901 will be described. For example, if the level of the pixel signal of the B pixel is “600” and the level of the pixel signal of the G pixel is “1200” at a location where the ICG in the image of the inspection object 901 is not excited to emit fluorescence. The level of the pixel signal of the R pixel (IR pixel) at this location is expressed by the following equation (10).

600/100 + 1200/100 = 18 (10)

Therefore, the white balance unit 331 adjusts the level of the pixel signal of the R pixel (IR pixel) by white balance processing as shown in the following equation (11) from the above equation (10) and the above equation (2). To do.

18 × 50 = 900 (11)

As a result, in the portion where the ICG is excited and not emitting fluorescence, the ratio of the level of the pixel signal of the B pixel, the G pixel, and the R pixel (IR pixel) is expressed by the following equation (12). Even when there is a difference in the pixel signal level between the B pixel and the G pixel, the portion where the ICG is excited and does not emit fluorescence becomes a reddish color (highlighted in red) and is displayed on the color monitor 40. There is nothing.

B pixel: G pixel: R pixel (IR pixel) = 600: 1200: 900
= 1: 2: 1.5 (12)

On the other hand, in the part where the ICG in the image of the inspection object 901 is excited and emits fluorescence, the level of the pixel signal of the R pixel (IR pixel) is G when the conditions in the image sensor 132 are the same as those described above. The level of the pixel signal of the pixel increases by the level shown in the following equation (13).

1200 / (15 × 2) = 40 (13)

Therefore, the level of the pixel signal of the R pixel (IR pixel) is expressed by the following equation (14) from the above equation (10) and the above equation (13).

18 + 40 = 58 (14)

For this reason, the white balance unit 331 changes the level of the pixel signal of the R pixel (IR pixel) to a multiple such as the following equation (15) from the above equation (14) and the above equation (2) by white balance processing. Adjust the level.

58 × 50 = 2900 (15)

As a result, at the location where the ICG is excited to emit fluorescence, the ratio of the level of the pixel signal of the B pixel, the G pixel, and the R pixel (IR pixel) is expressed by the following equation (16). The portion where the ICG is excited and fluoresced becomes a reddish color (highlighted in red) and is displayed on the color monitor 40.

B pixel: G pixel: R pixel (IR pixel) = 600: 1200: 2900
= 1: 2: 4.8 (16)

Thus, in the fluorescence observation endoscope apparatus 1, the IR image data that is the level of the weak pixel signal is set as the R image data. In the fluorescence observation endoscope apparatus 1, the white balance unit 331 performs white balance processing on the final G image data, B image data, and R image data (IR image data) output from the demosaicing unit 330. I do. Thus, in the fluorescence observation endoscope apparatus 1, the color monitor 40 emphasizes the reddish color in the portion where the ICG in the image of the inspection object 901 photographed in the fluorescence photographing mode is excited and fluoresced. Can be displayed. As a result, the fluorescence observation endoscope apparatus 1 excites the ICG administered to the examinee with the excitation light, and displays an image of the inspected object 901 that highlights the location where the excited ICG emits fluorescence. 40 can be displayed to the tester.

In order to increase the level of the pixel signal included in the R image data (IR image data) at a location where the ICG is excited and emits fluorescence, for example, the imaging mode of the fluorescence observation endoscope apparatus 1 is fluorescent. Near-infrared light emission intensity in the wavelength band of excitation light emitted from the light source device 20 in the photographing mode, and visible light emission in the blue (B) wavelength band and green (G) wavelength band. You may make the difference with an intensity | strength larger. That is, the near infrared light emission intensity in the wavelength band of the excitation light emitted from the light source device 20 is increased, and the difference between the emission intensity of visible light in the blue (B) and green (G) wavelength bands is increased. May be. As a result, the illuminated illumination light is reflected by the inspection object 901 and incident on the image sensor 132, and the visible light in the blue (B) wavelength band and the visible light in the green (G) wavelength band and the ICG are excited. The difference in light intensity between the fluorescence emitted and the fluorescence incident on the image sensor 132 is reduced, and the sensitivity (detection sensitivity) at which the image sensor 132 detects fluorescence can be improved.

More specifically, in the above-described calculation, the ratio of the intensity of visible light (green wavelength band light) incident on the image sensor 132 and the intensity of fluorescence is 15: 1, but the light source device 20 emits light. By making the emission intensity of near infrared light in the wavelength band of excitation light stronger than the emission intensity of visible light in the blue (B) wavelength band and visible light in the green (G) wavelength band, the ratio is reduced. can do. For example, if the ratio of the intensity of visible light (light in the green wavelength band) incident on the image sensor 132 and the intensity of fluorescence can be set to 10: 1, the above expression (6) and the above expression (13) The amount of increase can be increased, and the level of the pixel signal included in the R image data (IR image data) at the location where the ICG is excited to emit fluorescence can be increased.

Further, as described above, the image sensor 132 may be configured such that the dielectric multilayer filter layer 13203 is not provided (see FIG. 4). Even in the fluorescence observation endoscope apparatus 1 having the configuration in which the image sensor 132 having this configuration is provided in the imaging unit 13, a value capable of allowing an attenuation rate (light shielding rate) with respect to visible light in a wavelength band of 400 nm to 600 nm. If it can be reduced to, an image of the inspected object 901 that highlights the portion where the ICG is excited to emit fluorescence can be displayed on the color monitor 40 and presented to the inspector.

(Modification)
Here, the imaging unit 13 is provided with the image sensor 132 in which the dielectric multilayer filter layer 13203 is not provided, that is, the dielectric multilayer filter 13209 is not provided, thereby improving sensitivity to light and oblique incidence characteristics. The fluorescence observation endoscope apparatus 1 having the configuration will be described. First, an image sensor 132 (hereinafter referred to as “image sensor 133”) that does not include the dielectric multilayer filter 13209 will be described. Note that the configuration of the fluorescence observation endoscope apparatus 1 (hereinafter referred to as “fluorescence observation endoscope apparatus 2”) having a configuration in which the image sensor 133 is provided in the imaging unit 13 is the same as the fluorescence observation endoscope apparatus illustrated in FIG. 1, the image sensor 132 only replaces the image sensor 133, and detailed description thereof is omitted. The configuration of the image sensor 133 is the same as that of the image sensor 132 shown in FIG.

FIG. 12 is a cross-sectional view showing an example of another structure (structure of the image sensor 133) of the solid-state imaging device provided in the fluorescence observation endoscope apparatus 2 according to the embodiment of the present invention. In FIG. 12, similarly to the image sensor 132 illustrated in FIG. 4, a vertical portion of a part of the pixel portion 1321 when the R pixel, the G pixel, or the B pixel and the IR pixel are formed in the image sensor 133. The structure is shown. Similarly to the image sensor 132, the image sensor 133 illustrated in FIG. 12 is a back-illuminated solid-state imaging device.

As shown in FIG. 12, the structure of the image sensor 133 is a structure in which the dielectric multilayer filter layer 13203 in the structure of the image sensor 133 shown in FIG. Each layer constituting the pixel 1330 which is a pixel arranged at 133 is laminated. More specifically, a wiring layer 13201, a photoelectric conversion layer 13202, an on-chip color filter layer 13204, and a microlens 13205 are formed from the support substrate 13200 toward the light incident side. Since the configuration of each layer is the same as the configuration of each layer in the image sensor 133, detailed description is omitted.

In the image sensor 133, the arrangement of each on-chip color filter formed in the on-chip color filter layer 13204 and the characteristics (spectral characteristics) with respect to the wavelength band of light are also the same as those in the image sensor 133 shown in FIGS. 5A and 6A. This is the same as the arrangement and spectral characteristics of the on-chip color filter. However, in the image sensor 133, since the dielectric multilayer filter layer 13203 is deleted, the transparent material 13208 at a position corresponding to the pixel 1330a that is an R pixel, a G pixel, or a B pixel, and a pixel 1330b that is an IR pixel. The dielectric multilayer filter 13209 at the position corresponding to is not formed. Therefore, in the image sensor 133, the arrangement and spectral characteristics of the dielectric multilayer filter 13209 in the image sensor 132 shown in FIGS. 5B and 6B are not related. Therefore, the photoelectric conversion element 13207 of the IR pixel arranged in the image sensor 133 exposes light (fluorescence) in the near-infrared wavelength band transmitted by the spectral characteristic IR1 of the corresponding on-chip color filter 13211IR to perform photoelectric conversion. The pixel signal is output. Then, the image sensor 133 outputs serial RAW data representing the magnitude of the pixel signal exposed and photoelectrically converted by each pixel 1330 to the external processing unit 30 as an imaging signal.

The operations and processing methods of other components such as the light source device 20 and the image processing unit 33 provided in the fluorescence observation endoscope apparatus 2 are the same as those of the fluorescence observation endoscope apparatus 1. Therefore, in the fluorescence observation endoscope apparatus 2, similarly to the fluorescence observation endoscope apparatus 1, the portion of the inspected object 901 in which the ICG is excited and the fluorescent light is emitted has a reddish color (highlighted in red). The image is displayed on the color monitor 40. However, in the fluorescence observation endoscope apparatus 2, the result of the white balance processing performed by the white balance unit 331 is that the dielectric multilayer filter 13209 included in the image sensor 132 in the fluorescence observation endoscope apparatus 1 is deleted. A little different with it.

Here, the white balance part 331 which comprises the fluorescence observation apparatus of this invention in the fluorescence observation endoscope apparatus 2 is demonstrated. In the following description, as with the fluorescence observation endoscope apparatus 1, an example of level values displayed as locations where the ICG is excited to emit fluorescence will be described. In the following description, similarly to the fluorescence observation endoscope apparatus 1 described above, a level value representing the magnitude of a pixel signal obtained by photoelectric conversion of light corresponding to each pixel 1330 will be described.

For example, it is assumed that the ratio of the light transmittance of the on-chip color filter 13210B to the visible light in the blue (B) wavelength band and the light transmittance of the on-chip color filter 13211IR is 20: 1. Think. For example, it is assumed that the ratio of the light transmittance of the on-chip color filter 13210G to the visible light in the green (G) wavelength band and the light transmittance of the on-chip color filter 13211IR is 20: 1. Think about the case. In other words, each of the optical filters including the on-chip color filter 13211IR and the dielectric multilayer filter 13209 has a transmittance of 5% for visible light in the blue (B) wavelength band and visible light in the green (G) wavelength band, respectively. Suppose that The state before the white balance unit 331 performs white balance processing, that is, the level of the pixel signal at the location of the white subject included in each of the B image data and G image data output from the demosaicing unit 330, That is, a case is assumed in which the pixel signal levels of the B pixel and the G pixel are each “4000”.

In this case, the level of the pixel signal at the location of the white subject included in the R image data (IR image data), that is, the level of the pixel signal at the location of the white subject in the R pixel (IR pixel) is B pixel and G From the level of each pixel signal of the pixel, the following equation (17) is obtained.

(4000/20) + (4000/20) = 400 (17)

Therefore, the white balance unit 331 performs the white balance processing for the level “400” of the pixel signal of the R pixel (IR pixel) at the location of the white subject in the above equation (17), and the white balance of the R pixel (IR pixel). A white subject is reproduced by adjusting the level of the pixel signal at the location of the subject so as to be a multiple of the following equation (18).

4000/400 = 10 times (18)

In other words, a white object is reproduced by multiplying the gain value by which the level of the pixel signal of the R pixel (IR pixel) at the location of the white object is multiplied by 10 as shown in the above equation (18). .

At this time, for example, in the image of the inspection object 901 photographed in the fluorescence photographing mode, the pixel signal levels of the B pixel and the G pixel at a location where the ICG is not excited to emit fluorescence emit “600”, respectively. The level of the pixel signal of the R pixel (IR pixel) at this location is expressed by the following equation (19).

600/20 + 600/20 = 60 (19)

For this reason, the white balance unit 331 adjusts the level of the pixel signal of the R pixel (IR pixel) by white balance processing from the above equation (19) and the above equation (18) to the following equation (20). To do.

60 × 10 = 600 (20)

As a result, the ratio of the level of the pixel signal of the B pixel, the G pixel, and the R pixel (IR pixel) in the portion where the ICG is not excited and does not emit fluorescence is expressed by the following equation (21). , Reproduce the white subject.

B pixel: G pixel: R pixel (IR pixel) = 600: 600: 600
= 1: 1: 1 (21)

On the other hand, as in the case of the fluorescence observation endoscope apparatus 1 described above, in the image of the inspected object 901 photographed in the fluorescence photographing mode, in the image sensor 132 described above, at a location where the ICG is excited and emits fluorescence. Similar to the conditions, the ratio of the intensity of visible light incident on the image sensor 133 to the intensity of fluorescence is 15: 1, and the ratio of the sensitivity of visible light and fluorescence in each pixel 1330 arranged in the image sensor 133 is 2. Suppose the case of: 1. In this case, the level of the pixel signal of the R pixel (IR pixel) is expressed by the following formula with respect to the level of the pixel signal of the B pixel or G pixel by exposing (detecting) the fluorescence emitted by the ICG to emit fluorescence. Increases by the level shown in (22).

600 / (15 × 2) = 20 (22)

Therefore, the level of the pixel signal of the R pixel (IR pixel) is expressed by the following equation (23) from the above equation (19) and the above equation (22).

60 + 20 = 80 (23)

For this reason, the white balance unit 331 sets the level of the pixel signal of the R pixel (IR pixel) to a multiple such as the following expression (24) from the above expression (23) and the above expression (18) by white balance processing. Adjust the level.

80 × 10 = 800 (24)

As a result, at the location where the ICG is excited to emit fluorescence, the ratio of the pixel signal levels of the B pixel, G pixel, and R pixel (IR pixel) is as shown in the following equation (9). The portion where the ICG is excited and fluoresced becomes a reddish color (highlighted in red) and is displayed on the color monitor 40.

B pixel: G pixel: R pixel (IR pixel) = 600: 600: 800
= 1: 1: 1.3 (25)

In the case where the image is closer to the actual image of the inspection object 901, for example, the level of the pixel signal of the B pixel at a location where the ICG in the image of the inspection object 901 is excited and not emitting fluorescence is “600”, If the level of the pixel signal of the G pixel is “1200”, the level of the pixel signal of the R pixel (IR pixel) at this location is expressed by the following equation (26).

600/20 + 1200/20 = 90 (26)

Therefore, the white balance unit 331 adjusts the level of the pixel signal of the R pixel (IR pixel) by white balance processing as shown in the following equation (27) from the above equation (26) and the above equation (18). To do.

90 × 10 = 900 (27)

As a result, in the part where the ICG is excited and does not emit fluorescence, the ratio of the level of the pixel signal of the B pixel, the G pixel, and the R pixel (IR pixel) is expressed by the following equation (28). Even when there is a difference in the pixel signal level between the B pixel and the G pixel, the portion where the ICG is excited and does not emit fluorescence becomes a reddish color (highlighted in red) and is displayed on the color monitor 40. There is nothing.

B pixel: G pixel: R pixel (IR pixel) = 600: 1200: 900
= 1: 2: 1.5 (28)

On the other hand, in the part where the ICG in the image of the inspection object 901 is excited and emits fluorescence, the level of the pixel signal of the R pixel (IR pixel) is G when the conditions in the image sensor 133 are the same as those described above. The level is increased by the level shown in the following equation (29) with respect to the level of the pixel signal of the pixel.

1200 / (15 × 2) = 40 (29)

Therefore, the level of the pixel signal of the R pixel (IR pixel) is represented by the following equation (30) from the above equation (26) and the above equation (29).

90 + 40 = 130 (30)

For this reason, the white balance unit 331 sets the level of the pixel signal of the R pixel (IR pixel) to a multiple such as the following equation (31) from the above equation (30) and the above equation (18) by white balance processing. Adjust the level.

130 × 10 = 1300 (31)

As a result, at the location where the ICG is excited to emit fluorescence, the ratio of the level of the pixel signal of the B pixel, the G pixel, and the R pixel (IR pixel) is expressed by the following equation (32). The portion where the ICG is excited and fluoresced becomes a reddish color (highlighted in red) and is displayed on the color monitor 40.

B pixel: G pixel: R pixel (IR pixel) = 600: 1200: 1300
= 1: 2: 2.2 (32)

As described above, the fluorescence observation endoscope apparatus 2 includes the image sensor 133 in which the sensitivity to light and the oblique incidence characteristic are improved by not including the dielectric multilayer filter 13209 in the imaging unit 13. In the fluorescence observation endoscope apparatus 2, as in the fluorescence observation endoscope apparatus 1, IR image data that is a weak pixel signal level is used as R image data, and the final output from the demosaicing unit 330 is performed. The white balance unit 331 performs white balance processing on the G image data, B image data, and R image data (IR image data). As a result, in the fluorescence observation endoscope apparatus 2, as in the fluorescence observation endoscope apparatus 1, the location where the ICG in the image of the inspection object 901 photographed in the fluorescence imaging mode is excited to emit fluorescence. It can be displayed on the color monitor 40 with emphasis on the reddish color.

As can be seen from the above-described example (comparison between the above equation (16) and the above equation (32)), in the fluorescence observation endoscope device 2, the ICG is excited as compared with the fluorescence observation endoscope device 1. The degree of emphasizing the fluorescent emission part to a reddish color is small. That is, the sensitivity to the fluorescence emitted by the excitation of the ICG is lower than that of the fluorescence observation endoscope apparatus 1. For this reason, it is considered that the imaging unit 13 preferably includes an image sensor 132 that can detect fluorescence with higher accuracy, that is, can increase sensitivity to fluorescence. However, the image sensor 133 provided in the imaging unit 13 in the fluorescence observation endoscope apparatus 2 has higher sensitivity to light and oblique incidence characteristics than the image sensor 132 provided in the imaging unit 13 in the fluorescence observation endoscope apparatus 1. It is a solid-state imaging device. As described above, for example, the near-infrared light emission intensity in the wavelength band of the excitation light emitted from the light source device 20, the visible light in the blue (B) wavelength band, and the visible light in the green (G) wavelength band. By increasing the difference from the light emission intensity, the level of the pixel signal included in the R image data (IR image data) can be increased. Therefore, in the above-described example, although the degree of highlighting the portion where the ICG in the fluorescence observation endoscope apparatus 2 is excited and fluorescent light is emphasized in a reddish color, the emission intensity of near infrared light in the wavelength band of the excitation light is reduced. By adjusting, the fluorescence observation endoscope apparatus 2 can emphasize the portion where the ICG is excited and fluorescent light is emitted to the same degree as the fluorescence observation endoscope apparatus 1. Moreover, in the fluorescence observation endoscope apparatus 2, since the sensitivity to light and the oblique incidence characteristic are improved by the image sensor 133, it is possible to capture a more delicate image of the inspection object 901 including the normal imaging mode. Can do.

In addition to the above-described processing, the white balance unit 331 determines whether the level of the pixel signal of the B pixel or the G pixel is higher than the level of the pixel signal of the R pixel (IR pixel). May be further multiplied by “a value obtained by dividing the pixel signal level of the B pixel and the G pixel by an average level” to adjust the level. Thereby, the fluorescence detection sensitivity can be further improved.

For example, if the level of the pixel signal of the B pixel is “600” and the level of the pixel signal of the G pixel is “1200” at a location where the ICG in the image of the inspection object 901 is not excited to emit fluorescence. The level of the pixel signal of the R pixel (IR pixel) at this location is “90” from the above equation (26).

Therefore, the white balance unit 331 adjusts the level of the pixel signal of the R pixel (IR pixel) to “900” from the above equation (27) by white balance processing. Further, the white balance unit 331 calculates the higher level (here, the level of the pixel signal of the G pixel) among the levels of the pixel signal of the B pixel or the G pixel, which is obtained by the following equation (33). The pixel signal level of the B pixel and the G pixel is multiplied by a value obtained by dividing the level of the pixel signal of the B pixel and the G pixel by the average level to adjust the level of the pixel signal of the R pixel (IR pixel) as shown in the following equation (34).

1200 / ((1200 + 600) / 2) = 1.33 (33)
900 × 1.33 = 1200 (34)

As a result, in the portion where the ICG is excited and not emitting fluorescence, the ratio of the level of the pixel signal of the B pixel, the G pixel, and the R pixel (IR pixel) is expressed by the following equation (35). Even when there is a difference in the pixel signal level between the B pixel and the G pixel, the portion where the ICG is excited and does not emit fluorescence becomes a reddish color (highlighted in red) and is displayed on the color monitor 40. There is nothing.

B pixel: G pixel: R pixel (IR pixel) = 600: 1200: 1200
= 1: 2: 2 (35)

On the other hand, in the part where the ICG in the image of the inspection object 901 is excited and emits fluorescence, the level of the pixel signal of the R pixel (IR pixel) is G when the conditions in the image sensor 133 are the same as those described above. The level of the pixel signal of the pixel is increased by “40” from the above equation (29). Therefore, the level of the pixel signal of the R pixel (IR pixel) is “130” from the above equation (30). For this reason, the white balance unit 331 determines the level of the pixel signal of the R pixel (IR pixel) by the white balance process from the above equation (30), the above equation (18), and the above equation (33). The level is adjusted to a multiple such as 36).

130 × 10 × 1.33 = 1733 (36)

As a result, at the location where the ICG is excited to emit fluorescence, the ratio of the level of the pixel signal of the B pixel, the G pixel, and the R pixel (IR pixel) is expressed by the following equation (37). The portion where the ICG is excited and fluoresced becomes a reddish color (highlighted in red) and is displayed on the color monitor 40.

B pixel: G pixel: R pixel (IR pixel) = 600: 1200: 1733
= 1: 2: 2.9 (37)

As can be seen from the above-described example (comparison between the above equation (32) and the above equation (37)), in the white balance unit 133, “B pixel or G pixel” with respect to the level of the pixel signal of the R pixel (IR pixel). Detecting fluorescence by adjusting the level by further multiplying by “the value obtained by dividing the higher one of the pixel signal levels of the pixel signal by the level obtained by averaging the pixel signal levels of the B pixel and the G pixel” Sensitivity can be improved.

According to the embodiment, excitation light (near infrared light) in a first wavelength band (for example, a wavelength band of 700 nm to 800 nm) that emits fluorescence by irradiating the phosphor (ICG), and the first wavelength Visible light in a second wavelength band different from the band (for example, a wavelength band from 500 nm to 600 nm) and a third wavelength band different from the first wavelength band and the second wavelength band (for example, from 400 nm to 500 nm). Light source device (light source device 20) capable of simultaneously irradiating visible light in the wavelength band, a first filter that selectively transmits fluorescence (on-chip color filter 13211IR), and visible light in the second wavelength band A second filter that selectively transmits light (on-chip color filter 13210G), and a third filter that selectively transmits visible light in the third wavelength band (on-chip color filter). Image processing device (image sensor 133) in which a plurality of light receiving elements (pixels 1320) provided with any of the first and second pixels 13210B) are provided, and image processing is performed based on pixel signals obtained by detecting light corresponding to each of the pixels 1320. And an image processing apparatus (image processing unit 33) to perform.

Further, according to the embodiment, the imaging device (image sensor 132) reflects a dielectric multilayer filter (dielectric) that reflects light in the third wavelength band from the second wavelength band and transmits light in the fluorescent wavelength band. And a multilayer multilayer filter 13209), and the dielectric multilayer filter 13209 is disposed on the optical path of the on-chip color filter 13211IR, thereby forming a fluorescence observation apparatus (fluorescence observation endoscope apparatus 1).

Further, according to the embodiment, the intensity of the excitation light emitted by the light source device 20 is stronger than the intensity of each visible light emitted by the light source device 20, and the fluorescence observation apparatus (the fluorescence observation endoscope apparatus 1 or the fluorescence observation). An endoscope apparatus 2) is configured.

In addition, according to the embodiment, the excitation light cut is arranged on the optical path from the subject irradiated with the excitation light to the imaging device (the image sensor 132 or the image sensor 133) and prevents the excitation light from entering the pixel 1320. A fluorescence observation apparatus (fluorescence observation endoscope apparatus 1 or fluorescence observation endoscope apparatus 2) further including a filter (excitation light cut filter 131) is configured.

According to the embodiment, the fluorescence includes a wavelength of near infrared light (for example, 830 nm), the visible light in the second wavelength band includes a wavelength of green light (for example, 540 nm), and the third The visible light in the wavelength band includes a fluorescence observation apparatus (fluorescence observation endoscope apparatus 1 or fluorescence observation endoscope apparatus 2) including a wavelength of blue light (for example, 450 nm).

In addition, according to the embodiment, the image processing apparatus (the white balance unit 331 included in the image processing unit 33) is a pixel corresponding to the light detected by the pixel 1320 from the on-chip color filter 13211IR via the on-chip color filter 13210B. A fluorescence observation device (fluorescence observation endoscope device 1 or fluorescence observation endoscope) that performs image processing (white balance processing) based on a signal level obtained by multiplying each of the signals by a predetermined value (gain value). A mirror device 2) is constructed.

In addition, according to the embodiment, the image sensor 132 or the image sensor 133 has a fourth wavelength band (for example, 600 nm) different from the wavelength band through which each of the on-chip color filter 13210B and the on-chip color filter 13210B selectively transmits. A plurality of pixels 1320 provided with a fourth filter (on-chip color filter 13210R) that selectively transmits visible light in a wavelength band (˜700 nm wavelength band) is further arranged, and the light source device 20 is connected to the first wavelength band from the first wavelength band. 3 in response to light detected by the pixel 1320 from the on-chip color filter 13211IR through the on-chip color filter 13210B by the image processing apparatus (white balance unit 331 included in the image processing unit 33). Image processing based on the respective pixel signals. The fluorescence photographing mode for performing (white balance processing) and the light source device 20 emit light of the second wavelength band to the fourth wavelength band, and the image processing device (white balance unit 331 included in the image processing unit 33). However, mode switching for switching between a normal shooting mode in which image processing (white balance processing) is performed based on each pixel signal corresponding to light detected by the pixel 1320 from the on-chip color filter 13210G via the on-chip color filter 13210R. A fluorescence observation apparatus (fluorescence observation endoscope apparatus 1 or fluorescence observation endoscope apparatus 2) further including an apparatus (imaging mode changeover switch 14) is configured.

According to the embodiment, the fluorescence includes a wavelength of near infrared light (for example, 830 nm), the visible light in the second wavelength band includes a wavelength of green light (for example, 540 nm), and the third The visible light in the wavelength band includes a blue light wavelength (for example, 450 nm), and the visible light in the fourth wavelength band includes a red light wavelength (for example, 630 nm). An endoscope apparatus 1 or a fluorescence observation endoscope apparatus 2) is configured.

In addition, according to the embodiment, the image processing apparatus (the demosaicing unit 330 included in the image processing unit 33) is a pixel corresponding to the light detected by the pixel 1320 from the on-chip color filter 13210G via the on-chip color filter 13210R. A fluorescence observation apparatus (fluorescence observation endoscope apparatus 1 or fluorescence observation) that performs offset processing on each of the signals based on each of the pixel signals corresponding to the light detected by the pixel 1320 via the on-chip color filter 13211IR. An endoscope apparatus 2) is configured.

Further, according to the embodiment, the on-chip color filter 13211IR to the on-chip color filter 13210B are configured by a fluorescence observation apparatus (the fluorescence observation endoscope apparatus 1 or the fluorescence observation endoscope apparatus 2) formed of an organic material. Is done.

In addition, according to the embodiment, the on-chip color filter 13210R is configured with a fluorescence observation apparatus (fluorescence observation endoscope apparatus 1 or fluorescence observation endoscope apparatus 2) formed of an organic material.

In addition, according to the embodiment, a fluorescence observation endoscope apparatus (fluorescence observation endoscopy) in which a fluorescent substance composed of an indocyanine green derivative-labeled antibody (ICG) is administered to a living body (inspection subject) and diagnosed by an endoscope. A first wavelength band (for example, a wavelength band of 700 nm to 800 nm) that emits fluorescence by irradiating a fluorescent substance (ICG) administered to a subject to be examined. ) Excitation light, visible light in a second wavelength band (for example, a wavelength band of 500 nm to 600 nm) different from the first wavelength band, and third light different from the first wavelength band and the second wavelength band. A light source device (light source device 20) capable of simultaneously irradiating visible light in a wavelength band (for example, a wavelength band of 400 nm to 500 nm), and a first filter (Onch) that selectively transmits fluorescence. Color filter 13211IR), a second filter that selectively transmits visible light in the second wavelength band (on-chip color filter 13210G), and a third filter that selectively transmits visible light in the third wavelength band. (On-chip color filter 13210B), each of the pixels 1320 detects light corresponding to an imaging device (image sensor 132 or image sensor 133) in which a plurality of light receiving elements (pixels 1320) provided with any one of them are arranged. And an image processing device (image processing unit 33) that performs image processing based on the pixel signal.

In addition, according to the embodiment, the image sensor 132 or the image sensor 133 is inserted into the body of the subject to be examined in the fluorescence observation endoscope device 1 or the fluorescence observation endoscope device 2 (the tip of the insertion portion 11). The fluorescence observation endoscope apparatus 1 or the fluorescence observation endoscope apparatus 2 arranged in the imaging unit 13) provided in the unit is configured.

According to the embodiment, the fluorescence includes a wavelength of near infrared light (for example, 830 nm), the visible light in the second wavelength band includes a wavelength of green light (for example, 540 nm), and the third The visible light in the wavelength band includes the fluorescence observation endoscope apparatus 1 or the fluorescence observation endoscope apparatus 2 including the wavelength of blue light (for example, 450 nm).

In addition, according to the embodiment, the image sensor 132 or the image sensor 133 has a fourth wavelength band (for example, 600 nm) different from the wavelength band through which each of the on-chip color filter 13210B and the on-chip color filter 13210B selectively transmits. A plurality of pixels 1320 provided with a fourth filter (on-chip color filter 13210R) that selectively transmits visible light in a wavelength band (˜700 nm wavelength band) is further arranged, and the light source device 20 is connected to the first wavelength band from the first wavelength band. 3 in response to light detected by the pixel 1320 from the on-chip color filter 13211IR through the on-chip color filter 13210B by the image processing apparatus (white balance unit 331 included in the image processing unit 33). Image processing based on the respective pixel signals. The fluorescence photographing mode for performing (white balance processing) and the light source device 20 emit light of the second wavelength band to the fourth wavelength band, and the image processing device (white balance unit 331 included in the image processing unit 33). However, mode switching for switching between a normal shooting mode in which image processing (white balance processing) is performed based on each pixel signal corresponding to light detected by the pixel 1320 from the on-chip color filter 13210G via the on-chip color filter 13210R. The fluorescence observation endoscope apparatus 1 or the fluorescence observation endoscope apparatus 2 further includes a device (imaging mode changeover switch 14).

According to the embodiment, the fluorescence includes a wavelength of near infrared light (for example, 830 nm), the visible light in the second wavelength band includes a wavelength of green light (for example, 540 nm), and the third The visible light in the wavelength band includes a wavelength of blue light (for example, 450 nm), and the visible light in the fourth wavelength band includes a wavelength of red light (for example, 630 nm). 1 or the fluorescence observation endoscope apparatus 2 is configured.

In addition, according to the embodiment, the image processing apparatus (the demosaicing unit 330 included in the image processing unit 33) is a pixel corresponding to the light detected by the pixel 1320 from the on-chip color filter 13210G via the on-chip color filter 13210R. The fluorescence observation endoscope apparatus 1 or the fluorescence observation endoscope apparatus that performs offset processing on each of the signals based on each of the pixel signals corresponding to the light detected by the pixel 1320 via the on-chip color filter 13211IR. 2 is configured.

As described above, according to each embodiment of the present invention, in the solid-state imaging device, pixels that expose fluorescence emitted by a derivative-labeled antibody (fluorescent agent: fluorescent substance) such as ICG are arranged. . Thereby, in each embodiment of this invention, the pixel which exposes fluorescence can detect the fluorescence which the fluorescent substance excited and fluorescent-emitted. In each embodiment of the present invention, a dielectric multilayer filter having a reduced thickness (less than 1 μm in the embodiment) is formed in the solid-state imaging device at the position of the pixel that exposes the fluorescence arranged in the solid-state imaging device. To do. In each embodiment of the present invention, when observing the object under normal conditions (in the normal imaging mode in the embodiment), visible light in the blue (B) wavelength band and green (G) The light emission intensity of visible light in the wavelength band and visible light in the red (R) wavelength band have the same value, and white light that does not include light in the wavelength band of excitation light is irradiated as illumination light onto the object to be inspected. On the other hand, in each embodiment of the present invention, when observing an object to be inspected based on the fluorescence emitted by the fluorescent material and excited (in the embodiment, the fluorescence imaging mode), the wavelength band of blue (B) is used. The emission intensity of visible light and visible light in the green (G) wavelength band is weak at the same value, the emission intensity of near-infrared light in the excitation light wavelength band is high, and visible light in the red (R) wavelength band The object to be inspected is irradiated with illumination light that does not contain. Thereby, in each embodiment of the present invention, it is possible to perform observation of the inspection object in a normal state and observation of the inspection object by the fluorescence emitted from the fluorescent material and emitting fluorescence.

In each embodiment of the present invention, when observing an object to be inspected based on fluorescence emitted from a fluorescent material and excited, the weak fluorescence detected by the pixel that exposes the fluorescence is red (R). White balance processing is performed assuming that visible light in the wavelength band is detected. Thereby, in each embodiment of this invention, the location which the fluorescent substance emitted fluorescence can be emphasized to the reddish color in the image which observed the to-be-inspected object, and can be shown to a tester. Thus, the tester can easily confirm the location where the fluorescent material emits fluorescence.

As described above, in each embodiment of the present invention, as in the prior art, attenuation with respect to visible light is prevented so that visible light is not detected when fluorescence emitted from a fluorescent material is detected and fluorescence is detected. Even without providing an optical filter with a very high rate (light-shielding rate), it is possible to detect and present with high accuracy the fluorescence emitted by the fluorescent material and excited. More specifically, in the conventional technology, it is necessary to provide an optical filter having a very high attenuation rate (light shielding rate) that attenuates incident visible light to, for example, 0.3%, on a pixel that exposes fluorescence. However, in each embodiment of the present invention, it is possible to provide fluorescence with high accuracy only by providing an optical filter that does not have a high attenuation rate (light shielding rate) that attenuates incident visible light to 5% or 1%, for example. Can be detected. That is, in each embodiment of the present invention, even when a certain amount of visible light is allowed to be incident on the pixels that are exposed to fluorescence arranged in the solid-state imaging device, the fluorescence emitted by the fluorescent material is detected with high accuracy. can do. For this reason, in each embodiment of this invention, the thickness of an optical filter can be made thin and a solid-state imaging device can be reduced in size. Thereby, in each embodiment of this invention, the imaging part comprised including a solid-state imaging device in a fluorescence observation apparatus can be reduced in size. Accordingly, in each embodiment of the present invention, when the fluorescence observation apparatus is configured as a fluorescence observation endoscope apparatus, the solid-state imaging device is positioned at the distal end portion of the insertion portion that is inserted into the body of the subject to be examined. It can be easily arranged in the imaging unit.

Thereby, in the fluorescence observation endoscope apparatus, a function of bending the distal end portion of the insertion portion can be realized. That is, when the solid-state imaging device cannot be arranged in the imaging unit located at the distal end portion of the insertion unit, for example, a position where the solid-state imaging device is arranged by a relay lens that transmits an image by light in a relay form ( For example, since the image of the object to be inspected must be optically guided to the operation unit 12 or the like, it is difficult to freely curve the relay lens, and the function of bending the distal end of the insertion unit is installed. Can not do it. On the other hand, in each embodiment of the present invention, the solid-state imaging device can be disposed in the imaging unit located at the distal end portion of the insertion unit, and signals exchanged between the solid-state imaging device and other components ( For example, in the embodiment, since an imaging signal or a control signal (drive signal) can be transmitted as an electrical signal, the location of the electrical signal line can be freely bent, and the distal end of the insertion portion can be A function of bending can be mounted.

In addition, in each embodiment of the present invention, since the thickness of the optical filter formed in the solid-state imaging device can be reduced, it is possible to suppress deterioration in sensitivity to light and oblique incidence characteristics in each pixel arranged in the solid-state imaging device. be able to. Thereby, in each embodiment of the present invention, the object to be observed can be observed with higher accuracy. For example, when the fluorescence observation apparatus is configured as a fluorescence observation endoscope apparatus, the object to be inspected can be observed with higher accuracy.

In each embodiment of the present invention, an imaging signal output from the solid-state imaging device (in the embodiment, the image sensor 132 or the image sensor 133) provided in the imaging unit that constitutes the fluorescence observation apparatus of the present invention A case where each pixel is a pixel signal corresponding to a signal charge obtained by photoelectric conversion, that is, RAW data is shown. However, the format of the imaging signal output from the solid-state imaging device is not limited to the RAW data shown in each embodiment of the present invention. For example, in order to reduce the data amount of the imaging signal transmitted by the image sensor 132 through the imaging signal line 61, YC processing is performed on each pixel signal of the R pixel, the G pixel, and the B pixel, for example, a YCbCr signal , A so-called luminance color difference signal such as a YUV signal may be converted and output. In this case, the image sensor 132 converts the digital pixel signal output from the analog / digital conversion unit 1323 into a digital luminance / chrominance signal, and then captures the serial luminance / chrominance signal parallel / serial converted by the serializer 1324. A configuration in which the signal is output to the outside can be considered. In this case, in the external processing unit 30, after the deserializer 32 restores the serial luminance color difference signal to the original parallel luminance color difference signal, the demosaicing unit 330 in the image processing unit 33 performs the three-plate processing. A configuration is conceivable in which image data consisting only of pixel signals of any one of pixels, G pixels, or B pixels is generated.

In each embodiment of the present invention, the configuration in which the light source device constituting the fluorescence observation apparatus of the present invention emits light in a predetermined wavelength band by a set of a white light source and a dichroic mirror is shown. However, the configuration of the light source device is not limited to the configuration shown in each embodiment of the present invention. For example, instead of a pair of a white light source and a dichroic mirror, a light source that emits light in a predetermined wavelength band is provided, and a light emitting lens collects and emits the light emitted from each light source. May be. In this case, the light source device includes, for example, a blue light source that emits light in a blue (B) wavelength band (for example, visible light in a wavelength band of 400 nm to 500 nm) and light in a green (G) wavelength band (for example, A green light source that emits visible light in the wavelength band of 500 nm to 600 nm, a red light source that emits light in the red (R) wavelength band (eg, visible light in the wavelength band of 600 nm to 700 nm), and the wavelength of the excitation light An infrared light source that emits light in a band (for example, near infrared light in a wavelength band of 700 nm to 800 nm) is conceivable.

In each embodiment of the present invention, the case where the fluorescence observation apparatus of the present invention is configured as a fluorescence observation endoscope apparatus has been described. However, the fluorescence observation apparatus of the present invention is not limited to the configuration as the fluorescence observation endoscope apparatus shown in each embodiment. For example, the fluorescence observation apparatus of the present invention can be configured as a microscope apparatus. In this case, each component of the fluorescence observation apparatus of the present invention is arranged at an appropriate position in the microscope apparatus.

As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment and its modification. Additions, omissions, substitutions, and other modifications can be made without departing from the spirit of the present invention.
Further, the present invention is not limited by the above description, and is limited only by the scope of the appended claims.

According to each of the embodiments described above, it is possible to provide a fluorescence observation apparatus and a fluorescence observation endoscope apparatus that can realize downsizing of an imaging system and detection of fluorescence excited by a fluorescent substance with high accuracy. .

1,2 Fluorescence observation endoscope device (Fluorescence observation device, Fluorescence observation endoscope device)
10 Endoscopic scope part 11 Insertion part (insertion part)
12 Operation unit 13 Imaging unit (fluorescence observation device, fluorescence observation endoscope device)
130 Imaging Lens 131 Excitation Light Cut Filter (Fluorescence Observation Device, Fluorescence Observation Endoscope Device, Excitation Light Cut Filter)
132, 133 Image sensor (fluorescence observation device, fluorescence observation endoscope device, imaging device)
1320, 1320a, 1320b, 1330, 1330a, 1330b Pixel (light receiving element, imaging device)
13200 Support substrate 13201 Wiring layer 13202 Photoelectric conversion layer (imaging device)
13203 Dielectric multilayer filter layer (imaging device)
13204 On-chip color filter layer (imaging device)
13205 Microlens 13206 Wiring 13207 Photoelectric conversion element (light receiving element, imaging device)
13208 Transparent material 13209 Dielectric multilayer filter (imaging device, dielectric multilayer filter)
13210 On-chip color filter (imaging device, second filter, third filter, fourth filter)
13210R On-chip color filter (imaging device, fourth filter)
13210G On-chip color filter (imaging device, second filter)
13210B On-chip color filter (imaging device, third filter)
13211, 13211IR On-chip color filter (imaging device, first filter)
1321 Pixel unit (light receiving element, imaging device)
1322 Reading unit 13220 Control circuit 13221 Vertical scanning circuit 13222 Horizontal scanning circuit 1323 Analog / digital conversion unit 13230 Analog / digital conversion circuit 1324 Serializer 1325 Vertical signal line 1326 Horizontal signal line 14 Shooting mode changeover switch (fluorescence observation device, fluorescence observation endoscopy) Mirror device, mode switching device)
20 Light source device (fluorescence observation device, fluorescence observation endoscope device, light source device)
21 Light emission control unit (fluorescence observation device, fluorescence observation endoscope device, light source device)
220, 221, 222, 223, 224 White light source (fluorescence observation apparatus, fluorescence observation endoscope apparatus, light source apparatus)
231, 232, 233, 234 Dichroic mirror (fluorescence observation device, fluorescence observation endoscope device, light source device)
24 Light irradiation lens (fluorescence observation device, fluorescence observation endoscope device, light source device)
30 External processing unit (fluorescence observation device, fluorescence observation endoscope device, image processing device)
31 Mode determination unit (fluorescence observation device, fluorescence observation endoscope device, mode switching device)
32 Deserializer 33 Image processing unit (fluorescence observation device, fluorescence observation endoscope device, image processing device)
330 Demosaicing unit (fluorescence observation device, fluorescence observation endoscope device, image processing device)
331 White balance unit (fluorescence observation device, fluorescence observation endoscope device, image processing device)
332 Gamma correction unit 34 Digital / analog conversion unit 40 Color monitor 50 Optical signal cable 51 Light guide 60 Electrical signal cable 61 Imaging signal line 62 Mode signal line 900 Abdomen 901 Inspected object B1 Spectral characteristic G1 Spectral characteristic R1 Spectral characteristic IR1 Spectral characteristic DM1 spectral characteristic IR1 + DM1 spectral characteristic EL1 spectral characteristic B2 spectral characteristic G2 spectral characteristic R2 spectral characteristic B3 spectral characteristic G3 spectral characteristic IR3 spectral characteristic

Claims (17)

Excitation light in a first wavelength band that emits fluorescence by irradiating the phosphor, visible light in a second wavelength band different from the first wavelength band, the first wavelength band, and the second wavelength band A light source device capable of simultaneously irradiating visible light of a third wavelength band different from the wavelength band of
A first filter that selectively transmits the fluorescence; a second filter that selectively transmits visible light in the second wavelength band; and a third filter that selectively transmits visible light in the third wavelength band. An imaging device in which a plurality of light receiving elements provided with any of the filters are arranged,
An image processing device that performs image processing based on a pixel signal in which each of the light receiving elements detects corresponding light;
A fluorescence observation apparatus. The imaging device
A dielectric multilayer filter that reflects light in the third wavelength band from the second wavelength band and transmits light in the fluorescence wavelength band;
Further comprising
Disposing the dielectric multilayer filter on an optical path of the first filter;
The fluorescence observation apparatus according to claim 1. The intensity of the excitation light irradiated by the light source device is
Stronger than the intensity of each visible light irradiated by the light source device,
The fluorescence observation apparatus according to claim 1. An excitation light cut filter that is disposed on an optical path from the subject that irradiates the excitation light to the imaging device and prevents the excitation light from entering the light receiving element;
Further having
The fluorescence observation apparatus according to claim 1. The fluorescence is
Including the wavelength of near infrared light,
The visible light in the second wavelength band is
Including the wavelength of green light,
The visible light in the third wavelength band is
Including the wavelength of blue light,
The fluorescence observation apparatus according to claim 1. The image processing apparatus includes:
Image processing is performed based on a signal level obtained by multiplying each of the pixel signals corresponding to the light detected by the light receiving element from the first filter through the third filter by a predetermined value. ,
The fluorescence observation apparatus according to claim 1. The imaging device
A light receiving element provided with a fourth filter that selectively transmits visible light having a fourth wavelength band different from a wavelength band selectively transmitted from each of the first filter to the third filter. Multiple placement,
The light source device irradiates light of the third wavelength band from the first wavelength band, and the image processing device detects the light receiving element from the first filter via the third filter. A fluorescence imaging mode for performing the image processing based on each of the pixel signals corresponding to light; and the light source device irradiates light in the fourth wavelength band from the second wavelength band, and the image processing apparatus. A mode switching device that switches between the second imaging mode and the normal imaging mode in which the image processing is performed based on each of the pixel signals corresponding to the light detected by the light receiving element via the fourth filter,
Further having
The fluorescence observation apparatus according to claim 1. The fluorescence is
Including the wavelength of near infrared light,
The visible light in the second wavelength band is
Including the wavelength of green light,
The visible light in the third wavelength band is
Including the wavelength of blue light,
The visible light in the fourth wavelength band is
Including the wavelength of red light,
The fluorescence observation apparatus according to claim 7. The image processing apparatus includes:
For each of the pixel signals corresponding to light detected by the light receiving element from the second filter via the fourth filter, depending on light detected by the light receiving element via the first filter Further, an offset process is performed based on each of the pixel signals.
The fluorescence observation apparatus according to claim 7. From the first filter to the third filter,
Formed of organic materials,
The fluorescence observation apparatus according to claim 1. The fourth filter is:
Formed of organic materials,
The fluorescence observation apparatus according to claim 7. A fluorescence observation endoscopic device for performing an endoscopic diagnosis by administering a fluorescent substance comprising an indocyanine green derivative-labeled antibody to a living body,
Excitation light of a first wavelength band that emits fluorescence by irradiating the fluorescent material administered to the living body, visible light of a second wavelength band different from the first wavelength band, and the first A light source device capable of simultaneously irradiating a visible light having a wavelength band and a third wavelength band different from the second wavelength band;
A first filter that selectively transmits the fluorescence; a second filter that selectively transmits visible light in the second wavelength band; and a third filter that selectively transmits visible light in the third wavelength band. An imaging device in which a plurality of light receiving elements provided with any of the filters are arranged,
An image processing device that performs image processing based on a pixel signal in which each of the light receiving elements detects corresponding light;
A fluorescence observation endoscope apparatus. The imaging device
In the fluorescence observation endoscope device, disposed in the insertion portion to be inserted into the living body,
The fluorescence observation endoscope apparatus according to claim 12. The fluorescence is
Including the wavelength of near infrared light,
The visible light in the second wavelength band is
Including the wavelength of green light,
The visible light in the third wavelength band is
Including the wavelength of blue light,
The fluorescence observation endoscope apparatus according to claim 12. The imaging device
A light receiving element provided with a fourth filter that selectively transmits visible light having a fourth wavelength band different from a wavelength band selectively transmitted from each of the first filter to the third filter. Multiple placement,
The light source device irradiates light of the third wavelength band from the first wavelength band, and the image processing device detects the light receiving element from the first filter via the third filter. A fluorescence imaging mode for performing the image processing based on each of the pixel signals corresponding to light; and the light source device irradiates light in the fourth wavelength band from the second wavelength band, and the image processing apparatus. A mode switching device that switches between the second imaging mode and the normal imaging mode in which the image processing is performed based on each of the pixel signals corresponding to the light detected by the light receiving element via the fourth filter,
Further having
The fluorescence observation endoscope apparatus according to any one of claims 12 to 14. The fluorescence is
Including the wavelength of near infrared light,
The visible light in the second wavelength band is
Including the wavelength of green light,
The visible light in the third wavelength band is
Including the wavelength of blue light,
The visible light in the fourth wavelength band is
Including the wavelength of red light,
The fluorescence observation endoscope apparatus according to claim 15. The image processing apparatus includes:
For each of the pixel signals corresponding to light detected by the light receiving element from the second filter via the fourth filter, depending on light detected by the light receiving element via the first filter Further, an offset process is performed based on each of the pixel signals.
The fluorescence observation endoscope apparatus according to claim 15.

Images