WO2021152704A1 - Endoscope device, operating method for endoscope device, and image processing program - Google Patents

Abstract

This endoscope device includes: a light source device (100) that emits, at different timings, a first illuminating light group (GR1) that includes green light having a green wavelength band, and a second illuminating light group (GR2) that includes blue narrow-band light belonging to a blue wavelength band and that does not include green light; an imaging element (240) that has a color filter array (241) including color filters of a plurality of colors; and a processing circuit (310) that generates a display image on the basis of an image captured by the imaging element (240) when the first illuminating light group (GR1) was emitted and an image captured by the imaging element (240) when the second illuminating light group (GR2) was emitted. The color filter array (241) includes cyan and blue color filters. The processing circuit (310) generates the display image on the basis of a blue narrow-band image obtained from blue narrow-band light transmitted through the cyan and blue color filters.

Description

Endoscope device, operation method of endoscope device and image processing program

The present invention relates to an endoscope device, an operation method of the endoscope device, an image processing program, and the like.

In an endoscope device, a method of capturing an image using an image sensor having a complementary color filter is known. The complementary color filter is a filter that is sensitive to a plurality of wavelength bands of blue, green, and red. For example, a cyan color filter has a property of transmitting blue light and green light. Patent Document 1 discloses a method of generating a narrow band optical image or a white optical image by using an image sensor having a complementary color filter.

JP-A-2018-38637

The information obtained from light of a certain wavelength depends on the spectral sensitivity of the color filter of the image sensor. When light of multiple wavelengths having sensitivity to the same color filter is emitted at the same time, color mixing occurs. For example, in the method of Patent Document 1, green light and blue narrow band light are mixed in a cyan pixel corresponding to a cyan color filter. In Patent Document 1, a green image or a blue narrow band image is generated by performing subtraction by image processing, but it is difficult to completely correct the color mixing by image processing.

One aspect of the present disclosure comprises a first illumination light group including green light having a green wavelength band and a second illumination light group including blue narrow band light belonging to the blue wavelength band and not including the green light. An imaging element that has a light source device that emits light at different timings, a color filter array that includes color filters of a plurality of colors arranged in two dimensions, and an imaging element that images a subject, and the imaging when the first illumination light group is irradiated. The color filter array includes a processing circuit that generates a display image based on an image captured by the element and an image captured by the image pickup element when the second illumination light group is irradiated. , The cyan color filter and the blue color filter, the cyan color filter has an optical property of transmitting the green and blue light, and the processing circuit is the cyan color filter. , And an endoscopy device that produces the display image based on the blue narrow band image obtained by the blue narrow band light that has passed through the blue color filter.

Another aspect of the present disclosure is a method of operating an endoscope device including a color filter array including color filters of a plurality of colors arranged in two dimensions and an image pickup element for photographing a subject, wherein the color filter array is provided. Includes a cyan color filter and a blue color filter, and includes green light having the green wavelength band when the cyan color filter has optical characteristics to transmit green and blue light. The first illumination light group and the second illumination light group containing the blue narrow band light belonging to the blue wavelength band and not containing the green light were emitted at different timings, and the first illumination light group was irradiated. When a display image is generated based on an image captured by the image pickup element and an image captured by the image pickup element when the second illumination light group is irradiated, and the display image is generated. In addition, it relates to a method of operating an endoscope device that generates the display image based on the blue narrow band image obtained by the blue narrow band light transmitted through the cyan color filter and the blue color filter. ..

Still another aspect of the present disclosure is an image processing program for processing an image captured by an imaging element that images a subject, having a color filter array including color filters of a plurality of colors arranged in two dimensions. The color filter array includes a cyan color filter and a blue color filter, the cyan color filter has optical characteristics for transmitting green and blue light, and the first illumination light group has a first illumination light group. When the second illumination light group includes the blue narrow band light belonging to the blue wavelength band and does not include the green light, the first illumination light group includes the green light having the green wavelength band. A display image is generated based on the image captured by the image pickup element when irradiated and the image captured by the image pickup element when the second illumination light group is irradiated, and the display image is displayed. At the time of generation, a computer is made to perform a step of generating the display image based on the blue narrow band image obtained by the blue narrow band light transmitted through the cyan color filter and the blue color filter. Related to image processing programs.

Configuration example of the endoscope device. Configuration example of the color filter of the image sensor. An example of the spectral characteristics of a color filter. Detailed configuration example of the processing circuit. The figure explaining the surface sequential imaging performed by the endoscope apparatus of this embodiment. The spectrum of the first illumination light group. The spectrum of the second illumination light group. A flowchart illustrating processing in a processing circuit. The figure explaining the 1st interpolation processing. The figure explaining the 2nd interpolation processing. The explanatory view of the process which generates the white light image which performed the enhancement process as a display image. A flowchart illustrating an enhancement process for a white light image. The explanatory view of the process which generates the white light image and the narrow band light image as a display image.

Hereinafter, this embodiment will be described. The present embodiment described below does not unreasonably limit the content of the present invention described in the claims. Moreover, not all of the configurations described in the present embodiment are essential constituent requirements of the present invention.

1. 1. Endoscope device FIG. 1 is a configuration example of an endoscope device. The endoscope device includes an insertion unit 200, a control device 300, a display unit 400, an external I / F unit 500, and a light source device 100. As the endoscope device, for example, a flexible mirror used for a gastrointestinal tract or the like and a rigid mirror used for a laparoscope or the like can be assumed, but the endoscope device is not limited thereto. The insertion portion 200 is also referred to as a scope. The control device 300 is also referred to as a main body or a processing device. The display unit 400 is also called a display device. The external I / F unit 500 is also referred to as an operation unit or an operation device. The light source device 100 is also called a lighting unit or a lighting device.

The light source device 100 is a device that generates illumination light. The light source device 100 includes light sources LDV, LDB, LDG, LDR, and a combiner 120. Each of the light sources LDV, LDB, LDG, and LDR is an LED (Light Emitting Diode) or a laser light source. Hereinafter, the light generated by the light sources LDV, LDB, LDG, and LDR will be referred to as V light, B light, G light, and R light, respectively.

V light is narrow band light having a peak wavelength of 410 nm. The half width of V light is several nm to several tens of nm. The band of V light belongs to the blue wavelength band of white light and is narrower than the blue wavelength band. B light is light having a blue wavelength band in white light. G light is light having a green wavelength band in white light. R light is light having a red wavelength band in white light. For example, the wavelength band of B light is 430 to 500 nm, the wavelength band of G light is 500 to 600 nm, and the wavelength band of R light is 600 to 700 nm. For example, the image sensor 240 includes a plurality of color filters as described later. V light and B light pass through a blue color filter, G light passes through a green color filter, and R light passes through a red color filter. Further, V light, B light and G light pass through a cyan color filter.

The above wavelength is an example. For example, the peak wavelength of each light and the upper and lower limits of the wavelength band may be deviated by about 10%. Further, the B light, the G light and the R light may be narrow band light having a half width of several nm to several tens of nm. Further, the light source device 100 may include a light source (not shown) that irradiates light in another wavelength band. For example, the light source device 100 may include a light source LDA that irradiates A light. The A light is a narrow band light having a peak wavelength of 600 nm, and its half width is several nm to several tens of nm. The band of A light belongs to the red wavelength band of white light and is narrower than the red wavelength band.

The combiner 120 combines the light emitted by the LDV, LDB, LDG, and LDR, and causes the combined light to enter the light guide 210. The combiner 120 is composed of, for example, a dichroic mirror and a lens.

The light source device 100 emits one wavelength or a plurality of wavelengths of V light, B light, G light, and R light at one light emission timing. Hereinafter, the light having one wavelength or a plurality of wavelengths emitted at this one light emission timing is referred to as an illumination light group. The details of lighting will be described later.

The insertion portion 200 is a portion to be inserted into the body. The insertion unit 200 includes a light guide 210, an illumination lens 220, an objective lens 230, an image sensor 240, and an A / D conversion circuit 250. Further, the insertion unit 200 can include a memory 260. The image sensor 240 is also called an image sensor. The insertion portion 200 has a connector (not shown), and the insertion portion 200 is attached to and detached from the control device 300 by the connector.

The light guide 210 guides the illumination light from the light source device 100 to the tip of the insertion portion 200. The illumination lens 220 irradiates the subject with the illumination light guided by the light guide 210. In this embodiment, the subject is a living body. The reflected light from the subject is incident on the objective lens 230. The subject image is imaged by the objective lens 230, and the image sensor 240 images the subject image.

The image sensor 240 includes a plurality of pixels for photoelectric conversion of a subject image, and acquires pixel signals from the plurality of pixels. The image pickup device 240 is a color image sensor that can obtain pixel signals of a plurality of colors by one imaging.

FIG. 2 is a diagram illustrating a color filter array 241 of the image sensor 240. The color filter array 241 is a 4 × 4 filter unit 242 composed of 16 filters arranged in a two-dimensional grid pattern arranged side by side. The filter unit 242 is composed of four green color filters, two blue color filters, two red color filters, and eight cyan color filters. The green color filter corresponds to G in FIG. Hereinafter, the green color filter is also referred to as a G filter. Similarly, the blue color filter, the red color filter, and the cyan color filter correspond to B, R, and Cy in FIG. 2, respectively, and are hereinafter referred to as B filter, R filter, and Cy filter. Further, the pixel in which the G filter is arranged is referred to as a G pixel. Similarly, the pixels in which the B filter, the R filter, and the Cy filter are arranged are referred to as B pixel, R pixel, and Cy pixel, respectively.

FIG. 3 is a diagram showing the spectral characteristics of each color filter included in the color filter array 241. In FIG. 3, the transmittance curves are simulated and standardized so that the maximum values of the transmittances of each filter are equal. 3, the curve L _B represents the transmittance curve of the B filter, curve L _G represents a transmittance curve of G filters, the curve L _R represents the transmittance curve of the R filter, a curve L _Cy is Cy filter The transmittance curve is shown. Further, in FIG. 3, the horizontal axis represents the wavelength and the vertical axis represents the transmittance.

As shown in FIG. 3, B filter transmits light in the wavelength band H _B. G filter transmits light in the wavelength band H _G. Cy filter transmits the wavelength band H _B and the wavelength band H _G respective light absorbs light of a wavelength band H _R. That is, the Cy filter transmits light in the cyan wavelength band, which is a complementary color. R filter transmits light in the wavelength band H _R. In this specification, complementary colors refers to the color composed by light including at least two wavelength bands of the wavelength band _{H B, H G, H R} .

The A / D conversion circuit 250 A / D-converts an analog pixel signal from the image sensor 240 into a digital pixel signal. The A / D conversion circuit 250 may be built in the image sensor 240.

The control device 300 performs signal processing including image processing. Further, the control device 300 controls each part of the endoscope device. The control device 300 includes a processing circuit 310 and a control circuit 320.

The control circuit 320 controls each part of the endoscope device. For example, the user operates the external I / F unit 500 to set the presence / absence of emphasis processing. For example, when an instruction to perform emphasis processing is input, the control circuit 320 outputs an instruction to perform emphasis processing to the processing circuit 310. The processing circuit 310 emphasizes the blood vessel structure in the white light image based on, for example, an image obtained by V light. When the instruction not to perform the emphasis processing is input, the control circuit 320 outputs the instruction not to perform the emphasis processing to the processing circuit 310. The processing circuit 310 outputs the white light image to the display unit 400 without emphasizing it.

The memory 260 of the insertion unit 200 stores information about the insertion unit 200. The control circuit 320 controls each part of the endoscope device based on the information read from the memory 260. For example, the memory 260 stores information about the image sensor 240. The information about the image sensor 240 is, for example, information about the type of the image sensor 240 and the like. The control circuit 320 causes the processing circuit 310 to perform image processing corresponding to the information about the image sensor 240 read from the memory 260.

The processing circuit 310 generates a display image by performing image processing based on the pixel signal from the A / D conversion circuit 250, and outputs the display image to the display unit 400. The display unit 400 is, for example, a liquid crystal display device or the like, and displays a display image from the processing circuit 310.

FIG. 4 is a detailed configuration example of the processing circuit 310. The processing circuit 310 includes an interpolation processing unit 311 and a display image generation unit 313.

A pixel signal is input to the interpolation processing unit 311 from the A / D conversion circuit 250. The interpolation processing unit 311 acquires an image corresponding to each of V light, B light, G light, and R light based on the pixel signal. Hereinafter, the images corresponding to V light, B light, G light, and R light are referred to as V image, B image, G image, and R image, respectively. For example, the image sensor 240 includes the color filter shown in FIG. The V light is imaged by blue pixels and cyan pixels. The interpolation processing unit 311 generates a V image by performing interpolation processing based on the pixel signals of blue pixels and cyan pixels. Similarly, B light is imaged by blue pixels and cyan pixels, G light is imaged by green pixels and cyan pixels, and R light is imaged by red pixels. The interpolation processing unit 311 generates a B image, a G image, and an R image by performing interpolation processing based on the pixel signals corresponding to each light. The details of the interpolation process will be described later with reference to FIGS. 9 and 10.

The display image generation unit 313 generates a display image based on the B image, the G image, the R image, and the V image. The display image generation unit 313 generates a white light image using FIGS. 11 and 12, and generates a display image by performing enhancement processing based on the V image on the white light image. .. Alternatively, the display image generation unit 313 may generate a white light image and a narrow band light image as display images, as will be described later with reference to FIG. The display image generation unit 313 outputs the generated display image to the display unit 400.

The external I / F unit 500 is an interface for inputting from the user to the endoscope device. That is, it is an interface for operating the endoscope device, an interface for setting the operation of the endoscope device, and the like. For example, it includes buttons or dials, levers, etc. for operating the endoscope device.

FIG. 5 is a diagram illustrating surface sequential imaging performed by the endoscope device of the present embodiment. As shown in FIG. 5, the light source device 100 emits the first illumination light group GR1 in the first frame F1, emits the second illumination light group GR2 in the second frame F2, and emits the first illumination in the third frame F3. The light group GR1 is emitted, and the second illumination light group GR2 is emitted in the fourth frame F4. The frame is a period during which the image pickup device 240 performs imaging, and corresponds to a frame in moving image shooting.

The image sensor 240 images the subject irradiated with the first illumination light group GR1 in the first frame F1 and the third frame F3, and is irradiated by the second illumination light group GR2 in the second frame F2 and the fourth frame F4. Image the subject.

The processing circuit 310 generates the first display image IM1 based on the images captured in the first frame F1 and the second frame F2. Further, the processing circuit 310 generates a second display image IM2 based on the images captured in the second frame F2 and the third frame F3, and is based on the images captured in the third frame F3 and the fourth frame F4. , Generates the third display image IM3. The display images IM1 to IM3 are frame images in the moving image. After that, by performing the same operation, a moving image is taken and the moving image is displayed on the display unit 400.

6 and 7 are spectra of illumination light. FIG. 6 shows the spectrum of the light included in the first illumination light group, and FIG. 7 shows the spectrum of the light included in the second illumination light group.

The light source device 100 emits B light, G light, and R light as the first illumination light group GR1 in FIG. 5, and emits V light as the second illumination light group GR2 in FIG.

In the frames F1 and F3 from which the first illumination light group GR1 is emitted, the interpolation processing unit 311 generates an RGB image by the interpolation processing, outputs the B channel of the RGB image as the B image, and sets the G channel of the RGB image to G. It is output as an image, and the R channel of the RGB image is output as an R image.

In the frames F2 and F4 from which the second illumination light group GR2 is emitted, the interpolation processing unit 311 generates a B channel of an RGB image by the interpolation processing, and outputs the B channel as a V image. The interpolation processing unit 311 does not generate the G channel and the R channel, or does not output even if it is generated.

FIG. 8 is a flowchart illustrating the processing performed in the processing circuit 310. First, in S101, the processing circuit 310 acquires image data based on the emission of the illumination light group by the light source device 100. Image data here is to correspond to the image F ₂₀ in the image F ₁₀ or 10 in FIG. 9 to be described later, R filters of the imaging device 240, G filter, B filter, the output from the corresponding pixel in each filter of the Cy filter It is a set of signals to be generated. In the example of FIG. 5, the processing circuit 310 acquires the image data based on the emission of the first illumination light group GR1 in the odd frame, and acquires the image data based on the emission of the second illumination light group GR2 in the even frame. ..

Next, in S102, the interpolation processing unit 311 performs interpolation processing on the image data. Specifically, when the first illumination light group GR1 is emitted, the first interpolation processing is performed, and when the second illumination light group GR2 is emitted, the second interpolation processing is performed. Each interpolation process will be described later with reference to FIGS. 9 and 10. By acquiring image data for two frames and performing interpolation processing based on each image data, an R image, a G image, a B image, and a V image are acquired. Next, in S103, the display image generation unit 313 generates a display image based on the R image, the G image, the B image, and the V image. The display image generation process will be described later with reference to FIGS. 11 to 13.

Hereinafter, the operations and processes performed by the endoscope device of the present embodiment will be described with reference to the embodiments described later as appropriate.

The endoscope device of this embodiment includes a light source device 100, an image pickup device 240, and a processing circuit 310. The light source device 100 causes the first illumination light group GR1 and the second illumination light group GR2 to emit light at different timings. The first illumination light group GR1 includes green light having a green wavelength band. The second illumination light group GR2 includes blue narrow band light belonging to the blue wavelength band and does not include green light. The image sensor 240 has a color filter array 241 including color filters of a plurality of colors arranged two-dimensionally, and images a subject. The color filter array 241 includes cyan and blue color filters, and the cyan color filter has an optical property of transmitting green and blue light as shown in FIG. As described with reference to FIG. 5, the processing circuit 310 includes an image captured by the image pickup device 240 when the first illumination light group GR1 is irradiated and an image sensor 240 when the second illumination light group GR2 is irradiated. A display image is generated based on the captured image. In this case, the processing circuit 310 generates a display image based on the blue narrow band image obtained by the cyan color filter and the blue narrow band light transmitted through the blue color filter.

Note that green light corresponds to G light, and blue narrow band light corresponds to V light. The blue narrow band image is a V image. The green image captured by the green light corresponds to the G image. Further, the process of generating a display image based on the V image may be a process of generating a B channel image of a white light image using the V image, or B of a narrow band light image using the V image. It may be a process of generating a channel image, or a process of emphasizing a B channel image of a white light image using a V image. The narrow band optical image may be rephrased as an NBI (Narrow Band Imaging) image.

According to the present embodiment, the color filter array 241 of the image sensor 240 includes a Cy filter that transmits G light and B light. Since the wavelength bands of G light and B light include the peak wavelength of the absorption characteristic of hemoglobin, light in the wavelength band belonging to G light and B light is of high importance in in vivo observation. By using the image sensor 240 including the Cy filter, the light belonging to the G light and the B light can be received by a large number of pixels, so that it is possible to generate an image with high resolution. However, since the Cy filter transmits both G light and B light, color mixing occurs when light belonging to the green wavelength band and light belonging to the blue wavelength band are simultaneously irradiated. V light, which is narrow-band light, can image a structure that is not clearly imaged by G light or broad B light, but the contrast of the structure is lowered by mixing with G light. In that respect, according to the present embodiment, the light source device 100 emits light from the first illumination light group GR1 including G light and the second illumination light group GR2 including V light and not including G light at different timings. Let me. Since the G light is not irradiated at the irradiation timing of the second illumination light group GR2, the pixel value of the Cy pixel can be treated as the B pixel, and a high-resolution V image can be acquired. This makes it possible to improve the visibility of a specific structure in the displayed image. The specific structure is specifically the capillaries on the surface of the mucosa.

Further, according to the present embodiment, the image sensor 240 has color filters of a plurality of colors, and the light source device 100 causes the first illumination light group GR1 and the second illumination light group GR2 to emit light at different timings. By using the image sensor 240 having a plurality of color filters, the number of surface sequences can be reduced, so that color shift can be reduced. In a narrow sense, the first illumination light group GR1 and the second illumination light group GR2 are alternately irradiated, and the number of surface sequences is two. However, it is not prevented that the number of surface sequences is 3 or more.

As shown in FIG. 6, the first illumination light group GR1 includes red light having a red wavelength band and blue light having a blue wavelength band. Red light corresponds to R light, and blue light corresponds to B light. The red image captured by the red light corresponds to the R image, and the blue image captured by the blue light corresponds to the B image.

According to this embodiment, when the first illumination light group GR1 is irradiated, the B image, the G image, and the R image can be taken. Therefore, it is possible to generate a white light image having a natural color. Further, when the second illumination light group GR2 is irradiated, a V image can be taken. As a result, a display image can be generated based on the V image, the B image, the G image, and the R image. At this time, for example, the B channel of the display image is generated based on the V image and the B image.

Further, in the present embodiment, the processing circuit 310 generates a green channel image in the display image based on the G image and the V image obtained by the G light. The green channel is one of a plurality of channels constituting the display image. Specifically, the plurality of channels are a red channel, a green channel, and a blue channel, and correspond to the R channel, the G channel, and the B channel in FIGS. 11 and 13, respectively.

The G channel image of the displayed image has a higher contribution to the luminance component than the R channel and B channel images, and greatly affects the visibility of the displayed image. By using the V image to generate the G channel image, it is possible to improve the visibility of the structure such as capillaries.

Further, in the present embodiment, the processing circuit 310 generates a white light image as a display image based on the R image obtained by the R light, the G image, and the B image obtained by the B light. The processing circuit 310 performs enhancement processing on the G channel image based on the V image.

According to this embodiment, the enhancement process using the V image is performed on the G channel image in the white light image. Therefore, it is possible to present an image with improved visibility of structures such as capillaries as a display image while using a white light image having a natural color as a base.

For example, the processing circuit 310 extracts a feature amount related to the structure of the subject based on the G image and the V image. The feature quantity is an quantity whose value increases at the position where the structure of interest exists. For example, the feature amount is an edge component or the like. Specifically, the processing circuit 310 extracts an edge component by applying a high-pass filter to the V image. Further, the processing circuit 310 extracts the structure that is not reflected in the G image but is reflected in the V image by obtaining the difference between the G image and the V image. That is, the processing circuit 310 extracts a structure that is not reflected in the G image but is reflected in the V image based on the correlation between the G image and the V image. The processing circuit 310 emphasizes the displayed image based on this correlation.

Further, in the present embodiment, the processing circuit 310 emphasizes the structure of the subject in the displayed image by synthesizing the above feature amount with the G image. As described above, the blood vessels on the surface layer of the mucosa are photographed in the V image. Therefore, the feature amount extracted from the V image is the feature amount related to the blood vessels on the surface layer of the mucosa. By synthesizing this feature amount with the G image, the blood vessel structure in the displayed image can be emphasized. As will be described later with reference to FIG. 11, the enhancement process using the V image may be performed on the B channel image in addition to the G channel image of the display image.

Further, in the present embodiment, the processing circuit 310 generates a white light image as a display image based on the R image, the G image, and the B image, and has a narrow band as the display image based on the G image and the V image. An optical image may be generated.

The color of the white light image is natural, and the burden on the user when observing is small. In the narrow band optical image, the structure of capillaries and the like is emphasized, so that lesions and the like can be easily found. According to this embodiment, it is possible to present two images having different characteristics as display images. For example, the display unit 400 displays a white light image and a narrow band light image side by side on one screen.

Further, in the present embodiment, when the first illumination light group GR1 is irradiated, the processing circuit 310 performs the first interpolation processing on the pixel signal from the image sensor 240, and the second illumination light group GR2 is irradiated. When this happens, a second interpolation process different from the first interpolation process is performed on the pixel signal from the image sensor 240. The interpolation process here is the above-mentioned given color with respect to the pixel signal generated by the image pickup device 240 based on the light received by the pixel in which the given color filter is arranged among the color filters of a plurality of colors. This is a process of interpolating the signals of pixels in which a color filter different from the filter is arranged.

According to this embodiment, it is possible to perform interpolation processing according to the emitted illumination light. Specifically, when irradiating the first illumination light group GR1 in which the pixel value of the Cy pixel includes both the G component and the B component, the image based on the Cy pixel is used as a guide image for joint bilateral interpolation processing and guided filter interpolation processing. Etc. are interpolated. Further, when the second illumination light group GR2 whose pixel value of the Cy pixel contains the V component and does not mix with the G component is irradiated, the Cy pixel is treated as the B pixel. Specifically, the second interpolation that interpolates the signals of pixels other than Cy and B, that is, R pixel and G pixel, based on the pixel signal generated by the image sensor 240 based on the light received by the Cy pixel and the B pixel. Processing is done. In the second interpolation processing, processing such as bilinear interpolation, cubic interpolation, and direction discrimination interpolation is performed.

Further, in the present embodiment, the peak wavelength of blue narrow band light, that is, V light is within the range of 415 ± 20 nm.

At 415 ± 20 nm, the absorption characteristics of hemoglobin are maximized. That is, V light having a peak wavelength in the range of 415 ± 20 nm is light that is easily absorbed by hemoglobin. By using this V-light, it is possible to photograph a region having a high hemoglobin concentration, such as a blood vessel in a mucous membrane. Since V light is scattered in a relatively shallow place in the mucosa, blood vessels on the surface layer of the mucosa can be photographed by V light.

Further, the light source device 100 may emit light in another wavelength band such as A light as described above. Since A light has a longer wavelength than V light, it reaches deeper in the mucous membrane than V light. Therefore, deep blood vessels can be photographed by using A light. For example, A light is included in the second illumination light group GR2. A V image and an A image are acquired by the emission of the second illumination light group GR2, and the blood vessels in the deep mucosa can be emphasized by enhancing the display image based on the A image.

Further, the method of this embodiment can be applied to the operating method of the endoscope device. The method of operating the endoscope device is a method of operating the endoscope device having a color filter array 241 including color filters of a plurality of colors arranged two-dimensionally and including an image pickup element 240 for photographing a subject. The method of operation is green light having a green wavelength band when the color filter array 241 includes cyan and blue color filters and the cyan color filters have optical properties that allow green and blue light to pass through. The first illumination light group GR1 including the A display image is generated based on the image captured by the image pickup element 240 when the light is generated and the image captured by the image pickup element 240 when the second illumination light group GR2 is irradiated. Further, when the display image is generated, the display image is generated based on the blue narrow band image obtained by the cyan color filter and the blue narrow band light transmitted through the blue color filter.

The control device 300 of the present embodiment may be configured as follows. That is, each of the processing circuit 310 and the control circuit 320 is composed of the following hardware. Further, the processing circuit 310 and the control circuit 320 may be integrally configured by the following hardware. The hardware can include at least one of a circuit that processes a digital signal and a circuit that processes an analog signal. For example, hardware can consist of one or more circuit devices mounted on a circuit board or one or more circuit elements. One or more circuit devices are, for example, ICs and the like. One or more circuit elements are, for example, resistors, capacitors, and the like.

Further, each of the processing circuit 310 and the control circuit 320 may be realized by the following processors. Further, the processing circuit 310 and the control circuit 320 may be realized by one processor. That is, the control device 300 of the present embodiment includes a memory for storing information and a processor that operates based on the information stored in the memory. The information is, for example, a program and various data. The processor includes hardware. By controlling the light source device 100, the processor causes the first illumination light group GR1 and the second illumination light group GR2 to emit light at different timings. The processor is based on an image captured by the image sensor 240 when the first illumination light group GR1 is irradiated and an image captured by the image sensor 240 when the second illumination light group GR2 is irradiated. Generate a display image. At this time, the processor generates a display image based on the V image captured by the V light transmitted through the Cy filter and the B filter of the image sensor 240.

The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as GPU (Graphics Processing Unit) or DSP (Digital Signal Processor) can be used. The memory may be a semiconductor memory such as SRAM or DRAM, a register, a magnetic storage device such as a hard disk device, or an optical storage device such as an optical disk device. You may. For example, the memory stores instructions that can be read by a computer, and when the instructions are executed by the processor, the functions of each part of the control device 300 are realized as processing. The instruction here may be an instruction of an instruction set constituting a program, or an instruction instructing an operation to a hardware circuit of a processor. For example, the processor realizes the function of the processing circuit 310 in FIG. Alternatively, the processor realizes the functions of the processing circuit 310 and the control circuit 320 in FIG.

Further, each part of the endoscope device of the present embodiment may be realized as a module of a program that operates on a processor. For example, the program includes a light source control module that causes the first illumination light group GR1 and the second illumination light group GR2 to emit light at different timings, and an image captured by the image sensor 240 when the first illumination light group GR1 is irradiated. And a processing module that generates a display image based on the image captured by the image pickup device 240 when the second illumination light group GR2 is irradiated. The processing module generates a display image based on the V image captured by the V light transmitted through the Cy filter and the B filter of the image sensor 240.

Further, the program that realizes the processing performed by each part of the control device 300 of the present embodiment can be stored in, for example, an information storage medium that is a medium that can be read by a computer. The information storage medium can be realized by, for example, an optical disk, a memory card, an HDD, a semiconductor memory, or the like. The semiconductor memory is, for example, a ROM. The processing circuit 310 and the control circuit 320 perform various processes of the present embodiment based on the programs and data stored in the information storage medium. That is, the information storage medium stores a program for operating the computer as each part of the endoscope device of the present embodiment. A computer is a device including an input device, a processing unit, a storage unit, and an output unit. The program is a program for causing a computer to execute the processing of each part. The program is recorded on an information storage medium. Here, as the information recording medium, various recording media that can be read by an optical detection system, such as an optical disk such as a DVD or a CD, a magneto-optical disk, a hard disk, and a memory such as a non-volatile memory or a RAM, can be assumed.

2. Interpolation processing The interpolation processing unit 311 of the present embodiment performs different interpolation processing depending on whether the first illumination light group GR1 is emitted and the second illumination light group GR2 is emitted. Hereinafter, each interpolation process will be described.

2.1 First Interpolation Process FIG. 9 is a schematic diagram illustrating an outline of the first interpolation process. First, the interpolation processing unit 311 acquires the _{image F 10 corresponding to the image data.} The first illumination light group GR1 is an illumination light group including G light, and specifically includes B light, G light, and R light. The color filter array of the image sensor 240 includes a B filter, a G filter, an R filter, and a Cy filter. Therefore, when the first illumination light group GR1 is emitted, the B pixel outputs a signal corresponding to the B light. The G pixel outputs a signal corresponding to the G light. The R pixel outputs a signal corresponding to the R light. The Cy pixel outputs signals corresponding to G light and B light. _{The image F 10} corresponding to the image data is an image in which each pixel has a pixel value corresponding to light according to the type of the color filter.

The interpolation processing unit 311 generates an interpolation image of Cy pixels arranged with the highest density in the image sensor 240 as a guide image. Specifically, as shown in FIG. 9, the interpolation processing unit 311 has B pixels and G pixels based on the pixel values of each Cy pixel in _{the separated image F Cy 11} in which the pixel values of the Cy pixels are separated from _{the image F 10.} And the pixel value of the Cy pixel at the pixel position where each of the R pixels is arranged is calculated by the interpolation process. As a result, the interpolation processing unit 311 generates _{an interpolation image F Cy12} having pixel values of Cy pixels at all pixel positions. The interpolated image F _Cy12 is referred to as a guide image. As shown in FIG. 9, B pixels in the separation image F _Cy11, pixel positions, each arranged G pixels and R pixels are adjacent four directions is surrounded by Cy pixel. The four directions here are up, down, left, and right directions. Therefore, the interpolation processing unit 311 can generate _{a highly accurate guide image FCy12} by using well-known bilinear interpolation, cubic interpolation, direction discrimination interpolation, and the like.

Subsequently, the interpolation processing unit 311 generates an interpolation image of pixels of other colors based on _{the generated guide image F Cy12. FB 11} is a separated image in which the pixel values of the B pixels are separated from the image F _10. Interpolation processing unit 311, based on the guide image F _Cy12, Cy pixel in the separation image F _B11, by calculating the pixel values of the B pixels in the pixel position R pixels and G pixels are arranged, B to all pixels position _{An interpolated image FB12} having pixel values of pixels is generated. The interpolated image _FB12 is a B image corresponding to B light.

Likewise, the interpolation processing unit 311, based on the guide image F _Cy12, Cy pixel in the separation image F _G11, by calculating the pixel values of the G pixels of the pixel position R and B pixels are arranged, all the pixels _{An interpolated image FG12} having a pixel value of G pixel at a position is generated. The FG ₁₁ is a separated image in which the pixel values of the G pixels are separated from the image F _10. The interpolated image F _G12 is a G image corresponding to G light.

Similarly interpolation processing unit 311, based on the guide image F _Cy12, Cy pixel in the separation image F _R11, by calculating the pixel values of the R pixels in the pixel position G and B pixels are arranged, all pixel positions Generates _{an interpolated image FR12} having a pixel value of R pixel. _{FR 11} is a separated image in which the pixel values of the R pixels are separated from the image F _10. The interpolated image _FR12 is an R image corresponding to R light.

The interpolation method based on the guide image F _Cy12 is, for example, a known joint bilateral interpolation process or a guided filter interpolation process. As a result, the interpolation processing unit 311 can generate _{high-precision interpolated images FB12} , _FG12, and _FR12 of the B pixels, G pixels, and R pixels arranged at low densities in the image sensor 240, respectively.

Since the guide image F _Cy12 is composed of Cy pixels when the first illumination light group GR1 is emitted, it contains a G component and a B component. Therefore, the correlation between the Cy pixel and the B pixel and the correlation between the Cy pixel and the G pixel are very high. Therefore, the interpolation processing unit 311 can accurately perform processing of generating an interpolated image _{F B12} and _{F G12} based on the guide image _{F Cy12.} Further, in general, in white light, the high frequency component has a high correlation with each of the R pixel, the G pixel, and the B pixel. Therefore, the interpolation processing unit 311 can perform the interpolation processing with high accuracy even when the R pixel is interpolated by using the Cy image as the guide image F _Cy12.

2.2 Second interpolation process FIG. 10 is a schematic diagram illustrating an outline of the second interpolation process. First, the interpolation processing unit 311 acquires the _{image F 20 corresponding to the image data.} The second illumination light group GR2 is an illumination light group containing V light and not containing G light. Specifically, the second illumination light group GR2 includes only V light. Therefore, when the second illumination light group GR2 is emitted, the B pixel outputs a signal corresponding to the V light. The G pixel and the R pixel do not output a signal due to the emission of the second illumination light group GR2. The Cy pixel outputs a signal corresponding to V light. That is, when the second illumination light group GR2 is emitted, the Cy pixel can be treated as the B pixel.

_{F Cy 21 in} FIG. 10 is a separated image in which the pixel values of the Cy pixels are separated from the image F _20. As described above, since the Cy pixel can be treated as a B pixel in the second interpolation processing, the separated image F _Cy21 becomes an image having a signal corresponding to the B channel, that is, a pixel value corresponding to V light at the position of the Cy pixel. The separation images F _B21 separating the pixel values of the B pixels from the image F ₂₀ will image having pixel values corresponding to V light to the position of the B pixel.

Interpolation processing unit 311, a separation image _{F Cy21,} by combining the separation images _{F B21,} to generate an intermediate image _{F B22.} Intermediate image _{F B22,} in Cy pixel has a pixel value of the separation images _{F Cy21,} an image having pixel values of the separation image _{F B21} in the B pixel. As a result, it is possible to generate _{an intermediate image FB22} having a signal corresponding to V light for 10 pixels out of 16 pixels of 4 × 4 corresponding to the filter unit 242.

The interpolation processing unit 311 calculates the pixel value of the B pixel at the pixel position where each of the G pixel and the R pixel is arranged by the interpolation processing based on the pixel values of the B pixel and the Cy pixel in _{the intermediate image FB22.} As shown in FIG. 10, in the intermediate image _FB22, the pixel positions where the R pixels are arranged are surrounded by Cy pixels in four adjacent directions. The four directions here are up, down, left, and right directions. Further, in the intermediate image _FB22, the pixel positions where the G pixels are arranged are surrounded by Cy pixels or B pixels in six adjacent directions. The six directions here are up, down, left, and right, and two diagonal directions. Therefore, the interpolation processing unit 311 can generate _{a highly accurate interpolation image FB23} by using well-known bilinear interpolation, cubic interpolation, direction discrimination interpolation, and the like. The interpolated image _FB23 is a V image corresponding to V light.

As described above, when the second illumination light group GR2 is emitted, color mixing of G light and V light does not occur in the Cy pixel. When generating a V image, it is not necessary to perform image processing for removing the G component from the pixel value of the Cy pixel, so that it is possible to generate a highly accurate V image. Since the V image is an image that reflects the structure of the capillaries and the like on the surface layer of the mucous membrane, it is possible to display the structure of the capillaries and the like with high contrast in the display image by using the high-precision V image. .. In other words, by using a high-precision V-image, it is possible to improve the visibility of a specific structure.

3. 3. Display image generation process Next, the display image generation process in the display image generation unit 313 will be described. In the method of the present embodiment, a specific subject, specifically, capillaries on the surface layer of the mucous membrane may be displayed in an easily visible manner based on the V image, and the specific mode of the displayed image may be various modifications. It is possible to carry out. Hereinafter, two specific examples will be described.

3.1 White light image with enhancement processing FIG. 11 is a schematic diagram illustrating an outline of the display image generation processing. The display image generation unit 313 generates an enhanced white light image as a display image. Specifically, the display image generation unit 313 generates a white light image based on the R image, the G image, the B image, and the V image.

Specifically, the display image generation unit 313 inputs the R image to the R channel of the color image, inputs the G image to the G channel of the color image, and inputs the B image and the V image to the B channel of the color image. As a result, a white light image is generated as a color image. The B channel of the color image is, for example, an image obtained by adding and averaging the pixel values of the B image and the V image. However, the display image generation unit 313 may input only the B image to the B channel of the color image.

Then, the display image generation unit 313 extracts the feature amount from the V image and enhances the white light image based on the feature amount. The display image generation unit 313 extracts a feature amount indicating the structure of the surface blood vessel by extracting a high frequency component from the V image. The display image generation unit 313 enhances the white light image by adding the extracted high frequency component to the G channel of the white light image. Further, the display image generation unit 313 may enhance the white light image by adding the extracted high frequency components to the G channel and the B channel of the white light image.

FIG. 12 is a flowchart illustrating a process of calculating a feature amount used in the emphasis process. In step S31, the display image generation unit 313 obtains the average intensity of the V image, and corrects the average intensity of the G image with reference to the average intensity of the V image. By performing this correction, the display image generation unit 313 adjusts the average brightness of the G image to the average brightness of the V image.

In step S32, the display image generation unit 313 applies a high-pass filter to the G image whose average intensity has been corrected. The display image generation unit 313 outputs the output of the high-pass filter as a high frequency component of the G image.

In step S33, the display image generation unit 313 applies a high-pass filter to the V image. The display image generation unit 313 outputs the output of the high-pass filter as a high frequency component of the V image.

In step S34, the display image generation unit 313 calculates the difference between the high frequency component of the V image calculated in step S33 and the high frequency component of the G image calculated in step S32.

In step S35, the display image generation unit 313 calculates the difference between the V image and the G image whose average intensity has been corrected in step S31. This difference is called the intensity difference. Intensity is the pixel value in each image. The intensity difference is used to enhance the white light image, but if the intensity difference is used as it is for the enhancement process, the vascular structure may be overemphasized. Therefore, the intensity difference is suppressed in steps S36 to S38.

In step S36, the display image generation unit 313 corrects the average intensity of the R image with reference to the average intensity of the V image. The display image generation unit 313 adjusts the average brightness of the R image to the average brightness of the V image by performing this correction.

In step S37, the display image generation unit 313 calculates the ratio between the V image and the R image whose average intensity has been corrected. The ratio is the ratio of intensity, for example, the ratio is calculated for each pixel.

In step S38, the display image generation unit 313 suppresses the intensity difference calculated in step S35 by using the ratio calculated in step S37. For example, in step S37, when the ratio of the intensity of the V image to the intensity of the R image is calculated, the display image generation unit 313 divides the intensity difference by the ratio. This calculation is performed on each pixel, for example.

In step S39, the display image generation unit 313 synthesizes the difference of the high frequency component calculated in step S34 and the intensity difference suppressed in step S38. Synthesis is, for example, addition. This calculation is performed on each pixel, for example.

The display image generation unit 313 adds the composite value obtained in step S39 to the G channel of the white light image. This calculation is performed on each pixel, for example. In this way, the blood vessel structure in the white light image is emphasized based on the V image. Further, the display image generation unit 313 may add the composite value obtained in step S39 to the B channel of the white light image.

The display image generation unit 313 outputs the white light image after the enhancement process to the display unit 400. The processing performed by the display image generation unit 313 is not limited to the enhancement processing, and various image processing such as gradation modulation processing, enlargement processing, and noise reduction processing can be added.

3.2 White light image and narrow band light image The display image generation unit 313 generates two images, a white light image and a narrow band light image, as display images. Specifically, as shown in FIG. 13, the display image generation unit 313 inputs the R image to the R channel of the color image, inputs the G image to the G channel of the color image, and inputs the B image and the V image to the color image. A white light image is generated as a color image by inputting to the B channel of. The B channel of the color image is, for example, an image obtained by adding and averaging the pixel values of the B image and the V image. The white light image here is an image that has not been subjected to structure enhancement processing based on the V image.

Further, the display image generation unit 313 generates a narrow band optical image as a color image by inputting the G image to the R channel of the color image and inputting the V image to the G channel and the B channel of the color image.

The display image generation unit 313 outputs a white light image and a narrow band light image to the display unit 400. The display unit 400 displays, for example, a white light image and a narrow band light image side by side. By using a white light image, it is possible to display the subject in a natural color. Further, by using a narrow band optical image, it is possible to improve the visibility of a specific subject such as a lesion as compared with a white light image.

The display image generation unit 313 may output either a white light image or a narrow band light image to the display unit 400. For example, the display image generation unit 313 performs a process of switching the image to be output to the display unit 400 based on the user input.

Although the embodiments to which the present invention is applied and the modified examples thereof have been described above, the present invention is not limited to the respective embodiments and the modified examples as they are, and the present invention is within a range that does not deviate from the gist of the invention at the embodiment. The components can be transformed and embodied with. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments and modifications. For example, some components may be deleted from all the components described in each embodiment or modification. Further, the components described in different embodiments and modifications may be combined as appropriate. In this way, various modifications and applications are possible within a range that does not deviate from the gist of the invention. In addition, a term described at least once in the specification or drawing together with a different term having a broader meaning or a synonym may be replaced with the different term at any part of the specification or drawing.

100 ... Light source device, 120 ... Combined part, 200 ... Insertion part, 210 ... Light guide, 220 ... Illumination lens, 230 ... Objective lens, 240 ... Image sensor, 241 ... Color filter array, 242 ... Filter unit, 250 ... A / D conversion circuit, 260 ... memory, 300 ... control device, 310 ... processing circuit, 311 ... interpolation processing unit, 313 ... display image generation unit, 320 ... control circuit, 400 ... display unit, 500 ... external I / F unit, F1 to F4 ... Frame, GR1 ... 1st illumination light group, GR2 ... 2nd illumination light group, IM1 ... 1st display image, IM2 ... 2nd display image, IM3 ... 3rd display image

Claims (9)

A light source device that emits a first illumination light group including green light having a green wavelength band and a second illumination light group including blue narrow band light belonging to the blue wavelength band and not including the green light at different timings. When,
An image sensor that has a color filter array including color filters of a plurality of colors arranged in two dimensions and images a subject,
A display image is displayed based on an image captured by the image sensor when the first illumination light group is irradiated and an image captured by the image sensor when the second illumination light group is irradiated. The processing circuit to generate and
Including
The color filter array includes a cyan color filter and the blue color filter.
The cyan color filter has an optical property of transmitting the green and blue light.
The processing circuit
An endoscope device for generating the display image based on the blue narrow band image obtained by the blue narrow band light transmitted through the cyan color filter and the blue color filter. In claim 1,
The first illumination light group is an endoscopic device including red light having a red wavelength band and blue light having the blue wavelength band. In claim 1,
The processing circuit
An endoscopic device characterized in that an image of a green channel in the display image is generated based on the green image obtained by the green light and the blue narrow band image. In claim 2,
The processing circuit
A white light image is generated based on the red image obtained by the red light, the green image obtained by the green light, and the blue image obtained by the blue light.
An endoscopy device characterized in that an enhancement process is performed on a green channel image of the white light image based on the blue narrow band image, and the white light image after the enhancement process is output as the display image. In claim 2,
The processing circuit
A white light image is generated as the display image based on the red image obtained by the red light, the green image obtained by the green light, and the blue image obtained by the blue light.
An endoscope device for generating a narrow band optical image as the display image based on the blue narrow band image and the green image. In claim 4 or 5,
The processing circuit
When the first illumination light group is irradiated, the pixel signal from the image sensor is subjected to the first interpolation processing, and when the second illumination light group is irradiated, the pixel from the image sensor is irradiated. An endoscope device characterized in that a second interpolation process different from the first interpolation process is performed on a signal. In claim 1,
An endoscope device characterized in that the peak wavelength of the blue narrow band light is in the range of 415 ± 20 nm. A method of operating an endoscope device having a color filter array including color filters of a plurality of colors arranged two-dimensionally and including an image pickup element for imaging a subject.
When the color filter array includes a cyan color filter and a blue color filter, and the cyan color filter has optical characteristics for transmitting green and blue light.
The first illumination light group including the green light having the green wavelength band and the second illumination light group including the blue narrow band light belonging to the blue wavelength band and not containing the green light are emitted at different timings. ,
A display image is displayed based on an image captured by the image sensor when the first illumination light group is irradiated and an image captured by the image sensor when the second illumination light group is irradiated. Generate and
When generating the display image, the display image is generated based on the blue narrow band image obtained by the cyan color filter and the blue narrow band light transmitted through the blue color filter. How to operate the endoscopic device. An image processing program for processing an image captured by an image sensor that captures a subject and has a color filter array including color filters of a plurality of colors arranged in two dimensions.
The color filter array includes a cyan color filter and a blue color filter, and the cyan color filter has an optical property of transmitting green and blue light.
When the first illumination light group contains green light having the green wavelength band, and the second illumination light group contains blue narrow band light belonging to the blue wavelength band and does not include the green light.
A display image is displayed based on an image captured by the image sensor when the first illumination light group is irradiated and an image captured by the image sensor when the second illumination light group is irradiated. Generate and
When generating the display image, a step of generating the display image based on the blue narrow band image obtained by the blue narrow band light transmitted through the cyan color filter and the blue color filter is performed.
An image processing program that you run on your computer.

Images