WO2022239495A1 - Biological tissue observation system, biological tissue observation device, and biological tissue observation method - Google Patents

Abstract

Provided is a biological tissue observation system provided with: an event vision sensor (200) that detects a change in the luminance value of light of a first wavelength band emitted from biological tissue as an event; and an image processing unit (500) that generates a first image on the basis of a plurality of sensing data output at different timings from the event vision sensor.

Description

Biological tissue observation system, biological tissue observation device, and biological tissue observation method

The present disclosure relates to a biological tissue observation system, a biological tissue observation device, and a biological tissue observation method.

In recent years, with the development of surgical techniques, surgical instruments, etc., surgeries (so-called microsurgery) are frequently performed while observing the affected area with medical observation devices such as endoscopes and surgical microscopes. It is now being done. In addition, some of such medical observation devices not only have a configuration for optically observing an affected area, but also display an image of the affected area captured by an imaging device (image sensor) or the like on a display device such as a monitor. Apparatuses and systems (observation systems) for displaying electronic images have also been proposed.

In addition, in recent years, as an observation method using observation devices such as endoscopes and surgical microscopes, not only a method of observing the operative field with light in the visible light band, but also an observation method called special light observation has been proposed. there is Examples of special light observation include narrow band imaging (NBI), fluorescence imaging (FI), and infrared light observation (IRI Infra-Red Imaging).

Specifically, in fluorescence observation, a fluorescent substance (fluorescent substance) that has an affinity for lesions such as cancer is administered to the person to be examined (patient) in advance, and then irradiated with excitation light to excite the fluorescent substance. Then, the lesion is observed using a fluorescent image of fluorescence emitted from the fluorescent substance accumulated in the lesion. For example, Non-Patent Document 1 discloses an overview of fluorescence observation using indocyanine green (ICG) as a fluorescent substance and an example of surgery based on the fluorescence observation.

Mitsuo Kusano, "All about ICG Fluorescence Navigation Surgery", Intermedica Co., Ltd., 2008, p. 265

However, in the conventional special light observation method, the image sensor detects the absolute value of the brightness of the observation light (for example, fluorescence). may saturate the signal output from the In such a case, it is difficult to observe a minute difference or change in luminance of observation light having a luminance equal to or higher than a certain value in an observation image based on the signal. On the other hand, when the brightness of the observation light is low, the signal becomes weak, so that it is buried in noise and the like, making it difficult to observe the image itself.

Therefore, the present disclosure proposes a biological tissue observation system, a biological tissue observation apparatus, and a biological tissue observation method capable of capturing minute differences and changes in observation light without being affected by the brightness of the observation light itself. .

According to the present disclosure, an event vision sensor that detects, as an event, a change in luminance value of light in a first wavelength band emitted from living tissue, and a plurality of sensing data output from the event vision sensor at different timings. and an image processor that generates a first image based on the biological tissue observation system.

Further, according to the present disclosure, an event vision sensor that detects, as an event, a change in luminance value of light in a first wavelength band emitted from living tissue, and a plurality of sensing signals output from the event vision sensor at different timings. and an image processor that generates a first image based on the data.

Further, according to the present disclosure, the event vision sensor detects, as an event, a change in luminance value of light in the first wavelength band emitted from the biological tissue, and the image processing unit differs from the event vision sensor. Generating a first image based on a plurality of sensing data output with timing is provided.

2 is a block diagram showing an example configuration of an EVS 200 used in an embodiment of the present disclosure; FIG. 2 is a block diagram showing an example of a configuration of a pixel 302 located in a pixel array section 300 in the EVS 200 shown in FIG. 1; FIG. It is a figure showing an example of composition of observation system 10 for medical science concerning a 1st embodiment of this indication. 3 is a diagram showing an example of a functional configuration of a signal processing section 500 according to the first embodiment of the present disclosure; FIG. 4 is a flowchart of an observation method according to the first embodiment of the present disclosure; FIG. 4 is an explanatory diagram for explaining an example of an observation image (photograph) according to the first embodiment; 1 is an explanatory diagram for describing a first embodiment of the present disclosure; FIG. FIG. 10 is an explanatory diagram for explaining Modification 1 of the first embodiment of the present disclosure; FIG. 10 is a flowchart of an observation method according to modification 2 of the first embodiment of the present disclosure; FIG. FIG. 11 is an explanatory diagram (part 1) for explaining Modification 2 of the first embodiment of the present disclosure; FIG. 11 is an explanatory diagram (part 2) for explaining Modification 2 of the first embodiment of the present disclosure; FIG. 11 is an explanatory diagram (part 3) for explaining Modification 2 of the first embodiment of the present disclosure; It is a figure showing an example of composition of observation system 10a for medical science concerning a 2nd embodiment of this indication. FIG. 7 is a diagram showing an example of a functional configuration of a signal processing unit 500a according to the second embodiment of the present disclosure; 6 is a flow chart of an observation method according to a second embodiment of the present disclosure; FIG. 11 is an explanatory diagram for explaining an example of an observation image according to the second embodiment; FIG. FIG. 4 is a diagram showing an example of temporal changes in brightness of each site in cerebral artery bypass surgery. FIG. 18 is a diagram showing an example of a drawing result based on FIG. 17; FIG. 2 is a functional block diagram showing a configuration example of a hardware configuration of an information processing device that constitutes the medical observation system 10 according to the embodiment; It is a figure which shows an example of a schematic structure of an endoscope system. 21 is a block diagram showing an example of functional configurations of the camera and CCU shown in FIG. 20; FIG. It is a figure which shows an example of a schematic structure of a microsurgery system.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description. In addition, in this specification and drawings, a plurality of components having substantially the same or similar functional configuration may be distinguished by attaching different alphabets after the same reference numerals. However, when there is no particular need to distinguish between a plurality of components having substantially the same or similar functional configurations, only the same reference numerals are used.

In the following description, a tissue section or cell that is part of a tissue (eg, organ or epithelial tissue) obtained from a living body (eg, human body, plant, etc.) is referred to as a biological tissue. More specifically, in the following description, examples of living tissue include in-vivo tubular living tissue such as blood vessels and lymphatic vessels.

Note that the description will be given in the following order.
1. Introduction 2. About EVS 3 . First Embodiment 3.1 Configuration of Medical Observation System 3.2 Configuration of Signal Processing Section 3.3 Observation Method 3.4 Modification 1
3.5 Modification 2
4. Second Embodiment 4.1 Configuration of Medical Observation System 4.2 Configuration of Signal Processing Section 4.3 Observation Method5. Use case 6. Summary 7. An example of hardware configuration8. Application example 9 . supplement

<<1. Introduction>>
First, among the special light observations mentioned as an example of observation methods using observation devices such as endoscopes and surgical microscopes in the previous explanation, observation of fluorescent images of lesions, etc., by using fluorescent substances in particular. An outline of the fluorescence observation to be performed is explained. Next, based on the outline, the background leading to the creation of the medical observation system according to the embodiment of the present disclosure by the present inventors will be described.

In fluorescence observation, a fluorescent substance that has an affinity for lesions such as cancer is administered in advance to a subject (patient) to be examined, and by irradiating excitation light to excite the fluorescent substance, the fluorescent substance that has accumulated in the lesion is observed. The lesion can be observed from the fluorescent image of the fluorescence emitted from. Indocyanine green (ICG) is a representative example of a fluorescent substance used for such fluorescence observation. ICG can emit fluorescence having a wavelength around 820 nm (that is, light in the near-infrared band) when irradiated with light having a wavelength around 808 nm as excitation light. Furthermore, in recent years, as fluorescent substances used for fluorescence observation, from the viewpoint of reducing the effects (side effects) on the test subject due to the characteristics of more selectively accumulating in lesions such as cancer, and the administration (side effects). , and ICG, various fluorescent materials have been proposed. Further, among such fluorescent substances, substances that emit fluorescence in a wavelength band different from that of ICG have been proposed, and fluorescent substances that emit light belonging to the visible light wavelength band are also included.

In fluorescence observation, for example, by observing the presence or absence of fluorescence emitted by a fluorescent substance such as ICG and the time change of the fluorescence, the operator (user) can visually observe blood flow and lymphatic flow. . In addition, by analyzing the time change of the fluorescence emitted by the fluorescent substance (that is, the time change of the fluorescent substance flowing in the patient's body), for example, desired media such as blood vessels and lymph vessels (for example, blood and lymph) ) can also be distinguished between good and bad flow portions.

However, in the conventional fluorescence observation method, the image sensor detects the absolute value of the luminance of the fluorescence emitted from the fluorescent material. signal may saturate. In such a case, it is difficult to observe a minute difference or change in the luminance of fluorescent light having a luminance higher than a certain value in a fluorescent image based on the signal. On the other hand, when the luminance of the fluorescence is low, the signal becomes weak, so that it is buried in noise and the like, making it difficult to observe the fluorescence image itself. In addition, when it becomes difficult to observe a fluorescent image or a minute difference in a fluorescent image, for example, even if it is possible to determine the presence or absence of blood containing a fluorescent substance, minute changes in the blood flow or the like may occur. It is difficult to observe changes in flow direction, flow velocity, etc. Furthermore, in the case described above, observation may be made possible by adjusting the sensitivity (amplification factor) of the image sensor that detects fluorescence. It is difficult to adjust the sensitivity appropriately because the brightness is too high and the signal is saturated because the brightness is too high for observation.

Furthermore, as a problem when using fluorescence observation using a fluorescent substance such as ICG in surgical operations, when a drug containing a fluorescent substance (fluorescent agent) is administered and fluorescence imaging is performed, the fluorescence is transmitted through the bloodstream. Substances adhere to the walls of blood vessels and proteins in tissues, and may emit fluorescence for a while. Under such circumstances, when it is attempted to repeatedly perform the administration of a fluorescent agent and the fluorescence observation within a relatively short period of time, immediately after the administration of a new fluorescent agent, the fluorescent substance associated with the previous administration , may be mixed with the fluorescent substance associated with the new administration. Due to such a mixture, the fluorescence emitted by the fluorescent substance associated with the previous administration remains, making it difficult to distinguish and recognize the fluorescence emitted by the fluorescent substance associated with the new administration from the remaining fluorescence. Therefore, it is conceivable to observe the fluorescence emitted by the newly administered fluorescent substance after the previously administered fluorescent substance remaining in the observation area is released (washed out). However, in order to use such a method, there is a problem that the time required for fluorescence observation tends to be long because it takes time until the fluorescent substance associated with the previous administration is released.

In view of the situation described above, the present inventors came up with the idea of using an Event Vision Sensor (EVS) as an image sensor for fluorescence observation.

The EVS is a sensor that detects changes in luminance rather than detecting the absolute value of luminance, and has higher sensitivity than general RGB sensors. In addition, EVS has no concept of frame rate, and can output time stamp information (time) and pixel information (position, amount of change in brightness) when a change in brightness occurs exceeding a threshold. Therefore, even if the luminance is high, the EVS detects only the change in luminance, so that the output signal is not saturated. Therefore, according to EVS, minute differences and changes in fluorescence luminance can be captured with high temporal resolution without being affected by the absolute value of fluorescence luminance.

More specifically, for example, when a drug containing a fluorescent substance (fluorescent drug) is administered to a blood vessel or the like and blood flow is observed, the signal from the sensor may become saturated even if the brightness of the fluorescence increases over time. It is required to observe minute changes in the brightness of the fluorescence without the need to observe the intensity of the fluorescence. In such a case, by using the above-described EVS, it is possible to detect only changes in luminance, so that minute changes in fluorescence luminance can be observed without saturating the signal. In particular, when observing fluorescence from ICG, the purpose is often to observe its change over time. Therefore, by using EVS, the change in fluorescence itself can be captured without being affected by other stationary signals.

In this way, the present inventors created an embodiment of the present disclosure that uses EVS to capture minute changes in fluorescence without being affected by the absolute value of fluorescence luminance. The details of such embodiments of the present disclosure will be sequentially described below.

<<2. About EVS>>
Here, the EVS will be described with reference to FIGS. 1 and 2. FIG. FIG. 1 is a block diagram showing an example of the configuration of the EVS 200 used in the embodiment of the present disclosure, and FIG. 2 shows an example of the configuration of the pixels 302 located in the pixel array section 300 in the EVS 200 shown in FIG. It is a block diagram.

As shown in FIG. 1, the EVS 200 has a pixel array section 300 configured by arranging a plurality of pixels 302 (see FIG. 2) in a matrix. Each pixel 302 can generate a voltage corresponding to a photocurrent generated by photoelectric conversion as a pixel signal. Also, each pixel 302 can detect the presence or absence of an event by comparing the change in photocurrent corresponding to the amount of change in luminance of incident light (light emitted from the object) with a predetermined threshold. In other words, pixel 302 can detect an event based on the amount of luminance change exceeding a predetermined threshold.

Furthermore, as shown in FIG. 1 , the EVS 200 has a drive circuit 211 , an arbiter section (arbitration section) 213 , a column processing section 214 , and a signal processing section 212 as peripheral circuit sections of the pixel array section 300 .

When detecting an event, each pixel 302 can output a request to the arbiter unit 213 requesting output of event data representing the occurrence of the event. Then, each pixel 302 outputs the event data to the driving circuit 211 and the signal processing unit 212 when receiving a response indicating permission to output the event data from the arbiter unit 213 . Also, the pixel 302 that has detected an event outputs a pixel signal generated by photoelectric conversion to the column processing unit 214 .

The drive circuit 211 can drive each pixel 302 of the pixel array section 300 . For example, the drive circuit 211 detects an event, drives the pixel 302 that outputs the event data, and outputs the pixel signal of the corresponding pixel 302 to the column processing unit 214 .

The arbiter unit 213 arbitrates requests requesting the output of event data supplied from each of the pixels 302, responds based on the arbitration result (permission/non-permission of event data output), and resets event detection. A reset signal can be sent to the pixel 302 to do so.

The column processing unit 214 can perform processing for converting analog pixel signals output from the pixels 302 of the corresponding column into digital signals for each column of the pixel array unit 300 . The column processing unit 214 can also perform CDS (Correlated Double Sampling) processing on digitized pixel signals.

The signal processing unit 212 performs predetermined signal processing on the digitized pixel signals supplied from the column processing unit 214 and the event data output from the pixel array unit 300, and converts the signal-processed event data ( time stamp information, etc.) and pixel signals can be output.

A change in the photocurrent generated by the pixel 302 can be regarded as a change in the amount of light (luminance change) incident on the pixel 302 . Therefore, an event can also be said to be a luminance change of pixel 302 exceeding a predetermined threshold. Furthermore, the event data representing the occurrence of an event can include at least positional information such as coordinates representing the position of the pixel 302 where the change in the amount of light has occurred as an event.

Furthermore, the pixel 302 will be described with reference to FIG. In a pixel array section 300 configured by arranging a plurality of pixels 302 in a matrix, each pixel 302 has a light receiving section 304 , a pixel signal generation section 306 and a detection section (event detection section) 308 .

Specifically, the light receiving unit 304 can photoelectrically convert incident light to generate a photocurrent. Then, the light receiving unit 304 can supply a voltage signal corresponding to the photocurrent to either the pixel signal generating unit 306 or the detecting unit 308 under the control of the driving circuit 211 .

The pixel signal generation unit 306 can generate the signal supplied from the light receiving unit 304 as a pixel signal. Then, the pixel signal generation unit 306 can supply the generated analog pixel signals to the column processing unit 214 via vertical signal lines VSL (not shown) corresponding to columns of the pixel array unit 300 .

The detection unit 308 can detect whether an event has occurred based on whether the amount of change in photocurrent from the light receiving unit 304 has exceeded a predetermined threshold. The events can include, for example, an ON event indicating that the amount of change in photocurrent (amount of luminance change) has exceeded the upper limit threshold, and an OFF event indicating that the amount of change has fallen below the lower limit threshold. . Note that the detection unit 308 may detect only on-events.

When an event occurs, the detection unit 308 can output to the arbiter unit 213 a request to output event data representing the occurrence of the event. Then, when receiving a response to the request from the arbiter unit 213 , the detection unit 308 can output event data to the drive circuit 211 and the signal processing unit 212 .

In the embodiment of the present disclosure, by using such an EVS 200, only changes in the luminance of fluorescence are captured, so it is possible to capture minute changes in luminance without being affected by the luminance of fluorescence itself. .

In the following description, the case of application to fluorescence observation will be described as an example, but this is only an example, and the embodiments of the present disclosure are applicable to observation using light in the visible light band, and various It can also be applied to special light observation.

<<3. First Embodiment>>
<3.1 Configuration of medical observation system>
First, with reference to FIG. 3, an example of the configuration of a medical observation system (biological tissue observation system) 10 according to the first embodiment of the present disclosure will be described. FIG. 3 is a diagram showing an example of the configuration of the medical observation system 10 according to this embodiment. As shown in FIG. 3 , the medical observation system 10 mainly includes an EVS 200 , a light source (first illumination device) 400 and a signal processing section (image processing section) 500 . Each device included in the medical observation system 10 according to this embodiment will be described below.

(EVS200)
The EVS 200 can detect a change in luminance value of observation light (for example, fluorescence) emitted from the living tissue, which is the subject 700, as an event. As described above, the EVS 200 includes a pixel array section 300 having a plurality of pixels 302 arranged in a matrix and a detection section (event detector) 308 . Specifically, the EVS 200 can detect changes in the luminance value of observation light in a predetermined wavelength band (first wavelength band), for example, fluorescence with a wavelength around 820 nm (that is, near-infrared band). light) can be detected. Therefore, the EVS 200 measures, for example, the brightness of fluorescence emitted by a fluorescent substance (a drug containing a substance capable of emitting light in the first wavelength band) administered once or multiple times to the living tissue that is the subject 700. Changes can be detected.

In addition, the brightness of the fluorescence emitted from the fluorescent material administered to the subject 700 is weaker than, for example, the visible light reflected from the subject 700 or the excitation light that excites the fluorescent material. Therefore, in the present embodiment, as shown in FIG. 3, the EVS 200 has a cut filter (CF) 250 that cuts (does not transmit) visible light and excitation light in order to suitably detect fluorescence. is preferred. In addition, in this embodiment, instead of such a cut filter 250, a prism that separates observation light (for example, fluorescence) to be detected from visible light and excitation light and guides it to the EVS 200 may be used. Note that in the present embodiment, a fluorescence image (observation image) can be obtained by integrating the output signal from the EVS 200 as it is. The cut filter 250 for cutting light (visible light and excitation light) may not be provided.

(Light source 400)
The light source 400 is composed of, for example, an LED (Light Emitting Diode), a laser, or the like, and can irradiate excitation light that excites a fluorescent substance contained in a drug administered to the biological tissue, which is the subject 700 . The excitation light is light in a wavelength band (second wavelength band) different from the wavelength band (first wavelength band) of observation light (e.g., fluorescence) detected by the EVS 200. For example, ICG is used as a fluorescent substance. In this case, the excitation light for ICG is light having a wavelength near 808 nm. By being irradiated with such excitation light, ICG can emit fluorescence having a wavelength of around 820 nm.

(Signal processing unit 500)
The signal processing unit 500 can generate image data (observation image) (first image), which is a fluorescence image by fluorescence, based on a plurality of signals (sensing data) output from the EVS 200 at different timings. A detailed configuration of the signal processing unit 500 will be described later.

Note that the medical observation system 10 according to this embodiment is not limited to the configuration shown in FIG. It may also include other communication devices such as relay devices. Furthermore, in the present embodiment, the signal processing section 500 is not limited to being composed of an integrated device as shown in FIG. 3, and may be composed of a plurality of devices.

<3.2 Configuration of Signal Processing Unit>
Next, an example of the functional configuration of the signal processing unit 500 according to this embodiment will be described with reference to FIG. FIG. 4 is a diagram showing an example of the functional configuration of the signal processing section 500 according to this embodiment. The signal processing unit 500 includes a CPU (Central Processing Unit) and the like, and functions as a processing unit and a control unit. Specifically, as shown in FIG. 4, the signal processing unit 500 includes a signal acquisition unit 502, a frame memory (storage unit) 504, an image processing unit 506, an image adjustment unit 508, an output unit 510, and a control unit. It mainly has a section 520 , an adjustment section 522 and an input section 524 . Each functional unit included in the signal processing unit 500 according to this embodiment will be described below.

(Signal acquisition unit 502)
The signal acquisition unit 502 can acquire event data (time stamp information, etc.) and pixel signals, which are output signals (sensing data) from the EVS 200, and output them to an image processing unit 506, an adjustment unit 522, etc., which will be described later.

(Frame memory 504)
The frame memory 504 is composed of a magnetic storage device such as a HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like, and stores image data output from the image processing unit 506, which will be described later. can be done.

(Image processing unit 506)
The image processing unit 506 can generate image data (observation image) (first image), which is a fluorescence image, based on a plurality of signals (sensing data) output from the EVS 200 at different timings. Specifically, the image processing unit 506 reads the image data (sensing data) before the current frame from the frame memory 504, and adds the read image data to the EVS 200 after the image data via the signal acquisition unit 502. are added or sequentially added to generate new image data for the current frame. The image processing unit 506 then outputs the newly generated image data of the current frame to the image adjusting unit 508, which will be described later.

(Image adjustment unit 508)
The image adjustment unit 508 performs brightness adjustment (gain adjustment) and contrast adjustment (gamma adjustment) on the image data of the current frame output from the image processing unit 506, and performs output unit 510 and adjustment unit 522, which will be described later. can be output to

(Output unit 510)
The output unit 510 can output the image data from the image adjustment unit 508 to a display unit (not shown) in order to present the observation image to the user. Furthermore, the output unit 510 may output the image data to the frame memory 504 . In addition, in the medical observation system 10 according to the present embodiment, the display unit is configured by, for example, an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display, or the like.

(control unit 520)
A control unit 520 can control the EVS 200 and the light source 400 . For example, the control unit 520 can control the sensitivity (amplification) of the EVS 200, the threshold to be compared with the amount of change in brightness for detecting the presence or absence of an event, and the irradiation intensity of the light source 400.

(Adjuster 522)
The adjustment unit 522 determines (adjusts) a threshold (predetermined threshold) to be compared with the luminance change amount to detect the presence or absence of an event, and outputs a command signal based on the determined threshold to the control unit 520 described above. can be done.

For example, the adjustment unit 522 may determine the threshold based on the frequency of output of signals (sensing data) from the EVS 200 . Further, for example, the adjustment unit 522 may determine the threshold value based on the dynamic range of luminance or the gradation of the image data (first image) output from the image adjustment unit 508 .

In addition, for example, the adjustment unit 522 adjusts the velocity of the fluorescent substance (fluorescent agent) in the biological tissue that is the subject 700 (more specifically, the flow of the medium in the tubular biological tissue in the living body, such as the blood flow in the blood vessel) or The threshold may be determined based on the rate of change in fluorescence brightness. Furthermore, for example, the adjustment unit 522 determines the threshold value based on the type, size, position, state, displacement speed, deformation speed, surrounding state, etc. of the living tissue that is the subject 700 obtained from the image data. may In the present embodiment, the adjustment unit 522 may recognize the type, state, surrounding state, etc. of the living tissue using, for example, an image recognition model obtained by machine learning.

For example, the adjustment unit 522 may determine the threshold based on the depth of the blood vessel or organ that is the observation target. Specifically, when the observation target is located at a deep position, the brightness of the fluorescence decreases. Therefore, the adjusting unit 522 sets the threshold to be small, so that the EVS 200 can more reliably detect changes in the fluorescence. make it On the other hand, when the observation target is close to the superficial layer, there are many blood vessels, for example, and the brightness of the fluorescence is high. In addition, when the observation target is the digestive organ, there is a lot of fat around it, and the fat blocks the fluorescence observation, making it difficult to observe fluorescence.

Further, for example, the adjustment unit 522 may determine the threshold value based on the number of times of administration, type, or concentration of the fluorescent substance (fluorescent agent) input by the user via the input unit 524, which will be described later. In the present embodiment, the adjustment unit 522 may set the threshold to be sequentially smaller, for example, as the number of times of administration increases.

Furthermore, for example, the adjustment unit 522 may determine the threshold based on the irradiation intensity of the excitation light from the light source 400 . In the present embodiment, the adjustment unit 522 may set the threshold to be sequentially smaller, for example, as the irradiation intensity becomes lower. In this way, by adjusting the irradiation intensity and the threshold value in conjunction with each other, it is possible to detect changes in fluorescence luminance even when the irradiation intensity is low. Influence of excitation light on 700 can be suppressed.

(Input section 524)
The input unit 524 can receive input of data and commands from the user, and can output information input by the received input operation to the adjustment unit 522 . More specifically, the input unit 524 can be configured with a keyboard, touch panel, buttons, microphone, or the like.

It should be noted that in the present embodiment, the signal processing section 500 is not limited to being configured by the functional sections as shown in FIG. 4, and may further include other functional sections. In addition, in the present embodiment, the signal processing unit 500 is not limited to being composed of an integrated device as shown in FIG. 4, and may be composed of a plurality of separate devices.

<3.3 Observation method>
Next, an example of an observation method according to this embodiment will be described with reference to FIGS. 5 to 7. FIG. FIG. 5 is a flowchart of the observation method according to this embodiment, and FIG. 6 is an explanatory diagram for explaining an example of an observed image (photograph). FIG. 7 is an explanatory diagram for explaining the present embodiment, and more specifically, an explanatory diagram for explaining threshold setting.

As shown in FIG. 5, the observation method according to this embodiment can include a plurality of steps from step S101 to step S107. Details of each of these steps according to the present embodiment will be described below.

First, the signal processing unit 500 reads the image data of the previous frame stored in the frame memory 504 (the image data stored in the frame memory 504 immediately before) (step S101). In this embodiment, if the image data in the frame memory 504 has been reset (erased) immediately before, step S101 may be omitted. Next, the signal processing unit 500 adds the output signal (sensing data) output from the EVS 200 to the read image data (step S102).

Then, the signal processing unit 500 adjusts the brightness (gain) of the image data to which the signal has been added in step S102 (step S103). Next, the signal processing unit 500 adjusts the contrast (gamma) (step S104). Further, the signal processing unit 500 writes the image data adjusted in steps S103 and S104 to the frame memory 504 (step S105).

The signal processing unit 500 receives an input from the user or a result of whether or not the image data written in the frame memory 504 in step S105 described above satisfies preset conditions (for example, the number of additions of the output signal, the observation time etc.), it is determined whether or not necessary image data has been obtained (step S106). If the signal processing unit 500 determines that the necessary image data has been obtained (step S106: Yes), the process proceeds to step S107, and if it determines that the necessary image data has not been obtained (step S106: No), the process proceeds to step S101.

The signal processing unit 500 outputs the image data written in the frame memory 504 in step S105 (step S107). Then, the signal processing unit 500 ends the series of processes.

FIG. 6 shows image data (observation image) obtained by the observation method according to the present embodiment as described above. In detail, the left side of FIG. 6 shows an observation image obtained by this embodiment, and the right side shows an observation image obtained by a conventional fluorescence observation method (more specifically, a method using a conventional image sensor). Observation images are shown.

As shown on the right side of FIG. 6, in the observation image obtained by the conventional fluorescence observation method, the signal from the image sensor is saturated due to the luminance of the fluorescence having a luminance higher than a certain value, and the entire corresponding portion becomes white. It is not possible to recognize minute differences in fluorescence brightness at the observation site. On the other hand, in the observation image according to the present embodiment, only the change in the fluorescence luminance by the EVS 200 is detected, and the output signal is added to obtain an image. signal from is never saturated. Therefore, as shown on the left side of FIG. 6, minute differences in fluorescence brightness can be recognized in the observed image.

In this embodiment, as shown in FIG. 7, an event is output when there is a change in fluorescence luminance greater than the threshold value C, and image data is obtained by adding the signals of the output event. It will happen. Therefore, for example, when it is desired to set the luminance gradation in image data to 8 bits, the value obtained by subtracting the minimum luminance value from the maximum luminance value, which is the dynamic range (DR) of the image data, is divided into 256 (=8 bits). Threshold value C may be set as the value obtained by However, in the image data obtained by the EVS 200, since the luminance is on a log scale, it should be noted that the higher the luminance, the coarser the gradation. Also, in FIG. 7, an event detected by a change in brightness increasing is indicated by an upward arrow, and an event detected by a change in brightness decreasing is indicated by a downward arrow.

As described above, in this embodiment, by using the EVS 200, minute changes in fluorescence can be captured without being affected by the absolute value of fluorescence luminance.

<3.4 Modification 1>
Next, referring to FIG. 8, a case where the CF 250 is not provided will be described as Modification 1 of the present embodiment. FIG. 8 is an explanatory diagram for explaining Modification 1 of the present embodiment, and more specifically, an explanatory diagram for explaining threshold setting in this modification.

As described above, the brightness of the fluorescence emitted from the fluorescent material administered to the subject 700 is weaker than, for example, the visible light reflected from the subject 700 or the excitation light that excites the fluorescent material. Therefore, in the present embodiment described above, the EVS 200 preferably has the CF 250 that cuts (does not transmit) visible light and excitation light in order to suitably detect fluorescence.

However, in the embodiment of the present disclosure, a fluorescence image (observation image) can be obtained by integrating the output signal from the EVS 200 as it is. It is not necessary to provide the CF 250 that cuts the effective light (visible light and excitation light). Hereinafter, setting of the threshold value C when such a CF 250 is not provided will be described as this modified example.

If the CF 250 is not provided, the EVS 200 will detect, for example, the reflection of visible light from the subject 700 and the luminance change due to the excitation light, in addition to the fluorescence. As described above, the brightness of the fluorescence emitted from the fluorescent substance (fluorescent agent) administered to the subject 700 is very small compared to the visible light reflected from the subject 700. also tend to be very small. Therefore, in this modified example, it is preferable to set the threshold value C small in order to detect changes in fluorescence luminance.

Specifically, as shown in FIG. 8, the brightness of the image data obtained by this modified example is such that the brightness of the fluorescent light is above the constant range of the brightness components of the visible light and the excitation light (lower side of FIG. 8). The changes (lower side of FIG. 8) are sequentially added. Therefore, in this modified example, the threshold value C is set based on the dynamic range (DR) obtained by subtracting the minimum luminance value from the maximum fluorescence luminance value, thereby obtaining an image that is a fluorescent image having a desired gradation. data can be obtained.

Note that the present embodiment and Modification 1 are not limited to setting the threshold C based on the dynamic range (DR). For example, the threshold C may be determined based on the event detection frequency. good. For example, when the event detection frequency is higher than a predetermined frame rate (eg, 60 fps), the threshold C may be increased to lower the event detection frequency. Also, when the event detection frequency is lower than a predetermined frame rate (eg, 60 fps), the threshold C may be decreased to increase the event detection frequency.

<3.5 Modification 2>
Further, as a modification 2 of the present embodiment, an example applied to fluorescence observation when a fluorescent substance (fluorescent drug) as described above is administered multiple times will be described with reference to FIGS. 9 to 12. FIG. FIG. 9 is a flowchart of an observation method according to Modification 2 of this embodiment, and FIGS. 10 and 11 are explanatory diagrams for explaining Modification 2 of this embodiment. Furthermore, FIG. 12 is an explanatory diagram for explaining Modification 2 of the present embodiment, and is an explanatory diagram for explaining threshold setting in this modification.

In this modification, the image data stored in the frame memory 504 is reset at the timing of administering the fluorescent agent for the second and subsequent times, and a new output signal from the EVS 200 is added. It is possible to obtain an observation image, which is a fluorescence image emitted by . According to this modification, it is not necessary to perform observation after the fluorescent agent remaining in the observation area is released (after it is washed out), so it is possible to avoid an increase in the time required for observation.

Specifically, as shown in FIG. 9, the observation method according to this modification can mainly include a plurality of steps from step S201 to step S205. Details of each of these steps according to the present modification will be described below.

The user administers the first fluorescent agent (fluorescent substance) to the subject 700 (for example, a tubular biological tissue in the living body such as a blood vessel) (step S201). Next, the signal processing unit 500 sequentially adds the signals output from the EVS 200 to the image data (step S202).

The signal processing unit 500 determines whether a new fluorescent agent has been administered based on the input from the user (step S203). If it is determined that a new fluorescent agent has been administered (step S203: Yes), the signal processing unit 500 proceeds to step S204, and if it is determined that a new fluorescent agent has not been administered. (Step S203: No), the process proceeds to step S205.

Note that step S203 according to the present embodiment is not limited to determining whether or not a new fluorescent agent has been administered based on the input from the user. For example, the signal processing unit 500 performs image analysis on the image data to which the signal has been added in step S202 to determine the timing of administration of the fluorescent agent, that is, whether or not a new fluorescent agent has been administered. may be detected. More specifically, it is possible to detect the timing of the first administration of the fluorescent agent by detecting the rising edge of the luminance value in the image data to which the signal output from the EVS 200 is added. Furthermore, by detecting the timing at which the lowered luminance value starts to rise again in the temporal change of the luminance value in the image data, it is possible to detect the timing of the second and subsequent administrations of the fluorescent agent.

Then, the signal processing unit 500 clears the output image data stored in the frame memory 504, and returns to the process of step S202 described above (step S204). On the other hand, the signal processing unit 500 stores the output image data generated in step S202 described above in the frame memory 504 (step S205), and ends the series of processes.

Here, an example of image data according to this modified example will be described with reference to FIG. In FIG. 10, for example, V101 schematically shows an output image (fluorescence image) immediately before the first administration of the fluorescent agent. V103 schematically shows an output image (fluorescence image) after the first administration of the fluorescent agent. V105 schematically shows an output image (fluorescence image) immediately after the second administration of the fluorescent agent, obtained by a conventional fluorescence observation method (more specifically, a method using a conventional image sensor). (It is assumed that time has passed from V103 to V105). Further, V107 schematically shows an output image (fluorescence image) output after the second administration of the fluorescent agent, obtained by a conventional fluorescence observation method.

In the conventional fluorescence observation method, when the second administration of the fluorescent agent is performed, based on the output image V103 immediately before the administration, fluorescence emitted by the fluorescent substance contained in the newly administered fluorescent agent is detected as a result of detection. signals are added. Therefore, the output images V105 and V107 that are output thereafter are brighter than the output image V103 as a whole because the detection result of the fluorescence corresponding to the first administration of the fluorescent agent remains. As shown in 10, the image becomes whitish over the entire image. Therefore, the output image presented after the second administration of the fluorescent agent has, for example, a decrease in contrast, blown-out highlights, etc., and the visibility of the site to be observed due to the fluorescence is reduced.

On the other hand, in FIG. 10, V109 schematically shows an output image (fluorescence image) immediately after the second administration of the fluorescent agent according to this modified example. V111 schematically shows an output image (fluorescence image) output after the second administration of the fluorescent agent according to this modified example.

In this modified example, as shown in the output image V109, the image data stored in the frame memory 504 is reset at the timing of the second administration of the fluorescent agent. Furthermore, according to this modification, the EVS 200 detects the fluorescence in the portion where the luminance increased after the second administration of the fluorescent agent. In this case as well, the visibility of the site to be observed by fluorescence can be ensured in the same manner as in the observation after the first administration of the fluorescent agent.

In the example shown in FIG. 10, it is assumed that the first and second fluorescent agents are the same type of agent (fluorescent substance), but in this modified example, they are different types of agents (fluorescent substance). may Further, in this modification, the image data stored in the frame memory 504 at the timing of the second administration of the fluorescent agent may be stored separately without being discarded. By doing so, the user can compare the fluorescence image before the second administration of the fluorescent agent and the fluorescence image after the second administration of the fluorescence agent.

Furthermore, this modification can also be applied, for example, when evaluating blood flow before and after intestinal anastomosis by fluorescence observation using ICG as a fluorescent substance. In the evaluation of blood flow before and after intestinal anastomosis, it is required to recognize the boundaries between areas with blood flow and areas without blood flow.

Specifically, as shown in the upper part of FIG. 11, the first administration of the fluorescent agent is performed before the intestinal anastomosis. A blood flow assessment is performed. It should be noted that it takes about several minutes from administration of the fluorescent agent to observation of fluorescence emitted from the administered fluorescent agent. For example, V201 in FIG. 11 schematically shows a region where fluorescence emitted by ICG (fluorescent substance) contained in the first administered fluorescent agent is detected. That is, the expansion of the region V201 confirms the state of blood circulation (for example, how far the blood flow reaches), and the anastomosis site is determined according to the result of the confirmation.

Subsequently, the fluorescent agent is administered for the second time after the anastomosis, and the blood flow after the anastomosis is evaluated based on the fluorescence image according to the fluorescence detection result after the administration. For example, a fluorescence image V210 shown in the lower left of FIG. 11 is obtained by a conventional fluorescence observation method (more specifically, a method using a conventional image sensor) and is presented after the second administration of the fluorescent agent. An example is shown. In addition, V211 in the fluorescence image V210 schematically shows the anastomosis site, and V213 schematically shows the region corresponding to the first fluorescence component. On the other hand, V215 schematically shows the region where fluorescence emitted by ICG contained in the fluorescent agent administered for the second time was detected. Therefore, in the fluorescence image V210 obtained by the conventional fluorescence observation method, the first fluorescence component remains in the fluorescence image V210, and the fluorescence signal is added to the corresponding area V213. Therefore, it becomes difficult to determine which portion in the fluorescence image V210 corresponds to the second fluorescence component, and thus, the first administration of the fluorescent agent (before anastomosis) and the second administration of the fluorescent agent are difficult. It becomes difficult to compare with the time of administration (after anastomosis).

On the other hand, the fluorescence image V220 shown in the lower right of FIG. 11 shows an example of an output image presented after the second administration of the fluorescence agent, obtained by this modified example. Specifically, V221 in V220 schematically shows the anastomosis site, and V223 schematically shows the region corresponding to the second fluorescence component. In the fluorescence image V220 of this modified example, since there is no effect of the first fluorescence component, the region V223 corresponding to the second fluorescence component is the same as the fluorescence image V200 presented after the first administration of the fluorescent agent. It becomes possible to visually recognize with visibility. That is, according to this modification, it is possible to determine which portion in the fluorescence image V220 corresponds to the second fluorescence component. (before anastomosis) and the second time of administration of the fluorescent agent (after anastomosis).

In this modified example, in the case of fluorescence observation by administering the fluorescent agent for the second and subsequent times, since the fluorescence from the previous fluorescent agent remains, the signal intensity of the fluorescent component due to the newly administered fluorescent agent is relatively becomes smaller. Further, as shown in FIG. 12, the brightness of the image data obtained by this modification is such that the newly administered fluorescent agent is above the range of the brightness component of the fluorescence by the previous fluorescent agent (lower side in FIG. 12). Fluorescence changes due to (lower side of FIG. 12) are sequentially added. Therefore, in this modified example, the threshold C is set based on the dynamic range (DR) obtained by subtracting the minimum luminance value from the maximum luminance luminance value of the newly administered fluorescent agent, thereby obtaining the desired gradation. image data, which is a fluorescent image due to the newly administered fluorescent agent, can be obtained. In addition, in this modified example, it is preferable to set a small threshold value because the change in fluorescence luminance decreases as the number of times of administration increases.

<<4. Second Embodiment>>
<4.1 Configuration of Medical Observation System>
Further, in the embodiment of the present disclosure, not only the EVS 200 but also an RGB sensor capable of capturing a visible light image may be used together. In this case, the effects obtained by the EVS 200 can be enjoyed in the same manner as in the above-described first embodiment. Motion compensation can be performed to remove the fluorescence change component due to .

First, with reference to FIG. 13, an example of the configuration of a medical observation system (biological tissue observation system) 10a according to the second embodiment of the present disclosure will be described. FIG. 13 is a diagram showing an example of the configuration of a medical observation system 10a according to this embodiment. As shown in FIG. 13, the medical observation system 10a mainly includes an EVS 200, a light source (first and second lighting devices) 400a, a signal processing section 500a, and an RGB sensor 600. Each device included in the medical observation system 10a according to the present embodiment will be described below, but the description of the devices common to the first embodiment will be omitted.

(Light source 400a)
As in the first embodiment, the light source 400a can irradiate excitation light that excites a fluorescent agent (fluorescent substance) that has been administered to the biological tissue that is the subject 700 . Furthermore, the light source 400a can irradiate the subject 700 with observation light (eg, visible light (light having a wavelength of approximately 360 nm to approximately 830 nm) (light in a third wavelength band), for example. In terms of form, the light source 400a is not limited to irradiating excitation light and visible light. 900 nm to about 2500 nm) may be irradiated onto the subject 700. In the present embodiment, the light source 400a may be configured as an integrated device as shown in FIG. It may consist of two or more devices of the body, and is not particularly limited.

(Signal processing unit 500a)
As in the first embodiment, the signal processing unit 500a can generate an observed image of fluorescence (fluorescence image) based on a plurality of signals (sensing data) output from the EVS 200 at different timings. At that time, the signal processing unit 500a can perform motion compensation on the fluorescence image based on the visible light image based on the signal from the RGB sensor 600, which will be described later. A detailed configuration of the signal processing unit 500a will be described later.

(RGB sensor 600)
The RGB sensor 600 mainly includes a plurality of pixels (not shown) arranged in a matrix and a peripheral circuit section (not shown) that outputs an image based on light incident on each of the pixels as a pixel signal. Furthermore, the pixel signal output from the RGB sensor 600 is transmitted to the signal processing section 500a described above. Specifically, the RGB sensor 600 has a Bayer arrangement capable of detecting visible light (light in a third wavelength band different from the first wavelength band) such as blue light, green light, and red light, for example. For example, an image sensor capable of capturing a high-resolution image of 4K or higher is preferable. By using such an image sensor, a desired visible light image can be obtained with high resolution.

It should be noted that in this embodiment, the EVS 200 and the RGB sensor 600 are not limited to being configured as separate devices as shown in FIG. For example, the EVS 200 and the RGB sensor 600 are provided on two different substrates provided in one device (camera head), and the light dispersed by a prism (not shown) provided in the device is It may be configured to detect each. Alternatively, in this embodiment, for example, a plurality of pixels on one substrate may be configured to function as the EVS 200 and the RGB sensor 600 .

In addition, in the present embodiment, the medical observation system 10a may have an IR sensor (not shown) that detects infrared light, or a short wave infrared (SWIR) sensor such as an InGaAs sensor. It may have a sensor (not shown). For example, blood vessels located deep inside the body can be accurately captured by using short-wave infrared rays (light having a wavelength of about 900 nm to about 2500 nm).

Note that the medical observation system 10a according to the present embodiment is not limited to the configuration shown in FIG. 13, and may include, for example, various sensors as described above. Furthermore, in the present embodiment, the signal processing unit 500a is not limited to being composed of an integrated device as shown in FIG. 13, and may be composed of a plurality of devices.

<4.2 Configuration of Signal Processing Unit>
Next, an example of the functional configuration of the signal processing unit 500a according to this embodiment will be described with reference to FIG. FIG. 14 is a diagram showing an example of the functional configuration of the signal processing section 500a according to this embodiment. As shown in FIG. 14, the signal processing unit 500a includes signal acquisition units 502 and 532, a frame memory (storage unit) 504, an image processing unit 506, an image adjustment unit 508, an output unit 510, and a control unit 520. , an adjustment unit 522 , an input unit 524 , a motion detection unit (estimation unit) 534 , and a motion compensation unit (correction unit) 536 . Each functional unit included in the signal processing unit 500a according to the present embodiment will be described below, but the description of the functional units common to the first embodiment will be omitted.

(Image processing unit 506)
Unlike the first embodiment, the image processing unit 506 adds a signal related to the current frame output from the EVS 200 after the image data to the image data before the current frame that has undergone motion compensation in a motion compensation unit 536 (to be described later). (sensing data) are added to generate a new image of the current frame. The image processing unit 506 then outputs the newly generated image data of the current frame to the image adjusting unit 508, which will be described later.

(Signal acquisition unit 532)
The signal acquisition unit 532 can acquire pixel signals from the RGB sensor 600 and output them to the motion detection unit 534, which will be described later.

(Motion detector 534)
The motion detection unit 534 compares image data (visible light image) (second image) based on pixel signals from the RGB sensor 600 prior to the current frame and image data based on pixel signals of the current frame from the RGB sensor 600. By doing so, the relative motion between the RGB sensor 600 and the object 700 is estimated. Further, the motion detector 534 estimates the relative motion between the EVS 200 and the subject 700 based on the known positional relationship between the RGB sensor 600 and the EVS 200 . For example, the motion detection unit 534 calculates a motion vector between frames corresponding to each of the plurality of visible light image data for each predetermined unit data (for example, in units of pixels) among the plurality of visible light image data. Motion estimation methods include, for example, a block matching method and a gradient method. Further, the motion detection unit 534 may estimate the motion of the subject 700 between visible light image data based on each of a plurality of motion estimation methods, and combine the respective estimation results to improve motion estimation accuracy. The motion detection unit 534 then outputs the motion estimation result of the subject 700 in the image to the motion compensation unit 536, which will be described later.

(Motion Compensator 536)
The motion compensator 536 reads image data (fluorescence image) due to fluorescence from the frame before the current frame stored in the frame memory 504, and performs motion compensation based on the result of estimating the motion of the object 700 in the image output from the motion detector 534. , corrects the read image data by removing or shifting luminance changes due to motion (motion compensation). The motion compensation unit 536 then outputs the corrected image data to the image processing unit 506 .

Note that in the present embodiment, the signal processing unit 500a is not limited to being configured by the functional units as shown in FIG. 14, and may further include other functional units. In addition, in the present embodiment, the signal processing unit 500a is not limited to being composed of an integrated device as shown in FIG. 14, and may be composed of a plurality of separate devices.

<4.3 Observation method>
Next, an example of an observation method according to this embodiment will be described with reference to FIGS. 15 and 16. FIG. FIG. 15 is a flowchart of an observation method according to this embodiment, and FIG. 16 is an explanatory diagram for explaining an example of an observation image. Specifically, as shown in FIG. 15, the observation method according to this embodiment can mainly include a plurality of steps from step S301 to step S309. Details of each of these steps according to the present embodiment will be described below.

First, as in the first embodiment, the signal processing unit 500a converts the image data (fluorescence image) of the previous frame stored in the frame memory 504 (the image data stored in the frame memory 504 immediately before) into Read (step S301). Next, the signal processing unit 500a detects the movement of the subject 700 based on the pixel signals output from the RGB sensor 600 (step S302). Then, motion compensation is performed on the read image data based on the motion detected in step S302 (step S303).

Then, the signal processing unit 500a adds the signal output from the EVS 200 to the image data (fluorescence image) motion-compensated in step S303 (step S304).

After that, the signal processing unit 500a executes steps S305 to S309. to step S107, detailed description is omitted here.

As described above, in the present embodiment, not only the EVS 200 but also the RGB sensor 600 capable of capturing a visible light image are used together to detect the movement of the subject 700 or the like using the visible light image, Based on the detected motion, motion compensation can be performed to remove the change component of fluorescence due to motion. In addition, in this embodiment, since the relative movement of one between the EVS 200 and the subject 700 is estimated, it is possible to deal with not only the moving subject 700 but also the moving EVS 200 .

Further, in this embodiment, motion compensation is not limited to being performed based on the visible light image by the RGB sensor 600. As shown in FIG. 16, the visible light image (second image) and the fluorescence image (first image) may be added to generate image data. For example, in FIG. 16, the foreground portion (indicated by the white area with thick outline) is the fluorescence image and the background portion (indicated by the gray area with thin lines) is the visible light image. be.

<<5. Use case>
The above-described embodiments of the present disclosure can be used, for example, in anastomosis of blood vessels and large intestine, gastric tube preparation, esophageal reconstruction, cholecystectomy, etc., to evaluate tissues in vivo with blood flow, and in cerebral artery flow clipping and bypass surgery. , can be used to evaluate the state of blood-flowing tissues and blood flow turbulence. The embodiments of the present disclosure can also be used to identify target tissues in vivo, such as the liver, and to identify lymph nodes and lymph vessels other than blood flow.

For example, in the identification of lymph nodes, the sequential flow of the fluorescent agent causes the lymph nodes to light up in sequence, so the lymph node network can be identified by the order of light emission. Since it takes several minutes for the lymph nodes to light up sequentially, it is not necessary to detect fluorescence with high temporal resolution. Therefore, the threshold for comparison with the amount of change in luminance to detect the presence or absence of an event is set to a relatively low value. you don't have to

Also, the embodiment of the present disclosure can be applied to blood flow evaluation in cerebral artery bypass surgery, for example. Blood flow evaluation in cerebral artery bypass surgery will be described below with reference to FIGS. 17 and 18. FIG. FIG. 17 is a diagram showing an example of temporal changes in brightness of each site in cerebral artery bypass surgery, and FIG. 18 is a diagram showing an example of the result drawn based on FIG.

As described above, in blood flow evaluation in cerebral artery bypass surgery, it is required to evaluate the speed and direction of blood flow. changes. Therefore, by using the EVS200 with high temporal resolution, it is possible to capture rapid changes in fluorescence.

It should be noted that in the blood flow evaluation, as shown in FIG. 17, the elapsed time until each site emits fluorescence with maximum brightness is the blood flow evaluation index. Then, the elapsed time until the maximum brightness of each part shown in FIG. 17 can be imaged using, for example, hue and brightness. An example of such an image is shown in FIG.

<<6. Summary>>
As described above, according to each embodiment of the present disclosure, by using the EVS 200, minute changes in fluorescence can be captured without being affected by the absolute value of fluorescence luminance.

It should be noted that in the embodiment of the present disclosure described above, the subject 700 is not limited to living tissue, and may be a fine mechanical structure or the like, and is not particularly limited. In addition, the above-described embodiments of the present disclosure are not limited to application to applications such as medical care or research, and can be applied to observation devices that perform high-precision analysis using images. Therefore, the medical observation system 10 described above can be used as an observation system (observation device).

Furthermore, the medical observation system 10 described above can be used as a rigid endoscope, a flexible endoscope, an endoscope, a microscope, and the like.

<<7. Example of hardware configuration>>
Next, referring to FIG. 19, an example of a hardware configuration of an information processing device (for example, the signal processing unit 500 shown in FIG. 4, etc.) that executes various processes in the medical observation system 10 according to the present embodiment. , will be described in detail. FIG. 19 is a functional block diagram showing one configuration example of the hardware configuration of the information processing device that configures the medical observation system according to one embodiment of the present disclosure.

An information processing device 900 that configures the medical observation system according to the present embodiment mainly includes a CPU 901, a ROM 902, and a RAM 903. The information processing device 900 further includes a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, and a connection port 923. and a communication device 925 .

The CPU 901 functions as an arithmetic processing device and a control device, and controls all or part of the operations in the information processing device 900 according to various programs recorded in the ROM 902, RAM 903, storage device 919, or removable recording medium 927. The ROM 902 stores programs, calculation parameters, and the like used by the CPU 901 . The RAM 903 temporarily stores programs used by the CPU 901, parameters that change as appropriate during execution of the programs, and the like. These are interconnected by a host bus 907 comprising an internal bus such as a CPU bus. Note that each configuration of the signal processing unit 500 shown in FIG. 4 can be implemented by the CPU 901 .

The host bus 907 is connected via a bridge 909 to an external bus 911 such as a PCI (Peripheral Component Interconnect/Interface) bus. An input device 915 , an output device 917 , a storage device 919 , a drive 921 , a connection port 923 and a communication device 925 are connected to the external bus 911 via an interface 913 .

The input device 915 is operation means operated by the user, such as a mouse, keyboard, touch panel, button, switch, lever, and pedal. Further, the input device 915 may be, for example, remote control means (so-called remote controller) using infrared rays or other radio waves, or an external connection device such as a mobile phone or PDA corresponding to the operation of the information processing device 900. 929 may be used. Furthermore, the input device 915 is composed of, for example, an input control circuit that generates an input signal based on the information input by the user using the operation means and outputs the signal to the CPU 901 . A user of the information processing apparatus 900 can input various data to the information processing apparatus 900 and instruct processing operations by operating the input device 915 .

The output device 917 consists of a device capable of visually or audibly notifying the user of the acquired information. Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices and lamps, audio output devices such as speakers and headphones, and printer devices. The output device 917 outputs, for example, results obtained by various processes performed by the information processing device 900 . Specifically, the display device displays the results obtained by various processes performed by the information processing device 900 as text or images. On the other hand, the audio output device converts an audio signal including reproduced audio data, acoustic data, etc. into an analog signal and outputs the analog signal. Note that the output unit 510 shown in FIG. 4 can be implemented by the output device 917 .

The storage device 919 is a data storage device configured as an example of the storage unit of the information processing device 900 . The storage device 919 is composed of, for example, a magnetic storage device such as a HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like. The storage device 919 stores programs executed by the CPU 901 and various data. Note that the frame memory 504 shown in FIG. 4 can be implemented by either the storage device 919 or the RAM 903, or a combination thereof.

The drive 921 is a recording medium reader/writer, and is built in or externally attached to the information processing apparatus 900 . The drive 921 reads information recorded on a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the information to the RAM 903 . The drive 921 can also write records to a removable recording medium 927 such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. The removable recording medium 927 is, for example, DVD media, HD-DVD media, Blu-ray (registered trademark) media, or the like. The removable recording medium 927 may be a compact flash (registered trademark) (CF: CompactFlash (registered trademark)), a flash memory, an SD memory card (Secure Digital memory card), or the like. Also, the removable recording medium 927 may be, for example, an IC card (Integrated Circuit card) equipped with a contactless IC chip, an electronic device, or the like.

The connection port 923 is a port for direct connection to the information processing device 900 . Examples of the connection port 923 include a USB (Universal Serial Bus) port, IEEE1394 port, SCSI (Small Computer System Interface) port, and the like. Other examples of the connection port 923 include an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, and the like. By connecting the external connection device 929 to the connection port 923 , the information processing apparatus 900 acquires various data directly from the external connection device 929 and provides various data to the external connection device 929 .

The communication device 925 is, for example, a communication interface configured with a communication device or the like for connecting to a communication network (network) 931 . The communication device 925 is, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). Also, the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various types of communication, or the like. For example, the communication device 925 can transmit and receive signals and the like to and from the Internet and other communication devices according to a predetermined protocol such as TCP/IP. Also, the communication network 931 connected to the communication device 925 is configured by a network or the like connected by wire or wireless, and may be, for example, the Internet, home LAN, infrared communication, radio wave communication, satellite communication, or the like. .

An example of the hardware configuration capable of realizing the functions of the information processing device 900 constituting the signal processing unit 500 of the medical observation system 10 according to the embodiment of the present disclosure has been shown above. Each component described above may be configured using general-purpose members, or may be configured by hardware specialized for the function of each component. Therefore, it is possible to appropriately change the hardware configuration to be used according to the technical level at which the present embodiment is implemented. Although not shown in FIG. 19, various configurations corresponding to the information processing device 900 forming the signal processing unit 500 of the medical observation system 10 are of course provided.

It should be noted that it is possible to create a computer program for realizing each function of the information processing apparatus 900 that constitutes the medical observation system 10 according to the present embodiment as described above, and implement it in a personal computer or the like. A computer-readable recording medium storing such a computer program can also be provided. Recording media include, for example, magnetic disks, optical disks, magneto-optical disks, and flash memories. Also, the above computer program may be distributed, for example, via a network without using a recording medium. Also, the number of computers that execute the computer program is not particularly limited. For example, the computer program may be executed by a plurality of computers (for example, a plurality of servers, etc.) in cooperation with each other.

<<8. Application example >>
The technology according to the present disclosure can be applied to medical imaging systems. A medical imaging system is a medical system using imaging technology, such as an endoscope system or a microscope system.

[Endoscope system]
An example of an endoscope system will be described with reference to FIGS. 20 and 21. FIG. FIG. 20 is a diagram showing an example of a schematic configuration of an endoscope system 5000 to which technology according to the present disclosure can be applied. FIG. 21 is a diagram showing an example of the configuration of an endoscope 5001 and a CCU (Camera Control Unit) 5039. As shown in FIG. FIG. 20 illustrates a state in which an operator (for example, a doctor) 5067 who is a surgical participant is performing surgery on a patient 5071 on a patient bed 5069 using an endoscope system 5000 . As shown in FIG. 20, the endoscope system 5000 supports an endoscope 5001 as a medical imaging device, a CCU 5039, a light source device 5043, a recording device 5053, an output device 5055, and an endoscope 5001. and a support device 5027 .

In endoscopic surgery, an insertion aid called a trocar 5025 is punctured into a patient 5071. Then, the scope 5003 and surgical instrument 5021 connected to the endoscope 5001 are inserted into the body of the patient 5071 via the trocar 5025 . The surgical instrument 5021 is, for example, an energy device such as an electric scalpel, forceps, or the like.

A surgical image, which is a medical image of the inside of the patient's 5071 photographed by the endoscope 5001, is displayed on the display device 5041. The operator 5067 uses the surgical instrument 5021 to treat the surgical target while viewing the surgical image displayed on the display device 5041 . Note that the medical images are not limited to surgical images, and may be diagnostic images captured during diagnosis.

[Endoscope]
The endoscope 5001 is an imaging unit for imaging the inside of the body of a patient 5071. For example, as shown in FIG. A camera 5005 includes a zoom optical system 50052 that enables optical zoom, a focus optical system 50053 that enables focus adjustment by changing the focal length of an imaging unit, and a light receiving element 50054 . The endoscope 5001 converges light on the light receiving element 50054 through the connected scope 5003 to generate pixel signals, and outputs the pixel signals to the CCU 5039 through the transmission system. Note that the scope 5003 is an insertion portion that has an objective lens at its tip and guides light from the connected light source device 5043 into the body of the patient 5071 . The scope 5003 is, for example, a rigid scope for rigid scopes and a flexible scope for flexible scopes. The scope 5003 may be a direct scope or a perspective scope. Also, the pixel signal may be a signal based on a signal output from a pixel, such as a RAW signal or an image signal. Alternatively, a memory may be installed in the transmission system connecting the endoscope 5001 and the CCU 5039, and the parameters relating to the endoscope 5001 and the CCU 5039 may be stored in the memory. The memory may be arranged, for example, on the connection part of the transmission system or on the cable. For example, the parameters of the endoscope 5001 at the time of shipment and the parameters changed when the power is supplied may be stored in the memory of the transmission system, and the operation of the endoscope may be changed based on the parameters read from the memory. Also, an endoscope and a transmission system may be collectively referred to as an endoscope. The light receiving element 50054 is a sensor that converts received light into pixel signals, and is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type imaging element. The light-receiving element 50054 is preferably an imaging element having a Bayer array and capable of color imaging. The light receiving element 50054 is, for example, 4K (horizontal pixel number 3840×vertical pixel number 2160), 8K (horizontal pixel number 7680×vertical pixel number 4320) or square 4K (horizontal pixel number 3840 or more×vertical pixel number 3840 or more). It is preferable that the image sensor has a number of pixels corresponding to the resolution. The light receiving element 50054 may be a single sensor chip or a plurality of sensor chips. For example, a prism may be provided to separate the incident light into predetermined wavelength bands, and each wavelength band may be imaged by a different light-receiving element. Also, a plurality of light receiving elements may be provided for stereoscopic viewing. Further, the light receiving element 50054 may be a sensor including an arithmetic processing circuit for image processing in a chip structure, or may be a ToF (Time of Flight) sensor. The transmission system is, for example, an optical fiber cable or wireless transmission. The wireless transmission is sufficient as long as the pixel signals generated by the endoscope 5001 can be transmitted. Mirror 5001 and CCU 5039 may be connected. At this time, the endoscope 5001 may transmit not only the pixel signal but also information related to the pixel signal (for example, processing priority of the pixel signal, synchronization signal, etc.) at the same time. The endoscope may be configured by integrating a scope and a camera, or by providing a light-receiving element at the tip of the scope.

[CCU (Camera Control Unit)]
The CCU 5039 is a control device that comprehensively controls the connected endoscope 5001 and light source device 5043. For example, as shown in FIG. processing equipment. Also, the CCU 5039 may centrally control the connected display device 5041 , recording device 5053 and output device 5055 . For example, the CCU 5039 controls the irradiation timing and irradiation intensity of the light source device 5043 and the type of irradiation light source. In addition, the CCU 5039 performs image processing such as development processing (for example, demosaicing processing) and correction processing on the pixel signals output from the endoscope 5001, and outputs the processed pixel signals (for example, image processing) to an external device such as the display device 5041. ). Also, the CCU 5039 transmits a control signal to the endoscope 5001 to control driving of the endoscope 5001 . The control signal is, for example, information about imaging conditions such as magnification and focal length of the imaging unit. The CCU 5039 may have an image down-conversion function, and may be configured to output a high-resolution (eg, 4K) image to the display device 5041 and a low-resolution (eg, HD) image to the recording device 5053 at the same time.

In addition, the CCU 5039 is connected to external devices (eg, recording device, display device, output device, support device) via an IP converter that converts signals into a predetermined communication protocol (eg, IP (Internet Protocol)). good too. The connection between the IP converter and the external device may be configured by a wired network, or part or all of the network may be configured by a wireless network. For example, the IP converter on the CCU5039 side has a wireless communication function, and the received video is sent to an IP switcher or output via a wireless communication network such as the 5th generation mobile communication system (5G) or the 6th generation mobile communication system (6G). It may be sent to the side IP converter.

[Light source device]
The light source device 5043 is a device capable of emitting light in a predetermined wavelength band, and includes, for example, a plurality of light sources and a light source optical system that guides light from the plurality of light sources. The light source is, for example, a xenon lamp, an LED light source, or an LD light source. The light source device 5043 has, for example, LED light sources corresponding to the three primary colors R, G, and B, and emits white light by controlling the output intensity and output timing of each light source. Further, the light source device 5043 may have a light source capable of irradiating special light used for special light observation separately from the light source for irradiating normal light used for normal light observation. Special light is light in a predetermined wavelength band different from normal light that is light for normal light observation. is. Normal light is, for example, white light or green light. In narrow-band light observation, which is a type of special light observation, by alternately irradiating blue light and green light, the wavelength dependence of light absorption in body tissues can be used to detect specific tissues such as blood vessels on the surface of the mucous membrane. can be shot with high contrast. In addition, in fluorescence observation, which is a type of special light observation, excitation light that excites the drug injected into the body tissue is irradiated, and fluorescence emitted by the body tissue or the drug as a marker is received to obtain a fluorescence image. It is possible to make it easier for the operator to visually recognize body tissues and the like that are difficult for the operator to visually recognize with normal light. For example, in fluorescence observation using infrared light, a drug such as indocyanine green (ICG) injected into the body tissue is irradiated with infrared light having an excitation wavelength band, and the fluorescence of the drug is received to detect the body tissue. structure and the affected area can be easily visualized. Further, in fluorescence observation, an agent (for example, 5-ALA) that emits fluorescence in the red wavelength band when excited by special light in the blue wavelength band may be used. It should be noted that the light source device 5043 sets the type of irradiation light under the control of the CCU 5039 . The CCU 5039 may have a mode in which normal light observation and special light observation are alternately performed by controlling the light source device 5043 and the endoscope 5001 . At this time, information based on pixel signals obtained by special light observation is preferably superimposed on pixel signals obtained by normal light observation. Further, the special light observation may be infrared light observation in which infrared light is irradiated to look deeper than the surface of the organ, or multispectral observation utilizing hyperspectral spectroscopy. Additionally, photodynamic therapy may be combined.

[Recording device]
A recording device 5053 is a device for recording pixel signals (for example, an image) obtained from the CCU 5039, and is, for example, a recorder. A recording device 5053 records the image acquired from the CCU 5039 on an HDD, an SDD, or an optical disk. The recording device 5053 may be connected to a hospital network and accessible from equipment outside the operating room. Also, the recording device 5053 may have an image down-conversion function or an image up-conversion function.

[Display device]
The display device 5041 is a device capable of displaying an image, such as a display monitor. A display device 5041 displays a display image based on pixel signals obtained from the CCU 5039 . Note that the display device 5041 may function as an input device that enables line-of-sight recognition, voice recognition, and gesture-based instruction input by being equipped with a camera and a microphone.

[Output device]
The output device 5055 is a device for outputting information acquired from the CCU 5039, such as a printer. The output device 5055 prints on paper a print image based on the pixel signals acquired from the CCU 5039, for example.

[Support device]
The support device 5027 is an articulated arm including a base portion 5029 having an arm control device 5045 , an arm portion 5031 extending from the base portion 5029 , and a holding portion 5032 attached to the tip of the arm portion 5031 . The arm control device 5045 is configured by a processor such as a CPU, and operates according to a predetermined program to control driving of the arm section 5031 . The support device 5027 controls parameters such as the length of each link 5035 constituting the arm portion 5031 and the rotation angle and torque of each joint 5033 by means of the arm control device 5045 . control the position and posture of Thereby, the endoscope 5001 can be changed to a desired position or posture, the scope 5003 can be inserted into the patient 5071, and the observation area inside the body can be changed. The support device 5027 functions as an endoscope support arm that supports the endoscope 5001 during surgery. Thereby, the support device 5027 can take the place of the scopist who is an assistant holding the endoscope 5001 . Also, the support device 5027 may be a device that supports a microscope device 5301, which will be described later, and can also be called a medical support arm. Note that the control of the support device 5027 may be an autonomous control method by the arm control device 5045, or may be a control method in which the arm control device 5045 controls based on the user's input. For example, the control method is a master/slave method in which the support device 5027 as a slave device (replica device), which is a patient cart, is controlled based on the movement of the master device (primary device), which is the operator console at hand of the user. It's okay. Also, the control of the support device 5027 may be remotely controlled from outside the operating room.

An example of the endoscope system 5000 to which the technology according to the present disclosure can be applied has been described above. For example, the technology according to the present disclosure may be applied to microscope systems.

[Microscope system]
FIG. 22 is a diagram illustrating an example of a schematic configuration of a microsurgery system to which technology according to the present disclosure can be applied; In the following description, the same reference numerals are given to the same configurations as those of the endoscope system 5000, and duplicate descriptions thereof will be omitted.

FIG. 22 schematically shows an operator 5067 performing an operation on a patient 5071 on a patient bed 5069 using a microsurgery system 5300 . For simplicity, FIG. 22 omits illustration of the cart 5037 in the configuration of the microsurgery system 5300, and also shows a simplified microscope device 5301 instead of the endoscope 5001. As shown in FIG. However, the microscope device 5301 in this description may refer to the microscope section 5303 provided at the tip of the link 5035 or may refer to the entire configuration including the microscope section 5303 and the support device 5027 .

As shown in FIG. 22, at the time of surgery, using a microsurgery system 5300, an image of a surgical site captured by a microscope device 5301 is enlarged and displayed on a display device 5041 installed in the operating room. The display device 5041 is installed at a position facing the operator 5067, and the operator 5067 observes the state of the operation site by the image displayed on the display device 5041, for example, resection of the affected area. Various measures are taken against Microsurgery systems are used, for example, in ophthalmic and brain surgery.

Examples of the endoscope system 5000 and the microsurgery system 5300 to which the technology according to the present disclosure can be applied have been described above. Note that the system to which the technology according to the present disclosure can be applied is not limited to this example. For example, the support device 5027 can support other observation devices or other surgical tools instead of the endoscope 5001 or the microscope section 5303 at its distal end. As the other observation device, for example, forceps, forceps, a pneumoperitoneum tube for pneumoperitoneum, or an energy treatment instrument for incising tissue or sealing a blood vessel by cauterization can be applied. By supporting these observation devices and surgical instruments with a support device, it is possible to fix the position more stably and reduce the burden on the medical staff, compared to the case where the medical staff supports them manually. becomes possible. The technology according to the present disclosure may be applied to a support device that supports components other than such a microscope section.

The technology according to the present disclosure can be suitably applied to the CCU 5039, the endoscope 5001, or the microscope unit 5303 among the configurations described above. Specifically, by applying the above-described EVS 200 to the CCU 5039, the endoscope 5001, or the microscope unit 5303, minute changes in observation light can be captured without being affected by the absolute value of the luminance of the observation light. can be done.

<<9. Supplement >>
Note that the above-described embodiment of the present disclosure includes, for example, an observation method executed by the medical observation system 10 as described above, a program for operating the medical observation system 10, and a program in which the program is recorded. may include non-transitory tangible media. Also, the program may be distributed via a communication line (including wireless communication) such as the Internet.

Also, each step in the processing method of the embodiment of the present disclosure described above does not necessarily have to be processed in the described order. For example, each step may be processed in an appropriately changed order. Also, each step may be partially processed in parallel or individually instead of being processed in chronological order. Furthermore, the processing of each step does not necessarily have to be processed in accordance with the described method, and may be processed by another method by another functional unit, for example.

Of the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all of the processes described as being performed manually Alternatively, some can be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. are naturally within the technical scope of the present disclosure.

Also, the effects described in this specification are merely descriptive or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

Note that the present technology can also take the following configuration.
(1)
an event vision sensor that detects, as an event, a change in luminance value of light in the first wavelength band emitted from living tissue;
an image processing unit configured to generate a first image based on a plurality of pieces of sensing data output from the event vision sensor at different timings;
A biological tissue observation system.
(2)
The event vision sensor according to (1) above, wherein the event vision sensor detects light from a drug containing a substance capable of emitting light in the first wavelength band, which has been administered to the living tissue once or multiple times. biological tissue observation system.
(3)
further comprising a first lighting device that irradiates the biological tissue with excitation light that excites the substance;
The excitation light is light in a second wavelength band different from the first wavelength band,
The biological tissue observation system according to (2) above.
(4)
The biological tissue observation system according to (3) above, wherein the light in the first wavelength band is light in the near-infrared band.
(5)
The biological tissue observation system according to (3) or (4) above, wherein the event vision sensor has a filter that does not transmit the excitation light.
(6)
further comprising a storage unit that stores the sensing data,
The image processing unit adds the sensing data output after the output of the sensing data to the stored sensing data.
A biological tissue observation system according to any one of (3) to (5) above.
(7)
The image processing unit sequentially adds the plurality of sensing data output after the output of the sensing data to the stored sensing data.
The biological tissue observation system according to (6) above.
(8)
The image processing unit according to (6) or (7) above, wherein, based on the additional administration of the drug, the plurality of sensing data output after the additional administration of the drug is added. A biological tissue observation system.
(9)
The event vision sensor,
a pixel array section having a plurality of pixels arranged in a matrix;
an event detection unit that detects that a luminance change amount exceeds a predetermined threshold in each pixel;
having
A biological tissue observation system according to any one of (3) to (8) above.
(10)
The biological tissue observation system according to (9) above, further comprising an adjusting unit that adjusts the predetermined threshold.
(11)
The biological tissue observation system according to (10) above, wherein the adjustment unit adjusts the predetermined threshold based on the frequency of output of the sensing data.
(12)
The biological tissue observation system according to (10) above, wherein the adjustment unit adjusts the predetermined threshold value based on a dynamic range or gradation of luminance of the first image.
(13)
The biological tissue observation system according to (10) above, wherein the adjustment unit adjusts the predetermined threshold based on the number of administrations of the drug.
(14)
The biological tissue observation system according to (13) above, wherein the adjustment unit reduces the predetermined threshold value as the number of administrations of the drug increases.
(15)
The living tissue observation system according to (10) above, wherein the adjustment unit adjusts the predetermined threshold value based on the flow rate of the drug in the living tissue or the rate of change in the amount of light emitted from the substance.
(16)
(10) above, wherein the adjusting unit adjusts the predetermined threshold value based on at least one of the type, size, position, state, displacement speed, deformation speed, and surrounding state of the living tissue; The living tissue observation system described.
(17)
The biological tissue observation system according to (10) above, wherein the adjustment unit adjusts the predetermined threshold based on the type or concentration of the substance contained in the medicine.
(18)
The biological tissue observation system according to (10) above, wherein the adjustment unit adjusts the predetermined threshold based on the irradiation intensity of the excitation light.
(19)
The biological tissue observation system according to (18) above, wherein the adjustment unit reduces the predetermined threshold value as the irradiation intensity decreases.
(20)
Further comprising an image sensor that detects light in a third wavelength band different from the first wavelength band, reflected from the biological tissue.
A biological tissue observation system according to any one of (1) to (19) above.
(21)
The biological tissue observation system according to (20) above, wherein the event vision sensor and the image sensor are provided on different substrates.
(22)
The biological tissue observation system according to (20) above, wherein the event vision sensor and the image sensor are provided on the same substrate.
(23)
The living tissue observation system according to any one of (20) to (22) above, further comprising a second illumination device that irradiates the living tissue with light of the third wavelength band.
(24)
The biological tissue observation system according to any one of (20) to (23) above, wherein the light in the third wavelength band is visible light.
(25)
The biological tissue observation system according to any one of (20) to (24) above, wherein the image processing unit adds the first image to the second image from the image sensor.
(26)
an estimator for estimating relative motion between the event vision sensor and the living tissue based on a plurality of second images from the image sensor;
a correction unit that corrects deviations between the plurality of sensing data based on the estimation result;
further comprising
A biological tissue observation system according to any one of (20) to (24) above.
(27)
The living tissue observation system according to any one of (1) to (26) above, wherein the living tissue is an in vivo tubular tissue.
(28)
The biological tissue observation system according to any one of (1) to (27) above, which is any one of an endoscope, exoscopy, and microscope.
(29)
an event vision sensor that detects, as an event, a change in luminance value of light in the first wavelength band emitted from living tissue;
an image processing unit configured to generate a first image based on a plurality of pieces of sensing data output from the event vision sensor at different timings;
A biological tissue observation device comprising:
(30)
the event vision sensor detecting, as an event, a change in luminance value of light in the first wavelength band emitted from the living tissue;
an image processing unit generating a first image based on a plurality of sensing data output from the event vision sensor at different timings;
A biological tissue observation method, comprising:

10 medical observation system 200 EVS
211 drive circuit 212 signal processing unit 213 arbiter unit 214 column processing unit 250 cut filter 300 pixel array unit 302 pixel 304 light receiving unit 306 pixel signal generation unit 308 detection unit 400, 400a light source 500, 500a signal processing unit 502, 532 signal acquisition unit 504 frame memory 506 image processing unit 508 image adjustment unit 510 output unit 520 control unit 522 adjustment unit 524 input unit 534 motion detection unit 536 motion compensation unit 600 RGB sensor 700 subject

Claims (30)

an event vision sensor that detects, as an event, a change in luminance value of light in the first wavelength band emitted from living tissue;
an image processing unit configured to generate a first image based on a plurality of pieces of sensing data output from the event vision sensor at different timings;
A biological tissue observation system. 2. The event vision sensor of claim 1, wherein the event vision sensor detects light from a drug that has been administered to the biological tissue one or more times and includes a substance capable of emitting light in the first wavelength band. A biological tissue observation system. further comprising a first lighting device that irradiates the biological tissue with excitation light that excites the substance;
The excitation light is light in a second wavelength band different from the first wavelength band,
The biological tissue observation system according to claim 2. The biological tissue observation system according to claim 3, wherein the light in the first wavelength band is light in the near-infrared band. The living tissue observation system according to claim 3, wherein the event vision sensor has a filter that does not transmit the excitation light. further comprising a storage unit that stores the sensing data,
The image processing unit adds the sensing data output after the output of the sensing data to the stored sensing data.
The biological tissue observation system according to claim 3. The image processing unit sequentially adds the plurality of sensing data output after the output of the sensing data to the stored sensing data.
The biological tissue observation system according to claim 6. The biological tissue observation system according to claim 6, wherein the image processing unit adds the plurality of sensing data output after the additional administration of the drug based on the additional administration of the drug. The event vision sensor,
a pixel array section having a plurality of pixels arranged in a matrix;
an event detection unit that detects that a luminance change amount exceeds a predetermined threshold in each pixel;
having
The biological tissue observation system according to claim 3. The biological tissue observation system according to claim 9, further comprising an adjustment unit that adjusts the predetermined threshold. The biological tissue observation system according to claim 10, wherein the adjustment unit adjusts the predetermined threshold based on the frequency of output of the sensing data. 11. The biological tissue observation system according to claim 10, wherein the adjustment unit adjusts the predetermined threshold value based on a dynamic range of luminance or a gradation of the first image. The biological tissue observation system according to claim 10, wherein the adjustment unit adjusts the predetermined threshold based on the number of administrations of the drug. The biological tissue observation system according to claim 13, wherein the adjustment unit reduces the predetermined threshold value as the number of administrations of the drug increases. 11. The biological tissue observation system according to claim 10, wherein the adjustment unit adjusts the predetermined threshold value based on the flow velocity of the drug in the biological tissue or the rate of change in the amount of light emitted from the substance. 11. The adjusting unit according to claim 10, wherein the adjustment unit adjusts the predetermined threshold value based on at least one of the type, size, position, state, displacement speed, deformation speed, and surrounding state of the living tissue. biological tissue observation system. The biological tissue observation system according to claim 10, wherein the adjustment unit adjusts the predetermined threshold based on the type or concentration of the substance contained in the drug. The living tissue observation system according to claim 10, wherein the adjustment unit adjusts the predetermined threshold based on the irradiation intensity of the excitation light. The biological tissue observation system according to claim 18, wherein the adjustment unit reduces the predetermined threshold as the irradiation intensity decreases. Further comprising an image sensor that detects light in a third wavelength band different from the first wavelength band, reflected from the biological tissue.
The biological tissue observation system according to claim 1. The biological tissue observation system according to claim 20, wherein said event vision sensor and said image sensor are provided on different substrates. The living tissue observation system according to claim 20, wherein the event vision sensor and the image sensor are provided on the same substrate. The biological tissue observation system according to claim 20, further comprising a second illumination device that irradiates the biological tissue with the light of the third wavelength band. The living tissue observation system according to claim 20, wherein the light in the third wavelength band is visible light. The biological tissue observation system according to claim 20, wherein said image processing unit adds said first image to said second image from said image sensor. an estimator for estimating relative motion between the event vision sensor and the living tissue based on a plurality of second images from the image sensor;
a correction unit that corrects deviations between the plurality of sensing data based on the estimation result;
further comprising
The biological tissue observation system according to claim 20. The biological tissue observation system according to claim 1, wherein the biological tissue is an in vivo tubular tissue. The biological tissue observation system according to claim 1, which is any one of an endoscope, exoscopy, and microscope. an event vision sensor that detects, as an event, a change in luminance value of light in the first wavelength band emitted from living tissue;
an image processing unit configured to generate a first image based on a plurality of pieces of sensing data output from the event vision sensor at different timings;
A biological tissue observation device comprising: the event vision sensor detecting, as an event, a change in luminance value of light in the first wavelength band emitted from the living tissue;
an image processing unit generating a first image based on a plurality of sensing data output from the event vision sensor at different timings;
A biological tissue observation method, comprising:

Images