WO2025141691A1 - Object detection device, object detection method, object detection program, and endoscope system - Google Patents

Abstract

In order to provide an object detection device (10) that detects a specific object image from an image acquired by an endoscope 20 wherein, even if a detected specific object is lost from view, the same specific object can be easily detected, the present invention is provided with: a specific object image detection unit 14 that detects an image area including a specific object from an endoscope image based on first image data acquired by the endoscope 20; a storage control unit 13 that temporarily stores second image data including the detected specific object in a temporary storage unit 51; and a display control unit 12 that displays a reference image based on the second image data on a display device 30 when the disappearance of the specific object from the endoscope image is detected.

Description

Object detection device, object detection method, object detection program, and endoscope system

This invention relates to an object detection device, an object detection method, an object detection program, and an endoscope system that detect an image area of a specific object from an image acquired by an endoscope.

Conventionally, endoscope systems consisting of an endoscope that captures images of the inside of a subject and acquires image data, a processor that performs various image processing operations on the image data acquired by the endoscope, a display device that displays the image data processed by the processor as a visible image, and a storage device that records or stores the image data, have been widely used in, for example, the medical and industrial fields.

In the medical field, this type of endoscope system is widely used to perform various types of examinations to observe the inside of living organs, etc. In an examination using a medical endoscope system (hereinafter referred to as an endoscopic examination), for example, while the insertion part of the endoscope inserted inside the organ is being pulled out, images are captured by an imaging device provided at the tip of the insertion part. At this time, the endoscopic images acquired by the imaging device are displayed in real time, sequentially as moving images, using a display device. At the same time, the moving image data can also be stored in a storage device.

During such an endoscopic examination, the doctor performs so-called screening, searching for lesions such as polyps or tumors occurring on the inner walls of organs as specific objects (hereafter referred to as specific objects) while observing the real-time endoscopic image displayed on the display device (or, after the examination, reconstructing the endoscopic image based on recorded or stored endoscopic image data).

　Normally, specific objects such as lesions are very small and have a similar color to the inner walls of surrounding organs, so it takes skill to accurately distinguish and detect the image area of the specific object from the endoscopic image. In addition, in endoscopic examinations, the operation of the endoscope itself, such as pointing the tip of the endoscope's insertion part (the observation window of the imaging device) at the desired object to capture an image, requires skill.

For example, Japanese Patent Publication No. 2004-207842 discloses an image processing device in which a target image for identifying the object to be saved is registered in advance, and a series of image data being captured is analyzed in sequence based on the registered target image data. When an image containing a registered target is detected from the series of captured images, that captured image is extracted as the object to be saved and saved (recorded or stored).

On the other hand, in recent years, image processing technology for image diagnostic systems, known as computer-aided diagnosis (CAD), is being put into practical use. These systems use computers to process medical image data obtained by endoscopes and other instruments, and assist doctors in making diagnoses based on the analysis results and processed images obtained.

In the image processing technology presented in this type of conventional computer-aided diagnosis (CAD), for example, when a specific object is detected on the display screen of an endoscopic image, various object detection devices are being put into practical use that display the fact that a specific object has been detected by marking the image area that contains the specific object with a frame or other marking.

Generally, during endoscopic examinations in which the inside of a living organ is observed, the condition inside the organ, including the lesion (specific object), is constantly changing due to various disturbances, such as the patient's heartbeat and breathing, changes in body position, peristalsis of the organ, bleeding within the organ, food residue, etc.

Therefore, for example, when screening the inside of an organ during an endoscopic examination, even if a doctor detects a specific object, it is common for the doctor to lose sight of the specific object due to changes in the state of the inside of the organ caused by peristalsis, etc.

However, in endoscopic examinations, the doctor's main objective is not only to detect a specific object, but also to make a thorough observation of that specific object. Therefore, even if a specific object that has been detected is lost, there is always a desire to immediately detect the same specific object again.

However, even if the technology disclosed in the above-mentioned Japanese Patent Publication No. 2004-207842 is used to detect a specific object from an endoscopic image, if an image containing a registered target is detected, the captured image is only saved. Therefore, the technology disclosed in the publication etc. does not take into consideration, for example, re-detecting a specific object if it is lost after detection.

Furthermore, in conventional object detection devices using computer-aided diagnosis (CAD) technology, the marking display is generally cancelled if the specific object disappears from the image being displayed. Normally, a specific object that has been detected can be lost due to a change in the state inside the organ, such as when the specific object becomes separated from the tip of the endoscope (observation window) or when foreign matter gets between the tip of the endoscope (observation window) and the specific object. In such cases, it is very difficult to detect the same specific object again.

　Therefore, in conventional object detection devices, if a specific object that has been detected in an endoscopic image is lost, it takes time to detect the same specific object again, resulting in problems such as the endoscopic examination time being extended or the object being forgotten or overlooked. For these reasons, conventional endoscopic examinations have had the problem of placing a heavy burden on doctors, medical professionals, examiners, etc., and subjects (patients, etc.).

The object of the present invention is to provide an object detection device, an object detection method, an object detection program, and an endoscope system that can easily detect the same specific object again even if the detected specific object is lost in an object detection device that detects the image area of a specific object from an image acquired by medical equipment such as an endoscope or inspection equipment (such as an image inspection device).

In order to achieve the above object, an object detection device according to one embodiment of the present invention includes a specific object image detection unit that detects an image area including a specific object from an endoscopic image based on first image data acquired by an endoscope, a storage control unit that temporarily stores second image data including the detected specific object in a temporary storage unit, and a display control unit that displays a reference image based on the second image data on a display device when disappearance of the specific object is detected from the endoscopic image.

The object detection method of one aspect of the present invention includes a screening step of continuously displaying endoscopic images acquired continuously by an endoscope in a chronological order, a detection step of detecting a specific object from the endoscopic image, a temporary storage step of temporarily storing image data corresponding to an image area including the detected specific object, a reference image display step of displaying the image area including the specific object as a reference image when the disappearance of the specific object from the endoscopic image is detected, and a focus control step of terminating the display of the reference image and performing automatic focus adjustment control on the specific object when the specific object corresponding to the reference image and identical to the specific object that disappeared from the endoscopic image is detected again in the endoscopic image.

The object detection program of one embodiment of the present invention causes a computer to execute the following processes: continuously displaying endoscopic images; detecting a specific object from the endoscopic images; temporarily storing image data corresponding to an image area including the detected specific object; displaying the image area including the specific object as a reference image when the disappearance of the specific object from the endoscopic image is detected; and terminating the display of the reference image and performing automatic focus adjustment on the specific object when the specific object corresponding to the reference image and identical to the specific object that disappeared from the endoscopic image is detected again in the endoscopic image.

One embodiment of the endoscopic system of the present invention is an endoscopic system comprising an endoscope including an imaging unit having an imaging optical system that forms an optical image of an object and an imaging element that photoelectrically converts the optical image formed by the imaging optical system, and a processor, wherein the processor comprises a specific object image detection unit that detects an image area including a specific object from an endoscopic image based on first image data acquired by the imaging unit, a storage control unit that temporarily stores second image data including the detected specific object in a temporary storage unit, a display control unit that, when disappearance of the specific object is detected from the endoscopic image, displays a reference image based on the second image data in a second display area within the display screen of the display device that is different from the first display area in which the endoscopic image is displayed, and a focus control unit that drives and controls the imaging optical system, and when the object image detection unit detects that the specific object identical to the specific object that disappeared from the endoscopic image is detected again in the endoscopic image and that the same specific object continues to be displayed in the endoscopic image, the focus control unit drives and controls the imaging optical system to perform automatic focus adjustment control for the specific object.

An object detection method according to one embodiment of the present invention detects an image area including a specific object based on image data acquired by an image inspection device, temporarily stores image data of the specific object image including the detected specific object in a temporary storage unit, and, if the disappearance of the specific object is detected from an image based on the image data, displays a reference image based on the specific object image data on a display unit that displays an image based on the acquired image data.

The present invention provides an object detection device, object detection method, object detection program, and endoscope system that can easily detect the same specific object again even if the detected specific object is lost in an object detection device that detects the image area of a specific object from an image acquired by a medical device such as an endoscope or an inspection device (such as an image inspection device).

FIG. 1 is a block diagram showing an overall configuration of an endoscope system including an object detection device according to an embodiment of the present invention; FIG. 1 is a block diagram showing an outline of an internal configuration of an endoscope system including an object detection device according to an embodiment of the present invention; 1 is a flowchart illustrating an operation of an endoscope system including an object detection device according to an embodiment of the present invention; 2 is a display example of a display screen of a display device in the endoscope system of FIG. 1 ; 2 is a display example of a display screen of a display device at a time when a specific object is detected during an endoscopic examination using the endoscopic system of FIG. 1; 1. An example of a display on the display screen of the display device at a time when a detected specific object disappears during an endoscopic examination using the endoscopic system of FIG. 1 and a second image (reference image) is displayed. 2 is a display example of a display screen of a display device at a time when the same specific object is detected again during an endoscopic examination using the endoscopic system of FIG. 1; 8 is a display example of the display screen of the display device when the imaging unit is brought close to a specific object in the state of FIG. 7; 9 is a display example of the display screen of the display device when the imaging unit is brought closer to the specific object from the state of FIG. 8 . FIG. 2 is an explanatory diagram of an inference model obtained as a result of learning in the learning device; 10 is a flowchart showing a modified example of the operation of an endoscope system including an object detection device according to an embodiment of the present invention.

The present invention will be described below with reference to the illustrated embodiment, taking an endoscope system as an example. The drawings used in the following description are schematic. Therefore, in these drawings, each component is shown at a size that allows it to be recognized on the drawing. For this reason, the dimensional relationships and scale of each member on the drawing may be shown differently for each component. The present invention is not limited to the illustrated form with regard to the quantities, shapes, size ratios, relative positional relationships, etc. of each component shown in each drawing.

First, before describing the detailed configuration of an object detection device according to one embodiment of the present invention, the schematic configuration of an entire endoscope system including the object detection device according to this embodiment will be described below with reference to FIG. 1. FIG. 1 is a schematic diagram showing the overall configuration of an endoscope system including an object detection device according to one embodiment of the present invention.

As shown in FIG. 1, the endoscope system 1 is mainly configured with a processor 10, an endoscope 20, a display device 30, a light source device 40, a storage device 50, etc. This endoscope system 1 illustrates the general configuration of an endoscope system used in endoscopic examinations in which the inside of organs such as the upper digestive tract (esophagus, stomach, duodenum, etc.) or the lower digestive tract (large intestine, etc.) of a subject (patient, etc.) such as a living body are observed.

The endoscope 20 is an image inspection device that is composed of an insertion section 21, an operating section 22, a universal cable 23, etc.

The insertion section 21 is a component that is inserted into a subject such as a living organism. The insertion section 21 is formed by continuously connecting, in order from the tip side, a tip section 21a, a curved section 21b, and a flexible tube section 21c. The insertion section 21 is formed in a generally elongated tube shape. The insertion section 21 has an internal treatment tool insertion channel 21d, which is a duct for inserting an endoscopic treatment tool (not shown). This treatment tool insertion channel 21d is provided by passing through the insertion section 21 from its tip to its base end. The operation section 22 is connected to the base end side of the insertion section 21.

The tip portion 21a is a structural unit provided at the very tip of the insertion portion 21. Inside the tip portion 21a, various structural members such as an illumination unit 25 and an imaging unit 26 (not shown in FIG. 1; see FIG. 2) are provided.

Here, the illumination unit 25 is a structural unit that includes an optical element (illumination lens; not shown) that emits a light beam guided from the light source device 40 (described later) forward from the tip surface of the tip portion 21a to illuminate the observation area, including the lesion area, within the subject.

The imaging unit 26 is an electronic device unit that includes an optical lens (imaging optical system 26a; see FIG. 2) that forms an optical image of a specific object (such as a lesion, such as a tumor) to be observed inside the subject, and a photoelectric conversion element (imaging element 26b; see FIG. 2) that generates image information (such as still image data and moving image data) based on the optical image.

The bending portion 21b is a tubular portion that can be actively bent by a bending operation mechanism (not shown) that acts in conjunction with the operation of the bending operation member 22b described below. The flexible tube portion 21c is a tubular member that extends from the tip of the operation portion 22 and is connected to the base end of the bending portion 21b.

The operating section 22 is connected to the base end of the insertion section 21. The operating section 22 is configured with an operating section main body 22a, a bending operating member 22b, multiple operating members 22c, a treatment tool insertion port 22d, etc.

The operating unit body 22a has a generally box-like shape overall and constitutes a gripping portion that is held with the fingers of a user (such as a doctor) of the endoscope 20. As described above, the insertion portion 21 extends from the operating unit body 22a.

The bending operation member 22b and the multiple operation members 22c are operation members for performing various operations of the endoscope 20. The bending operation member 22b and the multiple operation members 22c are each provided at a predetermined position on the outer surface of the operation unit main body 22a.

The treatment tool insertion port 22d is provided at a predetermined position near the tip of the operating unit body 22a. The treatment tool insertion port 22d is the proximal opening of the treatment tool insertion channel 21d of the insertion section 21. The treatment tool insertion channel 21d is connected to the tip channel opening (not shown) of the tip section 21a on the tip side.

With this configuration, an endoscopic treatment tool (not shown) inserted through the treatment tool insertion port 22d is configured to pass through the treatment tool insertion channel 21d and then protrude outward from the tip channel opening of the tip portion 21a.

The universal cable 23 is a connection cord for connecting the endoscope 20 to the light source device 40 and the processor 10. For this purpose, the universal cable 23 is made of a tubular member extending from the side of the operation unit body 22a of the operation unit 22. A scope connector 23a is provided at the tip of the universal cable 23. This scope connector 23a is connected to the front panel of the light source device 40.

An electrical cable 23b extends from the scope connector 23a. A connector 23c is provided at the end of this electrical cable 23b. This connector 23c is connected to the front panel of the processor 10. Various signal transmission cables, optical fiber cables, etc. (not shown) are inserted into the universal cable 23.

The light source device 40 is a device that supplies illumination light to the illumination unit 25 (see FIG. 2) provided inside the tip 21a of the insertion section 21 of the endoscope 20. The illumination light emitted from the light source device 40 is transmitted to the illumination unit 25 of the tip 21a through the scope connector 23a, the universal cable 23, the operation section 22, and an optical fiber cable (not shown) that is arranged passing through the insertion section 21. The illumination light then passes through an illumination lens (not shown) included in the illumination unit 25 of the tip 21a, and is irradiated toward the observation target area in front of the tip 21a.

The processor 10 is a control device, a signal processing device, or a circuit unit that includes a control circuit and a signal processing circuit that control the entire endoscope system 1. The processor 10 is also configured to include the function of the object detection device of this embodiment (described in detail below; see FIG. 2).

The control circuit included in the processor 10 receives, for example, an operation instruction signal from the operating member 22c of the operation section 22 of the endoscope 20, and outputs various control signals for driving and controlling the imaging unit 26, the light source device 40, the lighting unit 25, etc. Also, the signal processing circuit included in the processor 10 receives, for example, an imaging signal from the imaging unit 26, and performs predetermined image signal processing, etc.

To this end, the processor 10 and the imaging unit 26 are electrically connected by a signal transmission cable (not shown). The signal transmission cable is inserted through the connector 23c, the electrical cable 23b, the scope connector 23a, the universal cable 23, the operation section 22, and the insertion section 21 to the imaging unit at the tip 21a.

With this configuration, the control signal output from the processor 10 and the imaging signal output from the imaging unit 26 are transmitted between the imaging unit 26 and the processor 10 through the signal transmission cable. One form of signal transmission cable is a composite cable in which multiple cables are bundled together and covered with an outer shield, outer tube, etc.

The display device 30 is a display device that receives image signals output from the processor 10 and displays endoscopic images in a predetermined format and various types of information. To this end, the display device 30 and the processor 10 are electrically connected using a video cable 24. The display device 30 may be in the form of a display device configured using, for example, a general liquid crystal panel.

The storage device 50 is a storage device that saves (records or stores) image data for storage that is generated for storage by various processes performed by the processor 10 based on the image information generated by the imaging unit 26.

As shown in FIG. 1, the storage device 50 is configured as one unit with the processor 10. As shown in FIG. 2, inside the storage device 50, a temporary storage unit 51 is provided in addition to the main storage unit (not shown) of the storage device 50. This temporary storage unit 51 is configured, for example, from a semiconductor memory, and is a component that functions as a temporary memory area that temporarily stores predetermined image data output from the image processing unit 11, etc.

In the configuration example shown in FIG. 1, the storage device 50 is integrally arranged inside the housing of the processor 10, but this is not limited to the configuration. For example, the storage device 50 may be configured as an external storage device that is configured using a housing separate from the processor 10.

In the example shown in FIG. 2, the temporary storage unit 51 is provided inside the storage device 50, but this is not a limited configuration. The temporary storage unit 51 can also be provided integrally inside the storage control unit 13 (see FIG. 2), which will be described later.

Furthermore, the processor 10 and the light source device 40 are not limited to being configured separately, as in the example configuration shown in FIG. 1. For example, the processor 10 and the light source device 40 may be configured integrally using a single housing.

The configuration of the lighting unit 25 is not limited to the above-mentioned configuration (a form in which the lighting light from the light source device 40 is transmitted to the tip portion 21a via an optical fiber cable or the like). As a configuration of the lighting unit 25 other than this configuration, for example, a light-emitting element such as an LED (Light Emitting Diode) as a lighting light source can be provided inside the tip portion 21a, and the power supply to the lighting light source (LED) and its light emission control can be controlled by a predetermined control circuit included in the processor 10.

The endoscope system 1 configured as described above basically has a configuration similar to that of a conventional endoscope system of the same type. Therefore, illustrations and detailed descriptions of other detailed configurations are omitted.

In an endoscope system 1 configured as described above, the object detection device of this embodiment is configured to be included in the processor 10. Next, the internal configuration of the processor in an endoscope system including an object detection device of one embodiment of the present invention will be described in detail below with reference to FIG. 2.

The processor 10 is configured with an image processing unit 11, a display control unit 12, a memory control unit 13, a specific object image detection unit 14, an operation determination unit 15, a focus control unit 16, an illumination control unit 17, a memory device 50 including a temporary memory unit 51, and the like.

The image processing unit 11 is a structural unit or circuit unit that receives image information output from the imaging unit 26 and performs various information processing based on the image information. The image processing performed here includes, for example, normal image information processing, display image data processing, storage image data processing, various types of image processing, etc.

Normal image processing includes A/D conversion processing, which receives the analog image signal output from the image sensor 26b and converts it into a digital image signal, and is a basic image information processing that is usually performed on image information. Normal image processing is the same as conventional processing, and a detailed description of this will be omitted.

Display image data processing is processing for generating image data for normal display. For example, image data for display is processing for generating first image data representing a first image (main endoscopic image) to be displayed in a first display area on the display screen of the display device 30, or processing for generating second image data (reference image data) representing a second image (reference image) to be displayed in a second display area different from the first display area on the same display screen.

In addition, FIG. 4 is a display example of the display screen of the display device in the endoscope system of this embodiment. In FIG. 4, reference numeral 100 indicates a display screen frame. Reference numeral 101 in FIG. 4 is a first display area. Reference numeral 102 in FIG. 4 is a second display area. Reference numeral 103 in FIG. 4 is an area for displaying information other than images. Reference numeral 104 in FIG. 4 is a predetermined alarm display area (described in detail later). The display example in FIG. 4 illustrates a case in which an image (reference numeral 101a; first image) is displayed only in the first display area 101.

The storage image data processing is a process for generating image data for recording (first image data, second image data, etc.).

The various types of image processing are image processing that are appropriately added to normal display image data. Specific examples of the various types of image processing include normal image adjustment processing such as brightness adjustment processing and white balance adjustment processing, and image enhancement processing such as contour enhancement processing and structural color enhancement processing (TXI; Texture and Color Enhancement Imaging).

The display control unit 12 is a structural unit or circuit unit that controls the display device 30 so that an appropriate display format is displayed according to the display image data output from the image processing unit 11. In addition to display control of the display image data (various settings such as display position, display area, display size, etc.), the display control unit 12 also controls the display of various information other than image information (text information, icon information, etc.). Here, the various information other than image information includes examination date and time information, subject (patient) information, medical record information, etc., as well as various alarm information (notification information) that is displayed as appropriate during an endoscopic examination, for example.

In more detail, the display control unit 12 performs display control to continuously display the endoscopic image in the first display area. In addition, when the disappearance of a specific object is detected from the endoscopic image, the display control unit 12 performs display control to display a reference image based on the second image data in a specified area (second display area) of the display device 30.

The disappearance here refers to cases where the specific object is displayed on the screen but is overlooked by the operator without noticing, where it is lost by unconsciously moving or operating the device, or where the object itself moves and becomes lost. In these cases, it is best to search for the object as soon as it disappears. This is because the area to be inspected is wide, and if it is overlooked once, it will be time-consuming to reinspect the same position. This disappearance can be detected and an alarm can be issued. This can be done by applying various methods to alert the operator, such as sound or vibration.

Furthermore, when a specific object contained in the reference image that is identical to a specific object that disappeared from the endoscopic image is detected again in the endoscopic image by the specific object image detection unit 14, the display control unit 12 performs display control to display a notification in a predetermined area (e.g., the alarm display area 104 (see FIG. 4)) on the display screen of the display device 30 to the effect that the same specific object contained in the reference image has been detected again.

The memory control unit 13 is a structural unit or circuit unit that controls the memory device 50 (including the temporary memory unit 51) so that the image data to be stored is saved (recorded or stored) in an appropriate storage format according to the image data to be stored output from the image processing unit 11.

The memory control unit 13 performs memory control to permanently store first image data representing an endoscopic image acquired by the endoscope 20 in a memory area of the storage device 50. The memory control unit 13 also performs memory control to temporarily store second image data (reference image data) including a specific object detected by the specific object image detection unit 14 (described later) in the temporary memory unit 51.

Furthermore, the storage control unit 13 also performs storage control to store various information related to the second image data (for example, information indicating the position of a specific object in an endoscopic image, unique characteristic information of a specific object (position, shape, size, color, etc.)) in association with the corresponding second image data.

In this way, by storing the characteristic information of the specific object in association with the second image data, even if multiple specific objects are present within one frame of an endoscopic image, for example, each of the multiple specific objects can be identified and stored.

The specific object image detection unit 14 is a structural unit or circuit unit that detects an image area including a specific object from an endoscopic image based on the first image data acquired by the endoscope 20.

In more detail, the specific object image detection unit 14 detects a specific object (such as a lesion, such as a tumor) in each frame of the endoscopic image that is displayed continuously based on the first image data output from the image processing unit 11.

The specific object detection unit 14 detects specific objects, for example, by performing similar image recognition based on multiple specific object images (such as case images) prepared in advance, or by performing data analysis using deep learning or machine learning to detect the desired specific object.

For example, in recent years, object detection technology using AI (Artificial Intelligence) has been developed that uses machine learning on endoscopic images acquired by an endoscope to obtain desired inference results (such as detection of specific objects such as tumors).

Machine learning involves learning the characteristics, time series information, spatial information, etc. of known input information, and then making inferences based on the learning results to obtain inference results about unknown matters. In other words, machine learning first obtains a trained model (also called an inference model) that makes it possible to infer a determinable output result from specific input information.

In this case, when generating a trained model, a large amount of information with a known relationship between input and output is used as training data in order to obtain highly reliable inference results. For example, in deep learning, a network is designed so that an expected output is obtained for a known input using a large amount of training data. The trained model obtained through this process can be used independently of the network that was used for training.

By providing the processor 10 with this type of inference model, it becomes possible to, for example, accurately and quickly detect specific objects in the specific object image detection unit 14, or to accurately and quickly determine endoscope operation in the operation determination unit 15.

The inference model is constructed roughly as follows. For example, a large amount of image data corresponding to the input and output is provided as training data to a specific network (not shown) for constructing the inference model. Here, image data of an image (such as a case image) in which a specific object is captured in an endoscopic image is input as the training data. In this case, an annotation may be set to surround the image area of the specific object with a frame or the like.

By learning from a large amount of training data, a specific network (not shown) is designed to obtain an output corresponding to an input. For example, FIG. 10 is an explanatory diagram of an inference model obtained as a result of learning in a specific learning device.

As shown in FIG. 10, when a group of endoscopic images (first teacher data group 201) including a specific object (such as a tumor) is input, information on the relative positional relationship between the object captured in the image (such as the lumen or inner wall of an organ, etc.) and the specific object, and additional information such as adding a frame display surrounding the image area of the specific object, are obtained along with reliability information. This allows the construction of a first inference model 200 that detects the specific object from the endoscopic image.

When endoscopic image data (first image data; moving image data) 202 during an endoscopic examination is input to the first inference model 200 constructed in this manner, the first inference model 200 detects an image area including a specific object in a frame in which the specific object is captured from the endoscopic image data 202, and outputs image data in which a frame display is added to the image area of the specific object (inference result image 203). This image 203 is temporarily stored as second image data, and can be displayed in the second display area of the display device 30 as necessary, providing a reference for the endoscope user during the endoscopic examination.

Furthermore, as shown in FIG. 10, when a group of search operation images during an endoscopic examination (second teacher data group 204; short moving images) are input, information regarding the search operation of a specific object during the endoscopic examination is obtained together with reliability information. This allows the construction of a second inference model 210 that determines the search operation of a specific object during an endoscopic examination.

When endoscopic image data (second image data; moving image data) 202 during an endoscopic examination is input to the second inference model 210 constructed in this manner, the second inference model 210 searches for and outputs a moving image (inference result moving image 205) including a search operation from the endoscopic image data 202. This moving image 205 is temporarily stored as second image data and can be displayed in the second display area of the display device 30 as necessary, providing a reference for the endoscope user during the endoscopic examination.

"Deep learning" is a multi-layered structure of the "machine learning" process using neural networks. A typical example is the "forward propagation neural network" which sends information from front to back and makes a judgment. In its simplest form, it has three layers: an input layer consisting of N1 neurons, a middle layer consisting of N2 neurons given by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be discriminated. The neurons in the input layer and middle layer, and the middle layer and output layer are each connected by a connection weight, and a bias value is added to the middle layer and output layer, making it easy to form logic gates. Three layers are sufficient for simple discrimination, but if there are many middle layers, it is also possible to learn how to combine multiple features in the machine learning process. In recent years, 9 to 152 layers have become practical in terms of the time it takes to learn, the accuracy of judgment, and the energy consumption.

Various well-known networks may be used as the network for machine learning. For example, R-CNN (Regions with CNN features) or FCN (Fully Convolutional Networks) that use CNN (Convolution Neural Network) may be used. This involves a process called "convolution" that compresses the features of an image, and operates with minimal processing and is strong in pattern recognition. In addition, a "recurrent neural network" (fully connected recurrent neural network) that can handle more complex information and allows information analysis in which the meaning changes depending on the order or sequence, and in which information flows in both directions, may be used.

To realize these technologies, conventional general-purpose calculation processing circuits such as CPUs and FPGAs can be used, but since much of the processing in neural networks involves matrix multiplication, GPUs (Graphic Processing Units) or Tensor Processing Units (TPUs), which are specialized for matrix calculations, may also be used. In recent years, such artificial intelligence (AI) dedicated hardware "neural network processing units (NPUs)" have been designed to be integrated and embedded with CPUs and other circuits, and may form part of the processing circuit.

Furthermore, in addition to deep learning, various well-known machine learning methods may be employed to obtain an inference model. For example, there are methods such as support vector machines and support vector regression. The learning here involves calculating the weights, filter coefficients, and offsets of the classifier, and other methods use logistic regression processing. When making a machine make a judgment, a human must teach the machine how to make the judgment. In this embodiment, a method is used to derive a judgment on an image using machine learning, but it is also possible to apply a rule-based method that applies rules acquired by humans through experience and heuristics to make a specific judgment.

Furthermore, when multiple specific objects are detected within one frame of an endoscopic image, for example, the specific object image detection unit 14 detects feature information for each specific object. Each piece of feature information thus detected is stored in the memory control unit 13 in association with the second image data of the corresponding specific object.

The operation determination unit 15 is a structural unit or circuit unit that determines the operation status of the operation unit 22 in the endoscope 20. The operation determination unit 15 determines the operation status of the operation unit 22, for example, based on the first image data output from the image processing unit 11. In this case, as described above, the operation determination may be performed using the second inference model 210. The operation determination unit 15 may also receive operation instruction signals from various operating members 22c provided in the operation unit 22, and determine the operation status of the operation unit 22 based on the received instruction signals.

The focus control unit 16 is a structural unit or circuit unit that drives and controls the imaging optical system 26a of the imaging unit 26. The focus control unit 16 controls the driving of the imaging optical system 26a based on the detection result of the specific object image detection unit 14 and the judgment result of the operation judgment unit 15, and executes, for example, a predetermined automatic focus adjustment operation. When a specific object is found, for example, focus control may be performed on it. Focus control may also be performed according to the contents of a memory that records the characteristics of the object.

The lighting control unit 17 is a structural unit or circuit unit that drives and controls the light source device 40 and the lighting unit 25. The lighting control unit 17 controls the light source device 40 or the lighting unit 25 in response to operation instruction signals from various operating members 22c provided in the operation unit 22, or instruction signals based on the detection results of the specific object image detection unit 14 or the judgment results of the operation judgment unit 15, and controls the switching of the light source type (white light, special light, etc.).

Here, light source types include, for example, white light that is irradiated onto the specimen during normal observation, as well as specific special light that is irradiated onto the specimen during image-enhancing observation such as narrow band imaging (NBI) or red dichromatic imaging (RDI).

It should be noted that all or part of the processor 10 includes hardware. Here, the processor 10 is configured with well-known components including, for example, a central processing unit (CPU), random access memory (RAM), read only memory (ROM), non-volatile memory, non-volatile storage, and non-transitory computer readable medium, as well as peripheral devices thereof.

The software programs executed by the CPU and fixed data such as data tables are stored in advance in ROM, non-volatile memory, non-volatile storage devices, etc. Then, the CPU reads out the software programs stored in ROM, etc., expands them into RAM, and executes them, and the software programs refer to various data, etc. as appropriate, thereby realizing the functions of the processor 10.

The processor 10 may also be configured with a semiconductor chip such as an FPGA (Field Programmable Gate Array). Furthermore, the processor 10 may also be configured with an electronic circuit.

Furthermore, the software program may be in a form in which the whole or part is recorded or stored as a computer program product on a portable disk medium such as a floppy disk, CD-ROM, or DVD-ROM, or on a non-transitory computer readable medium such as a card-type memory, HDD (Hard Disk Drive) device, or SSD (Solid State Drive) device.

The operation of the endoscope system 1 including the object detection device of this embodiment configured as described above will be described below. FIG. 3 is a flowchart explaining the operation of an endoscope system including an object detection device of one embodiment of the present invention. The flowchart shown in FIG. 3 is a diagram explaining the operation when an endoscopic examination is performed using the endoscope system 1.

First, it is assumed that the endoscope system 1 has been started up and is in a state in which an endoscopic examination can be performed. At this time, in step S1 of FIG. 3, the processor 10 checks whether the endoscope 20 in the endoscope system 1 is inserted inside an organ or the like to be examined and is in a state in which screening for an endoscopic examination can be started, or in which screening is already being performed (either state will be referred to as the screening state).

Here, whether or not the endoscope system 1 is in a screening state can be confirmed by, for example, checking that the endoscope system 1 has been started up in a predetermined procedure and that certain settings have been made for performing an endoscopic examination, or by checking the operation history of the endoscope system 1 after it has been started up. The state of the endoscope system 1 can also be determined by checking the image data of the endoscopic image acquired by the endoscope 20 itself or changes in the image data.

Note that FIG. 4 shows an example of the display screen of the display device 30 when screening is being performed during an endoscopic examination. The display example shown in FIG. 4 shows that the first image 101a (endoscopic image) is displayed in the first display area 101.

3, if it is confirmed that the endoscope system 1 is in a screening state, the process proceeds to the next step S2. If it is confirmed that the endoscope system 1 is not in a screening state, the process proceeds to step S11.
Although screening is described here as a representative operation for searching for a specific object, it is not necessary that the control from S2 onwards is performed only in the screening state, and similar processing may be performed in other states.

In step S11, the processor 10 confirms an instruction to end the endoscopic examination. Here, the instruction to end the endoscopic examination is, for example, a predetermined examination end instruction signal generated by operating a predetermined operating member among the processor 10 of the endoscopic system 1 or the operating members 22c provided on the operating unit 22 of the endoscope 20. In addition, the instruction to end the endoscopic examination can also be determined by, for example, confirming image data of the endoscopic image acquired by the endoscope 20.

If the instruction to end the test is confirmed in step S11, the process ends (END). If the instruction to end the test is not confirmed, the process returns to step S1 described above.

When it is confirmed that the screening state is in progress and the process proceeds to step S2, in this step S2, the processor 10 checks whether or not a specific object has been detected as a result of the screening being performed. In more detail, it checks whether or not an image area including a specific object has been detected from an endoscopic image based on the first image data acquired by the endoscope 20.

If a specific object is detected, the process proceeds to step S3. If a specific object is not detected, the process proceeds to step S11. The process from step S11 onwards is as described above.

FIG. 5 shows an example of the display on the display screen of the display device 30 at the time when a specific object is detected during an endoscopic examination. At this time, a first image 101a (endoscopic image) is displayed in the first display area 101. Simultaneously, a specific object 101b is displayed within the first image 101a displayed in the first display area 101. This specific object indicated by the reference symbol 101b is the specific object detected in the processing of step S2 described above.

Reference numeral 106 in FIG. 5 denotes a frame display corresponding to the image area including the detected specific object 101b. In this way, by displaying the frame 106 in the first display area 101, it is clearly indicated to the user (doctor, etc.) of the endoscope system 1 that the specific object 101b has been detected in the first image 101a. In this way, at the time of detection of the specific object 101b, for example, a display such as that shown in FIG. 5 is displayed.

In step S3 of FIG. 3, the processor 10 controls the memory control unit 13 to temporarily store the second image data, including the specific object image detected in the processing of step S2 described above, in the temporary memory unit 51.

Here, for example, if multiple image areas containing specific objects are detected, each of the detected images of the specific objects is stored as second image data in a separate file. At the same time, feature information for each of the multiple specific objects is acquired. Then, each piece of acquired feature information is stored in association with the corresponding second image data.

Furthermore, here, for example, if the same specific object is detected consecutively in an endoscopic image, the second image data is temporarily stored for each frame. In this case, it is not necessary to accumulate and store all the second image data. For example, only the most recent second image data, or a predetermined amount of second image data for the past few seconds including the most recent image data, may be stored in the temporary storage area.

Next, in step S4, the processor 10 checks whether the specific object detected in the processing of step S2 described above has disappeared (lost sight) from the endoscopic image being displayed. If it is confirmed that the detected specific object has disappeared from the endoscopic image, the process proceeds to the next step S5. If it is not confirmed that the detected specific object has disappeared from the endoscopic image, the process proceeds to step S12.

In step S12, the processor 10 refers to an internal clock (not shown) and waits for a predetermined time to elapse, and after the predetermined time has elapsed, proceeds to the processing of step S10.

The situation at this point is that the detected specific object continues to be displayed in the endoscopic image (the confirmation result of the processing in step S4 described above). Therefore, at this time, the user of the endoscopic system 1 (doctor, etc.) is able to fully observe the specific object in the endoscopic image being displayed. In this case, once it is confirmed in the processing in step S12 that the specific object has been displayed continuously for a predetermined period of time, the process proceeds to the processing in step S10 and the subsequent examination continues.

On the other hand, if the disappearance of a detected specific object is confirmed and processing proceeds to step S5, in this step S5, the processor 10 displays a second image based on the latest second image data temporarily stored in the processing of step S3 described above as a reference image in the second display area 102 on the display screen of the display device 30.

FIG. 6 shows an example of the display on the display screen of the display device 30 at the point in time (step S5) when the second image (reference image) is displayed in the second display area after the detected specific object has disappeared (step S4) during endoscopic examination.

First, in FIG. 6, the specific object 101b displayed in FIG. 5 has disappeared from the first image 101a (endoscopic image) displayed in the first display area 101. At this time, the display of the frame 106 also disappears at the same time as the specific object 101b disappears (step S4).

At the same time as the specific object 101b disappears, the second image 102a is displayed in the second display area 102 (step S5). At this time, the second image 102a displayed in the second display area 102 is a still image in a reduced form of the first image 101a (see FIG. 5) immediately before the specific object 101b disappeared. This second image 102a is a second image based on the latest second image data temporarily stored (temporarily recorded) in the processing of step S3 described above. At this point in time (after the specific object 101b disappears (S3) and at the point in time when the reference image is displayed (S4)), for example, the display will be as shown in FIG. 6.

Next, in step S6 of FIG. 3, the processor 10 checks whether the same specific object has been detected again within a predetermined time (e.g., about 5 seconds). Here, the same specific object is assumed to be the specific object that was once detected from the endoscopic image in the processing of step S2 described above, and that disappeared (lost sight, overlooked) from the endoscopic image in the processing of step S4 described above.

In some cases, it may be better to prioritize observation of this overlooked (lost) object when it is found again. This is because if the focus is not correct, there is a possibility that it will be overlooked again, and because rediscovering it means that it is what was being looked for and is considered to be an important object. In other words, this need can be met if the image area that contains the specific object contained in the reference image based on the specific object imaging result image data is temporarily recorded (temporarily stored) as a candidate for focusing.

If it is determined in step S6 that the same specific object has been detected again within a predetermined time, the process proceeds to step S7. If the same specific object has not been detected again, the process proceeds to step S10, and the subsequent inspections are continued.

In step S7, the processor 10 checks whether the same specific object detected again in the processing of step S6 described above has been continuously displayed for a predetermined time. If it is confirmed that the same specific object has been continuously displayed for a predetermined time, the processor proceeds to the processing of the next step S8. If the same specific object has not been continuously displayed for the predetermined time, the processor returns to the processing of step S4 described above.

In this case, when the same specific object is not continuously displayed for a predetermined period of time, it is assumed that the specific object cannot be stably observed on the display screen because, for example, it has disappeared again, or it has disappeared and been redetected momentarily repeatedly. In such a case, since the specific object cannot be observed adequately, it is assumed that it has disappeared again, and the process returns to step S4 to repeat the same process.

Figure 7 shows an example of the display on the display screen of the display device 30 at the time when the same specific object is detected again during an endoscopic examination. Here, the same specific object 101b is simultaneously displayed within the first image 101a displayed in the first display area 101 in Figure 7. At this time, the frame 106 is also displayed at the same time as the specific object 101b is redetected. Also, at this time, the display of the second image 102a in the second display area 102 is maintained.

At this time, there is a misalignment between the display of the first image 101a and the display of the second image 102a. This means that after the specific object disappears in the processing of step S4 described above, the user (doctor, etc.) of the endoscope system 1 operates the endoscope 20 from the state shown in FIG. 5 to perform an operation for redetection, and as a result, the angle of the first image, etc. changes. Therefore, in this case, it can be seen that the position of the redetected identical specific object 101b in the first display area 101 is different from the position of the specific object 101b in the second image 102a. In this way, even if the display position of the redetected specific object 101b on the screen is different, it can be determined that the redetected specific object 101b is the same specific object based on the feature information stored in association with the second image data.

In this way, if it is determined that the redetected specific object 101b is the same specific object, a predetermined notification display is made in a predetermined area of the display screen of the display device 30. For example, in the example shown in FIG. 7, a predetermined notification display, "! Redetected!", appears in the alarm display area 104. This may be assisted by audio.

This notification display notifies the user that the specific object 101b contained in the area surrounded by the frame 106 in the first display area 101 is the specific object that has been detected again this time, and that this specific object is the same specific object that was detected before it disappeared.

It is preferable that the notification display be a flashing or colored display, for example, so that the notification information is clearly indicated to the user (doctor, etc.) of the endoscope system 1. At this point (the point at which the specific object 101b is redetected), the display will be, for example, as shown in FIG. 7.

Next, in step S8 of FIG. 3, the processor 10 controls the display control unit 12 to stop displaying the reference image being displayed in the second display area. At the same time, the processor 10 controls the focus control unit 16 to execute automatic focus adjustment control (autofocus control; AF control) with a specific object as a specific target. The automatic focus adjustment control in this case may have, for example, an object tracking function. Then, the process proceeds to step S9.

In this way, by performing automatic focus adjustment control, it becomes easier to ensure that the same specific object that is detected again is always captured within the field of view of the endoscopic image in a focused state.

In such a situation, the user of the endoscope system 1 (such as a doctor) may perform an operation to observe the specific object in more detail. Specifically, for example, the user may perform an operation to bring the front surface of the tip 21a of the endoscope 20 (the objective lens of the imaging unit 26) closer to the specific object to be observed.

The closer the imaging unit 26 is to the specific object 101b, the larger the specific object 101b is displayed in the endoscopic image in the first display area 101, as shown in FIG. 8, for example. Even in this case, the area including the specific object 101b is displayed with a frame 106 enlarged to surround the specific object 101b according to the size of the specific object 101b.

Also, at the time shown in FIG. 8, the display of the second image in the second display area 102 has stopped, as shown in the processing of step S8 described above. Here, the timing for stopping the display of the second image is, for example, the time when it is confirmed that the same specific object that has been redetected has been continuously displayed for a predetermined period of time, as explained in the processing of step S7 above, but is not limited to this. For example, the time when a predetermined operation of the operation unit 22 is confirmed based on the output of the operation determination unit 15 after the same specific object has been redetected may be the timing for stopping the display of the second image.

In this case, the specified operation of the operation section 22 may be, for example, an operation of moving the imaging unit 26 closer to the specific object 101b to be observed. The operation may be determined, for example, by receiving an instruction signal from an operating member provided on the operation section 22 or an instruction signal from sensors provided on the endoscope 20. Alternatively, it may be determined based on the change over time in the endoscopic image acquired by the endoscope 20. That is, when the size of the specific object 101b to be observed is displayed gradually increasing over time in the endoscopic image, it may be determined that the imaging unit 26 is approaching the specific object 101b.

In general, in endoscopes, during normal observation such as screening, where the imaging unit and the object being observed are somewhat distant from each other (approximately a few centimeters), sufficient observation can be performed within the depth of field of the imaging optical system (so-called pan focus). However, as the imaging unit approaches the object being observed, the depth of field becomes shallow, making it desirable to perform detailed focus control.

As described above, in the endoscope system 1 of this embodiment, after a specific object is redetected, automatic focus adjustment control is performed to target the specific object as a specific target, so that a clear image in an accurate focus state can always be obtained even if an operation is performed to bring the imaging unit 26 closer to the specific object 101b.

Therefore, even when the imaging unit 26 is moved closer to the specific object 101b from the state shown in FIG. 8 and the display changes to the state shown in FIG. 9, focus control continues, so the specific object 101b is displayed in an enlarged state, and a clear image in a precise focus state can always be obtained.

The display example shown in FIG. 9 shows an example of a display in which the surface of the enlarged specific object 101b, such as capillaries, can be observed (displayed). FIG. 9 is an example of an endoscopic image displayed in such a state.

Returning to FIG. 3, in step S9, the processor 10 checks whether or not the individual detailed observation (inspection) of the specific object being observed has been completed. This check can be performed, for example, based on the output of the operation determination unit 15, and the time when a predetermined operation of the operation unit 22 is confirmed can be taken as the observation (inspection) end timing.

In this case, the predetermined operation of the operation section 22 may be, for example, an operation of moving the imaging unit 26 away from the specific object 101b to be observed. The determination of this operation is substantially the same as the determination of the above-mentioned approaching operation.

If it is confirmed in step S9 that detailed observation (inspection) of the specific object being observed has been completed, the process proceeds to the next step S10. If it is not confirmed that detailed observation (inspection) of the specific object has been completed (if detailed observation is continuing), the confirmation is repeated until it is confirmed that detailed observation (inspection) has been completed.

When the end of detailed observation (inspection) of the individual specific object is thus confirmed, then in step S10, the processor 10 controls the focus control unit 16 to switch the focus control setting of the currently set automatic focus adjustment control to the normal control setting. Thereafter, the process returns to step S1, and the process of searching for the next new specific object and observing (inspecting) it is repeated.

As described above, according to the present application, in an image inspection device such as an endoscope having a specific object image detection unit that detects an image area including a specific object based on acquired image data, the detected specific object imaging result image data is temporarily stored in a temporary storage unit, and when the disappearance of a specific object is detected, a display control unit is operated to display a reference image based on the specific object imaging result image data on the display unit that displays the acquired image data, so that even if the specific object is overlooked, it can be immediately rechecked. Here, the form in which the reference image is displayed together may be displayed in different areas of the same display, or both displays may partially overlap, or the reference image may be displayed separately on different displays.

As described above, according to the embodiment, in an endoscopic examination in which an image area of a specific object such as a tumor is detected from an endoscopic image and detailed observation of the specific object is performed, second image data including the detected specific object is temporarily stored. If the specific object disappears from the endoscopic image, a reference image based on the second image data is displayed as a reference image in the second display area of the display device 30. Then, if the same specific object is detected again in the endoscopic image, automatic focus adjustment control is performed on the same specific object.

With this configuration, the endoscope system 1 including the object detection device of one embodiment of the present invention can easily detect a specific object to be observed again based on a reference image of the specific object before it disappeared, even if the specific object is lost during an endoscopic examination.

In addition, if the same specific object is detected again, automatic focus adjustment control is performed on that specific object, so that the specific object can always be observed with a clear image that is accurately focused.

In addition, if a specific object is redetected after it has disappeared, a notification is displayed to the effect that the specific object has been redetected, allowing the user to easily notice that the same specific object has been detected again.

The specific object image detection unit 14 acquires feature information about the specific object, and the storage control unit 13 temporarily stores the feature information about the specific object in association with the second image data. With this configuration, even if multiple image areas containing specific objects are detected within one frame, each of the multiple detected specific object images can be stored separately and can be referenced separately. This allows for more precise inspection.

In the above embodiment, in the reference image display process (step S5 in FIG. 3), a still image in a reduced form of the first image (the image temporarily stored as the second image in step S3) immediately before the disappearance of the specific object is displayed in the second display area of the display device 30.

The form of the reference image is not limited to this example. The second image may be in the form of a search-assistance video image display (navigation display) for re-searching for a specific object that has disappeared.

In this case, the second image displayed in the second display area may be a short moving image of the period from when the specific object is detected to when it disappears, or for a few seconds after it disappears, or for a predetermined period of time (e.g., a few seconds) going back a certain amount of time from when it disappeared. Such short moving image data is temporarily stored, and is repeatedly displayed in the second display area.

In this way, it is possible to reconstruct the events leading up to the disappearance of a specific object. This may make it possible to clarify the cause of the disappearance of the specific object, such as the specific object being hidden by folds inside an organ due to peristalsis, or being covered by bleeding or food residue. Therefore, if the second image (reference image) is a short moving image, it becomes easier to determine the direction of the search operation after disappearance. This makes it easier to detect the same specific object again.

Furthermore, in the case of this playback display, for example, when a short moving image based on the second image data is played in the second display area, reverse playback may be performed. This makes it even easier to clarify the circumstances of the disappearance. This may make it even easier to redetect the same specific object.

On the other hand, in the embodiment described above, after the lost specific object is redetected (step S6 in FIG. 3), the continuous display of the same specific object for a predetermined period of time is confirmed (step S7 in FIG. 3), and automatic focus adjustment control is executed (step S8 in FIG. 3).

The operational control after the redetection of the lost specific object is not limited to the focus control described above. In the process of step S8 in FIG. 3, the following control may be performed instead of the focus control described above.

Firstly, for example, the lighting control unit 17 may control the driving of the lighting unit 25 and the light source device 40 to irradiate a region including a specific object with a predetermined special light. As a result, an image-enhanced observation image of a predetermined form is displayed in the first display region of the display device 30. Here, the image-enhanced observation image may be, for example, a narrowband observation image (NBI) or a red light observation image (RDI), etc.

With this configuration, the image-enhanced observation image can display specific objects more clearly, making them less likely to be lost. At the same time, the image-enhanced observation image can be used to perform more detailed observations.

Secondly, for example, control may be considered in which the image processing unit 11 executes a specified image adjustment process or a specified image enhancement process. As a result, an image reflecting the results of the specified image processing or the like is displayed in the first display area of the display device 30. Here, examples of image adjustment processes include brightness adjustment processes and white balance adjustment processes. Examples of image enhancement processes include contour enhancement processes and structural color enhancement processes.

The control timing of the illumination control unit 17 or the processing start timing of the image processing unit 11 may be, as described above, when a specific object disappears during an endoscopic examination, and then the same specific object is detected again and continues to be displayed, or, for example, when it is determined that a detailed observation operation is being performed on the specific object.

Here, a detailed observation operation of a specific object may be, for example, an operation of moving the tip 21a (imaging unit 26) of the endoscope 20 closer to the redetected specific object. Specifically, for example, when an operation of moving the tip 21a from its current position toward the position where the tip 21a was located at the time the second image data was acquired is detected, it is determined that an operation of moving the imaging unit 26 closer to the specific object is being performed.

As described above, when an operation such as moving the imaging unit 26 of the tip 21a of the endoscope 20 closer to the redetected specific object is detected, it is determined that an attempt is being made to perform a detailed observation operation on the specific object. At this time, in the embodiment described above, focus control is executed and automatic focus adjustment control is performed on the specific object (step S8 in FIG. 3).

In this case, if it is determined that the imaging unit 26 has approached within a predetermined distance of the specific object, the focus control may be further switched to the magnified observation mode. Part of the processing sequence in this case is shown in FIG. 11. FIG. 11 is a flowchart showing the operation of a modified version of this embodiment, which shows a part of the processing sequence in FIG. 3.

First, in FIG. 11, the processes in steps S1 to S6 are the same as those in FIG. 10 and are therefore not shown. Additionally, the processes in steps S7 and S8 in FIG. 11 are the same as those in steps S7 and S8 in FIG. 10.

That is, in step S7, the processor 10 confirms that the same specific object that has been redetected has been continuously displayed for a predetermined period of time. Next, in step S8, the processor 10 stops displaying the reference image in the second display area and executes focus control for automatic focus adjustment.

Then, in the next step S8a, the processor 10 determines whether or not an operation has been performed in which the imaging unit 26 approaches the specific object within a predetermined distance (e.g., about 3 millimeters) based on the output of the operation determination section 15. If an operation in which the imaging unit 26 approaches the specific object within the predetermined distance is determined, the process proceeds to step S8b. If an operation in which the imaging unit 26 approaches the specific object within the predetermined distance is not determined, the process proceeds to step S9.

Then, in step S8b, the processor 10 controls the focus control unit 16 to set the focus control appropriate for the specified magnified observation mode. Then, the process proceeds to step S9. The other processing steps are substantially the same as those in FIG. 10.

With this configuration, even when further magnified observation is performed, a clear image with accurate focus can always be obtained, allowing for better images of specific objects to be observed in detail.

The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the spirit of the invention. For example, an endoscope has been described as an example this time, but the invention can be applied to any device that sequentially obtains images and makes a judgment (such as an image inspection device). Furthermore, the above-described embodiment includes inventions at various stages, and various inventions can be extracted by appropriate combinations of the multiple components disclosed. For example, even if some components are deleted from all the components shown in the above embodiment, if the problem that the invention is intended to solve can be solved and the effects of the invention can be obtained, the configuration from which the components are deleted can be extracted as the invention. Furthermore, components from different embodiments may be appropriately combined. This invention is not restricted by its specific implementation except as limited by the attached claims.

Claims (20)

a specific object image detection unit that detects an image area including a specific object from an endoscopic image based on first image data acquired by an endoscope;
a storage control unit that temporarily stores second image data including the detected specific object in a temporary storage unit;
a display control unit that displays a reference image based on the second image data on a display device when disappearance of the specific object is detected from the endoscopic image;
An object detection device comprising: The object detection device according to claim 1, characterized in that the display control unit displays the reference image in a second display area on the display screen of the display device, the second display area being different from a first display area in which the endoscopic image is displayed. When the specific object included in the reference image is the same as the specific object that disappeared from the endoscopic image, and the specific object is detected again in the endoscopic image by the specific object image detection unit,
The object detection device according to claim 2 , wherein the display control unit displays, on the display device, a notification that the same specific object included in the reference image has been detected again. the storage control unit temporarily stores, in the temporary storage unit, moving image data of an image area including the specific object from a time point when the specific object disappeared from the endoscopic image until a certain time before the time point, as the second image data;
3. The object detection device according to claim 2, wherein the display control unit reproduces and displays moving image data based on the second image data in the second display area. a focus control unit that drives and controls an imaging optical system of an imaging unit included in the endoscope,
When the specific object image detection unit detects again in the endoscopic image a specific object that has disappeared from the endoscopic image and is identical to the specific object included in the reference image, and detects that the same specific object is still being displayed in the endoscopic image,
3. The object detection device according to claim 2, wherein the focus control unit drives and controls the imaging optical system, and executes automatic focus adjustment control for the specific object. The endoscope further includes an illumination control unit that drives and controls an illumination unit and a light source device included in the endoscope,
When the specific object image detection unit detects again in the endoscopic image a specific object that has disappeared from the endoscopic image and is identical to the specific object included in the reference image, and detects that the same specific object is still being displayed in the endoscopic image,
The lighting control unit drives and controls the lighting unit and the light source device to irradiate a special light onto an area including the specific object,
The object detection device according to claim 2 , wherein the display control unit displays an image-enhanced observation image in the first display area. 7. The object detection apparatus according to claim 6, wherein the image-enhanced observation image is a narrow band observation image (NBI) or a red light observation image (RDI). an image processing unit that performs image information processing on the first image data representing the endoscopic image;
an operation determination unit that determines an operation status of an operation unit of the endoscope;
Furthermore,
When the specific object image detection unit detects again in the endoscopic image a specific object that has disappeared from the endoscopic image and is identical to the specific object included in the reference image, and detects that the same specific object is still being displayed in the endoscopic image, or
When the operation determination unit determines that an observation operation is being performed on the specific object,
In either case,
The image processing unit executes an image adjustment process or an image enhancement process,
The object detection device according to claim 2 , wherein the display control unit displays an image output as a result of image processing by the image processing unit in the first display area of the display device. The image adjustment processing executed by the image processing unit is a brightness adjustment processing and a white balance adjustment processing,
9. The object detection device according to claim 8, wherein the image enhancement processing executed by said image processing unit is edge enhancement processing and structural color enhancement processing. The object detection device according to claim 8 , wherein the operation determination unit determines a detailed observation operation of the specific object based on a change over time in an image area including the specific object in the endoscopic image. The specific object image detection unit further acquires feature information regarding the specific object,
The object detection device according to claim 1 , wherein the storage control unit temporarily stores the feature information on the specific object in association with the second image data. a focus control unit that drives and controls an imaging optical system of an imaging unit included in the endoscope;
an operation determination unit that determines an operation status of an operation unit of the endoscope;
Further equipped with
When the specific object image detection unit detects the same specific object in the endoscopic image again, and the operation determination unit detects an operation of moving the tip end of the endoscope toward the position where the tip end of the endoscope was located at the time when the second image data was acquired,
3. The object detection device according to claim 2, wherein the focus control unit drives and controls the imaging optical system, and executes automatic focus adjustment control for the specific object. The specific object image detection unit
The object detection device described in claim 2, characterized in that it has a first inference model constructed by using a large amount of specific object image data as training data and annotating the image area containing the specific object by displaying a frame. The operation determination unit is
The object detection device described in claim 8, characterized in that it has a second inference model constructed using as training data a large amount of video image data consisting of a chronological arrangement of multiple endoscopic images acquired sequentially during an endoscopic examination using an endoscope, and in which a search operation for the specific object performed when the specific object detected during the endoscopic examination disappears is annotated. a screening step of continuously displaying endoscopic images acquired by an endoscope in a time series;
a detection step of detecting a specific object from the endoscopic image;
a temporary storage step of temporarily storing image data corresponding to an image area including the detected specific object;
a reference image display step of displaying an image area including the specific object as a reference image when disappearance of the specific object is detected from the endoscopic image;
a focus control step of terminating display of the reference image and executing automatic focus adjustment control for the specific object when the specific object corresponding to the reference image and identical to the specific object that has disappeared from the endoscopic image is detected again in the endoscopic image;
An object detection method comprising: On the computer,
A process for continuously displaying endoscopic images;
A process of detecting a specific object from the endoscopic image;
A process of temporarily storing image data corresponding to an image area including the detected specific object;
a process of displaying an image region including the specific object as a reference image when disappearance of the specific object is detected from the endoscopic image;
a process of ending display of the reference image and performing automatic focus adjustment on the specific object when the specific object corresponding to the reference image and identical to the specific object that disappeared from the endoscopic image is detected again in the endoscopic image;
An object detection program comprising: An endoscope system including an endoscope including an imaging unit having an imaging optical system that forms an optical image of an object and an imaging element that photoelectrically converts the optical image formed by the imaging optical system, and a processor,
The processor,
a specific object image detection unit that detects an image area including a specific object from an endoscopic image based on first image data acquired by the imaging unit;
a storage control unit that temporarily stores second image data including the detected specific object in a temporary storage unit;
a display control unit that, when disappearance of the specific object is detected from the endoscopic image, displays a reference image based on the second image data in a second display area, which is different from a first display area in which the endoscopic image is displayed, within a display screen of a display device;
a focus control unit that drives and controls the imaging optical system;
having
When the object image detection unit detects that the specific object that is the same as the specific object that disappeared from the endoscopic image is detected again in the endoscopic image, and the same specific object is continuously displayed in the endoscopic image,
The focus control unit controls the imaging optical system to perform automatic focus adjustment control for the specific object. Detecting an image area including a specific object based on image data acquired by an image inspection device;
Temporarily storing specific object imaging result image data including the detected specific object in a temporary storage unit;
An object detection method characterized in that, when disappearance of the specific object is detected from an image based on the image data, a reference image based on the specific object imaging result image data is also displayed on a display unit that displays an image based on the acquired image data. The object detection method according to claim 18, further comprising temporarily storing, as a disappearance process moving image, moving image data of an image area including the specific object from the time when the specific object disappeared from the endoscopic image until a certain time before that. The object detection method according to claim 18, further comprising provisionally recording an image area including the specific object contained in a reference image based on the specific object imaging result image data as a candidate for focusing.

Images