WO2024157360A1 - Treatment instrument detection device for endoscopic images, treatment instrument detection method for endoscopic images, and treatment instrument detection device program for endoscopic images - Google Patents

Abstract

The purpose of the present invention is to provide a treatment instrument detection device for endoscopic images which increases the accuracy with which the position of a treatment instrument in an endoscopic image can be ascertained. This treatment instrument detection device for endoscopic images comprises: an image acquisition unit which acquires a surgical image; an image recognition unit which uses a pre-trained deep learning model to perform image recognition on the surgical image; a treatment instrument extraction unit which generates a treatment instrument region comprising a segmentation mask and a bounding box within a treatment instrument image in the surgical image; and a treatment instrument region inpainting unit which inpaints an omission in the treatment instrument region due to an obstacle present in the surgical image. The treatment instrument region inpainting unit comprises an image inpainting unit which generates a colour processing image and inpaints the treatment instrument image. The treatment instrument region inpainting unit may comprise a rectangle inpainting unit which, for the segmentation mask of the treatment instrument output by the treatment instrument extraction unit, generates the smallest rectangle surrounding the treatment instrument region acquired by image processing and uses the smallest rectangle to perform inpainting.

Description

Treatment tool detection device for endoscopic images, treatment tool detection method for endoscopic images, and treatment tool detection device program for endoscopic images

The present invention relates to a treatment tool detection device for endoscopic images, a treatment tool detection method for endoscopic images, and a treatment tool detection device program for endoscopic images, and in particular to a treatment tool detection device, method, and program for endoscopic images that complements treatment tool recognition using artificial intelligence in surgical support for laparoscopic surgery.

In endoscopic surgery, by determining the exact position of the tip of a treatment tool within an endoscopic image, various applications become possible, such as enabling more accurate operation when using a surgical support robot system. Progress is being made in the development of technology to determine the location of treatment tools within endoscopic images (see Patent Documents 1 and 2, etc.).

The method disclosed in Patent Document 1 is a method of attaching a marker to the tip of a treatment tool and detecting the treatment tool. However, when retrofitting a marker to an existing treatment tool, it is necessary to consider the sterilization of the marker and the risk of the marker falling off. For this reason, there are problems such as the need for the treatment tool manufacturers to be responsive in order to popularize treatment tools with markers or that are marker-compatible. Furthermore, the method of attaching a marker involves issues such as the possibility of interfering with the operation of the treatment tool.

The method disclosed in Patent Document 2 is a method for detecting treatment tools in endoscopic images by combining a support vector machine (SVM), which is a type of machine learning, with a mathematical model.

The machine learning used in the method disclosed in Patent Document 2 is a support vector machine, which recognizes only predefined features, and therefore requires the learning of a very large number of features to recognize one treatment tool, such as changes in the texture of the treatment tool over time during surgery, parts of the treatment tool being hidden, and the presence of other treatment tools with the same characteristics as those used for learning. Furthermore, even if this configuration is simply replaced with a method capable of recognition under more complex conditions such as deep learning, if an endoscopic image contains multiple treatment tools of the same type and these are close to each other, it is difficult to recognize these treatment tools of the same type as individual treatment tools. Furthermore, in order to accurately track the three-dimensional position of the tip of the treatment tool during surgery, it is necessary to extract the tip point in more detail.

In consideration of the above problems, an invention has been filed that aims to provide a device for detecting the tip of a treatment tool in an endoscopic image, a method for detecting the tip of a treatment tool in an endoscopic image, and a program for detecting the tip of a treatment tool in an endoscopic image, which can appropriately detect the position of the tip of a treatment tool even in a complex environment where multiple treatment tools of the same type are included in an endoscopic image and are close to each other (see Patent Document 3).

Patent Document 3 is an invention relating to a device for detecting the tip of a treatment tool in an endoscopic image, which includes an image acquisition unit that acquires a surgical image, an image recognition unit that performs image recognition of the surgical image using a pre-trained deep learning model, a treatment tool extraction unit that generates a segmentation mask and a bounding box of the treatment tool in the surgical image, and a tip detection unit that detects the tip of the treatment tool using the generated segmentation mask and the generated bounding box, and the tip detection unit includes a root identification unit that identifies each of the four sides of the image edge region present in the surgical image and the side of the image edge region that has the largest coverage area with the segmentation mask as the root, an end position detection unit that detects the end portion of the treatment tool, an angle detection unit that detects the angle of the treatment tool, and a tip point detection unit that detects the tip point of the treatment tool.

JP 2017-164007 A Special Publication No. 2016-506260 Patent Application No. 2021-135936

The method described in Patent Document 3 is a technology that detects the tip point based on the segmentation mask and bounding box information of the treatment tool present in the acquired image, but due to obstacles reflected in the screen, the segmentation mask and bounding box may not be in contact with the edge area of the image, which is thought to affect the detection of the tip point. In particular, if the treatment tool is present on the screen, including a trocar (a cylindrical object called a trocar, etc.) for introducing the treatment tool into the abdominal cavity, the trocar part will not be recognized, resulting in a situation where the segmentation mask and bounding box cannot be accurately acquired, and this will affect the performance of tip detection.

In consideration of the above problems, the present invention aims to provide a device for detecting the tip of a treatment tool in an endoscopic image, a method for detecting the tip of a treatment tool in an endoscopic image, and a program for detecting the tip of a treatment tool in an endoscopic image, which can quickly and appropriately detect the position of the tip point of a treatment tool even in situations where an obstacle is reflected in the surgical image.

　That is, the endoscopic image treatment tool detection device of the embodiment includes an image acquisition unit that acquires a surgical image, an image recognition unit that performs image recognition of the surgical image using a pre-trained deep learning model, a treatment tool extraction unit that generates a treatment tool area in the surgical image consisting of a segmentation mask and a bounding box in the treatment tool image, and a treatment tool area completion unit that completes the treatment tool area due to an obstacle present in the surgical image. The treatment tool area completion unit includes an image completion unit that performs color processing on the surgical image to generate a color-processed image and completes the treatment tool area using the color-processed image. When generating the color-processed image, the image completion unit includes a color reduction unit that reduces a predetermined color from the surgical image to generate a reduced-color image, a binarization unit that binarizes the reduced-color image, an extraction unit that deletes areas unrelated to the target treatment tool present in the surgical image and extracts the treatment tool image from the color-processed image, a combination unit that generates a combined image by combining the treatment tool image acquired from the color-processed image with the inferred treatment tool image acquired through the deep learning model, and an expansion unit that generates a bounding box according to the combined image.

Furthermore, the treatment tool area complementing section of the endoscopic image treatment tool detection device may include a rectangle complementing section that generates a minimum rectangle surrounding the treatment tool area obtained by image processing for the treatment tool segmentation mask that is the output of the treatment tool extraction section, and performs complementing using the minimum rectangle. When performing complementing using the minimum rectangle, the rectangle complementing section may include a tip determination section that detects the tip of the treatment tool from the treatment tool segmentation mask that is the output of the treatment tool extraction section and determines whether or not there is an opening at the tip, a direction detection section that detects the extension direction of the treatment tool from the treatment tool segmentation mask that is the output of the treatment tool extraction section, a brightness detection section that detects the brightness around the treatment tool in the surgical image and detects the extension direction of the treatment tool based on the inference mask from the level of brightness, an extension section that extends the inference mask in the extension direction to generate an extended treatment tool image, a combination section that generates a combined image by combining the extended treatment tool image with the inferred treatment tool image obtained through the deep learning model, and an extension section that generates the bounding box according to the combined image.

Furthermore, in the treatment tool detection device for endoscopic images, the treatment tool area completion section may include an image completion section and a rectangle completion section, and may selectively output either of the outputs.

Furthermore, the image complementing section of the endoscopic image treatment tool detection device may be configured to remove background areas in the color-processed image from the brightness-adjusted image generated by adjusting the brightness of the surgical image.

Furthermore, the image complementation section of the endoscopic image treatment tool detection device may identify the treatment tool image in the color-processed image by comparing the treatment tool image with the inferred treatment tool image.

Furthermore, the image complementation section of the endoscopic image treatment tool detection device may reduce the image size of the surgical image to compress the time required for image processing.

In addition, the device for detecting the tip of a treatment tool in an endoscopic image of an embodiment includes an image acquisition unit that acquires a surgical image, an image recognition unit that performs image recognition of the surgical image using a pre-trained deep learning model, a treatment tool extraction unit that generates a segmentation mask and a bounding box of the treatment tool in the surgical image, a treatment tool area completion unit that completes missing treatment tool areas due to obstacles present in the surgical image, and a tip detection unit that detects the tip of the treatment tool using the generated segmentation mask and the generated bounding box, and the tip detection unit is characterized by including a root identification unit that identifies, as a root, each of the four sides of the image edge area present in the surgical image and the side of the image edge area that has the largest coverage area with the segmentation mask, an end position detection unit that detects the end portion of the treatment tool, an angle detection unit that detects the angle of the treatment tool, and a tip point detection unit that detects the tip point of the treatment tool.

The endoscopic image treatment tool detection device of the present invention includes an image acquisition unit that acquires a surgical image, an image recognition unit that performs image recognition of the surgical image using a pre-trained deep learning model, a treatment tool extraction unit that generates a treatment tool area in the surgical image consisting of a segmentation mask and a bounding box in the treatment tool image, and a treatment tool area completion unit that completes the treatment tool area due to an obstacle present in the surgical image, and the treatment tool area completion unit is provided with an image completion unit that performs color processing on the surgical image to generate a color-processed image and completes the treatment tool area using the color-processed image, and the image completion unit is provided with a color reduction unit that reduces a predetermined color from the surgical image to generate a color-reduced image when generating the color-processed image. The system includes a binarization unit that binarizes the color-reduced image, an extraction unit that removes areas in the surgical image that are unrelated to the treatment tool of interest and extracts the treatment tool image from the color-processed image, a combination unit that combines the treatment tool image obtained from the color-processed image with the inferred treatment tool image obtained through the deep learning model to generate a combined image, and an expansion unit that generates a bounding box according to the combined image. This further develops the method of using a segmentation mask and a bounding box to recognize treatment tools in endoscopic images, and adds evaluation of the color tone, shape, and presence or absence of an opening in the treatment tool area, thereby improving the accuracy of identifying the position of the treatment tool on the screen of the endoscopic image.

In addition, the device for detecting the tip of a treatment tool in an endoscopic image of the present invention includes an image acquisition unit for acquiring a surgical image, an image recognition unit for performing image recognition of the surgical image using a pre-trained deep learning model, a treatment tool extraction unit for generating a segmentation mask and a bounding box of the treatment tool in the surgical image, a treatment tool area completion unit for completing a missing treatment tool area due to an obstacle present in the surgical image, and a tip detection unit for detecting the tip of the treatment tool using the generated segmentation mask and the generated bounding box, and the tip detection unit detects the tip of the treatment tool in the surgical image. The system is equipped with a root identification unit that identifies as the root the side of the image edge region that has the largest coverage area with each of the four sides of the image edge region present in the image and the segmentation mask, an end portion detection unit that detects the end portion of the treatment tool, an angle detection unit that detects the angle of the treatment tool, and a tip point detection unit that detects the tip point of the treatment tool. By further developing the method of using a segmentation mask and bounding box to recognize treatment tools in endoscopic images and adding evaluation of the color tone, shape, and presence or absence of an opening in the treatment tool region, the accuracy of identifying the position of the treatment tool on the screen in the endoscopic image can be improved.

1 is a schematic diagram showing an example of the overall configuration of a treatment tool detection device for an endoscopic image according to an embodiment of the present invention. 2 is a diagram illustrating an image displayed on the endoscope display shown in FIG. 1 . FIG. 2 is a block diagram showing a configuration of a treatment tool detection device for an endoscopic image shown in FIG. 1 . 1 is a first block diagram showing a configuration of functional parts in a computer of an endoscopic image treatment tool detection device. FIG. 13 is a second block diagram showing the configuration of functional parts in a computer of the treatment tool detection device for an endoscopic image. FIG. FIG. 1 is a schematic diagram illustrating a segmentation mask and a bounding box. 7A and 7B are diagrams for explaining a method in which the root identification unit determines the side on which the treatment tool appears, and FIG. 7A is a schematic diagram of the entire inference image, and FIG. 7B is a schematic diagram of an enlarged view of the root of the treatment tool. 3A is a first schematic diagram showing generation of a color-processed image in a color-reduction section and a binarization section, FIG. 3B is a second schematic diagram, and FIG. 3C is a third schematic diagram. 5A is a first schematic diagram showing generation of a color-tone-processed image in an extraction unit, and FIG. 5B is a second schematic diagram showing generation of a color-tone-processed image in an extraction unit. 1A is a first schematic diagram showing generation of a complementary image in a combination section and an extension section, and FIG. 1B is a second schematic diagram showing generation of a complementary image in a combination section and an extension section. FIG. 11 is a flow diagram of a process for adjusting the brightness of a surgical image and deleting a background image in the image complementing unit. 13 is a flowchart showing a process of identifying a treatment tool image in a color-tone-processed image in an image complementing section. FIG. 1A is a schematic diagram showing the state of image size compression in an image complementing section at actual size, FIG. 1B is a schematic diagram showing 1/4 size, and FIG. 1C is a schematic diagram showing 1/8 size. 1A is a first schematic diagram showing detection of an opening of a treatment tool by a tip determination unit, and FIG. 1B is a second schematic diagram showing detection of an opening of a treatment tool by a tip determination unit. 1A is a first schematic diagram showing a first detection example of the extension direction of the treatment tool in the direction detection unit; FIG. 13 is a schematic diagram showing a second detection example of the extension direction of the treatment tool in the brightness detection unit; FIG. 13A is a first schematic diagram showing a third detection example of the extension direction of the treatment tool in the direction detection unit; FIG. 1A is a first schematic diagram showing the generation of a complementary rectangle in the extension section and the combination section, and FIG. 4 is a flowchart illustrating a treatment tool detection method for an endoscopic image according to an embodiment of the present invention. 13 is a flowchart of the leading end detection unit. 13A is a first flowchart and FIG. 13B is a second flowchart in a treatment tool region complementing section. 13 is a flowchart of an image complementing unit. 13 is a flowchart of a rectangle complementing unit.

Next, an embodiment of the present invention will be described with reference to the drawings. In describing the drawings relating to the embodiment, identical or similar parts are given the same or similar reference symbols. However, it should be noted that the drawings are schematic, and the relationship between thickness and planar dimensions, the thickness ratio of each component, etc., differ from the actual ones. Therefore, specific thicknesses and dimensions should be determined with reference to the following explanation. In addition, it goes without saying that there are parts in which the dimensional relationships and ratios differ between the drawings.

The embodiments are merely examples of devices and methods for embodying the technical ideas of the present invention, and the technical ideas of the present invention are not specified below in terms of the configuration, arrangement, layout, etc. of each component. The technical ideas of the present invention may be modified in various ways within the technical scope defined by the claims.

FIG. 1 shows an example of the overall configuration of an endoscopic image treatment tool detection device 10 (hereinafter referred to as the "treatment tool detection device") of an embodiment. The endoscopic image treatment tool detection device 10 also functions as a treatment tool tip detection device for endoscopic images. In the following explanation, the treatment tool tip detection device will be included in the treatment tool detection device 10. The treatment tool detection device 10 shown in FIG. 1 is composed of an endoscope 103, treatment tools 101 and 102, a processing device 105, and an endoscope display 106. In FIG. 1, the endoscope 103 and the treatment tools 101 and 102 are inserted into the abdominal cavity of a patient 107. The endoscope 103 inserted into the abdominal cavity of the patient 107 photographs an affected area 104, and the image obtained by photographing is transmitted to the processing device 105. The image transmitted from the endoscope 103 to the processing device 105 is displayed on the endoscope display 106.

The treatment tools 101 and 102 are, for example, grasping forceps, high-frequency knives, hemostatic clips, etc. Furthermore, in addition to the aforementioned forceps, knives, etc., the treatment tools 101 and 102 also include a trocar 108 that is used when introducing the treatment tools into the abdominal cavity of the patient 107. In FIG. 1, the treatment tools 101 and 102 are each housed inside the sheath of the trocar 108 (see the two-dot chain line), and the tips of the treatment tools 101 and 102 are exposed from the trocar 108.

The endoscope 103 has an image sensor and a light source (not shown), and the light source of the endoscope 103 irradiates light into the abdominal cavity of the patient 107, and the image sensor captures an image of the affected area 104, generating image data.

The processing device 105 is an endoscope system that includes various electronic computers (computing resources, so-called computers), such as a personal computer (PC), a mainframe, a workstation, a cloud computing system, etc.

Figure 2 shows an image captured by the endoscope 103 shown in Figure 1 and displayed on the endoscope display 106. The image shown in Figure 2 includes treatment tools 101 and 102 and an affected area 104. As an example, the treatment tools 101 and 102 shown in Figure 2 are the same type of grasping forceps. Treatment tool 101 represents a state in which the tip of the grasping forceps is open, and treatment tool 102 represents a state in which the tip of the grasping forceps is closed. In Figure 2, treatment tools 101 and 102 are also housed inside the sheath of trocar 108 (see dashed double-dashed line), and the tips of treatment tools 101 and 102 are exposed from trocar 108.

The treatment tool detection device for endoscopic images (treatment tool tip detection device for endoscopic images) 10 of the embodiment recognizes a treatment tool in an endoscopic image captured by an endoscope and detects the tip of the recognized treatment tool. For example, a high-frequency knife has one tip, so the treatment tool tip detection device of the embodiment detects one tip for the high-frequency knife. On the other hand, for example, grasping forceps has one tip when the tip gripper is closed. When the tip gripper is open, there are actually two tips, but even when the tip gripper is open, there are cases where the tip appears to be two or one in the endoscopic image depending on the orientation of the grasping forceps relative to the endoscope. The treatment tool tip detection device of the embodiment can detect treatment tools in endoscopic images regardless of the type, number, state, or orientation of the treatment tool. It can also detect treatment tools in the screen, including trocars.

The block diagram in FIG. 3 is an example of the configuration of the processing device 105 shown in FIG. 1. The processing device 105 in FIG. 3 is provided with a CPU 111 for executing various calculations, a ROM 112 for storing processing programs, a RAM 113 for storing data, etc., a storage unit 114 for storing various data and observation results, etc., and an I/O (input/output interface) 115, etc. The I/O 115 is an interface for communication (transmission and reception), a buffer, etc. The I/O 115 is used for transmitting and receiving image data to and from the endoscope 103, transmitting and receiving image data to and from the endoscope display 106, etc., and works in conjunction with the CPU 111. The RAM 113 is used for temporarily expanding the processing programs, image data imported from the I/O 115, etc. In addition to the CPU 111, a GPU specialized for image processing may be provided.

The memory unit 114 stores the deep learning model that is used by the image recognition unit 160 (described below) when performing image recognition.

The block diagrams of Figures 4 and 5 show the functional units within the CPU 111 of Figure 3. When each functional unit of the CPU 111 is realized by software, the CPU 111 realizes it by executing commands of a program, which is the software that realizes each function. In detail, each functional unit is an image acquisition unit 150, an image recognition unit 160, a treatment tool extraction unit 170, a tip detection unit 180, and a treatment tool area completion unit 200, and the tip detection unit 180 is equipped with a root identification unit 210, an end position detection unit 220, an angle detection unit 230, and a tip point detection unit 240. The treatment tool area completion unit 200 is equipped with an image completion unit 250 and a rectangle completion unit 260. The image completion unit 250 is equipped with a color reduction unit 251, a binarization unit 252, an extraction unit 253, a combination unit 254, and an expansion unit 255. The rectangle completion unit 260 includes a tip determination unit 261, a direction detection unit 262, a brightness detection unit 263, an extension unit 264, a combination unit 265, and an expansion unit 266. Each functional unit will be described below with reference to the drawings.

The image acquisition unit 150 acquires surgical images. The surgical images are images captured by the endoscope 103 shown in FIG. 1 and displayed on the endoscope display 106.

The image recognition unit 160 performs image recognition of surgical images using a pre-trained deep learning model.

The deep learning model used by the image recognition unit 160 of the treatment tool detection device 10 of the embodiment when performing image recognition of a surgical image is a model that has been trained in advance based on an instance segmentation technique and is stored in the storage unit 114. This deep learning model is a Convolutional Neural Network (CNN) that is constructed by adding annotations to image data of a surgical image captured in advance and using the annotated image data. In the embodiment, the deep learning model is constructed using at least image data that has annotations added to the treatment tool within the image data captured in advance. The annotation here refers to the creation of training data for the treatment tool when using the deep learning model (machine learning model).

The treatment tool extraction unit 170 generates a treatment tool area in the surgical image, the area being composed of a segmentation mask and a bounding box in the surgical image. The tip detection unit 180 detects the tip of the treatment tool using the generated segmentation mask and the generated bounding box.

As shown in FIG. 6, the treatment tool extraction unit 170 extracts a segmentation mask 402 and a bounding box 403 of a treatment tool 401 contained in the image data from the inference result of instance segmentation by the image recognition unit 160. Here, instance segmentation is a method of detecting individual segmentation masks and bounding boxes for multiple objects present in the image data.

The treatment tool segmentation mask is the pixel-unit area of the treatment tool in the endoscopic image, and is the area indicated by diagonal lines in Figure 6. The bounding box is rectangular data surrounded by the minimum x value, minimum y value, maximum x value, and maximum y value of the treatment tool segmentation mask, and is the rectangle indicated by dashed lines in Figure 6, and the four sides of the bounding box are parallel to the four sides of the endoscopic image.

The treatment tool area complementing unit 200 complements missing treatment tool areas caused by obstacles present in the surgical image. In surgical images, various tissues or other treatment tools may overlap the treatment tool as obstacles. Therefore, the treatment tool area complementing unit 200 complements the treatment tool area that is invisible due to these obstacles through image processing, enabling proper detection of the tip of the treatment tool.

The root identification unit 210 identifies as the root the side of the image edge region that has the largest coverage area between each of the four sides of the image edge region present in the surgical image and the segmentation mask.

The method by which the root identification unit 210 determines the side on which the treatment tool appears will be described with reference to Figures 7(A) and (B). Figure 7 shows an inference image 60 including a treatment tool 601. In Figure 7(A), an image edge line 602 is set on the inside at a distance r from the four sides ABCD of the inference image 60, and the area between the image edge line 602 and the outer frame line of the inference image 60 is set as the image edge 603. Of the four sides ABCD of the inference image 60, the side that covers the largest area between the image edge 603 and the treatment tool 601 is determined to be the side on which the treatment tool 601 appears.

FIG. 7(B) is a schematic diagram showing an enlarged view of the root of the treatment tool 601 in FIG. 7(A). A perpendicular line L1 is drawn from the corner closest to side C of the image edge line 602 to side C. Similarly, a perpendicular line L2 is drawn from the corner closest to side D of the image edge line 602 to side D. The area of a region 604 of the treatment tool 601 surrounded by the image edge line 602 and the perpendicular line L1 is compared with the area of a region 605 surrounded by the image edge line 602 and the perpendicular line L2. The area of region 605 is larger than the area of region 604, and therefore it is determined that the treatment tool 601 is emerging from side D.

The end position detection unit 220 detects the end portion of the treatment tool. The end position detection unit 220 detects the rearmost point of the treatment tool extending from the base toward the tip point of the treatment tool in the endoscopic image, i.e., the rear end point, as the end position. In addition, of the four corners of the bounding box, the corner closest to the tip of the treatment tool, i.e., the tip corner, and the corner furthest from it, i.e., the rear end corner, are detected as the end position.

The angle detection unit 230 detects the angle that the treatment tool in the endoscopic image makes with one of the four sides surrounding the endoscopic image as the angle of the treatment tool. Methods by which the angle detection unit 230 detects the angle of the treatment tool include a method of detection using the rear end point detected by the end position detection unit 220, a method of detection using the tip angle and rear end angle detected by the end position detection unit 220, and a method of detection without using any of the rear end point, tip angle, or rear end angle. Note that in the embodiment, these methods are applied regardless of the number of tips of the treatment tool.

The tip point detection unit 240 detects the tip point of the treatment tool in the endoscopic image. For example, the point on the treatment tool's segmentation mask that is farthest from the rear end point of the treatment tool is detected as the tip point of the treatment tool. Or, the point on the treatment tool's segmentation mask that is closest to the tip angle of the treatment tool is detected as the tip point of the treatment tool. These methods are limited to the case where the treatment tool has one tip. In addition, the intersection of a straight line that passes through the center of gravity of the treatment tool's segmentation mask and has the angle of the treatment tool as its inclination with the treatment tool's bounding box is detected as the tip point of the treatment tool. These detections of the tip point are just an example, and it is also possible to set a circle for determination at the tip point of the treatment tool and determine the angle of the arc portion that overlaps with each tip point.

In the treatment tool area completion unit 200 of this embodiment, the image completion unit 250 performs color processing on the surgical image to generate a color-processed image, and uses the color-processed image to complete the treatment tool area.

The image completion unit 250 performs the following processes in an integrated manner:

The color reduction unit 251 reduces a specific color from the surgical image to generate a color-reduced image. This process is explained in FIG. 8.

The binarization unit 252 binarizes the color-reduced image. This process is explained in FIG. 8.

The extraction unit 253 removes areas that are unrelated to the treatment tool in question that are present in the surgical image, and extracts the treatment tool image from the color-processed image. This process is explained in FIG. 9.

The combination unit 254 combines the inferred treatment tool image acquired through the deep learning model with the treatment tool image acquired from the color-processed image to generate a combined image 125 (see FIG. 10). This process is described in FIG. 10.

The expansion unit 255 generates a bounding box for the combined image. This process is explained in FIG. 10.

The image completion unit 250 also adjusts the brightness of the surgical image to remove background areas in the color-processed image.

In the embodiment of the treatment tool area completion unit 200, the rectangle completion unit 260 generates the smallest rectangle that encloses the treatment tool area for the treatment tool image in the color-tone-processed image, and performs completion using the smallest rectangle. The rectangle completion unit 260 performs the following processes in an integrated manner.

The tip determination unit 261 detects the tip of the treatment tool using an inference mask and determines whether or not there is an opening at the tip. This process is explained in Figure 14.

The direction detection unit 262 detects the extension direction of the treatment tool from the inference mask. This process is explained in Figure 15.

The brightness detection unit 263 detects the brightness around the treatment tool area in the endoscopic image and detects the extension direction of the treatment tool based on the brightness level. This process is explained in FIG. 16.

The extension unit 264 extends the inference mask in the extension direction to generate an extended treatment tool image. This process is explained in FIG. 18.

The combination unit 265 generates a combined image by combining the inferred treatment tool image acquired through the deep learning model with the extended treatment tool image. This process is explained in FIG. 18.

The expansion unit 266 expands the bounding box to fit the combined image. This process is explained in FIG. 18.

The inference mask referred to here is an image (data) that has been processed to separate the area in the endoscopic image where the treatment tool is present from unrelated areas and to cover (mask) the area where the treatment tool is present. Additionally, the minimum rectangle referred to here is a rectangle that includes all points inside the treatment tool and is sized and oriented so as to have the smallest area.

The generation of a color-processed image is illustrated as the schematic diagrams of Figures 8 and 9. The schematic diagrams of Figures 8 and 9 show an overview of the processing of a surgical image acquired through the image acquisition unit 150 and image-recognized by the image recognition unit 160. The processing is in the order of Figures 8(A), (B), (C), and then Figures 9(A) and (B).

Figure 8 (A) shows a surgical image 121, which is an image of the abdominal cavity captured by a camera (not shown) connected to the endoscope 103. A color-reduced image is generated by subtracting two of the three elements R (red), G (green), and B (blue) that make up each pixel of the surgical image. For example, G (green) is subtracted from R (red). As a result, the brightness of the color-reduced image represents the level of saturation in the surgical image. Figure 8 (B) shows a color-reduced image 122 after color reduction.

FIG. 8(C) is a binarized image (binarized image 123). For the color-reduced image in FIG. 8(B), a binarized image is generated by outputting 1 for pixels below a predetermined threshold and 0 for other pixels. As a result, areas with low saturation in the surgical image are extracted as treatment tool areas.

FIG. 9(A) shows a method for removing areas unrelated to the target treatment tool from a binarized image 123 (FIG. 8(C)). The treatment tool area 123a in the figure is a segmentation mask obtained through a deep learning model. Then, as shown in FIG. 9(B), the treatment tool and areas unrelated to the target treatment tool are removed to generate a color-processed image 124 (treatment tool mask image).

The schematic diagram in Figure 10 (A) shows an inferred treatment tool image obtained by image recognition of a surgical image using a deep learning model. Here, the inference mask 126 does not include the area stored in the trocar. Therefore, it is unclear which of the four sides of the image edge area the treatment tool is in contact with (which side it is emerging from). Therefore, the treatment tool mask 127 in the above-mentioned color-processed image 124 is combined. In this way, it becomes clear which side in the image the treatment tool is emerging from.

Then, as shown in FIG. 10B, the bounding box Bx1 corresponding to the initial inference mask 126 is expanded to the bounding box Bx2 of the combined image 125.

The process of deleting background regions in a color-processed image is shown as a flow diagram in Figure 11. A color-reduced image 122 is generated from the above-mentioned surgical image 121, and a binary image 123 is then generated. In addition to this process, a grayscale image 122g is generated from the surgical image 121, and a binary image 123g is generated from the grayscale image 122g.

Then, the common portion between binarized image 123 and binarized image 123g is obtained. As a result, the image of the bright parts of the biological tissue other than the treatment tool is deleted, and a corrected image (brightness-adjusted image 128) is generated in which only the treatment tool is clearly shown. This is performed to prevent the bright parts of the biological tissue from being mistakenly recognized as the treatment tool.

Furthermore, the treatment tool image in the color-processed image is identified by comparing the treatment tool image with the inferred treatment tool image. This process is shown as a flow diagram in FIG. 12. A color-reduced image 122 is generated from the above-mentioned surgical image 121, and then a binarized image 123 is generated. In addition to this flow, an inferred image 129p is generated from a segmentation mask obtained through the deep learning model. This includes all treatment tool areas contained in the surgical image 121. Since each treatment tool is distinguished in the inference result of instance segmentation, the treatment tool area is deleted from the inferred image 129p, and a non-target treatment tool image 129 is generated.

Then, the area that overlaps with the non-target treatment tool image 129 is deleted from the binarized image 123, and a corrected image (overlap adjusted image 130) is generated in which treatment tools other than the target treatment tool have been deleted.

In addition, the image completion unit 250 of the embodiment reduces the image size of the surgical image to compress the time required for image processing. Specifically, the image size is reduced for the binarized image 123 generated from the surgical image 121 via the color-reduced image 122. Figure 13 is a schematic diagram of an example of image size reduction. Figure 13(A) is a schematic diagram of the binarized image 123 at "actual size". Figure 13(B) is a schematic diagram of the binarized image 123 in (A) with one side at "1/4 size". Figure 13(C) is a schematic diagram of the binarized image 123 in (A) with one side at "1/8 size".

Once an image has been binarized, even if the image resolution is reduced, it does not affect the identification of the treatment tool area. By reducing the image size, it is possible to shorten the time required for image processing such as removing background areas. As shown in Figure 13, the binarized image with reduced image size is used as the binarized image for each process in Figures 8 to 12 described above.

The tip determination unit 261 detects the tip of the treatment tool from the treatment tool segmentation mask, which is the output of the treatment tool extraction unit, and determines whether or not there is an opening at the tip.

The schematic diagram in FIG. 14(A) shows a rectangular region 138 cut out as the smallest rectangle that encompasses the treatment tool region, with the long axis direction of the rectangle aligned horizontally. The treatment tool in rectangular region 138 has its tip open on the right side of the page. It is shaped like a sideways V. The white parts in the diagram correspond to the treatment tools.

In the schematic diagram of Figure 14 (B), the base of the horizontal V-shape (the space between the tips of forceps, etc.) is enlarged. Note that in the illustration, black and white are inverted, with the black parts in the illustration corresponding to the treatment tool. Then, ends e1 and e2 of the background, which is the white parts in the illustration, are recognized. The number of pixels for each of ends e1 and e2 is calculated. In the example shown, end e1 is narrower, so the number of pixels is less compared to end e2. From this, it is clear that end e2 is wider than end e1. Therefore, it is clear that there is an opening on the side of end e2 in the treatment tool.

The direction detection unit 262 determines the extension direction of the treatment tool based on the shape of the treatment tool segmentation mask, which is the output of the treatment tool extraction unit.

The schematic diagram in Figure 15 (A) shows a segmentation mask of a treatment tool whose tip opens and closes. The treatment tool in rectangular area 138 has its tip open to the right of the page. It is shaped like a horizontal V. The white part in the figure corresponds to the treatment tool. The root part (O) of the horizontal V-shape and the ends (P) and (Q) that spread out from the root part are defined. Here, the vector (OP) of the root part (O) and the end part (P) and the vector (OQ) of the root part (O) and the end part (Q) are calculated. Then, the directional vector (v) indicating the extension direction of the treatment tool is the opposite of the combined direction of vectors (OP) and (OQ). In other words, it is to the left of the page.

The schematic diagram in Figure 15 (B) shows a segmentation mask for a treatment tool whose tip does not open or close, with the white part in the diagram corresponding to the treatment tool. The areas (number of pixels) of the left and right segmentation masks are compared, with the center (Oi) of the smallest rectangle cut out as shown as the boundary. The direction with the larger area is determined to be the extension direction of the treatment tool. This is based on the structure of the treatment tool, in which the shaft of the treatment tool is thicker than the tip. Next, the widest position of the segmentation mask becomes the origin (O), and the intersection of the straight line extended from the origin (O) in the extension direction of the treatment tool determined earlier and the smallest rectangle becomes the end (R). The directional vector (v) indicating the extension direction of the treatment tool becomes the vector (OR).

Furthermore, the treatment tool area complementation unit 200 includes a brightness detection unit 263 that detects the brightness around the treatment tool area in the surgical image and detects the extension direction of the treatment tool from the brightness level. With the center of gravity O of the treatment tool segmentation mask as the origin, a straight line is drawn in the long axis direction of the rectangular area, and points on the line near the outside of the treatment tool mask are designated as X and Y (to make the position of the points easier to understand, point (Y) in Figure 16 is displayed off the straight line). Here, since point (X) on the treatment tool shaft has a lower brightness than point (Y) in the biological tissue, it can be seen that the extension direction of the treatment tool is in the direction of OX.

In addition, the determination of the extension direction of the treatment tool shown in the schematic diagram of FIG. 17 may be included. In FIG. 17(A), the segmentation mask of the treatment tool is displayed as if it is floating in the inferred treatment tool image 141. Then, the distances K1 and K2 between the edge of the screen and the image corresponding to the treatment tool are obtained. From a mutual comparison of the distances, the side with the closer distance (K1 side) becomes the root side of the treatment tool, and the extension direction is determined. As shown in FIG. 17(B), an extension image 142 with the extension direction corrected is generated.

FIG. 18 corresponds to an explanation of the processing in the extension unit 264. In FIG. 18(A), the inference mask 133 is corrected based on the extension direction determined by the direction detection unit in the combined image 125. Then, the origin (O) and direction vector (v) of the image of the treatment tool in the inference mask 133 are defined, and it is extended to the edge of the screen as shown in the addition unit 137 in FIG. 18(B). The width of the addition unit 137 is defined by the number of pixels of the cross section where the perpendicular line of the direction vector at the origin (O) intersects with the inference mask 133.

In FIG. 18(B), the combination unit 265 combines the inference treatment tool image (inference mask 133) acquired through the deep learning model with the addition unit 137 to generate a combined image (inference mask 134). In addition, the expansion unit 266 generates a bounding box (expanded bounding box 136) to match the combined image (inference mask 134).

The techniques described in Figures 14 to 17 may be used individually or in combination as appropriate. The optimal technique is selected taking into consideration various influences such as the treatment site (influence of organs and tissues) in the surgical image 121 and the type of treatment tool used.

In particular, the endoscopic image treatment tool detection device (endoscopic image treatment tool tip detection device) 10 of the embodiment generates a color-processed image to be used for the bounding box before generating a bounding box to improve the recognition accuracy of the treatment tool, making it easy to obtain information about which side of the screen in the endoscopic image the treatment tool is appearing from, which was not adequately handled by previous bounding box processing alone. For this reason, the treatment tool detection device 10 of the embodiment is expected to contribute to supporting laparoscopic surgery using artificial intelligence.

Now, using the flowcharts of Figures 19 to 23, the endoscopic image treatment tool detection method and the endoscopic image treatment tool detection program in the endoscopic image treatment tool detection device (endoscopic image treatment tool tip detection device) 10 will be described. The endoscopic image treatment tool detection method is executed by the CPU 111 of the processing device (computer) 105 based on the endoscopic image treatment tool detection program. The endoscopic image treatment tool detection program causes the processing device (computer) 105 in Figure 3 to execute an image acquisition function, an image recognition function, a treatment tool extraction function, and a tip detection function, a treatment tool area completion function, a root identification function, an end position detection function, an angle detection function, and a tip point detection function, an image completion function, a rectangle completion function, a color reduction function, a binarization function, an extraction function, a combination function, and an extension function, and a tip determination function, a direction detection function, a brightness detection function, an extension function, a combination function, and an extension function. Each function overlaps with the description of the treatment tool detection device 10 described above, so details will be omitted.

As can be seen from the flow chart in FIG. 19, the processing of the processing device (computer) 105 (CPU 111) includes various steps such as an image acquisition step (S150), an image recognition step (S160), a treatment tool extraction step (S170), and a treatment tool area completion step (S200). Of course, various steps necessary for the operation of the processing device (computer) 105 (CPU 111) itself, as well as output steps necessary for information output, a tip detection step (S180), etc. are naturally included.

The image acquisition function acquires a surgical image (S150; image acquisition step). The image recognition function performs image recognition of the surgical image using a pre-trained deep learning model (S160; image recognition step). The treatment tool extraction function generates a treatment tool area in the surgical image consisting of a segmentation mask and a bounding box in the treatment tool image (S170; treatment tool extraction step). The treatment tool area completion step (S200) executes each process of the image completion step (S250) or rectangle completion step (S260) in FIG. 21.

As can be seen from the flow chart of the tip detection step (S180) in FIG. 20, the processing by the processing device (computer) 105 (CPU 111) includes various steps such as a root identification step (S210), an end point detection step (S220), an angle detection step (S230), and a tip point detection step (S240).

The root identification function identifies as the root the side of the image edge region that has the largest coverage area between each of the four sides of the image edge region in the surgical image and the segmentation mask (S210; root identification step). The end position detection function detects the end portion of the treatment tool (end position detection step; S220). The angle detection function detects the angle of the treatment tool (angle detection step; S230). The tip point detection function detects the tip point of the treatment tool (tip point detection step; S240).

As shown in the flowchart of FIG. 21, the treatment tool area completion step (S200) is selective to execute an image completion step (S250) as shown in FIG. 21(A) or a rectangle completion step (S260) as shown in FIG. 21(B). Note that in the treatment tool detection device for endoscopic images (treatment tool tip detection device for endoscopic images) 10, the treatment tool area completion step (S200) can also execute both the image completion step (S250) and the rectangle completion step (S260).

FIG. 22 is a flowchart of the image completion step (S250) in the treatment tool area completion step (S200), and the processing by the processing device (computer) 105 (CPU 111) includes various steps of a color reduction step (S251), a binarization step (S252), an extraction step (S253), a combination step (S254), and an expansion step (S255).

The color reduction function reduces a specified color from the surgical image to generate a color-reduced image (S251; color reduction step). The binarization function binarizes the color-reduced image (S252; binarization step). The extraction function removes areas unrelated to the target treatment tool present in the surgical image and extracts the treatment tool image from the color-processed image (S253; extraction step). The combination function combines the inferred treatment tool image obtained through the deep learning model with the treatment tool image obtained from the color-processed image to generate a combined image (S254; combination step). The expansion function generates the bounding box to match the combined image (S255; expansion step).

FIG. 23 is a flow chart of the rectangle completion step (S260) in the treatment tool area completion step (S200), and the processing by the processing device (computer) 105 (CPU 111) includes various steps, such as a tip determination step (S261), a direction detection step (S262), a brightness detection step (S263), an extension step (S264), a combination step (S265), and an expansion step (S266).

The tip determination function detects the tip of the treatment tool from the treatment tool segmentation mask, which is the output of the treatment tool extraction unit, and determines whether or not there is an opening at the tip (S261; tip determination step). The direction detection function detects the extension direction of the treatment tool from the treatment tool segmentation mask, which is the output of the treatment tool extraction unit (S262; direction detection step). The brightness detection function detects the brightness around the treatment tool image in the color-processed image and detects the extension direction of the treatment tool from the high and low brightness (S263; brightness detection step). The extension function extends the minimum rectangle in the extension direction to generate an extended treatment tool image (S264; extension step). The combination function combines the extended treatment tool image with the inferred treatment tool image obtained through the deep learning model to generate a combined image (S265; combination step). The expansion function generates a bounding box to match the combined image (S266; expansion step).

The computer program of the present invention described above may be recorded on a processor-readable recording medium, and the recording medium may be a "non-transitory tangible medium" such as a disk, card, semiconductor memory, or programmable logic circuit.

The computer program can be implemented using, for example, a scripting language such as ActionScript or JavaScript (registered trademark), an object-oriented programming language such as Objective-C or Java (registered trademark), or a markup language such as HTML5.

10 Treatment tool detection device 60 Inference image 101, 102, 401, 601 Treatment tool 103 Endoscope 104 Affected area 105 Processing device 106 Endoscope display 107 Patient 108 Trocar 111 CPU
112 ROM
113 RAM
114 Storage unit 115 I/O
121 Surgical image 122 Color-reduced image 122g Grayscale image 123 Binarized image 123a Treatment tool area 123g Binarized image based on grayscale image 124 Tone-processed image 125 Combined image 126, 133, 134 Inference mask 127 Treatment tool mask 128 Brightness-adjusted image 129 Non-target treatment tool image 129p Inference image 130 Overlap-adjusted image 133, 134 Inference mask 135, 136 Bounding box 137 Addition section 138 Rectangular area 141 Inferred treatment tool image 142 Extension image 150 Image acquisition section 160 Image recognition section 170 Treatment tool extraction section 180 Tip detection section 200 Treatment tool area complementation section 210 Root identification section 220 End portion detection section 230 Angle detection section 240 Tip point detection section 250 Image complementation section 251 Color reduction unit 252 Binarization unit 253 Extraction unit 254 Combination unit 255 Expansion unit 260 Rectangle completion unit 261 Tip determination unit 262 Direction detection unit 263 Luminance detection unit 264 Extension unit 265 Combination unit 266 Expansion unit 402 Segmentation mask 403, Bx1, Bx2 Bounding box 602 Image edge line 603 Image edge 604, 605 Area

Claims (11)

an image acquisition unit for acquiring a surgical image;
An image recognition unit that performs image recognition of the surgical image using a pre-trained deep learning model;
a treatment tool extraction unit that generates a treatment tool area in the surgical image, the treatment tool area being composed of a segmentation mask and a bounding box in the treatment tool image;
a treatment tool region complementing unit that complements a missing treatment tool region due to an obstacle present in the surgical image,
The treatment tool region complement portion includes:
an image complementing unit that performs color processing on the surgical image to generate a color-processed image and complements a treatment tool region using the color-processed image;
The image completion unit, when generating the color-tone-processed image,
a color reduction unit that reduces a predetermined color from the surgical image to generate a reduced color image;
a binarization unit that binarizes the color-reduced image;
an extraction unit that deletes an area that is unrelated to the treatment tool and exists in the surgical image, and extracts the treatment tool image from the color-tone-processed image;
A combination unit that combines an inferred treatment tool image acquired through the deep learning model with the treatment tool image acquired from the color processing image to generate a combined image;
an extension unit that generates the bounding box to fit the combined image;
A treatment tool detection device for an endoscopic image, comprising: The treatment tool region complement portion includes:
a rectangle complementing unit that generates a minimum rectangle surrounding a treatment tool region obtained by image processing for a treatment tool segmentation mask that is an output of the treatment tool extraction unit, and performs complementation using the minimum rectangle;
The rectangle complementing unit, when generating a minimum rectangle surrounding the treatment tool region,
a tip determination unit that detects a tip of a treatment tool from a segmentation mask of the treatment tool that is an output of the treatment tool extraction unit and determines whether or not an opening of the tip is present;
a direction detection unit that detects an extension direction of the treatment tool from a segmentation mask of the treatment tool that is an output of the treatment tool extraction unit;
a brightness detection unit that detects the brightness of the area around the treatment tool in the surgical image and detects the extension direction of the treatment tool based on an inference mask from the brightness level;
an extension unit that extends the minimum rectangle in the extension direction to generate an extended treatment tool image;
A combination unit that combines the inference treatment tool image acquired through the deep learning model with the extension treatment tool image to generate a combination image;
an extension unit that generates the bounding box to fit the combined image;
The treatment tool detection device for an endoscopic image according to claim 1 , further comprising: The treatment tool region complement portion includes:
The image completion unit;
The treatment tool detection device according to claim 1 , further comprising the rectangular complementing section, and selectively outputs one of the outputs. The endoscopic image treatment tool detection device according to claim 1, characterized in that the image completion unit deletes background areas in the color-processed image from a brightness-adjusted image generated by adjusting the brightness of the surgical image. The treatment tool detection device for an endoscopic image according to claim 1, characterized in that the image complementing unit identifies the treatment tool image in the color-processed image by comparing the treatment tool image with the inferred treatment tool image. The endoscopic image treatment tool detection device according to claim 1, characterized in that the image completion unit reduces the image size of the surgical image to compress the time required for image processing. an image acquisition unit for acquiring a surgical image;
An image recognition unit that performs image recognition of the surgical image using a pre-trained deep learning model;
a treatment tool extraction unit that generates a segmentation mask and a bounding box of a treatment tool in the surgical image;
a treatment tool region complementing unit that complements a missing treatment tool region due to an obstacle present in the surgical image;
a tip detection unit that detects a tip of the treatment tool by using the generated segmentation mask and the generated bounding box,
The tip detection unit is
a root identifying unit that identifies, as a root, a side of an image edge region that has a maximum coverage area between each of four sides of an image edge region present in the surgical image and the segmentation mask;
an end portion detection unit for detecting an end portion of the treatment tool;
An angle detection unit that detects an angle of the treatment tool;
a tip point detection unit for detecting a tip point of the treatment tool, A computer in an endoscopic image treatment tool detection device
An image acquisition step of acquiring a surgical image;
An image recognition step of performing image recognition of the surgical image using a pre-trained deep learning model;
a treatment tool extraction step of generating a treatment tool region in the surgical image, the treatment tool region being composed of a segmentation mask and a bounding box in the treatment tool image;
a treatment tool area complementing step of complementing a missing treatment tool area due to an obstacle present in the surgical image;
The treatment tool region complementing step includes:
an image complementing step of performing color processing on the surgical image to generate a color-processed image and complementing a treatment tool region using the color-processed image;
The image complementation step includes, when generating the color-tone processed image,
a color reduction step of reducing predetermined colors from the surgical image to generate a color-reduced image;
a binarization step of binarizing the color-reduced image;
an extraction step of deleting an area that is not related to the treatment tool and is present in the surgical image, and extracting the treatment tool image from the color-processed image;
A combination step of combining an inferred treatment tool image acquired through the deep learning model with the treatment tool image acquired from the color processing image to generate a combined image;
and an expansion step of generating the bounding box in accordance with the combined image. The treatment tool region complementing step includes:
A rectangle complementation step is performed in which a minimum rectangle that encloses the treatment tool region obtained by image processing is generated for the treatment tool segmentation mask that is an output of the treatment tool extraction step, and the minimum rectangle is used to complement the treatment tool region;
The rectangle complementing step includes, when generating a minimum rectangle surrounding the treatment tool region,
a tip determination step of detecting a tip of a treatment tool from a segmentation mask of the treatment tool which is an output of the treatment tool extraction step, and determining whether or not an opening of the tip exists;
a direction detection step of detecting an extension direction of the treatment tool from a segmentation mask of the treatment tool which is an output of the treatment tool extraction step;
a brightness detection step of detecting a brightness around the treatment tool in the surgical image and detecting an extension direction of the treatment tool based on an inference mask from the brightness level;
an extension step of extending the minimum rectangle in the extension direction to generate an extended treatment tool image;
A combination step of combining the inference treatment tool image acquired through the deep learning model with the extension treatment tool image to generate a combination image;
augmenting the bounding box to fit the combined image;
The treatment tool detection method for an endoscopic image according to claim 8, further comprising: A computer in an endoscopic image treatment tool detection device
an image acquisition function for acquiring a surgical image;
An image recognition function that performs image recognition of the surgical image using a pre-trained deep learning model;
a treatment tool extraction function for generating a treatment tool region in the surgical image, the treatment tool region being composed of a segmentation mask and a bounding box in the treatment tool image;
A treatment tool area complementing function for complementing a missing treatment tool area due to an obstacle present in the surgical image is realized.
The treatment tool area complement function is
an image complementing function that performs color processing on the surgical image to generate a color-processed image and complements a treatment tool region using the color-processed image;
The image complementing function, when generating the color-tone processed image,
a color reduction function for reducing a predetermined color from the surgical image to generate a reduced color image;
A binarization function for binarizing the color-reduced image;
an extraction function for extracting the treatment tool image from the color-processed image by deleting an area that is unrelated to the treatment tool and exists in the surgical image;
A combination function of combining an inferred treatment tool image acquired through the deep learning model with the treatment tool image acquired from the color processing image to generate a combined image;
and an extended function for generating the bounding box in accordance with the combined image. The treatment tool area complement function is
A rectangle completion function is realized by generating a minimum rectangle surrounding a treatment tool region obtained by image processing for a segmentation mask of the treatment tool, which is an output of the treatment tool extraction function, and performing completion using the minimum rectangle;
The rectangle complement function, when generating a minimum rectangle surrounding the treatment tool area,
a tip determination function for detecting a tip of a treatment tool from a segmentation mask of the treatment tool which is an output of the treatment tool extraction function and determining whether or not an opening exists at the tip;
a direction detection function for detecting the extension direction of the treatment tool from a segmentation mask of the treatment tool which is an output of the treatment tool extraction function;
A brightness detection function for detecting the brightness of the periphery of the treatment tool in the surgical image and detecting the extension direction of the treatment tool based on an inference mask from the brightness level;
an extension function for extending the minimum rectangle in the extension direction to generate an extended treatment tool image;
A combination function of combining the inference treatment tool image acquired through the deep learning model with the extension treatment tool image to generate a combination image;
an extension that generates the bounding box for the combined image;
The treatment tool detection program for an endoscopic image according to claim 10, further comprising:

Images