WO2025069575A1 - Information processing device and control program - Google Patents

Abstract

This information processing device combines endoscopic video captured by an imaging device attached to an endoscope into a three-dimensional model of a treatment site generated from image data of the treatment site captured in advance, and displays a composite three-dimensional model obtained by combining the endoscopic video of the treatment site.

Description

Information processing device and control program

This disclosure relates to an information processing device and a control program.

In order to reduce the burden on patients, procedures using endoscopes have become widespread. When removing tumors or other lesions using endoscopic surgery, correctly identifying the treatment site is an important factor in, for example, reducing the procedure time.

JP 2015-107268 A discloses an endoscopic observation support device that displays a three-dimensional image of a tubular organ by superimposing a lesion extracted based on the path of the endoscope, the endoscopic image acquired by the endoscope, and the observation position of an ultrasound probe attached to the tip of the endoscope.

JP 2006-61274 A discloses an endoscope system that generates a virtual endoscope image by volume rendering and displays the virtual endoscope image superimposed on an actual endoscope image.

JP Patent Publication No. 2013-202313 discloses a surgery support device that displays the movement of surgical tools on a three-dimensional image in real time, displays the status of organ resection by the surgical tools, and issues warnings according to the position and speed of the surgical tools.

In order to determine the treatment area, a method is used in which the positional information of blood vessels and tumors extracted from image data taken by a radiological imaging device such as a CT (Computed Tomography) scanner is combined with the images taken by the endoscope. However, because the images taken by the endoscope are two-dimensional, there is no sense of depth in the images. This makes it difficult to grasp the sense of distance to the treatment area.

The present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing device and control program that can three-dimensionally grasp the difference between the positional relationship of the treatment site shown by image data of the treatment site and the positional relationship of the treatment site shown by an image captured by an endoscope.

The information processing device according to the first aspect includes a processor, and the processor controls the synthesis of a three-dimensional model of the treatment area generated from image data of the treatment area captured in advance with an image of the treatment area captured by an imaging device attached to an endoscope, and the display of the synthesized three-dimensional model with the image of the treatment area combined.

In the information processing device according to the second aspect, in the information processing device according to the first aspect, the processor controls the display of related information associated with each stage of treatment for the treatment area together with the three-dimensional model or the composite three-dimensional model, depending on the stage of treatment for the treatment area.

In the information processing device according to the third aspect, in the information processing device according to the second aspect, in the preoperative marking stage in which a treatment plan for the treatment area is formulated, the processor controls to superimpose and display at least one of the following on the corresponding position of the three-dimensional model: the position of the affected area, the preplanned resection line of the treatment area including the affected area, the resection range to be resected along the resection line, the position of the blood vessel passing through the treatment area, the preplanned ligation position of the blood vessel, and the blood flow cessation area where blood flow is halted by ligation of the blood vessel.

In the information processing device according to the fourth aspect, the processor in the information processing device according to the third aspect performs control to display the remaining volume of the organ containing the diseased area together with the three-dimensional model when the diseased area is resected along the resection line.

In the information processing device according to the fifth aspect, in the information processing device according to the second aspect, in the intraoperative marking stage in which the preoperative planning information determined in the preoperative marking stage in which a treatment plan for the treatment site is formulated is reconfirmed based on the characteristics of the treatment site obtained from the image of the treatment site synthesized into the synthetic three-dimensional model, the processor performs control such that, in response to a change in the resection line for the treatment site including the diseased area determined in the preoperative marking stage, the resection line in the preoperative marking stage and the updated resection line are each superimposed and displayed on the corresponding positions of the synthetic three-dimensional model, and the ligation position of the blood vessel determined in the preoperative marking stage is updated, and the updated ligation position of the blood vessel is superimposed and displayed on the corresponding position of the synthetic three-dimensional model.

In the information processing device according to the sixth aspect, in the information processing device according to the second aspect, in the case of a resection stage in which an affected area included in the treatment area is resected, the processor performs control to superimpose and display, on the corresponding positions of the composite three-dimensional model, at least one of the following: the position of the affected area, the resection line of the treatment area including the affected area, the resection range to be resected along the resection line, the position of the blood vessel passing through the treatment area, the ligation position of the blood vessel, the blood flow cessation area in which blood flow is halted by ligation of the blood vessel, and the position of the resection mark placed on the treatment area by a medical professional as a guide for resecting the affected area.

In the information processing device according to the seventh aspect, in the information processing device according to the sixth aspect, the processor controls to associate an identifier that identifies the ligation position of the blood vessel with the ligation position of the blood vessel and display it together with the synthetic three-dimensional model.

In the information processing device according to the eighth aspect, the identifier indicates the order in which blood vessels are ligated.

In the information processing device according to the ninth aspect, in the information processing device according to the sixth aspect, the processor performs control to distinguish between the ligation positions of blood vessels that have already been ligated and the ligation positions of blood vessels that have not yet been ligated, and to superimpose and display them at the corresponding positions of the composite three-dimensional model.

In the information processing device according to the tenth aspect, in the information processing device according to the ninth aspect, the processor performs control to distinguish between blood vessels in the blood flow stop region where blood flow has been stopped by ligation and blood vessels in the blood flow stop region where blood flow is present because the blood vessels have not yet been ligated, and to display them superimposed on the corresponding positions of the composite three-dimensional model.

In the information processing device according to the eleventh aspect, the processor in the information processing device according to the tenth aspect performs control to superimpose and display a video showing blood flow on the blood vessels in the blood flow stop region where blood flow is present.

In the information processing device according to the twelfth aspect, in the information processing device according to the ninth aspect, the processor controls to display the number of ligation positions of blood vessels that have already been ligated and the number of ligation positions of blood vessels that have not yet been ligated together with the composite three-dimensional model.

In the information processing device according to the thirteenth aspect, the processor in the information processing device according to the sixth aspect performs control to update the range of the blood flow halt region according to the ligation state of the blood vessel.

In the information processing device according to the 14th aspect, in the information processing device according to the 6th aspect, the processor performs control to output a warning if the resection line of the treatment area and the resection execution line represented by the resection mark are separated by a predetermined distance or more.

In the information processing device according to the fifteenth aspect, the processor of the information processing device according to the sixth aspect performs control to change the display form of the affected area, the resection range, the organ including the treatment site, and the blood vessels passing through the treatment site, which are displayed superimposed on the corresponding positions of the composite three-dimensional model, in accordance with instructions from a medical professional.

In the information processing device according to the 16th aspect, the processor of the information processing device according to any one of the first to 15th aspects performs control to display the composite 3D model from an angle specified by a medical professional.

In the information processing device according to the 17th aspect, in the information processing device according to the 16th aspect, the processor performs control to display the composite 3D model to which the new image has been added while compositing new images of the treatment site captured by the imaging device as the endoscope moves into the composite 3D model along with the movement of the endoscope.

In the information processing device according to the 18th aspect, in the information processing device according to the 17th aspect, the processor stores in a storage device historical information regarding the operation of an instrument performed by a medical professional while performing treatment on the treatment area together with the composite 3D model, and performs control to display the treatment details for the treatment area using the historical information and the composite 3D model in response to instructions from the medical professional.

The control program according to the 19th aspect is a program for causing a computer to execute control to combine a three-dimensional model of the treatment area generated from image data of the treatment area previously photographed with an image of the treatment area photographed by an imaging device attached to an endoscope, and to display a composite three-dimensional model into which the image of the treatment area has been combined.

According to the present disclosure, it is possible to three-dimensionally grasp the difference between the positional relationship of the treatment site shown by image data capturing the treatment site and the positional relationship of the treatment site shown by an image captured by an endoscope.

FIG. 1 is a diagram illustrating an example of the configuration of an endoscopic treatment system. FIG. 1 is a diagram showing an example of insertion of an endoscope. FIG. 2 is a diagram showing an example of an endoscopic image. FIG. 2 is a diagram illustrating an example of a three-dimensional model. 13 is an example showing an example of a composite three-dimensional model. FIG. 1 illustrates an example of the configuration of an information processing device. 13 is a flowchart showing an example of the flow of a preoperative marking process. FIG. 13 is a diagram showing a display example of preoperative planning information. 11 is a flowchart showing an example of the flow of a synthetic 3D model generation process. 13 is a flowchart showing an example of the flow of intraoperative sign processing. FIG. 13 is a diagram showing an example of display of intraoperative plan information. 13 is a flowchart showing an example of the flow of resection support processing. FIG. 13 is a diagram showing a display example of resection support information.

The present embodiment will now be described with reference to the drawings. Note that the same components and processes are given the same reference numerals throughout the drawings, and duplicate explanations will be omitted. Also, the dimensional ratios in the drawings have been exaggerated for the sake of explanation, and may differ from the actual ratios.

Furthermore, the stage before the endoscope is inserted into the patient's body and the procedure is performed is referred to as "preoperative," and the stage after the endoscope has been inserted into the patient's body is referred to as "intraoperative."

FIG. 1 is a diagram showing an example of the configuration of an endoscopic surgery system 100. As an example, the endoscopic surgery system 100 includes an image capturing device 1, an information processing device 2, and an endoscope 3.

The imaging device 1 is a device that captures images of the treatment site from outside the body in order to understand the positional relationship of the treatment site before inserting an endoscope 3 into the patient's body and removing a lesion such as a tumor (also called the "affected area"). Specifically, a CT scanning device that captures images of the treatment site using radiation, and an ultrasound imaging device that captures images of the treatment site using ultrasound are used as the imaging device 1. In this disclosure, as an example, an image of the treatment site is captured using a CT scanning device. The imaging device 1 converts the captured images into image data.

In this disclosure, the treatment site refers to an area that includes a lesion such as a tumor. In addition, the positional relationship of the treatment site refers to the positional relationship between the treatment site and the areas inside and around the treatment site, including, for example, the size of the treatment site, the position of the treatment site, the positions of blood vessels that pass through the treatment site, and the positional relationship with other organs around the treatment site.

The image data of the treatment area captured by the image capture device 1 is sent to the information processing device 2.

The endoscope 3 is a medical instrument that is inserted into the body through holes made in the nose, mouth, anus, abdomen, or chest, and captures images of organs 4 from inside the body using an imaging device (not shown) attached to the endoscope 3. Figure 2 is a diagram showing an example of the insertion of the endoscope 3 that captures images of organs 4 through a hole made in the abdomen. Also, Figure 3 is a diagram showing an example of an image captured by the endoscope 3. Hereinafter, the image captured by the endoscope 3 will be referred to as endoscopic image 5. A treatment tool, which is an example of an instrument for tissue sampling and tissue resection, can be attached to the tip of the endoscope 3.

The endoscope 3 notifies the information processing device 2 of the captured endoscopic video 5. Note that the endoscopic video 5 may be a video or a still image, and may be monochrome or color, but in this disclosure, as an example, the endoscopic video 5 is a color video.

The information processing device 2 in FIG. 1 generates a three-dimensional model 6 of the treatment area using image data received from the image capture device 1, and synthesizes the endoscopic image 5 received from the endoscope 3 onto the surface of the generated three-dimensional model 6 to generate a composite three-dimensional model 7 in which the image of the treatment area is synthesized onto the three-dimensional model 6.

FIG. 4 is a diagram showing an example of a three-dimensional model 6 of the treatment area. FIG. 5 is an example showing an example of a composite three-dimensional model 7 of the treatment area. As shown in FIG. 5, the information processing device 2 composites the endoscopic video 5 onto the surface of the three-dimensional model 6 generated from image data. The endoscopic video 5 is not composited onto the surface of the three-dimensional model 6 where the endoscopic video 5 is not present, and the surface of the treatment area represented by the three-dimensional model 6 is displayed.

The three-dimensional model 6 shown in FIG. 4 and the composite three-dimensional model 7 shown in FIG. 5 do not show the parts that pass through the inside of the treatment area, such as blood vessels, but because the image capturing device 1 also captures the structure inside the treatment area, it is possible for the three-dimensional model 6 and composite three-dimensional model 7 to show the parts that pass through the inside of the treatment area.

Next, we will explain an example of the configuration of the information processing device 2. Figure 6 is a diagram showing an example of the configuration of the information processing device 2.

As shown in FIG. 6, the information processing device 2 includes a control unit 10, a memory unit 12, an operation unit 13, a display unit 14, and an I/F (Interface) unit 15. The control unit 10, the memory unit 12, the operation unit 13, the display unit 14, and the I/F unit 15 are connected via a bus 16, and exchange various types of information with each other.

The control unit 10 controls the operation of the information processing device 2 based on instructions from a medical professional. The control unit 10 includes a CPU (Central Processing Unit) 10A, which is an example of a processor, a ROM (Read Only Memory) 10B, and a RAM (Random Access Memory) 10C.
M10B stores in advance various programs including a control program 11 that is read by the CPU 10A to generate the composite 3D model 7 and control the display of the generated composite 3D model 7, and various parameters that the CPU 10A refers to when controlling the operation of the information processing device 2. RAM 10C is used as a temporary work area for the CPU 10A.

The memory unit 12 stores, for example, image data, an endoscopic video 5, a three-dimensional model 6 of the treatment area generated by the control unit 10 from the image data, and a composite three-dimensional model 7 generated by the control unit 10 using the three-dimensional model 6 and the endoscopic video 5. The memory unit 12 is an example of a storage device in which stored information is maintained even if the power supplied to the memory unit 12 is cut off, and for example, a semiconductor memory such as an SSD (Solid State Drive) is used, but a hard disk may also be used.

The operation unit 13 is used by medical personnel to input instructions to the information processing device 2 and various parameters referenced when the information processing device 2 operates. There are no restrictions on the type of operation on the operation unit 13, and operations can be accepted, for example, from a keyboard, a touch panel, a touch pen, a mouse, etc.

The display unit 14 displays information related to the treatment using the endoscope 3, such as the synthetic 3D model 7, on the screen 30 (see, for example, FIG. 8).

The I/F unit 15 has a communication function and receives image data from the image capturing device 1, which is an example of an external device connected to a communication line (not shown) such as a LAN (Local Area Network), via wireless or wired communication. The I/F unit 15 also has a video input terminal and acquires the endoscopic video 5 from the endoscope 3. When the endoscope 3 transmits the endoscopic video 5 via the communication line, the video input terminal of the I/F unit 15 is not necessary.

Next, the process of the information processing device 2 that generates the composite 3D model 7 will be described in detail.

FIG. 7 is a flowchart showing an example of the flow of preoperative marking processing executed by the information processing device 2 when a three-dimensional model generation instruction to start generating a three-dimensional model 6 is received by a medical professional operating the operation unit 13. The CPU 10A of the information processing device 2 reads the control program 11 from the ROM 10B and executes the preoperative marking processing. It is assumed that the memory unit 12 has stored in advance image data of the patient's treatment site captured by the image capture device 1.

When performing a procedure on a patient using an endoscope 3, the procedure involves multiple stages. The stage in which a treatment plan for the treatment area is formulated based on image data of the patient captured by the image capture device 1 before the endoscope 3 is inserted into the patient is called the "preoperative labeling stage." The three-dimensional model 6 representing the treatment area is generated before the endoscope 3 is inserted into the patient, so the preoperative labeling process shown in Figure 7 is an example of processing in the preoperative labeling stage.

First, in step S10, the CPU 10A acquires image data of the patient's treatment area from the memory unit 12.

In step S20, the CPU 10A generates a three-dimensional model 6 of the treatment area using the image data acquired by the processing in step S10. The three-dimensional model 6 is generated, for example, by using three-dimensional model generation software provided by the developer of the image capture device 1. The CPU 10A displays the generated three-dimensional model 6 on the screen 30.

While looking at the three-dimensional model 6 displayed on the screen 30, the medical staff confirms the size of the affected area and the position of the blood vessels, and considers a treatment plan before performing the treatment, such as which blood vessels should be ligated and where to ligate them to remove the affected area, and how to excise the affected area. The medical staff notifies the CPU 10A of the contents of the treatment plan they have considered via the operation unit 13 as preoperative planning information.

The preoperative planning information includes at least one of the following: the location of the affected area, a resection line at the treatment site including the affected area, the resection range to be resected along the planned resection line, the location of blood vessels passing through the treatment site, the planned ligation location of the blood vessels, and the blood flow cessation area where blood flow will be halted by ligating the blood vessels. The preoperative planning information is an example of relevant information associated with the preoperative labeling stage.

In step S30, the CPU 10A acquires the notified preoperative planning information.

In step S40, the CPU 10A displays the preoperative planning information acquired by the processing of step S30 on the screen 30, superimposed on the corresponding position of the three-dimensional model 6.

FIG. 8 is a diagram showing an example of the display of preoperative planning information. For ease of explanation, the affected area is represented as "affected area 24", the resection line as "resection line 22", the blood vessel as "blood vessel 20", the ligation position of blood vessel 20 as "ligation position 21", and the blood flow stopped area as "blood flow stopped area 25". In particular, the affected area 24 estimated from the three-dimensional model 6 is represented as "affected area 24A", the resection line 22 estimated from the three-dimensional model 6 as "resection line 22A", and the ligation position 21 estimated from the three-dimensional model 6 as "ligation position 21A".

When the diseased area 24A is resected along the resection line 22A, the CPU 10A displays the remaining volume of the organ 4 including the diseased area 24A on the screen 30 that displays the preoperative planning information together with the three-dimensional model 6. The remaining volume of the organ 4 is the volume of the organ 4 that does not include the diseased area 24A, with the resection line 22A as the boundary. Therefore, the CPU 10A calculates the entire volume of the organ 4 including the diseased area 24A calculated from the three-dimensional model 6, and the remaining volume of the organ 4 from the resection line 22A indicated by the medical professional, and displays the calculated remaining volume of the organ 4 on the screen 30.

There are no restrictions on the position where the remaining volume of the organ 4 is displayed, but it is preferable that the CPU 10A displays the remaining volume of the organ 4 at a position that does not overlap with the three-dimensional model 6. The remaining volume of the organ 4 displayed on the screen 30 is also an example of preoperative planning information. Note that in addition to or instead of the remaining volume of the organ 4, the CPU 10A may display on the screen 30 the resected volume of the organ 4 when the diseased area 24A is resected along the resection line 22A, the weight of the resected organ 4, the remaining weight of the remaining organ 4, etc.

The preoperative labeling process shown in FIG. 7 is now complete. The information processing device 2 displays the preoperative planning information superimposed at the corresponding position on the three-dimensional model 6, so the medical staff can visualize the treatment plan in three dimensions before inserting the endoscope 3 into the patient, compared to when the medical staff simply considers the treatment plan in their head while looking at the three-dimensional model 6.

After the treatment plan for the treatment area is formulated in the preoperative marking stage, the endoscope 3 is actually inserted into the patient and the treatment begins. The stage at which the treatment is started by the medical professional is called the "intraoperative stage." The medical professional does not immediately begin treating the treatment area just because it is the intraoperative stage, but uses the endoscope 3 to confirm the actual characteristics of the treatment area and judge the validity of the treatment plan decided in the preoperative marking stage. Specifically, the medical professional reconsiders the validity of the resection line 22A decided in the preoperative marking stage based on the characteristics of the treatment area. Hereinafter, the stage of the intraoperative stage at which the preoperative plan information is reconsidered based on the actual characteristics of the treatment area photographed by the endoscope 3 is referred to as the "intraoperative marking stage," and the stage at which the diseased area 24 included in the treatment area is resected is referred to as the "resection stage."

FIG. 9 is a flowchart showing an example of the flow of the process for generating a composite 3D model 7 executed by the information processing device 2 when the endoscope 3 is inserted into a patient by a medical professional. The CPU 10A of the information processing device 2 reads the control program 11 from the ROM 10B and executes the process for generating a composite 3D model 7. Note that the memory unit 12 is assumed to have stored in advance image data of the treatment area of the patient captured by the image capture device 1. The memory unit 12 is also assumed to have stored in advance the 3D model 6 generated by the preoperative marking process shown in FIG. 7 and the preoperative planning information acquired in the preoperative marking process.

In step S100, the CPU 10A acquires the endoscopic image 5 from the endoscope 3.

In step S110, the CPU 10A composites the endoscopic video 5 acquired by the processing of step S100 onto the surface of the 3D model 6 corresponding to the same location shown in the endoscopic video 5, to generate a composite 3D model 7.

A known method is used for aligning the endoscopic video 5 with the three-dimensional model 6. Specifically, the CPU 10A inputs the acquired endoscopic video 5 into an image model that has been previously trained by associating the endoscopic video 5 with a position inside the body using deep learning such as a variational autoencoder (VAE), and identifies the location shown in the endoscopic video 5. Alternatively, the CPU 10A may first perform alignment to associate the location of the three-dimensional model 6 that the endoscopic video 5 is photographing, and then, based on the amount of movement of the endoscope 3 from the position where the alignment was performed, identify the location of the three-dimensional model 6 that the endoscopic video 5 is photographing after the movement. The amount of movement of the endoscope 3 may be calculated from measurement data obtained by an inertial sensor built into the endoscope 3, for example.

The CPU 10A continues to acquire the endoscopic video 5 until it receives an end instruction to end imaging of the treatment site using the endoscope 3.

In step S120, the CPU 10A determines whether the endoscope 3 has moved. Whether the endoscope 3 has moved or not may be determined, for example, based on whether a change is recognized in the endoscope video 5 captured in time series. The CPU 10A determines that the endoscope 3 has moved when a change is recognized in the endoscope video 5. Naturally, the CPU 10A may also determine whether the endoscope 3 has moved or not from measurement data obtained by an inertial sensor built into the endoscope 3.

If the endoscope 3 has moved, the process proceeds to step S110, where a new endoscopic video 5 of the treatment site acquired after the endoscope 3 has moved is composited onto the surface of the three-dimensional model 6 corresponding to the same location as that shown in the endoscopic video 5, generating a composite three-dimensional model 7. As a result, the display area of the endoscopic video 5 composited onto the surface of the three-dimensional model 6 increases.

On the other hand, if the determination process in step S120 determines that the endoscope 3 has not moved, the process proceeds to step S130.

In this case, in step S130, the CPU 10A determines whether or not an end instruction has been received from the medical staff via the operation unit 13. If an end instruction has not been received, the process proceeds to step S120. That is, the CPU 10A continues to generate a composite 3D model 7 to which a new endoscopic video 5 has been added by compositing a new endoscopic video 5 of the treatment site captured by the imaging device attached to the endoscope 3 as the endoscope 3 moves, into the composite 3D model 7 as the endoscope 3 moves, until the endoscope 3 finishes capturing the image of the treatment site.

If it is determined in the determination process of step S130 that an end instruction has been received, the process of generating the composite 3D model 7 shown in FIG. 9 is terminated.

Since the endoscopic video 5 captured by the endoscope 3 is a two-dimensional image, there is no sense of depth in the image. Therefore, the positional relationship of the treatment area recognized from the endoscopic video 5 may differ from the positional relationship of the treatment area represented by the three-dimensional model 6 that is displayed three-dimensionally. Furthermore, when recognizing the positional relationship of the treatment area from the endoscopic video 5, since the treatment area can only be confirmed from the image of the range displayed as the endoscopic video 5, it is difficult to grasp the treatment area three-dimensionally while looking down on it.

On the other hand, if the endoscopic video 5 corresponding to each position of the three-dimensional model 6 is synthesized on the surface of the three-dimensional model 6 as in the information processing device 2 disclosed herein, medical personnel can easily grasp in three dimensions the difference between the positional relationship of the treatment site shown by the image data capturing the treatment site and the positional relationship of the treatment site shown by the endoscopic video 5. Therefore, medical personnel can more easily identify which position of the treatment site is shown in the endoscopic video 5, compared to, for example, when the positional relationship of the treatment site is understood from the endoscopic video 5 alone.

Next, we will explain the processing of the information processing device 2 in the intraoperative labeling stage, in which the generated synthetic 3D model 7 is used to examine the validity of the treatment plan determined in the preoperative labeling stage.

FIG. 10 is a flowchart showing an example of the flow of intraoperative marking processing that is executed after the composite 3D model 7 is generated. The CPU 10A of the information processing device 2 reads the control program 11 from the ROM 10B and executes the intraoperative marking processing.

Note that the intraoperative labeling process is executed in parallel with the process of generating the composite 3D model 7 shown in FIG. 9. Therefore, if the endoscope 3 moves during the intraoperative labeling process, the range of the endoscopic video 5 that is synthesized into the composite 3D model 7 may expand.

In step S200 of FIG. 10, the CPU 10A displays the composite 3D model 7 on the screen 30.

In step S210, the CPU 10A acquires preoperative planning information from the memory unit 12 and displays the acquired preoperative planning information on the screen 30 by superimposing it on the corresponding position of the composite 3D model 7.

The medical staff refers to the endoscopic video 5 displayed on the surface of the synthetic 3D model 7 and considers whether the contents of the treatment plan formulated from the synthetic 3D model 7 are the same as those represented by the preoperative planning information. Because the synthetic 3D model 7 has the endoscopic video 5 synthesized therein, it is easier to grasp the characteristics and positional relationship of the treatment area compared to the 3D model 6 on which the endoscopic video 5 is not synthesized.

If, as a result of the consideration, it is determined that the position of the diseased area 24 recognized from the composite 3D model 7 and the position of the resection line 22A, etc., differ from their respective positions in the preoperative planning information, the medical staff will notify the CPU 10A via the operation unit 13 of an instruction to change the preoperative planning information.

Therefore, in step S220, CPU 10A determines whether an instruction to change the preoperative planning information has been received. If an instruction to change the preoperative planning information has been received, the process proceeds to step S230.

In step S230, the CPU 10A displays on the screen 30 the intraoperative planning information, which represents the treatment plan that reflects the changes to the preoperative planning information instructed by the change instruction, superimposed on the corresponding position of the composite 3D model 7.

For example, if CPU 10A receives an instruction to change resection line 22A, CPU 10A changes the position of resection line 22A in accordance with the change instruction. For ease of explanation, the resection line 22A after its position has been changed is referred to as "resection line 22B." Also, for example, if CPU 10A receives an instruction to change the size or position of affected area 24A, CPU 10A changes the size or position of affected area 24A in accordance with the change instruction. For ease of explanation, the affected area 24A after its size or position has been changed is referred to as "affected area 24B."

FIG. 11 is a diagram showing an example of the display of intraoperative planning information. The intraoperative planning information includes information indicated by the preoperative planning information. Also, as shown in FIG. 11, in response to a change in the resection line 22A, the CPU 10A displays the resection line 22A at the preoperative labeling stage and the updated resection line 22B, each superimposed on the corresponding positions on the composite 3D model 7. Also, in response to a change in the size or position of the affected area 24A, the CPU 10A displays the affected area 24A at the preoperative labeling stage and the updated affected area 24B, each superimposed on the corresponding positions on the composite 3D model 7.

Furthermore, when the resection line 22A is changed, the CPU 10A updates the ligation position 21A of the blood vessel 20 determined in the preoperative labeling stage, and displays the updated ligation position 21 of the blood vessel 20 superimposed on the corresponding position of the synthetic 3D model 7. Hereinafter, the updated ligation position 21 of the blood vessel 20 is referred to as the "ligation position 21B." Furthermore, when the resection line 22A is changed, the CPU 10A recalculates the remaining volume of the organ 4, and displays the recalculated remaining volume of the organ 4 on the screen 30.

Although not shown in FIG. 11, the blood flow stoppage area 25 included in the preoperative planning information may change due to the ligation position 21 of the blood vessel 20 being changed from ligation position 21A to ligation position 21B. In this case, the CPU 10A displays the updated blood flow stoppage area 25 by superimposing it on the corresponding position of the composite 3D model 7.

Such intraoperative planning information is an example of relevant information associated with the intraoperative labeling stage. By displaying the intraoperative planning information together with the composite 3D model 7, medical personnel can improve the accuracy of the resection location compared to when resecting the treatment area according to the preoperative planning information displayed together with the 3D model 6. In addition, because the composite 3D model 7 displays the preoperative planning information and the intraoperative planning information in a comparable state, medical personnel can proceed to the resection stage while checking the differences with the preoperative planning information.

If, for example, a medical professional has difficulty viewing the endoscopic image 5 of the treatment area displayed on the composite 3D model 7, the medical professional can change the viewpoint from which the composite 3D model 7 is viewed via the operation unit 13.

Therefore, in step S240 of FIG. 10, the CPU 10A determines whether or not a viewpoint change instruction to change the viewpoint from which the medical professional views the composite 3D model 7 has been received. If a viewpoint change instruction has been received, the process proceeds to step S250.

In step S250, the CPU 10A updates the position of the composite 3D model 7 so that the part of the composite 3D model 7 viewed from the viewpoint specified by the medical professional, i.e., the angle specified by the medical professional, is in the front, and displays it on the screen 30. The composite 3D model 7 can be rotated, for example, by applying a rotation matrix to the composite 3D model 7. In this way, the CPU 10A rotates the composite 3D model 7 and displays the composite 3D model 7 on the screen 30 with the arbitrary direction facing forward.

After changing the viewpoint from which the composite 3D model 7 is viewed, the process proceeds to step S260.

On the other hand, if it is determined in the determination process of step S240 that a viewpoint change instruction has not been received, the process proceeds to step S260 without executing the process of step S250.

In step S260, CPU 10A determines whether the medical professional has started to resect the diseased area 24, i.e., whether the operation has progressed to the resection stage. Before resecting the diseased area 24, the medical professional may mark the operation site using an electric scalpel to mark the position where the diseased area 24 will be resected. In this disclosure, the mark that marks the position where the diseased area 24 will be resected is referred to as a "resection mark 27." Therefore, CPU 10A may determine that the operation has progressed from the intraoperative marking stage to the resection stage when an operation of a resection instrument such as an electric scalpel attached to endoscope 3 has been performed. In this disclosure, as an example, it is assumed that the medical professional marks the operation site with a resection mark 27.

If the procedure has not yet progressed to the resection stage, the procedure proceeds to step S220 and the intraoperative labeling stage processing continues. On the other hand, if the procedure has progressed to the resection stage, the intraoperative labeling process shown in FIG. 10 ends.

Next, we will explain the processing of the information processing device 2 during the resection stage, when the medical staff resects the affected area 24.

FIG. 12 is a flowchart showing an example of the flow of the resection support process executed after the intraoperative marking process shown in FIG. 10 is completed. The CPU 10A of the information processing device 2 reads the control program 11 from the ROM 10B and executes the resection support process.

Note that the resection support process is executed in parallel with the process of generating the composite 3D model 7 shown in FIG. 9. Therefore, if the endoscope 3 moves during the resection support process, the range of the endoscopic video 5 that is composited into the composite 3D model 7 may expand.

In step S300, CPU 10A displays on screen 30 the intraoperative planning information finally formulated in the intraoperative labeling stage, superimposed on the corresponding position of the synthetic 3D model 7. Furthermore, if the acquired endoscopic video 5 includes a resection mark 27, CPU 10A displays the resection mark 27 at the corresponding position of the synthetic 3D model 7. The intraoperative planning information finally formulated in the intraoperative labeling stage, the resection mark 27, and the synthetic 3D model 7 are examples of resection support information. The resection support information is an example of related information associated with the resection stage.

FIG. 13 is a diagram showing an example of the display of resection support information. The virtual lines (not shown) connecting each of the resection marks 27 in FIG. 13 represent the resection execution lines along which the medical staff will actually perform the resection.

Furthermore, when there are multiple ligation positions 21B of a blood vessel 20, the CPU 10A associates an identifier for identifying the ligation position 21B with the ligation position 21B of each blood vessel 20 and displays them together with the composite three-dimensional model 7. In the display example shown in FIG. 13, the identifier "P1" is associated with the ligation position 21B of the first blood vessel 20, and the identifier "P2" is associated with the ligation position 21B of the second blood vessel 20. The numbers included in the identifiers indicate the order in which the blood vessels 20 are ligated. That is, in the display example shown in FIG. 13, the blood vessel 20 at the ligation position 21B associated with the identifier "P1" is ligated first, and the blood vessel 20 at the ligation position 21B associated with the identifier "P2" is ligated next. Therefore, medical personnel can check the ligation order of blood vessel 20 by referring to the identifier associated with ligation position 21B of blood vessel 20.

The identifiers of the ligation positions 21B are displayed based on the ligation order instructed by the medical staff during the intraoperative labeling stage. The CPU 10A may display the ligation order of the blood vessels 20 using a notation other than numbers. Specifically, the ligation order of the blood vessels 20 may be displayed in alphabetical order.

Before resecting the diseased area 24B, the medical professional will ligate the blood vessel 20 while referring to the resection support information, but depending on the situation, he or she may want to confirm whether or not the blood vessel 20 has been ligated. Therefore, when displaying the ligation position 21B of the blood vessel 20, the CPU 10A distinguishes between the ligation position 21B of the blood vessel 20 that has already been ligated and the ligation position 21B of the blood vessel 20 that has not yet been ligated, and displays them superimposed on the corresponding positions of the composite three-dimensional model 7.

For example, in FIG. 13, the blood vessel 20 at the ligation position 21B associated with the identifier "P1" has already been ligated, and the blood vessel 20 at the ligation position 21B associated with the identifier "P2" has not yet been ligated. In this case, as shown in FIG. 13, the CPU 10A displays the ligation position 21B associated with the identifier "P1" and the ligation position 21B associated with the identifier "P2" in different display forms. For example, the CPU 10A displays the ligation position 21B that has already been ligated as a straight line crossing the blood vessel 20, and displays the ligation position 21B that has not yet been ligated as an "x". Note that there are no restrictions on the display form of the ligation position 21B, as long as it is possible to distinguish between the ligation position 21B that has already been ligated and the ligation position 21B that has not yet been ligated.

By displaying the ligation positions 21B in this manner, medical personnel can check on the screen 30 which ligation positions 21B have been performed on the blood vessel 20 and which ligation positions 21B have not been performed on the blood vessel 20.

The CPU 10A also displays on the screen 30 the number of ligation positions 21B that have been ligated and the number of ligation positions 21B that have not yet been ligated together with the composite 3D model 7. In FIG. 13, the display "Remaining: 1" indicates the number of ligation positions 21B that have not yet been ligated, and the display "Completed: 1" indicates the number of ligation positions 21B that have been ligated.

The CPU 10A can determine from the endoscopic image 5 at which ligation position 21B the blood vessel 20 has been ligated, and can update the display form of the ligation position 21B and the number of ligated and unligated positions.

The CPU 10A may also distinguish between the blood vessel 20A in which blood flow has stopped due to ligation and the blood vessel 20B in which blood flow exists because the blood vessel has not yet been ligated, and superimpose and display them at the corresponding positions on the composite three-dimensional model 7. In FIG. 13, the blood vessel 20A located at the end of the blood flow direction from the ligation position 21B associated with the identifier "P1" is ligated at the ligation position 21B. On the other hand, the blood vessel 20B located at the end of the blood flow direction from the ligation position 21B associated with the identifier "P2" has not yet been ligated at the ligation position 21B. Therefore, the CPU 10A distinguishes between the blood vessel 20A and the blood vessel 20B, and superimposes and displays them at the corresponding positions on the composite three-dimensional model 7. There are no restrictions on the method for distinguishing between the blood vessel 20A and the blood vessel 20B, and for example, the display color of the blood vessel 20 may be changed.

Furthermore, for example, CPU 10A may display blood vessels 20A and 20B in a distinguished manner by superimposing a video showing the flow of blood on blood vessel 20B in which blood is present. The video showing the flow of blood may be an image showing the actual flow of blood or an animation. Such a video showing the flow of blood may be stored in advance in storage unit 12, for example.

By displaying on the screen 30 a blood vessel 20A in which blood flow has stopped and a blood vessel 20B in which blood flow is present, it is possible to reduce the occurrence of situations in which medical personnel forget to ligate a blood vessel 20, compared to when all blood vessels 20 are displayed in the same form regardless of their ligation status.

After displaying the resection support information on screen 30, in step S310 of FIG. 12, CPU 10A determines whether the resection position has shifted. A shift in the resection position means that there is a location between resection line 22B of the treatment area and the resection execution line represented by resection mark 27 that is more than a predetermined distance (hereinafter referred to as the "prescribed distance"). If the resection position has shifted, the process proceeds to step S320.

If there is a location between the resection line 22B and the resection execution line that is more than the specified distance away, if the medical staff resects the diseased area 24B along the resection execution line, they will end up resecting a location that is far from the planned resection line 22B. Therefore, in step S320, the CPU 10A outputs a warning. There are no restrictions on the form of the warning output as long as it is a method that the medical staff notices, and for example, a display on the screen 30 or a voice notification can be used.

The medical staff who received the warning notifies CPU 10A through operation unit 13 of whether or not to continue resecting diseased part 24B along the resection execution line despite the warning. Therefore, in step S330, CPU 10A determines whether or not a continue instruction has been received from the medical staff indicating that the diseased part 24B is to be resected without modifying the position of resection mark 27. If a continue instruction has not been received, this means that the medical staff will modify the position of resection mark 27 to change the resection position of diseased part 24B, so the process proceeds to step S300 and the display of resection support information continues. Note that CPU 10A may proceed to step S300 if a position modification instruction indicating that the position of resection mark 27 is to be modified is received from the medical staff.

If an instruction to continue is received, it is believed that the medical staff will not correct the position of the resection mark 27, but will instead use the resection mark 27 as a guide and intentionally resect a location away from the resection mark 27, so that the resection position approaches the resection line 22B. Therefore, the process proceeds to step S340, where the next process is executed.

On the other hand, if the determination process in step S310 determines that the resection position has not shifted, there is no need to issue a warning, and the process proceeds to step S340 without executing the processes in steps S320 and S330.

In step S340, the CPU 10A determines whether a new blood vessel 20 has been ligated at any of the ligation positions 21B. The CPU 10A may determine whether a new blood vessel 20 has been ligated from the endoscopic image 5. If a new blood vessel 20 has been ligated, the process proceeds to step S350.

In step S350, the CPU 10A displays the newly ligated ligation position 21B by a straight line crossing the blood vessel 20, and displays the ligation position 21B as having been ligated by superimposing it on the corresponding position on the composite 3D model 7. In other words, the CPU 10A updates the display of the ligation state to match the latest ligation state. The CPU 10A also updates the number of ligated and unligated ligation positions 21B displayed on the screen 30.

When the blood vessel 20 is ligated, blood flow stops in the blood vessel 20 at the end of the ligation position 21B in the blood flow direction, and blood flow into the organ 4 stops.

Therefore, in step S360, the CPU 10A updates the range of the blood flow stop region 25 based on the newly ligated ligation position 21B, and displays the updated blood flow stop region 25 by superimposing it on the corresponding position of the composite 3D model 7.

Furthermore, in step S370, the CPU 10A updates the display of the blood vessel 20A located at the end of the blood flow direction from the newly ligated ligation position 21B to a display form indicating that blood flow has stopped, and then proceeds to step S380.

On the other hand, if it is determined in the determination process of step S340 that the blood vessel 20 has not been newly ligated, there is no change in the ligation state, so the process proceeds to step S380 without executing the processes of steps S350 to S370.

In step S380, the CPU 10A determines whether or not a resection completion instruction has been received from the medical staff via the operation unit 13, indicating that the resection of the diseased area 24B has been completed. If a resection completion instruction has not been received, the process proceeds to step S340 since ligation of the blood vessel 20 may continue. In other words, until a resection completion instruction is received, the CPU 10A updates the display of the ligation status, the number of ligated and unligated ligation positions 21B, the range of the blood flow stop region 25, and the display of the blood flow status of the blood vessel 20 according to the latest ligation status each time ligation of the blood vessel 20 is performed.

On the other hand, if a resection completion notification is received, the resection support process shown in Figure 12 ends.

The CPU 10A may display a setting area in area 26 on the screen 30 for setting the display format of the affected area 24, the resection range, the organ 4 including the treatment site, and the blood vessels 20 passing through the treatment site. Each of the items in area 26, "Tumor," "Resection area," "Liver," "Artery," and "Vein," can be selected by the medical staff via the operation unit 13, and each time an item is selected, the CPU 10A switches the display of the separator ":" following the selected item between "display," "semi-transparent display," and "non-display" in sequence.

"Display" is a mode in which the part represented by the item is displayed on the screen 30. "Semi-transparent display" is a mode in which the part represented by the item is displayed transparently so that another part behind the part represented by the item is displayed on the screen 30. "Hide" is a mode in which the part represented by the item is not displayed on the screen 30.

In the example of FIG. 13, the tumor item is set to "display", so CPU 10A displays the affected area 24B with the tumor on screen 30. The resection area, liver, and artery items are also set to "display", so CPU 10A displays the part of organ 4 that is resected by resection line 22B, the liver, which is an example of organ 4, and the artery of blood vessels 20 on screen 30. On the other hand, the vein item is set to "hide", so CPU 10A does not display the vein of blood vessels 20 on screen 30.

In this way, the information processing device 2 can change the display format of the part corresponding to each item selected by the medical professional. Therefore, for example, it is possible to avoid displaying parts that get in the way of the medical professional checking the treatment area, so the characteristics of the treatment area can be checked in detail compared to a case where the display format of the parts constituting the composite 3D model 7 cannot be selected for each part. The display format setting area displayed in area 26 in FIG. 13 may also be displayed on screen 30 in the preoperative labeling stage shown in FIG. 8 and on screen 30 in the intraoperative labeling stage shown in FIG. 11.

As already explained, the endoscope 3 can be fitted with treatment tools such as a resection tool for resecting the diseased area 24 and a collection tool for collecting tissue from the diseased area 24. Therefore, the CPU 10A may store in the memory unit 12 a record of when and how the medical staff operated the treatment tool and what treatment was performed on the treatment site as a treatment tool operation history. The treatment tool operation history is an example of history information related to the operation of the treatment tool.

The time the treatment was performed can be obtained, for example, from a timer function built into the CPU 10A. The movement status of the treatment tool can be obtained, for example, from measurement data obtained by an inertial sensor built into the treatment tool. The details of the treatment using the treatment tool can be obtained, for example, from the endoscopic video 5.

The CPU 10A stores in the memory unit 12 the operation history of the treatment tool performed by the medical staff while performing treatment on the treatment area, together with the composite 3D model 7 including the treatment area.

If the operation history of the treatment tools and the composite 3D model 7 are stored in the memory unit 12 in this way, the treatment content instructed by the medical staff can be reproduced using the operation history of the treatment tools and the composite 3D model 7 and displayed on the screen 30, allowing the medical staff to check the past treatment content in three dimensions. Therefore, the medical staff can look back on the treatment content they performed and use any improvements they find in the next treatment, or learn how other medical staff perform treatments.

The above describes one form of the endoscopic procedure system 100 using an embodiment, but the disclosed form of the endoscopic procedure system 100 is only one example, and the form of the endoscopic procedure system 100 is not limited to the scope described in the embodiment. Various modifications or improvements can be made to the embodiment without departing from the gist of this disclosure, and forms with such modifications or improvements are also included in the technical scope of the disclosure.

For example, the internal processing order in each of the flowcharts for the preoperative marking process shown in FIG. 7, the synthetic 3D model 7 generation process shown in FIG. 9, the intraoperative marking process shown in FIG. 10, and the resection support process shown in FIG. 12 (hereinafter referred to as "processing of the information processing device 2") may be changed without departing from the scope of the present disclosure.

In the above embodiment, as an example, a form in which the processing of the information processing device 2 is realized by software processing has been described. However, processing equivalent to the flowchart showing the processing of the information processing device 2 may be processed by hardware. In this case, the processing speed can be increased compared to when the processing of the information processing device 2 is realized by software processing.

In the above embodiment, the term "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPU 10A) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

Furthermore, the processor operations in the above embodiments may not only be performed by a single processor, but may also be performed by multiple processors in physically separate locations working together. Furthermore, the order of each processor operation is not limited to the order described in the above embodiments, and may be changed as appropriate.

In the above embodiment, an example has been described in which the control program 11 is stored in ROM 10B. However, the storage destination of the control program 11 is not limited to ROM 10B. The control program 11 of the present disclosure can also be provided in a form recorded on a computer-readable storage medium.

For example, the control program 11 may be provided in a form recorded on an optical disk such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a Blu-ray disc. The control program 11 may also be provided in a form recorded on a portable semiconductor memory such as a USB (Universal Serial Bus) memory or a memory card. ROM 10B, CD-ROM, DVD-ROM, Blu-ray disc, USB, and memory card are examples of non-transitory storage media.

Furthermore, the control unit 10 may download the control program 11 from an external device connected to a communication line through the I/F unit 15, and store the downloaded control program 11 in the ROM 10B of the control unit 10. In this case, the CPU 10A of the control unit 10 reads the control program 11 downloaded from the external device from the ROM 10B and executes the processing of the information processing device 2.

The control program 11 of the present application can be provided as a program product. A program product includes any type of product for providing a program. For example, a program product includes a program provided over a network such as the Internet, and a non-transitory computer-readable recording medium such as a CD-ROM or DVD on which a program is stored.

The disclosure of Japanese Patent Application No. 2023-168676, filed on September 28, 2023, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

The following are additional notes related to this embodiment.

(Appendix 1)
A processor is provided.
The processor,
an information processing device which controls the display of a composite 3D model of a treatment area generated from image data of the treatment area photographed in advance, and the composite 3D model into which the image of the treatment area is photographed by an imaging device attached to an endoscope is composited.

(Appendix 2)
The information processing device according to claim 1, wherein the processor performs control to display related information associated with each of the stages together with the three-dimensional model or the composite three-dimensional model according to a stage of treatment for the treatment area.

(Appendix 3)
In the case of a preoperative labeling stage in which a treatment plan for the treatment site is formulated, the processor controls to superimpose and display at least one of the position of the affected area, a pre-planned resection line for the treatment site including the affected area, the resection range to be resected along the resection line, the position of the blood vessel passing through the treatment site, the pre-planned ligation position of the blood vessel, and a blood flow cessation region where blood flow is halted by ligation of the blood vessel on a corresponding position of the three-dimensional model.

(Appendix 4)
The information processing device according to claim 3, wherein the processor performs control to display, when the diseased area is resected along the resection line, a remaining volume of an organ including the diseased area together with the three-dimensional model.

(Appendix 5)
In the case of an intraoperative marking step of reconfirming the preoperative planning information determined in the preoperative marking step of formulating a treatment plan for the treatment area based on the characteristics of the treatment area obtained from the image of the treatment area synthesized in the synthetic three-dimensional model,
The information processing device described in Appendix 2, wherein the processor performs control to superimpose and display the resection line at the preoperative labeling stage and the updated resection line on the corresponding positions of the synthetic three-dimensional model in response to a change in the resection line of the treatment site including the affected area determined in the preoperative labeling stage, and to update the ligation position of the blood vessel determined in the preoperative labeling stage and superimpose and display the updated ligation position of the blood vessel on the corresponding position of the synthetic three-dimensional model.

(Appendix 6)
In the case of a resection stage in which an affected area included in the treatment area is resected, the processor controls to superimpose and display at least one of the following on a corresponding position of the synthetic three-dimensional model: a position of the affected area, a resection line of the treatment area including the affected area, a resection range to be resected along the resection line, a position of a blood vessel passing through the treatment area, a ligation position of the blood vessel, a blood flow halt area in which blood flow is halted by ligation of the blood vessel, and a position of an resection mark placed on the treatment area by a medical professional as a mark for resecting the affected area. The information processing device described in Appendix 2.

(Appendix 7)
The information processing device according to claim 6, wherein the processor performs control to associate an identifier for identifying a ligation position of a blood vessel with the ligation position of the blood vessel and display the identifier together with the synthetic three-dimensional model.

(Appendix 8)
The information processing device according to claim 7, wherein the identifier indicates a ligation order of blood vessels.

(Appendix 9)
The information processing device according to any one of appendices 6 to 8, wherein the processor controls to distinguish between the ligation positions of blood vessels that have already been ligated and the ligation positions of blood vessels that have not yet been ligated and to superimpose and display them at corresponding positions on the composite three-dimensional model.

(Appendix 10)
The information processing device according to any one of Supplementary Note 6 to Supplementary Note 9, wherein the processor performs control to distinguish between blood vessels in the blood flow stop region where blood flow has been stopped by ligation and blood vessels in the blood flow stop region where blood flow is present because the blood vessels have not yet been ligated, and to superimpose and display them at corresponding positions on the composite three-dimensional model.

(Appendix 11)
The information processing device according to claim 10, wherein the processor performs control to superimpose and display a video showing blood flow on the blood vessel in the blood flow stop region where blood flow exists.

(Appendix 12)
The information processing device according to any one of claims 6 to 11, wherein the processor controls to display the number of ligation positions of blood vessels that have already been ligated and the number of ligation positions of blood vessels that have not yet been ligated together with the composite three-dimensional model.

(Appendix 13)
The information processing device according to any one of claims 6 to 12, wherein the processor performs control to update a range of the blood flow stop region in accordance with a ligation state of a blood vessel.

(Appendix 14)
The information processing device according to any one of claims 6 to 13, wherein the processor performs control to output a warning when the resection line of the treatment area and the resection execution line represented by the resection mark are separated by a predetermined distance or more.

(Appendix 15)
The information processing device according to any one of claims 6 to 14, wherein the processor performs control to change the display forms of the affected area, the resection range, the organ including the treatment site, and the blood vessels passing through the treatment site, which are displayed superimposed on corresponding positions of the synthetic three-dimensional model in accordance with instructions from a medical professional.

(Appendix 16)
The information processing device according to any one of claims 1 to 15, wherein the processor controls to display the synthetic 3D model from an angle specified by a medical professional.

(Appendix 17)
The information processing device described in Appendix 16, wherein the processor controls to display the composite 3D model to which the new image has been added while compositing new images of the treatment site captured by the imaging device as the endoscope moves into the composite 3D model along with the movement of the endoscope.

(Appendix 18)
The processor stores, in a storage device, history information regarding the operation of a tool performed by a medical professional during treatment of the treatment site together with the synthetic three-dimensional model;
The information processing device according to claim 17, further comprising: a display unit configured to display, in response to an instruction from a medical professional, a treatment content for the treatment area using the history information and the composite three-dimensional model.

(Appendix 19)
A control program for causing a computer to execute control for combining a three-dimensional model of a treatment area generated from image data of the treatment area photographed in advance with an image of the treatment area photographed by an imaging device attached to an endoscope, and displaying a composite three-dimensional model into which the image of the treatment area has been combined.

(Appendix 20)
A non-transitory storage medium storing a program executable by a computer to execute a predetermined process,
The process,
A synthesis step of synthesizing an image of the treatment site photographed by an imaging device attached to an endoscope with a three-dimensional model of the treatment site generated from image data of the treatment site photographed in advance;
A display step of displaying a synthetic three-dimensional model synthesized with the image of the treatment area;
Non-transitory storage media including:

(Appendix 21)
A computer program product including a control program that causes a computer to execute control for combining a three-dimensional model of a treatment area generated from image data of the treatment area photographed in advance with an image of the treatment area photographed by an imaging device attached to an endoscope, and displaying a composite three-dimensional model into which the image of the treatment area has been combined.

Claims (19)

A processor is provided.
The processor,
An information processing device that performs control to combine an image of the treatment area captured by an imaging device attached to an endoscope with a three-dimensional model of the treatment area generated from image data of the treatment area captured in advance, and display the combined three-dimensional model with the image of the treatment area. The information processing device according to claim 1 , wherein the processor performs control to display related information associated with each of the stages, together with the three-dimensional model or the composite three-dimensional model, in accordance with a stage of treatment for the treatment location. 3. The information processing device according to claim 2, wherein in a preoperative labeling stage of formulating a treatment plan for the treatment site, the processor controls to superimpose and display at least one of the position of the affected area, a preplanned resection line for the treatment site including the affected area, a resection range to be resected along the resection line, the position of a blood vessel passing through the treatment site, a preplanned ligation position of the blood vessel, and a blood flow cessation region where blood flow is halted by ligation of the blood vessel on a corresponding position of the three-dimensional model. The information processing device according to claim 3 , wherein the processor performs control to display, when the diseased part is resected along the resection line, a remaining volume of an organ including the diseased part together with the three-dimensional model. In the case of an intraoperative marking step of reconfirming the preoperative planning information determined in the preoperative marking step of formulating a treatment plan for the treatment area based on the characteristics of the treatment area obtained from the image of the treatment area synthesized in the synthetic three-dimensional model,
3. The information processing device according to claim 2, wherein the processor performs control to superimpose and display the resection line at the preoperative labeling stage and the updated resection line on the corresponding positions of the synthetic 3D model in response to a change in the resection line of the treatment site including the diseased area determined in the preoperative labeling stage, and to update the ligation position of the blood vessel determined in the preoperative labeling stage and superimpose and display the updated ligation position of the blood vessel on the corresponding position of the synthetic 3D model. 3. The information processing device according to claim 2, wherein in a resection stage of resecting an affected area included in the treatment area, the processor controls to superimpose and display at least one of the position of the affected area, a resection line of the treatment area including the affected area, a resection range to be resected along the resection line, a position of a blood vessel passing through the treatment area, a ligation position of the blood vessel, a blood flow stoppage area where blood flow is stopped by ligation of the blood vessel, and a position of an resection mark placed by a medical professional at the treatment area as a mark for resecting the affected area on a corresponding position of the synthetic three-dimensional model. The information processing apparatus according to claim 6 , wherein the processor performs control to display an identifier for identifying a ligation position of a blood vessel together with the synthetic three-dimensional model in association with the ligation position of the blood vessel. The information processing device according to claim 7 , wherein the identifier indicates a ligation order of blood vessels. The information processing device according to claim 6 , wherein the processor performs control to distinguish between ligation positions of blood vessels that have already been ligated and ligation positions of blood vessels that have not yet been ligated and to superimpose and display them at corresponding positions on the composite three-dimensional model. The information processing device according to claim 9 , wherein the processor performs control to distinguish between blood vessels in the blood flow stop region where blood flow has been stopped by ligation and blood vessels in the blood flow stop region where blood flow is present because the blood vessels have not yet been ligated, and to superimpose and display them at corresponding positions on the synthetic three-dimensional model. The information processing device according to claim 10 , wherein the processor performs control to superimpose and display a moving image showing blood flow on the blood vessel in the blood flow stop region where blood flow exists. The information processing device according to claim 9 , wherein the processor performs control to display, together with the composite three-dimensional model, the number of ligation positions of blood vessels that have already been ligated and the number of ligation positions of blood vessels that have not yet been ligated. The information processing device according to claim 6 , wherein the processor performs control to update the range of the blood flow stop region in accordance with a ligated state of the blood vessel. The information processing device according to claim 6 , wherein the processor performs control to output a warning when the resection line of the treatment site and the resection execution line represented by the resection mark are separated by a predetermined distance or more. 7. The information processing device according to claim 6, wherein the processor performs control to change display forms of the diseased area, the resection range, the organ including the treatment site, and the blood vessels passing through the treatment site, which are displayed superimposed on corresponding positions of the synthetic three-dimensional model, in accordance with instructions from a medical professional. The information processing device according to any one of claims 1 to 15, wherein the processor performs control to display the composite three-dimensional model from an angle specified by a medical professional. The information processing device according to claim 16, wherein the processor performs control to display the composite 3D model to which a new image of the treatment site captured by the imaging device as the endoscope moves is added while compositing the new image into the composite 3D model as the endoscope moves. The processor stores, in a storage device, historical information regarding the operation of a tool performed by a medical professional during treatment of the treatment site together with the synthetic three-dimensional model;
The information processing device according to claim 17 , further comprising a control for displaying details of treatment for the treatment site using the history information and the composite three-dimensional model in response to an instruction from a medical professional. A control program that causes a computer to execute control to combine a three-dimensional model of the treatment area, generated from image data of the treatment area previously photographed, with an image of the treatment area photographed by an imaging device attached to an endoscope, and display a composite three-dimensional model with the image of the treatment area.