WO2023127719A1 - Program, information processing device, and information processing method - Google Patents

Abstract

A program according to one aspect of the present invention causes a computer (1) to execute a process in which circumferential surface images of a circumferential surface of a medical elongate body (4) inserted in a luminal organ are acquired in a time series and in which the amount of movement of the medical elongate body (4) is specified on the basis of the plurality of circumferential surface images acquired in the time series. Preferably, the program also causes the computer (1) to execute a process in which feature regions are extracted from the time-series circumferential surface images of the medical elongate body (4) and in which the amount of movement in the transverse direction or the rotational direction between the centers of gravity of the feature regions is specified on the basis of the extracted feature regions.

Description

Program, information processing device and information processing method

The present invention relates to a program, an information processing device, and an information processing method.

In recent years, medical elongate bodies have been inserted into hollow organs such as blood vessels to examine or treat lesions. For example, US Pat. No. 6,200,000 discloses a device that includes a catheter having markings to aid in proper alignment with the treatment area.

Japanese Patent Application Laid-Open No. 2021-058708

However, the invention according to Patent Document 1 has a problem that it is not possible to specify the amount of movement of the elongated body for medical use inserted into the hollow organ.

One aspect is to provide a program or the like that can specify the amount of movement of an elongated body for medical use inserted into a hollow organ.

A program according to one aspect obtains circumferential images of a circumferential surface of a medical elongated body to be inserted into a hollow organ in time series, and based on the plurality of circumferential images obtained in time series, the medical The computer is caused to execute a process of specifying the amount of movement of the elongated body for use.

In one aspect, it becomes possible to specify the amount of movement of the medical elongated body inserted into the hollow organ.

It is an explanatory view explaining composition of a catheter system. It is an explanatory view explaining a surrounding surface image. It is an explanatory view explaining processing which specifies a movement amount. It is an explanatory view explaining processing which specifies a movement amount. FIG. 11 is an explanatory diagram for explaining a movement amount correction process; FIG. 11 is an explanatory diagram for explaining a movement amount correction process; FIG. 10 is an explanatory diagram for explaining interest point update processing; FIG. 10 is an explanatory diagram for explaining interest point update processing; FIG. 10 is an explanatory diagram for explaining interest point update processing; FIG. 11 is an explanatory diagram showing an example of a display screen for movement amounts; FIG. 10 is an explanatory diagram showing an example of a display screen of a graph showing the relationship between the amount of movement and the elapsed time; FIG. 10 is an explanatory diagram showing an example of a screen displaying a circumferential image and a graph at the same time; 4 is a flow chart showing a processing procedure when outputting a movement amount of a medical elongated body; FIG. 10 is a flow chart showing a processing procedure of a preprocessing subroutine for a circumferential image; FIG. FIG. 11 is a flow chart showing a processing procedure of a subroutine of movement amount specifying processing; FIG. FIG. 10 is a flowchart showing a processing procedure of a subroutine of interest point update processing; FIG. 4 is a flowchart showing a processing procedure for outputting a circumferential image and a movement amount of a medical elongated body; 4 is a flow chart showing a processing procedure for outputting an angiographic image of a hollow organ and a movement amount of a medical elongated body. FIG. 10 is a flow chart showing a processing procedure when outputting a notification; FIG. FIG. 7 is an explanatory diagram illustrating the configuration of a catheter system according to Embodiment 2; FIG. 4 is an explanatory diagram showing an example of a record layout of a training data DB; It is an explanatory view explaining a movement amount specific model. FIG. 11 is a flowchart showing a processing procedure of a subroutine of movement amount identification processing using a movement amount identification model; FIG.

Hereinafter, the present invention will be described in detail based on the drawings showing its embodiments.

(Embodiment 1)
Embodiment 1 relates to an embodiment for specifying the amount of movement of a medical elongated body inserted into a hollow organ based on a peripheral surface image of the peripheral surface of the medical elongated body. Hollow organs are internal organs having a tubular structure, such as blood vessels, pancreatic ducts, bile ducts, gastrointestinal tracts or bronchi.

FIG. 1 is an explanatory diagram illustrating the configuration of the catheter system 10. FIG. A catheter system 10 of this embodiment includes an information processing device 1 , a Y connector 2 into which a medical elongate body 4 is inserted, and an imaging device 3 attached to the Y connector 2 . The information processing device 1 is a personal computer, tablet, smartphone, or the like. Hereinafter, the information processing apparatus 1 is read as the computer 1 .

The medical elongate body 4 includes guide wires, balloon catheters, stent catheters, and the like. The medical elongated body 4 is inserted through the inside of the Y connector 2 into the hollow organ.

The imaging device 3 is connected to the computer 1 by wire or wirelessly. The imaging device 3 is attached inside or outside the Y connector 2 so that the peripheral surface of the medical elongate body 4 can be observed. The imaging device 3 is, for example, a CCD (Charge Coupled Device) camera, a CMOS (Complementary Metal Oxide Semiconductor) camera, or the like. In this embodiment, an example in which the imaging device 3 is attached to the Y connector 2 will be described, but the present invention is not limited to this. For example, the imaging device 3 may be attached to a connection tube, an adapter, a catheter hub, a hemostatic valve, a three-way stopcock, or the like, which are components of the catheter system 10 (not shown).

A display device 5 and an input device 6 are connected to the computer 1 . The display device 5 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays a peripheral surface image of the peripheral surface of the medical elongate body 4 output from the computer 1, or the like. The input device 6 is an input device such as a keyboard, mouse, trackball, or microphone. The display device 5 and the input device 6 may be laminated integrally to form a touch panel. The input device 6 and the computer 1 may be configured integrally.

The computer 1 includes a control section 11, a storage section 12, a communication section 13, a display section 14, an input section 15, a reading section 16 and a large capacity storage section 17. Each configuration is connected by a bus B.

The control unit 11 includes an arithmetic processing unit such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), etc., and reads out the control program 1P (program product) stored in the storage unit 12. By executing it, various information processing, control processing, etc. related to the computer 1 are performed. Note that although FIG. 1 illustrates the controller 11 as a single processor, it may be a multiprocessor.

The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data necessary for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data and the like necessary for the control unit 11 to perform arithmetic processing. The communication unit 13 is an interface that performs communication between the computer 1 and the network. The display unit 14 is an interface that connects the display device 5 and the bus. The input unit 15 is an interface that connects the input device 6 and a bus.

The reading unit 16 reads a portable storage medium 1a including CD (Compact Disc)-ROM or DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the large-capacity storage unit 17 . Alternatively, the control unit 11 may download the control program 1P from another computer via the network N or the like and store it in the large-capacity storage unit 17 . Furthermore, the control unit 11 may read the control program 1P from the semiconductor memory 1b.

The large-capacity storage unit 17 includes recording media such as HDD (Hard disk drive) and SSD (Solid State Drive). In addition, in this embodiment, the storage unit 12 and the large-capacity storage unit 17 may be configured as an integrated storage device. Also, the large-capacity storage unit 17 may be composed of a plurality of storage devices. Furthermore, the large-capacity storage unit 17 may be an external storage device connected to the computer 1 .

The computer 1 may execute various information processing, control processing, etc. as a single computer, may be distributed among a plurality of computers, or may be executed through virtual machines. Various information processing and control processing related to the computer 1 may be executed by a server device or the like having a communication environment.

The computer 1 according to the present embodiment acquires circumferential images of the circumferential surface of the elongated medical body 4 inserted into the hollow organ in time series. The computer 1 specifies the amount of movement of the medical elongated body 4 based on a plurality of circumferential images acquired in time series. The computer 1 generates a graph showing the relationship between the identified movement amount and the elapsed time, and outputs the generated graph to the display device 5 .

Next, processing for specifying the amount of movement of the medical elongated body 4 based on the peripheral surface image of the peripheral surface of the medical elongated body 4 will be described. First, the acquisition processing of the circumferential image will be described.
FIG. 2 is an explanatory diagram for explaining a circumferential image. The computer 1 acquires peripheral surface images of the peripheral surface of the elongated medical body 4 inserted into the hollow organ from the imaging device 3 attached to the Y connector 2 in chronological order. The computer 1 performs preprocessing on a plurality of peripheral surface images acquired in time series.

Specifically, the computer 1 acquires the effective image area 11a from the acquired circumferential image. The effective image area 11a is an image area in which the medical elongate body 4 is shown. The computer 1 may accept designation of the effective image area 11a by an operator or the like. Alternatively, the computer 1 performs feature extraction processing such as edge detection processing, pattern matching processing, or the like based on each pixel information (color information, luminance information, etc.) included in the peripheral image, and extracts an effective image in the peripheral image. You may extract the area|region 11a. Furthermore, the computer 1 may use a segmentation network such as U-Net (Convolutional Networks for Biomedical Image Segmentation) to extract the effective image area 11a in the circumferential image. The computer 1 cuts out the obtained effective image area 11a.

The computer 1 converts the peripheral surface image from a color image (for example, an RGB (Red-Green-Blue color model) image) to a grayscale image (an image expressing black and white shading). Note that the execution order is not limited to that described above, and for example, the order of the extraction processing of the effective image area 11a and the conversion processing of the image type may be reversed. Note that the above-described extraction processing of the effective image area 11a and conversion processing of the image type are not essential and may be omitted. Note that the above-described processing may be performed manually by an operator.

The computer 1 validates the area 11b including the polygonal area formed on the circumferential surface of the medical elongate body 4 that advances and retreats while rotating, based on segmentation processing and labeling processing using an algorithm such as a binarization method. Extract from the image area 11a. Specifically, the computer 1 assigns a predetermined threshold to pixel values by segmentation processing using an algorithm such as a binarization method, and specifies regions exceeding the threshold and regions not exceeding the threshold. The computer 1 extracts the area 11b based on the labeling process of assigning the same label (number) to the pixels in which the area exceeding the threshold or the area not exceeding the threshold is continuous from the area specified by the binarization process.

As shown in the figure, a region including a plurality of rhombic regions formed on the peripheral surface of the long medical body 4 is extracted as the region 11b. Although a plurality of rhombic regions are formed on the entire circumference of the medical elongated body 4, only a portion thereof is shown in FIG. In addition, FIG. 2 shows a state in which the medical elongated body 4 advances and retreats while rotating, and the rhombus formed on the peripheral surface of the medical elongated body 4 changes into a triangle. The polygonal area formed on the peripheral surface of the elongated body 4 for medical use may be a triangle, a circle, a quadrangle, or the like other than a rhombus.

Note that the region extraction processing is not limited to the binarization method and labeling processing. For example, the Snakes method, which uses active contours (snakes) to segment an image, the Level set method, which expresses the inside and outside of a region using Level set functions, and the method based on deep learning, etc. Also good.

Next, the computer 1 specifies the amount of horizontal movement or the amount of rotational (vertical) movement between the centers of gravity of the areas extracted by the preprocessing.
3A and 3B are explanatory diagrams illustrating the process of specifying the amount of movement. FIG. 3A is an explanatory diagram for explaining the movement amount specifying process. FIG. 3B is an explanatory diagram showing three axial directions of the medical elongated body 4. As shown in FIG. The three axes include X-axis, Y-axis and Z-axis.

As shown in (1) and (2) of FIG. 3A, the computer 1 acquires the first frame of the peripheral surface image from the multiple peripheral surface images acquired in time series. The computer 1 receives the setting of the interest point 12a by the operator or the like for the acquired peripheral image of the first frame. For example, the computer 1 may receive the setting of the interest point 12a by the operator or the like at the position of the center of gravity of the circumferential image of the first frame. Note that the computer 1 may automatically set the point of interest 12a at the position of the center of gravity of the peripheral surface image of the first frame. Note that the interest point 12a is set only for the peripheral image of the first frame.

The computer 1 uses, for example, a binarization method and a labeling process to extract the feature region 12b containing the received interest point 12a. In addition, in FIGS. 3A and 3B, an example in which the characteristic region 12b is a rhombic region has been described, but the same can be applied to circular or other polygonal regions. The computer 1 calculates the position of the center of gravity of the extracted characteristic region 12b. For example, the computer 1 obtains the center of gravity of the feature region 12b by performing a center-of-gravity calculation using pixel values in the feature region 12b.

As shown in (3) and (4) of FIG. 3A, the computer 1 acquires a peripheral image (second frame peripheral image) subsequent to the first frame peripheral image. The computer 1 extracts the characteristic region 12b including the interest point 12a from the second frame of the circumferential image. The computer 1 calculates the position of the center of gravity 12e of the extracted characteristic region 12b.

The computer 1 performs calculation processing of the movement amount for the circumferential image. Specifically, the computer 1 calculates the difference between the position of the center of gravity 12e and the coordinate X-axis of the interest point 12a as a horizontal movement amount (pixels). The computer 1 calculates the difference between the position of the center of gravity 12e and the coordinate Y-axis of the interest point 12a as the movement amount (pixels) in the rotational direction.

Subsequently, the computer 1 acquires a peripheral image following the second frame peripheral image (third frame peripheral image). The computer 1 repeatedly executes the above-described processing on the acquired peripheral surface image of the third frame. In this way, the computer 1 performs the above-described processing on all circumferential images acquired in time series within a predetermined period (for example, 1 second), thereby obtaining a lateral direction corresponding to each circumferential image. Identify the amount of movement and the amount of movement in the rotational direction.

In addition, the movement amount in pixel units can be converted to the movement amount in actual distance. Specifically, regarding the amount of horizontal movement, the computer 1 multiplies the amount of horizontal movement in units of pixels by the actual distance per pixel. Regarding the amount of movement in the rotational direction, as shown in FIG. is calculated.

The computer 1 calculates the total amount of lateral movement and the total amount of rotational movement. The computer 1 outputs the calculated total amount of movement in the horizontal direction as the amount of movement in the horizontal direction, and outputs the total amount of movement in the rotational direction to the display device 5 as the amount of movement in the rotational direction.

Further, when the area 11b including the polygonal area formed on the peripheral surface of the medical elongated body 4 is inclined, the amount of lateral movement and the amount of movement in the rotational direction can be corrected using the area 11b. can.

4A and 4B are explanatory diagrams for explaining the movement amount correction process. FIG. 4A is an explanatory diagram for explaining the process of correcting the amount of movement in the horizontal direction. As shown, arrow 11c indicates the actual lateral movement direction of medical elongated body 4 . An arrow 11d indicates the lateral movement direction of the medical elongate body 4 along the horizontal axis (X-axis) of the circumferential image. Thus, when the area 11b is tilted, an error occurs between the calculated lateral movement amount and the actual lateral movement amount with respect to the lateral center line 12d of the area 11b.

In this case, the computer 1 acquires the tilt angle of the medical elongate body 4. For example, the computer 1 may calculate the inclination angle of the peripheral image based on a reference peripheral image prepared in advance. Alternatively, when a peripheral image is input, the computer 1 may acquire the inclination angle of the peripheral image using an inclination angle detection model that has been learned to output the inclination angle. The computer 1 can acquire the actual lateral movement amount of the medical elongated body 4 along the arrow 11c by correcting the lateral movement amount by geometric calculation from the acquired tilt angle. .

FIG. 4B is an explanatory diagram for explaining the process of correcting the amount of movement in the rotational direction. As shown, the arrow 11e indicates the actual direction of rotational movement of the medical elongate body 4. As shown in FIG. An arrow 11f indicates the rotational movement direction of the medical elongated body 4 along the vertical axis (Y-axis) of the circumferential image. Thus, when the region 11b is tilted, an error occurs between the calculated amount of movement in the rotational direction and the actual amount of movement in the rotational direction with respect to the vertical center line 12c of the region 11b. In this case, the computer 1 acquires the tilt angle of the medical elongate body 4 . The computer 1 can acquire the actual rotational movement amount of the medical elongate body 4 along the arrow 11e by correcting the movement amount in the rotational direction from the acquired tilt angle by geometric calculation. .

In addition, since the medical elongate body 4 advances and retreats while rotating, the area of the characteristic region 12b may decrease by a predetermined area threshold or more. For example, when the shape of the characteristic region 12b changes from a rhombus to a triangle (see FIG. 2), the area of the characteristic region 12b decreases by a predetermined area threshold or more. If the area of the feature region 12b decreases by more than a predetermined area threshold, then the point of interest 12a needs to be updated. When the interest point 12a is updated, the computer 1 re-extracts the feature region 12b including the interest point 12a, and calculates the movement amount based on the re-extracted feature region 12b.

　Figs. 5A, 5B and 5C are explanatory diagrams explaining the update process of the interest point 12a. FIG. 5A is an explanatory diagram of the characteristic region 12b whose area has not changed. FIG. 5B is an explanatory diagram of a characteristic region 12b having an area equal to or larger than a predetermined area threshold. FIG. 5C is an explanatory diagram of a characteristic region 12b whose area is less than a predetermined area threshold. Note that FIG. 5A is the same as (2) in FIG. 3A, so the description is omitted.

The computer 1 calculates the area (the number of pixels in the region) of the extracted characteristic region 12b. The computer 1 determines whether the calculated area of the characteristic region 12b is equal to or larger than a predetermined area threshold. When the computer 1 determines that the area of the characteristic region 12b is equal to or larger than the predetermined area threshold, it determines that the point of interest 12a does not need to be updated, and performs the movement amount calculation process. As shown in FIG. 5B, the long medical body 4 advances and retreats while rotating, and the shape of the characteristic region 12b changes into a polygon. Although the shape of the characteristic region 12b has changed to a polygon, the area of the characteristic region 12b is equal to or greater than the predetermined area threshold, so the computer 1 does not update the interest point 12a.

When the computer 1 determines that the area of the characteristic region 12b is less than the predetermined area threshold, it determines that the point of interest 12a needs to be updated. As shown in FIG. 5C, the medical elongated body 4 advances and retreats while rotating, the shape of the characteristic region 12b changes to a triangle, and the area of the characteristic region 12b is less than the predetermined area threshold. In this case, the computer 1 calculates the center of gravity 12e of the characteristic region 12b by center-of-gravity calculation using pixel values in the characteristic region 12b. The computer 1 updates the interest point 12a based on the position of the center of gravity 12e of the characteristic region 12b.

Specifically, the computer 1 acquires the vertical center line 12c of the region 11b formed on the peripheral surface of the medical elongate body 4. The computer 1 determines whether the position of the center of gravity 12e is to the left or right of the vertical center line 12c. When the computer 1 determines that the position of the center of gravity 12e is on the right side of the longitudinal center line 12c along the X-axis direction of the elongated body 4 for medical use, the computer 1 determines the position of the center of gravity 12e by the specified number of pixels (for example, 5 pixels). position to the left. When the computer 1 determines that the position of the center of gravity 12e is to the left of the vertical center line 12c along the X-axis direction of the medical elongated body 4, the computer 1 moves the position of the center of gravity 12e to the right by the specified number of pixels. do.

The computer 1 updates the position of the center of gravity after movement as the point of interest 12a, and based on the updated point of interest 12a, for example, uses a binarization method and a labeling process to determine a feature region 12b that includes the point of interest 12a. Re-extract. The computer 1 sets the re-extracted characteristic region 12b as the second characteristic region 12b. In this way, when the area of the characteristic region 12b has decreased by a predetermined area threshold or more, the point of interest 12a can be moved to the left and right regions (second characteristic regions 12b) of the characteristic region 12b.

Next, the movement amount display processing will be explained. The computer 1 outputs the specified lateral movement amount and rotational movement amount to the display device 5 . The display device 5 displays the amount of lateral movement and the amount of rotational movement output from the computer 1 .

FIG. 6 is an explanatory diagram showing an example of the movement amount display screen. The screen includes a circumferential image display field 13a and a movement amount display field 13b. The circumferential image display field 13 a is a display field for displaying a circumferential image of the circumferential surface of the medical elongate body 4 . The movement amount display field 13b is a display field for displaying the movement amount of the medical elongate body 4 in the horizontal direction and the movement amount in the rotational direction.

The computer 1 acquires peripheral surface images of the peripheral surface of the medical elongate body 4 within a predetermined period in time series. The predetermined period is, for example, 1 second, 2 seconds, or 5 seconds. For example, the computer 1 acquires a plurality of peripheral surface images per second in time series. The computer 1 identifies the amount of lateral movement and the amount of rotational movement of the elongated medical body 4 based on a plurality of circumferential images acquired in time series. It should be noted that the moving amount specifying process is the same as the above-described process, and thus the description thereof is omitted. The computer 1 outputs to the display device 5 the peripheral surface image of the medical elongated body 4 and the specified lateral movement amount and rotational movement amount.

On the display device 5, the peripheral image is displayed in the peripheral image display field 13a, and the lateral movement amount and the rotational movement amount of the medical elongate body 4 are displayed in the movement amount display field 13b. As shown, the lateral travel is 6 millimeters (symbol mm) and the rotational travel is 8 degrees (deg.). The movement amount display field 13b displays the absolute rotation angle or the relative rotation angle of the movement amount in the rotational direction. As shown in FIG. 3B, the rotation angle is calculated by geometrical calculation from the radius A of the elongated medical body 4 and the amount of movement in the rotation direction. Note that the amount of movement in the rotational direction may be displayed in units of millimeters or pixels.

In order to acquire time-series peripheral images from the imaging device 3 in real time, the computer 1 re-specifies the amount of movement of the long medical object 4 based on the new peripheral images. The computer 1 transmits to the display device 5 the new peripheral image and the re-specified amount of movement of the long medical object 4 . The display device 5 displays the circumferential image transmitted from the computer 1 and the amount of movement of the medical elongate body 4 . In this way, by synchronously displaying the peripheral image and the movement amount of the medical elongate body 4, the latest movement amount can always be displayed on the screen together with the peripheral image.

Also, the specified amount of movement can be displayed in a graph format.
FIG. 7 is an explanatory diagram showing an example of a display screen of a graph showing the relationship between the amount of movement and the elapsed time. The screen includes a first graph display field 14a and a second graph display field 14b. The first graph display field 14a is a display field for displaying a first graph showing the relationship between the amount of lateral movement of the medical elongate body 4 and the elapsed time. The second graph display field 14b is a display field for displaying a second graph showing the relationship between the amount of movement of the medical elongate body 4 in the rotational direction and the elapsed time.

The computer 1 acquires peripheral surface images of the peripheral surface of the medical elongate body 4 from the imaging device 3 in time series. The computer 1 identifies the amount of lateral movement and the amount of rotational movement of the elongated medical body 4 based on a plurality of circumferential images acquired in time series. The computer 1 generates a first graph and a second graph based on the specified lateral movement amount and rotational movement amount. Specifically, the computer 1 acquires the acquisition time of each circumferential image. The computer 1 generates a first graph based on the acquisition time of each circumferential image and the amount of lateral movement corresponding to the acquisition time. The computer 1 generates a second graph based on the acquisition time of each circumferential image and the amount of movement in the rotational direction corresponding to the acquisition time.

The computer 1 outputs the generated first and second graphs to the display device 5. As shown, on the display device 5, the first graph is displayed in the first graph display column 14a and the second graph is displayed in the second graph display column 14b. The horizontal axis of the first graph indicates time (eg, seconds), and the vertical axis of the first graph indicates the amount of horizontal movement (eg, mm). Note that the unit of the vertical axis of the first graph may be pixels. The horizontal axis of the second graph indicates time (eg, seconds), and the vertical axis of the second graph indicates the movement amount (eg, deg.). Note that the unit of the vertical axis of the second graph may be millimeters or pixels. In addition, in FIG. 7, although the example of the line graph was demonstrated, it is not restricted to this, For example, a bar graph etc. may be used.

In addition, the above-described first and second graphs and the circumferential surface image of the circumferential surface of the medical elongated body 4 can be displayed at the same time.
FIG. 8 is an explanatory diagram showing an example of a screen that simultaneously displays a circumferential image and a graph. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 7, and description is abbreviate|omitted. FIG. 8 further includes a peripheral surface image display field 14c. The peripheral image display field 14c is a display field for displaying a peripheral image of the peripheral surface of the long medical body 4 inserted into the hollow organ.

The computer 1 acquires peripheral surface images of the peripheral surface of the medical elongate body 4 from the imaging device 3 in time series. The computer 1 identifies the amount of lateral movement and the amount of rotational movement of the elongated medical body 4 based on a plurality of circumferential images acquired in time series. The computer 1 generates a first graph and a second graph based on the specified lateral movement amount and rotational movement amount.

The computer 1 simultaneously outputs to the display device 5 a plurality of peripheral surface images (moving images/videos) acquired in time series and the generated first and second graphs. As shown in the figure, on the display device 5, the peripheral surface image of the medical elongate body 4 is displayed in the peripheral surface image display field 14c, the first graph is displayed in the first graph display field 14a, and the second graph is displayed. is displayed in the second graph display field 14b.

Note that the computer 1 acquires time-series circumferential images from the imaging device 3 in real time, so the movement amount of the medical elongate body 4 is respecified based on new circumferential images. The computer 1 generates a first graph and a second graph from the re-specified movement amount. The computer 1 transmits the new circumferential image and the generated first and second graphs to the display device 5 . The display device 5 displays the peripheral surface image, the first graph, and the second graph transmitted from the computer 1 . In this way, by synchronously displaying the peripheral image and the first and second graphs, the latest movement amount can always be displayed on the screen together with the peripheral image.

Furthermore, the first and second graphs described above and the captured image of the hollow organ can be displayed at the same time. The captured image of the hollow organ includes an angiographic image of the hollow organ imaged by angiography, an image captured by an ultrasonic diagnostic catheter or an optical tomography catheter, and the like.

An example of an angio image will be described below, but it can be similarly applied to captured images of other types of hollow organs. The computer 1 generates the first graph and the second graph in the same manner as the graph generation process described above. A computer 1 acquires an angiographic image of a hollow organ from an angiography apparatus connected to the computer 1 . An angiography apparatus is a device that inserts a catheter into a hollow organ and injects a contrast agent to capture moving images of the condition of blood vessels. The computer 1 outputs to the display device 5 the generated first and second graphs and the acquired angiographic image of the hollow organ. The display device 5 displays the first and second graphs output from the computer 1 and the angiographic image of the hollow organ.

Note that the computer 1 acquires time-series circumferential images from the imaging device 3 in real time, so the movement amount of the medical elongate body 4 is respecified based on new circumferential images. The computer 1 generates a first graph and a second graph from the re-specified movement amount. At the same time, the computer 1 acquires the angiographic image of the hollow organ in real time from the angiography device. The computer 1 transmits the new angiographic image of the hollow organ and the generated first and second graphs to the display device 5 .

The display device 5 displays the angiographic image of the hollow organ transmitted from the computer 1, the first graph and the second graph. In this way, by synchronously displaying the angiographic image of the hollow organ and the first and second graphs, the latest movement amount can always be displayed on the screen together with the angiographic image of the hollow organ.

In addition, in this embodiment, an example of a graph showing the relationship between the amount of movement and the elapsed time has been described, but the present invention is not limited to this. For example, a graph or the like showing the relationship between the cumulative amount of movement and the elapsed time may be generated. In addition to the amount of movement, indicators such as movement speed (horizontal direction movement speed and rotational direction movement speed) may be used. For example, a moving speed histogram may be generated.

In addition, when the moving amount of the skilled operator is acquired from the past reference data, the computer 1 compares the acquired moving amount of the skilled operator with the moving amount specified by the above-described processing, and outputs the result. can be

FIG. 9 is a flow chart showing the processing procedure when outputting the movement amount of the medical elongated body 4 . The control unit 11 of the computer 1 transmits circumferential images of the circumferential surface of the elongated body 4 for medical use within a predetermined period (1 second, 2 seconds, or 5 seconds, etc.) from the imaging device 3 via the communication unit 13 in chronological order. acquire (step S101). For example, the control unit 11 acquires a plurality of circumferential images per second from the imaging device 3 in time series. The control unit 11 executes a preprocessing subroutine for a plurality of circumferential images acquired in time series (step S102). The control unit 11 executes a subroutine for processing to specify the amount of movement of the medical elongate body 4 (step S103). The subroutine for preprocessing and the subroutine for movement amount specifying processing will be described later.

The control unit 11 generates a first graph showing the relationship between the lateral movement amount of the medical elongated body 4 and the elapsed time based on the identified movement amount (step S104). Based on the specified movement amount, the control unit 11 generates a second graph showing the relationship between the movement amount in the rotational direction of the medical elongate body 4 and the elapsed time (step S105). The control unit 11 outputs the generated first graph and second graph to the display device 5 via the display unit 14 (step S106), and ends the process.

FIG. 10 is a flowchart showing the processing procedure of the preprocessing subroutine for the circumferential image. The control unit 11 of the computer 1 acquires an effective image area (image area in which the medical elongate body 4 is shown; effective image area 11a in FIG. 2) for the circumferential image (step S01). For example, the control unit 11 may extract an effective image area in the peripheral image by performing feature extraction processing such as edge detection processing or pattern matching processing based on each pixel information included in the peripheral image. Note that the control unit 11 may receive designation of the effective image area from the input device 6 by the operator or the like via the input unit 15 .

The control unit 11 cuts out the acquired effective image area (step S02). Note that the processing of steps S01 and S02 may be omitted. The control unit 11 converts the peripheral image from a color image to a grayscale image (step S03). In addition, it is not limited to the execution order of the process mentioned above. For example, after performing the process of step S03, you may perform the process of step S01 and step S02.

Based on segmentation processing and labeling processing using an algorithm such as a binarization method, the control unit 11 detects a polygonal region (rhombus, triangle, circle, quadrangle, etc.) formed on the peripheral surface of the long body 4 for medical use. (region 11b in FIG. 2) is extracted from the peripheral surface image (step S04). The control unit 11 terminates the preprocessing subroutine and returns.

FIG. 11 is a flow chart showing the processing procedure of the subroutine of the movement amount specifying processing. The control unit 11 of the computer 1 acquires one peripheral image from a plurality of peripheral images acquired in time series (step S11). The control unit 11 determines whether or not the acquired peripheral image is the first frame peripheral image (step S12). When the control unit 11 determines that the acquired circumferential image is not the first frame circumferential image (NO in step S12), the process proceeds to step S14, which will be described later.

When the control unit 11 determines that the acquired peripheral image is the first frame peripheral image (YES in step S12), the operator or the like is placed at the center of gravity of the peripheral image via the input unit 15, for example. receives the setting of the interest point from the input device 6 (step S13). Note that the interest point may be set at another position. The control unit 11 uses, for example, a binarization method and a labeling process to convert a characteristic region (a polygonal region formed on the circumferential surface of the elongated body 4 for medical use) containing the received interest point into a peripheral surface image. (step S14).

The control unit 11 calculates the centroid position and area of the extracted characteristic region (step S15). The control unit 11 calculates the difference between the center-of-gravity position and the point of interest on the coordinate X axis as the lateral movement amount (step S16). The control unit 11 calculates the difference between the center-of-gravity position and the point of interest on the coordinate Y axis as the amount of movement in the rotational direction (step S17). The control unit 11 determines whether or not the area of the characteristic region is equal to or larger than a predetermined area threshold (step S18).

When the control unit 11 determines that the area of the feature region is less than the predetermined area threshold (NO in step S18), it executes a subroutine for updating the interest point (step S19). Note that the subroutine of the interest point update processing will be described later. The control unit 11 transitions to the process of step S20, which will be described later. When the control unit 11 determines that the area of the characteristic region is equal to or larger than the predetermined area threshold (YES in step S18), the control unit 11 determines that the peripheral surface image is the last frame among the plurality of peripheral surface images. It is determined whether or not there is (step S20). When the control unit 11 determines that the peripheral image is not the last frame (NO in step S20), the process returns to step S11.

When the control unit 11 determines that the circumferential image is the last frame (YES in step S20), it calculates the total amount of lateral movement and the total amount of rotational movement (step S21). The control unit 11 corrects the total amount of movement in the horizontal direction and the total amount of movement in the rotational direction by geometric calculation from the tilt angle of the medical elongate body 4 (step S22). The control unit 11 outputs the corrected total horizontal movement amount as the horizontal movement amount, and the corrected total rotational movement amount as the rotational movement amount to the display device 5 (step S23). The control unit 11 terminates the subroutine for specifying the movement amount and returns.

FIG. 12 is a flowchart showing the processing procedure of a subroutine of interest point update processing. The control unit 11 of the computer 1 acquires the vertical center line (vertical center line 12c in FIG. 3A) of the region (region 11b in FIG. 3A) including the polygonal region formed on the peripheral surface of the medical elongate body 4. (Step S31). The control unit 11 determines whether or not the position of the center of gravity of the characteristic region is on the left side of the vertical center line along the X-axis direction of the medical elongate body 4 (step S32).

If the control unit 11 determines that the center-of-gravity position is to the left of the vertical center line (YES in step S32), the center-of-gravity position is shifted to the right by the specified number of pixels along the X-axis direction of the long body 4 for medical use. (step S33). The control unit 11 updates the position of the center of gravity after movement as the point of interest (step S34). The control unit 11 ends the interest point update processing subroutine and returns.

When the control unit 11 determines that the center-of-gravity position is not on the left side of the vertical center line (NO in step S32), it determines whether the center-of-gravity position of the characteristic region is on the right side of the vertical center line (step S36). If the control unit 11 determines that the center-of-gravity position is on the right side of the vertical center line (YES in step S36), it moves the center-of-gravity position to the left side of the medical elongate body 4 by the specified number of pixels along the X-axis direction. (step S37). The control unit 11 transitions to the process of step S34. If the control unit 11 determines that the position of the center of gravity is not on the right side of the vertical center line (NO in step S36), it ends the interest point update processing subroutine and returns.

FIG. 13 is a flow chart showing a processing procedure for outputting the circumferential image and the movement amount of the medical elongate body 4. FIG. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 9, and description is abbreviate|omitted. After executing the process of step S105, the control unit 11 of the computer 1 uses the display unit 14 to display a plurality of peripheral surface images (moving images/video) obtained in time series within a predetermined period (for example, 1 second) and , the generated first and second graphs are simultaneously output to the display device 5 (step S107). Control unit 11 terminates the process.

FIG. 14 is a flow chart showing a processing procedure for outputting an angiographic image of a hollow organ and the amount of movement of the elongated body 4 for medical use. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 9, and description is abbreviate|omitted. After executing the process of step S105, the control unit 11 of the computer 1 acquires an angiographic image of the hollow organ from the angiography apparatus connected to the computer 1 (step S111). The control unit 11 outputs the obtained angiographic image of the hollow organ and the generated first and second graphs to the display device 5 via the display unit 14 (step S112). Control unit 11 terminates the process.

Next, a process of outputting a notification when the identified movement amount exceeds a predetermined movement amount threshold will be described.
FIG. 15 is a flow chart showing a processing procedure for outputting a notification. The control unit 11 of the computer 1 acquires a predetermined horizontal movement amount threshold value and a rotational movement amount threshold value stored in advance in the storage unit 12 (step S121). Note that the movement amount threshold may be stored in the large-capacity storage unit 17 or an external device.

The control unit 11 acquires circumferential images of the circumferential surface of the long medical object 4 within a predetermined time (for example, 30 seconds) from the imaging device 3 via the communication unit 13 in chronological order (step S122). The control unit 11 specifies the amount of lateral movement and the amount of rotational movement of the elongated medical body 4 within a predetermined period of time based on the plurality of acquired circumferential images (step S123). It should be noted that the process of specifying the movement amount is the same as the process of specifying the movement amount described above, so the description thereof will be omitted.

The control unit 11 determines whether or not the lateral movement amount exceeds a predetermined lateral movement amount threshold (step S124). When determining that the lateral movement amount exceeds the threshold (YES in step S124), the control unit 11 outputs a notification to the display device 5 via the display unit 14 (step S125). The notification includes, for example, the specified movement amount and a predetermined movement amount threshold. Note that means such as voice notification, LED flash notification, vibration notification, or the like may be used. The control unit 11 returns to the process of step S122.

If the control unit 11 determines that the amount of lateral movement does not exceed the threshold (NO in step S124), it determines whether the amount of movement in the rotational direction exceeds a predetermined threshold of movement in the rotational direction ( step S126). If the controller 11 determines that the movement amount in the rotational direction has exceeded the threshold (YES in step S126), the process proceeds to step S125. If the controller 11 determines that the movement amount in the rotational direction does not exceed the threshold (NO in step S126), the process returns to step S122.

According to the notification process described above, the operator can be notified when the operator's movement operation is too fast. Alternatively, when an operator presses an operation button or the like to hold the medical elongated body 4 at the current position, if an unintended movement operation or the like occurs, the operator can be notified. can.

According to this embodiment, it is possible to specify the amount of movement of the medical elongated body 4 based on the peripheral surface image of the peripheral surface of the medical elongated body 4 inserted into the hollow organ.

According to the present embodiment, the first graph showing the relationship between the amount of lateral movement of the long medical body 4 and the elapsed time, and the relationship between the amount of movement in the rotational direction of the long medical body 4 and the elapsed time It is possible to output a second graph showing

According to this embodiment, it is possible to simultaneously output the circumferential image and the first and second graphs.

According to this embodiment, it is possible to simultaneously output the captured image of the hollow organ and the first and second graphs.

According to this embodiment, the operation of the medical elongated body 4 is quickly adjusted by outputting a notification when the movement amount of the medical elongated body 4 within a predetermined time exceeds a predetermined movement amount threshold. becomes possible.

(Embodiment 2)
Embodiment 2 relates to an embodiment in which the amount of movement of the medical elongated body 4 is specified by artificial intelligence (AI) based on a peripheral surface image of the peripheral surface of the medical elongated body 4 . In addition, description is abbreviate|omitted about the content which overlaps with Embodiment 1. FIG.

FIG. 16 is an explanatory diagram illustrating the configuration of the catheter system 10 according to Embodiment 2. FIG. In addition, the same code|symbol is attached|subjected about the content which overlaps with FIG. 1, and description is abbreviate|omitted. The large-capacity storage unit 17 of the computer 1 contains a movement amount identification model 171 and a training data DB (database) 172 .

The movement amount identification model 171 is a specifier that identifies (estimates) the movement amount of the medical elongate body 4 based on a plurality of circumferential images of the medical elongate body 4 acquired in time series. A trained model generated by learning. The training data DB 172 stores training data for constructing (creating) the movement amount identification model 171 .

FIG. 17 is an explanatory diagram showing an example of the record layout of the training data DB 172. FIG. The training data DB 172 includes input data strings and output data strings. The input data string stores a plurality of peripheral surface images of the long medical object 4 acquired in time series within a predetermined period. The predetermined period is, for example, 1 second, 2 seconds, or 5 seconds. In order to obtain the amount of movement of the long medical object 4, two or more circumferential images are stored in the input data string. The output data string stores the amount of lateral movement and the amount of rotational movement of the medical elongate body 4 .

FIG. 18 is an explanatory diagram for explaining the movement amount identification model 171. FIG. The movement amount identification model 171 is used as a program module that is part of artificial intelligence software. The movement amount identification model 171 receives a plurality of peripheral surface images of the medical elongated body 4 acquired in time series as input, and calculates the movement amount of the medical elongated body 4 (horizontal movement amount and rotational movement amount ) is an identifier with a built-up neural network.

The computer 1 generates the movement amount identification model 171 by performing deep learning for learning the feature amount of the circumferential surface of the medical elongate body 4 in the circumferential surface image as the movement amount identification model 171 . For example, the movement amount identification model 171 is a combination of CNN (Convolution Neural Network) and LSTM (Long-Short Term Memory). LSTM is a type of RNN (Recurrent Neural Network), and is a neural network that receives time-series data before a prediction target time as input and outputs a prediction value at the target time.

The movement amount identification model 171 includes a CNN layer that accepts input of a plurality of circumferential images of the medical elongate body 4 and extracts feature amounts, etc., and an LSTM layer that performs arithmetic processing on the feature amounts extracted from the CNN layer. (hidden layer) and an output layer for outputting the movement amount of the medical elongated body 4 predicted based on the feature amount.

A CNN layer and an LSTM layer are provided in parallel at each time point (eg, t1, t2, . . . tn). The CNN layer has an input layer that receives inputs of a plurality of circumferential images of the elongated body 4 for medical use, and an intermediate layer that has been learned by back propagation. The input layer has a plurality of neurons that receive input of pixel values of pixels included in circumferential images at a plurality of consecutive points in time, and passes the input pixel values to the intermediate layer. The intermediate layer has a plurality of neurons for extracting features of each circumferential image, and passes the extracted image features to the LSTM layer. The intermediate layer is composed of a convolution layer that convolves the pixel values of each pixel input from the input layer and a pooling layer that maps the pixel values convolved in the convolution layer, which are alternately connected. While compressing the pixel information of the image, the feature amount of the image is finally extracted.

The LSTM layer is called an LSTM block, temporarily stores its own calculation result at time t1, and when performing calculations on input data at time t2 next to time t1, it performs calculations on input data at time t1 before. Calculation is performed by referring to the result. By referring to the calculation result at the previous time point t1, calculation at the next time point t3 is performed from the time-series data up to the most recent time point t2. That is, the output data of the LSTM block at the previous time is used as the input data of the LSTM block at the next time.

The output layer has neurons for calculating output values based on the calculation results in the LSTM layer, and outputs the prediction result of the movement amount (horizontal movement amount and rotational movement amount) of the long body 4 for medical use. Output.

For example, the computer 1 uses training data accumulated in the training data DB 172 to generate the movement amount identification model 171 . Each record of the training data DB 172 is training data. The value of the output data string is the correct data (movement amount) to be output from the output layer. A plurality of circumferential surface images of the input data string are the input data. In the training data, a plurality of circumferential images of the long medical object 4 acquired in time series correspond to the amount of movement of the long medical object 4 (the amount of movement in the lateral direction and the amount of movement in the rotational direction). It is the data of the combination attached. Training data is generated based on a large amount of empirical data collected from operators who operate the medical elongate body 4 . Note that the training data may be data separately created manually.

The computer 1 performs learning using the acquired training data. Specifically, the computer 1 sequentially inputs a plurality of peripheral surface images of the long body 4 for medical use acquired in time series to each neuron corresponding to the input layer of the CNN layer in time series, and Calculate the predicted value of the movement amount of the medical elongated body 4 in . The computer 1 compares the actual moving amount of the elongated body 4 for medical use at the target time as a correct value with the predicted value, and constructs a moving amount specifying model 171 .

As described above, the computer 1 inputs a plurality of peripheral surface images, which are parameters, to the movement amount identification model 171 and performs learning. Specifically, the computer 1 sequentially inputs a plurality of circumferential images to each neuron of the input layer of the CNN layer. The computer 1 extracts the feature amount of each circumferential image in the intermediate layer of the CNN layer, and transfers the extracted image feature amount to the LSTM layer. The computer 1 performs operations in the LSTM layer. The computer 1 outputs the predicted value of the movement amount of the elongated body 4 for medical use calculated in the LSTM layer to the output layer and inputs it to the next neuron. That is, the predicted values of the movement amount of the long medical object 4 calculated in each LSTM layer are generated as time-series data.

The computer 1 compares the actual amount of movement of the elongated body 4 for medical use at the target time point with the predicted value as the correct value, and optimizes the parameters used for calculation in the LSTM layer so that the predicted value approximates the correct value. do. The parameters are, for example, weights (coupling coefficients) between neurons, coefficients of activation functions, and the like. Although the parameter optimization method is not particularly limited, for example, the computer 1 uses error backpropagation to optimize various parameters. The computer 1 acquires the predicted value of the movement amount of the long medical body 4 corresponding to each peripheral image as an output from each neuron of the output layer.

The computer 1 performs the above processing for each record stored in the training data DB 172 and learns the movement amount identification model 171 . Thereby, a model that can specify the amount of movement of the medical elongated body 4 can be constructed. Note that another computer (not shown) may perform the above-described learning process to generate the movement amount identification model 171 . In this case, the computer 1 acquires and deploys the movement amount identification model 171 generated by another computer. Note that the movement amount may be identified by using a WEB API (Application Programming Interface) using a machine learning model without constructing the movement amount identification model 171 .

The computer 1 identifies (predicts) the movement amount of the medical elongated body 4 using the generated movement amount identification model 171 . Specifically, the computer 1 acquires a plurality of peripheral surface images (for example, a first frame image, a second frame image, . The computer 1 inputs the plurality of acquired peripheral images to the movement amount identification model 171 and outputs a prediction result of the movement amount of the medical elongate body 4 .

For example, the amount of lateral movement of the elongated medical body 4 in the first frame image is "+2 mm", and the amount of movement in the rotational direction is "-12 deg.". The amount of lateral movement of the elongated medical body 4 in the second frame image is "+3 mm", and the amount of movement in the rotational direction is "-5 deg.". In this way, it is possible to output the prediction result of the lateral movement amount and the rotational movement amount in each circumferential surface image (frame image).

In addition, the movement amount identification model 171 is not limited to the combination of CNN and LSTM, RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), It can be implemented by other models such as SVM (Support Vector Machine), Bayesian Networks, Transformer Networks, Regression Trees or Random Forests.

In this embodiment, an example of the movement amount specifying model 171 that outputs both the amount of movement in the lateral direction and the amount of movement in the rotational direction has been described, but the present invention is not limited to this. For example, the computer 1 may generate a lateral movement amount identification model and a rotational movement amount identification model.

FIG. 19 is a flow chart showing the processing procedure of a subroutine of movement amount identification processing using the movement amount identification model 171 . The control unit 11 of the computer 1 inputs a plurality of circumferential surface images of the elongated body 4 for medical use acquired in the process of step S101 of FIG. 9 to the movement amount identification model 171 (step S41). The control unit 11 outputs the identification result of identifying the movement amount of the medical elongate body 4 from the movement amount identification model 171 (step S42). The control unit 11 terminates the subroutine for specifying the movement amount and returns.

According to the present embodiment, the movement amount of the medical elongate body 4 is specified using the movement amount identification model 171 based on a plurality of peripheral surface images of the medical elongate body 4 acquired in time series. becomes possible.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the meaning described above, and is intended to include all changes within the meaning and scope equivalent to the scope of the claims.

1 Information processing device (computer)
REFERENCE SIGNS LIST 11 control unit 12 storage unit 13 communication unit 14 display unit 15 input unit 16 reading unit 17 large-capacity storage unit 171 movement amount identification model 172 training data DB
1a portable storage medium 1b semiconductor memory 1P control program 2 Y connector 3 imaging device 4 medical elongate body 5 display device 6 input device 10 catheter system

Claims (12)

Acquiring peripheral images of the peripheral surface of a medical elongate body inserted into a hollow organ in time series,
A program that causes a computer to execute a process of identifying the amount of movement of the elongated body for medical use based on a plurality of circumferential images acquired in time series. extracting a feature region from the time-series circumferential image of the medical elongate body;
2. The program according to claim 1, wherein the amount of lateral or rotational movement between the centers of gravity of the characteristic regions is specified based on the extracted characteristic regions. 3. The program according to claim 2, wherein when the area of the characteristic region has decreased by a predetermined area threshold or more, a second characteristic region different from the characteristic region is set. 4. The program according to claim 2, wherein a polygonal area formed on the circumferential surface of the elongated body for medical use that advances and retreats while rotating is extracted as the feature area. When a plurality of circumferential images of the medical elongated body acquired in time series are input, a learning model trained to output a movement amount of the medical elongated body is provided with 5. The program according to any one of claims 1 to 4, wherein a plurality of circumferential images are input to identify the amount of movement of the elongated body for medical use. The program according to any one of claims 1 to 5, wherein the peripheral surface image is acquired from an imaging device that images the peripheral surface of the medical elongate body. A first graph showing the relationship between the amount of lateral movement of the elongated medical body and elapsed time, and a second graph showing the relationship between the amount of rotational movement of the elongated medical body and elapsed time. The program according to any one of claims 1 to 6, which outputs. 8. The program according to claim 7, which outputs the first graph, the second graph, and a peripheral surface image obtained by imaging the peripheral surface of the medical elongate body. 9. The program according to claim 7, which outputs the first graph, the second graph, and the picked-up image of the hollow organ. Acquiring the amount of movement of the medical elongated body within a predetermined time,
10. The program according to any one of claims 1 to 9, wherein a notification is output when the acquired movement amount exceeds a predetermined movement amount threshold. an acquisition unit that acquires, in time series, circumferential images of the circumferential surface of the elongated body for medical use that is inserted into the hollow organ;
An information processing apparatus comprising: a specifying unit that specifies the amount of movement of the elongated medical object based on a plurality of circumferential images acquired in time series. Acquiring peripheral images of the peripheral surface of a medical elongate body inserted into a hollow organ in time series,
An information processing method in which a computer executes a process of identifying the amount of movement of the elongated medical object based on a plurality of circumferential images acquired in time series.

Images