JP7623175B2 - Computer program, operation method of image processing device, and image processing device - Google Patents

Description

The present invention relates to a computer program, an image processing method, and an image processing device.

Conventionally, imaging diagnostic devices have been widely used for the diagnosis of arteriosclerosis, preoperative diagnosis during intravascular treatment using high-performance catheters such as balloon catheters or stents, or postoperative result confirmation. Imaging diagnostic devices include intravascular ultrasound imaging devices (IVUS: IntraVascular Ultrasound), optical coherence tomography imaging devices (OCT: Optical Coherence Tomography), etc., each of which has different characteristics.
In addition, an imaging diagnostic device that combines the functions of IVUS and OCT has been proposed, that is, an imaging diagnostic device that has an ultrasound transceiver unit capable of transmitting and receiving ultrasound and an optical transceiver unit capable of transmitting and receiving light (for example, Patent Documents 1 and 2).
With this type of imaging diagnostic device, it is possible to construct both an ultrasound tomographic image that takes advantage of IVUS's ability to measure deep areas, and an optical coherence tomographic image that takes advantage of OCT's ability to measure at high resolution, with a single scan.

Japanese Patent Application Publication No. 11-56752 Special Publication No. 2010-508973 Re-table 2014/162367 publication

However, the scale of the constructed ultrasound tomographic images and optical coherence tomographic images do not always match, which makes it difficult to interpret the tomographic images.

It is also possible to store the characteristics of the device and match the scale of the ultrasound tomographic image and the optical coherence tomographic image. However, in order to transmit and receive light using the optical transmitting and receiving unit, it is necessary to replace the blood in the blood vessels using a flushing liquid, and the ultrasound output from the ultrasound transmitting and receiving unit propagates through the flushing liquid rather than through the blood. Since the propagation speed of the ultrasound differs depending on the type of flushing liquid, the scale of the ultrasound tomographic image and the optical coherence tomographic image will not match. The type and concentration of the flushing liquid are not uniquely determined, and may be changed depending on the patient's condition. For this reason, there is a limit to the correction process of the propagation speed of the ultrasound according to the type of flushing liquid, and it is not possible to match the scale of the ultrasound tomographic image and the optical coherence tomographic image. Furthermore, if the propagation speed of the ultrasound is set incorrectly, ultrasound tomographic images and optical coherence tomographic images with inconsistent scales will be constructed.

In one aspect, the present invention aims to provide a computer program, an image processing method, and an image processing device that can align the scale of ultrasound tomographic images and optical coherence tomographic images obtained using a dual-type catheter equipped with IVUS and OCT functions.

In one aspect, the computer program causes the computer to execute a process of acquiring an ultrasonic tomographic image and an optical coherence tomographic image using a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves from the inner cavity of a tubular organ in a radially outward direction and an optical transceiver unit that transmits and receives light, recognizing a first inner lumen image of the tubular organ included in the acquired ultrasonic tomographic image, recognizing a second inner lumen image of the tubular organ included in the acquired optical coherence tomographic image, and correcting the scale of the ultrasonic tomographic image or the optical coherence tomographic image in the radial direction based on the recognized first inner lumen image and second inner lumen image so that the dimensions of the first inner lumen image and the second inner lumen image are consistent.

In one aspect, the image processing method acquires ultrasonic tomographic images and optical coherence tomographic images using a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves from the inner cavity of a tubular organ toward the outside in the radial direction and an optical transceiver unit that transmits and receives light, recognizes a first lumen image of the tubular organ contained in the acquired ultrasonic tomographic images, recognizes a second lumen image of the tubular organ contained in the acquired optical coherence tomographic images, and corrects the radial scale of the ultrasonic tomographic images or the optical coherence tomographic images based on the recognized first lumen image and second lumen image so that the dimensions of the first lumen image and the second lumen image are consistent.

In one aspect, the image processing device includes an acquisition unit that acquires ultrasonic tomographic images and optical coherence tomographic images using a sensor unit having an ultrasonic transceiver that transmits and receives ultrasonic waves from the inner cavity of a tubular organ toward the outside in the radial direction and an optical transceiver that transmits and receives light, a first recognition unit that recognizes a first lumen image of the tubular organ included in the ultrasonic tomographic image acquired by the acquisition unit, a second recognition unit that recognizes a second lumen image of the tubular organ included in the optical coherence tomographic image acquired by the acquisition unit, and a correction unit that corrects the scale of the ultrasonic tomographic image or the optical coherence tomographic image in the radial direction based on the first lumen image and the second lumen image recognized by the first recognition unit and the second recognition unit so that the dimensions of the first lumen image and the second lumen image are consistent.

In one aspect, the scale of ultrasound tomographic images and optical coherence tomographic images obtained using a dual-type catheter equipped with IVUS and OCT functions can be matched.

FIG. 1 is an explanatory diagram showing an example of the configuration of an image diagnostic apparatus. FIG. 2 is an explanatory diagram showing an example of the configuration of a catheter for diagnostic imaging. 1 is an explanatory diagram illustrating a cross section of a blood vessel through which a sensor portion is inserted; FIG. FIG. FIG. 1 is a block diagram showing an example of the configuration of an image processing device. A block diagram showing an example configuration of a first learning model. A block diagram showing an example configuration of a second learning model. FIG. 2 is a block diagram showing an example of the configuration of an image quality improvement learning model. 1 is a flowchart showing a method for generating an image quality improvement learning model. 4 is a flowchart showing an image processing procedure according to the first embodiment. 1 is an explanatory diagram showing a three-dimensional lumen image constructed based on an object extraction IVUS image and an object extraction OCT image. 1 is an explanatory diagram conceptually showing a method for matching three-dimensional intraluminal images by rigid registration and correcting the observation position, observation direction, and scale. FIG. 1 is an explanatory diagram conceptually showing a correspondence relationship between an IVUS image and an OCT image and a method of correcting the scale. 13 is an explanatory diagram conceptually showing a scale correction method for an inner cavity portion. FIG. 13 is an explanatory diagram conceptually showing a method of scale correction for the portion outside the lumen. FIG. 10 is a flowchart showing an image processing procedure according to a second embodiment. 10 is an explanatory diagram showing a method of correcting the correspondence relationship between ultrasonic line data and optical line data; FIG. FIG. 11 is an explanatory diagram showing a method of correcting the scale of ultrasonic line data and optical line data.

The computer program, image quality improvement learning model, learning model generation method, image processing method, and image processing device of the present disclosure will be described in detail below with reference to the drawings showing the embodiments. In each of the following embodiments, cardiac catheterization, which is an intravascular treatment, will be described as an example, but the tubular organs that are the subject of the catheterization treatment are not limited to blood vessels, and may be other tubular organs such as the bile duct, pancreatic duct, bronchi, and intestines. Furthermore, the present invention is not limited to these examples, but is indicated by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims. At least some of the embodiments described below may be combined in any manner.

In this embodiment, an imaging diagnostic device using a dual-type catheter equipped with the functions of both intravascular ultrasound (IVUS) and optical coherence tomography (OCT) will be described. The dual-type catheter is provided with a mode for acquiring ultrasonic tomographic images using IVUS only, a mode for acquiring optical coherence tomographic images using OCT only, and a mode for acquiring tomographic images using both IVUS and OCT, and these modes can be switched for use. Hereinafter, ultrasonic tomographic images and optical coherence tomographic images will be referred to as IVUS images and OCT images, respectively, as appropriate. In addition, IVUS images and OCT images will be collectively referred to as tomographic images.

(Embodiment 1)
FIG. 1 is an explanatory diagram showing a configuration example of an image diagnostic apparatus 100. The image diagnostic apparatus 100 of this embodiment includes an intravascular inspection apparatus 101, an angiography apparatus 102, an image processing apparatus 3, a display device 4, and an input device 5. The intravascular inspection apparatus 101 includes an image diagnostic catheter 1 and an MDU (Motor Drive Unit) 2. The image diagnostic catheter 1 is connected to the image processing apparatus 3 via the MDU 2. The display device 4 and the input device 5 are connected to the image processing apparatus 3. The display device 4 is, for example, a liquid crystal display or an organic EL display, and the input device 5 is, for example, a keyboard, a mouse, a trackball, or a microphone. The display device 4 and the input device 5 may be stacked together to form a touch panel. The input device 5 and the image processing apparatus 3 may also be configured as an integrated unit. Furthermore, the input device 5 may be a sensor that accepts gesture input or gaze input.

The angiography device 102 is connected to the image processing device 3. The angiography device 102 is an angiography device that images the blood vessels of a patient from outside the patient's body using X-rays while injecting a contrast agent into the patient's blood vessels, and obtains an angiogram, which is a fluoroscopic image of the blood vessels. The angiography device 102 is equipped with an X-ray source and an X-ray sensor, and images an X-ray fluoroscopic image of the patient by the X-ray sensor receiving X-rays irradiated from the X-ray source. The angiography device 102 outputs the angiogram obtained by imaging to the image processing device 3, and the angiography device 102 displays the angiogram and a tomographic image of the blood vessels imaged using the diagnostic imaging catheter 1 on the display device 4.

2 is an explanatory diagram showing an example of the configuration of the diagnostic imaging catheter 1. The area surrounded by the dashed line on the upper side in FIG. 2 is an enlarged view of the area surrounded by the dashed line on the lower side. The diagnostic imaging catheter 1 has a probe 11 and a connector portion 15 arranged at the end of the probe 11. The probe 11 is connected to the MDU 2 via the connector portion 15. In the following description, the side of the diagnostic imaging catheter 1 far from the connector portion 15 is described as the tip side, and the connector portion 15 side is referred to as the base end side. The probe 11 has a catheter sheath 11a, and at its tip, a guidewire insertion portion 14 through which a guidewire can be inserted is provided. The guidewire insertion portion 14 forms a guidewire lumen, which is used to receive a guidewire inserted in advance into a blood vessel and guide the probe 11 to the affected area by the guidewire. The catheter sheath 11a forms a tube portion that is continuous from the guidewire insertion portion 14 to the connector portion 15. A shaft 13 is inserted inside the catheter sheath 11a, and a sensor unit 12 is connected to the tip of the shaft 13.

The sensor unit 12 has a housing 12c, and the tip side of the housing 12c is formed in a hemispherical shape to suppress friction and catching with the inner surface of the catheter sheath 11a. In the housing 12c, an ultrasonic transmission/reception unit 12a that transmits ultrasonic waves into a blood vessel and receives reflected waves from the inside of the blood vessel, and an optical transmission/reception unit 12b that transmits near-infrared light into the blood vessel and receives reflected light from the inside of the blood vessel are arranged. In the example shown in Fig. 2, the ultrasonic transmission/reception unit 12a is provided on the tip side of the probe 11, and the optical transmission/reception unit 12b is provided on the base end side. That is, the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b are arranged in the housing 12c, separated by a predetermined length along the axial direction on the central axis of the shaft 13 (on the two-dot chain line in Fig. 2).
The ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b are arranged so that the transmission/reception directions of the ultrasonic waves and the near-infrared light are approximately 90 degrees to the axial direction of the shaft 13 (the radial direction of the shaft 13). It is preferable that the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b are attached slightly offset from the radial direction so as not to receive the reflected waves and the reflected light on the inner surface of the catheter sheath 11a. In this embodiment, for example, as shown by the arrow in Fig. 2, the ultrasonic transmission/reception unit 12a is provided in a position in which the direction of irradiation of ultrasonic waves is inclined toward the base end side with respect to the radial direction, and the optical transmission/reception unit 12b is provided in a position in which the direction of irradiation of near-infrared light is inclined toward the tip end side with respect to the radial direction.

An electric signal cable (not shown) connected to the ultrasonic transmission/reception unit 12a and an optical fiber cable (not shown) connected to the optical transmission/reception unit 12b are inserted into the shaft 13. The probe 11 is inserted into the blood vessel from the tip side. The sensor unit 12 and the shaft 13 can move forward and backward inside the catheter sheath 11a and can rotate in the circumferential direction. The sensor unit 12 and the shaft 13 rotate around the central axis of the shaft 13 as the rotation axis.

The MDU 2 is a driving device to which the connector 15 of the diagnostic imaging catheter 1 is detachably attached, and which drives a built-in motor in response to the operation of a medical professional to control the operation of the diagnostic imaging catheter 1 inserted into a blood vessel. For example, the MDU 2 performs a pull-back operation to rotate the sensor unit 12 and shaft 13 inserted into the probe 11 in a circumferential direction while pulling them toward the MDU 2 at a constant speed. The sensor unit 12 moves from the distal end side to the proximal end side and rotates, and scans the blood vessel continuously at a predetermined time interval to continuously capture a plurality of transverse slice images approximately perpendicular to the probe 11 at a predetermined interval. The MDU 2 outputs a reflected wave signal of the ultrasound received by the ultrasound transmitting/receiving unit 12a and a reflected light signal received by the light transmitting/receiving unit 12b to the image processing device 3.
The MDU 2 has a high-speed pullback mode in which the sensor unit 12 is retracted at high speed, and a low-speed pullback mode in which the sensor unit 12 is retracted at low speed. In the high-speed pullback mode, the MDU 2 drives the motor at high speed to retract the sensor unit 12 at high speed. In the low-speed pullback mode, the MDU 2 drives the motor at low speed to retract the sensor unit 12 at low speed. The image processing device 3 recognizes whether it is operating in the high-speed pullback mode or the low-speed pullback mode, and executes processing according to the pullback speed. Details will be described later.

The image processing device 3 acquires the reflected wave signal of the ultrasound output from the ultrasound transmission/reception unit 12a via the MDU 2 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data. The ultrasound line data is data indicating the reflection intensity of the ultrasound in the depth direction of the blood vessel as seen from the ultrasound transmission/reception unit 12a. The image processing device 3 constructs an IVUS image 60 (see FIG. 4) showing the transverse layer of the blood vessel based on the generated ultrasound line data.

The image processing device 3 also obtains interference light data by interfering the reflected light signal output from the optical transceiver 12b via the MDU 2 with the reference light obtained by separating the light from the light source, and generates optical line data based on the obtained interference light data. The optical line data is data indicating the reflection intensity of the reflected light in the depth direction of the blood vessel as seen from the optical transceiver 12b. The image processing device 3 constructs an OCT image 70 (see FIG. 4) showing the transverse layers of the blood vessel based on the generated optical line data.

Here, we will explain the ultrasonic line data and optical line data obtained by the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b, and the IVUS image 60 and the OCT image 70 constructed from the ultrasonic line data and the optical line data.

Figure 3 is an explanatory diagram showing a schematic cross-section of a blood vessel through which the sensor unit 12 has been inserted, and Figure 4 is an explanatory diagram of a tomographic image. First, using Figure 3, the operation of the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b within the blood vessel, and the ultrasonic line data and optical line data obtained by the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b will be explained.

When the sensor unit 12 and the shaft 13 are inserted into the blood vessel and imaging of a tomographic image is started, the sensor unit 12 rotates in the direction indicated by the arrow, with the central axis of the shaft 13 as the center of rotation. At this time, the ultrasonic transmission/reception unit 12a transmits and receives ultrasonic waves at each rotation angle. Lines 1, 2, ... 512 indicate the transmission and reception direction of ultrasonic waves at each rotation angle. In this embodiment, the ultrasonic transmission/reception unit 12a intermittently transmits and receives ultrasonic waves 512 times during a 360-degree rotation (one rotation) in the blood vessel. The ultrasonic transmission/reception unit 12a obtains one line of data in the transmission and reception direction by transmitting and receiving ultrasonic waves once, so that 512 pieces of ultrasonic line data extending radially from the center of rotation can be obtained during one rotation. The 512 pieces of ultrasonic line data are dense near the center of rotation, but become sparse as they move away from the center of rotation. Therefore, the image processing device 3 can generate pixels in the empty spaces of each line using a well-known interpolation process, thereby constructing a two-dimensional IVUS image 60 as shown in FIG. 4A.

Similarly, the optical transceiver 12b also transmits and receives near-infrared light (measurement light) at each rotation angle. The optical transceiver 12b also transmits and receives measurement light 512 times while rotating 360 degrees within the blood vessel, so that 512 optical line data extending radially from the center of rotation can be obtained during one rotation. For the optical line data, the image processing device 3 generates pixels in the empty spaces of each line by known interpolation processing, thereby constructing the two-dimensional OCT image 70 shown in FIG. 4A.

A two-dimensional tomographic image constructed from a plurality of ultrasonic line data in this manner is called one frame of IVUS image 60. A two-dimensional tomographic image constructed from a plurality of optical line data is called one frame of OCT image 70. Since the sensor unit 12 scans while moving inside the blood vessel, one frame of IVUS image 60 or OCT image 70 is acquired at each position of one rotation within the movement range. That is, one frame of IVUS image 60 or OCT image 70 is acquired at each position from the tip side to the base end side of the probe 11 within the movement range, so that multiple frames of IVUS image 60 or OCT image 70 are acquired within the movement range, as shown in FIG. 4B.

Note that the number of times that ultrasonic waves and light are transmitted and received in one rotation is an example, and the number of times that transmission and reception is not limited to 512. In addition, the number of times that ultrasonic waves are transmitted and received and the number of times that light is transmitted and received may be the same or different.

In order to minimize the amount of contrast agent required to obtain the OCT image 70, it is desirable to pull back the sensor unit 12 at a high speed. However, since it takes time for the transmitted ultrasound to be reflected at the target site and return, the intravascular inspection device 101 reduces the number of transmissions and receptions when the pullback speed is increased. In other words, the intravascular inspection device 101 reduces the number of acquisitions of ultrasound line data so that it can receive reflected waves at a required depth in the blood vessel. Note that, since the propagation speed of near-infrared light is sufficiently fast, the number of acquisitions of optical line data can be maintained regardless of the movement speed of the sensor unit 12.
For example, when the speed at which the sensor unit 12 is retracted is slow (slow pullback), the image processing device 3 acquires 512 pieces of ultrasonic line data and optical line data during one rotation of the sensor unit 12, whereas when the speed at which the sensor unit 12 is retracted is fast (high-speed pullback), the image processing device 3 acquires 256 pieces of ultrasonic line data and 512 pieces of optical line data during one rotation of the sensor unit 12.
As described above, the intravascular inspection device 101 and the image processing device 3 reduce the number of acquired ultrasound line data by a larger amount than the number of acquired optical line data when the retraction speed of the sensor unit 12 is fast. The number of acquired optical line data may be maintained or reduced.

Furthermore, the intravascular inspection device 101 may be configured to shorten the transmission and reception time of ultrasonic waves when the pullback speed is increased. In this case, the depth direction of the blood vessel to be imaged becomes shorter.
When the pullback speed is increased in this way, the image quality of the IVUS image 60 is reduced. Hereinafter, the high-image-quality IVUS image 60 constructed when the speed at which the sensor unit 12 is retracted is a first speed is referred to as a high-image-quality IVUS image 60b, and the low-image-quality IVUS image 60 constructed when the speed at which the sensor unit 12 is retracted is a second speed faster than the first speed is referred to as a low-image-quality IVUS image 60a. The high-image-quality IVUS image 60b and the low-image-quality IVUS image 60a mean that the image quality of one IVUS image 60 is higher or lower than the image quality of the other IVUS image 60 when comparing the image quality of both. When describing technical features that do not require a distinction between the high-image-quality IVUS image 60b and the low-image-quality IVUS image 60a, they are simply referred to as IVUS images 60 without distinction between them.

Figure 5 is a block diagram showing an example of the configuration of the image processing device 3. The image processing device 3 is a computer, and includes a processing unit 31, a memory unit 32, an ultrasonic line data generating unit 33, an optical line data generating unit 34, an input/output I/F 35, and a reading unit 36. The processing unit 31 is configured using one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), a TPU (Tensor Processing Unit), etc. The processing unit 31 is connected to each of the hardware components constituting the image processing device 3 via a bus.

The memory unit 32 has, for example, a main memory unit and an auxiliary memory unit. The main memory unit is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores data required for the processing unit 31 to execute arithmetic processing. The auxiliary memory unit is a storage device such as a hard disk, EEPROM (Electrically Erasable Programmable ROM), or flash memory. The auxiliary memory unit stores the computer program P executed by the processing unit 31, the first learning model 6, the second learning model 7, the image quality improvement learning model 8, and various data required for other processing. The auxiliary memory unit may be an external storage device connected to the image processing device 3. The computer program P may be written to the auxiliary memory unit during the manufacturing stage of the image processing device 3, or the image processing device 3 may acquire the computer program P distributed by a remote server device through communication and store it in the auxiliary memory unit. The computer program P may be recorded in a readable manner on a recording medium 30 such as a magnetic disk, optical disk, or semiconductor memory, and the reading unit 36 may read it from the recording medium 30 and store it in the auxiliary storage unit.

The ultrasound line data generation unit 33 acquires the reflected wave signal of the ultrasound output from the ultrasound transmission/reception unit 12a of the intravascular inspection device 101 as reflected wave data, and generates ultrasound line data based on the acquired reflected wave data.

The optical line data generating unit 34 obtains interference light data by causing interference between the reflected light signal output from the optical transmitting/receiving unit 12b of the intravascular inspection device 101 and the reference light obtained by separating the light from the light source, and generates optical line data based on the obtained interference light data.

The input/output I/F 35 is an interface to which the intravascular inspection device 101, the angiography device 102, the display device 4, and the input device 5 are connected. The processing unit 31 acquires an angio image from the angiography device 102 via the input/output I/F 35. The processing unit 31 also displays a medical image on the display device 4 by outputting a medical image signal of the IVUS image 60, the OCT image 70, or the angio image to the display device 4 via the input/output I/F 35. Furthermore, the processing unit 31 accepts information input to the input device 5 via the input/output I/F 35.

The image processing device 3 may be a multi-computer including multiple computers. The image processing device 3 may also be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following description, the image processing device 3 is described as being one computer. In this embodiment, the image processing device 3 is connected to an angiography device 102 that captures two-dimensional angio images, but is not limited to the angiography device 102 as long as it is a device that captures images of the patient's tubular organs and the diagnostic imaging catheter 1 from multiple directions outside the body.

The processing unit 31 of the image processing device 3 reads out and executes the computer program P stored in the memory unit 32, thereby generating ultrasound line data in the ultrasound line data generation unit 33, and executing a process of constructing an IVUS image 60 based on the generated ultrasound line data. The processing unit 31 also reads out and executes the computer program P stored in the memory unit 32, thereby generating optical line data in the optical line data generation unit 34, and executing a process of constructing an OCT image 70 based on the generated optical line data.

However, since the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b are arranged at different positions in the sensor unit 12 and have different transmission and reception directions of ultrasonic waves and light, even if the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b scan the blood vessel at the same imaging timing, a deviation occurs in the observation position and observation direction of the blood vessel. The deviation of the observation position is a deviation in the longitudinal direction and running direction of the blood vessel. The deviation of the observation direction is a deviation in the circumferential direction of the blood vessel. In addition, in order to transmit and receive light using the optical transmission/reception unit 12b, it is necessary to replace the blood in the blood vessel using a flushing liquid. Since the propagation speed of ultrasonic waves differs depending on the type of flushing liquid, the scales of the ultrasonic tomographic image and the optical coherence tomographic image will no longer match.
Therefore, the processing unit 31 reads out and executes the computer program P stored in the storage unit 32, thereby correcting deviations in the observation position and observation direction in the IVUS image 60 and the OCT image 70 when constructing the IVUS image 60 and the OCT image 70, and executing processing to match the scales of the IVUS image 60 and the OCT image 70. In this manner, the image processing device 3 of this embodiment can construct the IVUS image 60 and the OCT image 70 with the same observation position, observation direction, and scale, and can provide images that are easy to interpret.

As described above, in order to minimize the amount of contrast agent required to obtain the OCT image 70, it is desirable to pull back the sensor unit 12 at high speed, but in this case, a problem occurs in that the image quality of the IVUS image 60 deteriorates. Therefore, the processing unit 31 reads out and executes the computer program P stored in the memory unit 32, thereby performing a process of generating and constructing a high-image-quality IVUS image 60b using the low-image-quality IVUS image 60a and the OCT image 70 whose image quality does not deteriorate.

Next, we will explain the first learning model 6 and the second learning model 7 used to correct the deviations in the observation position and the observation direction and to correct the scale of the ultrasound tomographic image and the optical coherence tomographic image. We will also explain the image quality improvement learning model 8 that improves the image quality of the low-image-quality IVUS image 60a.

Figure 6 is a block diagram showing an example of the configuration of the first learning model 6. The first learning model 6 is a model that recognizes a predetermined object included in the IVUS image 60. The first learning model 6 can classify objects on a pixel-by-pixel basis by using an image recognition technique using, for example, semantic segmentation, and can recognize various objects included in the IVUS image 60. Specifically, the first learning model 6 recognizes a lumen image (first lumen image) in the IVUS image 60. The first learning model 6 also recognizes an object image (first object image) outside the lumen in the IVUS image 60, such as a blood vessel wall image, a plaque image, a stent image, etc. The blood vessel wall is the tunica media, and more specifically, the external elastic membrane (EEM). The first learning model 6 may be configured to recognize the center position of the sensor unit 12.

The first learning model 6 is, for example, a convolutional neural network (CNN) that has been trained by deep learning. The first learning model 6 has an input layer 6a to which an IVUS image 60 is input, an intermediate layer 6b that extracts and restores features of the IVUS image 60, and an output layer 6c that outputs an object-extracted IVUS image 61 that shows an object image included in the IVUS image 60 in pixel units. The first learning model 6 is, for example, a U-Net, a Fully Convolution Network (FCN), a SegNet, etc.

The input layer 6a of the first learning model 6 has a plurality of neurons that accept input of the IVUS image 60, that is, the pixel values of each pixel that constitutes the IVUS image 60, and passes the input pixel values to the intermediate layer 6b. The intermediate layer 6b has a plurality of convolution layers (CONV layers) and a plurality of deconvolution layers (DECONV layers). The convolution layer is a layer that compresses the dimensions of the IVUS image 60. The feature amount of the object image is extracted by the dimensional compression. The deconvolution layer performs a deconvolution process to restore the original dimensions. The restoration process in the deconvolution layer generates an object-extracted IVUS image 61 in which each pixel has a pixel value (class data) corresponding to the class of the object. The output layer 6c has a plurality of neurons that output the object-extracted IVUS image 61. As shown in FIG. 6, the object-extracted IVUS image 61 is an image that is classified, for example, color-coded, according to the type of object, such as a lumen image, a vascular wall image, a plaque image, and a stent image.

The first learning model 6 can be generated by preparing training data having an IVUS image 60 obtained by the ultrasound transmission/reception unit 12a and an object-extracted IVUS image 61 in which each pixel of the IVUS image 60 is annotated with class data according to the type of object, and performing machine learning on an untrained neural network using the training data. Specifically, the parameters of the neural network are optimized so that the difference between the object-extracted IVUS image 61 output when the IVUS image 60 of the training data is input to the untrained neural network and the image annotated as the training data is reduced. The parameters are, for example, weights (coupling coefficients) between nodes. The method of optimizing the parameters is not particularly limited, but for example, the processing unit 31 optimizes various parameters using the steepest descent method or the like.

According to the first learning model 6 trained in this way, by inputting an IVUS image 60 into the first learning model 6 as shown in FIG. 6, an object-extracted IVUS image 61 is obtained in which each part is classified on a pixel-by-pixel basis according to the type of object, such as a blood vessel lumen image, a blood vessel wall image, or a plaque image.

Figure 7 is a block diagram showing an example of the configuration of the second learning model 7. The second learning model 7 is a model that recognizes a specific object included in the OCT image 70. Like the first learning model 6, the second learning model 7 can classify objects on a pixel-by-pixel basis by using an image recognition technique using, for example, semantic segmentation, and can recognize various objects included in the OCT image 70. Specifically, the second learning model 7 recognizes a lumen image (second lumen image) in the OCT image 70. The second learning model 7 also recognizes object images (second object images) outside the lumen in the OCT image 70, such as blood vessel wall images, plaque images, and stent images.

The second learning model 7 is, for example, a convolutional neural network that has been trained by deep learning. The second learning model 7 has an input layer 7a to which an OCT image 70 is input, an intermediate layer 7b that extracts and restores features of the OCT image 70, and an output layer 7c that outputs an object-extracted OCT image 71 that shows an object image included in the OCT image 70 in pixel units. The second learning model 7 is, for example, a U-Net, a Fully Convolution Network (FCN), a SegNet, etc.

The input layer 7a of the second learning model 7 has a plurality of neurons that accept input of the OCT image 70, that is, the pixel values of each pixel that constitutes the OCT image 70, and passes the input pixel values to the intermediate layer 7b. The intermediate layer 7b has a plurality of convolution layers and a plurality of deconvolution layers. The convolution layer is a layer that compresses the dimensions of the OCT image 70. The feature amount of the object image is extracted by the dimensional compression. The deconvolution layer performs deconvolution processing to restore the original dimensions. The restoration processing in the deconvolution layer generates an object extraction OCT image 71 in which each pixel has a pixel value (class data) corresponding to the class of the object. The output layer 7c has a plurality of neurons that output the object extraction OCT image 71. The object extraction OCT image 71 is an image that is classified, for example, color-coded, according to the type of object, such as an intraluminal image, a vascular wall image, a plaque image, and a stent image.

The second learning model 7 can be generated by preparing training data having an OCT image 70 obtained by the optical transmission/reception unit 12b and an object-extracted OCT image 71 in which each pixel of the OCT image 70 is annotated with class data according to the type of object that corresponds to the pixel, and using the training data to train an untrained neural network. Specifically, similar to the learning of the first learning model 6, the parameters of the neural network are optimized so that the difference between the object-extracted OCT image 71 output when the OCT image 70 of the training data is input to the untrained neural network and the image annotated as the training data is reduced.

According to the second learning model 7 trained in this way, by inputting an OCT image 70 into the second learning model 7 as shown in FIG. 7, an object-extracted OCT image 71 is obtained in which each part is classified on a pixel-by-pixel basis according to the type of object, such as a blood vessel lumen image, a blood vessel wall image, or a plaque image.

Figure 8 is a block diagram showing an example of the configuration of the image quality improvement learning model 8. The image quality improvement learning model 8 is a model that generates a high-quality IVUS image 60b based on a low-quality IVUS image 60a and an OCT image 70 captured at approximately the same observation position. Even during high-speed pullback, the image quality improvement learning model 8 can generate a high-quality IVUS image 60b by supplementing and reconstructing the information of the low-quality IVUS image 60a with the OCT image 70, since the image quality of the OCT image 70 does not deteriorate.

The image quality improvement learning model 8 is, for example, a convolutional neural network that has been trained by deep learning. The image quality improvement learning model 8 has a first input layer 81a to which a low-quality IVUS image 60a is input, a second input layer 82a to which an OCT image 70 is input, an intermediate layer 8b that extracts and restores features of the low-quality IVUS image 60a and the OCT image 70, and an output layer 8c that outputs a high-quality IVUS image 60b restored based on the extracted features. The image quality improvement learning model 8 is, for example, a U-Net.

The first input layer 81a of the image quality improvement learning model 8 has a plurality of neurons that accept input of the pixel values of each pixel that constitutes the low-quality IVUS image 60a, that is, the IVUS image 60, and passes the input pixel values to the intermediate layer 8b. The second input layer 82a has a plurality of neurons that accept input of the OCT image 70, that is, the pixel values of each pixel that constitutes the OCT image 70, and passes the input pixel values to the intermediate layer 8b. The intermediate layer 8b has a plurality of convolution layers and a plurality of deconvolution layers. The convolution layer is a layer that compresses the dimensions of the low-quality IVUS image 60a and the OCT image 70. The feature amounts of the low-quality IVUS image 60a and the OCT image 70 are extracted by the dimensional compression. The deconvolution layer performs deconvolution processing to restore one high-quality IVUS image 60b. The output layer 8c has a plurality of neurons that output the high-quality IVUS image 60b.

Figure 9 is a flowchart showing a method for generating an image quality improvement learning model 8. Here, as an example, an example is described in which an image processing device 3 generates an image quality improvement learning model 8, but the type of computer that generates the image quality improvement learning model 8 is not particularly limited.

First, the image processing device 3 is set to a high-speed pullback mode in which the sensor unit 12 is pulled back at high speed, and acquires a low-quality IVUS image 60a and an OCT image 70 of a predetermined blood vessel region (step S101). The low-quality IVUS image 60a and the OCT image 70 obtained when high-speed pullback is performed are the IVUS image 60 and the OCT image 70 constructed by scanning the blood vessel at approximately the same observation position.
Then, the image processing device 3 sets the sensor unit 12 to a low-speed pullback mode in which the sensor unit 12 is pulled back at a low speed, and acquires a high-quality IVUS image 60b of the same vascular region (step S102). The image processing device 3 accumulates the low-quality IVUS image 60a and the OCT image 70 acquired in steps S101 and S102 when a high-speed pullback is performed, and the high-quality IVUS image 60b when a low-speed pullback is performed, as training data (step S103). The image processing device 3 accumulates a large amount of training data by repeating the processes of steps S101 to S103. Of course, the image processing device 3 may be configured to collect training data acquired and accumulated by other image diagnostic devices 100.

It is preferable that the low-quality IVUS image 60a, OCT image 70, and high-quality IVUS image 60b constituting the training data are images obtained from the same observation position and observation direction. A method for aligning the observation positions and observation directions of multiple frames of low-quality IVUS image 60a and OCT image 70 by comparing intraluminal images will be described later. By using this method, it is possible to align the observation positions and observation directions of the low-quality IVUS image 60a, OCT image 70, and high-quality IVUS image 60b, which are the training data.

Then, the image processing device 3 generates an untrained neural network by machine learning using the accumulated training data (step S104). Specifically, the parameters of the neural network are optimized so that the difference between the IVUS image 60 output when the low-quality IVUS image 60a and the OCT image 70 of the training data are input to the untrained neural network is reduced, and the high-quality IVUS image 60b of the training data is reduced. The parameters are, for example, weights (coupling coefficients) between nodes. The method of optimizing the parameters is not particularly limited, but for example, the processing unit 31 optimizes various parameters using the steepest descent method or the like.

According to the image quality improvement learning model 8 thus trained, as shown in FIG. 8, a high-quality IVUS image 60b is obtained by inputting a low-quality IVUS image 60a and an OCT image 70 captured at approximately the same observation position using the sensor unit 12.

Figure 10 is a flowchart showing the image processing procedure according to the first embodiment. The ultrasonic line data generating unit 33 and the optical line data generating unit 34 of the image processing device 3 generate multiple ultrasonic line data and optical line data based on the reflected wave data output from the ultrasonic transmission/reception unit 12a and the interference light data obtained by interference between the reflected light and reference light output from the optical transmission/reception unit 12b (step S111). The multiple ultrasonic line data and optical line data are assigned line numbers in chronological order of the observation time, for example. The line number corresponds to the observation time point. In other words, the line number corresponds to the observation position and observation direction.

Next, the processing unit 31 constructs a plurality of frames of IVUS images 60 based on the ultrasound line data, and constructs a plurality of frames of OCT images 70 based on the light line data (step S112). The plurality of frames of IVUS images 60 and OCT images 70 are assigned frame numbers, for example, in chronological order of the observation time. The frame numbers correspond to the observation positions. The plurality of frames of IVUS images 60 and OCT images 70 correspond to images obtained by observing blood vessels at a plurality of observation positions from the tip side to the base side of the probe 11. However, since the arrangement of the ultrasound transmission/reception unit 12a and the light transmission/reception unit 12b and the transmission and reception directions of ultrasound or light are different, the observation positions of the IVUS images 60 and the OCT images 70 assigned the same frame number do not necessarily match. In other words, the IVUS images 60 and the OCT images 70 are misaligned in the running direction of the blood vessels. For the same reason, the IVUS image 60 and the OCT image 70 are also misaligned in the circumferential direction of the blood vessel. Furthermore, to obtain the OCT image 70, the blood is replaced with a flushing liquid, and depending on the type of flushing liquid, etc., there is also a misalignment in the scale of the IVUS image 60 and the OCT image 70.

The processing unit 31, the ultrasound line data generating unit 33, and the optical line data generating unit 34 that execute the processes of steps S111 and S112 function as an acquisition unit that acquires an IVUS image 60 (ultrasonic tomographic image) and an OCT image 70 (optical coherence tomographic image).

Next, the processing unit 31 inputs the IVUS image 60 into the first learning model 6 to recognize the lumen image in the IVUS image 60 and the object image outside the lumen (e.g., a blood vessel wall image, a plaque image, a stent image) (step S113). The processing unit 31 executes image recognition processing for each of the multiple frames of the IVUS image 60. Note that the object-extracted IVUS image 61 output from the first learning model 6 is assigned the same frame number as the input IVUS image 60 or a similar frame number. The processing unit 31 that executes the processing of step S113 also functions as a first recognition unit that recognizes the lumen image (first lumen image) in the IVUS image 60.

The processing unit 31 also recognizes the lumen image in the OCT image 70 and the object image outside the lumen (e.g., a blood vessel wall image, a plaque image, a stent image) by inputting the OCT image 70 into the second learning model 7 (step S114). The processing unit 31 executes image recognition processing for each of the multiple frames of the IVUS image 60. Note that the object-extracted OCT image 71 output from the second learning model 7 is assigned the same frame number as the input OCT image 70 or a similar frame number. The processing unit 31 that executes the processing of step S114 also functions as a second recognition unit that recognizes the lumen image (second lumen image) in the OCT image 70.

The object images that can be recognized by the first learning model 6 and the second learning model 7 are not necessarily the same, but the processing unit 31 can at least recognize the lumen image in the IVUS image 60 and the lumen image in the OCT image 70. In addition, some of the object images, such as blood vessel walls, plaques, and stents, that are outside the lumen image are recognized as common object images in both the IVUS image 60 and the OCT image 70.

11 is an explanatory diagram showing three-dimensional lumen images 65, 75 constructed based on an object-extraction IVUS image 61 and an object-extraction OCT image 71. The object-extraction IVUS image 61 of multiple frames shows the lumen region at multiple observation positions in the running direction of the blood vessel, and the processing unit 31 can construct a three-dimensional lumen image 65 based on images of the lumen region included in the object-extraction IVUS image 61 of multiple frames, as shown in the upper diagram of FIG.
Similarly, the object extraction OCT images 71 of multiple frames show the lumen region at multiple observation positions in the direction in which the blood vessel runs, and the processing unit 31 can construct a three-dimensional lumen image 75 based on the images of the lumen region contained in the object extraction OCT images 71 of multiple frames, as shown in the lower diagram of Figure 11.

Next, the processing unit 31 specifies the translation amount, rotation amount, and scale correction coefficient of the three-dimensional lumen image 65 that matches the position, orientation, and dimensions of the three-dimensional lumen image 65, 75 by rigid registration (step S115). That is, the processing unit 31 specifies the correspondence of frame numbers in which the lumen image in the multiple frames of the IVUS image 60 matches the lumen image in the multiple frames of the OCT image 70. The processing unit 31 also specifies the rotation amount of the IVUS image 60 that matches the orientation of the lumen image in the IVUS image 60 and the orientation of the lumen image in the OCT image 70. The processing unit 31 also specifies the radial scale correction amount of the lumen portion in the IVUS image 60.

Fig. 12 is an explanatory diagram conceptually showing a method of matching three-dimensional lumen images 65, 75 by rigid registration and correcting the observation position, observation direction, and scale. In Fig. 12, the upper diagram is a three-dimensional lumen image 65 based on an object-extracted IVUS image 61, and the lower diagram is a three-dimensional lumen image 75 based on an object-extracted OCT image 71. The three-dimensional lumen image 65 and the three-dimensional lumen image 75 are misaligned in the observation position, observation direction, and scale in the radial direction of the blood vessel.
The processing unit 31 translates the three-dimensional lumen image 65 based on the object-extracted IVUS image 61, rotates it in the circumferential direction, and expands and contracts the blood vessel lumen in the radial direction by rigid registration , for example, as shown in the upper and center figures, to specify the translation amount, rotation amount, and scale correction coefficient of the lumen part at which the position, orientation, and dimensions of the three-dimensional lumen images 65 and 75 match. Specifically, the processing unit 31 calculates the similarity or dissimilarity between the three-dimensional lumen image 65 and the three-dimensional lumen image 75 in the three-dimensional coordinate system, and specifies the translation amount, rotation amount, and scale correction coefficient of the lumen part at which the similarity is maximized or the dissimilarity is minimized. For example, normalized cross-correlation or the like may be used as the similarity. For the dissimilarity, the sum of squared residuals or the like may be used. The processing unit 31 may calculate the similarity or dissimilarity of the contours of the three-dimensional lumen images 65 and 75.

13 is an explanatory diagram conceptually showing a method of correcting the correspondence and scale between the IVUS image 60 and the OCT image 70. Translation of the three-dimensional lumen image 65 by rigid registration corresponds to correction of the correspondence between the frame number of the IVUS image 60 and the frame image of the OCT image 70. Rotation of the three-dimensional lumen image 65 corresponds to image rotation processing in the IVUS image 60.

The processing unit 31 corrects the correspondence between the IVUS image 60 and the OCT image 70 (step S116). That is, the frame of the IVUS image 60 captured at the same observation position is associated with the frame of the OCT image 70. Specifically, the frame number of the IVUS image 60 captured at the same observation position is associated with the frame number of the OCT image 70. Since the amount of parallel movement identified in step S115 corresponds to the difference between the frame number of the IVUS image 60 captured at the same observation position and the frame number of the OCT image 70, the IVUS image 60 can be associated with the OCT image 70 by adding or subtracting the number of frames corresponding to the difference. The image processing device 3 stores the correspondence between the amount of parallel movement and the difference in frame numbers.

The processing unit 31 also corrects the orientation of the IVUS image 60 so that the orientation of the object image in the IVUS image 60 matches the orientation of the object image in the OCT image 70, i.e., so that the observation directions are the same (step S117). The amount of rotation identified in step S115 corresponds to the amount of deviation in the orientations of the IVUS image 60 and the OCT image 70, i.e., the amount of deviation in the rotation angle, so that the orientations of the IVUS image 60 and the OCT image 70 can be aligned by rotating the IVUS image 60 by the amount of deviation.

Although the correction of the observation position and observation direction by rigid registration of the three-dimensional lumen images 65, 75 has been described, the observation position, observation direction, and scale may also be corrected by directly comparing the object-extracted IVUS image 61 and the object-extracted OCT image 71 without constructing the three-dimensional lumen images 65, 75.

Before the translation, the IVUS image 60 of the nth frame and the OCT image 70 of the nth frame are associated with each other. n is an integer of 1 or more. The processing unit 31 calculates the similarity or difference between the lumen region in the object extraction IVUS image 61 of the corresponding frame number and the lumen region in the object extraction OCT image 71. The processing unit 31 calculates the similarity or difference between the lumen region for a predetermined number of object extraction IVUS images 61 and object extraction OCT images 71. Next, the processing unit 31 similarly calculates the similarity or difference between the lumen region in the object extraction IVUS image 61 of the (n+α)th frame and the lumen region in the object extraction OCT image 71 of the nth frame. α is an integer and corresponds to the amount of translation of the three-dimensional lumen image 65. The processing unit 31 verifies the correspondence between the object extraction IVUS image 61 and the object extraction OCT image 71 while incrementing the variable α by 1.
Furthermore, when comparing the lumen regions, the processing unit 31 rotates the orientation of the lumen region in the object extraction IVUS image 61 in the circumferential direction by an angle correction amount θ. The angle correction amount θ corresponds to the amount of rotation of the three-dimensional lumen image 65. The center of rotation is not the center of the object extraction IVUS image 61, but is rotated around a position corresponding to the center of the sensor unit 12. Furthermore, when comparing the lumen regions, the processing unit 31 expands or contracts the radial dimension of the lumen region in the object extraction IVUS image 61 by a scale correction coefficient k1. The processing unit 31 calculates the similarity or dissimilarity of the lumen region in the object extraction IVUS image 61 and the object extraction OCT image 71 while changing the angle correction amount θ and the scale correction coefficient k by a predetermined amount together with the variable α, and specifies the variable α, angle correction amount θ, and scale correction coefficient k1 that maximize the similarity or minimize the dissimilarity. Then, the processing unit 31 stores the specified variable α, angle correction amount θ, and scale correction coefficient k1.

In addition, the amount of deviation of the lumen region generally falls within a certain range depending on the arrangement of the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b and the characteristics of the available flushing liquid. For this reason, it is preferable to set a predetermined search range for each of the variable α, the angle correction amount θ, and the scale correction coefficient k1. It is preferable to configure the processing unit 31 to specify the variable α, the angle correction amount θ, and the scale correction coefficient k1 that maximizes the similarity or minimizes the dissimilarity of the lumen region within the search range.
In addition, since the frame numbers of the IVUS image 60 and the OCT image 70, which match the observation position, may differ depending on the arrangement of the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b, the irradiation direction of the ultrasonic wave and the near-infrared light, the size of the blood vessel, etc., it is preferable that the search range of the variable α includes positive and negative values. However, it is preferable that the width of the search range on the positive side of the variable α and the width of the search range on the negative side are asymmetric depending on the arrangement of the ultrasonic transmission/reception unit 12a and the optical transmission/reception unit 12b. This makes it possible to specify the variable α more efficiently.

Next, the processing unit 31 corrects the scale of the lumen portion in the IVUS image 60 (step S118).

FIG. 14 is an explanatory diagram conceptually showing a scale correction method for the lumen part. The upper diagram shows an object extraction IVUS image 61 and an IVUS image 60 before correction. The lower diagram shows an object extraction OCT image 71 and an IVUS image 60 after scale correction. The processing unit 31 identifies a scale correction coefficient k1 that matches the size of the lumen image in the IVUS image 60 with the size of the lumen image in the OCT image 70 by comparing the size of the lumen region in the object extraction IVUS image 61 with the size of the lumen region in the object extraction OCT image 71 through the above-mentioned processing. The correction coefficient k1 is a coefficient that corrects the radial length scale of the lumen image centered on the part corresponding to the sensor unit 12 in the IVUS image 60. The processing unit 31 performs scale correction by linearly converting the radial length of the lumen image in the IVUS image 60 using the correction coefficient k1. More specifically, if the distance between an arbitrary object image in the lumen region and the center of the sensor unit 12 is R1, then the distance R1' between the corrected object image and the center of the sensor unit 12 is expressed as R1 x k1. The processing unit 31 determines the correction coefficient k1 such that the contour when the lumen region of the object extraction IVUS image 61 is corrected with the correction coefficient k1 matches the contour of the lumen region in the object extraction OCT image 71, in other words, such that the similarity between the two contours is maximized or the difference is minimized.
Then, in step S118, the processing unit 31 uses the obtained correction coefficient k1 to correct the radial length scale of the portion corresponding to the lumen region in the IVUS image 60. Here, the scale of the lumen outer region is not corrected. If the lumen region shrinks, the lumen outer region is expanded in the circumferential direction by a known interpolation process. If the lumen region expands, the lumen outer region is narrowed in the circumferential direction by a thinning process. Note that the scale correction of the lumen region may be performed using IVUS line data, and the IVUS image 60 may be reconstructed using the scale-corrected IVUS line data.

Next, the processing unit 31 corrects the scale of the portion outside the lumen in the IVUS image 60 (step S119). The processing unit 31, which executes the processes in steps S115 to S119, functions as a correction unit that performs various correction processes such as observation position, observation direction, and scale correction.

15 is an explanatory diagram conceptually showing a method of correcting the scale of the outer part of the lumen. The upper figure shows an object extraction IVUS image 61 and an IVUS image 60 before correction, and the lower figure shows an object extraction OCT image 71 and an IVUS image 60 after scale correction. The processing unit 31 identifies a scale correction coefficient k2 that matches the size of an object image outside the lumen in the IVUS image 60 to the size of an object image outside the lumen in the OCT image 70 by comparing an arbitrary object image outside the lumen region in the object extraction IVUS image 61 with the size of the object image outside the lumen region in the object extraction OCT image 71. The correction coefficient k2 is a coefficient that corrects the radial length scale on the outer side of the lumen centered on the part corresponding to the sensor unit 12 in the IVUS image 60. The processing unit 31 performs scale correction by linearly converting the radial length of the image outside the lumen in the IVUS image 60 using the correction coefficient k2. More specifically, if the distance between an arbitrary object image outside the lumen region and the center of the sensor unit 12 is R2, the distance R2' between the corrected object image and the center of the sensor unit 12 is expressed as R1' + (R2 - R1) x k2. The processing unit 31 obtains a correction coefficient k2 that matches the object image outside the lumen when the lumen region of the object extraction IVUS image 61 is corrected with the correction coefficient k1 with the object image outside the lumen in the object extraction OCT image 71, in other words, that maximizes the similarity between the two object images or minimizes the dissimilarity between the two object images. The processing unit 31 uses the obtained correction coefficient k2 to correct the radial length scale of the part outside the lumen in the IVUS image 60. Note that the scale correction of the lumen outside region may be performed using IVUS line data, and the IVUS image 60 may be reconstructed using the IVUS line data after the scale correction.

Next, if the IVUS image 60 is a low-image-quality IVUS image 60a, the processing unit 31 uses the image quality improvement learning model 8 to perform image quality improvement processing of the low-image-quality IVUS image 60a (step S120). Specifically, as shown in FIG. 8, the low-image-quality IVUS image 60a is input to the first input layer 81a of the image quality improvement learning model 8, and the OCT image 70 is input to the second input layer 81b to generate a high-image-quality IVUS image 60b. Note that a more accurate high-image-quality IVUS image 60b can be obtained by inputting the low-image-quality IVUS image 60a and the OCT image 70, which have matching observation positions, observation directions, and scales, to the image quality improvement learning model 8. Note that the processing unit 31 that executes the processing of step S120 functions as an image quality improvement processing unit that improves the image quality of the IVUS image 60 using the image quality improvement learning model 8.

Then, the processing unit 31 displays the IVUS image 60 and the OCT image 70 with the observation position, observation direction, and scale corrections, as well as the improved image quality, on the display device 4 (step S121).

The processing unit 31 may be configured to first display the low-quality IVUS image 60a before image quality improvement on the display device 4, and each time a high-quality IVUS image 60b is generated, display the low-quality IVUS image 60a instead of the high-quality IVUS image 60b. The processing unit 31 may also perform image quality improvement processing on the priority droplet from the currently displayed low-quality IVUS image 60a.

As described above, the image processing device 3 and the like according to this embodiment 1 can align the observation positions and observation directions of the IVUS image 60 and the OCT image 70 obtained using a dual-type catheter equipped with IVUS and OCT functions.

The scale, particularly the radial scale, of the IVUS image 60 and the OCT image 70 obtained using a dual-type catheter with IVUS and OCT functions can be aligned.

Furthermore, a high-quality IVUS image 60b can be constructed even during high-speed pullback. Specifically, a high-quality IVUS image 60b can be constructed based on a low-quality IVUS image 60a and an OCT image 70. Therefore, it is possible to achieve both high-speed imaging of the IVUS image 60 and the OCT image 70 using a dual-type catheter equipped with IVUS and OCT functions, reduction in contrast agent, and high image quality.

In the first embodiment, an example of recognizing an intraluminal image in an IVUS image 60 and an OCT image 70 using the first learning model 6 and the second learning model 7 has been described, but the contour of the intraluminal image may be extracted by image processing such as binarization, edge detection, and pattern matching.

Also, an example has been described in which the scale correction is performed using different correction coefficients for the lumen portion and the lumen outer portion in the IVUS image 60, but it may be configured such that the radial scale is corrected only for the lumen portion.
Furthermore, although an example has been described in which one scale correction coefficient k1 is used to perform scale correction of the lumen images in all IVUS images 60, it is also possible to specify a correction coefficient k1 for each predetermined number of consecutive frames and configure the scale of the lumen images in the IVUS images 60 to be corrected.

Furthermore, although an example of performing correction of the observation position, observation direction, and radial scale correction of the IVUS image 60 and the OCT image 70 has been described, it may be configured to perform either one or two of the correction of the observation position and observation direction, and the radial scale correction.

Furthermore, in the present embodiment , an example has been described in which the translation amount, rotation amount, and scale correction amount of the three-dimensional lumen images 65, 75 are specified, but in the case where the scale correction coefficient is not specified , only the translation amount and the rotation amount may be specified. Note that while the linear transformation for performing translation , rotation, enlargement, or reduction of the three-dimensional lumen images 65, 75 has been mainly described, a configuration may be made in which a correction amount for aligning the three-dimensional lumen images 65, 75 is calculated by a high-order polynomial or other nonlinear transformation.

Furthermore, an example has been described in which the image quality of the IVUS image 60 is improved using a low-image-quality IVUS image 60a in which the observation position and direction have been matched and scale correction has been performed, and an OCT image 70. However, it is also possible to generate a high-image-quality IVUS image 60b by inputting the low-image-quality IVUS image 60a and the OCT image 70 into the image quality improvement learning model 8 without performing such correction.

Furthermore, in this embodiment 1, an example has been described in which the orientation and scale of the IVUS image 60 are corrected based on the OCT image 70, but it may also be configured to correct the orientation and scale of the OCT image 70 based on the IVUS image 60.
Specifically, in step S115, the processing unit 31 specifies the amount of rotation of the OCT image 70 that matches the orientation of the lumen image in the IVUS image 60 with the orientation of the lumen image in the OCT image 70. The processing unit 31 also specifies the amount of scale correction in the radial direction of the lumen portion in the OCT image 70. For example, the processing unit 31 translates the three-dimensional lumen image 75 based on the object-extracted OCT image 71, rotates it in the circumferential direction, and expands and contracts the blood vessel lumen in the radial direction by rigid registration , thereby specifying the amount of translation, the amount of rotation, and the scale correction coefficient of the lumen portion that match the positions, orientations, and dimensions of the three-dimensional lumen images 65, 75.
Then, in step S117, the processing unit 31 corrects the orientation of the OCT image 70 so that the orientation of the object image in the IVUS image 60 and the orientation of the object image in the OCT image 70 match, that is, so that the observation directions are the same. In addition, in step S118, the processing unit 31 corrects the scale of the inner cavity portion in the OCT image 70, and in step S119, corrects the scale of the outer cavity portion in the OCT image 70.

Furthermore, if there is a third medical image other than the IVUS image 60 and the OCT image 70, both the IVUS image 60 and the OCT image 70 may be configured to be corrected based on the third medical image so that the observation position, observation direction, and scale of the IVUS image 60 and the OCT image 70 are consistent with each other.

(Embodiment 2)
The image diagnostic device 100 according to the second embodiment differs from the first embodiment in that the ultrasound line data and the light line database are used to correct the observation position, observation direction, and scale. Since the other configurations of the image processing device 3 are similar to those of the image processing device 3 according to the first embodiment, the same reference numerals are used for similar parts, and detailed descriptions thereof will be omitted.

16 is a flowchart showing an image processing procedure according to embodiment 2. The processing unit 31 generates ultrasound line data and optical line data (step S211) in the same manner as in embodiment 1. Then, the processing unit 31 uses the ultrasound line data and optical line data to perform image recognition of a lumen image, a vascular wall image, plaque, a stent, and the like (steps S212 and S213).
In addition, the first learning model 6 and the second learning model 7 learn the ultrasonic line data and the light line data constituting one frame as training data, instead of the frame images of the IVUS image 60 and the OCT image 70 constructed as frame images. In other words, an image in which the ultrasonic line data constituting one frame are arranged is an image expressed in polar coordinates with the first axis being the circumferential direction and the second axis being the depth direction. When the ultrasonic line data constituting one frame is input to the first learning model 6, the object-extracted ultrasonic line data from which the object image is extracted is output. Similarly, when the OCT line data constituting one frame is input to the second learning model 7, the object-extracted OCT line data from which the object image is extracted is output.

Then, the processing unit 31 performs corrections of the observation position, observation direction, and scale on a line data basis in the same manner as in embodiment 1 (steps S214 to S218). Note that the processes of steps S215 and S216 may be performed in a single process. The processing unit 31 constructs an IVUS image 60 and an OCT image 70 based on the corrected ultrasound line data and optical line data (step S219). The processes of steps S220 and S221 are the same as those in embodiment 1.

17 is an explanatory diagram showing a method of correcting the correspondence between ultrasound line data and light line data. The upper figure is a schematic diagram showing ultrasound line data, and the lower figure is a schematic diagram showing a light line sensor. As in the first embodiment, the processing unit 31 assumes that the nth line of ultrasound line data and the nth line of light line data are associated with each other before translation. n is an integer of 1 or more. The processing unit 31 calculates, for example, the similarity or difference between the lumen area in the corresponding nth line to mth line of object extraction ultrasound line data and the lumen area in the object extraction light line data. m is an integer greater than n. Next, the processing unit 31 similarly calculates the similarity or difference between the lumen area in the (n+β)th line to (m+β)th line of ultrasound line data and the lumen area in the nth line to mth line of object extraction light line data. The variable β is an integer of 1 or more, and corresponds to the amount of translation and rotation of the three-dimensional lumen image 65. The correspondence between the ultrasound line data and the optical line data is verified while incrementing the variable β by 1. Then, the processing unit 31 identifies the variable β that maximizes the similarity or minimizes the dissimilarity. Then, the processing unit 31 stores the identified variable β.

Figure 18 is an explanatory diagram showing a method of correcting the scale of ultrasound line data and light line data. A scale correction coefficient k1 is specified to match the dimensions of the lumen image of the ultrasound line data so that the contour of the lumen region in the object extraction ultrasound line data matches the contour of the lumen region in the light line data. Similarly, a scale correction coefficient k2 for the outside of the lumen of the ultrasound line data is specified so that the dimensions of the object region outside the lumen in the object extraction ultrasound line data match the dimensions of the object region outside the lumen in the object extraction light line data. The processing unit 31 uses the obtained correction coefficient k1 to correct the radial length scale of the part corresponding to the lumen region in the ultrasound line data. The processing unit 31 also uses the correction coefficient k2 to correct the radial length scale of the part corresponding to the outside of the lumen region in the ultrasound line data.

According to the image processing device 3 etc. of this embodiment 2, as in the first embodiment, it is possible to align the observation position and observation direction of the IVUS image 60 and the OCT image 70 obtained using a dual-type catheter, and further to align the scale.

By performing image processing on the ultrasound line data and the ultrasound line data, the observation position, observation direction, and scale are corrected to construct the IVUS image 60 and the OCT image 70, so corrections can be made more accurately and efficiently to construct the IVUS image 60 and the OCT image 70.

In the second embodiment, an example is described in which the scale of the ultrasound line data is corrected based on the optical line database, but the scale of the optical line database may be corrected based on the ultrasound line data.

1 Diagnostic imaging catheter 2 MDU
Reference Signs List 3 Image processing device 4 Display device 5 Input device 6 First learning model 7 Second learning model 8 Image quality improvement learning model 11 Probe 11a
12 Sensor section 12a Ultrasound transmitting/receiving section 12b Optical transmitting/receiving section 12c Housing 13 Shaft 14 Guide wire insertion section 15 Connector section 30 Recording medium 31 Processing section 32 Storage section 33 Ultrasound line data generating section 34 Optical line data generating section 60 IVUS image 60a Low-quality IVUS image 60b High-quality IVUS image 61 Object-extracted IVUS image 65 Three-dimensional lumen image 70 OCT image 71 Object-extracted OCT image 75 Three-dimensional lumen image 100 Imaging diagnostic device 101 Intravascular inspection device 102 Angiography device

Claims (8)

While the blood in the hollow organ, which is a blood vessel, is being replaced with a flushing liquid , an ultrasonic tomographic image and an optical coherence tomographic image are acquired by a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves from the inner cavity of the hollow organ toward the outside in the radial direction and an optical transceiver unit that transmits and receives light,
Recognizing a first lumen image of the tubular organ included in the acquired ultrasonic tomographic image;
Recognizing a second lumen image of the tubular organ included in the acquired optical coherence tomographic image;
A computer program for causing a computer to execute a process of correcting the scale of the ultrasound tomographic image radially inward from the contour of the first lumen image or the scale of the optical coherence tomographic image radially inward from the contour of the second lumen image so that dimensions of the contour of the first lumen image and the contour of the second lumen image are consistent based on the recognized first lumen image and second lumen image. Recognizing a first object image of a predetermined object located radially outward from the first lumen image included in the acquired ultrasonic tomographic image;
Recognizing a second object image of the predetermined object located radially outward from the second lumen image included in the acquired optical coherence tomographic image;
The computer program of claim 1 , which causes a computer to execute a process of correcting the scale of the ultrasound tomographic image radially outward from the first lumen image or the scale of the optical coherence tomographic image radially outward from the second lumen image so that the dimensions of the first object image and the second object image are consistent based on the recognized first object image and second object image. The first lumen image in the ultrasonic tomographic image is recognized by inputting the acquired ultrasonic tomographic image into a first learning model that recognizes the first lumen image included in the ultrasonic tomographic image;
The computer program of claim 1 or claim 2, which causes a computer to execute a process of recognizing the second intraluminal image in the optical coherence tomographic image by inputting the acquired optical coherence tomographic image into a second learning model that recognizes the second intraluminal image included in the optical coherence tomographic image. The acquired ultrasonic tomographic image is input to a first learning model that recognizes the first lumen image included in the ultrasonic tomographic image and the first object image located radially outward from the first lumen image, thereby recognizing the first lumen image and the first object image in the ultrasonic tomographic image;
The computer program of claim 2, which causes a computer to execute a process of recognizing the second lumen image and the second object image in the optical coherence tomographic image by inputting the acquired optical coherence tomographic image into a second learning model that recognizes the second lumen image included in the optical coherence tomographic image and the second object image located radially outside the second lumen image. a plurality of the ultrasonic tomographic images and a plurality of the optical coherence tomographic images are acquired while moving the sensor unit along a running direction of the tubular organ;
By comparing the first lumen image included in the plurality of ultrasonic tomographic images with the second lumen image included in the plurality of optical coherence tomographic images, the ultrasonic tomographic images and the optical coherence tomographic images which are tomographic images at substantially the same observation position are identified;
The computer program of claim 1 , which causes a computer to execute a process of correcting the radial scale of the ultrasonic tomographic image or the optical coherence tomographic image so that the dimensions of the first lumen image and the second lumen image are consistent based on the first lumen image and the second lumen image contained in the identified ultrasonic tomographic image and the optical coherence tomographic image. The computer program of claim 5, which causes a computer to execute a process of correcting the scale so that the dimensions of the three-dimensional lumen image are consistent by rigid registration of a three-dimensional lumen image based on the recognized first lumen image and a three-dimensional lumen image based on the recognized second lumen image. 1. A method of operating an image processing device, comprising:
The image processing device includes:
While the blood in the tubular organ, which is a blood vessel, is being replaced with a flushing liquid , an ultrasonic tomographic image and an optical coherence tomographic image are constructed from data obtained by a sensor unit having an ultrasonic transceiver unit that transmits and receives ultrasonic waves from the inner cavity of the tubular organ toward the outside in the radial direction and an optical transceiver unit that transmits and receives light,
Recognizing a first lumen image of the tubular organ included in the acquired ultrasonic tomographic image;
Recognizing a second lumen image of the tubular organ included in the acquired optical coherence tomographic image;
A method for operating an image processing device, comprising: correcting, based on the recognized first and second lumen images, the scale of the ultrasound tomographic image radially inward from the contour of the first lumen image or the scale of the optical coherence tomographic image radially inward from the contour of the second lumen image so that dimensions of the contour of the first lumen image and the contour of the second lumen image are consistent. an acquisition unit that acquires ultrasonic tomographic images and optical coherence tomographic images using a sensor unit having an ultrasonic transmission/reception unit that transmits and receives ultrasonic waves from the inner cavity of the tubular organ toward the outside in the radial direction while the blood in the tubular organ, which is a blood vessel, is being replaced with a flushing liquid;
a first recognition unit that recognizes a first lumen image of the tubular organ included in the ultrasonic tomographic image acquired by the acquisition unit;
a second recognition unit that recognizes a second lumen image of the tubular organ included in the optical coherence tomographic image acquired by the acquisition unit;
and a correction unit that corrects the scale of the ultrasonic tomographic image radially inward from the contour of the first lumen image or the scale of the optical coherence tomographic image radially inward from the contour of the second lumen image based on the first lumen image and the second lumen image recognized by the first recognition unit and the second recognition unit so that dimensions of the contour of the first lumen image and the contour of the second lumen image are consistent.

Images