WO2025088983A1 - Image display device and image display control program - Google Patents

Abstract

This image display device acquires a region-of-interest image obtained by attaching markers to the body surface of a patient and capturing an image of a region of interest of the patient together with the markers. A line-of-sight guide unit guides the line of sight of a user to one of the markers in an image display unit. A marker identification unit identifies the marker in a region corresponding to the line of sight of the user guided by the line-of-sight guide unit. A conversion unit performs, for the region-of-interest image, conversion for aligning the marker in the region-of-interest image to the marker identified by the marker identification unit. The region-of-interest image converted by the conversion unit is displayed on the image display unit.

Description

Image display device and image display control program

The present invention relates to an image display device and an image display control program, and more specifically to an image display device and an image display control program for displaying an image of a region of interest.

When performing procedures on patients (injections, incisions, catheter insertion, etc.), doctors need to accurately grasp the position and condition of the patient's organs and tissues before carrying out the procedure. For example, when performing respiratory surgery, a procedure called thoracic drainage is performed. Thoracic drainage is a procedure in which a catheter is inserted into the patient's body and blood, pus, and air that has accumulated in the thoracic cavity is drained out of the body via the catheter. When inserting a catheter into the patient's body for thoracic drainage, the position and condition of the patient's organs must be accurately grasped to avoid the catheter reaching the organs.

When performing thoracic drainage, there is currently no established technology for accurately determining the position of organs and safely inserting a catheter into a patient's body. Generally, when inserting a catheter into a patient's body, an ultrasound device is used to confirm the position of the organs inside the body and the catheter is inserted while avoiding the organs. However, ultrasound devices cannot be used for thoracic drainage because ultrasound waves are reflected by the air in the thoracic cavity. To begin with, ultrasound devices are very difficult to use, and it is very difficult for the doctor in charge of inserting the catheter to perform various procedures while operating the ultrasound device.

In light of this situation, in thoracic drainage, a CT scanner is used to obtain and analyze a chest CT image as an image of the region of interest in advance, and doctors refer to this CT image beforehand when performing thoracic drainage. However, what is obtained from a chest CT image is an image of a cross-section of the body, and understanding the three-dimensional position of organs must depend on the doctor's experience and intuition. Thus, the method of obtaining and analyzing chest CT images in advance is not sufficient to accurately determine the position of the patient's organs, and the current situation is that the insertion of a catheter into the correct position during thoracic drainage depends on the doctor's level of skill. The above are problems with thoracic drainage, but similar problems can occur in other treatments. In other words, when performing various treatments, it is difficult to determine where in the patient's body the treatment tool should be inserted because it is not easy to understand the state of the tissues and organs inside the patient's body.

In recent years, with the advancement of xR technologies such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality), image display systems have been proposed that enable an image of a region of interest and an image of the patient's reality to be superimposed on the same screen on an image display device such as AR glasses (see, for example, Published Japanese Translation of PCT International Publication No. 2021-505226). Such a system allows a doctor to simultaneously observe an image of the region of interest and an image of the patient's reality via the image display device.

　In conventional image display systems, it is necessary to align a pre-captured image of the region of interest with an image of the actual patient (real image). However, simply aligning the image of the region of interest with the image of the patient poses the problem that the amount of calculation required for alignment and image processing after alignment becomes enormous, significantly affecting the display speed. If the alignment operation is imperfect, the image of the region of interest will not be displayed in the correct position on the image of the actual patient, and the above problem will still remain.

The present invention was made in consideration of these problems, and provides an image display device and an image display control program that make it possible to quickly align an image of a region of interest with an image of a real patient with a small amount of calculations.

In order to solve the above problems, the image display device according to the present invention comprises an image acquisition unit that acquires an area of interest image obtained by attaching a marker to the surface of a patient's body and capturing an image of the patient's area of interest together with the marker, an image display unit provided in the user's field of view that includes a gaze guidance unit that guides the user's gaze to one of the markers, a marker identification unit that identifies the marker in the area corresponding to the user's gaze guided by the gaze guidance unit, and a conversion unit that performs a conversion on the area of interest image to align the marker in the area of interest image with the marker identified by the marker identification unit, and the image display unit displays the area of interest image converted by the conversion unit.

The image display control program of the present invention is characterized in that it is configured to enable a computer to execute the steps of: attaching a marker to the surface of a patient's body and capturing an image of the patient's region of interest together with the marker to obtain an image of the region of interest; displaying the image of the region of interest together with the patient to which the marker is attached on an image display unit; guiding the user's line of sight to one of the markers on the image display unit; identifying the marker in the region of interest corresponding to the guided line of sight of the user; performing a transformation on the image of the region of interest to align the marker in the image of the region of interest with the identified marker; and displaying the image of the region of interest after the transformation on the image display unit.

The present invention provides an image display device and an image display control program that enable rapid alignment of an image of a region of interest with an image of a real patient with a small amount of calculation.

1 is a schematic diagram illustrating an overall configuration of a medical procedure support system 1 including an image display device according to a first embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of an image server 20. A block diagram illustrating an example of the configuration of the AR glasses 30. 13 is a conceptual diagram for explaining calculation of coordinates Xm1' to Xm3' of a marker MK in a CT image Pcc and coordinates Xm1 to Xm3 of the marker MK in the AR glasses 30. FIG. 1 is a schematic diagram showing an example of how to attach a marker MK to the body surface of a patient PT. 10 is a flowchart showing a procedure for capturing a CT image Pcc using the CT device 10 and calculating the coordinates of a marker MK. 13 is a flowchart showing a procedure for superimposing and displaying a CT image Pcc and an actual image Ppt of the patient PT through the AR glasses 30. 13 shows an example of a screen display when guiding the line of sight to a marker MK. FIG. 11 is a schematic diagram illustrating a second embodiment. FIG. 11 is a schematic diagram illustrating a second embodiment. 10 is a flowchart illustrating an operation of the system 1 according to the second embodiment. 10 is a flowchart illustrating an operation of the system 1 according to the second embodiment. 13 shows a modification of the second embodiment. FIG. 13 is a block diagram illustrating a medical procedure support system 1 according to a third embodiment. FIG. 13 is a schematic diagram illustrating a configuration of a medical procedure support system 1 according to a fourth embodiment. FIG. 13 is a schematic diagram illustrating a configuration of a medical procedure support system 1 according to a fifth embodiment. FIG. 13 is a schematic diagram illustrating a configuration of a medical procedure support system 1 according to a sixth embodiment. 13 is a schematic diagram showing an example of how to attach a marker MK to the body surface of a patient PT. FIG. 13 is a diagram for explaining determination of a degree of match of a patient. FIG. 13 is a diagram for explaining determination of a degree of match of a patient.

The present embodiment will be described below with reference to the attached drawings. In the attached drawings, functionally identical elements may be indicated by the same numbers. Note that the attached drawings show embodiments and implementation examples according to the principles of the present disclosure, but these are for the purpose of understanding the present disclosure and are in no way intended to limit the interpretation of the present disclosure. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application examples of the present disclosure in any way.

In this embodiment, the disclosure has been described in sufficient detail for a person skilled in the art to implement the disclosure, but it should be understood that other implementations and forms are possible, and that changes to the configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the disclosure. Therefore, the following description should not be interpreted as being limited to this.

[First embodiment]
The overall configuration of a medical procedure support system 1 including an image display device according to a first embodiment will be described with reference to Fig. 1. As an example, the medical procedure support system 1 illustrates a system for supporting the execution of thoracic drainage, but the present invention is not limited to this, and a similar configuration can also be adopted in systems for supporting other procedures, such as central venous pressure measurement (CVP) and epidural anesthesia.

The system in FIG. 1 may, as an example, comprise a CT device 10 as an image capturing device, an image server 20, AR glasses 30 as an image display device, and a display 40. In this system 1, a patient PT undergoing thoracic drainage first acquires a CT image Pcc (Pcc0) of the area of interest near the thoracic cavity in the CT device 10. The CT image Pcc is displayed on the AR glasses 30 when thoracic drainage is performed. Note that in this specification, an image that includes the area of interest and the surrounding organs, etc., such as this CT image Pcc (Pcc0), may be collectively referred to as a "area of interest image."

The patient PT has markers MK affixed to their body surface when imaging with the CT device 10 and when performing thoracic drainage. The markers MK are used to align the CT image Pcc with the actual image of the patient PT in the AR glasses 30. As an example, as shown in FIG. 1, at least three markers MK1 to 3 are affixed to the patient's body surface to surround the area of interest. Various calculations and operations based on the markers MK will be described later. At least three markers MK should be affixed to one patient PT, but it is possible to use more than three markers to achieve more accurate alignment.

After the CT images have been taken, the patient PT moves to the thoracic drainage treatment room and receives thoracic drainage from the doctor DR. The marker MK remains attached in the same position as during CT imaging in order to align the CT image Pcc as described below. The doctor DR in charge of the thoracic drainage wears AR glasses 30 and performs thoracic drainage by observing the actual image Ppt of the patient seen through the AR glasses 30 and the CT image Pcc superimposed on it. When superimposed, the CT image Pcc is subject to a specified transformation using a method described below.

The doctor DR holds the catheter 60 for thoracic drainage and inserts it into the appropriate location within the thoracic cavity based on the CT image Pcc and the patient's actual image Ppt. The doctor DR can also refer to the CT image Pcc displayed on the display 40 as an auxiliary. The doctor DR checks the position of the rib cage and deflated lungs in the CT image Pcc superimposed on the AR glasses 30, and recognizes the dead space within the thoracic cavity. The doctor DR then inserts the catheter 60 between the ribs. The doctor DR proceeds with the insertion of the catheter 60 while checking the CT image Pcc displayed on the AR glasses 30 to make sure that the insertion site is in the correct position and angle.

The CT device 10 is a device for capturing a CT image Pcc of the chest of the patient PT. Here, the CT image Pcc is used to grasp the position and condition of the organs around the area of interest of the patient PT when performing thoracic drainage. The CT device 10 is an example of an imaging device that captures images for grasping the position and condition of the patient's organs, and is not limited to this. Imaging devices other than the CT device 10, such as an MRI (Magnetic Resonance Imaging) device, may be used as long as they can obtain images for grasping the position and condition of the patient's organs.

Furthermore, the CT device 10 may be shared among multiple medical treatment support systems 1. In other words, the medical treatment support system 1 does not need to have its own CT device 10, and it is sufficient if it is configured to be able to acquire CT images taken by a CT device 10 located at an external institution, for example. The CT device 10 and the image server 20 may be connected by a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, a dedicated line, etc. Furthermore, the image server 20 and the AR glasses 30 do not need to be located in the same organization (hospital, etc.), and a system may be set up such that an organization managing the AR glasses 30 can access an image server 20 located in another organization as appropriate.

The image server 20 is a computer that manages the captured CT images Pcc and executes image processing for three-dimensional display in the AR glasses 30. The AR glasses 30 superimpose and display the CT images Pcc obtained from the image server 20 together with the actual image Ppt of the patient PT.

An example of the hardware configuration of the image server 20 will be described with reference to FIG. 2. As an example, the image server 20 may include a CPU (Central Processing Unit) 21 as an arithmetic and control device, an input/output interface 22, a RAM (Random Access Memory) 23, a ROM (Read Only Memory) 24, a storage device 25, a 3D modeling engine 26, and a 3D model display engine 27.

The CPU 21 is a calculation device that handles various calculations related to various operations of the image server 20. In addition to the CPU 21, a GPU (Graphical Processing Unit) that handles image processing may be provided. The input/output interface 22 is an interface device that handles input and output of data and signals between external devices including the CT device 10. The RAM 23 has a function of temporarily storing various calculation data and the like during execution of the image display control program stored in the image server 20. The ROM 24 stores the BIOS and firmware of peripheral devices. The storage device 25 is a storage device that stores the above-mentioned image display control program as well as the CT image Pcc captured by the CT device 10, and is, for example, a hard disk drive device or a solid state drive device. The RAM 23, ROM 24, and storage device 25 are examples of storage devices, and it goes without saying that other storage device configurations can be adopted.

The 3D modeling engine 26 is a calculation unit that converts the captured CT image Pcc0 (two-dimensional image) into a three-dimensional CT image Pcc. The 3D model display engine 27 is a calculation unit that converts the three-dimensional CT image Pcc into a data format suitable for display in the AR glasses 30. The 3D modeling engine 26 and the 3D model display engine may be realized by the CPU 21 and a program, or may be realized by an image processing processor or GPU separate from the CPU 21.

With reference to FIG. 3, an example of the configuration of the AR glasses 30 as an image display device will be described. As an example, the AR glasses 30 may include a display unit 31, a light guide unit 32, a speaker 33, a microphone 34, a sensor 35, a wireless communication unit 36, a visible light/infrared camera 37, and an AR glasses control unit 38. The AR glasses 30 also store an image display control program for controlling the image display. The AR glasses control unit 38 further includes a CPU 381, a RAM 382, and a ROM 383.

The display unit 31 is a display device that displays the three-dimensional image data Pcc received from the image server 20 via the wireless communication unit 36, and is, for example, a microdisplay. The wireless communication unit 36 and the RAM 382 (memory) function as an image acquisition unit that acquires CT images in the AR glasses 30. The CT image acquired by the image acquisition unit is displayed on the display unit 31 as an image display unit. The light guide unit 32 is a member that guides the light emitted by the display unit 31 to the user's eyes and displays the image of the image data Pcc in front of the user's eyes. As an example, the light guide unit 32 can be configured by a holographic lens that is arranged in front of the user's eyes, has translucency, and is capable of guiding light from the display unit 31. It goes without saying that the configuration of the display unit 31 and the light guide unit 32 can adopt a different configuration as long as the image data Pcc can be displayed superimposed on the actual image Ppt of the patient PT. For example, the display unit 31 and the light guide unit 32 can be replaced by a laser scanner or the like.

The speaker 33 is a device that emits sound based on audio data received from the image server 20 or an external device to communicate information to the user (doctor DR) by voice. The microphone 34 is a device that detects the voice (commands, etc.) emitted by the user and converts it into audio data. The audio data output by the microphone 34 is also transmitted to the image server 20 as appropriate, and can be collected and stored in the image server 20.

The sensor 35 is a sensor for detecting the movement, rotation, tilt, etc. of the AR glasses 30 and for detecting the depth of an object in front of the AR glasses 30, and includes, for example, an accelerometer, a gyro, a magnetometer, an infrared detection device, a depth sensor, etc.

The visible light/infrared camera 37 is an imaging device for performing the following operations, for example.
- Determine the spatial structure around the AR glasses 30 - Grasp the position Xg of the AR glasses 30 in the determined spatial structure - Detect the user's line of sight - Capture an image of the object in front of the user's eyes - Grasp the position Xm of the marker MK The type, number, placement, etc. of the cameras can be changed in various ways depending on the purpose and specifications.

The image display control program stored in the AR glasses 30, when executed, realizes a marker identification unit 391, a coordinate calculation unit 392, a gaze guidance unit 393, an instrument tip identification unit 394, and a conversion unit 395 within the AR glasses 30. Note that, in order to reduce the calculation load on the AR glasses 30, these programs may be executed on the image server 20 or another computer.

The marker identification unit 391 has a function of executing image processing to identify the marker MK affixed to the body surface of the patient PT in the view presented to the user by the AR glasses 30. The coordinate calculation unit 392 calculates the coordinates (first coordinates) of the marker MK included in the CT image Pcc and the coordinates (second coordinates) of the marker MK identified by the marker identification unit 391. As shown in FIG. 4, the coordinates Xm1' to Xm3' of the marker MK in the CT image Pcc are calculated as coordinates of a coordinate system Cx1 set based on a reference point in the CT device 10. On the other hand, the coordinates Xm1 to Xm3 of the marker MK identified by the marker identification unit 391 in the AR glasses 30 can be calculated as coordinates of a coordinate system Cx2, for example, based on the position Xg of the AR glasses 30. In this embodiment, in order to reduce the amount of calculation required for marker identification, the marker identification unit 391 performs the marker identification operation only in a partial area that corresponds to the line of sight guided by the line of sight guidance unit 393, which will be described later.

The gaze guidance unit 393 has the function of guiding the gaze of the user of the AR glasses 30 to the marker MK. As described below, the gaze guidance unit 393 guides the user to direct the gaze to one of the markers MK, and when the result of the guidance is that the gaze is directed to the marker MK, the marker identification unit 391 performs a marker identification operation in a portion of the area corresponding to the direction of the gaze. This limits the area in which the marker identification operation is performed, making it possible to reduce the amount of calculation. The instrument tip identification unit 394 is a part for identifying the tip of a treatment tool such as a catheter. The conversion unit 395 compares the coordinates Xm1' to Xm3' calculated by the coordinate calculation unit 392 with the coordinates Xm1 to Xm3, and performs a conversion to make the coordinates Xm1' to Xm3' match the coordinates Xm1 to Xm. By performing such a transformation, it becomes possible to accurately match the CT image Pcc to the actual image of the patient PT observed through the AR glasses 30, thereby assisting in performing accurate thoracic drainage. The calculation in the transformation unit 395 may be a linear transformation such as an affine transformation, but the nonlinear transformation algorithm may also be a nonlinear transformation using, for example, the B-spline method.

FIG. 5 shows an example of how to attach the markers MK to the body surface of the patient PT. A set of multiple markers, for example three markers (MK1 to MK3), is used. The markers MK1 to MK3 are attached to the body surface of the patient PT so as to surround the area of interest DA. Each of the markers MK1 to MK3 can be about 2 cm in diameter. They can be made of white alumina, for example. Any method can be used to fix them to the human body, but they can be fixed using medical tape (double-sided or single-sided). In addition, when imaging with the CT device 10, the markers MK1 to MK3 are preferably made of a material that has a lower X-ray transmittance than the human body. The markers MK1 to MK3 may be the same shape and color, but it is preferable to use different colors (red, blue, yellow, etc.), shapes (circle, square, triangle), brightness, etc. to make them easier to distinguish.

When performing thoracic drainage, as an example, markers MK1 to MK3 can be attached at the midpoint of each clavicle, with one marker MK1 and one marker MK2 attached to the hypochondrium (septum plexus). These points can be considered "fixed points" whose movement due to arm movement, breathing, etc. is smaller than that of the reference point, and whose movement in at least the horizontal direction is small and negligible. By selecting the fixed points as the attachment positions for markers MK1 to 3, it becomes possible to more accurately align the CT image Pcc. Here, the reference point can be set at a position where the amount of movement due to arm movement, breathing, etc. is approximately average.

The operation of the medical procedure support system 1 of the first embodiment will be described with reference to Figures 6 to 8. Figure 6 is a flowchart showing the procedure for taking a CT image Pcc using the CT device 10 and calculating the coordinates of the marker MK. Figure 7 is a flowchart showing the procedure for superimposing the CT image Pcc and the actual image Ppt of the patient PT on the AR glasses 30. Figure 8 shows an example of a screen display when guiding the line of sight to the marker MK.

When performing thoracic drainage using this medical procedure support system 1, first a marker MK is attached to the chest of the patient PT (step S11), and then a CT image Pcc0 of the patient's chest is taken with the CT device 10 (step S12). Note that when taking the CT image Pcc0, unlike typical CT scans, it is preferable for the patient PT to have both arms lowered toward the lower body during the scan. This is because subsequent thoracic drainage will be performed with both arms lowered, and it is therefore preferable to perform the CT scan in the same posture as that at that time.

The captured CT image Pcc0 is transferred (acquired) to the image server 20, where an intermediate file (Pcc') processed as voxel data is generated and saved (step S13). The voxel data is also transferred to the AR glasses 30. The coordinate calculation unit 392 in the AR glasses 30 identifies the positions of the three markers MK1 to MK3 based on this voxel data, and calculates their coordinates Xm1', Xm2', and Xm3' (step S14). Note that step S14 may be performed in parallel with the steps in the flowchart of FIG. 7 below. Note that the voxel data is volumetric data in which the CT value is entered in each lattice of a 3D cubic lattice space. The data of each lattice is called the voxel value. The coordinate Xmi of the marker MK can be obtained by extracting a voxel having a specific CT value for identifying the marker MK.

Once the voxel data has been acquired and stored in the image server 20, the patient PT moves to the treatment room to perform chest drainage (the marker MK remains attached to the chest), and the doctor DR activates the AR glasses 30 (step S21) and puts them on his or her head. After activation, the AR glasses 30 grasp the spatial structure S(x) around the AR glasses 30 according to the output of the various sensors 35 and the visible light/infrared camera 37, identify the position of the AR glasses 30, and calculate the coordinate Xg of that position (step S22).

Then, the line of sight guidance unit 393 of the AR glasses 30 displays a screen as shown in FIG. 8 for the doctor DR, who is the user, via the display unit 31, and sequentially guides the doctor DR's line of sight to one of the markers MK1 to MK3 (step S23). Specifically, an instruction OR1 saying "Please look at the marker in the upper left" is displayed in a part of the screen (for example, the lower left), and a line of sight display mark GDM indicating the direction of the doctor DR's line of sight detected according to the output of the visible light/infrared camera 37 and an area display mark CM indicating the area centered on the line of sight are displayed. If the doctor DR determines that the target marker MK1 is included near the center of the area display mark CM, he or she presses a confirmation button (not shown) to confirm the position of the area display mark CM. After that, the same operation is repeated for the markers MK2 and MK3, and the position of the area display mark CM including the markers MK1 to MK3 is identified (step S24). The confirmation of the area display mark may be performed by, for example, issuing a voice commanding confirmation to the microphone 34, in addition to using the confirmation button. Also, if it is determined that the gaze is fixed for a predetermined period of time or more (for example, 3 seconds or more), the gaze direction may be determined to be the position of the area display mark CM.

When the position of the area display mark CM is specified, the marker identification unit 391 is activated, and the position of the marker MK included in the area display mark CM is specified (step S25). Since the area of the area display mark CM is very small compared to the area of the entire view of the AR glasses 30, the amount of calculation can be suppressed. When the position of the marker MK is specified, the coordinate calculation unit 292 calculates the coordinate Xm (Xm1 to 3) of the marker MK (step S26). Steps S23 to S26 are repeated as many times as the number of markers MK.

Subsequently, after aligning the coordinates Xm1' to Xm3' with the coordinates Xm1 to Xm3 (step S27), the voxel data expressed in the coordinate system Cx1 is converted to an expression in the coordinate system Cx2 by affine transformation or the like (step S28). For the voxel data expressed in the coordinate system Cx2, it is assumed that light is emitted from the coordinate Xg, and the voxel values of the lattice through which the light passes are appropriately handled (volume ray casting). This results in a CT image Pcc, which is a three-dimensional image displayed on the AR glasses 30. Note that the coordinate Xg is defined for each of the left and right eyes of the AR glasses, and it is possible to display images obtained by emitting light from left and right positions, respectively. This results in an image with parallax, making stereoscopic vision possible.

The obtained three-dimensional image, CT image Pcc, is transmitted to the display unit 31 and displayed on the AR glasses 30 in a superimposed manner together with the actual image Ppt of the patient PT. When the program for the AR glasses 30 is being executed on the image server 20, the three-dimensional image data of the CT image is transmitted to the display unit 31 via the wireless communication unit 36, for example by wireless communication. The transmission rate is about 30 frames/second, and when the user (doctor DR) fixes his/her gaze near the area of interest DA as described above, the image can be displayed as being sufficiently smooth. When the transmission rate increases to 90 frames/second, the image can be displayed smoothly even when the user shifts his/her gaze.

If an attempt is made to extract the marker MK from the entire spatial structure recognized by the AR glasses 30, the amount of calculations becomes extremely large, and, for example, CT images can only be transmitted at about 3 frames per second. Therefore, in this embodiment, the gaze guidance unit 393 guides the gaze of the user (doctor DR) in the direction of the marker MK, a gaze display mark CM is set in the gaze direction, and the marker MK is recognized (searched) within the gaze display mark CM. This narrows the spatial range to be searched, and allows the marker MK to be recognized quickly (for example, about 50 to 100 times faster than when searching the first half). This operation can be performed sequentially for the three markers.

As described above, according to the first embodiment of the medical procedure support system 1, when the CT image obtained by the CT device 10 is superimposed and displayed on the AR glasses 30, the line of sight is guided by the line of sight guidance unit 393, and then the markers MK are identified in a limited area. This makes it possible to quickly align the CT image with a small amount of calculation.

[Second embodiment]
Next, a medical treatment support system 1 according to a second embodiment will be described with reference to FIG. 9 to FIG. 13. The overall configuration of the medical treatment support system 1 according to the second embodiment, and the configurations of the image server 20 and the AR glasses 30 may be the same as those of the first embodiment (FIG. 1 to FIG. 3), so duplicated descriptions will be omitted. However, the system 1 according to the second embodiment is different from the first embodiment in that it is configured to determine the breathing state of the patient PT and display a CT image according to that state. Specifically, as shown in FIG. 9, in addition to the markers MK1 to MK3 arranged at fixed points, a marker MK4 (a marker for respiration detection) arranged at a fluctuating point that fluctuates due to breathing is used. The movement of this marker MK4 is detected, and a CT image according to the movement is displayed on the AR glasses 30. Here, the "fluctuating point" is, for example, near the patient's nipple (in the case of a man). If chest breathing is strong, this position is a suitable attachment position (fluctuation point) for the marker MK4. In the case of a patient in whom abdominal breathing is more prevalent than thoracic breathing, it may be preferable to attach the marker MK4 near the navel on the abdomen.

The concept of acquiring CT images in the system 1 of the second embodiment will be described with reference to FIG. 10. For example, in the expiratory state where the exhalation is at its maximum, the inhalation state where the inhalation is at its maximum, and an intermediate state between the two, the coordinates of the marker MK4 are obtained and CT images Pcc10 to Pcc30 are acquired. In thoracic drainage, a deviation of up to about 1 cm can occur in the planar direction between the state in which breath is inhaled (inhalation state) and the state in which breath is exhaled (exhalation state). Even in the example of FIG. 10, a deviation can occur between the CT images Pcc10 to Pcc30. For this reason, in the system 1 of the second embodiment, multiple CT images Pcc10 to Pcc30 are acquired taking respiration into consideration, and are stored in association with the coordinate values of the markers MK.

The operation of the system 1 of the second embodiment will be described with reference to the flowcharts of Figures 11 and 12. The basic operation is similar to that of the first embodiment, so a duplicated explanation will be omitted, but as explained in Figure 10, it differs from the first embodiment in that multiple CT images are acquired and saved for each different respiratory state (steps S12' to S13'), and the coordinates Xmi' of the marker MK are calculated for each of the multiple CT images (step S14').

The procedure shown in FIG. 12 is basically the same as the procedure shown in FIG. 7, but differs from the first embodiment in that in steps S30 and S31, the position of the marker MK4 is identified in the view of the AR glasses 30, and the CT image corresponding to that position is switched (the CT image displayed is changed).

In the example of FIG. 10, separate CT images are taken for the intermediate state and are subject to switching display, but alternatively, as shown in FIG. 13, CT images can be taken for the exhalation and inhalation states, and the CT image of the intermediate state can be generated by interpolation processing based on the CT images of the exhalation and inhalation states.

As described above, the system of the second embodiment not only provides the same effects as the first embodiment, but also allows the display of CT images that correspond to the respiratory state of the patient PT, making it easier for the doctor DR to carry out medical procedures.

[Third embodiment]
Next, a medical treatment support system 1 according to a third embodiment will be described with reference to Fig. 14. The overall configuration of the medical treatment support system 1 according to the third embodiment may be the same as that of the first embodiment (Fig. 1), and therefore a duplicated description will be omitted. However, in this third embodiment, the 3D modeling engine 26 and the 3D model display engine 27 are omitted from the image server 20, and instead, a 3D modeling engine 395 and a 3D model display engine 396 are provided in the AR glasses 30. The function of 3D modeling of CT images is transferred to the AR glasses 30, and the basic operation is the same as that of the first embodiment.

[Fourth embodiment]
The configuration of the medical procedure support system 1 of the fourth embodiment will be described with reference to Fig. 15. The overall configuration of the system 1 is the same as that of the first embodiment, and the configurations of the image server 20 and the AR glasses 30 may also be the same, so duplicated descriptions will be omitted. The system 1 of the fourth embodiment differs from the first embodiment in that, in addition to the AR glasses 30, AR glasses 30T are provided that can observe images similar to those of the AR glasses 30 and are worn by the instructor DRT of the doctor DR.

The instructor DRT is in the vicinity of the doctor DR who is to be trained, and wears AR glasses 30T. Like the AR glasses 30, the AR glasses 30T are capable of observing the actual image Ppt of the patient PT, and like the AR glasses 30, are capable of superimposing and displaying a CT image Pcc. The instructor DRT can provide guidance to the doctor DR by looking at the display on the AR glasses 30T. The AR glasses 30 and AR glasses 30T may exchange data directly via wireless communication, or may send and receive data indirectly via the image server 20.

[Fifth embodiment]
The configuration of the medical treatment support system 1 according to the fifth embodiment will be described with reference to Fig. 16. Fig. 16 shows the overall configuration of the medical treatment support system 1 according to the fifth embodiment. The CT device 10 is not shown.

The medical procedure support system 1 of the fifth embodiment supports chest drainage by remote operation, where the target patient PT is in a hospital HP1, and the doctor DR in charge of the chest drainage is in a hospital HP2 that is geographically separated from the hospital HP1. The hospitals HP1, HP2, and the image server 20 are connected by a network NW.

A patient PT is present in hospital HP1, and is affixed with the same marker MK as in the previously described embodiment. Next to the patient PT is a robot 70 that is responsible for inserting a chest drainage catheter 60, and an actual image of the patient PT is captured by a camera 80. The image captured by the camera 80 is sent to the AR glasses 30 via a network NW. The robot 70 is operated by a controller 90 carried by a doctor DR in hospital HP2. The AR glasses 30 display the actual image Ppt of the patient PT sent from the camera 80 superimposed on the CT image Pcc, and are capable of performing the same alignment and conversion as in the previously described embodiment.

According to this fifth embodiment, the effects of the previous embodiments can also be obtained in medical procedures such as remote-controlled thoracic drainage.

[Modification]
In the above-described various embodiments, the procedure performed by the doctor DR is thoracic drainage, but as described above, the present invention is not limited to thoracic drainage and can be applied to other procedures. For example, a similar system can be used for injections into the veins of the arm or the arteries of the wrist. At least three markers MK can be attached around the veins of the forearm or the arteries of the wrist to take CT images, and the CT images can be displayed on the AR glasses 30 in the same manner.

Also, for epidural anesthesia, at least three markers MK are attached around the spine on the back, and a CT scan is similarly performed (chest CT scan is performed in the lateral position), which can then be used during epidural anesthesia. The CT image displayed on the AR glasses 30 identifies the spinal cord and dura mater, and epidural anesthesia can be performed by inserting a needle into the epidural space.

[others]
The present invention is not limited to the above-mentioned embodiment, and various modified examples are included. For example, the above-mentioned embodiment has been described in detail to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of a certain embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of a certain embodiment. In addition, it is possible to add, delete, or replace other configurations with respect to a part of the configuration of each embodiment. For example, in acquiring a CT image, image processing may be performed such as lowering the brightness of the CT image of the part where the catheter 60 exists, or increasing the transmittance of the catheter 60 relative to the actual image of the patient. In addition, a configuration may be adopted in which the transmittance of the CT image is changed by voice operation via the microphone 34 of the AR glasses 30. In addition, a configuration may be adopted in which the transmittance of the catheter 60 in the CT image can be changed according to the doctor's preference, not automatically.

Sixth embodiment
Next, a medical procedure support system 1 according to a sixth embodiment will be described with reference to Figs. 17 to 20. The overall configuration of the medical procedure support system 1 according to the sixth embodiment and the configuration of the image server 20 may be the same as those of the first embodiment (Figs. 1 and 2), and therefore a duplicated description will be omitted. However, the system 1 according to the sixth embodiment differs from the first embodiment in that, as shown in Fig. 17, when an image display control program stored in the AR glasses 30 is executed, the AR glasses 30 also function as a determination unit 397. The determination unit 397 will be described later.

Furthermore, as shown in FIG. 18, the system 1 of the sixth embodiment differs from the first embodiment in that in addition to the three markers MK1, MK2, and MK3, a marker MK4 is used to determine the degree of match between the patient whose CT image Pcc was taken and the patient when a procedure such as thoracic drainage is performed using the AR glasses 30.

Marker MK4 is used to identify the patient and is affixed to a position where the amount of change in position due to the patient's breathing is below a reference value. Here, a position where the amount of change in position due to the patient's breathing is below a reference value is a position that can be considered a fixed point as described in the first embodiment. In the example of FIG. 18, marker MK4 is affixed near the manubrium, but the position at which marker MK4 is affixed is not limited to near the manubrium.

In this embodiment, the CT device 10 captures an image of the region of interest DA of the patient PT together with four markers MK1 to MK4, and obtains a CT image Pcc as an image of the region of interest.

After the CT image Pcc has been taken, the patient PT moves to the treatment room where the doctor DR performs treatment such as thoracic drainage. The patient PT also has markers MK1 to MK4 attached to their body surface when the doctor DR performs treatment such as thoracic drainage.

The doctor in charge of the treatment wears the AR glasses 30 and performs the treatment while observing the actual image Ppt of the patient seen through the AR glasses 30 and the CT image Pcc superimposed on it.

When the image display control program stored in the AR glasses 30 is executed, the AR glasses 30 function as a marker identification unit 391, a coordinate calculation unit 392, a gaze guidance unit 393, an instrument tip identification unit 394, and a conversion unit 395, as described in the first embodiment. In this embodiment, the AR glasses 30 also function as a determination unit 397. Note that the processing of the determination unit 397 is executed, for example, between step S26 and step S27 in the flowchart of FIG. 7.

The determination unit 397 determines the degree of agreement between the patient displayed on the AR glasses 30 and the patient whose region of interest image was captured, based on the positions Xm1-Xm4 of the markers MK1-MK4 identified by the marker identification unit 391 and the positions Xm1'-Xm4' of the markers MK1-MK4 in the CT image Pcc as the region of interest image.

For example, the determination unit 397 determines the degree of correspondence between the patient displayed on the AR glasses 30 and the patient from whom the region of interest image was captured, based on the distance between marker MK4 as a specific marker identified from among the markers identified by the marker identification unit 391 and the other markers MK1-MK3, and the distance between marker MK4' in the CT image Pcc as the region of interest image and the other markers MK1'-MK3'.

Specifically, as shown in FIG. 19, the determination unit 397 calculates distances d14, d24, and d34 from the position Xm4 of the marker MK4 to the positions Xm1 to Xm3 of the other markers MK1 to MK3. Similarly, it calculates distances d14', d24', and d34' from the position Xm4' of the marker MK4' in the CT image Pcc as the region of interest image to the positions Xm1' to Xm3' of the other markers MK1' to MK3'.

Next, the determination unit 397 calculates the average value da of the distances d14, d24, and d34, and also calculates the average value da' of the distances d14', d24', and d34', and calculates the difference Δd (absolute value) between the average value da and the average value da'. If the difference Δd is equal to or less than a predetermined threshold, the determination unit 397 determines that the patient displayed on the AR glasses 30 and the patient whose region of interest image was captured match. On the other hand, if the difference Δd is greater than the predetermined threshold, it determines that the patient displayed on the AR glasses 30 and the patient whose region of interest image was captured do not match. In this case, a warning message indicating that the patients are different is displayed on the display unit 31 of the AR glasses 30.

Since the average values da and da' are considered to be patient-specific, by determining whether the difference between the two is below a threshold value, it is possible to determine whether the patient displayed on the AR glasses 30 matches the patient whose region of interest image was taken. This makes it possible to prevent a patient other than the patient whose CT image was taken from mistakenly receiving treatment from the doctor DR. The degree of agreement may be calculated based on the difference Δd, and the calculated degree of agreement may be displayed on the display unit 31 of the AR glasses 30. The degree of agreement is expressed as a percentage, for example, and can be calculated using a formula in which the degree of agreement is 100% when the difference Δd is 0, and the degree of agreement decreases as the difference Δ increases.

The determination unit 397 may also determine whether the patient displayed on the AR glasses 30 matches the patient from whom the region of interest image was captured by determining whether the sum of the squares of the differences between the corresponding distances is less than or equal to a predetermined threshold.

Specifically, the determination unit 397 calculates the difference Δd14 between the distances d14 and d14', the difference Δd24 between the distances d24 and d24', and the difference Δd34 between the distances d34 and d34'.

Then, the square sum of the difference Δd14, the difference Δd24, and the difference Δd34, that is, Δd14 ² + Δd24 ² + Δd34 ² is calculated. If the calculated square sum is equal to or less than the threshold, it is determined that the patient displayed on the AR glasses 30 and the patient whose region of interest image is captured match. On the other hand, if the calculated square sum is greater than the threshold, it is determined that the patient displayed on the AR glasses 30 and the patient whose region of interest image is captured do not match. Note that the degree of agreement may be calculated based on the calculated square sum, and the calculated degree of agreement may be displayed on the display unit 31 of the AR glasses 30. The degree of agreement is expressed, for example, as a percentage, and can be calculated using a formula in which the degree of agreement is 100% when the calculated square sum is 0, and the degree of agreement decreases as the calculated square sum increases.

The determination unit 397 may also determine the degree of correspondence between the patient displayed on the AR glasses 30 and the patient from whom the region of interest image was captured, based on the angle formed by the line segments connecting marker MK4 as a specific marker identified from among the markers identified by the marker identification unit 391 to the other markers MK1 to MK3, and the angle formed by the line segments connecting marker MK4' in the CT image Pcc as the region of interest image to the other markers MK1' to MK3'.

Specifically, as shown in FIG. 20, the determination unit 397 calculates the angle θ12 between a line segment L14 connecting the position Xm4 of the marker MK4 identified by the marker identification unit 391 to the position Xm1 of the marker MK1, and a line segment L24 connecting the position Xm4 of the marker MK4 to the position Xm2 of the marker MK2. It also calculates the angle θ13 between the line segment L14 and a line segment L34 connecting the position Xm4 of the marker MK4 to the position Xm3 of the marker MK3. It also calculates the angle θ23 between the line segment L34 and a line segment L24 connecting the position Xm4 of the marker MK4 to the position Xm2 of the marker MK2.

Similarly, the determination unit 397 calculates the angle θ12' between the line segment L14' connecting the position Xm4' of the marker MK4' in the CT image Pcc to the position Xm1' of the marker MK1 in the CT image Pcc, and the line segment L24' connecting the position Xm4' of the marker MK4 to the position Xm2' of the marker MK2' in the CT image Pcc. It also calculates the angle θ13' between the line segment L14' and the line segment L34' connecting the position Xm4' of the marker MK4' to the position Xm3' of the marker MK3' in the CT image Pcc. It also calculates the angle θ23' between the line segment L34' and the line segment L24' connecting the position Xm4' of the marker MK4 to the position Xm2' of the marker MK2'.

Next, the determination unit 397 calculates the average value θa of the angles θ12, θ13, and θ34, and also calculates the average value θa' of the angles θ12', θ13', and θ34', and calculates the difference Δθ (absolute value) between the average value θa and the average value θa'. If the difference Δθ is equal to or less than a predetermined threshold, the determination unit 397 determines that the patient displayed on the AR glasses 30 and the patient whose region of interest image was captured match. On the other hand, if the difference Δθ is greater than the predetermined threshold, it determines that the patient displayed on the AR glasses 30 and the patient whose region of interest image was captured do not match. In this case, a warning message indicating that the patients are different is displayed on the display unit 31 of the AR glasses 30.

Since the average values θa, θa' are considered to be patient-specific, by determining whether the difference between the two is below a threshold value, it is possible to determine whether the patient displayed on the AR glasses 30 matches the patient whose region of interest image was taken. This makes it possible to prevent a patient other than the patient whose CT image was taken from mistakenly receiving treatment from the doctor DR. The degree of agreement may be calculated based on the difference Δθ, and the calculated degree of agreement may be displayed on the display unit 31 of the AR glasses 30. The degree of agreement is expressed as a percentage, for example, and can be calculated using a formula in which the degree of agreement is 100% when the difference Δθ is 0, and decreases as the difference θ increases.

The determination unit 397 may also determine the degree of match between the patient displayed on the AR glasses 30 and the patient whose region of interest image was captured by determining whether the sum of the squares of the differences between the corresponding angles is equal to or less than a predetermined threshold value.

Specifically, the determination unit 397 calculates the difference Δθ12 between the angles θ12 and θ12', the difference Δθ13 between the angles θ13 and θ13', and the difference Δθ34 between the angles θ23 and θ23'.

Then, the square sum of the difference Δθ12, the difference Δθ13, and the difference Δθ23, that is, Δθ12 ² +Δθ13 ² +Δθ23 ² is calculated. If the calculated square sum is equal to or less than the threshold, it is determined that the patient displayed on the AR glasses 30 and the patient whose region of interest image is captured match. On the other hand, if the calculated square sum is greater than the threshold, it is determined that the patient displayed on the AR glasses 30 and the patient whose region of interest image is captured do not match. In addition, the degree of agreement may be calculated based on the calculated square sum, and the calculated degree of agreement may be displayed on the display unit 31 of the AR glasses 30. The degree of agreement is expressed, for example, as a percentage, and can be calculated using a calculation formula in which the degree of agreement is 100% when the calculated square sum is 0, and the degree of agreement decreases as the calculated square sum increases.

In the above, marker MK4 is set as the specific marker, but any of markers MK1 to MK3 may be set as the specific marker.

The determination unit 397 may also determine the degree of match between the patient displayed on the AR glasses 30 and the patient from whom the region of interest image was captured, based on the distance between the position of each marker M1 to MK4 identified by the marker identification unit 391 and the position of each marker MK1' to MK4' in the CT image Pcc as the region of interest image, which corresponds to the position of each marker MK1 to M4.

For example, the coordinates Xm1' to Xm4' in the coordinate system Cx1 of the markers MK1' to MK4' in the CT image Pcc are transformed into the coordinate system Cx2 of the AR glasses 30 by affine transformation or the like, and the degree of match between the patient displayed on the AR glasses 30 and the patient whose region of interest image was captured is determined based on the transformed coordinates Xm1" to Xm4" and the coordinates Xm1 to Xm4 of the markers MK identified in the AR glasses 30.

Specifically, for example, the sum of squares of the distance D1 between coordinate Xm1" and coordinate Xm1, the distance D2 between coordinate Xm2" and coordinate Xm2, the distance D3 between coordinate Xm3" and coordinate Xm3, and the distance D4 between coordinate Xm4" and coordinate Xm4 is calculated, and if the calculated sum of squares is equal to or less than a threshold, it is determined that the patient displayed on the AR glasses 30 matches the patient whose region of interest image was captured, and if the calculated sum of squares is greater than the threshold, it is determined that the patient displayed on the AR glasses 30 does not match the patient whose region of interest image was captured.

In addition, without performing affine transformation or the like, the sum of squares may be calculated in the same manner as above based on the markers MK1'-MK4' in the CT image Pcc and the coordinates Xm1-Xm4 of the markers MK identified in the AR glasses 30, and the degree of match between the patient displayed on the AR glasses 30 and the patient whose region of interest image was captured may be determined based on the calculated sum of squares. The degree of match may be calculated based on the calculated sum of squares and displayed on the display unit 31 of the AR glasses 30. The degree of match is expressed as a percentage, for example, and can be calculated using a formula in which the degree of match is 100% when the calculated sum of squares is 0, and the degree of match decreases as the calculated sum of squares increases.

As described above, the system of the sixth embodiment can determine whether the patient displayed on the AR glasses 30 matches the patient whose region of interest image was captured. This can prevent a patient other than the patient whose CT image was captured from mistakenly receiving treatment from the doctor DR.

In this embodiment, the marker MK4 is used to identify the patient, but it may also be used as a marker for accurate alignment when the actual image Ppt of the patient PT and the CT image Pcc are superimposed.

The disclosures of Japanese Patent Application Nos. 2023-182769 and 2024-052353 are incorporated herein by reference in their entirety. In addition, all documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims (18)

an image acquisition unit that acquires an image of a region of interest obtained by attaching a marker to a body surface of a patient and capturing an image of a region of interest of the patient together with the marker;
a line-of-sight guidance unit that guides the user's line of sight to one of the markers in an image display unit provided in the user's field of vision;
a marker identification unit that identifies the marker in an area corresponding to the user's line of sight guided by the line of sight guidance unit;
a conversion unit that performs a conversion on the region of interest image to align the marker in the region of interest image with the marker identified by the marker identification unit,
The image display device according to claim 1, wherein the image display unit displays the region of interest image converted by the conversion unit. 2. The image display device of claim 1,
The image display device according to claim 1 , wherein at least three of the markers are arranged at positions that surround a region of interest of the patient and where an amount of change in position due to respiration of the patient is smaller than a reference value. The marker further includes a respiration detection marker arranged at a position where a change in position due to respiration of the patient is larger than the reference value,
The image display device according to claim 2 , wherein the image of the region of interest displayed on the image display unit is changed in accordance with the movement of the marker for respiration detection. The image display device according to claim 3, which generates a new region of interest image by performing an interpolation process from a plurality of region of interest images in response to the movement of the respiratory detection marker. The image display device according to any one of claims 1 to 3, wherein the image acquisition unit acquires a plurality of images for a plurality of different stages of the patient's breathing. At least four markers are arranged, including a marker arranged at a position different from the at least three markers arranged,
the region of interest image is an image obtained by imaging the region of interest together with the at least four markers;
3. The image display device according to claim 2, further comprising a determination unit that determines a degree of match between a patient displayed on the image display unit and a patient from whom the region of interest image was captured, based on a position of each marker identified by the marker identification unit and a position of each marker in the region of interest image. The image display device according to claim 6, wherein the determination unit determines the degree of match between the patient displayed on the image display unit and the patient from whom the region of interest image was captured, based on the distance between the specific marker identified from among the at least four markers and the other markers, and the distance between the specific marker in the region of interest image and the other markers. The image display device according to claim 6, wherein the determination unit determines the degree of match between the patient displayed on the image display unit and the patient from whom the region of interest image was captured, based on the angle formed by the line segments connecting the specific marker identified from among the at least four markers to the other markers and the angle formed by the line segments connecting the specific marker in the region of interest image to the other markers. The determination unit determines, based on a distance between a position of each marker identified by the marker identification unit and a position of each marker in the region of interest image corresponding to the position of each marker,
The image display device according to claim 6 , further comprising: a determination unit configured to determine a degree of coincidence between the patient displayed on the image display unit and the patient from whom the region of interest image was captured. A step of attaching a marker to a body surface of a patient and capturing an image of a region of interest of the patient together with the marker;
displaying the region of interest image on an image display unit together with the patient to which the marker is attached;
guiding a user's gaze to one of the markers on the image display unit;
identifying the marker in a region corresponding to the directed line of sight of the user; and performing a transformation on the region of interest image to align the marker in the region of interest image with the identified marker;
and displaying the converted image of the region of interest on the image display unit. The image display control program according to claim 10, wherein at least three of the markers are placed in positions that surround the patient's region of interest and where the amount of change in position due to the patient's breathing is smaller than a reference value. The marker further includes a respiration detection marker arranged at a position where a change in position due to respiration of the patient is larger than the reference value,
The image display control program according to claim 11 , wherein the image of the region of interest displayed on the image display unit is changed in accordance with a movement of the marker for respiration detection. The image display control program according to claim 12, which generates a new region of interest image by performing an interpolation process from a plurality of region of interest images in response to the movement of the respiratory detection marker. The image display control program according to any one of claims 10 to 12, wherein the step of acquiring an image of the region of interest includes acquiring a plurality of images for a plurality of different stages of the patient's breathing. At least four markers are arranged, including a marker arranged at a position different from the at least three markers arranged,
the region of interest image is an image obtained by imaging the region of interest together with the at least four markers;
12. The image display control program according to claim 11, further comprising a step of determining a degree of correspondence between the patient displayed on the image display unit and the patient from whom the region of interest image was captured, based on the positions of each marker identified in the marker identifying step and the positions of each marker in the region of interest image. The image display control program according to claim 15, wherein the determining step determines the degree of match between the patient displayed on the image display unit and the patient from whom the region of interest image was captured, based on the distance between a specific marker identified from among the at least four markers and the other markers, and the distance between the specific marker and the other markers in the region of interest image. The image display control program according to claim 15, wherein the determining step determines the degree of match between the patient displayed on the image display unit and the patient from whom the region of interest image was captured, based on the angle formed by the line segments connecting the specific marker identified from among the at least four markers to the other markers and the angle formed by the line segments connecting the specific marker in the region of interest image to the other markers. The image display control program according to claim 15, wherein the determining step determines the degree of match between the patient displayed on the image display unit and the patient from whom the region of interest image was captured, based on the respective distances between the positions of the markers identified in the marker identifying step and the positions of the markers in the region of interest image that correspond to the positions of the markers.