[go: up one dir, main page]

WO2024170741A1 - Systèmes et procédés de manipulation automatisée de dispositif médical endovasculaire robotique - Google Patents

Systèmes et procédés de manipulation automatisée de dispositif médical endovasculaire robotique Download PDF

Info

Publication number
WO2024170741A1
WO2024170741A1 PCT/EP2024/053993 EP2024053993W WO2024170741A1 WO 2024170741 A1 WO2024170741 A1 WO 2024170741A1 EP 2024053993 W EP2024053993 W EP 2024053993W WO 2024170741 A1 WO2024170741 A1 WO 2024170741A1
Authority
WO
WIPO (PCT)
Prior art keywords
guidewire
command
actuator
trained
patient
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2024/053993
Other languages
English (en)
Inventor
Young-Ho Kim
Eric LLUCH
Mehmet Akif Gulsun
Ankur KAPOOR
Florin-Cristian Ghesu
Dorin Comaniciu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Corindus Inc
Original Assignee
Corindus Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Corindus Inc filed Critical Corindus Inc
Priority to CN202480012805.XA priority Critical patent/CN120693124A/zh
Priority to EP24706065.0A priority patent/EP4648705A1/fr
Publication of WO2024170741A1 publication Critical patent/WO2024170741A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A61B34/32—Surgical robots operating autonomously
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
    • A61B2034/107—Visualisation of planned trajectories or target regions
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2065—Tracking using image or pattern recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A61B2034/301—Surgical robots for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes

Definitions

  • PCI Percutaneous Coronary Intervention
  • AIS acute ischemic stroke
  • TIPS transjugular intrahepatic portosystemic shunt
  • a method includes obtaining, by a trained system, at least one movement command for an intervention device of a robotic surgical system to a location within a patient; determining, by the trained system, at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the intervention device; and applying the actuator command to the robotic surgical system.
  • the intervention device is a guidewire.
  • the at least one movement command is a desired location within the patient of a tip of the guidewire.
  • the method further includes obtaining an observed position of the guidewire from the robotic surgical system; and determining the estimated current position of the guidewire based on the observed position.
  • the observed position of the intervention device corresponds to an observed position of a proximal portion of the guidewire.
  • the estimated current position corresponds to an estimated position of a tip of the guidewire.
  • the determining determines a series of actuator commands, each actuator command of the series of actuator commands corresponds to a state of the guidewire.
  • the method further includes generating image data displaying the estimated current position of the guidewire within a vessel of the patient.
  • the estimated position of the guidewire is overlaid on a three-dimensional vessel segmentation.
  • Siemens Healthineers AG 3 Siemens Healthineers AG 3
  • the actuator command is at least one of a velocity command or a torque command for the actuator.
  • a trained system includes processing circuitry configured to cause the trained system to obtain at least one movement command for an intervention device of a robotic surgical system to a location within a patient, determine at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the intervention device, and apply the actuator command to the robotic surgical system.
  • the intervention device is a guidewire.
  • the at least one movement command is a desired location within the patient of a tip of the guidewire.
  • the processing circuitry is configured to cause the trained system to obtain an observed position of the guidewire from the robotic surgical system; and determine the estimated current position of the guidewire based on the observed position.
  • the observed position of the intervention device corresponds to an observed position of a proximal portion of the guidewire.
  • the estimated current position corresponds to an estimated position of a tip of the guidewire.
  • the processing circuitry is configured to cause the trained system to determine a series of actuator commands, each actuator command of the series of actuator commands corresponding to a state of the guidewire.
  • the processing circuitry is configured to cause the trained system to generate image data displaying the estimated current position of the guidewire within a vessel of the Siemens Healthineers AG 4 patient.
  • the estimated position of the guidewire is overlaid on a three-dimensional vessel segmentation.
  • the actuator command is at least one of a velocity command or a torque command for the actuator.
  • a trained system includes means for obtaining at least one movement command for an intervention device of a robotic surgical system to a location within a patient; means for determining at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the intervention device; and means for applying the actuator command to the robotic surgical system.
  • FIG.1 is a perspective view of a robotic procedure system according to one or more example embodiments;
  • FIG.2 is a block diagram of a robotic procedure system according to one or more example embodiments;
  • FIG.3 is a block diagram of a robotic procedure system depicting various actuating mechanisms according to according to one or more example embodiments;
  • FIG.4A illustrates a predictive control system according to one or more example embodiments;
  • FIG.4B illustrates a flow chart of generating a physics-based simulation of a guidewire according to one or more example embodiments; Siemens Healthineers AG 5
  • FIG.4C illustrates a flow chart of generating a new mesh with a desired vessel-like structure to restrict the movement of a portion of a guidewire according to one or more example embodiments;
  • Minimally-invasive clinical applications procedures commonly use 3D vessel geometries from 3D CTA (Computed Tomography Angiography) images. During these procedures, the clinician manipulates a wire through the catheter and across the blockage. The clinician only uses X-ray fluoroscopy intermittently to visualize and guide the catheter, guidewire, and other devices (e.g., angioplasty balloons and stents).
  • Various types of endovascular robot-assisted systems e.g., Corindus TM TechNIQ TM are being developed to provide efficient positional control of devices, helping clinicians to mitigate therapeutical risks.
  • the primary motions that a clinician can use to control the movement and direction of the wire include rotation and pushing/retracting from the proximal end of the wire outside the insertion point on the patient’s body.
  • Siemens Healthineers AG 7 As used herein, the direction distal is the direction toward the patient and the direction proximal is the direction away from the patient. The terms up and upper refer to the general direction away from the direction of gravity and the terms bottom, lower and down refer to the general direction of gravity. [0054] While the below example embodiments are described with reference to controlling a guidewire, it should be noted that other endovascular devices may be used. [0055] Referring to FIG.1, a robotic procedure system 10 is shown.
  • Robotic procedure system 10 may be used to perform catheter based medical procedures (e.g., percutaneous intervention procedures).
  • Percutaneous intervention procedures may include diagnostic catheterization procedures during which one or more catheters are used to aid in the diagnosis of a patient’s disease. For example, during one embodiment of a catheter based diagnostic procedure, a contrast media is injected into one or more coronary arteries through a catheter and an image of the patient’s heart is taken.
  • Percutaneous intervention procedures may also include catheter based therapeutic procedures (e.g., balloon angioplasty, stent placement, treatment of peripheral vascular disease, etc.) during which a catheter is used to treat a disease.
  • Robotic procedure system 10 is capable of performing any number of catheter based medical procedures with minor adjustments to accommodate the specific percutaneous devices to be used in the procedure.
  • robotic procedure system 10 may be used to diagnose and/or treat any type of disease or condition amenable to diagnosis and/or treatment via a catheter based procedure.
  • robotic procedure system 10 may be used for the treatment of hypertension utilizing a radiofrequency emitting catheter to deactivate certain nerves that enervate the kidneys to control hypertension.
  • Robotic procedure system 10 includes lab unit 11 and workstation 14.
  • Robotic procedure system 10 includes a robotic catheter system, shown as bedside system 12, located within lab unit 11 adjacent patient 21.
  • bedside system 12 may be equipped with the appropriate percutaneous devices (e.g., guidewires, guide catheters, working catheters, catheter balloons, stents, Siemens Healthineers AG 8 diagnostic catheters, etc.) or other components (e.g., contrast media, medicine, etc.) to allow the user to perform a catheter based medical procedure.
  • percutaneous devices e.g., guidewires, guide catheters, working catheters, catheter balloons, stents, Siemens Healthineers AG 8 diagnostic catheters, etc.
  • other components e.g., contrast media, medicine, etc.
  • a robotic catheter system such as bedside system 12 may be any system configured to allow a user to perform a catheter-based medical procedure via a bedside system by operating various controls such as the controls located at workstation 14.
  • Bedside system 12 may include any number and/or combination of components to provide bedside system 12 with the functionality described herein. [0057] Various embodiments of bedside system 12 are described in detail in P.C.T. International Application No. PCT/US2021/070042, filed November 14, 2021, PCT/US2020/041964, filed July 7, 2020, P.C.T. International Application No. PCT/US2009/042720, filed May 4, 2009, the entire contents of each of which are incorporated herein by reference.
  • bedside system 12 may be equipped to perform a catheter based diagnostic procedure.
  • bedside system 12 may be equipped with one or more of a variety of catheters for the delivery of contrast media to the coronary arteries.
  • bedside system 12 may be equipped with a first catheter shaped to deliver contrast media to the coronary arteries on the left side of the heart, a second catheter shaped to deliver contrast media to the coronary arteries on the right side of the heart, and a third catheter shaped to deliver contrast media into the chambers of the heart.
  • bedside system 12 may be equipped to perform a catheter based therapeutic procedure.
  • bedside system 12 may be equipped with a guide catheter, a guidewire, and a working catheter (e.g., a balloon catheter, a stent delivery catheter, ablation catheter, microcatheter, etc.).
  • bedside system 12 may be equipped with a working catheter that includes a secondary lumen that is threaded over the guidewire during a procedure.
  • bedside system 12 may be equipped with an over-the-wire working catheter that includes a central lumen that is threaded over the guidewire during a procedure.
  • bedside system 12 may be equipped with an intravascular ultrasound (IVUS) catheter.
  • IVUS intravascular ultrasound
  • any of the percutaneous devices of bedside system 12 may be equipped with positional sensors that indicate the position of the component within the body.
  • Bedside system 12 is in communication with workstation 14, allowing signals generated by the user inputs and/or control system of workstation 14 to be transmitted to bedside system 12 to control the various Siemens Healthineers AG 9 functions of bedside system 12. Bedside system 12 also may provide feedback signals (e.g., operating conditions, warning signals, error codes, etc.) to workstation 14. Bedside system 12 may be connected to workstation 14 via a communication link 38 that may be a wireless connection, cable connectors, or any other means capable of allowing communication to occur between workstation 14 and bedside system 12. [0061] Workstation 14 includes a user interface 30. User interface 30 includes controls 16. Controls 16 allow the user to control bedside system 12 to perform a catheter based medical procedure.
  • controls 16 may be configured to cause bedside system 12 to perform various tasks using the various percutaneous devices with which bedside system 12 may be equipped (e.g., to advance, retract, or rotate a guidewire, advance, retract, or rotate a working catheter, advance, retract, or rotate a guide catheter, inflate or deflate a balloon located on a catheter, position and/or deploy a stent, inject contrast media into a catheter, inject medicine into a catheter, or to perform any other function that may be performed as part of a catheter based medical procedure, etc.).
  • one or more of the percutaneous intervention devices may be steerable, and controls 16 may be configured to allow a user to steer one or more steerable percutaneous device.
  • bedside system 12 may be equipped with a steerable guide catheter, and controls 16 may also be configured to allow the user located at remote workstation 14 to control the bending of the distal tip of a steerable guide catheter.
  • catheter system 10 including a steerable guide catheter are disclosed in U.S. Patent No.9,452,277, the entire contents of which are incorporated herein by reference.
  • controls 16 include a touch screen 18, a dedicated guide catheter control 29, a dedicated guidewire control 23, and a dedicated working catheter control 25.
  • guidewire control 23 is a joystick configured to advance, retract, or rotate a guidewire
  • working catheter control 25 is a joystick configured to advance, retract, or rotate a working catheter
  • guide catheter control 29 is a joystick configured to advance, retract, or rotate a guide catheter.
  • touch screen 18 may display one or more icons (such as icons 162, 164, and 166) that control movement of one or more percutaneous devices via bedside system 12 or to receive various inputs from the user as discussed below.
  • Controls 16 may also include a balloon or stent control that is configured to inflate or deflate a balloon and/or a stent.
  • Each of the controls may include one or more buttons, joysticks, Siemens Healthineers AG 10 touch screens, etc., that may be desirable to control the particular component to which the control is dedicated.
  • robotic procedure system 10 includes a percutaneous device movement algorithm module or movement instruction module 114 that dictates how bedside system 12 responds to a user’s manipulation of controls 16 to cause a percutaneous device to move in a particular way.
  • Controls 16 may include an emergency stop button 31 and a multiplier button 33. When emergency stop button 31 is pushed a relay is triggered to cut the power supply to bedside system 12.
  • Multiplier button 33 acts to increase or decrease the speed at which the associated component is moved in response to a manipulation of guide catheter control 29, guidewire control 23, and working catheter control 25.
  • Multiplier button 33 may be a toggle allowing the multiplier effect to be toggled on and off. In another embodiment, multiplier button 33 must be held down by the user to increase the speed of a component during operation of controls 16.
  • User interface 30 may include a first monitor 26 and a second monitor 28. First monitor 26 and second monitor 28 may be configured to display information or patient specific data to the user located at workstation 14.
  • first monitor 26 and second monitor 28 may be configured to display image data (e.g., x-ray images, MRI images, CT images, ultrasound images, etc.), hemodynamic data (e.g., blood pressure, heart rate, etc.), patient record information (e.g., medical history, age, weight, etc.).
  • image data e.g., x-ray images, MRI images, CT images, ultrasound images, etc.
  • hemodynamic data e.g., blood pressure, heart rate, etc.
  • patient record information e.g., medical history, age, weight, etc.
  • first monitor 26 and second monitor 28 may be configured to display procedure specific information (e.g., duration of procedure, catheter or guidewire position, volume of medicine or contrast agent delivered, etc.).
  • the user may interact with or select various icons or information displayed on monitors 26 and 28 using a user input device or control (e.g., a mouse).
  • Monitor 26 and monitor 28 may be configured to display information regarding the position and/or bend of the distal tip of a steerable guide catheter. Further, monitor 26 and monitor 28 may be configured to display information to provide the functionalities associated with the various modules of controller 40 discussed below.
  • user interface 30 includes a single screen of sufficient size to display one or more of the display components and/or touch screen components discussed herein.
  • Siemens Healthineers AG 11 Robotic procedure system 10 also includes an imaging system 32 located within lab unit 11. Imaging system 32 may be any medical imaging system that may be used in conjunction with a catheter based medical procedure (e.g., non-digital x-ray, digital x-ray, CT, MRI, ultrasound, etc.).
  • imaging system 32 is a digital x-ray imaging device that is in communication with workstation 14. As shown in FIG.1, imaging system 32 may include a C-arm that allows imaging system 32 to partially or completely rotate around patient 21 in order to obtain images at different angular positions relative to patient 21 (e.g., sagittal views, caudal views, cranio-caudal views, etc.). [0066] Imaging system 32 is configured to take x-ray images of the appropriate area of patient 21 during a particular procedure. For example, imaging system 32 may be configured to take one or more x-ray images of the heart to diagnose a heart condition.
  • Imaging system 32 may also be configured to take one or more x-ray images during a catheter based medical procedure (e.g., real-time images) to assist the user of workstation 14 to properly position a guidewire, guide catheter, working catheter, stent, etc. during the procedure.
  • the image or images may be displayed on first monitor 26 and/or second monitor 28.
  • the user of workstation 14 may be able to control the angular position of imaging system 32 relative to the patient to obtain and display various views of the patient’s heart on first monitor 26 and/or second monitor 28. Displaying different views at different portions of the procedure may aid the user of workstation 14 properly move and position the percutaneous devices within the 3D geometry of the patient’s heart.
  • imaging system 32 may be any 3D imaging modality of the past, present, or future, such as an x-ray based computed tomography (CT) imaging device, a magnetic resonance imaging device, a 3D ultrasound imaging device, etc.
  • CT computed tomography
  • the image of the patient’s heart that is displayed during a procedure may be a 3D image.
  • Robotic procedure system 10 may include a control system, shown as controller 40. While a controller 40 is described, it should be understood example embodiments are not limited thereto.
  • processing or control circuitry may be used such as, but not limited to, one or more processors, one or more Central Processing Units (CPUs), one or more controllers, one or more arithmetic logic units (ALUs), one or more digital signal processors (DSPs), one or more microcomputers, one or more field programmable gate arrays (FPGAs), one or more System-on-Chips (SoCs), one or more programmable logic units (PLUs), one or more microprocessors, one or more Application Specific Integrated Circuits (ASICs), or any other device or devices capable of responding to and executing instructions in a defined manner.
  • controller 40 may be part of workstation 14.
  • Controller 40 is in communication with one or more bedside systems 12, controls 16, monitors 26 and 28, imaging system 32, and patient sensors 35 (e.g., electrocardiogram (“ECG”) devices, electroencephalogram (“EEG”) devices, blood pressure monitors, temperature monitors, heart rate monitors, respiratory monitors, etc.).
  • controller 40 may be in communication with a hospital data management system or hospital network 34, one or more additional output devices 36 (e.g., printer, disk drive, cd/dvd writer, etc.), and a hospital inventory management system 37.
  • Communication between the various components of robotic procedure system 10 may be accomplished via communication links 38.
  • Communication links 38 may be dedicated wires or wireless connections. Communication links 38 may also represent communication over a network.
  • Robotic procedure system 10 may be connected or configured to include any other systems and/or devices not explicitly shown.
  • robotic procedure system 10 may include IVUS systems, image processing engines, data storage and archive systems, automatic balloon and/or stent inflation systems, contrast media and/or medicine injection systems, medicine tracking and/or logging systems, user logs, encryption systems, systems to restrict access or use of robotic procedure system 10, robotic catheter systems of the past, present, or future, etc.
  • Further embodiments of robotic procedure system 10 including inflation and/or contrast media injection systems are disclosed in U.S. Patent No.9,545,497, Siemens Healthineers AG 13 issued January 17, 2017, which is incorporated herein by reference in its entirety.
  • Robotic procedure system 10 may include various actuating mechanisms that move an associated percutaneous device in response to a user’s manipulation of controls 16.
  • robotic procedure system 10 includes a guidewire actuating mechanism 50, a working catheter actuating mechanism 52, and a guide catheter actuating mechanism 54.
  • robotic procedure system 10 may include an actuating mechanism for inflating an angioplasty or stent delivery balloon and an actuating mechanism for delivering contrast agent.
  • guidewire actuating mechanism 50 and working catheter actuating mechanism 52 are incorporated within cassette 56 which is coupled to a base of bedside system 12.
  • Guidewire actuating mechanism 50 is coupled to guidewire 58 such that guidewire actuating mechanism 50 is able to cause guidewire 58 to advance, retract, and rotate.
  • Working catheter actuating mechanism 52 is coupled to working catheter 60 such that working catheter actuating mechanism 52 is able to cause working catheter 60 to advance, retract, and rotate.
  • Connector 62 couples guide catheter 64 to guide catheter actuating mechanism 54 such that guide catheter actuating mechanism 54 is able to cause guide catheter 64 to advance, retract, and rotate.
  • guidewire actuating mechanism 50, working catheter actuating mechanism 52, and guide catheter actuating mechanism may each include an engagement structure (e.g., one or more pairs of pinch wheels) suitable for engaging the respective percutaneous Siemens Healthineers AG 14 device such that the actuating mechanism is able to impart axial and/or rotational movement to the percutaneous device.
  • a Y-connector 66 is coupled to guide catheter actuating mechanism 54 via connector 68.
  • connector 68 may be a component separate from both Y-connector 66 and guide catheter actuating mechanism 54.
  • connector 68 may be part of (e.g., integral with) Y- connector 66 or part of actuating mechanism 54.
  • Y- connector 66 is also connected to cassette 56.
  • Y-connector 66 includes a first leg, a second leg, and a third leg.
  • the first leg of the Y-connector is connected to or in communication with the internal lumen of guide catheter 64.
  • the second leg is angled away from the longitudinal axis of guide catheter 64.
  • the second leg provides a port for the injection of fluids (e.g., contrast media, medicine, etc.) into the lumen of guide catheter 64.
  • the third leg of Y-connector 66 is coupled to a cassette 56 and receives both guidewire 58 and working catheter 60.
  • Guidewire actuating mechanism 50 includes a rotate actuator 70 and an advance/retract actuator 72.
  • Rotate actuator 70 is configured to cause rotation of guidewire 58 about its longitudinal axis.
  • the guidewire actuating mechanism 50 may further include one or more sensors 71 for measuring the velocity, torque, position and other parameters of the guidewire/actuator.
  • Advance/retract actuator 72 is configured to advance and/or retract guidewire 58 (i.e., to advance and/or retract along the longitudinal axis of the guidewire) within patient 21.
  • Working catheter actuating mechanism 52 includes a rotate actuator 74 and an advance/retract actuator 76.
  • Rotate actuator 74 is configured to cause rotation of working catheter 60 about its longitudinal axis.
  • the working catheter 60 may further include one or more sensors 71 for measuring the velocity, torque, position and other parameters of the guidewire/actuator.
  • Advance/retract actuator 76 is configured to advance and/or retract working catheter 60 (i.e., to advance and/or retract along the longitudinal axis of the working catheter) within patient 21.
  • Guide catheter actuating mechanism 54 includes a rotate actuator 78, an advance/retract actuator 80, and a bend actuator 82.
  • the guide catheter actuating mechanism 54 may further include one or more sensors 71 for measuring the velocity, torque, position and other parameters of the guidewire/actuator.
  • Rotate actuator 78 is configured to cause rotation of guide catheter 64 about its longitudinal axis.
  • Advance/retract Siemens Healthineers AG 15 actuator 80 is configured to advance and/or retract guide catheter 64 (i.e., to advance and/or retract along the longitudinal axis of the guide catheter) within patient 21.
  • guide catheter 64 may include one or more bend control elements that allow the user to cause bending of the distal tip of guide catheter 64.
  • bend actuator 82 causes the distal tip of guide catheter 64 to bend in response to a user’s manipulation of controls 16.
  • controls 16 and controller 40 located at workstation 14 are communicably coupled to various portions of bedside system 12 to allow the user and/or control system to control movement of guidewire 58, working catheter 60 and guide catheter 64 and any other percutaneous devices that bedside system 12 is equipped with.
  • controls 16 and controller 40 are coupled to guide catheter actuating mechanism 54 to allow the user to move guide catheter 64.
  • controls 16 and controller 40 are coupled to cassette 56 to allow the user to control guidewire 58 via guidewire actuating mechanism 50 and to control working catheter 60 via working catheter actuating mechanism 52.
  • Control signals 116 generated by the controls and controller at workstation 14 are communicated to bedside system 12 to control movement of percutaneous devices discussed herein.
  • Example embodiments provide (1) a scalable learning pipeline to train an AI predictive control system, providing a safety feature for long-term controls; (2) a long-term integrated predictive control for multiple devices in robotic neuro catheterization.
  • predictive controls alleviate the clinician’s fatigue and reduce the overall execution time, which is proportional to X-ray exposure time.
  • FIG.4A illustrates a predictive control system according to one or more example embodiments.
  • the system includes an offline phase 405 and an online implementation phase 450.
  • the offline phase 405 may be considered pre-processing and training and the online phase 450 may be considered a real-time processing phase.
  • the offline phase 405 includes a data lake 410, a pre- processing module 415, an engine simulator 420 and an AI predictive control system 460.
  • the data lake 410 leverages available CTA images and physical parameters for devices.
  • the data lake 410 may store physical parameters such as length of tip, shape and material for various types of guidewires, respectively, and other devices.
  • the data lake 410 may store patient data.
  • the data lake 410 may store 3D CTA brain and/or cardiac images for two planes.
  • the pre-processing module 415 is configured to perform 3D vessel segmentation and extract a centerline (e.g., in FIG.6B) with sectional information based on the CTA images from the data lake 410 and patient specific pre-op images 422 from an image processing module 480, which may be CTA Siemens Healthineers AG 17 images, but are not limited thereto.
  • image processing module 480 which may be CTA Siemens Healthineers AG 17 images, but are not limited thereto.
  • the type of images and images may be adjusted such that the simulator 420 may determine set of motions for the guidewire as described with reference to FIG.4B (e.g., 3D vessel segmentation, centerline determination and distance map generation).
  • the pre-processing module 415, engine simulator 420, the AI predictive control system 460 and the image processing module 480 and the functions thereof may be implemented by the controller 40.
  • the controller 40 is a processor
  • the processor may execute instructions stored in a memory 205 to perform the functions of the pre-processing module 415, engine simulator 420, the AI predictive control system 460 and the image processing module 480.
  • FIG.4B illustrates a flow chart of generating a physics-based simulation of a guidewire
  • FIGS.5A-5D illustrate images used to generate the physics-based simulation of the guidewire according to one or more example embodiments.
  • the pre-processing module 415 uses 3D CTA images of the patient (e.g., in FIG.5A) and generates a corresponding 3D segmentation of whole coronary vessels (e.g., in FIG.5B) at 415a.
  • the segmentation may be performed using 3d U-Net as described at Tsang, Review: 3D U-Net – Volumetric Segmentation (Medical Image Segmentation), Towards Data Science, April 2, 2019, the entire contents of which are incorporated by reference.
  • the pre-processing module 415 adds an insertion supporter (e.g., an introducer/sheath) of the guidewire as a boundary pipe, which is refined by a Poisson reconstruction filter to smooth irregularities in the reconstructed surface, making it more accurate and visually coherent within the overall 3D segmentation scene.
  • the pre-processing module 415 then computes a 3D level set distance map that contains a minimum distance to the vessels at each 3D point to define boundary conditions (e.g., in FIG.5C) at 415c.
  • the distance map may be generated according to Guo et al., DeepCenterline: a Multi-task Fully Convolutional Network for Centerline Extraction, arXiv:1903:10481v1, March 25, 2019, the entire contents of which are incorporated herein by reference.
  • the pre-processing module 415 is further configured to generate a stenosis grading for angiography (i.e., for Fully Automated Coronary Evaluation (FACE)), which can be used as varied vessel lumen and friction information for Siemens Healthineers AG 18 the engine simulator 420.
  • FACE Fully Automated Coronary Evaluation
  • the pre-processing module 415 may generate the stenosis grading as described in Avram, et al., CathAI: fully automated coronary angiography interpretation and stenosis estimation, npj Digital Medicine 6, Article number: 142 (2023), the entire contents of which are incorporated herein by reference.
  • the engine simulator 420 uses the information from the pre- processing module 415 (e.g., 3D vessel geometries, centerlines with section information, stenosis grading for frictional information, device properties) to create a physics-based simulation environment.
  • the engine simulator 420 is configured to simulate in real time the motion of the guidewire in vessel geometries and can generate synthetic x-ray images based on the simulated guidewire motion.
  • the engine simulator defines an initial position and direction of the guidewire at 420a.
  • the guidewires may be simulated using a Cosserat-rod model.
  • the engine simulator 420 applies a desired force to a proximal section of the guidewire using the Cosserat-rod model (at 420a), which interacts with the defined boundary conditions by solving partial differential equations.
  • the Cosserat-rod model allows all possible degrees of freedom of deformation: bend, twist, stretch and shear.
  • FIG.5D illustrates a guidewire 505 modeled by Cosserat- rod.
  • Cosserat-rod models the guidewire 505 in n segments 5051-505n.
  • the guidewire is modelled as a deformable curve with attached deformable vectors to characterize its orientation.
  • the orientation follows a system of coupled Partial Differential Equations to conserve both linear and angular momentum.
  • a Neural Ordinary Differential Equation (ODE) approach is used to solve the partial differential equations efficiently to achieve real-time simulations, since the time integration using Neural ODEs is more accurate than using conventional time integrators.
  • ODE Neural Ordinary Differential Equation
  • FIG.4C illustrates a flow chart of generating a new mesh with a desired vessel-like structure to restrict the movement of a portion of a guidewire according to one or more example embodiments.
  • Siemens Healthineers AG 19 [0098]
  • the simulator 420 obtains the 3D segmentation of the vessel structure of the patient (e.g., from the preprocessing).
  • the simulator 420 uses a boundary condition for the guidewire to be kept inside the vessel while having the base of the guidewire restricted while it is being inserted. Thus, a new boundary condition is developed.
  • the simulator 420 adds a vessel-like structure of a desired width to the entry point of large vessels at S492. Then, at S494, the simulator 420, cuts open meshes and finally applies a Poisson surface reconstruction filter is used to merge them in a watertight surface as shown at S496. [0099] To restrict the catheter within the generated watertight surface, at each time step of the simulation, the simulator 420 computes a wall response that the vessels are applying to the rod (guidewire) when it gets close to the lumen wall, i.e. (1) where d is the distance from the centerline of the guidewire to the closest point at the lumen and r is the radius of the guidewire.
  • the wall response force is given by where kw is a stiffness of the wall, vw is a dissipation coefficient, is a sum of all rod forces against the wall, is the elastic and is a dissipative term in normal velocity, where v is the applied velocity and is a normal velocity perpendicular to a vessel wall.
  • the stiffness kw and the dissipation coefficient vw control the effect of the wall boundary condition.
  • the minimal distance to the lumen’s surface, as well as the normal direction may be computed. However, this operation is very time consuming and highly dependent on a number of points (size) of the lumen.
  • the simulator 420 uses a novel boundary condition, in which the watertight surface is used to compute a 3D image on a bounding box that contains the lumen mesh, which contains the level set map defined at each voxel as seen in FIG 4D.
  • the level set map shown in the left of FIG.4D, is computed as a signed distance, so that its gradient is normal to the level set isocontours.
  • the signed distance is computed as exact in a neighborhood of the surface, and can be extrapolated farther away, for efficiency of computation.
  • the simulator 420 retrieves an approximation to its shortest distance to the lumen surface by just evaluating the 3D image of the level set map at that point, which is computationally efficient.
  • the normal direction is computed as well. Due to the definition of the level set map, this becomes straightforward by evaluating the gradient of the image at the desired point, as shown on the right of FIG.4D. [0104]
  • only one boundary condition is required, which avoids having to iterate over different boundary conditions defined at each point of the lumen’s surface. This allows the simulations to be computationally efficient.
  • the application of this new boundary condition implies a speed-up factor of 3000 bringing the simulation to real-time.
  • the vessels are continuously moving, making the guidance of the catheter more complicated.
  • the simulator 420 uses the level set method to move the vessel boundary, therefore having a continuously updated distance map that follows the vessel movement, which can be done using the boundary node method for cell motility as described in U.S. Patent No.10,282,638, the entire contents of which are incorporated by reference.
  • This provides the scalable simulated device states inside the vessels (e.g., in FIG.5D). More specifically, the engine simulator 420 includes a set of device motions that can be applied in each created environment.
  • the simulated device can consist of multiple segment points (e.g., one guidewire is defined as 300 segments points to model using the Cosserat-rod model). Each segment state can be defined as 3d position, velocity and applied force. The accumulated information of the segmented states indicate the device states.
  • the engine simulator 420 creates various states of the bedside system 12 using the device motions and associated environments as training data. Siemens Healthineers AG 21 [0107] In addition, the simulator 420 may incorporate the motion of the guidewire into a Digital Reconstruction Radiography (DRR) algorithm to create synthetic X-ray images in real time.
  • DDRR Digital Reconstruction Radiography
  • the motion of the guidewire at each exported time step may be integrated into the 3D CTA information by locating the closest voxels to the points of the guidewire and assigning them to have a maximum intensity.
  • a rendering image with guidewire points with an assigned maximum intensity may be generated by the simulator 420.
  • the simulator 420 uses a DDR algorithm, to simulate a C-arm and project the information onto 2D to generate synthetic X-ray images.
  • the simulator 420 may use the DDR algorithm described in U.S. Patent No.10,282,638.
  • an output image is optimized using neural style transfer, by taking a set of ground truth images that can come from a fluoroscopic image or angiographic image of the patient, for example, which define the style reference image by blending it with the content image coming from the DRR.
  • the pre-processing module 415 and engine simulator 420 provide a scalable environment and associated various device controls in simulation which allows a reduced amount of CTA data and/or images of the patient during real device control.
  • the online phase 450 includes the trained AI predictive control system 460, the bedside system 12 and the image processing module 480.
  • the AI predictive control system 460 and the image processing module 480 may be performed by the controller 40 or other hardware such as servers in the cloud.
  • the engine simulator 420 includes a set of device motions that can be applied in each created environment.
  • FIG.6A illustrates a flow chart of training the AI predictive control system 460.
  • the AI predictive control system 460 may be trained off-line based on collected data from the pre-processing module 415, and the physics engine simulator 420. Siemens Healthineers AG 22 Patient-specific data can also be applied to the engine simulator 420 to validate and reinforce the AI predictive control system 460.
  • the AI predictive control system 460 may be implemented as a recurrent neural network, long short-term memory (LSTM) network, a transformer architecture, a neural ordinary differential equations (ODE), other known deep learning architectures, a combination thereof or a sub-combination thereof.
  • the AI predictive control system 460 may be implemented by the controller 40.
  • the AI predictive control may be implemented by processing circuitry located remote from the bedside system 12 (e.g., in the cloud).
  • a user and/or simulator inputs a motion command ⁇ t to the robotic surgical system 12 and the AI predictive control system 460 for a time t.
  • the motion command ⁇ t is a command for the guidewire actuating mechanism 50 to advance/retract (translational movement) and rotate the guidewire and, thus, may also be referred to as a motor command or actuator command.
  • the motion command ⁇ t may include at least one of a velocity or torque command for the guidewire.
  • the simulator 420 maintains a true internal state x(t) which may be a 3d-tip position of the guidewire at the time t from performing the motion command ⁇ t .
  • the AI predictive control system 460 is configured to predict a long-term state estimation based on traces that is the history of sequential states.
  • the AI predictive control system 460 uses a recurrent neural network (e.g., RNN, LSTM, Transformer, N- ODE, etc.) to model a long-term state estimator.
  • the data generated from the simulator 420 is used to train the AI predictive control system 460.
  • the input data from the simulator 420 can be a 3d-tip position, a centerline, a vessel friction, device properties, and a traveled distance at time t as well as the ground truth for the tip position (i.e., the true internal state x(t)).
  • the AI predictive control determines a motion command ⁇ * based on our desired tip motion from the motion command ⁇ t .
  • the simulator 420 may determine an observed measured 3d position y(t) of the tip of the guidewire.
  • the simulator 420 knows the tip position/applied torque(action) which creates a mapping function between the 3d tip position and the action.
  • Siemens Healthineers AG 23 [0118]
  • the observed measured position y(t) may also be determined by the image processing module 480 using CTA images captured at the time t.
  • the true internal state may refer to the real observed value of the tip position of the guidewire, which may be observed using known sensors and/or models (e.g., from the simulator 420).
  • the AI predictive control system 460 maintains an estimated internal state ⁇ ( ⁇ ) which may be an estimate 3d-tip position of the guidewire at the time t from performing the motion command ⁇ t.
  • the AI predictive control system 460 outputs an estimated 3d position ⁇ ( ⁇ ) of the tip of the guidewire.
  • the estimated 3d position ⁇ ( ⁇ ) of the tip of the guidewire in a vessel may be displayed in overlaid on the 3d vessel segmentation or a 2d x-ray.
  • the training system determines an error/loss ⁇ y between the observed measured position y(t) and the estimated 3d position ⁇ ( ⁇ ) of the tip of the guidewire.
  • FIG.6A illustrates an AI neural network training process between ground truth (acquired from the simulator 420) and the AI predictive control system 460 (i.e., a target model).
  • the AI predictive control system 460 is trained to be close to ground truth based on all the input conditions.
  • the AI predictive control system 460 is trained using mean-squared error (MSE) based on the error/loss ⁇ y.
  • MSE mean-squared error
  • the AI predictive control system 460 may be generated by applying a sequence of 3d past tip position with vessel centerline, physical parameters, and traveled distance, and then creating a sequence of estimated states for future.
  • 100 CTA images for the right coronary artery are used as a simulator environment and 1000 randomly generated continuous sequential controls of ⁇ ⁇ are applied for 60 seconds, which generates 120 sequence sample points for each set of each sequential controls (one of the randomly generated continuous sequential controls) in each segmented vessel (one of the 100 CTA images).80% of the data may be used as a training set and 20% may be used for a testing set. We then present the safety Siemens Healthineers AG 24 ratio and difference between the estimated force and the ground-truth in the test set.
  • FIG.6B illustrates a 3D vessel geometry 610 with a centerline 615.
  • FIG.7A illustrates a flow chart of an implementation of an AI predictive control system according to one or more example embodiments.
  • the AI predictive control system 460 includes an inverse agent 705 and a forward agent 710.
  • the AI predictive control system 460 and the image processing module 480 may be performed by the controller 40 or other hardware such as servers in the cloud.
  • the inverse agent 705 and the forward agent 710 are trained as dual Transformer models such as described in Vaswani, et al., “Attention is all you need,” Advance in Neural Information Processing Systems, vol.30, 2017, the entire contents of which are incorporated by reference.
  • the inverse agent 705 generates a motion command of the guidewire ⁇ based on a difference between command position CMD and an estimated state of the tip of the guidewire ⁇ generated by the forward agent 710.
  • the forward agent 710 and the inverse agent 705 i.e., forward/inverse transformer models
  • the forward agent 710 and the inverse agent 705 are learned with 30 sequences of states as inputs, 1 output (many-to-one), 12 heads, 4 encoder/decoder layers, 128 dimensions of feedforward, and 0.1 dropouts.
  • the inverse agent 705 may generate a motion command of the guidewire ⁇ (i.e., control of guidewire at connection to robot such as translation and rotation).
  • x(t) is a 3d-tip position at time t
  • q(t) is motor state to apply at time t
  • e is vessel geometry (i.e., a lumen geometry from the level set distance map generated by pre-processing module 415) and a is one discrete action (i.e., a library of behaviors) at the proximal.
  • the AI predictive control system 460 Given the current position of the tip of the guidewire x tip and a desired location to go, the AI predictive control system 460 creates a sequence of Siemens Healthineers AG 25 motions minimizing an object cost (e.g., execution time, accurate motions) under constraints (e.g., no collision with a vessel and no puncture).
  • the AI predictive control system 460 uses reinforcement learning (RL) and includes the forward agent 710 for forward state estimation, which estimates a next state x(t+1) (e.g., position of tip of the guidewire), and the inverse agent 705 for inverse state estimation, which estimates a next action a(t+1) (e.g., at least one of torque or velocity).
  • a guidewire At a guidewire’s attachment point to the bedside system 12 (e.g., where a torquer is attached to the bedside system 12), a guidewire’s proximal state is where ⁇ is a translation and ⁇ is a rotation.
  • the distal state x ⁇ of the guidewire may be defined by its position and orientation in an ⁇ (3) transformation which is a 3d homogenous transformation matrix consisting of translation and rotation.
  • Physical parameters p of the guidewire may include diameter, Poisson ratio, Young modulus and number of elements. However, example embodiments are not limited thereto.
  • Let e ⁇ E represent the spatial environment of the vessel, including 3D vessel geometry, 3D vessel centerline, and sectional labels.
  • the AI predictive control system 460 Since the AI predictive control system 460 does not have a direct measurement for the tip of the guidewire, the AI predictive control system 460 simplifies x ⁇ ⁇ ⁇ R ⁇ with regard to a 3D vessel centerline by projecting it into a known 3D vessel centerline (e.g., from the pre-processing module 415). [0138] Therefore Siemens Healthineers AG 26 where ⁇ ⁇ ⁇ and ⁇ ⁇ ⁇ represent the translation along the centerline and the distance from the centerline to the tip, respectively.
  • the forward agent 710 implements a forward transition function FF and the inverse agent 705 implements an inverse transition function IF as follows: with control ⁇ ⁇ ⁇ ⁇ to transition from one state to another state given the parameters p and the spatial environment of the vessel e.
  • the proximal states ⁇ ⁇ ⁇ (e.g., position before motion), ⁇ ⁇ ⁇ (e.g., position after motion) are known from measurements of the guidewire actuator mechanism 50 (e.g., from motor encoder) and the proximal command ⁇ ⁇ ⁇ ⁇ and ⁇ are also known from measurements of the actuator mechanism 50.
  • the inverse agent 705 Given a desired tip state ⁇ ⁇ from the command CMD, and based on a current tip position (e.g., an observed tip position), the inverse agent 705 generates a sequence of proximal commands ⁇ ⁇ ( ⁇ ,..., ⁇ ) for the surgical system 12 such that the surgical system manipulates the guidewire to a desired tip state ⁇ ⁇ ( ⁇ ,... ⁇ ) . It should be noted, the inverse agent 705 also generates an estimated proximal position ⁇ ⁇ (in ⁇ ⁇ stead of measured position ⁇ ) to deliver proper commands to the bedside system 12.
  • the inverse agent 705 generates the sequence of proximal commands ⁇ ⁇ ( ⁇ ,..., ⁇ ) by minimizing a cost C as follows: where ⁇ is the current time and ⁇ is the long-term time step. More specifically, the AI control model 460 may be trained using mean-squared error (MSE) such that the AI control model 460 provides a sequence of actions minimizing the cost (e.g., errors). Siemens Healthineers AG 27 [0142] Thus, the AI predictive control system 460 is configured to estimate a sequence of state information and ⁇ ⁇ , ⁇ ( ⁇ ,..., ⁇ ) ⁇ ( ⁇ ,..., ⁇ ) ).
  • MSE mean-squared error
  • the image processing module 480 is configured to provide intermittent observation as well as provide an image illustrating the estimated states ⁇ and ⁇ .
  • the image processing module 480 (or the AI predictive control system 460) may generate an attention-based force map for the guidewire (or another percutaneous device) for display on one of the monitors 26 and 28.
  • the attention-based force map may be generated by the image processing module 480 (or the AI predictive control system 460) in accordance with Li, et al, Efficient Self-supervised Vision Transformers for Representation Learning, ICLR 2022, arXiv:2106.09785v2, July 6, 2022, the entire contents of which are hereby incorporated by reference.
  • the image processing module 480 provides an observation/measurement to the AI predictive control system 460.
  • the AI predictive control system 460 may compute uncertainty values based on errors between the centerline and the vessel wall, such that the AI predictive control system 460 can determine if the tip is in risk (i.e., too close to the wall in perpendicular direction).
  • the AI predictive control system 460 can issue a notification to the user and/or the image processing module 480 to perform an observation from X-ray if the tip is determined to be close to the vessel wall.
  • the image processing module 480 uses a dynamic road map (DRM) and FACE stenosis grading with 2D device tracking information.
  • DRM dynamic road map
  • the 2D tracking information is registered by the image processing module 480 onto a 3D vessel lumen based on registration with an original CTA image to generate an actual measurement of the guidewire y(t). Then, the measurement y(t) can be applied into AI predictive control system 460 (to generate ⁇ y) and reduce possible errors.
  • the image processing module 480 generates an image 708 which may reflect an estimated position of the guidewire and/or measured position of the guidewire permitting the user can supervise the procedure from Siemens Healthineers AG 28 uncertainty measures of AI predictive control.
  • the image 708 can be also applied for autonomous decision-making.
  • the AI predictive control system 460 may provide a command to manipulate the wire to cross a stenosis, making all state transitions via control and finding a sequence of motions that minimizes cost C, defined as the completion time to cross target area or distance to vessel wall.
  • the AI predictive control system 460 may calculate the sequence of motions using equation (7). [0148]
  • the AI predictive control system 460 generates a sequence of motion commands [ ⁇ ⁇ , ... , ⁇ ⁇ ⁇ ] from a long-term predictive control.
  • the cost function can be based on numerous factors such as completion time and safety permitting the AI predictive control system 460 to determine the sequence of motion commands [ ⁇ ⁇ , ... , in a real-time even without any observation.
  • the optimal controls for a long-term horizon time will be applied to the robotic manipulator 12 to control autonomously or let the user confirm that controls to execute.
  • One of advantage here is that if the safety boundary condition is not satisfied, then the predictive control can request new observation to the users, which allows the user to actively supervise the system or if it is autonomous system mode, then integrated image modality will be acquired new intermittent observation, as described above.
  • FIG.7B illustrates a method of operation of the AI predictive control system of FIG.7A.
  • the method of FIG.7B may be performed by the AI predictive control system 460 and, more specifically, the processing circuitry executing the AI predictive control system 460 (e.g., the controller 40).
  • the AI predictive control system 460 obtains at least one movement command for an intervention device of a robotic surgical system to a location within a patient (e.g., CMD) at S720.
  • the AI predictive control system 460 determines at least one actuator command (e.g., [ ⁇ ⁇ , ... , ⁇ ⁇ ⁇ ] ) for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the intervention device.
  • steps S720, S725 and S730 may be based on an image of the patient obtained at S715. For example, if the safety boundary condition is not satisfied, then the predictive control can request new observation to the users (e.g., new actual X-ray image), which allows the user to actively supervise the Siemens Healthineers AG 29 system or if it is autonomous system mode, then integrated image modality will be acquired new intermittent observation.
  • FIG.8 illustrates a forward agent shown in FIG.7A according to one or more example embodiments.
  • the forward agent 710 includes a transformer 805 and an autoregressive time-series transformer 810.
  • the autoregressive time-series transformer 810 includes a long-term predictive encoder 815 and a long-term predictive decoder 820.
  • the long-term predictive encoder 815 receives the estimated tip position ⁇ ⁇ ⁇ ⁇ , the proximal position ⁇ , the proximal state the estimated tip position ⁇ ⁇ ⁇ physical parameters p and spatial environment e. Based on the input to the long-term predictive encoder 815, the long-term predictive decoder 820 generates the estimated tip position ⁇ ⁇ ⁇ .
  • the transformer 805 is configured to generate at least one attention heat map 830 for image planes.
  • the attention heat maps may be generated based on at least one of real images 802 generated by the imaging system 32 or estimated images from the AI predictive control system 460.
  • the forward agent 710 receives a first plane image set 802a and a second plane image set 802a.
  • Each of the images sets 802a, 802b may include a DSA image and a fluoroscopic image.
  • the transformer 805 generates attention heat maps 830a, 830b for the first and second image planes, respectively.
  • the long-term predictive decoder 820 is configured to generate estimated heat maps 835a, 835b for each image plane using the estimated tip position ⁇ ⁇ ⁇ .
  • FIG.9 illustrates an inverse agent shown in FIG.7A according to one or more example embodiments.
  • the inverse agent 705 includes an autoregressive time-series transformer 910.
  • the autoregressive time-series transformer 910 includes a long-term predictive encoder 915 and a long-term predictive decoder 920.
  • the long-term predictive encoder 915 receives ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ , ⁇ ⁇ ⁇ , ⁇ , ⁇ as inputs.
  • the AI predictive control system provides an efficient approach to indicate when to turn on/off X-ray.
  • FIG.10 illustrates an example transformer model used to implement the AI predictive control system according to one or more example embodiments.
  • a transformer 1000 consists of an encoder 1010, decoder 1002, positional encodings 1003 with concatenation operation at the encoder 1010 and the decoder 1002, input embedding 1004, and output embedding 1005.
  • An input x is embedded with positional encoding.
  • the encoder 1010 maps an input sequence of symbol representations x to a sequence of representations.
  • a multi-head attention mechanism 1011 performs self-attention processing on the inputs.
  • the multi-headed attention mechanism 1011 encodes information about the relevant context, which allows the model to focus on relevant contexts at different length scales.
  • a feed-forward network 1013 performs a post processing operation to generate the sequence of representations z.
  • the decoder 102 generates an output sequence y of symbols one element at a time.
  • the transformer 1000 uses L-layers in the encoder 1010, to perform sequential operations (e.g., 110_1, 110_2, ... , 110_L) and L layers for the decoder 1002.
  • Layers of encoder 1010 are processed by Add and Norm components 1012, 1014 via skip feeds 1015, 1016 to perform residual connection followed by layer normalization.
  • Add and Norm components 1012, 1014 via skip feeds 1015, 1016 to perform residual connection followed by layer normalization.
  • layers of encoder 1010 are processed by Add and Norm components 1012, 1014 via skip feeds 1015, 1016 to perform residual connection followed by layer normalization.
  • first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of this disclosure.
  • the term "and/or,” includes any and all combinations of one or more of the associated listed items.
  • processing or control circuitry such as, but not limited to, one or more processors, one or more Central Processing Units (CPUs), one or more controllers, one or more arithmetic logic units (ALUs), one or more digital signal processors (DSPs), one or more microcomputers, one or more field programmable gate arrays (FPGAs), one or more System-on-Chips (SoCs), one or more programmable logic units (PLUs), one or more microprocessors, one or more Application Specific Integrated Circuits (ASICs), or any other device or devices capable of responding to and
  • processors such as, but not limited to, one or more processors, one or more Central Processing Units (CPUs), one or more controllers, one or more arithmetic logic units (ALUs), one or more digital signal processors (DSPs), one or more microcomputers, one or more field programmable gate arrays (FPGAs), one or more System-on-Chips (SoCs), one or more programmable logic units (
  • Siemens Healthineers AG 32 Although a flow chart may describe the operations as a sequential process, many of the operations may be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. A process may be terminated when its operations are completed, but may also have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.
  • the term “memory,” “storage medium,” “processor readable medium,” “computer readable storage medium” or “non- transitory computer readable storage medium” may represent one or more devices for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other tangible machine- readable mediums for storing information.
  • ROM read only memory
  • RAM random access memory
  • magnetic RAM magnetic RAM
  • core memory magnetic disk storage mediums
  • optical storage mediums optical storage mediums
  • flash memory devices and/or other tangible machine- readable mediums for storing information.
  • computer-readable medium may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other mediums capable of storing, containing or carrying instruction(s) and/or data.
  • example embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof.
  • the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as a computer readable storage medium.
  • a processor or processors When implemented in software, a processor or processors will perform the necessary tasks.
  • at least one memory may include or store computer program code, and the at least one memory and the computer program code may be configured to, with at least one processor, cause a network element or network device to perform the necessary tasks.
  • processor memory and example algorithms, encoded as computer program code, serve as means for providing or causing performance of operations discussed herein.
  • the terms “including” and/or “having,” as used herein, are defined as comprising (i.e., open language).
  • the term “coupled,” as used herein, is defined as connected, although not necessarily directly, and not necessarily mechanically. Terminology derived from the word “indicating” (e.g., “indicates” Siemens Healthineers AG 33 and “indication”) is intended to encompass all the various techniques available for communicating or referencing the object/information being indicated.
  • Some, but not all, examples of techniques available for communicating or referencing the object/information being indicated include the conveyance of the object/information being indicated, the conveyance of an identifier of the object/information being indicated, the conveyance of information used to generate the object/information being indicated, the conveyance of some part or portion of the object/information being indicated, the conveyance of some derivation of the object/information being indicated, and the conveyance of some symbol representing the object/information being indicated.
  • a method comprising obtaining, by a trained system, at least one movement command for an intervention device of a robotic surgical system to a location within a patient; determining, by the trained system, at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the intervention device; and applying the actuator command to the robotic surgical system.
  • Illustrative embodiment 2 The method of illustrative embodiment 1, wherein the intervention device is a guidewire.
  • Illustrative embodiment 3 The method of illustrative embodiment 2, wherein the at least one movement command is a desired location within the patient of a tip of the guidewire.
  • any one of illustrative embodiments 2-3 further comprising obtaining an observed position of the guidewire from the robotic surgical system; and determining the estimated Siemens Healthineers AG 34 current position of the guidewire based on the observed position.
  • Illustrative embodiment 5 The method of illustrative embodiment 4, wherein the observed position of the intervention device corresponds to an observed position of a proximal portion of the guidewire.
  • Illustrative embodiment 6. The method of illustrative embodiment 5, wherein the estimated current position corresponds to an estimated position of a tip of the guidewire.
  • Illustrative embodiment 8 The method of any one of illustrative embodiments 2-7, further comprising generating image data displaying the estimated current position of the guidewire within a vessel of the patient.
  • Illustrative embodiment 9. The method of any one of illustrative embodiments 2-8, wherein the estimated current position of the guidewire is overlaid on a three-dimensional vessel segmentation.
  • Illustrative embodiment 11 A trained system comprising processing circuitry configured to cause the trained system to obtain at least one movement command for an intervention device of a robotic surgical system to a location within a patient, determine at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the intervention device, and apply the actuator command to the robotic surgical system.
  • Illustrative embodiment 12. The trained system of illustrative embodiment 11, wherein the intervention device is a guidewire.
  • Illustrative embodiment 13 A trained system comprising processing circuitry configured to cause the trained system to obtain at least one movement command for an intervention device of a robotic surgical system to a location within a patient, determine at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current position of the intervention device, and apply the actuator command to the robotic surgical system.
  • the trained system of illustrative embodiment 12, wherein the at least one movement command is a desired Siemens Healthineers AG 35 location within the patient of a tip of the guidewire.
  • Illustrative embodiment 14 The trained system of any one of illustrative embodiments 12-13, wherein the processing circuitry is configured to cause the trained system to obtain an observed position of the guidewire from the robotic surgical system; and determine the estimated current position of the guidewire based on the observed position.
  • Illustrative embodiment 15 The trained system of illustrative embodiment 14, wherein the observed position of the intervention device corresponds to an observed position of a proximal portion of the guidewire.
  • Illustrative embodiment 17 The trained system of any one of illustrative embodiments 12-16, wherein the processing circuitry is configured to cause the trained system to determine a series of actuator commands, each actuator command of the series of actuator commands corresponding to a state of the guidewire.
  • Illustrative embodiment 18 The trained system of any one of illustrative embodiments 12-17, wherein the processing circuitry is configured to cause the trained system to generate image data displaying the estimated current position of the guidewire within a vessel of the patient.
  • Illustrative embodiment 20 The trained system of any one of illustrative embodiments 11-19, wherein the actuator command is at least one of a velocity command or a torque command for the actuator.
  • Illustrative embodiment 21 A trained system comprising means for obtaining at least one movement command for an intervention device of a robotic surgical system to a location within a patient; means for determining at least one actuator command for the robotic surgical system associated with the at least one movement command based on the movement command and an estimated current Siemens Healthineers AG 36 position of the intervention device; and means for applying the actuator command to the robotic surgical system.
  • Illustrative embodiment 22 The trained system of illustrative embodiment 21, wherein the intervention device is a guidewire.
  • Illustrative embodiment 23 The trained system of illustrative embodiment 22, wherein the at least one movement command is a desired location within the patient of a tip of the guidewire.
  • Illustrative embodiment 24 The trained system of any one of illustrative embodiments 22-23, further comprising means for obtaining an observed position of the guidewire from the robotic surgical system; and means for determining the estimated current position of the guidewire based on the observed position.
  • Illustrative embodiment 25 The trained system of any one of illustrative embodiments 22-23, further comprising means for obtaining an observed position of the guidewire from the robotic surgical system; and means for determining the estimated current position of the guidewire based on the observed position.

Landscapes

  • Health & Medical Sciences (AREA)
  • Surgery (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Robotics (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Manipulator (AREA)

Abstract

Un ou plusieurs modes de réalisation donnés à titre d'exemple concernent un procédé comprenant l'obtention, par un système entraîné, d'au moins une commande de mouvement pour un dispositif d'intervention d'un robot chirurgical à un emplacement à l'intérieur d'un patient ; la détermination, par le système entraîné, d'au moins une commande d'actionneur pour le robot chirurgical associé à la ou aux commandes de mouvement sur la base de la commande de mouvement et d'une position actuelle estimée du dispositif d'intervention ; et l'application de la commande d'actionneur au robot chirurgical.
PCT/EP2024/053993 2023-02-17 2024-02-16 Systèmes et procédés de manipulation automatisée de dispositif médical endovasculaire robotique Ceased WO2024170741A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202480012805.XA CN120693124A (zh) 2023-02-17 2024-02-16 用于自动化机器人血管内医疗设备操纵的系统和方法
EP24706065.0A EP4648705A1 (fr) 2023-02-17 2024-02-16 Systèmes et procédés de manipulation automatisée de dispositif médical endovasculaire robotique

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363485582P 2023-02-17 2023-02-17
US63/485,582 2023-02-17

Publications (1)

Publication Number Publication Date
WO2024170741A1 true WO2024170741A1 (fr) 2024-08-22

Family

ID=89983018

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2024/053993 Ceased WO2024170741A1 (fr) 2023-02-17 2024-02-16 Systèmes et procédés de manipulation automatisée de dispositif médical endovasculaire robotique

Country Status (3)

Country Link
EP (1) EP4648705A1 (fr)
CN (1) CN120693124A (fr)
WO (1) WO2024170741A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240411278A1 (en) * 2023-06-08 2024-12-12 Hong Kong Quantum Ai Lab Limited Artificial intelligence and robot based automated material synthesis system
US12498683B2 (en) * 2024-04-18 2025-12-16 Hong Kong Quantum Ai Lab Limited Artificial intelligence and robot based automated material synthesis system

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8480618B2 (en) 2008-05-06 2013-07-09 Corindus Inc. Catheter system
US8790297B2 (en) 2009-03-18 2014-07-29 Corindus, Inc. Remote catheter system with steerable catheter
US9452277B2 (en) 2012-09-06 2016-09-27 Corindus, Inc. Hemostasis valve for guide catheter control
US9545497B2 (en) 2008-12-12 2017-01-17 Corindus, Inc. Remote catheter procedure system
US10282638B2 (en) 2015-07-29 2019-05-07 Siemens Healthcare Gmbh Tracking for detection of TEE probe in fluoroscopy medical imaging
US20210093406A1 (en) * 2009-10-12 2021-04-01 Corindus, Inc. System and method for navigating a guide wire
WO2021258113A1 (fr) * 2020-06-19 2021-12-23 Remedy Robotics, Inc. Systèmes et procédés de guidage de dispositifs intraluminaux à l'intérieur du système vasculaire
US20220409309A1 (en) * 2021-06-29 2022-12-29 Siemens Healthcare Gmbh Method for generating an actuation signal for a robotic system for robot-assisted navigation of a medical object and overall system

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8480618B2 (en) 2008-05-06 2013-07-09 Corindus Inc. Catheter system
US9545497B2 (en) 2008-12-12 2017-01-17 Corindus, Inc. Remote catheter procedure system
US8790297B2 (en) 2009-03-18 2014-07-29 Corindus, Inc. Remote catheter system with steerable catheter
US20210093406A1 (en) * 2009-10-12 2021-04-01 Corindus, Inc. System and method for navigating a guide wire
US9452277B2 (en) 2012-09-06 2016-09-27 Corindus, Inc. Hemostasis valve for guide catheter control
US10282638B2 (en) 2015-07-29 2019-05-07 Siemens Healthcare Gmbh Tracking for detection of TEE probe in fluoroscopy medical imaging
WO2021258113A1 (fr) * 2020-06-19 2021-12-23 Remedy Robotics, Inc. Systèmes et procédés de guidage de dispositifs intraluminaux à l'intérieur du système vasculaire
US20220409309A1 (en) * 2021-06-29 2022-12-29 Siemens Healthcare Gmbh Method for generating an actuation signal for a robotic system for robot-assisted navigation of a medical object and overall system

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
AVRAM ET AL.: "CathAI: fully automated coronary angiography interpretation and stenosis estimation", NPJ DIGITAL MEDICINE, vol. 6, no. 142, 2023
GUO ET AL.: "DeepCenterline: a Multi-task Fully Convolutional Network for Centerline Extraction", ARXIV:1903:10481V1, 25 March 2019 (2019-03-25)
JIHOON KWEON: "Deep Reinforcement Learning for Guidewire Navigation in Coronary Artery Phantom", IEEE ACCESS, vol. 9, 13 December 2021 (2021-12-13), USA, pages 166409 - 166422, XP093158733, ISSN: 2169-3536, Retrieved from the Internet <URL:https://ieeexplore.ieee.org/ielx7/6287639/9312710/09648308.pdf?tp=&arnumber=9648308&isnumber=9312710&ref=aHR0cHM6Ly9pZWVleHBsb3JlLmllZWUub3JnL2RvY3VtZW50Lzk2NDgzMDg=> DOI: 10.1109/ACCESS.2021.3135277 *
LI ET AL.: "Efficient Self-supervised Vision Transformers for Representation Learning", ARXIV:2106.09785V2, 6 July 2022 (2022-07-06)
TSANG: "3D U-Net - Volumetric Segmentation (Medical Image Segmentation", TOWARDS DATA SCIENCE, 2 April 2019 (2019-04-02)
VASWANI ET AL.: "Attention is all you need", ADVANCE IN NEURAL INFORMATION PROCESSING SYSTEMS, vol. 30, 2017
VASWANI ET AL.: "Attention Is All You Need", ARXIV:1706.03762V7

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240411278A1 (en) * 2023-06-08 2024-12-12 Hong Kong Quantum Ai Lab Limited Artificial intelligence and robot based automated material synthesis system
US12498683B2 (en) * 2024-04-18 2025-12-16 Hong Kong Quantum Ai Lab Limited Artificial intelligence and robot based automated material synthesis system

Also Published As

Publication number Publication date
EP4648705A1 (fr) 2025-11-19
CN120693124A (zh) 2025-09-23

Similar Documents

Publication Publication Date Title
Wang et al. Remote‐controlled vascular interventional surgery robot
Chen et al. 3-D reconstruction of coronary arterial tree to optimize angiographic visualization
US11357573B2 (en) Optimum treatment planning during coronary intervention by simultaneous simulation of a continuum of outcomes
US7371067B2 (en) Simulation method for designing customized medical devices
Chen et al. Kinematic and deformation analysis of 4-D coronary arterial trees reconstructed from cine angiograms
AU2023350692A1 (en) System and method for coronary vascular diagnosis and prognosis with a biomechanical stress profiling index
CN116172699A (zh) 机器人导管导航系统的风险管理
Zhang et al. A novel spatial position prediction navigation system makes surgery more accurate
Scarponi et al. Autonomous guidewire navigation in dynamic environments
Barata et al. IVUS-based local vessel estimation for robotic intravascular navigation
Yao et al. Sim4EndoR: A reinforcement learning centered simulation platform for task automation of endovascular robotics
WO2024170741A1 (fr) Systèmes et procédés de manipulation automatisée de dispositif médical endovasculaire robotique
EP4322879B1 (fr) Navigation de dispositif d&#39;intervention
Evard Catheter localization utilizing a sensor-enabled guidewire design of a proof-of-concept system
Kim et al. AI-based Agents for Automated Robotic Endovascular Guidewire Manipulation
Shi et al. Guidewire tracking based on visual algorithm for endovascular interventional robotic system
Li et al. Deformable model-to-image registration toward augmented reality-guided endovascular interventions
Lee et al. Motion-adapted catheter navigation with real-time instantiation and improved visualisation
CN120105902B (zh) 用于对微导管输送模型进行训练的方法及相关产品
US20250356990A1 (en) Percutaneous coronary intervention planning
Ji et al. Guidewire navigation and delivery system for robot-assisted cardiology interventions
Koland et al. 3D Reconstruction of Coronary Vessel Trees from Biplanar X-Ray Images Using a Geometric Approach
Wang et al. From Patient-specific Digital Twin to Real-world Phantom: Autonomous Right Heart Catheterization
EP4623417A1 (fr) Fourniture d&#39;images de projection
Ainchil Cayuela 3D vessel reconstruction based on intra-operative intravascular ultrasound for robotic autonomous catheter navigation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24706065

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202480012805.X

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 202480012805.X

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 2024706065

Country of ref document: EP