[go: up one dir, main page]

WO2025054377A1 - Retouche et mise à jour de cible pour imagerie péropératoire - Google Patents

Retouche et mise à jour de cible pour imagerie péropératoire Download PDF

Info

Publication number
WO2025054377A1
WO2025054377A1 PCT/US2024/045456 US2024045456W WO2025054377A1 WO 2025054377 A1 WO2025054377 A1 WO 2025054377A1 US 2024045456 W US2024045456 W US 2024045456W WO 2025054377 A1 WO2025054377 A1 WO 2025054377A1
Authority
WO
WIPO (PCT)
Prior art keywords
dimensional
processors
images
coordinate system
flexible elongate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/US2024/045456
Other languages
English (en)
Inventor
Zhehao HU
Hui Zhang
Changxin Lai
Jorge Anton GARCIA
Trevor W. Laing
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intuitive Surgical Operations Inc
Original Assignee
Intuitive Surgical Operations Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intuitive Surgical Operations Inc filed Critical Intuitive Surgical Operations Inc
Publication of WO2025054377A1 publication Critical patent/WO2025054377A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present disclosure relates to planning and/or navigating minimally invasive medical procedures and, more specifically, to in-painting and other enhancements for intraoperative imaging.
  • Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects.
  • Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, physicians may insert minimally invasive medical instruments (including surgical, diagnostic, and/or therapeutic instruments) to reach a target tissue location.
  • minimally invasive technique is to use a flexible and/or steerable elongate device, such as a flexible catheter, that can be inserted into anatomic passageways and navigated toward a region of interest within the patient anatomy.
  • a combination of localization sensors disposed at a flexible elongate device and intraoperative imaging can greatly aid in planning and navigating a minimally invasive procedure.
  • combining sensor data with intraoperative images can enable accurate determination of a position, orientation, and/or pose of the flexible elongate device within the patent anatomy.
  • accessible intraoperative techniques such as tomosynthesis based on a limited set of projections, can suffer from low image quality and reconstruction artifacts. Improving the quality of intraoperative tomosynthesis images and extracting accurate intra-operative anatomical data remains a challenge using current techniques.
  • a tangible, non-transitory, computer readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain a first plurality of two-dimensional images depicting an anatomical structure and one or more objects.
  • the instructions may further cause the one or more processors to compute, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects.
  • the instructions may cause the one or more processors to cause a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.
  • a system for visualizing patient anatomy during a medical procedure comprises a display device, one or more processors, and one or more non- transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to obtain a first plurality of two- dimensional images depicting an anatomical structure and one or more objects.
  • the instructions may further cause the one or more processors to compute, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects.
  • the instructions may cause the one or more processors to cause a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.
  • a method for visualizing patient anatomy during a medical procedure comprises obtaining, by one or more processors, a first plurality of two- dimensional images depicting an anatomical structure and one or more objects.
  • the method may further comprise computing, by the one or more processors, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects.
  • the method may comprise causing, by the one or more processors, a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.
  • a tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to obtain a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device, and to identify, within each of at least two of the plurality of two-dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device.
  • the instructions may further cause the one or more processors to compute, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point, and to identify, within at least one of the at least two of the plurality of two-dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point.
  • the instructions may cause the one or more processors to receive, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point, and to register the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images. Still further, the instructions may cause the one or more processors to reconstruct, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure, and to identify, within the reconstructed three-dimensional image of the anatomical structure, a target. Still further, the instructions may cause the one or more processors to compute, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target.
  • a system for visualizing patient anatomy during a medical procedure comprises a display device, one or more processors, and one or more non- transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to obtain a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device.
  • the instruction may further cause the one or more processors to identify, within each of at least two of the plurality of two- dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device, and to compute, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates in an imaging coordinate system of the reference point. Still further, the instructions may cause the one or more processors to identify, within at least one of the at least two of the plurality of two-dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point.
  • the instructions may cause the one or more processors to receive, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point, and to register the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images. Still further, the instructions may cause the one or more processors to reconstruct, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure, and to identify, within the reconstructed three-dimensional image of the anatomical structure, a target.
  • the instructions may cause the one or more processors to compute, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target, and to cause the display device to display a graphical user interface depicting the computed location of the target in the instrument coordinate system.
  • a method for visualizing patient anatomy during a medical procedure comprises obtaining, by one or more processors, a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device.
  • the method may further comprise identifying, by the one or more processors, within each of at least two of the plurality of two- dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device, and computing, by the one or more processors, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point.
  • the method may comprise identifying, by the one or more processors, within at least one of the at least two of the plurality of two-dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point. Still further, the method may comprise receiving, by the one or more processors, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point, and registering, by the one or more processors, the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images.
  • the method may comprise reconstructing, by the one or more processors, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure, and identifying, by the one or more processors, within the reconstructed three- dimensional image of the anatomical structure, a target. Still further, the method may comprise computing, by the one or more processors, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target; and causing, by the one or more processors, a display device to display a graphical user interface depicting the identified or updated location of the target in the instrument coordinate system.
  • FIG. 1 A depicts an example system for navigating during a medical procedure within an operating environment.
  • FIG. 1 B is a simplified diagram of a flexible elongate device disposed within an anatomical structure.
  • FIGS. 1 C, 1 D are simplified diagrams depicting an example intraoperative imaging geometry for tomosynthesis.
  • FIG. 1 E is a simplified three-dimensional representation of a flexible elongate device disposed within an anatomical structure within the imaging geometry of FIGS. 1C, D.
  • FIG. 1 F depicts a portion of a projection image of a flexible elongate device disposed within an anatomical structure overlayed with fiducial markings.
  • FIGS. 1G, 1 H schematically illustrates projections of an example fiducial plate at two projection angles.
  • FIGS. 2A-D schematically illustrate an example procedure for in-painting fiducial markings in a projection image of a flexible elongate device disposed within an anatomical structure.
  • FIGS. 2E-H schematically illustrate an example procedure for in-painting a flexible elongate device in a projection image of an anatomical structure.
  • FIG. 3A schematically illustrates an example multi-projection image acquisition geometry for a flexible elongate device disposed within an anatomical structure.
  • FIG. 3B, 3C are example projection images for the image acquisition geometry of FIG. 3A.
  • FIG. 4A depicts a rigid body in two example coordinate systems.
  • FIG. 4B schematically illustrates an example coordinate transformation process.
  • FIG. 4C depicts an example geometry for flexible elongate device extraction and coordinate registration using shape data and projection images.
  • FIGS. 5A, 5B schematically illustrate example processes for flexible elongate device extraction, coordinate registration and target update using shape data and projection images.
  • FIG. 6A schematically illustrates an example style transfer for three-dimensional images.
  • FIG. 6B schematically illustrates an example training process for a style-transfer machine learning model.
  • FIGS. 13A, 13B depict, respectively, outputs of tomosynthesis imaging and conebeam computed tomography (CBCT) imaging of patient anatomy.
  • CBCT conebeam computed tomography
  • the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (e.g., up to six total degrees of freedom).
  • the term “shape” refers to a set of poses, positions, and/or orientations measured along an object.
  • distal refers to a position that is closer to a procedural site and the term “proximal” refers to a position that is further from the procedural site. Accordingly, the distal portion or distal end of an instrument is closer to a procedural site than a proximal portion or proximal end of the instrument when the instrument is being used as designed to perform a procedure.
  • the disclosure generally relates to systems and methods that facilitate user (e.g., physician) planning of, and/or user navigation during, an endoluminal medical procedure.
  • These systems and methods can provide an improvement in visualization of patient anatomy based on tomosynthesis of two-dimensional projection images from a limited set of angles.
  • the improvement in tomosynthesis output can lead to improved detectability of lesions and/or other targets within the patient anatomy.
  • the systems and methods described in the disclosure can facilitate registration between an imaging coordinate system and a coordinate system of an instrument (e.g., a flexible elongate device) disposed within patient anatomy.
  • the registration techniques described in the disclosure can improve the medical procedure accuracy and/or speed by facilitating, in turn, translating a target location from intraoperative images into the instrument coordinate system.
  • Fiducial markings which may originate from a fiducial plate placed beneath, above, or around the body of the patient, can aid in determining projection angles (e.g., the angular position of the arm of an x-ray imaging apparatus) but tend to interfere with tomographic reconstruction once the angles are determined.
  • In-painting the fiducial markings may comprise several steps. First, for each of the two-dimensional images, the system may generate a binary value mask that zeros out pixel values associated with the fiducial markings. In some examples, the system may generate the mask for a given image solely based on the image itself. In other examples, the system may use a sequence of images from different angles to aid in determining the mask for each of the angles. To generate the masks, the system may detect fiducials based on fiducial size, fiducial shape, and/or spacings between the fiducials. The system may use a variety of signal processing techniques to compensate for partial detections of fiducial markings.
  • the system may “dilate” a mask to ensure capturing all the pixels of the fiducial markings. After removing the fiducial markings, the system may fill in the removed pixels using one or more of a variety of in-painting techniques.
  • the system may use computer vision or image processing algorithms such as a Navier-Stokes gradient-based method or a fast-marching-based method.
  • the system may use machine learning (ML) for in-painting.
  • ML machine learning
  • the system may use progressive inpainting, attention-based in-painting, and/or pluralistic in-painting.
  • the system may use a variety of possible neural network structures such as an auto-encoder, a generative adversarial network (GAN), and/or a diffusion model to implement in-painting.
  • GAN generative adversarial network
  • systems and methods of the disclosure may apply in-painting techniques to remove an instrument (e.g., a flexible elongate device) disposed within patient anatomy.
  • the systems and methods of the disclosure may need to identify pixels associated with the instrument within the two-dimensional projections.
  • a system may identify pixels associated with the instrument based at least in part on user input. For example, the system may generate on a display device a graphical user interface (GUI) displaying a plurality of projections, and the user may select in at least two of the displayed projections pixels corresponding to the same reference point disposed at the instrument.
  • GUI graphical user interface
  • the system may compute the coordinates of the reference point in three dimensions by tracing back from the projected reference point along the projection rays and finding the intersection of the projection rays from different projections. Furthermore, the system may identify pixels forming a continuous curve associated with a section of the instrument in at least one of the projections. In some examples, the system may generate a GUI display and a projection and prompt a user to select a plurality of points along the curve associated with the section of the instrument. In other examples, the system may automatically determine the pixels forming a curve associated with the section of the instrument using one or more image processing techniques (e.g., thresholding, segmentation, contour detection, etc.).
  • image processing techniques e.g., thresholding, segmentation, contour detection, etc.
  • the system may back- project the two-dimensional curve into three dimensions of the imaging coordinate system to compute a three-dimensional curve associated with the section of the instrument.
  • the system may perform a reconstruction of the three-dimensional image based on projections and trace rays along the projection direction from the projection in which the pixels forming the curve are identified to find intensity maxima and/or other indications of the instrument.
  • the system may trace along projections of rays from the first projection image (where the pixels associated with the curve are identified) in a second projection image and identify pixels associated with the projection of the instrument onto the second projection image. In this manner, the system may reconstruct the curve associated with a section of the instrument in three dimensions based on at least two projection images, as described in more detail with reference to FIG. 4C.
  • the system may compute the three-dimensional curve associated with the instrument (e.g., at least a section of the instrument) disposed within an anatomical structure based at least in part on shape data obtained from a sensing system. Because the shape data from a sensing system may be in a different coordinate system from the imaging coordinate system, a method of using the shape data may include a registration process between the two coordinate systems. Registration of the coordinate systems along with using the shape data to identify the three-dimensional curve (associated with the instrument) in imaging coordinates is discussed in more detail with reference to FIGS. 4A-C.
  • the three-dimensional curve associated with the instrument may serve to generate in-painting masks for the instrument.
  • the system may project the curve onto each a set of two-dimensional projection images obtained from an imaging system to generate in-painting masks.
  • the in-painting masks may be dilated, and the in-painting process may be analogous to the in-painting process for fiducial markings, as described below with reference to FIGS. 2A-H.
  • the system may proceed to reconstruct a three- dimensional section of patient anatomy based on the set of two-dimensional images and respective angles.
  • the system may use back projection with suitable regularization and/or any other suitable algorithm for three-dimensional image reconstruction.
  • the system may be configured to enhance the reconstructed image using a style transfer technique. That is, the system may be configured to generate from a tomosynthesis image an enhanced image that has the style of a computed tomography (CT) or cone beam computed tomography (CBCT) image.
  • CT computed tomography
  • CBCT cone beam computed tomography
  • a method of operating the system may include training an ML model to transfer a CT/CBCT style to the tomosynthesis image.
  • the ML model may include a convolutional neural network (CNN) (e.g., U-net), a transformer network, a GAN, and/or a diffusion model.
  • CNN convolutional neural network
  • Training may include generating simulated tomosynthesis input by taking a subset of angles from CT/CBCT data, adding suitable noise to projections and respective angle uncertainties, generating tomosynthesis data and training based on the simulated tomosynthesis data to reduce loss with respect to the ground truth based on CT or CBCT data.
  • the system may then apply the trained style transfer model to real tomosynthesis data to reduce tomosynthesis artifacts and generate volumetric images or slices that resemble CT and/or CBCT outputs.
  • the techniques of this disclosure may enhance an inverse image obtained from a sparse set of measurements using a style transfer model trained on a less sparse set of measurements.
  • the system may render on a display device a graphical user interface depicting visualization data at any stage of the process: two-dimensional images with fiducials, the images with fiducials removed, a tomosynthesis image before style transfer, and/or a tomosynthesis image after style transfer.
  • the system may allow a user to select, e.g., via the graphical user interface, parameters and/or techniques (e.g., as briefly described above) for any stage of image processing.
  • a system may enhance one or more two-dimensional images which are not associated with respective viewing angles by in-painting one or more fiducial markings within the one or more two-dimensional images. Still in other examples, the system may enhance a tomosynthesis image using style transfer (e.g., transferring a CT/CBCT style to the tomosynthesis image).
  • style transfer e.g., transferring a CT/CBCT style to the tomosynthesis image.
  • fiducial in-painting need not be associated with tomosynthesis or style transfer
  • style transfer enhancement of tomosynthesis or other imaging modalities need not be associated with fiducial in-painting (e.g., can apply to methods where the system determines viewing angles without the use of fiducial plates).
  • the techniques for coordinate registration between the instrument coordinate system and the imaging coordinate system may aid (as described, for example, with reference to FIGS. 5A, B) in accurately determining a location of a target within patient anatomy and, thereby, increase speed and/or accuracy of a medical procedure.
  • FIG. 1 A depicts an example system 100 for navigating during a medical procedure within an operating environment 101.
  • the system 100 may obtain images from a portion of the operating environment 101 disposed within a field of view F (approximately demarcated by dashed lines) of an imaging unit 110. To that end, the system 100 may be in communicative connection with the imaging unit 110. Additionally, the system 100 may obtain data from one or more sensors disposed within the operating environment. To that end, the system 100 may be in communicative connection with the one or more sensors via a communicative connection with a sensing unit 115.
  • the system 100 includes a processing unit 120 and a display unit 130 in communicative connection with each other.
  • the imaging unit 110 and the sensing unit 115 are depicted as being distinct from the system 100, in other examples, the system 100 may include the imaging unit 110 and/or the sensing unit 115.
  • one or more processors of the processing unit 120 of the system 100 may be configured to receive images and/or processed image information from the imaging unit 110 and to receive data from the one or more sensors by way of the sensing unit 115.
  • the descriptions of example operations performed by the processing unit 120 below are to be understood to be executed by the one or more processors of the processing unit 120.
  • the one or more processors may include hardware specifically configured (e.g., hardwired or programmable) to carry out at least a portion of the example operations described in this disclosure. Additionally or alternatively, the one or more processors may be configured to carry out at least a portion of the example operations described in this disclosure by carrying out a set of software instructions.
  • the system 100 may include or be communicatively connected to a tangible, non-transitory, computer readable medium. The medium may store instructions which, when executed by the processing unit 120, perform any one or more of the example operations described below.
  • the instructions may cause the processing unit 120 to perform image processing operations on the images received from the imaging unit 110 and/or to perform computations (e.g., for coordinate registration) based on the data received by way of the sensing unit 115.
  • the instructions may cause the processing unit 120 to cause the display unit 130 to display, via a GUI, information based on the processing of images received from the imaging unit 110 and the processing of data received by way of the sensing unit 115.
  • the processing unit 120 may send the information, or send data representing the entire GUI including the information, to the display unit 130.
  • An operator e.g., a physician, another medical practitioner, or a fully-automated robotic surgery system
  • a medical procedure e.g., endoscopy, biopsy, pharmacological treatment, and/or treatment, such as ablation
  • the operator may control a flexible elongate device 140 inserted through an orifice O (or through a suitable surgical incision) into an anatomical structure A of a patient P disposed at a table T.
  • the medical procedure may include navigating the flexible elongate device 140 (indicated with solid lines outside and dashed lines inside the patient P) toward an ROI R within the anatomical structure A with the aid of information displayed at the display unit 130.
  • the ROI R may be a designated procedure site for visual examination, biopsy, treatment, or any other medical procedure.
  • the ROI R may be referred to as a region or a target.
  • One or more fiducial may be disposed at (e.g., physically contacting, integrated within, fixedly attached to, or removably attached to in a manner that, during operation/use, forms a rigid relationship with) the flexible elongate device 140.
  • the fiducials are configured to be visible in images obtained by the imaging unit 110 to thereby enhance visibility of the flexible elongate device 140 and/or to aid in identifying certain points along the device 140.
  • the fiducials may include elements of a variety of materials and/or structures such as metals, plastics, etched glass, dyes, radioactive or fluorescent markings, confined fluids (e.g., bubbles), etc.
  • the fiducial 142 may be integrated (e.g., etched, deposited, painted, or otherwise fixedly attached) onto the flexible elongate device 140. Additionally or alternatively, the fiducials may include elements removably disposed at the flexible elongate device 140. For example, the fiducials may be integrated onto a removable structure, such as a sleeve or a stylet, which in turn may be removably attached to the flexible elongate device 140.
  • a removable structure such as a sleeve or a stylet
  • One or more sensors may also be disposed at the flexible elongate device.
  • the sensors may be mechanical sensors, optical sensors, electromagnetic (EM) sensors, or any other suitable sensors.
  • the sensors may be integrated into the flexible elongate device 140, or removably attached to the flexible elongate device 140.
  • the sensors may be configured to communicate with the sensing unit 115.
  • the sensors are active sensors, configured to transmit electromagnetic (e.g., optical, radio, low-frequency magnetic) or radioactive radiation.
  • the sensing unit 115 may include components to receive the radiation emitted by the sensors and triangulate or trilaterate locations of the sensors in the sensor system coordinates.
  • the sensor system coordinates may be the coordinates of the robotic-assisted system configured to manipulate, control, or guide the flexible elongate device 140. As such, the sensor system coordinates may be referred to as instrument coordinates.
  • the sensors are passive and emit no radiation.
  • the passive sensors may sense radiation emitted by components of the sensing unit 115 disposed within the operating environment 101.
  • one or more electrified coils may be disposed within the operating environment 101 to generate static or dynamic magnetic fields.
  • the sensors may be configured to pick up variations in the dynamic magnetic fields or, as the sensors move, variations in the sensed static or quasistatic magnetic field and convert the sensed variations into electrical currents that are received by the sensing unit 115.
  • the sensing unit 115 may compute indications of positions of the sensors from the received electrical currents.
  • the sensors may be fiberoptic sensors disposed along the length of the flexible elongate device.
  • the fiberoptic sensors may include Bragg gratings and/or materials to enhance non-linear scattering.
  • the fiberoptic sensors may be configured to change spectral reflectivity based on material strain. Such sensors may scatter light emitted, for example, by the sensing unit 115 in a manner that indicates locations and degrees of bends in the flexible elongate device 140.
  • sensors may include accelerometers, gyroscopes, and/or magnetometers.
  • the point sensors may include inertial measurement units (IMUs) that combine multiple sensors (e.g., accelerometers, gyroscopes) and/or inertial and magnetic measurement units (IMMUs) that combine multiple sensors (e.g., accelerometers, gyroscopes, magnetometers).
  • IMUs and/or IMMUs may generate signals indicative of orientation of the flexible elongate device at a given point with respect to gravitational field and/or magnetic field (e.g., of Earth).
  • additional magnetic fields may be introduced to the sensor environment.
  • IMUs and/or IMMUs may generate signals indicative of motion of the flexible elongate device (e.g., caused by motion of an anatomical structure due to breathing and/or other factors, and/or independent motion of the flexible elongate device within the anatomical structure).
  • the sensing unit 115 may combine indications of orientation (e.g., up to three degrees of freedom) from IMUs with indications of position from other (e.g., EM) sensors to generate more complete data indicative of pose of the flexible elongate device.
  • the sensing unit 115 may combine data from sensors in multiple sensor coordinate systems.
  • the sensing unit 115 and/or the processing unit 120 may register multiple sensor coordinate systems with each other.
  • the system 100 may use any suitable combinations of the sensors described above to obtain shape data indicative of shape of the flexible elongate device 140 (e.g., pose, including location and orientation, at different points along the length of the flexible elongate device 140).
  • shape data indicative of shape of the flexible elongate device 140 e.g., pose, including location and orientation, at different points along the length of the flexible elongate device 140.
  • FIG. 1 B is a simplified diagram of the flexible elongate device 140 disposed within the anatomical structure A.
  • FIG. 1 B is included to give an expanded and more detailed view of a portion of the operating environment 101 disposed within the field of view F.
  • the anatomical structure A may be a lung of the patient P.
  • the flexible elongate device 140 may be inserted into and navigated by the operator toward the region R (e.g., target of the medical procedure), for example, for the purpose of investigating or treating a pathology in the region R.
  • the techniques described in the present disclosure can facilitate the navigation process by generating and displaying timely and accurate imaging and sensing of the flexible elongate device 140 and imaging of the anatomical structure A.
  • the processing unit 120 may generate a graphical user interface (GUI) or update GUI data for display on the display unit 130 to aid an operator with the medical procedure.
  • GUI graphical user interface
  • the processing unit 120 may generate data and/or control signals for a control unit of a robotic system configured to manipulate and/or navigate the flexible elongate device 140.
  • the processing unit 120 may be configured to generate, based on the combined imaging and sensing data, one or more alerts.
  • the alerts may include, for example, an alert indicating proximity to the region R, an alert indicating a potential navigation error, and alert indicating that confidence in location of the tip of the flexible elongate device 140 fell below a threshold level, etc.
  • Such intra-operative imaging data may include fluoroscopy X-ray data (e.g., generated by a C-arm X-ray device), and, particularly, tomosynthesis data (reconstructed from 2D X-ray images, such as fluoroscopy images, into a 3D volume),
  • X-rays emanating from the X-ray source 150 may pass through a fiducial plate 160 disposed, for example, under the patient P (or any other suitable location at or near the body of the patient P).
  • the fiducial plate 160 is shown separately to illustrate example fiducial markings disposed, as an example, as a grid throughout the plate 160.
  • the fiducial markings may be high X-ray density spheres and/or cylinders (with regular or varying diameters of 1 , 2, 5, 10 mm or any other suitable size) placed throughout the plate 160 at regular or varying spacing of 10, 20, 30, 40 mm or any other suitable spacing.
  • fiducial markings such as lines squares, or other suitable markings that have high X-ray attenuation or scattering may be included in the plate 160.
  • the system 100 may use the fiducial plate 160 to determine projection angles for 2D images (e.g., fluoroscopy images) detected by the detector 155 of the imaging unit 110.
  • FIG. 1 D defines an example imaging coordinate system.
  • the axis A may be referred to as the x-axis.
  • the orthogonal horizontal axis may be referred to as the y-axis.
  • the vertical axis may be referred to as the z-axis.
  • Planes Sx, Sy, and Sz are, respectively, orthogonal to the x-axis, the y-axis, and the z-axis, and may be referred to, respectively, as axial, sagittal, and coronal planes. It should be noted that the axial plane Sx is parallel to the plane of the arc C and does not correspond to any possible projection angle.
  • the coronal plane Sz corresponds to the projection acquired with the X- ray source 150 at the apex of the curve C.
  • the range of angles with respect to the apex of the curve C may not permit a projection corresponding to the sagittal plane, which would require an angle of ⁇ 90° with respect to vertical.
  • the fluoroscopy acquisition angles may lie, for example, between ⁇ 60 “with respect to vertical, subtending an arc that may be less than 120°.
  • a reconstructed tomosynthesis image may be presented in slices parallel to any one or more of the three planes Sx, Sy, and/or Sz.
  • FIG. 1 E is a simplified three-dimensional representation of a flexible elongate device 140 disposed within the anatomical structure A within the imaging geometry of FIGS. 1C, D. It is included to help visualize projections acquired by the tomosynthesis imaging system illustrated in FIG. 1 C with respect to the coordinate system of FIG. 1 D.
  • the flexible elongate device 140 may have higher X-ray density than the surrounding anatomical structure as may be seen in an example projection in FIG. 1 F.
  • FIG. 1 F depicts a portion of a projection image 180 of a flexible elongate device 141 (which may be device 140) disposed within an anatomical structure (e.g., anatomical structure A) overlayed with fiducial markings (e.g., from the fiducial board 160).
  • the flexible elongate device 141 appears darker than the surrounding tissue, but may be converted, in the negative, to the high-intensity portion of the image.
  • the high X-ray density fiducial marks e.g., mark 161
  • FIGS. 1G, H schematically illustrates projections (i.e., projected images) of the example fiducial plate 160 at two projection angles.
  • the projection angle may then be 0° with respect to the z- axis, i.e., normal to the plate, resulting in a regular grid of fiducial markings within the projected image of the plate 160.
  • the projection angle deviates from the normal, resulting in the columns of fiducial markings being closer that the rows, as well as other distortions.
  • Certain distortions may arise from the projection angle deviating from the normal with respect to the two symmetry axes (represented by dashed lines in FIGS. 1 G, H) of the fiducial plate 160.
  • the projection in FIG. 1 H may additionally have a perspective distortion resulting, for example, from slightly nonparallel X-rays admitted by the X-ray source 150.
  • the system 100 may use the geometric distortions in projections of the fiducial plate 160 to determine the X-ray incidence angle and, therefore, the projection angle and the position of the X-ray source 150 along the arc C.
  • the system 100 may rely on geometric formulas, such as the spacing of fiducial elements in a given direction being proportional to the cosine of the respective angle, to determine the projection angle. Additionally or alternatively, the system 100 may obtain and use calibration data from known projection angles.
  • FIGS. 2A-D schematically illustrate an example procedure for in-painting fiducial markings in a projection image of a flexible elongate device disposed within an anatomical structure.
  • FIG. 2A schematically illustrates a projection image with the anatomical structure A, having an ROI or target R, a flexible elongate device 140, and a grid of fiducial markings.
  • FIG. 2A can be thought of as a schematic example of the projection image 180 in FIG. 1 F.
  • One part of the in-painting process performed by one or more processors of a system may include identifying the fiducial markings in a projection image.
  • the system may use blob detection, spatial frequency analysis, circular Hough transforms, or any other suitable image processing algorithms.
  • the system may use machine learning models to detect fiducial markings.
  • the system may identify fiducial markings for each individual projection or, in some examples, may use correlations within a sequence of projections (e.g., within a fluoroscopy video) to identify the fiducial markings. Additionally or alternatively, the system may use prior information about the markings, such as size, shape, spacing to aid in detection and identification of the fiducial markings.
  • FIG. 2B Another part of the in-painting process performed by the one or more processors of the system includes generating a mask for masking and removing fiducial markings from projection images.
  • An example mask is illustrated in FIG. 2B.
  • the mask may be based on segmented fiducial markings in a projection image.
  • the segmented markings may be dilated to include pixels around the segmented regions, compensating for possible segmentation errors that resulted in missed pixels.
  • detecting fiducial markings may result in identifying the center points of the markings and, subsequently, using prior knowledge about the markings to generate the mask. That is, if the system identifies the projection angle and the center points of the markings in FIG. 2A, the system may generate the mask in FIG. 2B based on prior knowledge about size, shape, and/or spacing of the markings.
  • Another part of the in-painting process performed by the one or more processors of the system includes removing or zeroing out pixels from a projection image based on the generated mask, as illustrated in FIG. 2C.
  • the removal of the pixels leaves the values of the pixels unknown, zero, or any other suitable value, depending on algorithms and data structures used in in-painting of projection images.
  • the resulting gaps of information within a projection image are schematically illustrated in FIG. 2C.
  • the system may fill the gaps using a suitable in-painting technique, resulting in an in painted image schematically illustrated in FIG. 2D.
  • the result of in-painting the fiducial markings may preserve, as much as possible, the information pertaining to the projection of the anatomical structure A including the ROI or target R as well as the projection of the flexible elongate device 140.
  • In-painting techniques may include explicitly-defined image processing algorithms and/or be based on machine learning models (e.g., trained on projection image data).
  • the algorithms may include fluid-dynamics-based algorithms, such as a Navier-Stokes gradientbased method and/or diffusion-based methods. Additionally or alternatively, the in-painting algorithm may be based on a fast marching method.
  • ML-based techniques may include progressive in-painting, attention-based in-painting, and/or pluralistic-based in-painting. Progressive in-painting fills images in a step-wise manner based on surrounding pixels or structures. Attention-based in-painting considers the information from distant spatial locations. Pluralistic in-painting generates multiple results for a single image.
  • Additional information may guide the system’s decision regarding the choice of results generated by pluralistic in-painting.
  • ML model for inpainting may be based on an auto-encoder, a variational water-encoder (VAE), a generative adversarial network (GAE), or a diffusion model.
  • VAE variational water-encoder
  • GAE generative adversarial network
  • the system may be configured to in-paint other aspects and/or elements of projection images, such as spurious images of objects in the operative environment and/or an instrument (e.g., a flexible elongate device) disposed within patient anatomy.
  • an instrument e.g., a flexible elongate device
  • FIGS. 2E-H schematically illustrate an example procedure for in-painting a flexible elongate device in a projection image of an anatomical structure.
  • the system may implement the in-painting process for a flexible elongate device 240 (which may be the flexible elongate device 140) disposed within the anatomical structure A which includes the ROI or target R in a similar sequence to in painting fiducial markings discussed above.
  • FIG. 2E illustrates the original projection image
  • FIG. 2F illustrates a mask 241 corresponding to the flexible elongate device 240 within the projection image. Example processes for generating the mask 241 are discussed in more detail with reference to FIG. 4C.
  • FIG. 2G illustrates the projection image of FIG.
  • FIG. 2H illustrates the projection image of FIG. 2E with the flexible elongate device 240 removed and the gaps G filled in using, for example, the in-painting techniques discussed above.
  • the system may identify within the projection image 301 a reference point 310.
  • the reference point 310 may be the tip of the flexible elongate device 340 or another suitable reference point (e.g., identifiable by a fiducial disposed at the flexible elongate device 340, as described above with reference to FIG. 1 A).
  • the system may identify the reference point 310 automatically.
  • the system may prompt a user to identify the reference point 310 within the displayed projection image 301 , using for example, the GUI generated at the display.
  • the system may identify a reference point 320 within the projection image 302, which may correspond to the same physical point at the flexible elongate device 340 as the reference point 310.
  • the system may identify the point 320 automatically or based on user input, as described above. Furthermore, the system may identify within the projection image 302 additional points 322, 324 disposed at the flexible elongate device 340. Identifying the points 310-324 may include identifying two-dimensional coordinates within their respective projection images (e.g., images 301 , 302). The system may use the identified points 310-320 (e.g., the identified two- dimensional coordinates) to identify a curve corresponding to at least a portion of the flexible elongate device 340 within at least one of the projection images (e.g., image 302).
  • the system may use the identified points 310-320 and/or the identified two- dimensional curve to reconstruct a corresponding portion of the flexible elongate device 340 in three dimensions. Additionally or alternatively, the system may use at least two projection images (e.g., the projection images 301 , 302) to register, using shape data received from a sensing system (e.g., the sensing unit 1 15), an imaging coordinate system with a sensing or instrument coordinate system as described with reference to FIGS. 4A-C.
  • a sensing system e.g., the sensing unit 1 15
  • the rigid body 402 may represent a portion (e.g., a segment of length) of the flexible elongate device (e.g., flexible elongate device 140, 240, or 340). Although the flexible elongate device 140 is flexible, a short portion (e.g., an infinitesimal segment) may be considered rigid for all practical purposes.
  • a state (e.g., position and orientation) of the rigid body 402 in three-dimensional space can be described using the first coordinate system 404 and/or the second coordinate system 406.
  • a rigid body without symmetries e.g., body 402 has six degrees of freedom (6 DOFs) and its position and orientation can be described with six coordinates.
  • the rigid body 402 may have coordinates (x, y, z, 0, ⁇ j>, a), where x, y and z are position coordinates of a center 408 of the rigid body 402 along the axes of the first coordinate system 404 with respect to origin O.
  • the position coordinates may be designated for any point within the rigid body 402, or, in fact, any point in a rigidly defined geometric relationship to the body 402.
  • Coordinates 0, ⁇ j> and a can describe orientation of the body 402 with the aid of an orientation vector 409, which, in the example of FIG. 4A, originates in the center 408 and goes through the middle of one of the facets of the rigid body 402.
  • 0 may be an elevation angle with respect to z- axis
  • > may be an azimuthal angle parallel to the xy-plane
  • a may be the angle of rotation of the rigid body 402 around the orientation vector 409.
  • the three orientation coordinates may be roll, pitch and yaw of the rigid body 402 with respect to any suitable reference direction.
  • ) of first coordinate system 404 coordinates (x’, y’, z’, 0’, ⁇
  • Registering the first coordinate system 404 with the second coordinate system 406, at least in the vicinity of the rigid body 402, includes finding a mapping (e.g., a transformation, a mathematical relationship, etc.) at least between the coordinates (x, y, z, 0, ⁇
  • a mapping e.g., a transformation, a mathematical relationship, etc.
  • the mapping defines a corresponding point (u, v, w) in the vicinity of position (x, y, z) for any point (u’, v’, w’) in the vicinity of position (x’, y’, z’).
  • the mapping from the first coordinate system 404 to the second coordinate system 406 may include one or more scaling factors for the axes.
  • the mapping may include three translation variables, three rotation variables, and/or three scaling variables.
  • each of the variables may depend on position and the mapping may include deformations.
  • each of the coordinate systems 404 and 406 are independently calibrated to have accurate and consistent scaling within a shared operating volume. Coordinate registration process may then be defined in terms of three translation constants and three rotation constants for the shared operating volume. In other examples, gradual variations in scaling within at least one of the coordinate systems 404 and/or 406 may necessitate use of up to three translation variables and up to three rotation variables, each a function of location within the shared operating volume.
  • FIG. 4B schematically illustrates an example coordinate transformation process.
  • a processing unit e.g., the processing unit 120
  • the processing unit may then generate a mapping, M, between the two coordinate systems 404 and 406.
  • the processing unit may be configured to map a new position (u’, v’, w’) within the second coordinate system 406 on to a corresponding position (u, v, w) within the first coordinate system 404. Additionally or alternatively, the processing unit may be configured to map coordinates from the first coordinate system 404 to the second coordinate system 406. The mapping may be only valid in a region around (x, y, z).
  • the processing unit may extend the validity of mapping over any portion of a shared operating volume of the two coordinate systems 404 and 406.
  • the processing unit may implement the coordinate registration as a linear mapping: 0 sin/?' 1 0 O ' u'' 1 0 0 cos a — sin a v' + 0 cos/3. .0 sin a cos a . .w'.
  • a, p and y are rotation parameters (e.g., roll, pitch and yaw)
  • Sn, S22 and S33 are scaling parameters (which may be unity, as discussed above)
  • di, d2 and ds are displacement factors.
  • mapping parameters there may be nine mapping parameters, and six mapping parameters when scaling can be ignored.
  • the linear mapping may be a function of the input position coordinates (u’, v’, w’).
  • the processing unit may store and/or access a lookup table find an entry for mapping parameters corresponding to the input position coordinates. Because the lookup table can only have a limited number of recorded input coordinates (herein, recorded coordinates), the system may use the entry corresponding to the recorded coordinates nearest to the input coordinates. Alternatively, the system may interpolate mapping parameters corresponding to a set of recorded coordinates near the input coordinates.
  • the processing unit may store and/or access a polynomial, spline, or another suitable fit function relating input coordinates to the mapping parameters.
  • the mapping may relate the shape data for the flexible elongate device obtained from a sensing system and the flexible elongate device shape data in the imaging coordinate system as discussed with reference to FIG. 4C.
  • the system may use the mapping to update a target (e.g., target R) position within the instrument coordinate system, as described with reference to FIG. 5A.
  • FIG. 4C depicts an example geometry for flexible elongate device extraction and coordinate registration using shape data and projection images.
  • Two-dimensional projections 410, 420 of a three-dimensional curve 430 representative of at least a portion of a flexible elongate device e.g., flexible elongate device 140, 240, or 340.
  • the projections 410, 420 (which may be referred to as projected curves 410, 420) may be analogous to and/or representative of the two-dimensional projections of the flexible elongate device 340 in projection images 301 , 302.
  • the projections 410, 420 are depicted on orthogonal planes - the xz-plane and the xy-plane, respectively.
  • orthogonal projections may be obtained (e.g., at ⁇ 45°) in the imaging geometries described above, the discussion with reference to FIG. 4C can be generalized to non-orthogonal projections.
  • the three-dimensional image and coordinate system, defined by the x-, y-, and z- axes of FIG. 4C may be defined with respect to the origin located at a reference point disposed at the flexible elongate device.
  • the reference point 310 (which may be a tip of the flexible elongate device 340) in the projection image 301 and the reference point 320 in the projection image 302 may be projections of the same reference point along the flexible elongate device 340 and may define the origin, without any loss of generality, of the three-dimensional imaging coordinate system.
  • the projections 410, 420 and the three-dimensional curve 430 representation of the section of the flexible elongate device all intersect at the origin of the coordinate system.
  • a system may be configured to reconstruct the curve 430 based on two-dimensional projection images which include projected curves 410, 420.
  • the system may first identify within at least one of the projection images (e.g., corresponding to the projection onto the xy-plane) a projected curve (e.g., the projected curve 420).
  • the two-dimensional curve 420 may be represented as a centerline of the projection of the flexible elongate device. Identification of the centerline curve within a projection image may, in some examples be based on at least in part identifying several points (e.g., points 320, 322, and 324) along the curve.
  • the system may identify the points along the curve based on user input via a GUI displaying the respective projection or based on automatic extraction of the two-dimensional curve corresponding to the flexible elongate device from the respective projection image using suitable image processing algorithms.
  • the system may extract a two-dimensional curve (e.g., curve 410) in a second projection image based on the extracted two-dimensional curve (e.g., curve 420) in the first projection image.
  • the curves in the two projection images necessarily share the same coordinate along the axis defined by the intersection of the two projection planes (e.g., the x-axis with respect to the curves 410, 420). Consequently for a given point along one two-dimensional projected curve one can find a corresponding point on the second two- dimensional projected curve. For example, a point on the curve 420 may be projected along a projection ray 462 to the x-axis.
  • the system may find the corresponding point on the curve 410 by tracing along a projection ray 464 to find the intersection with the curve 410.
  • the system may identify the intersection as a high or a low (depending on the format of the projection image) intensity point in the projection in the projection image that intersects the projection ray 464.
  • the corresponding gray 468 intersects the curve 410 at at least two points.
  • the system may resolve the ambiguity by selecting the point that would be contiguous with previous points detected along the curve 410.
  • the corresponding point in three dimensions may be found, e.g., at an intersection 470 of back- projected rays emanating from the two corresponding points in two-dimensional projection images along the respective projection angles.
  • the system may use more than two projections to identify corresponding points in each of the projections and back project the identified points to find the correspondent point in three dimensions. In the presence of measurement uncertainties, the back projected rays may not intersect.
  • the system may then find a point in three dimensions that minimizes a metric (e.g., root mean square or L2-norm) of distances between the three-dimensional point in the back projection rays.
  • the curve 430 in three dimensions may be reconstructed from the curves 410, 420 in two-dimensional projections.
  • the system may reconstruct the curve 430 in three dimensions based on the curve 420 (or curve 410) by first performing a reconstruction of the imaging volume based on the full set of two-dimensional projections.
  • the curve 430 in the resulting three-dimensional reconstructed image may be computed by back-tracing or back-projecting along rays corresponding to the projection angle from points along the curve 420.
  • the system may identify corresponding points along the curve 430 based on suitable intensity thresholds and continuity requirements in a manner similar to identifying corresponding points on the two-dimensional curve 410, as discussed above.
  • the system may use the reconstructed curve 430 to register the imaging coordinate system with a sensing coordinate system.
  • the system may receive shape data from a sensing unit (e.g., sensing unit 115) and identify, within the shape data, sensing system coordinates corresponding to the reference point (e.g., tip of the flexible elongate device) defining the origin of the imaging coordinate system in FIG. 4C.
  • the first step of registration may be defining the translation portion of the transformation (e.g., transformation M described with reference to FIG. 4B), placing the reference point in at the common origin of the coordinate systems registered by the translation.
  • the shape data from the sensing system may then be represented by the dashed-line curve 480, assuming the same scale in the two coordinate systems, as discussed above.
  • the remaining portion of the registration may include finding roll, pitch, and yaw angles that align the shape data curve 480 to the curve 430.
  • an optimization algorithm may minimize a cost function indicative of misalignment or misregistration.
  • the system may compute the registration transformation between the shape data 480 and the curve 430 in the leastsquare sense with or without regularization accounting for prior transformation data and/or other suitable constraints.
  • the system may minimize a weighted least-square difference, valuing the alignment near the origin more heavily than the alignment away from the origin. Valuing the alignment near the origin more heavily may be particularly applicable to examples where the reference point at the origin of the registered coordinate systems in the tip of the flexible elongate device when the tip is in proximity to an anatomical target (e.g., target R).
  • an anatomical target e.g., target R
  • the registration between the imaging and sensing coordinate systems need not follow the extraction of the curve 430 in three-dimensions. Instead, the system may use the shape data 480 to simultaneously extract the curve 430 and register the coordinate systems. To that end, the system may compute coordinate transformations to align projections of the shape data 480 with at least one of the projected curves 410, 420. In this manner, the shape data 480 from the sensing unit may constrain the reconstruction of the curve 430.
  • FIGS. 5A, B schematically illustrate example processes 510, 520, 530, 540, 550, 554, 556, 558, and 560 for flexible elongate device extraction, coordinate registration and target update using shape data and projection images.
  • a system e.g., system 100
  • a processing unit e.g., processing unit 120
  • a display unit e.g., display unit 130
  • the system may obtain (e.g., from imaging unit 110) a set of two-dimensional projection images 502 corresponding to a respective set of projection angles obtained, for example, using the imaging geometry discussed with reference to FIG. 1C.
  • the system may select from the set 502 projection images 504, 506, which may be, for example, projection images 301 , 302.
  • the system may identify a corresponding pair of reference points (e.g., corresponding to a tip or a distal end of a flexible elongate device), such as, for example, points 310 and 320.
  • the system may generate on the display unit a GUI and prompt an operator to select the corresponding points. In other examples, the system may select the corresponding points automatically.
  • the system may execute a triangulation process 510 to compute coordinates in three dimensions of the imaging coordinate system of the reference point.
  • the output of the triangulation process 510 may be an imaging coordinate system 512 with the reference point as the origin, without any loss of generality, as described with reference to FIG. 4C.
  • the system may execute a 2D centerline extraction process 520 to identify within the projection 504 a curve 522 (e.g., curve 420) corresponding to at least a portion of the flexible elongate device including the reference point.
  • the system may generate on the display unit a GUI and prompt an operator to select a plurality of points on the curve 522. Additionally or alternatively, the system may use a one or more image processing algorithms to identify the curve 522.
  • the system may execute a back tracing process to generate, in the imaging system coordinates 512, a three-dimensional curve 532 (e.g., curve 430).
  • the back tracing process 530 may use the set of projection images 502, as described with reference to FIG. 4C.
  • the system may use the generated curve 532 along with shape data 535 obtained from a sensor unit (e.g., sensing unit 115) to compute a transformation 542 between the image coordinate system 512 and an instrument coordinate system, as discussed, for example, with reference to FIG. 4C. Additionally or alternatively, the system may use the registration process 540 in generating the curve 532.
  • a sensor unit e.g., sensing unit 115
  • the system may use the registration process 540 in generating the curve 532.
  • the system may use the curve 532 along with the set of projections 502 as inputs into the reconstruction with in-painting process 550.
  • the process 550 may include projecting the curve 532 back to the set of projection images 502 to generate in-painting masks.
  • the reconstruction with in-painting process 550 may generate a reconstructed three-dimensional image 552 of patient anatomy with a target 553.
  • the three-dimensional image 552 may remove artifacts due to the presence of the flexible elongate device disposed within patient anatomy, providing a clear image of the target 553.
  • the tomosynthesis reconstruction from the set of images 502 may be performed without in-painting. That is the process 550 may be replaced with a direct tomosynthesis reconstruction process, instead of a sequence of sub-processes 554, 556, and 558 discussed with reference to FIG. 5B.
  • the system may use the reconstructed image 552 along with the transformation 542 as inputs for a target update process 560.
  • the target update process 560 may include generating the three-dimensional image 552 and/or corresponding slices at a GUI and prompting an operator to identify the center of the target 563.
  • the system may automatically identify and/or segment the target 563 within the image 552.
  • the center of the target 553 in the image and coordinate system identified based on the image 552 may be mapped using the transformation 542 onto the instrument coordinate system 562.
  • the system may position the target 563 solely based on the transformation 542 and the target 553 from the image 552.
  • the system may use the target update process 560 along with prior information about the target 553 (e.g., from prior update based on intraoperative images and/or pre-operative images) to update coordinates of the target 563 within the instrument coordinate system 562.
  • FIG. 5B schematically illustrates sub-processes 554, 556, and 558 comprising the process 550, as discussed for example with reference to FIGS. 2E-H and 4C.
  • the mask generation process 554 may generate masks 555 based on the curve 532, by projecting a dilated version of the curve 532 onto the set of projections 502.
  • the in-painting process 556 may perform in painting as discussed above with reference to FIGS. 2A-H, outputting a set of projections 557 with the flexible elongate device removed. In some examples, the inpainting process may also remove fiducial markings from the set of projections 557.
  • the system may then reconstruct the in-painted projections 557 using any suitable tomosynthesis process 558 (e.g., using FDK reconstruction or any other suitable reconstruction technique), generating the three-dimensional image 552.
  • any suitable tomosynthesis process 558 e.g., using FDK reconstruction or any other suitable reconstruction technique
  • the system may additionally enhance the image 552 prior to identifying the target 553.
  • the system may use a style transfer process as discussed below with reference to FIGS. 6A, B.
  • FIG. 6A schematically illustrates an example style transfer for three-dimensional images.
  • a scene 600 (which may include, as discussed above, patient anatomy with an anatomical target, a device disposed in the patient anatomy, one or more fiducial to aid, for example, in image analysis and/or reconstruction) may be imaged using a choice of imaging modalities.
  • An example image 601 may be a result of imaging with a first imaging modality (e.g., CT or CBCT) and image 602 may be a result of a second imaging modality (e.g., tomosynthesis of fluoroscopy images).
  • a first imaging modality e.g., CT or CBCT
  • image 602 may be a result of a second imaging modality (e.g., tomosynthesis of fluoroscopy images).
  • the first modality may be a higher-fidelity modality than the second modality and, consequently, the image 601 may be a higher fidelity image than image 602.
  • a system e.g., system 100
  • the model 610 when effectively trained, may be seen as implementing denoising, deconvolution, image restoration and other image-enhancing processes.
  • the model 610 may use or be based on a convolutional neural network (CNN) (e.g., ll-net) and/or a transformer network. Additionally or alternatively, the model 610 may be a generative-adversarial network (GAN) model and/or a diffusion model. The model 610 may be based on a combination of models, generating multiple outputs from a single image input and selecting one of the output based on a choice of an operator or one or more computed metrics of the outputs.
  • CNN convolutional neural network
  • GAN generative-adversarial network
  • the system may render one or more outputs of the model on a display unit (e.g., display unit 130) of the system.
  • a display unit e.g., display unit 130
  • FIG. 6B schematically illustrates an example training process 620 for a styletransfer ML/DL model 630 (e.g., model 610).
  • the training process 620 may, but need not be implemented by the system that uses the model (e.g., system 100).
  • the training process may be performed by another suitable system with appropriate computing resources and the trained model may be transferred to a system for intraoperative imaging.
  • the training process 630 may start with selecting example high-fidelity volumetric data 630, for example from high-fidelity first-modality images (e.g., images such as image 1).
  • the high-fidelity volumetric data 630 may serve as the ground truth for the training process 620.
  • the volumetric data 630 may be, for example, reconstructed CT or CBCT data and may include associated projection data. Either from reconstructed data or from the original projection data associated with the high-fidelity volumetric model 630, the process may generate a simulated second modality sinogram (e.g., a simulated set of fluoroscopy images for tomosynthesis).
  • generating the simulated sinogram may include selecting projections from a limited set of projection angles (e.g., 120°, 100°, 110°, 90°, 80°, 70°, 60°, or any other suitable span).
  • generating the simulated sinogram may include simulating projections from interpolated volumetric data using, for example, and X-ray propagation model. Generating the simulated sinogram may include adding noise and/or distortions to the simulated projections and/or adding noise and/or distortions to the projection angles associated with the projections.
  • the process 620 may generated reconstructed second modality volumetric data 650 (e.g., tomosyntheses reconstruction) based on any suitable reconstruction algorithm (e.g., FDK).
  • the reconstructed data 650 serves as the training input to the model 660, and the model 660 is trained according to any suitable training techniques, to minimize loss with respect to the ground truth of the high-fidelity volumetric data 630.
  • the model 660 thus trained with the suitable volume of training data my then be used as the style-transfer model for enhancing images generated by the lower-fidelity modality (e.g., tomosynthesis) to generate higher-fidelity output that may look like higher-fidelity (e.g., CT or CBCT) reconstructions.
  • the higher-fidelity image output may facilitate, for example, identification of anatomical targets as discussed with reference to FIGS. 5A, B.
  • FIGS. 7-9B depict diagrams of a medical system that may be used for manipulating a medical instrument that includes a flexible elongate device according to any of the methods and systems described above, in some examples.
  • each reference above to the “system” may refer to a system (e.g., system 700) discussed below, or to a subsystem thereof.
  • FIG. 7 is a simplified diagram of a medical system 700 according to some examples.
  • the medical system 700 may include at least portions of the system 100 described with reference to FIG. 1 .
  • the medical system 700 may be suitable for use in, for example, surgical, diagnostic (e.g., biopsy), or therapeutic (e.g., ablation, electroporation, etc.) procedures. While some examples are provided herein with respect to such procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting.
  • the systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, portions of human or animal anatomy, non-surgical diagnosis, as well as for industrial systems, general or special purpose robotic systems, general or special purpose teleoperational systems, or robotic medical systems.
  • medical system 700 may include a manipulator assembly 702 that controls the operation of a medical instrument 704 in performing various procedures on a patient (e.g., patient P on table T, as in FIG. 1 ).
  • the medical instrument 704 may include the flexible elongated device 140 of FIG. 1 .
  • Medical instrument 704 may extend into an internal site within the body of patient P via an opening in the body of patient P.
  • the manipulator assembly 702 may be teleoperated, non-teleoperated, or a hybrid teleoperated and non-teleoperated assembly with one or more degrees of freedom of motion that may be motorized and/or one or more degrees of freedom of motion that may be non-motorized (e.g., manually operated).
  • the manipulator assembly 702 may be mounted to and/or positioned near patient table T.
  • a master assembly 706 allows an operator O (e.g., a surgeon, a clinician, a physician, or other user, as described above) to control the manipulator assembly 702.
  • the master assembly 706 allows the operator O to view the procedural site or other graphical or informational displays.
  • the manipulator assembly 702 may be excluded from the medical system 700 and the medical instrument 704 may be controlled directly by the operator O.
  • the manipulator assembly 702 may be manually controlled by the operator O. Direct operator control may include various handles and operator interfaces for hand-held operation of the medical instrument 704.
  • the master assembly 706 may be located at a surgeon’s console which is in proximity to (e.g., in the same room as) the patient table T on which patient P is located, such as at the side of the patient table T. In some examples, the master assembly 706 is remote from the patient table T, such as in in a different room or a different building from the patient table T.
  • the master assembly 706 may include one or more control devices for controlling the manipulator assembly 702.
  • the control devices may include any number of a variety of input devices, such as joysticks, trackballs, scroll wheels, directional pads, buttons, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, motion or presence sensors, and/or the like.
  • the manipulator assembly 702 supports the medical instrument 704 and may include a kinematic structure of links that provide a set-up structure.
  • the links may include one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place) and/or one or more servo controlled links (e.g., one or more links that may be controlled in response to commands, such as from a control system 712).
  • the manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on the medical instrument 704 in response to commands, such as from the control system 712.
  • the actuators may include drive systems that move the medical instrument 704 in various ways when coupled to the medical instrument 704.
  • one or more actuators may advance medical instrument 704 into a naturally or surgically created anatomic orifice.
  • Actuators may control articulation of the medical instrument 704, such as by moving the distal end (or any other portion) of medical instrument 704 in multiple degrees of freedom.
  • degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes).
  • One or more actuators may control rotation of the medical instrument about a longitudinal axis.
  • Actuators can also be used to move an articulable end effector of medical instrument 704, such as for grasping tissue in the jaws of a biopsy device and/or the like, or may be used to move or otherwise control tools (e.g., imaging tools, ablation tools, biopsy tools, electroporation tools, etc.) that are inserted within the medical instrument 704.
  • the control system 712 may include at least portions of the processing unit 120. Additionally or alternatively, the control system 712 may be in communicative connection with the processing unit 120. In some examples, the output of the processing unit 120 according to the techniques described above may cause the control system 712 to autonomously (without input from the operator O) control certain movements of the medical instrument 704.
  • the medical system 700 may include a sensor system 708 (which may include at least a portion of the sensing unit 115) with one or more sub-systems for receiving information about the manipulator assembly 702 and/or the medical instrument 704.
  • Such sub-systems may include a position sensor system (e.g., that uses electromagnetic (EM) sensors or other types of sensors that detect position or location); a shape sensor system for determining the position, orientation, speed, velocity, pose, and/or shape of a distal end and/or of one or more segments along a flexible body of the medical instrument 704; a visualization system for capturing images, such as from the distal end of medical instrument 704 or from some other location; and/or actuator position sensors such as resolvers, encoders, potentiometers, and the like that describe the rotation and/or orientation of the actuators controlling the medical instrument 704.
  • EM electromagnetic
  • the subsystems may include an imaging sub-system (e.g., using a color imaging device, an infrared imaging device, an ultrasound imaging device, an x-ray imaging device, a fluoroscopic imaging device, a computed tomography (CT) imaging device, a magnetic resonance imaging (MRI) imaging device, or some other type of imaging device), such as the imaging unit 110.
  • an imaging sub-system e.g., using a color imaging device, an infrared imaging device, an ultrasound imaging device, an x-ray imaging device, a fluoroscopic imaging device, a computed tomography (CT) imaging device, a magnetic resonance imaging (MRI) imaging device, or some other type of imaging device
  • CT computed tomography
  • MRI magnetic resonance imaging
  • the positions and orientations of sensors in the sensor system 708 may be determined in the sensor coordinate system.
  • the sensor coordinate system is integrated with or identical to the coordinate system of the manipulator assembly 702.
  • the medical system 700 may include a display system 710 (e.g., display unit 130) for displaying an image or representation of the procedural site and the medical instrument 704.
  • Display system 710 and master assembly 706 may be oriented so physician O can control medical instrument 704 and master assembly 706 with the perception of telepresence.
  • the display system 710 may include at least portions of the display unit 130.
  • the medical instrument 704 may include a visualization system, which may include an image capture assembly that records a concurrent or real-time image of a procedural site and provides the image to the operator O through one or more displays of display system 710.
  • the image capture assembly may include various types of imaging devices.
  • the concurrent image may be, for example, a two-dimensional image or a three- dimensional image captured by an endoscope positioned within the anatomical procedural site.
  • the visualization system may obtain intra-operative images in image system coordinates, distinct from the sensor system coordinates.
  • the visualization system may include endoscopic components that may be integrally or removably coupled to medical instrument 704.
  • a separate endoscope attached to a separate manipulator assembly, may be used with medical instrument 704 to image the procedural site.
  • the visualization system may be implemented as hardware, firmware, software or a combination thereof which interact with or are otherwise executed by one or more computer processors, such as of the control system 712.
  • Display system 710 may also display an image of the procedural site and medical instruments, which may be captured by the visualization system.
  • the medical system 700 provides a perception of telepresence to the operator O.
  • images captured by an imaging device at a distal portion of the medical instrument 704 may be presented by the display system 710 to provide the perception of being at the distal portion of the medical instrument 704 to the operator O.
  • the input to the master assembly 706 provided by the operator O may move the distal portion of the medical instrument 704 in a manner that corresponds with the nature of the input (e.g., distal tip turns right when a trackball is rolled to the right) and results in corresponding change to the perspective of the images captured by the imaging device at the distal portion of the medical instrument 704.
  • the perception of telepresence for the operator O is maintained as the medical instrument 704 is moved using the master assembly 706.
  • the operator O can manipulate the medical instrument 704 and hand controls of the master assembly 706 as if viewing the workspace in substantially true presence, simulating the experience of an operator that is physically manipulating the medical instrument 704 from within the patient anatomy.
  • the display system 710 may present virtual images of a procedural site that are created using image data recorded pre-operatively (e.g., prior to the procedure performed by the medical instrument system 200) or intra-operatively (e.g., concurrent with the procedure performed by the medical instrument system 200), such as image data created using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and/or the like.
  • CT computed tomography
  • MRI magnetic resonance imaging
  • PET positron emission tomography
  • fluoroscopy thermography
  • ultrasound ultrasound
  • OCT optical coherence tomography
  • thermal imaging impedance imaging
  • laser imaging laser imaging
  • nanotube X-ray imaging and/or the like.
  • the virtual images may include two-dimensional, three-dimensional, or higher-dimensional (e.g., including, for example, time based
  • display system 710 may display a virtual image that is generated based on tracking the location of medical instrument 704.
  • the tracked location of the medical instrument 704 may be registered (e.g., dynamically referenced) with the model generated using the preoperative or intra-operative images, with different portions of the model correspond with different locations of the patient anatomy.
  • the registration is used to determine portions of the model corresponding with the location and/or perspective of the medical instrument 704 and virtual images are generated using the determined portions of the model. This may be done to present the operator O with virtual images of the internal procedural site from viewpoints of medical instrument 704 that correspond with the tracked locations of the medical instrument 704.
  • the display system 710 may include the display unit 130 and may display images including the position, orientation, and/or pose of the medical instrument 704 in a joint coordinate system based on registering the sensor coordinate system with the imaging coordinate system according to the techniques described above with reference to FIGS. 2A- 6.
  • the medical system 700 may also include the control system 712, which may include processing circuitry (e.g., the processing unit 120) that implements the some or all of the methods or functionality discussed herein.
  • the control system 712 may include at least one memory and at least one processor for controlling the operations of the manipulator assembly 702, the medical instrument 704, the master assembly 706, the sensor system 708, and/or the display system 710.
  • Control system 712 may include instructions (e.g., a non-transitory machine-readable medium storing the instructions) that when executed by the at least one processor, configures the one or more processors to implement some or all of the methods or functionality discussed herein. While the control system 712 is shown as a single block in FIG.
  • control system 712 may include two or more separate data processing circuits with one portion of the processing being performed at the manipulator assembly 702, another portion of the processing being performed at the master assembly 706, and/or the like.
  • control system 712 may include other types of processing circuitry, such as application-specific integrated circuits (ASICs) and/or field- programmable gate array (FPGAs).
  • ASICs application-specific integrated circuits
  • FPGAs field- programmable gate array
  • the control system 712 may be implemented using hardware, firmware, software, or a combination thereof.
  • control system 712 may receive feedback from the medical instrument 704, such as force and/or torque feedback. Responsive to the feedback, the control system 712 may transmit signals to the master assembly 706. In some examples, the control system 712 may transmit signals instructing one or more actuators of the manipulator assembly 702 to move the medical instrument 704. In some examples, the control system 712 may transmit informational displays regarding the feedback to the display system 710 for presentation or perform other types of actions based on the feedback.
  • the virtual visualization system may obtain sensor data from the sensor system 708 that is used to compute an (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P.
  • the sensor system 708 may be used to register and display the medical instrument 704 together with the pre-operatively or intra-operatively recorded images.
  • PCT Publication WO 2016/191298 published December 1 , 2016 and titled “Systems and Methods of Registration for Image Guided Surgery”
  • the registration may be based on the techniques discussed above with reference to FIGS. 2A-6.
  • the sensor system 708 may be used to compute the (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P.
  • the location can be used to produce both macro-level (e.g., external) tracking images of the anatomy of patient P and virtual internal images of the anatomy of patient P.
  • the system may include one or more electromagnetic (EM) sensors, fiber optic sensors, and/or other sensors to register and display a medical instrument together with pre- operatively recorded medical images.
  • EM electromagnetic
  • Medical system 700 may further include operations and support systems (not shown) such as illumination systems, steering control systems, irrigation systems, and/or suction systems.
  • the medical system 700 may include more than one manipulator assembly and/or more than one master assembly.
  • the exact number of manipulator assemblies may depend on the medical procedure and space constraints within the procedural room, among other factors. Multiple master assemblies may be co-located or they may be positioned in separate locations. Multiple master assemblies may allow more than one operator to control one or more manipulator assemblies in various combinations.
  • FIG. 8A is a simplified diagram of a medical instrument system °0 according to some examples.
  • the medical instrument system 800 includes a flexible elongate device 802 (e.g., device 140), also referred to as elongate device 802, a drive unit 804, and a medical tool 826 that collectively is an example of a medical instrument 704 of a medical system 700.
  • the medical system 700 may be a teleoperated system, a non-teleoperated system, or a hybrid teleoperated and non-teleoperated system, as explained with reference to FIG. 7.
  • a visualization system 831 , tracking system 830, and navigation system 832 are also shown in FIG. 8A and are example components of the control system 712 of the medical system 700.
  • the medical instrument system 800 may be used for non-teleoperational exploratory procedures or in procedures involving traditional manually operated medical instruments, such as endoscopy.
  • the medical instrument system 800 may be used to gather (e.g., measure) a set of data points corresponding to locations within anatomic passageways of a patient, such as patient P.
  • the elongate device 802 is coupled to the drive unit 804.
  • the elongate device 802 includes a channel 821 through which the medical tool 826 may be inserted.
  • the elongate device 802 navigates within patient anatomy to deliver the medical tool 826 to a procedural site.
  • the elongate device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818.
  • the flexible body 816 may have an approximately 3 mm outer diameter. Other flexible body outer diameters may be larger or smaller.
  • Medical instrument system 800 may include the tracking system 830 for determining the position, orientation, speed, velocity, pose, and/or shape of the flexible body 816 at the distal end 818 and/or of one or more segments 824 along flexible body 816, as will be described in further detail below.
  • the tracking system 830 may include one or more sensors and/or imaging devices.
  • the flexible body 816 such as the length between the distal end 818 and the proximal end 817, may include multiple segments 824.
  • the tracking system 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, the tracking system 830 is part of control system 712 shown in FIG. 7.
  • the tracking system 830 may implement at least some of the techniques described with reference to FIGS. 1 A-6, and, to that end, may include at least portions of or be in communicative connection with the processing unit 120 of FIG. 1A.
  • Tracking system 830 may track the distal end 818 and/or one or more of the segments 824 of the flexible body 816 using a shape sensor 822.
  • the shape sensor 822 may be omitted.
  • the shape sensor 822 may include an optical fiber aligned with the flexible body 816 (e.g., provided within an interior channel of the flexibly body 816 or mounted externally along the flexible body 816).
  • the optical fiber may have a diameter of approximately 800 pm. In other examples, the diameter may be larger or smaller.
  • the optical fiber of the shape sensor 822 may form a fiber optic bend sensor for determining the shape of flexible body 816.
  • Optical fibers including Fiber Bragg Gratings may be used to provide strain measurements in structures in one or more dimensions.
  • FBGs Fiber Bragg Gratings
  • Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions are described in U.S. Patent Application Publication No. 2006/0013523 (filed July 13, 2005 and titled “Fiber optic position and shape sensing device and method relating thereto”); U.S. Patent No. 7,772,541 (filed on March 12, 2008 and titled “Fiber Optic Position and/or Shape Sensing Based on Rayleigh Scatter”); and U.S. Patent No. 8,773,650 (filed on Sept.
  • Sensors in some examples may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and Fluorescence scattering.
  • the shape of the flexible body 816 may be determined using other techniques. For example, a history of the position and/or pose of the distal end 818 of the flexible body 816 can be used to reconstruct the shape of flexible body 816 over an interval of time (e.g., as the flexible body 816 is advanced or retracted within a patient anatomy).
  • the tracking system 830 may alternatively and/or additionally track the distal end 818 of the flexible body 816 using a position sensor system 820.
  • Position sensor system 820 may be a component of an EM sensor system with the position sensor system 820 including one or more position sensors. Although the position sensor system 820 is shown as being near the distal end 818 of the flexible body 816 to track the distal end 818, the number and location of the position sensors of the position sensor system 820 may vary to track different regions along the flexible body 816.
  • the position sensors include conductive coils that may be subjected to an externally generated electromagnetic field. Each coil of position sensor system 820 may produce an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field.
  • the position sensor system 820 may measure one or more position coordinates and/or one or more orientation angles associated with one or more portions of flexible body 816.
  • the position sensor system 820 may be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, Z and three orientation angles indicating pitch, yaw, and roll of a base point. In some examples, the position sensor system 820 may be configured and positioned to measure five degrees of freedom, e.g., three position coordinates X, Y, Z and two orientation angles indicating pitch and yaw of a base point. Further description of a position sensor system, which may be applicable in some examples, is provided in U.S. Patent No. 6,380,732 (filed August 11 , 1999 and titled “Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked”), which is incorporated by reference herein in its entirety.
  • a processing unit may enhance the accuracy of positions obtained by the position sensor system 820 by combing data obtained by the position sensor system 820 with data obtained by an external imaging system (e.g., by way of the imaging unit 110) according to the techniques of this disclosure described above with reference to FIGS. 2A-6.
  • the tracking system 830 may alternately and/or additionally rely on a collection of pose, position, and/or orientation data stored for a point of an elongate device 802 and/or medical tool 826 captured during one or more cycles of alternating motion, such as breathing. This stored data may be used to develop shape information about the flexible body 816.
  • a series of position sensors such as EM sensors like the sensors in position sensor system 820 or some other type of position sensors may be positioned along the flexible body 816 and used for shape sensing.
  • a history of data from one or more of these position sensors taken during a procedure may be used to represent the shape of elongate device 802, particularly if an anatomic passageway is generally static.
  • FIG. 8B is a simplified diagram of the medical tool 826 within the elongate device 802 according to some examples.
  • the flexible body 816 of the elongate device 802 may include the channel 821 sized and shaped to receive the medical tool 826.
  • the medical tool 826 may be used for procedures such as imaging, surgery, biopsy, ablation, illumination, irrigation, suction, electroporation, etc.
  • Medical tool 826 can be deployed through channel 821 of flexible body 816 and operated at a procedural site within the anatomy.
  • Medical tool 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, grasper, brush, etc.), an ablation tool (e.g., a laser ablation tool, radio frequency (RF) ablation tool, cryoablation tool, thermal ablation tool, heated liquid ablation tool, etc.), an electroporation tool, and/or another surgical, diagnostic, or therapeutic tool.
  • the medical tool 826 may include an end effector having a single working member such as a scalpel, a blunt blade, an optical fiber, an electrode, and/or the like.
  • Other end types of end effectors may include, for example, forceps, graspers, scissors, staplers, clip appliers, and/or the like.
  • Other end effectors may further include electrically activated end effectors such as electrosurgical electrodes, transducers, sensors, and/or the like.
  • the medical tool 826 may be a biopsy tool used to remove sample tissue or a sampling of cells from a target anatomic location.
  • the biopsy tool is a flexible needle.
  • the biopsy tool may further include a sheath that can surround the flexible needle to protect the needle and interior surface of the channel 821 when the biopsy tool is within the channel 821 .
  • the medical tool 826 may be an image capture probe that includes a distal portion with a stereoscopic or monoscopic camera that may be placed at or near the distal end 818 of flexible body 816 for capturing images (e.g., still or video images).
  • the captured images may be processed by the visualization system 831 for display and/or provided to the tracking system 830 to support tracking of the distal end 818 of the flexible body 816 and/or one or more of the segments 824 of the flexible body 816.
  • the image capture probe may include a cable for transmitting the captured image data that is coupled to an imaging device at the distal portion of the image capture probe.
  • the image capture probe may include a fiber-optic bundle, such as a fiberscope, that couples to a more proximal imaging device of the visualization system 831 .
  • the image capture probe may be single-spectral or multi-spectral, for example, capturing image data in one or more of the visible, near-infrared, infrared, and/or ultraviolet spectrums.
  • the image capture probe may also include one or more light emitters that provide illumination to facilitate image capture.
  • the image capture probe may use ultrasound, x-ray, fluoroscopy, CT, MRI, or other types of imaging technology.
  • the image capture probe is inserted within the flexible body 816 of the elongate device 802 to facilitate visual navigation of the elongate device 802 to a procedural site and then is replaced within the flexible body 816 with another type of medical tool 826 that performs the procedure.
  • the image capture probe may be within the flexible body 816 of the elongate device 802 along with another type of medical tool 826 to facilitate simultaneous image capture and tissue intervention, such as within the same channel 821 or in separate channels.
  • a medical tool 826 may be advanced from the opening of the channel 821 to perform the procedure (or some other functionality) and then retracted back into the channel 821 when the procedure is complete.
  • the medical tool 826 may be removed from the proximal end 817 of the flexible body 816 or from another optional instrument port (not shown) along flexible body 816.
  • the elongate device 802 may include integrated imaging capability rather than utilize a removable image capture probe.
  • the imaging device (or fiber-optic bundle) and the light emitters may be located at the distal end 818 of the elongate device 802.
  • the flexible body 815 may include one or more dedicated channels that carry the cable(s) and/or optical fiber(s) between the distal end 818 and the visualization system 831 .
  • the medical instrument system 800 can perform simultaneous imaging and tool operations.
  • the medical tool 826 is capable of controllable articulation.
  • the medical tool 826 may house cables (which may also be referred to as pull wires), linkages, or other actuation controls (not shown) that extend between its proximal and distal ends to controllably bend the distal end of medical tool 826, such as discussed herein for the flexible elongate device 802.
  • the medical tool 826 may be coupled to a drive unit 804 and the manipulator assembly 702.
  • the elongate device 802 may be excluded from the medical instrument system 800 or may be a flexible device that does not have controllable articulation. Steerable instruments or tools, applicable in some examples, are further described in detail in U.S. Patent No.
  • the flexible body 816 of the elongate device 802 may also or alternatively house cables, linkages, or other steering controls (not shown) that extend between the drive unit 804 and the distal end 818 to controllably bend the distal end 818 as shown, for example, by broken dashed line depictions 819 of the distal end 818 in FIG. 2A.
  • at least four cables are used to provide independent up-down steering to control a pitch of the distal end 818 and left-right steering to control a yaw of the distal end 881 .
  • the flexible elongate device 802 may be a steerable catheter.
  • the drive unit 804 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of the teleoperational assembly.
  • the elongate device 802 and/or medical tool 826 may include gripping features, manual actuators, or other components for manually controlling the motion of the elongate device 802 and/or medical tool 826.
  • the elongate device 802 may be steerable or, alternatively, the elongate device 802 may be non-steerable with no integrated mechanism for operator control of the bending of distal end 818.
  • one or more channels 821 (which may also be referred to as lumens), through which medical tools 826 can be deployed and used at a target anatomical location, may be defined by the interior walls of the flexible body 816 of the elongate device 802.
  • the medical instrument system 800 may include a flexible bronchial instrument, such as a bronchoscope or bronchial catheter, e.g., for use in visual examination and diagnosis, biopsy, and/or treatment of a lung.
  • a flexible bronchial instrument such as a bronchoscope or bronchial catheter, e.g., for use in visual examination and diagnosis, biopsy, and/or treatment of a lung.
  • the medical instrument system 800 may also be suited for navigation and treatment of other tissues, via natural or surgically created connected passageways, in any of a variety of anatomic systems, including the colon, the intestines, the kidneys and kidney calices, the brain, the heart, the circulatory system including vasculature, and/or the like.
  • the information from the tracking system 830 may be sent to the navigation system 832, where the information may be combined with information from the visualization system 831 and/or pre-operatively obtained models to provide the physician, clinician, surgeon, or other operator with real-time position information.
  • the tracking system 830, the navigation system 832, and the visualization system 831 may cooperatively implement, at least partially, the functionality of the system 100 in implementing the techniques described with reference to FIGS. 1 A-6.
  • the real-time position information may be displayed on the display system 710 for use in the control of the medical instrument system 800.
  • the navigation system 832 may utilize the position information as feedback for positioning medical instrument system 800.
  • FIGS. 9A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly according to some examples.
  • a surgical environment 900 may include the patient P positioned on the patient table T.
  • Patient P may be stationary within the surgical environment 900 in the sense that gross patient movement is limited by sedation, restraint, and/or other means. Cyclic anatomic motion, including respiration and cardiac motion, of patient P may continue.
  • a medical instrument 904 is used to perform a medical procedure which may include, for example, surgery, biopsy, ablation, illumination, irrigation, suction, or electroporation.
  • the medical instrument 904 may also be used to perform other types of procedures, such as a registration procedure to associate the position, orientation, and/or pose data captured by the sensor system 708 to a desired (e.g., anatomical or system) reference frame.
  • the medical instrument 904 may be, for example, the medical instrument 704.
  • the medical instrument 904 may include an elongate device 910 (e.g., a catheter) coupled to an instrument body 912.
  • Elongate device 910 may be the elongate device 140 of FIG. 1.
  • Elongate device 910 includes one or more channels sized and shaped to receive a medical tool.
  • Elongate device 910 may also include one or more sensors (e.g., components of the sensor system 708).
  • a shape sensor 914 may be fixed at a proximal point 916 on the instrument body 912.
  • the proximal point 916 of the shape sensor 914 may be movable with the instrument body 912, and the location of the proximal point 916 with respect to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device).
  • the shape sensor 914 may measure a shape from the proximal point 916 to another point, such as a distal end 918 of the elongate device 910.
  • the shape sensor 914 may be aligned with the elongate device 910 (e.g., provided within an interior channel or mounted externally).
  • the shape sensor 914 may optical fibers used to generate shape information for the elongate device 910.
  • position sensors e.g., EM sensors
  • a series of position sensors may be positioned along the flexible elongate device 910 and used for shape sensing.
  • Position sensors may be used alternatively to the shape sensor 914 or with the shape sensor 914, such as to improve the accuracy of shape sensing or to verify shape information.
  • Elongate device 910 may house cables, linkages, or other steering controls that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918.
  • at least four cables are used to provide independent up- down steering to control a pitch of distal end 918 and left-right steering to control a yaw of distal end 918.
  • the instrument body 912 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of a manipulator assembly.
  • the instrument body 912 may be coupled to an instrument carriage 906.
  • the instrument carriage 906 may be mounted to an insertion stage 908 that is fixed within the surgical environment 900.
  • the insertion stage 908 may be movable but have a known location (e.g., via a tracking sensor or other tracking device) within surgical environment 900.
  • Instrument carriage 906 may be a component of a manipulator assembly (e.g., manipulator assembly 702) that couples to the medical instrument 904 to control insertion motion (e.g., motion along an insertion axis A) and/or motion of the distal end 918 of the elongate device 910 in multiple directions, such as yaw, pitch, and/or roll.
  • the instrument carriage 906 or insertion stage 908 may include actuators, such as servomotors, that control motion of instrument carriage 906 along the insertion stage 908.
  • a sensor device 920 which may be a component of the sensor system 708, may provide information about the position of the instrument body 912 as it moves relative to the insertion stage 908 along the insertion axis A.
  • the sensor device 920 may include one or more resolvers, encoders, potentiometers, and/or other sensors that measure the rotation and/or orientation of the actuators controlling the motion of the instrument carriage 906, thus indicating the motion of the instrument body 912.
  • the insertion stage 908 has a linear track as shown in FIGS. 9A and 9B.
  • the insertion stage 908 may have curved track or have a combination of curved and linear track sections.
  • FIG. 9A shows the instrument body 912 and the instrument carriage 906 in a retracted position along the insertion stage 908.
  • the proximal point 916 is at a position L0 on the insertion axis A.
  • the location of the proximal point 916 may be set to a zero value and/or other reference value to provide a base reference (e.g., corresponding to the origin of a desired reference frame) to describe the position of the instrument carriage 906 along the insertion stage 908.
  • the distal end 918 of the elongate device 910 may be positioned just inside an entry orifice of patient P.
  • the instrument body 912 and the instrument carriage 906 have advanced along the linear track of insertion stage 908, and the distal end 918 of the elongate device 910 has advanced into patient P.
  • the proximal point 916 is at a position L1 on the insertion axis A.
  • the rotation and/or orientation of the actuators measured by the sensor device 920 indicating movement of the instrument carriage 906 along the insertion stage 908 and/or one or more position sensors associated with instrument carriage 906 and/or the insertion stage 908 may be used to determine the position L1 of the proximal point 916 relative to the position L0.
  • the position L1 may further be used as an indicator of the distance or insertion depth to which the distal end 918 of the elongate device 910 is inserted into the passageway(s) of the anatomy of patient P.
  • FIG. 10 depicts a flow chart for an example method 1000 for visualizing patient anatomy during a medical procedure.
  • the method 1000 may be implemented by a system (e.g., system 100) which may include one or more processors (e.g., disposed at the processing unit 120) and a display device (e.g., the display unit 130). Furthermore, instructions for executing the method 1000 on the one or more processors may be stored on a tangible, non-transitory, computer readable medium.
  • the method 1000 includes obtaining a first plurality of two- dimensional images depicting an anatomical structure and one or more objects.
  • An example system may obtain the images from an imaging unit, such as, for example a fluoroscopy imaging unit.
  • the two-dimensional images may correspond to different projection angles, as discussed throughout the disclosure.
  • an example imaging setup may include a fiducial plate.
  • the method 1000 includes computing, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by inpainting at least a portion of pixels associated with at least one of the one or more objects.
  • the objects may include fiducial elements manifesting as fiducial markings in the set of projection images. Additionally or alternatively, the objects may include an instrument (e.g., a flexible elongate device disposed within patient anatomy. Pixels associated with the non- anatomical objects within the projections may cause artifacts when the two-dimensional projection images are used to reconstruct a three-dimension volumetric image. Thus, inpainting, as discussed in detail above with reference to images 2A-H, 4C and 5B may improve reconstructed image quality.
  • the system may in-paint fiducial markings to better identify the pixels associated with a flexible elongate device and, subsequently in-paint the pixels associated with at least a portion of the flexible elongate device. A reconstruction of three-dimensional anatomical structure after in-painting may facilitate identifying an anatomical target for the medical procedure, improving speed and accuracy of the procedure.
  • the method 1000 includes causing, a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.
  • the visualization may be for the purpose of evaluating a stage of the medical procedure and/or aiding with navigation.
  • the visualization may be a part of an interactive display, prompting an operator for an input, such as, for example, identifying an anatomical target.
  • the visualization may be a part of a GUI for identifying pixels associated with the flexible elongate device within the projection images, as discussed, for example with reference to FIGS. 3B, C and 4C.
  • the method 1000 includes computing, based at least in part on the second plurality of two-dimensional images, a first three-dimensional image of at least a portion of the anatomical structure.
  • the three-dimensional image may be based on reconstruction from projections (e.g., image 552 in FIG. 5A).
  • the method 1000 includes computing a second three- dimensional image based on the first three-dimensional image using a style-transfer machine learning model, and wherein the visualization is based at least in part on the computed second three-dimensional image.
  • the first reconstructed image may have low fidelity due to, for example, a limited set of projection angles.
  • Style transfer models taking the first three-dimensional image as an input and computing the second three-dimensional image as an output may significantly enhance image quality (e.g., as in FIGS. 14A, B below). Training a style transfer model is discussed above, particularly with reference to FIG. 6B, as well as below, with reference to FIG. 12.
  • FIGS. 11 A, B depict a flow chart for an example method 1100 for visualizing patient anatomy during a medical procedure.
  • the method 1100 may be implemented by a system (e.g., system 100) which may include one or more processors (e.g., disposed at the processing unit 120) and a display device (e.g., the display unit 130). Furthermore, instructions for executing the method 1100 on the one or more processors may be stored on a tangible, non-transitory, computer readable medium.
  • the method 1100 includes obtaining a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device.
  • the method 1100 includes identifying, within each of at least two of the plurality of two-dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device.
  • identifying the two-dimensional coordinated within projection images may include obtaining, by the one or more processors, input from an operator via a GUI.
  • image processing algorithms and/or ML models may aid in autonomously identifying, by a processing unit, the reference point, or, at least, candidate reference points within projection images.
  • the operator and/or the system may identify the reference point as the distal end (e.g., tip) of the flexible elongate device, and/or by way of a fiducial disposed at the flexible elongate device to indicate the reference point.
  • the method 1100 includes computing, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point.
  • the system may compute the three-dimensional coordinates by back-tracing rays from the two projections of the reference point, along the respective projection angles, to find an intersection of or a closest approach point of the back-traced rays.
  • the system may use more than two projection images with identified projected reference point coordinates to compute the coordinates of the reference point in three dimensions to make the computation more robust to noise and/or errors in identifying reference point projections. For example, an erroneously identified projected reference point may lead to a back-traced ray that passes far from the closest approach point of other back-traced rays.
  • the method 1100 includes identifying, within at least one of the at least two of the plurality of two-dimensional projection images, a curve (e.g., a centerline) corresponding to at least a portion of the flexible elongate device including the reference point. Identifying the curve may include obtaining, by the one or more processors, input from an operator via a GUI. For example, as discussed above (e.g., with reference to FIG. 3C) the GUI may render a projection image depicting a projection of the flexible elongate device, wherein the user may select several points corresponding to the device.
  • the one or more processors may segment the curve corresponding to the device using suitable image processing algorithms and user input. In other examples, the system may autonomously identify the curve.
  • the method 1100 includes receiving, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point.
  • the shape data may be generated by one or more sensors as discussed with reference to FIG. 1 A.
  • the method 1100 includes registering the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images. The registration process is discussed in detail with reference to FIGS. 4A-C.
  • the registration process may include first computing in three dimensions, based on projection images, the curve corresponding to the flexible elongate device and subsequently aligning the computed curve with the received shape data.
  • the shape data may serve as the input for the registration process, as discussed with reference to FIGS. 4C and 5A.
  • the method 1100 includes reconstructing, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure.
  • the reconstruction may be, for example, tomosynthesis based on a set of fluoroscopy images.
  • the reconstruction may include or be preceded by an in-painting step. Additionally or alternatively, the reconstruction may be enhance by using a style transfer model.
  • the method 1100 includes identifying, within the reconstructed three-dimensional image of the anatomical structure, a target.
  • identifying the target may include obtaining, by the one or more processors, input from an operator via a GUI, as discussed above, for example, with reference to FIG. 5A.
  • the system may autonomously identify the target within the reconstructed image using suitable image processing algorithms and/or an ML model. Additionally or alternatively, the system may use prior information (e.g., from pre-operative imaging) to identify the target within the reconstructed inter-operative image.
  • the method 1100 includes computing, in the instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target.
  • the location of the target may be computed based on user input identifying target center within the reconstructed image, based on identifying a center of a region (e.g., a collection of voxels) associated with the target, or by other suitable means.
  • computing the target location may be based on applying a registration transformation to the target location in the reconstructed image to compute the target location in instrument coordinates. Additionally or alternatively, computing the target location may include updating the target location from the previously identified target location in the instrument coordinate system.
  • the method 1100 includes causing the display device to display a graphical user interface depicting the computed location of the target in the instrument coordinate system. Depicting the target in the instrument coordinate system may aid the operator in conducting the medical procedure. Consequently, the techniques of the present disclosure to compute accurately and in a timely manner target location (which may move, for example, under the anatomical deformations caused by pressure from the operating instrument, breathing, and/or other physiological processes) may improve procedure speed and accuracy.
  • FIG. 12A depicts a flow chart for an example method 1200 for training a machine learning and/or deep learning model for style transfer as discussed, for example, with reference to FIGS. 6A, B.
  • the method may be implemented on one or more processors. At least some of the processors may be included in the processing unit 120. Additionally or alternatively, the one or more processors may be distributed (e.g., locally and/or in the cloud). At least one of the one or more processors may be a GPU.
  • the method 1200 includes computing, based on a three-dimensional ground-truth image, a set of two-dimensional projection images corresponding to a span of projection angles.
  • the span of projection angles may be no more than 120 degrees.
  • the method may include computing a set of two-dimensional images with a span of greater than 120 degrees.
  • tomosynthesis based on fluoroscopy commonly uses less than 120 degrees, larger spans are possible.
  • the method 1200 is applicable to enhancing images of other modalities.
  • the method 1200 may enhance a tomographic image obtained from sparsely-spaced angles covering a full range of 180 degrees.
  • the three-dimensional ground-truth images may be, for example, CT and/or CBCT images reconstructed, respectively, from CT and CBCT data.
  • Other possible ground truth images may include MRI, 3D ultrasound, or any other suitable 3D modality or combination of modalities. Additionally or alternatively, ground truth images may be at least in part computer-generated. Sets of two-dimensional projection images may be computed, for example, as projections of the ground-truth 3D images using suitable projection physics models.
  • a system for implementing the method 1200 may compute multiple sets of two- dimensional projection images based on a single 3D ground truth image. For example, one set may be computed for a set of projection angles spanning 120 degrees. Another set may be computed for a set of projection angles spanning 90 degrees. In any case, the angles may be distributed in any suitable way with any suitable angle spacing (e.g., 10°, 5°, 3°, 2°, 1 °, etc.).
  • a system may compute one set of projection images and use the one set to generate one or more additional sets. For example, the system may generate an additional set (or multiple) by adding noise and/or artifacts to the projection images.
  • an additional set may be generated from the first set by removing images associated with some of the projection images, e.g., on either side of the span of angles, every second angle, etc.
  • the additional set may have no more than half of the projections. That is, sparse projection sets may be generated from a more dense one.
  • the method 1200 includes computing, based on the set of two- dimensional projection images, a three-dimensional tomosynthesis training image.
  • a system may generate the tomosynthesis image using back-projection, filtered back-projection, iterative reconstruction, and/or any other suitable reconstruction method.
  • a training image need not be a tomosynthesis image. It may be an MRI image (e.g., degraded by adding noise or k-space errors with respect to the ground truth image), or any other suitable modality image for generating a style-transfer model.
  • a tomosynthesis image may be generated from a set of projection images based on a respective set angles.
  • a training system may add noise to at least some projection angles within the span of projection angles. That is, the projection images may be computed using one set of angles, but reconstructed using a slightly different set of angles, with systemic and/or random shifts with respect to the correct set on angles. In this manner, a training system may generate a model that is robust with respect to angle uncertainties.
  • the method 1200 includes using the three-dimensional tomosynthesis training image as an input into a style-transfer machine learning model to generate a three-dimensional output image.
  • the style-transfer machine learning model may include a CNN, a recursive neural network, a transformer network, a GAN, a diffusion model, or any other suitable model architecture and/or combination of models.
  • the style transfer model may generate at least one output for each of the input training images.
  • the model may be a combination of component models, and each of the component models may generate a separate output. The separate outputs may then be combined using a deterministic algorithm, for example.
  • the method 1200 includes adjusting the style-transfer machine learning model to reduce an indication of loss between the three-dimensional output image and the three-dimensional ground-truth image.
  • the training system may adjust model parameters using, for example, a gradient descent method. Additionally or alternatively, the system may adjust model architecture and/or other suitable hyperparameters.
  • the method 1200 may be used to train a stye-transfer ML/DL model that may, as a part of method 1000 or 1100, enhance visualization of patient anatomy and, thereby, aid in visualization of an anatomical target, as described below with reference to FIG. 12B.
  • FIG. 12B depicts a flow chart for an example method 1250 for visualizing patient anatomy using style transfer.
  • the method 1250 may be implemented using the system 100 or another suitable system.
  • the method 1250 may enhance an image reconstruction after in-painting and reconstruction techniques described throughout this disclosure.
  • the method 1250 may enhance reconstructed images independently of whether in-painting techniques are used.
  • the method 1250 includes obtaining a plurality of two-dimensional images associated with a first range of viewing angles, the two-dimensional images depicting an anatomical structure.
  • the two-dimensional images may be fluoroscopy images obtained over a span of angles of 60°, 90°, 120° or any other suitable range and with any suitable regular or uneven angle spacing (e.g., 10°, 5°, 3°, 2°, 1 °, etc.).
  • the method 1250 includes computing (e.g., by tomosynthesis), based at least in part on the plurality of two-dimensional images, a first three-dimensional image of at least a portion of the anatomical structure.
  • the computation may include inpainting of fiducial markings and/or other objects disposed within the anatomy, as discussed above.
  • the method 1250 includes computing a second three-dimensional image, based on at least a portion of the first three-dimensional image and using one or more style-transfer models.
  • the style transfer models may be any one of the style transfer models described above (e.g., with reference to FIG. 12A).
  • the one or more models may be trained as described above with reference to FIG. 12A.
  • source data for the ground-truth used in training the one or more models may be CT and/or CBCT data and may be associated with a range of viewing angles larger than the first range of viewing angles.
  • the one or more models may include models trained for inputs (e.g., training images) with different ranges of viewing angles, different angle sparsity, different noise levels, etc.
  • the method 1250 includes causing a display device to display a GUI depicting the second three-dimensional image.
  • the method 1250 may include causing a display device to display a GUI depicting style-transfer model options.
  • the method may further include obtaining user input with respect to option choice.
  • the GUI may present a choice and collect input for choosing among models trained on input images with different angle ranges, different angle sparsity, etc.
  • style-transfer model options may include target style for the second three- dimensional image.
  • the target style may be CT, CBCT, MRI, etc. That is, the output image may resemble an image obtained with the respective modality.
  • the style transfer may improve visualization of patient anatomy by improving the appearance of artifacts associated with tomosynthesis of limited angle range images.
  • FIGS. 13A, B depict, respectively, outputs of tomosynthesis imaging and CBCT imaging of patient anatomy.
  • panel 1310 depicts an axial slice
  • panel 1320 depicts a sagittal slice
  • panel 1330 depicts a coronal slice through the reconstructed tomosynthesis image.
  • panel 1340 depicts an axial slice
  • panel 1350 depicts a sagittal slice
  • panel 1360 depicts a coronal slice through the reconstructed CBCT image.
  • the axial and sagittal slices of the tomosynthesis image exhibit significant imaging artifacts (e.g., smearing) and have considerably lower fidelity than the axial and sagittal slices of the CBCT image.
  • the style-transfer techniques of the present disclosure may at least partially remedy the defects of tomosynthesis images as illustrated, for example, in FIGS. 14A, B.
  • FIGS. 14A, B illustrate performance of style-transfer models transferring tomosynthesis images into, respectively, a CT-style image and a CBCT -style image.
  • the style transfer models may be trained as discussed with reference to FIGS. 6A, B and 12.
  • FIGS. 14A, B illustrate that a style transfer example model based on U-net architecture may perform quite well (exhibit fidelity comparable to ground-truth images) with a moderate amount of reconstruction noise. With larger amounts of reconstruction noise, U-net architecture models develop blurring and artifacts that appear as holes within output images.
  • example models based on GAN architecture exhibit better performance than U-net architecture example models for image reconstruction with larger amounts of reconstruction noise.
  • control system 712 may be implemented in software for execution on one or more processors of a computer system.
  • the software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein.
  • the code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.).
  • the computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device.
  • the code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium.
  • the code may be executed by any of a wide variety of centralized or distributed data processing architectures.
  • the programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein.
  • wireless connections may use wireless communication protocols such as Bluetooth, near-field communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 802.11 , Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).
  • wireless communication protocols such as Bluetooth, near-field communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 802.11 , Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

Un système reçoit en provenance d'un dispositif d'imagerie un ensemble d'images de projection bidimensionnelles associées à des angles de projection respectifs. Les images de projection contiennent des projections d'une structure anatomique avec un dispositif allongé flexible disposé à l'intérieur de celle-ci, une cible anatomique et, éventuellement, des marquages de repère. Le système peut reconstruire une image tridimensionnelle à partir de l'ensemble d'images de projection bidimensionnelles. Dans certains exemples, avant la reconstruction, le système peut détecter et retoucher des pixels (à l'intérieur des images de projection) associés aux marquages de repère, au dispositif allongé flexible et/ou à d'autres objets. Le système peut améliorer la reconstruction à l'aide d'un modèle de transfert de style. De plus, le système peut recevoir des données de forme, dans le système de coordonnées du capteur/de l'instrument, associées au dispositif allongé flexible. Le système peut calculer une transformation entre le système de coordonnées d'imagerie et le système de coordonnées de l'instrument, identifier une cible à l'intérieur de l'image reconstruite et, à l'aide de la transformation calculée, mettre à jour l'emplacement cible dans le système de coordonnées de l'instrument.
PCT/US2024/045456 2023-09-08 2024-09-06 Retouche et mise à jour de cible pour imagerie péropératoire Pending WO2025054377A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363581526P 2023-09-08 2023-09-08
US63/581,526 2023-09-08

Publications (1)

Publication Number Publication Date
WO2025054377A1 true WO2025054377A1 (fr) 2025-03-13

Family

ID=92882571

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2024/045456 Pending WO2025054377A1 (fr) 2023-09-08 2024-09-06 Retouche et mise à jour de cible pour imagerie péropératoire

Country Status (1)

Country Link
WO (1) WO2025054377A1 (fr)

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6380732B1 (en) 1997-02-13 2002-04-30 Super Dimension Ltd. Six-degree of freedom tracking system having a passive transponder on the object being tracked
US20060013523A1 (en) 2004-07-16 2006-01-19 Luna Innovations Incorporated Fiber optic position and shape sensing device and method relating thereto
US7316681B2 (en) 1996-05-20 2008-01-08 Intuitive Surgical, Inc Articulated surgical instrument for performing minimally invasive surgery with enhanced dexterity and sensitivity
US7772541B2 (en) 2004-07-16 2010-08-10 Luna Innnovations Incorporated Fiber optic position and/or shape sensing based on rayleigh scatter
US8773650B2 (en) 2009-09-18 2014-07-08 Intuitive Surgical Operations, Inc. Optical position and/or shape sensing
US8900131B2 (en) 2011-05-13 2014-12-02 Intuitive Surgical Operations, Inc. Medical system providing dynamic registration of a model of an anatomical structure for image-guided surgery
US9259274B2 (en) 2008-09-30 2016-02-16 Intuitive Surgical Operations, Inc. Passive preload and capstan drive for surgical instruments
WO2016191298A1 (fr) 2015-05-22 2016-12-01 Intuitive Surgical Operations, Inc. Systèmes et procédés d'alignement pour chirurgie guidée par image
WO2019018736A2 (fr) 2017-07-21 2019-01-24 Intuitive Surgical Operations, Inc. Systèmes et procédés de dispositif allongé flexible
US20220156925A1 (en) * 2019-03-14 2022-05-19 Koninklijke Philips N.V. Dynamic interventional three-dimensional model deformation
WO2023129562A1 (fr) * 2021-12-29 2023-07-06 Noah Medical Corporation Systèmes et procédés d'estimation de pose d'un système d'imagerie

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7316681B2 (en) 1996-05-20 2008-01-08 Intuitive Surgical, Inc Articulated surgical instrument for performing minimally invasive surgery with enhanced dexterity and sensitivity
US6380732B1 (en) 1997-02-13 2002-04-30 Super Dimension Ltd. Six-degree of freedom tracking system having a passive transponder on the object being tracked
US20060013523A1 (en) 2004-07-16 2006-01-19 Luna Innovations Incorporated Fiber optic position and shape sensing device and method relating thereto
US7772541B2 (en) 2004-07-16 2010-08-10 Luna Innnovations Incorporated Fiber optic position and/or shape sensing based on rayleigh scatter
US9259274B2 (en) 2008-09-30 2016-02-16 Intuitive Surgical Operations, Inc. Passive preload and capstan drive for surgical instruments
US8773650B2 (en) 2009-09-18 2014-07-08 Intuitive Surgical Operations, Inc. Optical position and/or shape sensing
US8900131B2 (en) 2011-05-13 2014-12-02 Intuitive Surgical Operations, Inc. Medical system providing dynamic registration of a model of an anatomical structure for image-guided surgery
WO2016191298A1 (fr) 2015-05-22 2016-12-01 Intuitive Surgical Operations, Inc. Systèmes et procédés d'alignement pour chirurgie guidée par image
WO2019018736A2 (fr) 2017-07-21 2019-01-24 Intuitive Surgical Operations, Inc. Systèmes et procédés de dispositif allongé flexible
US20220156925A1 (en) * 2019-03-14 2022-05-19 Koninklijke Philips N.V. Dynamic interventional three-dimensional model deformation
WO2023129562A1 (fr) * 2021-12-29 2023-07-06 Noah Medical Corporation Systèmes et procédés d'estimation de pose d'un système d'imagerie

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LEE BRIAN C ET AL: "Breathing-Compensated Neural Networks for Real Time C-Arm Pose Estimation in Lung CT-Fluoroscopy Registration", 2022 IEEE 19TH INTERNATIONAL SYMPOSIUM ON BIOMEDICAL IMAGING (ISBI), IEEE, 28 March 2022 (2022-03-28), pages 1 - 5, XP034117535, DOI: 10.1109/ISBI52829.2022.9761705 *

Similar Documents

Publication Publication Date Title
US20220346886A1 (en) Systems and methods of pose estimation and calibration of perspective imaging system in image guided surgery
US12245833B2 (en) Systems and methods of continuous registration for image-guided surgery
US12121204B2 (en) Systems and methods of registration for image guided surgery
US20240041531A1 (en) Systems and methods for registering elongate devices to three-dimensional images in image-guided procedures
EP3334325B1 (fr) Systèmes et procédés d'enregistrement pour chirurgie guidée par l'image
JP2022140730A (ja) 画像誘導手術において位置合わせされた蛍光透視画像を使用するためのシステム及び方法
CN116585031A (zh) 用于智能种子配准的系统和方法
US20210259783A1 (en) Systems and Methods Related to Registration for Image Guided Surgery
US20230030727A1 (en) Systems and methods related to registration for image guided surgery
US20220142714A1 (en) Systems for enhanced registration of patient anatomy
EP3930616B1 (fr) Systèmes d'enregistrement d'anatomie de patient
WO2025054377A1 (fr) Retouche et mise à jour de cible pour imagerie péropératoire
WO2025054381A1 (fr) Transfert de style pour imagerie peropératoire
WO2025171214A2 (fr) Filtre directionnel et/ou contraintes de translation pour enregistrement de coordonnées
WO2025019569A1 (fr) Imagerie peropératoire pour navigation assistée par robot
WO2024163533A1 (fr) Extraction de dispositif allongé à partir d'images peropératoires

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24776098

Country of ref document: EP

Kind code of ref document: A1