[go: up one dir, main page]

WO2025029781A1 - Systèmes et procédés de segmentation de données d'image - Google Patents

Systèmes et procédés de segmentation de données d'image Download PDF

Info

Publication number
WO2025029781A1
WO2025029781A1 PCT/US2024/040140 US2024040140W WO2025029781A1 WO 2025029781 A1 WO2025029781 A1 WO 2025029781A1 US 2024040140 W US2024040140 W US 2024040140W WO 2025029781 A1 WO2025029781 A1 WO 2025029781A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
binary map
confidence
threshold
processors
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/US2024/040140
Other languages
English (en)
Inventor
Jorge ANTON GARCIA
Hui Zhang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intuitive Surgical Operations Inc
Original Assignee
Intuitive Surgical Operations Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Intuitive Surgical Operations Inc filed Critical Intuitive Surgical Operations Inc
Publication of WO2025029781A1 publication Critical patent/WO2025029781A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/457—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by analysing connectivity, e.g. edge linking, connected component analysis or slices
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images
    • G06V2201/031—Recognition of patterns in medical or anatomical images of internal organs

Definitions

  • Disclosed examples relate to segmenting image data.
  • the disclosed examples relate to systems and methods for segmenting image data associated with medical procedures performed on a subject.
  • Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects.
  • Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, physicians may insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic, and/or biopsy instruments) to reach a target tissue location.
  • minimally invasive medical instruments including surgical, diagnostic, therapeutic, and/or biopsy instruments
  • One such minimally invasive technique is to use a flexible and/or steerable elongate device, such as a flexible catheter, that can be inserted into anatomic passageways and navigated toward a region of interest within the subject’s anatomy.
  • Image data of the subject is used to plan a pathway towards the region of interest as well as to ensure proper navigation during the procedure.
  • the image data may be analyzed to segment sensitive structures to be avoided (e.g., vessels, pleura, fissures, etc.) from other objects in the image. Accordingly, there is a need for accurate segmentation of the sensitive structures to ensure efficient performance of the procedure.
  • a system for segmenting medical image data may include (i) one or more processors; and (ii) one or more non-transitory, computer-readable media storing instructions.
  • the instructions when executed by the one or more processors, cause the system to (1) obtain a plurality of image data depicting a portion of a subject; (2) input the plurality of image data into a deep learning model trained to generate a confidence map that indicates a confidence level that one or more portions of the image data depict an anatomical structure of the subject; (3) generate a first binary map based on the confidence map and a first confidence threshold; (4) generate a second binary map based on the confidence map and a second confidence threshold, wherein the second confidence threshold is lower than the first confidence threshold; (5) discard one or more connected components of the second binary map below a threshold size; and (6) cause a display device to display a graphical user interface (GUI) based on at least a portion of the plurality of image data, wherein the portion of the plurality of image data is labeled based on the first binary map and the second binary map.
  • GUI graphical user interface
  • a method for segmenting medical image data includes (1) obtaining, via one or more processors, a plurality of image data depicting a portion of a subject; (2) inputting, via the one or more processors, the plurality of image data into a deep learning model trained to generate a confidence map that indicates a confidence level that one or more portions of the image data depict an anatomical structure of the subject; (3) generating, via the one or more processors, a first binary map based on the confidence map and a first confidence threshold; (4) generating, via the one or more processors, a second binary map based on the confidence map and a second confidence threshold, wherein the second confidence threshold is lower than the first confidence threshold; (5) discarding, via the one or more processors, one or more connected components of the second binary map below a threshold size; and (6) causing, via the one or more processors, a display device to display a graphical user interface (GUI) based on at least a portion of the plurality of image
  • GUI graphical user interface
  • a non-transitory computer- readable medium storing instructions thereon.
  • the instructions when executed by one or more processors, cause the one or more processors to (1) obtain a plurality of image data depicting a portion of a subject; (2) input the plurality of image data into a deep learning model trained to generate a confidence map that indicates a confidence level that one or more portions of the image data depict an anatomical structure of the subject; (3) generate a first binary map based on the confidence map and a first confidence threshold; (4) generate a second binary map based on the confidence map and a second confidence threshold, wherein the second confidence threshold is lower than the first confidence threshold; (5) discard one or more connected components of the second binary map below a threshold size; and (6) cause a display device to display a graphical user interface (GUI) based on at least a portion of the plurality of image data, wherein the portion of the plurality of image data is labeled based on the first binary map and the second binary map.
  • GUI graphical user interface
  • FIG. 1 depicts a graphical user interface that may be generated and displayed to a user, according to some examples.
  • FIG. 2 depicts an example workflow for applying a deep learning model to generate a confidence map based on input image data, according to some examples.
  • FIG. 3 depicts an example process for generating binary maps based on a confidence map, according to some examples.
  • FIG. 4 depicts an example process for combining binary maps to produce an output segmentation map, according to some examples.
  • FIG. 5 depicts a graphical user interface that may be generated and displayed to a user, according to some examples.
  • FIG. 6 is an example flow diagram for segmenting medical image data, according to some examples.
  • FIG. 7 is a simplified diagram of a medical system in which techniques disclosed herein may be implemented, according to some examples.
  • FIG. 8A is a simplified diagram of a medical instrument system, including a flexible elongate device, which may be used in connection with the techniques disclosed herein, according to some examples.
  • FIG. 8B is a simplified diagram of a medical tool within the flexible elongate device of FIG. 8 A, according to some examples.
  • FIGS. 9A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly, according to some examples.
  • position refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates).
  • orientation refers to the rotational placement of an object or a portion of an object (e.g., one or more degrees of rotational freedom such as, roll, pitch, and yaw).
  • the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (e.g., up to six total degrees of freedom).
  • the term “shape” refers to a set of poses, positions, and/or orientations measured along an object.
  • distal refers to a position that is closer to a procedural site and the term “proximal” refers to a position that is further from the procedural site. Accordingly, the distal portion or distal end of an instrument is closer to a procedural site than a proximal portion or proximal end of the instrument when the instrument is being used as designed to perform a procedure.
  • the image data includes 2D and/or 3D images. While the term “pixel” is used herein to refer to a particular point within the image data, the term “pixel” should not be understood to imply that the corresponding image data is a 2D image. In embodiments where the image data is 3D image data, the term “pixel” should be understood to refer to a “voxel” of the 3D image.
  • This disclosure relates to segmenting image data to overcome problems associated with prior segmentation algorithms.
  • a deep learning model may be applied to generate a confidence map that indicates a confidence level that any given pixel is associated with a vessel or other anatomical feature.
  • deep learning models suffer from several drawbacks. For example, deep learning models depend on training data that are prone to error. As a result, techniques that rely on simple confidence thresholds produce a significant number of false positives. Additionally, these inaccuracies also tend to manifest in groupings of smaller, discrete connected components that belong to the same vessel. Accordingly, techniques that filter out connected components tend to inappropriately remove these smaller connected components that correspond to vessels.
  • the systems and methods described herein may apply two different confidence thresholds to the output of the deep learning model to generate two different binary maps (e.g., a matrix of values corresponding to the pixels of the image data, where a value of “1” corresponds to the pixel having a confidence above the threshold confidence value and a value of “0” corresponds to the pixel having a confidence below threshold confidence level.).
  • the systems and methods may then detect connected components (e.g., contiguous sets of pixels associated with a value of “1”) within the binary map. It should be appreciated that although the disclosure uses “1” and “0” to refer to the binary classification associated with a pixel, other binary classification representations are envisioned.
  • systems and methods discard connected components that are below a threshold size (e.g., reassign the values of the corresponding pixels to “0”) from the binary map associated with the lower confidence level (the “lower-confidence binary map”).
  • the systems and methods may then combine (e.g., multiply or perform an “AND” operation) the binary map associated with the higher confidence threshold (the “higher-confidence binary map”) with the modified lower-confidence binary map to produce the output binary map.
  • the higher-confidence binary map provides more accurate boundaries of the target anatomy whereas the lower-confidence binary map provides better information on the spread, length, and/or size of the target anatomy.
  • Removing the smaller connected components in only the lower-confidence binary map has several benefits. First, it maintains the higher confidence threshold in the boundary regions near connected components. That is, if a pixel bordering a connected component of the higher-threshold binary map is included in a connected component of the lower-threshold binary map, then the disclosed techniques are able to remove the false positive that would otherwise be present when relying solely on the lower-confidence threshold.
  • the pixel When the two binary maps are combined, the pixel will be zeroed out based on the “0” assigned to the pixel in the higher- threshold binary map. This ensures that the higher-confidence threshold is applied in the border regions of the connected components where improved demarcation between different types of anatomy (c.g., airway and vessel, vessel and lesion, a border of the lung) may improve the efficiency of the operation.
  • different types of anatomy c.g., airway and vessel, vessel and lesion, a border of the lung
  • the two connected components are connected in the lower-confidence binary map.
  • the size of the individual connected components in the higher-confidence binary map may be below the size threshold, whereas the combined connected component in the lower-confidence binary map is above the size threshold.
  • the size threshold was applied when relying solely on the higher-confidence binary map, the smaller connected components that actually exist may be inappropriately discarded. Said another way, applying the size threshold to discard connected components in only the lower- confidence binary map helps ensure that smaller connected components are not inappropriately discarded.
  • a false positive that is isolated from a connected component of the higher- threshold binary map will generally remain isolated in the lower-threshold binary map.
  • the false positive is discarded in the lower-threshold binary map and is not present when the two binary maps are combined.
  • the systems and methods are able to remove false positives based on the size of the connected component while preventing the lower confidence level from inappropriately connecting connected components that are proximate to one another.
  • the segmentation techniques disclosed herein can be executed by a typical medical computing device within a minute of collecting a set of image data from an intraoperative image sensor. Accordingly, the instant segmentation techniques can be performed fast enough to provide the above benefits to an intraoperative procedure.
  • the systems and methods adjust the confidence levels associated with the binary maps based on the particular procedure being performed. For example, it may be more important to have high confidence thresholds in the segmentation when performing a biopsy procedure, as opposed to when treating an ablation. Accordingly, in these embodiments, the systems and methods may enable the operator to tune the confidence thresholds to the particular procedure being performed. [0034] It will be understood that such improvements do not constitute an exhaustive list, and other improvements will be clear according to the various examples discussed herein.
  • an example GUI 100 is provided to a user (or multiple users) to facilitate a robotically-assisted medical procedure.
  • the GUI 100 enables a user to visualize, consider, and decide upon actions for moving/guiding and/or operating a minimally invasive medical instrument (e.g., a flexible elongate device and a treatment tool extendable therefrom) within the anatomy of the patient.
  • a minimally invasive medical instrument e.g., a flexible elongate device and a treatment tool extendable therefrom
  • the flexible elongate device may be steerable using various controls (e.g., controls physically manipulated by the user, such as a trackball, scroll wheel, mouse, etc., or virtual controls on GUI 100 or another GUI).
  • controls e.g., controls physically manipulated by the user, such as a trackball, scroll wheel, mouse, etc., or virtual controls on GUI 100 or another GUI.
  • the medical procedure is an endoluminal ablation procedure targeting a lesion within the patient’s lungs
  • the flexible elongate device is a catheter carrying/containing an ablation probe that is extendable from the catheter.
  • the ablation probe e.g., needle, balloon, and/or other structure
  • the ablation probe may perform ablation using radiofrequency ablation, microwave ablation, cryoablation, electroporation treatment, heat, or any other suitable ablation technique.
  • Example systems and devices/tools for an endoluminal ablation procedure are discussed in more detail below with reference to FIGS. 7- 9B.
  • GUI similar to the GUI 100 may instead be used for other portions of a patient’s anatomy (e.g., gastrointestinal procedures, cardiac procedures, etc.), and/or for medical procedures other than ablations, such as treatments involving injections into target lesions and/or biopsies.
  • a GUI similar to the GUI 100 may instead be used for other portions of a patient’s anatomy (e.g., gastrointestinal procedures, cardiac procedures, etc.), and/or for medical procedures other than ablations, such as treatments involving injections into target lesions and/or biopsies.
  • the GUI 100 may be generated by one or more processors of one or more computing devices and/or systems (e.g., one or more central processing units (CPUs) and/or one or more graphical processing units (GPUs)), which may in turn cause a display device (e.g., a dedicated or general-purpose monitor, or a head-mounted display unit, etc.) to display the GUI 100.
  • a display device e.g., a dedicated or general-purpose monitor, or a head-mounted display unit, etc.
  • the processor(s) may render the GUI 100 and send the corresponding signals/data to the display device for display.
  • the system can be any suitable system (controller(s), etc.) or systems that (collectively) include the one or more processors.
  • the example GUI 100 generally includes a visualization portion 102 and a control portion 104.
  • the visualization portion 102 depicts a model 110 of lung airways within the patient, with the model 110 including visual representations 112a of the patient vessels, visual representations 112b of the patient airways, and a virtual representation 114 of a target lesion.
  • model 110 may consist of only a single model or may be an amalgam of multiple models.
  • the system may model the lung airways and the target lesion separately (possibly based on different imaging modalities), and register the two models with each other for appropriate relative placement within the visualization portion 102.
  • the system may generate the model 110 based on pre-operative imaging data and/or intra-operative imaging data.
  • the pre-operative imaging data and/or intra-operative imaging data may be captured using any suitable imaging technology /modality or technologies/modalities, such as computed tomography (CT), cone-beam computed tomography (CBCT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and so on.
  • CT computed tomography
  • CBCT cone-beam computed tomography
  • MRI magnetic resonance imaging
  • fluoroscopy thermography
  • ultrasound ultrasound
  • OCT optical coherence tomography
  • thermal imaging impedance imaging
  • laser imaging laser imaging
  • nanotube X-ray imaging and so on.
  • the system generates an initial model 110 based on pre-operative imaging data, and then verifies or updates the model 110 based on intra-operative imaging data (e.g., to correct for inaccuracies in the initial model 110 such as the configuration of the lung airways or the lesion size and/or position, possibly due to changes that occurred since the pre-operative images were captured).
  • the process of updating the initial model 110 may include registering the intraoperative imaging data with the pre-operative imaging data and/or with the model 110 itself.
  • different imaging modalities are used to capture the pre-operative and intraoperative imaging data.
  • pre-operative imaging data may be captured using a CT imaging device
  • intra-operative imaging data may be captured using a CBCT or fluoroscopy imaging device.
  • the GUI 100 is used intraoperatively to create an ablation treatment plan during an ablation procedure, while a catheter is within a patient’s anatomy.
  • an imaging device e.g., a CT imaging device
  • the system Based on the pre-operative imaging data, the system generates the model 110, and identifies the target lesion (based on user segmentation or automatic segmentation using the pre-operative imaging data) for inclusion in the model 110 as target lesion 114.
  • the system identifies the vessels (e.g., the lung vasculature) for inclusion in the model 110 as vessels 112a.
  • the system registers the catheter to the model 110.
  • the user may use the model 110 to plan a route/path to the target lesion 114 via the airways 112b.
  • the user may perform a biopsy using the catheter, and the catheter may either be repositioned near the target lesion or left in place if already near the target lesion.
  • the system may capture additional imaging data using an intraoperative imaging device (e.g., a CBCT imaging device), and use the intraoperative imaging data to verify and/or update the pose of the anatomical structures included within the model 110. Accordingly, the system may perform the disclosed segmentation techniques on the intra-operative imaging data using the segmentation techniques described herein.
  • an intraoperative imaging device e.g., a CBCT imaging device
  • the system may perform the disclosed segmentation techniques on the intra-operative imaging data using the segmentation techniques described herein.
  • the visualization portion 102 may also include 2D image data 120 for a selected point in the model 110.
  • the selected point is associated with the target lesion 114.
  • the 2D image data 120 may include sets of image data of the selected point as captured from different axes.
  • the 2D image data 120 may include a first image 120a of the target lesion 114 along the axial axis, a second image 120b of the target lesion 114 along the coronal axis, and a third image 120c of the of the target lesion 114 along the sagittal axis. This provides the user additional context to assist in the developing a suitable ablation navigation workflow.
  • the system changes the visual appearance and other properties of the vessels 112.
  • the control 122 may enable the user to define a vessel size (e.g., thickness) threshold to change the visual appearance of the portion of the model 110 of the vessels 112 that satisfy the vessel size threshold. Accordingly, the vessels 112a that satisfy the vessel size threshold are depicted in a first manner (e.g., in red) and the vessels 112a that do not satisfy the vessel size threshold are depicted in a second manner (e.g., colorless).
  • control 122 may enable the user to define a size of volume 1 16 associated with the target lesion 114 in which the virtual representations of the vessels 112a arc to be depicted.
  • a virtual representation representative of the volume 116 may also change in size.
  • the system may only display the vessels 112a that are within the volume. This enables the system to indicate the location of the vessels 112a for the portion of the model 110 at which the procedure is to be performed without cluttering the rest of the model 110.
  • control portion 104 also includes a control 124 that enables the user to change an opacity associated with the depiction of the visual representations of the patient anatomy.
  • the controls 122, 124 enable the user to identify, for example, the vessels 1 12a without obscuring the visual representations of the patient airways 112b.
  • FIG. 2 depicts an example workflow 200 for applying a deep learning model 225 to generate a confidence map 230 based on input image data 220 (e.g., the pre-operative imaging data and/or intra-operative imaging data that forms the basis of the model 110).
  • input image data 220 e.g., the pre-operative imaging data and/or intra-operative imaging data that forms the basis of the model 110.
  • the deep learning model 225 may have any model structure suitable for classifying the image data 220.
  • the deep learning model may include a convolutional neural network (CNN), a fully-convolutional neural network (FCN), a U-net model, a Transformer model, and so on.
  • CNN convolutional neural network
  • FCN fully-convolutional neural network
  • U-net model
  • Transformer model a model suitable for classifying the image data 220.
  • the deep learning model 225 may be trained using labeled image data.
  • the deep learning model 225 is a single classifier model, such as a model that is trained only to identify the presence of vessels.
  • the deep learning model 225 is a multi-classificr model that is trained to apply multiple labels to the image data 220.
  • the multi-classifier model may include classifiers that specific to anatomical features (e.g., vessels, lung airways, lesions, etc.), classifiers associated with a shape of the structure (e.g., a tubularness, a size, etc.), classifiers associated with an appearance (e.g., a brightness), and so on.
  • the labeled image data may include labels associated with the label(s) of interest.
  • the labeled image data is obtained from publicly available datasets, such as the National Institute of Health’s National Lung Screening Trial (NLST) dataset or The Cancer Imaging Archive’s Non-Small Cell Lung Cancer (NSCLC) dataset. It should be appreciated that while the listed datasets specifically relate to lung anatomy segmentation, other public datasets may be applied to systems that segment other portions of the patient anatomy.
  • the output of the deep learning model 225 is the confidence map 230 that includes a confidence level assigned to the pixels of the image data 220.
  • the confidence levels may indicate a confidence of the deep learning model 225 that the pixel should be classified with the corresponding label.
  • the confidence levels included in the confidence map 230 may indicate a confidence that a pixel should be classified as a vessel.
  • the confidence map may output a vector of confidence values assigned to the pixel by each classifier.
  • the system may perform a sanity check on the labels assigned to the pixels to detect classification label sets that include incongruent labels.
  • an incongruent label set occurs if the deep learning model 225 classifies a pixel as being a vessel, but external to the lung.
  • the pixel is likely to be a false positive for the vessel classifier.
  • the system may correct the false positive by lowering the confidence level associated with the vessel classifier (e.g., setting the confidence level to a predetermined level, subtracting a predetermined value from the output confidence level, subtracting a value from the output confidence level relative to the confidence level for the “out of lung” classifier, etc.).
  • an incongruent label set occurs if the deep learning model 225 classifies a pixel being tubular and bright, but not a vessel.
  • the pixel is likely to be a false negative for the vessel classifier.
  • the system may correct the false negative by increasing the confidence level associated with the vessel classifier (e.g., setting the confidence level to a predetermined level, adding a predetermined value from the output confidence level, adding a value from the output confidence level relative to the confidence level for the “tubular” and/or “brightness” classifier, etc.).
  • the system that trains the deep learning model 225 may be a different system than the system that applies the deep learning model to the image data 220. That is, the deep learning model 225 may be a pre-trained model to which the disclosed segmentation techniques are applied to further improve the classifications provided by the pre-trained models. It should be appreciated that when confirming the congruence of label sets, the system may still apply a default confidence threshold associated with the pre-trained deep learning model 225 when deciding whether or not a label has been applied to a pixel. By checking for incongruent label sets using the default confidence thresholds, the system can remove false positives and/or false negatives from the data set before applying the disclosed segmentation techniques to the confidence map 230. This may further improve the ability for the disclosed segmentation techniques to remove false positives that arise during the segmentation process.
  • FIG. 3 depicts an example process for generating binary maps 335 based on a confidence map 330 (such as the confidence map 230 output by the deep learning model 225). It should be appreciated that FIG. 3 depicts only a subset of the confidence map 330, and that the confidence map 330 may include additional values extending in all three dimensions.
  • the system may generate a first binary map 335a based on a higher confidence threshold and a second binary map 335b based on a lower confidence threshold.
  • the system determines whether the confidence value is greater than or equal to confidence threshold. If so, the system sets the value to “1;” otherwise, the system sets the value to “0.” It should be appreciated that although the illustrated example uses .5 for the first confidence threshold and .4 for the second confidence threshold, this is just one example of suitable confidence thresholds, and the specific values may be tuned based on the particular needs of the user.
  • the system may identify connected components 337 therein.
  • connected components refers to contiguous sets of pixels or voxels that are above the confidence threshold associated with a binary map. That is, the contiguous sets of “1” values in the binary map. In some embodiments, a single pixel having a value of “1” not adjacent to any other pixels having a value of “1” may be considered a connected component.
  • the binary map includes three connected components 337a, 337b, and 337c.
  • connected component 337c is associated with a false positive output of the deep learning model. Accordingly, if the process of solely relying on a single binary map is applied, the false positive component 337c may be included in the output segmentation map.
  • the system further discards connected components that do not satisfy a size threshold (e.g., a predetermined number of pixels, a number of pixels relative to a size of a user-defined anatomical structure, etc.).
  • a size threshold e.g., a predetermined number of pixels, a number of pixels relative to a size of a user-defined anatomical structure, etc.
  • the threshold size may be defined such that the resulting segmented image data does not include a threshold number of connected components beneath a threshold size.
  • the system changes the values of the corresponding pixels from “1” to “0.” Accordingly, the false positive connected component 337c does not have a corresponding connected component in the binary map 335b.
  • the system may combine the binary map 335a with the binary map 335b by performing an “AND” operation 336.
  • the system multiplies the value in the binary map 335a by the value in the binary map 335b to calculate the corresponding value in the binary map 335c.
  • the system may then utilize the binary map 335c when generating the model 110 and/or indicating a location in the model 110 associated with, for example, the vessels 112.
  • the connected components 337a and 337b are now connected via additional pixels that satisfy the lower confidence threshold.
  • the connected component 337d indicates that a pathway exists between connected components 337a and 337b.
  • the ablation navigation workflow of FIG. 1 may inappropriately rely on the connected component 337d to nonetheless generate a pathway that traverses the discontiguous pixels between connected components 337a and 337b.
  • the system is able to remove the false positive 337c while still keeping the discrete connected components 337a and 337b separate.
  • the disclosed techniques related to combining the binary maps 335a, b enable the system to raise the confidence thresholds associated with a typical prc-traincd deep learning model by 4%.
  • FIG. 5 depicted is an example graphical user interface 500 for selecting an operation type.
  • the graphical user interface 500 may be displayed in the control portion 104 of the graphical user interface 100.
  • an ablation procedure may require highly accurate segmentation of sensitive tissue, such as vessels, to ensure that a lesion is treated while minimizing damage to the sensitive tissue.
  • a biopsy treatment may result in less damage to sensitive tissue than an ablation procedure. Accordingly, segmentation of sensitive tissue, such as vessels, may not need to be as accurate as for an ablation procedure. Accordingly, the confidence threshold associated with the binary map 335a and/or the confidence threshold associated with the binary map 335b may be made higher for a biopsy procedure than for an ablation procedure.
  • the system enables the user to select a procedure type. Based on the selection, the system may adjust the confidence thresholds used to generate the binary maps 335a, b, or in some instances may use a single binary map rather than combining binary maps. In some embodiments, the system may then obtain the most recent image data 220 and reapply the disclosed segmentation techniques using the updated thresholds.
  • FIG. 6 depicts an example flow diagram depicts 600 for segmenting medical image data, such as the image data 220 that is relied upon to generate the model 110.
  • the flow diagram 600 may be performed by one or more processors executing instructions stored in one or more computer-readable media (e.g., non-volatile memory), for example, such as various processor(s) of systems or subsystems discussed below in connection with FIGS. 7-9B.
  • processors executing instructions stored in one or more computer-readable media (e.g., non-volatile memory), for example, such as various processor(s) of systems or subsystems discussed below in connection with FIGS. 7-9B.
  • the system obtains a plurality of image data (such as the image data 120, 220) depicting a portion of a subject.
  • the image data may include pre-operative image data or intra-operative image data.
  • the image data includes 3D image data, such as computed tomography (CT) image data, cone-beam CT (CBCT) image data, positron emission tomography (PET) image data, ultrasound image data, or magnetic resonance imaging (MRT) image data.
  • CT computed tomography
  • CBCT cone-beam CT
  • PET positron emission tomography
  • ultrasound image data or magnetic resonance imaging (MRT) image data.
  • MRT magnetic resonance imaging
  • the system inputs the plurality of image data into a deep learning model (such as the deep learning model 225) that is trained to generate a confidence map (such as the confidence map 235) that indicates a confidence level that one or more portions of the image data depict an anatomical structure of the subject.
  • a deep learning model such as the deep learning model 225
  • a confidence map such as the confidence map 235
  • the anatomical structure may be a sensitive structure, such as a vessel, pleura, fissure, etc.
  • the deep learning model includes a U-net model.
  • the deep learning model is a multi-classifier model configured to apply two or more classifiers to each portion of the image data.
  • block 604 may include identifying incongruent sets of labels applied by the two or more classifiers to a particular portion of the image data and adjusting the confidence level for that portion.
  • the incongruent set labels include a set of labels indicating that the portion of the image data is a vessel and outside of the lung. In this example, the system may decrease the confidence level for that portion.
  • the incongruent set labels include a set of labels indicating at least one of a tubular characteristic above a threshold, a brightness characteristic above a threshold, or a vessel characteristic below a threshold. In this example, the system may increase the confidence level for that portion.
  • the system generates a first binary map based on the confidence map and a first confidence threshold.
  • the system may apply the techniques described with respect to generating the binary map 335a.
  • the system generates a second binary map based on the confidence map and a second confidence threshold and discards one or more connected components of the second binary map below a threshold size, respectively.
  • the system may apply techniques described with respect to generating the binary map 335b.
  • the second confidence threshold may be lower than the first confidence threshold.
  • the system causes a display device to display a graphical user interface (GUI) (such as the graphical user interface 100) based on at least a portion of the plurality of image data.
  • GUI graphical user interface
  • the portion of the plurality of image data may be labeled based on the first binary map and the second binary map.
  • the system may combine the first binary map and the second binary map, for example, by multiplying the first binary map and the second binary map.
  • the GUI may change the visual representation of the anatomical structures represented by the model 110 in accordance with the labels.
  • the GUI may include a thickness selection control and/or a volume size adjustment control (such as the control 122).
  • the system may apply the thickness criteria and/or the volume size criteria to update the display of the portion of the image data accordingly.
  • the GUI may include a procedure type control (such as the control 526) to indicate a procedure type.
  • the system may adjust the first and second thresholds based on the indicated procedure type.
  • FIGS. 7-9B depict diagrams of a medical system that may be used for manipulating a medical instrument according to any of the methods and systems described above, in some examples.
  • FIG. 7 is a simplified diagram of a medical system 700 according to some examples.
  • the medical system 700 may be suitable for use in, for example, surgical, diagnostic (e.g., biopsy), or therapeutic (e.g., ablation, electroporation, etc.) procedures. While some examples are provided herein with respect to such procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting.
  • the systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, portions of human or animal anatomy, non-surgical diagnosis, as well as for industrial systems, general or special purpose robotic systems, general or special purpose teleoperational systems, or robotic medical systems.
  • medical system 700 may include a manipulator assembly 702 that controls the operation of a medical instrument 704 in performing various procedures on a patient P.
  • Medical instrument 704 may extend into an internal site within the body of patient P via an opening in the body of patient P.
  • the manipulator assembly 702 may be teleoperated, nonteleoperated, or a hybrid teleoperated and non-teleoperated assembly with one or more degrees of freedom of motion that may be motorized and/or one or more degrees of freedom of motion that may be non-motorized (e.g., manually operated).
  • the manipulator assembly 702 may be mounted to and/or positioned near a patient table T.
  • a master assembly 706 allows an operator O (e.g., a surgeon, a clinician, a physician, or other user) to control the manipulator assembly 702.
  • the master assembly 706 allows the operator O to view the procedural site or other graphical or informational displays.
  • the manipulator assembly 702 may be excluded from the medical system 700 and the instrument 704 may be controlled directly by the operator O.
  • the manipulator assembly 702 may be manually controlled by the operator O. Direct operator control may include various handles and operator interfaces for handheld operation of the instrument 704.
  • the master assembly 706 may be located at a surgeon’s console which is in proximity to (e.g., in the same room as) a patient table T on which patient P is located, such as at the side of the patient table T. In some examples, the master assembly 706 is remote from the patient table T, such as in in a different room or a different building from the patient table T.
  • the master assembly 706 may include one or more control devices for controlling the manipulator assembly 702.
  • the control devices may include any number of a variety of input devices, such as joysticks, trackballs, scroll wheels, directional pads, buttons, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, motion or presence sensors, and/or the like.
  • the master assembly 706 may be or include an extended reality (XR) device, such as a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, or any other such device as described herein.
  • XR extended reality
  • VR virtual reality
  • AR
  • the manipulator assembly 702 supports the medical instrument 704 and may include a kinematic structure of links that provide a set-up structure.
  • the links may include one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place) and/or one or more servo controlled links (e.g., one or more links that may be controlled in response to commands, such as from a control system 712).
  • the manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on the medical instrument 704 in response to commands, such as from the control system 712.
  • the actuators may include drive systems that move the medical instrument 704 in various ways when coupled to the medical instrument 704.
  • one or more actuators may advance medical instrument 704 into a naturally or surgically created anatomic orifice.
  • Actuators may control articulation of the medical instrument 704, such as by moving the distal end (or any other portion) of medical instrument 704 in multiple degrees of freedom.
  • degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes).
  • One or more actuators may control rotation of the medical instrument about a longitudinal axis.
  • Actuators can also be used to move an articulable end effector of medical instrument 704, such as for grasping tissue in the jaws of a biopsy device and/or the like, or may be used to move or otherwise control tools (e.g., imaging tools, ablation tools, biopsy tools, electroporation tools, etc.) that are inserted within the medical instrument 704.
  • the manipulator assembly 702 may include or be a robotic-assisted platform as described in more detail above with regard to FIGS. 1-4.
  • the medical instrument 704 may be or include elements of a medical instrument as described above with regard to FIGS. 1-4.
  • the medical system 700 may include a sensor system 708 with one or more sub-systems for receiving information about the manipulator assembly 702 and/or the medical instrument 704.
  • Such sub-systems may include a position sensor system (e.g., that uses electromagnetic (EM) sensors or other types of sensors that detect position or location); a shape sensor system for determining the position, orientation, speed, velocity, pose, and/or shape of a distal end and/or of one or more segments along a flexible body of the medical instrument 704; a visualization system (e.g., using a color imaging device, an infrared imaging device, an ultrasound imaging device, an x-ray imaging device, a fluoroscopic imaging device, a computed tomography (CT) imaging device, a magnetic resonance imaging (MRI) imaging device, or some other type of imaging device) for capturing images, such as from the distal end of medical instrument 704 or from some other location; and/or actuator position sensors such as resolvers, encoders, potentiometers, and the like that describe
  • the medical system 700 may include a display system 710 for displaying an image or representation of the procedural site and the medical instrument 704.
  • Display system 710 and master assembly 706 may be oriented so physician O can control medical instrument 704 and master assembly 706 with the perception of telepresence.
  • the display system 710 and the master assembly 706 are depicted in FIG. 7 as separate blocks, both the display system 710 and the master assembly 706 may be part of the same device and/or operation control system.
  • the medical instrument 704 may include a visualization system, which may include an image capture assembly that records a concurrent or real-time image of a procedural site and provides the image to the operator O through one or more displays of display system 710.
  • the image capture assembly may include various types of imaging devices.
  • the concurrent image may be, for example, a two-dimensional image or a three-dimensional image captured by an endoscope positioned within the anatomical procedural site.
  • the visualization system may include endoscopic components that may be integrally or removably coupled to medical instrument 704. Additionally or alternatively, a separate endoscope, attached to a separate manipulator assembly, may be used with medical instrument 704 to image the procedural site.
  • the visualization system may be implemented as hardware, firmware, software or a combination thereof which interact with or are otherwise executed by one or more computer processors, such as of the control system 712.
  • Display system 710 may also display an image of the procedural site and medical instruments, which may be captured by the visualization system.
  • the medical system 700 provides a perception of telepresence to the operator O.
  • images captured by an imaging device at a distal portion of the medical instrument 704 may be presented by the display system 710 to provide the perception of being at the distal portion of the medical instrument 704 to the operator O.
  • the input to the master assembly 706 provided by the operator O may move the distal portion of the medical instrument 704 in a manner that corresponds with the nature of the input (e.g., distal tip turns right when a trackball is rolled to the right) and results in corresponding change to the perspective of the images captured by the imaging device at the distal portion of the medical instrument 704.
  • the perception of telepresence for the operator O is maintained as the medical instrument 704 is moved using the master assembly 706.
  • the operator O can manipulate the medical instrument 704 and hand controls of the master assembly 706 as if viewing the workspace in substantially true presence, simulating the experience of an operator that is physically manipulating the medical instrument 704 from within the patient anatomy.
  • the display system 710 may present virtual images of a procedural site that are created using image data recorded pre-operatively (e.g., prior to the procedure performed by the medical instrument system 800) or intra-operatively (e.g., concurrent with the procedure performed by the medical instrument system 800), such as image data created using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotubc X-ray imaging, and/or the like.
  • CT computed tomography
  • MRI magnetic resonance imaging
  • PET positron emission tomography
  • fluoroscopy thermography
  • ultrasound ultrasound
  • OCT optical coherence tomography
  • thermal imaging impedance imaging
  • laser imaging nanotubc X-ray imaging
  • nanotubc X-ray imaging and/or the like.
  • the virtual images may include two-dimensional, three-dimensional, or higher-dimensional (e.g.,
  • display system 710 may display a virtual image that is generated based on tracking the location of medical instrument 704.
  • the tracked location of the medical instrument 704 may be registered (e.g., dynamically referenced) with the model generated using the pre-operative or intra-operative images, with different portions of the model correspond with different locations of the patient anatomy.
  • the registration is used to determine portions of the model corresponding with the location and/or perspective of the medical instrument 704 and virtual images are generated using the determined portions of the model. This may be done to present the operator O with virtual images of the internal procedural site from viewpoints of medical instrument 704 that correspond with the tracked locations of the medical instrument 704.
  • the medical system 700 may also include the control system 712, which may include processing circuitry that implements the some or all of the methods or functionality discussed herein.
  • the control system 712 may include at least one memory and at least one processor for controlling the operations of the manipulator assembly 702, the medical instrument 704, the master assembly 706, the sensor system 708, and/or the display system 710.
  • Control system 712 may include instructions (e.g., a non-transitory machine -readable medium storing the instructions) that when executed by the at least one processor, configures the one or more processors to implement some or all of the methods or functionality discussed herein. While the control system 712 is shown as a single block in FIG.
  • control system 712 may include two or more separate data processing circuits with one portion of the processing being performed at the manipulator assembly 702, another portion of the processing being performed at the master assembly 706, and/or the like.
  • control system 712 may include other types of processing circuitry, such as application- specific integrated circuits (ASICs) and/or field-programmable gate array (FPGAs).
  • ASICs application-specific integrated circuits
  • FPGAs field-programmable gate array
  • the control system 712 may be implemented using hardware, firmware, software, or a combination thereof.
  • control system 712 may receive feedback from the medical instrument 704, such as force and/or torque feedback. Responsive to the feedback, the control system 712 may transmit signals to the master assembly 706. In some examples, the control system 712 may transmit signals instructing one or more actuators of the manipulator assembly 702 to move the medical instrument 704. In some examples, the control system 712 may transmit informational displays regarding the feedback to the display system 710 for presentation or perform other types of actions based on the feedback.
  • the control system 712 may include a virtual visualization system to provide navigation assistance to operator O when controlling the medical instrument 704 during an image-guided medical procedure.
  • Virtual navigation using the virtual visualization system may be based upon an acquired pre-operative or intra-operative dataset of anatomic passageways of the patient P.
  • the control system 712 or a separate computing device may convert the recorded images, using programmed instructions alone or in combination with operator inputs, into a model of the patient anatomy.
  • the model may include a segmented two-dimensional or three-dimensional composite representation of a partial or an entire anatomic organ or anatomic region.
  • An image data set may be associated with the composite representation.
  • the virtual visualization system may obtain sensor data from the sensor system 708 that is used to compute an (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P.
  • the sensor system 708 may be used to register and display the medical instrument 704 together with the pre-operatively or intra-operatively recorded images.
  • PCT Publication WO 2016/161298 published December 1, 2016 and titled “Systems and Methods of Registration for Image Guided Surgery”
  • PCT Publication WO 2016/161298 published December 1, 2016 and titled “Systems and Methods of Registration for Image Guided Surgery”
  • the sensor system 708 may be used to compute the (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P.
  • the location can be used to produce both macro-level (e.g., external) tracking images of the anatomy of patient P and virtual internal images of the anatomy of patient P.
  • the system may include one or more electromagnetic (EM) sensors, fiber optic sensors, and/or other sensors to register and display a medical instrument together with pre-operatively recorded medical images.
  • EM electromagnetic
  • Medical system 700 may further include operations and support systems (not shown) such as illumination systems, steering control systems, irrigation systems, and/or suction systems.
  • the medical system 700 may include more than one manipulator assembly and/or more than one master assembly.
  • the exact number of manipulator assemblies may depend on the medical procedure and space constraints within the procedural room, among other factors. Multiple master assemblies may be co-located or they may be positioned in separate locations. Multiple master assemblies may allow more than one operator to control one or more manipulator assemblies in various combinations.
  • FIG. 8A is a simplified diagram of a medical instrument system 800 according to some examples.
  • the medical instrument system 800 includes a flexible elongate device 802 (also referred to as elongate device 802), a drive unit 804, and a medical tool 826 that collectively is an example of a medical instrument 704 of a medical system 700.
  • the medical system 700 may be a teleoperated system, a non-teleoperated system, or a hybrid teleoperated and non-teleoperated system, as explained with reference to FIG. 7.
  • a visualization system 831, tracking system 830, and navigation system 832 are also shown in FIG. 8A and are example components of the control system 712 of the medical system 700.
  • the medical instrument system 800 may be used for non-teleoperational exploratory procedures or in procedures involving traditional manually operated medical instruments, such as endoscopy.
  • the medical instrument system 800 may be used to gather (e.g., measure) a set of data points corresponding to locations within anatomic passageways of a patient, such as patient P.
  • the elongate device 802 is coupled to the drive unit 804.
  • the elongate device 802 includes a channel 821 through which the medical tool 826 may be inserted.
  • the elongate device 802 navigates within patient anatomy to deliver the medical tool 826 to a procedural site.
  • the elongate device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818.
  • the flexible body 816 may have an approximately 3 mm outer diameter. Other flexible body outer diameters may be larger or smaller.
  • Medical instrument system 800 may include the tracking system 830 for determining the position, orientation, speed, velocity, pose, and/or shape of the flexible body 816 at the distal end 818 and/or of one or more segments 824 along flexible body 816, as will be described in further detail below.
  • the tracking system 830 may include one or more sensors and/or imaging devices.
  • the flexible body 816 such as the length between the distal end 818 and the proximal end 817, may include multiple segments 824.
  • the tracking system 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, the tracking system 830 is part of control system 712 shown in FIG. 7.
  • Tracking system 830 may track the distal end 818 and/or one or more of the segments 824 of the flexible body 816 using a shape sensor 822.
  • the shape sensor 822 may include an optical fiber aligned with the flexible body 816 (e.g., provided within an interior channel of the flexibly body 816 or mounted externally along the flexible body 816).
  • the optical fiber may have a diameter of approximately 800 pm. In other examples, the diameter may be larger or smaller.
  • the optical fiber of the shape sensor 822 may form a fiber optic bend sensor for determining the shape of flexible body 816.
  • Optical fibers including Fiber Bragg Gratings (FBGs) may be used to provide strain measurements in structures in one or more dimensions.
  • FBGs Fiber Bragg Gratings
  • the shape of the flexible body 816 may be determined using other techniques. For example, a history of the position and/or pose of the distal end 818 of the flexible body 816 can be used to reconstruct the shape of flexible body 816 over an interval of time (e.g., as the flexible body 816 is advanced or retracted within a patient anatomy).
  • the tracking system 830 may alternatively and/or additionally track the distal end 818 of the flexible body 816 using a position sensor system 820.
  • Position sensor system 820 may be a component of an EM sensor system with the position sensor system 820 including one or more position sensors.
  • the position sensor system 820 is shown as being near the distal end 818 of the flexible body 816 to track the distal end 818, the number and location of the position sensors of the position sensor system 820 may vary to track different regions along the flexible body 816.
  • the position sensors include conductive coils that may be subjected to an externally generated electromagnetic field. Each coil of position sensor system 820 may produce an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field.
  • the position sensor system 820 may measure one or more position coordinates and/or one or more orientation angles associated with one or more portions of flexible body 816.
  • the position sensor system 820 may be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, Z and three orientation angles indicating pitch, yaw, and roll of a base point. In some examples, the position sensor system 820 may be configured and positioned to measure five degrees of freedom, e.g., three position coordinates X, Y, Z and two orientation angles indicating pitch and yaw of a base point. Further description of a position sensor system, which may be applicable in some examples, is provided in U.S. Patent No. 6,380,432 (filed August 11, 1999 and titled “Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked”), which is incorporated by reference herein in its entirety.
  • the tracking system 830 may alternately and/or additionally rely on a collection of pose, position, and/or orientation data stored for a point of an elongate device 802 and/or medical tool 826 captured during one or more cycles of alternating motion, such as breathing. This stored data may be used to develop shape information about the flexible body 816.
  • a series of position sensors such as EM sensors like the sensors in position sensor 820 or some other type of position sensors may be positioned along the flexible body 816 and used for shape sensing.
  • a history of data from one or more of these position sensors taken during a procedure may be used to represent the shape of elongate device 802, particularly if an anatomic passageway is generally static.
  • FIG. 8B is a simplified diagram of the medical tool 826 within the elongate device 802 according to some examples.
  • the flexible body 816 of the elongate device 802 may include the channel 821 sized and shaped to receive the medical tool 826.
  • the medical tool 826 may be used for procedures such as diagnostics, imaging, surgery, biopsy, ablation, illumination, irrigation, suction, electroporation, etc.
  • Medical tool 826 can be deployed through channel 821 of flexible body 816 and operated at a procedural site within the anatomy.
  • Medical instrument 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, grasper, brush, etc.), an ablation tool (e.g., a laser ablation tool, radio frequency (RF) ablation tool, cryoablation tool, thermal ablation tool, heated liquid ablation tool, etc.), an electroporation tool, and/or another surgical, diagnostic, or therapeutic tool.
  • the medical tool 826 may include an end effector having a single working member such as a scalpel, a blunt blade, an optical fiber, an electrode, and/or the like.
  • Other end types of end effectors may include, for example, forceps, graspers, scissors, staplers, clip appliers, and/or the like.
  • Other end effectors may further include electrically activated end effectors such as electrosurgical electrodes, transducers, sensors, and/or the like.
  • the medical tool 826 may be a biopsy tool used to remove sample tissue or a sampling of cells from a target anatomic location.
  • the biopsy tool is a flexible needle.
  • the biopsy tool may further include a sheath that can surround the flexible needle to protect the needle and interior surface of the channel 821 when the biopsy tool is within the channel 821.
  • the medical tool 826 may be an image capture probe that includes a distal portion with a stereoscopic or monoscopic camera that may be placed at or near the distal end 818 of flexible body 816 for capturing images (e.g., still or video images).
  • the captured images may be processed by the visualization system 831 for display and/or provided to the tracking system 830 to support tracking of the distal end 818 of the flexible body 816 and/or one or more of the segments 824 of the flexible body 816.
  • the image capture probe may include a cable for transmitting the captured image data that is coupled to an imaging device at the distal portion of the image capture probe.
  • the image capture probe may include a fiber-optic bundle, such as a fiberscope, that couples to a more proximal imaging device of the visualization system 831.
  • the image capture probe may be single-spectral or multi- spectral, for example, capturing image data in one or more of the visible, near-infrared, infrared, and/or ultraviolet spectrums.
  • the image capture probe may also include one or more light emitters that provide illumination to facilitate image capture.
  • the image capture probe may use ultrasound, x-ray, fluoroscopy, CT, MRI, or other types of imaging technology.
  • the image capture probe is inserted within the flexible body 816 of the elongate device 802 to facilitate visual navigation of the elongate device 802 to a procedural site and then is replaced within the flexible body 816 with another type of medical tool 826 that performs the procedure.
  • the image capture probe may be within the flexible body 816 of the elongate device 802 along with another type of medical tool 826 to facilitate simultaneous image capture and tissue intervention, such as within the same channel 821 or in separate channels.
  • a medical tool 826 may be advanced from the opening of the channel 821 to perform the procedure (or some other functionality) and then retracted back into the channel 821 when the procedure is complete.
  • the medical tool 826 may be removed from the proximal end 817 of the flexible body 816 or from another optional instrument port (not shown) along flexible body 816.
  • the elongate device 802 may include integrated imaging capability rather than utilize a removable image capture probe.
  • the imaging device (or fiberoptic bundle) and the light emitters may be located at the distal end 818 of the elongate device 802.
  • the flexible body 215 may include one or more dedicated channels that carry the cable(s) and/or optical fiber(s) between the distal end 818 and the visualization system 831.
  • the medical instrument system 800 can perform simultaneous imaging and tool operations.
  • the medical tool 826 is capable of controllable articulation.
  • the medical tool 826 may house cables (which may also be referred to as pull wires), linkages, or other actuation controls (not shown) that extend between its proximal and distal ends to controllably bend the distal end of medical tool 826, such as discussed herein for the flexible elongate device 802.
  • the medical tool 826 may be coupled to a drive unit 804 and the manipulator assembly 702.
  • the elongate device 802 may be excluded from the medical instrument system 800 or may be a flexible device that does not have controllable articulation. Steerable instruments or tools, applicable in some examples, are further described in detail in U.S. Patent No.
  • the flexible body 816 of the elongate device 802 may also or alternatively house cables, linkages, or other steering controls (not shown) that extend between the drive unit 804 and the distal end 818 to controllably bend the distal end 818 as shown, for example, by broken dashed line depictions 819 of the distal end 818 in FIG. 8A.
  • at least four cables are used to provide independent up-down steering to control a pitch of the distal end 818 and left-right steering to control a yaw of the distal end 281.
  • the flexible elongate device 802 may be a steerable catheter.
  • the drive unit 804 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of the teleoperational assembly.
  • the elongate device 802 and/or medical tool 826 may include gripping features, manual actuators, or other components for manually controlling the motion of the elongate device 802 and/or medical tool 826.
  • the elongate device 802 may be steerable or, alternatively, the elongate device 802 may be non-steerable with no integrated mechanism for operator control of the bending of distal end 818.
  • one or more channels 821 (which may also be referred to as lumens), through which medical tools 826 can be deployed and used at a target anatomical location, may be defined by the interior walls of the flexible body 816 of the elongate device 802.
  • the medical instrument system 800 may include a flexible bronchial instrument, such as a bronchoscope or bronchial catheter, for use in examination, diagnosis, biopsy, and/or treatment of a lung.
  • a flexible bronchial instrument such as a bronchoscope or bronchial catheter
  • the medical instrument system 800 may also be suited for navigation and treatment of other tissues, via natural or surgically created connected passageways, in any of a variety of anatomic systems, including the colon, the intestines, the kidneys and kidney calices, the brain, the heart, the circulatory system including vasculature, and/or the like.
  • the information from the tracking system 830 may be sent to the navigation system 832, where the information may be combined with information from the visualization system 831 and/or pre-operatively obtained models to provide the physician, clinician, surgeon, or other operator with real-time position information.
  • the real-time position information may be displayed on the display system 710 for use in the control of the medical instrument system 800.
  • the navigation system 832 may utilize the position information as feedback for positioning medical instrument system 800.
  • Various systems for using fiber optic sensors to register and display a surgical instrument with surgical images are provided in U.S. Patent No. 8,300,131 (filed May 13, 2011 and titled “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”), which is incorporated by reference herein in its entirety.
  • FIGS. 9A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly according to some examples.
  • a surgical environment 900 may include a patient P positioned on the patient table T.
  • Patient P may be stationary within the surgical environment 900 in the sense that gross patient movement is limited by sedation, restraint, and/or other means. Cyclic anatomic motion, including respiration and cardiac motion, of patient P may continue.
  • a medical instrument 904 is used to perform a medical procedure which may include, for example, surgery, biopsy, ablation, illumination, irrigation, suction, or electroporation.
  • the medical instrument 904 may also be used to perform other types of procedures, such as a registration procedure to associate the position, orientation, and/or pose data captured by the sensor system 708 to a desired (e.g., anatomical or system) reference frame.
  • the medical instrument 904 may be, for example, the medical instrument 704.
  • the medical instrument 904 may include an elongate device 910 (e.g., a catheter) coupled to an instrument body 912.
  • Elongate device 910 includes one or more channels sized and shaped to receive a medical tool.
  • Elongate device 910 may also include one or more sensors (e.g., components of the sensor system 708).
  • a shape sensor 914 may be fixed at a proximal point 916 on the instrument body 912.
  • the proximal point 916 of the shape sensor 914 may be movable with the instrument body 912, and the location of the proximal point 916 with respect to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device).
  • the shape sensor 914 may measure a shape from the proximal point 916 to another point, such as a distal end 918 of the elongate device 910.
  • the shape sensor 914 may be aligned with the elongate device 910 (e.g., provided within an interior channel or mounted externally). In some examples, the shape sensor 914 may use optical fibers to generate shape information for the elongate device 910. [0102] In some examples, position sensors (e.g., EM sensors) may be incorporated into the medical instrument 904. A series of position sensors may be positioned along the flexible elongate device 910 and used for shape sensing. Position sensors may be used alternatively to the shape sensor 914 or with the shape sensor 914, such as to improve the accuracy of shape sensing or to verify shape information.
  • position sensors e.g., EM sensors
  • Elongate device 910 may house cables, linkages, or other steering controls that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918.
  • at least four cables are used to provide independent up-down steering to control a pitch of distal end 918 and left-right steering to control a yaw of distal end 918.
  • the instrument body 91 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of a manipulator assembly.
  • the instrument body 912 may be coupled to an instrument carriage 906.
  • the instrument carriage 906 may be mounted to an insertion stage 908 that is fixed within the surgical environment 900.
  • the insertion stage 908 may be movable but have a known location (e.g., via a tracking sensor or other tracking device) within surgical environment 900.
  • Instrument carriage 906 may be a component of a manipulator assembly (e.g., manipulator assembly 702) that couples to the medical instrument 904 to control insertion motion (e.g., motion along an insertion axis A) and/or motion of the distal end 918 of the elongate device 910 in multiple directions, such as yaw, pitch, and/or roll.
  • the instrument carriage 906 or insertion stage 908 may include actuators, such as servomotors, that control motion of instrument carriage 906 along the insertion stage 908.
  • a sensor device 920 which may be a component of the sensor system 708, may provide information about the position of the instrument body 912 as it moves relative to the insertion stage 908 along the insertion axis A.
  • the sensor device 920 may include one or more resolvers, encoders, potentiometers, and/or other sensors that measure the rotation and/or orientation of the actuators controlling the motion of the instrument carriage 906, thus indicating the motion of the instrument body 912.
  • the insertion stage 908 has a linear track as shown in FIGS. 9A and 9B.
  • the insertion stage 908 may have curved track or have a combination of curved and linear track sections.
  • FIG. 9 A shows the instrument body 912 and the instrument carriage 906 in a retracted position along the insertion stage 908.
  • the proximal point 916 is at a position LO on the insertion axis A.
  • the location of the proximal point 916 may be set to a zero value and/or other reference value to provide a base reference (c.g., corresponding to the origin of a desired reference frame) to describe the position of the instrument carriage 906 along the insertion stage 908.
  • the distal end 918 of the elongate device 910 may be positioned just inside an entry orifice of patient P.
  • the instrument body 912 and the instrument carriage 906 have advanced along the linear track of insertion stage 908, and the distal end 918 of the elongate device 910 has advanced into patient P.
  • the proximal point 916 is at a position LI on the insertion axis A.
  • the rotation and/or orientation of the actuators measured by the sensor device 920 indicating movement of the instrument carriage 906 along the insertion stage 908 and/or one or more position sensors associated with instrument carriage 906 and/or the insertion stage 908 may be used to determine the position LI of the proximal point 916 relative to the position L0.
  • the position LI may further be used as an indicator of the distance or insertion depth to which the distal end 918 of the elongate device 910 is inserted into the passageway(s) of the anatomy of patient P.
  • control system 712 may be implemented in software for execution on one or more processors of a computer system.
  • the software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein.
  • the code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.).
  • the computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device.
  • the code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium.
  • the code may be executed by any of a wide variety of centralized or distributed data processing architectures.
  • the programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein.
  • wireless connections may use wireless communication protocols such as Bluetooth, ncar-ficld communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 502.11, Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).
  • wireless communication protocols such as Bluetooth, ncar-ficld communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 502.11, Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).

Landscapes

  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Epidemiology (AREA)
  • Public Health (AREA)
  • Primary Health Care (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

L'invention concerne des systèmes et des procédés de segmentation de données d'image médicale. Le procédé peut consister à (i) obtenir une pluralité de données d'image représentant une partie d'un sujet ; (ii) entrer la pluralité de données d'image dans un modèle d'apprentissage profond entraîné pour générer une carte de confiance ; (iii) générer une première carte binaire sur la base de la carte de confiance et d'un premier seuil de confiance et une seconde carte binaire sur la base de la carte de confiance et d'un second seuil de confiance ; (iv) éliminer un ou plusieurs composants connectés de la seconde carte binaire au-dessous d'une valeur limite ; et (v) amener un dispositif d'affichage à afficher une interface utilisateur graphique (GUI) sur la base d'au moins une partie de la pluralité de données d'image, la partie de la pluralité de données d'image étant marquée sur la base de la première carte binaire et de la seconde carte binaire. Le procédé peut être mis en œuvre par le système.
PCT/US2024/040140 2023-07-31 2024-07-30 Systèmes et procédés de segmentation de données d'image Pending WO2025029781A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363516726P 2023-07-31 2023-07-31
US63/516,726 2023-07-31

Publications (1)

Publication Number Publication Date
WO2025029781A1 true WO2025029781A1 (fr) 2025-02-06

Family

ID=92458329

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2024/040140 Pending WO2025029781A1 (fr) 2023-07-31 2024-07-30 Systèmes et procédés de segmentation de données d'image

Country Status (1)

Country Link
WO (1) WO2025029781A1 (fr)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6380432B2 (en) 1999-10-22 2002-04-30 Elsicon Inc. Materials for inducing alignment in liquid crystals and liquid crystal displays
US20060013523A1 (en) 2004-07-16 2006-01-19 Luna Innovations Incorporated Fiber optic position and shape sensing device and method relating thereto
US20070133894A1 (en) * 2005-12-07 2007-06-14 Siemens Corporate Research, Inc. Fissure Detection Methods For Lung Lobe Segmentation
US7772541B2 (en) 2004-07-16 2010-08-10 Luna Innnovations Incorporated Fiber optic position and/or shape sensing based on rayleigh scatter
US7916681B2 (en) 2005-05-20 2011-03-29 Telefonaktiebolaget Lm Ericsson (Publ) Method and apparatus for communication channel error rate estimation
US8300131B2 (en) 2009-09-10 2012-10-30 Fujifilm Corporation Image pickup device for wide dynamic range at a high frame rate
US8773350B2 (en) 2011-08-31 2014-07-08 Sharp Kabushiki Kaisha Sensor circuit and electronic apparatus
US9259274B2 (en) 2008-09-30 2016-02-16 Intuitive Surgical Operations, Inc. Passive preload and capstan drive for surgical instruments
WO2016161298A1 (fr) 2015-04-02 2016-10-06 Ensco International Incorporated Guide monté sur étrier
WO2019018436A1 (fr) 2017-07-17 2019-01-24 Desktop Metal, Inc. Fabrication additive à l'aide de débits d'alimentation de matériau de construction variables

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6380432B2 (en) 1999-10-22 2002-04-30 Elsicon Inc. Materials for inducing alignment in liquid crystals and liquid crystal displays
US20060013523A1 (en) 2004-07-16 2006-01-19 Luna Innovations Incorporated Fiber optic position and shape sensing device and method relating thereto
US7772541B2 (en) 2004-07-16 2010-08-10 Luna Innnovations Incorporated Fiber optic position and/or shape sensing based on rayleigh scatter
US7916681B2 (en) 2005-05-20 2011-03-29 Telefonaktiebolaget Lm Ericsson (Publ) Method and apparatus for communication channel error rate estimation
US20070133894A1 (en) * 2005-12-07 2007-06-14 Siemens Corporate Research, Inc. Fissure Detection Methods For Lung Lobe Segmentation
US9259274B2 (en) 2008-09-30 2016-02-16 Intuitive Surgical Operations, Inc. Passive preload and capstan drive for surgical instruments
US8300131B2 (en) 2009-09-10 2012-10-30 Fujifilm Corporation Image pickup device for wide dynamic range at a high frame rate
US8773350B2 (en) 2011-08-31 2014-07-08 Sharp Kabushiki Kaisha Sensor circuit and electronic apparatus
WO2016161298A1 (fr) 2015-04-02 2016-10-06 Ensco International Incorporated Guide monté sur étrier
WO2019018436A1 (fr) 2017-07-17 2019-01-24 Desktop Metal, Inc. Fabrication additive à l'aide de débits d'alimentation de matériau de construction variables

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ALEXANDRU PAUL CONDURACHE ET AL: "Segmentation of retinal vessels with a hysteresis binary-classification paradigm", COMPUTERIZED MEDICAL IMAGING AND GRAPHICS, PERGAMON PRESS, NEW YORK, NY, US, vol. 36, no. 4, 8 February 2012 (2012-02-08), pages 325 - 335, XP028419734, ISSN: 0895-6111, [retrieved on 20120215], DOI: 10.1016/J.COMPMEDIMAG.2012.02.002 *
NAJJAR M A ET AL: "Memory-Efficient Architecture for Hysteresis Thresholding and Object Feature Extraction", IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE, USA, vol. 20, no. 12, 1 December 2011 (2011-12-01), pages 3566 - 3579, XP011411925, ISSN: 1057-7149, DOI: 10.1109/TIP.2011.2147324 *

Similar Documents

Publication Publication Date Title
US12121204B2 (en) Systems and methods of registration for image guided surgery
US20230088056A1 (en) Systems and methods for navigation in image-guided medical procedures
US11583353B2 (en) Systems and methods of continuous registration for image-guided surgery
US11080902B2 (en) Systems and methods for generating anatomical tree structures
CN110325138B (zh) 用于智能种子配准的系统和方法
US11779396B2 (en) Systems and methods for registering elongate devices to three dimensional images in image-guided procedures
EP3334325B1 (fr) Systèmes et procédés d'enregistrement pour chirurgie guidée par l'image
EP3795061A1 (fr) Systèmes et procédés d'alignement pour la chirurgie guidée par l'image
US20210259783A1 (en) Systems and Methods Related to Registration for Image Guided Surgery
US20230030727A1 (en) Systems and methods related to registration for image guided surgery
US20250268453A1 (en) Systems and methods for detecting an orientation of medical instruments
US20220142714A1 (en) Systems for enhanced registration of patient anatomy
WO2024186659A1 (fr) Génération d'images médicales haute résolution à l'aide d'un modèle d'apprentissage automatique
WO2025029781A1 (fr) Systèmes et procédés de segmentation de données d'image
US12502224B2 (en) Systems and methods for registering elongate devices to three-dimensional images in image-guided procedures
WO2024163533A1 (fr) Extraction de dispositif allongé à partir d'images peropératoires

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24758399

Country of ref document: EP

Kind code of ref document: A1