WO2025223980A1 - Method, system and computer program for identification and temporal tracking of equivalent lesions in medical images - Google Patents
Method, system and computer program for identification and temporal tracking of equivalent lesions in medical imagesInfo
- Publication number
- WO2025223980A1 WO2025223980A1 PCT/EP2025/060617 EP2025060617W WO2025223980A1 WO 2025223980 A1 WO2025223980 A1 WO 2025223980A1 EP 2025060617 W EP2025060617 W EP 2025060617W WO 2025223980 A1 WO2025223980 A1 WO 2025223980A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- lesion
- mask
- medical image
- annotation
- roi
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
- G06T7/0014—Biomedical image inspection using an image reference approach
- G06T7/0016—Biomedical image inspection using an image reference approach involving temporal comparison
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30056—Liver; Hepatic
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30084—Kidney; Renal
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
Definitions
- the present invention relates to medical images analysis. More specifically, the invention relates to a method, system and computer program for identification and temporal tracking of equivalent lesions in medical images such as computed tomography (CT) images, Magnetic Resonance Images (IMR), ultrasound images, etc.
- CT computed tomography
- IMR Magnetic Resonance Images
- the accurate identification and assessment of these anomalies are crucial for evaluating the potential associated risk.
- EP3610456-A1 provides a method for determining volumetric properties of one or more lesions in medical images.
- the method comprises receiving image data; determining one or more locations of one or more lesions in the image data; creating an image segmentation (i.e. mask or contour) comprising the determined one or more locations of the one or more lesions in the image data and using the image segmentation to determine a volumetric property of the lesion.
- the method uses convolutional neural networks to identify the lesion locations.
- EP2407927-B1 describes a method to evaluate the evolution of tumor lesions by segmentation into concentric areas and comparison of functional parameters over time. Images of lesions at different time points are provided, and lesions are delineated and registered. Subsequently, they are segmented into concentric areas, and changes in functional parameters between these areas are quantified and the changes are visualized in two- or three-dimensional models. The approach allows detailed quantitative analysis without increasing user interaction or requiring complex algorithms. In addition, the use of concentric area analysis to evaluate changes in tumor volume is highlighted.
- US2021209757-A1 provides a system focused on monitoring the progression of arterial or plaque-based diseases, such as coronary artery disease. It uses automatic analysis of medical images obtained at different times, and compares parameters derived from these images.
- the system provides an automated disease tracking tool using noninvasive medical images as input.
- the system accesses a first set of plaque parameters derived from a medical image at a first time point. These parameters are stored in a database. It then accesses a second medical image at a later time and automatically derives a second set of parameters. Changes in one or more plaque parameters are analyzed between the two sets, such as radiographic density, volume, geometry, location, volume to surface area ratio, heterogeneity index, radiographic density composition, and others.
- the system can also analyze changes in non-image-based metrics, such as serum biomarkers, genetics, omics, transcriptomics, microbiomics, and metabolomics.
- the object of present invention is to provide a new solution for accurate lesion identification and tracking that enables capturing changes in the dynamics of the lesions over time.
- the primary application of the proposed solution is to provide a medical staff with a temporal evolution of various parameters potentially associated with the malignancy of lesions in a specific organ of interest. These parameters may include changes in diameter, volume, or contrast uptake over time.
- the present invention proposes, according to a first aspect, a computer implemented method for identification and temporal tracking of equivalent lesions in medical images.
- the method comprises performing by one or more processing units the following steps: receiving/accessing a first medical image of a subject, the first medical image being made at a first time instant; identifying an organ of interest and at least one lesion in the first medical image by segmenting the first medical image, providing a first annotation mask as a result, where the organ of interest and the lesion are identified differently in the first annotation mask; preprocessing the first medical image and the first annotation mask, and cropping the first medical image and the first annotation mask to a region of interest (ROI) encompassing the organ of interest and the lesion; extracting anatomical landmarks from the first medical image by implementing an automated landmark detection process using a deep learning model; encoding the first medical image and their extracted anatomical landmarks into a latent anatomical space representation using a neural network; designating the first medical image and the first annotation mask as a reference system; receiving/access
- Present invention also proposes, according to another aspect, a system for identification and temporal tracking of equivalent lesions in medical images.
- the system comprises a memory and one or more processing units that are configured to execute/perform the method of the first aspect. That is, the proposed method and system ensure that lesions that disappear in an intermediate time point are not permanently discarded, but remain in the reference system, allowing them to be reassigned if they reappear in later medical images. Additionally, the invention ensures that subsequent medical images are registered against the reference system rather than only against the immediately prior image, thereby maintaining a consistent anatomical coordinate frame and allowing for longitudinal tracking across all time points.
- this copy can be loaded with all the lesion history.
- the original can be kept immutable and can be accessed at any time in case the original lesions of the first annotation mask should be viewed later.
- the organ of interest can be the liver, the pancreas or at least one kidney.
- the medical images can comprise CT, MRI, SPECT, and/or ultrasound images.
- the method further performs an expansion and resampling process on the first and second ROI temporary binary masks and the first and second annotation masks to restore their original dimensions .
- the method also computes a diameter and volume of the lesions in the second annotation mask and in the first annotation mask that share the same identifier based on a parametrization process including associating each lesion’s volume and diameter with the corresponding lesion tracked over time.
- the method computes the volume of the lesion in the second annotation mask and of the lesion in the first annotation mask; executes a checking criteria to check whether the computed volumes are equal or greater than a given threshold; and performs a morphological transformation of the lesions not fulfilling the checking criteria by subjecting them to a binary dilation process.
- the binary dilation process comprises using a 3x3x3 mm spherical kernel.
- the organ of interest is identified with a first value and each lesion is identified with a value that is greater than the first value.
- the value of a first identified lesion can be lower than the value of at least one second identified lesion.
- the hybrid registration algorithm performs affine and deformable transformations and further comprises a transformation in the latent anatomical space to improve robustness against physiological variations of the organ.
- the hybrid registration algorithm maintains the first ROI temporary binary mask fixed and deforms the second ROI temporary binary mask to match the first annotation mask.
- the hybrid registration algorithm may comprise a symmetric normalization (SyN) transformation.
- SyN symmetric normalization
- the preprocessing of the first and second medical images and of the first and second annotation masks comprises an isotropic resampling thereof.
- a computer program product is one embodiment that has a computer-readable medium including computer program instructions encoded thereon that when executed on at least one processor in a computer system causes the processor to perform the operations indicated herein as embodiments of the invention.
- present invention automates and facilitates accurate identification and temporal tracking of each lesion within the same subject/patient by extracting organ geometry from consecutive image studies/scans (e.g. in DICOM format) and by aligning these geometries through advance image registration techniques, such that this alignment eliminates potential variabilities associated with patient positioning or changes in the organ morphology and enhances accuracy.
- lesion tracking is improved thanks to the use of a unique identifier for each lesion.
- Fig. 1 is a flow diagram showing the different steps that can be implemented for identifying and temporarily tracking lesions, according to different embodiments of the present invention.
- Fig. 2 graphically illustrates how equivalent individual lesions are identified and tracked.
- Present invention discloses a method and system to identify equivalent individual lesions and automate their spatial follow-up in a sequence of medical studies/scans (e.g. CTs, MRIs, etc.) by performing: affine registration between organs containing the lesion/s; flexible registration to adapt the edges of the prior steps based on a registration algorithm; and morphologic operation (dilation) performed on the output mask of the prior step.
- a sequence of medical studies/scans e.g. CTs, MRIs, etc.
- Fig. 1 shows all the steps the proposed method can execute/perform for identifying and temporally tracking lesions, according to some embodiments.
- the method begins by receiving/accessing a first medical image (e.g. a CT scan) of a given subject.
- a first medical image e.g. a CT scan
- the method can locate and chronologically organize them.
- the method can provide an annotation mask for the (first) medical image by identifying the organ of interest (e.g. the liver) and one or more lesions through segmentation of the medical image.
- the organ of interest and the lesion/s are identified in a different manner.
- the organ of interest can be identified with a first value and each lesion can be identified with a value greater than the first value in ascending order for each identified lesion.
- the background of the received/accessed medical image can also be identified, for example, with a value lower than the first value. That is, an annotation mask with liver and two different lesions can have values “0” for background, “1” for liver and “2” and “3” for the associated lesions.
- Both the annotation mask and the received/accessed medical image undergo preprocessing before lesion comparison.
- this preprocessing involves isotropic resampling to address potential differences in pixel spacing and slice thickness among different medical studies, resulting in standardized dimensions of [1,1,1]mm for each dimension in both medical images and annotation masks.
- the medical image and annotation mask are cropped to a region of interest (ROI) encompassing the organ of interest and associated lesion/s. This step aims to reduce computational loads and ensure proper subsequent registration, which can be understood as the image alignment to a reference space or reference geometry.
- an automated landmark detection process is performed using a deep learning model to extract anatomical landmarks from the (first) medical image.
- the image, along with the extracted landmarks, is encoded into a latent anatomical space representation via a neural network.
- the anatomical landmarks may include 3D coordinates corresponding to the bifurcation of the portal vein, the confluence of the hepatic veins into the inferior vena cava, and/or the fundus of the gallbladder.
- the first annotation mask is defined as the reference geometry, including the organ and lesion/s.
- This geometry (including, following the previous example, the organ as “1” and the lesion/s with values greater than “1”) becomes the reference system for all subsequent studies, which are registered for lesion comparison.
- the method can iterate over all available medical images of the subject.
- the method can create a ROI temporary binary mask for each annotation mask previously provided.
- both the organ and the lesion/s are identified in the same manner. That is, both organ and lesion/s can be set to “1”. In case the background is also identified, this can be set to “0”.
- a registration can then be performed between two ROI temporary binary masks, with the reference system held fixed and the ROI temporary binary mask of the study being analyzed deformed to match the reference system.
- a hybrid registration algorithm is used, which includes computing a transformation in a latent anatomical space obtained via a neural network encoding of the medical images and their automatically extracted anatomical landmarks. This transformation achieves a global alignment, which is subsequently refined using the anatomical landmarks to optimize local correspondences.
- This approach ensures a robust registration process that accounts for both global anatomical context and localized structural variations, minimizing the impact of temporal changes in the organ's geometry. It is worth noting that the organ’s geometry and consistency can vary over time due to aging, surgical interventions, or other factors.
- the hybrid registration algorithm can be based on Demons algorithm (when two images between which there are large local deformations are to be registered), Fluid Registration (models the changes between the images as a fluid flow and uses differential equations to calculate the displacement between one and the other), B-Spline (divides the image into a mesh and applies B-Spline functions to align the images and smooth the deformation), etc.
- the hybrid registration approach yields a transformation of the ROI temporary binary mask of the study being analyzed into the coordinates of the reference system.
- the similarity score is determined by combining two or more of the following strategies: DICE coefficient between lesion masks, distance between lesion centroids, Hausdorff distance, and a structural similarity index (SSIM) computed in the latent anatomical space.
- lesions with a volume less than a given threshold can particularly undergo binary dilation, for instance, using a 3x3x3 spherical kernel, to ensure proper alignment with their counterparts in the reference system.
- a matrix containing the similarity scores for each pair can be created. Pairs with the maximum similarity score are assigned the same identifier in the annotation mask of the study being analyzed as in the first annotation mask.
- each lesion can be also performed using image analysis techniques. Again, by iterating over each lesion in the scan being analyzed, morphological parameter such as diameter, mean HU value, volume, etc. or other more complex parameters such presence of calcifications, scars, etc. can be extracted. Being numbered with respect to the reference system's mask, the results of said parameters evolution with respect to the reference system, for example, can be presented to the medical staff.
- Certain aspects of the present invention include process steps or operations and instructions described herein in an algorithmic and/or algorithmic-like form. It should be noted that the process steps and/or operations and instructions of the present invention can be embodied in software, firmware, and/or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by real-time network operating systems.
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Apparatus For Radiation Diagnosis (AREA)
Abstract
A method, system, and computer program for identification and temporal tracking of equivalent lesions in medical images are proposed. The method comprises identifying an organ and a lesion in a first image; preprocessing the first image and the first annotation mask, and cropping them to a ROI; extracting anatomical landmarks from the first image; encoding the first image and landmarks into a latent space; designating the first image and mask as a reference system; identifying the organ and a lesion in a second image; preprocessing the second image and the second annotation mask, and cropping them to a ROI; creating a first ROI temporary mask for the first mask and a second ROI temporary mask for the second mask; extracting anatomical landmarks from the second image; encoding the second image and landmarks into the latent space; registering the ROI temporary masks; applying a transformation map to the second mask; computing a similarity score between the lesions in the two masks; and checking whether the second mask comprises a new lesion, and, if so, adding a segmentation of the new lesion to the reference system and assign a new identifier to the new lesion.
Description
METHOD, SYSTEM AND COMPUTER PROGRAM FOR IDENTIFICATION AND TEMPORAL TRACKING OF EQUIVALENT LESIONS IN MEDICAL IMAGES
TECHNICAL FIELD
The present invention relates to medical images analysis. More specifically, the invention relates to a method, system and computer program for identification and temporal tracking of equivalent lesions in medical images such as computed tomography (CT) images, Magnetic Resonance Images (IMR), ultrasound images, etc.
BACKGROUND OF THE INVENTION
Liver, pancreas or kidney lesions pose a significant clinical challenge due to their diverse presentation. The accurate identification and assessment of these anomalies are crucial for evaluating the potential associated risk.
A detailed analysis of the radiological characteristics of those lesions in individual medical studies/scans provides information at a specific moment but lacks the dynamic and evolutionary perspective required to assess the nature and potential risk of the lesion.
Moreover, the precise identification of the lesions in consecutive medical studies is a challenge for the medical staff. Variability in patient orientation between studies, the constant evolution of the organ morphology, and the possible appearance of new lesions complicate the tracking of each individual lesion over time.
There are known some patents and patent applications in this field.
EP3610456-A1 provides a method for determining volumetric properties of one or more lesions in medical images. The method comprises receiving image data; determining one or more locations of one or more lesions in the image data; creating an image segmentation (i.e. mask or contour) comprising the determined one or more locations of the one or more lesions in the image data and using the image segmentation to determine a volumetric property of the lesion. The method uses convolutional neural networks to identify the lesion locations.
EP2407927-B1 describes a method to evaluate the evolution of tumor lesions by segmentation into concentric areas and comparison of functional parameters over time. Images of lesions at different time points are provided, and lesions are delineated and registered. Subsequently, they are segmented into concentric areas, and changes in
functional parameters between these areas are quantified and the changes are visualized in two- or three-dimensional models. The approach allows detailed quantitative analysis without increasing user interaction or requiring complex algorithms. In addition, the use of concentric area analysis to evaluate changes in tumor volume is highlighted.
US2021209757-A1 provides a system focused on monitoring the progression of arterial or plaque-based diseases, such as coronary artery disease. It uses automatic analysis of medical images obtained at different times, and compares parameters derived from these images. The system provides an automated disease tracking tool using noninvasive medical images as input. The system accesses a first set of plaque parameters derived from a medical image at a first time point. These parameters are stored in a database. It then accesses a second medical image at a later time and automatically derives a second set of parameters. Changes in one or more plaque parameters are analyzed between the two sets, such as radiographic density, volume, geometry, location, volume to surface area ratio, heterogeneity index, radiographic density composition, and others. The system can also analyze changes in non-image-based metrics, such as serum biomarkers, genetics, omics, transcriptomics, microbiomics, and metabolomics.
Other solutions are also known by US11551353-B2, EP1910997-B1, EP2939217-B1 , EP3664024-A1 , WO2023/239829-A2, and US20130181976-A1.
On the other hand, the scientific publication “Graph-based automatic detection and classification of lesion changes in pairs of CT studies for oncology follow-up” by Shalom Rochman et al. discloses a comprehensive graph-based method for the automatic detection and classification of lesion changes between current and prior CT scans. The method takes as input the current and prior CT images, along with their corresponding organ and lesion segmentations. Lesion change classification is formalized as a bipartite graph matching problem, wherein lesion pairings are determined using an adaptive overlap-based matching approach. The method identifies six types of lesion changes through a connected components analysis.
Document “Liver lesion changes analysis in longitudinal CECT scans by simultaneous deep learning voxel classification with SimU-Nef’ by Adi Szeskin et al. describes a fully automatic end-to-end pipeline for liver lesion changes analysis in consecutive (prior and current) abdominal CECT scans of oncology patients. The method uses a SimU-Net, a simultaneous
multi-channel 3D R2U-Net model trained on pairs of registered scans of each patient that identifies the liver lesions and their changes based on the lesion and healthy tissue appearance differences. It also implements a model-based bipartite graph lesions matching method for the analysis of lesion changes at the lesion level, and a method for longitudinal analysis of one or more of consecutive scans of a patient based on Simll-Net that handles major liver deformations and incorporates lesion segmentations from previous analysis.
Document “Explicit B-spline regularization in diffeomorphic image registration", Nicholas J. Tustison and Brian B. Avants, discusses different methods of achieving accurate and smooth deformations in medical image registration, with a focus on diffeomorphic methods and B- spline enhancements.
DESCRIPTION OF THE INVENTION
The object of present invention is to provide a new solution for accurate lesion identification and tracking that enables capturing changes in the dynamics of the lesions over time.
The primary application of the proposed solution is to provide a medical staff with a temporal evolution of various parameters potentially associated with the malignancy of lesions in a specific organ of interest. These parameters may include changes in diameter, volume, or contrast uptake over time.
To that end, the present invention proposes, according to a first aspect, a computer implemented method for identification and temporal tracking of equivalent lesions in medical images. The method comprises performing by one or more processing units the following steps: receiving/accessing a first medical image of a subject, the first medical image being made at a first time instant; identifying an organ of interest and at least one lesion in the first medical image by segmenting the first medical image, providing a first annotation mask as a result, where the organ of interest and the lesion are identified differently in the first annotation mask; preprocessing the first medical image and the first annotation mask, and cropping the first medical image and the first annotation mask to a region of interest (ROI) encompassing the organ of interest and the lesion; extracting anatomical landmarks from the first medical image by implementing an automated landmark detection process using a deep learning model; encoding the first medical image and their extracted anatomical landmarks into a latent anatomical space representation using a neural network; designating the first medical image and the first annotation mask as a reference system; receiving/accessing a second medical image of the subject, the second medical image being made at a second time instant
that is later than the first time instant; identifying the organ of interest and at least one lesion in the second medical image by segmenting the second medical image, providing a second annotation mask as a result, where the organ of interest and the lesion are identified differently in the second annotation mask; preprocessing the second medical image and the second annotation mask, and cropping the second medical image and the second annotation mask to a ROI encompassing the organ of interest and the lesion; creating a first ROI temporary binary mask for the first annotation mask and a second ROI temporary binary mask for the second annotation mask, the organ of interest and the lesion being identified in the same manner in the first ROI temporary binary mask and in the second ROI temporary binary mask; extracting anatomical landmarks from the second medical image by implementing the automated landmark detection process using the deep learning model; encoding the second medical image and their extracted anatomical landmarks into the latent anatomical space representation using a neural network; registering the first ROI temporary binary mask with the second ROI temporary binary mask using a hybrid registration approach that includes computing a transformation in the latent anatomical space to achieve global alignment, and refining the transformation using the anatomical landmarks to optimize local correspondences, obtaining a transformation map as a result; applying the transformation map to the lesion in the second annotation mask; computing a similarity score between the lesion in the first annotation mask and the lesion in the registered second annotation mask by combining at least two of the following strategies: a DICE coefficient between lesion masks, a distance between lesion centroids, a Hausdorff distance, a structural similarity index, SSIM, computed in the latent anatomical space, assigning a same identifier in the second annotation mask as in the first annotation mask to the lesions having a maximum similarity score; and checking whether the second annotation mask comprises at least one new lesion that is not present in the reference system, and, if so, adding a segmentation of the new lesion to the reference system and assigning a new identifier to the new lesion, wherein the reference system accumulates and stores the complete history of all lesion identifiers and their corresponding transformations throughout the image series. Accordingly, a continuous tracking of the lesions is ensured over time.
Present invention also proposes, according to another aspect, a system for identification and temporal tracking of equivalent lesions in medical images. The system comprises a memory and one or more processing units that are configured to execute/perform the method of the first aspect.
That is, the proposed method and system ensure that lesions that disappear in an intermediate time point are not permanently discarded, but remain in the reference system, allowing them to be reassigned if they reappear in later medical images. Additionally, the invention ensures that subsequent medical images are registered against the reference system rather than only against the immediately prior image, thereby maintaining a consistent anatomical coordinate frame and allowing for longitudinal tracking across all time points.
When the new segmentation of the lesion has been added to a copy of the first annotation mask, during subsequent iterations, this copy can be loaded with all the lesion history. In this way, the original can be kept immutable and can be accessed at any time in case the original lesions of the first annotation mask should be viewed later.
According to the invention, the organ of interest can be the liver, the pancreas or at least one kidney. Likewise, the medical images can comprise CT, MRI, SPECT, and/or ultrasound images.
In some embodiments, the method further performs an expansion and resampling process on the first and second ROI temporary binary masks and the first and second annotation masks to restore their original dimensions .
In some embodiments, the method also computes a diameter and volume of the lesions in the second annotation mask and in the first annotation mask that share the same identifier based on a parametrization process including associating each lesion’s volume and diameter with the corresponding lesion tracked over time.
In some embodiments, after the registering step, the method computes the volume of the lesion in the second annotation mask and of the lesion in the first annotation mask; executes a checking criteria to check whether the computed volumes are equal or greater than a given threshold; and performs a morphological transformation of the lesions not fulfilling the checking criteria by subjecting them to a binary dilation process.
In some embodiments, the binary dilation process comprises using a 3x3x3 mm spherical kernel.
In some embodiments, in the identifying step, the organ of interest is identified with a first value and each lesion is identified with a value that is greater than the first value. The value of a first identified lesion can be lower than the value of at least one second identified lesion.
In some embodiments, the hybrid registration algorithm performs affine and deformable transformations and further comprises a transformation in the latent anatomical space to improve robustness against physiological variations of the organ.
In some embodiments, the hybrid registration algorithm maintains the first ROI temporary binary mask fixed and deforms the second ROI temporary binary mask to match the first annotation mask.
In some embodiments, the hybrid registration algorithm may comprise a symmetric normalization (SyN) transformation.
In some embodiments, the preprocessing of the first and second medical images and of the first and second annotation masks comprises an isotropic resampling thereof.
Other embodiments of the invention that are disclosed herein also include software programs to perform the method embodiment steps and operations summarized above and disclosed in detail below. More particularly, a computer program product is one embodiment that has a computer-readable medium including computer program instructions encoded thereon that when executed on at least one processor in a computer system causes the processor to perform the operations indicated herein as embodiments of the invention.
Consequently, present invention automates and facilitates accurate identification and temporal tracking of each lesion within the same subject/patient by extracting organ geometry from consecutive image studies/scans (e.g. in DICOM format) and by aligning these geometries through advance image registration techniques, such that this alignment eliminates potential variabilities associated with patient positioning or changes in the organ morphology and enhances accuracy. Likewise, lesion tracking is improved thanks to the use of a unique identifier for each lesion.
BRIEF DESCRIPTION OF THE DRAWINGS
The previous and other advantages and features will be more fully understood from the following detailed description of embodiments, with reference to the attached figures, which must be considered in an illustrative and non-limiting manner, in which:
Fig. 1 is a flow diagram showing the different steps that can be implemented for identifying and temporarily tracking lesions, according to different embodiments of the present invention.
Fig. 2 graphically illustrates how equivalent individual lesions are identified and tracked.
DETAILED DESCRIPTION OF THE INVENTION AND OF PREFERRED EMBODIMENTS
Present invention discloses a method and system to identify equivalent individual lesions and automate their spatial follow-up in a sequence of medical studies/scans (e.g. CTs, MRIs, etc.) by performing: affine registration between organs containing the lesion/s; flexible registration to adapt the edges of the prior steps based on a registration algorithm; and morphologic operation (dilation) performed on the output mask of the prior step.
Fig. 1 shows all the steps the proposed method can execute/perform for identifying and temporally tracking lesions, according to some embodiments.
In some of these embodiments, the method begins by receiving/accessing a first medical image (e.g. a CT scan) of a given subject. In case several medical images of the subject are available, the method can locate and chronologically organize them.
Alongside the medical image, corresponding annotations thereof can be loaded in mask format. That is, the method can provide an annotation mask for the (first) medical image by identifying the organ of interest (e.g. the liver) and one or more lesions through segmentation of the medical image. Particularly, in the annotation mask, the organ of interest and the lesion/s are identified in a different manner. For instance, the organ of interest can be identified with a first value and each lesion can be identified with a value greater than the first value in ascending order for each identified lesion. Optionally, the background of the received/accessed medical image can also be identified, for example, with a value lower than the first value. That is, an annotation mask with liver and two different lesions can have values “0” for background, “1” for liver and “2” and “3” for the associated lesions.
Both the annotation mask and the received/accessed medical image undergo preprocessing before lesion comparison. In some embodiments, this preprocessing involves isotropic resampling to address potential differences in pixel spacing and slice thickness among different medical studies, resulting in standardized dimensions of [1,1,1]mm for each dimension in both medical images and annotation masks. Additionally, the medical image and annotation mask are cropped to a region of interest (ROI) encompassing the organ of interest and associated lesion/s. This step aims to reduce computational loads and ensure proper subsequent registration, which can be understood as the image alignment to a reference space or reference geometry.
Then, an automated landmark detection process is performed using a deep learning model to extract anatomical landmarks from the (first) medical image. The image, along with the extracted landmarks, is encoded into a latent anatomical space representation via a neural network. For example, in the case of the liver, the anatomical landmarks may include 3D coordinates corresponding to the bifurcation of the portal vein, the confluence of the hepatic veins into the inferior vena cava, and/or the fundus of the gallbladder.
Afterwards, the first annotation mask is defined as the reference geometry, including the organ and lesion/s. This geometry (including, following the previous example, the organ as “1” and the lesion/s with values greater than “1”) becomes the reference system for all subsequent studies, which are registered for lesion comparison.
With the reference system established, the method can iterate over all available medical images of the subject.
The method can create a ROI temporary binary mask for each annotation mask previously provided. In the ROI temporary binary masks, particularly, both the organ and the lesion/s are identified in the same manner. That is, both organ and lesion/s can be set to “1”. In case the background is also identified, this can be set to “0”.
A registration can then be performed between two ROI temporary binary masks, with the reference system held fixed and the ROI temporary binary mask of the study being analyzed deformed to match the reference system. In a particular embodiment, a hybrid registration algorithm is used, which includes computing a transformation in a latent anatomical space obtained via a neural network encoding of the medical images and their automatically extracted anatomical landmarks. This transformation achieves a global alignment, which is subsequently refined using the anatomical landmarks to optimize local correspondences. This approach ensures a robust registration process that accounts for both global anatomical context and localized structural variations, minimizing the impact of temporal changes in the organ's geometry. It is worth noting that the organ’s geometry and consistency can vary over time due to aging, surgical interventions, or other factors.
In other embodiments, the hybrid registration algorithm can be based on Demons algorithm (when two images between which there are large local deformations are to be registered), Fluid Registration (models the changes between the images as a fluid flow and uses differential equations to calculate the displacement between one and the other), B-Spline
(divides the image into a mesh and applies B-Spline functions to align the images and smooth the deformation), etc.
The hybrid registration approach yields a transformation of the ROI temporary binary mask of the study being analyzed into the coordinates of the reference system.
With all lesions positioned in the same reference system, they can be compared with those already present in the reference system. By iterating over all annotation masks, a similarity score is calculated for each pair of lesions in the study being analyzed and the reference system. According to the invention, the similarity score is determined by combining two or more of the following strategies: DICE coefficient between lesion masks, distance between lesion centroids, Hausdorff distance, and a structural similarity index (SSIM) computed in the latent anatomical space.
In some embodiments, given potential inaccuracies in registrations, lesions with a volume less than a given threshold (e.g. 200 pixels) can particularly undergo binary dilation, for instance, using a 3x3x3 spherical kernel, to ensure proper alignment with their counterparts in the reference system. A matrix containing the similarity scores for each pair can be created. Pairs with the maximum similarity score are assigned the same identifier in the annotation mask of the study being analyzed as in the first annotation mask.
Furthermore, if there are lesions not present in the reference system, they are added, accumulating the entire lesion history of the subject. This process involves adding the segmentation of the new lesion to the reference system, assigning it a unique identifier not already present in the reference geometry.
With the annotation mask of the study being analyzed having identifiers identical to those of the reference system in case of coincident lesions, individual parameterization of each lesion can be also performed using image analysis techniques. Again, by iterating over each lesion in the scan being analyzed, morphological parameter such as diameter, mean HU value, volume, etc. or other more complex parameters such presence of calcifications, scars, etc. can be extracted. Being numbered with respect to the reference system's mask, the results of said parameters evolution with respect to the reference system, for example, can be presented to the medical staff.
The present invention has been described in particular detail with respect to specific possible embodiments. Those of skill in the art will appreciate that the invention may be practiced in
other embodiments. For example, the nomenclature used for components, capitalization of component designations and terms, the attributes, data structures, or any other programming or structural aspect is not significant, mandatory, or limiting, and the mechanisms that implement the invention or its features can have various different names, formats, and/or protocols. Further, the system and/or functionality of the invention may be implemented via various combinations of software and hardware, as described, or entirely in software elements. Also, particular divisions of functionality between the various components described herein are merely exemplary, and not mandatory or significant. Consequently, functions performed by a single component may, in other embodiments, be performed by multiple components, and functions performed by multiple components may, in other embodiments, be performed by a single component.
Certain aspects of the present invention include process steps or operations and instructions described herein in an algorithmic and/or algorithmic-like form. It should be noted that the process steps and/or operations and instructions of the present invention can be embodied in software, firmware, and/or hardware, and when embodied in software, can be downloaded to reside on and be operated from different platforms used by real-time network operating systems.
The scope of the present invention is defined in the following set of claims.
Claims
1. A computer implemented method for identification and temporal tracking of equivalent lesions in medical images, the method comprising performing by one or more processing units the following steps: receiving/accessing a first medical image of a subject, the first medical image being made at a first time instant; identifying an organ of interest and at least one lesion in the first medical image by segmenting the first medical image, providing a first annotation mask as a result, where the organ of interest and the lesion are identified differently in the first annotation mask; preprocessing the first medical image and the first annotation mask, and cropping the first medical image and the first annotation mask to a region of interest, ROI, encompassing the organ of interest and the lesion; extracting anatomical landmarks from the first medical image by implementing an automated landmark detection process using a deep learning model; encoding the first medical image and their extracted anatomical landmarks into a latent anatomical space representation using a neural network; designating the first medical image and first annotation mask as a reference system; receiving/accessing a second medical image of the subject, the second medical image being made at a second time instant that is later than the first time instant; identifying the organ of interest and at least one lesion in the second medical image by segmenting the second medical image, providing a second annotation mask as a result, where the organ of interest and the lesion are identified differently in the second annotation mask; preprocessing the second medical image and the second annotation mask, and cropping the second medical image and the second annotation mask to a ROI encompassing the organ of interest and the lesion; creating a first ROI temporary binary mask for the first annotation mask and a second ROI temporary binary mask for the second annotation mask, the organ of interest and the lesion being identified in the same manner in the first ROI temporary binary mask and in the second ROI temporary binary mask; extracting anatomical landmarks from the second medical image by implementing the automated landmark detection process using the deep learning model; encoding the second medical image and their extracted anatomical landmarks into the latent anatomical space representation using a neural network;
registering the first and second ROI temporary binary masks using a hybrid registration approach comprising computing a transformation in the latent anatomical space to achieve global alignment, and refining the transformation using the anatomical landmarks to optimize local correspondences, obtaining a transformation map as a result; applying the transformation map to the lesion in the second annotation mask; computing a similarity score between the lesion in the first annotation mask and the lesion in the second annotation mask by combining at least two of the following strategies: a DICE coefficient between lesion masks, a distance between lesion centroids, a Hausdorff distance, a structural similarity index, SSIM, computed in the latent anatomical space; assigning a same identifier in the second annotation mask as in the first annotation mask to the lesions having a maximum similarity score; and checking whether the second annotation mask comprises at least one new lesion that is not present in the reference system, and, if so, adding a segmentation of the new lesion to the reference system and assigning a new identifier to the new lesion, the reference system accumulating and preserving the full history of all lesion identifiers and their corresponding transformation throughout the images, thereby ensuring the continuous tracking of lesions over time.
2. The method of claim 1 , further comprising computing a diameter and volume of the lesions in the second annotation mask and in the first annotation mask that have the same identifier based on a parametrization process comprising associating each lesion’s volume and diameter with the corresponding lesion tracked over time.
3. The method of any one of the previous claims, wherein after the registering step the method further comprises: computing a volume of the lesion in the second annotation mask and of the lesion in the first annotation mask; executing a checking criteria to check whether the computed volumes are equal or greater than a given threshold; and performing a morphological transformation of the lesions not fulfilling the checking criteria by subjecting them to a binary dilation process.
4. The method of claim 3, wherein the binary dilation process comprises using a 3x3x3 mm spherical kernel.
5. The method of any one of the previous claims, wherein in the identifying step:
the organ of interest is identified with a first value and each lesion is identified with a value that is greater than the first value, and the value of a first identified lesion is lower than the value of at least one second identified lesion.
6. The method of any one of the previous claims, wherein the hybrid registration algorithm performs affine and deformable transformations, and further comprises a transformation in the latent anatomical space to improve robustness against physiological variations of the organ.
7. The method of any one of the previous claims, wherein the hybrid registration algorithm maintains the first ROI temporary binary mask fixed and deforms the second ROI temporary binary mask to match the first annotation mask.
8. The method of any one of the previous claims, wherein the similarity score is computed by combining the four strategies.
9. The method of any one of the previous claims, wherein the preprocessing of the first and second medical images and of the first and second annotation masks comprises an isotropic resampling thereof.
10. The method of any one of the previous claims, wherein the organ of interest is a liver, a pancreas or at least one kidney.
11. The method of any one of the previous claims, wherein the medical images comprise Computed Tomography, CT, images.
12. A system for identification and temporal tracking of equivalent lesions in medical images, comprising a memory and one or more processing units, the processing units being configured to: receive/access a first medical image of a subject, the first medical image being made at a first time instant; identify an organ of interest and at least one lesion in the first medical image by segmenting the first medical image, providing a first annotation mask as a result, where the organ of interest and the lesion are identified differently in the first annotation mask; preprocess the first medical image and the first annotation mask, and crop the first medical image and the first annotation mask to a region of interest, ROI, encompassing the organ of interest and the lesion;
extract anatomical landmarks from the first medical image by implementing an automated landmark detection process using a deep learning model; encode the first medical image and their extracted anatomical landmarks into a latent anatomical space representation using a neural network; designate the first medical image and first annotation mask as a reference system; receive/access a second medical image of the subject, the second medical image being made at a second time instant that is later than the first time instant; identify the organ of interest and at least one lesion in the second medical image by segmenting the second medical image, providing a second annotation mask as a result, where the organ of interest and the lesion are identified differently in the second annotation mask; preprocess the second medical image and the second annotation mask, and crop the second medical image and the second annotation mask to a ROI encompassing the organ of interest and the lesion; create a first ROI temporary binary mask for the first annotation mask and a second ROI temporary binary mask for the second annotation mask, the organ of interest and the lesion being identified in the same manner in the first ROI temporary binary mask and in the second ROI temporary binary mask; extract anatomical landmarks from the second medical image by implementing the automated landmark detection process using the deep learning model; encode the second medical image and their extracted anatomical landmarks into the latent anatomical space representation using a neural network; register the first ROI temporary binary mask and the second ROI temporary binary mask using a hybrid registration approach that comprises computing a transformation in the latent anatomical space to achieve global alignment, and refining the transformation using the anatomical landmarks to optimize local correspondences, obtaining a transformation map as a result; apply the transformation map to the second annotation mask; compute a similarity score between the lesion in the first annotation mask and the lesion in the second annotation mask by combining at least two of the following strategies: a DICE coefficient between lesion masks, a distance between lesion centroids, a Hausdorff distance, a structural similarity index, SSIM, computed in the latent anatomical space; assign a same identifier in the second annotation mask as in the first annotation mask to the lesions having a maximum similarity score; and
check whether the second annotation mask comprises at least one new lesion that is not present in the reference system, and, if so, add a segmentation of the new lesion to the reference system and assign a new identifier to the new lesion, the reference system accumulating and preserving the full history of all lesion identifiers and their corresponding transformation throughout the images.
13. A non-transitory computer readable medium comprising code instructions that when executed by a computing system implement the method of any of claims 1 to 11.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP24382427.3 | 2024-04-22 | ||
| EP24382427 | 2024-04-22 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2025223980A1 true WO2025223980A1 (en) | 2025-10-30 |
Family
ID=90829274
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/EP2025/060617 Pending WO2025223980A1 (en) | 2024-04-22 | 2025-04-17 | Method, system and computer program for identification and temporal tracking of equivalent lesions in medical images |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2025223980A1 (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2407927B1 (en) | 2010-07-16 | 2013-01-30 | BVBA dr. K. Coenegrachts | A method and device for evaluating evolution of tumoral lesions |
| US20130181976A1 (en) | 2006-10-27 | 2013-07-18 | Carl Zeiss Meditec, Inc. | User interface for efficiently displaying relevant oct imaging data |
| EP1910997B1 (en) | 2005-08-01 | 2019-11-20 | Bioptigen, Inc. | Methods, systems and computer program for 3d-registration of three dimensional data sets obtained by preferably optical coherence tomography based on the alignment of projection images or fundus images, respectively |
| EP3610456A1 (en) | 2017-04-11 | 2020-02-19 | Kheiron Medical Technologies Ltd | Recist assessment of tumour progression |
| EP3664024A1 (en) | 2004-06-18 | 2020-06-10 | Siemens Healthcare GmbH | System and method for linking vois across timepoints for analysis of disease progression or response to therapy |
| US20210209757A1 (en) | 2020-01-07 | 2021-07-08 | Cleerly, Inc. | Systems, methods, and devices for medical image analysis, diagnosis, risk stratification, decision making and/or disease tracking |
| EP2939217B1 (en) | 2012-12-27 | 2022-06-22 | Koninklijke Philips N.V. | Computer-aided identification of a tissue of interest |
| US11551353B2 (en) | 2017-11-22 | 2023-01-10 | Arterys Inc. | Content based image retrieval for lesion analysis |
| WO2023239829A2 (en) | 2022-06-08 | 2023-12-14 | Progenics Pharmaceuticals, Inc. | Systems and methods for assessing disease burden and progression |
-
2025
- 2025-04-17 WO PCT/EP2025/060617 patent/WO2025223980A1/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3664024A1 (en) | 2004-06-18 | 2020-06-10 | Siemens Healthcare GmbH | System and method for linking vois across timepoints for analysis of disease progression or response to therapy |
| EP1910997B1 (en) | 2005-08-01 | 2019-11-20 | Bioptigen, Inc. | Methods, systems and computer program for 3d-registration of three dimensional data sets obtained by preferably optical coherence tomography based on the alignment of projection images or fundus images, respectively |
| US20130181976A1 (en) | 2006-10-27 | 2013-07-18 | Carl Zeiss Meditec, Inc. | User interface for efficiently displaying relevant oct imaging data |
| EP2407927B1 (en) | 2010-07-16 | 2013-01-30 | BVBA dr. K. Coenegrachts | A method and device for evaluating evolution of tumoral lesions |
| EP2939217B1 (en) | 2012-12-27 | 2022-06-22 | Koninklijke Philips N.V. | Computer-aided identification of a tissue of interest |
| EP3610456A1 (en) | 2017-04-11 | 2020-02-19 | Kheiron Medical Technologies Ltd | Recist assessment of tumour progression |
| US11551353B2 (en) | 2017-11-22 | 2023-01-10 | Arterys Inc. | Content based image retrieval for lesion analysis |
| US20210209757A1 (en) | 2020-01-07 | 2021-07-08 | Cleerly, Inc. | Systems, methods, and devices for medical image analysis, diagnosis, risk stratification, decision making and/or disease tracking |
| WO2023239829A2 (en) | 2022-06-08 | 2023-12-14 | Progenics Pharmaceuticals, Inc. | Systems and methods for assessing disease burden and progression |
Non-Patent Citations (6)
| Title |
|---|
| ADI SZESKIN, LIVER LESION CHANGES ANALYSIS IN LONGITUDINAL CECT SCANS BY SIMULTANEOUS DEEP LEARNING VOXEL CLASSIFICATION WITH SIMU-NET |
| NICHOLAS J. TUSTISONBRIAN B., EXPLICIT B-SPLINE REGULARIZATION IN DIFFEOMORPHIC IMAGE REGISTRATION |
| ROCHMAN SHALOM ET AL: "Graph-based automatic detection and classification of lesion changes in pairs of CT studies for oncology follow-up", INTERNATIONAL JOURNAL OF COMPUTER ASSISTED RADIOLOGY AND SURGERY; A JOURNAL FOR INTERDISCIPLINARY RESEARCH, DEVELOPMENT AND APPLICATIONS OF IMAGE GUIDED DIAGNOSIS AND THERAPY, vol. 19, no. 2, 4 August 2023 (2023-08-04), Berlin, DE, pages 241 - 251, XP093149686, ISSN: 1861-6429, DOI: 10.1007/s11548-023-03000-2 * |
| SHALOM ROCHMAN, GRAPH-BASED AUTOMATIC DETECTION AND CLASSIFICATION OF LESION CHANGES IN PAIRS OF CT STUDIES FOR ONCOLOGY FOLLOW-UP |
| TIAN LIN ET AL: "SAME++: A medical image registration framework enhanced via self-supervised anatomical embeddings", 25 February 2024 (2024-02-25), pages 1 - 11, XP093288681, Retrieved from the Internet <URL:https://arxiv.org/pdf/2311.14986> [retrieved on 20240225] * |
| VEROLI BENIAMIN DI ET AL: "A graph-theoretic approach for the analysis of lesion changes and lesions detection review in longitudinal oncological imaging", MEDICAL IMAGE ANALYSIS, 14 July 2024 (2024-07-14), GB, pages 103268, XP093288656, ISSN: 1361-8415, Retrieved from the Internet <URL:https://pdf.sciencedirectassets.com/272154/1-s2.0-S1361841524X00055/1-s2.0-S1361841524001932/main.pdf?hash=2764449b82e7ffd7693bfa0baca2617cbbea3c006e7fb4638183f824c2122837&host=68042c943591013ac2b2430a89b270f6af2c76d8dfd086a07176afe7c76c2c61&pii=S1361841524001932&tid=spdf-fdba6a9b-8d1e-4c86-b656-35f> [retrieved on 20240714], DOI: 10.1016/j.media.2024.103268 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zuluaga et al. | Multi-atlas propagation whole heart segmentation from MRI and CTA using a local normalised correlation coefficient criterion | |
| Linguraru et al. | Automated segmentation and quantification of liver and spleen from CT images using normalized probabilistic atlases and enhancement estimation | |
| US8953856B2 (en) | Method and system for registering a medical image | |
| EP1851722B1 (en) | Image processing device and method | |
| EP2365471B1 (en) | Diagnosis assisting apparatus, coronary artery analyzing method and recording medium having a coronary artery analzying program stored therein | |
| EP3352135B1 (en) | Method and apparatus for segmentation of blood vessels | |
| Wimmer et al. | A generic probabilistic active shape model for organ segmentation | |
| US20190197688A1 (en) | Similar case image search program, similar case image search apparatus, and similar case image search method | |
| CN111340756B (en) | Medical image lesion detection merging method, system, terminal and storage medium | |
| KR101028365B1 (en) | Pulmonary nodule multilevel matching method and apparatus for continuous computed tomography | |
| US12175676B2 (en) | Apparatus for identifying regions in a brain image | |
| EP4369290A1 (en) | Determining estimates of hemodynamic properties based on an angiographic x-ray examination | |
| KR20240147617A (en) | Method and apparatus for assessment of changes in quantitative diagnostic assistant information for regional airway | |
| Larrey-Ruiz et al. | Automatic image-based segmentation of the heart from CT scans | |
| US8306354B2 (en) | Image processing apparatus, method, and program | |
| Tahoces et al. | Deep learning method for aortic root detection | |
| WO2015078980A2 (en) | Method and system for determining the prognosis of a patient suffering from pulmonary embolism | |
| CN115797308A (en) | DCE-MRI-based breast tumor segmentation method | |
| Tan et al. | Automatic localization of the left ventricular blood pool centroid in short axis cardiac cine MR images | |
| WO2025223980A1 (en) | Method, system and computer program for identification and temporal tracking of equivalent lesions in medical images | |
| US20220398735A1 (en) | Method and system for automated processing, registration, segmentation, analysis, validation, and visualization of structured and unstructured data | |
| JP2002291733A (en) | Image diagnosis support method and system therefor | |
| Buongiorno et al. | Automatic quantification of left atrium volume for cardiac rhythm analysis leveraging 3D residual UNet for time-varying segmentation of ECG-gated CT | |
| Kurugol et al. | Centerline extraction with principal curve tracing to improve 3D level set esophagus segmentation in CT images | |
| BE1029246B1 (en) | Computer-implemented method for estimating an input function of a blood vein based on a sequence of three-dimensional volumetric scan images, a data processing device, a computer program product, and a computer-readable storage medium therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 25719422 Country of ref document: EP Kind code of ref document: A1 |