[go: up one dir, main page]

WO2025159640A1 - Procédé d'évaluation d'un risque de récidive du cancer du sein - Google Patents

Procédé d'évaluation d'un risque de récidive du cancer du sein

Info

Publication number
WO2025159640A1
WO2025159640A1 PCT/NL2025/050040 NL2025050040W WO2025159640A1 WO 2025159640 A1 WO2025159640 A1 WO 2025159640A1 NL 2025050040 W NL2025050040 W NL 2025050040W WO 2025159640 A1 WO2025159640 A1 WO 2025159640A1
Authority
WO
WIPO (PCT)
Prior art keywords
output
machine learning
image
patch
data processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/NL2025/050040
Other languages
English (en)
Inventor
Ebrahim HAZRATI
Mert KILIÇKAYA
Roque Rodriguez OUTEIRAL
Brian Michael HOXENG
Marc Arnaud HOGENBIRK
Diederik WEHKAMP
Annuska Maria Glas
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Agendia NV
Original Assignee
Agendia NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agendia NV filed Critical Agendia NV
Publication of WO2025159640A1 publication Critical patent/WO2025159640A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30024—Cell structures in vitro; Tissue sections in vitro
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the invention relates to a computer-implemented method for assessing a risk of breast cancer recurrence based on image data. Further, the invention relates to a computer program and a data-processing system for performing the method of assessing the risk of breast cancer recurrence based on image data.
  • BACKGROUND OF THE INVENTION Worldwide, breast cancer is the leading type of cancer in women, accounting for 25% of all cases. In 2018, it resulted in 2 million new cases and 627,000 deaths. Outcomes for breast cancer vary depending on the cancer type, the extent of disease, and the person's age.
  • MammaPrint is a routine prognostic test used to assess the risk of breast cancer recurrence (van ’t Veer et al., 2002. Nature 415: 530–536). MammaPrint comprises a set of distinct breast cancer marker genes.
  • AI encompasses a variety of computer aided techniques that are used to solve highly complex tasks. More specifically, AI typically refers to computers excelling in tasks that are commonly associated with human cognition/behaviors. Especially the advancement of deep learning in the recent years has led to great progress in a variety of tasks.
  • Some examples are image and speech recognition, natural language processing and pattern recognition.
  • AI is increasingly used to help physicians to develop more personalized treatment plans for their patients.
  • the invention solves the problem to provide a reliable and accurate as well as fast and cost-effective prognostic test to assess a risk of breast cancer recurrence.
  • the invention refers to an AI-based method to extract diagnostic and prognostic information from imaged slides of breast cancer tissue and to output a breast cancer recurrence score.
  • the invention circumvents the use of microarray-based analysis of gene expression levels in breast cancer samples.
  • SUMMARY OF THE INVENTION The present invention provides reliable and accurate assessment of the risk of breast cancer recurrence by analyzing images of stained tissues comprising breast cancer cells.
  • the invention provides a computer-implemented method for assessing a risk of breast cancer recurrence, the method comprises the steps of a ) inputting an image of a stained tissue sample comprising breast cancer cells from an individual into a processor; b ) pre-processing the image in order to achieve a tissue segment image of relevant image areas by the processor; c ) patch extraction, wherein N patches of the pre-processed tissue segment image are generated by the processor; d ) encoding using at least one machine learning data processing model the extracted N patches into a set of feature vectors, wherein the input image is associated with a set of N feature vectors by the processor; and e ) outputting using at least one further machine learning data processing model an overall label output predicting a breast cancer recurrence status by aggregating
  • the image used in the computer-implemented method of the invention is a whole slide image (WSI).
  • pre-processing of the image according to the invention comprises extracting uninformative image content comprising background and/or adipose tissue.
  • the pre-processing of the image comprises the steps of: a ) down-sampling of the image by a factor s; b) transforming the image to greyscale; c) transforming each pixel-value of the image to optical density (OD); d) creating a mask of pixels by defining an OD threshold level, preferably the OD threshold is > 0.1; e) excluding pixels at the boundary, preferably the boundary region is defined by 0.03*min (width, height); f) excluding objects which are smaller than 0.1 – 1* area largest object in mask; g) excluding objects of a certain length, preferably wherein objects are excluded of a length of (Lmax – Lmin)/ Lmax > 1.5 to 0.5, more preferably of a length of (Lmax – Lmin)/ Lmax > 0.95; and/or h) excluding marking, preferably wherein the marking is defined by a variance across RGB channels ⁇ 0.001.
  • the down-sampling factor s for example matches one dimension of the size of the extracted patches, such as having a down-sampling factor s of 256 for extracted patches of 256 x 256.
  • the computer-implemented method comprises that each patch is transformed to a feature vector of a certain length.
  • each feature vector encodes textural and/or morphological information of each patch.
  • the computer-implemented method comprises that after patch extraction, an additional quality control step is performed.
  • said additional quality control step comprises, for every patch, a tissue type classification and/or nuclei detection.
  • the encoding of the feature vectors is performed using a convolutional neural network (CNN) comprising an input layer, one or more hidden layers and an output layer, such as a ResNet50.
  • CNN convolutional neural network
  • the encoding of the feature vectors is performed by using a Resnet50 having a length 1024.
  • the encoding of the feature vectors is performed using self- supervised learning.
  • the at least one further machine learning data processing model used for outputting of the overall label output is at least one fully-connected network (FCN) comprising a gated attention mechanism.
  • the at least one further machine learning data processing model is an adapted version of the attention-based multiple Instance learning (A-MIL) model.
  • outputting the overall label output comprises the steps of: a) inputting the N feature vectors; b) propagating the input N feature vectors through a linear layer of a certain length, such as a length is 512; c) performing Rectified Linear Unit (ReLU) activation and Dropout; d) processing the resulting features by at least two parallel streams of gated attention mechanism, preferably wherein the first stream consists of a linear layer of length 384, followed by tangent hyperbolic (Tanh) activation and Dropout and the second stream consists of a linear layer of length 384, followed by Sigmoid activation and Dropout; e) multiplying the output of the at least two attention layers element- wise with each other; f) passing the multiplied result through the last layer of the gated attention, wherein the last layer is a
  • the computer-implemented method further comprises a) providing a slide-level prediction index predicted by a Patch-based End-to-End Module from the pre-processed image patches; and b) combining the overall label output and the output of step a) to generate a refined overall label output by using a further machine learning data processing model, preferably wherein the further machine learning data processing model used for combining is at least one further fully-connected network (FCN), more preferably wherein the at least one further FCN is an ensemble model.
  • FCN fully-connected network
  • combining of the overall label output and the image output of a low-resolution image version comprises the steps of: a) inputting the overall label output and the output of the Patch-based End-to-End Module in an ensemble model, comprising at least three linear layers, wherein each layer except the last layer is followed by ReLU activation and Dropout; and b) generating a refined overall label output.
  • the staining of the tissue sample which is used as image input in the method of the invention is heamatoxylin and eosin (H&E).
  • the invention also comprises training of the at least one machine learning data processing model used for encoding of feature vectors, outputting the overall label output, outputting the image output of a low-resolution image version and/or outputting the refined overall label output.
  • the training comprises the steps of: a) obtaining as ground truth a set of N patches having corresponding encoded N feature vectors; b) inputting as example data the set of N patches into a processor comprising the machine learning data processing model used for encoding the extracted N patches into a set of N feature vectors; c) performing the method of the invention comprising at least one machine learning data processing model of claim 1D on the input set of extracted N patches; d) measuring the error between the generated feature vectors of the extracted patches and the ground truth data; and e) updating the weights of the machine learning data processing model used for feature vector encoding for performing the training of the machine learning data processing model, preferably wherein the weights of the machine learning data processing model used for feature vector encoding are updated using backpropagation
  • training the method according to the invention for example comprises training the at least one machine learning data processing model used for encoding the extracted N patches into a set of feature vectors, wherein the training comprises the steps of: a) obtaining as ground truth data a set of N feature vectors having a corresponding label providing information about a risk of breast cancer recurrence in a tissue sample, preferably the label is a result of gene expression profiling; b) inputting as example data the set of N feature vectors s into a processor comprising the machine learning data processing model used for outputting the overall label output; c) performing the method according to step e) of claim 1 on the input set of N feature vectors; d) measuring the error between the generated overall label output and the ground truth data; and e) updating the weights of the at least one machine learning data processing model used for used for outputting the overall label output for performing the training of the machine learning data processing model, preferably wherein the weights of the at least one further machine learning data processing model used for outputting the overall label output are updated
  • training the method according to the invention for example comprises training the at least one further machine learning data processing model used for outputting the output of the Patch-based End-to-End Module, wherein the training comprises the steps of: a) obtaining as ground truth data an output of the Patch-based End-to-End Module of a stained tissue sample comprising breast cancer cells having a corresponding label providing information about a risk of breast cancer recurrence in the tissue sample, preferably the label is a result of gene expression profiling; b) inputting as example data the output of the Patch-based End-to-End Module of a stained tissue sample into a processor comprising the machine learning data processing model used for outputting the output of the Patch-based End-to-End Module; c) performing the method according to step a) of claim 12; d) measuring the error between the generated output of the Patch-based End-to- End Module and the ground truth data; and e) updating the weights of the machine learning data processing model used for outputting output of the Patch-based End-to-En
  • training the method according to the invention for example comprises training the machine learning data processing model used for outputting the refined overall label output, wherein the training comprises the steps of: a) obtaining as ground truth data an overall label output and an output of the Patch-based End-to-End Module having a corresponding label providing information about a risk of breast cancer recurrence of an image a stained tissue sample comprising breast cancer cells, preferably the label is a result of gene expression profiling; b) inputting as example data the overall label output and the output of the Patch-based End-to-End Module of stained tissue sample; c) performing the method according to step b) of claim 12 on the input overall label output and the input of the output of the Patch-based End-to-End Module of stained tissue sample; d) measuring the error between the generated refined overall label output and the ground truth data; and e) updating the weights of the machine learning data processing model used for outputting the refined label output for performing the training of the machine learning data processing model, preferably wherein the weights of the machine learning
  • the invention is also directed to a computer program having instructions which when executed by a computing device or system cause the computer device or system to perform the computer-implemented method according to the invention.
  • the invention is also directed to a data-processing system comprising means for carrying out the computer-implemented method according to the invention.
  • the invention also comprises a method of treating an individual who has been assessed as having a high risk of breast cancer recurrence according to the computer-implemented method, computer program product and/or data-processing system of the invention.
  • the method of treatment according to the invention comprises assessing the risk of breast cancer recurrence in an individual according to the computer-implemented method of the invention, and providing a therapy for preventing and/or treating breast cancer to the individual assessed as having a high risk of breast cancer recurrence.
  • BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 Exemplary schematic overview of methods of the invention.
  • A-MIL Attention-based Multiple instance learning
  • a down-sampled low-resolution image of the WSI is passed through a ResNet50 classifier (“Tiny WSI model”).
  • A-MIL and the ResNet50 classifier are both trained to output a label predicting a low/high risk of breast cancer recurrence.
  • the output of the Core AI model (overall label output) and the output of the Tiny WSI model (image output of a low-resolution image version) together form the input of a final ensemble model.
  • the multimodal aspect of the ensemble model is leveraged into a final overall label (refined overall label output) predicting a refined breast cancer recurrence status (“Digital Mamma Print Index”).
  • A-MIL Attention-based Multiple instance learning
  • A-MIL Attention-based Multiple instance learning
  • A-MIL Attention-based Multiple instance learning
  • A-MIL and the output of the Patch-based End-to-End module are both trained to output a label predicting a low/high risk of breast cancer recurrence.
  • the output of the Core AI model and the output of the Patch-based End-to-End module together form the input of a final ensemble model.
  • Fig. 2 Exemplary pre-processing of an WSI Tissue segmentation. Stroma and epithelial tissue, encapsulated by the contours is selected for further analysis. Adipose tissue and background areas are excluded.
  • Fig. 3 Exemplary patch extraction of a WSI. Tissue segmentation and patch extraction of a whole-slide-image (WSI). Fig.
  • Fig. 4 Exemplary network architecture of the Attention-based Multiple instance learning (A-MIL) model generating an overall label output Every feature vector which has been encoded to each of the extracted N patches is multiplied by its corresponding weight (attention score). The resulting low- dimensional embedding is then fed into a fully-connected network (FCN).
  • Fig. 5 Exemplary ensemble model architecture The respective outputs of the Tiny WSI (image output of a low-resolution image version) and of the Core AI models (overall label output) are fed as input into a fully-connected network (FCN), such as an ensemble model. The FCN generates a (final) refined overall output predicting a breast cancer recurrence status (Digital MammaPrint Index).
  • FCN fully-connected network
  • FIG. 6 Exemplary distribution of the final Digital Mamma Print Index (refined overall label output ) predicting a risk of breast cancer recurrence according to the method of the invention.
  • the horizontal axis represents the Digital Mamma Print Index which is the (final) refined overall label output of the ensemble model.
  • the vertical axis denotes density.
  • the continuous line represents the smooth probability density function.
  • Fig. 7 Schematic diagram of a neural network used for vector encoding A patch of an input image of a stained tissue samples obtained from an individual having breast cancer undergoes a series of convolutional and pooling layers with varying sizes. The output of the 2-D average pooling layer is an encoded feature vector derived from the input image.
  • Fig. 8 Tissue patches at different magnifications.
  • Fig. 9 An example input and output of the tissue type classification model. Each marker represents the predicted label for a 256x256 pixel tissue patch (circle: invasive, diamond: non-invasive, triangle: stroma, cross: artifact, square: infiltrated-invasive, pentagon: tumor-stroma).
  • Fig. 10 Nuclei detection. Extracted patches were processed by the nuclei detector to detect the nuclei and classify their type (circle: tumor cells, triangle: stroma cells, star: lymphocyte cells).
  • Fig. 11 Kaplan-Meier survival curve. On the y-axis the probability of distant recurrences is represented.
  • the invention for example includes computer implemented steps. For example, all steps of a method of the invention are computer-implemented steps.
  • Embodiments for example comprise a computer apparatus, wherein a method is of the invention is performed in said computer apparatus.
  • the invention for example extends to computer programs, particularly computer programs on or in a carrier, adapted for putting the invention into practice.
  • the program is for example in the form of source or object code or in any other form suitable for use in the implementation of a method according to the invention.
  • the carrier is any entity or device capable of carrying the program.
  • the carrier comprises a storage medium, such as a ROM, for example a semiconductor ROM or hard disk.
  • the carrier for example is a transmissible carrier such as an electrical or optical signal which is e.g. conveyed via electrical or optical cable or by radio or other means, e.g. via the internet or cloud.
  • the invention solves the problem of providing a novel, fast and cost-efficient method for predicting a risk of breast cancer recurrence.
  • the inventors of the invention surprisingly identified a method of assessing the risk of breast cancer recurrence by inputting an image of a stained tissue sample comprising breast cancer cells from an individual, pre-processing the image in order to achieve a tissue segment of relevant image areas, performing patch extraction, wherein N patches of the tissue segment are generated, encoding the extracted patches into a set of N feature vectors, and outputting an overall label output predicting breast cancer recurrence status by aggregating the contribution (weight/attention score) of each feature vector to the overall label output.
  • the invention is for example implemented using a machine or tangible computer-readable medium or article which optionally stores an instruction or a set of instructions that, if executed by a machine, cause the machine to perform a method and/or operations in accordance with the invention.
  • the invention is for example implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements include processors, microprocessors, circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, microchips, chip sets, etc.
  • the method of the invention uses images of stained tissue samples obtained from an individual having breast cancer.
  • Said tissue sample comprises breast cancer cells, or is suspected to comprise breast cancer cells.
  • the tissue sample is stained with a staining selected from hematoxylin and eosin (H&E), eriochrome and eosin, modified Papanicolaou or combinations thereof.
  • H&E hematoxylin and eosin
  • eriochrome and eosin modified Papanicolaou or combinations thereof.
  • the tissue sample is derived from an individual during or after a surgery for removing the breast cancer tissue or derived from a biopsy.
  • the image is obtained by virtual microscopy.
  • the image is obtained by whole-slide imaging.
  • the image is a whole-slide image (WSI).
  • the WSI has a rectangle or square format.
  • the WSI is in rectangle format having a size of 1000 x 750; 2000 x 1500; 3000 x 2250; 4000 x 3000; 5000 x 3750; 6000 x 4500; 7000 x 5250; 10,000 x 7500; 20,000 x 15,000; 30,000 x 22,500; 40,000 x 30,000; 40,000 x 30,000; 50,000 x 37,500; 60,000 x 45,000; 70,000 x 52,500; 80,000 x 60,000; 90,000 x 67,500; 100,000 x 75,000 or 200,000 x 150,000 pixels.
  • the WSI is in square format having a size of 1000 x 1000, 2000 x 2000; 3000 x 3000; 4000 x 4000; 5000 x 5000; 6000 x 6000; 7000 x 7000; 10,000 x 10,000; 20,000 x 20,000; 30,000 x 30,000; 40,000 x 40,000; 50,000 x 50,000; 60,000 x 60,000; 70,000 x 70,000; 80,000 x 80,000; 90,000 x 90,000; 100,000 x 100,000 or 200,000 x 200,000 pixels.
  • the WSI has a size up to 100,000 x 100,000 pixels.
  • the image optionally a WSI, has a size that does not directly fit into the available GPU memory of the hardware system or data-processing system performing a method of the invention.
  • the term ‘color channel of a pixel’ refers to one component of the color information associated with a certain pixel.
  • every pixel is represented by a single channel which is representing the intensity of light.
  • the pixel values are within a range of 0 to 255, corresponding with black and white respectively.
  • a pixel’s color may be determined by three channels, i.e. a red (R)-channel, green (G)-channel, blue (B)-channel.
  • the method of the invention comprises a step of pre-processing the image.
  • the pre-processing is performed in order to achieve a tissue segment image comprising relevant tissue areas, preferably comprising mainly or only relevant tissue areas.
  • the image is pre-processed which results in a tissue segment image which comprises only tissue areas informative for predicting a breast cancer recurrence status.
  • tissue segment image refers to a tissue area which is not informative for predicting a breast cancer recurrence, for example a tissue area comprising background and/or adipose tissue.
  • the tissue segment image obtained from pre-processing does not contain background and/or adipose tissue.
  • the image, optionally a WSI is pre-processed using a tissue segmentation pipeline.
  • the pre- processing is performed by an algorithm.
  • the pre-processing comprises the steps of a) down-sampling of the image by a factor s; b) transforming the image to greyscale; c ) transforming each pixel-value of the image to optical density (OD); d) creating a mask of pixels by defining an OD threshold level; e) excluding pixels at the boundary; f) excluding objects which are smaller than 0.1 – 1* area largest object in mask; g) excluding objects of a certain length; and/or h) excluding markings.
  • down-sampling factor refers to the factor by which an image’s resolution or dimensions are decreased during a down-sampling process.
  • the down-sampling factor s is from 8 – 1024.
  • the down-sampling factor s is 8; 16; 32; 64; 128; 192; 224; 256; 384; 512; 768; 1024; or 2048.
  • the down-sampling factor s is 256.
  • the down-sampled image has a resolution of 4 x 4.
  • Each pixel in the 4 x 4 down- sampled image corresponds to a 256 x 256 region in the original image.
  • patches of 256 x 256 can be extracted (vide infra), aligning with said regions in the down-sampled image.
  • the down- sampling factor matches a size of extracted patches.
  • the down- sampling factor s is 256 and the extracted patches are of size of 256 x 256 x 3, wherein “x 3” indicates the presence of three color channels (RGB).
  • each pixel-value of the image is transformed to optical density (OD).
  • each pixel value in a grayscale image is transformed by computing the negative base-10 logarithm of each pixel, with the exception of pixels with a value of zero.
  • a mask for example a binary mask, of pixels may be created based on two conditions: a) Pixels in the optical density-transformed image have values greater than a defined OD threshold level; and b) the variance of pixel values across the color channels of the original image is greater than a defined level.
  • the OD threshold level is in the range of > 1 - > 0.001, optionally > 1; > 0.8; > 0.7; > 0.6; > 0.5; > 0.4; > 0.3; > 0.2; > 0.1; > 0.09; > 0.08; > 0.07; > 0.06; > 0.05; > 0.04; > 0.03; > 0.02; > 0.01 or > 0.005.
  • the OD threshold level is > 0.1.
  • the boundary region of which pixels are excluded is defined as being in a range of 0.001 – 0.5*min (width, height).
  • the boundary region of which pixels are excluded is defined by 0.5; 0.4; 0.3; 0.2; 0.1; 0.09; 0.08; 0.07; 0.06; 0.05; 0.04; 0.03; 0.02; 0.01; 0.009; 0.008; 0.007; 0.006; 0.005; 0.004; 0.003; 0.002 or 0.001*min (width, height).
  • the boundary region of which pixels are excluded is defined by at least 0.03*min (width, height).
  • objects are excluded which are smaller than 1 – 0.001*area largest object in mask.
  • objects are excluded which are smaller than 0.9; 0.8; 0.7; 0.6; 0.5; 0.4; 0.3; 0.2; 0.1; 0.09; 0.08; 0.07; 0.06; 0.05; 0.04; 0.03; 0.02 or 0.01*area largest object in mask.
  • An object in a mask can be defined.
  • an object may be defined as all pixels with the same value, e.g. ‘1’ in a binary mask, that are directly connected such as horizontally, vertically and/or diagonally connected.
  • the person skilled in the art may for example use the function ‘skimage.measure.label’, which is part of the skimage.measure module in the scikit-image library (available at scikit-image.org/docs/stable/api/skimage.measure.html#skimage.measure.label).
  • object of a certain length are excluded wherein the length is (Lmax – Lmin)/ Lmax > 1.5 to 0.5, optionally wherein the length is (Lmax – Lmin)/ Lmax > 1.5; > 1.4; > 1.3; > 1.2; > 1.1; > 1.0; > 0.95; > 0.9; > 0.85; > 0.8; > 0.75; > 0.7; > 0.65; > 0.6; > 0.55; > 0.5; >0.45; > 0.4; > 0.35; > 0.3; > 0.25; > 0.2; > 0.15; or > 0.1.
  • markings which are excluded are pen markings, dirt, dust, droplets or any further element which is not relevant for performing a method of the invention.
  • any marking is excluded having a variance across RGB channels ⁇ 0.1 – 0.0001, optionally ⁇ 0.1; ⁇ 0.09; ⁇ 0.08; ⁇ 0.07; ⁇ 0.06; ⁇ 0.05; ⁇ 0.04; ⁇ 0.03; ⁇ 0.02; ⁇ 0.01; ⁇ 0.009; ⁇ 0.008; ⁇ 0.007; ⁇ 0.006; ⁇ 0.005; ⁇ 0.004; ⁇ 0.003; ⁇ 0.002; ⁇ 0.001; ⁇ 0.0009; ⁇ 0.0008; ⁇ 0.0007; ⁇ 0.0006; or ⁇ 0.0005.
  • tissue type classification may be performed by a convolutional neural network (CNN) such as a pre-trained CNN such as, for example, ResNet50, ResNet34, EfficientNet, DenseNet, RegNet, ResNeXt or a combination thereof.
  • CNN convolutional neural network
  • Nuclei detection may be performed using an AI model such as ‘You Only Look Once’ version 8 (YOLOv8), as is detailed herein below.
  • histopathological determination of slides or WSI by, e.g., a pathologist may be used to provide an estimate of the tumor percentage scoring of a slide or WSI.
  • Slides or WSI with an estimate of less than 50 % of invasive tumor cells. such as less than 40 %, less than 30 %, less than 20 %, or less than 10 % of invasive tumor cells, may be excluded from further analyses.
  • Patch extraction The method of the invention comprises a step of patch extraction.
  • the term “patches” as used herein refers to parts or pieces of the image, for example to parts or pieces of the pre-processed image.
  • the patch extraction comprises the generation of N patches of the pre-processed tissue segment image.
  • N matches number of extracted patches and for example depends on the size of the specimen.
  • the N patches are extracted from a pre-processed tissue segment WSI.
  • patch extraction is performed by extracting patches of the entire pre-processed tissue segment image, e.g., pre-processed tissue segment WSI.
  • patch extraction is performed by extracting only a selection, i.e., specific areas, of the pre-processed tissue segment image, e.g. pre-processed tissue segment WSI.
  • the pre-processed tissue segment image used for patch extraction has full resolution. Full-resolution means that the resolution of the obtained image is not changed for further image processing. For example, the full- resolution image which is used for patch extraction is not down-sampled.
  • patch extraction comprises mapping of the indices of the positive binary mask pixels to the coordinates of the image, optionally to the coordinates of the image such as WSI, wherein at each set of xy-coordinates a patch is extracted.
  • the extracted patches are for example of any size smaller than the input image, such as WSI.
  • the patch size is from 8 x 8 to 2048 x 2048 pixels.
  • the extracted patches for example comprise 3 color channels (RGB).
  • the extracted patches are of size 8 x 8; 16 x 16; 32 x 32; 64 x 64; 128 x 128; 196 x 196; 256 x 256; 384 x 384; 512 x 512; 768 x 768; 1024 x 1024; or 2048 x 2048 pixels.
  • patches are extracted having a size of 256 x 256 x 3, wherein “x 3” indicates the presence of three color channels (RGB).
  • an additional quality control step may be implemented.
  • images with certain characteristics may be excluded from further analysis. For example, for every patch a tissue type classification and/or nuclei detection may be performed.
  • the information obtained by the tissue type classification and/or nuclei detection may be used to exclude images from further analysis.
  • slides may be excluded by connected component analysis, if based on the analysis of only invasive tissue patches, the largest connected component consists of fewer than 100 patches and the second largest connected component has fewer than 50 patches. For example, if a slide has only 40 invasive patches and nothing else, the tumor percentage will be 100%. But such slide may be rejected because of the low number of invasive patches. Therefore, before calculating a tumor percentage, a connected component analysis may be performed and a slide may be rejected if the above conditions are not met.
  • tissue type classification for every patch information about the tissue type is obtained. These tissue types may be: invasive, infiltrated invasive, non- invasive, lymphocyte, stroma, tumor stroma, and other tissue types such as red blood cells, necrosis and artifacts.
  • tissue type classification task a Multi- Scale Feature Combination Model may be used, for example developed using PyTorch.
  • the model may incorporate techniques such as dropout and batch normalization.
  • the architecture may include an embedding layer to reduce the dimensionality of the feature representations and a SoftMax-based classifier for tissue type prediction.
  • a patch selection criteria may be used in which the number of patches per tissue type per WSI are maximalized or capped. For example, the number of invasive patches may be maximalized to 1024, the number of infiltrated invasive patches to 512, the number of non-invasive patches to 256, the number of lymphocyte patches to 256, the number of stroma patches to 128, the number of tumor stroma patches to 256.
  • YOLOv8 the advanced AI model called ‘You Only Look Once’ version 8 (YOLOv8) may be used to process each patch extracted during the patch extraction step.
  • YOLOv8s the small version of YOLOv8 (YOLOv8s, Jocher et al., 2023. Ultralytics, available at github.com/ultralytics/ultralytics) may be used.
  • Nuclei detection may also be performed using Mask R-CNN (He et al., 2017. arXiv:1703.06870), HoVer-Net (Graham et al. 2019. Medical Image Analysis 58: 101563) or RetinaNet (Lin et al., 2017. arXiv:1708.02002).
  • Mask R-CNN He et al., 2017. arXiv:1703.06870
  • HoVer-Net Graham et al. 2019. Medical Image Analysis 58: 101563
  • RetinaNet Li et al., 2017. arXiv:1708.02002.
  • the information about the tissue type and the detected nuclei obtained for the patches in an image may be combined into one measure for a certain image.
  • Said measure is for example a digital tumor percentage (dTu%) which can be calculated by dividing the total number of invasive nuclei as observed in all patches of an image by the total number of nuclei as observed in all patches of an image.
  • An image which has a dTu% below a certain threshold may be excluded from further analyses.
  • a dTu% below 20%, such as below 10% may be excluded from further analysis.
  • Feature vector encoding The method of the invention further contains a step of encoding the extracted N patches into a set of feature vectors, wherein the input image is associated with N feature vectors.
  • the encoding comprises that each of the N patches is transformed to a feature vector of a certain length.
  • each of the N patches is encoded with a feature vector, i.e., the number of feature vectors resembles the number of patches (N).
  • the length of each of the N feature vectors is from 1-10000 such as from 8 – 2048.
  • the length of each of the N feature vectors is 8; 16; 32; 64; 128; 256; 384; 512; 768; 1024 or 2048.
  • the length of each of the N feature vectors is 1024.
  • the encoding is performed for example by using a convolutional neural network (CNN).
  • a CNN comprises for example and input layer, one or more hidden layers, such as convolutional, pooling and/or flattening layers, and an output layer.
  • An input layer for example receives the input image with its pixel values.
  • the input layer for example has a stack of convolutional layers that are extracting the hierarchical features from the input image (e.g., LeNet-5 has 3 convolutional layers and ResNet50 has 49 convolutional layer).
  • Each convolutional layer for example consists of filters to capture patterns and spatial hierarchies in different scales.
  • Non-linear activation functions like ReLU (Rectified Linear Unit) are for example used after each convolutional operation to introduce non-linearity and enhance the model's capacity to learn complex features (ResNet50 has 32 activation layers).
  • Pooling layers down-sample (i.e., reduce) the spatial dimensions of the feature maps, reducing computational complexity and retaining essential information (e.g., ResNet50 has 16 pooling layers).
  • Down- sampling for example refers to a process in which the spatial resolution or dimensions of an image or a feature map are reduced. It involves for example reducing the number of pixels or data points along the width and height of the image or feature map.
  • down-sampling is for example achieved through operations like MaxPooling or AveragePooling. These operations involve dividing the input into non-overlapping or partially overlapping regions and summarizing each region by taking the maximum or average value, respectively.
  • Flattening layers are for example used to flatten the output from the convolutional layers into a one-dimensional vector, preparing it for the subsequent fully connected layers which are connected to the output layer.
  • a feature vector encodes textural and morphological information of the respective patch.
  • the CNN such as ResNet50
  • the CNN is designed to learn complex hierarchical features providing feature vectors which are rich of descriptive textural and morphological information.
  • the feature vectors comprise information regarding the form and/or structure of the tissues or cells within the image.
  • feature vectors comprise morphological details, such as shape, size, and arrangement of cells.
  • the feature vectors further comprise spatial, structural, texture and/or contextual information.
  • the feature vectors further comprise information how different structures are spatially arranged in the image, information on details about the texture or patterns within different regions of the image, information on specific structural patterns and abnormalities and/or information on relationships between different components in the image.
  • the CNN is for example a pre-trained CNN.
  • the weights of the CNN are obtained/trained by training the CNN to classify natural images from a database.
  • the CNN is selected from ResNet50, ResNet34, EfficientNet, DenseNet, RegNet, ResNeXt or a combination thereof.
  • the CNN is ResNet50, optionally a pre-trained ResNet50.
  • the ResNet50 used in the method of the invention is for example as described in He K., Zhang, X.; Shaoqing R. and Sun, J.; 2015; arXiv:1512.03385, Deep Residual Learning for Image Recognition.
  • the ResNet50 architecture for example comprises: 1) Input Layer: Standard input layer, wherein the image data is fed into the CNN. 2)
  • Initial Convolutional Layer The initial convolutional layer has 64 filters with a kernel size of 7x7 and a stride of 2. This layer is responsible for the initial feature extraction.
  • Residual Blocks ResNet50 is characterized by the use of residual blocks, each containing multiple convolutional layers. There are two main types of residual blocks with different numbers of layers.
  • ResNet50 After the stack of residual blocks, ResNet50 employs global average pooling, reducing the spatial dimensions to 1x1.
  • the ResNet50 has been pre-trained on ImageNet.
  • the pre-trained ResNet50 is obtained from Pytorch.
  • ResNet50 is pretrained on ImageNet50.
  • the weights of the ResNet50 are obtained by training it to classify natural images in the ImageNet database.
  • the N feature vectors are obtained by extracting the 3 rd residual block of a total of 5 blocks of a ResNet50 followed by an Average Pooling layer.
  • each of the N patches is transformed to a feature vector of length selected from 512; 768; 1024; 1280; 1536 or 2048.
  • each of the N patches is transformed to a feature vector of length 1024.
  • the N feature vectors are stored in PyTorch.
  • the feature vector encoding is performed by a feature extractor that is trained using self-supervised learning (SSL).
  • the resulting feature vector may comprise pathology self-supervised features.
  • SSL methods that can be applied are the DINOv2 framework (Vorontsov et al., 2024. Nat Med.
  • the method of the invention further comprises outputting an overall label output predicting a breast cancer recurrence status.
  • the overall label output is calculated and/or determined by aggregating the contribution (weight/attention score) of each of the N feature vectors to the overall label output.
  • the outputting of the overall label output is performed by using at least one fully-connected network (FCN) comprising a regression model, such as gated attention mechanism.
  • the gated attention mechanism is an adapted version of the attention-based multiple Instance learning (A-MIL) model.
  • A-MIL used in the method of the invention is for example as described in Ilse, M.; Tomczak J. M.; and Welling, M.; 2018; arXiv:1802.04712; Attention-based Deep Multiple Instance Learning.
  • such a model has a gated attention module and a fully connected network.
  • the gated attention module has for example three fully connected layers. Two branches for example process the input features differently using tanh and sigmoid activation functions, and the outputs are for example element-wise multiplied. The result is for example then passed through a fully connected layer.
  • the fully connected network for example takes the input features, processes them through a fully connected layer and uses the weights/attention score to produce a final output.
  • the A-MIL takes N feature vectors of a certain length as input.
  • the length of the N feature vectors for input is selected from 64; 128; 256; 384; 512; 768; 1024 and 2048.
  • the length of the N feature vectors for input is 1024.
  • the N feature vectors are propagated through a linear layer.
  • the linear layer is for example of a length of from 8 – 2048.
  • the length of the linear layer is selected from 8; 16; 32; 64; 128; 256; 384; 512; 768; 1024 and 2048.
  • the linear layer is for example of length 512.
  • the propagation is followed by Rectified Linear Unit (ReLU) activation and Dropout.
  • the resulting features are processed in two parallel streams of the gated attention mechanism.
  • the first stream comprises a linear layer.
  • the linear layer of the first stream is for example of a length of from 8 – 2048 followed by Tanh activation and Dropout.
  • the length of the linear layer of the first stream is selected from 8; 16; 32; 64; 128; 256; 384; 512; 768; 1024 and 2048 followed by Tanh activation and Dropout.
  • the linear layer of the first stream is of length 384, which is followed by Tanh activation and Dropout.
  • the second stream of the gated attention mechanism is a linear layer.
  • the linear layer of the second stream is for example of a length of from 8 – 2048 followed by Sigmoid activation and Dropout.
  • the length of the linear layer of the second stream is selected from 8; 16; 32; 64; 128; 256; 384; 512; 768; 1024 and 2048 followed by Sigmoid activation and Dropout.
  • the linear layer of the second stream is of length 384 followed by Sigmoid activation and Dropout.
  • Dropout of both layers is performed by multiplying each element with each other.
  • the result of the multiplication is passed through the last attention layer of the gated attention mechanism which is e.g. a linear layer of length 1.
  • the side level feature is for example of a length of from 8 – 2048 and propagated to a single node.
  • the side level feature is of a length selected from 8; 16; 32; 64; 128; 256; 384; 512; 768; 1024 and 2048 and propagated to a single node.
  • the side level feature is of length 512 and propagated to a single node.
  • the output of the single node is the overall output label predicting a breast cancer recurrence status (i.e., risk of breast cancer recurrence).
  • the gated attention module of the A-MIL for example calculates a contribution of each of the N feature vectors by outputting a weight (attention score) for each feature vector assigned to each of the N patches, wherein the sum of all weights (attention scores) per input image is 1.
  • Patches having a feature vector of high attention score (weight) are more important in predicting the overall label output.
  • high-attention scores (weights) originate from patches of a input image of a stained tissue sample which comprise tumor regions.
  • patches having a feature vector of high attention score (weight) are above the 75 th ; 80 th ; 85 th ; 90 th ; or 95 th percentile of all patches of an input image.
  • patches having a feature vector of high attention score (weight) are above 90 th percentile of all patches of an input image.
  • the contribution (weight/attention score) of each of the N feature vectors is for example aggregated by the A-MIL to the overall label output.
  • Providing an image output of a low-resolution image version The outputting of a method according to the invention for example further comprises providing an image output of a low-resolution image version of the stained tissue sample.
  • the low-resolution image version is for example achieved by down-sampling the original image of the stained tissue sample.
  • the low-resolution image version is down-sampled to a size/resolution where individual cells are no longer visible.
  • the low-resolution image version is down- sampled to a size selected from 8 x 8; 16 x 16; 32 x 32; 64 x 64; 128 x 128; 196 x 196; 256 x 256; 512 x 512; 768 x 768; 1024 x 1024; 2048 x 2048; or 4096 x 4096 pixels.
  • the low-resolution image version for example comprises 3 color channels (RGB).
  • the low-resolution image version is down-sampled to a size of 512 x 512 x 3, wherein “x 3” indicated the presence of three color channels (RGB).
  • the low-resolution image version of the stained tissue sample may be used as input to a machine learning data processing model to obtain the image output of the low-resolution image version.
  • the machine learning data processing model relies on the global appearance of the stained tissue sample.
  • the machine learning data processing model used is a convolutional neural network (CNN).
  • CNN convolutional neural network
  • the machine learning data processing model is selected from network architectures suitable for classification and regression, e.g. ResNet50, ResNet34, EfficientNet, ResNeXt, DenseNet, RegNet.
  • the machine learning data processing model is selected from vision transformers, e.g. ViT, DeiT, CaiT, Swin Transformer, T2T-ViT.
  • the classifier is a ResNet50.
  • ResNet50 is a deep convolutional neural network architecture with for example 50 layers. It has for example residual blocks, each containing two convolutional layers and utilizing skip connections to address the vanishing gradient problem.
  • the architecture for example comprises global average pooling and fully connected layers at the end, making it suitable for image classification tasks.
  • the ResNet50 used in the method of the invention is for example as described in He K., Zhang, X.; Shaoqing R. and Sun, J.; 2015; arXiv:1512.03385, Deep Residual Learning for Image Recognition.
  • Providing a slide-level prediction index by a Patch-based End-to-End Module for example further comprises providing a slide-level prediction index directly predicted from the pre- processed image patches or a subset thereof.
  • a so called “Patch- based end-to-end module” (Fig. 1B) may be used to learn to predict a slide-level prediction index from the pre-processed image patches or a subset thereof.
  • the input of the patch based end-to-end model is a subset of the pre-processed image patches.
  • said subset of pre-processed image patches is selected from the pre-processed image patches based on tissue type information in a process called tissue-based patch selection.
  • tissue type information in a process called tissue-based patch selection.
  • tissue types may be: invasive, infiltrated invasive, non-invasive, lymphocyte, stroma, tumor stroma, and other tissue types such as red blood cells, necrosis and artifacts.
  • the input comprises: ( ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ) dimensional patches sampled from pre-processed WSIs, where M is the total number of patches within the whole slide, and (H x W x C) are the height-width and the number of color channels in each patch, respectively (such as 256 x 256 x 3).
  • the tissue-based patch selection process takes an initial set of ⁇ patches and reduces it to a subset of ⁇ patches where ⁇ ⁇ ⁇ based on predefined criteria. To achieve this, it is assumed that access to a ⁇ ⁇ ⁇ discrete tissue classification result for each patch, where ⁇ represents the number of unique tissue categories.
  • a patch selection criteria may be devised based on the relative population of patches based on their tissue type.
  • the selection is guided by K-tuple criteria ( ⁇ , ⁇ ) where ⁇ denotes the relative percentage of patches to be sampled from each tissue category. The sum of all percentages satisfies the condition: ⁇
  • ⁇ ⁇ is the percentage of tissue category ⁇ .
  • the parameter ⁇ is a hyperparameter that can be tuned to optimize the overall performance of the downstream prediction model. Adjusting these percentages allows flexibility in emphasizing certain tissue types that might contribute more significantly to the prediction task. This tailored selection ensures that the sampled subset of ⁇ patches provides a balanced and informative representation of the slide while ignoring the uninformative part of the slide (e.g. fatty tissue).
  • a first block is an embedding block to represent the patches of the input image such as the input WSI. This may be in the form of a deep CNN or a Vision- Transformer.
  • a second block is an aggregation layer to aggregate the representation of all the patches to emphasize the important parts of the slide. This may be a Multiple Instance Learning (MIL)-based approach, such as ABMIL (Ilse et al., 2018. arXiv:1802.04712), TransMIL (Shao et al., 2021. Advances in Neural Information Processing Systems 34: 2136-2147) or CLAM (Lu et al., 2021. Nat Biomed Eng 5:555-570).
  • MIL Multiple Instance Learning
  • a third block is a regression layer such as a 1-layer multi-layer perception (MLP) to map the aggregated slide-level embedding to the slide-level prediction index.
  • MLP 1-layer multi-layer perception
  • the outputting of a method of the invention further comprises combining the overall label output, e.g.
  • the combining for generating a refined overall label output comprises inputting the overall label output and the image output of a low-resolution image version of the stained tissue sample in a (simple) fully-connected network (FCN), such as an ensemble model.
  • FCN fully-connected network
  • the outputting of a method of the invention further comprises combining the overall label output, e.g., generated by the gated attention mechanism, such as A-MIL, and the output of the patch-based end-to-end module for generating a refined overall label output.
  • the combining for generating a refined overall label output comprises inputting the overall label output and the output of the patch-based end-to-end module in a (simple) fully- connected network (FCN), such as an ensemble model.
  • FCN fully- connected network
  • the output of the patch-based end-to-end module used for generating a refined overall label output is at least one of the patch embeddings (E), the slide-level embeddings (S) and/or the patch-based predicted index.
  • the output of the patch-based end-to-end module used for generating a refined overall label output is at least two of the patch embeddings (E), the slide-level embeddings (S) and the patch-based predicted index
  • the output of the patch-based end-to-end module used for generating a refined overall label output is the patch embeddings (E), the slide- level embeddings (S) and the patch-based predicted index.
  • the output of the patch-based end-to-end module is concatenated with the feature vectors of other models, such as the self-supervised feature vector.
  • the ensemble model is a machine learning algorithm suitable for regression, such as linear regression, decisions trees, gradient boosting etc..
  • the ensemble model for example comprises 2 – 10 layers.
  • the ensemble model comprises at least two; at least three; at least four; at least five; at least six; at least seven; least eight; at least nine; at least 10; at least 15; at least 20; at least 25; at least 30; at least 35; at least 40; at least 45; at least 50; at least 60; at least 70; at least 80; at least 90; or at least 100 layers.
  • the input layer of the ensemble model has two neurons, i.e. is of length 2.
  • the input layer of the ensemble model has two neurons (length 2), wherein the overall label output and/or the image output of a low-resolution image version of the stained tissue sample and/or the output of the patch-based end-to-end module serve as input of the ensemble model.
  • the ensemble model has 1 – 100 hidden layers. The number of hidden layers of the ensemble model may be adjusted to the complexity of the performed task.
  • the ensemble model has a single hidden layer with 64 neurons.
  • the output of the ensemble model is of length 1.
  • the output layer of length 1 of the ensemble model outputs a refined overall label output.
  • each layer except the last layer is followed by at least one activation function such as ReLU, ELU, GLU, and/or Leaky ReLU activation and Dropout.
  • activation functions which are for example used in the method of the invention are for example further described in Dubey, S.; Singh, S. and Chaudhuri, B.; 2022; Neurocomputing, Activation functions in deep learning: A comprehensive survey and benchmark.
  • each layer except the last layer is followed by ReLU activation and Dropout.
  • the refined overall label output the overall label output and/or the image output of a low-resolution image version of the stained tissue sample and/or the output of the patch-based end-to-end module is > 0 the risk of breast cancer recurrence is low.
  • the refined overall label output the overall label output and/or the image output of a low-resolution image version of the stained tissue sample and/or the output of the patch-based end-to-end module is ⁇ 0 the risk of breast cancer recurrence is high.
  • the high risk of breast cancer recurrence group is further divided into 2 groups by taking the median of the high risk images of a stained tissue samples comprising breast cancer cells.
  • High risk group 2 is defined as having a refined overall label output, overall label output and/or image output of a low-resolution image version of the stained tissue sample and/or the output of the patch-based end-to-end module higher than the median value of the high risk images
  • High risk group 1 is defined as having a refined overall label output, overall label output and/or image output of a low-resolution image version of the stained tissue sample and/or the output of the patch-based end-to-end module lower than the median value of the high risk images.
  • Training the models The method according to the invention is for example a pretrained method, i.e. no training of the machine learning data processing models used in the method of the invention is required.
  • the method according to the invention for example comprises training of at least one or all machine learning data processing models used in the method of the invention.
  • Training of at least one machine learning data processing model according to the invention for example comprises a) obtaining ground truth data; b) inputting example data into at least one machine learning data processing model used in the method of the invention; c) performing the at least one machine learning processing model according to the invention on the input example data; d) measuring the error between the generated output and the ground truth data; and e) updating the weights of the at least one or all machine learning data processing model according to the invention for performing the training of the machine learning data processing model.
  • the steps b) to e) are repeated until the error is no longer decreasing.
  • ground truth data used for training at least one machine learning data processing model used in the method of the invention comprises a set of encoded N feature vectors corresponding with a set of N patches, a label providing information about a risk of breast cancer recurrence in a tissue sample corresponding with a set of N feature vectors, a label providing information about a risk of breast cancer recurrence in the tissue sample corresponding with a low- resolution image version of an image, and/or an output of the patch-based end-to end module of a stained tissue sample comprising breast cancer cells, a label providing information about a risk of breast cancer recurrence of an image of a tissue sample comprising breast cancer cells corresponding with the image of a stained tissue sample comprising breast cancer cells and/or a label providing information about a risk of breast cancer recurrence of an image of a tissue sample comprising breast cancer cells corresponding with the overall label output and
  • the label is for example the result of a gene expression signature measured.
  • the label is the output of an analyzed MammaPrint gene set, i.e. a Mamma Print index.
  • example data used for training at least one machine learning data processing model used in the method of the invention comprises a set of N patches of an image of a stained tissue sample comprising breast cancer cells having corresponding feature vectors, wherein the corresponding feature vectors serve as ground truth data.
  • example data used for training at least one machine learning data processing model used in the method of the invention comprises a set of encoded N feature vectors having a corresponding label providing information about a risk of breast cancer recurrence in a tissue sample, wherein the corresponding label serves as ground truth data.
  • example data used for training at least one machine learning data processing model used in the method of the invention comprises a low- resolution image version of an image and/or an output of the patch-based end-to end module of a stained tissue sample comprising breast cancer cells having a corresponding label providing information about a risk of breast cancer recurrence in a tissue sample of the image, wherein the corresponding label serves as ground truth data.
  • example data used for training at least one machine learning data processing model used in the method of the invention comprises an overall label output and an image output of a low-resolution image version of an image and/or an output of the patch-based end-to end module of a stained tissue sample comprising breast cancer cells having a corresponding label providing information about a risk of breast cancer recurrence in a tissue sample of the image, wherein the corresponding label serves as ground truth data.
  • example data used for training at least one machine learning data processing model used in the method of the invention comprises an image of a stained tissue sample comprising breast cancer cells having a corresponding label providing information about a risk of breast cancer recurrence in a tissue sample of the image, wherein the corresponding label serves as ground truth data.
  • training a machine learning data processing model used for encoding N patches of an image of a stained tissue sample into a set of N feature vectors comprises a) obtaining as ground truth data a set of encoded N feature vectors corresponding to a set of N patches; b) inputting as example data the set of the N patches into a processor comprising the machine learning data processing model used for feature vector encoding; c) performing the machine learning processing model on the input set of N patches; d) measuring the error between the generated feature vectors and the ground truth data; and e) updating the weights of the machine learning data processing model.
  • the steps b) to e) are repeated until the error is no longer decreasing.
  • Training a method of the invention for example comprises inputting as example data an image of stained tissue samples comprising breast cancer cells having a corresponding label providing information about a risk of breast cancer recurrence in said tissue sample.
  • the corresponding label providing information about a risk of breast cancer recurrence in said tissue sample for example provides ground truth data for measuring the error and updating the weights of the machine learning data processing model used in the method of the invention.
  • an image of stained tissue samples comprising breast cancer cells having a corresponding labels providing information about a risk of breast cancer recurrence in said tissue sample is used to train at least one or all machine learning data processing model used in the method of the invention.
  • the training comprises separating example data into three different subsets: a training, a validation and a test set.
  • the test set is for example kept aside during the training of a method of the invention.
  • examples of stained tissue samples comprised in the training and validation a set k-fold cross validation to train k models is for example applied.
  • each fold example data of stained tissue samples are for example randomly selected from a pool of example data of stained tissue samples. For example, 60,000 example data of stained tissue samples are selected for the training set and 5,000 example data of stained tissue samples are selected for the validation set.
  • training a method of the invention is performed on every example data of stained tissue samples from the training set, loss is calculated and backpropagation is used to update the weights of the machine learning data processing model, such as neural networks (NN), used in the method of the invention.
  • Loss functions are for example used to measure the error between the generated output of at least one machine learning data processing model, such as neural networks (NN), used in the method of the invention and the ground truth data, such as a predefined label corresponding to the example data.
  • a predefined label is for example obtained by gene expression analysis.
  • the output by gene expression analysis is the output of the MammaPrint Index measured.
  • Adam, AdamW, SGD, Adagrad, RMSprop and/or SparseAdam optimization is used to update the NNs’ weights of a method of the invention.
  • Adam optimization is used to updates the NNs’ weights of a method of the invention.
  • the average loss on the example data of stained tissue samples from the validation samples is for example reported after every epoch, after every second epoch, after every third epoch or only once.
  • the average loss is for example calculated as Mean Squared Error (MSE) loss which measures the average squared difference between the predicted values and the actual values.
  • MSE Mean Squared Error
  • n is the number of samples in the dataset.
  • yi is the actual (ground truth) value for the i-th sample.
  • ⁇ i is the predicted value for the i-th sample.
  • the validation process is repeated until the validation loss is no longer decreasing, i.e., until the model is not improving.
  • the average loss is calculated as Mean-absolute Error (MAE), Huber Loss, Quantile Loss, Weighted MSE and/or MAE.
  • MAE Mean-absolute Error
  • Huber Loss e.g., Huber Loss
  • Quantile Loss e.g., Weighted MSE
  • MAE Mean-absolute Error
  • Weighted MSE Weighted MSE
  • MAE Mean-absolute Error
  • NNs used e.g., NNs used are stored.
  • Adam, AdamW, SGD, Adagrad, RMSprop and/or SparseAdam optimization is used.
  • Adam optimization is used.
  • Adam optimization is used with a learning rate of 1xe -5 ; 1xe -4 ; 2xe -4 or 5xe -4 .
  • Adam optimization is used with a weight decay of from 1xe -5 - 1xe -3 , such as 1xe -5 ; 1xe -4 or 1xe -3 .
  • Adam optimization is used with a learning rate of 2e -4 and a weight decay of 1e -5 .
  • the classification model for outputting an overall label output predicting a breast cancer recurrence status, such as A-MIL, is for example trained with an example data batch size of 1 – 100.
  • the classification model for outputting an overall label output predicting a breast cancer recurrence status is trained with example data a batch size of at least 1; at least 3; at least 5; at least 10; at least 15; at least 20; at least 25; at least 30; at least 35; at least 40; at least 45; at least 50; at least 60; at least 70 or at least 80.
  • a batch for example refers to all of the N feature vectors of one image data example of stained tissue samples.
  • the training of the machine learning data processing model to classify the image output of a low-resolution image version of the stained tissue sample is similar to the training of the classification model for outputting an overall label output predicting a breast cancer recurrence status as described above.
  • training of the machine learning data processing model to classify the image output of a low-resolution image version of the stained tissue sample comprises a) obtaining as ground truth data a low-resolution image version of a stained tissue sample having a corresponding label providing information about a risk of breast cancer recurrence in the tissue sample of the low-resolution image; b) inputting as example data the low-resolution image version into a processor comprising the machine learning data processing model used for outputting the image output of a low-resolution image version; c) performing the machine learning data processing model; d) measuring the error between the generated image output of a low-resolution image version and the ground truth data; and e) updating the weights of the machine learning data processing model used for outputting the image output of a low-resolution image version for performing the training of the machine learning data processing model.
  • the weights of the machine learning data processing model used for outputting the image output of a low- resolution image version are updated using backpropagation.
  • the training is repeated until the error is no longer decreasing.
  • the machine learning data processing model to classify the image output of a low-resolution image version of the stained tissue sample is trained for example by using cross-entropy loss, Binary Cross-Entropy Loss and/or Focal loss.
  • the classification model to classify the image output of a low-resolution image version of the stained tissue sample is trained for example by using cross-entropy loss.
  • Adam, AdamW, SGD, Adagrad, RMSprop and/or SparseAdam optimization is used.
  • Adam optimization is used.
  • Adam optimization is used with a learning rate of 1xe -5 ; 1xe -4 ; 2xe -4 or 5xe -4 .
  • Adam optimization is used with a weight decay of from 1xe -5 – 1xe -3 , such as 1xe -5 ; 1xe -4 or 1xe -3 .
  • Adam optimization is used with a learning rate of 2e -4 and a weight decay of 1e -5 .
  • Adam optimization with a learning rate of 2e -4 and a weight decay of 1e -5 is used.
  • training of the machine learning data processing model to classify the output of the patch-based end-to end module of the stained tissue sample is similar to the training of the classification model for outputting an overall label output predicting a breast cancer recurrence status as described above.
  • training of the machine learning data processing model to classify the output of the patch-based end-to end module of the stained tissue sample comprises a) obtaining as ground truth data the output of the patch-based end-to end module of a stained tissue sample having a corresponding label providing information about a risk of breast cancer recurrence in the tissue sample image; b) inputting as example data the output of the patch-based end-to end module into a processor comprising the machine learning data processing model used for outputting the output of the patch-based end-to end module; c) performing the machine learning data processing model; d) measuring the error between the generated output of the patch-based end-to end module and the ground truth data; and e) updating the weights of the machine learning data processing model used for outputting the output of the
  • the weights of the machine learning data processing model used for outputting the output of the patch-based end-to end module are updated using backpropagation.
  • the training is repeated until the error is no longer decreasing.
  • the machine learning data processing model to classify the output of the patch-based end-to end module of the stained tissue sample is trained for example by using cross-entropy loss, Binary Cross-Entropy Loss and/or Focal loss.
  • the classification model to classify the output of the patch-based end-to end module of the stained tissue sample is trained for example by using cross-entropy loss.
  • Adam, AdamW, SGD, Adagrad, RMSprop and/or SparseAdam optimization is used.
  • Adam optimization is used.
  • Adam optimization is used with a learning rate of 1xe -5 ; 1xe -4 ; 2xe -4 or 5xe -4 .
  • Adam optimization is used with a weight decay of from 1xe -5 – 1xe -3 , such as 1xe -5 ; 1xe -4 or 1xe -3 .
  • Adam optimization is used with a learning rate of 2e -4 and a weight decay of 1e -5 .
  • Adam optimization with a learning rate of 2e -4 and a weight decay of 1e -5 is used.
  • training image augmentation is for example applied to enhance model generalizability.
  • Image augmentation for example comprises ColorJitter, RandomVerticalFlip, RandomHorizontalFlip and/or RandomRotation.
  • the RandomRotation is for example 180°.
  • all images are transformed to PyTorch tensor and normalized in each color channel.
  • std (0.229; 0.224; 0.225).
  • the transformation to tensors and/or normalization is applied to the images. For example, no augmentation is applied.
  • the classification model to classify the image output of a low-resolution image version and/or the output of the patch-based end- to end module of the stained tissue sample is trained with a batch size of 1 – 100, such as at least 1; at least 3; at least 5; at least 10; at least 15; at least 20; at least 25; at least 30; at least 35; at least 40; at least 45; at least 50; at least 60; at least 70 or at least 80.
  • a batch for example refers to all of the N feature vectors of one image data example of stained tissue samples.
  • Training the machine learning data processing model for outputting a refined overall label output is for example similar to the training of the machine learning data processing model for outputting an overall label output predicting a breast cancer recurrence status and/or training of the machine learning data processing model to classify the image output of a low- resolution image version and/or the output of the patch-based end-to end module of the stained tissue sample as described above.
  • training of the machine learning data processing model for outputting a refined overall label output comprises a) obtaining as ground truth data an overall label output and an image output of a low-resolution image and/or the output of the patch-based end-to end module having a corresponding label providing information about a risk of breast cancer recurrence of an image a stained tissue sample comprising breast cancer cells, preferably the label is a result of gene expression profiling; b) inputting as example data the overall label output and the image output of a low-resolution image version of the image and/or the output of the patch-based end-to end module of a stained tissue sample; c) performing the machine learning data processing model for outputting a refined overall label output on the input overall label output and the input image output of a low-resolution image version and/or the output of the patch-based end-to end module, d) measuring the error between the generated refined overall label output and the ground truth data; and e) updating the weights of the machine learning data processing model used for outputting the refined label
  • the weights of the machine learning data processing model used for outputting the refined label output are updated using backpropagation.
  • the steps b) to e) of the training are repeated until the error is no longer decreasing.
  • Training the ensemble model for example comprises mean-squared-error loss.
  • training the ensemble model comprises Adam optimizer using a learning rate of 1e-3.
  • ensemble is trained with a batch size of 1 – 5000; 50 – 3000; 100 – 2000 or 200 – 1500.
  • ensemble is trained with a batch size of at least 1; at least 10; at least 50; at least 75; at least 100; at least 150; at least 200; at least 250; at least 300; at least 450; at least 500; at least 550; at least 600; at least 700; at least 800; a at least 850; at least 900; at least 1000; at least 1500; at least 2000; at least 2500; at least 3000.
  • a batch for example refers to all of the N feature vectors of one image data example of stained tissue samples.
  • the training of a method is for example performed for all models used in the method separately.
  • the training of the method is for example performed for all models used in a method of the invention together.
  • Computer program product The invention for example comprises a computer program or computer program product having instructions which when executed by a computing device or system to perform each of the steps of a method of the invention. Further, the invention comprises for example a computer program product embodied on a non- transitory computer readable medium comprising instructions stored thereon to cause one or more processors to perform a method of the invention.
  • the computer program or computer program product is for example in the form of source or object code or in any other form suitable for use in the implementation of a method according to the invention.
  • the computer program or computer program product is for example stored on a carrier.
  • a carrier is for example any entity or device capable of carrying the computer program or computer program product.
  • the carrier comprises a storage medium, such as a ROM, for example a semiconductor ROM or hard disk.
  • the carrier for example is a transmissible carrier such as an electrical or optical signal which is e.g. conveyed via electrical or optical cable or by radio or other means, e.g. via the internet or cloud.
  • Device or apparatus The invention for example comprises a data-processing system, an apparatus and/or an device comprising means for carrying out a method of the invention.
  • the data-processing system, apparatus and/or device for example comprises processors, microprocessors, circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, microchips, chip sets, et cetera.
  • the data-processing system, apparatus and/or device is for example a computer, laptop, tablet, smartphone and/or a combination thereof.
  • the data-processing system, apparatus and/or an device comprising means for carrying out a method of the invention for example further comprises a imaging device, such as a microscope, for imaging stained tissue samples obtained from individuals. For example, the stained tissue samples obtained from individuals are prepared on slides.
  • the microscope is for example a stereomicroscope, compound microscope, fluorescence microscope, inverted microscope, a confocal microscope, scanning probe microscope, lightsheet microscope, spinning disk microscope and/or a combination thereof.
  • Methods of treatment The invention is also directed to a method of treating an individual who has been assessed as having a high risk of breast cancer recurrence according to the computer-implemented method, computer program product and/or data-processing system of the invention. For example, the individual is assessed as being at high risk of breast cancer recurrence if the refined overall label output, the overall label output and/or the image output of a low-resolution image version of the stained tissue sample and/or the output of the patch-based end-to-end module is ⁇ 0.
  • the method of treatment according to the invention for example comprises assessing the risk of breast cancer recurrence in an individual according to the computer-implemented method of the invention, and providing a therapy for preventing and/or treating breast cancer to the individual assessed as having a high risk of breast cancer recurrence.
  • the therapy for preventing and/or treating breast cancer for example includes any therapy of the state of the art which is known to be capable of preventing and/or treating breast cancer.
  • the therapy is for example selected from chemotherapy, immunotherapy, stem cell therapy, hormone therapy, such as endocrine therapy, radiotherapy, targeted therapy, performance of surgery, or a combination thereof.
  • the method of treatment comprises administrating to an individual assessed as having a high risk of breast cancer recurrence any active compound, drug and/or pharmaceutical compound of the state of the art which is known to be capable of preventing and/or treating breast cancer.
  • assessing the risk of breast cancer recurrence according to the computer- implemented method of the invention is performed prior to administration of an active compound, drug and/or pharmaceutical compound.
  • an active compound, drug and/or pharmaceutical compound for preventing and/or treating breast-cancer is selected from selective oestrogen receptor modulators (SERMs) such as acolbifene (Endoceutics), afimoxifene (BHR Pharma, Atossa Therapeutics), arzoxifene (Eli Lilly and company), apeledoxifene (Pfizer), clomifene (Sanofi), droloxifene (Pfizer), endoxifen (Atossa Therapeutics), lasofoxifene (Pfizer), ospemifene (Osphena), pipindoxifene (LEAPChem), raloxifene (Daiichi Sankyo), tamoxifen (Rosemont Pharmaceuticals) and toremifene (Orion Corporation); selective oestrogen receptor down-regulators (SERDs) such as amcenestrant (also called SAR439859, San, San
  • Immune checkpoint inhibitors used in the method of treatment according to the invention are for example selected from following non-limiting examples: CTLA-4 inhibitors such as antibodies, including ipilimumab (Bristol-Myers Squibb) and tremelimumab (MedImmune); PD1/PDL1 inhibitors such as antibodies, including pembrolizumab (Merck), sintilimab (Eli Lilly and Company), tislelizumab (BeiGene), toripalimab (Shangai Junshi Biosciense Company), spartalizumab (Novartis), camrelizumab (Jiangsu HengRui Medicine C), nivolumab and MDX- 1105 (Bristol-Myers Squibb), pidilizumab (Medivation/Pfizer), MEDI0680 (AMP- 514; AstraZeneca), cemiplimab (Regeneron) and PDR001 (Novart
  • Chemotherapeutic agents used in the method of treatment according to the invention are for example selected from following non-limiting examples: alkylating compounds such as bendamustine (Mundipharma Pharmaceuticals), busulfan (Pierre Fabre), carmustine (Bristol-Myers Squibb), chlorambucil (Aspen), cyclophosphamide (Baxter), dacarbazine (Pfizer), estramustine (Pfizer), ifosfamide (Baxter), lomustine (Kyowa Kirin Pharma), melphalan (GlaxoSmithKline), nimustine (Sankyo), procarbazine (Leadiant Biosciences), streptozotocin (Keocyt), temozolomide (Merck & Co), thiotepa (Adienne), treosulfan (Lamepro) and trofosfamide (Baxter); anthracyclines such as daunorubicin (Meda
  • a PARP inhibitor used in the method of treatment according to the invention are for example selected from following non-limiting examples: Olaparib (3-aminobenzamide, 4-(3-(1-(cyclopropanecarbonyl)piperazine-4-carbonyl)-4- fluorobenzyl)phthalazin-1(2H)-one; AZD-2281; AstraZeneca), rucaparib (6-fluoro-2- [4-(methylaminomethyl)phenyl]-3,10-diazatricyclo[6.4.1.04,13]trideca-1,4,6,8(13)- tetraen-9-one; Clovis Oncology, Inc.); niraparib tosylate ((S)-2-(4-(piperidin-3- yl)phenyl)-2H-indazole-7-carboxamide hydrochloride; MK-4827; GSK); talazoparib (11S,12R)-7-fluoro-11-(4-fluoropheny
  • Example 1 A precise example of performing the method of the invention using pretrained models and generating a refined overall label output
  • the method is applied on a digitized whole slide image (WSI) file of a patient with early stage breast cancer.
  • the first step is Tissue Detection.
  • Optical Density Conversion Transformation of pixel values to optical density (OD).
  • Thresholding Criteria Pixels with OD values > 0.1 are retained. Variance of pixel values across color channels must exceed 0.001.
  • Boundary Exclusion Removal of pixels at the boundary (0.001 x (the smaller dimension)).
  • Object Exclusion Elimination of objects smaller than 0.1 x (the area of the largest object).
  • Elongation Factor Exclusion Discarding objects with an elongation factor (R) greater than 0.95.
  • Tissue Mask Creation Generation of the final tissue mask for subsequent analysis. Following tissue detection, relevant patches are extracted based on the acquired tissue mask. See, an examples, Figures 2 and 3.
  • Tissue Mask Utilization Leveraging the tissue mask to extract patches of size (256 x 256 x 3) from the full-resolution slide image. 2) Patch Details: The number of extracted patches (N) is determined by the quantity of tissue present, resulting in a dataset of dimensions [N, 256, 256, 3]. Following patch extraction, the N patches are encoded into N feature vectors in a step called Feature Extraction. A pre-trained ResNet50 model on the ImageNet dataset for feature extraction is used. 1 ) Model Utilization: Each extracted patch is fed into the ResNet50 model. 2) Feature Vector Generation: Extraction of the output of the 3rd residual block followed by an Average Pooling Layer.
  • Feature Vector Transformation Conversion of each of the N patches into a feature vector of length 1024, resulting in a dataset of dimensions [N, 1024]. 4 ) These extracted feature vectors serve as input to the Core AI model.
  • Next step is to create a Tiny Whole Slide Image that serves as input to the Tiny WSI model.
  • 1 Transformation of the entire whole slide image (WSI) through re-sizing or down-sampling, resulting in an image of dimensions 512 x 512 pixels and 3 color channels.
  • Next step is to generate with the use of the extracted feature vectors an overall label output in the Core AI model and with the use of the down-sampled low- resolution image an image output of a low-resolution image version in the Tiny WSI model.
  • Tiny WSI model is a ResNet50 model where the input is the Tiny WSI (512 x 512 x 3 pixel) and the output is the image output of a low-resolution image version of the stained tissue sample.
  • Core AI model is visualized in Fig. 4. The input to the model is the slide’s encoded feature vectors, [N, 1024] and the output is the overall label output.
  • Next step is to generate a refined overall label output (Digital Mamma Print Index) by the ensemble model.
  • the ensemble model is visualized in Fig. 5.
  • the input to the ensemble model are the output of the Tiny WSI model (image output of a low- resolution image version of the stained tissue sample) and the output of the Core AI model (overall label output).
  • the output is digital MammaPrint Index (refined overall label output). This is the final reported digital MammaPrint Index. See, for example, Fig.6.
  • Example 2 One example of training the models with exemplary training data For training, digitized whole slide image (WSI) files of a patient with early stage breast cancer and MammaPrint Indices obtained by performing MammaPrint genomic test are used. The reported MammaPrint values are coupled with the slides and there is one reported value per patient.
  • WSI whole slide image
  • Model Inputs 1 ) The input to the ensemble model are the predicted MammaPrint Indices of the two Tiny WSI and Core AI models (outputs of these models). 2 ) Utilization of unaltered MammaPrint Indices for model training
  • Model Configuration Deployment of the Core AI model, as visualized in Fig. 4, for the training procedure.
  • Training Parameters The training of the ensemble model involves the utilization of typical training parameters, listed in table 3: Table 3. Typical training parameters.
  • Example 3 Another precise example of performing the method of the invention using pretrained models and generating a refined overall label output The method was applied on digitized whole slide image (WSI) files of patients with early stage breast cancer. Patients included female breast cancer patients with stage I or stage II disease who are lymph node negative or lymph node positive with up to 3 positive nodes, with a tumor size less than or equal to 5.0 cm, and for patients with stage III disease. An overview of the method is visualized in Fig. 1b and Fig.7. The first steps include Tissue Detection and Patch Extraction, which were performed according to Example 1. Hereafter, a tissue type classification task and nuclei detection was performed.
  • WSI whole slide image
  • a Multi-Scale Feature Combination Model was developed using PyTorch.
  • the model utilized ResNet50 as its backbone convolutional neural network (CNN) architecture.
  • CNN backbone convolutional neural network
  • the model included an embedding layer to reduce the dimensionality of the feature representations and a SoftMax-based classifier for tissue type prediction.
  • the model classifies the detected tissue to the following seven tissue types: invasive, infiltrated invasive, non-invasive, lymphocyte, stroma, tumor stroma, and rest (e.g., red blood cells, necrosis, artifacts, etc.).
  • the input to the model comprised three 256 ⁇ 256 pixel patches captured at different magnifications: 20x, 10x, and 5x (see Figure 8).
  • the model gains a more comprehensive understanding of tissue structures and features, enabling more accurate and informed tissue type classification.
  • This multiscale approach effectively captures both fine-grained details and larger-scale patterns within histopathology images, enhancing classification accuracy and robustness.
  • each dot represents the predicted label for a 256x256 pixel tissue patch.
  • Nuclei Detection For nuclei detection, the advanced AI model called ‘You Only Look Once’ version 8 (YOLOv8) was used to process each patch extracted during the Patch Extraction step. To balance inference time and performance, the small version of YOLOv8 (YOLOv8s, Jocher et al., 2023. Ultralytics, available at github.com/ultralytics/ultralytics) with 11.2 million parameters was chosen for the task.
  • YOLOv8s is capable of detecting and classifying nuclei into four categories: tumor, stroma, lymphocyte and others, as shown in Figure 10.
  • a dataset consisting of around 180k patches, extracted from 82 H&E slides was used to train the model. The dataset was randomly split into training and validation sets, with 80% of the data allocated for training and 20% for validation. During training, the hyperparameters were set as follows: input tensor size of 256, batch size of 60, and 50 epochs. Digital tumor percentage module Tissue type classification and nuclei detection models were applied to each identified tissue patches. As a result, for every tissue patch, information about its tissue type and the count and types of nuclei within it was obtained.
  • a patch selection criteria was applied in this example as follows: “invasive”: 1024; “infilterated_invasive”: 512, “noninvasive”: 256, “lymphocyte”: 256, “stroma”: 128, “tumor_stroma”: 256, “rest”: 0.
  • Pathology self-supervised features In contrast to the ImageNet pre-trained feature extractor of Example 1, in this example a pathology self-supervised feature extraction was performed.
  • Self-supervised learning (SSL) which leverages large amounts of unlabeled data, has shown to have superior performance in pathology specific tasks over supervised ImageNet pre-training in certain instances.
  • a well-known SSL framework is DINOv2 (Vorontsov et al., 2024. Nat Med.
  • T he input comprised: ( ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ) dimensional patches sampled from WSIs, where M is the total number of patches within the whole slide, and (H x W x C) are the height-width and the number of color channels in each patch, respectively (such as 256 x 256 x 3).
  • the tissue-based patch selection process takes an initial set of ⁇ patches and reduces it to a subset of ⁇ patches where ⁇ ⁇ ⁇ based on predefined criteria (see, for example, Fig. 7). To achieve this, access to a ⁇ ⁇ ⁇ discrete tissue classification result for each patch was assumed, where ⁇ represents the number of unique tissue categories.
  • ⁇ denotes the relative percentage of patches to be sampled from each tissue category.
  • ⁇ denotes the relative percentage of patches to be sampled from each tissue category.
  • the sum of all percentages satisfies the condition: ⁇
  • ⁇ ⁇ is the percentage of tissue category ⁇ .
  • the parameter ⁇ is a hyperparameter that can be tuned to optimize the overall performance of the downstream prediction model.
  • the regression layer is a simple 1-layer Multi-Layer Perceptron with weight dimensions ⁇ ⁇ 1 as our target prediction index is a scalar value. After the model makes a prediction, then this prediction is used to calculate the loss function, and finally update the respective layer parameters.
  • the Patch-based end-to-end module model was trained using a Mean Squared Error (MSE) loss to minimize the discrepancy between the predicted index and the ground truth: ⁇ where B is the batch size, and the ⁇ ⁇ , ⁇ is the ground truth predicted index of a given whole slide image of a patient ⁇ .
  • MSE Mean Squared Error
  • the patch embedding backbone is decomposed into layers ⁇ layer0, layer1, ... , layer ⁇ .
  • a selective combination of these layers e.g. last few layers of the embedder
  • This flexibility allows optimization of critical features while preventing over-fitting.
  • the model using ADAMW Lishchilov, 2017.
  • the ensemble model is the same as in previous examples, only the input differs.
  • the slide-level index prediction ypred obtained as output of patch-based end-to-end module is used.
  • the local patch-level features are concatenated to the patch-level features of the other models, here from the self-supervised model.
  • the global slide- level features and the slide-level index prediction ypred is concatenated to the slide- level features of the Core AI module before predicting the final digital MammaPrint index. This is the final reported digital MammaPrint Index.
  • Example 4 Prediction of recurrence of breast cancer using the digital MammaPrint Index in an independent dataset.
  • Example 3 The analysis is done using the model described in Example 3 on whole-slide images and the clinical outcomes of 261 patients, obtained from two separate cancer centers (NorthShore and Fox Chase Cancer Center) in the United States from 1992-2010. Of the 261 analyzed patients, 145 were classified as dMP High-Risk and 116 were classified as dMP Low-Risk. From the 261 patients, 15 developed distant recurrence, i.e. metastasis (event). A Kaplan-Meier survival analysis is shown in Figure 11, showing that the digital MammaPrint Index is a very accurate predictor of breast cancer recurrence. As can be seen, there was only one distant recurrence in the low risk group, while 14 distant recurrences were observed in the high risk group.
  • Fig.11 the probability of survival shown on the y-axis represents occurrences of breast cancer recurrence, more specifically distant recurrence, also called metastasis.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Investigating Or Analysing Biological Materials (AREA)

Abstract

L'invention concerne un procédé mis en oeuvre par ordinateur pour évaluer un risque de récidive du cancer du sein, un programme informatique ayant des instructions qui, lorsqu'elles sont exécutées par un dispositif ou un système informatique, amènent le dispositif ou le système informatique à mettre en oeuvre le procédé ainsi qu'un système de traitement de données comprenant des moyens pour mettre en oeuvre le procédé d'évaluation d'un risque de récidive du cancer du sein. Le procédé de l'invention est par exemple réalisé par traitement d'image de diapositive entière (WSI), le risque de récidive du cancer du sein étant évalué à partir d'une WSI d'un tissu tumoral.
PCT/NL2025/050040 2024-01-24 2025-01-24 Procédé d'évaluation d'un risque de récidive du cancer du sein Pending WO2025159640A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP24153728.1 2024-01-24
EP24153728 2024-01-24

Publications (1)

Publication Number Publication Date
WO2025159640A1 true WO2025159640A1 (fr) 2025-07-31

Family

ID=89715883

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/NL2025/050040 Pending WO2025159640A1 (fr) 2024-01-24 2025-01-24 Procédé d'évaluation d'un risque de récidive du cancer du sein

Country Status (1)

Country Link
WO (1) WO2025159640A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014151634A1 (fr) 2013-03-15 2014-09-25 Bristol-Myers Squibb Company Inhibiteurs macrocycliques des interactions protéine-protéine pd-1/pd-l1 et cd80(b7-1)/pd-l1
WO2015034820A1 (fr) 2013-09-04 2015-03-12 Bristol-Myers Squibb Company Composés utiles comme immunomodulateurs

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014151634A1 (fr) 2013-03-15 2014-09-25 Bristol-Myers Squibb Company Inhibiteurs macrocycliques des interactions protéine-protéine pd-1/pd-l1 et cd80(b7-1)/pd-l1
WO2015034820A1 (fr) 2013-09-04 2015-03-12 Bristol-Myers Squibb Company Composés utiles comme immunomodulateurs

Non-Patent Citations (19)

* Cited by examiner, † Cited by third party
Title
CARON ET AL., NEURIPS, vol. 33, 2020, pages 9912 - 9924
DUBEY, S.SINGH, SCHAUDHURI, B., NEUROCOMPUTING, 2022
EDUARDO CONDE-SOUSA ET AL: "HEROHE Challenge: assessing HER2 status in breast cancer without immunohistochemistry or in situ hybridization", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 8 November 2021 (2021-11-08), XP091095763 *
GRAHAM ET AL., MEDICAL IMAGE ANALYSIS, vol. 58, 2019, pages 101563
HE ET AL., ARXIV:1703.06870, 2017
HE K.ZHANG, X.SHAOQING RSUN, J.: "Deep Residual Learning for Image Recognition", ARXIV:1512.03385, 2015
ILSE, M.TOMCZAK J. M.WELLING, M., ARXIV:1802.04712, 2018
JOCHER ET AL., ULTRALYTICS, 2023, Retrieved from the Internet <URL:github.com/ultralytics/ultralytics>
KANG ET AL., PROCEEDINGS OF THE IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2023, pages 3344 - 3354
LIN ET AL., ARXIV:1708.02002, 2017
LU ET AL., NAT BIOMED ENG, vol. 5, 2021, pages 555 - 570
MAXIMILIAN ILSE ET AL: "Attention-based Deep Multiple Instance Learning", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 13 February 2018 (2018-02-13), XP081221608 *
MUSTAFA UMIT ONER ET AL: "Weakly Supervised Clustering by Exploiting Unique Class Count", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 18 June 2019 (2019-06-18), XP081585509 *
ROBERT-FLORIAN SAMOILESCU ET AL: "Model-agnostic and Scalable Counterfactual Explanations via Reinforcement Learning", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 4 June 2021 (2021-06-04), XP081984045 *
SHAO ET AL., ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS, vol. 34, 2021, pages 2136 - 2147
SRIVASTAVA ET AL., J MACHINE LEARNING RES, vol. 15, 2014, pages 1929 - 1958
TRIPATHI SUVIDHA ET AL: "An end-to-end breast tumour classification model using context-based patch modelling - A BiLSTM approach for image classification", COMPUTERIZED MEDICAL IMAGING AND GRAPHICS, PERGAMON PRESS, NEW YORK, NY, US, vol. 87, 4 December 2020 (2020-12-04), XP086455421, ISSN: 0895-6111, [retrieved on 20201204], DOI: 10.1016/J.COMPMEDIMAG.2020.101838 *
VAN 'T VEER ET AL., NATURE, vol. 415, 2002, pages 530 - 536
VORONTSOV ET AL., NAT MED, vol. 30, no. 2924, 2024, pages 2935

Similar Documents

Publication Publication Date Title
Van Rijthoven et al. HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images
US20240069026A1 (en) Machine learning for digital pathology
Sha et al. Multi-field-of-view deep learning model predicts nonsmall cell lung cancer programmed death-ligand 1 status from whole-slide hematoxylin and eosin images
EP4000014A1 (fr) Réseaux neuronaux convolutionnels pour classification d&#39;images histologiques de cancer
JP2025503388A (ja) 深層学習を用いた、組織の無標識仮想免疫組織化学染色
Kanchana et al. Enhancing skin cancer classification using efficient net b0-b7 through convolutional neural networks and transfer learning with patient-specific data
US12100150B2 (en) Droplet imaging pipeline
Swiderska-Chadaj et al. Convolutional neural networks for lymphocyte detection in immunohistochemically stained whole-slide images
Liu et al. Identification of lymph node metastasis in pre‐operation cervical cancer patients by weakly supervised deep learning from histopathological whole‐slide biopsy images
Koo et al. Non-annotated renal histopathological image analysis with deep ensemble learning
Tasnim et al. Classification of breast cancer cell images using multiple convolution neural network architectures
Pereira et al. Deep convolutional neural network applied to Trypanosoma cruzi detection in blood samples
Das et al. Deep Learning for Classification of Inflammatory Bowel Disease Activity in Whole Slide Images of Colonic Histopathology
Guetarni et al. A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models
JP2025517869A (ja) 三次リンパ組織様構造の機械学習識別、分類、及び定量化
WO2025159640A1 (fr) Procédé d&#39;évaluation d&#39;un risque de récidive du cancer du sein
WO2022226284A1 (fr) Quantification de l&#39;écosystème immun tumoral dans le cancer du poumon non à petites cellules (cpnpc) destinée à identifier des biomarqueurs cliniques de la réponse thérapeutique
CN115082718A (zh) 基于组织病理图像的胶质瘤分级方法、装置、设备及介质
Chen et al. EdgeNeXt-SEDP for cervical adenocarcinoma HPV-associated and non-HPV-associated diagnosis and decision support
Chauhan et al. Contrasting Low and High-Resolution Features for HER2 Scoring using Deep Learning
Zheng et al. UC-YOLOX: Enhancing urothelial carcinoma detection with an improved YOLOX architecture leveraging attention mechanisms
Alawfi Hybrid Capsule Network for precise and interpretable detection of malaria parasites in blood smear images
Zhou et al. Prediction of neoadjuvant therapy response in breast cancer based on interpretable artificial intelligence
Amar et al. Advanced glaucoma detection through retinal fundus image segmentation and stacking classifier
Hossain et al. Gastrointestinal Insights Redefined: An Integrated Hybrid Model Fusing Vision Transformer and Transfer Learning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25701723

Country of ref document: EP

Kind code of ref document: A1