WO2024233990A2 - Artificial intelligence system for tumor diagnostics of clinical magnetic resonance imaging datasets - Google Patents

Abstract

A deep-learning artificial intelligence (AI) framework that applies neuromorphic analysis to complex clinical MRI DICOM datasets to enable typing of brain tumors that are glioblastomas or of metastatic origin with high accuracy. The data processing is optimized to address MRI modality class imbalance, remove lack of comparability, and aid generalizability to create a computationally light, portable, highly accurate and reliable diagnosis platform. The present AI capabilities to continue learning from complex and diverse clinical MRI datasets brings MRI robustness in biomedical neurosurgical treatment pre-planning, with applications in other brain pathologies and expanded use in artificial generalizable intelligence.

Description

ARTIFICIAL INTELLIGENCE SYSTEM FOR TUMOR DIAGNOSTICS OF CLINICAL MAGNETIC RESONANCE IMAGING DATASETS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Nos. 63/465,719, filed May 11 , 2023, and 63/617,148, filed January 3, 2024, the entireties of which are hereby incorporated herein by this reference.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention generally relates to artificial intelligence systems and medical diagnostics. More particularly, the present invention relates to a system and method for diagnosing brain tumors from clinical magnetic resonance imaging (MRI) data images though the use of artificial intelligence models.

[0004] 2. Description of the Related Art

[0005] In general, Brain tumor neoplasms arise via cerebral central nervous system (CNS) primary origin or by metastasis from alternate sites of pathology, including tumors of the lung, breast and melanomas, and rank globally as the 10th leading cause of death in US. The average age adjusted incidence rate (AAIR, 2012-2016) of all CNS tumors is 23.4 per 100,000. Brain metastases are most common versus primary intracranial tumors and approximately 30% of brain and CNS tumors are malignant, with tumor types and grades varying with age and demographics. Of malignant primary brain tumors, gliomas dominate at ~75% of tumors and include glioblastoma multiforme (GBM, 14.6 %) that are most aggressive. The highly invasive nature of GBMs require radical tumor resection in treating the disease, with poor survival rates of <5% of patients surviving 5 years after diagnosis.

[0006] Neurosurgical strategies of treatment differ between cerebral metastases or glioblastomas, yet differentiating tumor types via magnetic resonance imaging (MRI) radiology remains challenging and impacts medical workup and treatment. A reliance on a range of information is required such as patient history of systemic malignancy and tumor features such as cerebral location near grey or white matter, multiplicity of cerebral sites, MR tumor morphology, and MR perfusion for tumor vascularity to provide predictive insight, although none of these factors are absolute. Intraoperative biopsies for genetic analysis further allow biomarker typing of tumors which assists in treatment. Despite these challenges, MRI remains a primary tool for initial guidance of tumor pathology.

[0007] One of the first demonstrated uses of MRI was its ability to resolve physiologic detail of normal versus pathological tumors in a rat model. A decade later, clinical use of 1.5 and 2.0 Tesla MRI on human patients was implemented and recent FDA approval of 7.0 Tesla MRI brings possibilities for very significant levels of physiological detail. The reliance on trained medical experts to evaluate MRI data is expected to become more strained with technology advances that require additional training, adding to time demands on radiologists to review and turnaround diagnostics on images. Given the increasing demand on radiologist and neurosurgeon clinicians in patient care, the early implementation of artificial intelligence for deep learning analysis of clinical data for MRI has become imperative.

[0008] Currently development of cutting-edge Al platforms has focused on detection of tumor pathology, such as the deep learning SCENIC algorithm developed by this group and tumor segmentation for detecting tumor edges. To what extent preprocessing of clinical data or synthesized data applied to Al development, data training and validation introduces noise or results in loss of data remains relatively unexplored. For example, complex operations such as skull stripping and MR bias field correction are sometimes applied. It is important to achieve comparable performance on medical data for generalizability and high transferability to clinical settings.

[0009] One previously developed system was as a deep learning neoplasm identification platform, with 98.3% classification accuracy on the publicly available BraTS Dataset. Although accurate for identifying tumor presence across MRI modalities, the previous design did not permit differentiation of glioblastomas or metastasized tumors (herein referred to as GBM or MET). Although clinical data is abundant, acquisition agreements with hospitals are required to ensure patient privacy through Institutional Review Board (IRB) agreements and clinical expertise and additional diagnostics must be present to verify tumor type.

[0010] It is thus to addressing the problems in existing Al models that can assist with prediction and treatment of brain tumors, as well as other cancer diagnoses, that the present invention is primarily directed. BRIEF SUMMARY OF THE INVENTION

[0011] Briefly described, the present system and method is a deep learning platform, called SIENNA, developed with access to a non-processed de-identified clinical DICOM dataset of clinically identified brain tumors, and non-tumor scans. Described herein are key challenges and strategies in working with clinical data that employ cutting-edge algorithms and tools to guide optimizations. This includes Synthetic Minority Over-Sampling Technique (SMOTE) to address class imbalance, “Hyperas” for optimizing hyper-parameters, histogram equalization for scan standardization, and adversarial training to ensure model robustness against different types of adversarial attacks. Capitalizing on the distinctive attributes of the convolutional neural network (CNN) architecture and its capacity to extract intricate hierarchical features from data, the clinical data optimized SIENNA model exhibits robust tissue pathology typing, demonstrated by discrimination of GBM versus MET tumor pathologies. The versatility of SIENNA in handling both raw and processed clinical data renders it an indispensable resource for enhancing clinical treatment workflow speed and efficacy to promote superior patient outcomes.

[0012] In one embodiment, the invention includes a system for determining the presence of cancerous tumors within diagnostic images of a human body. There is a scanning device generating one or more diagnostic images of a human body, such as an MRI device. The scanning device selectively transmits the one or more diagnostic images to other computer devices either through a wired or wireless connection, and there is a receiving computer device configured to process the received images. The computer device is configured to receive the one or more diagnostic images from the scanning device, create a plurality of tiles for each received image, compute a histogram for each tile, redistribute an intensity for each tile, create a cumulative distribution function scaling for each tile, conduct a bilinear interpolation to thereby stitch together the processed tiles, combine the processed tiles into in a final equalized image, and analyze the final equalized image to determine a presence of cancerous tumors within the diagnostic images.

[0013] In an embodiment, the invention also includes a computer-implemented method for determining the presence of cancerous tumors within diagnostic images of a human body. The method can be implemented by performing the steps of generating one or more diagnostic images of a human body at a scanning device, selectively transmitting the one or more diagnostic images from the scanning device to other computer devices, receiving the one or more diagnostic images from the scanning device, creating a plurality of tiles for each received image, computing a histogram for each tile, redistributing an intensity for each tile, creating a cumulative distribution function scaling for each tile, conducting a bilinear interpolation to thereby stitch together the processed tiles, combining the processed tiles into in a final equalized image, and then analyzing the final equalized image to determine a presence of cancerous tumors within the diagnostic images.

[0014] The present invention therefore provides an advantage in the diagnosis of tumors. Further, the present invention is industrially applicable as it provides a medical diagnostic system to aid healthcare providers in giving an accurate medical diagnosis and providing effective treatment. Other advantages, features, and benefits of the present invention will be apparent to one of skill in the art after review of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Fig. 1 is a diagram of one embodiment of the present system utilizing an MRI machine to provide diagnostic images of a human brain.

[0016] Fig. 2A is an overview and representation of MRI clinical dataset illustrating: a. Clinical workflow from MRI scans to SIENNA output interpretation by neurosurgeons, including collection, preprocessing, input, and output interpretation of data

[0017] Fig. 2B is a series of diagnostic images that are examples of de-identified raw MRI scans deemed noisy and unfit for training.

[0018] Fig. 2C is a series of MRI scans across different modalities present in the clinical dataset are shown along with representative images from GBM and MET.

[0019] Fig. 3A is a series of images and graphs illustrating histogram equalization. [0020] Fig. 3B is a series of MRI images adversarial examples used in the training process.

[0021] Fig. 3C is a table of Hyperas describing the various parameters and their respective plausible ranges of values provided for hyperparameter tuning.

[0022] Fig. 3D is a chart of the comparison of histogram equalization algorithms tested and evaluated using the SSIM index. [0023] Fig. 4A illustrates the SIENNA CNN Multilayer CNN

[0024] Fig. 4B is a table summarizing the layers and output of the CNN architecture - includes the number of layers, output shape, and the number of parameters.

[0025] Fig. 4C is a diagram illustrating the SIENNA data processing pipelines.

[0026] Fig. 5A is a series of MRI images highlighting the most frequently correctly and incorrectly classified samples.

[0027] Fig. 5B is a table of the performance evaluation on clinical data of the evaluation metrics estimated as an average of 100 runs of SIENNA.

[0028] Fig. 5C is a table of the performance evaluation of SIENNA for binary classification tasks on the extensively preprocessed BraTS 2020 dataset.

[0029] Fig. 5D is a graph of the comparative analysis of the performance of SIENNA and SCENIC, the state-of-the-art model, in the binary classification task of clinical data.

[0030] Fig. 6A is a series of MRI images of “patient 3” with known GBM, subjected to preprocessing, and the corresponding output of the SIENNA model for classification of tumor types.

[0031] Fig. 6B is a series of MRI images of “patient 1” with known MET, subjected to preprocessing, and the corresponding output of the SIENNA model for classification of tumor types.

[0032] Fig. 6C is a table that presents the results of SIENNA's patient-wise analysis, showing the F1 score along with the values of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) for each input.

DETAILED DESCRIPTION OF THE INVENTION

[0033] Referring now to the drawings in which like numbers represent like elements throughout the several views, Fig. 1 is a diagram of one embodiment of the present system 10 utilizing an MRI machine 12 as the scanning device to provide diagnostic images of a human brain, such as images shown in Fig. 2B-2C. In this embodiment, the system 10 for determining the presence of cancerous tumors within diagnostic images of a human body includes a scanning device, here MRI 12 generating one or more diagnostic images of a human body, which here are brain images from the head 16 or a patient, such as the images shown in Figs. 2B-2C. The scanning device (MRI 12) selectively transmits the one or more diagnostic images to other computer devices, such as computer device 14. Computer device 14 can be one or more physical computer platforms, or could be virtual drives located in the Cloud. And the transmission can be from a hardwired connection 18 or through a network and wireless connection.

[0034] The computer device 14 is then configured, via programming in software, middleware, firmware, or a combination thereof, to perform the steps of receiving the one or more diagnostic images from the scanning device (MRI 12), creating a plurality of tiles for each received image, as further described below, and then computing a histogram for each tile, as shown in Fig. 3A. The computer device 14 then, as further detailed below, distributes an intensity for each tile, creating a cumulative distribution function scaling for each tile, conducts a bilinear interpolation to thereby stitch together the processed tiles, and then combines the processed tiles into in a final equalized image, analyzing the final equalized image to determine a presence of cancerous tumors within the diagnostic image, as shown in Figs.5A-6C. [0035] In one embodiment, the analyzing the final equalized image is analyzing the final equalized image with a convolutional neural network. Further, the computer device 14 can be further configured to perform the step of training a predictive model based upon the analyzing of the final equalized images. Additionally, the computer device 14 can further be configured to perform the steps of producing a training data based upon the received one or more diagnostic images, introducing adversarial perturbations in the training data to train the predictive model.

[0036] When the one or more diagnostic images are MRI images, such as in Figs. 2B-2C, the computing of a histogram is computing a histogram equalization, as shown in Fig.3A. And as embodied here, the diagnostic images are one or more images of a human brain, but other images and scanning techniques can likewise be used in the present invention for medical diagnoses, as would be apparent to one of skill in the art.

Further, as embodied herein, the analyzing the final equalized image to determine a presence of cancerous tumors within the diagnostic images can be determining a presence of glioblastomas, a further described below. As also described herein, the training of the predictive model can include tuning of hyper-parameters from the group of a weight penalty factor, learning rate, and momentum.

[0037] One constructed embodiment of the present system 10 is called SIENNA, a deep learning platform developed with access to a non-processed de-identified clinical DICOM dataset of 386 scans, comprising 153 GBM, 125 MET clinically identified brain tumors, and 108 non-tumor scans. SIENNA system addresses key challenges and strategies in working with clinical data and employ cutting-edge algorithms and tools to guide optimizations. This includes Synthetic Minority Over- Sampling Technique (SMOTE) to address class imbalance, Hyperas for optimizing hyper-parameters, histogram equalization for scan standardization, and adversarial training to ensure model robustness against different types of adversarial attacks. Capitalizing on the distinctive attributes of the convolutional neural network (CNN) architecture and its capacity to extract intricate hierarchical features from data, our clinical data optimized SIENNA model exhibits robust tissue pathology typing, demonstrated here by discrimination of GBM versus MET tumor pathologies. The versatility of SIENNA in handling both raw and processed clinical data renders it an indispensable resource for enhancing clinical treatment workflow speed and efficacy to promote superior patient outcomes.

[0038] Fig. 2A is an overview and representation of MRI clinical dataset process flow, illustrating the clinical workflow from clinical data intake 22, to MRI image scans 24, to SIENNA processing 26, to output interpretation 28 by neurosurgeons, including collection, preprocessing, input, and output interpretation of data. Fig. 2B is a series of images 30 are examples of de-identified raw MRI scans deemed noisy and unfit for training. Fig. 2C is a series of images 40 of MRI scans across different modalities present in the clinical dataset are shown along with representative images from GBM and MET that are either tumor (top panel 42) or non-tumor (bottom panel 44).

[0039] The retention of data features during pre-processing of DICOM images. SIENNA is designed to fit seamlessly into the clinical workflow with DICOM images (Figs. 2A-2B) and to achieve greatest retention of raw features for distinguishing GBM versus MET tumor neoplasms (Figs. 2B-2C). Particular challenges intrinsic to working with clinical datasets were anticipated but avoided through common extensive preprocessing techniques such as re-orientation to LPS/RAI, N4 Bias correction, and Skull-stripping, which are seen with publicly available data. Clinical data evaluation includes addressing factors of data de-identification and clinical expert verification as well as natural patient and tumor variations contributing to noise that must be minimized, data annotation, and metadata creation. Here we evaluated de-identified clinical data obtained through an IRB-approved collaboration with Albany Medical College and structured in a patient-centric manner, with GBM and MET patient folders comprising multiple imaging modalities including T1 , T1- contrast-enhanced (T1-ce), T2-weighted (T2), and fluid-attenuated inversion recovery (FLAIR). Data from 20 patients is evaluated, 10 belonging to each of the GBM and MET categories.

[0040] Initial modality screening revealed value of T2 and FLAIR modalities for highest accuracy, consistent with their preferred use in MR diagnostics clinically. Given the high dimensionality of FLAIR and its ability to effectively distinguish tumor physiology, this modality was selected for all subsequent analyses. To facilitate effective training, validation, and testing of the SIENNA deep learning model, we segregated each patient Z-stack of horizontal MRI scans into tumoral and non- tumoral classes. Horizontal images that contained incomplete tissue data (uppermost images, primarily skull area) or heightened data complexity (lowermost images, primarily non-brain tissues) were removed (Fig. 2B) to reduce noise from limitations of a small dataset. All MRI scans were annotated with a uniform standardized naming format across the 20 patients to enable ease of analysis and verification during the testing phase. Shown are MRI scans across different MRI modalites (Fig. 2C), including GBM and MET tumors and non-tumor samples across modalities.

[0041] The variable sizes of brain tumors in patient datasets impacts the number of non-tumor data images within a given patient horizontal Z-stack. When the tumor is large, the tumor class constitutes the majority of the horizontal slices in a patient dataset and the non-tumor scan becomes a minority class. This class imbalance can lead to biased models that favor the majority class and fail to learn the patterns and features of the minority class adequately. To address this issue, data augmentation techniques can be employed to restore class balance within and across patient datasets. The Synthetic Minority Over-Sampling Technique (SMOTE) is a popular method for minority class oversampling, which involves generating synthetic samples by interpolating between minority class examples. By replicating samples using k- nearest neighbors of the low-frequency class, the SMOTE technique forces the network to allocate resources to the analysis of the minority class, thereby improving model performance. [0042] Figs. 3A-3D show a model optimization for stability and robustness using histogram equalization, Adversarial Training and Hyperas. These are applied selected techniques for enhancing stability and robustness of deep learning models, such as histogram equalization and adversarial training that are presented here alongside their respective evaluation metrics and results. Fig. 3A is a series of images 50 and graphs 52 illustrating histogram equalization, which showcases improved contrast equalization of the images in the dataset, represented along with histograms of original and enhanced scan. Fig. 3B is a series of MRI images illustrating original MRI scans 60 and visualizing some adversarial examples 62 used in the training process.

[0043] The variability in MRI scan intensities and quality across patients due to patient movement, technical characteristics of each MRI system, acquisition parameters and scanning conditions can create incomparability in datasets. When each of the scans have varying pixel intensities, the features the deep learning model learns will vary as well. We applied histogram equalization to address this issue by normalizing scan intensity and reducing sensitivity to illumination and other factors that can impact feature visibility (Figs. 2A, 2D). This facilitated better differentiation and recognition of distinct brain features by the model.

[0044] The present system allows augmentation of SIENNA by parameter and hyper-parameter optimization and adversarial training. SIENNA employs a multiple CNN framework to perform feature engineering and extraction of complex nonlinear dependencies from input images mapped to output classes. CNNs are the Al method of choice for complex medical data sets to bring greatest flexibility in handling a broad range of parameters and hyper-parameters critical for data optimization and analysis. While parameters like weight and bias are learned during training, hyperparameters such as weight penalty factor, learning rate, and momentum are tuned features. Model performance requires trial and error experimentation of different parameter combinations while retaining greatest generalizability.

[0045] Fig. 3C is a table of Hyperas describing the various parameters and their respective plausible ranges of values provided for hyperparameter tuning. We applied the HYPERAS algorithm to maximize performance on new data. The algorithm serves as a convenient interface for the Hyperopt library and automates the search for optimal hyper-parameters by exploring a range of user-defined search spaces. Various modeling strategies were applied to the data to estimate the achievable level of test error performance, with recall serving as the performance metric to be optimized by HYPERAS (Fig. 2C). In medical settings Al diagnostics may generate patient critical decisions in which avoiding misdiagnosis is essential. Image classification accuracy can be defined by metrics from the confusion matrix, such as true positive (TP), true negative (TN), false positive (FP), and false negative (FN) image analysis. To measure the discrepancy between the predicted and true values of the outputs by SIENNA, the loss function categorical cross-entropy, along with the false positive and false negative metrics, were used as the objective function during the optimization process. False positives and false negatives quantify misclassified scans and can be passed as loss functions to optimize the model towards decreased gradient.

[0046] Fig. 3D is a chart 64 of the comparison of histogram equalization algorithms - Visualization of different histogram equalization algorithms tested and evaluated using the SSIM index. CNN models offer a versatile set of functions to optimize but require avoidance of common deep learning issues of overfitting and outlier sensitivity in neural networks. To address this problem, adversarial training was utilized to introduce ‘adversarial examples’ as training samples, which are slightly modified to intentionally cause misclassification by machine learning classifiers. The augmentation process greatly enhanced SIENNA ability to distinguish between pertinent and irrelevant features, thereby improving the model's overall performance (Fig. 2B). However, it should be noted that noise introduced into the training data must be calibrated with careful consideration, as excessive noise has the potential to undermine the model's ability to identify meaningful patterns and attributes. An essential balance between the amount and nature of the noise introduced into the training dataset, gained by familiarity with the data features, helps to ensure that the model achieves optimal performance.

[0047] The SIENNA CNN architecture and data processing pipeline are shown in Figs. 4A-4C. The workflow chart of the SIENNA model indicates data optimization and model optimization at different stages of development, leading to the final optimized SIENNA model. Fig. 4A. illustrates a multilayer CNN 70, consisting of several convolutional and pooling layers, followed by fully connected layers and a final output layer. The depicted layers in the architecture possess both spatial and depth dimensions, which serve to illustrate the data shape and its transformation as it passes through the network.

[0048] Fig. 4B is a table summarizing the layers and output of the CNN architecture - includes the number of layers, output shape, and the number of parameters. It is also shown that SIENNA excels at binary and multiclass pathology classification tasks on diverse datasets. An assessment of SIENNA's performance in both binary and multiclass classification tasks. A non-interdependent multi-classification analysis can be used to reduce misdiagnosis risk. The subtasks classify input MRI scans into one of three categories, namely Non-Tumor, Glioblastoma (GBM), or Metastatic (MET). The positive classes for each subtask in the SIENNA model serve to emphasize the distinctive characteristics of each type of tumor, thereby enabling the model to discern between them. The final classification label of the input MRI scan is obtained using a comparative analysis of the results, for each subtask. This represents the SIENNA decision output for classification of the MRI scan as GBM or MET. Our evaluation of SIENNA’s performance involves the use of various performance metrics such as accuracy, F1 score, Area Under the Receiver Operating Characteristic Curve (AUROC), as well as the number of False Positives (FP) and False Negatives (FN) [29], The MR images for True Positive (TP), True Negative (TN) and FP and FN outcomes (Fig. 5A) and assessments (Fig. 5B) reveal the crucial role of an integration-based strategy in the development and optimization of deep learning models for complex medical imaging tasks. Furthermore, our 3-label multiclass classification model is more efficient and provides more detailed and nuanced predictions compared to a hierarchical 2-binary classification stacked model.

[0049] Fig. 4C is a diagram illustrating the SIENNA data processing pipelines. To further evaluate the performance of SIENNA, the model was tested on the highly pre-processed BraTS 2020 dataset of MRI data 74, which is a clinically acquired dataset of lower and higher grade GBMs. The BraTS dataset has been optimized to provide high-quality medical imaging data by using intensive preprocessing techniques like intensity normalization, bias field correction, and noise reduction. The dataset consists of NITTI files divided into four different MRI modalities, each containing 155 sliced images of dimension 240x240x3. Although this dataset is limited to the tumor type Glioma, it serves as a standardized resource for the development and evaluation of computer-aided diagnosis (CAD) systems for brain tumor segmentation. SIENNA was able to classify tumors accurately (performance metrics 76) on a subset of the BraTS2020 dataset consisting of 20 patients with a F1 score of 97% and 96% Accuracy (Figs. 5C-5D). One of the strengths of SIENNA is its versatility in accommodating both clinical and extensively pre-processed data, which makes it a valuable and adaptable tool for different medical applications. [0050] Figs. 5A-5D illustrate a collective data analysis of SIENNA's high accuracy rates in discriminating GBM and MET tumors. The figure illustrates the stratification of MRI scans into distinct categories and presents the evaluation metrics for the SIENNA model across various tasks, including classification accuracy for Task 1 (Tumor vs Non Tumor), Task 2(GBM vs Non-GBM), and Task 3(MET vs Non-MET) averaging an accuracy range of 88-91%.

[0051] Fig. 5A is a series of MRI images highlighting the most frequently correctly and incorrectly classified samples , with the top 20 MRI images that were classified with the highest and lowest accuracy for each task are shown. Specifically, a nontumor versus tumor images group 80, a glioblastoma version non-glioblastoma group 82, and a metastasis versus not non-matastasis group 84 are illustrated.

[0052] Fig. 5B is a table of the performance evaluation on clinical data that represents the evaluation metrics estimated as an average of 100 runs of SIENNA. Fig. 5C is a table of the performance evaluation of SIENNA for binary classification tasks on the extensively preprocessed BraTS 2020 dataset. Fig. 5D is a graph 86 of the comparative analysis of the performance of SIENNA and SCENIC, the state-of- the-art model, in the binary classification task of clinical data. The graph 86 shows accuracy in on a binary 0-1 scale.

[0053] Figs. 6A-6C illustrates a patient-wise performance evaluation of SIENNA. Fig. 6A is a series of MRI images of “patient 3” with known GBM, subjected to preprocessing, and the corresponding output of the SIENNA model for classification of tumor types. The test set 90 is interpreted by SIENNA and a true positive set 92 is created, along with a false positive set 94 and a true negative set 96.

[0054] Fig. 6B is a series of MRI images of “patient 1” with known MET, subjected to preprocessing, and the corresponding output of the SIENNA model for classification of tumor types. The test set 100 is interpreted by SIENNA and a true positive set 102 is created, along with a false positive set 104 and a true negative set 106.

[0055] Fig. 6C is a table that presents the results of SIENNA's patient-wise analysis (P1, P2, P3, P8, P9), showing the F1 score 108, along with the values of true positive (TP) 110, true negative (TN) 112, false positive (FP) 114, and false negative (FN) 116 for each input. The MRI images of the groups are then displayed.

[0056] Accordingly, SIENNA evaluates patient-specific GBM or MET tumor. A personalized assessment of the model’s performance on individual patient cases, typical in clinical practice, was performed for Figs. 6A-6C. Detailed patient-wise evaluation provides an opportunity to identify previously unrecognized patterns or relationships within the data, leading to new insights in the field of brain tumor diagnosis and treatment. The efficacy of the model in precise classification of brain tumors with minimal false positives and negatives was exhibited through the patientwise processing of input data for GBM-3 (Fig. 6A) and MET-1 (Fig. 6B) patient cases and evaluation of performance metrics (Fig. 6C). In summary, the development and optimization of SIENNA as a multiclass classification model for diagnostic distinction of clinical MRI scans as GBM and MET tumors demonstrates deep learning efficiency and high accuracy for brain tumor classification.

[0057] For SIENNA performance-enhancing design strategies and optimization techniques, model training, validation and testing. The fine-tuning of a neural network model for a specific problem statement and diverse data complexities relies on the optimal selection of hyper-parameters and model parameters, learned through training and updated via backpropagation. Parameters can be defined as internal configurations that determine the model's functionality and must be finetuned to create a robust and reliable deep learning platform. The training process is important in this aspect as it is the set from which the neural network is able to extract features to distinguish classes. Convolution is an efficient technique for feature extraction, reducing data dimensionality and generating feature maps using kernels as feature identifiers. However, data diversity and richness in the training set are crucial for optimal feature extraction. Unfortunately, there are no established criteria for distinguishing information-rich scans from clinical datasets, which are often provided in a patient-wise format of successive z-slice brain stacks. To address this, shuffling the datasets randomly can be helpful. During training, it is important to avoid presenting the model with successive scans belonging to the same class, as these are likely to contain similar information.

[0058] A large error between the target and network output can indicate new information in the data and represents the kind of data the model has not been trained upon. In cases where a certain MRI type presents unique complexities, images that are frequently misclassified can be exclusively added to the training set. In the context of deep learning, evaluating the performance of a model solely on the training set is insufficient as it only reflects its ability to memorize the training data, thereby limiting its ability to generalize to unseen data. This necessitates the use of a validation set, which measures the model's accuracy on an independent dataset during training, allowing it to learn and adapt to new features in the data distribution. This process is crucial for ensuring that the model does not overfit the training data and improves its performance on unseen data.

[0059] In one embodiment, the present system 10 utilizes convolutional neural network (CNN) formulations. In the context of a conventional multi-layer neural network, consider w_n and b_n to be the weight and bias parameters learned through the training process, and T_n as the activation function of the network which produces the final output. The output of this model can be represented as y_n = T_n([w_n, b_n] * x„) , where x_n is the input to the n^tfl layer. The output of one layer in a multi-layer neural network is used as the input for the subsequent layer until a final class prediction is made. Weights and bias of the model will be constantly updated with the aim of minimizing the average training cost C_train over the training set {x₁,x₂ x_n], using gradient based learning methodologies. The selection of model architecture and loss function can be deemed critical in enhancing the generalization performance, thereby being reliant on the rapidity and accuracy of gradient descent optimization.

[0060] Given the multi-class nature of our model, the categorical cross entropy was selected as the primary loss function for estimating errors. In addition, we employed the use of false positive and false negative metrics as auxiliary loss functions. This is particularly relevant in the field of medical diagnosis, where the implications of FP and FN errors can be significant, and the ability to optimize the model with respect to these metrics can be of critical importance. The total cost function can be represented as

Where

[0061] The gradient refers to the vector of partial derivatives of the loss function with respect to each learnable parameter of a neural network. In the process of optimizing the network’s performance, an update to the learnable parameters is performed through a single step along the gradient direction, which is aimed at minimizing the loss function. This step can be mathematically represented as

Where y, a scalar constant, is the learning rate that controls the tradeoff between the speed and stability of weight convergence. It determines the degree to which the weight parameters are updated at each iteration of the training process.

[0062] In practice, the gradients of error functions are computed on batches of data, and stochastic gradient descent (SGD) is a popular algorithm that can be used for this purpose. However, for our purposes, we have adopted the Adam optimization algorithm, which is a more sophisticated version of gradient descent and is available in the Python programming language. Medical MRI datasets exhibit a high degree of variability in data distribution over time, and a deep learning model must be able to adapt to these constantly changing functions to achieve generalizability and avoid introducing noise into the data analysis process. Stochastic gradient descent is a suitable algorithm for problem statements where data distribution changes over time. [0063] The present system 10 can also utilize adaptive histogram equalization techniques for enhancement of contrast across MRI data. The standardization of MRI scans is critical for accurate analysis and diagnosis, as variations in pixel intensities can result from various factors such as differences in scanning parameters and anatomical variations. These variations can make it difficult to compare scans between patients and may lead to errors in interpretation. Histogram equalization can mitigate these variations by normalizing MRI scans and improving comparability among patients. This technique equalizes the histograms, reducing the variations in intensity and enabling a more standardized comparison of scans between patients. Research has demonstrated that histogram equalization can be effective in improving the accuracy and reliability of MRI analysis, particularly in clinical settings where consistent and accurate diagnosis is critical and is applied here.

[0064] Consider an jpeg MRI image x(p,k), where p denotes the position coordinates and k denotes the color channel. It should be noted that MRI images are typically grayscale, but the pixel intensity range may vary due to differences in machine software or other parameters. Therefore, we represented the possible intensities of the MRI images as the range [G₀ (G_L - 1)] , where G₀ is the minimum intensity value and G_L - 1 is the maximum intensity value. The probability density function (PDF) of an intensity level i can be expressed as

PDF x(i) = n _i /n where G₀ ≤ i ≤ (G_L — 1) >

[0065] Where n _i represents the total number of pixels with intensity value i, and n represents the total number of pixels in the MRI image.

[0066] The cumulative probability function (CDF) of the corresponding PDF can be defined as: (written in the context of this problem; it is a generic expression.) C

DF x(i)

[0067] The transformation function T maps the cumulative distribution function of the MRI scan to a uniform distribution as:

T = G₀ + ( (G_L - 1) - G₀,.,) x CDF x(i) y = {T x(p, k) | V x (p, k) ∈ G}

[0068] While histogram equalization can be effective in improving the contrast and visibility of an image it may not always lead to perceptually better results. Therefore, evaluating the quality of these images after histogram equalization is needed. In our study, we utilized the Structural Similarity Index (SSIM) to evaluate the structural similarity between the original MRI scans and the enhanced scans and represents the quantification of image structure, contrast, and luminance through the respective formulations mentioned below. Here, x(p,k) and y(i,j) denote the two scans being compared.

[0069] The structure component computes the correlation between the two scans and can be defined as: structure(x y) =

(these notations not defined)

[0070] The contrast component measures the disparity in contrast between the two scans and can be defined as:

[0071] The luminance component calculates the variation in brightness between the two MRI scans and can be defined as:

[0072] Where c_n = (K_nG)² included to avoid instability K is a constant parameter that is typically set to a value less than 1 , and G is the maximum possible pixel value in the scan.

[0073] Together, these three components allow for a comprehensive analysis of the structural similarity between two scans, using the equation:

where l(x,y), c(x,y), and s(x,y) represent the luminance, contrast, and structure components, respectively, and a, p, and y are weighting parameters that determine the relative importance of each component.

[0074] The equation can be rewritten as:

[0075] Here, px and py are the means of the two scans being compared, and

are the variances and covariance of the two scans.

[0076] Effective intensity equalization of medical images is hypothesized to result in a more uniform histogram of the equalized image, thereby increasing its similarity with a reference image that has a uniform histogram. This is expected to be reflected in a higher SSIM value calculated between the equalized and reference images, indicating a greater degree of structural similarity between the two images. Our study, as depicted in Fig. 3D, compared different histogram equalization techniques including conventional histogram equalization, adaptive histogram equalization, and their respective modifications. Our findings suggest that the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm outperformed other techniques in enhancing contrast and highlighting anatomical structures in brain scan images. It effectively minimizes the over-amplification of noise in homogeneous regions of an image, something other equalization algorithms failed to do. This highlights the effectiveness of CLAHE as an optimal intensity normalization algorithm for MRI scans enhancement. The CLAHE algorithm partitions the scan into non-overlapping tiles of a specified size, in this case 8x8 pixels, resulting in 900 tiles. Smaller tile sizes are preferred for finer detail. Histogram equalization is performed for each tile, with a contrast limit applied to prevent over-amplification of noise in homogeneous regions. This contrast limit can be formulated as (these above sentences are generic ones).: clip limit = minClipLimit + N (CLIPS') x (N(AVG) — minClipLimit)

[0077] Where N(CLIPS) is the total number of clips, N(AVG) represents the average number of pixels in the mapped image and minClipLimit represents the minimum allowable clip limit. This value is typically set to a small positive number, such as 0.01 , to ensure that some amount of contrast amplification occurs for all tiles. The clip limit formula is used to determine the maximum allowable contrast amplification for each tile, based on the average number of pixels in the mapped image and the specified minimum clip limit. This helps to ensure that the contrast amplification is optimal for each tile, without introducing excessive noise.

[0078] In one embodiment, the method involves the computation of histogram, intensity redistribution, and cumulative distribution function scaling for each tile of the image, followed by bilinear interpolation to stitch together the processed tiles, resulting in a final equalized image. The effectiveness of the method is evaluated through visual comparison between the original and enhanced images, along with their corresponding histogram plots, as depicted in Fig. 3A. The histogram equalization process is demonstrated to significantly improve image quality, as evidenced by the changes in image intensity distribution.

[0079] Overfitting is a well-known obstacle in deep learning when training models with limited data. Adversarial training has emerged as a promising strategy to address this challenge, by leveraging adversarial examples to improve the model’s robustness and generalization performance. This method introduces adversarial perturbations in the training data, which enables the model to handle small variations in input and perform effectively with new data. We use the variable h_e to represent the neural network model, x to represent the generated adversarial example and y to represent the true label. The goal is to generate adversarial examples that maximize the loss function to help improve the model's performance. Therefore, the optimization problem can be represented as a maximization of loss:

Fix) — argmax L(y,h_θx) [0080] Here, L represents the loss function, which computes the difference between the predicted output h(,x ) and the true label y.

[0081] Let the perturbation function be: δ_∞ = {δ:: ||δ| ≤ ∞0}

[0082] To ensure that x is close to our original input x, we optimize over the perturbation 8 to x, and then by optimizing over <5:

F (x) = argmax L(y, h_θx + δ) where δ < E

[0083] We are optimizing over the perturbed input x + 8, which is expected to be closer to the original input x than a completely adversarial example would be. By doing so, we are able to generate adversarial examples that are both effective in improving the model's performance and close to the original input x.

[0084] For, the Fast Gradient Sign method (FGSM) was utilized to create adversarial examples, where a gradient calculated using a loss function is added as perturbations to the input data. grad(x)' =V L(y, h_θx + δ)

[0085] FGSM generates adversarial examples by computing the gradient of the loss function with respect to a perturbation δ via backpropagation. By applying gradient descent to δ, we can obtain the direction of perturbation that maximizes the difference between the model's prediction and the desired output. This approach is based on the sensitivity to variations, and the intuition that small changes to the input data can cause significant changes to the output of the model. By leveraging the gradient information of the loss function, we can perturb the input data in a way that maximizes the objective.

F(x) = argmax L(y, h_θx + [δ + grad(x)]) where δ ≤ ε [0086] Once the adversarial examples have been generated, they can be incorporated into the training process by including them as input data. The model's parameters are then updated based on the loss calculated from both the original and adversarial examples. By adjusting the magnitude of perturbations, varying levels of robustness can be achieved. Although this technique improves the generalization ability of models in deep learning, a trade-off exists between robustness and accuracy, where an overly robust model may decrease accuracy on normal data. Therefore, it is crucial to balance this trade-off, and acknowledge that adversarial training may not be suitable for certain data types or models due to computational costs.

[0087] Biomedical imaging provides an invaluable resource as an integral part of patient care. MRI has a historical link to tumor pathologies but remains nonoptimized analytically for clinical applications. Technology advances that increase Tesla power and provide vastly expanded detail, will only exacerbate the demands for trained medical experts to evaluate MRI data and require additional training, adding to time demands on radiologists to review and turnaround diagnostics on images. Given the increasing demand on radiologist and neurosurgeon clinicians in patient care and a high level of misdiagnosis from MRI alone that require multiple additional patient evaluations, the implementation of Al deep learning analysis of clinical data for MRI has become imperative. Currently development of cutting-edge Al platforms have focused on detection of presence or absence of tumor pathologies, such as the deep learning SCENIC algorithm developed by this group and tumor segmentation for detecting tumor edges. Here we develop SIENNA, a novel Al deep learning platform for distinguishing brain pathologies, applied here to GBM versus MET neoplasms. Our analysis reveals the challenges associated with applying extensively preprocessed clinical data for later model generalization.

[0088] SIENNA is a powerful deep learning Al algorithm trained on a small, minimally processed clinical dataset which achieves high accuracy typing (88-91%) of GBM and MET tumor pathologies, that is expected to increase in robustness with expanded data analysis, typical of Al platforms. Preceding the development of SIENNA, an analysis of post-analyzed tumors upon resection revealed high misdiagnosis (85%) by MRI alone, reflectiing the variability of these tumors and challenges even for clinical experts. SIENNA works on minimally processed MRI dicom data to fit seamlessly into the clinical workflow. The development of SIENNA also points to the value of numerous tools available to optimize deep learning platforms, such as Synthetic Minority Over-Sampling Technique (SMOTE) to address class imbalance, Hyperas for optimizing hyper-parameters, histogram equalization for scan standardization, and adversarial training to ensure model robustness against different types of adversarial attacks. By capitalizing on the distinctive attributes of the convolutional neural network (CNN) architecture and its capacity to extract intricate hierarchical features from data, our clinical data optimized SIENNA model exhibits robust tissue pathology typing, demonstrated here by discrimination of GBM versus MET tumor pathologies. The versatility of SIENNA in handling both raw and processed clinical data renders it an indispensable resource for enhancing clinical treatment workflow speed and efficacy to promote superior patient outcomes.

[0089] SIENNA's ability to handle raw clinical data with minimal preprocessing, along with publicly available data, sets it apart from other state-of-the-art methods. Furthermore, SIENNA shows potential for future adaptation and application to a range of brain physiology and pathology tasks in clinical biomedical imaging. To improve the model's performance and robustness, analysis on larger clinical datasets, including MRI data from different manufacturers, will be necessary. Furthermore, the integration of Al-based pre-diagnostic predictions, classification of additional brain tumor types and stages, and comorbidity analysis can be incorporated to enhance the model's utility. SIENNA’s Al capabilities can continue to learn from complex and diverse clinical MRI datasets brings MRI robustness in biomedical neurosurgical treatment pre-planning. Moreover, the versatility of SIENNA's Al framework allows for its potential application in other brain pathologies and expanded use in artificial generalizable intelligence.

[0090] Also attached hereto as Appendix A is source code to enable one embodiment of the present invention that can be executed on computer device 14. [0091] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below, if any, are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of one or more aspects of the invention and the practical application, and to enable others of ordinary skill in the art to understand one or more aspects of the invention for various embodiments with various modifications as are suited to the particular use contemplated.

* * * * * README :

# SIENNA

## Abstract

Biomedical imaging diagnostics and clinical workflow are realizing significant benefits by integrating Artificial intelligence (Al) and machine learning (ML) to streamline clinical communication and processes, unify expertise, and expand the scope of retrievable information from data. For magnetic resonance imaging (MRI) of brain tumors of glioblastoma (GBM) or metastasized (MET) origins, overlapping and complex tumor features hinder multi-classification .

MRI tumor-type misdiagnosis by experts has a significant error rate, according to recent retrospective studies as high as 85%. Multiple diagnostic protocols currently compensate, delaying clinical workups and treatments . ML applied to minimally processed MRI clinical DICOM datasets is largely unexplored for tumor diagnostics, with most studies relying on highly processed public datasets . This hinders clinical integration due to overfitting and low generalizability of the existing ML approaches trained on those datasets and lowers the trustworthiness of such ML models for new patients and less processed data . To enable Al deployment in the clinical setting we developed SIENNA, a deep-learning Al diagnostics framework with state-of-the-art data processing and training of clinical MRI DICOM datasets for neuromorphic analysis . SIENNA cross-validation forces our model to reduce overfitting. Limit false positives and false negatives, and achieve commendable performance for new patients, constituting a real-world clinical scenario. SIENNA&#39; s demonstrates average accuracy on clinical DICOM MRI data across 3 tasks of 92% (Non-Tumor), 91% (GBM), and 93% (MET) , with the distribution of accuracies skewed higher and lower bound at 75%. SIENNA achieves high accuracy typing of brain tumor GBM and MET pathologies, capability to work with small clinical datasets, and is a computationally light and portable Al companion diagnostic to advance the use of MRI in patient care.

## Hardware Requirements

The training process was conducted on a Microsoft Windows 11 workstation equipped with an Intel(R) Core (TM) i7-10750H six-core CPU, 16 GB of system RAM, and a single NVIDIA GTX 1650 GPU boasting 4 GB of GPU RAM.

## Python Environment

Python libraries and modules required are listed in requirements .txt.

<pre>pip install -r requirements .txt ' </pre>

## SIENNA Architecture

### Source code for training and testing SIENNA:

1. SIENNA. py: Defines the SIENNA architecture class .

2. feature_extraction . py : Defines a convolutional neural network (CNN) model using Keras and TensorFlow. It includes hyperparameter tuning using Hyperas and Hyperopt .

3. adversarial training. py: Defines an AdversarialGenerator class for generating and training adversarial examples using TensorFlow and Keras . It includes functionality to generate perturbations, conduct adversarial training, and evaluate model accuracy on adversarial examples . The code also provides options for plotting perturbation maps and comparing original and adversarial images .

5. intensity_redistribution . py: Pre-processing of input images . PREMO is defined here.

6. performance_evalution . py : Evaluates cross-validation performance.

REQUIREMENTS: opencv-python>=4. 5 . 5 numpy>=1.23.4 hyperopt==0.2. 5 hyperas==0.4.1 np_utils==0.6.0 scikit-image>=0.22.0 mat plot lib >=3. 6.0 tensorflow>=2. 13.0 mat plot lib==3.7.1 opencv==4. 8.0 imblearn==0.11 .0

SIENNA: import os import cv2 import numpy as np import tensorflow as tf import feature_extraction from adversarial_training import AdversarialGenerator from intensity redistribution import adaptive MRI enhancement from performance_evaluation import InbuiltMetricsCalculator class SIENNA: def init (self, multi=False, binary=False) : if multi: self .mult i_c las sificat ion ( ) def multi_classification(self ) : trials, best_model, best_run,x_train, y_train, x_test, y_test,x_cv,y_cv=feature_extraction. run_hyperparameter_tunning( ) best_model . save( ' D: \\10-02-2023\\SIENNA\\model\\optimised_model. h5 ' ) loss_f unction = tf . keras . losses .Catego ricalCros sent ropy (from_logits= False) adv generator = AdversarialGenerator(model=best_model, x=x_train, y=y_train, loss_function=loss_function, eps=0. 1, stabilization_factor=le-8, plot_perturbation= False, plot_example= False, training_iterations=l) optimizer = tf.keras. optimizers. Adam() adv generator . adversarial training(optimizer) custom_metrics_calculator = InbuiltMetricsCalculator(best_model, x_train, np.argmax(y_train, axis=l), z) results_default_threshold = custom_metrics_calculator . evaluate(overall=True) print( "Results using Default Threshold (0.5):") for class_name, metrics in results_default_threshold.items() : print(f "Class : {class_name}" ) print(f "Accuracy : {metrics['Accuracy' ]}") print(f"Fl Score: {metrics['Fl score']}") print(f"AUROC: {metrics[' AUROC ' ]}" )

# Print the number of TP, TN, FP, FN, and their corresponding image names for class 0 class_index = 0 tp_image_names = custom_metrics_calculator .get_image_names(class_index, true_positive=True) tn image names = custom metrics calculator .get_image_names(class_index, true_negative=True) fp_image_names = custom_metrics_calculator .get_image_names(class_index, f alse_positive=T rue) fn_image_names = custom_metrics_calculator .get_image_names(class_index, f alse_negative=T rue) print( "\nClass 0 - True Positives (TP):") print(f "Number of TP: {len(tp_image_names[ 'True Positives'])}") print(tp_image_names["True Positives"] ) print( "\nClass 0 - True Negatives (TN):") print(f "Number of TN: {len(tn_image_names[ 'True Negatives'])}") print(tn_image_names["True Negatives"] ) print( "\nClass 0 - False Positives (FP):") print(f "Number of FP: {len(fp_image_names[ ' False Positives'])}") print (f p image names [ " False Positives" ] ) print( "\nClass 0 - False Negatives (FN):") print(f "Number of FN: {len(fn_image_names[ ' False Negatives'])}") print (f n_image_names [ " False Negatives" ] ) if > name == " main ": app = SIENNA(multi=True)

> import tkinter as tk from tkinter import filedialog from PIL import Image, ImageTk import tensorflow as tf import numpy as np import cv2 from intensity_redistribution import adaptive_MRI_enhancement

# Load your model model = tf . ker as .models . load_model( 'D: \\10-02-2023\\SIENNA\\model\\optimised_model. h5 ' ) def predict_image(image_path) : image = cv2. imread(image_path) img = cv2. resize(image, (240, 240) ) he = adaptive_MRI_enhancement(img) custom_clahe_image = he . PREMO( ) custom_clahe_image = custom_clahe_image / 255 sample = np. asarray(custom_clahe_image) sample = np. expand_dims (sample, axis=0) # Add a batch dimension prediction = model . predict(sample) predicted_class = np.argmax(prediction) return predicted class def predict_button_clicked( ) : file_path = filedialog . askopenfilename( ) if file_path: prediction = predict_image(file_path) print (predict ion) result_label . config(text=f "Prediction : {prediction}" ) root = tk. Tk( ) root .title( "Image Prediction" ) predict_button = tk. Button(root, text="Predict", command=predict_button_clicked) predict_button . pack( ) result_label = tk. Label(root, text="" ) result_label . pack( ) root .mainloop( )

PERFORMANCE_EVALUATION : import tensorflow as tf import matplotlib. pyplot as plt import numpy as np import os import cv2 from keras . models import loadjnodel from tensorflow. keras . utils import to_categorical from sklearn. metrics import confusion_matrix, roc_auc_score from sklearn. metrics import accuracy_score, fl_score from sklearn. metrics import roc_curve, auc import matplotlib print(f "TensorFlow version: {tf. version }") print(f "Matplotlib version: {matplotlib. version }") print(f "NumPy version: {np. version }") print(f "OpenCV version: {cv2. version }") import warnings warnings. filterwarnings( 'ignore' ) # Ignore all warnings import os os. environ [ 'TF_CPP_MIN_LOG_LEVEL' ] = '3' import tensorflow as tf tf . compat . vl. logging. set_ver bos ity(tf . compat . vl . logging. ERROR) import warnings from matplotlib import MatplotlibDeprecationWarning warnings .filterwarnings ( "ignore", category=MatplotlibDeprecationWarning) model = load_model("D: \10-02-2023\SIENNA\model\optimised_model . h5" ) data_dir = 'D:\10-02-2023\SIENNA\data' x = [] y = [] z = [] class mapping = {'N': 0, 'G': 1, 'M' : 2} for class_folder in os . listdir (data_dir) : if class_folder in class_mapping: class_path = os . path. join(data_dir, class_folder) class_label = class_mapping[class_folder] for image_filename in os . listdir (class_path) : if image_filename. lower( ) . endswith( (" . png" , ".jpg")): image_path = os . path . join(class_path, image_filename) image = cv2.imread(image_path) image = cv2. resize(image, (240, 240)) image = image / 255.0 x.append(image) y .append (class_label) z .append (image filename) x = np.array(x) y = np.array(y) z = np.array(z) y = to_categorical(y, num_classes=3) data_dir = '/content/drive/MyDrive/adv' x = [] y = [] z = [] class_mapping = {'N': 0, 'G': 1, 'M' : 1} for class_folder in os . listdir (data_dir) : if class_folder in class_mapping: class_path = os. path. join(data_dir, class_folder) class_label = class_mapping[class_folder] for image_filename in os . listdir(class_path) : if image filename. lower( ) . endswith( (" . png" , ".jpg")): image_path = os . path . join(class_path, image_filename) image = cv2.imread(image_path) image = cv2. resize(image, (240, 240)) image = image / 255.0 x.append(image) y .append (class_label) z .append (image_filename) x = np.array(x) y = np.array(y) z = np.array(z) y = to_categorical(y, num_classes=2) class InbuiltMetricsCalculator : def > init (self, model, x_data, y_data, image_names) : self. model = model self.x_data = x_data self.y_data = y_data self . image_names = image_names self . num_classes = np. max(y_data) + 1 self.y_pred = np.argmax(self .model. predict(self.x_data), axis=l) self. cm = confusion_matrix(self .y_data, self.y_pred) def calculate_metrics(self , class_index, threshold=0.5) : tp = self . cm[class_index, class_index] fp = np. sum(self . cm[ : , class_index] ) - tp fn = np.sum(self . cm[class_index, :]) - tp tn = np.sum(self .cm) - tp - fp - fn accuracy = (tp + tn) / np. sum(self . cm) precision = tp / (tp + fp) recall = tp / (tp + fn) flscore = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0 y_scores = self .model. predict(self . x_data) [: , class_index] y_pred_threshold = (y_scores >= threshold) .astype(int) y_true_binary = (self.y_data == class_index) . astype(int) auroc = roc_auc_score(y_true_binary, y_pred_threshold) result = {

"True positives": tp,

"False positives": fp,

"False negatives": fn,

"True negatives": tn, "Accuracy": accuracy, "Precision": precision, "Recall": recall, "Fl score": flscore, "AUROC": auroc,

} return result def evaluate(self , threshold=0.5, overall=False) : results = {} for i in range(self . num_classes) : class_results = self . calculate_metrics (i, threshold) results[f"Class {i}"] = class_results if overall: if self . num_classes == 2: y scores = self.model . predict(self. x data) [: , 1] y_true_binary = (self.y_data == 1) .astype(int) auroc = roc_auc_score(y_true_binary, y_scores) accuracy = accuracy_score(y_true_binary, (y_scores >= threshold) . astype(int) ) flscore = fl_score(y_true_binary, (y_scores >= threshold) . astype(int) ) results [ "Overall Metrics"] = {

"AUROC": auroc,

"Accuracy": accuracy, "Fl score" : flscore } else: overall accuracy = np.mean( [metrics[ "Accuracy" ] for metrics in results .values () ] ) overall fl score = np.mean( [metrics["Fl score"] for metrics in results .values () ] ) y_true_binary = np.eye(self . num_classes) [self .y_data] y_scores = self.model. predict(self.x_data) fpr, tpr, > = roc_curve(y_true_binary . ravel( ), y_scores . ravel( ) ) overall_auroc = auc(fpr, tpr) results [ "Overall Metrics"] = {

"Accuracy": overall_accuracy, "Fl score": overall_fl_score, "AUROC": overall_auroc

} return results def get_image_names(self , class_index, true_positive=False_J true_negative=False, false_positive=False false_negative=False) : y_true_binary = (self.y_data == class_index) . astype(int) y_pred_threshold = (self .model . predict (self .x_data) [: , class_index] >= self .threshold) . astype(int) tp_indices = np.where( (y_true_binary == 1) & (y_pred_threshold == 1))[0] tn_indices = np.where( (y_true_binary == 0) & (y_pred_threshold == 0))[0] fp_indices = np . where( (y_true_binary == 0) & (y_pred_threshold == 1))[0] fn_indices = np . where( (y_true_binary == 1) & (y_pred_threshold == 0))[0] tp_image_names = [self . image_names [i] for i in tp_indices] tn_image_names = [self . image_names [i] for i in tn_indices] fp_image_names = [self . image_names [i] for i in fp_indices] fn_image_names = [self . image_names [i] for i in fn_indices] return {

"True Positives": tp_image_names if true_positive else [], "True Negatives": tn_image_names if true_negative else [], "False Positives": fp_image_names if false_positive else [], "False Negatives": fn_image_names if false_negative else [], def draw_plots (self , class_indices=None auroc=False, accuracy=False, fl_score=False, recall=False, overall_auc=False) : if class_indices is None: class_indices = range(self . num_classes) if ovenall_auc: if self . num classes == 2: y_true_binary = self.y_data y scones = self .model . pnedict(self . x data) [: , 1] fpr, tpr, > = roc_curve(y_true_binaryj y_scores) noc_auc = auc(fpr\ tpn) plt.figure(figsize=(10j 6)) plt. plot(fpr, tprj, lw=2j label=f 'Overall AUROC = {roc_auc: .2f}' ) else: y_true_binary = np.eye(self . num_classes) [self .y_data] y_scores = self .model. predict(self.x_data) fpr, tpr, > = roc_curve(y_true_binary. ravel()j y_scores . ravel( ) ) roc_auc = auc(fpr.> tpr) plt.figure(figsize=(10j 6)) plt. plot(fpr, tpr, lw=2j label=f 'Overall AUROC = {roc_auc: .2f}' ) if not overall_auc or len(class_indices) > 0: for i in class_indices : y_true_binary = (self .y_data == i) .astype(int) y scores = self .model . predict(self . x data) [: , i] fpr, tpr, > = roc_curve(y_true_binary, y_scores) roc auc = auc(fpr, tpr) plt.figure(figsize=(10, 6)) plt. plot(fpr, tpr, lw=2, label=f' Class {i} (AUROC = {roc_auc : .2f}) ' ) plt.plot([0, 1], [0, 1], color= ' navy ' , lw=2, linestyle= ' - - ' ) plt.xlim( [0.0, 1.0]) plt.ylim( [0.0, 1.05]) plt. xlabel( ' False Positive Rate') plt. ylabel( 'True Positive Rate') plt .title( ' Receiver Operating Characteristic (ROC) Curve') plt . legend(loc= ' lower right') plt . show( ) if accuracy: if self . num classes == 2: class_names = ["Binary"] accuracies = [self .calculate_metrics(l)["Accuracy"]] else: class_names = [f'Class {i}' for i in class_indices] accuracies = [self .calculate_metrics(i)["Accuracy"] for i in class_indices] plt.figure(figsize=(10, 6)) plt . bar(class_names, accuracies, color= ' skyblue ' ) plt.xlabel( 'Class' ) plt.ylabel( 'Accuracy' ) plt .title( 'Accuracy by Class') plt . xticks ( rotation=45 ) plt . show( ) if fl score: if self . num_classes == 2: class names = ["Binary"] fl_scores = [self . calculate_metrics(l) [ "Fl score"]] else: class_names = [f'Class {i}' for i in class_indices] fl_scores = [self . calculate_metrics(i) [ "Fl score"] for i in class_indices] plt.figure(figsize=(10, 6)) plt . bar(class_names, fl_scores, color= ' lightgreen ' ) plt.xlabel( 'Class' ) plt .ylabel( ' Fl Score') plt.title( ' Fl Score by Class') plt . xticks ( rotation=45 ) plt.show() if recall : if self . num_classes == 2: class names = ["Binary" ] recalls = [self . calculate_metrics(l) [ "Recall" ] ] else: class_names = [f ' Class {i} ' for i in class_indices] recalls = [self . calculate_metrics(i) [ "Recall" ] for i in class_indices] plt .figure(figsize=(10, 6)) plt . bar(class_names, recalls, color= ' lightcoral ' ) plt.xlabel( ' Class ' ) plt.ylabel( ' Recall ' ) plt . title( ' Recall by Class ' ) plt . xticks ( rotation=45 ) plt. show( )

INTENSITY_REDISTRIBUTION : import cv2 import numpy as np from skimage import exposure import matplotlib. pyplot as plt from skimage. metrics import structural_similarity as ssim n n n

This code implements various image enhancement techniques for MRI images . It includes :

* PREMO (Pixel Redistribution Enhancement, Masking, Optimization) for MRI intensity equalization

* Traditional Histogram Equalization

* Adaptive Histogram Equalization

* Contrast- Limited Adaptive Histogram Equalization (CLAHE)

* Contrast Stretching

The code also provides functions to calculate structural similarity (SSIM) and mutual information (MI) between images . class adaptive_MRI_enhancement : def > init (self, image, comparitive_study=False, plot_hist=False) :

Initialize the intensity equalizer.

Args : image: scan to be equalized . comparitive_study: Whether to plot comparitive study. plot_hist: plot histograms of equalized scans. self . image=image self . comparitive_study=companitive_study self . plot_hist=plot_hist def PREMO(self, clip_limit=l.5,gamma=3) :

Pixel Redistribution Enhancement, Masking, Optimization(PREMO) for MRI intensity equalization

Args: image: numpy array clip_limit: pixel threshold to clip gamma: gamma correction parameter II II II gray = cv2. cvtColor (self . image, cv2.COLOR_BGR2GRAY) h, w = gray. shape

#otsu's thresholding thresh_val, thresh_img = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + CV2.THRESH OTSU) kernel = np.ones((5, 5), np.uint8)

#noise removal using morhological operations mask = cv2.morphologyEx(thresh_img, cv2.MORPH_OPEN, kernel, iterations=3) #cumulative distribution function (CDF) hist, > = np.histogram(gray.flatten(), bins=256, range=(0, 255)) cdf = hist . cumsum( ) cdf = (cdf - cdf.minO) / (cdf.max() - cdf.min()) * 255

#mapping intensities to CDF mapping = np.interp(np.arange(256), np.arange(256), cdf) out = np.interp(gray.flatten(), np.arange(256), mapping) . reshape (gray . shape) out = np.clip(out, 0, 255) if clip_limit > 0: hist, > = np.histogram(out.flatten(), bins=256, range=(0, 255)) excess = np.sum(hist) - clip limit * gray. size if excess > 0: limit val = np.argmax(hist.cumsum() > excess) out = np.clip(out, 0, limit_val)

# Gamma Correction out = (out - np.min(out)) / (np.max(out) - np.min(out)) * 255 out = np.power(out / 255.0, gamma) * 255.0

#binary mask out = np. where(mask == 0, gray, out)

#intensity range 0-255 out = np.clip(out, 0, 255) out = cv2.cvtColor(out.astype(np.uint8), cv2.COLOR_GRAY2BGR) if self . plot_hist: self . plot_histograms(out) return out def ssim(self , imgl, img2): II II II

SSIM index to quanitfy similarity between two images II II II imgl gray, img2_gray = np.mean(imgl, axis=2), np.mean(img2, axis=2) meanl,mean2 = np.mean(imgl_gray) , np.mean(img2_gray) sigmal, sigma2 = np. std(imgl_gray), np. std(img2_gray) covariance = np . cov(imgl_gray . ravel() , img2_gray.ravel()) ssim = (2 * meanl * mean2 + 0.01) * (2 * covariance[0, 1] + 0.03) / ((meanl

** 2 + mean2 ** 2 + 0.01) * (sigmal ** 2 + sigma2 ** 2 + 0.03)) return ssim def plot_histograms(self ,out) : II II II

Function to plot histograms of scans before and after equalisation II II II fig, ax = plt . subplots (nrows=2, ncols=2, figsize=(18, 8)) ssim = self . ssim(self. image, out) ax[0, 0] . imshow(cv2. cvtColor(self . image, cv2.COLOR_BGR2RGB)) ax[0, 0] . set_title( 'Original ' ) ax[0, 0] .grid(False) ax[0, 1] .imshow(cv2.cvtColor(out, cv2. COLOR_BGR2RGB) ) ax[0, 1] . set_title(f Equalised Scan (SSIM={ssim })') ax[0, 1] .grid(False) hist_original, bins_original, > = ax[l, 0] .hist(self. image. ravel(), bins=256, range=[6, 255], color= ' #BB0707 ' ) ax[l, 0] . set_ylabel( ' Frequency ' ) ax[l, 0] . set_xlabel( ' Pixel Values') ax[l, 0].text(6, -max(hist_original) * 0.12, 'Dark Pixels', color= ' black ' , ha= ' center ' ) ax[l, 0].text(255, -max(hist_original) * 0.12, 'Bright Pixels', color= ' black' , ha='center') hist_clahe, bins_clahe, > = ax[l, 1] . hist(out . ravel( ), bins=256, range=[6, 255], color= ' #244FA2 ' ) ax[l, 1] . set_ylabel( ' Frequency ' ) ax[l, 1] . set_xlabel( ' Pixel Values') ax[l, l].text(6, -max(hist clahe) * 0.12, 'Dark Pixels', color=' black' , ha= ' center ' ) ax[l, l].text(255, -max(hist_clahe) * 0.12, 'Bright Pixels', color= ' black' , ha= ' center ' ) plt . reParams [' axes .facecolor ' ] = '#D0E2EF' plt . reParams [' axes .grid ' ] = True plt . reParams [' grid . linestyle ' ] = ':' plt. rcParamsf 'grid. color' ] = 'white' plt . tight_layout ( ) plt . show( ) def mutual_information(self , imgl, img2): joint_histogram = np.zeros((256, 256)) for i in range(imgl . shape[0] ) : for j in range(imgl . shape[l] ) : joint_histogram[imgl[i, j], img2[i,j]] += 1 joint histogram /= imgl.size joint_prob = joint_histogram / np. sum( joint_histogram) probl, prob2= np.sum( joint prob, axis=l), np. sum( joint prob, axis=0) mutual_info = 0 for i in range(256): for j in range(256): if joint_prob[i, j] > 0: mutual_info += joint_prob[i, j] * np. log( joint_prob[i, j] / (problfi] * prob2[j])) return mutual_info def entropy(self , img) : histogram = np. zeros(256) for i in range(img . shape[0] ) : for j in range ( img . shape [1] ) : histogram[img[i, j] ] += 1 histogram /= img. size prob = histogram / np. sum(histogram) entropy = 0 for i in range(256): if probfi] > 0: entropy -= probfi] * np . log(prob[i] ) return entropy def ambe(self, imgl, img2) : mi = self .mutual_information(imgl, img2) entl, ent2= self .entropy(imgl), self .entropy(img2) ambe = 2 * mi / (entl + ent2) return ambe def rgb_to_lab(self , img) : lab = cv2. cvtColor (img, cv2.COLOR_BGR2LAB)

L, A, B = cv2. split(lab) L = L / 255 return L, A, B def traditional_HE(self) :

1, a, b = self . rgb_to_lab(self. image) hist,_= np. histogram(l .flatten( ), bins=256, range=(0, 255)) cdf = hist . cumsum( ) cdf_m = (cdf - cdf .min())*255/(cdf .max()-cdf .min()) img2 = cdf_m[l] hist,_ = np.histogram(img2.flatten(),bins=255,range=(0,255)) cdf_2 = hist.cumsum() updated_img = cv2. cvtColor(cv2.merge((img2, a, b)), cv2.COLOR_LAB2BGR) return updated_img def adaptive_HE(self ) :

1, a, b = self . rgb_to_lab(self . image) img adapteq = exposure . equalize_adapthist(l, clip_limit=0. 1) img_adapteq = (img_adapteq *255) .astype( ' uint8 ' ) updated_lab_img2 = cv2.merge( (img adapteq, a, b) ) img_adapteq = cv2. cvtColor (updated_lab_img2, cv2.COLOR_LAB2BGR) return img_adapteq def contrast_HE(self ) :

1, a, b = self . rgb_to_lab(self . image) p2, p92 = np. percentile(l, (2, 92) ) img_rescale = exposure . rescale_intensity(l, in_range=(p2, p92)) updated_lab_img2 = cv2.merge( (img_rescale, a, b) ) contrast_stretch = cv2. cvtColor(updated_lab_img2, cv2.COLOR_LAB2BGR) return contrast_stretch def clahe_HE (self ) : clahe = cv2. createCLAHE (clipLimit=l, tileGridSize=(8,8) ) clahe_imgl = clahe. apply(self . image) return clahe imgl

FEATURE_EXTRACTION : from hyperopt import Trials, STATUS_OK, tpe from hyperas import optim from hyperas . distributions import choice, uniform from keras . callbacks import EarlyStopping, Modelcheckpoint import tensorflow as tf from tensorflow. keras import datasets, layers, models from tensorflow. keras . layers import Dropout from keras import backend as K import cv2 import numpy as np import os from imblearn . over_sampling import SMOTE from intensity redistribution import adaptive MRI enhancement from keras . utils import to_categorical from sklearn.model_selection import train_test_split def create_model(x_train, y_train, x_test, y_test) : II II II

Training and tunning SIENNA' s convolutional neural network (CNN)

Args : x_train: The training data input features (images) y_train: The training data labels (corresponding classes for each image) x_test : The testing data input features (images) y_test : The testing data labels (corresponding classes for each image) gpus = tf. conf ig. experimental . list_physical_devices( 'GPU ' ) #GPU configuration if gpus: tf . config . experimental . set_memory_growth(gpus [0] , T rue)

#dynamic model initiation model = models .Sequential )

#first conv block, with pre-defined parameter search spaces to be tuned by HYPERAS model. add(layers .Conv2D(filters={{choice( [10, 12, 16, 24, 33, 39, 48, 64,74, 87, 96,110, 115,120, 128, 200])}}, kernel_size={{choice( [(2,2), (3, 3), (4,4),

(5,5), (6,6), (7,7), (8,8)])}} , st rides= (4, 4) , padding= ' same ' , kernel_regularizer =tf . keras . regularizers .12 ( l={{choice( [ (0.01), (0.001), (0.0001), (0.00001), (0.005), (0.0005), (0.5)])}}), input_shape=(240, 240, 3), name= ' cvl ' )) model. add (layers .Activation ({{choice( [ ' relu ' , ' selu ' , ' elu ' , 'tanh ' , 'exponential' , ' LeakyReLU' ] )}}, name= ' activl ' )) model. add (layers .MaxPooling2D( (2, 2), padding= ' same' , name= ' mpooll ' )) model. add (Dropout ( {{choice ( [0.1,0.2, 0.3, 0.4,0.5] )}},name=' drpl' ))

#second conv block, with pre-defined parameter spaces to be tuned by HYPERAS model. add(layers .Conv2D(filters={{choice( [10, 12, 16, 24, 33, 39, 48, 64,74, 87, 96,110, 115,120, 128, 200])}}, kernel_size={{choice( [(2,2), (3, 3), (4,4),

(5,5), (6,6), (7,7), (8,8)])}} , strides=(4,4) , padding= ' same ' , kernel_regularizer =tf .keras . regularizers .12 ( l={{choice( [ (0.01), (0.001), (0.0001), (0.00001), (0.005), (0.0005), (0.5)])}}), input_shape=(240, 240, 3), name= ' cv2 ' )) model. add (layers .Activation ({{choice( [ ' relu ' , ' selu ' , ' elu ' , 'tanh ' , 'exponential' , ' LeakyReLU' ] )}}, name= ' activ2 ' )) model. add (layers .MaxPooling2D( (2, 2), padding= ' same' , name= ' mpool2 ' )) model. add (Dropout ({{choice ([0.1, 0.2, 0.3, 0.4,0.5] )}},name='drp2' ))

#third conv block, with pre-defined parameter spaces to be tuned by HYPERAS model. add(layers .Conv2D({{choice( [10, 12, 16, 24, 33, 39, 48, 64,74, 87, 96,110, 115,120, 128, 200])}}, kernel_size={{choice( [ (2, 2) , (3, 3), (4,4),

(5,5), (6,6), (7,7), (8,8)])}} , st rides= (4, 4) , padding= ' same ' , kernel regularizer =tf .keras . regularizers .12 ( l={{choice( [ (0.01), (0.001), (0.0001), (0.00001), (0.005), (0.0005), (0.5)])}}), input_shape=(240, 240, 3), name= ' cv3 ' )) model. add (layers .Activation ({{choice( [ ' relu ' , ' selu ' , ' elu ' , 'tanh ' , 'exponential' , ' LeakyReLU' ] )}}, name= ' activ3 ' )) model. add(layers .MaxPooling2D( (2, 2), padding= ' same' ), name= 'mpoolS ' ) model. add (Dropout ({{choice ([0.1, 0.2, 0.3, 0.4,0.5] )}},name='drp2' )) model . add ( layers . Flatt en( ) ) model. add (layers .Dense (512, {{ choice ( [ ' relu ' , ' sigmoid ' ] )}}, name= ' fc ' ) ) model. add (Dropout ({{choice ([0.1, 0.2, 0.3, 0.4, 0.5])}})) model. add (layers .Dense (3, name= ' 2f c ' ) ) model . add ( layers .Activation ( " softmax" ) ) model. compile(loss= ' catego rical_c ros sent ropy ' , optimizer= 'Adam ' , metrics=[ ' acc ' ,tf. keras .metrics . FalseNegatives( ), tf . keras .metrics . FalsePositives( ), tf . ker as .metrics . Recall ( )] )

#callback for early stopping callbacks = [EarlyStopping(monitor= ' val loss ' , patience=2) , Modelcheckpoint (filepath=f ' model/optimised_model. h5 ' , monitor= ' val_loss ' , save_best_only=T rue) ]

#model training history = model. fit(x_train,y_train, batch_size={{choice( [16, 32, 64] )}}, epochs={{choice( [15, 25] )}}, verbose=2, validation_data=(x_test, y_test), callbacks=callbacks) validation_acc = max(history. history [ ' val_acc ' ] ) score, acc, fn,fp, recall = model .evaluate(x_test, y_test) return { ' loss ' : score+fp+fn, ' status ' : STATUS_OK, ' model ' : model} def data() :

#data path tumor_train_flair = 'data/AMC Named Files/tumor/ ' nontumor_train_flair = 'data/AMC Named Files/no tumor/ ' x,y=[L [] clip_limit = 0.5 grid_size = (4, 4) for i in os . listdir(tumor_train_flair) :

#label according to tumor type. GBM labelled as 1, MET labelled as 2 if 'GBM' in i : labeltumour=l else: if 'MET' in i: labeltumour=2 for j in os . listdir(tumor_train_flair+i) : if(str(j)== 'Ax- Flair' or str( j )== ' Series-AxFLAIR ' ) : for k in os . listdir(tumor_train_flair+i+ ' / ' +j) : if k != ' Zaxis ' : image = tumor_train_flair+i+ ' / ' +j+ ' / ' +k img=cv2. imread(image) img=cv2. resize(img, (240,240) )

#apply premo he = adaptive_MRI_enhancement(img) custom_clahe_image =he . PREMO( ) custom_clahe_image=custom_clahe_image/255 x. append(custom_clahe_image) y. append (labeltumour) for i in os . listdir(nontumor_train_flair) : for j in os . listdir(nontumor_train_flair+i) : if (str(j)== 'Ax- Flair' or str(j )==' Series-AxFLAIR ' ) : for k in os . listdir(nontumor_train_flair+i+ ' / ' +j) : if k ! = ' Zaxis ' : image = nontumor_train_flair+i+' / ' +j+' / '+k img=cv2. imread(image) img=cv2. resize(img, (240,240)) #apply premo he = adaptive MRI enhancement(img) custom_clahe_image =he.PREMO() custom_clahe_image=custom_clahe_image/255 x. append(custom_clahe_image) y . append(0) y_train = to_categorical(y, 3) from sklearn. utils import shuffle x_shuffle, y_shuffle = shuffle(x, y_train)

#training-vaalidation-test split as 60-20-20 xa, x_test, ya, y_test = train_test_split(x_shuffle,y_shuff le, test_size=0.2,train_size=0.8) x_train, x_cv, y_train, y_cv = train_test_split(xa,ya,test_size =

0.25,train_size =0.75) x_train = np. asarray(x_train) y_train = np. asarray(y_train) x_test = np.asarray(x_test) y test = np.asarray(y test) x_cv = np. array(x_cv) y_cv = np. array(y_cv)

#SMOTE data augementation to balance minority classes MET and Non-Tumor sm = SMOTE () train_rows=len(x_train) x_train = x_train . reshape(train_rows, -1) x_train, y_train = sm. fit_resample(x_train, y_train) x_train = x_train . reshape( -1, 240, 240, 3) return x_t rain, y_t rain, x_test,y_test,x_cv,y_cv def round_int(x) : if x in [float("-inf"),float("inf")] : return float("nan") return int(round(x)) def recall(y_true, y pred) : true_positives = K. sum(K. round(K. clip(y_true * y_pred, 0, 1))) possible positives = K . sum(K. round(K. clip(y_true, 0, 1))) recall = true_positives / (possible_positives + K.epsilon()) return recall def fn(y_true, y_pred): return K.sum(K.cast(y_true*(l-y_pred), 'float'), axis=0) def fp(y_true, y_pred): return K.sum(K.cast((l-y_true)*y_pred, 'float'), axis=0) def run_hyperparameter_tunning() : trials = Trials() x_train, y_train, x_test, y_test,x_cv,y_cv= data() #running parameter optiimization best_run, best_model = optim.minimize(model=create_model, data=data, f u net ions= [recall] , algo=tpe. suggest , max_evals=30, trials=trials) return trials, bestjnodel, best_run,x_train, y_train, x_test, y_test,x_cv,y_cv

ADVERSARIAL_TRAINING : import tensorflow as tf import matplotlib. pyplot as plt import numpy as np import os import cv2 from keras. models import load_model from tensorflow. keras . utils import to_categorical import warnings warnings. filterwarnings( 'ignore' ) # Ignore all warnings import os os. environ [ 'TF_CPP_MIN_LOG_LEVEL' ] = '3' import tensorflow as tf tf . compat . vl. logging. set_ver bos ity(tf . compat . vl . logging. ERROR) import warnings from matplotlib import MatplotlibDeprecationWarning warnings .filterwarnings ( "ignore", category=MatplotlibDeprecationWarning) class AdversarialGenerator : def > init (self, model, x, y, loss_f unction, eps=0.3, stabilization_factor=le-8, plot_perturbation=False, plot_example=False) :

Initialize the AdversarialGenerator .

Args: model: SIENNA model. x: original images for which adversarial examples will be generated, y: True labels corresponding to the original images. loss_function: Loss function to optimize during adversarial example generation . eps: Perturbation magnitude. stabilization_factor : Perturbation Normalization constant. plot_perturbation: Whether to plot the generated perturbation maps. plot_example: Whether to plot original and adversarial examples. self. model = model self.x = x self.y = y self . loss_function = loss_function self.eps = eps self . stabilization factor = stabilization factor self . plot_perturbation = plot_perturbation self . plot example = plot example def generate(self ) : x_adv = tf . identity(self .x) # Create a copy of images. y_adv = tf . identity(self .y) # Create a copy of labels. with tf .GradientTape() as tape: tape. watch (x_adv) y_pred = self . model(x_adv, training=False) # Make predictions on the present model. loss = self . loss_f unction(y_adv, y_pred) # Calculate loss. gradients = tape.gradient(loss, x_adv) # Gradient Computation. gradients /= (tf .math. reduce_std(gradients) + self . stabilization_factor) # Gradient Normalization. x_adv += self.eps * gradients # Apply perturbations to x_adv. if self . plot_perturbation : self ._plot_pertur bat ion (gradients) if self . plot_example: self ._plot_example( self .x, x_adv) return x_adv, y_adv def _plot_perturbation(self , gradients):

# Plot the perturbation maps for adversarial examples num_images = gradients . shape[0] plt .figure(figsize=(num_images * 5, 5)) for i in range(num_images) : noise = np. clip(gradients [i] , -1, 1) plt . subplot(l, num_images, i + 1) plt. imshow(noise, cmap='gray' if noise. ndim == 2 else None) plt. axis ( 'off' ) plt .tight_layout() plt . show( ) def _p 1 ot_exam p 1 e ( s e If j original, adversarial) :

# Plot original and adversarial examples. num_images = original. shape[0] plt .figure(figsize=(num_images * 10; 10)) for i in range(num_images) : original image = np.clip(original[i], 0, 1) plt. subplot(2j num_imageSj i + 1) plt. imshow(original_image) plt.axis( 'off' ) adversarial_image = np . clip(adversarial[i], 0_} 1) plt. subplot(2j num_images, num_images + i + 1) plt. imshow(adversarial_image) plt.axis( 'off' ) plt . tight_layout ( ) plt . show( ) class AdversarialGenerator : def > init (self; model; x, y, loss function; eps=0.3; stabilization_factor=le-8; plot_penturbation= False; plot_example= False; training_iterations=l) : II II II Initialize the AdversarialGenerator .

Args: model: SIENNA model. x: original images for which adversarial examples will be generated, y: True labels corresponding to the original images. loss_function: Loss function to optimized. eps: Perturbation magnitude. stabilization_factor : Small constant for gradient normalization. plot_perturbation: Whether to plot the generated perturbation maps, plot example: Whether to plot original and adversarial examples. training_iterations : Number of adversarial training iterations. II II II self. model = model self.x = x self.y = y self . loss_function = loss_function self.eps = eps self . stabilization_factor = stabilization_factor self . plot_perturbation = plot_perturbation self . plot_example = plot_example self .training_iterations = training_iterations def adversarial_training(self ; optimizer): for iteration in range(self .training_iterations) : print(f"Training iteration: {iteration+1}" ) x adVj y adv = self .generate() # Generate examples of current iter. with tf .GradientTape( ) as tape: predictions = self .model(x_adv, training=True) # Predict prediction on current model. loss = self . loss_f unction(y_adVj predictions) # Compute Loss. grads = tape.gradient(losSj self .model .trainable_variables) # Gradient Computation optimizer . apply_gradients( zip (grads , self .model .trainable_variables)) if self . plot_perturbation : self ._plot_pertur bat ion (self .gradients) if self . plot_example: self ._plot_example( self .x, x_adv) def generate(self ) : x_adv = tf . identity(self .x) # Create copy of x. y_adv = tf . identity(self .y) # Create copy of y. with tf .GradientTape() as tape: tape. watch (x_adv) y_pred = self . model(x_adv, training=False) # Model prediction. loss = self . loss_f unction(y_adVj y_pred) # Loss Computation. self .gradients = tape. gradient(loss, x_adv) # Gradient Computation self .gradients /= (tf .math. reduce_std(self. gradients) + self . stabilization_factor) # Gradient Normalization. x_adv += self.eps * self .gradients # Apply perturbations to x_adv. return x_adv, y adv def evaluate_adversarial_examples(self, x_adv, y adv): predictions = self .model. predict(x_adv) accuracy = np.mean(np.argmax(predictions, axis=l) == np.argmax(y_adv, axis=l)) print(f 'Accuracy on adversarial examples: {accuracy: .2%}' ) def _plot_perturbation(self , gradients):

# Plot the perturbation maps for adversarial examples num_images = gradients . shape[0] plt .figure(figsize=(num_images * 5, 5)) for i in range(num_images) : noise = np.clip(gradients[i]j -1, 1) plt. subplot(l, num_images, i + 1) plt.title(f' Noise {i+1}') plt. imshow(noise, cmap='gray' if noise. ndim == 2 else None) plt.axis( 'off' ) plt . tight_layout ( ) plt . show( ) def _plot_example(self , original, adversarial) :

# Plot original and adversarial examples. num_images = original. shape[0] plt .figure(figsize=(num_images * 10, 10)) for i in range(num_images) : original_image = np.clip(original[i], 0, 1) plt . subplot(2, num_images, i + 1) plt. title(f 'Original {i+1}') plt. imshow(original_image) plt.axis( 'off' ) adversarial_image = np . clip(adversarial[i] , 0, 1) plt. subplot(2, num_images, num_images + i + 1) plt.title(f' Adversarial {i+1}') plt. imshow(adversarial_image) plt.axis( 'off' ) plt . tight_layout ( ) plt . show( )

Claims

CLAIMS What is claimed is:

1. A system for determining the presence of cancerous tumors within diagnostic images of a human body, comprising: a scanning device generating one or more diagnostic images of a human body, the scanning device selectively transmitting the one or more diagnostic images to other computer devices; and a computer device configured to perform the steps of: receiving the one or more diagnostic images from the scanning device; creating a plurality of tiles for each received image; computing a histogram for each tile; redistributing an intensity for each tile; creating a cumulative distribution function scaling for each tile; conducting a bilinear interpolation to thereby stitch together processed tiles; combining the processed tiles into in a final equalized image; and analyzing the final equalized image to determine a presence of cancerous tumors within the diagnostic images.

2. The system of claim 1 , wherein the analyzing the final equalized image is analyzing the final equalized image with a convolutional neural network.

3. The system of claim 1 , wherein the computer device further configured to perform the step of training a predictive model based upon the analyzing of the final equalized images.

4. The system of claim 3, wherein the computer device further configured to perform the steps of: producing a training data based upon the received one or more diagnostic images; and introducing adversarial perturbations in the training data to train the predictive model.

5. The system of claim 1 , wherein: wherein the one or more diagnostic images are MRI images; and computing a histogram is computing a histogram equalization.

6. The system of claim 5, wherein the diagnostic images are one or more images of a human brain.

7. The system of claim 6, analyzing the final equalized image to determine a presence of cancerous tumors within the diagnostic images is determining a presence of glioblastomas.

8. The system of claim 4, wherein the training of the predictive model includes tuning of hyper-parameters from the group of: weight penalty factor, learning rate, and momentum.

9. A computer-implemented method for determining the presence of cancerous tumors within diagnostic images of a human body, comprising: generating one or more diagnostic images of a human body at a scanning device; selectively transmitting the one or more diagnostic images from the scanning device to other computer devices; receiving the one or more diagnostic images from the scanning device; creating a plurality of tiles for each received image; computing a histogram for each tile; redistributing an intensity for each tile; creating a cumulative distribution function scaling for each tile; conducting a bilinear interpolation to thereby stitch together processed tiles; combining the processed tiles into in a final equalized image; and analyzing the final equalized image to determine a presence of cancerous tumors within the diagnostic images.

10. The method of claim 9, wherein analyzing the final equalized image is analyzing the final equalized image with a convolutional neural network.

11. The method of claim 9, further including training a predictive model based upon the analyzing of the final equalized images.

12. The method of claim 11 , further including: producing a training data based upon the received one or more diagnostic images; and introducing adversarial perturbations in the training data to train the predictive model.

13. The method of claim 9, wherein: generating one or more diagnostic images is generating one or more diagnostic MRI images; and computing a histogram is computing a histogram equalization.

14. The method of claim 13, wherein generating one or more diagnostic MRI images is generating one or more diagnostic MRI images of a human brain.

15. The method of claim 11 , analyzing the final equalized image to determine a presence of cancerous tumors within the diagnostic images is determining a presence of glioblastomas.

16. The method of claim 12, wherein the training of the predictive model further includes tuning of hyper-parameters from the group of: weight penalty factor, learning rate, and momentum.

17. A system for determining the presence of cancerous tumors within diagnostic images of a human body, comprising: a computer device configured to perform the steps of: receiving one or more diagnostic images from a scanning device a generating one or more diagnostic images of a human body, the scanning device selectively transmitting the one or more diagnostic images to other computer devices; creating a plurality of tiles for each received image; computing a histogram for each tile; redistributing an intensity for each tile; creating a cumulative distribution function scaling for each tile; conducting a bilinear interpolation to thereby stitch together processed tiles; combining the processed tiles into in a final equalized image; and analyzing the final equalized image to determine a presence of cancerous tumors within the diagnostic images.

18. The system of claim 17, wherein the analyzing the final equalized image is analyzing the final equalized image with a convolutional neural network.

19. The system of claim 17, wherein the computer device further configured to perform the step of training a predictive model based upon the analyzing of the final equalized images.

20. The system of claim 19, wherein the computer device further configured to perform the steps of: producing a training data based upon the received one or more diagnostic images; and introducing adversarial perturbations in the training data to train the predictive model.