[go: up one dir, main page]

WO2025106998A1 - Procédés et systèmes de réglage optimal de soins pour des patients à l'aide d'un apprentissage automatique - Google Patents

Procédés et systèmes de réglage optimal de soins pour des patients à l'aide d'un apprentissage automatique Download PDF

Info

Publication number
WO2025106998A1
WO2025106998A1 PCT/US2024/056428 US2024056428W WO2025106998A1 WO 2025106998 A1 WO2025106998 A1 WO 2025106998A1 US 2024056428 W US2024056428 W US 2024056428W WO 2025106998 A1 WO2025106998 A1 WO 2025106998A1
Authority
WO
WIPO (PCT)
Prior art keywords
patient
discharge
act
data
sub
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/US2024/056428
Other languages
English (en)
Inventor
Enhao LIU
Hao Howard ZHANG
Colleen Michelle ENNETT
Warren Dean D'SOUZA
William Porter BAME
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Maryland Medical System Corp, University of
Original Assignee
Maryland Medical System Corp, University of
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Maryland Medical System Corp, University of filed Critical Maryland Medical System Corp, University of
Publication of WO2025106998A1 publication Critical patent/WO2025106998A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/20—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/01—Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/26—Government or public services
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
    • G16H10/60—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for patient-specific data, e.g. for electronic patient records
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/30—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for calculating health indices; for individual health risk assessment

Definitions

  • At least one non-transitory computer-readable medium storing thereon sequences of computer-executable instructions for quantizing a likelihood of discharging a patient from a hospital is disclosed.
  • the sequences of computer-executable instructions include instructions that instruct at least one processor to: receive patient information about the patient, the patient information including information about a current hospitalization of the patient; receive hospital-unit-volume information corresponding to a rolling window of time, the hospital-unit-volume information indicating a number of patients admitted to, transferred into, or discharged from a hospital unit in which the patient currently resides; convert the patient information and the hospital-unit-volume information into a machine-interpretable format tailored to at least one binary output machine learning algorithm; execute the binary output machine learning algorithm to generate a plurality of binary models that predict the likelihood of discharging the patient within each timeframe of a plurality of timeframes; combine the plurality of binary models into a single combined discharge model, wherein combining the plurality of binary models includes optimizing weightings of discharge predictions for each binary model of the plurality of binary models; determine, using the single combined discharge model, at least one weighted discharge readiness quantization for the patient; transform the at least one weighted discharge readiness quantization into at least one discharge prediction
  • receiving the patient information includes accessing an electronic medical record (EMR) of the patient in real-time.
  • EMR electronic medical record
  • the rolling window of time corresponds to a most recent 12-hour span of time.
  • the plurality of binary models predict the likelihood of discharging the patient within one or more of a 12-hour timeframe, a 24-hour timeframe, a 36-hour timeframe, a 48-hour timeframe, a 60-hour timeframe, and a 72-hour timeframe.
  • the instructions further instruct the at least one processor to optimize the plurality of binary models for each timeframe according to one or more optimization metrics.
  • the optimization metrics include accuracy, precision, recall, and F1-score.
  • the instructions further instruct the at least one processor to execute a Bayesian optimization algorithm to fine-tune one or more hyperparameters of the at least one binary output machine learning algorithm.
  • fine-tuning the one or more hyperparameters of the at least one binary output machine learning algorithm includes selecting values for the one or more hyperparameters that maximize an area-under-the-precision-recall curve (AUC-PR) for imbalanced datasets.
  • optimizing the weightings of discharge predictions for each binary model of the plurality of binary models includes: assigning a weight to a discharge prediction of each timeframe of the plurality of timeframes to produce a plurality of weighted discharge predictions; and identifying an optimal value of each weight.
  • identifying the optimal value of each weight includes determining a value of each weight that maximizes a median of a sum of the plurality of weighted discharge predictions. In another example of the at least one non-transitory computer-readable medium, identifying the optimal value of each weight includes imposing: a first restriction on the sum of the plurality of weighted discharge predictions to maximize a distribution of discharge predictions for patients likely to be discharged; and a second restriction on the sum of the plurality of weighted discharge predictions to minimize a distribution of discharge predictions for patients not likely to be discharged.
  • the first restriction includes imposing a minimum-value threshold on a first quartile of the sum of the plurality of weighted discharge predictions.
  • the second restriction includes imposing a maximum-value threshold on a third quartile of the sum of the plurality of weighted discharge predictions.
  • transforming the at least one weighted discharge readiness quantization into the at least one discharge prediction for the patient includes converting a likelihood of patient discharge into a value on a standardized scale of a plurality of standardized values.
  • the at least one discharge prediction is the value on the standardized scale of the plurality of standardized values.
  • the instructions further instruct the at least one processor to train each binary model of the plurality of binary models using at least one of Random Forest, Gradient Boosting, Support Vector Machines, or Neural Networks.
  • the binary output machine learning algorithm includes one or more gradient-boosted decision trees, transformer-based models, at least one custom deep learning architecture, Random Forest, support vector machine (SVM), k-nearest neighbors (KNN), or Lasso/Ridge regression.
  • the at least one custom deep learning architecture includes at least one of a convolutional neural network, a recurrent neural network, or a hybrid model.
  • the instructions further instruct the at least one processor to: receive at least one of new patient information or new hospital-unit-volume information; and re-train the binary output machine learning algorithm using the at least one of the new patient information or the new hospital-unit- volume information.
  • the instructions further instruct the at least one processor to: execute the re-trained binary output machine learning algorithm to determine at least one updated weighted discharge readiness quantization for the patient; transform the at least one updated weighted discharge readiness quantization into at least one updated discharge prediction for the patient; and control the user interface to provide the at least one updated discharge prediction to the user.
  • the instructions further instruct the at least one processor to: receive at least one of new patient information or new hospital-unit-volume information; execute the binary output machine Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) learning algorithm with respect to the at least one of new patient information or the new hospital- unit-volume information to determine at least one updated weighted discharge readiness quantization for the patient; transform the at least one updated weighted discharge readiness quantization into at least one updated discharge prediction for the patient; and control the user interface to provide the at least one updated discharge prediction to the user.
  • a method of quantizing a likelihood of discharging a patient from a hospital including: receiving patient information about the patient, the patient information including information about a current hospitalization of the patient; receiving hospital-unit-volume information corresponding to a rolling window of time, the hospital-unit-volume information indicating a number of patients admitted to, transferred into, or discharged from a hospital unit in which the patient currently resides; converting the patient information and the hospital-unit- volume information into a machine-interpretable format tailored to at least one binary output machine learning algorithm; executing the at least one binary output machine learning algorithm to generate a plurality of binary models that predict the likelihood of discharging the patient within each timeframe of a plurality of timeframes; combining the plurality of binary models into a single combined discharge model, wherein combining the plurality of binary models includes optimizing weightings of discharge predictions for each binary model of the plurality of binary models; determining, using the single combined discharge model, at least one weight
  • FIG.1 illustrates a process depicting an inpatient’s potential journey through various care settings in a hospital according to an example
  • FIG.2 illustrates a block diagram of an anticipatory system according to an example
  • FIG.3 illustrates a machine-learning process to predict the length of stay of a patient admitted into the hospital according to an example
  • FIG.4 illustrates a machine-learning process to predict the deterioration risk of a patient admitted into the hospital according to an example
  • FIG.5 illustrates a graphical user interface associated with predicting deterioration risks of patients in a hospital according to an example
  • FIG.6 illustrates a machine-learning process to predict a patient’s readiness for a care downgrade at a future time according to an example
  • FIG.7 illustrates a machine-learning process to predict a patient’s clinical readiness for discharge at a future time according to an example
  • FIG.8 illustrates a graphical user interface associated with predicting clinical readiness for discharging patients in a hospital in the next 24 hours according to an example
  • a Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) hospital may include an organ-transplant unit, a general medicine unit, a trauma unit, a pediatrics unit, and so forth. Large hospitals may have dozens of discrete units. Controlling patient flow throughout a hospital from patient arrival to departure (and all movements in between) may involve vigilant monitoring of patients’ conditions and anticipating future clinical events to optimally manage patients and ensure they get the proper care at the right time and location. Currently, such control mostly relies on the clinical intuition and experience of healthcare professionals and is more reactive than anticipatory.
  • FIG.1 illustrates a process 100 depicting an inpatient’s potential journey through various care settings in a hospital according to an example.
  • Certain acts of the process 100 are described as occurring in sequence solely for purposes of explanation rather than limitation. However, in many implementations, acts of the process 100 may be executed in different orders and/or in parallel with one another. As one non-limiting example, one or more of acts 110, 114, 118, 124, 126, and/or 128 may be executed in parallel and/or in a different order than that depicted in FIG. 1. In various examples, many of the acts of the process 100 (including, for example, acts 110, 114, 118, 124, 126, and/or 128) may be executed repeatedly and/or continuously throughout execution of the process 100.
  • the process 100 starts with admitting a patient to a hospital.
  • the patient is admitted to an acute care bed.
  • the process 100 may proceed to a different act, such as act 116 or act 120 depending on the patient’s medical condition.
  • acts 110, 116, and 120 may include moving a patient to an acute-care bed, an intensive care unit (ICU) bed, or an intermediate care unit (IMC) bed, respectively.
  • the act 102 may include determining which hospital unit is most likely to meet the patient’s needs (for example, between the units at the acts 110, 116, or 120) and proceeding to the corresponding act.
  • the length of stay (LOS) of the patient in the hospital may be estimated.
  • a first machine-learning process may be employed to evaluate the patient’s LOS, which will be described in more detail below with respect to FIG.3.
  • the process 100 may then proceed to act 114, in one example, to predict the patient’s risk of in-hospital deterioration to assist in the decision-making of upgrading the patient’s care setting.
  • a second machine-learning process may be employed to predict the deterioration Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) likelihood of the patient, which will be described in more detail below with respect to FIGS.4 and 5.
  • the process 100 may proceed to act 124 to evaluate the patient’s clinical readiness for discharge. On the other hand, if the predicted deterioration likelihood is not low (“Otherwise”), the process 100 may proceed to act 116 to transfer or move the patient to, for example, an ICU bed. In one example, act 118 may follow the act 116 or act 120 to predict the patient’s readiness for care downgrade to assist in the decision-making of potentially downgrading the patient’s care setting in the hospital. A third machine-learning process may be employed to predict the downgrade readiness of the patient, which will be described in more detail below with respect to FIG.6.
  • the process 100 returns to the act 116 or 120 to maintain the patient in the ICU or IMC bed.
  • the process 100 proceeds to act 120 or 122 to downgrade the patient’s care setting to an IMC bed or an acute care bed, respectively.
  • the patient’s clinical readiness for discharge at a future time is predicted to assist in the decision-making of when to discharge the patient from the hospital.
  • a fourth machine-learning process may be employed to predict the clinical discharge readiness of the patient, which will be described in more detail below with respect to FIGS.7 and 8.
  • the process 100 returns to the act 122 to maintain the patient in the acute care bed.
  • the process 100 proceeds to act 126.
  • the operational readiness for discharging the patient is predicted to assist the decision-making of when to discharge the patient from the hospital.
  • a fifth machine-learning process may be employed to predict the operational readiness for discharging the patient, which will be described in more detail below with respect to FIG.9. If the predicted operational readiness is low (“Not Ready”), the process 100 returns to the act 122 to maintain the patient in the acute care bed.
  • the process 100 proceeds to act 128.
  • the patient’s discharge disposition is predicted to assist the decision-making of where to discharge the patient from the hospital.
  • a sixth machine- learning process may be employed to predict the discharge disposition of the patient, which will Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) be described in more detail below with respect to FIGS.10 and 11. If the predicted discharge disposition destination is not available (“Not Ready”), the process 100 returns to the act 122 to maintain the patient in the acute care bed. On the other hand, if the predicted discharge disposition destination is available (“Otherwise”), the process 100 proceeds to act 130.
  • the patient’s readmission risk is predicted to assess how likely a patient is to be readmitted to a hospital after discharge.
  • the readmission risk may be evaluated over a certain length of time after discharge, such as within one day, one week, one month, 30 days, 60 days, and so forth.
  • a seventh machine-learning process may be employed to predict the readmission likelihood of the patient, which will be described in more detail below with respect to FIGS.12 and 13. If the predicted readmission risk is not high (“Otherwise”), the process 100 proceeds to act 132 to discharge the patient.
  • the process 100 may return to the act 122 if the patient agrees to stay in the hospital, or still proceed to act 132 to discharge the patient if the patient insists on leaving the hospital. However, if the patient is still discharged, care personnel may intervene to address the risk and employ mitigation measures to reduce the risk of readmission for that patient.
  • the act 130 may be performed after the patient has been discharged from the hospital.
  • act 134 in one example, after a patient is discharged at the act 132, the patient may need to be readmitted to the hospital because, for example, the patient’s medical condition has deteriorated.
  • the patient may then be readmitted at the act 102 to re-experience at least a portion of the process 100.
  • one or more acts in the process 100 may be reordered or may extend over a period when multiple other acts are performed.
  • some or all of the acts 112, 114, 118, 124, 126, 128, and/or 130 may be performed in parallel.
  • acts 112, 114, 118, 124, 126, 128, and/or 130 may be repeatedly and/or continuously performed throughout a patient’s stay in the hospital.
  • an anticipatory system for assisting in clinical decision-making for optimal care settings may include one or more computing devices.
  • FIG.2 illustrates a block diagram 200 of an anticipatory system 201 according to an example.
  • the anticipatory system 201 includes at least one controller 202 (“controller 202”), at least one user interface 204 (“user interface 204”), memory and/or storage 206 (“memory 206”), and at Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) least one network interface 208 (“network interface 208”).
  • the anticipatory system 201 may be coupled to one or more external devices 210 (“external devices 210”) via the network interface 208. Medical personnel, such as the personnel tasked with changing care settings for patients, may use the anticipatory system 201 to aid in the decision-making of care setting changes.
  • the anticipatory system 201 may aid medical personnel by generating clinical predictions (for example, a patient’s admission likelihood, LOS, deterioration likelihood, downgrade readiness, discharge likelihood, operational readiness for discharge, discharge disposition, and readmission risk) for the medical personnel to consider as part of their medical evaluation and interventions.
  • the controller 202 may construct, optimize, and/or execute one or more machine-learning models to generate the clinical predictions based on information stored in the memory 206, and/or based on information received from the external devices 210, and display the clinical predictions on the user interface 204.
  • the user interface 204 may include a display device 212 (for example, a computer monitor, laptop display, smartphone display, or other such displays) to display the clinical predictions. Accordingly, references to the user interface 204 displaying information may include the display device 212 displaying information. Medical personnel may also input information via the user interface 204, such as (near) real-time information indicative of the patients’ status updates.
  • the controller 202 may recognize the (near) real-time information and automatically re-execute the one or more models to update the clinical predictions in real-time based at least in part on the (near) real-time information.
  • the process 100 proceeds to act 112 to predict the patient’s estimated LOS in the hospital.
  • the act 112 may be performed at any time throughout a patient’s inpatient stay to predict and update the LOS before discharge.
  • LOS is defined as the number of days an inpatient occupies a hospital bed from admission through discharge. Having the ability to accurately estimate LOS prior to patient discharge can assist with tactical and short-term decision-making, such as discharge planning, staffing shortfalls, and a variety of other flow and throughput issues.
  • LOS is not easy to estimate or predict.
  • FIG.3 illustrates a machine-learning process 400 to execute the act 112 to predict the LOS of a patient admitted into the hospital according to an example.
  • the process 400 starts by extracting and compiling various types of data.
  • the sub-act 402 may include sub-acts 402a and 402b to extract several years (for example, five years) of clinical, operational, and financial data, primarily focused on patients and patient encounters, via various acquisition methods or standards.
  • the sub-act 402a extracts data via the health level seven (HL7) standards including the fast healthcare interoperability resources (FHIR) standard and the sub-act 402b extracts data via various web services.
  • Other methods may be included in the sub-act 402, such as the KB_SQL queries, application programming interface (API)/web service calls, and event-driven real-time interfaces.
  • API application programming interface
  • An electronic medical record is an application and database that acquires and stores clinical, operational, and financial data related to healthcare and health system operations.
  • most data extracted at the sub-act 402 is extracted from the EMR. Categories of data acquired from the EMR include patient demographics, patient history, encounter details, medications, lab tests, vital signs, radiology results, nursing documentation, social determinants of health (SDOH), and financial details.
  • data are extracted using the inpatient (or expected to be inpatient) or observation encounter en, where n ⁇ ⁇ 1,...,N ⁇ during the past five years, where each data sample represents a flat longitudinal patient. More recent data, especially physiologic measurements, are considered most relevant while older data are treated as less relevant and often summarized.
  • the extracted and compiled data are preprocessed.
  • the sub-act 404 may include sub-acts 404a and 404b.
  • the response variables for the ground truth of a LOS model are estimated based on after-the-fact LOS data.
  • an objective of the act 112 is to provide not only point LOS estimates but also likely LOS ranges.
  • M2212-7001WO(UMMS23-0003/WO.10) For example, a series of predictive LOS models may be used, each trained only on data available at a specific time point for each encounter. Each LOS model estimates the remaining encounter LOS and also produces a likely upper/lower bound.
  • a 48-hour LOS model may be trained using only data available between admission and the 48th hour of encounter and outputs LOS-48 and likely upper and lower bounds.
  • a total of 20 models may be created using a set of time cut-offs, for example, 12 hours, 24 hours, 36 hours, 48 hours, 60 hours, 72 hours, 84 hours, 96 hours, 108 hours, 120 hours, 144 hours, 168 hours, 192 hours, 216 hours, 240 hours, 264 hours, 288 hours, 312 hours, 336 hours, and 360 hours of encounter.
  • each model beyond 120 hours may contain fewer and fewer encounters.
  • the extracted and compiled data are feature-encoded and feature-engineered.
  • the goal of both feature-encoding and feature-engineering is to maximize the predictive power of the extracted data. Many techniques for both encoding and engineering may be used.
  • encoded variables may include single variables such as numeric, boolean, ordinal, and categorical data features.
  • Numerical data features include extracted data in the raw form such as height, weight, and BMI.
  • Boolean data features may be encoded by uniformly converting extracted data to 1 (yes) and 0 (no) for information of, for example, whether a patient is on IV therapy, is in the ICU, or has previous encounters with a hospital or hospital system.
  • Ordinal data features may be converted from the extracted data into a series of sequential integers while preserving their original rankings (for example, pain score, admission level of care, and Branden score).
  • the data features may be encoded by either one-hot encoding (dummy encoding) or frequency encoding.
  • data features relating to admission source, mode of arrival, and primary insurance type may use one-hot encoding
  • data relating to ZIP code, financial class, and type of breathing assistance may use frequency encoding.
  • the data features may be encoded by either hash encoding or grouping (especially when an appropriate industry standard “grouper” is available).
  • some EMRs may define tens of thousands of order codes, some with industry- standard coding (ICD or CPT), and some without. Because not all of them can be grouped in useful or meaningful ways, hash encoding is used to reduce the dimensionality while attempting to preserve enough detail to facilitate predictive modeling.
  • medication data is generally stored as national drug code (NDC) codes, of which there are tens of thousands.
  • NDC national drug code
  • ATC anatomical therapeutic chemical
  • WHO World Health Organization
  • Diagnoses are generally represented as international classification of diseases (ICD) codes. Again, there are tens of thousands of ICD codes. To group these into manageable categories, clinical classifications software refined (CCSR) body systems and clinical categories are used.
  • CCSR clinical classifications software refined
  • variables that change over time are also encoded from the data features and evaluated based on performance. Although many healthcare data variables change over time, vital signs, for example, and the resulting chronological vectors are often irregular and/or sparse. Nevertheless, these types of data are still treated as time series variables. All data recorded beyond each time point is excluded. In addition, data are not limited to the current encounter, but anything recorded more than one year before the exclusion date is excluded.
  • the features created include first value, most recent value, time since the most recent value is recorded, total number of values recorded, and average number of values recorded per day. Additional features are created for each numeric time series variable, which include median value, minimum value, maximum value, variance, difference between the median value and the most recent value, difference between the median value and the fist value, difference between the first value and the most recent value, first quartile Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) boundary, third quartile boundary, time spent in each quartile, time spent in each quartile as a percentage of total encounter time, current quartile, and the slope of a regression model fit to the values (for example, heart rate, pain score, and white blood count).
  • Additional features may be created for each boolean time series variable, including time spent in positive and negative states (in fractional days) and time spent in positive and negative states as a percentage of total encounter time (for example, time on a ventilator, in ICU, and for receiving antibiotic). Additional features may be created for each ordinal variable (after being transformed or encoded), including time spent in each ordinal level (in fractional days) and time spent in each ordinal level as a percentage of total encounter time (for example, pain score, Richmond agitation-sedation scale [RASS] level, and patient mobility).
  • RASS Richmond agitation-sedation scale
  • each categorical variable including time spent in each category (in fractional days) and time spent in each category as a percentage of total encounter time (for example, affect and behavior, respiratory within defined limits [WDL], and central line status).
  • time spent in each category in fractional days
  • time spent in each category as a percentage of total encounter time (for example, affect and behavior, respiratory within defined limits [WDL], and central line status).
  • WDL respiratory within defined limits
  • central line status for example, affect and behavior, respiratory within defined limits [WDL], and central line status.
  • a series of summarized variables are produced from less recent data.
  • the summarized variables may include encounter history, diagnostic history, and medication history.
  • Encounter history may include previous encounter data, number of inpatient encounters, average number of inpatient encounters per year, days since most recent inpatient encounter, number of emergency department (ED)/observation encounters, average number of ED/observation encounters per year, days since most recent ED/observation encounter, number of scheduled outpatient encounters, days since most recent outpatient encounter, number of inpatient days over the past five years (or other periods of time), number of surgical procedures performed over the past five years (or other periods of time), days since most recent surgical procedure, discharge disposition and location for the most recent encounter, number of different home addresses recorded, and/or other information.
  • ED emergency department
  • Diagnostic history may include CCSR categories for all diagnoses recorded prior to the current encounter (one Boolean feature per category) and CCSR body systems for all diagnoses recorded prior to the current encounter (one Boolean feature per body system).
  • Medication history may include ATC pharmacological groups for all medications recorded prior to the current encounter (one boolean per group) and ATC therapeutic groups for all medications recorded prior to the current encounter (one boolean per group).
  • the type of machine-learning algorithm for the LOS model is selected from various types of algorithms.
  • the sub-act 406 includes sub-acts 406a and 406b.
  • various binary models are trained, each using a different type of algorithm.
  • data folds and splits are constructed for an augmented version of k-fold cross-validation.
  • the data set resulting from feature encoding and engineering is first split into two sets, one containing the oldest 85% of encounters (train-0), and the other containing the most recent 15% of encounters (test-0).
  • Train-0 is further split into five “folds” of relatively equal size.
  • Two constraints are imposed on the train-0 splitting process: to avoid cross-encounter data leakage, each fold is limited to a unique set of patients, including all encounters for those patients; each sub-set contains at least one sample of each category for all categorical variables.
  • Each type of algorithm is trained using only split 1, with folds 2-5 used for training and fold 1 used for measuring performance. In one example, only four representative time points were used, which are, for example, 24 hours, 48 hours, 72 hours, and 96 hours.
  • Hyperparameters for each model are chosen based on a coarse-grained grid search. If more than one variable/feature encoding was relevant, each may be tried sequentially.
  • Features are culled separately for each model using algorithm-specific methods, requirements, and constraints. In some examples, further transformation and/or augmentation of the data may be required. For example, some algorithms cannot handle missing values so various imputation methods are used to address them. Some algorithms perform best when numeric data have been normalized so appropriate normalization techniques may be applied to those features.
  • the evaluated types of algorithms include Quantile Random Forest, Quantile Gradient Boosting, and Deep Learning (minimizing Pinball Loss).
  • the performance of each tuned model is evaluated using a set of metrics. Although the target variable is binary, the desired model outputs are LOS point estimates and a likely LOS range.
  • quantile loss may be chosen as the primary metric with mean average error (MAE) as Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) a secondary metric.
  • MAE mean average error
  • error distributions may be analyzed to understand the conditions under which each model performs best and worst.
  • Quantile Gradient Boosting using the XGBoost “extreme gradient boosting” library with a pinball loss objective function may be chosen as the final algorithm type for constructing the LOS binary models.
  • one or more other machine-learning algorithms may be chosen alone or in various combinations at the sub-act 406b.
  • the hyperparameters of the selected type of algorithm are optimized and feature culling is performed, both by training, to construct, optimize, validate, and update the final LOS models.
  • the sub-act 408 may include sub-acts 408a, 408b, 408c, 408d, and 408e.
  • the final models may be constructed. While a single 80/20 split for only four time points (for example, 24 hours, 48 hours, 72 hours, and 96 hours) may be used to evaluate modeling algorithms at the sub-act 406, five separate models using different 4+1 combinations are first constructed at the sub-act 408a for the hyperparameter optimization of each final LOS model.
  • Hyperparameters for each of the five separate models may be tuned using coarse grid search at the sub-act 408b.
  • the hyperparameters defined below are optimized to improve the performance of the five LOS models.
  • Number of trees may be limited to a particular threshold amount; for example, the number of trees may be capped at 2,000.
  • Maximum tree depth is the maximum depth of the trees which controls the model complexity and may be specified. Its range is uniformly set between any of various numbers, such as between 3 and 8.
  • Learning rate determines the step size during model training (for example, sampled from a logarithmic scale between 0.001 and 0.3) to allow for both small and large step sizes.
  • Subsample ratio represents the fraction of training samples used for building each tree, uniformly sampled between, for example, 0.5 and 1 with a step size of 0.1.
  • Column sample by tree specifies the fraction of features to be randomly sampled for each tree, sampled uniformly between, for example, 0.5 and 1 with a step size of 0.1.
  • Other hyperparameters may also be specified.
  • feature culling may also be performed for each of the five LOS models at the sub-act 408b. The importance of each feature for each LOS model may be determined using Shapley additive explanations (SHAP) analysis.
  • SHAP Shapley additive explanations
  • SHAP is a model-agnostic Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) approach that uses coalitional game theory to evaluate each model feature representing how much that feature contributes to the model’s output.
  • Features with zero contribution, based on the SHAP analysis, are dropped, and models are retrained without them to rule out anomalous behavior.
  • the sub-act 408a may include investigating and, when necessary, mitigating biases of various types. Subsets of results from all fine-tuned models are used to check for biases by comparing them against the performance of the overall model.
  • the performance on a subset of results including only female patients may be compared to the performance of the entire model and the performance of the non-female population.
  • a bias check may be performed for various categories including race, ethnicity, and sex. If an unacceptable level of bias is discovered, attempts to mitigate that bias may be made. Mitigation may be done by over-sampling the biased population or re-weighting training samples to put more emphasis on the biased population.
  • the final LOS models may be built using train-0 (a combination of all folds) and subsequently tested against test-0, a chronologically “most recent encounters” hold-out set, at the sub-act 408b.
  • Hyperparameters for the final LOS models may be derived by averaging hyperparameters across all five corresponding split-based LOS models.
  • the feature set for each final LOS model is a union of feature sets from all five corresponding split-based LOS models.
  • the final LOS models may be built using the corresponding time point’s complete data set.
  • the final LOS models may be validated. If the behavior of a final LOS model, including performance and levels of bias, is consistent with the split-based LOS models obtained at the sub-act 408a or sub-act 408b (“YES”), the process 400 proceeds to a sub- act 408d.
  • the process 400 returns to the sub-act 408a to find and fix the causes and reconstruct the final LOS model.
  • testing at the sub-act 408c reveals a time bias which is affecting model performance. Investigation of this unwanted behavior at the sub-act 408a reveals that LOS distributions are different during part of the COVID-19 pandemic (specifically March 2020 through January 2023).
  • the final LOS model behavior including performance and levels of bias must be consistent with individual split-based LOS Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) models (“YES”) at the sub-act 408a or sub-act 408b. If not (“NO”), further investigation and remediation at the sub-act 408a would be necessary.
  • the new data of the first type may include relevant information such as whether certain care units become opened or closed which will affect the workflow through the hospital, or whether a new initiative is introduced at the hospital that affects the workflow of the nurses or doctors and changes where and when data are documented.
  • the sub-act 408d may be configured to be performed periodically, for example, daily, weekly, monthly, or quarterly, to search for new data of the first type. If new data are available (“YES”), the process 400 proceeds to the sub-act 408e to extract, process, and insert the new first-type data to update the dataset. After the dataset is updated, the process 400 returns to the sub-act 408b to retrain the final model with the updated dataset. If no new first-type data are available (“NO”) at the sub-act 408d, the process 400 proceeds to a sub-act 410 to execute the final model closest to the encounter time to output the LOS predictions.
  • NO no new first-type data are available
  • the closest model is the 48-hour model; accordingly, the 48-hour model may be used to estimate the patient’s total LOS.
  • the closest model is the 48-hour model; accordingly, the 48-hour model may be used to estimate the patient’s total LOS.
  • Hourly LOS estimations and likely ranges may be provided to a discharge planning application. While the creation of the predictive LOS models may include only static historical data and the new first-type data, LOS predictions may change over time (sometimes quickly) as new EMR data of a second type becomes available. For example, the new second-type data may include new information associated with the patient or the care unit where the patient is currently bedded.
  • the LOS predictions may be produced and sent to the EMR in as timely a fashion as possible, which requires near real-time data acquisition.
  • Industry-standard mechanisms such as HL7 and FHIR, are the primary near-real-time acquisition methods.
  • Additional data acquisition methods may also be used, such as data analytics reporting tools, API web services, and KB_SQL queries. This allows an automatic near-real-time two-way data pipeline.
  • the process 400 proceeds through sub-act 318 depending on whether new second-type data becomes available.
  • the process 400 stays at the sub-act 314 to keep searching for new second- type data.
  • the sub-act 408d and the sub-act 318 may search for new data of the same type to be updated but may be performed at different periodicities. For example, the sub- act 318 may be performed more frequently than the sub-act 408d.
  • the process 100 proceeds to act 114 to predict the patient’s risk of in-hospital deterioration, such as serious negative health outcomes including organ failure, respiratory instability, sepsis, and even death, which may require an upgrade of the patient’s level of care.
  • FIG.4 illustrates a machine-learning process 500 to execute the act 114 to predict the deterioration risk of a patient admitted into the hospital according to an example.
  • the process 500 starts by extracting and compiling various types of data.
  • the sub-act 502 may include sub-acts 502a, 502b, and 502c to extract several years (for example, five years) of clinical, operational, and financial data, primarily focused on patients and patient encounters, via various acquisition methods or standards.
  • the sub-act 502a extracts data via the HL7 standards including the FHIR standard
  • the sub-act 502b extracts data via interconnect web services such as the EMR web services
  • the sub-act 502c extracts data via the KB_SQL queries.
  • Other methods may be included in the sub-act 502, such as API/web service calls and event-driven real-time interfaces.
  • An EMR is an application and database that acquires and stores clinical, operational, and financial data related to healthcare and health system operations.
  • most data extracted at the sub-act 502 is extracted from the EMR. Categories of data acquired from the EMR include patient demographics, patient history, health and medical encounter details, medications, lab tests, vital signs, radiology results, nursing documentation, social determinants of health (SDOH), and financial details.
  • data are extracted using the inpatient (or expected to be inpatient) or observation encounter en, where n ⁇ ⁇ 1,...,N ⁇ during the past five years, where each data sample represents a flat longitudinal single patient.
  • the extracted and compiled data are preprocessed.
  • the sub-act 504 may include sub-acts 504a and 504b.
  • the response variables for the ground truth of a deterioration risk model are estimated based on certain related clinical events. For example, these events may include an unplanned level of care elevation (acute to IMC or ICU, or IMC to ICU), a call to the rapid response team, a cardiac arrest, and death.
  • an objective of the act 114 is to identify patients up to a certain threshold time in advance of deterioration, such as four hours in advance of deterioration. Accordingly, patient data of deterioration events beyond the threshold time may be excluded. For patients who did not deteriorate, the median event time of all patients who did deteriorate may be used as the time threshold. After all variables have been analyzed individually and in concert to discover patterns, ranges, fluctuations over time, absences, distributions, correlations with other variables, and so forth, the most appropriate encoding techniques are chosen, and a data set created. Variables for which multiple encoding techniques are needed or wanted (for specific algorithms or discovery) are included multiple times in the data set, once for each encoding type.
  • the extracted and compiled data are feature-encoded and feature-engineered.
  • the goal of both feature-encoding and feature-engineering is to maximize the predictive power of the extracted data.
  • Many techniques for both encoding and engineering may be used.
  • the efficacy of each technique may be analyzed iteratively along with model and feature selection (as discussed in greater detail below) to identify one or more optimal techniques based on the chosen metric(s).
  • Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) In one example, for constructing the deterioration risk model, various types of variables are encoded from the data features and evaluated based on performance.
  • encoded variables may include numeric, boolean, ordinal, and categorical data features.
  • Numerical data features include extracted data in the raw form such as height, weight, and BMI.
  • Boolean data features may be encoded by uniformly converting extracted data to 1 (yes) and 0 (no) for information of, for example, whether a patient is on IV therapy, is in the ICU, or has previous encounters with a hospital or hospital system.
  • Ordinal data features may be converted from the extracted data into a series of sequential integers while preserving their original rankings (for example, pain score and admission level of care).
  • categorical data features when the number of categories is reasonable, the data features may be encoded by either one-hot encoding (dummy encoding) or frequency encoding.
  • data features relating to admission source, mode of arrival, and primary insurance type may use one-hot encoding
  • data relating to ZIP code, financial class, and type of breathing assistance may use frequency encoding.
  • the data features may be encoded by either hash encoding or grouping (especially when an appropriate industry standard “grouper” is available).
  • some EMRs may define tens of thousands of order codes, some with industry- standard coding, and some without. Because not all of them can be grouped in useful or meaningful ways, hash encoding is used to reduce the dimensionality while attempting to preserve enough detail to facilitate predictive modeling.
  • medication data is generally stored as NDC codes, of which there are tens of thousands.
  • ATC pharmacological and therapeutic groupings are used, as provided by the WHO. Diagnoses are generally represented as ICD codes. Again, there are tens of thousands of ICD codes.
  • CCSR body systems and clinical categories are used.
  • variables that change over time are also encoded from the data features and evaluated based on performance. Although many healthcare data variables change over time, vital signs, for example, and the resulting chronological vectors are often irregular and/or sparse. Nevertheless, these types of data are still treated as time series variables.
  • the features created include first value, most recent value, Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) time since the most recent value is recorded, total number of values recorded, and average number of values recorded per day.
  • Additional features are created for each numeric time series variable, which include median value, minimum value, maximum value, variance, difference between the median value and the most recent value, difference between the median value and the fist value, difference between the first value and the most recent value, first quartile boundary, third quartile boundary, time spent in each quartile, time spent in each quartile as a percentage of total encounter time, current quartile, and the slope of a regression model fit to the values (for example, heart rate, pain score, and white blood count). Additional features may be created for each boolean time series variable, including time spent in positive and negative states (in fractional days) and time spent in positive and negative states as a percentage of total encounter time (for example, time on a ventilator, in ICU, and for receiving antibiotic).
  • Additional features may be created for each ordinal variable (after being transformed or encoded), including time spent in each ordinal level (in fractional days) and time spent in each ordinal level as a percentage of total encounter time (for example, pain score, RASS level, and patient mobility). Additional features may be created for each categorical variable, including time spent in each category (in fractional days) and time spent in each category as a percentage of total encounter time (for example, affect and behavior, WDL, and central line status).
  • a series of summarized variables are produced from less recent data (that is, patient data of deterioration events beyond the threshold period, such as four hours).
  • the summarized variables may include encounter history, diagnostic history, and medication history.
  • Encounter history may include previous encounter data, number of inpatient encounters, average number of inpatient encounters per year, days since most recent inpatient encounter, number of ED/observation encounters, average number of ED/observation encounters per year, days since most recent ED/observation encounter, number of scheduled outpatient encounters, days since most recent outpatient encounter, number of inpatient days over the past five years (or other periods of time), number of surgical procedures performed over the past five years (or other periods of time), days since most recent surgical procedure, discharge disposition and location for the most recent encounter, number of different home addresses recorded, count of all deterioration events prior to the most recent encounter, days since the most recent deterioration event, and/or other information.
  • Diagnostic history may Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) include CCSR categories for all diagnoses recorded prior to the current encounter (one boolean feature per category) and CCSR body systems for all diagnoses recorded prior to the current encounter (one boolean feature per body system).
  • Medication history may include ATC pharmacological groups for all medications recorded prior to the current encounter (one boolean per group) and ATC therapeutic groups for all medications recorded prior to the current encounter (one boolean per group).
  • the type of machine-learning algorithm for the deterioration risk model is selected from various types of algorithms.
  • the sub-act 506 includes sub-acts 506a and 506b.
  • various binary models are trained, each using a different type of algorithm.
  • folds and splits are constructed for an augmented version of k-fold cross-validation.
  • the data set resulting from feature encoding and engineering is first split into two sets, one containing the oldest 85% of encounters (train-0), and the other containing the most recent 15% of encounters (test-0).
  • Train-0 is further split into five “folds” of relatively equal size.
  • Two constraints are imposed on the train-0 splitting process: to avoid cross-encounter data leakage, each fold is limited to a unique set of patients, including all encounters for those patients; each sub-set contains at least one sample of each category for all categorical variables.
  • Each type of algorithm is trained using only split 1, with folds 2-5 used for training and fold 1 used for measuring performance.
  • Hyperparameters for each model are chosen based on a coarse-grained grid search.
  • Features are culled separately for each model using algorithm- specific methods, requirements, and constraints. In some examples, further transformation and/or augmentation of the data may be required. For example, some algorithms cannot handle missing values so various imputation methods are used to address them. Some algorithms perform best when numeric data have been normalized so appropriate normalization techniques may be applied to those features. Some algorithms perform poorly if highly correlated features are used so a cross-correlation matrix is generated, and over-correlated features are dropped.
  • the evaluated types of algorithms include Ridge Regression with greedy FTRL feature selection, Support Vector Regression (linear, RBF, and polynomial), Random Forest, and/or Gradient Boosting (CatBoost and XGBoost).
  • the performance of each tuned model is evaluated using a set of metrics.
  • the desired model output is a probability expressed as a number between 0.0 (risk is 0%) and 1.0 (risk is 100%).
  • the data are unbalanced, with the prevalence of deterioration risk generally ranging between 8% and 12%.
  • AUC-ROC may be chosen as the primary measure of “goodness” partly because of its resistance to unbalanced data.
  • AUC-PR may be added as a secondary metric to provide a sanity check.
  • error distributions are analyzed to understand the conditions under which each model performs best and worst.
  • Gradient Boosting specifically Extreme Gradient Boosting (XGBoost)
  • XGBoost Extreme Gradient Boosting
  • one or more other machine-learning algorithms may be chosen alone or in various combinations at the sub-act 506b.
  • the hyperparameters of the selected type of algorithm are optimized and feature culling is performed, both by training, to construct, optimize, validate, and update the final admission likelihood model.
  • the sub-act 508 may include sub-acts 508a, 508b, 508c, 508d, and 508e.
  • the final model is constructed.
  • the hyperparameters defined below are optimized to improve the performance of the XGBoost model.
  • Number of trees may be limited to a particular threshold amount; for example, the number of trees may be capped at 2,500.
  • Maximum tree depth is the maximum depth of the trees which controls the model complexity. Its range is uniformly set between any of various numbers, such as between 3 and 8.
  • Learning rate determines the step size during model training (for example, sampled from a logarithmic scale between 0.001 and 0.3) to allow for both small and large step sizes.
  • Minimum child weight specifies the minimum sum of instance weights required for a child node to split, helping control overfitting. Its range and step Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) size may be selected according to various design concerns; in one example, the range is set between 1 and 25 with a step size of 5. Minimum split loss (Gamma) defines the minimum loss reduction required for partitioning a leaf node and controls regularization. Its value and step size may be selected according to various design concerns; in one example, the range minimum split loss is sampled between 0.5 and 1 with a step size of 0.05.
  • Subsample ratio represents the fraction of training samples used for building each tree, uniformly sampled between, for example, 0.5 and 1 with a step size of 0.1.
  • Column sample by tree specifies the fraction of features to be randomly sampled for each tree, sampled uniformly between, for example, 0.5 and 1 with a step size of 0.1.
  • Scale positive class weight is a parameter that may be particularly beneficial for handling imbalanced classes in a binary setup. It adjusts the balance of positive and negative weights to handle imbalanced data, sampled uniformly between, for example, 1 and 5 to give more weight to the positive class. Lambda (regularization rate in L2 regularization) may also be specified.
  • the hyperparameters for each of the five models are fine-tuned using a differential evolution (DE) optimization procedure.
  • DE is an extremely parallelizable technique so the workload may be spread across several servers to reduce run time.
  • the goal of the optimization is to identify the optimal hyperparameter set ⁇ ⁇ that maximizes the AUC-ROC score, expressed as an objective function ⁇ ⁇ ⁇ ⁇ .
  • ⁇ ⁇ arg m ⁇ a ⁇ x ⁇ ⁇ , where ⁇ represents the hyperparameters to be optimized (for example, learning rate, maximum depth, regularization terms, and so forth), and ⁇ is the space of possible hyperparameters.
  • ⁇ represents the hyperparameters to be optimized (for example, learning rate, maximum depth, regularization terms, and so forth)
  • ⁇ is the space of possible hyperparameters.
  • TPE tree-structured Parzen estimator
  • TPE models the likelihood of good and bad hyperparameter configurations based on previous observations.
  • ⁇ be the likelihood of high-performing hyperparameters (those improving AUC-ROC) and ⁇ ⁇ ⁇ ⁇ be the likelihood of low-performing hyperparameters (those reducing AUC-ROC).
  • feature culling may also be performed at the sub-act 508b.
  • the importance of each feature for each model is determined using SHAP analysis.
  • SHAP is a model-agnostic approach that uses coalitional game theory to evaluate each model feature representing how much that feature contributes to the model’s output.
  • Features with zero contribution, based on the SHAP analysis, are dropped, and models are retrained without them to rule out anomalous behavior.
  • the sub-act 508a may include investigating and, when necessary, mitigating biases of various types.
  • Subsets of results from all fine-tuned models are used to check for biases by comparing them against the performance of the overall model. For example, the performance on a subset of results including only female patients may be compared to the performance of the entire model. For example, a bias check may be performed for the various categories including race, ethnicity, sex, language, and religion. If an unacceptable level of bias is discovered, attempts to mitigate that bias may be made. Mitigation may be done by over-sampling the biased population or re-weighting training samples to put more emphasis on the biased population.
  • the final model is built using train-0 (a combination of all folds) and subsequently tested against test-0, a chronologically “most recent encounters” hold-out set, at the sub-act 508b.
  • Hyperparameters for this final model are derived by averaging hyperparameters across all five split-based models.
  • the feature set for the final model is a union of feature sets from all split-based models.
  • the final model is built using the complete data set. Hyperparameters and the feature set are the same as for the train-0 model. Isotonic regression coefficients are calculated specifically for the final model.
  • the final model is validated.
  • the process 500 proceeds to a sub-act 508d. Otherwise (“NO”), the process 500 returns to the sub-act 508a to find and fix the causes and reconstruct the final model.
  • the sub-act 508d in one example, it is determined whether new real-time data of a first type are available for updating the dataset for training.
  • the new data of the first type may include relevant information such as whether certain care units become opened or closed which will affect the workflow through the hospital, or whether a new initiative is introduced at the hospital that affects the workflow of the nurses or doctors and changes where and when data are documented.
  • the sub-act 508d may be configured to be performed periodically, for example, daily, weekly, monthly, or quarterly, to search for new data of the first type. If new data are available (“YES”), the process 500 proceeds to the sub-act 508e to extract, process, and insert the new first-type data to update the dataset. After the dataset is updated, the process 500 returns to the sub-act 508b to retrain the final model with the updated dataset.
  • the process 500 proceeds to a sub-act 510 to execute the final model to output the deterioration likelihood prediction.
  • the final deterioration risk model may output a report showing the changes in the deterioration risk prediction over time in the context of a variety of patient-specific clinical measures being implemented at different times.
  • the report may be displayed via the user interface 204 and may be in textural and/or graphical forms. Discretization of the output prediction at the sub-act 514 may still be needed because machine-learning models, even those that produce continuous results in the range of 0 to 1, do not generally produce outputs that resemble probabilities. Probabilistic alignment may be desirable for two reasons.
  • non-probabilistic results can cause end-user cognitive dissonance. For example, people not trained in machine learning, statistics, or similar disciplines may not understand numbers that have no inherent meaning beyond rank.
  • model retraining may be done regularly (for example, quarterly) to incorporate new data and discover new patterns. Retrained models are likely to produce outputs with different characteristics (ranges, distributions, etc.) than their predecessors.
  • the model outputs are transformed to more closely match probabilities while preserving overall rank. If the error distribution of the final model exhibits sigmoidal characteristics, isotonic regression may be Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) chosen as the transformation technique.
  • the pseudo-probabilities are discretized by thresholds pyramidally into four categories, where category 4 (highest risk) contains the smallest number of patients, and category 1 (lowest risk) contains the largest number of patients. This facilitates color coding and at-a-glance interpretation for end-users. In other examples, other numbers of discrete categories may be used for the pseudo-probabilities of the final model. While the creation of the predictive model may include only static historical data, per- patient deterioration risk scores may change over time (sometimes quickly) as new EMR data of a second type becomes available. For example, the new second-type data may include new information associated with the patient or the care unit where the patient is currently bedded.
  • the deterioration risk scores may be calculated and sent to the EMR in as timely a fashion as possible, which requires near-real-time data acquisition.
  • Industry-standard mechanisms such as HL7 and FHIR, are the primary near-real-time acquisition methods.
  • Several other data interoperability and acquisition methods are also used, such as data analytics reporting tools, API web services, and KB_SQL queries. This allows an automatic near-real-time two-way data pipeline.
  • the process 500 proceeds through sub-act 318 depending on whether new second-type data becomes available.
  • each readmission risk score may be updated in the EMR system display every few minutes (for example, every 8 to 12 minutes), depending on the availability of new data, to reduce the amount of variability in the displayed score over time and enhance the usability of the deterioration risk scores for clinical users.
  • the sub-act 318 is executed to extract, process, and insert the new data to update the training dataset.
  • the process 500 then returns to the sub-act 510 to re-execute the final model. If there is no new second-type data available at the sub-act 318, the process 500 stays at the sub-act 318 to keep searching for new second-type data.
  • the sub-act 508d and the sub-act 318 may search for new data of the same type to be updated but may be performed at different periodicities.
  • the sub- act 318 may be performed more frequently than the sub-act 508d.
  • the machine-learning process 500 for executing the act 114 to predict the patient’s deterioration risk may be performed via the user interface 204.
  • the user interface Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) 204 may display a graphical user interface (GUI) associated with the act 114.
  • GUI graphical user interface
  • FIG.5 illustrates a GUI 600 associated with predicting deterioration risks of patients in a hospital according to an example.
  • a column 602 displays the colored or textured icons representing the patient deterioration risk scores predicted by the final model in descending order (from top to bottom) to assist in the decision-making of moving patients to higher levels of care in the next four hours. For example, if the deterioration risk score of a patient is in category 3 or 4, the corresponding icon may be displayed in red or of a first texture. If the deterioration risk score is in category 2, the corresponding icon may be displayed in yellow or of a second texture. If the deterioration risk score is in category 1, the corresponding icon may be displayed in green or of a third texture.
  • the process 100 proceeds to an act 124 at which the patient’s clinical readiness for discharge may be evaluated (described in more detail below). However, if the deterioration risk of the patient is predicted to be not low (medium or high), the process 100 proceeds to an act 116 to upgrade the patient’s level of care. For example, the patient’s level of care may be upgraded to ICU (or IMC or other levels higher than acute care bed at the act 110). In one example, following the act 116, the process 100 proceeds to an act 118 to predict the patient’s readiness for a care downgrade.
  • FIG.6 illustrates a machine-learning process 700 for executing the act 118 to predict the patient’s readiness for a care downgrade from a higher level of care (such as an ICU bed at the act 116 or IMC bed at the act 120) to a lower level of care (such as an IMC bed at act 120 or acute care bed at act 122) at a future time according to an example.
  • the process 700 starts with extracting and compiling patient data of different types in various ways.
  • the sub-act 702 includes a sub-act 702a and a sub-act 702b.
  • the sub-act 702a may include receiving or extracting patient information (for example, from an EMR), the patient information including both discrete and non-discrete data available in the EMR about patients for current inpatient encounters e n , where n ⁇ ⁇ 1,...,N ⁇ , such as information about the patient for the current hospitalization, time- features, event frequency features, prescription medications, patient’s healthcare insurer, comorbidities, and so forth.
  • patient information may be received in real-time.
  • the sub-act 702b may include extracting non- Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) structured data.
  • the sub-act 702 may include compiling, receiving, or extracting hospital- unit-volume data on hospital unit volume vn, where n ⁇ ⁇ 1,...,N ⁇ for each inpatient encounter en, which measures the number of patients admitted to, or discharged from the hospital unit where the patient currently resides within a rolling window of time (for example, the past 12 hours).
  • the hospital-unit-volume information may be received in real-time.
  • Patient data extraction and compilation at sub-act 702 may be performed in various ways.
  • the sub-act 702 may include systematically extracting input data from a variety of sources following standard medical data protocols such as HL7, FHIR, web services, connections to databases, data reporting schemes, and so forth. These protocols ensure consistent and seamless integration of data from multiple healthcare settings and devices.
  • the sub-act 702 may include automatic data collection. Many data points, such as vital signs (heart rate, blood pressure), lab results, and medication administration, are collected automatically from medical devices integrated with EMR systems (for example, bedside monitors, laboratory systems that process bloodwork samples, and drug dispensing systems). These automatic inputs minimize manual errors and enhance the efficiency of data gathering.
  • the sub-act 702 may include extracting patient data from manual data entry. Some data, such as clinician's assessments including height, weight, blood pressure, care plans including medications, family history, allergies, and patient-reported outcomes including pain score and symptoms, may be manually entered by healthcare professionals. To maintain consistency and accuracy, standard templates and checklists are used across different healthcare providers.
  • the sub-act 702 may include extracting patient data using wearables and remote devices. In certain examples, data from patient-worn devices (for example, glucose monitors and fitness trackers) or remote monitoring systems can also be integrated into the EMR to provide continuous, real-time data streams.
  • data may be extracted from diverse patient populations across different demographic groups, medical Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) conditions, and care settings.
  • patient demographics such as age, gender, ethnicity, and socioeconomic status are commonly captured during hospital admission processes.
  • Diagnosis codes (ICD-10), clinical procedure codes, and patient treatment plans are also included to capture the diversity of patient conditions and healthcare needs.
  • the data compilation at the sub-act 702 may include integrating information from various healthcare settings, including hospitals, clinics, and specialized care facilities. This comprehensive approach provides a holistic view of patient clinical encounters and outcomes.
  • the data extraction at the sub-act 702 may follow data quality assurance protocols.
  • sub-act 704 In addition to automated checks for missing values and outliers, manual validation may be employed by clinical experts to confirm the accuracy of key variables, especially those that are manually entered or highly complex (for example, diagnostic codes and treatment plans).
  • privacy regulations such as HIPAA may be maintained. All data may be encrypted in transit and at rest, and de-identification techniques may be used when sharing or analyzing data, protecting sensitive patient information while allowing for meaningful analytics.
  • the extracted and compiled patient data are preprocessed.
  • the sub-act 704 includes sub-acts 704a, 704b, and 704c.
  • the response variables for ground truth are defined.
  • the compiled dataset is preprocessed partly to establish reliable response variables, which serve as the ground truth labels for each binary model predicting downgrade readiness at specific future time intervals. For example, twelve binary models may be employed, each predicting the downgrade readiness at 2, 4, 6, 8, 10, ... , 24 hours in the future, respectively.
  • the primary objective of defining the response variables is to accurately label each instance as either “ready for downgrade” or “not ready” within these designated timeframes.
  • actual transfer times recorded in the EMRs are used as direct indicators of readiness for transfer to a lower or downgraded level of care. In cases where transfer times are not available or applicable, clinically validated proxies are utilized to represent a patient’s readiness for downgrade.
  • proxies may include specific clinical events or status changes, such as completion of stabilization protocols, achievement of certain physiological thresholds, or documented physician assessments indicating readiness for transfer. Each proxy is carefully chosen in consultation with clinical experts to ensure accuracy and relevance.
  • Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) By defining these response variables in this manner, the models may be trained with labels rooted in real clinical outcomes, enhancing the predictive performance and practical utility of each binary model across all timeframes. This systematic approach provides a strong foundation for developing models that can reliably forecast patient downgrade readiness, allowing for more informed and timely clinical decision-making in ICU and IMC settings.
  • the extracted patient data are feature-engineered to be ready for the following data transformation.
  • the sub-act 704b includes selecting features of current encounter information including critical aspects of the patient’s current hospital stay. This information includes patient demographics and clinical vital signs, hospital services and diagnoses, admission details, procedures and scheduled surgeries, medications and treatment plans, therapist evaluations and mobility assessments, and lab orders and results.
  • Patient demographics and clinical vital signs may include core data, such as age, weight, height, body surface area (BSA), BMI, blood pressure, pulse, and temperature, which establish a baseline clinical profile.
  • Hospital service and diagnoses may include information about current hospital services, primary diagnosis, admission reason, and overall acuity level, which provide insight into the patient’s condition and care requirements.
  • Admission details may include factors such as admission type, source, level of care, unit-room-bed location, and arrival means (for example, emergency or scheduled admission), which give context to initial patient needs and subsequent changes in care level.
  • Procedures and scheduled surgeries may include any recent procedures, ongoing treatments, or upcoming surgeries (for example, scheduled transitions from invasive to non- invasive support), which are critical indicators of care trajectory. These planned interventions often impact the patient’s readiness for a lower level of care, and their inclusion enhances the models’ ability to anticipate transfer timelines.
  • Medications and treatment plans may include key aspects of the patient’s medication regimen, including recent changes in dosages or additions of critical medications (for example, antibiotics, sedatives, or vasoactive agents), which reflect real- Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) time clinical management and signal improvements or deteriorations relevant to downgrade readiness.
  • Therapist evaluations and mobility assessments may include mobility scores, such as AM-PAC “6 Clicks” scores alongside therapist notes on the patient’s ability to perform daily activities, which offer important insight into functional status and progress toward transfer readiness.
  • Lab orders and results may include data on lab tests (for example, blood gas analyses and electrolyte panels) which provide snapshots of vital organ function and stability and support decisions related to downgrade readiness based on clinical stability.
  • the sub-act 704b may include selecting features of the most recent encounter information. To capture a holistic view of the patient’s care trajectory, important data from the patient’s most recent hospitalization, if available, is included in the feature engineering. This data may include historical stay attributes including the total length of stays, the level of care provided, and the hospital services utilized during the previous encounter, which provides context to the current clinical scenario. The data may also include previous diagnoses and clinical events. Previous diagnoses and significant clinical events (for example, ICU admission history and major surgeries) help create a timeline of patient acuity and recovery patterns, which informs the likelihood of downgrade readiness in the current encounter. The sub-act 704b may include selecting derived features from time-related dynamics and event frequencies.
  • Derived features are generated to provide insights into time-related dynamics and event frequencies. These time-related dynamics track key aspects of a patient's healthcare clinical events, such as the number of days between the current and previous visits, the total hours spent in the hospital, and the time since the last mobility evaluation. Moreover, day-of- week effects are accounted for to capture potential variations in care based on the day the visit occurred.
  • Event frequency features measure the frequency and cumulative occurrences of specific clinical events. For instance, the recent event counts, which represent the number of lab orders, transfers, or patient updates within a recent timeframe (for example, the past 12 hours) may be tracked. Cumulative event counts may also be calculated to monitor the total occurrences of key events (for example, lab orders and transfers) since admission.
  • the sub-act 704b may include selecting derived features for hospital unit volume and bed turnover. These derived features are considered by tracking the number of patients admitted to, transferred into, or discharged from the hospital unit where the patient currently resides during a Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) defined period (for example, the past 12 hours or 24 hours). In addition, data on bed turnover rates and capacity utilization trends provide additional context on hospital pressures, helping predict whether a downgrade transfer is both feasible and appropriate within the desired timeframe.
  • the engineered data are transformed for input to the machine-learning models.
  • Sub-act 704c may include converting the patient information and/or hospital-unit-volume information into a machine-interpretable format tailored to a desired machine-learning model (for example, XGBoost).
  • XGBoost machine-learning model
  • a variety of transformation techniques may be applied based on the nature of the selected features.
  • the normalization technique may be used. This technique centers the data by removing the median and scales it based on the interquartile range, making it particularly resistant to outliers which are common in clinical datasets.
  • a patient’s BMI which might range from very low (for example, 15) to high (for example, 40), is normalized by subtracting the median BMI and scaling it based on the interquartile range (IQR).
  • IQR interquartile range
  • blood pressure readings such as systolic values
  • systolic values typically range between 90 and 180 mmHg but can exhibit extreme values due to acute conditions. By normalizing these values, they are centered around the median systolic blood pressure and scale based on the IQR. This helps mitigate the influence of unusually high or low readings, providing a more stable dataset for model training.
  • blood glucose levels measured in mg/dL, can range from normal fasting levels around 70–99 mg/dL to very high values (for example, 200+ mg/dL) in cases of diabetes or stress-induced hyperglycemia.
  • This transformation converts categorical variables into one-hot encoded vectors, ensuring that the models can effectively interpret and analyze them. In doing so, rare categories (those occurring fewer than 100 times) are grouped and any unknown categories are treated as infrequent to ensure that even lesser-seen data is accounted for.
  • hospital service type categories like “cardiology,” “oncology,” and “neurology” are each converted into separate binary columns (one for each service). If a patient is in Cardiology, their entry will have a “1” in the cardiology column and a “0” elsewhere.
  • means of arrival categories like “ambulance,” “walk-in,” or “transferred” are transformed similarly.
  • rare categories may be grouped under “other” to avoid sparsity issues in the models. If an unexpected category appears (for example, new means of arrival), the unexpected category may be treated as “other” to prevent the models from breaking due to unknown data.
  • the level of care such as “ICU,” “IMC,” or “nursery” are also one- hot encoded. Each level of care is assigned its own binary column, enabling the models to identify the intensity of care required. Patients in ICU, for example, will have a “1” in the ICU column and a “0” in others. Grouping rare levels under “other” ensures that less frequent levels do not affect model performance.
  • the AM-PAC (activity measure for post-acute care) G-code metrics are encoded based on functional ability, reflecting the likelihood of discharge disposition.
  • Lower-functioning codes for example, CN, CM, and CL
  • higher-functioning codes for example, CJ, CI, and CH
  • This numeric scale preserves the ranking, with CN mapped to 0 and CH to 6, to capture the progressive scale of functional ability.
  • Unknown codes are assigned a default value of -1.
  • the acuity level reflects the urgency of medical attention required, with levels ordered as “less urgent,” “urgent,” “emergent,” and “immediate.” These are assigned numeric values to capture the hierarchy, with 0 for “less urgent,” 1 for “urgent,” 2 for “emergent,” and 3 for “immediate.” This encoding ensures the model accurately interprets increasing acuity. Any unknown acuity level is assigned -1 to maintain data consistency. For textual data, which often include rich but unstructured data like the reason for visits, textual transformation is used. Textual transformation converts the text into a matrix of term frequency-inverse document frequency (TF-IDF) scores.
  • TF-IDF term frequency-inverse document frequency
  • the transformation captures the most significant terms that best represent the textual data, enabling the models to process and interpret these insights effectively.
  • a text field describing the patient’s reason for visiting the hospital for example, “chest pain” and “shortness of breath”
  • TF-IDF scores For example, “chest pain” and “shortness of breath”
  • common stop words for example, “the” and “is”
  • the most relevant terms for example, the top Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) 20 most relevant words
  • text fields of clinical notes are transformed into a matrix of TF-IDF scores, describing a patient's health status, treatment plan, and medical history such as quality improvement measures on deep vein thrombosis (DVT) prophylaxis, urinary catheter existence, ambulation status, and nutrition.
  • text fields of operative notes that describe a patient’s surgery such as type of surgery, estimated blood loss, type of anesthesia, and urine output are transformed into a matrix of TF-IDF scores.
  • a machine-learning algorithm is selected from multiple types of algorithms for each binary model.
  • the sub-act 706 may include sub-acts 706a and 706b.
  • the sub-act 706a includes training the twelve binary models described above to predict downgrade readiness in different future time frames using algorithms of different types.
  • the algorithm selection at the sub-act 706 may include training and evaluating only one or more of the twelve binary models using algorithms of different types.
  • Training separate models for each timeframe allows each model to focus on the clinical features most pertinent to that specific interval, enabling more precise and context-aware predictions.
  • the rationale for using multiple models lies in the complexity and variability of ICU or IMC patient conditions. Attempting to capture downgrade readiness across all time intervals with a single model would risk oversimplification and could dilute prediction accuracy because the relevance of specific clinical signals may fluctuate depending on the patient’s stage of recovery. For instance, vital signs, medication adjustments, or laboratory values can vary greatly in significance depending on whether a downgrade prediction is for the short term (for example, 2- 4 hours) or slightly longer intervals (for example, 12-24 hours). Multiple models allow us to better capture these nuances and account for clinical variability across the ICU or IMC care spectrum.
  • each timeframe uses distinct models for each timeframe to account for the unique distributions of clinical data, as indicators like heart rate variability, respiratory support needs, or lab trends may evolve differently over time.
  • M2212-7001WO(UMMS23-0003/WO.10) a model predicting downgrade readiness within the next 2 hours may weigh rapid changes in respiratory support more heavily whereas a model for a 24-hour interval might prioritize stabilized lab values and consistent vital signs.
  • a combined optimization approach is employed, which synthesizes each model’s output into a single, actionable downgrade readiness score. This ensemble method leverages the strengths of each timeframe-specific model while minimizing reliance on any single set of features.
  • a plurality of binary models each corresponding to one of the time intervals (from 2 hours to 24 hours), is executed and trained using different types of algorithms with default parameter settings, such as Random Forest, XGBoost, Support Vector Machines (SVM), and Neural Networks.
  • the models are trained based on the data with features obtained from sub-act 704.
  • a 5- fold cross-validation is employed using a dataset where the ground truth of whether the actual care downgrade occurred within the specified time frames is known.
  • the dataset is split into five parts, with each model trained on four folds and tested on the remaining fold. This is repeated five times to ensure that each part of the dataset is used as a test set exactly once, providing a robust assessment of model performance.
  • a comprehensive comparison based on a set of metrics is performed for the various machine-learning algorithms to predict the patient’s readiness for care downgrade within the next 2, 4, 6, ... , and 24 hours.
  • the binary models are evaluated based on key metrics including accuracy, precision, recall, F1-score, and the receiver operating characteristic-area under the curve (ROC-AUC).
  • ROC-AUC receiver operating characteristic-area under the curve
  • ROC-AUC measures a model’s ability to distinguish between two classes (ready for downgrade or not ready).
  • XGBoost may outperform other algorithms by being able to handle complex and imbalanced clinical datasets.
  • XGBoost may be able to prevent Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) overfitting through regularization.
  • XGBoost may be selected at sub-act 706b as the optimal algorithm for each of the binary models, ensuring reliable and high-performance predictions for patient downgrade readiness across all time intervals in ICU or IMC settings.
  • one or more other machine-learning algorithms may be chosen alone or in various combinations at the sub-act 706b.
  • a Bayesian optimization procedure may be implemented to fine-tune the hyperparameters of each binary model. This optimization ensures that each model maximizes its predictive performance while minimizing overfitting, allowing good generalization to unseen data. This allows further refining of the models’ accuracy to ensure that each binary model is a reliable tool for predicting whether a patient will be ready for transfer to a lower level of care within the specific time intervals.
  • the hyperparameters defined below are optimized to improve the performance of the XGBoost model.
  • Maximum tree depth is the maximum depth of the trees which controls the model complexity.
  • maximum tree depth may be set to a range between, for example, 3 and 10.
  • Learning rate determines the step size during model training, sampled from a logarithmic scale between a range of values (for example, 0.001 and 0.3) to allow for both small and large step sizes.
  • Minimum child weight specifies the minimum sum of instance weights required for a child node to split, helping control overfitting. In at least one example, minimum child weight may be set within a range between 1 and 6.
  • Gamma defines the minimum loss reduction required for partitioning a leaf node and controls regularization. In at least one example, the gamma value may be sampled between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.05).
  • Subsample ratio represents the fraction of training samples used for building each tree, uniformly sampled between a range of values (for example, a range of 0.5 and 1) with a specified step size (for example, a step size of 0.05).
  • Column sample by tree specifies the fraction of features to be randomly sampled for each tree, sampled uniformly between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.1).
  • Scale positive class weight is a parameter particularly beneficial for handling imbalanced classes in a binary setup. Scale positive class weight adjusts the balance of positive and negative weights to handle imbalanced data, sampled uniformly between a range of values (for example, 1 and 9) to give more weight to the positive class.
  • the evaluation metric for fine- tuning the hyperparameters is selected as the Area Under the Precision-Recall Curve (AUC-PR), which is more suitable for handling imbalanced classes.
  • AUC-PR Area Under the Precision-Recall Curve
  • the goal of the sub-act 708 is to identify the optimal hyperparameter set ⁇ ⁇ that maximizes the AUC-PR score, expressed as an objective function ⁇ ⁇ ⁇ ⁇ .
  • ⁇ ⁇ ⁇ ⁇ represent the AUC-PR score of the XGBoost model with hyperparameters ⁇ .
  • ⁇ ⁇ arg m ⁇ a ⁇ x ⁇ ⁇ , where ⁇ represents the hyperparameters to be optimized (for example, learning rate, maximum depth, and regularization terms), and ⁇ is the space of possible hyperparameters.
  • ⁇ represents the hyperparameters to be optimized (for example, learning rate, maximum depth, and regularization terms)
  • ⁇ is the space of possible hyperparameters.
  • TPE models the likelihood of good and bad hyperparameter configurations based on previous observations.
  • ⁇ be the likelihood of high-performing hyperparameters (those improving AUC-PR) and ⁇ be the likelihood of low-performing hyperparameters (those reducing AUC-PR).
  • the surrogate model is used to predict the performance of new hyperparameter configurations.
  • the surrogate model is iteratively refined as more real-time data is collected.
  • the Bayesian optimization procedure start at step one of initialization, in which a few initial random hyperparameter configurations are evaluated to gather baseline data. These configurations are chosen randomly within the specified hyperparameter space ⁇ and provide the initial observations necessary to build a surrogate model.
  • the surrogate model is fitted based on the observed data, constructing the ⁇ ⁇ ⁇ ⁇ and Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) ⁇ distributions.
  • the surrogate model serves as a probabilistic estimator that guides the search for optimal hyperparameters, predicting how different configurations will perform without needing to evaluate each one directly.
  • the next hyperparameter configuration is selected by maximizing the acquisition function, which balances exploration (testing hyperparameters where uncertainty is high) and exploitation (testing configurations likely to yield high performance based on the surrogate model). This ensures that the search efficiently converges towards optimal hyperparameters, even in complex search spaces.
  • the model is fit using XGBoost with a maximum of 500 boosting rounds per trial. Additionally, if there is no improvement in model performance after 100 boosting rounds, early stopping is triggered to prevent further unnecessary iterations. Performance may be evaluated using the AUC-PR score, which is particularly relevant for an imbalanced healthcare dataset, focusing on precision and recall.
  • the optimization continues by repeating steps two to four.
  • the total number of trials may be limited to a maximum value (for example, 50) to ensure that the search process remains efficient without overfitting the hyperparameter space. If, within these trials, the model reaches a plateau where further boosting rounds yield no significant improvement in AUC-PR scores, the optimization may be stopped and the hyperparameter fine-tuning is completed.
  • a formulation of a final downgrade readiness at a specific future time is constructed, optimized, validated, and updated.
  • a formulation to predict downgrade readiness in, for example, the next 6 to 12 hours is constructed that combines the weighted predictions from each model trained for all time intervals. The combination of the weights of these binary model predictions is optimized to generate an accurate and calibrated final downgrade readiness prediction for each patient.
  • the sub-act 710 includes sub-acts 710a, 710b, 710c, 710d, 710e, and 710f. At sub-act 710a, in one example, the formulation of the final downgrade readiness in the next 6 or 12 hours is constructed.
  • the primary goal of the formulation optimization of the final downgrade likelihood ⁇ is to maximize the median, &'( ⁇ across the training dataset. All ⁇ values computed from various patients and at various times ⁇ in the training data are gathered and sorted into a distribution. The median &'( ⁇ ⁇ ⁇ ⁇ ⁇ is the middle value in the sorted distribution, representing the typical likelihood across the population of all ⁇ ⁇ ⁇ ⁇ values.
  • two key constraints are imposed on the maximization of &'() ⁇ ⁇ ⁇ ⁇ * to optimize ⁇ ⁇ ⁇ ⁇ .
  • the distribution of the final downgrade readiness may be maximized for patients likely to be ready, particularly focusing on predictions updated during early morning hours before the anticipated transfer, to increase the accuracy of identifying patients ready for downgrade within a configurable target timeframe, for example, the next 6 or 12 hours.
  • the downgrade likelihood ⁇ is maintained sufficiently high when predicting downgrade readiness within the configurable target timeframe, for example, the next 6 or 12 hours, for enhancing early identification accuracy.
  • the distribution of final downgrade likelihood ⁇ ⁇ ⁇ ⁇ may be minimized for patients who are predicted to remain in the ICU 116 or IMC 120, reducing the risk of mistakenly categorizing patients as ready for downgrade when they should remain in the higher level of care.
  • a significant amount of randomly generated weight sets for example, 100,000 are generated and evaluated.
  • a formulation optimization procedure will be followed that maximizes the median, &'( ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , while ensuring that the two key constraints on quartiles +1) ⁇ * and +3) ⁇ * are met.
  • step two of evaluating &'( ⁇ ⁇ ⁇ ⁇ ⁇ for each set of weights the likelihood of care downgrade within the next $ hours (2, 4, 6, 8, ... , and 24 hours) is computed for each patient during their stays in hospital at any given time ⁇ . Then, &'( ⁇ ⁇ ⁇ ⁇ ⁇ can be obtained from the entire training data sets.
  • step three of checking constraints it is verified that the two quartile constraints are satisfied. If a set of weights meets the constraints and yields an improved &'( ⁇ , it is retained as the best candidate set.
  • the steps one to three may be repeated for each set and through the entire volume of generated weight sets.
  • the formulation optimization procedure stops when all sets of weights have been evaluated.
  • the best-performing set of weights is selected for the final downgrade likelihood ⁇ ⁇ ⁇ ⁇ based on the highest &'( ⁇ ⁇ ⁇ ⁇ ⁇ that satisfies the constraints.
  • the final model formulation is validated.
  • the process 700 proceeds to the sub-act 710e. Otherwise (“NO”), the process 700 returns to the sub-act 710a to find and fix the causes and reformulate the final model.
  • the new data of the first type may include relevant information such as whether certain care units become opened or closed which will affect the workflow through the hospital, or whether a new initiative is introduced at the hospital that affects the workflow of the nurses or doctors and changes where and when data are documented.
  • the sub-act 710e may be configured to be performed periodically, for example, daily, weekly, monthly, or quarterly, to search for new data of the first type. If new data are available (“YES”), the process 700 proceeds to the sub-act 710f to extract, process, and insert the new first-type data to update the dataset. After the dataset is updated, the process 700 returns to the sub-act 710c to retrain the final model with the updated dataset.
  • the process 700 proceeds to a sub-act 712 to execute the final model to output the downgrade likelihood prediction and discretize the prediction into final scores.
  • the output or value of the optimized final downgrade readiness ⁇ ⁇ ⁇ ⁇ is discretized.
  • a discretization process may be implemented to transform the continuous final downgrade readiness (ranging from 0 to 1) into a number of distinct categories (for example, five categories of 1, 2, 3, 4, and 5).
  • This discretization may be achieved by selecting cutoff values at the 20%, 35%, 50%, and 65% percentiles to categorize the values of the optimized final downgrade likelihood ⁇ ⁇ ⁇ ⁇ .
  • category 1 may correspond to a final downgrade likelihood value of 0 ⁇ 20%
  • category 2 may correspond to a final downgrade likelihood value of 20 ⁇ 35%
  • category 3 may correspond to a final downgrade likelihood value of 35 ⁇ 50%
  • category 4 may correspond to a final downgrade likelihood value of 50 ⁇ 65%
  • category 5 may correspond to a final downgrade likelihood value of 65 ⁇ 100%.
  • different cutoff values between 0% and 100% may be used for the discrete categorization, and/or different numbers of categories may be implemented.
  • the resulting discrete downgrade readiness categories provide an intuitive and straightforward representation of a patient’s likelihood of care downgrade at a particular time in the future. This allows for consistent tracking over time and ensures that the downgrade Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) readiness scores over customizable timeframes are both interpretable and actionable within clinical settings.
  • the creation of the downgrade likelihood model may include only static historical data
  • the final scores may change over time (sometimes quickly) as new EMR data of a second type becomes available.
  • the new second-type data may include new information associated with the patient or the care unit where the patient is currently bedded.
  • the downgrade likelihood scores may be calculated and sent to the EMR in as timely a fashion as possible, which requires near-real-time data acquisition.
  • each downgrade likelihood score may be updated in the EMR system display every few minutes (for example, every 8 to 12 minutes), depending on the availability of new data, to reduce the amount of variability in the displayed score over time and enhance the usability of the scores for clinical users.
  • the sub-act 318 is executed to extract, process, and insert the new data to update the training dataset.
  • the process 700 then returns to the sub-act 712 to re- execute the final model. If there is no new second-type data available at the sub-act 318, the process 700 stays at the sub-act 318 to keep searching for new second-type data.
  • the sub-act 710e and the sub-act 318 may search for new data of the same type to be updated but may be performed at different periodicities. For example, the sub- act 318 may be performed more frequently than the sub-act 710e.
  • the process 700 for executing the act 118 to predict the readiness for care downgrade is performed after a patient is placed in either the ICU 116 or IMC 120. In other examples, the process 700 for executing the act 118 may be performed after a patient is placed in any care units of an elevated care level. Referring to FIG.1, after the patient is downgraded to the IMC 120 and then again to an acute care bed 122, the process 100 proceeds to an act 124 in which a machine-learning process is executed to predict the patient’s clinical readiness for future discharge.
  • FIG.7 illustrates a Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) machine-learning process 800 for executing the act 124 to predict the patient’s clinical readiness for discharge at a future time according to an example.
  • the process 800 starts by extracting and compiling patient data of different types in various ways, including patient information and hospital-unit-volume information.
  • the sub-act 802 includes a sub-act 802a and a sub-act 802b.
  • the sub-act 802a may include receiving or extracting patient information including both discrete and non-discrete data available in the EMR about patients for current inpatient encounters en, where n ⁇ ⁇ 1,...,N ⁇ , including information about the patient for the current hospitalization, time- event frequency features, prescription medications, patient’s healthcare insurer, comorbidities, and so forth. Patient information may be received in real-time from an EMR.
  • the sub-act 802b may include extracting non-structured data.
  • discharge related information may be extracted from a variety of types of notes, for example, consult note, hospitalist note, and case management note, for each inpatient encounter en, where n ⁇ ⁇ 1,...,N ⁇ .
  • the sub-act 802 may include compiling or receiving hospital-unit-volume information including data on hospital unit volume v n , where n ⁇ ⁇ 1,...,N ⁇ for each inpatient encounter e n , which measures the number of patients admitted to, into, or discharged from the hospital unit where the patient currently resides within a rolling window of time (for example, past 12 hours).
  • hospital-unit-volume information may be received and/or updated in real-time.
  • Patient data extraction and compilation at sub-act 802 may be performed in various ways.
  • the sub-act 802 may include systematically extracting input data from a variety of sources following standard medical data protocols such as HL7, FHIR, web services, connections to databases, data reporting schemes. These protocols ensure consistent and seamless integration of data from multiple healthcare settings and devices.
  • the sub-act 802 may include automatic data collection. Many data points, such as vital signs (heart rate, blood pressure, and so forth), lab results, and medication administration, are collected automatically from medical devices integrated with EMR systems (for example, bedside monitors, laboratory systems that process bloodwork samples, and drug dispensing systems).
  • the sub-act 802 may include extracting patient data from manual data entry. Some data, such as clinician's assessments including height, weight, blood pressure, care plans including medications, family history, allergies, and patient-reported outcomes including pain score and symptoms, may be manually entered by healthcare professionals. To maintain consistency and accuracy, standard templates and checklists are used across different healthcare providers.
  • the sub-act 802 may include extracting patient data using wearables and remote devices. In certain examples, data from patient-worn devices (for example, glucose monitors, and fitness trackers) or remote monitoring systems can also be integrated into the EMR to provide continuous, real-time data streams.
  • data may be extracted from diverse patient populations across different demographic groups, medical conditions, and care settings. For example, patient demographics such as age, gender, ethnicity, and socioeconomic status are commonly captured during hospital admission processes. Diagnosis codes (ICD-10), clinical procedure codes, and patient treatment plans are also included to capture the diversity of patient conditions and healthcare needs.
  • the data compilation at the sub-act 802 may include integrating information from various healthcare settings, including hospitals, clinics, and specialized care facilities. This comprehensive approach provides a holistic view of patient clinical encounters and outcomes.
  • the data extraction at the sub-act 802 may follow data quality assurance protocols.
  • sub-act 802 In addition to automated checks for missing values and outliers, manual validation may be employed by clinical experts to confirm the accuracy of key variables, especially those that are manually entered or highly complex (for example, diagnostic codes, and treatment plans).
  • data may be encrypted in transit and at rest, and de-identification techniques may be used when sharing or analyzing data, protecting sensitive patient information while allowing for meaningful analytics.
  • the extracted and compiled patient data are preprocessed.
  • the sub-act 804 includes sub-acts 804a, 804b, and 804c.
  • the response variables for ground truth are defined.
  • the compiled dataset is preprocessed partly to establish reliable response variables, which serve as the ground truth labels for each binary model predicting discharge likelihood at specific future time intervals.
  • reliable response variables For example, six binary models may be employed, each predicting the discharge likelihood at 12, 24, 36, 48, 60, or 72 Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) hours in the future, respectively.
  • the primary objective of defining the response variables is to accurately label each instance as either “clinically ready for discharge” or “not ready” within these designated timeframes.
  • actual discharge times recorded in the EMR are used as a direct indicator of discharge readiness. In cases where discharge time is not available or applicable, clinically accepted proxies are utilized to represent a patient’s readiness for discharge.
  • proxies may include specific clinical events or status changes, such as completion of treatment plans or physician-documented readiness for discharge, which are validated in consultation with clinical experts to ensure accuracy and relevance.
  • the models may be trained with labels rooted in real clinical outcomes, enhancing the predictive performance and practical utility of each binary model across all timeframes. This systematic approach provides a strong foundation for developing models that can reliably forecast patient discharge readiness, allowing for more informed clinical decision-making.
  • the extracted patient data are feature-engineered to be ready for the following data transformation. The feature-engineering process is carefully designed to extract maximum value from patient data and enhance the predictive power of machine-learning models. The selection of data features is driven by clinical relevance, ensuring that the most meaningful variables are included.
  • the sub-act 804b includes selecting features of current encounter information including critical aspects of the patient’s current hospital stay. This information includes hospital services provided, the patient's diagnosis, reason for visit, level of care, acuity level, means of arrival, admission type & source, and unit-room-bed location. Additionally, essential clinical measures may be included, such as age, weight, height, BSA, BMI, blood pressure, pulse, and temperature, as well as relevant lab orders and results. To further enrich the data, therapist evaluations and notes are included, such as mobility levels and AM-PAC “6 Clicks” scores, which measure a patient’s ability to perform daily activities. The sub-act 804b may include selecting features of the most recent encounter information.
  • the historical data may include the total length of stays, Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) the level of care provided, the hospital services utilized, and the diagnoses from that encounter.
  • the sub-act 804b may include selecting derived features from time-related dynamics and event frequencies. Derived features are generated to provide insights into time-related dynamics and event frequencies.
  • Event frequency features measure the frequency and cumulative occurrences of specific clinical events. For instance, the recent event counts, which represent the number of lab orders, transfers, or patient updates within a recent timeframe (for example, the past 12 hours) may be tracked. Cumulative event counts may also be calculated to monitor the total occurrences of key events (for example, lab orders and transfers) since admission.
  • the sub-act 804b may include selecting derived features for hospital unit volume.
  • the engineered data are transformed for input to the machine-learning models.
  • Transforming the engineered data may include converting the patient information and/or the hospital-unit-volume information into a machine-interpretable format tailored to a selected machine-learning model (for example, XGBoost).
  • XGBoost machine-learning model
  • a variety of transformation techniques may be applied based on the nature of the selected features. For numerical features, such as clinical measurements, the normalization technique may be used.
  • This technique centers the data by removing the median and scales the data based on the interquartile range, making the technique particularly resistant to outliers which are common in clinical datasets.
  • a patient which might range from very low (for example, 15) to high (for example, 40), is normalized by subtracting the median BMI and scaling it based on the IQR. This makes the data more robust against outliers, such as extreme BMI values for patients with rare conditions.
  • blood pressure readings such as systolic values, typically range between 90 and 180 mmHg but can exhibit extreme values due to acute conditions.
  • blood glucose levels measured in mg/dL
  • glucose values can range through median- centering and IQR scaling, the skew from extreme glucose readings is reduced, ensuring that rare, high glucose levels do not disproportionately impact the model.
  • the length of hospital stay often varies widely, with some patients staying only a few days and others remaining for weeks.
  • boolean features that represent binary outcomes, they are converted into 0 and 1 values to enable seamless integration into machine learning models. This binary format allows the models to interpret the presence or absence of specific patient conditions or statuses effectively.
  • a patient’s current stay in the ICU is recorded as a boolean feature.
  • a patient’s current stay may be transformed into a binary indicator (1 for ICU, 0 for not in ICU) to ensure that the models can understand the importance of ICU status in predicting clinical discharge readiness.
  • the feature of whether a patient is experiencing their first hospitalization is encoded as a boolean variable.1 represents a first-time visit and 0 indicates a repeat visit. This information is useful as first-time admissions may differ in complexity and discharge likelihood compared to subsequent visits.
  • whether a patient requires surgery during their stay is also recorded as a boolean feature, with 1 indicating that surgery is needed and 0 indicating that it is not. This feature helps the model account for patients with surgical needs, who may have longer or more complex stays.
  • For categorical features, one-hot transformation may be employed.
  • This transformation converts categorical variables into one-hot encoded vectors, ensuring that the models can effectively interpret and analyze them. In doing so, rare categories (for example, those occurring fewer than 100 times) are grouped and any unknown categories are treated as infrequent to ensure that even lesser-seen data is accounted for.
  • hospital service type categories like “cardiology,” “oncology,” and “neurology” are each converted into separate binary columns (one for each service). If a patient is in Cardiology, their entry will have a “1” in the cardiology column and a “0” elsewhere.
  • means of arrival categories like “ambulance,” “walk-in,” or “transferred” are transformed similarly.
  • rare categories may be grouped under “other” to avoid sparsity issues in the models. If an unexpected category appears (for example, new means of arrival), the unexpected category may be treated as “other” to prevent the models from breaking due to unknown data.
  • the level of care such as “ICU,” “IMC,” or “nursery” are also one- hot encoded. Each level of care is assigned its own binary column, enabling the models to identify the intensity of care required. Patients in ICU, for example, will have a “1” in the ICU column and a “0” in others. Grouping rare levels under “other” ensures that less frequent levels do not affect model performance.
  • a default numeric value for example, -1
  • numeric values are assigned, such as 2, 1, and 0, respectively. This preserves the ranking while allowing the models to understand the progression of mobility states. If an unknown mobility level is encountered, it is assigned a default numeric value, such as -1, ensuring that the models can still process the data without disruption.
  • the AM-PAC (activity measure for post-acute care) G-code metrics are encoded based on functional ability, reflecting the likelihood of discharge disposition.
  • Lower-functioning codes for example, CN, CM, and CL
  • M2212-7001WO(UMMS23-0003/WO.10) indicate a higher likelihood of subacute rehab
  • higher-functioning codes for example, CJ, CI, and CH
  • This numeric scale preserves the ranking, with CN mapped to 0 and CH to 6, to capture the progressive scale of functional ability.
  • Unknown codes are assigned a default value of -1.
  • the acuity level reflects the urgency of medical attention required, with levels ordered as “less urgent,” “urgent,” “emergent,” and “immediate.” These are assigned numeric values to capture the hierarchy, with 0 for “less urgent,” 1 for “urgent,” 2 for “emergent,” and 3 for “immediate.” This encoding ensures the model accurately interprets increasing acuity. Any unknown acuity level is assigned -1 to maintain data consistency. For textual data, which often include rich but unstructured data like the reason for visits, textual transformation is used. Textual transformation converts the text into a matrix of TF-IDF scores.
  • the transformation captures the most significant terms that best represent the textual data, enabling the models to process and interpret these insights effectively.
  • a text field describing the patient’s reason for visiting the hospital for example, “chest pain” and “shortness of breath”
  • TF-IDF scores By converting the text to lowercase and ignoring common stop words (for example, “the” and “is”), the most relevant terms (for example, the top 20 most relevant terms) are focused on.
  • text fields of clinical notes are transformed into a matrix of TF-IDF scores, describing a patient's health status, treatment plan, and medical history such as quality improvement measures on DVT prophylaxis, urinary catheter existence, ambulation status, and nutrition.
  • text fields of operative notes that describe a patient’s surgery such as type of surgery, estimated blood loss, type of anesthesia, and urine output are transformed into a matrix of TF-IDF scores.
  • a machine-learning algorithm is selected from multiple types of algorithms.
  • one or more of the algorithm types may include a binary model by default and may sometimes be specified to include a multi-class model.
  • the sub-act 806 may include sub-acts 806a and 806b.
  • the sub-act 806a includes training a plurality of binary models to predict clinical discharge likelihoods using algorithms of Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) different types.
  • the binary models may include six individual binary models, each binary model predicting whether a patient will be ready for discharge from the hospital at a respective future time (for example, in the next 12, 24, 36, 48, 60, or 72 hours from now).
  • This multi-model approach implemented in the process 800 allows for greater granularity, as different clinical factors become more or less relevant at varying stages of patient recovery. For example, a patient ready for discharge in the next 12 hours may present different clinical indicators than one ready in the next 72 hours, making specialized models essential for capturing these nuances. By training distinct models for each time frame, each model may focus on the specific clinical signals relevant to its discharge window. This specialization allows for more precise predictions, as each model can be fine-tuned to prioritize key indicators such as vital signs, lab results, or medication effectiveness that are most significant at a particular stage in the patient's recovery. In contrast, attempting to use a single model for all time intervals may introduce unnecessary complexity and potentially dilute the accuracy of predictions across different time frames.
  • each binary model corresponding to one of the time intervals is executed and/or trained using different types of algorithms with default parameter settings, including Random Forest, XGBoost, SVM, and Neural Networks.
  • the models are trained based on the data with features obtained from sub-act 804.
  • a 5-fold cross-validation is employed using a dataset where the ground truth of whether the actual discharge occurred within the specified time frames is known.
  • Sub-act 806a may include executing the XGBoost algorithm to generate a plurality of binary models that predict the likelihood of discharging the patient within each of a plurality of timeframes, such as within 12 hours, within 24 hours, and so forth.
  • a comprehensive comparison based on a set of metrics is performed for various machine-learning algorithms to predict the likelihood of patient discharge within the next 12, 24, 36, 48, 60, and 72 hours.
  • the binary models are evaluated and/or optimized based on key metrics including accuracy, precision, recall, F1-score, and the ROC-AUC.
  • Accuracy measures overall prediction correctness. Precision focuses on minimizing false positives by ensuring that patients predicted as dischargeable were indeed ready for discharge. Recall prioritizes the capture of all truly dischargeable patients, minimizing false negatives.
  • ROC-AUC measures a model’s ability to distinguish between two classes (discharged or not).
  • XGBoost may be preferred for handling complex and imbalanced clinical datasets.
  • XGBoost may be able to prevent overfitting through regularization.
  • XGBoost may be selected at sub-act 806b as the optimal algorithm for each of the binary models, ensuring reliable predictions for patient discharge readiness across the 12, 24, 36, 48, 60, and 72-hour intervals.
  • one or more other machine-learning algorithms may be chosen alone or in various combinations at the sub-act 806b.
  • a Bayesian optimization procedure may be implemented to fine-tune the hyperparameters of each binary model.
  • the hyperparameters defined below are optimized to improve the performance of the XGBoost model.
  • Maximum tree depth is the maximum depth of the trees which controls the model complexity. In various examples, maximum tree depth may be set to a range between, for example, 3 and 10.
  • Learning rate determines the step size during model Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) training, sampled from a logarithmic scale between a range of values (for example, 0.001 and 0.3) to allow for both small and large step sizes.
  • Minimum child weight specifies the minimum sum of instance weights required for a child node to split, helping control overfitting. In at least one example, minimum child weight may be set within a range of between 1 and 6.
  • Gamma defines the minimum loss reduction required for partitioning a leaf node and controls regularization. In at least one example, the gamma value may be sampled between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.05).
  • Subsample ratio represents the fraction of training samples used for building each tree, uniformly sampled between a range of values (for example, a range of 0.5 and 1) with a specified step size (for example, a step size of 0.05).
  • Column sample by tree specifies the fraction of features to be randomly sampled for each tree, sampled uniformly between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.1).
  • Scale positive class weight is a parameter particularly beneficial for handling imbalanced classes in a binary setup. Scale positive class weight adjusts the balance of positive and negative weights to handle imbalanced data, sampled uniformly between a range of values (for example, 1 and 9) to give more weight to the positive class.
  • the evaluation metric for fine-tuning the hyperparameters is selected as the AUC-PR, which is more suitable for handling imbalanced classes.
  • the goal of the sub-act 808 is to identify the optimal hyperparameter set ⁇ ⁇ that maximizes the AUC-PR score, expressed as an objective function ⁇ .
  • ⁇ represent the AUC-PR score of the XGBoost model with hyperparameters ⁇ .
  • TPE For surrogate model fitting, a TPE is used as the surrogate model to approximate the objective function ⁇ . TPE models the likelihood of good and bad hyperparameter configurations based on previous observations. Let ⁇ be the likelihood of high-performing hyperparameters (those improving AUC-PR) and ⁇ be the likelihood of low-performing Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) hyperparameters (those reducing AUC-PR). These distributions are updated based on observed evaluations, and the surrogate model is used to predict the performance of new hyperparameter configurations. In various examples, the surrogate model is iteratively refined as more real-time data is collected.
  • an acquisition function is used to guide the selection of the next hyperparameters for evaluation.
  • This ratio directs the optimization process, balancing exploration (trying hyperparameters where the model is uncertain) and exploitation (choosing hyperparameters that are likely to perform well based on the surrogate model).
  • the Bayesian optimization procedure starts at step one of initialization, in which a few initial random hyperparameter configurations are evaluated to gather baseline data. These configurations are chosen randomly within the specified hyperparameter space ⁇ and provide the initial observations necessary to build a surrogate model.
  • the surrogate model is fitted based on the observed data, constructing the ⁇ ⁇ ⁇ ⁇ and ⁇ ⁇ ⁇ ⁇ distributions.
  • the surrogate model serves as a probabilistic estimator that guides the search for optimal hyperparameters, predicting how different configurations will perform without needing to evaluate each one directly.
  • the next hyperparameter configuration is selected by maximizing the acquisition function, which balances exploration (testing hyperparameters where uncertainty is high) and exploitation (testing configurations likely to yield high performance based on the surrogate model). This ensures that the search efficiently converges towards optimal hyperparameters, even in complex search spaces.
  • the model is fit using XGBoost with a maximum number of boosting rounds per trial (for example, 500 boosting rounds per trial). Additionally, if there is no improvement in model performance after a certain number of boosting rounds (for example, 100 boosting rounds), early stopping is triggered to prevent further unnecessary iterations. Performance is evaluated using the AUC-PR Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) score, which is particularly relevant for an imbalanced healthcare dataset, focusing on precision and recall. At step five of iteration, the optimization continues by repeating steps two to four.
  • the total number of trials may be limited to a maximum (for example, 50) to ensure that the search process remains efficient without overfitting the hyperparameter space. If, within these trials, the model reaches a plateau where further boosting rounds yield no significant improvement in AUC-PR scores, the optimization is stopped and the hyperparameter fine-tuning is completed.
  • a formulation of a final clinical discharge likelihood at a specific future time is constructed, optimized, validated, and updated.
  • a formulation is constructed that combines the weighted predictions from each model trained for different time intervals (12, 24, 36, 48, 60, and 72 hours). The primary objective is to optimize the weights of these binary model predictions to generate an accurate and calibrated discharge likelihood for each patient.
  • the sub-act 810 includes sub-acts 810a, 810b, 810c, 810d, 810e, and 810f. At sub-act 810a, in one example, the formulation of the final clinical discharge likelihood in the next 24 hours is constructed.
  • Constructing the final clinical discharge likelihood may include combining the plurality of binary models into a single combined discharge model which may, in turn, include optimizing weightings for each discharge prediction of each binary model.
  • ⁇ refers to the observation timestamp during the patient’s stay in the hospital.
  • ⁇ predicts the final likelihood of discharge within the next $ hours from that specific time ⁇ and may be referred to as a weighted discharge readiness quantization for the patient.
  • the primary goal of the formulation optimization of the final discharge likelihood ⁇ ⁇ ⁇ ⁇ is to maximize the median, &'( ⁇ ⁇ ⁇ ⁇ ⁇ across the training dataset.
  • All ⁇ ⁇ ⁇ ⁇ ⁇ values computed from various patients and at various times ⁇ in the training data are gathered and sorted into a distribution.
  • the median &'( ⁇ is the middle value in the sorted distribution, representing the typical likelihood across the population of all ⁇ values.
  • two key restrictions are imposed on the maximization of &'() ⁇ * to optimize ⁇ .
  • the distribution of the final discharge likelihood ⁇ for patients likely to be discharged may be maximized, particularly focusing on predictions updated at a specific time, for example, at 6 AM the day before discharge, to increase the accuracy of identifying patients ready for discharge the following day.
  • the optimized final likelihood is maintained sufficiently high when predicting discharge one day before the actual discharge happens.
  • the distribution of the optimized final discharge likelihood ⁇ may be minimized for patients who are predicted to remain in the hospital, reducing the risk of mistakenly categorizing patients as ready for discharge when they should remain under care.
  • 100,000 sets may be generated to explore a broad range of potential combinations.
  • step two of evaluating &'( ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , for each set of weights the likelihood of discharge within the next $ hours (12, 24, 36, 48, 60, and 72 hours) is computed for each patient during their stays in hospital at any given time ⁇ . Then, &'( ⁇ ⁇ ⁇ ⁇ ⁇ can be obtained from the entire training data sets.
  • step three of checking constraints it is verified that the two quartile constraints are satisfied. If a set of weights meets the constraints and yields an improved &'( ⁇ , it is retained as the best candidate set. The steps one to three are repeated for each set and through the entire volume of generated weight sets.
  • the formulation optimization procedure stops when all sets of weights have been evaluated.
  • the best-performing set of weights is selected for the final discharge likelihood ⁇ ⁇ ⁇ ⁇ based on the highest &'( ⁇ ⁇ ⁇ ⁇ ⁇ that satisfies the constraints.
  • the final model formulation is validated. If the behavior of the final model formulation is consistent with those obtained at the sub-act 810a or sub-act 810c (“YES”), Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) the process 800 proceeds to the sub-act 810e. Otherwise (“NO”), the process 800 returns to the sub-act 810a to find and fix the causes and reformulate the final model.
  • the new data of the first type may include relevant information such as whether certain care units become opened or closed which will affect the workflow through the hospital, or whether a new initiative is introduced at the hospital that affects the workflow of the nurses or doctors and changes where and when data are documented.
  • the sub-act 810e may be configured to be performed periodically, for example, daily, weekly, monthly, or quarterly, to search for new data of the first type. If new data are available (“YES”), the process 800 proceeds to the sub-act 810f to extract, process, and insert the new first-type data to update the dataset.
  • the process 800 After the dataset is updated, the process 800 returns to the sub-act 810c to retrain the final model with the updated dataset. If no new first-type data are available (“NO”) at the sub-act 810e, the process 800 proceeds to a sub-act 812 to execute the final model to output the discharge likelihood prediction and discretize the prediction into final scores.
  • the score or value of the optimized final discharge likelihood ⁇ ⁇ ⁇ ⁇ is discretized.
  • a discretization process may be implemented to transform the continuous final discharge likelihood (ranging from 0 to 1) into a number of distinct categories (for example, five categories of 1, 2, 3, 4, and 5).
  • This discretization may be achieved by selecting cutoff values at the 20%, 35%, 50%, and 65% percentiles to categorize the values of the optimized final discharge likelihood ⁇ ⁇ ⁇ ⁇ .
  • category 1 may correspond to a final discharge likelihood value of 0 ⁇ 20%
  • category 2 may correspond to a final discharge likelihood value of 20 ⁇ 35%
  • category 3 may correspond to a final discharge likelihood value of 35 ⁇ 50%
  • category 4 may correspond to a final discharge likelihood value of 50 ⁇ 65%
  • category 5 may correspond to a final discharge likelihood value of 65 ⁇ 100%.
  • sub- act 712 may include transforming the weighted discharge readiness quantization ⁇ into a discharge prediction, that is, a value (for example, 1, 2, 3, 4, or 5) on a standardized scale (for example, 1-5).
  • a discharge prediction for example, a value (for example, 1, 2, 3, 4, or 5) on a standardized scale (for example, 1-5).
  • different cutoff values between 0% and 100% may be used for the discrete categorization, and/or a different number of discrete categories may be implemented.
  • the resulting discrete discharge likelihood categories provide an intuitive and straightforward representation of a patient’s likelihood of discharge at a particular time in the future. This allows for consistent tracking over time and ensures that the discharge scores are both interpretable and actionable within clinical settings.
  • the final scores may change over time (sometimes quickly) as new EMR data of a second type becomes available.
  • the new second-type data may include new information associated with the patient or the care unit where the patient is currently bedded.
  • the discharge likelihood scores may be calculated and sent to the EMR in as timely a fashion as possible, which requires near-real-time data acquisition.
  • Industry-standard mechanisms such as HL7 and FHIR, are the primary near-real-time acquisition methods.
  • Several other data acquisition methods are also used, such as data analytics reporting tools, API web services, and KB_SQL queries. This allows an automatic near-real-time two-way data pipeline.
  • the process 800 proceeds through sub-act 318 depending on whether new second-type data becomes available.
  • each discharge likelihood score may be updated in the EMR system display every few minutes (for example, every 8 to 12 minutes), depending on the availability of new data, to reduce the amount of variability in the displayed score over time and enhance the usability of the scores for clinical users.
  • the sub-act 318 is executed to extract, process, and insert the new data to update the training dataset.
  • the process 800 then returns to the sub-act 812 to re- execute the final model.
  • the process 800 stays at the sub-act 318 to keep searching for new second-type data.
  • the sub-act 810e and the sub-act 318 may search for new data of the same type to be updated but may be performed at different periodicities.
  • the sub- act 318 may be performed more frequently than the sub-act 810e.
  • the machine-learning process 800 for executing the act 124 to predict the patient’s clinical readiness for discharge at a future time may be performed via the user interface 204.
  • the user interface 204 may display a GUI associated with the act 124.
  • FIG.8 illustrates a GUI 900 for predicting clinical readiness for discharging patients in a hospital in the next 24 hours according to an example.
  • a first column 902 displays the discretized final discharge likelihood Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) scores of patients currently in the hospital in a transparent and straightforward manner to assist in the decision-making of whether any of the patients are ready to or should be discharged in the next 24 hours.
  • a second column 904 displays the unit and bed numbers of the patients to identify those patients. For example, the patient in bed 129-A of unit 1W has a discharge likelihood score of 4 to be discharged from the hospital in the next 24 hours and the patient in bed 131-A of unit 1W has a discharge likelihood score of only 2. In other examples, discharge-likelihood scores may span a range other than 1 to 5.
  • the process 100 proceeds to an act 126 to predict the operational readiness of the hospital for discharging the patient.
  • the act 126 may only be executed for the patients having a high enough discharge likelihood score.
  • Discharge planning may be relevant to patient flow since delays in discharge can lead to hospital-wide congestion. After patients are identified as clinically ready for discharge, there still could be operational delays before the patients physically leave the hospital unless a set of conditions or criteria is met to safely discharge the patients.
  • FIG.9 illustrates a machine-learning process 1000 for executing the act 126 to predict the operational readiness of the hospital for discharging a patient according to an example.
  • the process 1000 starts by extracting and compiling patient data of different types in various ways.
  • the sub-act 1002 includes a sub-act 1002a and a sub-act 1002b.
  • the sub-act 1002a may include extracting structured data from EMR.
  • EMR electronic medical record
  • a wide array of structured patient data from EMR systems are systematically extracted, including medical condition and recovery status, laboratory results, and treatments.
  • Input data are extracted from a variety of sources following standard medical data protocols such as HL7, FHIR, web services, connections to databases, and data acquisition schemes. These protocols ensure consistent and seamless compilation of data from multiple healthcare settings and devices.
  • the sub-act 1002a may include automatic data collection.
  • the sub-act 1002a may include manual data entries. Some data such as acuity of illness, functional assessment, physician and nurse assessments, and patient feedback and self- assessments such as physical mobility, understanding of their care instructions, SDOH barriers, and confidence in managing post-hospital care are manually entered by healthcare professionals. To maintain consistency and accuracy, standard templates and checklists are used across different healthcare providers.
  • the sub-act 1002a may employ wearables and remote devices.
  • data from patient-worn devices can also be integrated into the EMR to provide continuous, real-time data streams.
  • data may be extracted from diverse patient populations across different demographic groups, medical conditions, and care settings. For example, patient demographics such as age, gender, ethnicity, and socioeconomic status are commonly captured during hospital admission processes. Diagnosis codes (ICD-10), clinical procedure codes, and patient treatment plans are also included to capture the diversity of patient conditions and healthcare needs.
  • the data compilation at the sub-act 1002a may include integrating information from various healthcare settings, including hospitals, clinics, and specialized care facilities. This comprehensive approach provides a holistic view of patient clinical encounters and outcomes.
  • the data extraction at the sub-act 1002a may follow data quality assurance protocols. Ensuring high-quality data is paramount. In addition to automated checks for missing values and outliers, manual validation may be employed by clinical experts to confirm the accuracy of key variables, especially those that are manually entered or highly complex (for example, diagnostic codes, and treatment plans). Throughout the data extraction and compilation process at the sub-act 1002a, data may be encrypted in transit and at rest, and de-identification techniques may be used when sharing or analyzing data, protecting sensitive patient information while allowing for meaningful analytics. Act sub-act 1002b, in one example, non-structured data, such as free text in clinical notes, are extracted.
  • NLP natural language processing
  • Medication management data are used to confirm that all medications are reconciled, and the patient is stable on their medication regimen.
  • Rehabilitation status data are used to confirm that the patient can safely manage their activities of daily living.
  • Social and home support data are used to confirm that the family members or caregivers are ready to provide support or home care has been arranged.
  • Insurance authorization and payment data are used to ensure that insurance or other payment mechanisms are in place, which can sometimes delay discharge.
  • Post-discharge arrangement data are used to confirm that home care, transportation, or the availability of rehabilitation facilities, as these are logistical factors that need to be addressed before discharge can happen.
  • the non-structured data may include consult notes, hospitalist notes, and case management notes.
  • Consult notes are medical documents created by a consulting healthcare provider, usually a specialist, who is asked to evaluate a patient by the referring provider.
  • a consult note may include the specialist's assessment, findings, and recommendations for further management or treatment. From this record of the consultation, insights and guidance may be provided for continuing care, such as suggested next-step treatments, lab tests or imaging, and follow-ups.
  • Hospitalist notes are typically part of the patient's daily medical record and include updates on the patient’s condition, treatment plan, and any changes in their status during the hospital stay.
  • the extracted data may provide assessment and diagnosis of the patient’s current condition, outline of medications, therapies, tests, or procedures planned for the patient, as well as any discharge considerations.
  • the NLP techniques may include text removal and combination, sentence segmentation, lowercasing and stope word removal, spelling correction and abbreviation expansion, handling Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) temporal information, and discharge keywords.
  • Text removal and combination includes removing duplicate notes and combining long notes separated into different database entries for continuation and completeness.
  • Sentence segmentation includes splitting text into sentences to make analysis easier.
  • Lowercasing and stop word removal includes converting text to lowercase and removing irrelevant words.
  • Spelling correction and abbreviation expansion includes correcting typos and expanding medical abbreviations using a medical lexicon.
  • Handling temporal information includes creating a timeline for each patient for key events (for example, treatment date and recovery status), and then aligning and standardizing dates to enable extraction of trends over time, particularly for lab results and medication adherence tracking.
  • Discharge keywords include detecting and extracting information specially designed to detect patient discharge hurdles, such as “safe discharge plan,” “physical therapy and/or occupational therapy discharge recommendations,” and “transportation needs at discharge.”
  • the extracted and compiled patient data are preprocessed.
  • the sub-act 1004 includes sub-acts 1004a, 1004b, and 1004c.
  • the response variables for ground truth are defined.
  • the compiled dataset is preprocessed to establish reliable response variables, which serve as the ground truth labels for a binary model to predict operational discharge readiness within, for example, the next 24 hours.
  • This variable effectively, the dataset undergoes a thorough preprocessing phase that leverages data elements within the EMR system. This preprocessing focuses on identifying both direct discharge indicators (such as actual discharge timestamps) and proxy indicators that reflect operational readiness when actual discharge times may not be available. The most straightforward proxy for operational readiness is the actual discharge timestamp.
  • Proxy indicators may include clinical notes and medical orders, disposition documentation, consult and procedure completion times, and nursing and case management notes.
  • Clinical notes and medical orders include clinician notes that indicate a plan to discharge or notations within medical orders, Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) signaling the discontinuation of key treatments or medications and providing insights into when the patient was operationally ready for discharge.
  • Disposition documentation includes entries within the EMR where discharge dispositions (for example, home and rehabilitation facility) have been established or updated, which may imply operational readiness even before the actual discharge.
  • Consult and procedure completion times include the timing of key consults, procedures, or diagnostics relevant to discharge clearance, suggesting that all necessary steps for discharge are complete.
  • Nursing and case management notes include observations or documentation from nursing or case management staff indicating readiness to transition the patient out of the current care setting. Notes such as “awaiting transport” or “discharge pending” are operational indicators that can help define readiness in the absence of exact timestamps. Proxy indicators are refined through close collaboration with physicians, discharge planners, and other domain experts to ensure their clinical relevance and operational alignment.
  • proxies and direct indicators are selected, they are validated against historical cases to assess how closely they align with actual operational readiness. This validation step helps to ensure that the chosen proxies, such as clinical notes and disposition entries, consistently reflect true discharge readiness and allow the model to generalize effectively.
  • the extracted patient data are feature-engineered to be ready for the following data transformation. The feature-engineering process is carefully designed to extract maximum value from the compiled data and enhance the predictive power of machine-learning models. The selection of data features is driven by clinical relevance, ensuring that the most meaningful variables are included.
  • the sub-act 1004b includes selecting features of relevant aspects of the patient’s current hospital stay. These features may include, for example, hospital services provided, the patient's diagnosis, reason for visit, level of care, acuity level, means of arrival, admission type and source, unit-room-bed location, and so forth. Additionally, relevant orders may be included, such as imaging, lab, procedure, therapy and diet, and the corresponding results. To further enrich the data, therapist evaluations and notes may be included, such as Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) mobility levels and AM-PAC “6 Clicks” scores, which measure a patient’s ability to perform daily activities.
  • the sub-act 1004b may include selecting derived features from time-related dynamics and event frequencies. Derived features are generated to provide insights into time-related dynamics and event frequencies. These time-related dynamics track key aspects of a patient's healthcare clinical events, such as the number of days between the current and previous visits, the total hours spent in the hospital, and the time since the last mobility evaluation. Moreover, day-of-week effects are accounted for to capture potential variations in care based on the day the visit occurred. Event frequency features measure the frequency and cumulative occurrences of specific clinical events. For instance, the recent event counts, which represent the number of lab orders, transfers, or patient updates within a recent timeframe (for example, the past 12 hours) may be tracked.
  • Cumulative event counts may also be calculated to monitor the total occurrences of key events (for example, lab orders and transfers) since admission.
  • the sub-act 1004b may include selecting derived features from notes. These derived features are considered to identify discharge barriers, including recovery status (for example, medically ready for discharge), pending laboratory results, trends of vital signs, surgical or procedural recovery milestones, functional assessment from physical therapy and occupational therapy, and physician and nurse assessments.
  • the engineered data are transformed for input to the machine-learning models. A variety of transformation techniques may be applied based on the nature of the selected features. For numerical features, such as clinical measurements, the normalization technique may be used.
  • This technique centers the data by removing the median and scales the data based on the interquartile range, making the technique particularly resistant to outliers which are common in clinical datasets.
  • numerical features may include the number of pending items, such as the total number of imaging, lab, procedure, therapy, and diet orders that are not completed. This number directly indicates operational readiness for discharge.
  • the length of hospital stay often varies widely, with some patients staying only a few days and others remaining for weeks. Normalizing the length of stay by removing the median and scaling by the IQR ensures that the model can handle this variation Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) effectively, reducing the impact of exceptionally long stays that might otherwise distort the feature distribution.
  • Boolean features that represent binary outcomes are converted into 0 and 1 values to enable seamless integration into machine learning models. This binary format allows the models to interpret the presence or absence of specific patient conditions or statuses effectively.
  • whether a patient is medically ready for discharge is selected as a feature encoded as a boolean variable.1 represents medically ready and 0 indicates otherwise.
  • a patient’s current stay in the ICU is recorded as a boolean feature. The current stay information may be transformed into a binary indicator (1 for ICU, 0 for not in ICU) to ensure that the models can understand the importance of ICU status in predicting clinical discharge readiness.
  • boolean feature In another example, whether a patient requires surgery during their stay is also recorded as a boolean feature, with 1 indicating that surgery is needed and 0 indicating that it is not. This feature helps the model account for patients with surgical needs, who may have longer or more complex stays.
  • one-hot transformation may be employed. This transformation converts categorical variables into one-hot encoded vectors, ensuring that the models can effectively interpret and analyze them. In doing so, rare categories (for example, those occurring fewer than 100 times) are grouped and any unknown categories are treated as infrequent to ensure that even lesser-seen data is accounted for.
  • hospital service type categories like “cardiology,” “oncology,” and “neurology” are each converted into separate binary columns (one for each service).
  • a patient is in Cardiology, their entry will have a “1” in the cardiology column and a “0” elsewhere.
  • means of arrival categories like “ambulance,” “walk-in,” or “transferred” are transformed similarly. To handle rare categories (for example, those occurring fewer than 100 times), rare categories are grouped under “other” to avoid sparsity issues in the models. If an unexpected category appears (for example, new means of arrival), the unexpected category is treated as “other” to prevent the models from breaking due to unknown data.
  • the level of care such as “ICU,” “IMC,” or “nursery” are also one- hot encoded. Each level of care is assigned its own binary column, enabling the models to identify the intensity of care required.
  • Patients in ICU will have a “1” in the ICU Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) column and a “0” in others. Grouping rare levels under “other” ensures that less frequent levels do not affect model performance. For ordinal features, the inherent ranking of these categories is preserved by converting them into numeric values based on a predefined order. If the model encounters an unknown category, a default numeric value (for example, -1) is assigned to allow the transformation process to handle unexpected inputs without disrupting the model performance.
  • a default numeric value for example, -1
  • the readiness for hospital discharge scale may be administered close to the discharge on a range of values (for example, on a scale from 0 to 10), with higher scores indicating greater readiness.
  • scales such as Barthel index, which ranges from 0 to 100, and Katz index, which ranges from 0 to 6, may be employed to access a patient’s ability to perform activities of daily living (ADLs) and evaluate the patient’s physical readiness for discharge.
  • AM-PAC activity measure for post-acute care
  • G-code metrics are encoded based on functional ability, reflecting the likelihood of discharge disposition.
  • Lower-functioning codes for example, CN, CM, and CL
  • higher-functioning codes for example, CJ, CI, and CH
  • This numeric scale preserves the ranking, with CN mapped to 0 and CH to 6, to capture the progressive scale of functional ability.
  • the acuity level reflects the urgency of medical attention required, with levels ordered as “less urgent,” “urgent,” “emergent,” and “immediate.” These are assigned numeric values to capture the hierarchy, with 0 for “less urgent,” 1 for “urgent,” 2 for “emergent,” and 3 for “immediate.” This encoding ensures the model accurately interprets increasing acuity. Any unknown acuity level is assigned -1 to maintain data consistency. For textual data, which often include rich but unstructured data like the reason for visits, textual transformation is used. Textual transformation converts the text into a matrix of TF-IDF scores.
  • a machine-learning algorithm is selected from multiple algorithms.
  • the sub-act 1006 may include sub-acts 1006a and 1006b.
  • the sub- act 1006a includes training binary classification models to predict operational readiness for discharge using algorithms of different types. For example, a binary model (predicting high or low) of each type may be employed to predict whether a patient is operationally ready to be discharged within the next 24 hours.
  • binary models of different types are trained using different types of algorithms with default parameter settings, including Random Forest, XGBoost, SVM, and Neural Networks. The models are trained based on the data with features obtained from sub-act 1004.
  • a ten- fold cross-validation is employed using a dataset where the ground truth of whether the actual discharge occurred within 24 hours is known.
  • the dataset is randomly split into ten parts, with each model trained on nine folds and tested on the remaining fold. This may be repeated ten times to ensure that each part of the dataset is used as a test set exactly once, providing a robust assessment of model performance. The random split was repeated fifty times to further ensure a more stable and reliable estimate of model performance.
  • the binary models of different types are evaluated based on key metrics including accuracy, precision, recall, and F1-score.
  • XGBoost may be selected at sub-act 1006b as the optimal algorithm for the binary model, ensuring reliable predictions for operational readiness for discharge in the next 24 hours.
  • one or more other machine-learning algorithms may be chosen alone or in various combinations at the sub-act 1006b.
  • the hyperparameters of the selected type of machine- learning algorithm may be optimized through training and periodically updated.
  • the sub-act 1008 may include sub-acts 1008a, 1008, and 1008c.
  • a Bayesian optimization procedure may be implemented to fine-tune the hyperparameters of the selected binary model.
  • the hyperparameters defined below are optimized to improve the performance of the XGBoost model.
  • Maximum tree depth is the maximum depth of the trees which controls the model complexity. In various examples, maximum tree depth may be set to a range between, for example, 3 and 10. Learning rate determines the step size during model training, sampled from a logarithmic scale between a range of values (for example, 0.001 and 0.3) to allow for both small and large step sizes.
  • Minimum child weight specifies the minimum sum of instance weights required for a child node to split, helping control overfitting. In at least one example, minimum child weight may be set within a range of between 1 and 6.
  • Gamma defines the minimum loss reduction required for partitioning a leaf node and controls Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) regularization. In at least one example, the gamma value may be sampled between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.05).
  • Subsample ratio represents the fraction of training samples used for building each tree, uniformly sampled between a range of values (for example, a range of 0.5 and 1) with a specified step size (for example, a step size of 0.05).
  • Column sample by tree specifies the fraction of features to be randomly sampled for each tree, sampled uniformly between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.1).
  • Scale positive class weight is a parameter particularly beneficial for handling imbalanced classes in a binary setup. Scale positive class weight adjusts the balance of positive and negative weights to handle imbalanced data, sampled uniformly between a range of values (for example, 1 and 9) to give more weight to the positive class.
  • the evaluation metric for fine-tuning the hyperparameters is selected as the AUC-PR, which is more suitable for handling imbalanced classes. In other examples, other metrics may be added or may replace AUC-PR.
  • the goal of the sub-act 1008a is to identify the optimal hyperparameter set ⁇ ⁇ that maximizes the AUC-PR score, expressed as an objective function ⁇ ⁇ ⁇ ⁇ . Let ⁇ ⁇ ⁇ ⁇ represent the AUC-PR score of the XGBoost model with hyperparameters ⁇ .
  • ⁇ ⁇ arg m ⁇ a ⁇ x ⁇ ⁇ , where ⁇ represents the hyperparameters to be optimized (for example, learning rate, maximum depth, and regularization terms), and ⁇ is the space of possible hyperparameters.
  • ⁇ represents the hyperparameters to be optimized (for example, learning rate, maximum depth, and regularization terms)
  • ⁇ is the space of possible hyperparameters.
  • TPE models the likelihood of good and bad hyperparameter configurations based on previous observations. Let ⁇ ⁇ ⁇ ⁇ be the likelihood of high-performing hyperparameters (those improving AUC-PR) and ⁇ ⁇ ⁇ ⁇ be the likelihood of low-performing hyperparameters (those reducing AUC-PR).
  • the Bayesian optimization procedure may start at step one of initialization, in which a few initial random hyperparameter configurations are evaluated to gather baseline data. These configurations are chosen randomly within the specified hyperparameter space ⁇ and provide the initial observations necessary to build a surrogate model.
  • the surrogate model is fitted based on the observed data, constructing the ⁇ and ⁇ distributions.
  • the surrogate model serves as a probabilistic estimator that guides the search for optimal hyperparameters, predicting how different configurations will perform without needing to evaluate each one directly.
  • the next hyperparameter configuration is selected by maximizing the acquisition function, which balances exploration (testing hyperparameters where uncertainty is high) and exploitation (testing configurations likely to yield high performance based on the surrogate model). This ensures that the search efficiently converges towards optimal hyperparameters, even in complex search spaces.
  • the model is fit using XGBoost with a maximum number of boosting rounds per trial (for example, 500 boosting rounds per trial). Additionally, if there is no improvement in model performance after a certain number of boosting rounds (for example, 100 boosting rounds), early stopping is triggered to prevent further unnecessary iterations.
  • Performance is evaluated using the AUC-PR score, which is particularly relevant for our imbalanced healthcare dataset, focusing on precision and recall.
  • the optimization continues by repeating steps two to four.
  • the total number of trials may be limited to a maximum number (for example, 50) to ensure that the search process remains efficient without overfitting the hyperparameter space.
  • Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) If, within these trials, the model reaches a plateau where further boosting rounds yield no significant improvement in AUC-PR scores, the optimization is stopped and the hyperparameter fine-tuning is completed.
  • the new data of the first type may include relevant information such as whether certain care units become opened or closed which will affect the workflow through the hospital, or whether a new initiative is introduced at the hospital that affects the workflow of the nurses or doctors and changes where and when data are documented.
  • the sub-act 1008b may be configured to be performed periodically, for example, daily, weekly, monthly, or quarterly, to search for new data of the first type. If new data are available (“YES”), the process 1000 proceeds to the sub-act 1008c to extract, process, and insert the new first-type data to update the dataset.
  • the process 1000 After the dataset is updated, the process 1000 returns to the sub-act 1008a to retrain the final model with the updated dataset. If no new first-type data are available (“NO”) at the sub-act 1008b, the process 1000 proceeds to a sub-act 1010 to execute the final model to output the operational readiness prediction.
  • the determination of an appropriate probability cutoff is critical for classifying whether a patient is operationally ready to be discharged within the next 24 hours. This cutoff ensures that the final model’s predictions are both clinically actionable and aligned with operational priorities.
  • the receiver operating characteristic (ROC) curve of the final model may first be analyzed.
  • the ROC curve plots sensitivity (true positive rate) against specificity (false positive rate) for various probability cutoffs.
  • the ROC curve visualizes the trade-off between the final model’s ability to correctly identify patients operationally ready for discharge (sensitivity) and its ability to correctly exclude patients not operationally ready for discharge (specificity).
  • Youden’s index (J) of the ROC curve may be calculated to identify the optimal cutoff that best balances sensitivity and specificity.
  • the cutoff that maximizes Youden’s Index (J) may be selected as the optimal cutoff, as it ensures the best balance between the two metrics.
  • the clinical and operational circumstances of the patient may be incorporated into the determination of the cutoff.
  • the threshold may be adjusted upward to favor specificity.
  • clinical stakeholders and/or operational teams may be engaged to validate and refine the cutoff based on practical circumstances.
  • the selected cutoff may be validated and then implemented. For example, the selected cutoff may be tested on a hold-out dataset to ensure robust performance.
  • ROC curve is analyzed to reveal that a cutoff of 0.65 maximizes Youden’s index (J). This cutoff implies that patients with a predicted operational readiness probability above 0.65 are classified as having a “High” likelihood of operational readiness for discharge while those below 0.65 are classified as having a “Low” likelihood. Adjustments may also be made to ensure alignment with clinical safety protocols.
  • An example of the prediction output of the operational readiness for discharge in the next 24 hours and an associated list of pending items remaining to be completed prior to discharge is shown in Table 1.
  • the sub-act 318 is executed to extract, process, and insert the new data to update the training dataset. If new second-type data are available, the sub-act 318 is performed to extract, process, and insert the new data to update the input dataset. Then the process 1000 returns to the sub-act 1010 to re- execute the final model. If no new second-type data are available at sub-act 1010, the process 1000 stays at the sub-act 318 to keep searching for new second-type data.
  • the sub-act 1008c and the sub-act 318 may search for new data of the same type to be updated but may be performed at different periodicities. For example, the sub- act 318 may be performed more frequently than the sub-act 1008c.
  • the process 100 proceeds to an act 128 to predict the discharge disposition of the patient. Discharge disposition may refer to the destination to which a patient may be discharged, such as a long-term care facility, skilled nursing facility, rehabilitation facility, home, and others.
  • the act 128 may only be executed for the patients having a high enough discharge likelihood score and/or a high operational readiness.
  • the act 128 leverages the EMR data and overcomes the issue of high cardinality when predicting many classes.
  • a customizable re-grouping module is employed to consolidate a number of discharge dispositions (for example, over 100 nuanced discharge dispositions) into major groups, reducing complexity and improving prediction accuracy.
  • FIG.10 illustrates a machine-learning process 1100 to execute the act 128 to predict the discharge disposition of a patient at a future time according to an example.
  • the process 1100 starts by extracting and compiling patient data of different types in various ways.
  • the sub-act 1102 includes a sub-act 1102a and a sub-act 1102b.
  • the sub-act 1102a may include extracting both discrete and non-discrete data available in the EMR about patients for current inpatient encounters en, where n ⁇ ⁇ 1,...,N ⁇ , including information about the patient for the current hospitalization, time-related features, event frequency features, prescription medications, patient’s healthcare insurer, comorbidities, and so forth.
  • the sub-act 1102b may include extracting non-structured data.
  • discharge related information may be extracted from a variety of types of notes, for example, consult notes, hospitalist notes, and case management note, for each inpatient encounter en, where n ⁇ ⁇ 1,...,N ⁇ .
  • In 1102 may include compiling data on hospital unit volume vn, where n ⁇ ⁇ 1,...,N ⁇ for each inpatient encounter en, which measures the number of patients admitted into, or discharged from the hospital unit where the patient currently resides within a defined time period (for example, past 12 hours).
  • the sub-act 1102 may include systematically extracting input data from a variety of sources following standard medical data protocols such as HL7, FHIR, web services, connections to databases, data reporting schemes, and so forth. These protocols ensure consistent and seamless integration of data from multiple healthcare settings and devices.
  • the sub-act 1102 may include automatic data collection. Many data points, such as vital signs (for example, heart rate and blood pressure), lab results, and medication administration, are collected automatically from medical devices integrated with EMR systems (for example, bedside monitors, laboratory systems that process bloodwork samples, and drug dispensing systems).
  • the sub-act 1102 may include extracting patient data from manual data entry. Some data, such as clinician's assessments including height, weight, blood pressure, care plans including medications, family history, allergies, and patient-reported outcomes including pain score and symptoms, may be manually entered by healthcare professionals. To maintain consistency and accuracy, standard templates and checklists may be used across different healthcare providers.
  • the sub-act 1102 may include extracting patient data using wearables and remote devices. In certain examples, data from patient-worn devices (for example, glucose monitors and fitness trackers) or remote monitoring systems can also be integrated into the EMR to provide continuous, real-time data streams.
  • data may be extracted from diverse patient populations across different demographic groups, medical conditions, and care settings. For example, patient demographics such as age, gender, ethnicity, and socioeconomic status are commonly captured during hospital admission processes. Diagnosis codes (ICD-10), clinical procedure codes, and patient treatment plans are also included to capture the diversity of patient conditions and healthcare needs.
  • the data compilation at the sub-act 1102 may include integrating information from various healthcare settings, including hospitals, clinics, and specialized care facilities. This comprehensive approach provides a holistic view of patient clinical encounters and outcomes.
  • the data extraction at the sub-act 1102 may follow data quality assurance protocols. Ensuring high-quality data is paramount.
  • sub-act 1102 In addition to automated checks for missing values and Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) outliers, manual validation may be employed by clinical experts to confirm the accuracy of key variables, especially those that are manually entered or highly complex (for example, diagnostic codes, and treatment plans).
  • data may be encrypted in transit and at rest, and de-identification techniques may be used when sharing or analyzing data, protecting sensitive patient information while allowing for meaningful analytics.
  • the extracted and compiled patient data are preprocessed.
  • the sub-act 1104 includes sub-acts 1104a, 1104b, and 1104c.
  • the response variables for ground truth are defined for training multi-class (that is, not binary) prediction models that estimate patient discharge dispositions. Unlike binary classification, where a model determines a simple “ready” or “not ready” status, this multi-class approach predicts a range of possible discharge outcomes, such as “home,” “skilled nursing facility (SNF),” “rehabilitation,” or other broad disposition categories.
  • SNF secreted nursing facility
  • the creation of accurate response variables may be important, as it provides the multi-class model with the ground truth needed to learn and predict these complex patient outcomes.
  • Discharge data may contain many disposition categories (for example, over 100 nuanced disposition categories), which vary across healthcare institutions and patient populations. Managing this level of detail can add complexity, potentially leading to overfitting and reduced model interpretability.
  • a customizable re-grouping module 1004a1 may be constructed to consolidate these highly specific discharge dispositions into broader, clinically meaningful categories.
  • a first category of rehabilitation facilities may include specific dispositions like “discharge to on-site distinct rehabilitation unit from chronic for inpatients” or “discharge to off-site rehabilitation center specializing in post-acute care.”
  • a second category of psychiatric facilities may encompass dispositions like “discharge to a psychiatric facility or an off-site psychiatric unit of another acute care hospital” and “transfer to on-site behavioral health rehabilitation program.”
  • a third category of long-term care (LTC) facilities may cover specific placements such as “discharge to a LTC facility” or “transfer to nursing home with SNF level of care.”
  • a fourth category of home care and community-based services may include “discharge to home with basic home health assistance,” “home discharge with comprehensive home care services,” and “discharge to hospice care at home.”
  • a fifth category of specialized care settings may consist
  • the re-grouping module 1104a1 is also designed to be adaptable across diverse healthcare settings. Each institution or provider can customize the grouping based on their operational needs and disposition categories, enhancing the model’s flexibility and relevance to different environments. To create the response variables, actual recorded discharge dispositions in the EMR are used and cross-validated with clinical notes and hospital standards to ensure each patient’s discharge disposition is accurately represented. This approach ensures that the multi-class model is trained on reliable, real-world data, establishing high-quality labels for the multi-class output.
  • the extracted patient data are feature-engineered to be ready for the following data transformation. The feature-engineering process is carefully designed to extract maximum value from patient data and enhance the predictive power of machine-learning models.
  • the sub-act 1104b includes selecting features of current encounter information including critical aspects of the patient’s current hospital stay. This information may include hospital services provided, the patient's diagnosis, reason for visit, level of care, acuity level, means of arrival, admission type and source, and unit-room-bed location. Additionally, essential clinical measures may be included, such as age, weight, height, BSA, BMI, blood pressure, pulse, and temperature, as well as relevant lab orders and results.
  • the sub-act 1104b may include selecting features of the most recent encounter information. In addition to the current encounter, important historical data from the patient’s most recent hospital stay is considered. The historical data may include the total length of stays, the level of care provided, the hospital services utilized, and the diagnoses from that encounter. Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) By including this historical context, the model can understand the patient’s care trajectory and further refine predictions based on both recent and current clinical information.
  • the sub-act 1104b may include selecting derived features from time-related dynamics and event frequencies.
  • Derived features are generated to provide insights into time-related dynamics and event frequencies. These time-related dynamics track key aspects of a patient's healthcare clinical events, such as the number of days between the current and previous visits, the total hours spent in the hospital, and the time since the last mobility evaluation. Moreover, day-of-week effects are accounted for to capture potential variations in care based on the day the visit occurred.
  • Event frequency features measure the frequency and cumulative occurrences of specific clinical events. For instance, the recent event counts, which represent the number of lab orders, transfers, or patient updates within a recent timeframe (for example, the past 12 hours) may be tracked. Cumulative event counts may also be calculated to monitor the total occurrences of key events (for example, lab orders and transfers) since admission.
  • the sub-act 1104b may include selecting derived features for hospital unit volume. These derived features are considered by tracking the number of patients admitted to, transferred into, or discharged from the hospital unit where the patient currently resides during a defined period (for example, the past 12 hours or 24 hours).
  • the engineered data are transformed for input to the machine-learning models.
  • transformation techniques may be applied based on the nature of the selected features. For numerical features, such as clinical measurements, the normalization technique may be used. This technique centers the data by removing the median and scales it based on the interquartile range, making it particularly resistant to outliers which are common in clinical datasets.
  • a patient’s BMI which might range from very low (for example, 15) to high (for example, 40), is normalized by subtracting the median BMI and scaling it based on the IQR. This makes the data more robust against outliers, such as extreme BMI values for patients with rare conditions.
  • blood pressure readings such as systolic values, typically range between 90 and 180 mmHg but can exhibit extreme values due to acute conditions. By normalizing these values, they are centered around the median systolic blood pressure and scale Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) based on the IQR.
  • blood glucose levels measured in mg/dL
  • the skew from extreme glucose readings is reduced, ensuring that rare, high glucose levels don’t disproportionately impact the model.
  • the length of hospital stay often varies widely, with some patients staying only a few days and others remaining for weeks.
  • boolean features that represent binary outcomes, they are converted into 0 and 1 values to enable seamless integration into machine learning models. This binary format allows the models to interpret the presence or absence of specific patient conditions or statuses effectively.
  • a patient’s current stay in the ICU is recorded as a boolean feature.
  • a patient’s stay may be transformed into a binary indicator (1 for ICU, 0 for not in ICU) to ensure that the models can understand the importance of ICU status in predicting clinical discharge readiness.
  • the feature of whether a patient is experiencing their first hospitalization is encoded as a boolean variable.1 represents a first-time visit and 0 indicates a repeat visit. This information is useful as first-time admissions may differ in complexity and discharge likelihood compared to subsequent visits.
  • whether a patient requires surgery during their stay is also recorded as a Boolean feature, with 1 indicating that surgery is needed and 0 indicating that it is not. This feature helps the model account for patients with surgical needs, who may have longer or more complex stays.
  • one-hot transformation may be employed. This transformation converts categorical variables into one-hot encoded vectors, ensuring that the models can effectively interpret and analyze them.
  • rare categories may be grouped under “other” to avoid sparsity issues in the models. If an unexpected category appears (for example, new means of arrival), the unexpected category may be treated as “other” to prevent the models from breaking due to unknown data.
  • the level of care such as “ICU,” “IMC,” or “nursery” are also one- hot encoded. Each level of care is assigned its own binary column, enabling the models to identify the intensity of care required. Patients in ICU, for example, will have a “1” in the ICU column and a “0” in others. Grouping rare levels under “other” ensures that less frequent levels do not affect model performance.
  • the inherent ranking of these categories is preserved by converting them into numeric values based on a predefined order.
  • a default numeric value for example, -1 is assigned to allow the transformation process to handle unexpected inputs without disrupting the model performance.
  • numeric values are assigned, such as 2, 1, and 0, respectively. This preserves the ranking while allowing the models to understand the progression of mobility states.
  • the unknown mobility level is assigned a default numeric value, such as -1, ensuring that the models can still process the data without disruption.
  • the AM-PAC (activity measure for post-acute care) G-code metrics are encoded based on functional ability, reflecting the likelihood of discharge disposition.
  • Lower-functioning codes for example, CN, CM, and CL
  • higher-functioning codes for example, CJ, CI, and CH
  • This numeric scale preserves the ranking, with CN mapped to 0 and CH to 6, to capture the progressive scale of functional ability.
  • Unknown codes are assigned a default value of -1.
  • the acuity level reflects the urgency of medical attention required, with levels ordered as “less urgent,” “urgent,” “emergent,” and “immediate.” These are assigned numeric values to capture the hierarchy, with 0 for “less urgent,” 1 for “urgent,” 2 for “emergent,” and 3 for “immediate.” This encoding ensures the model accurately interprets increasing acuity. Any unknown acuity level is assigned -1 to maintain data consistency. For textual data, which often include rich but unstructured data like the reason for visits, textual transformation is used. Textual transformation converts the text into a matrix of TF-IDF scores.
  • the transformation captures the most significant terms that best represent the textual data, enabling the models to process and interpret these insights effectively.
  • a text field describing the patient’s reason for visiting the hospital for example, “chest pain” and “shortness of breath”
  • TF-IDF scores By converting the text to lowercase and ignoring common stop words (for example, “the” and “is”), the top most relevant terms are focused on.
  • text fields of clinical notes are transformed into a matrix of TF-IDF scores, describing a patient's health status, treatment plan, and medical history such as quality improvement measures on deep vein thrombosis (DVT) prophylaxis, urinary catheter existence, ambulation status, and nutrition.
  • text fields of operative notes that describe a patient’s surgery such as type of surgery, estimated blood loss, type of anesthesia, and urine output are transformed into a matrix of TF-IDF scores.
  • a machine-learning algorithm is selected from multiple algorithms of different types.
  • the sub-act 1106 may include sub-acts 1106a and 1106b.
  • the sub-act 1106a includes training multiple multi-class models to predict discharge disposition using algorithms of different types.
  • each multi-class model is trained using different types of algorithms with default parameter settings, including Random Forest, XGBoost, SVM, and Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) Neural Networks.
  • the models may be trained based on the data with features obtained from sub- act 1104. To evaluate the performance of these models, in one example, a 5-fold cross-validation is employed using a dataset where the ground truth of the discharge disposition is known.
  • the dataset is split into five parts, with each model trained on four folds and tested on the remaining fold. This is repeated five times to ensure that each part of the dataset is used as a test set exactly once, providing a robust assessment of model performance and reducing the risk of overfitting to any specific subset of data.
  • a comprehensive comparison based on a set of metrics is performed for various machine-learning algorithms of different types to predict the discharge disposition of a patient at the time of the discharge.
  • the multi-class models are evaluated based on key metrics tailored to healthcare analytics, including accuracy, precision, recall, F1-score, and ROC-AUC. Accuracy measures overall prediction correctness.
  • ROC-AUC measures a model’s ability to distinguish between classes. In multi-class settings, ROC-AUC is calculated using the one-vs-rest approach for each class and then averaged. A higher ROC-AUC indicates better model performance in correctly ranking disposition categories and minimizing incorrect classifications.
  • XGBoost may be strong in handling complex, multi-class, and imbalanced clinical datasets, and may be robust against overfitting through regularization. Furthermore, XGBoost’s efficiency in processing large, nuanced datasets may allow it to generalize well across diverse patient cases, capturing key patterns across a broad range of discharge dispositions. As a result, XGBoost may be selected at sub-act 1106b as the optimal algorithm for the multi-class model. In other examples, one or more other machine-learning algorithms may be chosen alone or in various combinations at the sub-act 1106b.
  • the hyperparameters of the selected type of machine- learning algorithm may be optimized by training and periodically updated.
  • the sub-act 1108 may include sub-acts 1108a, 1108b, and 1108c.
  • a Bayesian optimization procedure may be implemented to fine-tune the hyperparameters of the selected multi-class model. This optimization ensures that the model maximizes its predictive performance while minimizing overfitting, allowing it to generalize well to unseen data. This allows further refining of the model’s accuracy to ensure that the model is a reliable tool for predicting the most likely discharge disposition for the patient.
  • the hyperparameters defined below are optimized to improve the performance of the XGBoost model.
  • Maximum tree depth is the maximum depth of the trees which controls the model complexity. In various examples, maximum tree depth may be set to a range between, for example, 3 and 10. Learning rate determines the step size during model training, sampled from a logarithmic scale between a range of values (for example, 0.001 and 0.3) to allow for both small and large step sizes.
  • Minimum child weight specifies the minimum sum of instance weights required for a child node to split, helping control overfitting. In at least one example, minimum child weight may be set within a range of between 1 and 6. Gamma defines the minimum loss reduction required for partitioning a leaf node and controls regularization.
  • the gamma value may be sampled between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.05).
  • Subsample ratio represents the fraction of training samples used for building each tree, uniformly sampled between a range of values (for example, a range of 0.5 and 1) with a specified step size Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) (for example, a step size of 0.05).
  • Column sample by tree specifies the fraction of features to be randomly sampled for each tree, sampled uniformly between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.1).
  • Maximum delta step is particularly beneficial for handling imbalanced classes in a multi-class setup. Setting a limit on the maximum step size that each leaf node’s weight can take helps stabilize the model’s updates, preventing excessive shifts that could favor the majority class.
  • Maximum delta step may be defined using a quantized uniform distribution to explore integer values between 0 and 10.
  • the evaluation metric for fine- tuning the hyperparameters is selected as the AUC-PR, which is more suitable for handling imbalanced classes.
  • the goal of the sub-act 1108a is to identify the optimal hyperparameter set ⁇ ⁇ that maximizes the AUC-PR score, expressed as an objective function ⁇ .
  • ⁇ ⁇ ⁇ represent the AUC-PR score of the XGBoost model with hyperparameters ⁇ .
  • ⁇ represents the hyperparameters to be optimized (for example, learning rate, maximum depth, and regularization terms)
  • ⁇ is the space of possible hyperparameters.
  • TPE is used as the surrogate model to approximate the objective function ⁇ . TPE models the likelihood of good and bad hyperparameter configurations based on previous observations.
  • ⁇ ⁇ ⁇ ⁇ be the likelihood of high-performing hyperparameters (those improving AUC-PR) and ⁇ ⁇ ⁇ ⁇ be the likelihood of low-performing hyperparameters (those reducing AUC-PR). These distributions are updated based on observed evaluations, and the surrogate model is used to predict the performance of new hyperparameter configurations. In various examples, the surrogate model is iteratively refined as more real-time data is collected. For acquisition maximization, an acquisition function is used to guide the selection of the next hyperparameters for evaluation.
  • This ratio directs the optimization process, balancing exploration (trying hyperparameters where the model is uncertain) and exploitation (choosing hyperparameters that are likely to perform well based on the surrogate model).
  • the Bayesian optimization procedures start at step one of initialization, in which a few initial random hyperparameter configurations are evaluated to gather baseline data. These configurations are chosen randomly within the specified hyperparameter space ⁇ and provide the initial observations necessary to build a surrogate model.
  • the surrogate model is fitted based on the observed data, constructing the ⁇ and ⁇ distributions.
  • the surrogate model serves as a probabilistic estimator that guides the search for optimal hyperparameters, predicting how different configurations will perform without needing to evaluate each one directly.
  • the next hyperparameter configuration is selected by maximizing the acquisition function, which balances exploration (testing hyperparameters where uncertainty is high) and exploitation (testing configurations likely to yield high performance based on the surrogate model). This ensures that the search efficiently converges towards optimal hyperparameters, even in complex search spaces.
  • the model is fitted using XGBoost with a maximum number of rounds per trial (for example, 500 boosting rounds per trial). Additionally, if there is no improvement in model performance after a certain number of boosting rounds (for example, 100 boosting rounds), early stopping may be triggered to prevent further unnecessary iterations. Performance may be evaluated using the AUC-PR score, which is particularly relevant for an imbalanced healthcare dataset, focusing on precision and recall.
  • the optimization continues by repeating steps two to four. In one example, the total number of trials may be limited to a maximum number (for example, 50) to ensure that the search process remains efficient without overfitting the hyperparameter space.
  • the optimization may be stopped and the hyperparameter fine-tuning is completed.
  • the new data of the first type may include relevant information such as whether certain care units become opened or closed which will affect the workflow through the hospital, or whether a new initiative is introduced at the hospital that affects the workflow of the nurses or doctors and changes where and when data are documented.
  • the sub-act 1108b may be configured to be performed periodically, for example, daily, weekly, monthly, or quarterly, to search for new data of the first type. If new data are available (“YES”), the process 1100 proceeds to the sub-act 1108c to extract, process, and insert the new first-type data to update the dataset. After the dataset is updated, the process 1100 returns to the sub-act 1108a to retrain the final model with the updated dataset. If no new first-type data are available (“NO”) at the sub-act 1108b, the process 1100 proceeds to a sub-act 1110. At the sub-act 1110, in one example, the process 1100 proceeds based on whether the predicted disposition may be characterized as a sensitive scenario.
  • sub-act 1112 may include determining whether the predicted disposition is on a list of pre-defined sensitive dispositions.
  • the sensitive predictions are converted to “to be determined” to allow healthcare providers to reassess critical cases without relying solely on the model’s output. This safeguard ensures that sensitive cases receive the necessary caution and attention.
  • the sub-act 1112 may include sub-acts 1112a and 1112b.
  • sub-act 1112a in one example, if case managers or healthcare staff document discharge-related dispositions in the EMR system, these human-entered inputs automatically overwrite the model’s predictions. This priority on case managers’ inputs helps avoid miscommunications by ensuring the final output aligns with real-time human judgment.
  • the sub-act 1112a may include detecting and integrating these human-entered data points to seamlessly override the model’s output as needed.
  • the post-processing of the prediction results may be adaptable to accommodate additional rules for specific scenarios or guidelines, such as flagged cases for extended observation or special discharge handling based on institutional requirements.
  • the creation of the discharge disposition model may include only static historical data
  • the final prediction may change over time (sometimes quickly) as new EMR data of a second type becomes available.
  • the new second-type data may include new information associated with the patient or the care unit where the patient is currently bedded.
  • the discharge disposition prediction may be produced and sent to the EMR in as timely a fashion as possible, which requires near-real-time data acquisition.
  • Industry-standard mechanisms such as HL7 and FHIR, are the primary near-real-time acquisition methods.
  • Several other data acquisition methods are also used, such as data analytics reporting tools, API web services, and KB_SQL queries. This allows an automatic near-real-time two-way data pipeline.
  • the process 1100 proceeds through the sub-act 318 depending on whether new real-time data of a second type becomes available for updating the model hyperparameters.
  • each prediction may be updated in the EMR system display every few minutes (for example, every 8 to 12 minutes), depending on the availability of new data, to reduce the amount of variability in the displayed score over time and enhance the usability of the predictions for clinical users. If there is new second-type data available from, for example, a near-real-time data pipeline via HL7, FHIR, or other means, the sub-act 318 is executed to extract, process, and insert the new data to update the training dataset.
  • the sub-act 318 is performed to extract, process, and insert the new data to update the input dataset. Then the process 1100 returns to the sub-act 1114 to re- execute the final model. If no new second-type data are available at sub-act 318, the process 1100 stays at the sub-act 318 to keep searching for new second-type data.
  • the sub-act 1108c and the sub-act 318 may search for new data of the same type to be updated but may be performed at different periodicities. For example, the sub- act 318 may be performed more frequently than the sub-act 1108c.
  • the machine-learning process 1100 for executing the act 126 to predict the patient’s discharge disposition may be performed via the user interface 204.
  • the user interface 204 may display a GUI associated with the act 126.
  • FIG.11 illustrates a GUI 1200 for predicting discharge dispositions of patients in a hospital according to an example.
  • a first column 1202 displays the predicted discharge disposition results for patients currently in the hospital in a transparent and straightforward manner to assist in the decision-making of where the patients may be discharged at their respective discharge times.
  • a second column 1204 displays the unit and bed numbers of the patients to identify those patients. For example, the patient in bed 534-B of unit 5W should be discharged home but the patient in bed 3-1 of unit 2C should be discharged to an SNF.
  • the process 100 proceeds to act 130 to predict the patient’s risk of readmission after the patient is discharged from the hospital for a certain period.
  • the act 130 may be executed shortly after the patient’s arrival at the hospital at the act 102 and continues throughout the patient’s inpatient stay and up until after the patient’s discharge at act 132 for a certain period.
  • a patient may need to be readmitted to the hospital after departure or discharge. A readmission occurs when a patient is admitted, discharged, and then subsequently readmitted.
  • the terms “admitted,” “discharged,” and “readmitted” are specific to inpatients, although they are frequently misused even by healthcare professionals.
  • readmissions are planned. For example, some orthopedic surgeries require a follow-up inpatient encounter to remove temporarily implanted hardware. Some readmissions are either unrelated to or undertaken long after the initial admission. In healthcare, unplanned readmissions that occur within a relatively short amount of time, that is, excluding special cases like those described above, are widely viewed as undesirable. It may be a sign of previous inadequate or incomplete care. There are relevant state and federal metrics based on readmission rates that are used to rank hospitals. Sometimes these metrics are used to impose monetary incentives/disincentives. In Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) addition, insurance companies will often not pay claims for unplanned readmissions under certain circumstances.
  • An example type of readmission is an unplanned 30-day readmission, the likelihood of which needs to be predicted before patient discharge.
  • a 30-day readmission refers to a readmission that occurs within 30 days after the discharge date.
  • Traditional readmission risk scores are calculated at discharge time, which allows no opportunity to plan for preventive measures prior to discharge.
  • transitional nurse navigators can identify high-risk patients several days before discharge to allow for patient education efforts and other medical or supportive interventions before discharge to prevent readmission. Also, transitional nurse navigators will no longer need to sift through all bedded inpatients to determine which ones have high-risk conditions, have currently been readmitted, and might be amenable to interventions to reduce readmission risk.
  • FIG.12 illustrates a machine-learning process 1300 to execute the act 130 to predict the readmission risk of a patient if the patient is discharged from the hospital according to an example.
  • the process 1300 starts by extracting and compiling various types of data.
  • the sub-act 1302 may include sub-acts 1302a, 1302b, and 1302c to extract several years of clinical, operational, and financial data, primarily focusing on patients and patient encounters, from a hospital’s or hospital system’s EMR and/or other sources, including cross-organization data repositories such as a s nationwide health information exchange (HIE).
  • HIE s nationwide health information exchange
  • EMR and HIE data may be extracted at sub-act 1302a and 1302b, respectively, using a variety of standard extract-transform-load (ETL) techniques and data acquisition methods, including SQL queries, API/web service calls, and event-driven real-time interfaces.
  • API calls and real-time interfaces may adhere to well-known international standards such as HL7, DICOM, and FHIR.
  • An EMR is an application and database that acquires and stores clinical, operational, and financial data related to healthcare and health system operations.
  • most data extracted at the sub-act 1302 is extracted from the EMR at the sub-act 1302a.
  • Categories of data acquired from the EMR include patient demographics, patient history, encounter details, medications, lab tests, vital signs, radiology results, nursing documentation, SDOH, and financial details.
  • Act the sub-act 1302a data are extracted using the inpatient (or expected to be inpatient) encounter e n , where n ⁇ ⁇ 1,...,N ⁇ as the cornerstone data point such that all other data are related Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) to that specific inpatient encounter. While the focus on the predicted risk score is inpatients (hospitalized patients), the readmission risk score is also calculated for currently in-house non- inpatients (in the ED and/or observation unit) who might become inpatients, and also previous inpatients within 30 days of discharge.
  • HIE is a cross-organization data repository offering data that may not be generally available to a single healthcare organization because healthcare organizations often do not share data directly with each other. For readmissions, a typical healthcare organization only has access to readmission data for patients who are both admitted and subsequently readmitted to one of that organization’s facilities.
  • cross-organization data repositories for example, HIE
  • HIE cross-organization data repositories extracted at the sub-act 1302b fills the gap, providing admission and readmission data for patients whose index admission or subsequent readmission occurred outside the boundaries of the healthcare organization.
  • publicly available data may be extracted from the U.S. Census Bureau (for example, the decennial census supplemented by ACS), CDC (social vulnerability index), and Neighborhood Atlas (social vulnerability index).
  • the sub-act 1304 may include sub-acts 1304a and 1304b.
  • the response variables for the ground truth of a 30-day (unplanned) readmission risk model are defined based on data extracted from the HIE, along with readmission-specific diagnosis codes and hospital admission or readmission facilities.
  • readmission risk models for unplanned readmission after a different period for example, 60 days or 90 days may be employed.
  • the extracted and compiled data are feature-encoded and feature-engineered.
  • the goal of both feature-encoding and feature-engineering is to maximize the predictive power of “raw” data.
  • Many techniques for both encoding and engineering may be used, so it may be advantageous to gauge the efficacy of each and retain only the best performing techniques for the model type and metric(s) chosen. This is done iteratively along with model and feature selection (as described in more detail below).
  • Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) In one example, for constructing a 30-day readmission model, the following variables are encoded from the data features and evaluated based on performance.
  • Single variables include numeric, boolean, ordinal, and categorical data features.
  • Numerical data features include extracted data such as height, weight, and BMI.
  • Boolean data features may be encoded by uniformly converting extracted data to 1 (yes) and 0 (no) for information of, for example, whether a patient is on IV therapy, in the ICU, or has previous encounters with a hospital or hospital system.
  • Ordinal data features may be converted from the extracted data into a series of sequential integers while preserving their original rankings (for example, pain score and admission level of care).
  • the data features may be encoded by either one-hot encoding (dummy encoding) or frequency encoding.
  • data features relating to admission source, mode of arrival, and primary insurance type may use one-hot encoding
  • data relating to ZIP code, financial class, and type of breathing assistance may use frequency encoding.
  • the data features may be encoded by either hash encoding or grouping (especially when an appropriate industry standard “grouper” is available).
  • an EMR may define tens of thousands of order codes, some with industry-standard coding, and some without. Because not all of them can be grouped in useful or meaningful ways, hash encoding is used to reduce the dimensionality while attempting to preserve enough detail to facilitate predictive modeling.
  • medication data is generally stored as NDC codes, of which there are tens of thousands.
  • ATC pharmacological and therapeutic groupings are used, as provided by the WHO. Diagnoses are generally represented as ICD codes. Again, there are tens of thousands of ICD codes.
  • CCSR body systems and clinical categories are used.
  • variables that change over time are also encoded from the data features and evaluated based on performance. Although many healthcare data variables change over time, vital signs, for example, the resulting chronological vectors, are often irregular and/or sparse. Nevertheless, these types of data are still treated as time series variables.
  • the features created include first value, most recent value, Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) time since the most recent value is recorded, total number of values recorded, and average number of values recorded per day.
  • Additional features are created for each numeric time series variable, which include median value, minimum value, maximum value, variance, difference between the median value and the most recent value, difference between the median value and the fist value, difference between the first value and the most recent value, first quartile boundary, third quartile boundary, time spent in each quartile (in minutes), time spent in each quartile as a percentage of total encounter time, current quartile, and the slope of a regression model fit to the values (for example, heart rate, pain score, and white blood count). Additional features are created for each boolean time series variable, including time spent in positive and negative states (in minutes) and time spent in positive and negative states as a percentage of total encounter time (for example, time on a ventilator, in ICU, and for receiving antibiotic).
  • each ordinal variable (after being transformed or encoded), including time spent in each ordinal level (in minutes) and time spent in each ordinal level as a percentage of total encounter time (for example, pain score, Richmond agitation-sedation scale (RASS) level, and patient mobility). Additional features are created for each categorical variable, including time spent in each category (in minutes) and time spent in each category as a percentage of total encounter time (for example, affect and behavior, respiratory WDL, and central line status).
  • the type of machine-learning algorithm for the 30-day readmission model is selected from various types of algorithms.
  • the sub-act 1306 includes sub-acts 1306a and 1306b.
  • various binary models are trained, each using a different type of algorithm.
  • folds and splits are constructed for an augmented version of k-fold cross-validation.
  • the data set resulting from feature encoding and engineering is first split into two sets, one containing the oldest set of encounters (train-0) (for example, the oldest 85% of encounters), and the other containing the most recent set of encounters (test-0) (for example, the most recent 15% of encounters).
  • Train-0 is further split into five “folds” of relatively equal size.
  • each fold is limited to a unique set of patients, including all encounters for those patients; each sub-set contains at least one sample of each category for all categorical variables.
  • Hyperparameters for each model are chosen based on a coarse-grained grid search.
  • Features are culled separately for each model using algorithm-specific methods, requirements, and constraints.
  • further transformation and/or augmentation of the data may be required. For example, some algorithms cannot handle missing values so various imputation methods are used to fill the gaps.
  • Some algorithms perform best when numeric data have been normalized so appropriate normalization techniques are applied to those features.
  • Some algorithms perform poorly if highly correlated features are used so a cross-correlation matrix is generated, and over-correlated features are dropped. Some algorithms perform better when continuous numeric features are split into binary “buckets.” Some algorithms may not be equipped to handle the number of features in the training data so feature culling techniques (for example, FTRL) may be applied.
  • FTRL feature culling techniques
  • the evaluated types of algorithms include Simple Regression (Ridge and Lasso), Gradient Boosted Trees (normal via CatBoost and “extreme” via XGBoost), Support Vector Machines (for example, linear only without “kernel trick” variations), Random Forest (classical and “extra trees” variant), and Neural Networks (for example, long short-term memory [LSTM] and recurrent neural network [RNN]).
  • the performance of each tuned model is evaluated using a set of metrics. Although the target variable is binary, the desired model output is a probability expressed as a number between 0.0 (risk is 0%) and 1.0 (risk is 100%).
  • the data may be unbalanced, with the prevalence of 30-day unplanned readmission generally ranging between 5% and 20%.
  • AUC-ROC may be chosen as the primary measure of quality partly because of its resistance to unbalanced data and because AUC-ROC is the metric widely accepted in assessing readmission risk.
  • AUC-PR may be added as a secondary metric to provide a sanity check.
  • error distributions are analyzed to understand the conditions under which each model performs best and worst. In one example, at the sub-act 1306b, the gradient-boosted-tree models turn out to be less brittle, require significantly fewer resources to train and maintain, and have less positive-case error.
  • XGBoost variant gradient boosted trees
  • one or more other machine-learning algorithms may be chosen alone or in various combinations at the sub-act 1306b.
  • the hyperparameters of the selected type of algorithm are optimized and updated, and feature culling is performed, both by training, for a final 30-day readmission model.
  • the sub-act 1308 may include sub-acts 1308a, 1308b, 1308c, 1308d, and 1308e.
  • the final model is constructed.
  • the hyperparameters defined below are optimized to improve the performance of the XGBoost model.
  • Maximum tree depth is the maximum depth of the trees which controls the model complexity. In various examples, maximum tree depth may be set to a range between, for example, 3 and 8.
  • Learning rate determines the step size during model training, sampled from a logarithmic scale between a range of values (for example, 0.001 and 0.3) to allow for both small and large step sizes.
  • Minimum child weight specifies the minimum sum of instance weights required for a child node to split, helping control overfitting.
  • minimum child weight may be set within a range of between 1 and 25 with a step size of 5.
  • Gamma defines the minimum loss reduction required for partitioning a leaf node and controls regularization.
  • the gamma value may be sampled between a range of values (for example, 0.5 and 1) with a specified step size (for example, a step size of 0.05).
  • Subsample ratio represents the fraction of training samples used for building each tree, uniformly sampled between a range of values (for example, a range of 0.5 and 1) with a specified step size (for example, a step size of 0.1).
  • Scale positive class weight is a parameter particularly beneficial for handling imbalanced classes in a binary setup. Scale positive class weight adjusts the balance of positive and negative weights to handle imbalanced data, sampled uniformly between a range of values (for example, 1 and 5) to give more weight to the positive class.
  • the hyperparameters for each of the five models are fine-tuned using a Bayesian optimization procedure.
  • the goal of the optimization is to identify Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) the optimal hyperparameter set ⁇ ⁇ that maximizes the AUC-ROC score, expressed as an objective function ⁇ ⁇ ⁇ ⁇ .
  • ⁇ ⁇ ⁇ ⁇ represent the AUC-ROC score of the XGBoost model with hyperparameters ⁇ .
  • TPE For surrogate model fitting, a TPE is used as the surrogate model to approximate the objective function ⁇ ⁇ ⁇ ⁇ .
  • TPE models the likelihood of good and bad hyperparameter configurations based on previous observations. Let ⁇ ⁇ ⁇ ⁇ be the likelihood of high-performing hyperparameters (those improving AUC-ROC) and ⁇ ⁇ ⁇ ⁇ be the likelihood of low-performing hyperparameters (those reducing AUC-ROC). These distributions are updated based on observed evaluations, and the surrogate model is used to predict the performance of new hyperparameter configurations. In various examples, the surrogate model is iteratively refined as more real-time data is collected.
  • an acquisition function is used to guide the selection of the next hyperparameters for evaluation.
  • This ratio may direct the optimization process, balancing exploration (trying hyperparameters where the model is uncertain) and exploitation (choosing hyperparameters that are likely to perform well based on the surrogate model).
  • the Bayesian optimization procedures start at step one of initialization, in which a few initial random hyperparameter configurations are evaluated to gather baseline data. These configurations are chosen randomly within the specified hyperparameter space ⁇ and provide the initial observations necessary to build a surrogate model.
  • the surrogate model is fitted based on the observed data, constructing the ⁇ and ⁇ ⁇ ⁇ ⁇ distributions.
  • the surrogate model serves as a probabilistic estimator that guides the search for optimal hyperparameters, predicting how different configurations will perform without needing to evaluate each one directly.
  • the next hyperparameter configuration is selected by maximizing the acquisition function, which balances exploration (testing hyperparameters where uncertainty is high) and exploitation (testing configurations likely to yield high performance based on the surrogate model). This ensures that the search efficiently converges towards optimal hyperparameters, even in complex search spaces.
  • the model is fit using XGBoost with a maximum number of boosting rounds per trial (for example, 2500 boosting rounds per trial). Additionally, if there is no improvement in model performance after a certain number of boosting rounds (for example, 100 boosting rounds), early stopping is triggered to prevent further unnecessary iterations. Performance is evaluated using the AUC- ROC score, which is particularly relevant for our imbalanced healthcare dataset, focusing on precision and recall.
  • the optimization continues by repeating steps two to four. In one example, the total number of trials may be limited to a maximum number (for example, 50) to ensure that the search process remains efficient without overfitting the hyperparameter space.
  • feature culling may also be performed at the sub-act 1308b.
  • the importance of each feature for each model is determined using SHAP analysis.
  • SHAP is a model-agnostic approach that uses coalitional game theory to evaluate each model feature representing how much that feature contributes to the model’s output.
  • Features with zero contribution, based on the SHAP analysis, are dropped, and models are retrained without them.
  • the sub-act 1308a may include investigating and, when necessary, mitigating biases of various types.
  • Subsets of results from all fine-tuned models are used to check for biases by comparing them against the performance of the overall model. For example, the performance on a subset of results including only female patients may be compared to the performance of the entire model. For example, a bias check may be performed for the Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) various categories including race, ethnicity, sex, language, religion, and census tract. If an unacceptable level of bias is discovered, attempts to mitigate that bias may be made. Mitigation may be done by over-sampling the biased population or re-weighting training samples to put more emphasis on the biased population.
  • the final model is built using train-0 (a combination of all folds) and subsequently tested against test-0, a chronologically “most recent encounters” hold-out set at the sub-act 1308b.
  • hyperparameters for this final model are derived by averaging hyperparameters across all five split-based models.
  • the feature set for the final model is a union of feature sets from all split-based models.
  • the final model is built using the complete data set. Hyperparameters and the feature set are the same as for the train-0 model. Isotonic regression coefficients are calculated specifically for the final model.
  • the final model is validated.
  • the process 1300 proceeds to a sub-act 1308d. Otherwise (“NO”), the process 1300 returns to the sub-act 1308a to find and fix the causes.
  • the new data of the first type may include relevant information such as whether certain care units become opened or closed which will affect the workflow through the hospital, or whether a new initiative is introduced at the hospital that affects the workflow of the nurses or doctors and changes where and when data are documented.
  • the sub-act 1308d may be configured to be performed periodically, for example, daily, weekly, monthly, or quarterly, to search for new data of the first type. If new data are available (“YES”), the process 1300 proceeds to the sub-act 1308e to extract, process, and insert the new first-type data to update the dataset. After the dataset is updated, the process 1300 returns to the sub-act 1308b to retrain the final model with the updated dataset. If no new first-type data are available (“NO”) at the sub-act 1308d, the process 1300 proceeds to a sub-act 1310 to execute the final model to output the readmission risk prediction.
  • NO no new first-type data are available
  • Discretization of the output prediction from the sub-act 1314 may still be needed because machine-learning models, even those that produce continuous results in the range of 0 to 1, do Attorney Docket No.: M2212-7001WO(UMMS23-0003/WO.10) not generally produce outputs that resemble probabilities.
  • Probabilistic alignment is desirable for two reasons. First, non-probabilistic results can cause end-user cognitive dissonance. People not trained in machine learning, statistics, or similar disciplines may not be familiar with numbers that have no inherent meaning beyond rank. Second, model retraining may be done regularly (for example, quarterly) to incorporate new data and discover new patterns. Retrained models are likely to produce outputs with different characteristics (ranges, distributions, and so forth) than their predecessors.
  • the model outputs are transformed to more closely match probabilities while preserving overall rank. If the error distribution of the final model exhibits sigmoidal characteristics, isotonic regression may be chosen as the transformation technique. Other transformation techniques such as Platt scaling may also be considered.
  • the pseudo-probabilities are discretized pyramidally into 5 categories, where category 5 (highest risk) contains the smallest number of patients, and category 1 (lowest risk) contains the largest number of patients. This facilitates color coding and at-a-glance interpretation for end-users. In other examples, other numbers of discrete categories may be used for the pseudo-probabilities of the final model.
  • the creation of the predictive model may include only static historical data
  • per- patient deterioration risk scores may change over time (sometimes quickly) as new EMR data of a second type becomes available.
  • the new second-type data may include new information associated with the patient or the care unit where the patient is currently bedded.
  • the readmission risk scores may be calculated and sent to the EMR in as timely a fashion as possible, which requires near-real-time data acquisition.
  • Industry-standard mechanisms such as HL7 and FHIR, are the primary near-real-time acquisition methods.
  • Several other data acquisition methods are also used, such as data analytics reporting tools, web services, and KB_SQL queries. This allows an automatic near-real-time two-way data pipeline.
  • the process 1300 proceeds through sub-act 318 depending on whether new second-type data becomes available.
  • each readmission risk score may be updated in the EMR system display every few minutes (for example, every 8 to 12 minutes), depending on the availability of new data, to reduce the amount of variability in the displayed score over time and enhance the usability of the deterioration risk scores for clinical users.
  • the sub-act 318 is executed to extract, process, and insert the new data to update the training dataset.
  • the process 1300 then returns to the sub-act 1310 to re-execute the final model. If there is no new second-type data available at the sub-act 318, the process 1300 stays at the sub-act 318 to keep searching for new second-type data. In other examples, the sub-act 1308d and the sub-act 318 may search for new data of the same type to be updated but may be performed at different periodicities. For example, the sub- act 318 may be performed more frequently than the sub-act 1308d.
  • the machine-learning process 1300 for executing the act 130 to predict the patient’s readmission risk may be performed via the user interface 204.
  • the user interface 204 may display a GUI associated with the act 130.
  • FIG.13 illustrates a GUI 1400 associated with predicting 30-day readmission risks of patients at a hospital system level according to an example.
  • a first column 1402 displays the predicted readmission results in descending order (from top to bottom) for patients to assist in the decision-making of which patients may have a high risk for readmission to the hospital.
  • a second column 1404 displays the discretized readmission risk scores of the patients in a transparent and straightforward manner. In this example, because all the patients have a relatively high readmission score (for example, a readmission score of 4 or 5), all the readmission risk scores may be highlighted by a background in red or of a first texture.
  • a third column 1406 displays the current location information for each patient. For example, if the patient is still in the hospital (that is, the process 1300 is performed before the patient’s discharge), the background of the patient’s location information is highlighted in yellow. Otherwise, if the patient is located outside the hospital such as at home possibly against medical advice (that is, the process 1300 is performed after the patient’s discharge), the background of the patient’s location information is not highlighted.
  • the fourth column 1408 and the fifth column 1410 may display the patient's last clinical service and the associated admission complaint, respectively, to provide context to the readmission risk prediction.
  • the anticipatory system 201 includes a controller 202, a user interface 204, memory 206, and a network interface 208, which may be coupled to the external devices 210.
  • the controller 202 may execute various operations discussed above.
  • the controller 202 may also execute one or more instructions stored on one or more non-transitory computer- readable media.
  • the memory 206 and/or the external devices 210 may include or be coupled to the non-transitory computer-readable media which store instructions 214.
  • the controller 202 may execute the instructions 214 to execute various operations discussed above, including the processes 100, 400, 500, 700, 800, 1000, 1100, and 1300.
  • the controller 202 may include one or more processors or other types of controllers.
  • the controller 202 is or includes at least one processor.
  • the controller 202 performs at least a portion of the operations discussed above using an application-specific integrated circuit tailored to perform particular operations in addition to, or in lieu of, a processor.
  • examples in accordance with the present disclosure may perform the operations described herein using many specific combinations of hardware and software and the disclosure is not limited to any particular combination of hardware and software components.
  • Examples of the disclosure may include a computer-program product configured to execute methods, processes, and/or operations discussed above.
  • the computer-program product may be, or include, one or more controllers and/or processors configured to execute instructions to perform methods, processes, and/or operations discussed above.
  • the user interface 204 may include one or more user inputs, one or more user outputs, or a combination thereof.
  • the user interface 204 may include user-input devices such as a keyboard, a mouse, a touchscreen, and so forth. Users, such as medical personnel, may use the user inputs to input information indicative of a patient’s current status in the hospital and/or a current status of various units in the hospital.
  • the user interface 204 may also include user- output devices such as a computer monitor, sound-emitting devices such as speakers or buzzers, light-emitting devices such as light-emitting diodes, and so forth.
  • the user interface 204 may include one or more GUIs. Users, such as medical personnel, may use the user outputs to receive various information related to clinical decision-making on changing patients’ care settings.
  • the memory 206 may include non-transitory computer-readable media, such as memory devices (for example, read-only memory, random-access memory, and so forth) and/or storage devices (for example, hard drives, optical disks, and so forth).
  • the network interface 208 may include one or more devices or components to interface with external devices, such as the external devices 210.
  • the network interface 208 may include wired-communication components that enable wired communication (for example, wired-communication ports to receive a communication wire), wireless-communication components that enable wireless communication (for example, antennas), a combination thereof, and so forth.
  • the external devices 210 may include any devices or components which are physically external to the anticipatory system 201, such as external computing devices (for example, other computing devices within a hospital), cloud computers, servers, external data storage, and so forth.
  • the external devices 210 may include devices substantially similar to the anticipatory system 201.
  • a hospital may include several computing devices implemented as or including the anticipatory system 201, each of which may be networked together.
  • each hospital unit may include a dedicated team of discharge professionals who manage patient discharges, and each of these teams (or even each member of each team) may have one or more computing devices substantially similar to the anticipatory system 201.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Business, Economics & Management (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Public Health (AREA)
  • Primary Health Care (AREA)
  • Computational Linguistics (AREA)
  • Epidemiology (AREA)
  • General Business, Economics & Management (AREA)
  • Molecular Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Resources & Organizations (AREA)
  • Tourism & Hospitality (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Databases & Information Systems (AREA)
  • Development Economics (AREA)
  • Pathology (AREA)
  • Mathematical Optimization (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Game Theory and Decision Science (AREA)
  • Mathematical Analysis (AREA)
  • Educational Administration (AREA)
  • Computational Mathematics (AREA)

Abstract

L'invention concerne un procédé qui consiste à : recevoir des informations de patient comprenant des informations concernant une hospitalisation actuelle d'un patient; recevoir des informations de volume d'unité d'hôpital correspondant à une fenêtre de temps de roulement, indiquant un nombre de patients admis, transférés dans, ou sortis d'une unité d'hôpital recevant le patient; convertir les informations de patient et les informations de volume d'unité d'hôpital en un format interprétable par machine pour au moins un algorithme d'apprentissage automatique de sortie binaire; exécuter l'au moins un algorithme pour générer des modèles binaires qui prédisent la probabilité de sortie du patient à l'intérieur de chacune de multiples trames temporelles; combiner les modèles binaires en un modèle combiné unique, comprenant l'optimisation de pondérations de prédictions de sortie pour chaque modèle binaire; déterminer, à l'aide du modèle combiné unique, au moins une quantification de disposition à une sortie pondérée; transformer l'au moins une quantification de disposition à une sortie pondérée en au moins une prédiction de sortie; et commander une interface utilisateur pour fournir l'au moins une prédiction de sortie à un utilisateur.
PCT/US2024/056428 2023-11-16 2024-11-18 Procédés et systèmes de réglage optimal de soins pour des patients à l'aide d'un apprentissage automatique Pending WO2025106998A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363599753P 2023-11-16 2023-11-16
US63/599,753 2023-11-16

Publications (1)

Publication Number Publication Date
WO2025106998A1 true WO2025106998A1 (fr) 2025-05-22

Family

ID=95743358

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2024/056428 Pending WO2025106998A1 (fr) 2023-11-16 2024-11-18 Procédés et systèmes de réglage optimal de soins pour des patients à l'aide d'un apprentissage automatique

Country Status (1)

Country Link
WO (1) WO2025106998A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120260940A (zh) * 2025-06-09 2025-07-04 中国人民解放军西部战区总医院 一种老年综合征智能科研数据管理方法及系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200013490A1 (en) * 2017-03-03 2020-01-09 Rush University Medical Center Patient predictive admission, discharge, and monitoring tool
US20200211701A1 (en) * 2018-12-31 2020-07-02 Cerner Innovation, Inc. Decision Support Tool For Determining Patient Length Of Stay Within An Emergency Department
US20210098090A1 (en) * 2019-09-30 2021-04-01 GE Precision Healthcare LLC System and method for identifying complex patients, forecasting outcomes and planning for post discharge care

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200013490A1 (en) * 2017-03-03 2020-01-09 Rush University Medical Center Patient predictive admission, discharge, and monitoring tool
US20200211701A1 (en) * 2018-12-31 2020-07-02 Cerner Innovation, Inc. Decision Support Tool For Determining Patient Length Of Stay Within An Emergency Department
US20210098090A1 (en) * 2019-09-30 2021-04-01 GE Precision Healthcare LLC System and method for identifying complex patients, forecasting outcomes and planning for post discharge care

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120260940A (zh) * 2025-06-09 2025-07-04 中国人民解放军西部战区总医院 一种老年综合征智能科研数据管理方法及系统

Similar Documents

Publication Publication Date Title
Tomašev et al. Use of deep learning to develop continuous-risk models for adverse event prediction from electronic health records
US11664097B2 (en) Healthcare information technology system for predicting or preventing readmissions
Banerjee et al. Development and performance of the pulmonary embolism result forecast model (PERFORM) for computed tomography clinical decision support
US20210005321A1 (en) System and method for predicting patient risk outcomes
US12170148B2 (en) Learning platform for patient journey mapping
Sesen et al. Lung Cancer Assistant: a hybrid clinical decision support application for lung cancer care
Carrell et al. Improving methods of identifying anaphylaxis for medical product safety surveillance using natural language processing and machine learning
Chava Revolutionizing Healthcare Systems with Next-Generation Technologies: The Role of Artificial Intelligence, Cloud Infrastructure, and Big Data in Driving Patient-Centric Innovation
WO2025106998A1 (fr) Procédés et systèmes de réglage optimal de soins pour des patients à l'aide d'un apprentissage automatique
US20240194351A1 (en) Tool for predicting prognosis and improving survival in covid-19 patients
US20240194354A1 (en) Tool for predicting prognosis and improving survival of patients
Ikhalea et al. A Conceptual Framework for AI-Driven Early Detection of Chronic Diseases Using Predictive Analytics
Neysiani et al. Data science in health informatics
Jamshidi et al. A Hybrid Health Journey Recommender System Using Electronic Medical Records.
AlBashayreh Evaluating patient-centered outcomes in palliative care and advanced illness using natural language processing and big data analytics
Landi et al. The evolution of mining electronic health records in the era of deep learning
Yan et al. Technology road mapping of two machine learning methods for triaging emergency department patients in Australia
Wang et al. Learning and predicting from dynamic models for COVID-19 patient monitoring
Alnaber Comparative Analysis of Prediction Models for Diabetic Patient Readmission Using Explainable AI for Feature Selection and Two-Stage Optimization Techniques
Luz et al. Machine Learning in Healthcare: Analyzing Performance of Algorithms for Diabetes Risk Prediction
Suganthi et al. Data Analytics in Healthcare Systems–Principles, Challenges, and Applications
Sharma et al. Transforming healthcare with AI: an adequate method for diabetes prediction using machine learning techniques
Ozery-Flato et al. Characterizing Subpopulations with Better Response to Treatment Using Observational Data–an Epilepsy Case Study
dos Santos Improving the Robustness of Multimodal AI with Asynchronous and Missing Inputs
Demigha The Complexity of Medical and Radiological Imaging Data and Their Difficult Daily Management

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24892451

Country of ref document: EP

Kind code of ref document: A1