DeepSOFA: A Real-Time Continuous Acuity Score Framework using Deep
Learning
Authors: Benjamin Shickel1, Tyler J. Loftus2, Tezcan Ozrazgat-Baslanti3,5, Ashkan Ebadi3,5,
Azra Bihorac3,5*, Parisa Rashidi4,5
Affiliations:
1

Department of Computer and Information Science and Engineering, University of Florida,
Gainesville, FL, USA.
2

Department of Surgery, University of Florida, Gainesville, FL, USA.

3

Department of Medicine, University of Florida, Gainesville, FL, USA.

4

Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA.

5

Precision and Intelligent Systems in Medicine (PRISMAP), University of Florida, Gainesville,
FL, USA.
*Azra Bihorac MD MS, Department of Medicine, Division of Nephrology, Hypertension, and
Renal Transplantation, PO Box 100224, Gainesville, FL 32610-0254. Telephone: (352) 2739009; Fax: (352) 392-5465; Email: abihorac@medicine.ufl.edu.
One Sentence Summary: By leveraging state-of-the-art deep learning techniques, we developed
a novel ICU acuity score (DeepSOFA) that accurately predicts in-hospital mortality at any hour
during a patient’s ICU trajectory based on real-time trends in physiological time series.
Abstract: Traditional methods for assessing illness severity and predicting in-hospital mortality
among critically ill patients require manual, time-consuming, and error-prone calculations that
are further hindered by the use of static variable thresholds derived from aggregate patient
populations. These coarse frameworks do not capture time-sensitive individual physiological
patterns and are not suitable for instantaneous assessment of patients’ acuity trajectories, a
critical task for the ICU where conditions often change rapidly. Furthermore, they are ill-suited
to capitalize on the emerging availability of streaming electronic health record data. We propose
a novel acuity score framework (DeepSOFA) that leverages temporal patient measurements in
conjunction with deep learning models to make accurate assessments of a patient’s illness
severity at any point during their ICU stay. We compare DeepSOFA with SOFA baseline models
using the same predictors and find that at any point during an ICU admission, DeepSOFA yields
more accurate predictions of in-hospital mortality.

Introduction
Patients entering a hospital’s intensive care unit (ICU) often have a life-threatening
condition or the propensity to develop one at any moment. Early recognition of evolving degrees
of illness severity in the ICU is invaluable. Timely and accurate illness severity assessments
may identify patients in need of life-saving interventions and inform shared decision-making
processes among providers, patients, and families, promoting optimal resource utilization and
value of care. Unfortunately, there are few accurate and flexible methods for determining realtime illness severity. Our primary aim is the development of an acuity score framework that
encompasses the full scope of a patient’s physiological measurements over time to generate
dynamic in-hospital mortality predictions.
One of the most cited metrics in assessing ICU patient acuity is the Sequential Organ
Failure Assessment (SOFA) score (1). The SOFA score considers six different organ or organ
system components (cardiovascular, respiratory, nervous, liver, coagulation, and renal) and uses
worst measurements over a given interval (typically 24 hours) in conjunction with static value
thresholds to assign a numerical score for each component. The sum of these component scores
yields the overall SOFA score, which can be used to assess illness severity and predict mortality
(2–4). While the SOFA score is easy to use and provides a quick assessment of a patient’s
overall condition and mortality risk, its accuracy is hindered by the use of fixed cutoff points for
each component score. As an alternative to existing linear models (5–7), we seek more precise
and accurate models using granular data and temporal trends, with predictions that update in
real-time as new data becomes available. Badawi et al. (8) recently performed hourly
recalculations of several acuity scores for ICU patients at 208 hospitals in the Philips eICU
Research Institute database, reporting that hourly SOFA scores accurately predicted ICU
mortality with area under the receiver operating characteristic curve (AUC) 0.86. This sentinel
finding suggests that hourly acuity score calculations may be used for predictive analytics with
reasonable accuracy and may promote early detection of changes in illness severity despite the
likely presence of transient and self-limited fluctuations in clinical data (9).
Our solution uses deep learning, a branch of machine learning referring to sets of
machine learning models and architectures that learn optimal features directly from the data
itself, focusing on increasingly complex representations of raw data by combining layers of
nonlinear data transformations (10, 11). Deep learning models automatically discover latent
patterns and form high-level representations from large amounts of raw data without the need for
manual feature extraction based on a priori knowledge or practitioner intuition, which is timeconsuming and error-prone. These methods have revolutionized natural language processing,
speech recognition, and computer vision. Deep learning models have also been applied within
healthcare, using computer vision for identifying diabetic retinopathy (12) and recognizing skin
cancer with accuracy similar to that of a board-certified dermatologist (13), and using deep
models to predict the onset of heart failure (14), pain responses (15), and ICU mortality (16).
Deep learning is especially well-suited to real-time mortality prediction for its ability to
capitalize on the emerging availability of streaming electronic health record data, especially
given the recent success of Badawi et al. (8) in using hourly acuity score updates to predict
mortality. Ideally, mortality prediction in the ICU may combine real-time data with deep
learning prediction models, though this approach has not been previously reported.
We compared deep learning-based mortality prediction models trained on hourly
measurements to baseline models using traditional SOFA score definitions. Contrary to the

exclusive use of worst values during a given time period in the original SOFA score, we used the
entirety of a patient’s data stream over the same time period. The DeepSOFA model employs a
recurrent neural network (RNN), a deep learning algorithm suited for multivariate temporal input
data. All experiments were performed on medical and surgical ICU patients at our institution
with external validation performed within the MIMIC-III dataset and a prospective cohort of
ICU patients at our institution. Model efficacy was assessed at several different time-points
during each individual ICU admission. Because continuous measurements may provide more
granularity than daily measurements in representing acuity and clinical trajectory and because
deep models are capable of learning the complex, non-linear associations among input variables
in an automated fashion, we hypothesized that DeepSOFA would yield greater accuracy in
predicting mortality compared with traditional SOFA models, that this advantage would be
maintained when automated in a prospective fashion in real time.
Results
Real-time mortality prediction
Given our focus on real-time acuity assessment and mortality prediction, we place equal
emphasis on model performance at every time-point along a patient’s ICU trajectory. Because
measurements were resampled on an hourly basis, mortality predictions and area under the
receiver operating characteristic curve (AUC) were generated at every hour beginning with ICU
admission and continuing until hospital discharge or death. Performance metrics were assessed at
several time-points as proportions of the total length of each encounter.
Figure 1 shows ROC curves and corresponding AUC after a percentage (in quartiles) of
each test encounter’s time series has been incorporated by each model, illustrating the
performance progression of each model after viewing increasing amounts of test data.
Percentages are based on the total length of each encounter. For example, for an encounter
lasting 120 hours, AUC is calculated at 30, 60, 90, and 120 hours, illustrated as 25%, 50%, 75%,
and 100% of the encounter, respectively.
Figure 2 shows the calculation of AUC relative to the time of hospital discharge or death.
For an encounter of length T hours, AUC at zero hours before discharge is calculated after seeing
the entirety of the encounter’s time series, i.e. at t = T. Similarly, AUC at 100 hours before
discharge is calculated for each encounter at t = (T - 100), where T is different for each test
encounter. For encounters shorter than 100 hours, we use the encounter’s first mortality
prediction at t = 0 for calculating overall AUC. In all models, accuracy increased over time.

Figure 1. Receiver operating characteristic curves and AUC after 25%, 50%, 75%, and 100% of each test encounter’s time series has been seen. The
development cohort used for each DeepSOFA model is shown in parentheses. Shaded regions represent 95% confidence intervals based on 1000
bootstrapped iterations.

Figure 2. AUC calculated at every hour during the final 100 hours of each encounter. The development cohort used for each DeepSOFA model is shown
in parentheses. For encounters shorter than the specified time, AUC was assessed using the first available prediction. Shaded regions represent 95%
confidence intervals based on 1000 bootstrapped iterations.

DeepSOFA score
While the model performance data reported in the previous section is important for
assessing the overall effectiveness and validity of deep models for predicting in-hospital
mortality among ICU patients, the knowledge gained by deep learning models must be translated
into an actionable form suitable for clinical application. Ideally, the deep model would provide a
single value representing the probability of in-hospital mortality, and would provide this value at
any time-point during an ICU admission using all currently available data that. Thus, similar to
the traditional SOFA score, we derived a deep learning-based measure of illness severity referred
to as the DeepSOFA score. Whereas SOFA is a coarse acuity score based on fixed measurement
thresholds, the DeepSOFA score is the culmination of more complex temporal models that learn
micro-trends and patterns from the entirety of individual patient time-series data. The
DeepSOFA score can be applied at any time during a patient’s ICU stay and captures available
information from the entire duration of the encounter.
The DeepSOFA score was derived from the output of the final layer of the recurrent
neural network. Because the final layer uses the sigmoid activation function, the DeepSOFA
score represents the probability of mortality spanning a continuous range from 0.0 to 1.0, with
1.0 representing 100% probability of in-hospital mortality.
Figure 3 illustrates the evolution of the DeepSOFA score and traditional SOFA scores as
functions of the percentage of each testing encounter available to the models. As more data
becomes available, the predicted probability of mortality is expected to increase for death
encounters and decrease for survival encounters. Unexpectedly, for several models, including
the deep models and the logistic 24h SOFA model, early changes in mortality prediction \ were
often inverse to the outcome, i.e. predicted mortality decreased among non-survivors and
increased among survivors. After the initial 5-10% of ICU encounter data was available,
DeepSOFA predicted probability of mortality increased for death encounters and decreased for
survival encounters. DeepSOFA overestimated the probability of death among survivors
compared with most baseline SOFA models, but projected the highest probability of death
among non-survivors.
Figure 4 shows a frequency distribution of acuity scores generated by all models at the
time of death for all non-survivors, meaning the entirety of each test encounter was available to
all models. Intuitively, the final acuity score should be high for patients whose ICU stay resulted
in death. The deep models featured the greatest proportion of patients 90%-100% predicted inhospital mortality.

Figure 3. Mean predicted mortality over time by outcome. The development cohort used for each DeepSOFA model is shown in parentheses. Shaded
regions represent 95% confidence intervals based on 100 bootstrapped iterations.

Figure 4. Distribution of final predicted patient mortality for non-survivors at time of death. The development cohort used for each DeepSOFA model is
shown in parentheses.

Discussion
Our purpose was to explore the application of deep learning methodology to the task of
predicting in-hospital mortality among ICU patients using components of the Sequential Organ
Failure Assessment (SOFA) score, a well-known measure of patient acuity, comparing deep
models to traditional SOFA models. The DeepSOFA recurrent neural network was trained on
full temporal time series, as opposed to SOFA’s worst value thresholds, and compared with a
baseline model based on bedside inference of traditional SOFA scores.
DeepSOFA achieved superior as accuracy in predicting in-hospital mortality compared
with traditional SOFA models. Despite early losses in discrimination between survivors and nonsurvivors, DeepSOFA was more accurate than traditional SOFA model at the earliest time-points
and maintained a relatively proportional advantage over other models throughout the duration of
ICU encounters. The superior accuracy of deep models is partially attributable to their ability to
learn latent structure and complex relationships from low-level data, including temporal trends in
the case of recurrent neural networks. Due to their internal memory mechanisms, recurrent
mortality prediction models based on sequential time series learn temporal patterns from
potentially long-term dependencies in time series variables. These often-complex relationships
are lost in linear models, especially when applying worst value thresholds like SOFA score
calculations. The memory of a GRU network also allows models to be updated in real-time based
on single sets of measurements, without needing to retrain the entire model.
Our DeepSOFA score, the acuity score derived from the output of our deep network, also
shows benefits over the traditional SOFA score in recognizing declining patient health. Figure 5
shows an example non-survivor from the UFHealth cohort, from time of ICU admission to time
of death. Compared with the traditional SOFA model, DeepSOFA was able to assign
increasingly more severe acuity to the patient as early as 100 hours before discharge, compared
with traditional SOFA scores which only reflected illness severity during the final 20 hours of
the encounter when the window opportunity for critical intervention had drastically shortened.

Figure 5. DeepSOFA scores, predicted mortality from traditional SOFA scores, and commonly measured vital signs measured at each hour for an
example non-survivor from the UFHealth cohort. The dashed vertical line indicates time of death, which for this patient occurred 111 hours after initial
ICU admission.

Previous work has often employed multivariable regression models in predicting
mortality for ICU patients. The Simplified Acute Physiology Score (SAPS) (17, 18) and
Mortality Probability Model (MPM) (19) have each been used to predict in-hospital mortality
using data available within one hour of ICU admission. Afessa et al. (20) evaluated the accuracy
of SAPS III and MPM III in predicting mortality among large cohorts (SAPS III: 16,784 patients
from 281 hospitals across five continents; MPM III: 124,855 patients from 98 hospitals in the
United States), and found that each had strong accuracy (SAPS III: AUC 0.85, MPM III: AUC
0.82). In the same study, the Acute Physiology and Chronic Health Evaluation (APACHE) IV
(21) score was used with data from the first 24 hours of ICU admission for 110,558 patients from
45 hospitals in the United States, and achieved AUC 0.88. Although these methods have
produced reasonably accurate predictions of in-hospital mortality, their accuracy is inferior to
that of deep models, and their clinical application is cumbersome compared with automated
models with the capacity for integration of streaming electronic health record data.
This study is limited by using data from hospital single country. Patient populations and
practice patterns from our institution and MIMIC-III contributors may differ from that of other
ICU settings, limiting the generalizability of our findings. Future studies should apply the
DeepSOFA score to live streaming electronic health record data and investigate the feasibility
and efficacy of expanding the variables included in the deep model beyond those included in
SOFA score components.
In summary, deep learning models trained on time series data were more accurate than
SOFA models for predicting in-hospital mortality among ICU patients. These results suggest that
the SOFA score can be augmented with more nuanced and intelligent mechanisms for assessing
patient acuity. Deep learning technology may be used to generate accurate real-time prognostic
data to identify patients in need of life-saving interventions and inform shared decision-making
processes among providers, patients, and families, promoting optimal resource utilization and
value of care.
Materials and Methods
Study design
Using the University of Florida Health (UFH) Integrated Data Repository as Honest
Broker, we created a single-center longitudinal dataset (referred to as UFHealth) that was
extracted directly from the electronic medical record system, derived from 553,808 distinct
hospital encounters related to 84,355 unique patients requiring hospital admission at University
of Florida Health between June 1, 2011 and April 1, 2016. The dataset includes structured and
unstructured clinical data, demographic information, vital signs, laboratory values, medications,
diagnoses, and procedures. Among these hospital encounters, there were 36,563 distinct
encounters related to 28,822 unique patients in which the patient entered the ICU at least once
before discharge. For purposes of this study, an ICU encounter began when a subject first
entered the ICU, with the first mortality predictions generated one hour after ICU admission and
ended when the subject left the hospital. Prediction modeling was limited to data accrued during
ICU admission. For patients transferred out of the ICU to an intermediate care unit or hospital
ward, the end-point of death or hospital discharge was assessed at the conclusion of that hospital
admission. This study was approved by the University of Florida Institutional Review Board and

Privacy Office. For external validation, identical selection criteria were applied to the publicly
available MIMIC-III1 database of critical care patients, arriving with a final cohort consisting of
57,786 distinct encounters from 46,520 unique patients. Both cohorts are summarized in Table 1.

Number of patients
Age, mean (SD)
Female gender, n (%)
Body mass index, mean (SD)
Charlson comorbidity index, median (25th, 75th)
Hospitalized days, median (25th, 75th)
Race, n (%)
White
African-American
Hispanic
Asian
Other/Missing
ICU type, n (%)
Medical ICU
Surgical ICU
Neurological ICU
Cardiac ICU
Burn ICU
Pediatric ICU
Maximum SOFA score, mean (SD)
Cardiovascular
Respiratory
Central nervous system
Coagulation
Liver
Renal
Total SOFA score

UFHealth
(n = 36563)
28822
58.89 (17.22)
16328 (44.66%)
29.22 (47.20)
2.00 (0.00, 3.00)
6.23 (3.38, 11.88)

MIMIC
(n = 57164)
38606
74.48 (53.76)
29880 (43.25%)
28.36 (8.19)
----6.78 (4.04, 12.06)

27768 (75.95%)
5922 (16.20%)
1189 (3.25%)
254 (0.69%)
1430 (3.91%)

49249 (71.28%)
6456 (9.34%)
2373 (3.43%)
1594 (2.31%)
9420 (13.63%)

9446 (25.83)
11797 (32.26)
7693 (21.04)
6591 (18.03)
678 (1.85)
358 (0.98)

26400 (38.21)
19487 (28.20)
----23205 (33.59)
---------

1.23 (1.13)
0.84 (1.26)
1.48 (1.50)
0.82 (1.11)
0.26 (0.72)
3.22 (1.14)
6.40 (3.52)

1.48 (1.21)
1.39 (1.54)
2.14 (1.60)
0.76 (1.00)
0.42 (0.94)
3.61 (0.88)
8.11 (3.91)

Table 1. Demographic summary of both UFHealth and MIMIC cohorts.

Data processing
For both cohorts, we extracted all raw time series data for the 13 variables outlined in the
original SOFA score. Although additional variables would have likely improved mortality
prediction performance, we limited the deep learning models to exclusively use SOFA variables
to facilitate direct comparison to traditional SOFA baseline models. Variable time series began at
24 hours before ICU admission and ended at time of hospital discharge. The primary outcome of
interest was in-hospital mortality.
Following variable extraction, measurement outliers were removed using the same fixed
ranges used by MIMIC. Raw time series were then resampled to an hourly frequency, taking the
mean value where multiple measurements existed for the same encounter during the same 1-hour
window. Following resampling, gaps in the resulting time series were filled by forward1

https://mimic.physionet.org

propagating previous values for vital signs and laboratory tests and substituting 0 for vasopressor
rates and ventilation status. For all remaining missing values, including when a variable was
missing entirely from an encounter, we imputed clinically normal ranges defined by experts.
Table 2 includes a description of the variables, including summary statistics and preprocessing
and imputation rules.

Encounters Missing, % *
Variable

UFHealth

Mean arterial blood pressure, mmHg
3.8
Dopamine, μg/kg/min
96.4
Dobutamine, μg/kg/min
97.6
Epinephrine, μg/kg/min
93.6
Norepinephrine, μg/kg/min
88.5
Partial pressure of oxygen (PaO2),45.2
mmHg
Fraction of inspired oxygen (FiO2),48.1
%
Mechanical ventilation, binary indicator
73.8
Bilirubin, mg/dL
42.2
Platelets, count x 10^3/mm^3
4.3
Creatinine, mg/dL
3.7
Urine output, mL
2.2
GCS, score
5.8

Value, Mean (SD) †

Frequency (Hours), Mean (SD) †‡ǁ

Valid Value Range §

MIMIC

UFHealth

MIMIC

UFHealth

MIMIC

Low, High

14.4
93.9
98.3
96.8
87.5
42.8
45.6
61.4
45.0
6.7
13.5
10.8
14.6

83.1 (19.6)
6.2 (4.2)
4.7 (2.7)
0.1 (0.2)
0.2 (0.3)
129.8 (60.5)
41.3 (14.4)
1.0 (0.0)
2.2 (5.0)
217.5 (152.0)
1.3 (1.3)
168.5 (195.5)
12.3 (3.2)

78.9 (17.2)
6.7 (17.9)
5.1 (4.0)
0.0 (0.2)
0.2 (0.7)
125.8 (77.5)
48.9 (22.1)
1.0 (0.0)
4.6 (7.1)
237.6 (158.2)
1.5 (1.5)
123.1 (2422.9)
11.5 (3.7)

1.1 (2.9)
1.8 (39.2)
1.5 (7.8)
1.7 (38.3)
1.8 (25.1)
10.5 (56.1)
1.6 (16.0)
92.6 (309.7)
45.9 (90.5)
18.8 (14.9)
18.2 (14.7)
1.6 (5.5)
3.0 (5.8)

0.8 (4.4)
1.7 (19.2)
1.4 (14.4)
1.5 (15.7)
1.9 (18.9)
8.6 (28.8)
2.1 (10.1)
99.9 (155.1)
31.6 (52.1)
19.0 (28.2)
17.6 (21.4)
1.6 (9.0)
3.4 (11.6)

(0, 300]
(0, ∞)
(0, ∞)
(0, ∞)
(0, ∞)
(0, 800]
[20, 100]
----[0, 150]
[0, 10000]
[0, 150]
----[3, 15]

Time-Series Imputation ¶
Forward Propagation
Imputation
Before Imputation?
Value
Yes
85.0
No
0.0
No
0.0
No
0.0
No
0.0
Yes
97.5
Yes
21.0
No
0.0
Yes
0.5
Yes
300.0
Yes
1.0
Yes
60.0
Yes
15.0

* We denote a variable missing from an encounter if there is no recorded value from time of ICU admission to hospital discharge.
† Measurement value and frequency are computed for recorded values only, and do not consider encounters missing specified variable.
‡ Frequency computed for encounters which included at least two recorded values for specified variable. Mean frequency computed first on a per-encounter basis, then across all encounters.
ǁ Frequency for dopamine, dobutamine, epinephrine, norepinephrine, and mechanical ventilation based on start time of implied duration.
§ Values falling outside of specified range are removed. Parentheses imply exclusivity, square brackets imply inclusivity.
¶ If specified, we carry previous values forward to fill in gaps in the hourly-sampled time series. For all variables, any remaining gaps are filled using specified imputation value. If a variable is missing from an
encounter, we use the specified imputation value at every hour.

Table 2. Summary of variables.

Models
For predicting in-hospital mortality, we used a recurrent neural network (RNN) with
gated recurrent units (GRU), a deep learning model tailored for working with temporal data.
Internally, the RNN continuously updates its parameters based on multivariate inputs from both
the current time step and previous time steps. As such, a mortality prediction incorporates
patterns detected across the entirety of an ICU encounter, with recognition of longer-range
temporal relationships aided by the addition of GRUs. We hypothesized that variable trends
automatically extracted by the RNN would yield higher predictive performance than worst-value
SOFA calculations.
We trained a separate deep learning model on the development set of each cohort and
used both models for prediction on each of the two testing cohorts. When training the GRU
models, we used 10% of the development data as validation for early termination of training.
We compared our deep models with mortality predictions derived from overall published
mortality rate correlations (2). At any hour during a patient encounter, this model associated the
current maximum SOFA score beginning at ICU admission with a predicted probability of
mortality. At each hour, SOFA scores were calculated based on the previous 24 hours of data.
We refer to our baseline model as Maximum SOFA/Bedside, as the known mortality rates we
use for death probabilities are what bedside practitioners would use to infer patient acuity from
calculated SOFA score.
Evaluation
For each of our two ICU cohorts, we divided each set of encounters into a random split
containing 70% for development and 30% for testing. All results were reported based on the
testing set of each cohort. All models were trained on the development set of each cohort,
including cross-institution experiments in which only the training cohort of the source dataset is
used for evaluation on the target dataset’s test cohort.
For all models in our experiments, we enforced a prediction at every hour during test
cohort encounters, beginning at one hour after ICU admission and ending at the time of final
hospital discharge or death. For all metrics, we report 95% confidence intervals using 100
bootstrapped iterations of sampling mortality prediction probabilities with replacement.
Supplementary Materials
Fig. S1. DeepSOFA component set AUC comparison during final 100 hours of each encounter.
Fig. S2. DeepSOFA component set AUC comparison by percentage of each encounter seen.
References and Notes:
1. J. L. Vincent, R. Moreno, J. Takala, S. Willatts, A. De Mendonca, H. Bruining, C. K.
Reinhart, P. M. Suter, L. G. Thijs, The SOFA (Sepsis-related Organ Failure Assessment) score to
describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems
of the European Society of Intensive Care Medicine, Intensive Care Med 22, 707–710 (1996).

2. F. Ferreira, D. Bota, A. Bross, C. Mélot, J. Vincent, Serial evaluation of the sofa score to
predict outcome in critically ill patients, JAMA 286, 1754–1758 (2001).
3. D. Mendonca, Use of the SOFA score to assess the incidence of organ dysfunction / failure in
intensive care units : Results of a multicenter , prospective study, 26, 1793–1800 (1998).
4. L. Minne, A. Abu-Hanna, E. de Jonge, Evaluation of SOFA-based models for predicting
mortality in the ICU: A systematic review., Crit. Care 12, R161 (2008).
5. S. Kim, W. Kim, R. W. Park, A Comparison of Intensive Care Unit Mortality Prediction
Models through the Use of Data Mining Techniques, 17, 232–243 (2011).
6. G. Meyfroidt, F. Güiza, J. Ramon, M. Bruynooghe, Machine learning techniques to examine
large patient databases, Best Pract. Res. Clin. Anaesthesiol. 23, 127–143 (2009).
7. G. Clermont, D. C. Angus, S. M. DiRusso, M. Griffin, W. T. Linde-Zwirble, Predicting
hospital mortality for patients in the intensive care unit: A comparison of artificial neural
networks with logistic regression models, Crit. Care Med. 29 (2001).
8. O. Badawi, X. Liu, E. Hassan, P. J. Amelung, S. Swami, Evaluation of ICU Risk Models
Adapted for Use as Continuous Markers of Severity of Illness Throughout the ICU Stay., Crit.
Care Med. 46, 361–367 (2018).
9. D. M. Maslove, With Severity Scores Updated on the Hour, Data Science Inches Closer to the
Bedside*, Crit. Care Med. 46, 480–481 (2018).
10. Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521, 436–444 (2015).
11. I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (MIT Press, 2016).
12. V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S.
Venugopalan, K. Widner, T. Madams, J. Cuadros, R. Kim, R. Raman, P. C. Nelson, J. L. Mega,
D. R. Webster, Development and validation of a deep learning algorithm for detection of diabetic
retinopathy in retinal fundus photographs, JAMA - J. Am. Med. Assoc. 316, 2402–2410 (2016).
13. A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, S. Thrun,
Dermatologist-level classification of skin cancer with deep neural networks, Nature 542, 115–
118 (2017).
14. E. Choi, A. Schuetz, W. F. Stewart, J. Sun, Using recurrent neural network models for early
detection of heart failure onset., J. Am. Med. Inform. Assoc. 292, 344–350 (2016).
15. P. Nickerson, P. Tighe, B. Shickel, P. Rashidi, Deep neural network architectures for
forecasting analgesic response, Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBS 2016–
October, 2966–2969 (2016).
16. H. Du, M. M. Ghassemi, M. Feng, The effects of deep network topology on mortality
prediction, 38th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. EMBC 2016 2016–Octob, 2602–
2605 (2016).
17. P. G. H. Metnitz, R. P. Moreno, E. Almeida, B. Jordan, P. Bauer, R. A. Campos, G.
Iapichino, D. Edbrooke, M. Capuzzo, J. R. Le Gall, SAPS 3-From evaluation of the patient to
evaluation of the intensive care unit. Part 1: Objectives, methods and cohort description,
Intensive Care Med. 31, 1336–1344 (2005).
18. R. P. Moreno, P. G. H. Metnitz, E. Almeida, B. Jordan, P. Bauer, R. A. Campos, G.

Iapichino, D. Edbrooke, M. Capuzzo, J. R. Le Gall, SAPS 3 - From evaluation of the patient to
evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital
mortality at ICU admission, Intensive Care Med. 31, 1345–1355 (2005).
19. T. L. Higgins, D. Teres, W. S. Copes, B. H. Nathanson, M. Stark, A. a Kramer, Assessing
contemporary intensive care unit outcome: an updated Mortality Probability Admission Model
(MPM0-III)., Crit. Care Med. 35, 827–35 (2007).
20. B. Afessa, O. Gajic, M. T. Keegan, Severity of Illness and Organ Failure Assessment in
Adult Intensive Care Units, Crit. Care Clin. 23, 639–658 (2007).
21. J. E. Zimmerman, A. a Kramer, D. S. McNair, F. M. Malila, Acute Physiology and Chronic
Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill
patients., Crit. Care Med. 34, 1297–1310 (2006).

Acknowledgments: The authors acknowledge Gigi Lipori, MBA and Paul Nickerson, MS for
assistance with data retrieval. Funding: A.B., T.O.B., and P.R. were supported by R01
GM110240 from the National Institute of General Medical Sciences. A.B. and T.O.B.
were supported by Sepsis and Critical Illness Research Center Award P50 GM-111152
from the National Institute of General Medical Sciences. T.O.B. has received grant
(97071) from Clinical and Translational Science Institute, University of Florida. This
work was supported in part by the NIH/NCATS Clinical and Translational Sciences
Award to the University of Florida UL1 TR000064. T.L. was supported by a postgraduate training grant (T32 GM-008721) in burns, trauma and perioperative injury
awarded by the National Institute of General Medical Sciences (NIGMS). B.S. was
supported by internal funds at J. Crayton Pruitt Family Department of Biomedical
Engineering, University of Florida. The content is solely the responsibility of the authors
and does not necessarily represent the official views of the National Institutes of Health.
The funders had no role in study design, data collection and analysis, decision to publish,
or preparation of the manuscript. A.B., B.S, and T.O.B. had full access to all the data in
the study and take responsibility for the integrity of the data and the accuracy of the data
analysis. The Titan X Pascal partially used for this research was donated by the NVIDIA
Corporation. Author contributions: BS, TL, AB, and TOB had full access to the data in
the study and take responsibility for the integrity of the data and the accuracy of the data
analysis. Study design performed by BS, TL, AB, and PR. BS and TL drafted the
manuscript. Analysis was performed by BS, TL, TOB, AB, and PR. Funding was
obtained by AB and PR. Administrative, technical, material support was provided by AB
and PR. Study supervision was performed by AB and PR. All authors contributed to the
acquisition, analysis, and interpretation of data. All authors contributed to critical revision
of the manuscript for important intellectual content. Competing interests: The authors
report no conflict of interest. Data and materials availability: The MIMIC-III database
is publicly available. UFHealth cohort data are available from the University of Florida
Institutional Data Access / Ethics Committee for researchers who meet the criteria for
access to confidential data and may require additional IRB approval.

Supplementary Materials:

Figure S1. DeepSOFA component set AUC comparison during final 100 hours of each encounter. A separate recurrent neural network was trained for
each set of variables. Shaded regions represent 95% confidence intervals based on 100 bootstrapped iterations.

Figure S2. DeepSOFA component set AUC comparison by percentage of each encounter seen. A separate recurrent neural network was trained for
each set of variables. Shaded regions represent 95% confidence intervals based on 100 bootstrapped iterations.

