Anthony J. Weekes,Jaron D. Raper,Dasia Esener,Jillian Davison,Jeremy S. Boyd,Christopher Kelly,Jason T. Nomura,Alyssa M. Thomas,Kathryn Lupez,Carly A. Cox,Patrick M. Ockerse,Stephen Leech,Jakea Johnson,Eric Abrams,Kathleen Murphy,Nathaniel S. O'Connell
Comparing predictive performance of pulmonary embolism risk stratification tools for acute clinical deterioration
clinical deterioration,health care,outcome assessment,prognosis,pulmonary embolism,risk assessment
Abstract Objectives Existing pulmonary embolism (PE) risk scores were developed to predict death within weeks, but not more proximate adverse events. We determined the ability of 3 PE risk stratification tools (simplified pulmonary embolism severity index [sPESI], 2019 European Society of Cardiology guidelines [ESC], and PE short‐term clinical outcomes risk estimation [PE‐SCORE]) to predict 5‐day clinical deterioration after emergency department (ED) diagnosis of PE. Methods We analyzed data from six EDs on ED patients with confirmed PE. Clinical deterioration was defined as death, respiratory failure, cardiac arrest, new dysrhythmia, sustained hypotension requiring vasopressors or volume resuscitation, or escalated intervention within 5 days of PE diagnosis. We determined sensitivity and specificity of sPESI, ESC, and PE‐SCORE for predicting clinical deterioration. Results Of 1569 patients, 24.5% had clinical deterioration within 5 days. sPESI, ESC, and PE‐SCORE classifications were low‐risk in 558 (35.6%), 167 (10.6%), and 309 (19.6%), respectively. Sensitivities of sPESI, ESC, and PE‐SCORE for clinical deterioration were 81.8 (78, 85.7), 98.7 (97.6, 99.8), and 96.1 (94.2, 98), respectively. Specificities of sPESI, ESC, and PE‐SCORE for clinical deterioration were 41.2 (38.4, 44), 13.7 (11.7, 15.6), and 24.8 (22.4, 27.3). Areas under the curve were 61.5 (59.1, 63.9), 56.2 (55.1, 57.3), and 60.5 (58.9, 62.0). Negative predictive values were 87.5 (84.7, 90.2), 97 (94.4, 99.6), and 95.1 (92.7, 97.5). Conclusions ESC and PE‐SCORE were better than sPESI for detecting clinical deterioration within 5 days after PE diagnosis.
Comparing predictive performance of pulmonary embolism risk stratification tools for acute clinical deterioration

Existing pulmonary embolism (PE) risk scores were developed to predict death within weeks, but not more proximate adverse events. We determined the ability of 3 PE risk stratification tools (simplified pulmonary embolism severity index [sPESI], 2019 European Society of Cardiology guidelines [ESC], and PE short‐term clinical outcomes risk estimation [PE‐SCORE]) to predict 5‐day clinical deterioration after emergency department (ED) diagnosis of PE.

We analyzed data from six EDs on ED patients with confirmed PE. Clinical deterioration was defined as death, respiratory failure, cardiac arrest, new dysrhythmia, sustained hypotension requiring vasopressors or volume resuscitation, or escalated intervention within 5 days of PE diagnosis. We determined sensitivity and specificity of sPESI, ESC, and PE‐SCORE for predicting clinical deterioration.

Of 1569 patients, 24.5% had clinical deterioration within 5 days. sPESI, ESC, and PE‐SCORE classifications were low‐risk in 558 (35.6%), 167 (10.6%), and 309 (19.6%), respectively. Sensitivities of sPESI, ESC, and PE‐SCORE for clinical deterioration were 81.8 (78, 85.7), 98.7 (97.6, 99.8), and 96.1 (94.2, 98), respectively. Specificities of sPESI, ESC, and PE‐SCORE for clinical deterioration were 41.2 (38.4, 44), 13.7 (11.7, 15.6), and 24.8 (22.4, 27.3). Areas under the curve were 61.5 (59.1, 63.9), 56.2 (55.1, 57.3), and 60.5 (58.9, 62.0). Negative predictive values were 87.5 (84.7, 90.2), 97 (94.4, 99.6), and 95.1 (92.7, 97.5).

ESC and PE‐SCORE were better than sPESI for detecting clinical deterioration within 5 days after PE diagnosis.

The emergency department (ED) decision to admit a patient with acute pulmonary embolism (PE) is usually based on the patient's predicted risk of death or clinical deterioration. Established risk stratification tools, such as the pulmonary embolism severity index (PESI), simplified pulmonary embolism severity index (sPESI), and European Society of Cardiology (ESC) guidelines, have been validated to predict death and limited outcomes within 30 days after ED diagnosis of PE, but not short‐term clinical deterioration.

Some clinicians argue that risk of clinical deterioration within 5 days is more important than risk of death alone after an acute PE. , , A PE risk stratification tool that focuses on shorter‐term, clinically relevant outcomes enables informed decisions that reduce clinical uncertainty and prioritize patient safety. The risk of longer term outcome(s) does not inform a clinician's immediate concerns about clinical deterioration, need for hospital‐based interventions, anticoagulation safety, or disposition. Currently, most patients classified low‐risk by sPESI are admitted, which suggests clinicians are uncomfortable discharging PE patients with the current risk stratification tools. ,

We derived and validated PE‐SCORE in a previously published report. In the current study, we compared the ability of sPESI, ESC, and PE‐SCORE to predict 5‐day clinical deterioration after ED‐diagnosed PE.

We performed a secondary analysis of pooled data from the Pulmonary Embolism Short‐term Clinical Outcomes registry and Short‐term Clinical Deterioration after Acute Pulmonary Embolism registry (ClinicalTrials.gov NCT02883491 and NCT03915925, respectively). The registry data were prospectively collected from August 2016 to November 2020 from six geographically separate academic EDs (Salt Lake City, UT; Nashville, TN; Charlotte, NC; Orlando, FL; Newark, Delaware; San Diego, CA). The two registry databases were populated by the same six EDs listed above using the same inclusion and exclusion criteria for both registries. Patients 18 years or older with image‐confirmed acute PE diagnosed within 12 h of initial presentation were eligible. Exclusion criteria included: age less than 18 years old, refusal to participate in the study, PE determined to be chronic, received empiric anticoagulation or underwent an escalated intervention more than 12 h preceding the PE diagnosis, incidental PE (not related to the primary diagnostic workup or ED presentation) and segmental/subsegmental PE. Both registries involved the same predictors and outcomes. The primary outcome was short‐term clinical deterioration, defined as death, respiratory failure, cardiac arrest, new dysrhythmia, sustained hypotension receiving catecholamine or volume resuscitation support, or escalated intervention within 5 days of ED‐diagnosed PE. Deaths were delineated as PE‐related and not PE‐related and determined by the site investigator. We defined respiratory failure as respiratory distress requiring emergent interventions and mechanical ventilation. These interventions included unscheduled noninvasive positive pressure ventilation, intubation, and surgical airway management. Cardiac arrest was any pulseless rhythm requiring cardiopulmonary resuscitation. New dysrhythmias were defined as cardiac rhythms not present on initial evaluation, and included bradycardia, stable ventricular tachycardia, supraventricular tachycardia, and atrial flutter or atrial fibrillation with rapid ventricular response. Hypotension was defined as sustained systolic blood pressures <90 mm Hg treated with >500 milliliters fluid or adrenergic agents.

Determining risk of short‐term clinical deterioration in patients diagnosed with pulmonary embolism in the emergency department is important for clinician decision‐making. This secondary analysis of multicenter registry data demonstrated greater accuracy for identifying patients at low‐risk for 5‐day clinical deterioration or death using risk stratification tools that incorporate assessment of right ventricular dysfunction.

Eligible patients were followed for 5‐day clinical deterioration events and the following data were entered into the registries: demographics, PE risk factors, comorbidities, and clinical findings. During the ED evaluation, clinicians performed point‐of‐care cardiac ultrasound and we collected laboratory measurements of cardiac biomarkers (troponin and brain natriuretic peptide levels). Institutional review boards (IRBs) of participating sites approved both registry studies. PE‐SCORE was developed and validated from these registry databases. Sample size determinations were previously reported.

For this study, we included all patients from enrolled in the two registries. We excluded patients missing data required for any of the three risk stratification tools.

Table 1 provides a comparison of the three PE risk stratification tools: sPESI, ESC, and PE‐SCORE. sPESI is a prognostic tool that was developed from 11 candidate variables; the final tool had six components, which were used in this study. Those components were age, systolic blood pressure, heart rate, cardiopulmonary disease, history of cancer, and oxygen saturation. A total of zero points was classified as low‐risk. A total of 1 to 6 points on the 6‐point sPESI tool was considered not low‐risk of clinical deterioration within 5 days. We used sPESI as the reference standard in this study, which was coded as a binary categorical predictor variable. The ESC risk stratification tool combines sPESI or PESI with additional components, which include hemodynamic instability, right ventricle (RV) dysfunction by echocardiography or computed tomography (CT) pulmonary angiography, and cardiac troponin levels. ESC has 4 risk classifications: high, intermediate‐high, intermediate, and low. For purposes of this secondary analysis, we coded ESC as a binary variable (low‐risk of clinical deterioration within 5 days vs. not low‐risk as shown in Table 1). Low‐risk ESC was defined as being low‐risk by sPESI criteria (0 points) and the absence of RV dilatation by CT or echocardiography. Not low‐risk was defined by hemodynamic instability on presentation or sPESI greater than 0 points or RV dilatation by CT or echocardiography or troponin elevation. We developed PE‐SCORE from 138 candidate variables, of which 9 components made up the final tool: creatinine >2.0 mg/dL, dysrhythmia at presentation, suspected systemic infection, systolic blood pressure <100 mm Hg, heart rate (<50 or >110 bpm), syncope, medical or social reason for hospitalization, bedside echocardiography features of RV dilatation with or without septal deviation or RV systolic dysfunction, and CT RV:LV ratio of 1.0 or more. As a logistic regression and points tool, PE‐SCORE was validated with area under the receiver operating characteristic curve (AUC) of 0.78–0.80. A score of 0 points is low‐risk of 5‐day clinical deterioration and scores of 1–10 points are not low‐risk. Both ESC and PE‐SCORE include imaging assessment for RV dilatation. We used previously reported definitions for RV dilatation for ESC and PE‐SCORE. , , Severe RV dilatation was determined by visual estimation or measurement of an RV:LV ratio of 1.0 or greater in one or more transthoracic windows. For CT, an RV:LV ratio of 1.0 or greater (as determined by radiologist) was considered RV dilatation.

Our primary outcome was clinical deterioration within 5 days of ED‐diagnosed PE (as defined in the parent study described above).

We performed pairwise comparisons of sPESI with ESC and PE‐SCORE. For each risk stratification tool, we reported sensitivity, specificity, negative and positive predictive values, and F1 scores. F1 is calculated by (positive predictive value × sensitivity)/(positive predictive value + sensitivity). When using binary classification, F1 is a measure of the accuracy of a model. We used receiver operating characteristics (ROC) to derive the area under the curve (AUC) with 95% confidence intervals (CIs) for clinical deterioration. We used Wald tests following the guidelines of Roldan‐Nofuentes and Sidaty‐Regad to compare differences in sensitivity, specificity, and predictive values for sPESI paired with ESC and with PE‐SCORE. We used the method described by DeLong to compare AUCs of sPESI with ESC and with PE‐SCORE. F1 scores were compared using bootstrapping and P values were estimated from the bootstrapped samples. A reclassification was assigned if there was a change in classification when applying ESC or PE‐SCORE criteria to that first assigned by the sPESI tool. As recommended by Kerr et al., we reported on net reclassification for binary risk categories as the change in true‐positives for events and false‐positives for nonevents for ESC and PE‐SCORE. We reported the change in true‐positive rate as the change in sensitivity. False‐positive rate is 1 minus specificity. Differences with false‐positive rate can be inferred directly from differences between model specificity. Finally, we reported discrimination of the 3 tools with AUC and the F1 score. We used R and RStudio software for all analyses.

The combined databases contained 1736 patients with acute PE between August 2016 and November 2020. We included 1569 PE patients who had all components necessary to determine sPESI, ESC, and PE‐SCORE classifications. Table 2 shows patient characteristics and comorbid conditions as grouped by the primary outcome. Age, gender, and race were similar between outcome groups, whereas troponin and RV abnormalities by CT and echocardiography were significantly higher in patients with clinical deterioration events.

Of the 1569 patients, 385 (24.5% [295% confidence interval (CI), 2.4%–26.8%]) experienced 1 or more components of clinical deterioration within 5 days: 64 (16.6%) had cardiac arrest, 37 (9.6%) of whom did not achieve return of spontaneous circulation; 131 (34%) had respiratory failure; 99 (25.7%) had new dysrhythmia; 73 (19%) had sustained hypotension treated with volume resuscitation and/or vasopressors; and 113 (29.4%) patients were given reperfusion intervention. There was 1 death within 5 days among those classified low‐risk by sPESI (segmental PE without right ventricular abnormalities on imaging). The death was caused by a perforated gastrointestinal ulcer and was considered unrelated to PE. There were no deaths within 5 days for those classified low‐risk by ESC or PE‐SCORE criteria.

Table 3 displays the number of patients classified as low‐risk and not low‐risk by each risk stratification tool and whether the group experienced clinical deterioration within 5 days or not. Both ESC and PE‐SCORE classified more patients as not low‐risk than sPESI. sPESI exhibited a sensitivity of 81.8%, specificity of 41.2%, positive predictive value of 31.2%, and negative predictive value of 87.5%. ESC was the most sensitive for excluding low‐risk. Almost every patient who experienced clinical deterioration within 5 days was correctly classified as not low‐risk. As a trade‐off, ESC had very low specificity of 13.7%, approximately half that of PE‐SCORE and sPESI. Most patients who were truly low‐risk were not correctly identified (high false‐positive rate) by ESC (Table 4). The sensitivity of PE‐SCORE for excluding low‐risk patients was 96.1% (370 of 385 patients with clinical deterioration were correctly predicted as not low‐risk). However, PE‐SCORE's specificity was 24.8% (only 294 of 890 without clinical deterioration were correctly predicted as low‐risk). PE‐SCORE's positive predictive value was 29.4% (false‐positive rate of 70.6%), and its negative predictive value was 95.1%, demonstrating a low false‐negative rate (Table 4). The discrimination of sPESI (AUC 0.615 [0.591, 0.639]) and PE‐SCORE (AUC 0.605 [0.589, 0.620]) for clinical deterioration within 5 days were similar (P = 0.44), whereas ESC (AUC 0.562 [0.551, 0.573]) was lower (P < 0.001) (Table 4). Figure 1 shows reclassification of initial sPESI assignments by the newer tools. sPESI classified 558 patients as low‐risk of clinical deterioration within 5 days. The newer tools, ESC and PE‐SCORE, subsequently reclassified 77.1% and 64.5% as not low‐risk, respectively. Of 1011 patients classified as not low‐risk by sPESI, ESC and PE‐SCORE reclassified 0.6% and 11.3% as low‐risk, respectively.

A limitation of this study is that we used the same registry databases PE‐SCORE was developed and validated on. In the original PE‐SCORE report, the tool's prognostic accuracy was assessed across the range of possible risk thresholds (across the scale of 0–9 points) or with its original logistic regression model. However, for this study, we restricted the PE‐SCORE points model to a low‐risk threshold (low‐risk = zero points). Additionally, although PE‐SCORE was developed and validated on the current database, it has not been externally validated. A second limitation is that sPESI and ESC were designed to identify patients at risk of death and 30‐day adverse outcomes; they were not designed to identify those at risk for short‐term clinical deterioration. Conversely, PE‐SCORE was designed to predict clinical deterioration within 5 days of PE diagnosis. The next logical step in comparing the prognostic accuracy of these risk stratification tools would involve an external validation of PE‐SCORE while collecting data on sPESI and ESC. Another limitation is our modified ESC risk stratification strategy used only sPESI criteria and did not incorporate the option for using Hestia criteria for low‐risk consideration. Like PE‐SCORE, Hestia guidelines factor in social considerations for risk classification, but Hestia is more expansive, including criteria such as pain control and bleeding risk. A further limitation of this study is that clinical decision‐making for each patient's disposition was not based on prospective use of sPESI, ESC, and PE‐SCORE tools, as it should be for an impact study with a new cohort of PE patients.

In this study, we compared the ability of sPESI, ESC, and PE‐SCORE risk stratification tools to identify clinical deterioration within 5 days of ED‐diagnosed PE. ESC and PE‐SCORE had higher sensitivities, lower specificities, and higher negative predictive values than sPESI for predicting clinical deterioration within 5 days. sPESI does not include cardiac imaging or biomarkers. Although some guidelines support outpatient treatment of PE without testing for RV abnormalities, other research suggests patients with RV abnormalities, who are deemed low‐risk by validated tools, are at higher risk of mortality than those without RV abnormalities. , , Our findings support use of the more recent risk stratification tools, including the 2019 ESC guidelines and our group's recently derived PE‐SCORE, which incorporate RV assessments to determine low‐risk for 5‐day clinical deterioration. , Furthermore, PE‐SCORE was trained and validated for clinical deterioration events within 5 days. A systematic review and meta‐analysis of PE risk stratification tools by Elias et al. used earlier versions of ESC than the 2019 ESC guidelines we evaluated in this study. Elias et al. reported the proportion of patients classified low‐risk by sPESI and ESC were 36.3% and 89.8%, respectively. However, only 2 of 71 studies included in the meta‐analysis studied clinical deterioration outcomes within the index hospitalization. A strength of our study is its focus on clinical deterioration outcomes of interest to clinicians deciding on disposition for immediate discharge or inpatient PE management. In our study, the proportions of low‐risk were similar for sPESI, but much lower for the 2019 version of ESC compared to Elias et al. Earlier versions of ESC did not require RV assessment as a criterion for low‐risk assignment. In Elias et al., in‐hospital mortality was much higher among those classified low‐risk by earlier versions of ESC compared with sPESI (5.0% [3.6%–7.1%] and 0.3% [0%–2.3%], respectively). Emerging evidence of increased mortality of patients with abnormal cardiac biomarkers and RV by CT or echocardiography ushered in revisions by ESC. The 2019 ESC risk stratification strategy includes abnormal RV (by imaging) or myocardial injury (by troponin) as exclusion criteria for low‐risk assignment. In our cohort, death was uncommon (2.4%), and no patient assigned to the low‐risk group by sPESI, ESC, or PE‐SCORE died from PE within 5 days. Our findings complement recent reports showing added prognostic value of RV assessment for predicting mortality or clinical deterioration events in otherwise low‐risk PE cohorts. , In our study, clinical deterioration was a comparatively frequent occurrence at 24.5% (95% CI, 2.4%–26.8%). This outcome prevalence is similar to another report, which demonstrated nearly identical rates of clinical deterioration (26%), even among initially normotensive patients with acute PE. These findings build on others from our registry databases, which show important clinical deterioration events occur within days of an acute PE. In Raper et al., 12% of PE patients classified low‐risk by sPESI criteria experienced 5‐day clinical deterioration. In that study, RV assessments (CT or echocardiography) and laboratory biomarkers improved prognostic accuracy for 5‐day clinical deterioration versus prognostic tools without RV assessment. In this multicenter registry data analysis, we noted the 2 newer tools, which incorporate RV assessments, had better sensitivity and negative predictive value than sPESI for predicting clinical deterioration events within 5 days of PE diagnosis. Overall, we found PE‐SCORE classified significantly more patients as not low‐risk than sPESI, and most reclassifications to low‐risk by PE‐SCORE did not develop clinical deterioration. In comparing the prognostic ability of tools, it is noteworthy that ESC incorporates troponin, sPESI, and RV imaging, resulting in a more stringent sieve (and higher sensitivity) than sPESI alone. However, these additional parameters also result in a very high false‐positive rate. PE‐SCORE incorporates RV imaging findings with similar effects on sensitivity, lending support to previous work demonstrating RV assessment methods as predictors of clinical deterioration. Although PE‐SCORE could be criticized for overestimating potential deterioration (70% false‐positive rate), it has a high sensitivity and high negative predictive value—most patients who experienced clinical deterioration were correctly classified. Overall, ESC and PE‐SCORE were both more sensitive than sPESI, but PE‐SCORE provided a more balanced prognostic profile than ESC in this cohort, with a significantly lower false‐positive rate and an AUC of 0.60 (Table 4). If externally validated, the high negative predictive value of PE‐SCORE for short‐term clinical deterioration may support the clinical decision for outpatient management of low‐risk PE patients. Our report suggests PE prognostic tools are not interchangeable for the specific purpose of determining low‐risk for acute PE‐related clinical deterioration outcomes. Tools that incorporated RV assessments had higher negative predictive values and sensitivity for acute clinical deterioration than sPESI. The statistical differences demonstrated significantly improved accuracy in classifying patients at low‐risk for clinical deterioration within 5 days over sPESI.

Anthony J. Weekes designed the study. All PESCOR investigators enrolled and collected data. Nathaniel S O'Connell and Anthony J. Weekes provided statistical analysis and interpretation of the data. Anthony J. Weekes drafted the manuscript. All authors contributed substantially to article revision for important intellectual content. Anthony J. Weekes takes responsibility for the article as a whole.

Jason T. Nomura is an ultrasound consultant for Philips. The other authors disclose no conflicts of interest.