Evaluation
It is critical to ensure that AI systems are safe, ethical, and without bias in the clinical domain. For the
proposed approach, we performed comprehensive automatic evaluations and a novel, rigorous,
patient safety-focused clinical evaluation. The unique clinical evaluation framework was designed to
(1) screen for and identify the common, specific correctness issues in LLMs observed in longform
clinical summarization and (2) assess the potential patient safety implications associated with any
incorrectness identified using a modified version of the World Health Organization’s International
Classification for Patient Safety.45
Automated Evaluations
We used the summarization evaluation metrics of recall-oriented understudy for gisting evaluation
(ROUGE),46 bidirectional encoder representations from transformers score (BERTScore),47 and
source chunking approach for large-scale inconsistency evaluation (SCALE).48 ROUGE computes the
overlap of n-grams between the generated and reference summaries. For longform document
summarization, the following ROUGE scores are considered to be close to the reference summaries:
ROUGE-1, above 0.4; ROUGE-2, above 0.2; and ROUGE-L, above 0.3.46 BERTScore leverages the
pretrained contextual embeddings from BERT and matches words to compute a similarity score for
each token in the candidate sentence with each token in the reference sentence. We used SCALE,48 a
natural language inference–based approach, to measure the faithfulness between the source
document and the generated text. Further background is provided about SCALE in eAppendix 2 in
Supplement 1.
Statistical Analysis
Based on prior work, 3 board certified EM physician leaders (M.M., A.F., and P.S.) with experience in
formal quality and patient safety review processes performed retrospective reviews of ED-based
EHR records of 50 individual ED patient encounters, randomly selected from the test dataset.49
Based on prior published clinical evaluations of LLM, as well as the study feasibility of using EM
physician quality and patient safety leaders, 50 ED patient encounters were evaluated.50 Reviewers
Figure. Data Flow of Generating Emergency Department (ED) Handoff Summary
EHR data
Transition of
care notes
Consult
notes
Progress
notes
Procedure
notes
Vitals Medications
Radiology
results
Labs
HPI
Extract recent HPI
Regex
processing
Regex
processing
Regex
processing
Regex
processing
Course of treatment
Salience selection
LLM 1
transformer
Remove duplicates
LLM 2
transformer
Get most salient medications and
laboratory tests, vitals, consults, and 
images with pattern-matching rules
Automated summary
HPI:
Patient is a 64-year-old male with history of alcoholic hepatitis sent in 
from urgent care for hypotension in setting of N/V/D 3 days ago, also 
with fall outside of ED
Events:
• Presented to the ED on January 6, 1972, for hypotension in setting of 
N/V/D 4 days
• On January 7, 1972, patient is to be admitted to medicine SDU
Plans:
Laboratory tests: CBC, CMP, trop, cultures, VBG, imaging: CTH. Aggressive 
IVF, monitor.
Disposition:
Admit to medicine-general, intermediate care (stepdown), for dizziness
Medications:
• Calcium gluconate-NaCl 2-0.67 GM/100ML-% IV solution 2g
• Octreotide acetate 100 mcg/mL solution (wrapper): 50mcg
Labs:
CBC: Hgb: 4.9 panic (!), Hct: 17.2 panic (!), WBC: 14.13 high (!), platelet: 
174
Vitals:
• HR: 105, BP: 94/62,  RR: 18, SPO2: 100, temp 97.9F
Consults:
• IP consult to gastroenterology
CBC indicates complete blood count; CMP, comprehensive metabolic panel; CTH, computed tomography of the head; EHR, electronic health record; Hct, hematocrit; Hgb,
hemoglobin; HPI, history of present illness; HR, heart rate; IP, inpatient; IVF, intravenous fluid; N/V/D, nausea, vomiting, and diarrhea; RR, respiratory rate; SDU, step down unit; SPO2,
peripheral capillary oxygen saturation; WBC, white blood cell; WBG, whole blood glucose.