evaluation frameworks may not address the anticipated effect LLM performance limitations could
have on patient safety.38-41
In this study, we aim to expand on prior work of clinical summarization to rigorously evaluate
the outcomes of a fine-tuned model developed to generate accurate and safe summaries of the care
rendered during an ED visit, with the long-term goal of integrating automated, structured EM-to-IP
handoff notes into an EHR-based electronic handoff admission workflow (see eAppendix 1 in
Supplement 1). We fine-tune pretrained LLMs on well curated datasets of structured and
unstructured EHR data from the ED encounter to summarize the patient’s ED care. We improved the
correctness of model generations and customized the summaries in a structured format designed
by a team of EM and internal medicine physician leaders for optimal usefulness. We proposed a novel
patient safety-focused LLM evaluation framework to examine the LLM-generated handoff notes’
quality and accuracy and the downstream patient safety implications of any identified inaccuracies.
To evaluate noninferiority, we compared the LLM-generated handoff notes with the preexisting
physician-written EM-to-IP handoff notes as the active control, using both the proposed patient
safety-focused clinical evaluation framework and automated benchmark-driven methods. We used
the physician-written EM-to-IP handoff notes as the active control and used the scores from both
evaluation frameworks for the margin of inferiority of the intervention.
Methods
Data Collection
The study, with review and approval from the Weill Cornell institutional review board (IRB), was
conducted at an urban academic 840-bed quaternary-care hospital in New York City, with
approximately 71 000 adult ED visits and 21 000 admissions annually. EHR data from 1600
individual EM patient encounters leading to acute hospital admission were randomly selected from
visits occurring between April and September of 2023. We limited our analysis to EM patient
encounters occurring after April 2023, as the study site had updated the EM-handoff at that time.
Encounters before this date used an earlier version of the EM-handoff note that would have provided
suboptimal data for training labels. We used these data to fine-tune a pretrained LLM, which then
generated an abstractive EM-handoff note. For the 1600 patient encounters (the study participants),
Weill Cornell Medicine IRB approved a waiver of informed consent because the study used
retrospective data and posed minimal risk to patients. We used Strengthening the Reporting of
Observational Studies in Epidemiology (STROBE) reporting guidelines.
EM-to-IP Handoff Note Template
The EM-to-IP handoff note template used in the study is a replication of the current manual handoff
note structure used at the study site. The generated EM handoff note consists of components
generated by a rule-based pattern-matching approach (laboratory tests, vitals, medications, consult
orders, and radiology impressions) and components generated by the trained abstractive
summarization model (history of present illness [HPI], differential diagnoses, immediate care plans,
in-ED events, and disposition). Each summary also included a header with the timestamp of ED triage
and discharge, patient’s birth date, patient’s unique identifier, patient’s encounter number, and the
total time of patient’s stay in the ED.
Data Curation for Automated ED Note Generation
The EHR data were bifurcated into 2 datasets linked by the patient encounter number: 1 for the rule-
based pattern-matching approach and the other for the LLM fine-tuning discussed in further detail
in eAppendix 1 inSupplement 1. The rule-based framework was designed by the 3 board certified EM
physicians (M.M., A.F., and P.S.). Fine tuning of the pretrained LLM consisted of the notes inTable 1:
EM clinician notes, consultation notes, EM progress note entries, and EM procedure notes. The