Original Investigation| Emergency Medicine
Developing and Evaluating Large Language Model–Generated Emergency Medicine
Handoff Notes
Vince Hartman, MS; Xinyuan Zhang, PhD; Ritika Poddar, MS; Matthew McCarty, MD; Alexander Fortenko, MD, MPH; Evan Sholle, MS; Rahul Sharma, MD, MBA;
Thomas Campion Jr, PhD; Peter A. D. Steel, MA, MBBS
Abstract
IMPORTANCE An emergency medicine (EM) handoff note generated by a large language model
(LLM) has the potential to reduce physician documentation burden without compromising the safety
of EM-to-inpatient (IP) handoffs.
OBJECTIVE To develop LLM-generated EM-to-IP handoff notes and evaluate their accuracy and
safety compared with physician-written notes.
DESIGN, SETTING, AND PARTICIPANTSThis cohort study used EM patient medical records with
acute hospital admissions that occurred in 2023 at NewYork-Presbyterian/Weill Cornell Medical
Center. A customized clinical LLM pipeline was trained, tested, and evaluated to generate templated
EM-to-IP handoff notes. Using both conventional automated methods (ie, recall-oriented
understudy for gisting evaluation [ROUGE], bidirectional encoder representations from transformers
score [BERTScore], and source chunking approach for large-scale inconsistency evaluation [SCALE])
and a novel patient safety-focused framework, LLM-generated handoff notes vs physician-written
notes were compared. Data were analyzed from October 2023 to March 2024.
EXPOSURE LLM-generated EM handoff notes.
MAIN OUTCOMES AND MEASURESLLM-generated handoff notes were evaluated for (1) lexical
similarity with respect to physician-written notes using ROUGE and BERTScore; (2) fidelity with
respect to source notes using SCALE; and (3) readability, completeness, curation, correctness,
usefulness, and implications for patient safety using a novel framework.
RESULTS In this study of 1600 EM patient records (832 [52%] female and mean [SD] age of 59.9
[18.9] years), LLM-generated handoff notes, compared with physician-written ones, had higher
ROUGE (0.322 vs 0.088), BERTScore (0.859 vs 0.796), and SCALE scores (0.691 vs 0.456),
indicating the LLM-generated summaries exhibited greater similarity and more detail. As reviewed by
3 board-certified EM physicians, a subsample of 50 LLM-generated summaries had a mean (SD)
usefulness score of 4.04 (0.86) out of 5 (compared with 4.36 [0.71] for physician-written) and mean
(SD) patient safety scores of 4.06 (0.86) out of 5 (compared with 4.50 [0.56] for physician-written).
None of the LLM-generated summaries were classified as a critical patient safety risk.
CONCLUSIONS AND RELEVANCEIn this cohort study of 1600 EM patient medical records,
LLM-generated EM-to-IP handoff notes were determined superior compared with physician-written
summaries via conventional automated evaluation methods, but marginally inferior in usefulness
(continued)
Key Points
Question Can a large language model
(LLM) generate emergency medicine
(EM)-to-inpatient (IP) handoff notes
that are useful and safe for EM care?
Findings In this cohort study of 1600
EM patient medical records using a
novel evaluation framework, the
LLM-generated EM-to-IP handoff notes
had a mean usefulness of 4.04 out of 5
(compared with 4.36 for
physician-written) and a mean patient
safety of 4.06 out of 5 (compared with
4.50 for physician-written) with no
critical patient safety risks.
Meaning These findings suggest the
value of a manual, patient safety–
focused clinical evaluation of LLM
models and the potential of
LLM-generated handoff notes to create
a new standard of care in EM.
+ Invited Commentary
+ Supplemental content
Author affiliations and article information are
listed at the end of this article.
Open Access.This is an open access article distributed under the terms of the CC-BY License.
JAMA Network Open.2024;7(12):e2448723. doi:10.1001/jamanetworkopen.2024.48723 (Reprinted) December 3, 2024 1/12
Downloaded from jamanetwork.com by guest on 01/13/2025