Abstract (continued)
and safety via a novel evaluation framework. This study suggests the importance of a physician-in-
loop implementation design for this model and demonstrates an effective strategy to measure
preimplementation patient safety of LLM models.
JAMA Network Open.2024;7(12):e2448723. doi:10.1001/jamanetworkopen.2024.48723
Introduction
Handoffs, where patient information is exchanged between health professionals during a transfer of
clinical responsibility, have been identified as a critical source of medical errors.1,2 The Joint
Commission, the Accreditation Council for Graduate Medical Education, and the Association of
American Medical Colleges have all recommended the development of high-quality and standardized
handoff processes to address the substantial patient risk of this ubiquitous event.3,4 Implementing
handoff tools has previously demonstrated significant reductions in medical errors.5,6 High-quality
handoffs from emergency medicine (EM) to inpatient (IP) services (EM-to-IP) are challenged by
medical complexity, diagnostic uncertainty, rapidly evolving care plans, and time constraints.7-10 The
EM-to-IP handoff structure is not well standardized, frequently communicated verbally, and poorly
adhered to in emergency departments (EDs), including in medical centers with formalized handoff
systems.11-14 Prior research has demonstrated that suboptimal EM-to-IP handoff is associated with
adverse events, EM leaders and front-line clinicians themselves view the EM-to-IP handoff as high
risk, and an electronic health record (EHR)-based technology is commonly mentioned as the most
desired assistive tool in improving ED transitions of care.15-18 Limited work to date has demonstrated
EM electronic handoff tools as feasible, efficient, and effective.19-21 In April 2023, EM and internal
medicine leadership of the study site collaboratively developed and launched a mandatory,
EHR-based handoff workflow via a standardized EM-to-IP handoff note template, designed for real-
time completion by the EM care team at time of admission. At 3 and 6 months postlaunch, informal
evaluation of new EM-to-IP handoff notes through random medical record review and unstructured
clinician feedback sessions revealed variable completeness, quality, and subsequent usefulness of
the handoff notes.
In recent years there has been an accelerated interest in using LLMs to automate clinical tasks
in an effort to unburden physicians and reduce burnout.22 Computer-generated text within clinical
notes using natural language processing (NLP) have been overall shown to improve note completion
rates, physician satisfaction, and patient outcomes.23 Since 2018, NLP has made rapid advancements
in health care with the discovery of the transformer model architecture, the building block of large
language models (LLMs). LLMs can automate workflows such as discharge summaries,24 radiology
reports,25 patient messaging,26 after-visit summaries,27 and ambient dictation28 with various levels
of perceived quality in each workflow.29 LLMs are particularly effective at summarizing large
unstructured clinical datasets, such as ED patient medical records.30 A common concern of LLMs is
their ability to hallucinate data, or LLMs generating output text that is not factually consistent with
the original source content.31 Much work has been done in health care to reduce hallucinations
through building larger-parameter models trained on trillions of datasets, and then instruction fine-
tuning the LLM on smaller, well-curated datasets.32,33 LLMs can also be designed with explainability
by citing inferred content back to the reference source notes.34 For short-context length notes, using
few-shot prompt engineering approaches with large language models like GPT-4 can produce
summaries that outperform standard physician documentation in completeness and error
frequency.35 However, factual inconsistencies in the summaries produced by LLMs increase as the