records and perform manual comparative evaluations of 50 LLM-generated and 50 clinician-
generated summary notes within the context of complex ED encounters. This approach is unlikely
scalable, invoking complex postimplementation governance questions that remain unanswered in
the medical literature and invoke the need for future research focused on the possibility of AI
performing the clinical evaluations.56 Lastly, the relatively infrequent but potentially significant
patient safety implications of model output incorrectness and incompleteness warrants further
model refinement and repeat clinical evaluation, as described in the eAppendix 1 inSupplement 1
overview of our rigorous preimplementation model development and testing framework.
Conclusions
This study’s results suggest promise for future thoughtful integration of LLM-generated EM-to-IP
handoff notes into clinical admission workflows, as well as the associated potential downstream
quality and efficiency gains. Our novel clinical evaluation framework demonstrates an effective
preimplementation strategy to measure potential patient safety implications of incorrectness
identified in LLM-generated clinical care summaries, which will guide future model refinement and
implementation strategies. In the absence of a current written standard of care in EM, this innovation
could represent a transformative advancement in the quality of EM-to-IP transitions of care.
ARTICLE INFORMATION
Accepted for Publication:October 7, 2024.
Published: December 3, 2024. doi:10.1001/jamanetworkopen.2024.48723
Open Access:This is an open access article distributed under the terms of theCC-BY License. © 2024 Hartman V
et al.JAMA Network Open.
Corresponding Author:Vince Hartman, MS, Abstractive Health, 333 E 56 St, Apt 7N, New York, NY 10022 (vince
@abstractivehealth.com).
Author Affiliations:Abstractive Health, New York, New York (Hartman, Zhang, Poddar); Department of
Emergency Medicine, NewYork-Presbyterian/Weill Cornell Medicine, New York (McCarty, Fortenko, Sharma,
Steel); Department of Population Health, NewYork-Presbyterian/Weill Cornell Medicine, New York (Sholle,
Campion); Clinical and Translational Science Center, Weill Cornell Medicine, New York, New York (Campion).
Author Contributions:Mr Hartman and Dr Zhang had full access to all of the data in the study and take
responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design:Hartman, Zhang, Poddar, McCarty, Fortenko, Campion, Steel.
Acquisition, analysis, or interpretation of data:All authors.
Drafting of the manuscript:Hartman, Zhang, Poddar, McCarty, Campion, Steel.
Critical review of the manuscript for important intellectual content:All authors.
Statistical analysis:Hartman, Zhang, Poddar, Sholle.
Obtained funding:Hartman, Campion.
Administrative, technical, or material support:Hartman, Zhang, Poddar, Sholle, Sharma, Campion, Steel.
Supervision: Zhang, Poddar, McCarty, Sharma, Campion, Steel.
Conflict of Interest Disclosures:Dr Hartman reported holding equity in Abstractive Health during the conduct of
the study and holding a patent for automated summarization of a hospital stay using machine learning issued to
Abstractive Health. No other disclosures were reported.
Funding/Support: Our research received support from NewYork-Presbyterian and Weill Cornell Medicine,
including the Joint Clinical Trials Office and Clinical and Translational Science Center (grant No. UL1TR002384).
Role of the Funder/Sponsor:The funder had no role in the design and conduct of the study; collection,
management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and
decision to submit the manuscript for publication.
Data Sharing Statement:See Supplement 2.
JAMA Network Open |Emergency Medicine Developing and Evaluating LLM-Generated Emergency Medicine Handoff Notes
JAMA Network Open.2024;7(12):e2448723. doi:10.1001/jamanetworkopen.2024.48723 (Reprinted) December 3, 2024 9/12