curation (4.24 [0.58] vs 4.76 [0.48]), readability (4.00 [0.64] vs 4.64 [0.49]), correctness (4.52
[0.64] vs 4.90 [0.39]), and patient safety (4.06 [0.86] vs 4.50 [0.56]).
In extrapolating the estimated worst-case scenario impact of these performance gaps on
patient safety, the 3 expert clinicians determined none of the identified model performance issues
were anticipated to create a level 1 (life-threatening) safety event (see examples of worst case
scenarios in eTable 2 inSupplement 1). While the incompleteness and faulty logic identified in the
automated summaries received mean (SD) safety scores of 4.20 (0.93) and 4.60 (0.75), respectively;
13 (8.7%) and 11 (7.3%) of these events, respectively, were determined to have the potential to create
a level 2 patient safety event following EM-to-IP handoff, substantially higher compared with the
physician-written summaries (0%). All of the 5 hallucinations had patient safety scores between 4
and 5 and a mean (SD) score of 4.96 (0.14), which is defined as the hallucinations posing mild to no
patient safety risk. LLM-generated notes demonstrated a higher rate of incorrectness (9.6%)
compared with the physician-written notes (2.0%), although very few hallucinations.
ICC were 0.79 for completeness, 0.70 for curation, 0.59 for readability, 0.76 for correctness,
and 0.74 for usefulness. These numbers suggest good reliability of agreement for completeness,
curation, correctness, and usefulness and suggest fair reliability for readability among the 3 raters.
Discussion
The study demonstrated success in generating EM-to-IP handoff notes using both a fine tuned,
pretrained LLM and rule-based approaches within an end user–developed note template. It is
important to note that (largely due to time constraints within the EM care delivery model) the
performance of EM-to-IP handoff notes was not the current standard of care in EM. The study site’s
unique electronic handoff process enabled a comparison between physician-written and
LLM-generated handoff notes. Traditional automated evaluations of the model output suggested
Table 3. Mean Clinical Quality Evaluation, Large Language Model (LLM)–Generated and Physician-Written
Criteria
LLM-generated Physician-written
Mean score (SD)
Likert rating 1-5, No. (%)a
Mean score (SD)
Likert rating 1-5, No. (%)a
1 23 45 1 2345
Completeness 4.00 (0.88) 0 12 (8) 31 (20.7) 69 (46) 38 (25.3) 4.16 (0.84) 0 3 (2) 31 (20.7) 48 (32) 68 (45.3)
Curation 4.24 (0.58) 0 1 (0.7) 13 (8.7) 85 (56.7) 51 (34) 4.76 (0.48) 0 0 6 (4) 39 (26) 105 (70)
Readability 4.00 (0.64) 0 8 (5.3) 17 (11.3) 87 (58) 38 (25.3) 4.64 (0.49) 0 0 5 (3.3) 38 (25.3) 107 (71.3)
Correctness 4.52 (0.64) 0 0 13 (8.7) 39 (26) 98 (65.3) 4.90 (0.39) 0 0 2 (1.3) 12 (8) 136 (90.7)
Usefulness 4.04 (0.86) 0 12 (8) 30 (20) 59 (39.3) 49 (32.7) 4.36 (0.71) 0 5 (3.3) 13 (8.7) 53 (35.3) 79 (52.7)
a Likert scores and score distributions over 50 notes for 3 annotators. There are no 1 ratings for either physician or LLM summaries in the 150 evaluationresults.
Table 4. Mean Clinical Safety Evaluation, Large Language Model (LLM)–Generated and Physician-Written
Criteria
LLM-generated Physician-written
Mean (SD)
Likert score 1-5, No. (%)a
Mean (SD)
Likert score 1-5, No. (%)a
1 2 3 45 1 2345
Completeness 4.20 (0.93) 0 13 (8.7) 19 (12.7) 58 (38.7) 60 (40) 4.50 (0.65) 0 0 17 (11.3) 43 (28.7) 90 (60)
Curation 4.82 (0.32) 0 1 (0.7) 3 (2) 21 (14) 125 (83.3) 4.90 (0.31) 0 0 3 (2) 8 (5.3) 139 (92.7)
Readability 4.74 (0.37) 0 1 (0.7) 6 (4) 23 (15.3) 120 (80) 4.94 (0.14) 0 0 0 10 (6.7) 140 (93.3)
Correctness:
hallucination
4.96 (0.14) 0 0 0 5 (3.3) 145 (96.7) 5.00 0 0 0 0 150 (100)
Correctness:
knowledge gap
4.88 (0.48) 0 3 (2) 2 (1.3) 6 (4) 139 (92.7) 4.90 (0.42) 0 1 (0.7) 5 (3.3) 3 (2) 141 (94)
Correctness:
faulty logic
4.60 (0.75) 0 11 (7.3) 12 (8) 13 (8.7) 114 (76) 4.94 (0.24) 0 0 2 (1.3) 2 (1.3) 146 (97.3)
Correctness: bias 5.00 0 0 0 0 150 (100) 5.00 0 0 0 0 150 (100)