subsequently evaluated 2 ED-to-inpatient handoff notes for each patient: (1) the physician-written
note and (2) the LLM-generated note.
On a Likert scale of 1 to 5, where 1 is unacceptable and 5 is excellent, the 3 physicians rated the
completeness, curation, readability, and correctness of the summary as shown in eTable 1 in
Supplement 1. Physicians rated the usefulness of the summary, defined as the capability of the
summary being incorporated into a workflow where a physician would make edits before final
completion, mitigating potential future self-referential learning loops and the downstream adverse
consequences.51 Likewise, the raters assessed potential patient safety implications of unmitigated
model errors using a scale from 1 to 5, where 1 denotes life-threatening risks and 5 denotes no
identified patient safety risk for completeness, curation, readability, and the 4 subcategories within
correctness (hallucination, faulty logic, knowledge gap, and bias), as well as the overall patient safety
risk.45 Evaluators arrived at prestudy consensus that a usefulness Likert score of at least a 3 out of 5
indicated that the LLM-generated summary likely demonstrated baseline acceptability for such a
workflow. To extrapolate a theoretical worst case scenario, the physicians rated the safety of the
LLM-generated summary as defined as the capability of the summary to fully replace a physician-
written note (unmitigated).
To improve consistency and agreement, the 3 reviewers met to familiarize themselves with the
framework and evaluated 10 separate cases from the test dataset that were not included in the
clinical evaluation results. Additionally, after independently scoring the summaries, they met to
ensure consensus interpretation of the multidimensional scoring framework. Interrater reliability was
calculated using intraclass correlation coefficient (ICC), using a 2-way random effects model for
consistency with the Pingouin statistical package version 0.5.4 in Python (Python Software
Foundation). The ICC measures the similarity of the 3 raters to confirm the consistency and validity
of the evaluation protocol; the scores are from 0 to 1, where 1 indicates unanimous agreement and 0
represents no agreement.52 Data were analyzed from October 2023 to March 2024.
Results
Automated Tasks
Of 1600 patients, the mean (SD) age was 59.8 (18.9) years and 832 (52%) were female. InTable 2,
ROUGE and BERTScore compare the summaries with the testing set from our annotations, and
SCALE score compares the summaries with the source notes. From automatic evaluation results, we
observed that LLM-generated summaries had better scores than the physician summaries, such that
ROUGE-2 was 0.322 vs 0.088, BERT-precision was 0.859 vs 0.796, and SCALE was 0.691 vs 0.456,
suggesting the LLM-generated summaries were more similar and more detailed than the physician
summaries.
Clinical Evaluation Tasks
The clinical evaluation results for LLM-generated summaries and physician-written summaries are
shown inTable 3and Table 4. The mean clinical quality scores of the automated summaries are in a
comparable range (4-5) to those of the physician summaries. However, theautomated summaries
were observed to be of lower quality compared with the physician-written summaries with regards
to mean (SD) usefulness (4.04 [0.85] vs 4.36 [0.71]), completeness (4.00 [0.88] vs 4.16 [0.84]),
Table 2. Automated Evaluation Scores, Large Language Model (LLM)–Generated and Physician-Written
Summary type R-1 a R-2a R-La BERT-p BERT-r SCALE
LLM-generated 0.494 0.322 0.391 0.859 0.876 0.691
Physician-written 0.251 0.088 0.154 0.796 0.827 0.456
Abbreviations: BERT, bidirectional encoder representations from transformers; p, precision-based scores; r, recall-based scores; R, recall-oriented understudy for gisting evaluation;
SCALE, source chunking approach for large-scale inconsistency evaluation.