superior performance. However,while the manual clinical evaluation demonstrated the majority of
the LLM-generated notes were of promising comparative quality (scores of 4-5), they were, on
average, inferior to the clinician-written notes.
Our novel clinical evaluation’s findings suggest the majority of identified quality limitations and
incorrectness would have minimal impact on patient safety, even when extrapolated to the worst-
case scenario of the LLM-generated summary content not being reviewed and edited by a clinician
before completion. This was designed to address contemporary LLM concerns of user trust, reliance
and expertise.49 As such, none of the incorrect output text elements reached life-threatening risk.
However,incompleteness and faulty logic identified in the automated summaries were not always
negligible, with just under 1 in 10 of these performance gaps determined to have the potential to
create significant patient safety risk compared with the physician-written summaries. These critical
implementation safety findings will inform (1) directionality of further model refinement; (2) further
clinical evaluation of postrefinement model output; and (3) irrespective of downstream model
performance, an EHR-implementation plan constrained to a user-interface design that will allow EM
clinicians to review and edit the LLM-generated handoff note as a draft before finalizing (see
eAppendix 1 inSupplement 1). This physician-in-the-loop process has also been identified as critical
in other recent work implementing LLMs into clinical workflows.29,53
While the automated methods of SCALE and MPNet-based sentence transformers
demonstrated a cursory view of the faithfulness performance of the models, the clinical evaluation
provided the nuanced context of the true factuality of our system on a word by word level. When
comparing with the source notes, the automatic evaluations rewarded the summaries with more
details, more semantic similarities, and more entailment logics, while physician-written notes tended
to be more concise with more shortcuts and clinical jargon, which are penalized by automatic
evaluation metrics. In addition, LLM-generated summaries are completely based on the source
notes, while physician-written summaries are often composed with additional knowledge that
cannot be found from the source notes.
The divergence of the automated and clinical evaluation results of an LLM intended for
integration into a critical clinical workflow is an important finding. First, this observed finding
validates the importance of clinical evaluations in addition to conventional automated evaluations to
determine accuracy.54 While other LLM clinical evaluation frameworks have been described to
measure conventional model output quality categories (such as incorrectness domains and other
performance gaps),30,35 to our knowledge, our novel framework is the first to incorporate
anticipated patient safety implications for each individual category deficiency.
Limitations
There were several limitations to the study that were primarily driven from constraints of
infrastructure, as well as regulations, legal governance, and labor requirements. At the study location,
the data were required to remain on premise at all times and the infrastructure that was provided
had a GPU limitation of 24 GB. Given these infrastructure restrictions, the best open-source model
available during the study was LLM 2. Furthermore, we were not able to demonstrate the comparable
difference between our fine-tuned LLM 2 model and third party LLMs32,55 because of the study
location’s restrictions and concerns with the data retention policies. Nevertheless, our study
demonstrates the potential capability of integrating state-of-the-art open source LLMs at
organizations that are less open to integrating third-party LLMs.
While the dataset was smaller, we made significant efforts to reduce model variance and