LLM-model training, an informatics professional (V.H.) worked over a period of 200 hours with 3
board certified emergency medicine physician leaders with experience in formal quality and patient
safety review processes (M.M., A.F., and P.S.) to improve the dataset through manual curation and
annotation. As the task of EM-handoff note generation is not dependent on racial characteristics of
the patients, we removed all mentions of race during the annotation stage as a means to avoid race
bias; therefore, the model was trained to generate text without race-based assumptions. Although
resource intensive, a small and carefully curated dataset of at least 1000 examples has been shown
to be sufficient to produce remarkable results for the language model chosen.42 Given the size of
our dataset, we created a train and test dataset with a ratio of 1500:100, with a higher ratio of data
placed in the training set and eschewed a validation set to lower the variance of the models. We used
k-fold cross validation on the training dataset to avoid sampling bias for the hyperparameter
optimization of the LLMs.
Models
For this study, we chose the LLMs Robustly Optimized BERT Approach (RoBERTa; hereafter referred
to as LLM 1)43 for saliency content selection and Large Language Model Meta AI 2 (Llama-2; hereafter
referred to as LLM 2) 7B44 for abstractive summarization. Further information about the models and
technology specifications is provided in detail in eAppendix 1 inSupplement 1.
Data Processing
As LLM 2 only has a context size of 4096 tokens,44 we used 2 steps to process the EM notes to both
shorten the input size while maintaining content salience. First, we adopted a number of heuristic
strategies for prioritization and filtration: (1) clinical note types (hierarchy presented in Table 1), (2)
time of authorship, and (3) duplicate sentence detection. Second, we used an LLM 1–based saliency
model to infer EM note sentences based on likelihood of content contribution to the EM-to-IP
handoff notes.
Model Training and Inference
Our summarization model is a fine-tuned decoder-only causal language model based on LLM 2. We
used different prompts for the separate types of summarization: HPI and EM handoff. Additional
information about the model training and inference process is provided in eAppendix 1 in
Supplement 1.
Using a combination of generative AI powered by our fine-tuned LLM 2 model and a set of
heuristic rules, our summarization system produced ED handoff notes with various sections for
downstream clinical tasks. The inference process is shown in theFigure.
Table 1. Types of Data Included From the Emergency Department (ED) Patient Electronic Health Recorda
Type of data Description
Descriptive Date of birth, medical record number, encounter number, and total time of stay in ED
Encounter ED arrival date and time, IP admit date and time
Laboratory tests
(all results available)
Examples: hemoglobin, hematocrit, white blood cell count, neutrophil count, platelets, sodium,
potassium, chloride, bicarbonate, creatinine, blood urea nitrogen, troponin, D dimer, lactate,
urinalysis, ketone, blood, nitrite, leucocytes, and red blood cells
Laboratory tests
(only if abnormal)
Examples: β-human chorionic gonadotropin hormone, all serum drug levels (alcohol level,
salicylate level, Tylenol level), magnesium, lipase, and erythrocyte sedimentation rate
Notes (in order of
hierarchy)
EM clinician notes, consultation notes, EM progress notes, and EM procedure notes
Vitals Height, weight, temperature, heart rate, blood pressure, and peripheral capillary
oxygen saturation
Orders Medications, consults, and radiology results
Abbreviations: EM, emergency medicine; IP, inpatient.
a Automated EM handoff notes are generated from
the curation of the data through both rule-based and
large language model–summarization approaches.
JAMA Network Open |Emergency Medicine Developing and Evaluating LLM-Generated Emergency Medicine Handoff Notes