text: **JAMA Network Open | Emergency Medicine** Developing and Evaluating LLM-Generated Emergency Medicine Handoff Notes 
  
  **Evaluation**
  It is critical to ensure that AI systems are safe, ethical, and without bias in the clinical domain. For the 
  
  proposed approach, we performed comprehensive automatic evaluations and a novel, rigorous, 
  
  patient safety-focused clinical evaluation. The unique clinical evaluation framework was designed to 
  
  (1) screen for and identify the common, specific correctness issues in LLMs observed in longform 
  
  clinical summarization and (2) assess the potential patient safety implications associated with any 
  
  incorrectness identified using a modified version of the World Health Organization’s International 
  Classification for Patient Safety.45 
  
  **Automated Evaluations**
  We used the summarization evaluation metrics of recall-oriented understudy for gisting evaluation 
  (ROUGE),46 bidirectional encoder representations from transformers score (BERTScore),47 and 
  source chunking approach for large-scale inconsistency evaluation (SCALE).48 ROUGE computes the 
  overlap of n-grams between the generated and reference summaries. For longform document 
  
  summarization, the following ROUGE scores are considered to be close to the reference summaries: 
  ROUGE-1, above 0.4; ROUGE-2, above 0.2; and ROUGE-L, above 0.3.46 BERTScore leverages the 
  pretrained contextual embeddings from BERT and matches words to compute a similarity score for 
  each token in the candidate sentence with each token in the reference sentence. We used SCALE,48 a 
  natural language inference–based approach, to measure the faithfulness between the source 
  
  document and the generated text. Further background is provided about SCALE in eAppendix 2 in 
  
  Supplement 1. 