text: eAppendix 1 in Supplement 1). This physician-in-the-loop process has also been identified as critical 
  in other recent work implementing LLMs into clinical workflows.29,53 
  
  While the automated methods of SCALE and MPNet-based sentence transformers 
  
  demonstrated a cursory view of the faithfulness performance of the models, the clinical evaluation 
  
  provided the nuanced context of the true factuality of our system on a word by word level. When 
  
  comparing with the source notes, the automatic evaluations rewarded the summaries with more 
  
  details, more semantic similarities, and more entailment logics, while physician-written notes tended 
  
  to be more concise with more shortcuts and clinical jargon, which are penalized by automatic 
  
  evaluation metrics. In addition, LLM-generated summaries are completely based on the source 
  
  notes, while physician-written summaries are often composed with additional knowledge that 
  
  cannot be found from the source notes. 
  
  The divergence of the automated and clinical evaluation results of an LLM intended for 
  
  integration into a critical clinical workflow is an important finding. First, this observed finding 
  
  validates the importance of clinical evaluations in addition to conventional automated evaluations to 
  determine accuracy.54 While other LLM clinical evaluation frameworks have been described to 
  measure conventional model output quality categories (such as incorrectness domains and other 
  performance gaps),30,35 to our knowledge, our novel framework is the first to incorporate 
  anticipated patient safety implications for each individual category deficiency. 