43. Liu Y, Ott M, Goyal N, et al. Roberta: a robustly optimized bert pretraining approach.arXiv. Preprint posted
online July 26, 2019. doi:10.48550/arXiv.1907.11692
44. Touvron H, Martin L, Stone K, et al. Llama 2: open foundation and fine-tuned chat models.arXiv. Preprint
posted online July 19, 2023. doi:10.48550/arXiv.2307.09288
45. World Health Organization. Conceptual framework for the international classification for patient safety
version 1.1: final technical report January 2009. 2010. Accessed October 23, 2024.https://www.who.int/publications/
i/item/WHO-IER-PSP-2010.2
46. Lin CY. Rouge: a package for automatic evaluation of summaries. 2004. Accessed October 23, 2024.https://
aclanthology.org/W04-1013/
47. Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. Bertscore: Evaluating text generation with bert.arXiv.
Preprint posted online February 24, 2020. doi:10.48550/arXiv.1904.09675
48. Lattimer BM, Chen P, Zhang X, Yang Y. Fast and accurate factual inconsistency detection over long documents.
Presented at: Conference on Empirical Methods in Natural Language Processing. December 6-10, 2023; Singapore.
49. Schaye V, Miller L, Kudlowitz D, et al. Development of a clinical reasoning documentation assessment tool for
resident and fellow admission notes: a shared mental model for feedback.J Gen Intern Med. 2022;37(3):507-512.
doi:10.1007/s11606-021-06805-6
50. Zaretsky J, Kim JM, Baskharoun S, et al. Generative artificial intelligence to transform inpatient discharge
summaries to patient-friendly language and format.JAMA Netw Open. 2024;7(3):e240357. doi:10.1001/
jamanetworkopen.2024.0357
51. Choudhury A, Chaudhry Z. Large language models and user trust: consequence of self-referential learning loop
and the deskilling of health care professionals.J Med Internet Res. 2024;26:e56764. doi:10.2196/56764
52. Gisev N, Bell JS, Chen TF. Interrater agreement and interrater reliability: key concepts, approaches, and
applications. Res Social Adm Pharm. 2013;9(3):330-338. doi:10.1016/j.sapharm.2012.04.004
53. Baxter SL, Longhurst CA, Millen M, Sitapati AM, Tai-Seale M. Generative artificial intelligence responses to
patient messages in the electronic health record: early lessons learned.JAMIA Open. 2024;7(2):ooae028. doi:10.
1093/jamiaopen/ooae028
54. Goodman KE, Paul HY, Morgan DJ. LLM-generated clinical summaries require more than accuracy.JAMA.
2024;331(8):637-638. doi:10.1001/jama.2024.0555
55. Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report.arXiv. Preprint posted online March 4, 2024. doi:10.
48550/arXiv.2303.08774
56. Chung HW, Hou L, Longpre S, et al. Scaling instruction-finetuned language models.J Mach Learn Res. 2024;
25(70):1-53.
SUPPLEMENT 1.
eAppendix 1.Technology Specifications, Model Training, and Inference
eAppendix 2.Automated Method of SCALE
eTable 1.Definitions From the Clinical Assessment
eTable 2.Worst Case Scenario of Incorrectness Examples From LLM-Generated Notes
SUPPLEMENT 2.
Data Sharing Statement
JAMA Network Open |Emergency Medicine Developing and Evaluating LLM-Generated Emergency Medicine Handoff Notes
JAMA Network Open.2024;7(12):e2448723. doi:10.1001/jamanetworkopen.2024.48723 (Reprinted) December 3, 2024 12/12
Downloaded from jamanetwork.com by guest on 01/13/2025