Are current methods of evaluating medical AI systems comprehensive enough to ensure their clinical reliability? @npjDigitalMed #ShanghaiArtificialIntelligenceLaboratory

"Towards evaluating and building versatile large language models for medicine"

• This study introduces MedS-Bench, a comprehensive benchmark spanning 11 clinical tasks to evaluate 9 leading LLMs (including MEDITRON, Llama 3, Mistral, GPT-4, Claude-3.5), and develops MedS-Ins, a large-scale instruction-tuning dataset with 58 medical corpora totaling 5M instances with 19K instructions across 122 tasks.

• The researchers identify that while LLMs perform well on multiple-choice medical questions, they struggle with real clinical tasks, highlighting concerns about deployment in healthcare settings and the need for more robust evaluation methods beyond traditional benchmarks.

• The methodology involved creating MedS-Ins by filtering medical-related tasks from 1616 natural instruction tasks, prompting existing BioNLP datasets, and using GPT-4 to enhance task instructions, followed by instruction tuning on a multilingual LLM base model (MMed-Llama 3) using zero-shot and few-shot prompting strategies.

• The results show that MMedIns-Llama 3 outperformed existing models across multiple tasks, achieving 63.9% accuracy on multiple-choice questions, 46.82/48.38 BLEU/ROUGE scores in text summarization, 83.77% accuracy in information extraction, and particularly strong performance in named entity recognition with an F1-score of 79.29.

#MedicalAI #LargeLanguageModels #HealthcareAI #ClinicalNLP

Authors: Chaoyi Wu, Pengcheng Qiu et. al Yanfeng Wang & Weidi Xie
Link: