Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA
Vision and Language;AI for Healthcare;Benchmark
Yes
No ethics review needed.
1. Is the construction of adversarial negative samples reasonable? Could additional strategies be introduced to ensure the validity of these adversarial samples?
2. Does class imbalance in the dataset affect the model's generalization ability? How can better diagnostic performance be achieved for organs beyond the chest? Additionally, validation across multiple modalities is recommended to assess the impact of imbalanced data distribution on each modality.
3. What is the impact of adding open-ended tasks on the model's performance? How can report generation or open-ended question-answering be incorporated into ProbMed? Clearly, open-ended questions are more representative of common clinical scenarios.
4. Could GPT-4-generated visual descriptions introduce bias or hallucinations?
1. By introducing adversarial negative samples, the dataset tests model robustness, offering a more challenging evaluation standard.
2. ProbMed includes various diagnostic tasks across different imaging modalities and body organs, providing a evaluation setting for models.
3. The model’s performance improvement strategies are validated through methods like chain-of-thought reasoning and visual description enhancement, providing a foundation for future advancements.
This paper introduces ProbMed, a Med-VQA dataset with adversarial negative samples to reveal multimodal models' limitations in complex medical tasks. Chain-of-thought reasoning and GPT-4-generated visual descriptions are shown to enhance fine-grained diagnostic performance.
1. The dataset has a significantly higher number of chest X-ray images than other types, which may lead to poorer model performance on other organs. Has the model's performance across different modalities been experimentally verified?
2. How does the study prevent hallucinations when using GPT-4 for caption analysis, abnormality identification, and positional descriptions through few-shot prompting?
3. ProbMed uses closed-ended questions and lacks open-ended tasks, limiting the dataset's comprehensiveness for evaluation.
4. Constructing adversarial samples by selecting random entities (such as alternative organs, modalities, or hallucinated conditions with "no" answers) may increase testing difficulty but could also introduce mistakes, resulting in samples that aren’t truly adversarial.
Yes
No ethics review needed.
I have mentioned them in weaknesses
It is a well-written paper with many experiments to evaluate the models on the proposed dataset. Several popular models have been studied and also authors propose mitigation strategies to improve the models' performance. Overall, the paper targets important questions and has the potential to be a good contribution to the field.
The paper introduces the ProbMed dataset for the rigorous evaluation of large multimodal models in clinical domains. The authors show that models perform worse than random on medical diagnosis questions when subjected to adversarial questions. They also show that the poor visual understanding of LMMs is a primary bottleneck, and can be mitigated by adding visual descriptions generated by GPT-4o.
After reading the paper, I have a few questions and concerns as follows:


1. (major) If I understood it correctly, the dataset has been curated using the publically available data on the internet. My main concern is the possible data contamination in larger closed or open-source models. When some models have been trained on the data and some not, evaluation using this dataset loses its fairness. 

2. If I have gotten it right, all the adversarial questions are in the form of negated questions and their response is "no". Having a model that has been previously trained on the data, can we ensure that it does not cheat?

3. (major) I strongly suggest adding a distribution of the responses that are "yes" or "no". how many questions have an answer of "no" and how many "yes" within each category in the dataset? This can be done through a qualitative distribution plot in the paper. 

4. (major) The adversarial question design is creative, but it has issues as well. Within clinical data. there are always cases that have co-occurrence of multiple forms of the disease, but in the original caption, we only have one of them as according to a clinician it is the important one. In this regard, when we create adversarial questions, this important fact has been ignored. So, each question actually needs to be validated by a medical expert. I have seen that 100 samples were examined by experts, but that number is significantly small compared to the size of the dataset. In fact, the paper lacks a thorough and careful expert study to ensure correctness.
Yes
No ethics review needed.
- It is interesting that augmentation with visual descriptions generated by GPT-4o can improve the performance. Can open-sourced multi-modal models (e.g., Qwen2-VL-72b-Instruct, LLaMa 3.2 Vision) improve the performance as well?

- Sec. 4.3.2 it is not clear how CheXagent is used to enhance the model performance. Could you please provide more details on how CheXagent was integrated with other vision-language models?
- A new dataset, PromMed, was curated for medical VQA benchmarking, which contains adversarial question pairs
- Comprehensive experiments on multiple VLMs
- Insightful findings: SOTA VLMs perform worse than random guessing on specialized diagnostic questions
This work curates a medical VQA dataset with pairing original questions and negation questions  and evaluations of SOTA VLMs show the poor visual understanding abilities. Furthermore, the authors show that this issue could be eliminated by adding external visual descriptions generated by GPT-4o.
- Some questions are too trivial in the dataset. The ultimate goal of multimodal models is deployment in real clinical practice when the accuracy is good enough. However, no clinician will ask questions on basic modalities (e.g., CT, MR) or organs because they are too trivial. Arguments such as “CheXagent, trained exclusively on chest X-rays, achieved the highest accuracy in determining abnormalities and conditions. However, its performance in general tasks like identifying image modality and organs was lower” should be modified because ChexXagent was not designed for tasks like identifying image modality and organs. These tasks are also not clinically significant because of their trivialities. I would suggest that the authors focus their analysis and discussion on the more clinically relevant and challenging aspects of the dataset, such as identifying specific abnormalities or conditions. 

- Minor: References format is not consistent. Some preprint papers even missed arxiv id
Yes
No ethics review needed.
1. For table 3, my current understanding is that for questions that form a pair of original + adversarial question:
Getting the original question correct, but adversarial question wrong would equate to an accuracy of 50% in the "averaged accuracy", and 0% in the second case. But in either case, the tally of the original questions that do not have an adversarial pair remains consistent? Wouldn't this obfuscate the true impact of the adversarial examples since it's diluted by the original, non-adversarial "no" questions?
2. Are adversarial questions included in the tally of 57k QA pairs, and used in evaluation by default? e.g. in table 4, figure 4, 5, etc.
3. Besides the question + image itself, is there any additional formatting / parsing used to evaluate the various models? especially for open-source MLLMs that may not be perfect at instruction following, what happens if a model outputs a full sentence response instead of a simple yes / no answer.
4. How consistent are these results, especially for larger frontier models where it may not be possible guarantee deterministic output. Is there a way quantify the variance in the performance if e.g. gpt4o / gemini are evaluated more than once?
The dataset is large, high quality, diverse, balanced, and is curated in a methodical manner, with plenty of useful metadata (such as question categorization, location annotation, etc.), making it an incredibly valuable benchmarking resource for the community. With these metadata intact, future works can easily build on top of the released data to curate more open-ended / multiple choice style questions if needed. This in my opinion is the paper's biggest strength. Additionally, the authors share some interesting findings about how much performance degrade when simple adversarial perturbations were applied to the original questions, which shows that even the frontier models today are quite fragile and limited in their ability to perform medical decision making reliably. The investigation into COT, especially COT with external visual descriptions are also quite insightful, revealing a clear path forward to improve open-source, domain-specific MLLMs.
Authors curated a large, high quality, balanced dataset of 57k close-ended (yes / no) VQA questions for  6,303 radiology medical images sourced from existing open-source datasets. The benchmark can be sliced across a number of different categories and for about half of the questions for which the answer is "yes", an "adversarial" question is also created by perturbing the entities / locations referenced in the original questions and changing the answers to "no". The dataset is used to evaluate a range of closed-source frontier MLLMs as well as open-source general purpose and domain-specific MLLMs. 

Overall the authors find that when the adversarial pair is introduced, all models saw significant drops in performance (> 20% in accuracy) although general purpose frontier models like gemini / gpt4o appear to be more robust and saw lower decrease in performance.  
The authors further find that when performance is broken down by category, general purpose models can still struggle significantly, and may perform worse on specialized domain-specific MLLMs. Amongst other findings, the authors also investigate whether COT can enhance performance, finding that while open-source models generally saw an improved performance, larger frontier models do not, and open-source models can benefit from using higher quality visual descriptions generated by larger frontier models as part of the COT process, further boosting performance.
My main complaint with the paper is that I find the presentation of the results difficult to follow at times. For example, it is not perfectly clear to me whether the "adversarial" questions are included in the tally of 57k QA pairs, and are also included as part of the evaluation benchmark by default - or if the 57k QA pairs represent the default, "clean" benchmark, with additional X number of adversarial questions (presumably = the number of questions with "yes" as answer) used to report the numbers in table 3 and the appendix only. It is also not super clear to me what the authors mean in table 3, where the authors distinguish between "Averaged Accuracy" and "Accuracy (%) with Adversarial Pairs". An illustrative example of how these metrics are computed would be helpful. 

Additionally, a limitation of the study is that the benchmark / adversarial questions are only limited to binary yes / no settings, when I can imagine the methodology easily extending to multiple choice questions, which seems like a missed opportunity. Although I believe this is something that future works can build on given the well curated nature of the dataset.
State-of-the-art multimodal models, including GPT-4V and Gemini Pro, perform worse than random guessing on specialized medical diagnosis questions, highlighting the need for more robust evaluation methods to ensure reliability in medical diagnosis.
@inproceedings{
anonymous2024worse,
title={Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical {VQA}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=QByW8EYEtt},
note={under review}
}
Large Multimodal Models (LMMs) have shown remarkable progress in medical Visual Question Answering (Med-VQA), achieving high accuracy on existing benchmarks. However, their reliability under robust evaluation is questionable. This study reveals that state-of-the-art models perform worse than random guessing on medical diagnosis questions when subjected to simple probing evaluation. To address this critical evaluation problem, we introduce the Probing Evaluation for Medical Diagnosis (ProbMed) dataset to rigorously assess LMM performance in medical imaging through probing evaluation and procedural diagnosis. Particularly, probing evaluation features pairing original questions with negation questions with hallucinated attributes, while procedural diagnosis requires reasoning across various diagnostic dimensions for each image, including modality recognition, organ identification, clinical findings, abnormalities, and positional grounding. Our evaluation reveals that top-performing models like GPT-4o, GPT-4V, and Gemini Pro perform worse than random guessing on specialized diagnostic questions, indicating significant limitations in handling fine-grained medical inquiries. We further investigate the underperformance of open-source models (e.g., LLaVA, LLaVA-Med, and Med-Flamingo) through an ablation study. This study reveals that poor visual understanding is a primary bottleneck, which can be mitigated by adding visual descriptions generated by GPT-4o, leading to an average performance improvement of 9.44%. These findings underscore the urgent need for more robust evaluation methods and domain-specific expertise to ensure LMM reliability in critical medical fields.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Vision and Language AI for Healthcare Benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/30bf267317fa106d4130c978d7a1787e17d80a5d.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/4786041fc1a5ace8c27c5888b5ea14fc69a02089.zip
Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission