On the Robustness of Vision-Language Models Against Distractions
Model Evaluation;Vision-Language Models;Multimodal;Distraction Robustness
Margin violation -- this paper has reduced margins on both the left and right sides to fit more content into 10 pages.
Submission Desk Rejected by Program Chairs
@misc{
liu2024on,
title={On the Robustness of Vision-Language Models Against Distractions},
author={Ming Liu and Hao Chen and Jindong Wang and Wensheng Zhang},
year={2024},
url={https://openreview.net/forum?id=EGjTCIcSnW}
}
Although vision-language models (VLMs) have achieved significant success in various applications such as visual question answering, their resilience to prompt distractions remains as an under-explored area. Understanding how distractions affect VLMs is crucial for improving their real-world applicability, as inputs could be filled with noisy and irrelevant information in many practical scenarios. This paper aims to assess the robustness of VLMs against both visual and textual distractions in the context of science question answering. Built on the \emph{ScienceQA} dataset, we developed a new benchmark that introduces distractions in both the visual and textual contexts. To evaluate the reasoning capacity of VLMs amidst these distractions, we analyzed the performance of ten state-of-the-art models, including GPT-4o. Our findings reveal that most VLMs are vulnerable to various types of distractions, experiencing noticeable degradation in reasoning capabilities when confronted with distractions. Notably, models such as InternVL2 demonstrates a higher degree of robustness to these distractions. We also found that models exhibit greater sensitivity to textual distractions than visual ones. Additionally, we explored various mitigation strategies, such as prompt engineering, to counteract the impact of distractions. While these strategies improved model resilience, our analysis shows that there remain significant opportunities for further improvement.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Ming_Liu21 ~Hao_Chen15 ~Jindong_Wang4 ~Wensheng_Zhang2
Ming Liu Hao Chen Jindong Wang Wensheng Zhang
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Model Evaluation Vision-Language Models Multimodal Distraction Robustness
I certify that there is no acknowledgement section in this submission for double blind review.
liu|on_the_robustness_of_visionlanguage_models_against_distractions
/pdf/292311945b4cd386bd349ccd1168316ce0908ab2.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
On the Robustness of Vision-Language Models Against Distractions
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission