Dissecting Misalignment of Multimodal Large Language Models via Influence Function
Multimodal Large Language Model;Interpretability
Yes
No ethics review needed.
- The number of datasets and baselines used in the experimental section is limited, which fails to sufficiently demonstrate the generalizability and effectiveness of the proposed method.

- What is the necessity of using the Influence Function?
- **Dual-Perspective Data Valuation**: The proposed ECIF is the first dual-perspective data valuation method for multimodal large language models.

- **Applicable to Various Tasks**: The authors developed algorithms for different tasks, including identifying the most valuable data for specific tasks, detecting data misalignment, and tracing mispredictions.

- **Efficient Sample Influence Removal**: Experimental results show that ECIF is more efficient than retraining at removing the influence of samples and identifying influential data in the training set.
The paper introduces the Extended Influence Function for Contrastive Loss (ECIF) to address data misalignment in multimodal large language models (MLLMs). ECIF is designed to efficiently evaluate data influence for contrastive learning, considering both positive and negative samples. It provides a closed-form approximation, avoiding retraining, and improves the transparency and interpretability of MLLMs by accurately assessing data impact and alignment.
- **Questionable Design Rationale**: The authors applied the Influence Function to contrastive learning, but the overall design's rationale is debatable.

- **Insufficient Experiments**: The experiments are limited, with too few baselines and datasets, leading to insufficient empirical validation.

- **Lack of Clear Explanation**: Despite the use of many formulas, the paper lacks a clear explanation of why the Influence Function was used.
Yes
No ethics review needed.
Please see the weakness part.
1. ECIF is a novel extension of influence functions tailored specifically for contrastive loss, allowing it to consider both positive and negative samples' impacts, which is essential for contrastive learning in MLLMs.
2. By providing a closed-form approximation, ECIF avoids the need for retraining, making it practical and computationally efficient for large models.
3. The paper demonstrates ECIF's effectiveness across multiple tasks, including data valuation, misalignment detection, and misprediction trace-back, showcasing its versatility.
This paper proposes the Extended Influence Function for Contrastive Loss (ECIF), a method designed to improve data valuation in multimodal large language models (MLLMs). ECIF addresses the limitations of traditional influence functions by accounting for the roles of both positive and negative samples in contrastive learning, enabling a more accurate assessment of data impact and model alignment.
1. ECIF assumes a linear combination of influences from positive and negative samples, which may not accurately reflect real-world complexities in multimodal contrastive tasks.
2. The experiments focus on relatively clean and well-defined datasets, limiting the assessment of ECIF's robustness in more challenging, noisy, or open-domain multimodal datasets.
Yes
No ethics review needed.
N/A
(1) The paper addresses the misalignment of text-image pairs in training data and its impact on model performance.

(2) The proposed ECIF method provides an approach to evaluate the contribution of training data in contrastive learning models.
This paper proposed extended influence function for contrastive loss. It evaluates the training data contribution in MLLMs. The proposed method tackles the challenges of applying influence functions to contrastive learning models, and propose algorithms for data evaluation, misalignment detection, and misprediction trace-back tasks. The experiments show that ECIF improves the transparency and interpretability of MLLMs.
(1) The dataset used in the experiment is too simple; All of them are old and toy datasets. More complex datasets should be used, for example Imagenet instead of Imagenette.

(2) The improvement is very limited. In Table 1, the accuracy is dropped but still marked in bold. It will confuse the readers. It would be better to clarify the criteria for bolding results, especially in cases where accuracy decreases.
@inproceedings{
anonymous2024dissecting,
title={Dissecting Misalignment of Multimodal Large Language Models via Influence Function},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Y07R8h6m8e},
note={under review}
}
Multi-modal Large Language models (MLLMs) are always trained on data from diverse and unreliable sources,  which may contain misaligned or mislabeled text-image pairs. This frequently causes robustness issues and hallucinations, leading to performance degradation. Data valuation is an efficient way to detect and trace these misalignments. Nevertheless, existing methods are computationally expensive for MLLMs. 
While computationally efficient, the classical influence functions are inadequate for contrastive learning models because they were originally designed for pointwise loss. Additionally, contrastive learning involves minimizing the distance between the modalities of positive samples and maximizing the distance between the modalities of negative samples. This requires us to evaluate the influence of samples from both perspectives. To tackle these challenges, we introduce the Extended Influence Function for Contrastive Loss (ECIF), an influence function crafted for contrastive loss. ECIF considers both positive and negative samples and provides a closed-form approximation of contrastive learning models, eliminating the need for retraining. Building upon ECIF, we develop a series of algorithms for data evaluation in MLLM, misalignment detection, and misprediction trace-back tasks. Experimental results demonstrate our ECIF advances the transparency and interpretability of MLLMs by offering a more accurate assessment of data impact and model alignment compared to traditional baseline methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal Large Language Model Interpretability
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/142a23b121dc9c8ead3a9e3e00e7e8f8f623d4de.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Dissecting Misalignment of Multimodal Large Language Models via Influence Function
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission