Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation
Multimodal learning;Uncertainty estimation
Yes
No ethics review needed.
* Why use MMML specifically for the introductory introductory presentation? The reader cannot understand where the representation is. Please explain the MMML framework in detail. Would it still be similar if it was replaced with another model?
* The theoretical proofs and methodology of this article are solid and valid.

* The Error Propagation and Uncertainty Estimation proposed by the authors is interesting and unique.

* The authors implemented diverse experiments on multimodal datasets from different domains.
This paper presents several key contributions through the development of the SURE (Scalable Uncertainty and Reconstruction Estimation) framework. First, SURE introduces a novel approach for handling missing modalities by incorporating latent space reconstruction
techniques into pretrained multimodal models. Second, SURE addresses the critical challenge of evaluating the reliability of both reconstructed inputs and final outputs by integrating uncertainty estimation. This is achieved through (1) a Pearson Correlation-based loss function and (2) the first application of error propagation in deep network training, allowing SURE to effectively quantify
uncertainties from multiple sources. Third, SURE is adaptable to a wide range of pretrained multimodal networks, demonstrating its robustness across various datasets and tasks, ultimately enhancing both interpretability and performance in multimodal learning scenarios.
* A great deal of multimodal works [1,2,3,4,5] related to modal absence has been neglected, leading to results and limitations that are difficult to measure.
These multimodal efforts are the main current SOTA in the field of studying modal absence across different modeling paradigms, including learning robust joint representations and reconstructions. It would be useful for the authors to clarify and compare the differences with these works.

* The writing of the article is not good enough. The author does not discuss and explain in depth many of the concepts and structures introduced. For example, there is no clear explanation of what exactly constitutes the reconstruction module described in the methods section.

* The authors consider a limited number of modal absence scenarios, and the study of modal absence should encompass both intramodal absence and intermodal absence obeying real-world scenarios. Then, the authors seem to follow a specific setup, leading to a lack of generalization of the experimental results.

[1] Li, Mingcheng, et al. "Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In AAAI.

[3] Zheng Lian, Lan Chen, Licai Sun, Bin Liu, and Jianhua Tao. Gcnet: graph completion network for incomplete multimodal learning in conversation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

[4]Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabas P ´ oczos. Found in translation: Learn- ´
ing robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 6892–6899, 2019.

[5] Li, Mingcheng, et al. "A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 9. 2024.
Yes
No ethics review needed.
1. A detailed caption is missing for Figure 3, making it difficult to understand the symbols and the flow of the proposed pipeline since their descriptions are spread across sections 2.2 to 2.4.

2. How does the number of reconstructors \( r_i \) scale with the number of modalities? Figure 3 suggests that a large number of reconstructors may be necessary to handle all possible missing-modality scenarios, which could affect the model’s scalability. A detailed explanation would clarify how the framework manages this growth efficiently.
1. The proposed approach introduces uncertainty estimation to adapt pretrained multimodal models for downstream tasks with missing modalities. This method not only enhances overall performance on downstream tasks but also enables the estimation and interpretation of uncertainty arising from both the model's inherent stochasticity and incomplete inputs.

2. The proposed approach is validated across multiple benchmarks and compared against previous methods.

3. The approach is designed to be compatible with a wide range of pretrained multimodal models, making it adaptable and applicable across diverse architectures and tasks.
This paper introduces SURE (Scalable Uncertainty and Reconstruction Estimation), a framework designed to improve pretrained multimodal models when modalities are missing during training or evaluation. SURE focuses on two key areas: reconstructing missing modalities within the latent space and quantifying uncertainty for both these reconstructions and subsequent predictions. A novel Pearson Correlation Coefficient loss function and statistical error propagation are used to estimate uncertainties, improving the reliability of the reconstructed inputs and final predictions. SURE is architecture-agnostic and can adapt to various pretrained multimodal models, achieving robust and interpretable results in experiments on sentiment analysis, genre classification, and action recognition.

Overall, this paper introduces an interesting perspective for enhancing pretrained multimodal models when modalities are missing. The framework integrates techniques for estimating uncertainty and is adaptable across various architectures. However, it requires clearer theoretical justification, especially regarding how uncertainty estimation contributes to improved predictive performance. Additionally, inconsistencies in benchmark results and limited evaluation of uncertainty estimation indicate areas that need refinement. Addressing these issues would enhance both the impact and comprehensibility of the framework.
1. It is unclear how and why uncertainty estimation enhances overall performance. While Section 2.3 suggests that incorporating Bayesian learning can help model the aleatoric uncertainty, there is a lack of theoretical explanation for how this contributes to improvements in predictive performance, such as accuracy.

2. In section 3.2, the authors mentioned that the Gaussian assumption allows for a closed-form solution to uncertainty estimation, suggesting that a strong alignment or correlation between uncertainty and prediction error leads to better performance. However, if the underlying distribution does not follow the Gaussian assumption, is this correlation still necessary for achieving an optimal solution? If not, optimizing the PCC loss to align uncertainty with prediction error may need further justification.

3. The performance of IMDer on the CMU-MOSI dataset seems to be inconsistent to the results reported in the original paper. For example, the F1 score for full inputs reported in the original IMDer paper is 85.1 v.s. 62.0 reported in this paper. The authors may need to clarify the difference.

4. The evaluation of uncertainty estimation in the paper is limited. Currently, the paper assesses uncertainty estimation by calculating the correlation between uncertainty and prediction error. However, it does not intuitively explain why a high correlation between these metrics is desirable or how it contributes to the model’s reliability.
Yes
No ethics review needed.
1. What does MMML stand for?
2. According to Table 3 and 3, it seems that pretraining contributes to the majority of the performance gain of SURE. For example, the performance of the proposed SURE without pretrained weights in Table 4 seems to be worse than the performance of previous methods in Table 3. However, with pretrained weights, the preformance is significantly increased and surpasses previous methods.
3. The integration of latent reconstructors and multiple uncertainty estimations could add computational overhead. A quantitative analysis of the computational costs compared to other methods would help in understanding SURE’s efficiency and scalability.
4. The authors claim that the proposed approach is scalable (**Scalable** Uncertainty and Reconstruction Estimation). However, no justification or experimental evidence is provided to support this claim.
1. The paper brings up an important issue in multimodal learning—handling missing modalities, which is common in real-world applications where all data sources are not consistently available.
2. The framework provides both reconstruction and output uncertainty estimates, which help assess the reliability of predictions.
3. The paper includes extensive experiments across diverse tasks (sentiment analysis, genre classification, and action recognition) and datasets.
This paper focuses on an important challenge in multimodal learning—handling missing modalities, which is common in practical applications. The proposed SURE framework (Scalable Uncertainty and Reconstruction Estimation) aims to enhance pretrained multimodal models by incorporating latent-space reconstruction and uncertainty estimation. The inclusion of uncertainty estimates for both reconstruction and final predictions is valuable, as it provides insights into the model's confidence, especially when modalities are incomplete.

However, while the paper presents extensive experiments, its motivation and novelty are somewhat ambiguous. Moreover, claims regarding “scalability”, “distribution-free” and other aspects lack sufficient justification.
1. The motivation behind the paper is ambiguous. Although the introduction discusses the benefits of adapting pretrained multimodal models for downstream tasks, the rest of this paper primarily centers on uncertainty estimation. The authors could consider aligning the main content more closely with the stated motivation to improve coherence.
2. Figure 1 suggests that pretraining alone may be sufficient for handling incomplete inputs. Therefore, it remains unclear how missing modalities or incomplete inputs impact the adaptation of pretrained multimodal models.
3. The novelty of this paper may be limited, given that latent-space reconstruction and Bayesian uncertainty estimation are well-established techniques [1][2][3]. The authors should elaborate on how these approaches uniquely enhance the adaptation of pretrained multimodal models for downstream tasks, as their current combination appears somewhat trivial.
4. The authors describe the proposed Pearson Correlation Coefficient (PCC) Loss as a 'distribution-free' loss function. However, the notation in equation (4) suggests that the PCC loss still relies on the Gaussian assumption. Additionally, the uncertainty constraint for PCC loss also appears to be derived from this Gaussian assumption. The authors should further clarify the “distribution-free” property of the proposed PCC loss.

[1] Abdar, Moloud, et al. "A review of uncertainty quantification in deep learning: Techniques, applications and challenges." Information fusion 76 (2021): 243-297.

[2] Lian, Zheng, et al. "GCNet: Graph completion network for incomplete multimodal learning in conversation." IEEE Transactions on pattern analysis and machine intelligence 45.7 (2023): 8419-8432.

[3] Ma, Mengmeng, et al. "Smil: Multimodal learning with severely missing modality." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 3. 2021.
Yes
No ethics review needed.
See Weaknesses.
1. The proposed method achieves good results on multiple datasets.
This paper introduces latent space reconstruction and uncertainty estimation to address missing modality issues. Experiments validate the effectiveness of the proposed method.
1. The authors leverage a pre-trained model and only fine-tune the reconstruction module and the prediction head. This pipeline is similar to prompt-based methods [1,2]. Therefore, these baselines should be incorporated in the experiments.
2. The authors claim that existing methods overlook the unreliability of reconstruction features which may damage the quality of the final outputs. However, in [2], they also perform a reconstruction process in the latent space and design prompts to enhance the reliability of the reconstructed features.  Meanwhile, [2] also leverages a pre-trained model. Therefore, the differences should be explained to highlight your contributions.
3. Why do you choose Pearson correlation instead of other correlation coefficient? Have you tried to use other nonlinear correlation coefficient?
4. Why not train the reconstruction module and classifier together?
5. The authors should report the results of all missing cases in Table 1-3. Besides, performances of different missing ratios can be added to enhance the paper.

[1] multimodal prompting with missing modalities for visual recognition, cvpr2023

[2] multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition, acl 2024
SURE enhances pretrained multimodal models by addressing missing modalities and small-scale datasets, combining latent space reconstruction with uncertainty estimation for both reconstruction and downstream tasks.
@inproceedings{
anonymous2024are,
title={Are you {SURE}? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=IT7LSnBdtY},
note={under review}
}
Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present **SURE** (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks.  Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal learning Uncertainty estimation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5c1c68890eed6e2766700a7d1a60e73ed0da9162.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission