Sparsely multimodal data fusion
multimodal;representation learning;contrastive;transformer;masked attention;sentiment;sparsity;embedding space
Yes
No ethics review needed.
The paper suggests novel contributions, but they are not sufficiently highlighted. Can you expand on how MCA distinguishes itself from existing methods in terms of theoretical innovations or novel mechanisms, beyond empirical performance? Providing explicit motivations for architectural choices would also make the contributions clearer and more impactful.
1. It addresses the challenge of missing modalities.

2. It extends the masked multimodal attention (MMA), improving its capability to handle incomplete data.

3. Improves the quality of learned embeddings.
The paper proposes a method called Modal Channel Attention (MCA) to enhance masked multimodal transformer models for learning from multimodal data, even in the presence of missing modalities. The approach extends existing masked multimodal attention (MMA) by incorporating channels for incomplete modalities, which significantly improves the quality of learned embeddings.
1. The contributions primarily emphasize performance improvements and extensions to the existing MMA. More detailed explanations of the specific contributions and the importance of the novelty would be beneficial. 

2. The technical section is quite brief (half a page), and it would be helpful if the figure showing the architecture and self-attention provided more detail. Moreover, part of the technical section is devoted to describing the architecture by listing details like activations, hidden size, and the number of attention heads. However, no explicit motivations are provided for these choices. Additionally, the figures should have been more curated to avoid text pixelation. Given the current shape of the paper, the novelties seem to be present, but the paper fails to properly highlight them, relying mostly on an empirical evaluation.
Yes
No ethics review needed.
No
The topic is very interesting and important.
This paper is not ready for publication in ICLR
The presentation is poor.
Yes
No ethics review needed.
Refer to the summary
Refer to the summary
**Summary of this paper:**

This work proposes an extension to the masked multimodal transformer model which incorporates modal-incomplete channels in the multihead attention mechanism called modal channel attention (MCA).

**Strengths:**

1.	This paper addresses an interesting problem by extending the masked multimodal attention model architecture to work with modality-incomplete datasets, a common scenario in real-world applications.

2.	The paper is straightforward.
 
3.	Experimental results demonstrate the effectiveness of the proposed method.

**Weakness:**

There is no visual comparison between the proposed architecture and previous structures, which would help clarify the improvements and differences.

**Comments, Suggestions And Typos:**

1.	It is recommended to add a figure in the introduction to briefly introduce the task of this paper, or to add a figure somewhere to compare the proposed model architecture with previous MMA architecture.

2.	Figure 1 is not good-looking, the authors are encouraged to improve the quality of the figure.

3.	For clarity and rigor, it is recommended to express important definitions or functions (i.e., modal sparsity) by formulas.

4.	The citation formats for figures differ in this paper.
Refer to the summary
Yes
No ethics review needed.
1.	In the MCA model, are important modalities treated differently from minor modalities? Or are different modalities given different weights in the attention mechanism?
2.	At which critical point does the performance of MCA significantly degrade as modal sparsity increases? Are there optimisation methods to maintain embedding quality at higher sparsity?
1. MCA provides a novel method for dealing with modal sparsity in multimodal transformers, addressing a common limitation in multimodal learning. The approach enhances the MMA model by adding a modality-aware attention channel, innovatively allowing robust embedding creation from incomplete datasets.
2. The study is rigorous, using two datasets (CMU-MOSEI and TCGA) to evaluate the model's effectiveness across varied tasks, including sentiment analysis and cancer classification. The experiments are thorough, with evaluations under multiple levels of modal sparsity and comparisons with established models.
3. This model has important implications for real-world applications where complete multimodal data is often unavailable, such as in medical data analysis and surveillance systems. MCA enables more resilient model applications by tolerating missing modalities without significant loss of performance, thus broadening the scope of multimodal transformers.
Traditional multimodal learning methods often assume that all modes are uniformly present in training and reasoning, which may be impractical in practical applications. To solve this problem, the authors propose a modal channel attention (MCA) model, which is an extension of the masked multimodal attention (MMA) approach. By integrating a dedicated attention mechanism, MCA achieves efficient fusion and embedding of multimodal data, even in the case of sparse modes. We evaluate MCA on the CMU-MOSEI and TCGA datasets, and the results show that MCA outperforms MMA in creating aligned, uniform embeddings with different modal sparsity. In downstream tasks such as multimodal sentiment analysis and tumor type prediction, MCA also shows higher recall rates and classification/regression performance, making it a versatile tool for real-world multimodal data fusion.
1. This paper does not explore in depth the interactions between modes to further explore the specific impact of different combinations of modes on MCA model performance. You may wish to consider exploring whether particular combinations are more conducive to improving model performance in order to gain a fuller understanding of the relative importance of each mode.
2. Although MCA is described as an efficient solution, this paper does not further quantify its computational efficiency and resource consumption on larger or more complex datasets to effectively assess the feasibility of its practical application.
3. The attention mechanism of the MCA model in dealing with missing modalities is still complex and not easy to understand intuitively. I would like to see the interpretability of the model in applications, which would help build user trust in the model's decisions.
4. The comparison experiments in this paper are limited to comparisons with the MMA model, which makes it difficult to fully demonstrate the relative strengths of the MCA model in multimodal approaches. You can evaluate the advantages and limitations of the MCA model more comprehensively by increasing the number of comparison experiments with multimodal methods other than MMA.
Yes
No ethics review needed.
No ethics review needed.
* As far as I know, CMU-MOSEI primarily provides information in three modalities, including audio, video, and text. I am not sure what exactly the four modalities the authors are referring to when they use them.

* The core structure presented in this paper is very similar to previous work [1], leading to limited improvements and contributions. I suggest that the authors clarify the differences between them in detail to highlight their contribution.

[1] Shvetsova, N., Chen, B., Rouditchenko, A., Thomas, S., Kingsbury, B., Feris, R. S., ... & Kuehne, H. (2022). Everything at once-multi-modal fusion transformer for video retrieval. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition (pp. 20020-20029).
* This paper demonstrates the importance of how to cope with representation learning in the case of incomplete modalities in real applications.

* The authors provide extensive implementation details and interesting experiments to demonstrate the need for the proposed structure.

* The authors' review of the relevant literature is well organized.
This paper presents a sparse multimodal fusion method in the presence of modal incompleteness. An extension to the masked multimodal transformer model is proposed which incorporates modal-incomplete channels in the multi-head attention mechanism called modal channel attention (MCA). The validity and usefulness of the method is verified by experiments on two multimodal datasets with different metrics and scenarios.
* This paper is clearly not ready for publication, including confusing typography, inadequate page counts for major sections, and formatting errors. For example, Fig and Figure are mixed up and line 221 is missing punctuation.

* The experimental section focuses on the comparative analysis of different metrics. However, the authors lacked an effective comparison with previous work to highlight the advantages of the proposed methodology.

* In addition, this study does not have enough qualitative analysis and ablation studies to explore the effect of different combinations of modal deletions in the proposed structure on performance.
Representation learning of multimodal data with improved rank and recall when samples have only a fraction of the set of modalities
@inproceedings{
anonymous2024sparsely,
title={Sparsely multimodal data fusion},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=iSLDihAfYi},
note={under review}
}
In order to create unified representations of multimodal data, masked multimodal transformer architectures use attention masking to constrain attention to occur between unimodal and fused intermediate representations. In this study, it is demonstrated that they can create an aligned embedding space when samples include only a fraction of data modalities. This is done by measuring the quality of generated embedding spaces as a function of modal sparsity. An extension to the masked multimodal transformer model is proposed which incorporates modal-incomplete channels in the multihead attention mechanism called modal channel attention (MCA). Two datasets with 4 modalities are tested, CMU-MOSEI for multimodal sentiment recognition and TCGA for multiomics. Models are shown to learn uniform and aligned embedding spaces with only two out of four modalities in most samples. It was found that, even with no modal sparsity, the proposed MCA mechanism improves the quality of generated embedding spaces, recall metrics, and subsequent performance on downstream tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
multimodal representation learning contrastive transformer masked attention sentiment sparsity embedding space
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/44fc6f17150c4b3abed0b223ba06207076c1c972.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/18051eb5b1effb2554bb4caf871bf8c0c7e43512.zip
Sparsely multimodal data fusion
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission