Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)
Deep Learning;Contrastive Learning;Neural Collapse;Image Classification
Yes
No ethics review needed.
please refer to the weakness part.
1.	The paper tackles the critical issue of dimensional collapse within semi-supervised learning scenarios.
2.	This paper provides some theoretical clues to support their claims.
3.	The paper is generally well structured and easy to follow.
This paper addresses the dimensional collapse problems within the semi-supervised contrastive learning paradigm. This paper first provides theoretical insights into the influence of large learning rates and cosine similarities on dimensional collapse. To mitigate this issue, the authors introduce a method, termed CLOP, aimed at enhancing subspace separation.  Experimental results on CIFAR and Tiny-ImageNet demonstrate the performance improvements of the algorithm.
1.	The paper appears to conflate two distinct concepts: dimensional collapse and neural collapse. Dimensional collapse in semi-supervised learning (SSL) typically refers to the limited dimensionality of learned SSL features, while neural collapse is associated with a desirable state in supervised training marked by several good qualities such as intra-class alignment and inter-class separation (properties NC1-NC4).
2.	The claim regarding orthogonal structures achieving the most distinguishable classes relative to the Simplex ETF structure is unclear. Simplex ETF theoretically maximizes angular separation, reaching a pairwise cosine value of -1/(k-1), whereas orthogonal structures attain pairwise cosine values of zero.
3.	The issue of dimensional collapse is primarily addressed by the SSL community; however, the paper's focus is on semi-supervised methods, yet it lacks comparison with other semi-supervised baselines.
4.	The proposed method should be compared with ETF-based(or uniform variants) methods[1] to adequately demonstrate its effectiveness. Moreover, to comprehensively address dimensional collapse, additional self-supervised experiments are recommended, including comparisons with ETF-based SSL methods [2] and other methods targeting dimensional collapse [3,4].
5.	Experiments are limited to small-scale datasets, such as CIFAR and Tiny-ImageNet, which may restrict the generalizability of the findings.
6.	The performance improvements are not significant on larger batch sizes.

[1] Targeted supervised contrastive learning for long-tailed recognition.

[2] Combating Representation Learning Disparity with Geometric Harmonization.

[3] Understanding dimensional collapse in contrastive self-supervised learning.

[4] Variance-invariance-covariance regularization for self-supervised learning.
Yes
No ethics review needed.
Please address concerns above.
A theoretical upper bound of the learning rate for collapse is provided.
This paper considers the problem of neural collapse in contrastive learning. The authors theoretically investigate the effect of large learning rate in contrastive learning with cosine similarity and provide a bound for the learning rate to avoid the collapse. The authors also propose a semi-supervised contrastive loss. Experimental results on small image datasets support the claim.
- The definition of neural collapse is incorrect. Neural collapse refers to the phenomenon that within-class variance becomes zero. Although it would result in low-rank representations, "embeddings converge into a lower-dimensional space" is insufficient to properly describe neural collapse. Please refer to [Papyan et al.] that mainly discuss neural collapse, which is not cited in this paper.

[Papyan et al.] Prevalence of Neural Collapse during the terminal phase of deep learning training. PNAS 2021.

- The bound seems to be not practical. 2 is too large learning rate for contrastive learning.

- While experimental results are sensitive to the batch size, there is no discussion about the batch size throughout the theoretical analysis. 

- Experimental setting is limited to small image datasets without transfer learning, so its generalizability is questionable.

- No ablation study on the choice of the similarity function, while the authors claim that the proposed method is better than the one with consine similarity.

- Does Lemma 2 still hold when Eq. (3) is used?
Yes
No ethics review needed.
Please see weaknesses
*Theoretical Analysis:* The paper provides a theoretical analysis of neural collapse, focusing on the impact of large learning rates in contrastive learning settings. The theoretical insights could offer useful guidance for practitioners concerned with collapse issues in representation learning.

*New Loss Design:* The proposal of CLOP, a loss function to enforce orthogonality among embeddings, is a potentially valuable direction for addressing neural collapse. It could add diversity to the current strategies used in self-supervised and semi-supervised learning frameworks.
The paper introduces a contrastive learning method called CLOP, which aims to address the issue of neural collapse by leveraging orthonormal prototypes. The authors theoretically analyze the collapse phenomenon in self-supervised and semi-supervised contrastive learning setups and propose a novel loss function to mitigate collapse by enforcing orthogonal subspace constraints. Experiments on CIFAR-100 and Tiny-ImageNet suggest that the proposed method stabilizes performance under varying batch sizes and learning rates.
*Motivation:* The primary motivation of CLOP—to prevent neural collapse and enhance feature diversity—bears significant similarity to VICReg, which also aims to mitigate collapse by enforcing variance, invariance, and covariance constraints on representations. Although CLOP takes an orthogonal approach, it does not sufficiently differentiate itself from VICReg in either theoretical justification or practical application. The theoretical analysis mainly emphasizes the necessity for spatial separation, an objective already addressed by VICReg, leading to questions about the novelty of the approach.

*Insufficient Comparative Analysis:* While VICReg is briefly acknowledged, the paper lacks a robust empirical comparison with VICReg to establish CLOP’s superiority or complementary advantages. In particular, the experiments fail to demonstrate a distinct performance edge over VICReg, which diminishes the contribution of the proposed method. For this paper to make a convincing case for CLOP, direct comparisons, especially in scenarios prone to collapse, would be necessary.

*Experimental Design Issues:* The choice of CIFAR-100 and Tiny-ImageNet as benchmark datasets provides limited insights into the scalability and effectiveness of CLOP in larger and more complex real-world scenarios. Furthermore, while CLOP is tested under different batch sizes and learning rates, the experiments do not include variations in data augmentation or pretext tasks, which are known to impact the success of contrastive learning methods.
We conduct a theoretical analysis of neural collapse in contrastive learning and introduce a novel loss function to mitigate this problem.
@inproceedings{
anonymous2024preventing,
title={Preventing Collapse in Contrastive Learning with Orthonormal Prototypes ({CLOP})},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=SsWMJ42hJO},
note={under review}
}
Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, neural collapse, where embeddings converge into a lower-dimensional space, poses a significant challenge, especially in semi-supervised and self-supervised setups. In this paper, we first theoretically analyze the effect of large learning rates on contrastive losses that solely rely on the cosine similarity metric, and derive a theoretical bound to mitigate this collapse. Building on these insights, we propose CLOP, a novel semi-supervised loss function designed to prevent neural collapse by promoting the formation of orthogonal linear subspaces among class embeddings. Unlike prior approaches that enforce a simplex ETF structure, CLOP focuses on subspace separation, leading to more distinguishable embeddings. Through extensive experiments on real and synthetic datasets, we demonstrate that CLOP enhances performance, providing greater stability across different learning rates and batch sizes.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Deep Learning Contrastive Learning Neural Collapse Image Classification
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e3c4298b30eb20fd1d76f009e38922bc0675d536.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/5989ebbf2cfaf2639883e01dc36af36edd0bbef1.zip
Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission