Multiple Descents in Unsupervised Auto-Encoders: The Role of Noise, Domain Shift and Anomalies
double descent;unsupervised learning;autoencoders;domain adaptation;bias-variance curve;anomalies
Yes
No ethics review needed.
Please refer to the weakness section for the questions.
The paper investigates double descent in the unsupervised setting, which did not receive attention as much as supervised learning or self-supervised learning.

The paper has the following contributions:
1. The paper shows how the four different variables (sample noise, feature noise, domain shift, and anomalies) affect the double descent.
2. The paper connects observations of the synthetic and real-world datasets.
3. It is interesting to see that although there are four different variables, many of them share the same intuition (the magnitude or portion of the noise controls the vertical height and the horizontal position).
The paper investigates the double descent phenomenon in the unsupervised setting. The experiments on synthetic datasets explore the relation between double descent and four variables: sample noise, feature noise, domain shift, and anomalies. Moreover, some observations remain for real-world datasets.
The paper is not smooth to read. Some details are not mentioned or explained. For example, providing some details regarding the under-complete AEs could help the readers understand why the experiments should be based on under-complete AEs. Besides, the experiment sections contain many figures and they could be rearranged for a smoother reading experience.

As for the experiment section specifically:
1. How is Fig.2 different from the traditional figure of test loss against the number of parameters? They seem to convey the same information. Does it mean the relation between the two in unsupervised learning is the same as supervised learning?
2. Some curves are missing. For example, in Fig.3a various levels of sample noises are shown, but many of them are missing in Fig.3b. Could you provide all the curves?
3. I recommend the authors put a paragraph or a table to summarize the relation between the double descent and the four variables as they share very similar behavior and intuition. It will be easier for the reader to access the experiment outcome.
4. For the paper to be more informative, the author could compare the double descent behaviors in unsupervised learning against the supervised learning regime. Could you conclude if there is any different behavior in the unsupervised setting than the ones observed in the other settings?
Yes
No ethics review needed.
1. Does $d_ij$ in Equation 3 have the same meaning as the dependence score in the Method part? I think it is confusing because the first one should be a conditioned set, not a score. Additionally, where does the $max_ij$ come from in the Method section? Is it part of the greedy algorithm for constructing the tree structure? Please notify the symbol clearly before using it.

2, Could you explain why the performance on certain datasets, such as MNIST, is much lower than MLP-based methods? Moreover, in Table 3, it seems that the F1 score for KNN is consistently the best among classical OD methods across most of datasets, often surpassing your method by a significant margin in both low-dimensional and high-dimensional cases.

3, What are your thoughts on the potential of classical compared to MLP OD methods? If you believe that improving classical methods from performance side offers more potential than increasing interpretability through deep learning, could you briefly state your reasons in the discussion section?
1, The motivation of this paper is stated very clearly, addressing the curse of high dimensionality, using univariate empirical modeling, and selecting the vine copula method to capture paired interactions hierarchically to model high-dimensional distributions.
2, This method is highly interpretable; in addition to providing a score to determine whether an observation is an outlier, it also offers insights into which features contribute to this conclusion and whether the outlier status is due to violations of individual feature distributions or interactions between features.
3, The paper provides a discussion about the choice of the hyperparameter T, which is critical when using truncated vine copulas. In Section 6.1, it explains how the hyperparameter T affects performance and computational time, as well as the trade-off between information loss and overfitting.
4, The paper is well-written and easy to follow, particularly in terms of its motivations and theoretical background.
This paper proposes a new method called Vine Copula-Based Outlier Detection (VC-BOD) based on Copula to detect outliers in high-dimensional data with classical manner. Compared to MLP methods, it increases the interpretability of the reason why observations are detected as outliers. To be specific, it can provide information about which feature conducts the anomalies and whether it is violating its own distribution or interacts uncommonly with other features. By utilizing Vine Copula and under Sklar’s theorem, it divides the problem by individual marginal distribution and paired feature Copula to capture the interactions and dependencies. 
By carefully selecting the hyperparameter, it can learn the major dependencies across features while stay computationally tractable under high dimensional setting. Overall, VC-BOD is a powerful, interpretable alternative to deep-learning-based models for outlier detection, especially in scenarios that require transparency and explainability.
1, The related work section is overly detailed, including some methods that don’t have a strong connection to the paper’s motivation.
2. The notation used in the paper is somewhat confusing and needs clarification.
3, The analysis of experimental results is insufficient, and the results do not demonstrate outstanding performance across all classical outlier detection (OD) methods, particularly in comparison to KNN.
Yes
No ethics review needed.
- In practice, noise is not always Gaussian. How would different noise distributions impact the behavior of multiple descent?
- Could the paper provide theoretical insights or hypotheses on why multiple descent occurs under conditions of low SNR, domain shift, and anomalies, and how critical regions might be characterized more systematically?
- Are there practical ways to predict the occurrence of multiple descent based on data characteristics or noise conditions, making it more applicable for real-world scenarios? 
- Given that SNR is challenging to compute from data without specific model assumptions about signal and noise, what are some methods or approximations to determine SNR in real-world data?
- For the KNN-DAT experiment (Fig 12), what would be the relationship of test loss curve with different levels of source-target distribution shift? (e.g. measured by distribution distance such as KL divergence or Wasserstein distance) Does the relationship confirm with the simulated experiment?
* This paper addresses an important and timely topic in understanding the generalization behavior of over-parameterized models.
The paper is well-written and easy to follow. It is interesting to see that under low SNR, over-parameterized models’ ability to fit (or memorize) noise can actually improve generalization when encountering domain shifts.
* The experiments are thorough, and the analysis of results provides insightful observations.
This paper investigates the phenomenon of multiple descent in unsupervised learning paradigms, specifically using under-complete auto-encoders (AEs). Contrary to prior studies that suggest double descent does not occur in unsupervised learning, this work empirically demonstrates the presence of multiple descent under conditions of low SNR, significant anomalies, and domain shifts. Additionally, it identifies scenarios where multiple descent affect test loss, providing new insights to model selection and training with over-parameterized networks.
-  While the paper highlights the roles of noise, domain shift, and anomalies in multiple descent, providing further theoretical insights into the underlying mechanisms could enrich the understanding of why this phenomenon occurs.
- Defining critical regions for multiple descent more explicitly, possibly through some kind of scaling law like function characterizations, could make the findings more actionable and help readers choose appropriate model complexity and stopping condition in practical settings.
Yes
No ethics review needed.
Is it possible to define the double descent phenomenon for unsupervised learning models other than autoencoders?

How can the use of test loss to define predictive performance in unsupervised learning be justified? From the perspective of distribution learning, shouldn't distributional similarity serve as the performance metric?
This study also provides empirical examples related to the generalization performance of overparameterized models in unsupervised learning. These findings will encourage further research on model complexity in unsupervised learning.
This paper experimentally discovers the double descents (multiple descents) phenomenon in autoencoder models and explains this phenomenon to sample noise, feature noise, domain shift, and outliers. The double descent phenomenon represents a specific pattern in the generalization performance of a model based on its size. Notably, this phenomenon has provided significant motivation for studying the generalization performance of overparameterized models.
Although the paper claims to present extensive simulation results demonstrating the double descent phenomenon in unsupervised learning, the experiments primarily seem to involve regression tasks. This interpretation arises because the study examines the performance changes in autoencoders by adding noise to data or features, which can essentially be understood as fitting an arbitrary target variable, similar to a regression model. In this sense, the results could be viewed within the same context as previous findings observed in supervised learning.

Moreover, discovering multiple descent patterns may not offer a new perspective based solely on the current simulation results. Additional discussion and analysis would likely be necessary to substantiate this claim.

Before discussing the double descent phenomenon in unsupervised learning, it is essential first to define the performance metric in the context of unsupervised learning. If unsupervised learning is limited to distribution learning, I suggest defining the performance metric based on the distance to the target distribution and examining how performance varies with model size. Suitable models for unsupervised learning would include those capable of distribution learning, such as VAE, GAN, or diffusion models. Alternatively, a simpler experimental approach could involve using a clustering task.
Yes
No ethics review needed.
see weaknesses
- The presented paper shows some rigorous analysis of the double descent phenomenon under different settings
- The paper is well written and easy to follow.
The authors study the phenomenon of double descent appearing in over-parametrized machine learning models. Where recent findings focused on supervised learning problems, here the authors focus on unsupervised learning specifically studying autoencoders. Considering the evolution of training loss with respect to different data modifications (sample noise, feature noise, domain shift, and anomalies), they find multiple descent phenomena appearing under different settings, yielding insights into the behaviour of learning of autoencoders studying both synthetic and real data.
- The takeaway of this paper is not exactly clear. What can we do with these insights on Autoencoders, also given that the CNN AE -- probably the more relevant model in nowadays application -- presented in the appendix does not show the typical double descent with progressing training? The discovered connection between model size and performance does not seem so surprising to me. An extensive discussion and recent application of AE s would strongly benefit the paper. As a constructive suggestion, the authors could look into Sparse Autoencoders, which recently are in heavy use in XAI [1] and even have been adapted by Google in their Gemini project [2], and insights into their training dynamics would be of great use. Note that a smaller scale example on a standard vision benchmark could suffice here.
- The paper lacks a more theoretical discussion of *why* what we see is happening. The recent works shedding light onto this and showing that double descent is not in conflict with classical statistical learning theory could serve as a starting point [3,4] and should also be appropriately addressed in intro and related work.

[1] L. Gao et al., Scaling and evaluating sparse autoencoders (https://cdn.openai.com/papers/sparse-autoencoders.pdf)

[2] https://deepmind.google/discover/blog/gemma-scope-helping-the-safety-community-shed-light-on-the-inner-workings-of-language-models/

[3] A Curth* et al., A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning NeurIPS 2023

[4] A Curth* Classical Statistical (In-Sample) Intuitions Don’t Generalize Well: A Note on
Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs arXiv 2024

*no relation to the reviewer
@inproceedings{
anonymous2024multiple,
title={Multiple Descents in Unsupervised Auto-Encoders: The Role of Noise, Domain Shift and Anomalies},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=YBv9EExJPk},
note={under review}
}
The phenomenon of double descent has recently gained attention in supervised learning. It challenges the conventional wisdom of the bias-variance trade-off by showcasing a surprising behavior. As the complexity of the model increases, the test error initially decreases until reaching a certain point where the model starts to overfit the train set, causing the test error to rise. However, deviating from classical theory, the error exhibits another decline when exceeding a certain degree of over-parameterization. We study the presence of double descent in unsupervised learning, an area that has received little attention and is not yet fully understood. We conduct extensive experiments using under-complete auto-encoders (AEs) for various applications, such as dealing with noisy data, domain shifts, and anomalies. We use synthetic and real data and identify model-wise, epoch-wise, and sample-wise double descent for all the aforementioned applications. Finally, we assessed the usability of the AEs for detecting anomalies and mitigating the domain shift between datasets. Our findings indicate that over-parameterized models can improve performance not only in terms of reconstruction, but also in enhancing capabilities for the downstream task.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
double descent unsupervised learning autoencoders domain adaptation bias-variance curve anomalies
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/24c7fc33f3b6a1c0c1f186e467c7e28a688eacec.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/831e53acedc80d0919013d7503757e7bc8fc1ec5.zip
Multiple Descents in Unsupervised Auto-Encoders: The Role of Noise, Domain Shift and Anomalies
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission