Deconstructing Denoising Diffusion Models for Self-Supervised Learning
denoising diffusion models;denoising autoencoder;self-supervised learning
Yes
No ethics review needed.
**1.** The authors mainly focused on investigating the representation learning abilities of DiTs. Are there similar observations on U-Net-based diffusion models? 

**2.** Based on the experimental results, can we conclude that adding noise primarily impacts the generation capabilities of diffusion models rather than their representation learning ability? Are there any insights for this observation?
**1.** In general, this paper contributes significantly to the intersections of diffusion models and representation learning.  The findings open up new avenues for future research in leveraging diffusion processes to enhance representation quality across diverse applications

**2.** The authors conducted extensive experiments to support their results. The paper is well-written and easy to follow.
In this paper, the author studied the representation learning abilities of denoising-autoencoder-based diffusion models (DDMs). Throughout extensive ablation studies, they explored how various components of modern DDMs influence self-supervised representation learning. At the core of their philosophy is to deconstruct a DDM, changing it step-by-step into a classical DAE. This research process demonstrates that the main critical component for a DAE to learn good representations is a tokenizer that creates a low-dimensional latent space.
The reviewer has the following major concerns about this paper: 

**1.** It is not comprehensive to study the representation ability of diffusion models only by considering the classification of downstream tasks. The authors should provide more experiments on other tasks to support their conclusions. 

**2.** Although the observations of this work are really new and interesting, the authors seem to not fully discuss the implications of these findings.
Yes
No ethics review needed.
1. What are the FID scores for other modifications beyond Table 1, such as operating in the image space with PCA?

2. Does better classification accuracy always lead to worse FID scores? In other words, are the tasks of generation and representation learning (for recognition) fundamentally contradictory to each other? Or could we unify them?

3. Why is the middle layer of the transformer chosen as the representation for linear probing?
Studying the representation learning ability of denoising diffusion models is important and could provide useful insights for unifying models for generation and discrimination.

Overall, the paper is very well written. The experiments are solid, and the message is clear. The observation that a low-dimensional latent space is a critical component for representation learning is useful.
This paper studies the representation learning ability of denoising diffusion models. Through a set of ablation studies that deconstruct a denoising diffusion model into a classical denoising autoencoder (DAE), the authors observe that only a very few modern components (such as a low-dimensional latent space) are critical for learning good representations. Experiments also show that a latent DAE, which largely resembles the classical DAE, can perform competitively in self-supervised learning.
1. While the study begins with denoising diffusion models, it ultimately leads to models that demonstrate strong representations for classification but not for generation. The FID is reported only in Table 1, which reveals a significant contradiction between classification accuracy and FID. 

3. For the goal of representation learning for classification without fine-tuning, the obtained latent DAE achieves slightly worse performance than MAE and contrastive learning.

4. The representation is extracted from the middle layer of the transformer for linear probing. Previous studies have found that the middle layer may not provide the best representation of a diffusion model for classification.
Yes
No ethics review needed.
1. L 247: The paper claims "self-supervised learning performance is not correlated to generation quality". However, the selected tasks such as linear probes only consider coarse representations that are useful for high-level tasks. What about more fined representations such as segmentation, and positions of a specific object, etc, would it be correlated to generation quality?
2. L 369: why the DAE is expected to work directly on the image space, could you please explain the importance of working on the image space?
(1) The paper has identified a thorough set of different components between classical DAE and modern diffusion models.

(2) The paper sheds light on a potential framework for understanding the representation ability of diffusion models - DAE.

(3) The paper is the first work trying to identify key representation components from generative models and could inspire future works.

(4) The paper has some interesting findings, such as the low-rank tokenizer is important, and the high-level representation ability may not correlate with the generation ability.
The paper studies the representation ability of generative denoising diffusion models (DDM). It specifically aims to identify crucial components for DDMs' representation ability during the process of removing modern components in DDMs until it becomes a simpler model very similar to classic Denoising Autoencoder (DAE). Notably, DAE is originally proposed for representation learning. The paper is highly empirical, conducting various experiments on different components, such as different loss terms, different tokenizers, class-conditioning, noise schedule, whether to predict clean data, etc. It can remove many components designed for generation, and show high-level representation abilities are not strictly related to generation ability.
(1) The paper only presents empirical findings, with no theoretical analysis or practical applications.

(2) The complex possible choices of components make the experiment order not strictly natural and logical. 

(3) Missing experiments on some possible choices of components make the conclusions of the paper not that strong, for example, it's hard to conclude whether predicting clean images is more helpful than predicting noise for representation learning.
Yes
No ethics review needed.
1. Could you explain on why noise scheduling can be considered a form of data augmentation? Is there any ablation study showing that the effects of noise scheduling and data augmentation are comparable?
2. Regarding the section on predicting the original image (Lines 392–406), shouldn't the matrix $V$ be of size $D \times d$? Also, if the weights $w_i$ are all 1, the loss would again become the reconstruction loss on the latent space. This suggests that the relative scale of these weights is important. Do you have any ablation studies on this aspect?"
1. The authors aim to bridge modern self-supervised learning with classical methods by deconstructing diffusion denoising models (DDMs) and propose a simple but effective representation learner similar to traditional denoising autoencoders (DAEs).

2. Through this novel deconstructing process, the authors provide key insights. For instance, they highlight the importance of the low-dimensional latent space into which the tokenizer maps the patches, as it plays a crucial role in learning robust representations. Additionally, they demonstrate that adding noise in the latent space is more effective than in the pixel space, a finding that invites further exploration of latent space structures.
In this paper, the authors perform an extensive ablation study on modern Denoising Diffusion Models (DDMs) to identify the critical components for effective representation learning, comparing these models to traditional denoising autoencoders (DAEs). The authors begin by comparing DDMs and DAEs on their design and purpose, indicating that generation (DDM objective) is not directly connected with good representations (DAE objective), and suggesting a possible trade-off between these them.

Then the authors test various modifications to make DDMs more similar to DAEs while preserving high-quality representations. Notably, they find that the tokenizer is beneficial mainly for its dimensionality reduction role and that a simplified approach, like patch-wise PCA, can serve this function without compromising performance. They also identify less critical components: noise scheduling (analogous to data augmentation), class conditioning (which may lessen the model’s need to capture class-specific, fine-grained semantics), and predicting noise versus the original image.

Based on these findings, the authors propose **l-DAE**, a DAE that operates in the latent domain. They validate the effectiveness of l-DAE through experiments on classification and object detection tasks, showing its competitive performance.
1. In the paper, the authors make numerous claims but test them only on a specific task (ImageNet classification) and model (DiT-L). Furthermore, the deconstructing process faces a limitation: the components may be correlated, making a sequential analysis potentially inadequate.
2. Some of the claims may be overlooked. For instance, the statement that 'multiple levels of noise is analogous to a form of data augmentation' (lines 416–418) may be overly simplified. Prior research (https://arxiv.org/abs/1406.3269) has shown that combining representations at different noise levels can lead to significant improvements."
deconstructing a modern denoising diffusion model to a classical denoising autoencoder for self-supervised learning
@inproceedings{
anonymous2024deconstructing,
title={Deconstructing Denoising Diffusion Models for Self-Supervised Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=9oMB6wnFYM},
note={under review}
}
In this study, we examine the representation learning abilities of Denoising Diffusion Models (DDM) that were originally purposed for image generation. Our philosophy is to deconstruct a DDM, gradually transforming it into a classical Denoising Autoencoder (DAE). This deconstructive process allows us to explore how various components of modern DDMs influence self-supervised representation learning. We observe that only a very few modern components are critical for learning good representations, while many others are nonessential. Our study ultimately arrives at an approach that is highly simplified and to a large extent resembles a classical DAE. We hope our study will rekindle interest in a family of classical methods within the realm of modern self-supervised learning.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
denoising diffusion models denoising autoencoder self-supervised learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c5a63e42b92850e1995401b826d0a422ff97ca83.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Deconstructing Denoising Diffusion Models for Self-Supervised Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission