On the Relation Between Linear Diffusion and Power Iteration
Diffusion models
Yes
No ethics review needed.
(1) what are $x_j$ and $n$ in eq. (26)? 
(2) where $v_j$ right after eq. (26) should be $u_j$.
(3) line 487: what is $J$ in $sin(\theta_J)$? is it $i$?
The authors aim to provide an analysis of the diffusion process, an important question for diffusion models. This is achieved using a simple model.

Overall, the writing is easy to follow.
This paper studies the generation process of diffusion models. Focusing on linear models where the diffusion process is similar to performing noisy PCA, the authors show that the generation process of the linear diffusion model is similar to the power iteration method in that it converges to the leading eigenvector of the underlying data distribution.
Providing theoretical studies for complex problems in deep learning and deep generative models is often very challenging. Thus, it is common practice to study a simpler problem that can shed light on the underlying mechanisms of more complex ones. While this work falls into this category, I found that the results for the linear diffusion model diverge significantly from the phenomena observed in real cases.

(1) Theorem 4.3 shows that the diffusion process only converges to the dominant eigenvector (the one corresponding to the largest eigenvalue). Thus, it gives a bad estimation for the true distribution with a dimension larger than 1. While eq. (25) also provides a formulation when noise is injected in the intermediate steps, (i) it does not provide an estimation guarantee for the entire spectrum, (ii) many recent diffusion models such as DDIMDMP-solver do not require the injection of noise in the intermediate steps.

(2) The results in Section 5 (e.g, Figure 7) on real cases only demonstrate the convergence to the entire spectrum, not the dominant eigenvector. This also supports the gap between the analysis in Theorem 4.3 and the phenomenon observed in real cases. 

(3) Initially, I thought Section 5 was intended to support the argument that the diffusion process converges to the "true distribution" as in Theorem 4.3. However, this is not the case, as Figure 7 plots the convergence of the network Jacobian evaluated at $x_t$. Thus, even at $t= 0$, it depends on the learned network and does not represent the true distribution of the data; the convergence probably holds as long as the network is Lipschitz. In other words, the convergence does not indicate the quality of the diffusion process. For example, a network that outputs all zeros would also exhibit fast convergence. I may be misinterpreting the results; if so, I'd appreciate clarification.
Yes
No ethics review needed.
Please see my questions above.
The paper has a clear structure and is easy to read. The topic is also interesting since studying the evolution of the denoisers along diffusion trajectory is important but less well explored in the literature.
This paper explore diffusion reverse sampling process by analyzing a linear diffusion model. The main result is that the reverse sampling process leads to samples that concentrate on the first eigenvector of the covariance matrix of the underlying data.
I have doubt on both the theoretical and practical aspects of this work. The theoretical results do not seem correct (correct me if I am wrong) and the connection to practice is weak. I list my questions as below:

*Major questions:

(i) Can the author explain how do they conclude Theorem 4.3 ? The author claimed that the final projection operator is a diagonal matrix with a spectrum concentrate around the first eigenvalue (line 409-412) and they supported this claim qualitatively in Figure 5. However, I cannot find a close-form expression for this projection matrix as in the limit of $\tau\rightarrow \infty$. From (24) I would assume 
\begin{align}
\mathbb{E}\mathcal{P_{\tau}}=U_0 c_0^\tau\text{diag}[1, (\frac{c_1}{c_0})^\tau,...] U_0^T,
\end{align}
, which do not really hold since equation (20) only holds for small t. If we assume this expression holds, we have 
\begin{align}
\mathbb{E}x_gx_g^T=(U_0c_0^{2\tau}\text{diag}[1, (\frac{c_1}{c_0})^{\tau},...])(\mathbb{E}[\epsilon_T\epsilon_T^T])(\text{diag}[1, (\frac{c_1}{c_0})^{\tau},...]U_0^T).
\end{align}

By the assumption that diffusion stepsize is $\frac{1}{T}$ and there are $T$ steps in total (line 360-361), we have $\epsilon_T\sim\mathcal{N}(0,\frac{1}{T})$. Therefore $\mathbb{E}x_gx_g^T=U_0\text{diag}[c_0^{2\tau}/T, c_1^{2\tau}/T,...]U_0^T$. Then how does this term equal to $u_0u_0^T$ as in equation (17)? Shouldn't all eigenvectors go to zero as $T\rightarrow \infty$ ? 


(ii) In section 5 the author claims that correlation of the low indices (lower frequencies) withstands higher noise levels. However, it is shown in Figure 7 that many of the dark curves remain flat during a wide range of time steps and only drop when t approaches 0, which is very different from linear case (Figure 3) that show a gradual evolution of the sine angles. Therefore I disagree with that the decaying behavior of the nonlinear model  is similar to that in the linear case.

*Minor questions:
(i) The Figure 7 is very difficult to read since all the useful information is concentrated near $t=0$. I suggest the authors to redraw the plot using log scale.

(ii) The paper could benefit from putting more details on the theoretical results in an appendix. Not sure whether authors are allowed to add one.
Yes
No ethics review needed.
- Most experiments for linear case are done on MNIST- could the authors clarify to what extent the assumptions made in the spiked covariance model apply to real-world datasets? For example, in cases where the low-dimensional subspace assumption does not hold.
- Moreover, since power iteration is such a major theme of the paper, a small discussion of the power iteration method in the introduction would help enhance readability.
- The term $u_i^t$, has never been defined. Moreover, a consistent notation of keeping time in the superscript and index in the subscript  (or vice-versa) for both vectors and matrices will help avoid confusion.
The paper provides a novel perspective on diffusion models for understanding the generation process by drawing parallels with power iteration. The work opens up new research directions for improving the interpretability and performance of generative models by highlighting the eigenvector alignment properties in diffusion models. The paper is well-written with most claims justified theoretically and backed by empirical evidence.
The explores the relationship between the power iteration method and diffusion models with a linear denoiser, that can be modeled as a PCA projection, enabling analysis via the spiked covariance model. The authors demonstrate that, in the reverse process, high eigen-value components emerge earlier, and leading eigenvectors align with the data distribution.
The paper does a good job at conveying the idea and intuition from the simulations, however, the proof of the main result (Theorem 4.3), could be refined more. Particularly, I can not fully understand the proof in the high noise regime i.e., $\tau \leq t \leq T$. A more explicit explanation for the following statement would enhance the rigor: how exactly does Assumption 4.2 guarantee that $U_\tau^\top U_{\tau+1}$ is diagonal just enough not to spoil the diagonality of the next partial operator $\mathbb E \mathcal P_{τ+1}$? Can the authors justify this mathematically?
We analyze linear diffusion models, showing that generation acts as a correlation machine, aligning noise with data's principal components. We connect it to power iteration, explaining low-frequency emergence and convergence in deep denoisers
@inproceedings{
anonymous2024on,
title={On the Relation Between Linear Diffusion and Power Iteration},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=mKM9uoKSBN},
note={under review}
}
Recently, diffusion models have gained popularity due to their impressive generative abilities. These models learn the implicit distribution given by the training dataset, and sample new data by transforming random noise through the reverse process, which can be thought of as gradual denoising. In this work, we examine the generation process as a ``correlation machine'', where random noise is repeatedly enhanced in correlation with the implicit given distribution. 
To this end, we explore the linear case, where the optimal denoiser is known to be the PCA projection. This enables us to connect the theory of diffusion models to the spiked covariance model, where the dependence of the denoiser on the noise level and the amount of training data can be expressed analytically, in the rank-1 case.
In a series of numerical experiments, we extend this result to general low rank data, and show that low frequencies emerge earlier in the generation process, where the denoising basis vectors are more aligned to the true data with a rate depending on their eigenvalues. This model allows us to show that the linear diffusion model converges in mean to the leading eigenvector of the underlying data, similarly to the prevalent Power Iteration method. 
Finally, we empirically demonstrate the applicability of our findings beyond the linear case, in the Jacobians of a deep, non-linear denoiser, used in general image generation tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/44b0c35720851420a0739e066fe4f6f810d169a4.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
On the Relation Between Linear Diffusion and Power Iteration
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission