On the Interpolation Effect of Score Smoothing
score-based diffusion models;score smoothing;data interpolation;generalization vs memorization;subspace recovery
Yes
No ethics review needed.
1. Line 057 $\sigma$ should be $t$ instead? otherwise there's no reference to $t$ in the abbreviation/RHS of the equation.
2. Whilst big O notation is standard, I think its a bit cleaner and more common to write these results  with $\leq$
3. As mentioned in the weakness its clear the smoothed score induces a controllable loss, but I think to complete the story the authors need to discuss / motivate why would we learn anything akin to the specific choice of smoothed score used in this work ?  I understand this is mentioned in the limitations but maybe something earlier in the introduction would be helpful.
1. The paper starts of a simpler and more accessible 1d 2 data points analysis to motivate their insights to the reader before moving onto the more technical results.
2. The paper's insights into their smoothed ESF approximation seem valuable to the community. In particular, it shows that the KL regarding a uniform distribution is bounded (unlike if using the ESF directly) and quite neat.
This paper theoretically explores  the score of variance exploding SDEs (VE-SDEs) to showcase the success in score based diffusion models by arguing that the smoothing of the data-score generating novel samples that interpolates across the training data subspace.
I think the paper could do a better job at motivating their results for a wider audience:

1. Equation (8) can and should be defined before Lemma 1 , in Lemma 1 you throw in the smoothed score defined in an implicit way and then you redefine it and label it it equation 8, it would be better to introduce equation 8 first and then use the notation defined in equation 8 to introduce lemma 1. 
2. It seems plausible that a network would learn a smoothed version of the score (as discussed in limitations), but you never strongly motivate / discuss this, earlier discussion of this would motivate your choice of analysis better.  Either way it would be helpful for you to motivate and introduce the smoothed score a bit earlier in the text.
Yes
No ethics review needed.
See weaknesses.
The paper considers a simple one-dimensional model to study the smoothed score function. Mathematical properties are derived and numerical experiments are conducted.
This work studies the the interpolation effect of score smoothing and provides numerical experiments.
Major questions: 
1. The biggest concern is the gap between the problem that the authors attempt to study and the model proposed in this paper. The starting point is memorization vs. generalization in diffusion models. The authors conjecture the memorization is from the smoothed score function. It raised two natural questions: 1) how do you know the NN-learning score estimator has such smoothing effect in practice? 2) Even though the score estimator used in practice is smoothed, how is this property linked to the generalization, i.e., overcoming the memorization? These two questions are not well-studied in the paper.
2. The paper considers a(n) (essentially) one-dimensional model. In this case, the (empirical) score function has a closed-form. However, the authors do not study this score function while turning to its smoothed version. Then I would like to ask, even if we know all the properties of the smoothed score function, how can we have more knowledge about the true score function, even in this simple case?
3. The authors conduct numerical experiments to compare three score functions/estimators. However, it is not clear how to interpret the results. The distribution after adding noise must be the same as standard Gaussian is the stationary distribution in all three cases. Then, what is the new information by plotting the density and histograms?

Minor questions:
1. Why do you introduce $\hat{x}_t^{(n)}$, $\bar{s}_t^{(n)}$ and $$\widehat{s}^{(n)}_{t, \tau}$$, three versions of the score function? Also, if the target distribution is $p_0^{(n)}$, why to you need a smoothed version $\hat{p}_t^{(n)}$? Please clarify the relation between them and emphasize the necessity of introducing these auxiliary functions/distributions.
2. The target distribution studies seems to be relevant to Gaussian mixture models. There are some papers studying learing GMMs using diffusion models, e.g., https://arxiv.org/abs/2404.18869 and https://arxiv.org/abs/2404.18893. Any relation to the literature?
Yes
No ethics review needed.
More broadly, the smoothing effect can be related to generalization error and the inductive bias of deep neural networks. Could the authors briefly comment on how the analyses can help understand these phenomena?
The paper is well-organized and has a clear presentation. The theoretical results on two points and one-dimensional subspace are intuitive and easy to follow.
The paper investigated the smoothing effect in score estimation and presented theoretical analyses on a simplified data model. Numerical results were also presented to support the argument.
It is not clear how the analyses of one one-dimensional subspace can be extended to more practical data. 
- For example, how can the analyses be connected back to address memorization/hallucination behaviors mentioned in the introduction?
- In line 475, I cannot agree the analysis is "distribution-agnostic," as it still relies on a highly simplified assumption of the practical data.
Yes
No ethics review needed.
1. The motivation for the chosen local smoothing technique is not entirely intuitive. The author shows that their smoothed score function is close to the true score in the $L^2$ sense, which is usually assumed for the NNs in practice. If one can further show that 'if two different score functions are both close to the true score in the $L^2$ sense, then the dynamics driven by them are close to each other', then studying a specific smoothed score could directly help us understanding the others.

2. The relationship between $\kappa$ and $t$ in section 3 is a bit confusing. In Proposition 2, $\kappa$ is defined by $t$, while in proposition 3, $t$ is restricted in $[0, \kappa]$. In practice, should $t$ be specified first?
The paper presents a novel perspective on how score smoothing enables generative models to interpolate between data points, avoiding full memorization—a valuable insight for understanding generalization in diffusion models. Numerical experiments illustrate the model’s interpolation behavior, validating that a smoothed ESF leads to sample distributions that interpolate among training points. This supports the theoretical analysis and enhances understanding of the smoothing effects on generative dynamics.
This paper investigates the interpolation effect that arises when smoothing the empirical score function (ESF) in score-based diffusion models, specifically exploring its impact on generative sample diversity. The authors focus on a theoretical model where training data lie in a one-dimensional subspace and use a local smoothing approach on the ESF, demonstrating that smoothing allows for interpolation between training points and mitigates memorization. They offer a mathematical analysis under simplified settings and provide numerical illustrations to validate their findings. Their method seeks to clarify why score smoothing may improve generalization in generative models by facilitating smoother transitions within the data's underlying structure.
1. A typo at line 57: $t$ should be inside $\sqrt{\cdot}$.

2. The motivation for the chosen local smoothing technique is not entirely intuitive. While they study an approximation in the $L^2$ sense, it remains unclear why this specific smoothing function is preferable over alternative smoothing methods. The paper acknowledges this limitation but could further clarify. See questions below.

3. The relationship between $\kappa$ and $t$ in section 3 is a bit confusing. See questions below.
We show mathematically how a smoothing of the empirical score function can lead the denoising diffusion dynamics to generate samples that interpolate the training data within an underlying linear subspace.
@inproceedings{
anonymous2024on,
title={On the Interpolation Effect of Score Smoothing},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ETX8NTEuCj},
note={under review}
}
Score-based diffusion models have achieved remarkable progress in various domains with an ability to generate new data samples that do not exist in the training set. In this paper, we examine a hypothesis that this phenomenon manifests an interpolation effect caused by a smoothing of the empirical score function. Focusing on settings where the training set lies in a one-dimensional linear subspace, we take a distribution-agnostic perspective and study the interplay between score smoothing and the denoising dynamics with mathematically solvable models. We demonstrate how score smoothing can lead to the generation of samples that interpolate among the training data within the subspace while avoiding a full memorization of the training set.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
score-based diffusion models score smoothing data interpolation generalization vs memorization subspace recovery
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f14c611f1e90ac2f7312b9712398b6334a1d1f61.pdf
learning theory
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
On the Interpolation Effect of Score Smoothing
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission