FreqPrior: Improving Diffusion Models with Frequency Filtering Gaussian Noise as Prior
video diffusion models; Fourier transform; noise prior; frequency filtering
Yes
No ethics review needed.
The title should explicitly mention "Video Diffusion Models" to clarify that the method is specifically designed for video generation and not applicable to image diffusion models. This will avoid any confusion and make the scope of the paper clearer to readers.

The paper should provide detailed measurements of GPU memory usage before and after applying the proposed FreqPrior method, particularly focusing on peak memory consumption. Given that 3D FFT can be memory-intensive, especially for resolutions higher than 512x512, this information is crucial for understanding the practical feasibility of the method. Include tables or graphs showing the GPU memory usage for different resolutions and compare them with the baseline methods. This will help readers assess the trade-offs between memory consumption and performance improvements.

The paper should explore the impact of different Classifier-Free Guidance (CFG) strengths when using FreqPrior. Since CFG is a common technique used in diffusion models to enhance generation quality, understanding how FreqPrior interacts with varying CFG strengths is important for practical applications.
The main contributions are:

FreqPrior refines noise in the frequency domain using a new filtering technique that addresses different frequency signals while maintaining a noise prior distribution close to a standard Gaussian distribution. This method helps preserve important low-frequency signals, enhancing semantic fidelity.

 The authors propose a partial sampling process that perturbs the latent space at an intermediate timestep during the noise prior generation. This approach significantly reduces inference time without compromising the quality of the generated videos.

The paper provides a comprehensive theoretical analysis of the variance decay issue in existing methods, which contributes to the loss of details and motion dynamics. The authors show that the covariance error of their method is negligible, indicating that their noise prior closely approximates a Gaussian distribution.
The paper introduces FreqPrior, a novel noise initialization strategy for text-to-video diffusion models.  FreqPrior refines noise in the frequency domain using a new filtering technique that addresses different frequency signals while maintaining a noise prior distribution close to a standard Gaussian distribution. This method helps preserve important low-frequency signals, enhancing semantic fidelity. The authors propose a partial sampling process that perturbs the latent space at an intermediate timestep during the noise prior generation. This approach significantly reduces inference time without compromising the quality of the generated videos. The paper provides a comprehensive theoretical analysis of the variance decay issue in existing methods, which contributes to the loss of details and motion dynamics. The authors show that the covariance error of their method is negligible, indicating that their noise prior closely approximates a Gaussian distribution.
The title should explicitly mention "Video Diffusion Models" to clarify that the method is specifically designed for video generation and not applicable to image diffusion models. This will avoid any confusion and make the scope of the paper clearer to readers.

The paper should provide detailed measurements of GPU memory usage before and after applying the proposed FreqPrior method, particularly focusing on peak memory consumption. Given that 3D FFT can be memory-intensive, especially for resolutions higher than 512x512, this information is crucial for understanding the practical feasibility of the method. Include tables or graphs showing the GPU memory usage for different resolutions and compare them with the baseline methods. This will help readers assess the trade-offs between memory consumption and performance improvements.


The paper should explore the impact of different Classifier-Free Guidance (CFG) strengths when using FreqPrior. Since CFG is a common technique used in diffusion models to enhance generation quality, understanding how FreqPrior interacts with varying CFG strengths is important for practical applications.
Yes
No ethics review needed.
1. My first question is the experimental analysis, why only Vbench is provided?

2. The second question is theoretical evidence for why non-normalized Gaussian distribution will cause the worse motion dynamics.

3. Have you considered or tested other types of frequency filtering (e.g., adaptive filtering methods) to optimize the noise prior? What is the generalization capability of such frequency filtering? It would be important to demonstrate their broader applicability

4. Have you measured the standard deviation for your generated videos with different seeds? It contains lots of randomness in video generation. Whether this work select videos based on human visualization? If not, it takes which principles for results selection?
1. This work identifies the importance of standard Gaussian distribution in the sampling process for video generation. 

2. This work introduces a new frequency decomposition strategy for random variables.

3. Extensive experiments and theoretical derivation provide a great illustration for the motivation.
The paper presents a novel approach for enhancing noise priors in text-to-video diffusion models. The authors introduce a new frequency filtering method to refine noise priors, improving video quality by preserving essential details and dynamics better than existing baselines such as Gaussian noise, mixed noise, progressive noise, and FreeInit. The core motivation is to keep the standard Gaussian distribution for the frequency-based sampling refinement process. The method requires additional sampling iterations but offers notable performance improvements across multiple metrics evaluated on the VBench benchmark. The experiments are conducted using three open-source text-to-video models (VideoCrafter, ModelScope, and AnimateDiff), and the results highlight that the proposed method outperforms the baselines in both quantitative and qualitative aspects.
1. Despite this work has shown the side effects of non-uniformed sampling noise distribution, it is still hard to understand why this will destroy the motion dynamics from the theoretical perspectives. 

2. The evaluation of this work is only based on VBench, which is somehow not sufficient. It is suggested to include more comparisons in terms of FID, FVD, etc. Whether the conclusion will stand under these metrics.

3. This work lacks user study and does not provide the detailed prompts used for video generation. Since the video quality measurement for AIGCs is not absolutely reliable, providing a user study for video generation is essential. 

4. How to obtain the equation (7), it needs a detailed explanation.
Yes
No ethics review needed.
What is the performance on more recent ODE-based diffusion models?
1. The FreqPrior approach addresses detail loss and motion dynamics issues better than previous methods (e.g., FreeInit), leading to improved video fidelity.
2.  The partial sampling process significantly reduces inference time by around 23% compared to similar methods.
3.  FreqPrior achieves higher scores in quality and semantics in evaluations, especially on the VBench benchmark.
To address the problem of variance-decreasing in FreeInit, the authors propose to re-design the low-pass filter in FreeInit and use two sets of noise to maintain the variance of intermediate diffusion variables. Experiments show that the proposed method is able to preserve more details than FreeInit.
The paper argues that the variance-decreasing problem in FreeIniit causes it to generate over-smoothed results. But the provided evidence is weak. Although the demo cases at the beginning of this paper support this conclusion, more videos in the supplement files do not verify it.  According to Table 2,  the quantitative improvements over FreeIniit is also marginal.
Yes
No ethics review needed.
Questions are listed in the weakness.
1. Comprehensive theoretical analysis of the variance decay issue of existing methods and addressing the issue by novel filtering technique is interesting and novel.

2. Extensive experiments validate the novel filtering method refine the text-to-video diffusion models significantly.
Building on FreeInit, this method introduces a novel frequency filtering approach to obtain an improved noise prior that enhances high-frequency signals and approximates a Gaussian distribution, refining text-to-video diffusion models. 
Additionally, by implementing partial sampling instead of the full sampling used in FreeInit, it effectively reduces the sampling time.
1. This work builds upon FreeInit, so the implementation of FreeInit should remain consistent with the original. However, while the original FreeInit uses 4 extra iterations, the comparisons in this work are made with only 2 extra iterations.

* What would the results be if both FreeInit and FreqPrior were implemented with 4 extra iterations? Would FreqPrior still outperform FreeInit?

2. Applying this method to recent T2V models could enhance the completeness of the paper.
* If high-quality T2V models are available, making low-frequency matching unnecessary, would this method still be effective?
* Additionally, if possible, could the method demonstrate effectiveness on the latest T2V models, such as T2V-Turbo or Pyramidal Flow?
@inproceedings{
anonymous2024freqprior,
title={FreqPrior: Improving Diffusion Models with Frequency Filtering Gaussian Noise as Prior},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8x0SGbCpzs},
note={under review}
}
Text-driven video generation has advanced significantly due to developments in diffusion models. Beyond the training and sampling phases, recent studies have investigated noise priors of diffusion models, as improved noise priors yield better generation results. One recent approach employs Fourier transform to manipulate noise, marking the initial exploration of frequency operations in this context. However, it often generates videos that lack motion dynamics and imaging details. In this work, we provide a comprehensive theoretical analysis of the variance decay issue present in existing methods, contributing to the loss of details and motion dynamics. Recognizing the critical impact of noise distribution on generation quality, we introduce FreqPrior, a novel noise initialization strategy that refines noise in the frequency domain.  Our method features a novel filtering technique designed to address different frequency signals while maintaining the noise prior distribution that closely approximates a standard Gaussian distribution. Additionally, we propose a partial sampling process by perturbing the latent at an intermediate timestep during finding the noise prior, significantly reducing inference time without compromising quality. Extensive experiments on VBench demonstrate that our method achieves the highest scores in both quality and semantic assessments, resulting in the best overall total score. These results highlight the superiority of our proposed noise prior.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
video diffusion models; Fourier transform; noise prior; frequency filtering
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/59a40b594e1ffbedebf58da0e403398c4edec2d8.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/f64ac479f0b9982fec28a43a28d403221adbec6a.zip
FreqPrior: Improving Diffusion Models with Frequency Filtering Gaussian Noise as Prior
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission