Rectified Diffusion Guidance for Conditional Generation
Diffusion Models;Conditional Generation;Classifier-Free Guidance
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
Yes
No ethics review needed.
1. The unbiasedness is a great property to have. However, from the toy case experiment, the proposed method seems to affect the variance recovery. Would the proposed method hurt diversity in image generation?  

2. In text-to-3D generation, e.g., DreamFusion, the CFG coefficient is a lot larger. Can the analysis explain this and can similar improvements made in these scenarios? 

3. While ReCFG’s use of a lookup table aids in theoretical correctness, it adds a layer of complexity to the sampling process. For text-to-image models, where open-vocabulary synthesis requires flexibility across diverse prompts, the need to precompute or interpolate values for novel prompts could hinder ReCFG’s feasibility. I am wondering how much is the extra computation, especially for more complicated generation tasks.
The paper provides a rigorous analysis of a known issue with CFG, quantifying expectation shift and proposing ReCFG as a theoretically grounded alternative. This analysis contributes to understanding the mathematical foundation of guided sampling in diffusion models. The resulting ReCFG is designed as a post-hoc solution that can be applied to pretrained models without the need for additional training.

The experiments on the toy data is helpful for understanding the method and subsequent real data experiments also demonstrates the effectiveness.
This paper introduces an enhancement to Classifier-Free Guidance (CFG), a widely used approach in diffusion model sampling, by identifying and addressing an expectation shift flaw in CFG. The authors propose "Rectified Classifier-Free Guidance" (ReCFG), a variation that relaxes the original guidance coefficient constraints to align more closely with diffusion theory. By incorporating a lookup table to adjust guidance coefficients, ReCFG aims to maintain conditional fidelity without additional retraining or loss of inference speed. The authors present theoretical derivations and empirical results demonstrating improved conditional fidelity on several state-of-the-art diffusion models, including applications on ImageNet and CC12M.
My main concern is the scope of the experiment and demonstrated experimental improvements. CFG has been a very important component in most AIGC model. Even GPT-type sequential image modeling uses CFG for better performance. Targeting such a component, the current work doesn't demonstrate enough practical impact in more complex generative tasks such as text to image generation. 
Performance-wise, the empirical gain measured by the FID or CLIP scores in ImageNet and CC12M are not that significant. In this case, some user study or visual comparison would be helpful in showcasing the effectiveness.
Yes
No ethics review needed.
1. I guess one needs to solve (28) for each time step t, which may introduce more computation. Will that be an issue?
2. Whether the coefficients obtained by optimizing (37) can lead to (28) since it also take expectations with respect to t?
3. By (28) and the equation before (34), it seems that we also have $E_{x_t}(\epsilon_\theta(x_t,t,c)) = E_{x_t}(\epsilon_\theta(x_t,t)) = 0$. Now suppose you have $\Delta_t =0$. From the proof of Theorem 3, specifically (S107) and (S112), we have $E_{x_0}(x_0) = E_{c,x_0}(x_0)$. Does that mean in this case, the conditional expectation is the same as the unconditional expectation? If it is true, does that mean if $E_{x_0}(x_0) \neq  E_{c,x_0}(x_0)$, $\Delta_t \neq 0$ and you will always have expectation shift?
1. The paper provides a theoretical analysis, identifying a flaw in the popular CFG technique and proposing a mathematically grounded solution, also provides rigorous proof of the benefits of ReCFG.
2. The introduction of ReCFG offers a novel solution to the expectation shift problem associated with CFG in diffusion models.
The paper examines rectified classifier-free guidance (ReCFG) for diffusion models, addressing theoretical flaws in existing models that use Classifier-Free Guidance (CFG). By introducing new coefficients for combining conditional and unconditional score functions, ReCFG adheres closely to diffusion theory, eliminating expectation shifts in conditional distributions and enhancing model fidelity without the need for retraining.
1. Since it is mainly a theoretical paper, I think it would be beneficial to add more explanation after Theorem 3, while probably reduce the length of Theorem 2. 
2. If I understand correctly, the method needs to solve (28) for each time step t, which may increase the computation.
3. Although it is a theoretical paper, it would be better to add more experiment results, e.g., some illustrations of generated images.
Yes
No ethics review needed.
- It is difficult to understand the connection between theorems 1 and 2 as the fact that $\mathbb{E}\_{x_t}[\nabla_{x_t} \log q_{t, \gamma}(x_t \mid c)] \neq 0$ doesn't appear anywhere in theorem 2. However, theorem 3 illustrates this difference perfectly. Why putting theorem 3 so late in the manuscript ? I believe that putting theorem 3 early would help readers better undertsand the logical flow of the paper's intuitions.
- I find the results in Table 1 and 2 very tight between the algorithms, could you provide means and standard deviations ?
- According to Table 3, it seems that the expectation ratio is often close to 1. Doesn't it advocate for $\gamma_0 = 1 - \gamma_1$ in the end ?
- The paper is well written
- The proposed approach is novel
- The paper is well illustrated with intuitive theorems
This paper addresses conditional generative modeling and highlights the limitations of traditional Classifier-Free Guidance (CFG). It demonstrates that denoising with CFG cannot be viewed as a reciprocal diffusion process. To tackle this issue, the authors propose a relaxed version of CFG that introduces a new type of guidance. They show that this alternative guidance can be unbiased and provide a methodology for calculating the necessary guidance parameters. The paper also illustrates the superiority of their proposed method through evaluations on standard benchmarks, showcasing its effectiveness in overcoming the limitations of classic CFG.
- The proposed method doesn't give a closed form expression but an estimator
- The method is computationally intensive
Yes
No ethics review needed.
* Figure 1: Is the expectation ratio essentially the softmax? The figure's message is unclear. Distinguishing the expectation ratios between the first and last sampling images is challenging based on the comparison provided. Consider providing a more detailed explanation of what the expectation ratio represents and how it's calculated and some visualization, such as using a different color scale or providing separate figures for the first and last sampling steps.

* Figure2: Seems like this method achieves unbiasness at the cost of losing diversity(much smaller variance). If caclualting KL divergence, it is hard to tell which distribution is closer to the ground-truth. 

* If relaxing the constriant of CFG: two coefficients sum to one; I doubt it increases or decreases the magnitude of the predicted noise norm, which may degenerate the sampling process. 

* In Table1\&2: are the results of the baseline CFG good enough to be persuasive? Please choose some well-accepted benchmark models results on these datasets and ensure the hyperparameters are comparable, for example, 25 or 256 sampling steps with ddpm or ddim.

* From Table3, it suggests users have to prepare the expectation ratio for each class and each sampling steps. The combination of class and sampling steps makes it too cumbersome. Furthmore, the such small digit makes people worry the numerical stability of this method. Therefore, the ablation study should be presented to prove the numerical stability of the expectation ratio. Please consider conducting some ablation studies, such as:
Comparing results when using class-specific ratios vs. a global average ratio
Analyzing how performance changes when rounding the ratios to different decimal places
Examining the impact of using fewer sampling steps on both performance and computational complexity
The analysis of CFG is extensive.
The paper thoroughly examines Classifier-Free Guidance (CFG) in diffusion, highlighting the bias introduced by mean shift due to violations of the zero mean assumption in data samples. It suggests adjusting the CFG coefficient to mitigate this shift. Some empirical experiments seem to support these findings, revealing the mean-shift issue and proposing an easily implementable solution to address this bias.
* The proposed method appears numerically unstable due to the minute coefficient deviation from the default value of 1.0, casting doubt on the experiment's reliability. Furthermore, this slight difference in coefficients may indicate that the violation of the zero-mean assumption in the original CFG has negligible practical implications.  Please consider conducting an ablation study showing how results change with small perturbations to the coefficients.

* The merits of unbiasness scarfices the diversity(dramatic smaller variance compared to the ground truth), as indicated in Figure2. Provide a quantification of the trade-off between unbiasedness and diversity would be more informative. 

* In essence, once relaxing the constriant of CFG: two coefficients sum to one; I doubt it increases or decreases the magnitude of the noise norm, which may collapse the sampling process. Providing an analysis of how the proposed method affects the noise norm throughout the sampling process, compared to standard CFG, would be informative. 

* No visual demonstration is provided. Please provide some sample images generated with standard CFG vs. the proposed method, or visualizations of the sampling process over time.
The paper confirms the phenomenon of expectation shift of CFG, and proposes a novel perspective to alleviate such issue, achieving better conditional fidelity.
@misc{
xia2024rectified,
title={Rectified Diffusion Guidance for Conditional Generation},
author={Mengfei Xia and Nan Xue and Yujun Shen and Ran Yi and Tieliang Gong and Yong-jin Liu},
year={2024},
url={https://openreview.net/forum?id=Y4kJp8GQmV}
}
Classifier-Free Guidance (CFG), which combines the conditional and unconditional score functions with two coefficients summing to one, serves as a practical technique for diffusion model sampling. Theoretically, however, denoising with CFG *cannot* be expressed as a reciprocal diffusion process, which may consequently leave some hidden risks during use. In this work, we revisit the theory behind CFG and rigorously confirm that the improper configuration of the combination coefficients (*i.e.*, the widely used summing-to-one version) brings about expectation shift of the generative distribution. To rectify this issue, we propose ReCFG with a relaxation on the guidance coefficients such that denoising with ReCFG strictly aligns with the diffusion theory. We further show that our approach enjoys a ***closed-form*** solution given the guidance strength. That way, the rectified coefficients can be readily pre-computed via traversing the observed data, leaving the sampling speed barely affected. Empirical evidence on real-world data demonstrate the compatibility of our post-hoc design with existing state-of-the-art diffusion models, including both class-conditioned ones (*e.g.*, EDM2 on ImageNet) and text-conditioned ones (*e.g.*, SD3 on CC12M), without any retraining. We will open-source the code to facilitate further research.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Mengfei_Xia1 ~Nan_Xue1 ~Yujun_Shen1 ~Ran_Yi1 ~Tieliang_Gong2 ~Yong-jin_Liu1
Mengfei Xia Nan Xue Yujun Shen Ran Yi Tieliang Gong Yong-jin Liu
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion Models Conditional Generation Classifier-Free Guidance
I certify that there is no acknowledgement section in this submission for double blind review.
xia|rectified_diffusion_guidance_for_conditional_generation
/pdf/499c7b3eb05732ef22b3d892b6ac55187a8bfe6e.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Rectified Diffusion Guidance for Conditional Generation
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission