Guided Score Identity Distillation for Data-Free One-Step Text-to-Image Generation
stable diffusion;data-free distillation;single-step generation;classifier-free guidance
Thank you for your constructive feedback. Before addressing each of your concerns in detail, we’d like to clarify a key misunderstanding regarding SwiftBrush, which we had initially found unclear ourselves. Specifically, we sought clarification from the authors on their statement: *"During training, a guidance scale of 4.5 is used for both teachers."* 

The authors confirmed that *“Classifier-Free Guidance (CFG) was not applied to the LoRA teacher in Equation 6. We directly use the output from the LoRA teacher and subtract it with random Gaussian noise ε.”*

In other words, CFG in SwifthBrush is used during the inference phase of the fake score network but is not applied during training (the "training" in SwiftBrush refers to the training of the generator). Thus, we stand by our claim that **we are the first to incorporate CFG into the training of the fake score network.**
Yes
N/A
No ethics review needed.
Please refer to the Weaknesses.
The paper introduces two distinct strategies for CFG in SiD: the long CFG and short CFG strategies, each with unique effects on model performance.
1. In the long CFG, it is applied to the pretrained score network (the teacher model) with a higher guidance scale (typically $\tau$>1), which encourages the pretrained model to generate images more aligned with the textual prompt, which in turn compels the student generator to follow this alignment. The primary effect is a strong improvement in text adherence and quality, with the capability to achieve a balanced FID and CLIP score. 
2. In the short CFG, it is applied to the fake score network (the student model) with a reduced guidance scale (typically $\tau$ values between 0 and 1). By reducing CFG, this approach lessens the fake score network’s alignment with the text prompt, which incentivizes the student generator to improve text alignment and quality. This strategy tends to result in a trade-off where lower CLIP scores (text alignment) can occur, but FID scores (distribution match) remain competitive.
This paper focuses on a novel approach to accelerating text-to-image diffusion models through Score Identity Distillation with Long-Short Guidance (SiD-LSG), specifically targeting data-free one-step generation. By combining Score Identity Distillation with Classifier-Free Guidance (CFG), the authors efficiently distill Stable Diffusion models into one-step generators without needing real data. The proposed method incorporates both long and short CFG strategies, which balance FID and CLIP scores, enabling efficient and high-quality text-to-image generation. Experimental results demonstrate the effectiveness of the proposed method on the COCO dataset, achieving a record-low data-free FID of 8.15.
1. The main contribution of this paper is introducing a variant of CFG into SiD, i.e., long and short CFGs, which aim to balance semantic alignment and the quality of generated images. However, there seems to be a contradiction, as no single $\tau$ value optimally balances both CLIP and FID scores (see results in Tables 1 and 2). Are there intrinsic reasons for this phenomenon?

2. A more intuitive ablation study would better showcase the effectiveness of the proposed long and short CFG strategies. Specifically, in Tables 1 and 2, rather than using SiD-LSG alone, could you provide the results (both CLIP and FID) for SiD with the same $\tau$ value across different configurations: SiD (short CFG only), SiD (long CFG only), SiD (CFG only), and SiD (original)?
Yes
No ethics review needed.
See the weaknesses.
- The paper adapts Score identity Distillation (SiD) to work with text-to-image models.
- SiD-LSG shows state-of-the-art distillation results in a data-free setting. With a small CFG, it achieves impressive FID scores (8.15 for SD1.5 and 9.52 for SD2.1). With a large CFG, it achieves high CLIP scores, with sacrification in FID ones.
- The paper presents a comprehensive set of experiments.
The paper proposes a new distillation technique (SiD-LSG) for data-free one-step text-to-image generation. First, it adapts Score identity Distillation (SiD) to work with text-to-image models. Second, it proposes a Long and Short Classifier-Free Guidance (LSG) scheme to improve the distillation effectiveness. SiD-LSG shows state-of-the-art distillation results in a data-free setting. With a small CFG, it achieves impressive FID scores (8.15 for SD1.5 and 9.52 for SD2.1). With a large CFG, it achieves high CLIP scores, with sacrification in FID ones.
- L268-269: The claim "we are the first to incorporate CFG into the training of the fake score network" is incorrect. SwiftBrush uses CFG 4.5 for both teachers (see Implementation Details in Section 4.1 in their paper).
- While the paper proposes an interesting short guidance setting ($\kappa 2 = \kappa 3 < 1$), it does not apply to the long and short guidance ($\kappa 2 = \kappa 3 > 1$). I do not see the connection between the short guidance setting and the long and short guidance setting.
- In the end, the long and short guidance scheme is very simple and uninteresting ($\kappa 1 = \kappa 2 = \kappa 3 = \kappa 4 = 1.5). It just applies the same CFG for both teachers, which SwiftBrush already did.
- Some recent works (SwiftBrush v2, DMD2) should be mentioned and compared with.
- The training protocol is expensive. The batch size is large (512 by default), requiring 64 gradient accumulation rounds on H100 (80G), FP32 setting. SiD-LSG necessitates gradient backpropagation through the score networks, which requires massive memory usage. Hence, it can only use the batch size per GPU as 1 on H100 (80G), FP32 setting.
- Fig 3, 4, 6, 9: The curves for FID scores and CLIP scores are hard to differentiate. They have the same color and with just a small transparency difference. The authors should improve these figures.
- Fig. 6, which is discussed in Section 3.2, is in the Appendix. It is not good. The authors should move the figure to the main paper and squeeze the text to fit in the 10-page limit.
Yes
No ethics review needed.
- Does this approach valid for v-prediction networks? how does this alter the theory?
- The paper is well written and easy to follow
- The novelty and relation to SiD is clear.
- The theory and mathematical background is solid.
- The method is novel and important.
- The ablation is comprehensive.
- The results of the methods are SoTA.
The authors propose an enhancement to Score Identity Distillation (SiD), a recently introduced method for distilling a pre-trained diffusion network into a single-step generator. This paper’s method specifically targets pre-trained text-to-image models based on DDPM, adapting SiD to distill two epsilon-prediction networks: SD1.5 and SD2.1.base.

Following the SiD framework, the one-step generator ($\theta$) is optimized by training an additional fake multi-step generator ($\psi$). Furthermore, classifier-free guidance (CFG) is integrated into the distillation process, allowing for varied CFG values that emphasize different target objectives. This added flexibility contributes to enhanced model performance.

The results achieve state-of-the-art FID scores and demonstrate strong CLIP scores on the COCO-2014 validation prompts.
- Given SiD the method novelty is limited to the adaptation of the SiD theory to DDPM and t2i models.
- The distillation is on relatively old pre-trained networks.
Yes
No ethics review needed.
If a model itself does not use CFG, then the proposed method will not work？ For example, FLUX does not use CFG for inference but uses word embedding instead. Just open discussion.
1.  The main contribution is The proposed method enhances the Score Identity Distillation technique by incorporating Classifier-Free Guidance , although these two techniques have been proposed in other studies.

2. By reducing the number of steps required for image generation to one, the method significantly improves the efficiency of the generation process, and the proposed method achieves state-of-the-art performance in terms of Fréchet Inception Distance (FID) on the COCO-2014 validation set, setting a new benchmark for data-free one-step distillation.
This paper mainly studies how to accelerate the diffusion-based generative model to generate high-quality images. The paper presents a novel method called Guided Score Identity Distillation with Long-Short Guidance (SiD-LSG) for data-free one-step text-to-image generation, which enhances the Score Identity Distillation (SiD) technique by incorporating Long and Short Classifier-Free Guidance (LSG).
1.  Regarding training efficiency, SiD needs to generate false data for training, and adding CFG will further increase the model overhead, resulting in more training time and resources.

2. The data-free nature of the SiD and SID-LSG means it does not leverage real-world data to further refine the generated images, which could potentially improve photorealism and text alignment.
We have developed long and short guidance strategies within Score identity Distillation to efficiently distill Stable Diffusion models into one-step text-to-image generators, all in a data-free manner.
@inproceedings{
anonymous2024guided,
title={Guided Score Identity Distillation for Data-Free One-Step Text-to-Image Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=HMVDiaWMwM},
note={under review}
}
Diffusion-based text-to-image generation models trained on extensive text-image pairs have demonstrated the ability to produce photorealistic images aligned with textual descriptions. However, a significant limitation of these models is their slow sample generation process, which requires iterative refinement through the same network. To overcome this, we introduce a data-free guided distillation method that enables the efficient distillation of pretrained Stable Diffusion models without access to the real training data, often restricted due to legal, privacy, or cost concerns. This method enhances Score identity Distillation (SiD) with Long and Short Classifier-Free Guidance (LSG), an innovative strategy that applies Classifier-Free Guidance (CFG) not only to the evaluation of the pretrained diffusion model but also to the training and evaluation of the fake score network. We optimize a model-based explicit score matching loss using a score-identity-based approximation alongside our proposed guidance strategies for practical computation. By exclusively training with synthetic images generated by its one-step generator, our data-free distillation method rapidly improves FID and CLIP scores, achieving state-of-the-art FID performance while maintaining a competitive CLIP score. Notably, the one-step distillation of Stable Diffusion 1.5 achieves an FID of **8.15** on the COCO-2014 validation set, a record low value under the data-free setting.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
stable diffusion data-free distillation single-step generation classifier-free guidance
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0abfa82d8d328b14026a2b7940ff470dc4c447d4.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Guided Score Identity Distillation for Data-Free One-Step Text-to-Image Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission