Adversarial Score Identity Distillation: Rapidly Surpassing the Teacher in One Step
Diffusion distillation;deep generative models
We appreciate the reviewer’s recognition of SiDA’s SOTA performance.


1. *Can SiDA be used to distill DiT-based models?*


    **Response:** Our research primarily focuses on distilling U-Net-based EDM models, which are well-recognized and benchmarked in diffusion-based generation and distillation. We have now also added results for EDM2, which shows a significantly lower FID on ImageNet 512 x 512 compared to all existing DiT-based models, as detailed in Table 2 of the EDM2 paper and our forthcoming revision.


     Despite this, we acknowledge the versatility of DiT-based architectures. SiDA is not inherently limited to U-Net backbones and shows potential for adapting to transformer-based architectures. For example, in the context of transformer-based diffusion models, we could divide the architecture into “encoder” and “decoder” components. Insights from studies like REPA (Yu, Sihyun, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. "Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think." arXiv preprint arXiv:2410.06940 (2024).) suggest that aligning self-supervised features from the encoder, particularly those from middle layers (6th to 8th in a 24-layer model), with a diffusion model could enhance DiT/SiT model training.


    While integrating SiDA with transformer-based models is intriguing, this exploration is beyond the scope of our current research. We plan to pursue this direction in future projects and invite researchers with the necessary computing resources to adapt our SiDA code, which will be open-sourced, to distill DiT-based models and contribute their findings to the community.


2. *Can SiDA be applied to high-resolution models?*


     **Response:** Yes, SiDA excels in high-resolution settings, achieving remarkably low FIDs on ImageNet 512x512 without CFG and through one-step generation. We have documented our current results for distilling EDM2 and will provide further updates after revising our paper based on the reviewers' feedback.


3. *Why is $L^1(\theta)$ still necessary with an additional discriminator, i.e., why does SiDA still require tuning $\alpha$?*


     **Response:** Omitting $L^1(\theta)$, or setting $\alpha=0$, was feasible in SiD but resulted in unsatisfactory FID scores. With the inclusion of adversarial loss, this configuration in both SiDA and SiD$^2$A now yields satisfactory outcomes, yet still underperforms compared to the recommended settings of $\alpha=1.0$ for SiDA and $\alpha=1.0$ or $1.2$ for SiD$^2$A. This indicates that, despite the support from adversarial loss, maintaining $\alpha>0$ is important to correct the approximation bias in the SiD gradient effectively.
Response to Reviewer Ajww
Dear Reviewers, 

We appreciate your valuable feedback. While we are preparing a comprehensive revision, we would like to promptly address the shared concern regarding the absence of results for high-resolution images. To this end, we have adapted the SiDA codebase to distill EDM2 models of various sizes pretrained on ImageNet 512×512, yielding the following results:

| **Method** | **CFG** | **NFE** | **XS (125M)** | **S (280M)** | **M (498M)** | **L (777M)** | **XL (1.1B)** | **XXL (1.5B)** |
|------------|---------|---------|---------------|--------------|--------------|--------------|---------------|----------------|
| EDM2       | N       | 63      | 3.53          | 2.56         | 2.25         | 2.06         | 1.96          | 1.91           |
| EDM2       | Y       | 63      | 2.91          | 2.23         | 2.01         | 1.88         | 1.85          | 1.81           |
| sCT        | Y       | 1       | -             | 10.13        | 5.84         | 5.15         | 4.33          | 4.29           |
| sCT        | Y       | 2       | -             | 9.86         | 5.53         | 4.65         | 3.73          | 3.76           |
| sCD        | Y       | 1       | -             | 3.07         | 2.75         | 2.55         | 2.40          | 2.28           |
| sCD        | Y       | 2       | -             | 2.50         | 2.26         | 2.04         | 1.93          | 1.88           |
| SiD        | N       | 1       | 3.353 ± 0.041 | 2.707 ± 0.054| 2.060 ± 0.038| 1.907 ± 0.016| 1.888 ± 0.030 | -              |
| SiDA       | N       | 1       | 2.228 ± 0.037 | 1.756 ± 0.023| 1.546 ± 0.023| 1.501 ± 0.024| 1.428 ± 0.018 | -              |
| SiD²A      | N       | 1       | **2.156 ± 0.028** | **1.669 ± 0.019** | **1.488 ± 0.038** | **1.413 ± 0.022** | **1.379 ± 0.017** | -              |

The results for EDM2 are sourced from Karras et al. (2024), while those for sCT and sCD are obtained from Lu and Song (2024). Although Lu and Song's work is concurrent with ours, we include their results here to highlight the superior performance of SiDA.

We observe that SiD²A achieves an FID of 1.379 on EDM2-XL, marking the lowest score reported to date. This result is notable as it is obtained without classifier-free guidance (CFG), post-hoc exponential moving average (EMA), and utilizes only a single generation step. Furthermore, both SiDA and SiD²A applied to EDM2-S (280M parameters) surpass the performance of the largest teacher model, EDM2-XXL (1.5B parameters), as well as sCD and sCT methods. We also note that SiD, the baseline model adapted by us for EDM2, achieves performance comparable to that of the teacher model. This observation aligns with previous findings on smaller datasets and EDM models, demonstrating SiD's effectiveness across varying scales. 

**References:**

- Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. "Analyzing and Improving the Training Dynamics of Diffusion Models." In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 24174–24184, 2024.

- Cheng Lu and Yang Song. "Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models." *arXiv preprint arXiv:2410.11081*, 2024.

We will provide a more detailed response and a revised paper in due course.
Preliminary Response: Addressing Concerns on High-Resolution Image Results
Yes
No ethics review needed.
-	Why $L^1(\theta)$ is still needed with the additional discriminator, i.e., why SiDA still needs tuning $\alpha$?
-	The authors repurpose the fake score network as an additional discriminator with direct using the average of the encoder features as the discriminator score. This design introduces the adversarial objective to the original SiD with minimal additional parameter cost.
-	The proposed method SiDA significantly outperforms SiD and achieves superior 1-step generation performance on various datasets.
The authors propose SiDA (Score identity Distillation with Adversarial Loss), an enhanced version of Score identity Distillation (SiD) that aims to improve diffusion model training efficiency. The key innovation is the introduction of an adversarial loss that helps distinguish between real and generated latents at different noise levels. By reusing the fake score network in the original SiD, the authors propose a discriminator design that utilizes the encoder feature map to obtain the discriminator score. The proposed method achieves SOTA results on CIFAR-10, ImageNet-64, and other datasets.
-	The discriminator design is heavily based on the U-Net encoder-decoder architecture. It is not straightforward to generalize to transformer-based diffusion models, e.g., DiT, where there is no clear partition of the encoder and decoder.
-	The current submission lacks the evaluation of Diffusion models on higher resolution, e.g., ImageNet 256x256. The cost overhead from both discriminator training and maintaining an additional fake score network can be exaggerated on high-resolution data.
Yes
No ethics review needed.
- How does the method perform as the number of available real images changes? What happen if you use only 10% of the training real images?
- Why not apply latent diffusion for larger images? Using VAE to compress the image would help to see if the method can scale to bigger images, without increasing the training time
- Does the approach generalize to teacher models other than VP-EDM?
- The paper arbitrarily sets $\lambda_{sid}$ = 100 and $\lambda_{adv}$ = 0.01 without discussion or ablation, although these hyperparameters likely play an important role in the method. How do these values influence the performance metrics?
- In Figure 1, for $\alpha$=1.2 (the yellow line), after approximately 1000 iterations, the Inception Score (IS) drops and the FID increases sharply; additionally, SiDA with $\alpha$=1.2 is absent from Table 1. Does this imply that the model fails to converge?
- Achieves state-of-the-art performance with only one inference step
- The methods demonstrates improvement over the teacher network thanks to the real images
- Training remains relatively stable despite the use of an adversarial loss
The paper introduces Score Identity Distillation with Adversarial loss. The method aims to distill a teacher diffusion model to reduce the number of inference steps to only one. The primary contribution is the combination of SiD with the use of real images and adversarial objective, which not only reduces the number of required inference steps but also improves generation quality, achieving a lower FID score across Cifar10, ImageNet-64, FFHQ-64 and AFHQ-64.
- Requires real images for training, which is not necessary for other distillation methods
- Experiments are demonstrated only on very small images resolution
- Ablation studies are limited to the parameter $\alpha$
- A schematic of the framework would greatly improve the paper's readability.
Yes
No ethics review needed.
Please see the Weakness section.
- The discussions in the experiment section are very comprehensive. The detailed analysis of the training statistics across the training steps is appreciated. 

- The paper presented the background such as SiD and adversarial loss in diffusion well.
This paper introduces an improved version of score identity distillation and proposes training SiD with an adversarial objective. 
The main motivation is that the lack of real data in the original training of SiD can potentially lead to discrepancies to the original data distribution. The authors propose to reuse the encoder part of the fake score model as the backbone of discriminator, and use its predicted realness and fakeness scores to improve the generation results further, at no cost of additional model parameters. By combining SiD with adversarial training, together with additional techniques such as initializing the student model with the SiD pretrained model, the proposed method achieved improved one-step diffusion results on multiple datasets.
- Presentation

In the technical parts of the paper, which is mainly the Section 4. I frequently find the presentations not clear enough. My biggest confusion arises from the 'Loss Construction that Respects the Weighting of Diffusion Distillation Loss' paragraph. I understand that balancing loss terms can be tricky, and I acknowledge that this is an important topic to discuss. But from what the author presents, I didn't find how the proposed weight 'honors the existing weighting mechanisms' and, more importantly, automatically balances the loss regardless of the datasets. I read the Eq 7 and Eq 8 and found nothing particularly new. How is Eq 7 fundamentally different from PatchGAN? 

- Novelty

Although the paper targets a problem of SiD that is not previously discussed, the solution of adding an adversarial term is by no means a novelty under the context of diffusion distillation. The adversarial object has been proven very effective in many works, such as UFOGen and SD3-turbo. The proposed method here is a simple mixture of two existing methods. 

- Evaluation and performance

Most of the experiment settings used in this paper are small-scale. And the authors only compared the proposed method with very few other methods in the 512x512 ImageNet experiment. Since efficiency is one of the main advantages the authors are trying to advocate, it is necessary to include larger-scale evaluations, such as text-to-image experiments, to allow for fair comparisons with recent methods. 

I also have a slight concern about the advantage of efficiency. It is true that, according to Figure 3, the FID of the proposed method decreases faster compared to the baseline SiD in the later half of the training; I'd like to point out that this is achieved with an additional loss term, meaning each iteration probably optimizes the model with a larger gradient magnitude. Also, the adversarial loss term will inevitably introduce more computation, so the model is not necessarily able to finish training faster compared to SiD.
Yes
No ethics review needed.
+ Why can incorporating average "fakeness" within each GPU batch into each pixel's loss solve the gap between diffusion distillation and adversarial learning?
+ Why does the proposed method can achieve faster convergence than baseline when trained from scratch?
+ This paper proposes an intuitive extension of score identity distillation and effectively improves the generation performance on small-scale datasets, such as CIFAR and ImageNet 64x64.
+ The proposed method achieves better performance than the main baseline, SiD and other diffusion methods.
This paper proposes an intuitive method, which mainly focuses on the improvement of score identity distillation method by introducing additional real images for adversarial training. The improvement is two fold. First, additional real images can boost generation performance over data-free score identity distillation by discrimating real and synthetic images. Second, it achieves faster convergence rate relative to the baselines. The experimental results on small-scale dataset support the superority of the proposed method.
+ More state-of-the-art generative methods are missing, such as DiT, RDM and MAR.
+ The method only conducts experiments on small-scale datasets or low-resolution images. I'm concern about the effectiveness of this method on large-scale dataset with high-resolution images. More results on large-scale datasets with high-resolution images consolidate the effectiveness of this method.
+ OSGAN[R1] adopts an one-step adversarial training on generative task. The comparion with OSGAN is needed to distinguish their differences and highlight the contributions of this paper.
+ This paper doesn't mention any plan about open source of code and trained weights. In spite of encouraging performance, I'm concern about the reproducibility of this paper. 
+ Some motivations are not well clarified in the paper. See Questions section.
+ Some unnecessary equations, such as Eq between 149 and 151, can be removed.

[R1] Training Generative Adversarial Networks in One Stage. CVPR 2021.
@inproceedings{
anonymous2024adversarial,
title={Adversarial Score Identity Distillation: Rapidly Surpassing the Teacher in One Step},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=lS2SGfWizd},
note={under review}
}
Score identity Distillation (SiD) is a data-free method that has achieved state-of-the-art image generation performance by leveraging only a pretrained diffusion model, without the need for any training data. In this paper, we introduce SiDA (SiD with Adversarial Loss), which not only enhances generation quality but also improves distillation efficiency by incorporating real images and adversarial loss. SiDA utilizes the encoder from the generator's score network as a discriminator, enhancing its ability to differentiate between real images and those generated by SiD. The adversarial loss is batch-normalized within each GPU and then combined with the original SiD loss, effectively integrating the average "fakeness" per GPU batch into the pixel-based SiD loss. This allows SiDA to distill a single-step generator either from scratch or by fine-tuning an existing one. SiDA converges significantly faster than its predecessor when trained from scratch and quickly surpasses the original model’s performance after an initial warmup period when fine-tuning from a SiD checkpoint. This method has established new benchmarks for low FID scores when distilling EDM diffusion models pretrained on CIFAR-10 (32x32) and ImageNet (64x64), achieving FID scores of **1.499** on CIFAR-10 unconditional, **1.396** on CIFAR-10 conditional, and **1.110** on ImageNet 64x64.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion distillation deep generative models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/66cfafe07c19ca7385459ce0a05b8ce5dd786b0b.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Adversarial Score Identity Distillation: Rapidly Surpassing the Teacher in One Step
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission