$\Delta$-DiT: Accelerating Diffusion Transformers without training via Denoising Property Alignment
Diffusion Model;Training-Free;Acceleration;Diffusion Transformer
Yes
No ethics review needed.
N/A
1. The paper proposes a simple method for speeding up the generation process of transformer-based diffusion models, which is training-free and thus can be easily applied.
2. The results look good from the numbers on the COCO dataset.
This paper proposes a method specifically tailored for transformer-based diffusion models to speed up the inference process. The authors first propose to utilize $\Delta$-Cache to skip several transformer blocks to speed up the inference process. Then the authors discovered the alignment between the transformer blocks and the denoising process e.g., the shallow blocks and the early stage of the diffusion denoising process mostly account for the outline generation. By utilizing this discovery, the authors then propose $\Delta$-DiT to further refine the caching method to improve the generation quality.
1. It seems to me that the alignment ($\Delta$-DiT) does not standout against the baseline method $\Delta$-Cache from Table 1 which makes me a bit concerned about the effectiveness of the alignment. I'm curious have the authors also tried some other ways to apply the $\Delta$-Cache in different blocks/steps.
2. The authors conduct experiments on PixelArt-$\alpha$ and DiT-XL. Could the authors also provide results on more recent models such as SD3 which is also transformer-based?
Yes
No ethics review needed.
- Despite applying caching, there seem to be many cases where the FID score actually decreases. What might be the reason for this result? Could this outcome imply an experimental error or a lack of accuracy in the FID metric? Are there experimental results using more recent metrics, such as CMMD [1]? 
- At which exact timestep do you perform computations for caching? Are these computations performed at uniform intervals? 
- What is the unit of the DiT Block when calculating Delta? 
- What is the memory overhead required to run Delta-Diffusion? 

[1] Jayasumana, Sadeep, et al. "Rethinking FID: Towards a Better Evaluation Metric for Image Generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
- This paper is well-written and works effectively with a relatively simple idea. 
- The proposed Delta-DiT can accelerate the existing DiT model by more than 1.6 times without additional training. 
- It is also beneficial that it works in few-step denoising scenarios, such as with LCM.
This paper presents a framework called Delta-DiT, designed to accelerate the Diffusion Model architecture of DiT. Delta-DiT enables faster denoising operations by caching the differences between specific blocks within the DiT model, eliminating the need for actual computation. The proposed method outperforms existing solvers, such as DPM-Solver, as well as other diffusion acceleration methods like FasterDiffusion.
- There is a lack of experimental results demonstrating that Delta-Dit is actually faster than existing methods like DPM-Solver and Faster Diffusion. To compare performance accurately, it should be evaluated on a Pareto curve with latency and FID as axes. The paper only provides a single point of comparison in a table, which may be cherry-picked. 
- There are no experimental results using Unet. While the paper primarily targets DiT, Unet is still widely used in diffusion architecture, making a significant limitation to this work.  
- Error in Related Works: In the case of So et al., they are also targeting DiT. 
- The acceleration results in Table 3 are too small.
Yes
No ethics review needed.
1. The proposed delta-cache is training-free and can be used without training data. The main experimental results in the paper show that caching some feature changes can even improve the generation performance, which seems counter-intuitive. Could the author provide a deeper analysis of this?

Overall, this is an interesting work. The reviewer is willing to adjust the score.
1. The proposed acceleration method is training-free, making it more practical and easy to apply.

2. The idea of reusing the change instead of the block feature is interesting.

3. Figures 2 and 3 analyze the property of different generation stages, which is insightful.
This paper studies cache-based acceleration for Diffusion Transformers (DiTs) and proposes a training-free method, delta-cache. To compensate for the lack of long shortcuts in DiTs, delta-cache stores the incremental changes in feature maps. Furthermore, delta-cache aligns with the denoising property by caching deep blocks during the early stages and shallower blocks in the later stages. Experiments show that delta-cache achieves 1.6x speedup in 20-step generation while improving image quality.
1. The choice of bound b and the number of cached blocks Nc is empirical. Figures 6 and 7 show that the results are sensitive to the values of b and Nc. Is there a better optimization method for delta-cache?

2. The method achieves 1.6x acceleration in 20-step generation, which seems insignificant compared to other compression methods such as post-training quantization and time-step distillation. Is it possible to achieve more than 2x acceleration without sacrificing performance? Could the author clarify the advantage of cache-based methods on DiTs?
Yes
No ethics review needed.
Please see the weaknesses.
The paper addresses the timely issue of accelerating diffusion processing and introduces the ∆-Cache method, which shows potential in enhancing the diffusion model's speed. The method is evaluated on two datasets.
This paper introduces a method to accelerate Transformer-based diffusion models  for image generation. It uses the ∆-Cache method to store incremental changes in diffusion blocks, speeding up the generation process.
1- The presentation of the paper is not clear. It is hard to understand the proposed model. For example, ∆-Cache and the DiT blocks are introduced with insufficient context. Even in the abstract, jumps between ideas such as the role of shallow and deep blocks without clear transitions, complicating the flow. In the proposed method section, there are repetitions on the performance of the blocks.  These parts can be compact to improve the presentation. Several parts of the paper are difficult to undrestand, for example, page 2, line 86 -94, page 5 line 237-240, 

2- As I compared the results in Table 1 and Table 3, the performance improvement (speedup and latency) is marginal (Ours (b = 12), Faster Diffusion, PIXART, and TGATE have very close speedup). Yes, it is right that the proposed model has slightly better speed and performance, but it is not feasible. The authors are encouraged to discuss the practical implications of these improvements to better contextualise their importance in real-world applications.

3-  In page 6, line 302, what do you mean by " imaginary parts of a complex number"?
@inproceedings{
anonymous2024deltadit,
title={\${\textbackslash}Delta\$-DiT: Accelerating Diffusion Transformers without training via Denoising Property Alignment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pDI03iK5Bf},
note={under review}
}
Diffusion models are now commonly used for producing high-quality and diverse images, but the iterative denoising process is time-intensive, limiting their usage in real-time applications. As a result, various acceleration techniques have been developed, though these primarily target UNet-based architectures and are not directly applicable to Transformer-based diffusion models (DiT). To address the specific challenges of the DiT architecture, we first analyze the relationship between the depth of DiT blocks and the quality of image generation. While skipping blocks can lead to large degradations in generation quality, we propose the $\Delta$-Cache method, which captures and stores the incremental changes of different blocks, thereby mitigating the performance gap and maintaining closer alignment with the original results. Our analysis indicates that the shallow DiT blocks primarily define the global structure of images such as compositions, and outlines, while the deep blocks refine details. Based on this, we introduce a denoising property alignment method that selectively bypasses computations of different blocks at various timesteps while preserving performance. Comprehensive experiments on PIXART-$\alpha$ and DiT-XL demonstrate that $\Delta$-DiT achieves a $1.6\times$ speedup in 20-step generation and enhances performance in most cases. In the 4-step consistent model generation scenario, and with a more demanding $1.12\times$ acceleration, our approach significantly outperforms existing methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion Model Training-Free Acceleration Diffusion Transformer
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/70768cf26b7dc2e1cf0fcdc7dbe961374d2e251c.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
$\Delta$-DiT: Accelerating Diffusion Transformers without training via Denoising Property Alignment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission