De-biasing Diffusion: Data-Free FP8 Quantization of Text-to-Image Models with Billions of Parameters
Diffusion;Quantization;Floating-Point;Data-Free;Text-To-Image
Yes
No ethics review needed.
* The concept of stochastic weights is unclear to me. Are the weights stored using $8+p$ bits while computations are still performed with 8-bit precision?
* In the paper, the authors apply their methods only to the 50-step FLUX.1-dev model. I wonder if the proposed method is also effective on the 4-step FLUX.1-schnell model.
* The authors evaluate their method directly on FLUX, a state-of-the-art diffusion model, making the method practical for real-world use.
* In addition to FID, the authors report PSNR and SSIM metrics to compare their generated images with those from 16-bit models, showcasing the robustness of the proposed approach.
* The proposed method is supported by strong mathematical analysis.
In this paper, the authors introduce a de-biasing approach for quantizing large-scale text-to-image diffusion models to 8 bits. They identify and mathematically analyze the bias in the existing quantization methods, which negatively impacts quantization performance, especially for long denoising sequences. To mitigate this, they use simple stochastic rounding to reduce bias. Beyond applying stochastic rounding to activations, they extend this method to weights, introducing the concept of Stochastic Weight Rounding (WSR). Experiments on SDXL and FLUX demonstrate the effectiveness of the proposed approach.
* As noted in lines 407–417, using stochastic weights requires storing additional $p$ bits. Therefore, for a fair comparison, I suggest that the authors compare their method to W$(8+p)$A$8$ baselines. It is possible that the observed improvement primarily results from the additional weight bits.
* Writing and Presentation:
  * In Figures 1, 2, and 3, there are too many curve lines in each figure, making them difficult to interpret. Additionally, some colors are similar, and some lines overlap, making it hard to distinguish between them.
  * The authors should include visual comparisons within the main text to better showcase their results.
  * To enhance clarity, the authors should consider adding a figure that provides an overview of their main method. Currently, it is hard to grasp the key ideas from the pure text.
  * Line 131: Consider replacing "U-Net" with a different term, as FLUX uses a diffusion transformer.
  * Figure 3: The left figure lacks a label for the y-axis.
  * Line 169: The quotation marks for M$x$E$y$ are incorrect.
Yes
No ethics review needed.
Please see weaknesses.
The paper studies the effect of fp8 quantization on diffusion models and presents various observations and recommendations for practical use. The authors convincingly show that main issue in FP8 quantization can be attributed to quantization bias and as such, propose to de-bias it with stochastic rounding schemes. Such a rounding scheme can be easily implemented for activations, but for weights it will be counter-productive; as such authors propose wight-stochastic-rounding (WSR) scheme that maintains FP8 weight copy + p-bit mantissa and certain uniform mixing scheme.  Authors  provide comparisons in terms FID/PSNR for every proposed de-biasing technique indicating that suggesting methods have merit and singling out M4E3 version of FP8 scheme as most suitable candidate for deployment.

The main strengths of the paper is thoughtfulness of the investigation and presentation: for every claim/equation in the paper there is sufficient empirical or theoretical explanation given in main paper or appendix.
The paper presents the study of FP8 quantization (in various special forms) for diffusion models in data-free setting; it overviews various techniques to overcome quantization error/bias of aforementioned schemes, and has actionable recommendations for practical use. The presentation of the paper, including overall writing, and other evidence (proofs, plots, and supporting information) is very informative and is one of the strengths of the paper.
There are only few weaknesses of the paper, and most of them were mentioned by authors in the manuscript:
1. Practical applicability of some of the proposed schemes. In the current form, only the stochastic rounding of activations can be deployed on devices; any other schemes need significant effort in software/hardware support. SR itself gives measurable improvements, but not as good as SR+WSR (based on table results)
2. Assumptions on #diffusion steps used throughout manuscript (i.e., steps > 50) are unrealistic in my opinion simply due to the nature of discussion: if we are talking about efficiency and on-device deployment, such models are most probably will be deployed in few step regime
3. p=4 for WSR scheme might be too big (in my opinion) for any practical deployment despite the shown advantage: wsr with p=4 means 12bit storage for every weight tensor, as such, there might be alternative allocation or scheme for same 12bit per weight storage. I think, for any WSR benefits, authors should try coming up with less taxing scheme on additional storage.
3. User study. I think, having user study on perceived quality difference across proposed techniques would significantly strengthen the conclusions of the paper.
Yes
No ethics review needed.
* The paper focuses on fp8 quantization, where performance drops relative to uncompressed model are not very pronounced. Would be interesting to see whether the conclusions are valid for the case of more aggressive quantization (say W4/A8) - i.e one should focus on reducing the bias, rather than the variance term. I suggest having at least one experiments with stronger compression to showcase the difference between RTN performance and SR. 

* Noticeable decrease of FID metric relative to uncompressed model given high PSNR (small difference between the output of original and quantized model) looks suspicious. Do you have any explanation for this phenomenon? Do you adopt the same random seed for generation with compressed and original model? How robust is the FID value to the change of seed? 

* What is the sampler used for image generation? Is it the default sampler in diffusers config or DDIM?
* Bias-variance decomposition of quantization error makes sense and provides some useful insights about the possible solution to reduce the quantization error, 

* The superiority of the proposed stochastic rounding for weights and activations compared to naive rounding-to-nearest is validated on several modern architectures and inference setups.
This work proposes stochastic rounding of weights and activations to mitigate the bias incurred by quantization in the iterative denoising process. The introduced approach is evaluated on fp8 quantization of modern diffusion models - SDXL and FLUX.
* The overall novelty is limited. Stochastic rounding is an established technique in various applications [1, 2] and the current work adopts it for diffusion models.

* The evaluation setup is not very compelling. Unless the drop of generation quality is very significant, metrics such as FID do not reflect it. PSNR/SSIM between the generations produced by the original model and quantized one may be not a proper measure, as a compressed model can generate samples different from the teacher, yet of high quality. I would suggest adding user preference study at least for some cases as a reliable assessment. Specifically, given two images an assesor should tell, whether one image is better than other (or both are equal in terms of quality). 

* WSR(p=4) increases amount of memory used for model storage by 50% relative to fp8. I would suggest to check whether mixed fp8/fp16 quantization is still inferior in terms of quality. 

**Minor**

Line 84: PTDQ -> PTQD

---
[1] Gupta, Suyog, et al. "Deep learning with limited numerical precision." International conference on machine learning. PMLR, 2015.
[2] Wang, Naigang, et al. "Training deep neural networks with 8-bit floating point numbers." Advances in neural information processing systems 31 (2018).
Yes
No ethics review needed.
See above weaknesses

References:

[1] He, Yefei, et al. "Ptqd: Accurate post-training quantization for diffusion models." Advances in Neural Information Processing Systems 36 (2024).

[2] Podell, Dustin, et al. "Sdxl: Improving latent diffusion models for high-resolution image synthesis." arXiv preprint arXiv:2307.01952 (2023).

[3] Tang, Siao, et al. "Post-training quantization with progressive calibration and activation relaxing for text-to-image diffusion models." arXiv preprint arXiv:2311.06322 (2023).
- Floating point quantization for diffusion models is positive direction.
- The paper is handling the larger models, SDXL and Flux.1. 
- The approach is generally intuitive and easy to understand.
- This paper builds on top of PTQD [1] in adjusting the denoising schedule, which is good as not a lot of diffusion quantization paper adopt its approach.
- Figure 2 is effective.
This manuscript proposes a FP-based Diffusion Model quantization scheme. Specifically, the paper uses Stochastic Rounding, originally used for training, to assist the quantization calibration process. The technique uses PSNR as a measure to try and achieve calibration where images produced by a quantized denoising U-Net/Transformer are very similar to that of the full precision model. This approach is applied to SDXL and Flux.1.
- The primary similarity metric used by the authors for calibration is PSNR. They cite the SDXL [2] paper which states that FID has limitations due to Mean-Opinion-Scores. This is usually because the generates images are compared to real images (FID-GT). However, [3] propose FID-FP32, a variant of FID which compares the distribution of generated images from the quantized model, against those of the full precision model (which is treated as the baseline). If FID-32 is 0, the distribution of images from the quantized model is the same as that of the full precision model. Since this metric discards the weaknesses of the original FID, it is arguably a better metric for the objective than PSNR for what the authors aim to achieve, yet this paper does not seem to be aware of it.
- Most of the experimental results generate too few images (100 or 1k, e.g., Fig 1, Fig 3, Tab 2) to provide trustworthy and meaningful results. 
- Line 463-464 "the minimal recommended size" this is incorrect. See https://huggingface.co/docs/diffusers/en/using-diffusers/sdxl which states "anything below 512x512 is not likely to work".
- The paper only considers W8 precision, whereas other methods consider W4 [3].
We propose a simple, data-free method to improve diffusion models with FP8 quantization, with significant improvement for all open source text-to-image models.
@inproceedings{
anonymous2024debiasing,
title={De-biasing Diffusion: Data-Free {FP}8 Quantization of Text-to-Image Models with Billions of Parameters},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=nExUJBF5tR},
note={under review}
}
Diffusion neural networks have become the go-to solution for tasks involving automatic image generation, but the generation process is expensive in terms of memory, energy, and computational cost. Several works have aimed to reduce the cost via quantization to 8 bits or less. Despite that, half-precision (FP16) is still the default mode for large text-to-image generation models --- where reducing numerical precision becomes more challenging. In this work, we show that the reduction of quantization bias can be more important than the reduction in quantization (mean square) error. We propose a data-free method for model quantization, with the goal of producing images that are indistinguishable from those generated by large, full-precision diffusion models. We show that simple methods like stochastic rounding can decrease the quantization bias and improve image generation quality with little to no cost. To close the remaining gap between full-precision and quantized models, we suggest a feasible method for partial stochastic-rounding of weights. When using the MS-COCO dataset as the baseline, we show our quantization methods achieve as good FID scores as the full-precision model. Moreover, our methods decrease the quantization-induced distortion of the images generated by the full-precision model, with the distortion decreasing with the number of diffusion steps.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Diffusion Quantization Floating-Point Data-Free Text-To-Image
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2549d75c1ffc5fba39e03edb6b6e20989aeb6d65.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e80f977d7f9b4c6eac7fffc27e2996f8e83fb0e5.zip
De-biasing Diffusion: Data-Free FP8 Quantization of Text-to-Image Models with Billions of Parameters
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission