Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
auto-regressive image generation;acceleration;training-free;image generation
Yes
No ethics review needed.
Please see the Weaknesses
1. This paper adapts speculative decoding to auto-regressive (AR) image generation, achieving substantial speed improvements with minimal impact on image quality, addressing a key efficiency challenge in this area.

2. The proposed method demonstrate performance gains on state-of-the-art models, including Lumina-mGPT and Anole, underscoring its applicability and effectiveness.
This paper proposes Speculative Jacobi Decoding (SJD) for accelerating the inference of auto-regressive (AR) image generation. SJD introduces a probabilistic acceptance criterion, using the ratio of the probability conditioned on the current draft tokens to that of the previous iteration for speculative verification, enabling faster convergence while preserving diversity in sampling-based decoding. This approach reduces the number of steps required, improving inference speed without compromising image quality.
1. Need for analysis on image randomness in SJD. The randomness in SJD introduces variability in generated images, but there is limited analysis on the extent of this randomness. Unlike speculative decoding in the language domain, which offers some guarantees on reproducibility, SJD would benefit from theoretical bounds or deeper empirical analysis to clarify the extent of variation in its results.

2. Demonstrating similar speed-up and preservation of image quality on additional models, such as LlamaGen, would strengthen this paper.

3. Need for additional qualitative evaluation: Additional qualitative results, as in Fig. 5, comparing original images with those generated by the proposed method, along with steps and latency comparisons on Lumina-mGPT and Anole, would be beneficial. Particularly, a detailed discussion on representative failure cases and their causes is needed to clarify potential issues in image quality.

3. While this paper tends to emphasize step counts in terms of speed-up, latency is a more accurate metric for evaluating it. Therefore, using latency as the primary measure would enable a clearer comparision. 

4. Improved clarification in Fig. 4 would enhance understanding of the proposed method.
Yes
No ethics review needed.
- The acceptance is determined if it improves prediction compared to the conditions of the previous Jacobi iteration. I wonder if the conditional probability given the draft tokens from the previous iteration is too low; it may have a chance to accept tokens even if it still has a very low conditional probability but a little bit higher than that, which may result in inaccurate token acceptance.
- Clear motivation about the limitation of existing performant auto-regressive T2I generation models and Jocobi decoding strategy.
- Achieving about 2 times faster inference latency with fewer iteration steps, with a small drop in image quality compared to original T2I backbones.
- This approach can be applied to multiple different auto-regressive T2I generation models.
- Provide several meaningful ablations to better understand the robustness of the proposed methods with varying scales of hyperparameters.
Recent auto-regressive models generate quality images but require thousands of next-token prediction steps during inference, which is significant in terms of inference speed. To address this, this paper proposes Speculative Jacobi Decoding (SJD), a training-free probabilistic parallel decoding algorithm. SJD introduces a probabilistic convergence criterion, allowing for sampling-based token decoding that maintains diversity and accelerates inference while overcoming the limitation of original Jacobi Decoding - deterministic, greedy decoding, which limits diversity to be incorporated into the image generation framework. This approach enables multiple token predictions per step in a probabilistic way (i.e., inserting randomness), reducing required steps while preserving image quality.
- The proposed approach requires over-computation in their Jacobi iteration. It computes conditional prediction scores for all tokens in the window but only accepts several earlier tokens by design - so resources used for computing later tokens are the first unaccepted tokens and later ones, even if some of them are technically accepted. This is inevitable since next-token prediction should be conditioned on all previous tokens that should be accepted (except for the beginning) and cannot allow some undesired holes of unaccepted tokens. While the approach proceeds these unaccepted tokens to new draft tokens for the next iterations, the effectiveness of this technique is not studied throughout the paper. At least, comparing the impact of this reuse of over-computed tokens with several token initializations with spatial priors (and random as well) would be meaningful.

- Initialization with spatial prior (ISP) doesn't show real acceleration in both terms of latency and steps compared to SJD w/o ISP. Figure 9 shows some different behavior, but the gap looks marginal in the main table (Table 1). I would say this is not appealing in terms of idea since this is from a random idea without strong grounding, less relevant to the overall story of this paper, and the performance is also less impactful.

- Analyses/ablations look incomplete. For Figures 6,7,8 and 9, I believe this should also be meaningful to provide quantitative performance of image quality rather than reporting only step compression rate. Reporting only the efficiency of the proposed approach does not give meaningful insight.

- The experimental results with Anole in Table 1 also look incomplete. I think more rows (i.e., B and C in Lumina-mGPT cases) are necessary for this table.

- (minor) typo: Each column presents --> Each row presents
Yes
No ethics review needed.
Q1: Could the authors provide the conparison of perplex between SJD and other decoding methods?


Q2: I'm curious about how many image samples are used to compute the FID scores. Could the authors provide more evaluation details?
I doubt that the authors have use some tricks in evalution.  For example, the image quantity maybe different to compared method
Q3: Could the author provide more  results on small  models?
Not everyone has access to large models, it's more convicing to conduct experiments on small models
This paper introduces reject sampling to accelerate auto-regressive text-to-image generation.
SJD enables the model to predict multiple tokens at each step, accepting them based on a probabilistic criterion, reducing the number of inference steps compared to conventional next-token-prediction.
The paper proposes a spatial locality-aware token initialization strategy to further enhance acceleration in specific scenarios.
Experiments on multiple auto-regressive text-to-image generation models demonstrate SJD's effectiveness in accelerating inference without compromising visual quality.
SJD iteratively decodes multiple tokens in parallel within a sliding window. The model computes the conditional probability for a sequence of draft tokens in a single forward pass. A probabilistic criterion then determines which tokens to accept, adding them to the fixed pre-filling sequence. Unaccepted tokens are resampled for the next iteration. The spatial locality-aware token initialization strategy leverages the spatial relationships in images for initializing new tokens. SJD effectively accelerates auto-regressive text-to-image generation models, achieving approximately 2× speedup for Lumina-mGPT and Anole without significant visual quality degradation.
1) SJD relax the greedy acceptence criteron of next token by reject sampling. Such acceleration will sample results with smaller probabilities and JD is also able to update multiple tokens at a time.  And relaxation of acceptence criteron lead to suboptimal results. Moreover, this paper is infact very simple with little experiments. It's more like a trick rathor than a paper.
2) The results that JD only accelerate t2i models by a rate of 1.04 is not convicing, which is much worse than results provided by JD and other papers.
3) The evaluation details are not provided. As far as I'm concerned, SJD may leads The nolvelty seems incremental. to suboptimal image quality and better image diversity. Hence, the FID score would be better if the image quality decrease a little and the image diversity increase.
4) There are too little models tested. It's better to apply SJD to more large models and small models.
Not that, I'm not familar to large model accelaration, I will consider other reviews' comments.
@inproceedings{
anonymous2024accelerating,
title={Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=LZfjxvqw0N},
note={under review}
}
The current large auto-regressive models can generate high-quality, high-resolution images, but these models require hundreds or even thousands of steps of next-token prediction during inference, resulting in substantial time consumption. In existing studies, Jacobi decoding, an iterative parallel decoding algorithm, has been used to accelerate the auto-regressive generation and can be executed without training. However, the Jacobi decoding relies on a deterministic criterion to determine the convergence of iterations. Thus, it works for greedy decoding but is incompatible with sampling-based decoding which is crucial for visual quality and diversity in the current auto-regressive text-to-image generation. In this paper, we propose a training-free probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation. By introducing a probabilistic convergence criterion, our SJD accelerates the inference of auto-regressive text-to-image generation while maintaining the randomness in sampling-based token decoding and allowing the model to generate diverse images. Specifically, SJD facilitates the model to predict multiple tokens at each step and accepts tokens based on the probabilistic criterion, enabling the model to generate images with fewer steps than the conventional next-token-prediction paradigm. We also investigate the token initialization strategies that leverage the spatial locality of visual data to further improve the acceleration ratio under specific scenarios. We conduct experiments for our proposed SJD on multiple auto-regressive text-to-image generation models, showing the effectiveness of model acceleration without sacrificing the visual quality. Code will be released upon acceptance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
auto-regressive image generation acceleration training-free image generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/252a4682f4e876c38d702b83f5d32c6d8e0a6ab9.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission