HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
autoregressive models;image generation;text-to-image
Yes
No ethics review needed.
In Table 4, I am interested to know if increasing the diffusion steps for HART during inference—similar to the approach used in MAR—would lead to an improvement in generation quality. Clarifying this aspect would help in understanding the impact of diffusion steps on HART's performance.
The hybrid tokenization approach presented in this paper demonstrates impressive reconstruction quality, outperforming methods that rely solely on discrete tokens. Additionally, the necessity of residual tokens is convincingly validated through detailed ablation studies, which provide strong empirical support for their inclusion. The model architecture—an autoregressive model with discrete tokens and conditioned diffusion on continuous tokens—exemplifies an effective design choice that balances efficiency and performance. This hybrid approach not only achieves comparable speeds to VAR in class-conditional image generation tasks but also scales effectively to large datasets, as evidenced by its application to tasks like text-to-image generation.
This paper addresses limitations in autoregressive models (AR models), particularly the challenges in image reconstruction due to discretized tokens and the high computational costs of generating high-resolution images. To overcome these issues, the authors introduce a novel hybrid tokenizer that combines both discrete and continuous tokens. Specifically, discrete tokens are generated in an AR style, while continuous tokens are generated in a diffusion style conditioned on the discrete tokens. Compared to the VAR tokenizer, this approach demonstrates enhanced reconstruction FID performance. Moreover, in generative modeling tasks, it shows superior efficiency compared to current state-of-the-art models.
While Figure 7 demonstrates that residual tokens in HART are indeed easier to learn than full tokens in MAR, the results from the class-conditional image generation task seem to contradict this intuition. MAR, which generates full continuous tokens, achieves competitive FID performance despite having fewer parameters than HART, which focuses on generating residual tokens—ostensibly a less complex task. 

This raises questions about the specific aspects that contribute to the observed performance gap between the two models. It would be interesting to explore whether this difference could be mitigated by increasing the diffusion steps for HART, similar to the approach in MAR. Further clarification or analysis in this area would provide valuable insights into the comparative efficiency and effectiveness of the models.
Yes
No ethics review needed.
See the weaknesses, and my major concern is that: 
 
Can you provide a more in-depth theoretical explanation of HART's hybrid tokenization and why it outperforms traditional discrete-only AR approaches? Not only to put some tricks to improve the performance. This would help clarify whether the hybrid approach could serve as a generalized framework for other AR T2I tasks.
- The model demonstrates efficiency improvements over state-of-the-art diffusion models. The residual diffusion approach minimizes memory and processing costs, achieving a reduction in latency and MACs. As a result, HART’s latency and throughput metrics are efficient, offering a speed and computational advantage.

- the hybrid tokenizer combining discrete and continuous tokens is efficient in autoregressive (AR) image generation, providing higher fidelity in image reconstruction and generating fine details often missed by discrete-only models. This method addresses the usual limitations of discrete tokenizers by retaining important image details, especially at high resolutions (1024×1024）
The paper presents HART, a Hybrid Autoregressive Transformer model for efficient, high-resolution text-to-image generation. Key innovations include a hybrid tokenization approach that combines discrete and continuous tokens, enabling finer detail capture while reducing computational overhead. HART achieves better efficiency, with lower latency and higher throughput than comparable models, and directly generates 1024x1024 images without super-resolution. However, the paper could benefit from a stronger theoretical foundation, exploration of alternative text conditioning methods, more comparative analysis with similar models like VAR-CLIP and STAR, and clearer differentiation from MAR, especially regarding inference optimizations. Overall, HART is a promising work in autoregressive T2I generation.
- **Limited Theoretical Foundation**:
    - The paper relies predominantly on experimental results, without providing a thorough theoretical basis for the proposed methodology. A more detailed theoretical analysis would strengthen the paper’s rigor and enhance the broader applicability of the approach.
- **Ambiguity in Differentiation Between HART and other work, e.g. MAR**:
    - The distinction between HART and MAR is not fully clear, especially concerning optimizations like sampling efficiency and inference techniques. For example, while HART achieves optimal quality with just 8 sampling steps due to its diffusion setup, it is unclear why MAR could not potentially achieve similar results with similar diffusion adaptations. Additionally, some statements (lines 329-335) imply that certain optimizations, such as KV-caching, are unique to HART, but MAR could likely implement these as well. Further clarification on these points would provide a more accurate comparison between the two models and help readers understand the unique contributions of HART

- **Unexplored Alternatives for Text Conditioning**:
    - The current approach employs text tokens as the sequence start token, but it does not explore or compare this choice with other established methods, such as cross-attention mechanisms commonly used in diffusion models for text-to-image (T2I) tasks. Including comparisons with alternative conditioning methods would clarify the advantages or limitations of the current setup.
- **Insufficient Comparative Analysis**:
    - The paper would benefit from a broader evaluation against similar autoregressive T2I models, such as VAR-CLIP and STAR. A horizontal comparison would add depth to the experimental results, positioning HART more clearly within the landscape of current models.
Yes
No ethics review needed.
Please explain the above questions.
1. This paper proposes a feasible scheme to solve the key problem of AR model: effective modeling of image quality, which is ahead of other methods in FID.
2. The paper is logical, the process of problem solving is very smooth, and the experimental setting is also reasonable.
This paper proposes a feasible scheme for AR model to produce high quality images effectively, and has a great improvement in performance and throughput.
1. There is doubt about the use of 50% for modeling discrete and continuous tokens, what happens if it is alternate, please explain the necessity of 50% here, and if there is a 30% accident, whether the other probability will break the balance, in other words, whether 50% is the best balance between the probabilities of the two modeling methods.
2. The overall solution of this paper is based on the random selection of two mature pipelines. There is a lot of randomness in training in this way, will it lead to poor stability of the network? Besides, this kind of scheme is not the most ideal for balancing the combination of continuous tokens and the separation of tokens, and it is a cheating scheme.
Yes
No ethics review needed.
see Weakness.
1. The paper is well-written and easy to follow.
2. The idea of using a compact diffusion model to model the residual continuous token is both simple and effective. I believe this design will offer valuable insights to the community.
3. The experiments are carefully designed. The results and conclusions are convincing.
This paper introduces HART, an innovative autoregressive transformer designed for efficient visual generation. It employs a hybrid token space to represent images, with the discrete component modeled by a VAR model and the continuous component modeled by a compact diffusion model. Extensive experiments demonstrate that HART outperforms previous baselines in both image quality and efficiency.
The paper notes that a lower rFID does not necessarily indicate a better gFID, which is also supported by the ablation experiments. However, I feel that there is a lack of analysis regarding this phenomenon. For instance, it would be beneficial to provide insights into which design elements are crucial for obtaining tokens that achieve both good rFID and gFID simultaneously, and the other designs lead to a bad gFID.
Yes
No ethics review needed.
- The illustration of Figure 6. First, the arrow direction of scalable-resolution autoregressive transformer is wrong. Second, the last layer hidden states are not figured out in Figure 6. Third, the HART attention mask (right) does not contain much information. The attention mechanism of each stage is not easy to obtain for the readers.
+ This paper proposes a hybrid tokenizer, addressing the poor image reconstruction quality of existing discrete tokenizer. Also, the hybrid tokenizer is simple and easy to implement.
+ To model tokens generated by the hybrid tokenizer, this paper introduce a scalable-resolution discrete AR model and a light-weight residual diffusion model. Based on these modules, the proposed method can generate 1024px image efficiently.
+ The authors conduct exhaustive experiments to validate the proposed method. Compared with existing AR and diffusion models, the proposed method achieves superior performance on text-to-image and class-conditioned image generation tasks.
This paper presents an efficient and high-resolution autoregressive visual generation model with a hybrid tokenizer. The hybrid tokenizer includes a discrete tokenizer and a continuous tokenizer. The discrete tokenizer that is widely used in existing AR models is applied to model image structure, while the continuous tokenizer is used to represent residual image details. To model these image token priors, this paper also introduces a scalable discrete AR model and a light-weight residual diffusion model. Based on above modules, the proposed method beats VAR in rFID and outperforms sota diffusion models in both FID and CLIP score.
- The condition of residual diffusion. As shown in line 320~321, the condition consists of last layer hidden states from AR transformer and the discrete tokens predicted in the last sampling step. To my knowledge, the discrete tokens predicted in the last sampling step only contain residual image details. It is not enough to predict the residual tokens. Thus, the hidden states are important. Does the hidden states come from all sampling steps?
- Efficiency enhancements (Training). In line 346~349, the fist sentence is about the overhead of residual diffusion module. But the following solution (discarding 80% tokens) is used to mitigate training overhead of VAR, which is also illustrated in Appendix A.1. In my opinion, there is a mismatch between these two parts.
- Alternating training in hybrid tokenizer. To my knowledge, with this training scheme, visual decoder is robust to the error of predicted tokens. For generation model, the error is large in the early training. Thus, the decoder with this scheme performs better than others, as shown in Figure 7 (middle and right). Does the great advantage of alternating training exist after generation model training?
The first autoregressive model that can directly generate 1024x1024 images. It takes advantage of hybrid tokenization and residual diffusion to model both continuous and discrete tokens.
@inproceedings{
anonymous2024hart,
title={{HART}: Efficient Visual Generation with Hybrid Autoregressive Transformer},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=q5sOv4xQe4},
note={under review}
}
We introduce Hybrid Autoregressive Transformer (HART), the first autoregressive (AR) visual generation model capable of directly generating 1024x1024 images, rivaling diffusion models in image generation quality. Existing AR models face limitations due to the poor image reconstruction quality of their discrete tokenizers and the prohibitive training costs associated with generating 1024px images. To address these challenges, we present the hybrid tokenizer, which decomposes the continuous latents from the autoencoder into two components:  discrete tokens representing the big picture and continuous tokens representing the residual components that cannot be represented by the discrete tokens. The discrete component is modeled by a scalable-resolution discrete AR model, while the continuous component is learned with a lightweight residual diffusion module with only 37M parameters. Compared with the discrete-only VAR tokenizer, our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K, leading to a 31% generation FID improvement from 7.85 to 5.38. HART also outperforms state-of-the-art diffusion models in both FID and CLIP score, with 4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Code will be released upon publication.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
autoregressive models image generation text-to-image
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9791f1b1a7287c9b9878ab28a022dc63a8d7cb64.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission