Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining
Autoregressive Image Generation;Multi-modality;LLM
Yes
No ethics review needed.
The comparisons and evaluations are insufficient for the work. Specifically, a few questions are still to be addressed:
* How does Lumina-mGPT compare with diffusion-based methods in terms of image generation?
* How does Lumina-mGPT perform on text-only tasks, comparing with other text-only LLMs?
* The proposed Unambiguous image Representation, when used with Flexible Progressive Supervised Finetuning (FP-SFT), allows the method to generate images of varying resolutions.
* While prior method Chameleon can only perform vision-language and text-only tasks, Lumina-mGPT achieves visual recognition tasks (e.g., segmentation, depth prediction) and controllable generation as well as image editing, which makes Lumina-mGPT a unified model for various downstream applications.
* The performance on VQA benchmarks significantly improves over baseline Chameleon, demonstrating the effectiveness of the proposed model.
This work presents Lumina-mGPT, a series of autoregressive models for multimodal understanding and generation. With a pre-trained decoder-only transformer, Lumina-mGPT allows for a unified framework for multi-model modeling. The work shows that multimodal generative pretraining is the key towards general multimodal capabilities. Furthermore, flexible progressive supervised fine-tuning, as proposed by the work, allows high-aesthetic image generation at flexible resolutions. Finally, omnipotent supervised fine-tuning enables Lumina-mGPT to achieve task unification in visual generation and understanding.
* The contributions of this work is not clearly described. While the work claims mGPT to be a key insight, the importance of mGPT is not discovered by the work, as the mGPT model is adapted from Chameleon. The difference is that Lumina-mGPT performs fine-tuning on Chameleon.
* The proposed Unambiguous image Representation (Uni-Rep) is only applied at the supervised fine-tuning stage. This creates gaps between the image representation in pre-training and fine-tuning. 
* The method claims that baseline Chameleon shows degraded visual quality compared to diffusion methods (L242-244). However, this work does not perform quantitative evaluations on the quality of the generated images to illustrate whether the method is able to overcome this limitation. Metrics such as FID and/or results on benchmarks such as T2I-CompBench [1] are needed to show the performance improvements against other related works.
* No comparisons are provided with text-only LLMs on text-only benchmarks such as MMLU. The example dialogues are insufficient in evaluating the model's capabilities in text-only tasks.
* No human evaluation is performed to assess the quality of multi-modal generation.

[1] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation. K. Huang, et al. https://arxiv.org/abs/2307.06350
Yes
No ethics review needed.
- In the appendix, you observe that *"with the CFG increasing, the quality of generated images improves, proving the effectiveness of the classifier-free guidance in this context."*. This finding appears to contrast with typical outcomes in diffusion models, where increasing the CFG beyond a certain threshold often results in diminished image quality. Could you elucidate the reasons behind this differing impact of CFG in your model compared to traditional diffusion models?
+ Omni-SFT within Lumina-mGPT effectively unifies a diverse array of tasks within a single model framework, demonstrating its extensive multitasking capabilities across various complex applications such as text-to-image synthesis, image captioning, image editing, spatial-conditional image generation, and more.
+ The use of mGPT for initial training, followed by sophisticated fine-tuning strategies like FP-SFT and Omni-SFT, establishes a solid foundation for generating high-quality, photorealistic images while adeptly handling a wide range of multimodal tasks.
+ Lumina-mGPT exhibits exceptional flexibility in generating images across different resolutions and aspect ratios—a significant advantage over many existing models.
+ By effectively bridging the gap between autoregressive and diffusion methods, Lumina-mGPT achieves remarkable visual aesthetics and detailed image rendering without the need for cascading models,
+ The paper ingeniously integrates various advanced concepts from the literature and existing models to enhance the training and inference capabilities of Lumina-mGPT. This includes adopting classifier guidance from diffusion models and stabilization techniques used in large language models, which collectively contribute to the robustness and efficiency of the framework.
This paper introduces Lumina-mGPT, a multimodal autoregressive model that excels in generating photorealistic images from textual descriptions, as well as performing a variety of other vision and language tasks. Built on a pre-trained decoder-only transformer architecture, the model adeptly processes multimodal token sequences. Key innovations of Lumina-mGPT include Effective Multimodal Generative Pretraining (mGPT) using extensive interleaved text-image datasets, Flexible Progressive Supervised Fine-tuning (FP-SFT), and Omnipotent Supervised Finetuning (Omni-SFT). These developments enable the model to produce images of varying resolutions and accommodate a broad spectrum of vision-language tasks, marking a notable advancement in flexibility and task integration over traditional autoregressive approaches.
- The paper falls short in providing comprehensive comparative metrics, such as FID scores, with only limited comparisons featured in Table 2 against the Chameleon model. A more extensive range of benchmarking against current state-of-the-art (SoTA) models is essential to objectively evaluate the model’s performance and its advancements over existing methodologies. Including a broader set of metrics would significantly clarify the model's positioning and contribution within the broader field.
- Although the paper showcases zero-shot capabilities in enhancing visual details, it only offers qualitative results. Incorporating quantitative evaluations would provide a more robust comparison against other models and substantiate the claimed improvements.
- While Omni-SFT demonstrates the model's capability to handle diverse tasks, the absence of quantitative results in a controlled evaluation test limits the understanding of its performance, especially in comparison to specialized models. Providing such data would help gauge the effectiveness of Omni-SFT and its relative performance across different tasks.
- The paper highlights the performance benefits of Omni-SFT fine-tuning on top of the Lumina model but lacks a comparative analysis with scenarios where Omni-SFT is fine-tuned starting from the initial mGPT. A direct comparison would help validate the advantage of transitioning from Lumina to Omni for task unification, ensuring that this methodological choice yields tangible benefits.
Yes
No ethics review needed.
The Large World Model (LWM) paper introduces a multimodal autoregressive model capable of processing extensive sequences of video and book data, with a context length of millions of tokens. This enables it to perform tasks such as language understanding, image analysis, and video generation. In contrast, the Lumina-mGPT model is focused specifically on photorealistic text-to-image generation. Is this understanding accurate? Based on reading both papers, LWM appears to use a less effective VQ-VAE compared to Lumina-mGPT's VQ tokenizers; does this difference explain the variation in visual quality? Could the authors provide an analysis to identify the areas contributing the most to these performance gains?

While Lumina-mGPT demonstrates impressive performance across various tasks, additional examples in less conventional vision-language settings would help illustrate its adaptability and generalization. Could the authors provide results or comparisons on more challenging benchmarks or diverse vision-language tasks (e.g., object counting, spatial reasoning) that may better highlight Lumina-mGPT's strengths and weaknesses across the multimodal spectrum?
The paper presents a systematic approach to fine-tuning through Flexible Progressive Supervised Fine-Tuning (FP-SFT) and Omnipotent Supervised Fine-Tuning (Omni-SFT). These methods are well-supported by empirical results, demonstrating that Lumina-mGPT generates high-quality, high-resolution images across various aspect ratios. Additionally, an extensive evaluation of zero-shot performance and attention visualizations offers insights into the model's generalizability and internal mechanisms.

The paper is well-organized and clearly articulates its goals, methodology, and findings. The authors effectively communicate complex technical concepts, ensuring that each design choice—such as the unambiguous image representation—is both motivated and illustrated, enhancing readability. While certain sections, such as the detailed explanations of supervised fine-tuning processes, may require technical background to fully appreciate, the overall clarity remains high.

Lumina-mGPT's primary contribution lies in bridging the gap between autoregressive (AR) models and diffusion-based image quality, with its open-source release further amplifying its impact.
The paper "Lumina-mGPT" introduces a multimodal autoregressive model designed for flexible, photorealistic image generation from text, leveraging a decoder-only transformer and multimodal Generative PreTraining (mGPT). Lumina-mGPT unifies text and image processing within a single framework, using Flexible Progressive Supervised Finetuning (FP-SFT) for high-resolution, flexible-aspect image synthesis, and Omnipotent Supervised Finetuning (Omni-SFT) for a wide range of vision-language tasks, including segmentation and multiview generation. The model’s novel unambiguous image representation enhances its ability to generate variable-resolution images, and extensive multimodal pretraining enables impressive zero-shot capabilities across visual tasks.
The introduction and abstract emphasize the decoder-only architecture for text and image as a novel contribution, which is strange, given that several prior works—some even cited in this paper—have already employed similar architectures.

The effectiveness of Flexible Progressive Supervised Finetuning (FP-SFT) would benefit from more detailed analysis, especially around why certain parameter settings or resolution stages were chosen. Currently, the progressive finetuning approach appears somewhat arbitrary, and an ablation study exploring different finetuning configurations or showing specific benefits per stage would substantiate these choices.
Yes
No ethics review needed.
What exactly is different about Lumina-mGPT from other large VLMs? Is the training different? Is there some novelty in how the training is done?
The paper finetunes the Chamaleon model to add image generation and other multimodal tasks to the model. The original Chameleon model can also do this but the image generation model was not publicly released.
The paper also proposes a simple representation of the images to support various aspect ratios by adding information about height and width, as well as end-of-line tokens after each row of (latent) pixels.
The finetuning on multimodal tasks shows that the training approach also generalizes to tasks besides image generation.
The paper proposes to finetune an autoregressive decoder-only model (initialized from Meta's Chameleon model) for image generation based on multimodal inputs. For this the paper initializes their model from the Chameleon checkpoint and finetunes the model for image generation and also adds additional multimodal tasks (e.g.,image editing, dense predictions like segmentation maps, spatial conditions, VQA, etc) to the training. The resulting model can generate photorealistic images in various aspect ratios and can also perform the multimodal tasks it was finetuned on.
There is limited novelty in the paper. Most of the work seems to come from the original Chameleon checkpoint with the image training being a relatively straight-forward finetuning approach.

The finetuning with multimodal tasks is also a relatively straight forward extension and other works have already shown that AR vision models can handle many different tasks and also can perform in-context learning (e.g., Sequential Modeling Enables Scalable Learning for
Large Vision Models, CVPR 2024).

Overall it's not clear to me what exactly is novel and different from Chameleon or other large VLMs.
Yes
As this is an open-source large model for generation purposes, which may or may not be commercialized, I would suggest a further review of the potential legal and regulatory concerns.
No ethics review needed. Yes, Legal compliance (e.g., GDPR, copyright, terms of use)
Please see the questions from my review comments above. Overall, my preliminary rating is based on the pros and cons of the current work.
- The proposed model achieves very good image generation results on the T2I task in the AR paradigm, as stated as one of the main contributions.
- The paper presents a rather detailed discussion of the implementations and fine-tuning recipes, including failure cases, which is appreciated.
- This work is open-source, with the promise to release the model checkpoints and codes, thus, it may be of interest to the researchers in the open-source community on large models.
This paper introduces a large multimodal autoregressive model, Lumina-mGPT, with a particular focus on its text-to-image (T2I) generation capability after fine-tuning. The model leverages pre-trained token representations from Chameleon 7B and 30B (Meta) and incorporates two fine-tuning techniques—Flexible Progressive Supervised Fine-tuning (FP-SFT) and Omnipotent Supervised Fine-tuning (Omni-SFT)—to adapt the model for T2I tasks and multiple vision and language tasks.

Overall, I think this paper has both clear strengths and limitations, which I will detail in my comments below.
- While this may sound like a platitude, I must note that there is limited novelty in this work. As the authors acknowledge, a well-trained token representation for multimodal data is crucial in this setting, motivating their choice of Chameleon. The proposed fine-tuning techniques, however, follow commonly used methods and widely accepted design choices. Overall, the work reads very much like a technical report to me. However, I recognize that both intellectually novel approaches and engineering-heavy work have their place in the community. That being said, I am fine with either as long as the design choices are well-justified. 
- A more concerning issue is the claim of the “emergent zero-shot capabilities.” While zero-shot is indeed becoming something of a buzzword, I don’t believe the observed reconstruction ability within the VQ-VAE module qualifies as true “zero-shot.” This term should be defined and examined rigorously, with careful attention to the training and inference data at different stages, especially within complex frameworks that combine multiple large models.
- While I appreciate the discussion on failure cases, Figure 6 actually highlights that this work does not address the fundamental challenges in generative modeling—specifically, the limitations in distribution learning with limited data. In other words, much of the performance boost appears to come from improved pre-trained data representations and high-quality, sufficient data during fine-tuning, rather than advancements in the model’s actual ability to learn distributions.
@inproceedings{
anonymous2024luminamgpt,
title={Lumina-m{GPT}: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=RXeP5ajK2S},
note={under review}
}
We present Lumina-mGPT, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions. Unlike existing autoregressive image generation approaches, Lumina-mGPT employs a pretrained decoder-only transformer as a unified framework for modeling multimodal token sequences. Our key insight is that a simple decoder-only transformer with multimodal Generative PreTraining (mGPT), utilizing the next-token prediction objective on massive interleaved text-image sequences, can learn broad and general multimodal capabilities, thereby illuminating photorealistic text-to-image generation. Building on these pretrained models, we propose Flexible Progressive Supervised Finetuning (FP-SFT) on high-quality image-text pairs to fully unlock their potential for high-aesthetic image synthesis at any resolution while maintaining their general multimodal capabilities. Furthermore, we introduce Ominiponent Supervised Finetuning (Omni-SFT), transforming Lumina-mGPT into a foundation model that seamlessly achieves omnipotent task unification. The resulting model demonstrates versatile multimodal capabilities, including visual generation tasks like text-to-image/multiview generation and controllable generation, visual recognition tasks like segmentation and depth estimation, and vision-language tasks like multi-turn visual question answering. Additionally, we conduct extensive analysis to illustrate the inference-time mechanisms of Lumina-mGPT and its emergent zero-shot capabilities. We release all code and checkpoints, hoping to facilitate the progress toward building artificial general intelligence.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Autoregressive Image Generation Multi-modality LLM
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8868007045aed678ddaca9cb1ae513503e25345b.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission