Improving Visual Commonsense in Language Models via Multiple Image Generation
Language Models;Image generation;Machine learning
Yes
No ethics review needed.
As weakness
VLMIG’s late-fusion approach, which combines textual and visual data just before final prediction, allows it to perform well on both visual and textual commonsense tasks. This ensures that LLMs benefit from visual grounding while retaining their core language abilities. The model excels across multiple task types, including object commonsense tasks (e.g., color and shape recognition) and reading comprehension. This versatility makes it a well-rounded approach for both visual and language-based tasks. VLMIG consistently outperforms existing models, including VLMs and VaLMs, in visual commonsense benchmarks, showing that multiple image generation and aggregation is a promising direction for enhancing multimodal reasoning capabilities. The authors conduct thorough experiments, including ablation studies, to validate the effects of different components, such as multi-image generation and the late-fusion layer. This helps substantiate the robustness and adaptability of the proposed model.
The paper introduces a method called VLMIG, aimed at enhancing visual commonsense reasoning in large language models (LLMs) by incorporating multiple generated images into the inference process. The method is particularly innovative in its use of a late-fusion mechanism, allowing the language model to consider both text and visual information without compromising on text-based tasks. VLMIG integrates various image predictions generated by a pre-trained text-to-image model, weighting them to improve visual commonsense understanding. Experimental results demonstrate VLMIG’s effectiveness across several tasks, such as visual commonsense reasoning and traditional language benchmarks, outperforming current baselines and maintaining strong performance in non-visual tasks.
1. The method relies heavily on the quality of generated images. Poor text-to-image generation could potentially harm performance, but this potential failure mode isn't thoroughly discussed. The effectiveness of VLMIG depends on the quality and relevance of the generated images. If the text-to-image model generates incorrect or low-quality images, this could lead to suboptimal or incorrect predictions, as the ensemble approach relies on the integration of various visual predictions
2. The approach requires generating and processing multiple images during inference, which likely increases computational costs and latency significantly. Although high-quality images can be generated quickly, this approach still results in considerable computational and time costs, especially when k is large. This could limit the method's practicality in time-sensitive applications
3. The paper doesn't adequately address how the approach scales with larger language models or longer text inputs, particularly given the computational requirements of processing multiple images.
4. There isn't sufficient analysis of the contribution of each component (VTP, LFAL, multiple image generation) to the overall performance.
5. Only test on VLMIG with GPT-2 backbone for table 1 may be really far away from current VLM community.
Yes
No ethics review needed.
For the different vLMIG models in Table 1 & 2, how many layers are used in the fusion module?
The main strengths of the paper are as follows:

* The empirical studies are conducted on variety of models (BERT, GPT-2, Llama-8b, Gemma-2B) which is commendable and also shows the broader applicability of the method.

* The empirical improvements on the datasets presented in the paper are significant, especially at smaller scale models (BERT, GPT-2, Gemma-2B).

* The proposal method is simple and easy to integrate in any existing LLMs.
This paper proposes a new method for Visually-augmented Language Models (VaLM). The method proposes using fixed (frozen) pre-trained LLM and visual encoders and trainable visual projection and multimodal fusion layers. The pre-trained components never see inputs from the other modality. The fusion layer feeds into the output (softmax) layer. A second contribution of the paper is to use multiple generated images at inference with weighted-average vectors. The method shows empirical improvement in visual understanding tasks while showing neutral-positive behavior on pure text-understanding tasks.
The main weaknesses of the paper are:

* While the proposed method shows significant improvements in visual understanding tasks at small scale, the results at larger scale (namely Llama-8b in Table 2) show very marginal improvements over the baseline which suggests either (a) the late fusion network requires many more parameters (more # of transformer layers) as the base model becomes more capable, or (b) at larger base models, we need a more end-to-end approach where we feed concatenated text+image tokens at the input layer as opposed to the proposed late fusion approach.

* The visual understanding dataset used in Table 1 (object common sense) seems too simple to form a very informed opinion of the proposed method. The authors mention running this method on more complex datasets as future work but we may need to see some results during the discussion phase if possible.
Yes
No ethics review needed.
1. How many A100 hours are used for your training?

2. Why an out-of-date Wikitext-103 dataset is adopted as the training corpora, maybe ShreadPajama is another better choice? Or you adopt this dataset for the density of knowledge?
1. This work resolved a significant issue in current LLaVA-style MLLMs, in which the lightweight visual SFT always leads to significant catastrophic forgetting and performance degradation on text-only capabilities. VILA (Lin et al., 2023) resolved this via large-scale multimodal pre-training, while the efficient training pipeline for resolving this issue is not explored well. Concluding from the results in Table-2, LLaVA-Next also gets a 2.9 scores degradation compared with its LLM backbone vicuna-7b, while the vLMIG achieved a slightly +0.2 score improvement.
This work proposed vLMIG, a novel visually-augmented language modeling framework for visual commonsense reasoning.
1. This architecture design only slightly differs from Flamingo, in which Flamingo adopts the cross-attention and this work adopts joint self-attention following VaLM. An ablation study on adopting cross-attention layer design is also appreciated to investigate some potential in architectural design.

2. An ablation study on the number of LFALs can also strengthen the contribution of this work. Flamingo adds cross-attention to each layer, while vLMIG only adds one LFAL. If there is a scaling effect on the number of introduced LFALs, then the users can decide how many LFALs to add to achieve a great tradeoff between performance and efficiency.

3. A significant baseline of LLaVA-llama3-8b is missed. You can consider to try to eval the https://huggingface.co/lmms-lab/llama3-llava-next-8b model and add the results to Table 2.

4. The efficiency is still another concern due to the image generation at inference stage in Table 3. Maybe retrieval is a better way? Or some explorations on efficient image generation can be also added to this work.
Yes
No ethics review needed.
1. Are some paramaters frozen during training? The performance of vLMIG based on Llama3-8B is even improved on QuAC while only training on Visual Genome which exhibit limited text diversity.
2. Maintaining text understanding capability during training VLMs is studied comprehensively, such as MM1 and Deepseek-VL. I am curious if these VLMs can directly give good performance on tasks requiring visual commonsense knowledge while keeping a nice performance on textual tasks.
1. Performance improvements  on object commonsense tasks are significant compared with baseline VaLMs.
2. The proposed method is simple to implement and ablation studies is comprehensive.
This paper propose a novel late-fusion method incorporating multiple images during training and inference to augment visual commonsense of LLMs. During inference, vLMIG leverage a Text2Image model to generate multiple images based on input text and use features of these image to enhance the KV in fusion layer upon the LLM.
1. The reproducibility of results are questionable since the papers fails to clearly introduce the exact setting of the experimental results. For example, Sec. 4.1 states VG and Laion-220K are both used while #371 says only VG is used. 
2. The comparison setting might be unfair. How to decide whether an incoming question needs image to augment or not? For questions that are irrelevant to visual knowledge, generated images can possibly harm the performance. Specifically, does the model generated images during evaluation on datasets like SQuAD.
We improve large language models' visual commonsense by generating multiple images from text prompts and integrating them into decision-making via late fusion, boosting performance on visual commonsense reasoning and NLP tasks.
@inproceedings{
anonymous2024improving,
title={Improving Visual Commonsense in Language Models via Multiple Image Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=QP3EvD1AVa},
note={under review}
}
Commonsense reasoning is fundamentally based on multimodal knowledge. However, large language models (LLMs), trained using textual data only, are limited with their ability to incorporate essential visual information. In contrast, Visual Language Models (VLMs), which excel at visually-oriented tasks, often fail at non-visual tasks such as textual commonsense reasoning. 
This divergence highlights a critical challenge - the integration of robust visual understanding with foundational text-based reasoning. To this end, we introduce a method aimed at enhancing LLMs' visual commonsense while maintaining textual modeling and commonsense reasoning performance. Specifically, our method generates multiple images based on the input text prompt and integrates these into the model's decision-making process by mixing their prediction probabilities. To facilitate multimodal grounded language modeling, we employ a late-fusion layer that combines the projected visual features with the output of a pre-trained LLM conditioned on text only. This late-fusion layer enables predictions based on comprehensive image-text knowledge as well as text only when required. We evaluate our approach using several visual commonsense reasoning tasks together with traditional NLP tasks, including common sense reasoning and reading comprehension. Our experimental results demonstrate significant superiority over existing baselines. When applied to recent state-of-the-art LLMs (e.g., Llama3), we observe improvements not only in visual commonsense but also in NLP benchmarks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Language Models Image generation Machine learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6227b636aa8abea5661de78aabd92ec7731f41d1.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e26177f955794dff697aef932f38c3df2aff8f2b.zip
Improving Visual Commonsense in Language Models via Multiple Image Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission