Beyond Fixed Resolution: Enhancing VLLMs with Adaptive Input Scaling
Visual Large Language Model
Yes
No ethics review needed.
Does discrete image representation in VLLMs also suffer from the same image resolution problems?
This paper presents an interesting investigation in the choice of image resolution for different vision-language tasks with VLLMs. It reveals that image resolution would influence the performance for different downstream tasks. To solve the resolution variants problem, the authors futher propose a parameter-efficient fine-tuning techinque to tailor pretrained VLLMs to different image resolution. The experiments are extensive and well-organized.
This paper presents an investigation of the optimal image resolution for different vision-language tasks. It then propose a parameter-efficient fine-tuning technique to extend pretrained VLLMs to the target resolution.
Investigation on only LLaVA models might impair the generalization of this analysis.
Yes
No ethics review needed.
See the weakness.
This paper explores the resolution preferences of different vision-language tasks and formulates an empirical formula to determine a relatively appropriate resolution. It also proposes a new parameter-efficient fine-tuning technique to enhance the visual input resolution of pre-trained models.
This paper investigates the resolution preferences of different vision-language tasks and proposes an empirical formula to determine optimal resolution. It also presents a novel parameter-efficient fine-tuning technique to extend the visual input resolution of pre-trained models, which is validated by extensive experiments.
1. The author's starting point is good. However, the assumption that all existing VLLMs have a fixed resolution is invalid. Many existing VLLMs are of dynamic resolution, such as MiniCPM-V2 and Qwen2VL. The author seems to have directly ignored such methods and there is no discussion on them at all.

2. Regarding task selection, I don't seem to see that the author has selected tasks that are highly dependent on resolution for statistical evaluation, such as the DocVQA dataset. From my experience, this is a task scenario that is highly dependent on resolution. Statistical data in this task scenario can provide some inspiration.
Yes
No ethics review needed.
1. Intuitively, the selection of optimal resolution should not only be related to image complexity and uncertainty variance but also to the specific QA. For example, asking whether a photo contains a panda versus asking about the number of bamboo in a panda's arms would require different levels of image detail, which is more crucial than image complexity itself. The paper lacks sufficient analysis on this aspect.
2. According to Eq.3 in the paper, the task-wise optimal resolution is determined by C(T) and V(T), where C(T) and V(T) are the means of all samples in the task. What are the statistical distributions of these two values in each task? If there is significant variance, would it affect the significance of the mean itself?
3. The article does not provide specific information about the training data. It would be helpful to include more details about the experiments in the experimental section. Conversely, the case study section seems a bit lengthy.
1. The overall logic is sound, and the experimental results demonstrate that the method proposed in the paper alleviates the shortcomings of fix resolution in downstream VQA tasks.
2. The article provides comprehensive experiments and analysis on the calculation and validation of optimal resolution.
Regarding the subpar performance of VLLM in downstream tasks when using fixed resolution for image processing, this work proposes a task-wise resolution selection method and adapts the model to optimal resolution on each task through post-training. By measuring image complexity and model uncertainty variance across resolutions and combining the two, an empirical formula for calculating optimal resolution based on the baseline resolution is obtained. In the resolution adaptation process, by adding LORA and training only certain key parameters, the performance of the proposed 7B model in downstream tasks surpasses all baselines and is comparable to the 13B model.
1. The research findings of the article are relatively superficial, lacking deeper exploration. The relationship between the task and its optimal resolution in this paper seems more like overfitting to the benchmark itself. It is more important to focus on the sample-wise resolution selection or to derive the relationship between optimal resolution and more abstract task categories, rather than specific benchmarks.
2. There is a lack of comparison with other dynamic resolution methods, such as the dynamic number of tiles used in InternVL[1].
3. There are too few captions for the figures and tables. To understand the details of the figures and tables, one needs to refer back to a specific section.  

[1] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
Yes
No ethics review needed.
See weakness.
The method lifts the performance of LLaVA. The proposed task-driven dynamic resolution is meaningful.
This paper proposes a "two-stage" adaptive resolution pipeline. The authors introduce an empirical formula for choosing an optimal resolution. Experiments show that this method is effective on LLAVA.
1. The pipeline is not end-to-end and looks unsightly. It seems like two VLMs cascaded.
2. The baseline, only LLaVA, is limited.
3. I feel it would be best for the authors to design the dynamic resolution of the task-driven mode in an end-to-end manner. The current approach is too heavy in image preprocessing
@inproceedings{
anonymous2024beyond,
title={Beyond Fixed Resolution: Enhancing {VLLM}s with Adaptive Input Scaling},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=gc70LAWjwe},
note={under review}
}
Real-world vision-language applications demand varying levels of perceptual granularity. However, most existing visual large language models (VLLMs), such as LLaVA, pre-assume a fixed resolution for downstream tasks, which leads to subpar performance. To address this problem, we first conduct a comprehensive and pioneering investigation into the resolution preferences of different vision-language tasks, revealing a correlation between resolution preferences with 1.image complexity, and 2.uncertainty variance of the VLLM at different image input resolutions. Building on this insight, we propose an empirical formula to determine the optimal resolution for a given vision-language task, accounting for these two factors as the zeroth-order and first-order terms in the Taylor expansion on a given image input. Second, based on rigorous experiments, we propose a novel parameter-efficient fine-tuning technique to extend the visual input resolution of pre-trained VLLMs to the identified optimal resolution. Extensive experiments on various vision-language tasks validate the effectiveness of our method.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Visual Large Language Model
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/84b4668c4f636e5ff5609d537b4254f2aef1a20c.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Beyond Fixed Resolution: Enhancing VLLMs with Adaptive Input Scaling
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission