Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks
Multimodal Large Language Models;Text-rich Image;Multi-image
Yes
No ethics review needed.
- What is the key innovation in the adaptive encoding? It seems that the hyperparameter to be adaptively chosen is M, so does that means it simply figure out a sub-image size that can fit in the sequence length of various MLLMs.

- How is such approach compared to the current token reduction approach? Is that fair comparison if baselines can only take part of the input tokens restricted by the sequence length when your approach can successfully feed all the tokens forward?
- The authors deliver a specified text-rich multi image instruction tuning dataset which is highly comprehensive with coverage of diverse domains (documents, slides, charts, tables, webpages).

- The paper presentation is clear and informative enough to demonstrate their key ideas.
This paper proposes LEOPARD, a vision language model specifically designed for handling text-rich multi-image tasks. The main contributions are: (1) A large instruction-tuning dataset of about 1M samples focused on text-rich multi-image scenarios, and (2) an adaptive high-resolution multi-image encoding module that dynamically allocates visual sequence length based on image characteristics. The paper addresses two key challenges: the scarcity of high-quality instruction datasets for text-rich multi-image tasks, and the difficulty in balancing image resolution with sequence length constraints. The authors evaluate their model across 13 benchmarks, demonstrating superior performance on text-rich multi-image tasks while maintaining competitive results on single-image and general vision-language tasks.

Overall, this could be a good contribution to the field, with some clarifications needed on the methodology and experiments. Given these clarifications in an author response, I would be willing to increase my score.
- The proposed methodology and model seems just a combination of some existing works like llava-interleave and pixel shuffling. The novelty is just not clear enough to me at least based on current manuscript. While the adaptive high-resolution encoding is presented as novel, the paper lacks clear theoretical justification for why it should perform better than fixed allocation strategies. 
The relationship to and improvements upon prior work need to be more explicitly discussed to establish the technical novelty beyond combining known methods.

- The experiments try to evaluate the contribution of  both the fine-tuning dataset and adaptive encoding method but the ablation studies presented are insufficient to validate the key claims. More comprehensive experiments are needed comparing different feature allocation strategies, various compression ratios, and alternative approaches to handling multiple high-resolution images. The contribution of the dataset versus the methodology is not clearly separated in the results.
Yes
No ethics review needed.
## Strength
1. The paper is well-written and easy to follow
2. The proposed method works well on the targeted tasks.

## Weakness.
My main concern is the limited contribution and novelty of this paper.
1. The paper contains two parts, the MLLM LEOPARD and its training data LEOPARD-INSTRUCT. However, 

    a) the MLLM is a simple combination of an off-the-shelf multi-image strategy and an off-the-shelf high-resolution strategy. I could hardly find any novel point about it.

    b) The training data is a combination of current datasets and newly collected data. I do not find any unique design in the data collection process. I believe it would be helpful for the research community, but it is not a sufficient contribution to be a top conference paper.

2. The experiment results are not solid. There are many new MLLMs that could handle both multi-images and high-resolution images. For example, the NVLM, InternVL-2, etc. The methods compared in the paper are somewhat out of date.
Please refer to Questions
In this paper, the authors propose LEOPARD, an MLLM specifically designed for text-rich, multi-image tasks. To realize this, the authors collected large-scale training data LEOPARD-INSTRUCT.
Please refer to Questions
Yes
No ethics review needed.
1. Seems there's no quality control strategies in the GPT-4o based data generation. In L226, the authors said that the error rate of GPT-4o generated QA is less than 10%. My questions include: 
1.1 How do those errors look like? Hallucination?
1.2. How will those errors affect the model performance? 
1.3. How to design some approaches to reduce such errors? 
2. Recently, Qwen2-VL has proposed to process visual inputs under dynamic visual resolution (no patching). It would be better if the comparison between adaptive high-res encoding (proposed in this paper) and Qwen2-VL visual encoding strategy can be presented.
3. Some methods are missing in comparisons, such as Qwen2-VL, InternLM-XComposer2.5, InternVL2.
This paper has two major contributions:
1. LEOPARD-INSTRUCT: The paper creates a large-scale, high-quality multimodal instruction-tuning dataset specifically curated for text-rich, multi-image scenarios. This dataset addresses the scarcity of such resources and provides a robust foundation for training and improving MLLMs on complex, real-world tasks involving multiple interconnected images.
2. Adaptive High-Resolution Multi-Image Encoding: The paper develops an adaptive high-resolution multi-image encoding module that optimizes the allocation of visual sequence length based on input images' aspect ratios and resolutions. It allows LEOPARD to handle high-resolution images without compromising detail or clarity.
The paper introduces LEOPARD, an MLLM designed for text-rich multi-image tasks. 
It addresses two key challenges: the lack of high-quality datasets for tuning models on such tasks and the difficulty in balancing image resolution with sequence length limitations. 
The authors create LEOPARD-INSTRUCT, with ~1M data instances, tailored for text-rich, multi-image scenarios, and an adaptive high-resolution multi-image encoding module to optimize visual sequence length. 
Experiments show LEOPARD outperforms existing MLLMs on text-rich multi-image benchmarks and maintains competitiveness on single-image tasks and general domain evaluations.
The paper's weaknesses include:
1. Lack of detailed quality control for data generated by GPT-4o, which could impact model performance
2. Limited comparison with the latest visual encoding strategies like Qwen2-VL's dynamic resolution approach.
@inproceedings{
anonymous2024leopard,
title={Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=oSJqRF0Tkg},
note={under review}
}
Text-rich images, where text serves as the central visual element guiding the overall understanding, are prevalent in real-world applications, such as presentation slides, scanned documents, and webpage snapshots. Tasks involving multiple text-rich images are especially challenging, as they require not only understanding the content of individual images but reasoning about inter-relationships and logical flows across multiple visual inputs.
Despite the importance of these scenarios, current multimodal large language models (MLLMs) struggle to handle such tasks due to two key challenges: (1) the scarcity of high-quality instruction tuning datasets for text-rich multi-image scenarios, and (2) the difficulty in balancing image resolution with visual feature sequence length. Low-resolution encoding impairs the recognition of embedded text, while high-resolution encoding quickly exceeds the model’s maximum sequence length under multi-image settings.
To address these challenges, we propose Leopard, a MLLM designed specifically for handling vision-language tasks involving multiple text-rich images. 
First, we curated about one million high-quality multimodal instruction-tuning data, tailored to text-rich, multi-image scenarios.
Second, we developed an adaptive high-resolution multi-image encoding module to dynamically optimize the allocation of visual sequence length based on the original aspect ratios
and resolutions of the input images.
Experiments across a wide range of benchmarks demonstrate our model's superior capabilities in text-rich, multi-image evaluations and competitive performance in general domain evaluations.
We are committed to open-source models and will release all collected data, code, and checkpoints to the community.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal Large Language Models Text-rich Image Multi-image
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/93a5e17e3c8dfd981c0d6bb3a85394b1f7e91485.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission