Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark
Large Multimodal Models;Visual Question Answering
Yes
No ethics review needed.
Please see the weakeness. In addition,
* How many templates were used to generate questions?
* What advantages does the VHs benchmark offer compared to recent multi-image QA benchmarks?
* Novel Multi-Image QA Benchmark: The authors introduce an interesting multi-image QA benchmark, Visual Haystacks, designed around a vision-centric "needle-in-a-haystack" scenario, providing a fresh and challenging setting for the LMM evaluation.

* Comprehensive Model Evaluation:  The paper conducts a thorough evaluation of LMMs on the VHs benchmark, uncovering important insights into current models, such as vulnerability to visual distractors, challenges with multi-image understanding, and tendencies toward positional visual bias.

* Novel Visual RAG Framework: The authors introduce a novel visual RAG framework that combines a compressor and a retriever. The compressor efficiently processes up to 10,000 images on a single 40GB A100 GPU, while the retriever identifies the top-k most relevant images for a given question, enhancing the framework’s scalability and efficiency.
The authors presents Visual Haystacks (VHs), a new vision centric benchmark designed to assess the performance of Large Multimodal Models (LMMs) in the multi-image question answering (QA) task. In addition, the author proposes a new visaul-RAG framework, MIRAGE, to enhance the task performance.
* Limited Object Diversity: The authors constructed the VHs benchmark using objects from the COCO dataset, which contains only 80 object categories. This limited selection may restrict the diversity and comprehensiveness of the benchmark, potentially affecting its ability to evaluate models across a broader range of visual scenarios.

* Restricted Question Diversity: The authors appear to rely on a few simple templates to generate questions, which may restrict the variety of question types in the benchmark.

* More like Object Detection than QA Reasoning: Many questions in the benchmark (e.g., "For the image with a truck, is there a dog?") seem to primarily assess the model’s object detection abilities rather than its visual QA reasoning skills. It is questionable if the benchmark requires the advanced visual QA reasoning skills from the models.

* Missing Related Work: The paper does not reference several recent multi-image QA benchmarks, for example: 
 1. CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs
 2. MANTIS: Interleaved Multi-Image Instruction Tuning
 3. MUIRBENCH: A Comprehensive Benchmark for Robust Multi-Image Understanding. 

Additionally, a similar multi-image retrieval approach was introduced in "ColPali: Efficient Document Retrieval with Vision Language Models", but this work was also not cited.
Yes
Is it possible for the author to release their code under the MIT license, considering it is derived from the Apache 2.0-licensed LLaVA codebase? Could the author elaborate this point?
No ethics review needed.
- Could you please address the points raised in the above weakness?
- Could you please add some randomly sampled failure cases made by GPT or Gemini? Sometimes failure cases can tell more than good cases. 
- Could you please address the ethics concerns around the code license?
- I generally feel the direction is important to our community where design meaningful Visual Haystack benchmark for evaluating VLM. 
- Some interesting points are discovered when evaluating models on the proposed benchmark. Since random guess could achieve 50% accuracy in the proposed benchmark, some open-sourced VLMs performance significantly drop even the Haystack size is very small. However, those models maintain high scores in some public evaluation-datasets. 
- Some detailed experiments are conducted such as needle position and running time. 
- The proposed benchmark are made publicly available under MIT license, which is good for community.
This paper addresses the limitations of Large Multimodal Models (LMMs) in multi-image question answering, where handling large visual contexts does not ensure effective retrieval and reasoning across images. Current benchmarks reveal biases and challenges in MIQA, such as poor cross-image reasoning and sensitivity to information placement. To overcome these, the authors propose "Visual Haystacks (VHs)," a vision-centric benchmark that tests retrieval and reasoning over multiple images, highlighting models' struggles with visual distractors and multi-image reasoning. They also introduce MIRAGE, an open-source Multi-Image Retrieval Augmented Generation framework capable of handling up to 10,000 images on a single GPU, achieving significant improvements over existing models and setting new standards in MIQA benchmarks like RetVQA. Key contributions include VHs, systematic LMM evaluation, and MIRAGE's scalable MIQA capabilities.
- Benchmark construction is still mainly centered around recognition tasks, based on benchmark design principles listed in Line129~138. Basically, it requires a strong recognition among all the input images, rather than true visual reasoning. 
- Based on the Figure 2 and 3, certain models, such as Gemini, GPT and the proposed MIRAGE, consistently perform better on the proposed multi-needle challenges compared to single-needle tasks. However, the multi-needle challenges are intentionally designed to be more difficult, as they demand additional reasoning across multiple images. Does this mean failure in designing the benchmark?
- Since the benchmark is constructed in way of examining recognition, therefore the proposed method contain ad-hoc modules, such as "a retriever module then calculates relevance scores, ensuring that only the most relevant images are passed to the LLM for final reasoning." Does this design hold for general visual reasoning tasks? For example, many of the tested single image dataset used in this paper, do not need this retriever module at all. 
- The proposed framework achieved not-very-good performance on some of the tested datasets. Also, there are many datasets that not being tested such as SEED, MME, and CHAIR.
Yes
No ethics review needed.
1. I'm confused about the difference between the MIRAGE model in Table 1 and the Q-former Model in Table. Doesn't MIRAGE utilize Q-former. 
2. See above.
1. This paper introduced a new visual needle in a haystack benchmark which composed of 1k yes/no questions. 
2. Evaluated on both open-source and close-source models and gained three insightful findings. 
3. Introduced a new baseline called MIRAGE for better handling of visual haystack tasks.
This paper introduces a long context, visual needle in a haystack benchmark which composed of 1k yes/no questions changeling the model to reasoning and find the target object in the images. It evaluated on both open-source and closed-source LMMs and reveal several critical findings such as susceptibility to visual distractors, difficulty in multi-image reasoning, and a bias in image positioning. It introduces a new baseline called MIRAGE (Multi-Image Retrieval Augmented Generation) for better handling of VH tasks.
1. The questions are only limited to yes/no questions. 
2. The question template are very limited, seems only three. 
3. MIRAGE has a significant performance drop in 4 out of 7 general VQA tasks. 
4. The approach of MIRAGE, deselecting unrelated (distracting) images somehow circumvents the VH challenge, as the this challenge lies in how model can reasoning in long context.  
5. The task of finding a target object seems still not simulating a real world scenario of long context visual reasoning task.
We introduce "Visual Haystacks," a vision-centric NIAH benchmark to evaluate LMMs, show that current models struggle with this task, and propose MIRAGE, a framework that improves performance by integrating retrieval with the VQA process.
@inproceedings{
anonymous2024visual,
title={Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=9JCNPFL1f9},
note={under review}
}
Large Multimodal Models (LMMs) have made significant strides in visual question-answering for single images. Recent advancements like long-context LMMs have allowed them to ingest larger, or even multiple, images. However, the ability to process a large number of visual tokens does not guarantee effective retrieval and reasoning for multi-image question answering (MIQA), especially in real-world applications like photo album searches or satellite imagery analysis. In this work, we first assess the limitations of current benchmarks for long-context LMMs. We address these limitations by introducing a new vision-centric, long-context benchmark, "Visual Haystacks (VHs)". We comprehensively evaluate both open-source and proprietary models on VHs, and demonstrate that these models struggle when reasoning across potentially unrelated images, perform poorly on cross-image reasoning, as well as exhibit biases based on the placement of key information within the context window. Towards a solution, we introduce MIRAGE (Multi-Image Retrieval Augmented Generation), an open-source, lightweight visual-RAG framework that processes up to 10k images on a single 40G A100 GPU—far surpassing the 1k-image limit of contemporary models. MIRAGE demonstrates up to 13% performance improvement over existing open-source LMMs on VHs, sets a new state-of-the-art on the RetVQA multi-image QA benchmark, and achieves competitive performance on single-image QA with state-of-the-art LMMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Multimodal Models Visual Question Answering
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/017250e0038c1f1f4fa14ed6c9efcfe9fa573e31.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission