Benchmarking Visual Cognition of Multimodal LLMs via Matrix Reasoning
Visual Cognition;Matrix Reasoning;Psychometrics;Visual Reasoning;Multimodal LLMs
Yes
No ethics review needed.
1. What are the insights from the paper? Why are we interested in testing matrix reasoning of VLMs and why do we want VLMs to succeed at this task? Besides observing that VLMs aren't good at this task, what should VLM researchers and developers should take away from this work?
1. Paper is well structured and experiments are comprehensive in terms of the number of models evaluated.
2. MLLM evaluation is an important challenge and the paper seeks to address that question with a connection to human psychometric evaluation.
This paper studies the high-level visual cognition abilities that require multi-image reasoning, by investigating matrix reasoning. A new benchmark is proposed:  MARS-VQA which contains 1440 instances and VCogBench to evaluate matrix reasoning abilities.  The paper finds that similar to previous findings with RAVEN's matrices and other similar tests, state of the art MLLMs struggle at matrix reasoning and perform worse or slightly better than random (25%) performance on a four-way classification task.
1. The claim that matrix reasoning has been "proven to be used to test human intelligence" or that "matrix reasoning is an important reflection of many fundamental capabilities of human intelligence" are to say the least, as controversial as saying "IQ Tests" are a true reflection of human intelligence.
2. The benchmark is restricted to shapes but could have potentially also used natural images. In my opinion, making claims about human visual cognition where the test data is purely symbolic is an overclaim. It could be an evaluation of human symbolic cognition.
3. The experiments include prompts designed for this task -- the influence of this choice of prompts on the performance is unclear.
Yes
No ethics review needed.
Due to the incremental contributions of this paper, I tend to be borderline negative in the current stage. Please refer to the weakness section for detailed comments.
1. The paper is clear and easy to follow.

2. The evaluations are performed on various MLLMs of both APIs and open-source models.

3. It reveals the current MLLMs still need to improve on matrix reasoning.
This paper introduces a new dataset MaRs-VQA and a new benchmark VCog-Bench to evaluate the zero-shot visual cognition capability of MLLMs with a matrix reasoning task. It requires MLLMs to discern relationships among patterns in a set of images and extrapolate to predict subsequent patterns. This paper proposes two evaluation pipelines of the proposed VCog-Bench: (1) Multi-image reasoning via CoT and (2) direct image input and text output. The evaluations are performed on various MLLMs of both APIs and open-source models.
1. The contribution of this paper is incremental. 
- The proposed MaRs-VQA and VCog-Bench are all sourced from existing well-built datasets, including MaRs-IB, RAVEN, and CVR.
The proposed multi-image reasoning via CoT method is an application of CoT to a particular task. It is not a general solution for other tasks. 
- The conclusion of the limitation of current MLLMs is not supported by sufficient evidence and is not convincing. How the author attains the conclusion that current MLLMs have Limited Use of Visual Information and Restricted Visual Working Memory needs to be clarified. 
2. The experiments show that the MLLMs perform much worse than humans. It is unknown if it is because the MLLMs do not understand the task to perform. The author may evaluate MLLMs with in-context learning, which can take one or two QA paris as examples. 
3. Additionally, it will be interesting to discuss if the MLLMs can easily attain the ability to solve matrix reasoning via training in a small number of cases. For example, the model can be trained on a small subset of MaRs-VQA and evaluated on VCog-Bench.
Yes
No ethics review needed.
Please refer to the weakness part.
1. The proposed MaRs-VQA dataset and VCog-Bench benchmark help establish the cognitative training and evaluation pipeline for multi-modal large language model.

2. The evaluations experiments are comprehensive and include extenstive MLLMs.

3. Some of the insights found in experiments can inspire more future investigations.
This paper introduces MaRs-VQA as a new matrix reasoning VQA dataset, and VCog-Bench as a visual cognition benchmark to evaluate the matrix reasoning performance of 16 existing MLLMs in the zero-shot setting. The thorough experiments qualitatively reveal the visual cognition gap between MLLMs and humans in matrix reasoning problems.
1. The motivation is unclear. It is not clear how the proposed MaRs-VQA differs from the privious ones. From Table 1, it seems the most remarkable difference is the introduction of RGB image.

2. The authors claim that "This setting makes current matrix reasoning assessment an ill-posed problem because such tests accurately reflect reasoning capability only when subjects engage without prior training, i.e., in zero-shot inference settings." It is quite confusing since the training-testing paradigm is the common methodology. As long as the training and test sets do not overlap, why does this pattern not make sense?

3. In multi-image reasoning evaluation of Table 2, only GPT and Claude evaluated. Why the open-source models not included?

4. In Table 4, the authors claim that "The difficulty level is based on the complexity of color, size, geometry, positional relationships, and object counting." However, it is not clear how the difficulty level is related to the mentioned elements. More details should be presented.
Yes
There do not appear to be any significant ethical concerns with this work.
No ethics review needed.
- Could the authors explore how changes in the visual complexity of MaRs-VQA impact model performance? Are models sensitive to subtle changes, such as color gradient variations or object overlaps?
- The paper showed zero-shot performance as an evaluation criterion. Could few-shot learning approaches be explored as a transitional step to understand whether MLLMs can improve their performance incrementally?
- How might MLLMs be modified to better retain visual information during the reasoning process? Want to hear the authors' opinion.
- The paper focuses on a challenging area of visual cognition relevant for assessing human-like intelligence. Unlike many studies that emphasize perception-based tasks (e.g., object detection), this work addresses higher-level reasoning and working memory. The MaRs-VQA dataset, based on established psychological tests, offers a diverse and validated dataset for visual cognition.
- The work highlights a gap between human and machine cognition in abstract visual reasoning, even for advanced models like GPT-4o. By focusing on more sophisticated visual cognition, the paper encourages the development of models that can reason about images abstractly, not just recognize them. This has important implications for the progress of AGI.
- The paper is well-structured, with clear definitions of each dataset, model, and experimental setup. The comparison between human and model performance is effective in showcasing current limitations and areas for improvement.
This paper introduces a new dataset called MaRs-VQA and a benchmark named VCog-Bench to evaluate how well MLLMs understand and reason about visual information. It focuses on testing these models using matrix reasoning tasks, which are inspired by established intelligence tests like RPM and the WISC. These tasks require advanced visual reasoning, which is still challenging for current AI models. The paper compares how different MLLMs perform in these tasks compared to humans, showing a clear gap in capabilities. The contributions include making MaRs-VQA, the largest dataset designed by psychologists for matrix reasoning, and introducing VCog-Bench as a new standard for evaluating visual cognition. The experiments show the current limitations of MLLMs in dealing with abstract visual problems and provide suggestions for future improvements.
- The paper does not explore how changes in the visual complexity of MaRs-VQA impact model performance. It is unclear whether models are sensitive to subtle changes (e.g., color gradient variations or object overlaps), which could provide important insights into model robustness and areas for improvement.
- One of the weaknesses is the lack of detailed analysis into why specific models fail at particular reasoning tasks. However, considering that the primary goal of the paper is to propose a new dataset, this focus on benchmarking rather than in-depth analysis of is understandable.
We propose a new dataset MaRs-VQA and a new benchmark VCog-Bench to evaluate the zero-shot visual cognition capability of MLLMs and compare their performance.
@inproceedings{
anonymous2024benchmarking,
title={Benchmarking Visual Cognition of Multimodal {LLM}s via Matrix Reasoning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=QrhB9HcgnL},
note={under review}
}
Recently, Multimodal Large Language Models (MLLMs) and Vision Language Models (VLMs) have shown great promise in language-guided perceptual tasks such as recognition, segmentation, and object detection. However, their effectiveness in addressing visual cognition problems that require high-level multi-image reasoning and visual working memory is not well-established. One such challenge is matrix reasoning -- the cognitive ability to discern relationships among patterns in a set of images and extrapolate to predict subsequent patterns. This skill is crucial during the early neurodevelopmental stages of children and is proven to be used to test human intelligence. Inspired by the matrix reasoning tasks in Raven’s Progressive Matrices (RPM) and Wechsler Intelligence Scale for Children (WISC), we propose a new dataset MaRs-VQA and a new benchmark VCog-Bench to evaluate the zero-shot visual cognition capability of MLLMs and compare their performance with existing human intelligent investigation. Our comparative experiments with different open-source and closed-source MLLMs on the VCog-Bench revealed a gap between MLLMs and human intelligence, highlighting the visual cognitive limitations of current MLLMs. We believe that the public release of VCog-Bench, consisting of MaRs-VQA, and the inference pipeline will drive progress toward the next generation of MLLMs with human-like visual cognition abilities.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Visual Cognition Matrix Reasoning Psychometrics Visual Reasoning Multimodal LLMs
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/402e062ad2175a19a65f06f5b2510ba6d36c87d3.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Benchmarking Visual Cognition of Multimodal LLMs via Matrix Reasoning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission