GridAgent: A 2D Grid-Based Game Framework And Benchmark For Multimodal Large Language Models
MLLM; benchmark; game
Yes
No ethics review needed.
Q1: GPT-4o performs worse than random and other, much smaller MLLMs on PU, PL, and SO tasks. Can you provide a discussion on why this is the case? Can you also check if other models which are larger than 7B also similarly perform worse on these tasks? 

Q2: How does the Level of difficulty affect LLMs performance? Could you add results across different difficult levels for those MLLMs? 

Q3: Do agentic workflows like ReAct and Reflexion improve the performance of MLLMs in these multi-turn tasks? 

Q4: Could you provide more details about how those specific abilities are connected to the Wechsler Intelligence Test?
S1: The authors propose a taxonomy for evaluating visual abilities of MLLMs and ground it in existing intelligence test (Wechsler Intelligence Test) setup which has been used to evaluate human cognition

S2: The task setups and scenarios are reproducible and demonstrate the shortcoming of current MLLMs on visual reasoning
The authors introduce GridAgent, a new benchmark for evaluating "execution", "perception reasoning", "memory", "learning" and "planning" abilities of Multimodal Large Language Models. The authors base this taxonomy on the Wechsler Intelligence Test. The benchmark includes 15 different tasks setups targeting one or more of the abilities.
W1: This work provides limited insight compared to previous works. [1] Provides a more in-depth analysis of memory capabilities of MLLMs. Embodied and Computer control benchmarks like [2], [3], [4] provide more real-world insight into the planning and execution abilities of MLLMs in visual setups. [5] provides a detailed analysis of perception reasoning abilities of MLLMs including analysis of Visual, Text and Joint reasoning abilities and failure modes. 

W2: Reasoning strategies like Chain-of-Thought [6], Self-consistency [7] are necessary for invoking reasoning abilities of LLMs. Similarly multi-turn tasks (like the ones introduced in this environment) are more suited to agentic formulation of LLMs like ReAct [8], and Reflexion [9]. The results sections lacks these details (did you use COT?) and evaluations of agentic frameworks. 

W3: Human evaluation settings are unclear and not well-motivated. Firstly, “Five players (including two authors) participated in the games through the GridAgent interface, with each completing five rounds across all tasks. " It is not well motivated to have authors who are intimately familiar with the task setups to serve as testers. Secondly, "The results show that any adult, once familiar with the rules and putting in serious effort, is fully capable of completing all the games." Where are these results? Lastly, "Consequently, we set the human baseline to 1 for all tasks.” This implies that the human baselines were set to 1.0 without actually completing the same set of tasks that the MLLMs solved. These facts need to be clarified and more details about the human evaluation need to be mentioned. 

W4: Lack of Variability studies: Does the performance of MLLMs across multiple trials remain consistent? Authors could evaluate this on a subset of tasks. 

W5: Lack of studies about scaling with model sizes: Does the performance of MLLMs on these tasks improve with more model parameters? All considered models are 7B parameters or less providing no information about whether larger model sizes correlated with better performance. 

[1] Wang, H., Shi, H., Tan, S., Qin, W., Wang, W., Zhang, T., Nambi, A., Ganu, T., & Wang, H. (2024). Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models. arXiv preprint arXiv:2406.11230. https://arxiv.org/abs/2406.11230

[2] Gan, C., Zhou, S., Schwartz, J., Alter, S., Bhandwaldar, A., Gutfreund, D., Yamins, D. L. K., DiCarlo, J. J., McDermott, J., Torralba, A., & Tenenbaum, J. B. (2021). The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI. arXiv preprint arXiv:2103.14025. https://arxiv.org/abs/2103.14025

[3] Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., & Yu, T. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv preprint arXiv:2404.07972.

[4] Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P. Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024). VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. arXiv preprint arXiv:2401.13649.

[5] Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., & Zhou, T. (2024). HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. arXiv preprint arXiv:2310.14566. https://arxiv.org/abs/2310.14566

[6] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint arXiv:2201.11903. https://arxiv.org/abs/2201.11903

[7] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv preprint arXiv:2203.11171. https://arxiv.org/abs/2203.11171

[8] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv preprint arXiv:2210.03629. https://arxiv.org/abs/2210.03629

[9] Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv preprint arXiv:2303.11366. https://arxiv.org/abs/2303.11366
Yes
No ethics review needed.
I’d also recommend authors to add evaluation examples where some of these models succeeded in supplementary. Examples from the best model should be good to add

I believe the benchmark is interesting and valuable to the community if authors address my concerns I'd be happy to increase my rating.
1. The paper is well written and easy to follow
2. The tasks in the benchmark are thought through and carefully designed to test the five key capabilities the paper is focused on. I also like how authors created a set of tasks that require composing these capabilities to solve some of the tasks.
3. The experiments section covers most popular state-of-the-art MLLMs apart from Gemini. The zero-shot evaluation results on the tasks clearly demonstrate that these models are lacking zero-shot generalization ability to solve the proposed tasks.
The paper presents GridAgent, a benchmark inspired by Wechsler Intelligence Test to  evaluate five MLLM capabilities: execution, perception reasoning, memory, learning, and planning. To do so, the paper proposes 12 games in 2D grid based environment which are diverse semantic environments, with randomized layouts and varying difficulty to test generalization and robustness of MLLM models. Proposed tasks evaluate multimodal reasoning and understanding abilities of existing MLLMs. Finally, the paper presents quantitative comparison of state-of-the-art proprietary and open-source MLLMs on the GridAgent benchmark and find that on majority of these tasks current MLLMs achieve performance closer to random chance performance. Additionally, the paper presents some preliminary analysis that suggests current MLLMs lack training for image-only perception tasks and it is difficult for these models to solve tasks that require contradictory knowledge from what is learned by these models.
1. For completeness I’d recommend authors to add results for latest Gemini model in comparison as well.
2. The results demonstrated in the paper are under zero-shot setting. I’d be curious to see how well all of these models ( or a subset of these  for which it is possible given constraints on context length) perform on these tasks with 1-2 in-context examples. We have seen remarkable results with In-Context learning for these long-context models and it’d be good to validate if these tasks can be solved with in-context examples. I believe it should help for SO, FI, and PL tasks. Adding these will also make paper stronger and help highlight the difficulty of tasks in the benchmark
3. Some details about the resolution of image input and how it is fed to each of these models is missing in the paper and supplementary. My understanding is - for all tasks where agent needed to take multiple actions to complete the task the input was fed to the model step by step. Is that the case? If yes, I’d recommend authors to add more details about how the evaluation was done, the resolution of image observations used, etc to the paper.
4. Did the authors ever run experiments with higher resolution image inputs for some of the tasks like Filing? I’d imagine these models should perform better on the task if input image is much higher resolution. If not, I think it would be a good experiment to add to the paper. This would clearly highlight the lack of perception reasoning ability if the input resolution is not a bottleneck.
Yes
No ethics review needed.
- The explanation of certain tasks, such as Perception Reasoning, lacks clarity regarding how performance metrics are derived and how these align with human cognition benchmarks.
- Section 5 exhibits some structural confusion, indicating significant issues in the writing structure of the Methodology section. The structure could be better organized to flow logically into the methodology and experimental analysis.
- The current comparative approach between MLLMs and humans is somewhat simplistic; exploring alternative comparison methods could provide deeper insights.
- The benchmark is well-structured, effectively targeting key competencies in MLLMs and providing a robust evaluation framework.
- The introduction of randomized game layouts enhances the test's generalization and robustness by minimizing overfitting to training data.
- The paper presents detailed empirical evaluations of multiple MLLMs, offering clear comparative insights.
- The use of diverse tasks ensures a comprehensive assessment of the models’ abilities, extending beyond simple task-solving to encompass more complex multimodal reasoning, while minimizing overlap with pre-training data, thus enhancing validity.
The paper introduces GridAgent, a 2D grid-based benchmark framework designed to evaluate Multimodal Large Language Models (MLLMs) across five core capabilities: execution, perception reasoning, memory, learning, and planning. This is accomplished through twelve distinct game tasks. The results indicate that while models like GPT-4o perform well in specific tasks, none achieve human-level performance across all categories.

The article currently still has some obvious issues and writing structure that need to be addressed. Nonetheless, this contribution is significant, and with further clarification during the rebuttal phase, it could be accepted for the conference.
- There is a limited discussion regarding the selection of specific game tasks and their comprehensive coverage of the intended capabilities.
- For some tasks, such as Puzzle and Sorting, model performance results close to random, raising questions about the alignment of task complexity with the current capabilities of models.
- Although the paper mentions the randomness in game layouts to avoid overfitting, it lacks sufficient analysis on how varying levels of task complexity affect model performance or whether simpler versions of the benchmark could serve as a more effective baseline. This introduces potential biases due to the artificial nature of the tasks.
- The comparison between MLLMs and human benchmarks appears overly simplistic, lacking a detailed exploration of potential improvements to models in order to bridge the performance gap.
@inproceedings{
anonymous2024gridagent,
title={GridAgent: A 2D Grid-Based Game Framework And Benchmark For Multimodal Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=jpypMKAsO6},
note={under review}
}
Multimodal Large Language Models (MLLMs) integrate the linguistic capabilities of LLMs with the ability to process multimodal data, enabling them to address a wider array of tasks. However, a comprehensive and standardized benchmark for evaluating MLLMs' complex visual reasoning performance in multimodal tasks has yet to be established. We introduce GridAgent, a versatile 2D grid-based framework that serves as a benchmark for assessing five essential capabilities of MLLMs: execution, perception reasoning, memory, learning, and planning. The framework includes twelve unique game tasks specifically designed to avoid overlap with the model's pre-training corpus. Each task targets at least one core competency and is enriched with diverse semantic information. Additionally, the game layouts are randomly generated, ensuring a more rigorous and authentic assessment of the MLLMs' capabilities. Experimental results indicate that although certain MLLMs excel in specific capabilities, none exhibit a comprehensive skill set comparable to the human baseline. Our work can be seen at: https://iclr2025gridagent.github.io/GridAgent-website.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
MLLM; benchmark; game
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9c24cac797c4dc8eb4e3fdac0675702d5c6a8df9.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/b1beee4af32a2a1da58d655e30a79853e2119401.pdf
GridAgent: A 2D Grid-Based Game Framework And Benchmark For Multimodal Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission