AgentQuest: Benchmarking LLM and VLM Agents on Long-Horizon Interactive Tasks
LLM;VLM;Agents;Benchmark;RL
Yes
No ethics review needed.
1. How might the benchmark adapt to new multimodal architectures that integrate more advanced vision processing, such as video-based reasoning?

2. Could the “knowing-doing” gap (where models have knowledge but fail in practical application) be addressed by incorporating memory or reinforcement learning methods in future iterations of the benchmark?
1. AgentQuest integrates diverse environments, testing a wide range of agentic skills from navigation to complex reasoning, making it a well-rounded benchmark.

2. The benchmark’s emphasis on agentic capabilities addresses key limitations in current LLMs and VLMs, which are crucial for autonomous AI applications.

3. The benchmark’s metrics allow for detailed analysis of model performance, identifying specific areas like spatial reasoning and long-term planning where models struggle.
The paper presents AgentQuest, a benchmark to evaluate LLMs and VLMs on complex, long-horizon tasks. By consolidating multiple reinforcement learning environments (e.g., NetHack, TextWorld, BabyAI), AgentQuest provides a unified testbed for assessing skills like spatial reasoning, long-term planning, and exploration. Findings show that while LLMs perform adequately on simpler tasks, they struggle with complex ones, especially when handling visual inputs. AgentQuest highlights gaps between current model capabilities and human-level performance, encouraging further research to enhance agentic qualities in AI.
1. It is weird that VLMs underperform when visual information is included. It indicates that current architectures may not be well-suited for complex vision-based reasoning. Or the current prompt is not suitable for the model. I think this point deserves a careful study.

2. More long-context models should be included for evaluation, for example Qwen2, Llama-3-8B-Instruct-80K, LongAlpaca-7B, LongChat-v1.5-7B-32k.
Yes
No ethics review needed.
- Does the AgentQuest support multi-agent LLM decision-making?
- How does AgentQuest deal with LLM hallucination? For instance, the LLM/VLM returns an invalid action.
- Comprehensive experiments. This paper is well-supported by comprehensive experiments and a diverve selection of environments that cover a wide range of agentic challenges. 
- The proposed benchmark incorporates several typical environments and could facilitate rapid verification of LLM/VLM-based decision-making. Considering that existing LLM/VLM-based decision-making methods are evaluated by their own tasks or environments, such a unified benchmark will be beneficial and valuable to the whole community.
- The authors provide analysis into the experimental results on AgentQuest, including Spatial Reasoning, Long-term planning, Discovering and Leveraging Environment Dynamics, and Knowing-Doing Gap. The analysis provide insight into this problem, and could facilitate relevant research in this area.
- The writing is clear and easy to follow. The motivation is clearly presented.
- The code is provided to make the benchmark reproducible.
The paper introduces AgentQuest, a suite of six reinforcement learning environments for testing the agentic capabilities of long-context LLMs and VLMs. The benchmark includes diverse environments, ranging from simple tasks such as BabyAI, to complex tasks such as NetHack. The authors provide baseline evaluations of state-of-the-art LLMs/VLMs on AgentQuest using zero-shot prompting. The authors perform a qualitative analysis of the results across capabilities and identify an intriguing knowing-doing gap where the models cannot employ the knowledge they possess. Finally, the authors develop an open-source toolkit for AgentQuest.
- More related decision-making environments should be discussed and compared. For instance, the authors include the Crafter in the benchmark, however there are also Minecraft environments such as Minedojo[1] and MineRL[2]. There are also LLM-based multi-agent cooperation benchmarks such as Overcook[3]. 
- More visualizations of the environment, agent behavior, and agent trajectories can be provided so that the analysis can be more intuitive.
- Does the AgentQuest support multi-agent LLM decision-making?
- How does AgentQuest deal with LLM hallucination? For instance, the LLM/VLM returns an invalid action.
- More cases should be provided for the limitations of VLMs.


[1] Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., ... & Anandkumar, A. (2022). Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35, 18343-18362.

[2] Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., & Salakhutdinov, R. (2019). Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440. 

[3] Liu, J., Yu, C., Gao, J., Xie, Y., Liao, Q., Wu, Y., & Wang, Y. (2023). Llm-powered hierarchical language agent for real-time human-ai coordination. arXiv preprint arXiv:2312.15224.
Yes
No ethics review needed.
Creating a comprehensive benchmark to test model agent capabilities is very meaningful; however, the authors' work has certain limitations. It would be helpful if the authors could provide further explanation.
1. AgentQuest integrates various reinforcement learning environments, covering tasks from simple to highly complex, to comprehensively evaluate the agency capabilities of LLMs and VLMs.

2. Different metrics have been designed to provide quantitative analysis of model performance across multiple dimensions, including interactivity, spatial reasoning, and long-term planning.

3. According to the authors, AgentQuest is convenient for researchers to expand and apply, helping to advance the field of intelligent agents and improve model usability in complex, dynamic environments.
This paper introduces a new benchmark designed to assess the agent ability of both LLMs and VLMs, with a particular focus on their performance in complex environments. By incorporating diverse tasks and refined metrics, this benchmark highlights the strengths and limitations of current models, especially in visual decision-making tasks where shortcomings are evident. Evaluation results indicate that while current models perform well on simple tasks, they continue to face significant challenges with complex ones. AgentQuest provides a platform for advancing research in intelligent agents.
1. The main issue is that this paper, as a benchmark for evaluating the agent capabilities of models, only assesses base models. Many recent studies focus on enhancing models’ agent capabilities, but this paper does not evaluate these works. While base models are important, they weren’t specifically trained to improve agent abilities. This may be one reason why they perform poorly on many tasks. Evaluating the latest agent-focused methods would better reflect the field's progress.

2. As far as I know, this paper simply combines several existing benchmarks without further integration. While bringing existing benchmarks together to evaluate model performance may be important, it doesn’t qualify as innovative work. The authors need to provide further explanation on this point.
Yes
No ethics review needed.
1. What does the "reasoning" refer to in L72 "action = self.llm_cilent(reasoning)"?
1. The presentation of this benchmark is timely.
2. The paper presents problems about the knowing-doing gap, which might spark more research.
This paper presents a benchmark to evaluate LLMs' capabilities in video games. Six video games are included. One metric that captures how close the LLMs are to completing a task is proposed for evaluation. State-of-the-art LLMs are used for evaluation. The paper presents findings from quantitative and qualitative experiments.
My major concern is the comprehensiveness of the proposed benchmark. Current VLMs or MLLMs are usually trained on image-text pairs in the real world. The appearance of the involved video games might differ largely from the training data. It is not very surprising, at least to the reviewer, that providing game images leads to worse performance. More scenarios, especially general ones, should be involved to support the conclusion. This can also further reflect the overall capability of MLLMs.
@inproceedings{
anonymous2024agentquest,
title={AgentQuest: Benchmarking {LLM} and {VLM} Agents on Long-Horizon Interactive Tasks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=fp6t3F669F},
note={under review}
}
Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies—areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce AgentQuest, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). 
We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as models perform worse when visual representations of the environments are provided. We release AgentQuest as an open and user-friendly benchmark to facilitate future research and development in the agentic community.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM VLM Agents Benchmark RL
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e51c80001b7ca9e3bbd9b0e44f317e0227b8166d.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/99a6173cca1c70af08eba8a68d3580709f6664d3.zip
AgentQuest: Benchmarking LLM and VLM Agents on Long-Horizon Interactive Tasks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission