Rational Decision-Making Agent with Learning Internal Utility Judgment
Decision Making;Autonomous Agent;Large Lanugage Model;Elo Rating
Yes
No ethics review needed.
See weakness
The proposal of using Elo score is reasonable and points out an interesting direction of rethinking how evaluation metrics can guide the decision making of LLM agents.

Meanwhile, the experiments are good and extensive.
The paper studies how to improve the performance of LLM-agents on sequential decision making tasks. It discussed the drawbacks of relying on pre-defined external metrics to guide the decision making of LLM-agents. Furthermore, they propose to use Elo score together with Experience Exploration techniques.
1) the paper is rather empirical and heuristic, with combination of two heuristic techniques.
2) Elo itself also has significant issues [1]. For examples, it may provide no useful signals sometimes. In other words, such a black-box use of Elo without significant extensions/more detailed discussions on its potential limitations is a sign of limited novelty.

[1] Re-evaluating evaluations. NeurIPS 2018
Yes
No ethics review needed.
Since the paper lacks convergence analysis of this approach, I wonder how much additional computational overhead this method requires compared to existing approaches. The time cost of conducting multiple comparisons and updating Elo scores, especially for complex decision scenarios, could be substantial and needs to be evaluated against baseline methods.
The main strength of this paper are:
1. The paper proposes a novel idea by shifting away from the conventional reliance on external performance measures to an internal utility judgment approach for LLM-based agents. 
2. The framework provides a practical solution for developing internal rationality in LLM-based agents.
3. The experimental results demonstrate promising performance across multiple datasets,
This paper introduces a novel approach named RaDAgent that addresses the limitation of LLM-based agents' dependence on external performance measures for decision-making. Unlike existing methods that rely on potentially unreliable or unavailable external metrics, RaDAgent develops internal rationality through an iterative framework combining Experience Exploration and Utility Learning. Besides, the framework incorporates the Elo Rating system to learn and assign utilities to decision steps through pairwise comparisons, enabling agents to make autonomous decisions based on learned internal judgment.
The  main weakness of this paper are:
1. The Elo rating system might oversimplify the complexity of decision utilities. Besides, the pairwise comparison approach could be time-consuming for complex tasks.
2. The convergence for the Elo-based utility learning lacks theoretical analysis. While the paper demonstrates empirical effectiveness of the Elo rating system in learning decision utilities, it fails to provide rigorous theoretical guarantees about the convergence properties of this approach. An analysis of whether and how quickly the Elo scores converge to stable values would strengthen the theoretical foundation of the approach and provide better understanding of its reliability in different scenarios.
Yes
No ethics review needed.
Please refer to the "weakness" section.
Overall, the idea of this work is novel and intuitive.

Overall, this paper is well-written and well-presented.

The proposed RaDAgent method significantly outperforms the baseline methods across most of tasks.
This work proposes RaDAgent (Rational Decision-Making Agent) that utilizes experience exploration and utility learning. Notably, Elo scores are assigned to each step to evaluate its utilities, which helps the agent to derive the optimal decisions. The authors provide empirical evidence that validates the efficacy of the proposed method on several tasks such as Game of 24, WebShop, and ToolBench. They also show that LLMs and the decision-making approaches are complementary, mutually enhancing each other to achieve superior performance outcomes.
I would recommend putting the Related Work section after the Introduction so that readers can gain more background information of this work. Indeed, explaining how the literature or baselines work can help the readers better understand the motivation of authors' idea.

The authors need to enhance the language and fix typos/grammar issues, e.g., in line 96-97, "take action", line 139 "Elo score" should be plural, etc.

Line 97-98, in general, one would use different functions to represent the policy and transition functions in MDP. I would expect a better introduction of MDP on page 2. For example, one can use a tuple (S, A, R, P, \mu) to represent an MDP. Please see more details in RL paper or Sutton's RL book. In addition, \pi is often used to denote the policy (action generator).

Even though mentioning in the Limitation section by authors, I am still interested in if RaDAgent can be implemented in other (at least one state-of-the-art) LLMs (e.g., Claude, Llama), especially not the GPT-series. Also, why the authors choose GPT-series as their backbone LLM? If RaDAgent does not work for other LLMs like Claude, what could be the possible reasons?
@inproceedings{
anonymous2024rational,
title={Rational Decision-Making Agent with Learning Internal Utility Judgment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GEBkyKZOc4},
note={under review}
}
With remarkable advancements, large language models (LLMs) have attracted significant efforts to develop LLM-based agents capable of executing intricate multi-step decision-making tasks. Existing approaches predominantly build upon the external performance measure to guide the decision-making process but the reliance on the external performance measure as prior is problematic in real-world scenarios, where such prior may be unavailable, flawed, or even erroneous. For genuine autonomous decision-making for LLM-based agents, it is imperative to develop rationality from their posterior experiences to judge the utility of each decision independently. In this work, we propose RaDAgent (Rational Decision-Making Agent), which fosters the development of its rationality through an iterative framework involving Experience Exploration and Utility Learning. Within this framework, Elo-based Utility Learning is devised to assign Elo scores to individual decision steps to judge their utilities via pairwise comparisons. Consequently, these Elo scores guide the decision-making process to derive optimal outcomes. Experimental results on the Game of 24, WebShop, ToolBench and RestBench datasets demonstrate RaDAgent’s superiority over baselines, achieving about 7.8% improvement on average. Besides, RaDAgent also can reduce costs (ChatGPT API calls), highlighting its effectiveness and efficiency.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Decision Making Autonomous Agent Large Lanugage Model Elo Rating
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7989f78845a3a51e021f0babd36d99996c454629.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/744318746bc33b40367c9f8eebda7400a4d56e4a.zip
Rational Decision-Making Agent with Learning Internal Utility Judgment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission