DPM: Dual Preferences-based Multi-Agent Reinforcement Learning
multi-agent reinforcement learning;preference-based reinforcement learning;RLAIF;RLHF
Yes
No ethics review needed.
* The relative size of L^T versus L^A may not be the same as one is summing many more terms in (6). Is this addressed with some normalization?
* Is it fair to compare with sparse reward scenarios? I ask because essentially, the approach is building its own dense reward function via trajectory and agent preferences and so perhaps it should be compared from a dense perspective.
* The temporal component of training through an LLM is not discussed (somewhat discussed in the conclusion, but not explicitly). How does it compare the computation of a densely constructed rewards as one needs to evaluate a NN for each agents intrinsic reward. This may scale badly as the number of agents become large compared to human constructed reward functions although subsets may be used as mentioned.
* It would be interesting (although not necessary) to do an analysis comparing the reward generated via DPM vs the dense reward for the problem. Perhaps the DPM generated reward has some sense of interpretability.
*I strongly appreciate Figure 2.

*For feedback, they use an LLM for feedback of the RL agent. For completeness, the limitations of such a choice must be at least mentioned in the introduction, but I enjoyed the comparison with Human feedback in Appendix D.2 and preference Alignment.

*The paper structure overall is very clear and the idea is simple but intuitive.

*The ablation study is quite nice. I appreciated this as well.
In this work, the authors propose generating agent-based intrinsic reward functions in MARL whose functions are generated by combining a preference score based both on pairwise trajectory comparison as well as a comparison between agent decisions. The use an LLM approach to generate the preference comparisons and compare on both SMAC and SMACv2. The contribution is simple, but interesting and the paper explores many different comparisons which is well done.

The exposition and writing can be improved as well as small mistakes such as Figure 1. They compare against many baselines, but in a completely sparse reward setting where as the baselines were developed with a ”weakly” sparse reward which perhaps makes the comparisons unfair. Further, they have omitted some recent work in Sparse Reward MARL - see below. Additionally, only SMAC and SMACv2 are compared, but other MARL environments would be interesting to add - although, in this reviewer’s opinion are not necessary. Additionally, the time constraint of training through an LLM is only very briefly mentioned at the end and perhaps further analysis can be conducted on the interpretability of their learned reward functions compared to the dense human version.
* Figure 1b has no blue line or perhaps it is always zero of which it is useless and hard to see.
* Figure 1a is extremely confusing to follow.
* There is a real lack of related work section compared to what has been done in MARL on sparse versus dense reward functions. No details about the ideas the baselines have introduced in the sparse MARL space are discussed.
• What is Ψ in P_ψ, \hat{r}_ψ for example in equation (1) - I assume it is the learnable weights, but this should be stated.
* Probably should add that the loss is negative log-liklihood in Eqn. (2) as other approaches can be used
* In SMAC, it is very challenging to win with the completely sparse approach. Thus, I think a comparison with the ”less weak” sparse manner is more fair as I imagine the baselines will do better, but the results of DPM+QMIX will not change much. This ”less weak” sparse reward is very natural to construct of which a non-expert could reasonably create. Comparing for example with ”https://arxiv.org/pdf/2405.18110”, it is clear that for 5m v.s. 6m, the result is extremely similar in terms of performance if one uses their weakly sparse reward
* A comparison with the following sparse reward solution to SMAC in ICML 2023 should at least be mentioned or discussed. (”Lazy Agents: A New Perspective on Solving Sparse Reward Problem in Multi-agent Reinforcement Learning”)
* There are a few typos and the flow of the paper is incoherent at times. (End of Section 2.1 for example). Shortened List of typos:
– The title should be singular. "Dual Preference Based MARL"
– Section 2.1, second paragraph. Third sentence should be "in which subgoal-based..."
– Section 2.1, second paragraph. Fourth sentence should be "caused by"
– Section 2.2, 3rd, sentence "a scripted teacher which assigns"
– Fix the bibliography. Many things such as ”ai”, "Smacv2" etc. should be capitalized.
– Please use "LLM" as a singular vs plural correctly. For example in first pargarph of figure 3.1, one needs the article "an" in many places.
– Section 3.3., "common PbRL research"
Yes
No ethics review needed.
1. How much preference feedback is being used in Figure 4? Is the same amount of the two types of preference feedback being used?

2. In the preference alignment case study, it is unclear what is being aligned. Can the authors elaborate on what is being compared?
1. The results in SMAC and SMAC v2 show that adding DPM can significantly outperform other standard MARL baselines.  

2. I liked how the authors included additional generalization experiments, demonstrating that DPM can work well with other MARL algorithms, not just QMIX.
This work introduces a preference learning method for the multi-agent RL setting. In particular, the approach, DPM, leverages traditional trajectory comparison feedback along with agent comparison preference feedback, which ranks agents' actions.
1. Unclear what the contributions are / limited novelty:

This method's significant contribution seems to be its use of a new feedback methodology, where a teacher ranks specific actions. 

2. Experimental Design:

The authors did compare DPM against several MARL baselines but not against the other preference learning algorithm for multi-agent systems [1]. I think the authors need to compare against this algorithm or explain why this would not be a valid comparison.

The evaluation is limited to one environment domain, SMAC. Why not include other prominent MARL environment suites, such as Multi Particle Environments (MPE) or Multi-Agent Mujoco?

3. Use of LLM as preferences:

The authors use LLMs to obtain preferences, which is a design choice. However, the reasoning for this design choice is limited. The authors note that DPM uses preferences from LLM to address the issues associated with human preferences. Can the authors elaborate on these issues? In addition, the authors should describe the limitations of using an LLM in this context.  

We should also note that one primary objective in preference learning is to learn reward functions that encode human preferences (i.e., it is important that humans are actually providing the feedback). Can DPM work with human preferences? I’m a bit concerned that humans would have difficulty ranking specific actions in the agent comparison component of the approach. Especially in the continuous action setting, actions from one time step to the next are unlikely to change drastically, making it difficult for a human to detect differences. 


[1] https://ojs.aaai.org/index.php/AAAI/article/view/29666
Yes
No ethics review needed.
1. In the manuscripts, the author mentioned DPM adopts multiple reward models. How many distinct reward models does DPM employ? What are their specific roles (trajectory vs. agent preferences)? And how do these models differ in structure or function?
2. What is the required number of preference pairs per scenario? And how does the number of pairs affect performance?
1. The preference-based method appears to be novel in MARL settings and it is shown effective in the experiments.
2. The case study and ablation study section provides comprehensive insights.
This paper addresses the challenge of sparse rewards in multi-agent reinforcement learning (MARL) by introducing Dual Preferences-based Multi-Agent Reinforcement Learning (DPM). DPM is a dual preferences-based method that evaluates both complete trajectories and individual agent contributions. Experimental results on the StarCraft Multi-Agent Challenge (SMAC) and SMACv2 demonstrate that DPM significantly outperforms existing sparse-reward baselines.
1. My primary concern lies in evaluation.

    - DPM is mainly evaluated against exploration methods in MARL. This comparison lacks fairness since DPM receives additional environment knowledge through LLM prompt templates (including game objectives, cues on how to win the game, and state interpretation) while baseline algorithms (except densely rewarded QMIX) do not have access to such information.
    - The evaluation relies solely on StarCraft-based benchmarks, lacking diversity. Additional benchmarks with different characteristics would strengthen the evaluation.

2. The motivation for agent ranking requires better justification. The direct comparison of agent actions in partially observable or heterogeneous settings (where agents have different types and abilities) needs more explanation.
    
3. The use of multiple reward functions introduces additional training overhead. Additionally, since preference pairs are generated from the replay buffer throughout the training, the computational costs of LLM queries should be reported.

4. Minor comments:
    - Line 179, typo with actions $o_t$
Yes
No ethics review needed.
Please refer to my weaknesses part.
The paper is well written and most parts of the paper are easy to follow. The paper conducts a comprehensive set of experiments with ablation studies. The idea to access an agent's contribution to a team by utlising agents preference labelled by LLM is novel and the results demonstrate its effectiveness.
This paper proposes a preferences-based MARL algrorithm named DPM which aims at solving the sparse reward problems in MARL by utlising LLM to label trajectory preferences across trajectories and agent preferences within one time step. The preferences across agents help to identify agents' contribution within one step. The experimental results on SMAC and SMACv2 show that DPM outperform many strong baselines in sparse reward setting and achieve close performance to QMIX in dense reward setting.
Some essential details or results are missing, so I scored the paper's soundness as fair:

1. How long is a trajectory used in trajectory comparison. Is it a complete episode or part of the episode? If it is the former, how does it align with the extrinsic reward provided by the environment in sparse reward setting.

2.  Could you compare the DPM performance based on human feedback with LLM feedback on the same graph?

3. In line 344, you mention spaese-reward based algorithms perform poorly becasue of a harsher sparse setting, could you compare the sparse settings explicitly?

4. Could you analyse the reason that DPM outperforms QMIX dense in 10gen-terran? does it mean LLM can do better than extrinsic dense rewards?

5. Could you provide some actual examples of the LLM's prompt and cooresponding response on trajectories and agents comparison?

6. Do you think the results of Figure 8(d) contradicts with your motivation that we should assess the agent's individual contribution to the team for better performance? How can the trend that "feedback from only a subset of agents, especially with fewer agents, results in higher win rates compared to using feedback from all agents." support your motivation. Could you analyse the reason for the trend? Also, could you suggest a way to selecet the number of agents we should compare in large scale games and which agents should be compared? 

Some potential limitations are not discussed or addressed, so I scored the paper's contribution as fair:
7. How do you think the tradeoff between the cost (time and money) and the performance achieved by DPM? Given the same number of floating-point arithmetic calculations, will there be alternative apporach?
8. Does DPM rely on the prior knowledge of the games? In many sparse reward problems, even human experts may find it diffcult to compare agents' contribution within one step. 

I am willing to alter my socres if my questions are well adressed.
Yes
No ethics review needed.
See Weaknesses.
- This paper is well written and easy to follow.
- The idea of extending PbRL into multi-agent settings by comparing / ranking agents is inspiring, which can be considered as an implicit credit assignment.
- Experiment results show that DPM can well “reconstruct” the dense rewards in SMAC scenarios, thus achieving comparable performance with the baseline that learns from original dense rewards.
- This paper provides an example of transforming SMAC game information into texts, which can be beneficial for relevant future research.
This paper addresses the sparse reward problem in cooperative MARL utilizing PbRL techniques. Specifically, besides comparing trajectories like SARL settings, this paper compares / ranks agents in a state to better learn the reward models. It also uses LLMs for automated comparison to reduce the reliance on human input, and achieves good results on benchmarks SMAC and SMACv2.
- The reward calculation for running QMIX is not clearly illustrated. According to Line 307, intrinsic rewards are $\hat{r}$ and $\hat{R}\_t$, which are not defined in previous texts. I assume that $\hat{r} = \hat{R}\_t = \sum\_{j \in N} \hat{r}_{\psi} (s_t, a_t^j, o_t^j)$ according to Line 229. However, this reward function does not take $a_t^{-j}$ as input, which does not align with MARL settings where the rewards for individual actions depend on teammates’ actions.
- Even if the individual rewards are optimal, the sum operation will lose their information. For example, the reward sum is the same when individual rewards are (1, 0, …, 0) and (0, 0, …, 1). Authors should develop techniques to handle this problem, or apply DPM to MARL algorithms, like MAPPO[1], that can directly learn from individual rewards.
- There has already been a series of work[2,3,4,5] that utilizes LLMs to build dense rewards to solve sparse reward tasks. This paper should introduce these work and include them as baselines.
- StarCraftII is a popular game and SMAC is a popular benchmark. It is likely that LLMs already know the original dense reward function and use it for comparison. However, we should be more interested in whether DPM can solve novel sparse reward MARL tasks. It would be better if the authors discuss about this open problem.
- It seems unnecessary to include both versions of SMAC as benchmark, especially when the earlier version has “deterministic limitations”.
- More experiment details should be highlighted, including baseline introduction, the process of reward model learning, results with different LLMs, and the cost of LLM api for conducting the entire research and a single DPM running.

[1] Chao Yu, et al. The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games.

[2] Tianbao Xie, Siheng Zhao, et al. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning.

[3] Yecheng Jason Ma, et al. Eureka: Human-Level Reward Design via Coding Large Language Models.

[4] Shengjie Sun, Runze Liu, et al. A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning.

[5] Yun Qu, et al. Choices are More Important than Efforts: LLM Enables Efficient Multi-Agent Exploration.
@inproceedings{
anonymous2024dpm,
title={{DPM}: Dual Preferences-based Multi-Agent Reinforcement Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VzuPnoSKQ1},
note={under review}
}
Preference-based Reinforcement Learning (PbRL), which optimizes reward functions using preference feedback, is a promising approach for environments where handcrafted reward modeling is challenging. Especially in sparse-reward environments, feedback-based reward modeling achieves notable performance gains by transforming sparse feedback signals into dense ones.
However, most PbRL research has primarily focused on single-agent environments, with limited attention to multi-agent environments.
In this paper, we propose Dual Preferences-based Multi-Agent Reinforcement Learning (DPM), which extends PbRL to multi-agent tasks by introducing _dual_ preferences comparing not only whole trajectories but also individual agent contributions during transitions. Furthermore, DPM replaces human preferences with those generated by LLMs to train the reward functions. Experimental results in the StarCraft Multi-Agent Challenge (SMAC) and SMACv2 environments demonstrate significant performance improvements over baselines, indicating the efficacy of DPM in optimizing individual reward functions and enhancing performances in sparse reward settings.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
multi-agent reinforcement learning preference-based reinforcement learning RLAIF RLHF
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6b91d97260258cb477fe947d24f6d42a723da52f.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/a2c1fbd24c3b97f462023aef5e17f83989dda7a2.zip
DPM: Dual Preferences-based Multi-Agent Reinforcement Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission