Advantage Alignment Algorithms
Multi-agent Reinforcement Learning;Opponent Shaping;Social Dilemmas;General-Sum Games
We wanted to thank the reviewer for their feedback and for their appreciation of our paper. We also believe that the quality of the paper is good. Having said that, we now address the reviewer’s technical concerns:

1. There is indeed a $\beta$ factor missing here, and we will correct this, thank you.
2. To proceed from equation 15 to equation 16, we simply use the policy gradient theorem, in particular the Advantage form of the policy gradient which is detailed in [1]. We will add this clarification on our derivation as well.
3. In general if we have a double sum $\sum_{i=0} ( \sum_{j=i+1} a_i b_j )$, this is equivalent to $\sum_{j=0} \sum_{i<j} a_i b_j = \sum_{j=0} b_j \sum_{i<j} a_i$, this is because we are essentially summing over all tuples of natural numbers $(i,j)$ for which $j>i$. Now setting $a_i = \gamma^i A_i^i$ and $b_j = \gamma^{1+j} A^2_j \nabla_{\theta_1} \log \pi^1_j$ gives us the desired result.
4. We expand the steps as follows: $$
\begin{aligned} \nabla_{\theta}V(\theta) &= E_{\tau \sim \pi} \sum_{t=0}^{\infty}\gamma^t A(a_t|s_t) \nabla_\theta \log \pi_{\theta}(a_t|s_t)\\\\
&= \sum_{t=0}^{\infty} \gamma^t E_{\tau \sim \pi} [A(a_t|s_t) \nabla_\theta \log \pi_{\theta}(a_t|s_t)]\\\\
&= \sum_{t=0}^{\infty} \gamma^t \sum_{s_t, a_t} P(a_t,s_t) A(a_t|s_t) \nabla_\theta \log \pi_{\theta}(a_t|s_t) \\\\
&= \sum_{t=0}^{\infty} \gamma^t \sum_{s_t} P(s_t) \sum_{a_t} \pi_{\theta}(a_t|s_t) A(a_t|s_t) \nabla_\theta \log \pi_{\theta}(a_t|s_t) \\\\
&= \sum_{t=0}^{\infty} \gamma^t \sum_s P(s_t=s) \sum_a \pi_{\theta}(a|s) A(a|s) \nabla_\theta \log \pi_{\theta}(a|s) \\\\
&= \sum_s \left(\sum_{t=0}^{\infty} \gamma^t P(s_t=s)\right) \sum_a \pi_{\theta}(a|s) A(a|s) \nabla_\theta \log \pi_{\theta}(a|s) \\\\
&= \sum_s d_{\gamma}(s) \sum_a \pi_{\theta}(a|s) A(a|s) \nabla_\theta \log \pi_{\theta}(a|s) \\\\
&= \sum_{(s,a)} d_{\gamma}(s) \pi_{\theta}(a|s) A(a|s)\nabla_\theta \log \pi_\theta(a|s)
\end{aligned}
$$ Where from equation 2 to 3 we used the fact that for any function of a state-action pair \( f(a, s) \), we have: $$
\begin{aligned}
E_{\tau \sim \pi} f(a_t, s_t) &= \sum_\tau P(\tau) f(a_t, s_t) \\\\
&= \sum_{a_t, s_t}  f(a_t, s_t) \sum_{t_1 \neq t} \sum_{t_2 \neq t} \dots P(\tau) \\\\
&= \sum_{a_t, s_t} P(a_t, s_t) f(a_t, s_t)
\end{aligned}
$$where in the last step we have just marginalised $P(\tau)$ over all time-steps except for $t$, setting $f(a_t, s_t) = \nabla_\theta \log \pi_{\theta}(a_t|s_t)$ gives the intended result.
5. Assuming orthonormality simply means that $\nabla_{\theta} \log \pi_\theta(a|s) \cdot \nabla_{\theta} \log \pi_\theta(a'|s') = 0$ for all $(a,s) \neq (a', s')$, hence the sum over all state-action pairs collapses to a single term, which contains the $(a', s')$ pair. The gradient dot product for that pair is assumed to be $1$ again by the orthonormality assumption. We will note that this orthonormality assumption is exactly true in the tabular case. i.e. if the parameters $\theta_i$ of our policy are exactly $\log \pi(a|s)$ for all possible $(a,s)$, then this assumption is exact.
6. Here $A^*_{LOLA}(s_t, a_t, b_t)$ is the new advantage that should be put into the policy gradient theorem in order to get the update direction for a LOLA agent. It's not equal to $V^1$.
7. This is the assumption that LOLA makes about the opponent, it is true by hypothesis. LOLA takes $\Delta \theta_2 = \alpha\nabla_{\theta_2} V^2$ because it assumes that the opponent is a naive learner (i.e. doesn't incorporate opponent-shaping information), and hence would take a gradient step by greedily maximizing its value function.
8. Thank you, we will fix it.
9. In figure 1b, the state CD corresponds to a previous state which is "player 1 cooperated, and player 2 defected", DC is the reverse, "player 1 defected, and player 2 cooperated". This figure illustrates that our algorithm finds the tit-for-tat policy, which cooperates if the opponent cooperated in the last turn, but defects otherwise. This asymmetry between CD and DC is very important, cooperating in both states means that our agent doesn't punish opponent defection.

We hope that our responses have satisfactorily addressed the reviewer's concerns, and we appreciate your positive comments regarding the originality, quality, clarity, and significance of our work. In this light we kindly ask the reviewer to reconsider their score.

**References**

[1] Agarwal, A., Jiang, N., Kakade, S., and Sun, W. (2021). Reinforcement Learning: Theory and Algorithms. Preprint.
Addressing technical concerns
Yes
No ethics review needed.
Additional clarification questions:
- Fig6 seems to indicate that Advantage Alignment is not really stable (increasingly noisy rewards, partial collapse towards the end of training against AC and AD). Is it sensible to hyperparameters? Or is there another reason for this behavior?
- Fig4 selects the best agent out of 10 seeds. Given the variance observed for the Negotiation Game in Fig6 (which granted is not the same game as Fig4), how representative are the results in Fig4?
- I am curious how Advantage Alignment can play against AD or AC if it can't change its policy (AD and AC are fixed policies). Doesn't this break Assumption 1?


Overall, I recommend for acceptance, for the strengths listed above (theoretically sound, well motivated, experimentally justified).
Strengths:
- This paper follows a long line of work on opponent shaping (LOLA, POLA, COLA, LOQA) and builds on LOQA to propose a new opponent shaping algorithm. While it feels a bit incremental, I find the work very well motivated, and theoretically justified. Extensive experiments on different domains and with relevant baselines confirm its practical performance.
This work aims to encourage cooperation in normal form games through opponent shaping. This is done by proposing Advantage Alignment, where the product of the advantages of the player's value-network and the opponent's value-network are used to update the policy. This means that, when advantages are aligned (i.e., they have the same sign), the log probabilities increase or decrease accordingly. The authors show theoretically that LOLA and LOQA, 2 other opponent shaping algorithms, also follow the advantage alignment principles. They then experimentally validate advantage alignment on Coin Game, Negotiation Game, and the high-dimensional Commons Harvest Open environment.
Concerns:
- Having access to the opponent's value function results in a specific setting where all the player's preferences are public. The negotiation game completely changes if we know the preferences of the opponent, and we could devise a simple strategy that maximizes the average utility of both players. Since the utilities in this experiment are orthogonal to each other, there does not seem to be any real dilemma. For these reasons, I believe the insights gained from the Negotiation Game experiments to be limited.
- Advantage Alignment uses the product of advantages to align the policy. When there are more than 2 players involved (as in Commons Harvest Open), does the alignment depend on the product of the advantages of all players? If so, if a single agent does not cooperate, the whole alignment product fails. This would limit the scaling of Advantage Alignment to $n$ players.
Yes
No ethics review needed.
- For the coin game, in the always defect case, LOQA receives slightly higher return than Advantage Alignment. Do you have any ideas for why this is?
-  In figure 3, there is mention of green values to show cooperation, but I do not see the green values in the figure.
- In Appendix 6, you mention that equation 59 uses the sum of past advantages of the agent up to the current time step and the advantage of the opponent at the current time step. Did you experiment with much tuning for number of terms to include in the sum of past advantages or number of future terms?
This paper is very well-written, and I appreciated how the authors helped the reader build intuition and understanding of the significance of their technique in section 4. The experiments served to reiterate the strength of their algorithm's performance, and the authors gave solid context for why each environment was selected.
This paper focuses on opponent shaping.  They propose an opponent shaping formulation that aligns the advantages of interacting agents, thereby simplifying the mathematical formulation of opponent shaping and reducing the associated computational complexity. In this formulation, the probability of future mutually beneficial actions is increased when their interaction has been positive. They demonstrate their methodologies' effectiveness on several social dilemmas and connect opponent shaping's past successes with this new advantage alignment formulation.
The main concern I have with the paper is its individuality from the LOQA work, which concerns a similar technique, applied on similar problems, that achieves similar results. Both the more complex experiments (negociation and and harvest open) and the attached proofs in the appendix helped differentiate some of the beneficial aspects of Advantage Alignment in terms of scalability.
Yes
N/A
No ethics review needed.
See concerns in weaknesses. If the these concerns can be resolved, I will consider to raise the score. However, for the moment I can only give a reject due to those concerns.
1. This paper proposed an original idea that derives the advantage alignment to implement the policy gradient with respect to the opponent (the opponent shaping term), which can reduce the computational complexity. This paradigm has been shown to have connections to previous opponent shaping approaches, which is a progress in this research direction.
2. The general quality of this paper is good. Although there are some technical points that I require some furtehr clarifications, most of proofs are correct to my best knowledge. The theorem proofs are actually dependent on assumptions raised in this paper, and assumptions are provided with explanations. The experiments were conducted on sufficient banchmarks, in comparison with multiple baselines. Furthermore, the results not only include the numeric results, but also involve some demonstrations of test case, which make the paper more comprehensive. 
3. This paper is well written. It has a clear motivation in Introduction and a thorough introduction of social delimma and opponent shaping. For those people who are not doing research in this direction, it is still friendly enough for them to catch up. 
4. As for the significance of the research problem, social delimma, it is surely a significant problem. The main reason from my own perspective is that it can simulate a class of social problems. About the opponent shaping direction, the significance to me is sceptical, since the requirement of opponent's knowledge seems like a bit strong assumption. However, I do not deny that this is a necessary step towards the weaker and more realistic assumptions. As a result, this paper is significant within the resesrch community of opponent shaping.
This paper investigated a long-standing research problem in multi-agent systems and game theory, named social dilemma. To address this problem, this paper stood on the side of opponent shaping and proposed a method that attempted to align advantages of various agents, to achieve a compromise between social welfare and self-interests. The main assumption lies in the connection between the opponent's policy gradient and the agent's policy parameters. Building on this, this paper proposed a paradigm called Advantage Alignment, and a corresponding RL algorithm based on PPO. This paper also unified other approaches of opponent shaping, such as LOLA and LOQA, in the paradigm of Advantage Alignment. The experimental results on various tasks showed the effectiveness of the proposed Advantage Alignment.
I have some technical concerns about this paper, which are specifically listed as follows:
1. In line 720, could you explain why the $\beta$ term is missing?
2. In line 739, could you give more details about how equation (16) is derived from equation (15)?
3. In line 755, could you give more details about how to transform from (19) to (8), step by step?
4. In line 773, could you give more details about how equation (24) is derived from equation (23)?
5. In line 783, even with the assumption of orthonormal gradients, it is still not clear why equation (27) can be derived from equation (26). Could you please clarify this step by step?
6. In line 792, is $A^{*}_{\text{LOLA}}$ equal to $V^{1}$?
7. In line 809, could you explain why $\alpha \nabla_{\theta_{2}} V^{2}(\theta_{1}, \theta_{2})$ is equal to $\Delta \theta_{2}$?
8. In line 1001, there is a typo: equation equation -> equation.

Additionally, I also have some concerns about the experimental analysis:

9. In the results shown in Figure 1b, could you give some intuitive interpretation on the asymmetry between the results of CD and DC? For example, is it related to your theoretical claims?
Yes
No ethics review needed.
- Could the authors explain how LOLA and LOQA and AdAlign are related? I see the derivation in Theorem 1 and 2 but do not understand the implication and how they differ in terms of their training objectives.
- It is not clear why increasing the log prob when both agents' advantages are negative is desirable (Eq. 8 and Fig. 1a). Can the authors elaborate on this?
- The proposed derivation is novel, and provides novel insight on opponent shaping methods
- The paper is theoretically grounded and well motivated.
- The resulting derivation unifies prior works nicely (Theorem 1 and 2).
- Broad suite of baselines and evaluation environments.
The paper proposes a novel method for opponent shaping based on the advantage functions of the interacting agents. The derivation unifies prior works through the lens of advantage alignment. Empirically, the proposed method is effective in learning complex coordination behaviors under various multi-agent environments.
- I wish Section 2 and 3 are more detailed, and cover more technical background on prior works. The additional information would allow the reader appreciate the contribution much better.
- The evaluation protocol is not provided. For readers that have little background in opponent shaping, it is not clear if the agent is trained prior to the evaluation or being trained (and thus adapted) during the evaluation episodes.
- It is not clear why increasing the log prob when both agents' advantages are negative is desirable (Eq. 8 and Fig. 1a).
- To my understanding, the proposed method does not outperform the baselines LOQA algorithm. The authors should discuss the differences and why it is desirable to use the proposed method over LOQA.
- The paper claims that the proposed method is more efficient than the baseline but does not elaborate on this aspect.

Minor comments:
- The advantage function is not defined mathematically.
- I believe Eq.2 is not the right expression, though I think this does not affect the main derivation of the paper.
- The connection between Eq.5 and Eq.6 is not explained.
We introduce Advantage Alignment, a new family of algorithms for opponent shaping in general-sum games, designed to promote cooperation and avoid suboptimal outcomes.
@inproceedings{
anonymous2024advantage,
title={Advantage Alignment Algorithms},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=QFO1asgas2},
note={under review}
}
Artificially intelligent agents are increasingly being integrated into human decision-making: from large language model (LLM) assistants to autonomous vehicles. These systems often optimize their individual objective, leading to conflicts, particularly in general-sum games where naive reinforcement learning agents empirically converge to Pareto-suboptimal Nash equilibria. To address this issue, opponent shaping has emerged as a paradigm for finding socially beneficial equilibria in general-sum games. In this work, we introduce Advantage Alignment, a family of algorithms derived from first principles that perform opponent shaping efficiently and intuitively. We achieve this by aligning the advantages of interacting agents, increasing the probability of mutually beneficial actions when their interaction has been positive. We prove that existing opponent shaping methods implicitly perform Advantage Alignment. Compared to these methods, Advantage Alignment simplifies the mathematical formulation of opponent shaping, reduces the computational burden and extends to continuous action domains. We demonstrate the effectiveness of our algorithms across a range of social dilemmas, achieving state-of-the-art cooperation and robustness against exploitation.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multi-agent Reinforcement Learning Opponent Shaping Social Dilemmas General-Sum Games
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c8657cf8f1251022b24f4a6e447c8026251016f3.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/2538d619820ba8b2cf347732f1a8e42f24a538fa.pdf
Advantage Alignment Algorithms
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission