ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
Offline Reinforcement Learning;Multi-Agent Reinforcement Learning;Stationary Distribution Correction Estimation
Yes
No ethics review needed.
The ablation study shows that performance decreases when using a more complex network. Why is that the case? If more data were available or the model was improved, could the results potentially differ?
- Effective Distribution Correction in Multi-Agent Settings: ComaDICE improves upon traditional DICE by extending stationary distribution correction to multi-agent environments. Through f-divergence-based alignment between behavior and target policies, it effectively handles the complex distributional shifts unique to multi-agent interactions, enhancing policy reliability and performance.
- Theoretical Foundation: A thorough mathematical analysis provides convergence and stability proofs through f-divergence correction, reinforcing the algorithm’s reliability in multi-agent scenarios.
- Stable Value Decomposition: ComaDICE decomposes global values into convex local objectives, enhancing training stability and aligning local agent optimization with the global objective to address coordination and stability challenges specific to multi-agent environments.
This paper proposes an algorithm introducing stationary distribution correction to address the distributional shift problem in offline cooperative multi-agent reinforcement learning (MARL). In multi-agent environments, this issue is intensified due to the large joint state-action space and the interdependencies among agents. To tackle this, ComaDICE minimizes the f-divergence between the stationary distributions of the learning and behavior policies. Additionally, by leveraging the Centralized Training with Decentralized Execution (CTDE) framework, it decomposes the global value functions into local values for each agent, ensuring that the optimization of each agent’s local policy is consistent with the global learning objective.
- ComaDICE’s theoretical analysis relies on non-negative weights and convex activations in the mixing network for stability. While this aids convergence, it may limit the model’s ability to capture complex inter-agent dynamics. In the practical algorithm, this limitation is intensified by the use of a single-layer linear mixing network, which further restricts representational capacity in highly interactive environments.
- High Computational Cost: ComaDICE incurs a significant computational cost due to the precise f-divergence-based adjustments needed for each agent’s local policy in stationary distribution correction. This can lead to reduced efficiency, especially in environments with a large number of agents.
Yes
No ethics review needed.
1. What is the main purpose for the value decomposition and learning individual Q-functions? 

1. Related to the first question, what is the purpose of Theorem 4.2, if the original loss is also concave? If we use the mixing functions, does $ \mathcal{\tilde L}$ still find the global optimum?

1. Can ComaDICE solve the XOR game and Bridge mentioned in AlberDICE? 

1. Related to “Strengths”, can solving ComaDICE learn the global optimum in the underlying MDP even with the mixing network?


1. Is the Individual Global Max (IGM) assumption required in order to introduce the mixing network? In other words, does ComaDICE assume that the underlying optimal Q function assume IGM?

1. What is the main difference in the final algorithm with OMIGA [3] ?

1. Why are the notations mixed between using joint observations $\mathbf{o}$ and $s$? Is the setting a Dec-POMDP or a specialized setting where the joint observations constitute the state?

1. Please write in a different color during the rebuttal which part is the novel part. In particular, which part of the proofs are a novel contribution and which part is coming from OptiDICE [2]

1. Please address both my comments in the Strengths and Weaknesses section in detail.


### References
[1] AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation (Matsunaga et, al. NeurIPS 2023)

[2] OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation (Lee et, al. ICML 2021)

[3] Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization (Wang et, al. NeurIPS 2023)
It is known that every MDP has a deterministic optimal policy, which, if extended to the multi-agent setting, can be factorized into independent policies. A very optimistic reading of the paper would interpret that the decomposition procedure is actually learning optimal value functions over decentralized policies. If this interpretation is correct, the ComaDICE presents a very scalable and principled offline MARL algorithm for learning decentralized value functions and policies without any restrictive IGM assumption as in previous work such as ICQ. However, it is worth noting that this is more of a statement of the potential of the paper rather than a strength of its current version, as these points are not specifically addressed in the current draft.

Furthermore, ComaDICE can be more scalable in comparison to AlberDICE [1] which requires alternating optimization.
The paper proposes ComaDICE, a stationary distribution correction estimation approach to addressing OOD states and actions in Offline MARL. The derivation starts with the LP formulation of RL in the joint state-action space, which results in a concave objective for learning state-value functions similar to OptiDICE [2]. The objective also uses value decomposition with a monotonic mixing network for the advantage function similar to OMIGA [3]. ComaDICE is evaluated on SMACv2 and MaMuJoCo and shows comparable performance with baselines such as BC and OMIGA.
### Problem Statement
It is not clear what the main problem is that ComaDICE is solving. It is briefly mentioned in the introduction that OOD states and actions are a problem in offline MARL. However, this was addressed in detail by other work such as CFCQL/OMIGA/AlberDICE mentioned in the Related Work (especially AlberDICE [1] which is the most similar). For instance, AlberDICE considers some coordination problems (XOR/Bridge/etc.) where OOD joint actions may be common, these is no consideration of these settings as well as comparison to AlberDICE both algorithmically and empirically. Thus, it is not entirely clear from the current draft of the paper what the main problem is that ComaDICE is solving, and if it is indeed OOD actions, which part of the algorithm in particular is alleviating this.

### Novelty
The derivation is based on OptiDICE [2] but this is not explicitly mentioned, which is misleading. 
For instance, Proposition 4.1 seems equivalent to Proposition 1 of OptiDICE. Furthermore, an extension of OptiDICE to solve Offline MARL was considered in AlberDICE [1] so it is not clear why ComaDICE should be preferred over AlberDICE. Furthermore, the final algorithm closely resembles OMIGA [3].

### Algorithm
It is unclear what the purpose of value decomposition in learning the Q functions is. It seems the advantage can be computed by the learned state-value function $\nu$ and run WBC (as mentioned in Appendix C of AlberDICE[1]). Also, Line 346 defines $A_\nu^{tot}$ as the sum of reward and state-value functions.

### Lack of Relevant Baselines
AlberDICE and OptiDICE are missing as the main baselines, as well as CFCQL which addresses OOD actions but in a different manner. These are all mentioned in the Related Work section but not compared.

### Writing
The paper is generally not well-written. All of the aforementioned weaknesses of the paper should be addressed in detail and the writing should not raise any of these concerns.
Yes
No ethics review needed.
* The discussion of the DICE method and its goals in the Related Work and Preliminary Knowledge section is not sufficiently in-depth to clearly locate the innovations of this paper with respect to existing work.
* The proof section of DICE skips some key steps,  has a confusingly organized derivation and notation that does not clearly state the purpose and rationale for each step of mathematical transformation. Interpretation between Lagrange functions and offline reinforcement learning problem formulation is in lack.
* The paper fails to explore the common sparse reward scenario in offline reinforcement learning, and the analysis of the quality requirements of behavioral strategies is insufficient, limiting the feasibility of the method in practical applications.
* The introduction of the DICE algorithm within the domain of multi-agent reinforcement learning represents a significant contribution characterized by its novelty and innovative approach.
* The experimental settings are comprehensive, demonstrating the potential of the proposed algorithm effectively.
In this paper, ComaDICE algorithm is proposed for offline multi-agent reinforcement learning, extending the DICE method to multi-agent scenarios and using value function decomposition and policy extraction methods, the idea is novel and the experimental results show the potential of the method.
* The contribution emphasizes the value decomposition and claims to prove the equivalence of the local policy product to the globally optimal policy, but the paper is too repetitive and superficial in its exposition of the decomposition method, neither explicitly defining the local subtasks and their corresponding policies, nor the relationship between them, nor providing a complete proof of this equivalence.
* The experimental results for hybrid networks contradict mainstream research findings (single-layer outperforms double-layers), a phenomenon that hints at a possible fundamental flaw in the application of the DICE methodology to MARL, but the paper lacks an in-depth discussion of this.
Yes
No ethics review needed.
Q1. Do the authors assume that $s$ can be sufficiently represented from $o$?
Notations for states and observation are confusingly used. For example, in line 234 and 237, $\nu(s)$ and $q(s,a)$ are defined as a collection of functions that requires individual observations,$\nu_i(o_i)$ and $q(o_i,a_i)$ respectively If so, the assumption should be explicitly clarified in the POMDP description.

Q2. Figure 2 lacks information on which task was selected for each benchmark. (e.g. 5_vs_5 or 10_vs_10 in protoss, or expert or medium data quality in Hopper) Could you clarify?
This work provides a comprehensive performance evaluation through experiments conducted on an extensive set of benchmarks, including the challenging multi-agent RL benchmark SMAC-v2. The experimental results indicate that the proposed approach shows superior performance relative to the baselines considered in the manuscript.
This work introduces ComaDICE (offline Cooperative MARL with DICE), an approach for offline cooperative multi-agent reinforcement learning (RL) that leverages the DICE method. ComaDICE formulates the offline cooperative multi-agent RL problem as a constrained optimization and employs a DICE-based method to compute a global Lagrangian multiplier, $\nu^{tot}$. Given the large state and action spaces typical in multi-agent RL, practical optimization of $\nu^{tot}$ may not be feasible. To address this challenge, ComaDICE employs value function decomposition to decompose the global Lagrangian multiplier into individual Lagrangian multipliers for each agent.
The primary concern regarding this work is its novelty compared to previous work.

Specifically, AlberDICE by Matsunaga et al. (2023) [A] conveys a similar idea, i.e. adoption of DICE for offline cooperative multi-agent RL. The key difference of ComaDICE from AlberDICE appears to be employing individual Langrangian multipliers $\nu_i$ and the mixing network for value function decomposition: while AlberDICE utilizes a simple yet principled resampling method for obtaining $\nu_i$, ComaDICE employs the value factorization as discussed in Section 4.2, which requires the reliance on the additional mixing network $\mathcal{M}_\theta$ further necessitates additional training.

Although AlberDICE is briefly mentioned in Section 2, the paper does not adequately discuss the theoretical or empirical advantages of adopting the value decomposition instead of the resampling approach. In addition, the AlberDICE paper presents a simple multi-agent task where value function decomposition leads to a substantial performance loss.
 
The absence of such a comparison significantly weakens the perceived contribution of this study, as it fails to establish a clear improvement of differentiation from previous work. In this sense,
(1) A detailed discussion comparing ComaDICE and AlberDICE should be added,
(2) AlberDICE should be included as a baseline in all experiments,
(3) ComaDICE should be tested on the XOR game in the AlberDICE paper  


[A] Matsunaga et al., “AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation.”, NeurIPS 2023.

[Minor comments]
In Sections 6.1 and 6.3, SMACv1 environment is discussed; however, corresponding experimental results for SMACv1 are not included in the main manuscript. It would be more appropriate to relocate these discussions to the appendix and direct readers there for further details, or incorporate the result on SMACv1 to the main manuscript, potentially replacing some of the redundant results on SMACv2 in Table 1 or Figure 1.
This paper introduces ComaDICE, a novel offline cooperative multi-agent reinforcement learning algorithm that uses stationary distribution shift regularization to improve performance in complex environments like MuJoCo and StarCraft II.
@inproceedings{
anonymous2024comadice,
title={Coma{DICE}: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=5o9JJJPPm6},
note={under review}
}
Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we propose a novel type of regularizer in the space of stationary distributions to address the distributional shift more effectively. Our algorithm, ComaDICE, provides a principled framework for offline cooperative MARL to correct the stationary distribution of the global policy, which is then leveraged to derive local policies for individual agents. Through extensive experiments on the offline multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Offline Reinforcement Learning Multi-Agent Reinforcement Learning Stationary Distribution Correction Estimation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5e0515a3b8ce78c6b761966cb8e5b5999d21c4d1.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/9b2a5d631dd353b4c5090825b42756d3eea17754.zip
ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission